Debugging C Programs with GDB – Part 1

When you write C code, you’re playing with power! You’re bound to let this power go to your head and shoot yourself in the foot here and there. At some point(s) your program is going to do something that just doesn’t quite make sense.

The bad news is that your program doesn’t make any sense because you’ve written flaws into it. That’s fine, you’ve either written janky C programs, or not written any C. The good news is that GDB is here to help us learn from our mistakes!

Through the next few posts I’ll share some tips on basic GDB usage, explore a bit of history and dig more into how the C programs on my machine are actually working.

Building for Debugging

To kick things off, I’m going to just slap together a quick C program and a Makefile to assist in building it and running my debugger.

// test.c
#include <stdio.h>

char *done = "Done!";

int main(int argc, char *argv[]) {
  int i;

  for (i = 0; i < 10; i++) {
    printf("Iteration %d\n", i);
  printf("%s\n", done);

  return 0;

This program has a simple for loop and a few print statements and I’ll use GDB to inspect what it’s doing a bit more. To provide more information to the debugger about this program I’ll use the -g flag when building it.

# Makefile
CC=gcc -g -o $@ -Wall $<

all: test

test: test.c

debug: test
  gdb -q ./test

For maximum laziness, I added a debug target to my Makefile here so that I can use make debug to jump right it. I gave gdb the -q option to quiet down since it normally has a lot to say on startup.

That’s about all I need to get my program ready for debugging!

Basic Commands

Now we get to the hard part. GDB has a bajillion features so getting started can be daunting. Probably one of the best commands to learn first is the run command, as so far the program has been looked at a little bit, but isn’t actually running at the moment.

You can also provide arguments to the program by providing arguments to run. This program doesn’t care about arguments, but don’t let that stop you from giving it some anyway!

The excitement of just running a program in GDB is very short lived, I want to be able to stop the program somewhere and poke around a bit. The list command can spit out a listing of the program.

Initially gdb will show the first 10 lines of the source. You could run list again to see the next 10 lines but GDB has a friendly feature where hitting enter will automatically rerun your last command, so I used that to continue reading the full source.

Looking at this listing, I think a good place to pause and look around would be at the printf() call within my for loop. To have GDB stop here I’ll use the break command and I’ll give it the argument 10 to indicate I’d like to set a breakpoint at line 10.

Now when I give it a run, it’ll stop the program when it hits that line.

To resume the program, until the next breakpoint is hit, you can use the continue command. Another little time-saver trick with gdb is that many commands have shortcuts, such as c for continue.

Peeking Into The Code

The ability to set breakpoints and resume execution is a good start, but even better is getting a look around at this point in time to glean more about what the program is doing. It’s time to start looking beyond the C code and see what the program is actually doing in assembly, the state of the CPU in the context of our program and what’s going on in memory.

First let’s look at the assembly version of the main function. I’ll use the disassemble command for that, and I’ll tell it that main is what I’m interested in disassembling.

Oh noes! Assembly!

Assembly code get’s a bad rep, but it’s not as bad as people think it is. You might not want to write a large application in assembly, and that’s reasonable, but if you want to be a strong C programmer you need to know enough assembly to figure out what your program is up to.

x86_64 assembly has two different syntaxes to choose from, AT&T syntax and Intel syntax. They both work just fine but GDB defaults to AT&T syntax and I prefer the Intel syntax so I’ll use the command set disassembly-flavor intel to get it to my liking.

That looks better! Now let’s briefly look at a few things. Looks like my main function is 21 instructions long, alright… a smidge more than half of the operations are mov (move) instructions and I see a few branching operations, jmp (jump), call (call a subroutine), jle (jump if less than or equal to) and ret (return from subroutine).

One thing I find interesting is the instruction at offset <+64>, call 0x400430 <puts@plt>. I did not use the puts() function in my code! The compiler caught on that my last printf() statement doesn’t need to be a format string and optimized the result a little bit.

Let’s get back to inspecting what this program is up to, I’m currently still in the middle of my paused program, and I’m at the very start of one of my loop iterations. In this disassembly output I can see I’m at offset <+24>, as indicated by the little => arrow, this is the next instruction the program will run.

The mov instruction moves a value from one place to another, similar to the assignment operator  = in most programming languages. In this case the full instruction is mov eax,DWORD PTR [rbp-0x4] which is basically eax = DWORD PTR [rbp - 0x4]. Ignoring the right side of that for now, we’re assigning a value to something called eax. This eax thing is a CPU register, which is basically a variable in the hardware of the CPU. We can look at all the registers with the info command by saying info registers.

Okay so there are a bunch of registers, and eax is not one of them… GREAT! This is because the x86 architecture has been through a lot, way back in the day (early 70s) Intel released their 8008 CPU that had some 8-bit registers with names like A (for Accumulator).

When Intel got to the 8086 in the late 70s they made the A register twice the size (16-bits) and started calling it the AX register. To help with software compatibility with older system the AX register could be used as an 8-bit register with AH representing the higher 8 bits and AL the lower 8 bits.

Then the mid-80s showed up and Intel was like MOAR BITS and released their 80386 that had 32-bit registers, now they refer to the A register as EAX (there’s our guy!), again preserving backward compatibility by allowing the 16 and 8 bit registers to remain the same. Now-a-days our 64-bit processors are king, so we have the 64-bit register RAX, but can still use EAX, AX, AH, and AL.

All that history lesson to give full context on why mov eax, <stuff>  is going to modify our rax register!

Now, to run just that one instruction, I’ll use the nexti command. I’ll then check the registers again with the shorthand version of info registers and just look at the eax register: i r eax

If I continue my program, I’ll notice that this number correlates with something in my program.

The eax register is getting set to the i value I’m setting during my for loop!

In the next post I’ll continue digging into this program and discover more about the disassembled version of my C program and show off some more GDB commands along the way!

Exploring ELF

Hello there! This post is going to be about the Executable and Linkable Format (ELF).  This is one of the most important binary formats out there and is designed to store object code in a way that can be used on a wide variety of processor types and operating systems. It’s the most popular format used to contain compiled programs and libraries.

Let’s take a file apart and look at what it’s made of!

ELF Header of a C Program

To kick things off I will build a very small C program, test.c

int main(int argc, char *argv[]) {
  return 0;

I’ll give it a quick and lazy build with make test and look at the first 64 bytes of it with hexdump.

We can see right away this is an ELF file, the first 4 bytes are "\x7fELF".  The ELF format supports 32-bit and 64-bit machines. This header will be 52 bytes long on a 32-bit machine and 64 bytes long on a 64-bit machine to support the longer memory addresses. For this post I’ll only be looking at 64-bit files.

Here’s the structure of the 64-bit header from the ELF-64 documentation:

typedef struct {
  unsigned char e_ident[16]; /*  ELF  identification  */
  Elf64_Half e_type;         /*  Object  file  type  */
  Elf64_Half e_machine;      /*  Machine  type  */
  Elf64_Word e_version;      /*  Object  file  version  */
  Elf64_Addr e_entry;        /*  Entry  point  address  */
  Elf64_Off e_phoff;         /*  Program  header  offset  */
  Elf64_Off e_shoff;         /*  Section  header  offset  */
  Elf64_Word e_flags;        /*  Processor-specific  flags  */
  Elf64_Half e_ehsize;       /*  ELF  header  size  */
  Elf64_Half e_phentsize;    /*  Size  of  program  header  entry  */
  Elf64_Half e_phnum;        /*  Number  of  program  header  entries  */
  Elf64_Half e_shentsize;    /*  Size  of  section  header  entry  */
  Elf64_Half e_shnum;        /*  Number  of  section  header  entries  */
  Elf64_Half e_shstrndx;     /*  Section  name  string  table  index  */
} Elf64_Ehdr

A visual representation of this structure:

The e_ident field contains all the information needed to understand what format and version of an ELF file this is. As I mentioned before, the first 4 bytes of this should be \x7f, E, L, F, the magic number for all ELF files. The remaining bytes of the e_ident field will indicate if the file is 32 or 64-bit, if the file is in little endian or big endian format and a few other details about what the file was built for.

The e_type field indicates what type of data is represented in the file, if it’s a object (1), executable (2), shared object (3) or core file (4). The e_machine field represents what instruction set the file is in. e_version is always 1 for now.

Up this this point the header is the same for 32-bit and 64-bit machines. The next three fields e_entry (program entry point), e_phoff (program header offset) and e_shoff (section header offset) are used to refer to memory addresses so are twice as long on a 64-bit machine. The e_entry field is a pointer to the start of program in virtual memory. If you’re unfamiliar with what virtual memory is I suggest this video from Android Authority’s YouTube channel.  The e_phoff and e_shoff indicate the offsets within the ELF file to the program and section headers tables.

The rest of the header contains some flags, counts and sizes relating to the program and section headers in the file, I suggest checking out the elf header documentation on Wikipedia for more detail on the header.

We can easily see all of this header detail in a human readable way with the readelf program.

A Simpler ELF File

I want to dive a bit deeper into the file, but even an empty C program like this contains a ton of data compared to a small Hello, world! program written in assembly. Here’s my hello world assembly source for a x86_64 Linux machine:

section .text

  global _start


  ; write(1, msg, len)
  mov rax, 1
  mov rdi, 1
  mov rsi, msg
  mov rdx, len

  ; exit(0)
  mov rax, 60
  mov rdi, 0

section .data

msg db "Hello, world!",0xa
len equ $ - msg

I won’t dive into the syntax of this assembly, but it makes one call to Linux to write Hello, world!\n to stdout (1) and  a second call to exit the program with 0. I’ll assemble it into an elf64 object file with nasm -f elf64 hello.asm and link it into an executable with ld -s -o hello hello.o.

This file is a lot smaller than the C program, 512 bytes vs 8552. There is also a lot less data in the ELF program and sections headers as indicated by the  header.

There are 2 program headers instead of 9 and 4 sections headers instead of 31, so much less to look at!

ELF Sections

Let’s look at more of what readelf can tell us about this ELF file, starting with closer inspection of the sections headers.

$ readelf -a hello
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x4000b0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          256 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         2
  Size of section headers:           64 (bytes)
  Number of section headers:         4
  Section header string table index: 3

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         00000000004000b0  000000b0
       0000000000000027  0000000000000000  AX       0     0     16
  [ 2] .data             PROGBITS         00000000006000d8  000000d8
       000000000000000e  0000000000000000  WA       0     0     4
  [ 3] .shstrtab         STRTAB           0000000000000000  000000e6
       0000000000000017  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), l (large)
  I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)

There are no section groups in this file.

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x00000000000000d7 0x00000000000000d7  R E    200000
  LOAD           0x00000000000000d8 0x00000000006000d8 0x00000000006000d8
                 0x000000000000000e 0x000000000000000e  RW     200000

 Section to Segment mapping:
  Segment Sections...
   00     .text 
   01     .data 

There is no dynamic section in this file.

There are no relocations in this file.

The decoding of unwind sections for machine type Advanced Micro Devices X86-64 is not currently supported.

No version information found in this file.

Looking first at the section table, there are 4 section entries. The sections headers have the following structure:

typedef struct {
  uint32_t   sh_name;
  uint32_t   sh_type;
  uint32_t   sh_flags;
  Elf32_Addr sh_addr;
  Elf32_Off  sh_offset;
  uint32_t   sh_size;
  uint32_t   sh_link;
  uint32_t   sh_info;
  uint32_t   sh_addralign;
  uint32_t   sh_entsize;
} Elf32_Shdr;

Precise detail on this structure, and what the fields are for can be found pretty easily in the Linux man page for elf. Going on the output of readelf, the first section is is a NULL section. I don’t quite understand why this first section is empty, but it is part of the ELF standard.

The next two sections are  .text and .data, which were defined in the assembly code. These are both of type PROGBITS which means all the data in the section is defined and formatted for the programs own usage, it is a chunk of arbitrary bytes as far as ELF is concerned.

The address listed is where the section should be loaded in the program’s virtual memory space when the program is loaded by the operating system, if that section should be loaded into memory space of the process. The offset indicates how many bytes into the ELF file is the start of this section and size is the length of that section. So we can see here that .text will be loaded at 0x4000b0 for the process, which is 0xb0 bytes into the ELF binary file on the filesystem and that chunk of data is 0x27 bytes long.

The flags tells us a few other bits about the section, both .text and .data have the allocate flag, so it should reside in the programs memory space when the program is executed. The .text section is the only section marked as executable, so the system should only allow code execution to occur within that section. The .data section is the only section with the writable flag, so the memory in that range can be modified by the running program.

The program headers are pretty similar to the section headers, they describe information the system needs to run the program. They describe program segments and generally point to the same regions of memory and the binary file as the sections do, but with some detail that’s specific to executable and shared objects. I’ll defer again to the manual for all the raw details.

That covers my introduction to the ELF format. Hopefully this provides good enough context to explore a bit more on your own. I suggest trying to see if you can build an ELF file from scratch or modify one that is already built for some binary hacking shenanigans! In either case, I hope you enjoyed the read and would value any feedback or question you leave in the comments.

Barebones Linux ISO

My post on Building a Barebones Linux System has been the most popular post on my blog so far. Last week a reader left a comment asking how the system described in that post could be loaded onto an ISO file along with GRUB to make it into a bootable image. I think that’s a pretty good follow-up so today I will share the process on how to do just that!

Continuing the Barebones Journey

It’s been a bit since I posted the original Barebones Linux post and in  Getting Busybox With It I added BusyBox to the system and structured my build into a Makefile to preserve that effort (available on my Github). That is what I’ll use as the starting point for this post.

Back then I used the Linux Kernel version 4.13.3 and BusyBox version 1.27.2, while I’m pulling this up again I’ll change a few of my Makefile variables to use the latest versions for today, Linux 4.15 and BusyBox 1.28.0.  After updating my versions I let the Makefile run and realized I needed to update my BusyBox config. I opted for the changes that the BusyBox build system suggested and gave it another test in QEMU to make sure it was all good to go.

Now to build an ISO for it! I looked to see what resources I could find on ow to get that done. I saw what had to offer and found a lot of good info on how grub-mkresue works here. I took a peek at the grub2 manual and it looks easier than I expected it would be.

Creating a GRUB ISO

As an initial test I wanted to verify that a normal grub2 rescue image would book in QEMU. I used grub-mkrescue -o test.iso to build this image.

Then to test it I ran qemu-system-x86_64 -m 2048 -cdrom test.iso -boot d.

Easy enough. There is no grub.cfg so it gives you this basic rescue shell, but I’m glad it works. The grub documentation and examples I found show that you can give grub-mkrescue a directory that it will include in the build ISO and I decided to give that a shot.

I returned to my barebones repo directory and created a directory iso to store my files that I’d like to go in the ISO, and the boot/grub directory within that to store my grub config.

I copied my built vmlinuz and initramfs files to iso/boot/ and wrote the smallest grub.cfg to test it.

menuentry 'barebones' {
  linux	/boot/vmlinuz
  initrd /boot/initramfs

With all the files where they should be I run grub-mkrescue again to make the build.

Then I’ll give my barebones.iso a test in QEMU and there’s my menu entry:

Selected my barebones OS and let it do it’s thing:

Huzzah! IT’S ALIVE!

I thought there was going to be a bit more to this but I was pleasantly surprised!