Debugging C Programs with GDB – Part 2

In my previous post I covered a few basics around building a C program for debugging, looking at the code listing and assembly of the program, setting breakpoints and peeking into the registers.

For Part 2, I’ll be using more gdb commands to explore the assembly code that’s involved in building up and initializing the stack frame at the start of a function.

Stacking Up

I want to dig down into every bit of what this program is doing, I’ll begin by pulling up the disassembly version of the program once more.

If I use break main to set a breakpoint at the start of my main function, GDB will be practical and set this breakpoint at the address 0x400575, 15 bytes into the assembled version of main. It’ll do this because the first 6 instructions of the function are setting up the stack for the function and normally you can trust that the compiler has done a good job handling that for you.

I want to really start at the beginning, so I’m going to instead set my breakpoint with a pointer to the first byte of the function with the command break

The first assembly instruction seen here is push rbp. This is pushing the value stored in the rbp (base pointer) register onto the stack, but where is the stack? The rsp (stack pointer) register tell us where the top of the stack currently is.

These first 3 instructions are building a new call stack frame. Since this program uses libc the program starts by running code to initialize a few things then calls the main function that was created in the source code. The compiler adds these instructions to build out the stack frame for the given function.

The stack when printf is being called from main

As mentioned before, the rsp register points to the points to the top of the stack. By running nexti (next instruction), I can execute one machine operation at a time. After letting push rbp run, the current value of rbp is added to the top of the stack. While this happens, the stack grows and rsp will be updated to the new top of the stack.

The value of rbp hasn’t changed, but the value of rsp is 8 bytes smaller. In most CPU architectures the stack grows “down” like this, where the lowest call frame represents the current frame.

Peeking Into Memory

We should now be able to see the rbp value at the top of the frame. By using the x command to print data in memory, we can take a look at the frame. x has a few useful options, while you’re learning GDB they can be tricky to remember so I recommend keeping a cheat sheet handy until you’ve got it down.

To look at the value at the top of the stack, I’ll use x/1xg 0x7fffffffda10. This is asking to see 1 unit of data, in hexadecimal format, considering 64-bit “giant” words. You can choose instead to dereference the rsp register directly in the command by using $rsp for the address.

The value of rbp is indeed at the top of the stack

With the commands covered so far we can more easily inspect what each instruction is doing. The next instruction is mov rbp, rsp, this moves the value of rsp into rbp. This is to set the base pointer to the top of the previous stack frame which is used at the end of a function to restore the previous stack state.

Now rbp and rsp are set to the same value

The next instruction, sub rsp,0x20, lowers the value of rsp by 32 (4 64-bit words). This is the size of the frame being built for main. Using x again, I’ll look at the 4 words in the stack, plus the next word after (the rbp value that was pushed to the stack).

There’s already data in this stack! This is data left over from previous execution and is considered uninitialized, as this function doesn’t know what data was been left behind here. When gcc warns that 'i' is used uninitialized in this function [-Wuninitialized], it is because the program is using a variable that’s not in a known state.

Pointers IRL

The next 3 instructions are going to be initializing the new stack frame region using pointers. If you’ve been confused about pointers in C, pointers in assembly might help demystify that concept for you.  Let’s consider the next 3 instructions:

=> 0x000000000040056e <+8>:	mov    DWORD PTR [rbp-0x14],edi
   0x0000000000400571 <+11>:	mov    QWORD PTR [rbp-0x20],rsi
   0x0000000000400575 <+15>:	mov    DWORD PTR [rbp-0x4],0x0

These are assignment operators that refer to addresses relative to a memory address that is stored in a register. In this case they are modifying the data in the new stack frame (addresses below rbp). Sometime before main was called, the edi and rsi registers were set to some values that the compiler wants kept around.

The first instruction here is setting a 32-bit (Double Word) at the memory address 20 bytes below rbp. I’ll need to give x a w parameter so that it’ll know I’m now looking for a 32-bit word.

Basically the same thing going on for the next instruction, but setting the memory address 32 bytes below the base pointer to the 64-bit value currently in rsi.

These registers (*di, *si) are index registers that can be used for various string operations. They are also part of the x86_64 calling convention to be used when calling functions. In this case, these registers contain the parameters for main, my argc and argv variables.

We can use the print command to look at the variables as they were named in the source code. The & operator can be used similar to how it used in C to verify that these variables are stored at those locations in the stack.

The next instruction, mov DWORD PTR [rbp-0x4],0x0, is the i variable used within the for loop inside of main, here being initialized to the value we specified, 0.

At this point the stack for main is initialized and main is ready to do its thing. In the next post I’ll continue the GDB exploration to investigate the rest of what this function is up to!

Running Nomad on ppc64le

Recently I’ve been doing some experimentation with Nomad, a tool that helps manage applications running on a cluster of machines. I first ran through their getting started guide, but wanted to continue my education by deploying it on some of my Barreleye systems to see if it could be used for benchmarking and other lab workloads.

I quickly found that there’s no official support for ppc architectures, but since this isn’t a production environment I wasn’t going to let that stop me.

Note on PPC64 vs PPC64LE

Before POWER8, the POWER architecture was exclusively big-endian. This became a barrier for users as many projects were not designed to consider the endianness of the processor and some applications and libraries would not run properly. There’s an assumption the system is little-endian (since x86 dominates the server and desktop market) and this can be fairly time consuming to address.

IBM responded to customer feedback about this, and introduced bi-endian support in POWER8 to allow the user to decide if they want their OS and applications to use little or big endianness. Since the release of POWER8, the Linux community has mostly moved over to ppc64le, as bugs related to processor endianness are no longer a factor.

For the most part, building code for ppc64le is typically identical to building code for x86_64!

But First, Go!

As Nomad does not officially support ppc, I’ll need to build it from source. The README clearly states that we’ll need Go version 1.9 or newer.

The system I’m working on is kicked with Ubuntu 16.04.4 LTS, and if you bring in the golang-go it will install go 1.6. Alternatively, there is a golang-1.9 package that’ll slap 1.9.2 on your system.

In my case though I’m going to manually install the latest stable version, 1.10, since I know there’s been good work going on to improve the Go assembler and performance for PowerPC. Besides, manually installing Go is easy as pie.

To install go I’ll download the latest built archive for ppc64le, extract it to /usr/local/ and append my PATH environment variable for all users by editing my /etc/profile file. Then I’ll create a go directory in my home directory and update my own .bashrc file to set my GOPATH variable to its path, while also appending $GOPATH/bin to my users PATH variable.

Off the Beaten Path

Next I’ll follow the Installing Nomad documentation on installing via source to see how far I get.

So the only hangup here is there’s no build target in the Makefile for ppc64le. The amd64 target is a pretty close guess to it, so I’ll duplicate that target and make some minor modifications to it.

There it is! I’m still a bit of a noob with Nomad and I’ve only run very basic workloads with it, but everything is working as expected so far. I look forward to doing a bit more tinkering with it in the near future.

Debugging C Programs with GDB – Part 1

When you write C code, you’re playing with power! You’re bound to let this power go to your head and shoot yourself in the foot here and there. At some point(s) your program is going to do something that just doesn’t quite make sense.

The bad news is that your program doesn’t make any sense because you’ve written flaws into it. That’s fine, you’ve either written janky C programs, or not written any C. The good news is that GDB is here to help us learn from our mistakes!

Through the next few posts I’ll share some tips on basic GDB usage, explore a bit of history and dig more into how the C programs on my machine are actually working.

Building for Debugging

To kick things off, I’m going to just slap together a quick C program and a Makefile to assist in building it and running my debugger.

// test.c
#include <stdio.h>

char *done = "Done!";

int main(int argc, char *argv[]) {
  int i;

  for (i = 0; i < 10; i++) {
    printf("Iteration %d\n", i);
  printf("%s\n", done);

  return 0;

This program has a simple for loop and a few print statements and I’ll use GDB to inspect what it’s doing a bit more. To provide more information to the debugger about this program I’ll use the -g flag when building it.

# Makefile
CC=gcc -g -o $@ -Wall $<

all: test

test: test.c

debug: test
  gdb -q ./test

For maximum laziness, I added a debug target to my Makefile here so that I can use make debug to jump right it. I gave gdb the -q option to quiet down since it normally has a lot to say on startup.

That’s about all I need to get my program ready for debugging!

Basic Commands

Now we get to the hard part. GDB has a bajillion features so getting started can be daunting. Probably one of the best commands to learn first is the run command, as so far the program has been looked at a little bit, but isn’t actually running at the moment.

You can also provide arguments to the program by providing arguments to run. This program doesn’t care about arguments, but don’t let that stop you from giving it some anyway!

The excitement of just running a program in GDB is very short lived, I want to be able to stop the program somewhere and poke around a bit. The list command can spit out a listing of the program.

Initially gdb will show the first 10 lines of the source. You could run list again to see the next 10 lines but GDB has a friendly feature where hitting enter will automatically rerun your last command, so I used that to continue reading the full source.

Looking at this listing, I think a good place to pause and look around would be at the printf() call within my for loop. To have GDB stop here I’ll use the break command and I’ll give it the argument 10 to indicate I’d like to set a breakpoint at line 10.

Now when I give it a run, it’ll stop the program when it hits that line.

To resume the program, until the next breakpoint is hit, you can use the continue command. Another little time-saver trick with gdb is that many commands have shortcuts, such as c for continue.

Peeking Into The Code

The ability to set breakpoints and resume execution is a good start, but even better is getting a look around at this point in time to glean more about what the program is doing. It’s time to start looking beyond the C code and see what the program is actually doing in assembly, the state of the CPU in the context of our program and what’s going on in memory.

First let’s look at the assembly version of the main function. I’ll use the disassemble command for that, and I’ll tell it that main is what I’m interested in disassembling.

Oh noes! Assembly!

Assembly code get’s a bad rep, but it’s not as bad as people think it is. You might not want to write a large application in assembly, and that’s reasonable, but if you want to be a strong C programmer you need to know enough assembly to figure out what your program is up to.

x86_64 assembly has two different syntaxes to choose from, AT&T syntax and Intel syntax. They both work just fine but GDB defaults to AT&T syntax and I prefer the Intel syntax so I’ll use the command set disassembly-flavor intel to get it to my liking.

That looks better! Now let’s briefly look at a few things. Looks like my main function is 21 instructions long, alright… a smidge more than half of the operations are mov (move) instructions and I see a few branching operations, jmp (jump), call (call a subroutine), jle (jump if less than or equal to) and ret (return from subroutine).

One thing I find interesting is the instruction at offset <+64>, call 0x400430 <puts@plt>. I did not use the puts() function in my code! The compiler caught on that my last printf() statement doesn’t need to be a format string and optimized the result a little bit.

Let’s get back to inspecting what this program is up to, I’m currently still in the middle of my paused program, and I’m at the very start of one of my loop iterations. In this disassembly output I can see I’m at offset <+24>, as indicated by the little => arrow, this is the next instruction the program will run.

The mov instruction moves a value from one place to another, similar to the assignment operator  = in most programming languages. In this case the full instruction is mov eax,DWORD PTR [rbp-0x4] which is basically eax = DWORD PTR [rbp - 0x4]. Ignoring the right side of that for now, we’re assigning a value to something called eax. This eax thing is a CPU register, which is basically a variable in the hardware of the CPU. We can look at all the registers with the info command by saying info registers.

Okay so there are a bunch of registers, and eax is not one of them… GREAT! This is because the x86 architecture has been through a lot, way back in the day (early 70s) Intel released their 8008 CPU that had some 8-bit registers with names like A (for Accumulator).

When Intel got to the 8086 in the late 70s they made the A register twice the size (16-bits) and started calling it the AX register. To help with software compatibility with older system the AX register could be used as an 8-bit register with AH representing the higher 8 bits and AL the lower 8 bits.

Then the mid-80s showed up and Intel was like MOAR BITS and released their 80386 that had 32-bit registers, now they refer to the A register as EAX (there’s our guy!), again preserving backward compatibility by allowing the 16 and 8 bit registers to remain the same. Now-a-days our 64-bit processors are king, so we have the 64-bit register RAX, but can still use EAX, AX, AH, and AL.

All that history lesson to give full context on why mov eax, <stuff>  is going to modify our rax register!

Now, to run just that one instruction, I’ll use the nexti command. I’ll then check the registers again with the shorthand version of info registers and just look at the eax register: i r eax

If I continue my program, I’ll notice that this number correlates with something in my program.

The eax register is getting set to the i value I’m setting during my for loop!

In the next post I’ll continue digging into this program and discover more about the disassembled version of my C program and show off some more GDB commands along the way!