epoll() Tutorial – epoll() In 3 Easy Steps!

It wasn’t very long ago that it was a feat of greatness to get a single webserver setup to support 10,000 concurrent connections. There were many factors that made it possible to develop webservers, such as nginx, that could handle more connections with greater efficiency than their predecessors. One of the biggest factors was the advent of constant-time polling ( O(1) ) mechanisms for monitoring file descriptors introduced into most operating systems.

In the No Starch Press book, The Linux Programming Interface, section 63.4.5 provides a table of observations that describes the time it takes to check different quantities of file descriptors via some of the most common polling methods.

As this shows, the performance benefits of epoll are decent enough to have an impact on even as few as 10 descriptors. As the number of descriptors increases, using regular poll() or select() becomes a very unattractive option compared to epoll().

This tutorial will run through some of the basics of using epoll() on Linux 2.6.27+.

Prerequisite knowledge

This tutorial assumes you’re familiar and comfortable with Linux, the syntax of C and the usage of file descriptors in UNIX-like systems.



Getting started

Make a new directory to work out of for this tutorial, here’s the Makefile we’re using.

all: epoll_example
 
epoll_example: epoll_example.c
  gcc -Wall -Werror -o $@ epoll_example.c
 
clean:
  @rm -v epoll_example

Throughout this post I’ll be using functionality described by the following headers:

#include <stdio.h>     // for fprintf()
#include <unistd.h>    // for close(), read()
#include <sys/epoll.h> // for epoll_create1(), epoll_ctl(), struct epoll_event
#include <string.h>    // for strncmp

Step 1: Create epoll file descriptor

First I’ll go through the process of just creating and closing an epoll instance.

#include <stdio.h>     // for fprintf()
#include <unistd.h>    // for close()
#include <sys/epoll.h> // for epoll_create1()
 
int main()
{
	int epoll_fd = epoll_create1(0);
 
	if(epoll_fd == -1)
	{
		fprintf(stderr, "Failed to create epoll file descriptor\n");
		return 1;
	}
 
	if(close(epoll_fd))
	{
		fprintf(stderr, "Failed to close epoll file descriptor\n");
		return 1;
	}
	return 0;
}

Running this should work and display no output, if you do get errors you’re either probably running a very old Linux kernel or your system needs real help.

This first example uses epoll_create1() to create a file descriptor to a new epoll instance given to us by the mighty kernel. While it doesn’t do anything with it quite yet we should still make sure to clean it up before the program terminates. Since it’s like any other Linux file descriptor we can just use close() for this.

Level triggered and edge triggered event notifications

Level-triggered and edge-triggered are terms borrowed from electrical engineering. When we’re using epoll the difference is important. In edge triggered mode we will only receive events when the state of the watched file descriptors change; whereas in level triggered mode we will continue to receive events until the underlying file descriptor is no longer in a ready state. Generally speaking level triggered is the default and is easier to use and is what I’ll use for this tutorial, though it’s good to know edge triggered mode is available.

Step 2: Add file descriptors for epoll to watch

The next thing to do is tell epoll what file descriptors to watch and what kinds of events to watch for. In this example I’ll use one of my favorite file descriptors in Linux, good ol’ file descriptor 0 (also known as Standard Input).

#include <stdio.h>     // for fprintf()
#include <unistd.h>    // for close()
#include <sys/epoll.h> // for epoll_create1(), epoll_ctl(), struct epoll_event
 
int main()
{
	struct epoll_event event;
	int epoll_fd = epoll_create1(0);
 
	if(epoll_fd == -1)
	{
		fprintf(stderr, "Failed to create epoll file descriptor\n");
		return 1;
	}
 
	event.events = EPOLLIN;
	event.data.fd = 0;
 
	if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, 0, &event))
	{
		fprintf(stderr, "Failed to add file descriptor to epoll\n");
		close(epoll_fd);
		return 1;
	}
 
	if(close(epoll_fd))
	{
		fprintf(stderr, "Failed to close epoll file descriptor\n");
		return 1;
	}
	return 0;
}

Here I’ve added an instance of an epoll_event structure and used epoll_ctl() to add the file descriptor 0 to our epoll instance epoll_fd. The event structure we pass in for the last argument lets epoll know we’re looking to watch only input events, EPOLLIN, and lets us provide some user-defined data that will be returned for events.



Step 3: Profit

That’s right! We’re almost there. Now let epoll do it’s magic.

#define MAX_EVENTS 5
#define READ_SIZE 10
#include <stdio.h>     // for fprintf()
#include <unistd.h>    // for close(), read()
#include <sys/epoll.h> // for epoll_create1(), epoll_ctl(), struct epoll_event
#include <string.h>    // for strncmp
 
int main()
{
	int running = 1, event_count, i;
	size_t bytes_read;
	char read_buffer[READ_SIZE + 1];
	struct epoll_event event, events[MAX_EVENTS];
	int epoll_fd = epoll_create1(0);
 
	if(epoll_fd == -1)
	{
		fprintf(stderr, "Failed to create epoll file descriptor\n");
		return 1;
	}
 
	event.events = EPOLLIN;
	event.data.fd = 0;
 
	if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, 0, &event))
	{
		fprintf(stderr, "Failed to add file descriptor to epoll\n");
		close(epoll_fd);
		return 1;
	}
 
	while(running)
	{
		printf("\nPolling for input...\n");
		event_count = epoll_wait(epoll_fd, events, MAX_EVENTS, 30000);
		printf("%d ready events\n", event_count);
		for(i = 0; i < event_count; i++)
		{
			printf("Reading file descriptor '%d' -- ", events[i].data.fd);
			bytes_read = read(events[i].data.fd, read_buffer, READ_SIZE);
			printf("%zd bytes read.\n", bytes_read);
			read_buffer[bytes_read] = '\0';
			printf("Read '%s'\n", read_buffer);
 
			if(!strncmp(read_buffer, "stop\n", 5))
				running = 0;
		}
	}
 
	if(close(epoll_fd))
	{
		fprintf(stderr, "Failed to close epoll file descriptor\n");
		return 1;
	}
	return 0;
}

Finally we’re getting down to business!

I added a few new variables here to support and expose what I’m doing. I also added a while loop that’ll keep reading from the file descriptors being watched until one of them says ‘stop’. I used epoll_wait() to wait for events to occur from the epoll instance, the results will be stored in the events array up to MAX_EVENTS with a timeout of 30 second. The return value of epoll_wait() indicates how many members of the events array were filled with event data. Beyond that it’s just printing out what it got and doing some basic logic to close things out!

Here’s the example in action:

$ ./epoll_example 

Polling for input..

hello!

1 ready events
Reading file descriptor '0' -- 7 bytes read.
Read 'hello!
'

Polling for input...

this is too long for the buffer we made

1 ready events
Reading file descriptor '0' -- 10 bytes read.
Read 'this is to'

Polling for input...
1 ready events
Reading file descriptor '0' -- 10 bytes read.
Read 'o long for'

Polling for input...
1 ready events
Reading file descriptor '0' -- 10 bytes read.
Read ' the buffe'

Polling for input...
1 ready events
Reading file descriptor '0' -- 10 bytes read.
Read 'r we made
'

Polling for input...

stop

1 ready events
Reading file descriptor '0' -- 5 bytes read.
Read 'stop
'

First I gave it a small string that fits in the buffer and it works fine and continues iterating over the loop. The second input was too long for the read buffer, and is where level triggering helped us out; events continued to populate until it read all of what was left in the buffer, in edge triggering mode we would have only received 1 notification and the application as-is would not progress until more was written to the file descriptor being watching.

I hope this helped you get some bearings on how to use epoll(). If you have any problems, questions or feedback I’d appreciate you leaving a comment!

Cross Compiling C Code for ARM

Hello! In this brief post I will share how you can cross compile some simple C code for running on an ARM based device. I have a desire to write a few simple things in C for tinkering with OpenBMC on Barreleye G2 and figured I would share my process.



The Build Box

To build this program I’m going to use a freshly built Ubuntu 16.04.3 VM, that way I know for sure what dependencies are needed. My host system is also running Ubuntu 16.04.3 and I’m using Virt Manager as an interface to libvirt that is serving my VMs via QEMU and KVM.

Starting the install

After a few minutes I am ready to go!

The Codes

The code for this is pretty simple, I’ll just build the ol’ hello world!

#include <stdio.h>

int main() {
  printf("Hello, ARM!\n");
  return 0;
}

I’ll SSH into my VM so that my commands are easier to follow. I start by creating the source file with vim.

Building and Testing

Ubuntu offers many pre-built cross compilers. All you need to do is run a sudo apt update to ensure your package listing is up to date and a sudo apt install -y build-essential gcc-arm-linux-gnueabi will get you what you need.

To build, I’ll use the arm-linux-gnueabi-gcc compiler instead of just gcc. I’ll verify it was indeed for an ARM machine with readelf, then scp it from the VM to my host system.

Next, to test it, I’ll scp it over to my OpenBMC system, ssh into it and give it a go!

There it is! I’ve successfully cross compiled a simple program to run on an ARM machine. If you have any questions about this please let me know in the comments!

Making with Make

I enjoy writing in a variety of low level and compiled languages. One of the tools I use almost every time regardless of the language, or sometimes mix of languages, is make. In this article, I want to share some of the ways that make can be used and some of the tips and tricks I employ the most when using make.

I’ll be writing and running these examples on my Ubuntu 16.04.3 laptop with GNU Make 4.1.

Make Without Makefiles

I almost always use make with a Makefile, as it’s significantly more customizable that way, but I think it’s important to know that make does have some usage you can employ even without a Makefile.

Without a Makefile, make can still follow some of it’s implicit rules for building some files. Let’s say we have the following test.cpp file ready to build:

#include <stdio.h>

int main() {
  printf("Hello from test.cpp!\n");
  return 0;
}

You could build the test program with a simple call to make test.

Or if you’d like to build a test.o object file, you can do make test.o and if that’s present when you do make test it’ll build the binary using the already built object file.

There is a pretty good smattering of things you can build this way, check the documentation if you’re interested in more detail on the rules and supported targets.



Looking at Makefiles

Most projects will have a depth beyond what make is able to determine with all it’s smarty-pantsness, and in those cases the beloved Makefile is there to make life easy.

If you’ve pulled down and built a common open source project and looked at the Makefile, or generated one with a tool like autotools or cmake, you may have looked at it with much confusion.

As an example I’ll look at libuv, my favorite cross platform asynchronous I/O library. After cloning the repo down, running ./autogen.sh to generate the build configuration script, then running the ./configure script I get a nearly 5000 line Makefile. To me it looks like mostly gibberish and some tests. In all fairness there is a lot of good things happening in there but it’s not good for learning how to write a Makefile.

A Makefile doesn’t always need to be that cray-cray. For my own projects I try to keep it pretty simple, though over time it generally becomes more complex. An example of mine, from my post on building a barebones Linux system, is on my github here. I’m not the only person crazy enough to stick with a handwritten Makefile; the Redis database also uses a handwritten Makefile, and Redis is production quality and awesome AF.

Let’s start looking at the basics of making your own Makefile!

Makefile Basics

Most of the time when you use make, it will be looking for a file named Makefile to find your targets. If you run make without a Makefile, you’ll be greeted with this lovely message:

make: *** No targets specified and no makefile found. Stop.

If you had say, an empty Makefile, you’ll see something along the lines of:

make: *** No targets. Stop.

The first thing you should be aware of regarding Makefile syntax is that tabs are part of the syntax! I’ve seen a few developers start building a Makefile and be like “WTF!” when nothing works because their text editor is configured to insert spaces when they hit tab.

For the record, I use both 😀

The general format of a Makefile is a a list of targets with optional dependencies and commands

<target>: [dependency] [dependency]
<tab>[command]
<tab>[command]

As an example, I’ll define a target test that will not have dependencies and that target will run some echo commands.

test:
  echo test!
  echo IT WERKS!!!!

With this in my Makefile, if I run make with no arguments it’ll run my first target test. The common convention is for a Makefiles to start with the target all.

If I add a second target moartest and I want to run that one, I’ll need to specify it during my command as make moartest.

test:
  echo test!
  echo IT WERKS!!!!

moartest:
  echo woah now, so fancy

Makefile Dependencies

One of my favorite things about make is the way it handles dependencies. If you’re using it for building a project you can organize the steps however you’d like and structure a hierarchy where one step runs before another.

I’ll extend my previous example to add my moartest target as a dependency of the test target.

test: moartest
  echo test!
  echo IT WERKS!!!!

moartest:
  echo woah now, so fancy

Now when I run make, test will be inspected since it is the first target and since test has moartest in its list of dependencies that make will first look for that target and if it’s commands execute successfully the commands for test will also be ran.

If for some reason the dependency commands should fail, make will error out at that point. To simulate this I will add an exit 1 command to my moartest target.

test: moartest
  echo test!
  echo IT WERKS!!!!

moartest:
  echo woah now, so fancy
  exit 1

If the target name is a file, and that file already exists, the target will be skipped. Here’s an example where my randomcrap target generates a file that’s a dependency of my test target.

test: randomcrap
  echo we have random!

randomcrap:
  dd if=/dev/urandom of=randomcrap bs=1024 count=1

A dependency doesn’t need to be a another target, in many cases it’s useful if a dependency is some source file. make will look to see if that source file has been updated and will re-run the target only when it seems necessary.

Consider this example:

test: copiedfile
  echo we have the latest copy!

copiedfile: originalfile
  cp originalfile copiedfile

And observe how make response to the absence of the source file, how it skips the file when it’s already the same as the original, and how updates to the original will be noticed during the subsequent run.



Variables in Makefiles

It is often useful to have some variables in your Makefile. Variables can be set with the NAME=VALUE syntax. In my first example here I’ll compile the following hello.c program:

#include <stdio.h>

int main(int argc, char *argv[]) {
  printf("Well hello you proverbial world you.\n");
  return 0;
}

To make my compilation of the program a bit more flexible I’ll make a COMPILER variable to setup what compiler I’d like it to use.

COMPILER = gcc

all: hello-world

hello-world: hello.c
  $(COMPILER) -o hello-world hello.c

Now if I wanted to switch my various build targets to use clang instead, I can just modify my COMPILER variable.

I could even move my program name and source file to their own variables, and reference that variable as my target and its dependencies.

COMPILER = clang
PROGRAM = hello-world
SOURCE = hello.c

all: $(PROGRAM)

$(PROGRAM): $(SOURCE)
  $(COMPILER) -o $(PROGRAM) $(SOURCE)

Outside of the variables you define yourself, there are also some automatic variables that can be pretty handy.

The three I use the most are $@ which if used in a command will be the name of the target, $< which will be the first dependency for that target and $^ which will be all of the dependencies for the target.

all: automagic

automagic: automation magic
  echo "target: $@"
  echo "first dependency: $<"
  echo "all dependencies: $^"

These variables can be combined in interesting and useful ways. The automatic variables can even be embedded in your normal variables. Let’s say we have some C program that has a header and a code file, you’d want to rebuild the program if the header was changed but not include the header as an argument to the compiler. You could make your own compiler rule that includes most of the settings you want and define a pattern where the first dependency is included in the commands for the target.

PROGRAM = myprogram
COMPILE_PROGRAM = gcc -Wall -o $@ $<

all: $(PROGRAM)

$(PROGRAM): main.c main.h
  $(COMPILE_PROGRAM)

Additional Command-fu

There are two other things I think that are useful to know when writing the commands for the targets.

So far we’ve seen all our output repeated, which is normally quite handy for debugging. If you feel like making your output a little prettier you can start your command with @ to squelch the output.

all:
  @echo "one moment"
  @sleep 5
  @echo "okay i'm back"
  @sleep 2

Another good thing to be aware of is that each command is ran from your current working directory. If you want to do something like make a directory, jump into it and do more work inside of it, you’ll have to run multiple commands in a single go.

all:
  @mkdir subdirectory
  @cd subdirectory
  @pwd
  @cd subdirectory; pwd

If you find yourself with really long lines in your Makefile you can always add a backslash (\) before your new line to ask the make parser to ignore that as you’re just trying to make things pretty.

And that will wrap up my post on Makefiles! I hope you find this useful and I’d love to receive your questions and feedback in the comments. Keep Tinkering!