Running Nomad on ppc64le

Recently I’ve been doing some experimentation with Nomad, a tool that helps manage applications running on a cluster of machines. I first ran through their getting started guide, but wanted to continue my education by deploying it on some of my Barreleye systems to see if it could be used for benchmarking and other lab workloads.

I quickly found that there’s no official support for ppc architectures, but since this isn’t a production environment I wasn’t going to let that stop me.

Note on PPC64 vs PPC64LE

Before POWER8, the POWER architecture was exclusively big-endian. This became a barrier for users as many projects were not designed to consider the endianness of the processor and some applications and libraries would not run properly. There’s an assumption the system is little-endian (since x86 dominates the server and desktop market) and this can be fairly time consuming to address.

IBM responded to customer feedback about this, and introduced bi-endian support in POWER8 to allow the user to decide if they want their OS and applications to use little or big endianness. Since the release of POWER8, the Linux community has mostly moved over to ppc64le, as bugs related to processor endianness are no longer a factor.

For the most part, building code for ppc64le is typically identical to building code for x86_64!

But First, Go!

As Nomad does not officially support ppc, I’ll need to build it from source. The README clearly states that we’ll need Go version 1.9 or newer.

The system I’m working on is kicked with Ubuntu 16.04.4 LTS, and if you bring in the golang-go it will install go 1.6. Alternatively, there is a golang-1.9 package that’ll slap 1.9.2 on your system.

In my case though I’m going to manually install the latest stable version, 1.10, since I know there’s been good work going on to improve the Go assembler and performance for PowerPC. Besides, manually installing Go is easy as pie.

To install go I’ll download the latest built archive for ppc64le, extract it to /usr/local/ and append my PATH environment variable for all users by editing my /etc/profile file. Then I’ll create a go directory in my home directory and update my own .bashrc file to set my GOPATH variable to its path, while also appending $GOPATH/bin to my users PATH variable.

Off the Beaten Path

Next I’ll follow the Installing Nomad documentation on installing via source to see how far I get.

So the only hangup here is there’s no build target in the Makefile for ppc64le. The amd64 target is a pretty close guess to it, so I’ll duplicate that target and make some minor modifications to it.

There it is! I’m still a bit of a noob with Nomad and I’ve only run very basic workloads with it, but everything is working as expected so far. I look forward to doing a bit more tinkering with it in the near future.

Barebones Linux on Barreleye G1

Following up on the previous article, this post will be building a similar barebones Linux for the POWER8-based OpenPOWER Barreleye system.

For this round, I will be building the system on a freshly kicked x86_64 Ubuntu 16.04 VM to capture all dependencies needed to follow the same steps.

Cross Compiling Linux

To get things started I will pull in the various dependencies needed to build the system.

sudo apt install -y build-essential gcc-powerpc64le-linux-gnu libssl-dev bc

With these dependencies in place, I’ll create a new directory to work out of and pull down the kernel the same way as the last post. A minor difference is today the latest stable version is 4.13.1, so I’ll be rolling with that version.

mkdir barebones-ppc64le
cd barebones-ppc64le/
tar xf linux-4.13.1.tar.xz
cd linux-4.13.1/

Next I setup some environment variables to prepare for cross compilation.

export ARCH=powerpc
export CROSS_COMPILE=/usr/bin/powerpc64le-linux-gnu-

The next big step is to configure the kernel. For this platform, I have not yet had success using one of the default configs that comes with the kernel source tree. The good news is that the OpenPOWER firmware of this platform is Linux based and open source. The kernel configuration from the firmware build repo is a known good configuration for this exact system, so that gives me assurance it will work pretty well.

I’ll download the skiroot kernel config, run the kernels make oldconfig to get the kernel build happy with the configuration (piping yes '' to make to accept the default answers to its many questions), and build the kernel using the number of threads returned to me by nproc.

wget -O .config
yes '' | make oldconfig
make -j`nproc`
cp vmlinux ..
cd ..

Building initramfs and Testing

Building the initramfs is also nearly identical to the previous post, just using the cross compiler instead of the default. This is the program that will be my init process.

#include <stdio.h>
#include <unistd.h>
int main() {
  while (1) {
    printf("Hello from the land of Barreleye\n");

And the steps I used to compile and pack into my initramfs

powerpc64le-linux-gnu-gcc -o init -static test.c
echo init | cpio -o --format=newc > initramfs

With that all done, I will upload it to a webserver I have in our lab and boot up my Barreleye system to Petitboot, the firmware stacks primary interface. I’ll exit to a shell, wget the kernel and initramfs and kexec load and execute!

Since I’m connecting into the system from home to the BMC console the interface is a wee bit slow to respond to input, but it does the trick.

Testing my build on a Barreleye system

All looks good! Please feel free to comment or leave any questions you may have.