Hello AFU on Alpha-Data KU3

Picking up on the Hello AFU project, I’ve recently gone through the motions of building the Hello AFU project for an actual CAPI device and tested it out. This post documents the process I followed to build and deploy this on real hardware.

Requirements

To complete this process you’ll need a few things:
* A POWER8 based machine, for me I’m using a Barreleye server
* An Alpha-Data KU3 card
* The latest HDK archive from Alpha Data’s support site, at this time that file is named ADMPCIEKU3_CAPI_HDK_REL18MAR16.zip
* A licensed version of Xilinx’s Vivado



Preparing files for the build

First off, we need to extract the HDK

unzip ADMPCIEKU3_CAPI_HDK_REL18MAR16.zip

In the HDK by default, there will be some AFU source files in adku060_capi_1_1_release/Sources/afu/ we’ll jump in there and delete them, then copy over the SystemVerilog files from the hello-afu repository

cd adku060_capi_1_1_release/Sources/afu/
rm *
cp ~/projects/hello-afu/*.sv .

Next, open the project file adku060_capi_1_1_release/Sources/prj/psl_fpga.prj in a text editor to change a few lines. Remove all of the lines that start with verilog work, then add lines to reference the source files we copied into the afu directory. Some bash-fu for that:

cd ../prj
sed -i '/^verilog work/d' psl_fpga.prj
for i in `ls ../afu/*.sv | cut -d'/' -f3`; do echo "verilog work \"afu/$i\"" >> psl_fpga.prj; done

That should have us setup to build our AFU in leiu of the one that comes with the HDK!

Build and flash the binfile

With our files in the right spot and our project file modified, we just need to run a few of the tcl scripts in the HDK through vivado.

vivado -mode batch -source psl_fpga.tcl -notrace
vivado -mode batch -source write_bitstream.tcl -notrace

The first run here does the heavy lifting of synthesis, place and route, etc. The second command generates the actual binfile and bitfile that we can use to flash the device. The first command takes a significant amount of time on my i7-equipped laptop, about 40 minutes, the second command completed in about 9 seconds. Maybe someday we’ll have a CAPI-based accelerator for synthesis and place & route! Now that the building is complete I have my bitfile at capi-adku060/psl_fpga_flash.bin

To flash this to your device to a card that already has the PSL working you can use the capi-flash-script utility. If your card is factory-fresh or in a bad state, you can use a JTAG programmer and Vivado’s Hardware Manager to flash directly from your laptop, or remotely via xvcserver.

Using the AFU

After I flashed my AFU, I ensured libcxl was setup on my server. Since I’m running Ubunt 16.04 I simply installed it via apt.

apt-get install -y libcxl-dev

Next I rebooted the machine so that everything is nice and fresh, as part of the PCIe reset the bitfile from the KU3’s flash chip will be flashed onto the FPGA. I can verify the card is in a good state because I have my cxl device at /dev/cxl/afu0.0d.

I run my test_afu binary from the hello-afu project and boom! The same result as I get from simulation, woo-hoo!

Leave a Reply