How I scripted testing for SystemVerilog

In Python and other languages, I’m used to having test suites and a handy runner to pull it all together. While digital logic designers are usually pretty good about writing test benches to go along with the RTL code, I didn’t find a lot of resources out on the interwebs that described a process for automating tests against SystemVerilog code. In my desire to have something similar to continuous integration testing for RTL I explored what options I’d have for scripting the process.

Design Under Test

The motivator for this excersize is that I was wanted to extend my shift register implementation to support a configurable depth. Initially it would shift data after one clock cycle, but it can be useful to have other delays as well. I also wanted to use this as an example for the SystemVerilog package manager I’ve been slapping together, so I wanted some way to tell early if I do something dumb that’ll impact other projects that use this shift register.

So I extended my module to include this new depth parameter:

module shift_register
    #(parameter width = 1,
      parameter depth = 1) (
  input logic clock,
  input logic [0:width-1] in,
  output logic [0:width-1] out);

  logic [0:depth-1][0:width-1] data;

  always_ff @ (posedge clock) begin
    {out, data} <= {data, in};
  end
endmodule

To test this I wrote this basic test bench:

module test_shift_register_default();

  logic clock;
  logic data_in;
  logic data_out;

  shift_register dut(clock, data_in, data_out);

  initial begin
    clock <= 0;
    data_in <= 0;
    #3 data_in <= 1;
    #4 assert (data_out == 0);
    data_in <= 0;
    #4 assert(data_out == 1);
    #5 $finish();
  end

  // 2ns clock
  always #2 clock <= ~clock;

endmodule

This test runs some basic assertions to validate the vanilla version of this shift module, I wrote tests for a few parameter permutations as well.



Vivado Batch Mode

Since the cards I have at work to tinker with use Xilinx based FPGAs, I decided to see how I could go through this process using Vivado. I read partially through an official Vivado simulation tutorial; in particular, chapter 3 on running simulations in “batch mode”. After some tinkering with the process I found this was a lot easier than I initially expected.

The first thing I needed to do was use xvlog to parse and build my SystemVerilog code.

$ xvlog --sv shift_register.sv tests/*.sv
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/shift-register/shift_register.sv" into library work
INFO: [VRFC 10-311] analyzing module shift_register
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/shift-register/tests/test_16bit_delayed.sv" into library work
INFO: [VRFC 10-311] analyzing module test_shift_register_16bit_delayed
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/shift-register/tests/test_8bit.sv" into library work
INFO: [VRFC 10-311] analyzing module test_shift_register_8bit
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/shift-register/tests/test_default.sv" into library work
INFO: [VRFC 10-311] analyzing module test_shift_register_default
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/shift-register/tests/test_delayed.sv" into library work
INFO: [VRFC 10-311] analyzing module test_shift_register_delayed

This creates a xsim.dir that has a file built for each module in my project, and a few log files.

Next, I use xelab to create a simulation snapshot. I’m not sure what a simulation snapshot is, but I know the xsim binary enjoys working with them! Generally I’m just giving it an argument for the module I want to use as my top level entity.

$ xelab test_shift_register_default
Vivado Simulator 2015.4
Copyright 1986-1999, 2001-2015 Xilinx, Inc. All Rights Reserved.
Running: /opt/Xilinx/Vivado/2015.4/bin/unwrapped/lnx64.o/xelab test_shift_register_default 
Multi-threading is on. Using 2 slave threads.
Starting static elaboration
Completed static elaboration
Starting simulation data flow analysis
Completed simulation data flow analysis
Time Resolution for simulation is 1ns
Compiling module work.shift_register
Compiling module work.test_shift_register_default
Built simulation snapshot work.test_shift_register_default

****** Webtalk v2015.4 (64-bit)
  **** SW Build 1412921 on Wed Nov 18 09:44:32 MST 2015
  **** IP Build 1412160 on Tue Nov 17 13:47:24 MST 2015
    ** Copyright 1986-2015 Xilinx, Inc. All Rights Reserved.

source /home/kwilke/projects/shift-register/xsim.dir/work.test_shift_register_default/webtalk/xsim_webtalk.tcl -notrace
INFO: [Common 17-206] Exiting Webtalk at Thu Mar 17 08:45:07 2016...

This adds some more data to the xsim.dir and brings them to a state that I can run the tests in. At this point I can use xsim with the -R flag to run simulation until it ends, then quit.

$ xsim -R test_shift_register_default

****** xsim v2015.4 (64-bit)
  **** SW Build 1412921 on Wed Nov 18 09:44:32 MST 2015
  **** IP Build 1412160 on Tue Nov 17 13:47:24 MST 2015
    ** Copyright 1986-2015 Xilinx, Inc. All Rights Reserved.

source xsim.dir/work.test_shift_register_default/xsim_script.tcl
# xsim {work.test_shift_register_default} -maxdeltaid 10000 -autoloadwcfg -runall
Vivado Simulator 2015.4
Time resolution is 1 ns
run -all
$finish called at time : 16 ns : File "/home/kwilke/projects/shift-register/tests/test_default.sv" Line 24
exit
INFO: [Common 17-206] Exiting xsim at Thu Mar 17 08:48:42 2016...

Automating the process

Initially, I whipped up a Makefile to allow me to run a make test to test my project. Since the xsim program doesn’t exit with an error status code for assertion failures, I was piping it’s output to a small shell script. This process was a little wonky.

At this point I decided to add it to Packilog so I could control the process a little easier with Python. I found this to be a pleasant and familar type of flow.

$ packilog -t
Using vivado testing driver
Building tests.
Running test module "test_shift_register_default"
Simulating test module 'test_shift_register_default'.

****** xsim v2015.4 (64-bit)
  **** SW Build 1412921 on Wed Nov 18 09:44:32 MST 2015
  **** IP Build 1412160 on Tue Nov 17 13:47:24 MST 2015
    ** Copyright 1986-2015 Xilinx, Inc. All Rights Reserved.

source xsim.dir/work.test_shift_register_default/xsim_script.tcl
# xsim {work.test_shift_register_default} -maxdeltaid 10000 -autoloadwcfg -runall
Vivado Simulator 2015.4
Time resolution is 1 ns
run -all
$finish called at time : 16 ns : File "/home/kwilke/projects/shift-register/tests/test_default.sv" Line 24
exit
INFO: [Common 17-206] Exiting xsim at Thu Mar 17 08:53:11 2016...

Result: PASS
Running test module "test_shift_register_delayed"
Simulating test module 'test_shift_register_delayed'.

****** xsim v2015.4 (64-bit)
  **** SW Build 1412921 on Wed Nov 18 09:44:32 MST 2015
  **** IP Build 1412160 on Tue Nov 17 13:47:24 MST 2015
    ** Copyright 1986-2015 Xilinx, Inc. All Rights Reserved.

source xsim.dir/work.test_shift_register_delayed/xsim_script.tcl
# xsim {work.test_shift_register_delayed} -maxdeltaid 10000 -autoloadwcfg -runall
Vivado Simulator 2015.4
Time resolution is 1 ns
run -all
$finish called at time : 32 ns : File "/home/kwilke/projects/shift-register/tests/test_delayed.sv" Line 28
exit
INFO: [Common 17-206] Exiting xsim at Thu Mar 17 08:53:13 2016...

Result: PASS
Running test module "test_shift_register_8bit"
Simulating test module 'test_shift_register_8bit'.

****** xsim v2015.4 (64-bit)
  **** SW Build 1412921 on Wed Nov 18 09:44:32 MST 2015
  **** IP Build 1412160 on Tue Nov 17 13:47:24 MST 2015
    ** Copyright 1986-2015 Xilinx, Inc. All Rights Reserved.

source xsim.dir/work.test_shift_register_8bit/xsim_script.tcl
# xsim {work.test_shift_register_8bit} -maxdeltaid 10000 -autoloadwcfg -runall
Vivado Simulator 2015.4
Time resolution is 1 ns
run -all
$finish called at time : 16 ns : File "/home/kwilke/projects/shift-register/tests/test_8bit.sv" Line 24
exit
INFO: [Common 17-206] Exiting xsim at Thu Mar 17 08:53:25 2016...

Result: PASS
Running test module "test_shift_register_16bit_delayed"
Simulating test module 'test_shift_register_16bit_delayed'.

****** xsim v2015.4 (64-bit)
  **** SW Build 1412921 on Wed Nov 18 09:44:32 MST 2015
  **** IP Build 1412160 on Tue Nov 17 13:47:24 MST 2015
    ** Copyright 1986-2015 Xilinx, Inc. All Rights Reserved.

source xsim.dir/work.test_shift_register_16bit_delayed/xsim_script.tcl
# xsim {work.test_shift_register_16bit_delayed} -maxdeltaid 10000 -autoloadwcfg -runall
Vivado Simulator 2015.4
Time resolution is 1 ns
run -all
$finish called at time : 21 ns : File "/home/kwilke/projects/shift-register/tests/test_16bit_delayed.sv" Line 24
exit
INFO: [Common 17-206] Exiting xsim at Thu Mar 17 08:53:28 2016...

Result: PASS
Package Test Result: PASS

Hopefully the test driver pattern in Packilog will allow for easy cross-tool testing, as I would like to get support for a variety of common tools. ‘m pretty happy with this flow so far because it allows me to write my code in my editor of choice and use a simple command line program to pull it all together.

Leave a Reply

Your email address will not be published. Required fields are marked *