Skip to content

Chapter 11: Labs and a Design Study

Reading about pipelines, out-of-order execution and caches gets you halfway. The other half is watching them run. Every tool in this chapter is free and open-source and can be installed on a laptop. You start with a pipeline that you can single-step, move down to a cycle-level simulator that models a real out-of-order core, then generate and simulate an actual RISC-V system-on-chip, and finally look at the individual wires of a tiny core. Nothing here needs an FPGA or a fabrication run, only a terminal. The chapter ends with a measured design study in which you answer one microarchitecture question with evidence.

The four labs are ordered by depth. Do them in order the first time. Before every run, write down what you expect. A measurement teaches you something only when it can surprise you.

A word on commands

The commands in this chapter are those of the tools' own instructions at the time of writing. Class and option names change between releases (the names of Chipyard configurations and of gem5 statistics in particular), so check each project's current documentation, and search the configuration folder or stats.txt for the nearest match when a name is not found. On Windows, run the labs of this chapter, except the first, inside WSL2, not in the native command prompt or PowerShell: the toolchains assume a POSIX environment. Builds are large; do not cancel a compile that "looks stuck".

11.1 The ladder of tools

Lab Tool What it is Reinforces
1 Ripes A visual pipeline simulator: minutes to start Chapter 2
2 gem5 A cycle-level simulator with an out-of-order core: measures IPC and miss rates Chapters 3 and 4
3 Chipyard Generates real Rocket and BOOM systems as Verilog Chapters 6 and 8
4 Verilator and GTKWave Fast simulation of a tiny core's hardware description, with waveforms Chapter 6

11.2 Lab 1: Ripes, a pipeline that moves one clock at a time

Ripes is a free graphical RISC-V simulator. It animates the datapath cycle by cycle: instructions slide through fetch, decode, execute, memory and write back; forwarding paths light up; hazards insert bubbles; and a cache model shows hits and misses. It models single-cycle and five-stage in-order processors, with or without forwarding and hazard detection, a branch model, and a parametrisable L1 cache (size, associativity, replacement and write policy). It is in-order only: for out-of-order behaviour you need gem5.

Getting it. Ripes ships as one prebuilt binary for Windows, macOS and Linux, so no compiler is needed to start: fetch a ready-made release from the project's page (github.com/mortbopet/Ripes, under Releases). On Linux it is an AppImage. It can also be built from source, which needs Qt 6, CMake and a C++17 compiler.

The program: add up an array. Open the Editor tab, paste the program below, then in the Processor tab choose the five-stage processor with forwarding and hazard detection, and step one clock at a time. The loop is a classic source of a load-use hazard and a taken branch on each iteration.

.data
arr:    .word 3, 1, 4, 1, 5, 9, 2, 6
        .text
        .globl  main
main:   la      a0, arr         # a0 = address of arr[0]
        li      a1, 8           # a1 = count
        li      a2, 0           # a2 = running sum
loop:   lw      t0, 0(a0)       # load arr[i]
        add     a2, a2, t0      # use it at once: a load-use hazard
        addi    a0, a0, 4       # next element
        addi    a1, a1, -1      # one fewer to go
        bnez    a1, loop        # back if not done: a control hazard
end:    j       end

Predict first. What will a2 hold at the end? Where will a bubble appear on every pass? What happens after each taken bnez? The answers: \(a2 = 3+1+4+1+5+9+2+6 = 31\); one bubble between lw and add; and the wrongly fetched instructions after a taken branch are flushed.

As you step, watch three things in the datapath view.

  1. Forwarding paths. After lw t0, the add depends on t0. With forwarding on, a wire from a later stage lights up and feeds the value to the ALU input.
  2. The load-use bubble. A load's result is not ready until MEM, so even with forwarding the dependent add stalls exactly one cycle, and you see a bubble injected.
  3. The branch. bnez resolves late, so a taken branch flushes the wrongly fetched instructions. The cycle counter and the pipeline diagram update live.

Example 11.1: predicting the cycle count by hand

Make the assumptions explicit: five stages with full forwarding; the branch is resolved in EX; the front end always fetches the next instruction in line (predict not taken), so a taken branch flushes two instructions and costs 2 cycles. Count only the eight passes of the loop and ignore the set-up instructions and the fill and drain.

  • Each pass has 5 instructions: lw, add, addi, addi, bnez.
  • add uses t0 straight after lw: 1 load-use stall per pass.
  • The first seven passes end in a taken branch: 2 more cycles each. The eighth falls through: no penalty.

Passes 1 to 7 take \(5 + 1 + 2 = 8\) cycles each, and pass 8 takes \(5 + 1 = 6\): a total of \(7 \times 8 + 6 = 62\) cycles for 40 instructions, a CPI of \(62/40 = 1.55\).

Now move addi a0,a0,4 between lw and add. The add is then two instructions behind the lw, and gets t0 by forwarding with no stall. A pass takes \(5 + 2 = 7\) cycles when the branch is taken and 5 when it is not: \(7 \times 7 + 5 = 54\) cycles, a CPI of \(54/40 = 1.35\). Scheduling the code by hand saved one cycle per pass: 8 cycles in all.

Ripes will report different absolute numbers (its own model of the branch, the set-up instructions, the fill and drain), but the differences between the configurations should have the same shape.

Suggested experiments.

Experiment Do this Expect to see
Forwarding on or off Run on the model without forwarding and note the cycles; then with forwarding Without it each dependence stalls up to 2 cycles; with it only the load-use pair stalls, for 1
Kill the bubble Move addi a0,a0,4 between the lw and the add The bubble disappears: you scheduled the code by hand
Cache Enable the data cache, then shrink it or change its associativity First-touch misses, then hits; a direct-mapped cache conflicts sooner than a 2-way one
Longer loop Grow the array to 64 or more words The CPI settles above 1.0; the gap is the hazard penalty of Chapter 2

11.3 Lab 2: gem5, measuring a real out-of-order core

gem5 is the research-grade, cycle-level simulator used throughout academia and industry. Where Ripes shows one in-order pipeline, gem5 models complete systems and, crucially, a real out-of-order core, so that you can measure the effects of Chapters 3 and 4. It offers CPU models along a speed and detail scale:

  • AtomicSimple: the fastest, functional, no timing;
  • TimingSimple: in order, with real memory timing;
  • Minor: an in-order pipeline;
  • O3: out of order, with a reorder buffer, issue queue, load/store queue, renaming and speculation.

It also models a configurable memory hierarchy (levels, associativity, replacement), buses and crossbars, branch predictors and DRAM controllers. You configure the machine in a Python script and gem5 runs your binary on it.

Building it (Ubuntu or Debian, or WSL2): a C++ toolchain, SCons and a few libraries, then the build of the RISC-V variant. The first build is long.

sudo apt install build-essential scons python3-dev git m4 zlib1g-dev \
                 libprotobuf-dev protobuf-compiler libboost-all-dev
git clone https://github.com/gem5/gem5.git
cd gem5
scons build/RISCV/gem5.opt -j$(nproc)    # long the first time: be patient

Running a program on the out-of-order core. In "syscall emulation" mode gem5 runs a statically linked user program directly, with no operating system. The shipped configuration script wires up a CPU, caches and memory from command-line switches.

# a small static RISC-V program
riscv64-unknown-elf-gcc -O2 -static hello.c -o hello

build/RISCV/gem5.opt configs/deprecated/example/se.py \
    --cmd=hello --cpu-type=O3CPU \
    --caches --l2cache \
    --l1d_size=32kB --l1i_size=32kB --l1d_assoc=8 \
    --l2_size=256kB --l2_assoc=8

grep ipc m5out/stats.txt                          # instructions per cycle
grep "dcache.overallMissRate" m5out/stats.txt
grep -iE "branchPred.*(mispred|lookups)" m5out/stats.txt

The results land in m5out/, and the numbers are in stats.txt as name-and-value pairs. Names differ slightly across gem5 versions; if a grep finds nothing, open stats.txt and search for the closest match. As a sanity check, on O3 well-behaved code should show an IPC above 1.0, while on TimingSimple the IPC can never exceed 1.0. If your O3 run looks like an in-order core, you left --cpu-type at its default: always pass it explicitly.

Experiment Change Compare
Cache size sweep --l2_size = 64kB, 256kB, 1MB L2 miss rate and IPC: diminishing returns, the memory wall as a graph
Associativity --l1d_assoc = 1, 2, 8 Conflict misses fall as associativity rises
In order against out of order TimingSimpleCPU against O3CPU IPC and cycles: how much latency out-of-order execution hides
Branch cost A branch-heavy program Mispredictions × flush penalty ≈ lost cycles

Copy m5out/ aside after each run: the next run overwrites it.

11.4 Lab 3: Chipyard, a real SoC from Rocket and BOOM

Chipyard is Berkeley's open framework for building RISC-V systems-on-chip. It bundles generators (rocket-chip with the in-order Rocket core, BOOM, caches, the TileLink interconnect, peripherals and more), all written in Chisel and elaborated to synthesisable Verilog. Unlike gem5, a C++ model, Chipyard produces the actual hardware description that you could push through a physical-design flow to a real chip. This is where Chapters 6 and 8 become tangible.

A configuration is a Scala class that composes generators into an SoC. RocketConfig gives an in-order Rocket tile; a large BOOM configuration (for example LargeBoomV3Config, whose name varies by release) gives a wide out-of-order BOOM tile. Both attach L1 caches to an L2 and system memory over a TileLink fabric, with UART, GPIO and SPI peripherals on a peripheral bus. You then simulate the generated Verilog with Verilator.

Setting up. Chipyard is heavy: it pulls a large RISC-V toolchain and many Chisel dependencies through a one-time setup script. It needs Linux or WSL2, plenty of disk and time.

git clone https://github.com/ucb-bar/chipyard.git
cd chipyard
./build-setup.sh riscv-tools    # once: toolchain and dependencies (long)
source env.sh                   # in every new terminal

One program, two microarchitectures. Simulation lives under sims/; the Verilator flow is in sims/verilator, and the CONFIG make variable selects the SoC. The first build elaborates Chisel into Verilog and compiles the simulator, which is slow; later runs reuse it.

cd sims/verilator
make CONFIG=RocketConfig                   # build the simulator (slow once)
make CONFIG=RocketConfig run-binary \
     BINARY=$RISCV/riscv64-unknown-elf/share/riscv-tests/isa/rv64ui-p-add

make CONFIG=LargeBoomV3Config run-binary \
     BINARY=$RISCV/riscv64-unknown-elf/share/riscv-tests/isa/rv64ui-p-add

Changing the microarchitecture is a one-line change of configuration: BOOM shares the same TileLink and peripheral scaffolding, so only the tile changes. Compare the cycle counts of the two runs: same ISA, same binary, different microarchitecture. (A tiny single-instruction test mostly measures start-up, so for a more telling comparison run a longer program such as a benchmark.)

Where TileLink shows up. After elaboration, look under generated-src/ for the Verilog. Modules such as TLXbar and TLBuffer are the TileLink crossbar and pipeline registers, and the system, memory and peripheral buses wire the tile to the L2 cache, DRAM and the UART and GPIO. This is the interconnect chapter rendered as real, synthesisable modules: you can open the Verilog and trace a load from the core to memory through the crossbar.

11.5 Lab 4: Verilator and GTKWave, down to the wires

To really see a core work you simulate its hardware description and stare at the signals. Verilator compiles synthesisable Verilog into a fast C++ model; you drive it with a small test bench, dump a waveform file (VCD), and open it in GTKWave. The core to use is PicoRV32, the tiny, size-optimised, in-order core of Chapter 8, which is a single readable Verilog file, so nothing is hidden.

sudo apt install verilator gtkwave
git clone https://github.com/YosysHQ/picorv32.git
cd picorv32
make test_verilator       # build the model and run its test firmware
gtkwave testbench.vcd     # open the recorded waveform

If your checkout gates tracing behind a flag, enable it explicitly (for example --trace among the Verilator flags). If you write your own harness, recording a waveform always takes the same few lines: open a trace, then dump it every half clock.

Verilated::traceEverOn(true);
VerilatedVcdC* tfp = new VerilatedVcdC;
top->trace(tfp, 99);  tfp->open("testbench.vcd");
// each half clock:  top->clk = !top->clk; top->eval(); tfp->dump(t++);
tfp->close();

What to look for. Add clk, resetn and the memory interface: mem_valid, mem_ready, mem_addr, mem_wdata and mem_rdata, and the program counter. You will watch instruction fetches march through memory, see the valid and ready pair gate every access (the same handshake idea as AXI in Chapter 6) and confirm the reset behaviour. Zoom into one memory transaction and follow it from request to answer. If the waveform file is empty, tracing was not switched on: add --trace to Verilator or the traceEverOn and dump lines to your harness.

11.6 The open-source toolbox

Tool What it is for
Ripes Seeing an in-order pipeline, forwarding, hazards and a cache
gem5 Measuring IPC, miss rates and mispredictions, in order and out of order
Chipyard (with rocket-chip and BOOM) Generating Rocket and BOOM systems-on-chip as real hardware descriptions
Verilator Fast simulation of hardware descriptions, which records waveforms
GTKWave Viewing waveforms signal by signal
Spike The official RISC-V instruction-set simulator: a fast "golden" check of correctness, with no timing

11.7 Capstone: measure one design decision

Choose one change, implement it, and quantify the effect with a before-and-after number.

  • A. A prefetcher. In gem5, enable a stride prefetcher on a loop that walks memory, and measure the change in miss rate and IPC.
  • B. A wider core. In Chipyard, move from a small to a large BOOM configuration and compare cycles on the same binary (and CoreMark if you can). Where do the extra resources stop helping?
  • C. Associativity. In gem5, vary the L1 associativity (1, 2, 4, 8) at a fixed size: where do conflict misses stop mattering?
  • D. A new device. In Chipyard, attach a simple memory-mapped register device to the peripheral bus, write a bare-metal program that pokes its registers, and find the TileLink transaction in a waveform.

In every case state a hypothesis first, run the baseline, run the variant, and report the difference as a number. That before-and-after number is the whole point of a microarchitecture lab.

How to write it up

Every experiment report should contain:

  1. the question and your prediction;
  2. the set-up: machine, clock, operating system, tool and compiler versions, flags;
  3. a table of results, with at least 3 runs of each and the median reported;
  4. one chart;
  5. what matched your prediction, what did not, and why.

The design study is two pages: question, hypothesis, set-up, baseline, variant, the difference (with its spread over runs) and what you would try next. A marking guide that rewards the right things is: a clear question and prediction (15%); a reproducible set-up (20%); correct measurements, repeated (30%); honest interpretation (25%); presentation (10%).

11.8 Summary

  • Ripes makes the five-stage pipeline, forwarding and hazards visible; predict the cycle count by hand first.
  • gem5 is cycle-level: run a binary on the out-of-order O3 core and read the IPC, cache miss rates and branch mispredictions from stats.txt, sweeping cache size and associativity and comparing in-order with out-of-order. Always pass --cpu-type explicitly.
  • Chipyard generates real synthesisable Rocket and BOOM SoCs wired over TileLink, simulated with Verilator.
  • Verilator and GTKWave take you to the waveform of a tiny core such as PicoRV32. Spike is the golden functional check.
  • The capstone is simple: change one thing and measure the difference. Report the set-up honestly and repeat the runs.

11.9 Exercises

Exercise 11.1. You run the same binary on gem5's TimingSimple and O3 models and both report an IPC of 0.8. What is the likely mistake?

Answer

The O3 run did not actually use the out-of-order CPU. Pass --cpu-type=O3CPU explicitly.

Exercise 11.2. Which tool would you use to confirm that a program's answer is correct, without caring about timing?

Answer

Spike, the golden RISC-V instruction-set simulator: fast, functional, no timing.

Exercise 11.3 (Ripes). Run the array-sum program on the five-stage processor without forwarding and note the cycle count. Repeat with forwarding and hazard detection. Then move addi a0,a0,4 between the load and its use and run again. Screenshot one load-use bubble and one flush after a taken branch, and annotate them.

Answer

The register a2 holds \(3+1+4+1+5+9+2+6 = 31\) every time. The cycle counts should satisfy: no forwarding > forwarding > forwarding with the reordered code. In the reordered version each lw is followed by an independent instruction, so the load-use bubble disappears; the taken-branch flush remains.

Exercise 11.4 (gem5). Write a C program that sums a large array (for example 4 MB) several times, and compile it statically for RISC-V. On O3CPU, sweep --l2_size over 64kB, 256kB, 1MB and 4MB, recording the IPC and the L2 miss rate. At one size, compare TimingSimpleCPU with O3CPU. Plot IPC against L2 size. Where does it flatten, and why there?

Answer

IPC rises and the L2 miss rate falls as the cache grows, sharply once the array fits in the L2 cache and then flattening (diminishing returns). TimingSimple's IPC can never exceed 1; O3 should be clearly higher on the same binary. Exact numbers depend on your gem5 version, so report it.

Exercise 11.5. Use the hand model of Example 11.1 to predict the loop cycle count, with forwarding, for an array of 16 words instead of 8. What is the CPI?

Answer

There are 16 passes of 5 instructions: 80 instructions. The first 15 passes each take \(5 + 1 + 2 = 8\) cycles and the last takes \(5 + 1 = 6\): \(15 \times 8 + 6 = 126\) cycles. CPI \(= 126/80 = 1.575\). With the load-use bubble scheduled away: \(15 \times 7 + 5 = 110\) cycles, CPI \(= 1.375\).

Exercise 11.6. In the Verilator lab, why does a waveform file that is empty point to a missing switch rather than a bug in PicoRV32?

Answer

Verilator records nothing unless tracing is enabled both at build time (the --trace option) and in the harness (traceEverOn(true) and the dump calls). An empty file therefore means that tracing was never switched on, not that the core misbehaved.

Exercise 11.7 (design study). Pick option A, B, C or D of Section 11.7, state a hypothesis with a predicted size of effect, and describe exactly what you will run for the baseline and the variant, how many repetitions you will use, and what result would make you reject your hypothesis.

Answer

There is no single answer; a good one has a measurable prediction ("a stride prefetcher will raise the IPC of a sequential array sum by more than 10%"), a baseline and a variant that differ in exactly one parameter, at least three runs of each with the median and spread reported, a stated tool version and flags, and a rejection criterion ("if the IPC changes by less than the run-to-run spread, I conclude the prefetcher does not help this program"). A reader should be able to repeat the experiment from the report alone.