Skip to content

Chapter 9: Benchmarks and Metrics

How fast is fast? The Iron Law of Chapter 1 gives the definition, but in practice processors are compared with a zoo of numbers: MIPS, FLOPS, Dhrystone, CoreMark, Embench, SPEC, LINPACK, STREAM, MLPerf. This chapter says what each one measures, how to turn it into a fair comparison, which kind of average to use, and how results get bent. The aim is that you can read a performance claim critically and write one honestly.

You should know the Iron Law and CPI (Chapter 1), and the cores of Chapter 8 give context for the embedded benchmarks. The memory system of Chapter 4 explains one of the pitfalls.

9.1 What "faster" means

Time is the only honest yardstick. Everything else is a stand-in for it.

\[\text{performance} = \frac{1}{\text{execution time}} \qquad\qquad X \text{ is } n \text{ times faster than } Y \iff n = \frac{\text{time}(Y)}{\text{time}(X)}\]

For example, if a program takes 15 s on machine Y and 10 s on X, then X is \(15/10 = 1.5\) times faster.

Two different questions need two different measures. How long does my one job take? is latency. How many jobs does the machine finish per hour? is throughput. A laptop user cares about the first; a web server about the second. Every number in this chapter is either a time or a rate derived from one.

9.2 Rate metrics, and why they mislead

MIPS, millions of instructions per second, is the instruction count divided by (time × \(10^6\)). It is easy to quote, but an instruction on one ISA does different work from an instruction on another, and even on one machine a program of many simple instructions scores a higher MIPS figure while finishing later.

FLOPS are floating-point operations per second (MFLOPS, GFLOPS, TFLOPS). They are meaningful for number crunching, when everyone counts operations the same way, on the same algorithm.

Example 9.1: higher MIPS, slower machine

Machine A runs a job as 2.0 billion simple instructions in 1.0 s: \(2.0\times10^9/10^6 = 2\,000\) MIPS. Machine B runs the same job as 1.2 billion richer instructions in 0.8 s: \(1.2\times10^9/(0.8\times10^6) = 1\,500\) MIPS.

B has the lower MIPS, and finishes \(1.0/0.8 = 1.25\) times sooner. MIPS rewarded A for doing more, smaller steps.

9.3 Synthetic benchmarks: Dhrystone

A synthetic benchmark is a small artificial program written to imitate the mix of operations in typical software, not a real application. It is small and portable, easy to compile for any processor, even one with no operating system. But "typical" is a guess, and a small program behaves unlike a large one.

Benchmark Year Exercises Reported as
Whetstone 1972 Floating-point arithmetic and maths functions MWIPS
Dhrystone 1984 Integer arithmetic, records, string copies and compares DMIPS

Dhrystone, by Reinhold Weicker, runs one loop over and over, and the raw result is Dhrystones per second. The VAX-11/780, marketed as a "1 MIPS" machine, scored 1 757 Dhrystones per second. So the score is divided by 1 757 and called DMIPS: "how many VAXes is this?". Dividing by the clock in MHz gives DMIPS/MHz, work per clock tick, a CPI-like figure that compares designs running at different clock rates.

Example 9.2: DMIPS

A core scores 351 400 Dhrystones per second at 100 MHz. $\(\frac{351\,400}{1\,757} = 200 \text{ DMIPS}, \qquad \frac{200\ \text{DMIPS}}{100\ \text{MHz}} = 2.0\ \text{DMIPS/MHz}\)$

Dhrystone is being retired, for three reasons.

  • It is tiny. Its code and data fit in any cache, so it never feels the memory wall.
  • It is compiler-sensitive. Clever compilers can remove or simplify parts of it, and much of the score comes from string copying, which some libraries hand-optimise for exactly this test.
  • It has no enforced rules. Vendors chose their own compiler, flags and libraries, so two "DMIPS" figures are rarely comparable.

Treat a DMIPS number as a rough hint only, and only when the compiler and flags are stated.

9.4 CoreMark and Embench

CoreMark (EEMBC, 2009) is a small, free benchmark for processor cores, designed to fix Dhrystone's flaws. It runs four kinds of work:

  • lists: find and sort in linked lists (pointer chasing);
  • matrices: multiply and add (loops and arithmetic);
  • a state machine: parse input (branchy and hard to predict);
  • CRC: a checksum of every result, so that nothing can be optimised away.

Its results are computed from data that the compiler cannot see in advance, and self-checked. The run rules are strict: run for at least 10 seconds, do not change the source, and report the compiler and flags with every score. The score is iterations per second, and CoreMark/MHz divides it by the clock.

Example 9.3: CoreMark scores

4 000 iterations in 10 s on a 100 MHz core gives a score of \(4\,000/10 = 400\) and \(400/100 = 4.0\) CoreMark/MHz.

How should CoreMark/MHz be read? These are rough orders of magnitude for one core, from published results; the exact figures depend on the compiler and flags, which is the point of reporting them.

Kind of core Rough CoreMark/MHz Why
Tiny multicycle or bit-serial well below 1 Many cycles per instruction
Simple in-order microcontroller about 2 to 4 Pipelined, about one instruction per cycle
Superscalar in-order about 3 to 5 Sometimes two per cycle
Wide out-of-order about 5 to 8 and more Many per cycle, good prediction

A design with a high CoreMark/MHz may reach only a low clock rate. The chip's real speed is CoreMark/MHz × MHz: the Iron Law again.

Other embedded benchmarks:

  • Embench is a free, open suite (2019) of about 20 real embedded programs. It reports speed relative to a reference core and code size, both as geometric means, because in embedded systems memory costs money.
  • CoreMark-PRO is EEMBC's bigger sibling: several integer and floating-point workloads with larger data, closer to real applications.
  • ULPMark is EEMBC's energy benchmark for microcontrollers: how much energy a standard task costs.

9.5 Big-system benchmarks

SPEC CPU 2017, from the Standard Performance Evaluation Corporation, consists of 43 benchmarks built from real applications: compilers, compression, physics, weather, chess and more. It has four suites, integer and floating point, each measured two ways.

  • SPECspeed: one copy of each program. How quickly does one job finish? (Latency.)
  • SPECrate: many copies at once. How much work per hour? (Throughput; it fills every core.)

Each result is a ratio against a fixed reference machine's time, and the overall score is the geometric mean of the ratios. Base results use the same conservative compiler flags for all programs; peak allows tuning per program. There are strict run and reporting rules, and published, reviewed results.

The specialists each measure one thing well.

Benchmark Measures Unit Used for
LINPACK (HPL) Solving a huge dense system of linear equations FLOPS The TOP500 supercomputer list
STREAM Sustained memory bandwidth: copy, scale, add and triad over big arrays GB/s The memory wall, the roofline (Chapter 5)
MLPerf Training and inference of standard AI models Time to train, queries per second AI chips and systems
Whole-device applications Browsing, photos, games Scores Phones and laptops

LINPACK rewards floating-point units and says little about a web server; STREAM ignores computation entirely.

Question Reach for
How efficient is this small core, clock for clock? CoreMark/MHz, Embench
How big will my embedded program be? Embench code size
How fast is this server or desktop CPU on real code? SPEC CPU (speed or rate)
How many floating-point operations can this machine sustain? LINPACK (HPL)
Is my program limited by memory? STREAM, plus the roofline
How good is this AI accelerator? MLPerf
How fast is my application? Run your application: always the best benchmark

9.6 Summarising and comparing

Which mean?

For raw times of a fixed workload, the total (or the arithmetic mean) of the times is honest: it is how long the work takes. For results normalised to a reference machine, use the geometric mean: the \(n\)th root of the product of \(n\) ratios. It gives the same ranking whichever machine is chosen as the reference. The arithmetic mean of ratios does not.

Example 9.4: the arithmetic mean of ratios can contradict itself

Two machines X and Y run two programs. Times in seconds: X takes 1 s on program 1 and 10 s on program 2; Y takes 10 s and 1 s.

Reference X. Y's time ratios are \(10/1 = 10\) and \(1/10 = 0.1\). Their arithmetic mean is \((10 + 0.1)/2 = 5.05\), so Y looks about five times slower.

Reference Y. X's time ratios are \(1/10 = 0.1\) and \(10/1 = 10\). The arithmetic mean is again 5.05, so now X looks five times slower than Y.

Each machine is declared five times slower than the other. The geometric mean in either direction is \(\sqrt{10 \times 0.1} = 1\): the machines are equal on this workload, whichever is the reference.

Faster is not everything: efficiency

  • Performance per watt (CoreMark/mW, SPECrate per watt) decides battery life and data-centre electricity bills.
  • Energy per operation (picojoules per instruction). Energy is power × time, so a fast chip that finishes and sleeps can use less energy than a slow, frugal one.
  • Performance per area (CoreMark/mm²). Silicon costs money; this is how many cores fit on a die.

Example 9.5: speed against efficiency

Core X: 4.0 CoreMark/MHz at 1 000 MHz and 0.5 W gives \(4\,000\) CoreMark, and \(4\,000/0.5 = 8\,000\) CoreMark per watt. Core Y: 6.0 CoreMark/MHz at 2 000 MHz and 3 W gives \(12\,000\) CoreMark, and \(12\,000/3 = 4\,000\) CoreMark per watt.

Y is 3 times faster; X does twice the work per joule. Which is "better" depends on the product: a server or a watch.

9.7 Benchmarking honestly

Trap What happens
Benchmark-specific tuning Compilers or libraries recognise a famous benchmark and special-case it
Hidden flags The headline uses aggressive flags that your code will not
Cherry-picking Only the tests you win are quoted
Fits in cache A tiny benchmark never touches main memory, so it hides the memory wall
Unequal systems More memory, a faster disk or a higher power limit on one side
Peak against sustained A burst at turbo clock, before the chip heats up and slows down

A reporting checklist:

  1. State the exact workload and input size.
  2. State the system: core, clock (fixed or turbo), memory and cache sizes.
  3. State the software: compiler and version, every flag, the libraries.
  4. Repeat the runs; report the median and the spread; let caches and clocks warm up first.
  5. Use the right mean, and show the per-program results too.
  6. Make it reproducible: someone else should get the same number.

The best benchmark is the program you actually care about. Every other score is an estimate of it.

9.8 Summary

  • Performance is 1 ÷ time. Rate metrics such as MIPS mislead across ISAs and programs; FLOPS only for like-for-like number crunching.
  • Dhrystone gives DMIPS (divide by 1 757) and DMIPS/MHz: historic, tiny and easily gamed. CoreMark fixes it with self-checking, run rules and CoreMark/MHz; Embench adds open, real programs and code size.
  • SPEC CPU 2017 uses real applications, speed and rate, and ratios against a reference; LINPACK, STREAM and MLPerf each measure one thing.
  • Use the geometric mean for normalised results, and compare efficiency per watt, per joule and per mm² as well as speed.
  • Report workload, system, compiler and flags, and repeat runs. The best benchmark is your own program.

9.9 Exercises

Exercise 9.1. A core at 250 MHz scores 1 230 000 Dhrystones per second. What are its DMIPS and DMIPS/MHz?

Answer

\(1\,230\,000/1\,757 \approx 700\) DMIPS; \(700/250 = 2.8\) DMIPS/MHz.

Exercise 9.2. A CoreMark run completes 30 000 iterations in 15 s on a 500 MHz core. What are the score and CoreMark/MHz? Is the run valid?

Answer

Score \(= 30\,000/15 = 2\,000\); CoreMark/MHz \(= 2\,000/500 = 4.0\). The run lasted 15 s, at least the required 10 s, so the duration rule is met (the compiler and flags must still be reported).

Exercise 9.3. Why is a geometric mean used for SPEC's ratios?

Answer

The results are ratios against a reference machine. The geometric mean of ratios gives the same ranking whichever reference is chosen, whereas an arithmetic mean of ratios can flip, as Example 9.4 shows.

Exercise 9.4. Machine A takes 12 s on program 1 and 3 s on program 2; machine B takes 4 s and 9 s. Using A as the reference, compute B's time ratios and their arithmetic and geometric means. Repeat with B as the reference. What do you observe?

Answer

Reference A: B's ratios are \(4/12 = 0.333\) and \(9/3 = 3\). Arithmetic mean \(= (0.333 + 3)/2 = 1.667\); geometric mean \(= \sqrt{0.333 \times 3} = \sqrt{1} = 1\).

Reference B: A's ratios are \(12/4 = 3\) and \(3/9 = 0.333\). Arithmetic mean \(= 1.667\) again; geometric mean \(= 1\).

The arithmetic mean says each machine is 1.67 times slower than the other, a contradiction; the geometric mean says they are equal in both directions, as they are (total time is 15 s for A and 13 s for B, so with raw times B is slightly ahead).

Exercise 9.5. Core P: 3.0 CoreMark/MHz at 500 MHz, 0.2 W. Core Q: 5.0 CoreMark/MHz at 1 500 MHz, 2.5 W. Find the speed and CoreMark per watt of each, and say which suits a wearable device and which a server.

Answer

P: \(3.0 \times 500 = 1\,500\) CoreMark; \(1\,500/0.2 = 7\,500\) per watt. Q: \(5.0 \times 1\,500 = 7\,500\) CoreMark; \(7\,500/2.5 = 3\,000\) per watt. Q is 5 times faster; P does \(7\,500/3\,000 = 2.5\) times the work per joule. P suits a wearable device (battery life), Q a server (speed, within a power budget).

Exercise 9.6. Why is Dhrystone "tiny" a problem for comparing processors with different caches and memory systems?

Answer

Its code and data fit in any cache, so every processor runs it entirely out of the L1 cache and none of them experiences its memory wall. Two processors with very different memory systems can therefore post similar scores while performing very differently on real programs.

Exercise 9.7 (practical). Obtain the free CoreMark benchmark from EEMBC, build it with make (which runs it and writes run1.log and run2.log), and record the score, the compiler and flags printed in the log, and your CPU's base clock. Compute CoreMark/MHz. Rebuild with at least two optimisation levels (for example -O0 and -O3), three runs each. Pool results from your group's machines into a table with a geometric mean normalised to one reference machine.

Answer

A valid run lasts at least 10 s; CoreMark says so in its log if it did not. -O0 is typically several times slower than -O2 or -O3 on the same machine: the compiler is part of the result. Your geometric-mean ranking should not change when you pick a different reference machine; an arithmetic mean of ratios might.

Exercise 9.8. Your CoreMark score went up 40% between two runs. Before believing it, what three things do you check?

Answer

That the compiler and flags were the same; that the machine's clock really was the same (turbo, power mode, temperature, other programs running); and that both runs were valid (at least 10 s) and repeated. A 40% jump from one run is more likely a difference in the set-up than a real improvement.