Lesson 45: Comparative Study, Benchmarking and Capstone Project¶
45.1 Learning Outcomes¶
By the end of this hour a student will be able to:
- Compare all major frameworks against a structured feature matrix.
- Define and compute the standard hardware benchmarks: quantum volume, CLOPS, EPLG, algorithmic qubits.
- Design a fair cross-platform comparison and avoid the standard pitfalls.
- Use OpenQASM 3 for portability between frameworks.
- Plan and execute the capstone project for this course.
45.2 Framework Feature Matrix¶
| Feature | Qiskit | Cirq | PennyLane | Q# | Braket | TFQ | pytket |
|---|---|---|---|---|---|---|---|
| Language | Python | Python | Python | Q# | Python | Python | Python/C++ |
| Autodiff | partial | no | yes | no | no | yes | no |
| ML integration | Qiskit ML | via TFQ | all frameworks | no | no | TF only | no |
| Hardware access | IBM | many (plugins) | Azure | many | many | ||
| Simulator quality | good | good | good | good | excellent (SV1) | excellent (qsim) | via others |
| Compiler quality | good | good | basic | good | basic | basic | best |
| Noise modelling | excellent | good | good | good | good (DM1) | basic | via others |
| Error mitigation | excellent | manual | some | some | some | no | some |
| Resource estimation | basic | no | no | excellent | no | no | some |
| Pulse control | yes | yes | no | no | no | no | no |
| Documentation | excellent | good | excellent | good | good | fair | good |
| Community size | largest | large | large | medium | medium | small | medium |
| Active development | very | active | very | active | active | slowed | active |
The pragmatic conclusions.
45.3 Hardware Benchmarks¶
Quantum Volume (IBM, 2019)¶
Definition. The largest \(2^{n}\) such that the device can run a random square circuit on \(n\) qubits with depth \(n\) and achieve a heavy-output probability above \(2/3\) with \(97.5\%\) confidence.
A "square" circuit has depth equal to width, so QV probes qubit count, gate fidelity, connectivity and compiler quality together in a single number.
| Device | Quantum Volume | \(n_{\max}\) |
|---|---|---|
| IBM Falcon r5 | 128 | 7 |
| Quantinuum H2 | \(2^{20} \approx 10^{6}\) | 20 |
| IonQ Forte | \(2^{22}\) (claimed) | 22 |
Note what QV is not. Quantinuum's H2 has ~56 qubits but a QV of \(2^{20}\), while IBM's Condor has 1121 qubits and a far lower QV. Qubit count alone is nearly meaningless; QV captures fidelity and connectivity too. It saturates for large devices, though, which is why newer metrics were introduced.
CLOPS — Circuit Layer Operations Per Second¶
with \(M\) templates, \(K\) parameter updates, \(S\) shots, \(D\) layers. CLOPS measures speed, which QV ignores entirely — critical for variational algorithms that run thousands of circuits.
Typical: IBM Eagle ≈ 2 000–5 000 CLOPS; trapped-ion devices are orders of magnitude lower because of slow gates.
EPLG — Error Per Layered Gate¶
derived from layer fidelity on a chain of \(n\) qubits. IBM introduced EPLG because QV saturates: EPLG scales to 100+ qubits and isolates the two-qubit error that actually limits circuits.
Algorithmic Qubits (IonQ)¶
Uses real algorithms (Bernstein–Vazirani, QFT, Grover, phase estimation, VQE, QAOA) rather than random circuits. More application-relevant than QV, but vendor-defined — treat cross-vendor comparisons with care.
Choosing a benchmark¶
| Question | Metric |
|---|---|
| Can it run deep circuits? | Quantum Volume, EPLG |
| Can it run many circuits quickly? | CLOPS |
| Will my algorithm work? | Algorithmic Qubits, or just run it |
| How good are the two-qubit gates? | EPLG, gate error from calibration |
| How many qubits? | qubit count — the least informative metric |
45.4 Designing a Fair Benchmark¶
The five pitfalls¶
| Pitfall | Why it invalidates the comparison | Fix |
|---|---|---|
| Different compilers | you are benchmarking compilers, not hardware | compile all with the same tool (e.g. TKET) |
| Different shot counts | statistical error differs | fix shots across devices |
| Hardware-native problems | favours whichever device matches the problem graph | use both native and random topologies |
| Anecdotal timing | ignores queue, compile and readout time | report end-to-end wall clock |
| One problem size | proves nothing about scaling | vary \(n\) and report the trend |
A defensible protocol¶
1. Choose a benchmark SUITE (QASMBench, MQT Bench, or a task-relevant set)
2. Compile EVERY circuit with the SAME compiler (pytket) to each device's native gate set
3. Fix shots (e.g. 4096) across all devices
4. Run each configuration >= 3 times on different days (hardware drifts daily)
5. Report:
- success metric with error bars
- two-qubit gate count after compilation
- end-to-end wall-clock time
- cost
6. Publish the code and the raw data
Step 4 is the one most often skipped. Superconducting device calibration changes daily; a single-day measurement can differ by 20–30 % from the weekly mean.
45.5 Portability via OpenQASM 3¶
# Qiskit -> QASM
from qiskit import qasm3
qasm_str = qasm3.dumps(qiskit_circuit)
# QASM -> Cirq
from cirq.contrib.qasm_import import circuit_from_qasm
cirq_circuit = circuit_from_qasm(qasm_str)
# QASM -> pytket
from pytket.qasm import circuit_from_qasm_str
tk = circuit_from_qasm_str(qasm_str)
# Braket accepts QASM directly
from braket.ir.openqasm import Program
device.run(Program(source=qasm_str), shots=1000)
Portability caveats to check every time.
- Endianness. Qiskit is little-endian; Cirq and Q# are big-endian. Always test with an asymmetric state (L37, L39).
- Gate-set coverage. Exotic gates may not round-trip; decompose to a common set first.
- Classical control. QASM 3 supports it; QASM 2 does not.
- Custom gates. Definitions may be inlined or lost.
Always verify a round trip on a small circuit before trusting a large one.
45.6 The Capstone Project¶
Brief¶
Implement a quantum algorithm from Units I–III on at least two platforms, benchmark it honestly, and report your findings.
Deliverables.
- Code — reproducible, on two or more frameworks, in a public repository.
- Report (8–12 pages) — algorithm, implementation, results, honest analysis.
- Presentation (15 minutes) with a live demonstration.
Suggested projects¶
| # | Project | Units drawn on | Difficulty |
|---|---|---|---|
| 1 | Grover's search with a real predicate (e.g. Sudoku) | I, II | medium |
| 2 | Shor factoring 15, 21, 35 with full continued fractions | II | medium |
| 3 | QAOA on Max-Cut, hardware vs simulator | III | medium |
| 4 | VQE for \(\mathrm{H}_2\) / LiH bond dissociation curve | III | hard |
| 5 | QSVM vs classical SVM on a real dataset | III | medium |
| 6 | Quantum teleportation and superdense coding on hardware | I | easy |
| 7 | 3-qubit error-correcting code with injected errors | I | medium |
| 8 | Barren-plateau study: gradient variance vs \(n\) | III | medium |
| 9 | Cross-platform benchmark of one algorithm on 3 devices | IV | medium |
| 10 | Resource estimation for a chosen algorithm in Q# | IV | hard |
Marking rubric¶
| Criterion | Marks |
|---|---|
| Correct implementation (verified against theory) | 25 |
| Multi-platform implementation and comparison | 15 |
| Quality of analysis (error budget, statistics, scaling) | 20 |
| Honest assessment of limitations and classical baselines | 15 |
| Report clarity and structure | 15 |
| Presentation and demonstration | 10 |
| Total | 100 |
Note the 15 marks for honesty. A project reporting "our QSVM achieved 91 % accuracy versus scikit-learn's 97 %, and here is why" scores higher than one claiming quantum advantage without a classical baseline. Every quantum result must be reported alongside the best classical alternative. This is the single most important professional habit this course can instil.
45.7 Solved Problems¶
Problem 1 — Computing quantum volume¶
A device passes the QV test at \(n=6\) but fails at \(n=7\). State its quantum volume. If gate errors improve 2×, estimate the new QV.
Solution.
Estimating the improvement. A QV circuit on \(n\) qubits has depth \(n\) and roughly \(n^2/2\) two-qubit gates (each layer pairs \(n/2\) qubits, over \(n\) layers):
The heavy-output test needs circuit fidelity roughly above \(2/3\):
Calibrating from the observed data. \(n_{\max} = 6\) gives
After a 2× improvement, \(\epsilon' = 0.01125\):
Why the leverage is so high. \(n_{\max}\propto\epsilon^{-1/2}\) while \(V_Q = 2^{n_{\max}}\), so \(V_Q\) grows exponentially in \(\epsilon^{-1/2}\). This is why vendors invest so heavily in gate fidelity rather than qubit count — and why a 1000-qubit device with mediocre gates has a lower QV than a 20-qubit device with excellent ones.
Problem 2 — Designing a fair three-platform comparison¶
Compare Grover's algorithm on IBM, IonQ and a simulator. Specify the protocol completely and predict the results.
Solution.
Algorithm. Grover on \(n = 3,4,5\) qubits, single marked item, \(k = \lfloor\frac{\pi}{4}\sqrt{2^n}\rfloor\) iterations.
Metric. \(\Pr[\text{marked}]\), compared against the theoretical \(\sin^2\big((2k+1)\theta\big)\) (L22).
The protocol, with each choice justified.
| Step | Choice | Rationale |
|---|---|---|
| Circuit construction | Qiskit | one source of truth |
| Compilation | pytket for all three | removes the compiler as a confound |
| Shots | 4096 everywhere | equal statistical error |
| Repetitions | 3 runs on 3 different days | captures calibration drift |
| Devices | ibm_brisbane, IonQ Aria-1, Aer simulator |
two modalities plus the ideal reference |
| Reporting | mean ± std dev, plus CNOT count, wall clock and cost | complete picture |
Predicted results.
| \(n\) | \(k\) | Theory | Simulator | IonQ | IBM |
|---|---|---|---|---|---|
| 3 | 2 | 0.945 | 0.945 | 0.89 ± 0.02 | 0.72 ± 0.05 |
| 4 | 3 | 0.961 | 0.961 | 0.81 ± 0.03 | 0.48 ± 0.07 |
| 5 | 4 | 0.999 | 0.999 | 0.68 ± 0.04 | 0.22 ± 0.08 |
Reasoning behind the prediction.
- The simulator matches theory exactly — this is the correctness check. If it does not, the implementation is wrong, and no hardware result means anything.
- IonQ degrades gracefully. The multi-controlled \(Z\) in the oracle and diffuser needs all-to-all connectivity; IonQ has it, so no SWAPs. Its 0.4 % two-qubit error compounds slowly.
- IBM degrades sharply. Heavy-hex connectivity forces SWAPs for the multi-controlled gates. For \(n=5\) the \(C^4Z\) decomposition plus routing can exceed 60 CNOTs; at \(\epsilon = 8\times10^{-3}\) that gives \(F\approx0.62\), and the signal degrades further from readout error.
- Error bars widen with \(n\) because deeper circuits are more sensitive to daily calibration drift.
What to report alongside.
| Device | \(n=5\) CNOTs | Wall clock (incl. queue) | Cost |
|---|---|---|---|
| Simulator | — | 2 s | $0 |
| IonQ | 44 | 25 min | $369 |
| IBM | 68 | 3 h (queue) | $0 (free tier) |
The honest conclusion. "IonQ's all-to-all connectivity gives markedly better Grover fidelity at every size, at ~85× the cost per shot and comparable wall-clock time once queueing is included. Neither device is close to the noiseless result at \(n=5\). Classical search over \(2^5 = 32\) items is instantaneous."
Problem 3 — Benchmarking a variational algorithm¶
Design a benchmark of a 6-qubit VQE across simulator, IBM and IonQ. What must be reported?
Solution.
System. \(\mathrm{H}_2\) in the STO-3G basis, mapped by Jordan–Wigner to 4 qubits (or 6 with a larger active space). Exact ground-state energy known classically — essential, because it gives ground truth.
Protocol.
- Classical reference. Compute the FCI (full configuration interaction) energy exactly. This is the target.
- Simulator, noiseless. Run VQE with the UCCSD ansatz. This validates the implementation. If it does not converge to the FCI energy, stop and debug — do not proceed to hardware.
- Simulator with a device noise model (
AerSimulator.from_backend). Predicts the hardware result before spending money. - Hardware, parameters fixed. Do not optimise on hardware. Take the optimal \(\vec\theta\) from step 2 and simply evaluate the energy. This isolates hardware noise from optimiser behaviour and is far cheaper.
- Hardware with error mitigation (readout + ZNE, L38).
What must be reported.
| Quantity | Why |
|---|---|
| Absolute energy error vs FCI (in Hartree) | the only meaningful accuracy measure |
| Chemical accuracy achieved? (\(<1.6\ \mathrm{mHa}\)) | the field's standard threshold |
| Circuit depth and CNOT count after compilation | explains the error |
| Shots used and total wall clock | the real cost |
| Energy with and without mitigation | quantifies mitigation's value |
| The classical cost of the same calculation | the honest baseline |
Predicted results.
| Configuration | Energy error (Ha) | Chemical accuracy? |
|---|---|---|
| Exact (FCI) | 0 | — |
| Simulator, noiseless | \(10^{-8}\) | ✓ |
| Simulator + noise model | 0.031 | ✗ |
| IBM raw | 0.048 | ✗ |
| IBM + readout mitigation | 0.021 | ✗ |
| IBM + readout + ZNE | 0.009 | ✗ |
| IonQ + mitigation | 0.004 | ✗ (close) |
The essential final line of the report. "Classical FCI solves this system exactly in \(<1\) millisecond on a laptop. The quantum computation took 4 hours and 12 million shots to reach 0.004 Ha — 2.5× worse than chemical accuracy. This experiment validates the implementation and quantifies current hardware limitations; it does not demonstrate any advantage."
Problem 4 — Interpreting vendor benchmark claims¶
Three vendors publish these claims. Assess each critically.
(a) "Our device has 1121 qubits — the world's largest." (b) "We achieved quantum volume \(2^{20}\)." (c) "Our processor performed a calculation in 200 seconds that would take a supercomputer 10 000 years."
Solution.
(a) 1121 qubits (IBM Condor).
Assessment: nearly meaningless on its own. Qubit count says nothing about whether useful circuits can run. The questions to ask:
- What is the two-qubit gate error? (If \(10^{-2}\), a 100-CNOT circuit has \(F = 0.37\).)
- What is the quantum volume? (Condor's is far below Quantinuum's 56-qubit H2.)
- What is the connectivity? (Heavy-hex, degree ~2.5 — SWAP-heavy.)
- Has any algorithm been run on all 1121 qubits? (No.)
Verdict: an important engineering milestone for fabrication and control, not a computational capability claim. IBM itself de-emphasised raw qubit count in favour of EPLG for exactly this reason.
(b) Quantum volume \(2^{20}\) (Quantinuum H2).
Assessment: a strong and meaningful claim. QV is a well-defined, published protocol that simultaneously tests fidelity, connectivity and compilation. \(2^{20}\) means random square circuits on 20 qubits at depth 20 succeed — genuinely hard.
Still worth checking:
- Was the protocol followed exactly (heavy-output > 2/3 at 97.5 % confidence)?
- Was the compilation heavily specialised for the QV circuits? (QV permits arbitrary compilation, which is a known and much-debated loophole — the metric rewards compiler effort as well as hardware.)
- What is the CLOPS? Trapped ions have excellent QV but slow gates; for variational algorithms needing \(10^{6}\) circuits, speed may matter more than depth.
Verdict: credible and informative, with the compilation caveat noted.
(c) "200 seconds versus 10 000 years" (Google Sycamore, 2019).
Assessment: the most contested claim in the field, and instructive precisely for that reason.
- The task was random circuit sampling — deliberately constructed to be hard classically and with no known application. It is a proof of principle, not a useful computation.
- The classical estimate was contested immediately. IBM argued that using disk storage rather than RAM reduces it to ~2.5 days. Later tensor-network improvements (Pan, Chen, Zhang 2021) reproduced the samples in hours on a GPU cluster.
- The comparison target moved. As classical algorithms improved, the claimed advantage shrank by many orders of magnitude — which is the normal and healthy scientific process.
- The physics result stands. Sycamore did perform a coherent 53-qubit computation with a measured fidelity of \(0.2\%\) — a genuine and impressive experimental achievement.
Verdict: a real experimental milestone, overstated as a computational claim. The lesson for students: any "quantum supremacy" claim is a claim about the current best classical algorithm, and classical algorithms improve.
The three questions to ask of any quantum advantage claim.
Problem 5 — A complete capstone project plan¶
Plan Project 3 (QAOA Max-Cut, hardware vs simulator) in full: timeline, milestones, deliverables and the risks.
Solution.
Objective. Implement QAOA for Max-Cut on Erdős–Rényi graphs of 4–8 nodes; compare \(p=1,2,3\) on a simulator, a noisy simulator and real hardware; benchmark against Goemans–Williamson.
Week-by-week plan.
| Week | Task | Deliverable |
|---|---|---|
| 1 | Literature: Farhi et al. 2014; GW 1995; Bravyi et al. 2020 | annotated bibliography |
| 2 | Implement QAOA in Qiskit; brute-force optimum for verification | working code, \(n\le8\) |
| 3 | Classical parameter optimisation on the simulator; grid + COBYLA | optimal \((\gamma,\beta)\) tables |
| 4 | Implement classical baselines: GW SDP, greedy, random | baseline results |
| 5 | Port to PennyLane; verify identical results | second implementation ✓ |
| 6 | Noisy simulation with FakeSherbrooke; predict hardware |
noise study |
| 7 | Compile with pytket; measure CNOT reduction | compilation study |
| 8 | Run on IBM hardware (\(p=1,2\); \(p=3\) if depth allows) | raw hardware data |
| 9 | Apply error mitigation; ablate its components | mitigated results |
| 10 | Analysis, plots, error budget | figures |
| 11 | Write the report | draft |
| 12 | Revise; prepare presentation | final submission |
Key figures the report must contain.
- Approximation ratio versus \(p\), for simulator / noisy simulator / hardware / GW.
- Approximation ratio versus graph size \(n\).
- \(\langle C\rangle\) heat map over \((\gamma,\beta)\) — the QAOA energy landscape.
- CNOT count before and after pytket compilation.
- Error budget: how much of the gap from ideal is due to CNOT error, readout error and depth.
Predicted headline results.
| Method | Approximation ratio |
|---|---|
| Random assignment | 0.50 |
| Greedy | ~0.65 |
| QAOA \(p=1\), simulator | ~0.75 |
| QAOA \(p=3\), simulator | ~0.88 |
| Goemans–Williamson | ~0.92 |
| QAOA \(p=1\), hardware | ~0.68 |
| QAOA \(p=3\), hardware | ~0.55 (noise dominates) |
The central finding to expect and to state plainly. Increasing \(p\) improves the simulated result monotonically but degrades the hardware result beyond \(p\approx2\), because the added depth costs more in noise than it gains in expressivity. There is an optimal \(p\) for a given noise level — a genuinely interesting and reportable result.
Risks and mitigations.
| Risk | Mitigation |
|---|---|
| Hardware queue delays | start hardware runs in week 8, not week 11; use FakeBackend meanwhile |
| Optimisation fails to converge | use parameter transfer from small instances (L34, Problem 5) |
| Results too noisy to interpret | reduce to \(n=4\), \(p=1\) — a clean result beats an ambitious mess |
| Scope creep | fix \(n\le8\) and \(p\le3\) in week 1 and do not expand |
The report's concluding paragraph, drafted in advance.
"QAOA was implemented on two frameworks and validated against brute-force optima. On a noiseless simulator the approximation ratio improves monotonically with \(p\), reaching 0.88 at \(p=3\) — still below the classical Goemans–Williamson guarantee of 0.878 on these instances. On hardware, performance peaks at \(p=2\) and degrades thereafter as circuit depth outpaces coherence. Error mitigation recovered approximately 40 % of the noise-induced gap. No quantum advantage was observed or expected at this scale; the contribution of this work is a quantified characterisation of where hardware noise overtakes algorithmic improvement."
45.8 Practice Problems¶
- A device passes QV at \(n=8\). What is \(V_Q\), and what error rate does the estimate imply? [Ans: 256; \(\epsilon\approx0.0127\)]
- Name three flaws in "our quantum computer solved it in 1 second".
- Which metric best predicts VQE performance: QV, CLOPS or qubit count? Justify.
- Convert a Qiskit circuit to QASM 3 and import it into pytket.
- Why must all devices in a benchmark use the same compiler?
- Design a fair comparison of two error-mitigation techniques.
- What are the three questions to ask of any quantum advantage claim?
45.9 Summary and Course Conclusion¶
Framework selection¶
Benchmarks¶
- Quantum Volume tests depth, fidelity and connectivity jointly; \(n_{\max}\propto\epsilon^{-1/2}\), so halving the error quadruples \(V_Q\).
- CLOPS measures speed (critical for variational algorithms); EPLG scales beyond QV's saturation; Algorithmic Qubits is application-oriented but vendor-defined.
- Qubit count is the least informative metric.
Fair comparison¶
Same compiler, same shots, multiple days, multiple sizes, end-to-end timing, published data.
What you have learned in this course¶
| Unit | Capability |
|---|---|
| I | Manipulate qubits, gates and entanglement; compute measurement statistics; explain interference, no-cloning, teleportation and error correction |
| II | Derive and analyse Deutsch–Jozsa, Bernstein–Vazirani, Simon, QFT, phase estimation, Shor, Grover and HHL; assess the quantum threat to cryptography and the post-quantum response |
| III | Encode data, build quantum kernels and variational models, compute gradients, diagnose barren plateaus, and formulate optimisation problems for QAOA and annealing |
| IV | Use the major frameworks, transpile and mitigate errors, estimate fault-tolerant resources, and benchmark honestly |
The disposition this course has tried to instil¶
Quantum computing is a real and important field with a small number of proven advantages (Shor, Grover, quantum simulation), a large number of proposed ones that remain unproven, and a substantial gap between current hardware and useful application.
The most valuable professional skill is not writing a quantum circuit. It is telling the difference between a demonstration and an advantage — and reporting that difference honestly.
Always report the classical baseline.