Author: Guillaume Lessard / qector.store
Series: QECTOR Decoder v3 companion notes, Post 17
Source: QECTOR Decoder v3 Reference Manual v1.0.0
Date: August 2026
DOI: 10.5281/zenodo.21941046
Tags: reproducible research, QEC benchmarks, artifacts, latency, logical error rate
Abstract
A quantum-decoder benchmark mixes a mathematical problem, a stochastic workload, compiled code, and real hardware. A number without that context is not portable evidence. The QECTOR v1.0.0 manual defines the metadata, statistical, memory, and claim rules that turn a local run into a reviewable artifact. This post turns those rules into a practical checklist for research teams, systems engineers, and product reviews.
1. Start with a scoped sentence
Write the claim before writing the benchmark script. A safe claim looks like:
On this checked-in artifact, for this code, noise model, seed, batch shape, and environment, decoder A produced this measured result.
An unsafe claim looks like:
Decoder A is universally faster or always more accurate.
The manual excludes the second style. It also records that earlier benchmark artifacts were withdrawn after a core fingerprint change, so provenance belongs in every claim.
2. Separate correctness from performance
Answer the correctness questions before the timing questions:
Did every reachable syndrome produce H @ c == s?
Did the batch path equal the per-shot path where that is claimed?
Were observables scored rather than raw correction vectors?Only after those gates pass should the run report latency, throughput, memory, or logical-error statistics.
3. Cold path and hot path
Decoder construction covers graph creation, weight preprocessing, allocations, and setup. That is the cold path. Repeated decode() calls on a pre-built decoder with in-memory syndromes are the hot path.
Both are useful, but they answer different questions:
| Measurement | Includes | Appropriate for |
|---|---|---|
| Cold path | Build and setup | Per-job startup or service provisioning |
| Hot path | Repeated decode only | Steady-state repeated workloads |
| End to end | Input, transfers, decode, output | Application-level capacity |
Never present a hot-path number as end-to-end performance without labeling the omitted work.
4. Required benchmark metadata
The manual's artifact table requires:
code family and distance/size
rounds, checks, qubits, detectors
physical error rate and noise source
code-capacity or circuit-level tag
decoder class and all mode flags
shots, trials, warmup, and seed
metric definition
OS, CPU, RAM, Python, Rust, package, GPU, runtime
git commit or release tag
raw JSON/CSV path and SHA-256If a number depends on a GPU or driver, record the exact device and runtime. If memory is measured, say whether it is Python allocation, process RSS, native heap, or VRAM.
5. Statistical reporting
For latency, report the distribution: n, mean, median, standard deviation, minimum, maximum, p50, p90, p95, p99, and a confidence interval on the mean.
For logical error rates, report errors, shots, the LER, and a Wilson interval. With zero observed errors, do not claim a zero logical-error probability; report the interval and the sampling context.
For decoder comparisons, use the same sampled detector data, DEM settings, observable map, and noise model. A different random sample supports an independent replication, but it is not a paired head-to-head comparison unless the pairing protocol is stated.
6. Reproducible command design
A good command leaves no hidden degree of freedom:
benchmark --code rotated_surface --distance 5
--noise circuit_level --p 0.001
--decoder blossom --weighted
--shots 100000 --seed 42
--out results/run.jsonStore the command next to the output. The environment block and the commit hash are what let a reviewer tell a changed workload apart from a changed implementation.
7. Benchmark images need a disposition
The manual keeps structural figures and excludes charts tied to withdrawn or non-surviving hardware measurements. A locally generated chart is still useful, but its status must be explicit:
structural illustration
local smoke measurement
checked-in evidence artifact
published comparative resultOnly the last two should support public numerical claims, and only together with the accompanying artifact.
8. A release-review checklist
Before quoting a result:
- Pin the release or commit.
- Capture dependency versions and hardware.
- Run the local correctness and import smoke tests.
- Label hot, cold, and end-to-end timing.
- Store raw JSON/CSV and hash it.
- Attach a Wilson interval for LER.
- State the safe and unsafe wording.
- Remove customer data, proprietary circuits, and secrets from logs.
Item 8 matters for service deployments: a benchmark artifact can accidentally disclose inputs that were never meant to be public.
Takeaway
Reproducibility is not a citation added after the graph. It is the link between the claim, the command, the environment, the raw artifact, and the hash. That discipline makes a local QEC result useful to another lab and keeps product language honest.
Reference
Guillaume Lessard, QECTOR Decoder v3: Syndrome-Faithful Decoding - Foundations, Algorithms, and Architecture of a Fifteen-Backend Quantum Error Correction Engine, v1.0.0, August 2026. DOI: 10.5281/zenodo.21941046.