Your Benchmark Is Wrong Until You Can Make Your Harness Fail on Purpose

A clean table that took six tries

A measurement run comes back with a tidy table: six cells, six numbers, all green. The pacing guard failed on the first attempt. It failed again on the second. On the sixth attempt it passed, and the table shipped. Nothing in the output recorded that it took six runs to get one clean one.

That single omission is the most expensive bug class in engineering right now, because it is invisible from the outside. If your checkout page returns the wrong price, a customer complains within the hour. If your benchmark returns 1.02 instead of 1.00, it goes into a blog post, a sales deck, and an architecture decision, and gets quoted back at you for two years. A benchmark harness is the one piece of software whose output nobody can independently check. That is why it stays wrong.

Here are the recurring shapes of harness defects, the ones that show up in everything from a two-hour load test to a 4,000-run LLM evaluation, plus a checklist you can run in a day. The examples come from a recent investigation into Server-Sent Events (SSE) buffering, where a developer built a rig to test whether reverse proxies hold back streamed tokens, then found fourteen things wrong with the rig before it produced a single trustworthy number.

Defect 1: retrying until the guard passes

Re-running a measurement until your sanity check goes green is not persistence. It is selection bias with extra steps. You are selecting for the moments when the machine happened to be quiet.

The numbers are brutal. On one host, six cells looked publishable after six attempts. With retries removed — exactly one honest attempt per cell — one of six cells was measurable at all. The other five were not. Six tries had converted "this laptop cannot measure this" into a table someone would have cited.

The fix is not just to stop retrying. Make the policy visible in the output: state that the suite makes exactly one attempt per cell, and label a failed cell UNMEASURABLE with the guard's own reason, instead of letting it quietly disappear.

Defect 2: a guard with a tenth of the power it claims

The rig audited the emitter's pacing after each cell. The emitter kept a send log keyed by request ID. The runner reused a single request ID across all ten runs, so each run overwrote the previous log. The guard audited the last run. The other nine went into the median unaudited.

On paper, ten runs of protection. In reality, one. This is the defect that should scare you most, because the guard reported success the entire time. An instrument that cannot show you its own coverage is decoration. Print the IDs you actually audited. Assert the count. If you instrument ten things, your output should contain ten receipts.

Defect 3: the instrument that creates the failure it reports

Two examples from the same rig, and both destroyed the cells most likely to contain the finding:

• Timestamps recorded after the write. Writes block. When a buffering proxy applied backpressure, the emitter's write blocked, and the guard logged that as emitter drift — which fails the guard and voids the cell. Read that again in terms of consequences: the buffering cells, the ones you built the rig to measure, were the cells most likely to be thrown out as instrument error. Recording the timestamp before the write measures when the emitter woke up on schedule, not when the downstream felt like accepting bytes.
• A healthcheck that poisoned the loop. The emitter's Docker healthcheck spawned a fresh Python interpreter inside the container once per second, forever, next to a loop whose entire job was millisecond-accurate pacing. The drift spikes that voided five cells were 10.44 to 10.75ms against a 10.00ms tolerance. Swap in a cheap, infrequent probe and the same host measured 0.22ms.

Obvious when written down. Invisible in a compose file. Ask one question of every probe, logger and sidecar: could this thing change the number I am about to publish?

Defect 4: the metric nobody tested

The headline number was "frames per read": SSE frames received divided by the number of read calls that delivered at least one frame. One frame alone scores 1.0. Forty-one frames in a single read scores 41.0. Everything rests on incrementing the arrival counter once per read, not once per frame.

There was no test for that. Four tests covered the client, and all four used a naturally incremental stream where per-read and per-frame produce the same answer — so a client with the increment in the wrong place passed all four.

The falsification that fixed it: build a relay that deliberately drains an entire upstream response and flushes it in one go, then assert both sides of the contrast. Buffered measured 21.00, direct measured 1.00. Then move the increment inside the per-frame loop and confirm the test now fails. That test is pinned in git. The version run by hand before committing proved nothing to anyone but its author.

Defect 5: the silent undercount

The frame counter looked for blank-line terminators across read boundaries, keeping a leftover tail between calls. It counted with a non-overlapping scan and trimmed the tail with a rightmost search. Those two can disagree about which bytes a terminator occupied. Given three or more consecutive newlines split across a read, the tail got trimmed past a newline the counter had not consumed, and frames were quietly absent from the count that every published number derived from.

The check that catches this is boring and decisive: a fuzz test over a two-character alphabet of newline and X — 20,000 trials — asserting that the split total always equals the whole-string total. Zero mismatches after the fix, and it had passed the bug before it.

What a harness you actually trust then finds

Once the rig held up, the findings were sharp. Four proxies, pinned to exact patch versions, in front of the same emitter sending a frame every 50ms:

• Direct, nginx 1.31.5, Caddy 2.11.4, Traefik v3.7.13: 2 to 3ms to first token, frames arriving 50ms apart, about 1.02 frames per read.
• HAProxy 3.4.4 on a stock config: 206ms to first token, zero gap between frames, 5.12 frames per read. Bursts of five, a fifth of a second late.

Then the part that matters more than the headline: change the frame size from about 60 bytes to about 1.1KB and HAProxy's behaviour changes completely — 53ms to first token and 1.46 frames per read. Slow the emit rate and the coalescing fades: 13.67 frames per read at a 5ms interval, 5.12 at 50ms, 2.05 at 200ms. The measurement is a property of the stream through the proxy, not a property of the proxy. Quoting "HAProxy delivers five events per read" as a fact about HAProxy would be wrong.

Two more numbers that keep you honest. The direct, unproxied path measures 1.02 frames per read — that is the noise floor, one merged read in 41 is the client and the kernel, and anyone reporting 1.02 as distinct from 1.00 is reading noise. And the same host throws isolated 30 to 45ms scheduling spikes with nothing in the path at all, so no ~40ms first-token stall can be pinned on a proxy from that rig. The author nearly published one.

Also worth knowing: X-Accel-Buffering: no — the header everyone reaches for to stop SSE buffering — changed nothing through HAProxy. 214ms against 206ms, ratio identical. It is an nginx convention, and nginx was not the proxy doing the buffering. Which is the whole point of measuring instead of repeating advice.

The five-rule harness checklist

1. One attempt per cell, stated in the output. No silent retries. Label failures UNMEASURABLE with the reason.
2. Falsify the metric before you trust it. Build an artificial input that must produce an extreme value, and assert both ends of the contrast.
3. Give every instrument a receipt. Print the IDs it audited and assert the count. Coverage you cannot see is coverage you do not have.
4. Hunt probes that perturb the loop. Every logger, healthcheck and sidecar inside the measured path is a suspect.
5. Size the payload realistically. Token streams are small frames arriving steadily. Benchmark with 1KB chunks and you will measure the wrong thing and never know.

The honest trade-offs

This discipline costs time you may not have. Single-attempt measurement means some afternoons produce no publishable number at all, and that is a hard sell when someone is waiting for a chart. Falsification tests can eat a full day for one metric. And on cheap hardware, the noise floor can be larger than the effect you are trying to detect — in which case the correct output is "we cannot measure this here", which is a real finding but not the one anyone wanted.

So scope it. If the number will change an architecture decision, a purchase, or a public claim, do all five. If it is a quick sanity check nobody will quote, skip the ceremony.

Why this is also a hiring signal

Anyone can run a benchmark. Very few engineers can tell you how their benchmark could be wrong, and fewer still volunteer it before you ask. In interviews, "what would falsify this result?" is a better question than any algorithm puzzle, because it cannot be answered by memorisation and it predicts what happens when that person owns your release metrics. If you are building a team — in-house or near-shore — hire for the instinct to distrust the instrument, not the instinct to defend the chart.

Key Takeaways

• Benchmark harnesses are the only software whose output no one can independently verify, which is exactly why they stay wrong.
• Retrying until the guard passes turned six publishable cells into one honest measurement. Make the single-attempt policy visible in the output.
• A guard that audited one run of ten looked like ten runs of protection. Print receipts and assert coverage counts.
• Instruments that perturb the loop void the cells most likely to contain your finding — check every probe inside the measured path.
• Falsify every metric with an artificial input before you publish it. The frame-counting metric passed four tests with the core logic inverted.
• A measured result belongs to the stream, not the tool: HAProxy looks catastrophic at 60-byte frames and mild at 1.1KB.
• Hire people who ask what would make the number wrong.