Your Docs Example Is a Deployment You Never Tested

You are setting up a canary rollout. You want a header to land on the new backend and not the old one, and the feature that makes it possible shipped last month. So you copy the YAML straight out of the vendor's own launch announcement, paste it into your config, and restart. The process dies on boot with field not found, node: middlewares.

Your first instinct is that you broke something. You did not. The announcement's example puts the middlewares key one level too deep — nested inside the load balancer block, next to the server list. The vendor's own source code disagrees with the vendor's own blog post. The service type has a middlewares field. The load balancer type does not. Move that one key up a level, so it sits on the service instead of inside the load balancer, and the same container starts cleanly on the first try.

That is worth sitting with. Not because a YAML key was misplaced, but because the artifact we trust most when learning a new tool — the official example — is the artifact almost nobody runs.

The Example Is a Deployment Nobody Tests

Docs examples live in a strange place. They look like code. They are typed like code, indented like code, and copied like code. But they are written as prose, reviewed as prose, and versioned like a tweet. The person writing the announcement understands the feature perfectly. They are describing its shape from memory, and their mental model is one level off. Nothing catches it, because nothing executes it.

A shipped library has a test suite that screams when the API moves. A blog post has a marketing deadline. When a field relocates, the library breaks loudly in your CI and the blog post keeps quietly lying for two years.

The part that stings is the arithmetic. Verifying that config takes twelve lines and one container. The cost of not verifying it is paid by every reader, one at a time, forever — and paid on the exact day they are deciding whether to adopt the tool.

You can also learn a lot about a project from whether its launch post parses. Not because a misplaced key predicts an outage, but because it shows where the team's attention actually goes. A project that runs its own announcement examples has a habit the prose cannot fake. A project that does not has told you which layer to double-check before you trust the next claim.

The Error Message That Saved Your Week

Before you write off the whole experience, notice what actually happened. The proxy refused to start. It did not accept the config, ignore the unknown key, and boot normally while your canary silently ran without its header.

That distinction matters more than the bug. Consider two universes:

• Strict validation: the process fails at boot, in a log line you can read, naming the exact offending node. You fix it in two minutes.
• Silent tolerance: the process starts fine. Your header never appears on the canary. You spend an afternoon staring at weights and a backend that swears it received nothing, and you eventually blame the wrong layer entirely.

Strict schema validation on configuration is not pedantry. It is the difference between a five-minute fix and a five-hour investigation. If you own a config format, reject unknown fields by default and fail at load time rather than at request time. Ask yourself which unknown keys you are currently swallowing, and what a user would see if one of them was a typo that mattered.

The same strictness appears a second time in the same feature. A weighted service child cannot carry its own middleware inline. Try it and you get the identical error, because the type representing a weighted child has no middleware field at all — only a name, a weight, headers, and a couple of status fields. The fix is not a workaround; it is the intended shape. Pull the child out into a standalone service, attach the middleware there, and reference it from the weighted list.

What You Learn Only By Running It

Fix the syntax and the real question arrives: does the middleware stay on one branch? That is the actual claim of the feature, and it is the one worth measuring rather than trusting.

The rig is small. Two tiny backends that echo back whatever headers they receive. One weighted service in front with a 9:1 split. A middleware on the canary branch that sets a single header and does nothing else. Then fire 200 sequential requests and record which backend answered and whether the header came along.

The result is reassuring. The canary backend answered 20 of 200 requests — 10.0 percent, matching the 9:1 weight — and carried the header on all 20. The stable backend answered 180 and carried it on none. No leaks in either direction. Repeat with 500 requests fired 50 at a time to rule out a race between branch selection and the middleware chain, and you get 50 and 450 with the same perfect isolation.

Now the unsettling part. Watch the raw order of which backend answers, and it is not random sampling. It is a fixed cycle, roughly one canary request in every ten, repeating in the same slot: nine stable, one canary, nine stable, one canary. Deterministic weighting is fine — arguably better, since it guarantees spread over time. But it changes how you test by hand.

Say you want to eyeball a canary deployment with five requests. Under a fixed repeating cycle, whether you see the canary depends entirely on where in the cycle you happen to start. Land those five requests in the wrong phase and you get zero canary hits and conclude, wrongly, that the branch is dead or the weight is broken. When you write the runbook step for checking a canary manually, the magic number is not five. It is enough requests to cover at least two full cycles, plus a check that the counts resemble the weight you configured.

Which Documentation Deserves a Test

Nobody is going to run an entire docs site in CI. That is a fantasy, and chasing it produces a brittle suite everyone disables. But examples are not equally valuable, and they are not equally expensive to protect. A workable split:

1. Full, copy-pasteable configurations. Launch posts, quickstarts, upgrade guides. Write them as real files, load them with the real binary in CI, and assert two things: it boots, and the behavior you promised actually happens.
2. Behavioral claims in prose. The sentence saying router middlewares run before service middlewares is testable. Assert the observable outcome — the header value that reaches the backend — not merely that the config parses.
3. Inline fragments and partial snippets. A three-line excerpt that was never meant to run. Either promote it to a full example or accept that it is commentary and label it as such. Do not pretend a fragment is copy-pasteable.

The tiering matters because the cost of being wrong is not evenly distributed. A broken snippet for a rarely used parameter costs one person one afternoon. A broken example in the post announcing your headline feature costs every reader, at the moment they are deciding whether to bring you into their stack.

It is also a cheap, high-signal way to evaluate a tool you are considering. Take the quickstart, run it exactly as written against the version they tell you to install, and count how many edits it takes to get a working result. Three edits means expect rough edges. Zero edits means somebody over there is doing the boring work, and the rest of the product probably reflects it.

The Trade-offs, Honestly

Documentation tests rot too. They pin a version, and the moment the product moves, the test fails for reasons unrelated to your code. That is a real maintenance tax, and it is why teams abandon the practice.

The honest answer is to treat it like any other flaky-but-valuable test. Pin the exact version the example claims to target. Keep the tests small — one container, one config file, a dozen lines of CI. When they fail on an upgrade, that failure is the point: it tells you the docs are now wrong for the version people are installing today.

Then there is the uncomfortable version of this lesson, the one aimed at your own repository. Your README example is a deployment nobody tests. Your onboarding doc is a config nobody runs. The commands you paste into a pull request comment are prose that will outlive the API they were written against. The vendor who misled you shipped exactly the mistake you are probably shipping right now, one level up the stack.

The fix is not a documentation culture. It is one small habit: if a human is expected to paste it, a machine should have run it first.

Key Takeaways

• Copy-pasteable examples are code. If nothing runs them, they are fiction with indentation.
• When an official example fails, the cause is usually structural — a key at the wrong level — not a typo on your side.
• Strict config validation is a feature. Failing at boot and naming the offending node beats a silent no-op discovered in production.
• Weighted routing is often deterministic, not random. Test with enough requests to cover two full cycles, not five.
• Prioritize by cost of being wrong: full configs get golden tests, behavioral claims get assertions, fragments get labelled as fragments.
• Doc tests age badly. Pin the version they target and accept the maintenance cost as the price of not lying to every reader.