Your Model Didn’t Change. Your Second Code Path Did.

Your offline evaluation says 94% accuracy. Production says 87%. Nobody retrained anything. Nobody moved a threshold. The model file is byte-for-byte identical to the one you scored last month. The gap is not the model — it's the second copy of your own logic that nobody owns.

The bug class with no stack trace

Training and serving are usually two different programs, written by two different people, at two different times. The training pipeline runs offline in Python, reading a warehouse table. The serving path runs online in whatever your service is written in — Java, Go, TypeScript — and its inputs come from a request payload plus a couple of lookups.

When those two programs compute slightly different inputs for the same real-world event, you have train/serve skew. The model was fit on one distribution and is now being asked to predict on another one. It doesn't complain. It returns a value with the same confidence it always did. That's what makes this the most expensive silent failure in production machine learning: it looks exactly like a model that is merely mediocre.

Four places the second code path gets written

• Feature computation. The classic. Your offline job computes "orders in the last 30 days" from a nightly snapshot; online you compute it live from a different table. One includes the event being scored, the other doesn't. One fills missing values with 0, the other drops the row.
• Text and image preprocessing. Whitespace, unicode normalization, a tokenizer version bumped on one side only, EXIF rotation applied in one pipeline and not the other. This is where LLM applications hide their ugliest bugs, because prompts are strings and strings never fail loudly.
• Prompt assembly. Your evaluation harness concatenates system prompt, retrieved context, and user message one way. Your service template does it another way — different ordering, an extra separator token, a tool schema injected only in production. You are now evaluating a different program than the one you ship. It's the same mistake as benchmarking a CI runner on a two-second toy script and assuming the 3x holds inside a Docker-heavy pipeline.
• Post-processing. Thresholds, tie-breaks, rounding, class priors, the "if score > 0.8 route to a human" rule written in three places. The model outputs are identical and the decisions are not.

The arithmetic that makes this urgent

A mismatch on 0.3% of requests sounds like rounding error. At 10 million predictions a day, it's 30,000 inputs a day your model has never seen.

Worse, skew is biased, not random. Random noise averages out across a day. A timezone bug in a feature calculation does not — it hits one region, one customer segment, one hour of the day. So you end up with a metric that looks fine on average and is terrible for a specific group of users. That's the shape of problem that turns into a support escalation instead of a dashboard alert.

And you cannot find it by reading the model. Everything that goes wrong is upstream of it.

The fix is boring: one code path, owned by serving

The durable fix is not better documentation of two implementations. It's deleting one of them.

Make the serving path the only implementation and have your offline pipeline import it. Backfills and evaluations then call the same function your API calls, with the same version of the same library. If your training stack is Python and your service is Go, you have three honest options, ordered by how much pain you're willing to accept:

1. Share a library. Move feature and preprocessing logic into a package the offline job imports and the service calls. Slowest to build the first time, cheapest forever after.
2. Freeze a signed contract. If two teams genuinely cannot share code, define a versioned, typed schema of the exact model input — protobuf is a fine choice — plus golden test vectors. Both implementations must produce byte-identical output for those vectors before either ships a change.
3. Generate one from the other. Compile or transpile a single source of truth into both runtimes. This works well for tokenizers and feature transforms; it works badly for anything with business logic hiding inside it.

What you should not do is the current default: two implementations, one Slack thread, and a promise to keep them in sync.

Prove parity with a differential test, not a code review

"Same logic, different language" is unfalsifiable until you run it. So run it.

Keep a golden set: a few thousand real, raw production payloads — the request as it arrived, before any transformation. On every pull request that touches either path, push all of them through both and compare the resulting model inputs and final outputs. Not "mostly the same." Identical, or explicitly within a tolerance you wrote down and can defend, like 1e-9 for float vectors.

Set the failure budget at zero mismatches. A 99% pass rate on a parity test is a test that tells you nothing. If you can't get to zero yet, log the delta on live traffic in shadow mode and rank the mismatches by how much they would move a decision — that's a triage list you can work in order of business impact.

Two details people skip:

• Version the golden set alongside the model card. When the parity test fails a year from now, someone needs to know which inputs it was supposed to protect.
• Treat any edit to the serving path as a model release. If you change the preprocessing code, you have changed your model's behavior even if the weights file hashes identically. Re-run the full evaluation set, not just unit tests.

The uncomfortable corollary: a moving artifact

Everything above assumes you have one frozen model file to compare against. That assumption is starting to wobble. Projects like mini-AGI — a continual-learning model that trains on a single stream of data and pages its weights off disk, so the parameter count is bounded by free storage rather than VRAM — make a design choice worth quoting: it reads through exactly the same code path it serves on, and "reading and being trained are the same event."

That's elegant, and it removes skew at the source. It also means there is no frozen artifact to diff. So your parity test has to grow a third axis: not just "does the offline path match the online path," but "does the same request produce the same answer when it is not also a training step." That's a different and harder test. Replay a captured request through the serving path with learning disabled and assert the output is stable. If you run anything that updates in production, you need that check before the feature ships, not after a user notices the answers drifting.

A 30-minute audit you can run this week

1. Pick your highest-traffic model. Write down where its input gets computed. If the answer is "in two places," you have found your project.
2. Grab 500 raw production requests. Run them through both paths. Count the exact mismatches. That number is your baseline — announce it.
3. For every mismatch, name the rule that differs. Rules are fixable. "The pipelines are just different" is not a diagnosis.
4. Choose one of the three options above and put an owner on it. Sharing a library is usually right. Picking anything with a name in it beats picking nothing.
5. Add the differential test to CI with a zero-mismatch gate before you fix everything, so the count can only go down.

None of this is glamorous. It's plumbing. But the alternative is a system where your evaluation numbers describe a program you don't run, and your production numbers describe a model you never trained. Pick which one you'd rather explain to a customer.

Key Takeaways

• Train/serve skew is a silent failure: the model is fine, but the inputs it receives in production are not the inputs it learned from.
• The four usual culprits are feature computation, preprocessing, prompt assembly, and post-processing rules — every place a rule gets written twice.
• A 0.3% mismatch rate becomes 30,000 wrong inputs a day at 10M predictions, and skew is biased toward specific segments rather than spread evenly.
• Fix it by deleting an implementation: share a library, freeze a signed contract with golden vectors, or generate both paths from one source.
• Gate CI on a differential test with zero mismatches against a versioned golden set; a 99% pass rate means nothing.
• Treat serving-path edits as model releases — the weights hash is not the only thing that determines behavior.
• If your model keeps learning in production, add a third parity axis: replay captures with learning disabled and assert the answer is stable.