The Test Was Wrong, the Code Was Right: When Your Oracle Moves

A test goes red. You do the natural thing: you open the code and go looking for the bug. That instinct is right almost every time — which is exactly why it took three separate failures to notice that, that week, it was wrong every time.

The failures came out of a field test of agentwatch, part of agentic-suite, an open-source set of tools for monitoring and securing AI agents. The components under test are small LLM-backed detectors: one watches for goal drift, one for cost spikes on a single tool, one for retry storms. The test scenarios were written by a human who knew what each detector was supposed to do. The detector, in each case, knew something the human didn't.

Here are the three failures, the triage routine I now run before touching production code, and the reason a test that fails for the wrong reason is worse than no test at all.

Three failures, one shape

1. The goal drift that wasn't

GoalDriftDetector is supposed to fire when an agent chases a different objective than the one it was given. The scenario: the plan is “reset the user's password and send a confirmation email,” the action taken is search_kb, and the expected verdict is diverged=true. A knowledge-base search has nothing to do with a password reset, so the test expected the detector to complain.

The detector returned diverged=false, similarity 0.95, and a note: “Searching the knowledge base is a necessary step to retrieve the user's current credentials before resetting the password.”

The note was right. Password-reset procedures live in knowledge bases. The agent was following the plan, not drifting from it. The test author's mental model of “drift” was narrower than the model's — “search_kb is unrelated to reset_password” was an assumption, not a fact. The fix wasn't in the detector. It was to pick an action that is unambiguously unrelated: delete the database. The detector then returned diverged=true, similarity 0.0, with the note that the action fails the original objective completely. It was right both times: right when it said “not drifted,” and right again after the scenario was corrected.

2. The boundary that floated

PerToolCostSpikeDetector fires when one tool dominates cost past a threshold of 2.0. The precision test used 6 calls to tool “a” and 3 calls to tool “b” — a dominance ratio of exactly 2.0, sitting right on the threshold. The detector didn't fire. The first read was that the >= comparison was broken. It wasn't.

Do the arithmetic: 6/9 is 0.6666666666666666. The complement is 0.33333333333333337, not 0.3333333333333333. Divide one by the other and you get 1.9999999999999998. That is not greater than or equal to 2.0, and it never was. In IEEE 754 — the floating-point standard behind every mainstream CPU — six ninths divided by three ninths is not two. The detector did the correct thing: 1.9999… is below the threshold, so it stayed quiet.

The fix was in the test: 7 calls and 2 calls instead of 6 and 3. That gives 0.7778 / 0.2222 = 3.5, comfortably past the threshold and nowhere near the ambiguity zone. The rule of thumb worth memorizing: test numeric thresholds at n+1, not at n. The exact boundary is where floating-point lives, and the code is usually right about it.

3. The blind spot in the docs

RetryStormDetector fires on five or more retries. The scenario: six retries where five succeed — transient errors that resolve on their own — expected not to fire. It fired. The first instinct was that the detector should be smarter: it should distinguish transient retries from systemic ones.

Then the docs got read. The predecessor's anomaly-validation-matrix.md says it plainly: “Known blind spots for v0.1.0: Transient error vs systemic — no distinction between retries that succeed vs those that keep failing. A run with 5 retries that all succeed still fires.” The detector's entire logic is total_retries >= 5. It never looks at success rate. Documented, deliberate, not a bug.

The test had been written against an imagined version of the product. The fix was to invert the expectation — and to rename the test so it says “known blind spot” out loud.

The pattern, and why it's more dangerous than a bug

Three different layers failed: world knowledge in the first case, arithmetic in the second, documentation in the third. One shape: the expectation was wrong, not the implementation.

That matters because of an asymmetry almost nobody warns you about. Fix a real bug and your system improves. Fix your code to satisfy a wrong test and your system gets worse — you have just encoded a false belief about how the system behaves and stamped it with a green check. From that point on, the test actively protects the mistake. Six months later someone “refactors” the correct behavior away and CI thanks them for it.

And the failure mode is quiet. A missing test leaves a gap you can see in a coverage report. A test asserting the wrong thing looks like diligence.

A three-minute triage before you touch the code

1. Read the output, not the boolean. In the first case the verdict was wrong and the reason was right. If your component returns a note, a diff, a confidence score, or a ranked list, read it before you form a theory. The summary is not the evidence.
2. Classify the assertion by what it encodes. Numbers — thresholds, ratios, tolerances — fail at boundaries: assume floating-point first, it costs two minutes to rule out. Relationships between two things (“search_kb is unrelated to a password reset”) fail on domain knowledge: check whether the relationship is actually true. Product policy (“retries should be suppressed”) can only fail one way, against the spec, so go read the spec.
3. Recompute by hand. A calculator settles the arithmetic case in seconds. If your hand computation and the code agree, the test's expectation is now the suspect.
4. Ask the counterfactual. If I “fix” the code to make this test pass, can I defend the new behavior to a user, in one sentence, without mentioning the test? If not, the test is the defect.

One habit I've adopted from this: never change production code and its test in the same commit unless you can state, in a single sentence, which one was wrong and why.

Which oracle outranks which

An oracle is whatever you trust to tell you the right answer. In a plain unit test it's the assertion, because you computed the expected value independently of the code. With an LLM-backed component, the oracle moves — and it moves by category, not by vibe.

• The spec outranks everything. What the product decided to do is not something a model gets a vote on. The third case was settled by one paragraph in a markdown file.
• Deterministic math outranks intuition. The second case was settled by arithmetic. No model, no debate, no meeting.
• The model can outrank your domain intuition — but only about the world. The first case was a question about how password resets actually work, and the model had the better-informed view. That inversion is real, and it's narrow: the model knows the world, not your product's intent.
• Your intuition is the weakest input, not the strongest. It generated all three bad expectations in the first place.

Keep the wrong test — relabeled

The reflex after the third case is to delete the test. Don't. A test that documents a known limitation is one of the most useful artifacts you can leave behind, if you name it honestly. Assert the behavior that exists today and put the limitation in the test name: fires on transient retries (known blind spot, delete this test when fixed). Now the day someone improves the detector, CI fails and points at the exact file and line that says what to do next. You've converted a wrong test into a tripwire.

When not to bother

Being honest about the cost: this triage is not free, and you shouldn't run it on every red build. It's worth the three minutes when the test is new and unmerged, when the assertion encodes a numeric boundary, or when the component is non-deterministic. It's usually a waste when a test that has been green for a year suddenly fails after a code change you made — that failure is telling you about your change, not about the oracle. Spend the suspicion where the odds are.

What this means if models write your tests

Most teams now have agents writing code and, increasingly, writing the tests for that code. That changes the arithmetic. If the same model, working from the same assumption, writes both the implementation and the assertion, a green test proves they agree. It does not prove either one is right. Consistency is not correctness, and a suite generated from one wrong premise is a very confident mirror.

So the review skill that matters shifts from “does this test pass” to “what does this test assume, and is that assumption written down anywhere?” That question is cheap in review and expensive to skip. It's also a good interview prompt: hand a candidate a failing test and a correct implementation, and watch whether they reflexively rewrite the implementation.

None of this means you should distrust your tests. It means you should have a procedure for the rare case where the test is the suspect — because the rare case stays invisible until you look. Test at n+1, quote the spec, and read the reason before the verdict.

Key Takeaways

• When the component under test is an LLM, the test is not automatically the oracle. Sometimes it's the suspect.
• Check three things first: floating-point boundaries (test at n+1, not n), domain relationships the model may understand better than you, and documented limitations you never read.
• Fixing code to satisfy a wrong test is the one failure mode where a red test makes your system worse — it encodes a false belief and hides it behind a green check.
• Read the reasoning, not just the verdict; recompute numbers by hand; let the spec outrank the model on anything about product intent.
• Don't delete a wrong test — rename it to document the known limitation and turn it into a tripwire for the future fix.
• When the same model writes the code and the tests, green means consistent, not correct.