Five Agents, One Opinion: The Independence Assumption Inside Your LLM Ensemble

You send the same question to five AI agents. Four of them land on the same answer. You call it consensus, you write the number into a dashboard, and you move on.

There is a decent chance you just asked one model the same question five times. That is not five judgments. It is one judgment sampled five times, and if that is what is happening underneath, every piece of math you built on top of it is standing on air.

What a quorum is supposed to buy you

The idea comes from distributed systems. If you have five servers and two of them are broken or lying, three agreeing servers are still enough to trust the answer. This is why engineers use odd numbers, why three beats two, and why formulas like Q > 2f exist: your quorum size has to outnumber twice the number of faulty participants.

None of that math cares about how smart your servers are. It only cares about one thing: failures have to be independent. Two nodes fail independently when one failing tells you almost nothing about whether the other one will. Same bug, different machine, different moment. The moment your failures are correlated — one power strip, one shared config file — a quorum stops being protection and becomes decoration.

Now replace the servers with language models.

The assumption nobody tests

Five agents built on the same base model, the same system prompt, the same temperature, and the same question are not independent. They share everything that could make them wrong. When one of them misreads an ambiguous sentence, so do the other four, because it is the same sentence and effectively the same reader.

You do not have to take that on faith. You can measure it in an afternoon.

In a detailed write-up on DEV Community, an engineer describes exactly this discovery while building a pharmacy decision system on top of Jev, a model that returns probabilities instead of prose — ask it whether oxycodone is a controlled substance and you get 0.98, not a paragraph saying so. Numbers can be measured. Paragraphs cannot. That one property is what makes the rest of the experiment possible.

He measured the spread across five identical prompts sent to five separate agents: 0.010. Then he measured how much the same question moves when you change nothing meaningful at all — a floor of 0.042.

The spread between his five agents was four times smaller than the model's own random jitter. He had built a Byzantine fault-tolerant voting system on top of a single voter.

Measure the noise floor before you argue about the threshold

The noise floor is the amount a score can move when the meaning of the question has not moved at all. It is not error. It is the texture of a non-deterministic system, and it is the number your decision rule should be built on.

Three cheap tests give it to you:

1. Identity test. Send the exact same prompt twenty times. Measure the spread. That is your floor.
2. Reorder test. Shuffle the order of the options or the sub-questions. Measure again. Order should not matter. Measure how much it does.
3. Paraphrase test. Rewrite the question three different ways with the same meaning. Measure the spread across the group, not just within each version.

In that pharmacy build, the three numbers came out at 0.042 for identity, 0.059 for reordering, and 0.073 for the paraphrase group. Every one of those is noise you were previously treating as signal.

The floor belongs in your decision rule

Once you have the floor, the threshold stops being arbitrary. A vote is stable when its distance from 0.5 is larger than the floor. A score of 0.54 in a system with a 0.042 floor is not a weak yes. It is a shrug, and it should not be allowed to cast a deciding vote.

That single change — a stability check instead of a fixed cutoff — is often the difference between a system that decides and a system that declines. Declining is the part most teams forget to build, even though it is the outcome that keeps you out of trouble.

Paraphrasing is the only lever that buys you real independence

Identical prompts across five agents gave a spread of 0.010. Paraphrased prompts, on the hard cases, pushed the spread to 0.080 — above the noise floor, which is exactly what you want. You are no longer sampling one reader. You have five readers looking at five wordings of the same problem.

So the rule is blunt: if you require a quorum, every vote has to be a different paraphrase of the question. Five identical prompts do not count as five. They count as one with a confidence interval.

Then audit your paraphrases, because two of them probably are not paraphrases

That same experiment turned up a paraphrase that had quietly become a different question. It asked whether pregnancy could be excluded "on the basis of this record alone." The word alone reads as a challenge to whether one test is sufficient. The model read it correctly and scored 0.36 on a negative test. The prompt was wrong, not the model.

If you are going to spend effort on an ensemble, spend some of it re-reading your paraphrases against the source of truth. A paraphrase that smuggles in a new question is not a second opinion. It is a second question.

Sometimes your label is wrong and the model is right

The second bug from that build was human. A case involved a late period, a declined pregnancy test, and an isotretinoin order. The engineer labeled it "escalate," because the order obviously needed a pharmacist. The model said "pregnancy is not ruled out" — 115 times out of 120.

The model was right. "Pregnancy is not ruled out" is precisely what triggers the hold. The label had confused a property of the prescription with a property of the question, and the fix was to relabel the case, not to retrain the model.

Build your ambiguous tier expecting it to catch you first. It usually does.

Rising escalation is a feature, not a regression

When chaos was injected — truncated records, adversarial text, rate limits, agents crashing mid-round — the escalation rate went from 5% to 18%. That is a six-sigma-sized move, and it is the direction you want.

The kernel sent more decisions to a human as the evidence got worse. Its most instructive moment was declining a trivial question: a documented penicillin anaphylaxis with a new amoxicillin order, where two agents got rate-limited and a third came back at 0.54, inside the noise floor. Quorum not met. Human loop.

If your system's escalation rate does not move when the quality of the underlying evidence moves, then your escalation path is not wired to your confidence at all. It is wired to something else, probably a queue depth.

When not to build any of this

Be honest about the trade. Five agents means five times the spend, more latency, more orchestration, and a paraphrase set you now have to maintain. For reversible, cheap, low-stakes decisions — tagging a support ticket, summarizing a changelog — one agent and a log line is the correct architecture. Use the ensemble on the irreversible ones: money, health, access control, anything you would have to explain to a regulator or a customer.

Also note what the pharmacy numbers do and do not prove. Zero wrong verdicts across 1,080 golden rounds bounds the true error rate below roughly 0.28% at 95% confidence. That is not zero. It is a ceiling, and pretending otherwise is how benchmarks get quoted out of context.

A five-step checklist

1. Measure the floor. Run the identity, reorder, and paraphrase tests. Write the three numbers down.
2. Reshape your votes. If you want a quorum, every agent must see a different paraphrase. Identical prompts get grouped, not counted.
3. Replace the 0.5 cutoff with a stability rule. A vote only counts when its margin from 0.5 clears the floor.
4. Make declining a first-class outcome. Track your escalation rate by evidence quality and expect it to climb when inputs degrade.
5. Audit before you blame. When something looks wrong, check the label and the paraphrase before you touch the model.

None of this requires a fancy framework. It requires accepting that your agents are correlated sources — more like five interns who read the same memo than five independent experts. You can still build a trustworthy system on correlated sources. You just cannot pretend they are independent while you do it, because the math you are relying on will not be there when you need it.

Key Takeaways

• Quorum math assumes independent failures. Agents sharing a base model, prompt, and temperature are the opposite of independent.
• Measure your noise floor with three tests — identity, reorder, paraphrase — before you tune a single threshold.
• If a vote's distance from 0.5 is smaller than the floor, it carries no information. That is an escalation, not a weak yes.
• Only paraphrasing raises real spread. Identical prompts across five agents measured 0.010 against a 0.042 floor.
• Escalation rates that rise as evidence degrades are working as designed. Flat escalation rates mean you built a queue, not a guardrail.
• Check your labels and your paraphrases first. In one documented build, two of the three bugs found were the engineer's, not the model's.