Stop Letting the Model Do the Math: Your Tools Should Return Answers, Not Rows

Ask a small language model how many of eleven IDs in a list are greater than or equal to ten, and it will get it wrong twenty times out of twenty. Not sometimes. Every single call. That is not a benchmark curiosity; it is a design signal. Eleven small integers, the kind of thing you do in your head while waiting for coffee, is already past the point where a model's arithmetic can be trusted. And yet almost every agent stack I look at hands the model raw rows and asks it to total them up.

The fix is not a better model. It is a different division of labor: your tools do the arithmetic, the model does the explaining. If that sounds obvious, look at how few agent stacks actually work that way. Most of them return rows, then let the model do the addition in the final sentence, and everyone downstream treats that total as sourced because it arrived with a citation attached.

Your agent is a lossy channel, not a calculator

Every hop between the thing that knows the answer and the person who reads it is lossy. Tokenization splits numbers into pieces. Sampling can drift. Context gets truncated. And the model is not reading your data — it is reading a description of your data, then recomputing from that description. You would not let a colleague re-derive last quarter's revenue from a screenshot of the dashboard, but that is exactly the pipeline we build when a tool returns a list and a prompt says: tell the user how much this costs.

Here is a concrete shape of the problem. A cloud waste scanner produces one report: 120 findings across 17 regions, 31 of them priced, total estimated waste of $6.90 per month, or $82.80 a year. Ask an agent how much that costs and it will sum the priced rows itself. With two rows you get away with it. With 31 you do not. The failure is silent, the answer is formatted, and the number is wrong.

The failure that survives review

A wrong total looks exactly like a right total. It has a source. It has units. It is confident. Compare that to the failure we usually worry about, where the model invents a resource that does not exist — that kind of error is at least oddly specific, and someone usually catches it. A mis-added total is plausible by construction, so it gets copied into the ticket and the slide deck and nobody looks again.

The second, sneakier failure is the silent zero. Suppose the check in your report is called unattached-ebs and someone asks for unattached EBS volumes. The filter matches nothing. A tool that returns an empty list, and a model that summarizes it, produce a precise zero: no savings there. That reads as good news. A precise zero with a source attached is harder to catch than a miscount, because there is nothing to compare it against.

Three jobs your tools should do instead of the model

1. Return the answer, not the ingredients

A well-shaped tool takes a report path and a filter, and comes back with the arithmetic already done: how many findings matched, monthly and annual cost, how many are free or approximate, the single costliest finding, plus per-check and per-region breakdowns that are already grouped and already sorted. Every how-much, how-many, and which-is-biggest question gets answered from one call. The model's only job is to quote it.

The same rule works from the other direction. The scan tool returns totals, breakdowns, and the two or three costliest findings inline, and leaves the 120 rows in the report file. That is partly about correctness and partly about context: a busy account's rows would bury the total you actually care about.

2. Echo the filter that produced the number

Every summary response should carry the predicate that generated it: which checks you asked for, how many findings exist in the report, how many matched — and, critically, the names you asked for that do not exist in the report at all. Add an explicit not matched block showing what the filter excluded and what it cost. Now a zero is auditable instead of reassuring. The rule here is simple: check the predicate the model sent, not just the number it quoted. A wrong filter returns an exact figure with a source attached.

3. Refuse to be ambiguous

If two shapes of answer are possible, return both, explicitly labeled. Ambiguity is the one thing the model is worst at resolving and the one thing you are best at resolving at write time, when you have the data model in front of you. Do not defer a schema decision to inference.

Let the model decide what to look at, not what is true

The division that holds up is: the model routes, sequences, and translates; deterministic code owns the truth. In one agentic system built for a media pipeline, the model investigates a backlog, queries three different monitoring tools, and writes up a recommendation in plain English. But the actual choice of which piece of work is safe to delay comes from a short rule written as ordinary code: throw out anything the client already approved, anything the director flagged, anything other work is waiting on, then pick whatever is left with the most slack before its own deadline.

In live tests the fixed rule never broke — zero violations across every run. The AI matched the rule's pick three times out of five. Sixty percent is not a number you put on a slide. It is the right number to report, because the system's safety never depended on the model being correct. It depended on the rule being unbypassable. That is the pattern worth copying: the model can be wrong about which item it recommends and the system still cannot do the wrong thing.

The two-front-door test

If the same engine serves a human at a terminal and an agent over a tool interface, they must agree. Write an end-to-end test that runs both paths and asserts the same total. This catches the whole class of drift where you fix the CLI, ship it, and forget the tool server — or the reverse, which is more common now that the agent path is the one under active development. One engine, two doors, one number, one test. It is a boring test and it is the one that keeps you from explaining a discrepancy to a customer.

Enforce the boundary in CI, not in a README

If your agent server is read-only, prove it in the build. A handful of lines that read the server's source and assert it never references the apply or delete path will fail the day someone wires it up, just for testing. Convention is weak; a test is a wall. The destructive half of the system should build a plan — the API calls in order, the backup step where the provider allows one, an irreversible flag on every step with no recovery window — and then stop. Applying the plan stays a human action with a per-resource prompt.

When it is fine to let the model compute

Not every number needs a tool. Be honest about where the line is:

• Small closed sets. If the answer follows from fewer than roughly five values you handed the model in the same message, a tool is overhead. Eleven is already too many.
• Prose-grade questions. Roughly how bad is this, is this getting worse, does this look normal — these do not need arithmetic, they need a sentence.
• Numbers a human will re-derive anyway. If the source data is on the same screen as the answer, the cost of an error is small.
• Never for anything durable. A number going into a ticket, a contract, a bill, an SLA report, or a dashboard somebody else reads without the raw data behind it gets computed in code.

A rule of thumb that has held up for me: count the joins. If answering the question means combining values from more than one place, or iterating over more than a handful of items, that work belongs in deterministic code with a test next to it.

The trade-off you are actually accepting

You are buying a bigger tool surface. Every figure you compute is a function you maintain, a name you have to keep stable, and one more entry in a tool list that can get long enough for the model to pick the wrong one. Deterministic code is not free: you now own the arithmetic, which means you own the tests for it. And you lose flexibility — a tool that returns a single number cannot be asked a follow-up question you did not anticipate, while a tool that returns rows can be sliced a dozen ways.

That is the honest trade: flexibility for auditability. For exploratory work, take the flexibility. For anything touching money, contracts, or a customer-facing number, take the auditability and accept the extra functions. The agent gets to be useful without being trusted with the one part it is reliably bad at.

Key Takeaways

• A small model got an eleven-integer comparison wrong 20 times out of 20. Arithmetic is not a place to rely on inference.
• Wrong totals survive review because they look sourced, formatted, and confident. Silent zeros are worse: a precise zero reads as good news.
• Make tools return figures, counts, breakdowns, and minima/maxima — already grouped and sorted — not raw rows for the model to add up.
• Always echo the filter: which checks matched, which names do not exist in the report, and what the filter excluded.
• Let the model route, sequence, and explain; let fixed rules written as plain code make the call that touches money or contracts.
• If a terminal and an agent share one engine, prove they return the same number in an end-to-end test.
• Enforce read-only boundaries in CI with a test that reads the source, not in a README that nobody rereads.
• Count the joins. More than a handful of values, or more than one source, means the calculation belongs in code you own.