Your AI Coding Tool Cut 89% of Tokens. Your Bill Didn’t Move.

A tool with 79,000 GitHub stars says it removes up to 90% of the terminal output your coding agent reads. An X post claiming it cuts Claude Code tokens by 60% reached 313,000 views. Then someone spent $1,500 of their own token budget testing it, and the bill barely moved — in one configuration it went up 17%.

RTK (Rust Token Killer) sits between your agent and your shell. When the agent runs ls, git, or a test command, RTK rewrites the call and returns a terser version of the output. Instead of a full file listing with owners and timestamps, the agent sees names, sizes, and permissions. That is a genuinely sensible idea. Compressing what an agent reads is one of the few levers you have over an AI coding bill.

The problem is what happened next: the tool’s own savings counter told a story the invoice didn’t. That gap is not unique to RTK. It is the default failure mode of nearly every “cut your AI costs” pitch you will be handed this year, and learning to see it is now a core engineering skill.

The metric that lies: bytes removed versus dollars burned

RTK ships a command called rtk gain. It reports raw command output minus filtered output, measured in bytes, divided by four. That is an estimate of tokens removed from context. It is not a count of billed tokens, and the README quietly admits it: cutting 90% of bash output “is not the same as cutting your bill by 90%.”

Here is the structural reason it can’t be. Removing a chunk of text changes what the model sees. It also changes what the model does next — how many turns it takes, which files it re-reads, whether it gets confused and retries. Every one of those turns is billed. A counter that measures only removed output assumes the rest of the session stays identical. It never does.

The write-up from Quesma is worth reading in full, because it shows exactly how far the counter drifts. Across 445 attempts, RTK reported 349.2 million tokens saved — an 89% reduction. Two calls in a single task, head -1 train.txt requested twice, accounted for 69% of that total. RTK compared the one-line reads against the size of the whole file. Those commands were never going to return the whole file. The counter was crediting a difference that could not exist.

What the benchmark actually found

The team ran Terminal-Bench 2.1, a benchmark built around heavy terminal interaction — precisely where a tool like RTK should shine. Two stacks: Claude Code with Fable 5.0, and OpenCode with DeepSeek V4 Pro. Every task ran five times with RTK and five times without, same models, same platform, same timeouts. After dropping a few tasks that hit model refusals, that is 1,740 attempts.

Spending on passed attempts: Fable went from $596 to $546, but its pass rate slipped from 84% to 83%. DeepSeek went from $26 to $31, with pass rate falling from 71% to 69%. Divide total spend — including failed attempts — by number of passes, and Fable came out 3% cheaper while DeepSeek came out 7% more expensive.

Then they weighted every task equally, because one expensive task can swamp dozens of cheap ones. On that measure, Fable was 1% more expensive, with a confidence interval that included zero. DeepSeek’s average task cost rose 17%. Restricting to the 36 DeepSeek tasks where all ten attempts passed, the increase was still 18%.

There are two honest readings. The first: on one model and one benchmark, RTK was roughly cost-neutral, and neutral is not nothing — it may relieve context pressure in other setups. The second: the headline “up to 90%” and the measured reality live in different universes. If you bought on the headline, you bought a story.

One task decided the whole result

Almost all of Fable’s apparent win came from a single task, winning-avg-corewars. Both configurations passed every attempt; with RTK the agent finished in about half as many turns. Remove that one task and the remaining savings were under 1%. On DeepSeek, the same task went the other way — more turns, more money.

This is the small-sample trap in agent benchmarking, and it will bite you the first time you A/B a prompt change on five tasks. Agent runs have enormous variance. One task with a lucky early tool call can swing an entire report.

Compression can cost you turns

The most instructive failure in the run is a loop. On one DeepSeek task, the agent ran find with a flag the plugin version didn’t support. RTK rewrote it to rtk find, which failed with “Use find directly.” The next retry was rewritten again. The agent accumulated 339 consecutive errors over roughly 12 minutes. It still passed the task — at about nine times the cost of the matching baseline attempt. That specific bug was fixed in a later release, but the failure mode is general: anything sitting between your agent and the world can turn one bad rewrite into a retry spiral.

The ceiling problem

Even a flawless compressor has a maximum win, and that ceiling is lower than you think. Without RTK, terminal output made up about 11% of Fable’s input tokens and 40% of DeepSeek’s. Everything else — your system prompt, file reads, search results, the conversation so far — dominates.

RTK can only touch shell commands. Claude Code and OpenCode expose file reading and searching as separate tools, and those bypass it entirely. In these runs, only 31% of Claude Code’s terminal calls and 51% of OpenCode’s went through RTK at all. Meanwhile, roughly half of Claude Code’s bash calls already limited their own output with head, tail, or wc. The tool was competing with habits the agent already had.

None of this makes RTK bad. It makes the framing bad. A 10% lever applied to 30% of your traffic is a 3% lever, and 3% sits well inside the noise of a single afternoon of agent runs.

A five-rule test for any “cut your AI bill” tool

You will be pitched more of these, not fewer. Apply the same test every time.

1. Demand cost per passing task, not cost alone. A tool that halves your spend while dropping pass rates by five points has not saved you money — it has moved the work to you. Quesma’s pass-rate deltas were only one and two points, but they were measured, and they belong on the same chart as the dollars.
2. Weight tasks equally and report the distribution. Give the mean cost per task, plus how much the worst task contributed. If one task drives the headline, you have an anecdote, not a result.
3. Run both arms at least five times and publish confidence intervals. Agent runs are noisy. A single-run A/B is a coin flip with a chart.
4. Distrust any savings counter that reports bytes. Bytes divided by four is a guess about tokens, and it says nothing about the turns it changes. When a dashboard shows “tokens saved,” ask what happened to session length.
5. Check what the tool cannot reach. If it only rewrites shell output, measure what share of your input tokens shell output actually is — 11% on one stack, 40% on another. The same tool is a rounding error on one and a real lever on the other.

Corroboration matters too. JetBrains ran a SkillsBench evaluation and found no savings at all. Two independent results pointing away from the headline is a signal, not a coincidence.

Where the real levers are

If terminal compression is a 3% lever, what is a 20% lever? The unglamorous stuff: shrink what you put in front of the model on every single turn (system prompts, tool schemas, always-included context files), bound what comes back (pagination, explicit limits, structured summaries instead of raw logs), and turn on prompt caching so repeated prefixes stop being billed at full price.

The trade-offs are real. Aggressive context trimming costs you information, and information is what makes an agent pass. There is no free lunch here — only a bill you can read and a bill you can’t.

So run the experiment on your own repo, with your own tasks, and count passes as well as dollars. Ten tasks, five runs each, two arms. That is a day of work and a few hundred dollars. It is a far better deal than adopting a 79,000-star tool because its own counter said 89%.

Key Takeaways

• Token savings and cost savings are different measurements. A tool can remove 89% of the bytes your agent reads and leave your bill flat — or raise it.
• Any compression layer changes the agent’s next turns. Cost per passing task is the only number that captures that.
• Report cost per task with the distribution, at least five runs per arm, and confidence intervals. Single-run agent benchmarks are noise.
• Know your ceiling before you buy. Terminal output was 11% of input tokens on one stack and 40% on another, and shell-only tools can’t touch file reads or search.
• Bugs in the shim between your agent and the shell can spiral into hundreds of retries. Watch turn counts, not just output sizes.