Your Self-Hosting Bill Is a Utilization Number, Not a Model Choice

A local model on hardware you own sounds like the end of the token bill. Then someone asks how many developers it will serve at 3 p.m. on a Tuesday, and the room goes quiet. Self-hosting isn't priced by how smart the model is. It's priced by how much of the machine is working while it is smart.

That number — utilization — rarely makes it into the proposal. Everything else you're comparing, from benchmark scores to parameter counts to tokens per second, sits downstream of it. Get utilization wrong and the rest of the analysis is decoration.

Capability is priced in steps, not in parameters

The intuition going in is that better models cost proportionally more to run. They don't. Quality improves gradually across the open-weight field, but the hardware underneath moves in jumps, because the binding constraint isn't the number of parameters. It's whether the weights fit in the memory of the machine you already have. Cross that line and you aren't paying for a few billion more parameters. You're paying for the next machine.

Published experiments on moving coding workloads off frontier APIs show this clearly. One mid-sized model cost roughly four times more to serve than a smaller one while improving measured application performance by about 2 percent. A much larger model cost only marginally more than the tier below it and solved substantially more tasks. Both facts are invisible on a leaderboard, which ranks models by capability and says nothing about what a resolved task costs you.

That's the substitution you need to make. Stop ranking models by tokens per second or by benchmark position. Rank them by cost per resolved task — the fully loaded cost of the hardware, the electricity, and the engineer time, divided by the number of tasks the model actually closes. It's a worse-feeling metric and a much better decision.

Concurrency is the constraint nobody budgets for

Behind an API, concurrency is someone else's problem. You get a reasonably predictable response time and you stop thinking about it. When you run the box, it becomes the binding constraint.

Here's the part that surprises people: your model weights are not the only thing living in memory. Every active session holds a running copy of its conversation — the context, plus the model's internal working memory for that conversation — so it doesn't have to re-read the whole thing on every turn. That memory is called the KV cache, and it sits in the same expensive memory as the weights. A long-horizon coding agent that has been poking at a repository for an hour is holding a large slice of your machine while its human sits there reading a diff.

So your real capacity isn't that the weights fit. It's that the weights plus N live sessions fit. Add the intraday shape of developer work and the picture gets worse. Internal engineering demand concentrates in a few overlapping hours; evenings and weekends drop close to zero. You're paying capital depreciation on that hardware around the clock for a few hours of actual work. A frontier API pools uncorrelated demand across thousands of customers in different time zones. You can't.

Unless, that is, your team is deliberately spread. A nearshore setup with pods in two or three time zones — and an async-first habit of writing decisions down instead of holding meetings — flattens your demand curve in exactly the way that helps here. The utilization question stops being four peak hours and becomes eleven useful ones. That's a structural advantage of distributed teams that almost never shows up in a cost model, and it's worth measuring before you compare hardware quotes.

The order things break in

Worth knowing which wall you hit first:

• Latency goes first. Response time becomes unacceptable long before the hardware physically runs out of room. Your users will tell you the box is full when it technically isn't.
• Concurrency goes second. Beyond a certain number of simultaneous sessions, requests queue. Queuing is fine for a batch job and miserable for an interactive coding agent.
• Throughput goes last. This is the number on the spec sheet, and it's the one you'll be least likely to bump into.

The harness is a line item, not a preference

Run the same model on the same hardware against the same task set with three different agent harnesses — the orchestration layer that decides how prompts get built, cached, and replayed — and the resolve rate barely moves. A couple of tasks either way. But wall time can differ by a factor of two.

When you rent GPUs, wall time is your bill. So the harness isn't a tooling preference. It's a cost line item with your name on it.

And it can move under you. In one reported case, a point release changed how prefix caching was handled — prefix caching is the trick where the model reuses the shared beginning of a prompt instead of recomputing it from scratch — and the caching broke. Suddenly every request paid full price for tokens it had been getting nearly free. Rental cost went up by a factor of five. Nothing about the model changed. Nobody's code changed. A dependency shipped a small optimization and the effective price of your product moved by 400 percent.

The lesson isn't to pick the cheapest harness. It's to pick the harness your developers actually want to use, and then meter it. Cost per resolved task, per release. If you don't have that chart, a patch on a Tuesday can quietly double your burn rate and you'll find out at the end of the quarter.

The four numbers to collect before you decide

1. Peak concurrent sessions. Measured, not guessed. Instrument your current agent usage and count how many sessions are open at the 95th percentile — not the average, and not the theoretical maximum.
2. Memory floor per session. Weights, plus average live context, plus KV overhead. Multiply by peak concurrency. If that total exceeds the memory of a hardware tier, you've just priced yourself into the next tier.
3. Cost per resolved task. Tasks your team actually closes, not requests served. This is the only number that compares cleanly against a per-token API bill.
4. Realistic utilization. Hours the box is doing useful work, divided by hours you're paying for it, measured over a month. Not peak and not ideal. Actual.

Run the break-even at that fourth number. If your business case needs near-continuous saturation to work, it doesn't work — and the failure mode is slow, because the hardware looks great in week one while everyone is trying it out.

When self-hosting wins anyway

Cost alone usually doesn't survive contact with the spreadsheet. What does survive are the reasons that aren't about money:

• Data that isn't allowed to leave the building. Compliance requirements, client contracts, or a security review that will never say yes to a third-party endpoint.
• A workload nobody can rate-limit. If your product depends on sustained throughput that a provider can throttle or reprice, that dependency is a business risk, not a line item.
• Prices that don't move mid-quarter. A budget you can defend is worth something, even when it's a bigger number than the API's.

That isn't an argument against self-hosting. It's an argument for being honest about which of those three is actually driving the decision. If it's none of them, you're making a cost argument, and cost arguments need utilization math that holds.

The cheapest step is two weeks of metering

Before you buy anything, spend two weeks measuring what you already do. Token spend per closed task on your current API. Sessions open at peak. Average context size at the end of a session. Then replay a sample of those sessions against a self-hosted box with a memory cap and watch where it queues.

That's a small amount of work next to a six-figure hardware commitment, and it turns a debate about model quality into a conversation about three numbers. The model was never the hard part.

Key Takeaways

• Self-hosting economics are decided by utilization and memory tier, not by benchmark position or parameter count.
• Memory is priced in steps. Crossing a hardware tier costs more than a few billion extra parameters ever will.
• Capacity is weights plus live sessions. Long agent contexts hold expensive memory while a human reads a diff.
• Internal demand peaks for a few hours a day, but you pay for hardware 24. Distributed, async-first teams flatten that curve — a real advantage worth measuring.
• The harness barely changes resolve rate and can change wall time by 2x. When you rent GPUs, wall time is the bill. Meter it per release.
• Justify self-hosting with data residency, unthrottleable workloads, or price stability. If the only reason is cost, run the utilization math first.