Your Agent Harness Shipped 244 Releases This Year. What Did You Pin?

Nothing in your repository changed between Monday and Friday. Same tests, same prompt file, same developer. And yet the agent that wrote a clean diff on Monday spent Friday afternoon rewriting the same function three times.

You went looking for the cause in your code, because that is where causes usually live. It wasn't there. It was in a dependency you never added to a lockfile: the program wrapped around the model.

The part of your agent that isn't the model

People talk about Claude Code, Cursor, or Pi as if they were the model. They aren't. The model is the thing that predicts tokens. The harness is the program around it. It decides what text gets assembled before every request, which tools the model is allowed to call, how tool output is fed back, when the conversation gets compacted, and what happens when a command fails. Same model, different harness, very different results.

That harness updates itself. Usually by default, usually without asking, and often several times a week. A developer who recently moved their setup off Claude Code to Pi counted the cost in a public write-up: 244 releases in a year, and more than three-quarters of them changed the system prompt or the tool descriptions. That is a release every day and a half, and most of them changed the instructions the model reads before it reads your code.

Sit with that ratio for a second. Most of those releases were not parser bug fixes. They were edits to the standing instructions that shape how the model behaves on your repository.

A system prompt change is a code change

The system prompt is the block of instructions a harness puts in front of every conversation. It is invisible in your diff, but it is the most privileged text in the entire request: it sits where models are trained to weight instructions most heavily, which is precisely the position that outranks your repository conventions.

Two things follow.

First, your project rules can lose to house style. You wrote a rules file that says the team wraps at 100 columns and uses tabs, and the harness shipped a new paragraph that prefers something else. Nobody reviewed that change against your conventions, because it didn't arrive as a pull request. It arrived as an auto-update.

Second, instructions cost tokens whether or not they help you. The same write-up points out the default shell tool's description ran to almost 3,000 tokens, much of it general advice about git and opening pull requests that current models already know cold. You pay that on every request, before a single line of your code is read.

None of this is an argument that vendors are careless. Frequent releases fix real problems fast. It is an argument that a fast-moving harness puts behavior changes on a channel almost nobody monitors.

Four things can change your agent's output. Only one gets pinned.

• The model. Usually pinned deliberately, sometimes deprecated out from under you. You mostly track this one.
• The harness. Auto-updating, changes weekly, rarely pinned. This is the one that surprises you.
• Your config. Rules files, skills, connected tools, permission settings. In your repo, but often reviewed by nobody.
• The world. A dependency released, a flaky test got flakier, the repo grew. The drift you blame on the agent.

When output quality moves, engineers reach for the model first because it has a name and a version number. In practice the harness and your own uncommitted config are the more common culprits — and the harness is the one you can pin without giving up anything.

If quality drops, change one variable at a time, cheapest first. Reverting a harness version takes a minute. Re-litigating your architecture takes a week.

Pin the harness, not just the model

Pin everything is not a policy; it's a refusal to decide. Here is a tiered version that survives contact with a real team.

Tier 1 — Pinned: anything that produces commits

Work that lands on main or ships to production runs on an exact harness version. Not latest, not a range. Install it in the container image or the devcontainer, next to the language runtime, and bump it on a schedule. Commit the harness config to the repo so behavior changes are reviewable in a pull request like everything else.

Tier 2 — Canary: one volunteer on the latest channel

One engineer runs the newest release all week and reports what broke. This is the cheapest early-warning system you will ever build, and it costs one person's tolerance for rough edges. Rotate the volunteer each quarter so both the burden and the knowledge spread.

Tier 3 — Free: exploration

Scratch repos, spike branches, throwaway scripts. Let the harness do whatever it wants there. Experimentation is where new features earn their place, and you don't want a pinned version strangling it.

The missing piece: a behavior test suite

You cannot evaluate an upgrade on vibes, and you don't need a benchmark. You need ten tasks. Write down ten real asks from your repo — add a timestamp column to this migration, find why this endpoint 500s on empty input, refactor this module to match the pattern in the neighboring one — each with a pass or fail you can judge by eye in a minute or two. Fifteen minutes to build, ten to run. If a new harness version clears fewer than eight of ten, you have your answer before your team finds out the hard way.

Almost nobody has this list, which is why should we upgrade turns into an argument about anecdotes. The team with the ten tasks upgrades on Tuesday and moves on with their afternoon.

The exit cost is the real design constraint

Here is the test that decides whether any of this is optional: how much of your workflow lives inside a feature only one vendor has?

Look at the rules-file mess. A common pattern is three near-identical instruction files for three harnesses, kept in sync by hand or by symlink. That is pure friction, and it is a tell: your project's conventions are trapped in a vendor's filename instead of a format. Keep the real rules in one neutral file and symlink each vendor's expected name to it. When you switch, you delete a symlink.

The same test applies to tooling. If your team's build, test, and deploy steps are ordinary commands — a Makefile target, a script in the repo, a documented CLI — then any harness can call them, and switching costs an afternoon. If they live as a proprietary workflow feature in one tool, switching costs a quarter. The developer who moved to Pi wrote the extension they were missing in about twenty minutes. That number isn't a coincidence: their build steps were plain commands.

Adopt the harness's clever features for convenience, never for the only copy of a process. The blast radius of a vendor's direction change is exactly the set of things you can only do their way.

The honest trade-off

Pinning has a real price, and it would be dishonest to sell it as free.

• You delay fixes. A harness bug that annoys you daily might be fixed upstream the week after you pin, and you won't see it until your bump window.
• You skip improvements. Some of those 244 releases made agents genuinely better at large repos. Staying three months behind means staying three months behind.
• You take on a job. Someone has to read the changelog — call it fifteen minutes a week — or the pin quietly becomes a frozen liability instead of a decision.

The counterargument is real too: a harness that never moves rots against new model versions, and new model versions are where most capability gains actually come from. The goal isn't stability for its own sake. It's knowing which changes reached you, and choosing when to take them.

A drill you can run this week

1. Ask each engineer what harness version they're on. If nobody knows, you don't have a policy, you have a lottery.
2. Pin the version for anything that produces commits, in the same place you pin the language runtime.
3. Name one canary, and put their review date somewhere a calendar can see it.
4. Write the ten-task list. It's an afternoon of thinking that pays out on every upgrade for a year.
5. Move project rules into one neutral file and symlink the vendor name to it.
6. Move the build and test steps you depend on into plain commands in the repo, so a switch costs an afternoon instead of a quarter.

None of this is dramatic, and that's the point. The interesting failures in AI-assisted development are rarely the model being wrong. They're the case where the ground moved, nobody logged it, and a team spent a week blaming the wrong thing.

Key Takeaways

• The model is a dependency. The harness around it is a second, faster-moving dependency that rarely appears in a lockfile.
• A system prompt change is a behavior change: it edits the highest-privilege instructions in every request, including the ones that outrank your repo conventions.
• Pin the harness for anything that produces commits, canary the latest channel with one volunteer, and let exploration run free.
• Ten real tasks with pass or fail beats a benchmark and beats an argument. Build the list once, run it before every upgrade.
• Exit cost is a design constraint: keep rules in a neutral file and critical steps as plain commands, so switching is an afternoon.
• Pinning costs you fixes and features and adds about fifteen minutes of changelog reading a week. Pay it deliberately, or drift by accident.