Your Agent Remembers the Sediment: Decide What Persists at Write Time

There is a failure mode in agent systems that never shows up as an error. The agent works well in week one. By week four it is citing a plan you abandoned in March, treating an outage you fixed as a current outage, and applying a "user preference" that actually came from a marketing page it scraped. Nothing is red in your logs. Every step is individually defensible. The system has been remembering the wrong things — and the reflex fix, "make it forget more," treats the symptom instead of the wound.

The wound is upstream of retrieval. Most agent memory stacks are one of two things: a transcript store that everything gets appended to, or a vector index that everything gets embedded into. (Embedding just means turning text into a list of numbers so a machine can compare passages by similarity.) Both designs are archives with a search bar. Neither is memory.

During a task, your agent produces three very different things

A single nontrivial task generates three kinds of material, and they have almost nothing in common:

• Attempts. Half-written plans, an approach it tried and backed out of, a hypothesis that died on contact with a test. Essential while the task is running. Worthless the moment it ends.
• Raw output. Tool responses, retrieved documents, error strings, stack traces. Verbose, often transient, frequently irrelevant — and occasionally containing text from an untrusted source that reads like an instruction.
• Conclusions. The short list of things that are now true and will still be true tomorrow: a decision, a verified fact, a commitment made to another system.

Only the third category deserves to survive the task. But writing is easy and deciding is hard, so the default write path treats all three identically: append it, embed it, move on.

Put rough numbers on it. A modest task — 40 tool calls and 8 retrievals, with tool output averaging 600 tokens — produces somewhere around 30,000 tokens of raw material. The conclusions from that same task, written down properly, are maybe 400 tokens. If your write path cannot tell those apart, you are pushing roughly 98% noise into the index you will later search by similarity. The retriever is not the thing that broke. It faithfully returns the best match for the query. You just filled the pond with silt.

Why a filter at read time cannot fix this

The obvious objection: store everything, and sort out relevance at retrieval. Three mechanisms make that fail, and they compound.

1. Summaries inherit their inputs' errors. The raw log said "connection refused, retrying." A summarization pass renders that as "service down." The summary is shorter, cleaner, more confident — and wrong in a way that is now harder to detect, because the original hedged language is gone.
2. Wrong beliefs often retrieve better than right ones. A stale conclusion tends to be stated as a general rule ("the billing API rate-limits at 100 requests per minute"), while the correction is specific and hedged ("as of the November incident, the limit was raised for our tenant"). Similarity search rewards the general, confident phrasing. You have built a system that prefers its own mistakes.
3. Derivation compounds the error. Facts derived from contaminated memory get summarized again, embedded again, and cited by later tasks. Nothing downstream can tell which upstream entry was sediment. By the time your read-side filter looks at it, the contamination is already an input to a decision.

This is the same reason databases enforce constraints at write time rather than relying on every future query to be careful. You cannot un-bake a summary. A winemaker racks wine off its spent yeast for exactly this reason — not because the sediment was bad, but because its job is finished and leaving it in contact changes the wine. A recent DEV post made that analogy well; what I want to argue about is the engineering consequence.

Write-side custody: decide at the door, and be able to say why

Write-side custody means the component that writes memory owns the decision about what it admits — and can explain, for every entry, why it exists and where it came from. Not a filter bolted onto the read path. A gate on the write path, with an audit trail.

Four classes of writes

Every candidate write gets exactly one class. The classes have different storage, different retention, and different permissions about what the agent is allowed to reason from.

1. Evidence. Raw transcripts, tool output, retrieval results. Written to an archive, never embedded, never retrievable by the agent — only by a human debugging or an auditor. Retention by policy, typically 30 to 90 days. This is where the 30,000 tokens go, and where they should have gone all along.
2. State. Facts that are true right now and are allowed to change: which version is deployed, which ticket is open, which customer is on which plan. Every state entry carries a source and an expiry, and the agent re-verifies rather than re-uses when it is stale.
3. Commitments. Things the agent did that other systems now depend on: a file written, a PR opened, an email drafted. Immutable, with an ID, because something outside your system is pointing at them.
4. Beliefs. The only class the agent is allowed to reason from freely. Each belief must point to at least one evidence or state record, carry a confidence and a timestamp, and be revocable without archaeology.

If your memory store is one undifferentiated pile, you cannot distinguish class 1 from class 4, and the retriever will happily serve both with equal confidence. The classification itself is usually cheap: most candidates are obviously evidence (a tool response), obviously state (a version string), or obviously commitments (a write operation). You only need a model call on the genuinely ambiguous ones.

Three questions before anything is promoted

1. Would this still be true tomorrow if everything else changed? "The build failed" is not durable — it is state with a short expiry. "This repo requires Node 22" is durable until someone upgrades it. If you cannot state the condition under which the entry becomes false, you do not have a belief, you have a log line.
2. Who said it, and are they allowed to say it? A measurement from your own tool is evidence about the world. A sentence from a fetched web page is not a principal; it is content. A user instruction is a principal. Collapsing these is how scraped text becomes a stored preference three weeks later.
3. If this is wrong, what breaks, and how would I find out? Reversible beliefs can be written with lower confidence and shorter TTLs. Irreversible ones — anything that will steer money, access, or a customer-facing message — should not be promoted at all without a human or a second independent source.

The gate that matters most: instruction-shaped content

Anything that reads like a directive — "always," "ignore," "from now on," "the correct procedure is" — gets a stricter gate than anything that reads like a fact. Accept instructions only from a named trusted principal. Never from retrieved content, never from tool output, never from a document the agent fetched on its own. An agent that follows a scraped instruction is a boundary problem, not a memory problem. But memory turns a one-time boundary failure into a permanent one, which is why the two problems get debugged together and should not be.

What this costs

Write gates are not free, and pretending otherwise is how they get ripped out six weeks later.

• Latency and tokens. Classification adds work per task, not per tool call. Budget roughly one extra cheap call per task. If you are running a model over every tool result, you have doubled your spend to save yourself from a policy problem.
• Lost value. A write-time gate is sometimes wrong. You will drop something you needed. That is why evidence lives in the archive with a pointer: recovery is a re-derivation, not a data restore. The gate is allowed to be wrong as long as the underlying material still exists.
• Ownership. You now have a policy, and policies rot. Someone has to own the classification rules the way someone owns the schema.

The middle path that usually works: rule-based classification for the obvious cases, a model only on ambiguous ones, an explicit archive for everything rejected, and a hard rule that instruction-shaped content never gets promoted without a trusted source.

One boundary is not enough

Some fine material always gets through. A conclusion that looked confirmed turns out to rest on one shaky source. A summary carries a trace of the noise it was built from. That is not an argument for skipping the gate; it is an argument for a second pass.

In practice: give beliefs a TTL (30 days is a reasonable default) and require re-confirmation to extend it. Re-check state entries on a schedule. And when a new belief contradicts an old one, do not silently overwrite — record both, mark the older one contested, and put the conflict in a queue. Contradiction is the most valuable signal your memory system will ever produce, and the standard write path throws it away by mutating a row.

What to measure

You cannot manage this without a number. Sample 50 retrieved entries a week and label each one: correct, stale, wrong-source, or should-not-exist. Track that last bucket — it is your contamination rate. Above roughly 2%, you have a write-gate problem, not a retrieval problem, and tuning the retriever will only make the wrong entries arrive faster.

Track two more: how often the agent re-derives something it already knew (too low means you are over-gating and burning money), and the token cost per durable memory entry (a healthy ratio is stark — a few thousand tokens of raw material per surviving belief is normal, not wasteful).

The uncomfortable part is that a write-side boundary makes your agent visibly less capable in the short term. It stops "remembering" things it used to retrieve instantly. Then, three weeks later, it stops confidently repeating last month's dead end — and you discover that the feature you thought was memory was actually a very fast way to launder noise into policy.

Key Takeaways

• Agent memory fails by over-promoting, not by forgetting. Dead ends, transient errors, and scraped text are the sediment, and it does not settle out on its own.
• Read-side filtering cannot repair a contaminated store: summaries inherit errors, confident wrong entries often retrieve better, and derived facts compound the damage.
• Classify every write into evidence, state, commitments, or beliefs — and let the agent reason only from beliefs, each of which points back to a source.
• Gate instruction-shaped content harder than facts. Instructions come from named principals, never from retrieved documents or tool output.
• Accept the trade-offs: one extra cheap call per task, an archive for everything you reject, and a named owner for the classification policy.
• Sample 50 retrieved entries weekly. More than ~2% in the should-not-exist bucket means fix the gate, not the retriever.