Similarity cannot tell you which fact is current

A correction that names its subject is easy to retrieve. A long history is what actually breaks it.

If you have shipped an agent that has to stay current, you have met this failure. The model tells a user that a company's chief executive is someone who left two years ago. The support assistant quotes a policy that was replaced last quarter. Somebody asks "is it the model" and the answer is no, the model is fine, the right document is even in the index. The system retrieved a true sentence. It just retrieved the wrong true sentence, because two facts about the same thing looked equally relevant and nothing in the pipeline knew which one was live.

We wanted to know how often this really happens, and where, so we built a benchmark on real fact-changes and measured it instead of guessing. The result is more specific than the usual warning. The dramatic version of the failure is rarer than the synthetic demos suggest. The real version is narrow, mechanical, and fixable with something cheap.

The failure that does not happen as often as you have been told

There is a popular demo where you bury a corrected fact in a pile of notes, ask the question, and watch RAG confidently serve the stale value. It is a good scare. It is also, on real data, substantially an artefact of how those benchmarks phrase the correction.

When an update names its subject, the way a real record does, it stays retrievable. "So-and-so became chief executive of Acme in 2026" shares the entity and the role with the query, so similarity ranks it where it belongs. We pushed this hard: a correction that names its subject stays findable even when it is buried under a large pile of unrelated distractors. Volume alone is not the enemy. A single, well-formed correction is not the enemy either. If your corpus asserts each fact once, in sentences that name what they are about, plain top-k retrieval mostly does the right thing, and the scary demo does not reproduce.

That matters because it tells you where not to spend money. You do not need a memory architecture to survive an occasional correction. You need one specific thing to survive a specific situation, and most teams cannot tell whether they are in it.

The failure that does happen: a history of near-identical facts

Here is the situation that actually breaks retrieval. An entity accumulates a long history of the same attribute over time. A footballer's clubs, season after season. A club's head coaches across a decade. A vendor's account owners as the territory gets reshuffled. Every one of those statements is about the same entity and the same attribute, so every one of them is near-identical in embedding space.

Now similarity has no reliable notion of temporal validity. The current value and ten stale values sit at roughly the same distance from the query, because they differ only in the part the embedding weights least. An embedding can sometimes catch a date token or a turn of phrase, but it has no dependable signal for which value is the live one. Top-k returns a handful of them, the current value often misses the budget, and the reader serves whichever statement happened to rank highest. Nothing in pure cosine similarity says "this one is current and those ten are history," because being current is not a semantic property. It is a temporal one.

We measured how often this leaks. Pooled across three relations on real Wikidata fact-changes, plain RAG served a stale, superseded value 12.8% of the time, with answer accuracy at 80.8%. That is not catastrophic, and it is well below the failure rates that buried-needle synthetic benchmarks tend to advertise. It is a steady, quiet wrongness on exactly the entities that have the richest histories, which are often the entities your users care about most.

We measured it with the contamination ruled out

The trap in any benchmark like this is that the model already knows the answer from pre-training, so a high score measures world knowledge, not memory use. We controlled for it directly. The source is real Wikidata start and end qualifiers, which give auditable temporal gold for free, and the entities are obscure on purpose, journeyman footballers and lower-division clubs that the model has no parametric opinion about.

The control is a closed-book run: the same questions with the memory emptied. If the model answers correctly with no context, it is leaking pre-training, and that case is thrown out. On the succession set, the model got 0 of 390 correct closed-book. So every open-book point is retrieval and ranking doing the work, not the model reciting something it already memorised. Without that check, you cannot tell a real retrieval win from a contamination artefact, and it gets skipped more often than it should.

The fix is a recency lane, and it is cheap

The fix is not a memory system. It is a second retrieval lane that runs alongside the semantic one and costs almost nothing.

Keep the same reader budget. Fill most of the slots the way you already do, by similarity. Fill the rest with the most recent statement that is actually about the queried entity, found by cheap term overlap, with no per-statement model call. The semantic lane finds the context; the recency lane guarantees the current value is in the candidate set even when its cosine score is mediocre. Then let the reader see both.

On the same benchmark, that one change cut stale-serving from 12.8% to 3.8% and lifted accuracy from 80.8% to 90.8%. Because both systems ran the same cases, the paired test is the honest one: the recency lane fixed 38 stale answers and introduced 3 new ones, which is not a wash, it is a real effect (two-sided sign test, p < 0.001). It is the same shape of fix as context-prepended chunking or a metadata filter. Unglamorous, interpretable, cheap, and far higher leverage than the next embedding model.

The benchmark ran on real Wikidata fact-changes across three relations (n = 390), with start and end qualifiers as the gold label, one shared reader (gemini-3.5-flash) and embedder (gemini-embedding-001), and the same fixed reader budget (top-k = 5) for both systems on the same cases. "Stale" is the rate of serving a value whose validity window has closed. Contamination was ruled out by a closed-book pass: the model answered 0 of 390 correctly with the memory emptied. Point estimates are shown; 95% Wilson intervals were calculated in the run output. The benchmark is open, so you can rebuild the cases and rerun both systems on the same budget: StaleRAG.

The honest limits, because they are the point

A single pooled number flatters this. The per-relation split, in the table above, is where the truth lives.

The failure and the fix are concentrated in dense, clean histories. Head coaches went to zero stale answers and football clubs dropped by roughly two thirds, while heads of government, where plain RAG already scored 94%, barely moved because there was almost nothing to fix. So this is partly a property of long, dense, low-variance histories, not a universal RAG defect. If your data does not have those histories, this fix is solving a problem you do not have.

It is also a mitigation, not a cure. The pooled leak dropped to 4%, not to zero, and the lane made three cases worse. The entity matching is term overlap, which assumes statements name their entity cleanly, the way structured records do. It will degrade on messy prose full of pronouns and name variants, which we have not yet tested. And recency reranking is a known technique, not an invention. The contribution is not the lane. It is knowing, from measurement on real data, exactly when reaching for it pays and when it is dead weight.

Where this leaves you

If you are running an agent that has to stay current, the order to actually work in:

1. Check whether you even have histories. Pull the entities your users ask about most and count how many times each attribute has changed. If most facts are asserted once, you do not have this failure, and a memory layer would be solving nothing.
2. Measure stale-serving as its own metric. Not accuracy. The rate at which the system serves a value that has since been superseded. You cannot fix what you are not separately counting.
3. Add a recency lane only where the histories are dense. It is small to add when your records already carry entity and date fields, it does not touch your reader, and it is interpretable enough to debug.
4. Do not reach for a memory architecture first. The expensive option rarely beats the cheap lane on the failure that is actually hurting you. Prove you need more before you build more.

Being current is not a semantic property, so a semantic index cannot rank for it. That is the whole failure in one line. The teams that ship reliable agents are not the ones with the cleverest memory layer. They are the ones who measured where their retrieval actually serves yesterday's truth, and added the cheapest lane that fixes it.

That is also why the first week of a retrieval engagement goes into measurement, not architecture. Counting how often each system serves a superseded value, broken out by entity type, is what tells us whether you need a recency lane or whether your data never had the histories that would justify one. Most teams never run that count, so their freshness failures stay invisible until a user catches the agent quoting a fact that stopped being true a year ago.