A million token context window is a wider desk. It is not a filing cabinet, and treating it as one is the source of a lot of disappointment.

The confusion is understandable. For a while, a big window genuinely does substitute for memory, and the substitution is so convenient that teams build on it without noticing they have taken on a debt.

The distinction

Context is what the model can see right now. Memory is what it can retrieve later, across sessions, and be right about.

The two get conflated because a big window makes the memory problem invisible for a while. You can stuff an entire case history into the prompt and it works, until the history outgrows the window, or the session ends and everything in it is gone, or a second agent needs the same information and has no way to reach it.

Memory has properties context does not. It survives restarts. It can be queried by something other than the model. It can be corrected. A summary sitting in a context window has none of those.

The quality cost of filling the window

There is a cost to a full window that is easy to miss because it never announces itself.

Every irrelevant document competes for attention with the one that matters. Past a certain density, adding more context measurably degrades the answer rather than improving it. The model does not warn you when this happens. It just gets vaguer, hedges more, and starts producing answers that are defensible rather than specific.

This is why the instinct to retrieve twenty chunks to be safe is not safe. Four good chunks beat twenty mediocre ones, and the difference is visible in the output long before anyone thinks to blame retrieval volume.

• Retrieval breadth and answer quality are not monotonically related.

• Vagueness is the symptom of an overfull window, not a model limitation.

• Cost and latency scale with what you send, every single turn.

The durable pattern

The pattern that holds up is to keep the window small and deliberate. Retrieve what this turn needs, write outcomes to a real store, and let the next turn retrieve again.

That store can be a custom object, a summary field on the record, or a proper vector index. The technology matters far less than the two properties: it survives the session, and it can be read by something other than the model.

In a Salesforce context this is usually simpler than teams expect. The record already exists. The history already has a home. Writing a structured summary back to a field is often the entire memory layer, and it has the considerable advantage that a human can read and correct it.

• Write conclusions to a field or object, not into a rolling transcript.

• Retrieve per turn rather than carrying everything forward.

• Make stored memory human readable so it can be audited and fixed.

• Set a hard budget for how much context any turn may consume.

What long windows are genuinely good for

None of this means large windows are not useful. They are, for a specific shape of task.

Anything that requires reasoning across a whole document at once benefits enormously: comparing two contracts, finding an inconsistency across a long thread, summarising something that resists chunking. These are single pass tasks where the alternative is a worse decomposition.

The mistake is using that capability as a persistence layer for a multi turn process, which is a different problem that happens to be temporarily solvable the same way.

The short version

Treat the window as working memory with a hard budget, and treat memory as something you write down.

Do that and most context problems turn back into retrieval problems, which are the kind you can actually engineer, measure, and fix.