Fewer tokens, longer memory

The result

On a 60-turn conversation, questions about something said earlier were answered correctly 18 of 18 times (before: 0 of 18). The prompt shrank 71% (118,657 to 34,697 tokens across 24 questions) and model cost fell 92%. Measured 15 October 2026.

That is a figure for questions that depend on a long history. On an ordinary single question about attached data the saving is small and mostly within run-to-run noise, so we do not claim a general “N times fewer tokens”.

What changed

• Recall instead of replay. Before, every question carried the last 8 messages whole. Now it carries the last two turns and the earlier turns that match the question, as short excerpts picked sentence by sentence. Matching uses the words and numbers of the question; it adds no model call and no new storage.
• A direct route for “what did I say earlier”. A question that clearly asks about the past, with nothing new to compute, is answered on the small model instead of the full agent loop.
• Long tool results stay out of the prompt. A result over 2,400 characters goes in as its head plus a marker, and the rest is fetched only if the answer needs it. This is what makes web questions consistently smaller.
• Your permissions still apply. Anything hidden by your AI-data-access settings is removed before recall runs, so it cannot come back.

How far back it reaches

Without any model calls, the six planted facts were reachable inside the prompt 6 of 6 times at 60, 120, 250, 500 and 1,000 turns, at about 2,800 characters of history per question every time. The old shape reached none of them. Finding the turn took about 14 ms at 250 turns and 57 ms at 1,000.

How it was measured

• The real agent loop against the real model deployments. Tokens and cost come from the provider’s usage figures, summed over every model call.
• A generated 60-turn sales conversation with six planted facts (a return rate, a promo code, a revenue figure, a churn rate, a forecast, an agreed exclusion rule) and many nearby turns with other regions, months and numbers that a careless lookup would confuse with them. Six questions need an old turn and two need only the latest. Each was asked three times, 24 runs per setting. The answers are planted by the generator, never read from a model.
• Prices are third-party-reported list prices and will be checked against the provider’s price sheet.

What this does not claim

• It is one generated conversation and 24 runs per setting: a large effect, a small sample.
• Matching is by words. A question that paraphrases heavily (“that thing with the March spike”) will recall less; a search by meaning over the same turns is the next step.
• Memory across separate chats (the stored library) was not part of this measurement.
• On 28 ordinary single questions the prompt was about 17% lower on average, but most of that is the agent choosing a different route on different runs. Pass rates were equal within noise.
• The provider’s own prefix cache already served 25 to 30% of prompt tokens before this change; we did not raise it and do not count it as a saving.

Availability

This is being switched on account by account, so you may not see it yet. It can be turned off at any time and nothing is stored or migrated, so turning it off returns the previous behaviour on the next question. Separately, how much your plan can keep in memory is on the Memory page and in plans.