The window is the working memory

A twelve-question task in a window too small for it: what the agent knows grows with each read and shrinks when the window is trimmed, compacted or searched again, and what each choice costs in tokens.

Loading the animation…

Concept

A model has no memory between calls. Everything it can use to choose its next step is in the prompt: the system prompt, the task, and whatever the agent has put there since, its tool results above all. That prompt is the context window, and for an agent it plays the part of working memory. It is also the bill: every model call is charged for the whole window again, so a task's input cost is the window's size summed over its calls, $C_{\text{in}} = \sum_t |W_t|$ .

The animation runs a research task of twelve questions, two from each of the six articles in the corpus. For each question the agent searches (BM25, chapter 2) and reads the best chunk, then the next, until it has read the chunk that holds the answer. Each read is one model call. With no limit the window grows to 3,033 tokens and the task costs 26,616 input tokens; every fact stays in view, so all 12 questions are answered.

Give the window a budget of 1,500 tokens and something has to go. Three policies:

Truncate: drop the oldest read. Cheap, 18,094 tokens, but the facts in the dropped reads go with them: at the end the agent can answer only 5 of the 12 questions. What it "knows" (the dots) grows with each read and shrinks with each drop.
Compact: when the window overflows, one extra model call summarises every read but the latest into a line per fact, and the summary replaces them. The agent keeps all 12 facts for 16,768 tokens, 63% of the unbounded cost, in 2 compactions.
Retrieve: truncate, and when a question's fact has gone, search for it again before answering. Every question is answered (12), but the agent reads 30 chunks instead of 15 and spends 39,175 tokens, more than with no limit at all, because each extra read pays for a full window.

That summariser is perfect: it keeps one line per fact. Switch the summariser to lossy (each line kept with probability 0.75 at every compaction, an illustrative rate) and compaction at 1,500 tokens keeps only 7 of the 12 facts; chapter 7 measures what survives over a longer task.

Smaller budgets make the trade sharper: at 1,000 tokens truncation keeps 3 facts; at 3,000 the whole task almost fits and the policies barely differ. The rest of the site takes these pieces apart: how retrieval finds the right chunk (chapters 2 to 4), how the text is cut into chunks in the first place (chapter 5), what to pack into the window and where (chapter 6), what survives compaction (chapter 7), what an agent remembers between sessions (chapter 8), and when a long prompt beats retrieval (chapter 9).

Maths

Let $W_t$ be the window at call $t$ and $T$ the number of calls. The input cost is $C_{\text{in}} = \sum_{t=1}^{T} |W_t|$ . With no budget, $|W_t|$ grows by each read's length $\ell_t$ , so the cost grows with the square of the number of reads: $C_{\text{in}} \approx \sum_t (|W_0| + \sum_{s<t} \ell_s)$ .

Truncation caps $|W_t| \le B$ and so caps the cost at $T B$ , but a fact survives only while its read is among the most recent $\approx (B - |W_0|)/\bar\ell$ reads. Compaction replaces $k$ reads of total length $\sum \ell$ by a summary of length $|S| \ll \sum \ell$ , at the price of one more call (here 18 calls against 16). Re-retrieval keeps $|W_t| \le B$ but adds calls, each of cost up to $B$ : it buys recall with $T$ .

Code

The truncation policy, cut from the vendored engine (windowRun): drop the oldest read (or the oldest summary line) until the window fits, never the read just added.

const truncate = (qi: number | null) => {
  while (used(items) > budget) {
    const k = items.findIndex((it) => it.kind === "read" || it.kind === "summary");
    if (k < 0 || k === items.length - 1) break;