Long context or retrieval?

Fifty questions over a document set of up to a million tokens: the whole set in every prompt, with and without the prompt cache, against the top k chunks, in dollars, seconds and answers found.

Loading the animation…

Concept

Context windows of a million tokens raise an obvious question: why retrieve at all? Put the whole document set in the prompt and let the model find the answer. The engine prices that against retrieval, question by question, from the engine's dated list prices and cache rules (the same accounting as Agent Harnesses Explained, prompt caching) and token counts measured on this corpus: a system prompt of 33 tokens, questions of 13.6 tokens on average and chunks of 180.

Fifty questions, a minute apart, about a set of a million tokens, at Claude Sonnet 4.6's list prices:

The whole set in every prompt costs $3.00 a question, $150.04 for the fifty, and with this simple latency model the first token arrives after 200 s of prefill, every time.
With the prompt cache, the first call writes the set to the cache at a premium ($3.75) and every later call reads it at a tenth of the input price ($0.30): $18.49 for the fifty, 8.11 times cheaper, and the later first tokens arrive in 0.403 s. The cache pays for itself from the second question, as long as the questions come within its lifetime.
Retrieval of the top five reranked chunks costs $0.00359 a question, $0.18 for the fifty: another 103 times less than the cached long prompt, and its cost does not depend on the size of the set at all.

What does retrieval give up? On this corpus, the answer is in the top five chunks for 99.5% of the questions (top three: 99.0%; measured in chapter 4). The long prompt always contains the answer, but containing it is not the same as using it: chapter 6 showed why an answer buried in the middle of a long input is used less often. The long prompt wins when questions need many scattered parts of the set at once, or when the set is small: at the corpus's own size (37,499 tokens) the cached long prompt costs $0.73 for the fifty questions, and retrieval's machinery may not be worth building.

Maths

Per question, the long prompt costs $(N + q)\,p_{\text{in}}$ and retrieval $(k \bar c + q)\,p_{\text{in}}$ , so their ratio is about $N / (k \bar c)$ : here $10^6 / (5 \times 180) \approx 1{,}100$ before output tokens. Caching divides the long prompt's per-question cost by $p_{\text{in}} / p_{\text{read}}$ (10 at these list prices) after a first call that costs $p_{\text{write}} / p_{\text{in}}$ times more (1.25 for the Claude models; GPT-5 mini charges no write premium). The cached long prompt overtakes the uncached one at the second question, and never overtakes retrieval while $N\,p_{\text{read}} > k \bar c\,p_{\text{in}}$ , that is while the set is more than $k \bar c \,p_{\text{in}}/p_{\text{read}} \approx 9{,}000$ tokens.

Time to first token is the overhead plus the uncached prompt over the prefill rate: $N / 5{,}000$ seconds for an uncached million-token prompt.

Code

Each question's cost, cut from the vendored engine (runQuestions): the cached prefix is read from the cache once it has been written and is still alive.

if (s === "long+cache") {
  if (alive >= 0 && now - alive <= price.ttl_ms) {
    const c = Math.floor(prefix / price.block) * price.block;
    cached = c >= price.min_tokens ? c : 0;
  }