A million tokens of context, and why retrieval still matters

GPT-6 Astra documents a 1,050,000-token context window, a maximum input of 922,000 tokens, and a 128,000-token output limit. Claude Fable 5.1 lists a one-million-token context window and 128,000-token maximum output. These limits leave room for substantial source material, but the advertised window is not the same thing as an unrestricted input budget.

Every time context windows jump, the same claim goes around: retrieval is dead, just put everything in the prompt. It was wrong at 100,000 tokens and it is wrong at a million, but for more interesting reasons than before.

What genuinely changed

A large context window removes a category of engineering that only ever existed to work around a limit.

Small chunks can lose relationships between distant passages. A question about a termination clause and a payment schedule needs both pieces and their surrounding conditions. A top-five retrieval step may miss one; full-document input is one alternative to test, alongside better retrieval and document expansion.

If the whole document fits within your cost, latency, and privacy constraints, compare full-document input with retrieval on the same questions. Keep the simpler design only if it preserves the evidence and answer quality you need.

Five reasons retrieval does not go away

Cost. Large requests need the applicable context tier, not just the headline rate. For example, 500,000 uncached input tokens on Astra fall above its 272,000-token threshold: at $20 per million for that tier, input costs $10 before output, tool use, or any cache-write treatment. You cannot send a million input tokens to a model whose maximum input is 922,000. Retrieval can reduce the amount you need to send.

Latency. Larger inputs can increase time to the first useful answer. Caching, model choice, output length, and tools also affect latency. Measure the complete request rather than assuming a fixed slowdown.

Precision. More context is not a guarantee of a better answer. Include tests with superseded policies, distracting documents, and evidence spread across sections. Compare whether each design selects the correct source and explains its conclusion.

Freshness. A context window is filled at request time from somewhere. That somewhere is a retrieval system, whether or not you call it one. The question is never “retrieval or not” but “how good is the thing that chooses what to include”.

Auditability. Log the documents, versions, and passages supplied to the model. Retrieval can make that selection explicit, but full-document input can also be auditable if you retain its provenance and require references to the relevant passages.

What good looks like now

A useful design to compare is retrieval that expands selected passages into larger sections or whole documents. It can preserve local detail without sending an entire collection on every request.

Concretely, test three complete policies against fifteen isolated paragraphs when questions depend on whole-policy context. For a large collection of independent facts, smaller passages may work better. The unit of retrieval should follow the question and the evidence, not a slogan about context size.

This tends to improve answers and reduce engineering at the same time, which is rare enough to be worth noticing.

Three patterns that use a big window well

Whole-document reasoning. Compare entire-document input when the task depends on cross-references or internal consistency. Check that the answer actually connects the relevant passages, and include the full input cost in the comparison.

Long-running work with accumulated state. Agentic tasks that run for a while build up history: what was tried, what failed, what the codebase turned out to look like. A big window means that history survives instead of being summarized away at every step, and summarization is where agents most often lose the thread.

Few-shot with real examples. A larger window can hold complete examples and edge cases. Add them when evaluations show that they help; neither a fixed example count nor a lower cost than fine-tuning is guaranteed.

How to decide what goes in

A simple test that survives contact with production: for every block of context, be able to say what question it answers.

If you cannot — if a section is in the prompt because it seemed relevant, or because it was already in the variable — take it out and measure. In most systems a meaningful share of the context is there for no articulable reason, and removing it makes answers better and cheaper at the same time.

The related discipline is clear organization: distinguish the instructions, question, and source material, and define what should happen when the context does not contain the answer. Test that fallback with missing and contradictory evidence. One abstention instruction is a specification, not a guarantee against invention.

The thing nobody budgets for

Large context windows make it very easy to spend money without deciding to.

A prompt that quietly includes a whole repository because it was simpler than choosing files is a decision, made by nobody, that shows up on an invoice six weeks later. And it is invisible in testing, because during testing you ran it eleven times.

The discipline is to count tokens per unit of work rather than watching only a monthly total. A 400,000-token input is twenty times the size of a 20,000-token input. Its price need not be twenty times as high: context tiers, cache reads, and writes change the arithmetic.

Output is the other half, and it is smaller

The context window and output limit are different limits. The two models named above each list a 128,000-token maximum output. Reserve space for the response and, where applicable, reasoning; do not treat the headline window as room for that many input tokens plus an unlimited reply.

That asymmetry is fine for most work and awkward for one specific pattern: transforming a very large document into a similarly large document. Translating a long manual, or rewriting an entire codebase file by file, hits the output ceiling long before the input one. The answer is the same as it has always been — process in pieces, and keep the pieces aligned to something meaningful like a chapter or a module rather than to an arbitrary token count.

The question underneath

Large context windows are an engineering convenience. They do not change what makes an AI feature worth building, and they have a mild tendency to obscure it — because when everything fits, it is tempting to skip the step where you decide what actually matters.

That step is still the work. Which decision is this improving, what does the model need to see in order to improve it, and how will you know if it got it wrong. A million tokens makes the second question easier to answer badly and no easier to answer well.

Source check: 10 October 2026. Checked Astra and Fable 5.1 context/output limits and Astra’s long-context input pricing. Sources: GPT-6 Astra; Claude Fable 5.1. The recommendations are editorial guidance; no comparative performance or cost benchmark was run for this update.

Try this

Measure one representative request with the target model: instructions, tools, retrieved material, history, user input and output. Compare a whole-document version with a retrieval version against the same evidence-based questions, including an unanswered one. Record omissions, citations, latency, applicable context-tier charges and cache usage before choosing a design.