Claude: Prompt Caching for the Person Paying

Your system sends a large block of identical bytes on every request: instructions, tool definitions, reference material, examples. Then it sends the one short thing the user asked. Prompt caching lets the provider keep that stable block prepared so repeat sends are billed at a discount, and it is the only cost lever that has no quality cost at all.

The interesting part is not that caching exists. It is that failing to hit it is completely invisible. Nothing errors, nothing degrades, and the only symptom arrives on the invoice weeks later.

Why this exists

Caching works on a prefix, and a prefix is all-or-nothing from the first byte that differs. One changed character near the top invalidates everything after it and you get no benefit whatsoever — while still returning a correct answer, at full price, with no warning anywhere in the response your code looks at.

The ordering mistake behind it is always mundane. A current date at the top of the system prompt. A request ID added for logging. A user’s name placed above the shared instructions instead of below. Retrieved documents in whatever order the search returned them.

The nastiest version is this one: tool definitions serialized by iterating a map whose order is not guaranteed across processes. Inside one worker the order is stable, so the cache hits and everything looks fine in development, where there is a single process and a warm cache. In production, across restarts and a pool of workers, each process builds a different prefix and the hit rate collapses. The code is identical in both places. That is why this one survives review.

How to use it

Measure the overlap before you write anything. Print the full payload for two consecutive real requests and diff them byte for byte. The identical portion is what you are currently paying full price to resend. That number is the size of the prize, and you have it before writing any code.

Order strictly from stable to variable — system instructions, tool definitions, reference material, examples, conversation history, then the current question — and put the cache boundary at the last byte that is identical across requests, not at a section break that feels tidy.

Sort everything that arrives from somewhere else. Retrieval results, tool lists, key order in any serialized structure. Determinism is the whole requirement.

Then make the regression impossible to merge. Write a test that builds the payload for two different inputs and asserts the first N bytes are identical. This is the step most teams skip and the one that matters, because the change that breaks a cache — a field added near the top — looks harmless in review in every respect except this one. Then log the cached-token count beside the regular input count, put the ratio next to spend, and alert when it falls.

One consequence worth using: once the prefix is cached, longer and clearer instructions become the cheap option. Teams trim prompts to save money and pay for it in quality; with caching in place, that trade largely disappears.

Where it goes wrong

Output is not cached. On Claude Opus 5, output is billed at five times input; on Fable 5.1, the same five-to-one. If your responses are long, caching will not touch that, and a tighter output format will.

Low volume can lose. Cache entries expire. If your requests arrive further apart than the entry lives, you pay to write the cache repeatedly and never read it. Check whether your provider bills a cache write above the base input rate — if it does, caching an occasional-use system can cost more than not caching it at all.

Unique input has no prefix to share. If every request carries a different large document, only the instructions above it benefit.

And caching makes waste affordable, which is not the same as removing it. A bloated prompt that now costs less is still bloated, and human review time — usually the largest cost in the whole system — is unaffected by any of this.