Prompt caching, for people who pay the bill

Here is a cost pattern that is nearly universal and almost never examined.

Your AI system sends a request. That request contains a system prompt of a few hundred words, a set of tool definitions, some reference material, a handful of worked examples, and — at the very end — the thing the user actually asked.

The next request contains exactly the same system prompt, exactly the same tool definitions, exactly the same reference material, exactly the same examples, and a different final question.

Whether you paid to process that block again depends on the provider, model, cache configuration, and whether a reusable entry was still available. Before assuming repeated input is being billed at the ordinary rate, inspect the response usage.

What caching does

Prompt caching lets the provider hold a prefix of your request in a prepared state, so subsequent requests that begin with the same content can reuse it instead of processing it again. Reads against a cached prefix are billed at a substantial discount relative to normal input tokens.

Caching reuses computation for matching input rather than deliberately removing context or switching models. It is not a cache of the final answer, and repeated requests can still generate different outputs. Keep evaluations when changing prompt structure: the optimization itself and the prompt rewrite are different changes.

Why it is worth more than it sounds

The instinct is that a system prompt is small and therefore not worth optimizing. That instinct is calibrated on the wrong number.

As an illustrative calculation, suppose a support assistant sends a stable 15,000-token prefix and a 60-token question. The reusable portion is about 99.6% of that input. Measure your actual payload; relevant knowledge-base documents may change from one request to the next.

That makes the repeated input worth examining, but it is not the whole bill. Output, reasoning, tools, cache writes, and misses can all matter. Estimate reuse over the cache lifetime before calling a large prefix a saving.

Caching is a good early cost experiment because it can reduce repeated-input work without discarding the source material. Its value relative to a different model depends on the workload and actual hit rate.

The requirements behind a cache hit

Caching matches a prefix, subject to provider rules for eligible content, minimum length, breakpoints, and expiration. OpenAI GPT-5.6 and later support implicit or explicit breakpoints; Claude requires cache_control configuration. Check the exact model before expecting reuse.

Keep reusable content stable before changing content. An edit prevents reuse of a cached prefix that includes it, but an earlier eligible prefix can still hit. A changed timestamp near the start can be costly; it does not prove that every cache benefit disappears.

Respect the provider’s request structure rather than manually concatenating every field in one universal order. Claude renders tools, system content, then messages. Within the parts you control, keep stable material consistent and place changing material after the reusable portion when the task permits.

Common candidates to inspect are per-request timestamps, identifiers, early personalisation, and randomly reordered documents. Keep document ordering deterministic where that preserves relevance and meaning; do not reorder a ranking solely to chase a cache hit.

How to tell whether it is working

Do not assume. Both providers report cache usage in the response metadata, and this is the only reliable way to know.

Record cache-read and cache-write tokens alongside uncached input, output, latency, and spend. Define whether your metric counts requests with any hit or the fraction of input tokens reused; those are different measures. Alert on meaningful changes against comparable traffic.

This is a good candidate for a metric with a written definition and an owner, because otherwise it becomes one of those numbers that exists on a chart and means nothing to anybody.

What it does not solve

Caching reduces the cost of input you send repeatedly. It does nothing about three other things, and it is worth being clear so it does not get oversold internally.

Output tokens. Output is typically billed at several times the input rate and is not cached, because it is generated fresh each time. If your system produces long responses, that is a separate problem with a separate answer — usually asking for a tighter output format.

Genuinely unique input. If every request contains a different large document, there is no shared prefix to cache. You may still benefit for the instructions above it, but the document itself is paid for each time.

Human review time. Cache pricing does not directly reduce the work of reviewing an answer. Include that cost separately; whether it dominates depends on the application and its review process.

How it interacts with everything else you might do

Caching changes the arithmetic on several decisions people make around it, usually in the same direction.

It can lower the repeated cost of useful instructions. Extra examples or reference material still have write and reuse costs and may affect behavior. Add them when evaluations justify them, then compare the total price across realistic reuse.

It changes the retrieval calculation. If a set of reference documents is shared across many requests, putting them in the cached prefix can beat retrieving different fragments per request — simpler, and often cheaper once the discount applies. That is worth measuring before building a retrieval pipeline you may not need.

It rewards deliberate conversation design. Appending can preserve a reusable prefix; rewriting earlier history may shorten it. Compaction can still be worthwhile when it reduces irrelevant context or total cost. Measure both effects instead of treating cache hit rate as the only objective.

The order to do things in

If the AI bill feels too high, use the following sequence as an investigation plan. Let measured costs and quality constraints determine which change comes first.

First inspect cache usage and input hygiene. Then compare output length, effort, and model choice against quality requirements. There is no universal order that is free of trade-offs: changing the prompt or context can change the result even when the intention is only to save money.

Most teams start at the far end of that list, debating models, and never do the first two — which is how you end up having a difficult conversation about answer quality to recover a saving that was sitting in the prompt structure all along.

Where to start

Three steps, in order, none of which take long.

Measure the overlap. Compare representative payloads in a safe development workflow, with sensitive content protected. Count the reusable portion and check whether it meets the model’s cache minimum. Do not add raw customer prompts to general logs just to estimate savings.

Configure and price reuse. Choose the provider-supported cache behavior, keep stable content consistent, and account for writes and expiration. A prefix that is never reused before it expires may not save money.

Instrument the hit rate. Log it from day one, alert on it falling. A cache is a thing that silently stops working, and silent failures are the expensive kind.

The useful outcome is a measured change in cost per accepted result. Preserve the prompt’s meaning, check whether reuse occurs, and compare the actual bill before calling the optimization a success.

Source check: 10 October 2026. Checked prefix matching, partial invalidation, cache configuration, lifetime/eligibility requirements, and usage categories. Sources: OpenAI prompt caching; Claude prompt caching. The recommendations are editorial guidance; no comparative performance or cost benchmark was run for this update.

Try this

In a protected development workflow, compare two representative request structures without dumping private customer content into general logs. Identify eligible repeated prefixes, then inspect actual cache reads and writes across repeated requests. Include expiration, output and retry costs. Compare the observed bill before treating overlap as realized savings.