Prompt Caching and How to Think About It

You do not cache a prompt in the abstract.

You cache the beginning of a request.

That small distinction explains most prompt caching surprises. A cache hit is not about semantic similarity. It is about whether the beginning of this request matches a beginning you cached earlier.

The first request writes the cache.

The launch strategist prompt is the stable part. You expect to send it again and again, so you mark it with Anthropic's cache control.

system = [{
    "type": "text",
    "text": "You are a launch strategist...",
    "cache_control": {"type": "ephemeral", "ttl": "1h"}
}]

That marker means: cache everything up to here. The first request writes the cache. The payoff comes on later requests.

The second request reuses the same prefix.

The conversation changed, but the cached prefix did not. The marker is still after the same launch strategist prompt, so Claude can read that part from cache.

The cache key includes what comes before the system prompt.

In the Messages API, the useful order to remember is:

tools -> system -> messages

If your marker is on the system block, the prefix may be tools + system, not just system. Add a tool, remove a tool, or change a tool definition, and you can create a different cache.

JSON schemas can surprise you for the same reason.

Structured output is useful. It can also change the request shape. Depending on the SDK feature, your schema or grammar may become part of what the model sees before ordinary messages.

That means normal chat and JSON-shaped calls may land in separate cache lanes. This is not bad. It is just something to expect and measure.

Now make the repeated prompt huge.

A real app may carry a persona, policy rules, examples, domain documents, customer notes, and formatting rules. That stable context might be 10,000 tokens, 50,000 tokens, or much more.

Prompt caching turns that repeated stable prefix into a one-time write followed by cheaper reads.

This is the part most people get right. The next part is where costs sneak back in.

The conversation can become the new problem.

Caching the system prompt does not automatically cache the conversation. If the marker stays after the system prompt, every old message after that marker is still ordinary input.

Early in a chat, that is fine. Later, the conversation can become larger than the original prompt.

Stop the cached prefix before the part that changes.

Some workflows replay a conversation and then add a fresh task instruction:

[ system ][ conversation so far ][ new task instruction ]

The conversation so far is stable across related calls. The new task instruction changes each call. Put the marker at the end of the stable conversation, not after the changing instruction.

Warm the cache before the user needs it.

A cache write can be slow and expensive. If your app has a large stable prompt, you can warm the cache when the user opens the chat, before they send the first real message.

But warm-up only helps if it matches the real request shape. If real chat uses tools, warm with the same tools. If real chat uses a different model or schema path, expect a different cache.

Do not look only at total input tokens.

For each model call, log three buckets:

• ordinary input tokens,
• cache creation tokens,
• cache read tokens.

A request can contain many tokens and still be cheap if most are cache reads. A smaller request can cost more if it keeps writing new caches.

Five rules are enough.

1. Prompt caching caches a prefix, not a vibe.
2. The prefix includes request structure: tools, system, messages.
3. Changing anything before the marker can create a new cache.
4. Put the marker after stable content, before changing content.
5. Log cache reads and writes separately.

Once you can point to the prefix, you can usually predict whether the next call will hit, miss, or create a new cache lane.

Different APIs, same request shape.

Anthropic, OpenAI, and Gemini expose different controls. The common idea is still the request prefix: put reusable content first, and changing content later.