Home › Blog › Prompt Caching, Explained

Prompt Caching, Explained

By Victor, founder of apicuts

If your app sends the same long system prompt, tool list or document on every request, you're paying full price to resend it. Prompt caching lets the provider reuse that prefix and bill it at a lower rate.

How it works

The provider stores the processed start of your prompt. When the next request begins with exactly the same tokens, it reuses that work and bills those tokens at a reduced rate. The match is on the prefix: change anything near the start and everything after it misses the cache.

At each provider

• OpenAI: caching is automatic for prompts of 1,024 tokens or more. The usage data reports cached_tokens, and the discount depends on the model.
• Anthropic: mark the end of the stable prefix with cache_control, or turn on automatic caching. Cache reads cost about a tenth of normal input on most models; cache writes cost 1.25× input for the default 5-minute lifetime or 2× for one hour. The usage data reports cache_creation_input_tokens and cache_read_input_tokens.
• Google Gemini: recent models cache repeated prefixes automatically, and explicit context caching lets you store a large document or instruction for a set time, with a storage fee.

Why caching silently misses

• A timestamp, request ID or user name sits in the system prompt.
• Tool definitions arrive in a different order on each request.
• JSON is serialized with keys in an unstable order.
• The stable part is shorter than the provider's minimum cacheable length.
• Requests sharing the prefix are further apart than the cache lifetime.

How to check it works

Track the share of input tokens read from cache for each route. On a healthy chat or agent route it should be high and steady. If it drops to zero after a deploy, something at the start of a prompt changed: that's exactly what cost regression protection catches.

Frequently asked questions

Related