Prompt Caching for Claude APIs: Verify Hits and Fix Silent Misses (2026)
Prompt caching is a prefix-matching system, not a switch that makes every request cheaper. Learn where to place cache breakpoints, how to prove a write and a later hit, and why apparently identical requests can miss.
1. The mental model: cache an ordered prefix, not a bag of text
Anthropic's cache follows the prompt in a defined order: tools, then system, then messages. A breakpoint represents the entire prefix through the marked block. The next request can read that entry only when the relevant earlier content still matches. Reusing the same paragraph somewhere else is not enough. If a tool definition, system instruction, earlier message, model choice, or relevant thinking configuration changes before the breakpoint, the later prefix may need a new cache entry. This makes prompt organization more important than the number of cache markers. Put durable instructions, tool schemas, examples, or a repeatedly queried document first. Put timestamps, request IDs, current user input, and frequently changing retrieval results after the last stable breakpoint. Think in terms of one immutable prefix followed by a mutable suffix.
2. Automatic caching versus explicit breakpoints
Anthropic currently supports two ways to enable caching. Automatic caching adds a top-level cache_control field and moves the effective breakpoint as a conversation grows. It is convenient for ordinary multi-turn chat. Explicit caching places cache_control on a particular content block and is better when you know exactly which material remains stable. Explicit control is usually easier to diagnose in batch processing, RAG, and agent systems because the boundary is visible in the request. Anthropic allows multiple explicit breakpoints when different sections change at different rates, but more markers do not rescue an unstable prefix. A write occurs at a breakpoint; backward lookup searches for entries that earlier requests actually wrote. The safest starting pattern is one breakpoint on the final block of a long, stable system prefix, with the current question after it.
{
"model": "YOUR_MODEL_ID",
"max_tokens": 128,
"system": [
{
"type": "text",
"text": "LONG_STABLE_INSTRUCTIONS_OR_DOCUMENT",
"cache_control": { "type": "ephemeral" }
}
],
"messages": [
{
"role": "user",
"content": "A question that may change on every request"
}
]
}3. Read all three input fields correctly
The response usage object is the evidence. On a new entry, cache_creation_input_tokens reports tokens written at the breakpoint. On a hit, cache_read_input_tokens reports tokens retrieved from an existing entry. input_tokens is not the whole prompt when caching is active; it represents uncached tokens after the final breakpoint. Calculate total input as input_tokens + cache_creation_input_tokens + cache_read_input_tokens. For a fresh prefix, the first response should normally show creation greater than zero and read equal to zero. A later sequential request within the cache lifetime should move the reusable prefix into the read field. The uncached question remains in input_tokens. A response can be valid while both cache fields are zero—for example, when the marked prefix is below that model's minimum cacheable length. Never infer a hit from latency alone, and do not infer total prompt size from input_tokens alone.
4. Run a two-request test that produces auditable evidence
Use a long reference file that comfortably exceeds the selected model's documented minimum. The script below creates one unique test prefix, holds it unchanged in memory, and sends two requests in sequence. The changing question is placed after the breakpoint. Because the second request starts only after the first response has returned, it can see the cache entry created by the first. Inspect the printed fields, not the answer text. For a fresh run, request one should show cache_creation_input_tokens greater than zero. Request two should show cache_read_input_tokens greater than zero. If a matching entry already exists, the first call may itself be a read; the unique run tag prevents that in this test. This example uses cocodot's verified Anthropic-format endpoint, which accepts an x-api-key and returns cache usage fields when the upstream response contains them.
import json
import os
import time
import urllib.request
BASE = "https://cocodot.co/api/ai"
KEY = os.environ["COCODOT_API_KEY"]
MODEL = os.environ["MODEL_ID"]
with open("reference.txt", "r", encoding="utf-8") as f:
document = f.read()
# Generated once, then kept identical for both calls.
run_tag = str(time.time_ns())
stable_prefix = f"Cache verification run: {run_tag}\n\n{document}"
def call(question):
body = {
"model": MODEL,
"max_tokens": 64,
"system": [{
"type": "text",
"text": stable_prefix,
"cache_control": {"type": "ephemeral"},
}],
"messages": [{"role": "user", "content": question}],
}
req = urllib.request.Request(
BASE + "/v1/messages",
data=json.dumps(body).encode("utf-8"),
headers={
"x-api-key": KEY,
"anthropic-version": "2023-06-01",
"content-type": "application/json",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=120) as response:
return json.load(response)["usage"]
first = call("Give this reference a five-word label.")
print("first ", first)
second = call("Name the reference's main subject.")
print("second", second)
assert first.get("cache_creation_input_tokens", 0) > 0
assert second.get("cache_read_input_tokens", 0) > 05. Six common reasons a cache silently misses
First, a timestamp, nonce, session ID, or request ID is inserted before the breakpoint on every call. Second, tool definitions are emitted in a different order or their schemas change. Third, dynamic RAG results are placed in the system prefix even though they vary by query. Fourth, an earlier conversation turn is edited, removed, summarized, or reordered. Fifth, the marked prefix is shorter than the selected model's minimum cacheable length. Sixth, the next call arrives after the entry expires. Model and thinking-setting changes can also alter the effective prompt and invalidate cached message prefixes. Fixes follow directly from the causes: generate volatile metadata after the breakpoint, sort tools deterministically, keep stable policy before changing retrieval, append new conversation turns instead of rewriting old ones, test with a clearly long prefix, and measure the actual interval between requests. Changing irrelevant HTTP formatting is not the core issue; the model-facing content and its order are.
6. Tool definitions and growing conversations need deliberate boundaries
For tools, Anthropic's documented pattern is to place cache_control on the last tool in the tools array; that breakpoint covers the preceding tool definitions as one prefix. This works well for coding agents whose tool schemas are large and mostly stable. It fails when the application rebuilds the tool array in nondeterministic order or changes descriptions on every request. For a growing conversation, preserve prior turns exactly and append the newest user message. Automatic caching can move the boundary as history grows; explicit breakpoints give more control when stable tools, stable instructions, and conversation history change at different rates. Anthropic permits up to four explicit breakpoints, but start with one or two and add another only for a real update-frequency boundary. Complexity makes misses harder to explain. Always inspect usage after an agent framework upgrade, because frameworks may compact history or regenerate tool schemas without making that change obvious.
7. TTL, minimum length, and concurrency explain many false alarms
The default Anthropic ephemeral cache lifetime is five minutes, measured from the start of the request that writes or reads the entry. A hit refreshes that lifetime. The API also defines a one-hour TTL for workloads whose follow-up requests arrive later; use the provider's current documentation before choosing it. Minimum cacheable length varies by model, so a small test prompt can return zeros in both cache fields even when the request syntax is valid. Concurrency creates another trap: a new cache entry becomes available only after the first response begins. If ten identical requests start simultaneously, the later nine may not read the entry being created by the first. For verification, wait until request one completes before sending request two. For production fan-out, pre-warm the shared prefix or gate the first burst. These checks distinguish a configuration failure from a timing or threshold issue.
8. Verify the gateway, then monitor the effect continuously
A relay must preserve cache_control on the request for upstream Anthropic prompt caching to occur. It must also return the cache usage fields if clients are to verify hits and account for them correctly. cocodot's Anthropic-format endpoint at https://cocodot.co/api/ai/v1/messages accepts Anthropic-format requests and returns cache_creation_input_tokens and cache_read_input_tokens when those fields are present in the upstream response. That is a transport capability, not a promise that every prefix will hit. The sequential test above verifies the effect. In production, record all three input fields per request and alert when a workload that normally repeats a stable prefix suddenly produces zero reads across consecutive eligible calls. Before calling it an outage, check prefix changes, elapsed time, model, minimum length, tool ordering, and concurrency. Anthropic documents the current semantics at https://platform.claude.com/docs/en/build-with-claude/prompt-caching. Usage proves whether caching happened; current provider billing documentation, not a hard-coded multiplier in application prose, should determine financial treatment.
Where prompt caching is useful—and what must remain stable
| Workload | Good cached prefix | Changing content | Main risk |
|---|---|---|---|
| Coding agent | Tool definitions and system rules | User task and tool results | Tool order or schema changes |
| Document Q&A | One repeatedly queried document | Each question | Replacing the document every call |
| Long conversation | Earlier conversation turns | Newest turn | Editing or compacting old turns |
| RAG application | Stable instructions and shared corpus | Per-request retrieval results | Putting volatile retrieval before the breakpoint |
| Batch jobs | Shared policy and examples | Each record | Prefix below the model's minimum length |
| Infrequent task | Any reusable prefix | Scheduled input | Next request arrives after the TTL |