Prompt Cache Optimization (OpenAI)

Operating procedure for getting maximum prompt-cache hit rate on OpenAI API workloads, distilled from OpenAI's "Prompt Caching 201" cookbook. Applied correctly, caching cuts time-to-first-token (TTFT) latency by up to ~80% and input token cost by up to 90%, with discounts up to 98.75% on some models. Caching is automatic and free — your job is to stop breaking it: stabilize prefixes, route requests stickily, and measure cached_tokens.

Scope note: the source explicitly applies to models before GPT-5.6. For GPT-5.6 and later, consult the current platform Prompt Caching guide before relying on these mechanics.

When to use

Use this skill when:

• Designing or reviewing prompt/message layout for any repeated OpenAI API call — system prompts, tool definitions, schemas, conversation loops, agents.
• usage.prompt_tokens_details.cached_tokens is low or zero on traffic that repeats a prefix.
• TTFT latency or input token spend is a problem at scale (long contexts benefit most: 67% faster TTFT at 150k+ tokens vs 7% at 1024 tokens, measured over 2,300 iterations).
• Choosing between Chat Completions vs Responses API, or Batch vs Flex Processing.
• Setting truncation/compaction policy for long conversations or Realtime (voice) sessions.

Do NOT use when:

• Prompts are under 1024 tokens and cannot reasonably be lengthened — caching never activates below 1024 tokens.
• Requests share no repeated prefix (fully unique one-off prompts) — there is nothing to cache.
• Targeting GPT-5.6+ — verify against the current platform guide first.

Rules

1. Know the eligibility mechanics before optimizing. Caching works automatically, with no extra fee, for prompts of 1024+ tokens; hits occur in 128-token increments; a hit requires an exact prefix match AND landing on the same machine. The cacheable prefix includes messages, images, audio, tool definitions, and structured-output schemas. Mechanically, the system reuses cached key/value attention tensors instead of recomputing the forward pass over the prefix.
2. Put durable content first, volatile content last. Order the request: instructions → tool definitions → schemas → stable context → dynamic/user/session content at the end. "Even small changes in early tokens will invalidate exact prefix matching and prevent cache hits." A single early space or timestamp kills the whole cache.
3. Never inject dynamic values early in the prompt. No timestamps-for-debugging, no session IDs, no per-request values anywhere before the volatile tail. If a value must vary per request, move it to the end of the input.
4. Cross the 1024-token threshold deliberately. A 900-token prompt never gets a cache hit. Lengthen it to 1100 tokens: at a 50% cache rate you save 33% on token costs; at 70% you save 55%. Padding a short prompt with stable content can be net cheaper.
5. Use prompt_cache_key for routing stickiness. Requests route to inference engines by a hash of the first ~256 tokens; prompt_cache_key combines with that hash so identical prefixes land on the same engine. A coding customer improved hit rate from 60% to 87% by adding it.
6. Respect the ~15 RPM capacity per prefix + prompt_cache_key combination. Exceeding ~15 requests/minute spreads traffic across additional machines, each a one-time cache miss. Choose key granularity to target ~15 RPM: per-user keys when a user's conversations share a prefix (e.g., same codebase); per-conversation keys when users run many unrelated parallel threads; or hash-and-mod a user id into buckets sized to land near 15 RPM.
7. Restrict tools with allowed_tools, never by editing the tools array. Keep the full static tool list in the cached prefix and pass per-call restrictions as allowed_tools with tool_choice, e.g. {"mode":"auto", "tools":["get_weather","get_location"]}. Modifying or reordering the tools array busts the cache; allowed_tools lives in request metadata and does not.
8. Prefer the Responses API over Chat Completions for reasoning models. Internal OpenAI benchmarks show 40–80% better cache utilization, because the Responses API persists chain-of-thought between turns via previous_response_id or encrypted reasoning items. Without reasoning models there is no caching improvement — don't migrate for cache reasons alone in that case.
9. Prefer Flex Processing over Batch API for latency-insensitive bulk work. Flex keeps the same 50% token discount as Batch but adds control over request rate (RPM), extended prompt caching, and prompt_cache_key. In a 10,000-identical-request head-to-head, Flex achieved an 8.5% higher cache hit rate, translating to a 23% input-token cost reduction. Note: caching on the Batch API lacks model parity — pre-GPT-5 models are not supported, so for o3/o4-mini workloads switch to Flex to get caching.
10. Treat context engineering and caching as a measured trade-off. Dropping, summarizing, or compacting earlier turns breaks the cache — compaction rewrites the prefix. If compacting (e.g., context_management=[{"type": "compaction", "compact_threshold": 100000}] on the Responses API), use evaluations to pick the method and frequency, balancing the cost of lost caching against intelligence gains. Minimize the tool surface for decision quality; prune per-request with allowed_tools, not by editing the array.
11. Use extended prompt caching for prefixes reused over hours. Set prompt_cache_retention: "24h" on the request. For gpt-5.5, gpt-5.5-pro, and all future models, 24h is the default and in-memory mode is unsupported. Only key/value tensors are persisted in GPU-local storage; original input content stays in memory only — relevant for data-retention reviews.
12. On the Realtime API, replace default truncation with retention_ratio. Realtime caching works like the Responses API for audio, text, and images (32k context, 4,096 max output → 28,224 context tokens before truncation). Default "auto" truncation removes the minimum old messages each turn, shifting the conversation start and causing repeated cache misses. Instead send session.update with "truncation": {"type": "retention_ratio", "retention_ratio": 0.7} — keeps ~70%, drops the oldest ~30% in larger, less frequent chunks. Trade-off: the model may forget earlier dialogue sooner, but prefix stability holds across turns.
13. Measure before and after every change. Read cached_tokens in usage.prompt_tokens_details on every response (e.g., prompt_tokens: 2006, cached_tokens: 1920); aggregate via request logs or the Usage dashboard. Iterate deliberately — never assume a layout change helped without the numbers.
14. Append, never rewrite (the Codex lesson). The Codex team treats prompt structure as a first-class performance surface: system instructions, tool definitions, sandbox configuration, and environment context stay identical and consistently ordered; new messages append rather than modify earlier content. Copy this discipline in any long-running agent loop.

Checklist

• Prompt is ≥1024 tokens (lengthen with stable content if just under — see Rule 4).
• Layout order: instructions → tools → schemas → stable context → volatile/user content last.
• Zero dynamic values (timestamps, IDs, whitespace) before the volatile tail.
• Tools array is static and consistently ordered; per-call restriction uses allowed_tools + tool_choice.
• prompt_cache_key set, with granularity (per-user / per-conversation / hashed buckets) targeting ~15 RPM per prefix+key.
• Reasoning models on the Responses API with previous_response_id or encrypted reasoning items, not Chat Completions.
• Latency-insensitive bulk jobs on Flex Processing (not Batch), with extended caching + prompt_cache_key.
• Prefixes reused across hours: prompt_cache_retention: "24h" set (default on gpt-5.5+).
• Compaction/truncation policy chosen via evals, accounting for cache breakage cost.
• Realtime sessions use retention_ratio truncation (e.g., 0.7), not "auto".
• Conversation turns append-only; earlier content never edited mid-conversation.
• cached_tokens from usage.prompt_tokens_details monitored per request and on the Usage dashboard; hit rate tracked before/after changes.

Anti-patterns

The source's cache-miss causes — eliminate each:

• Timestamps or other dynamic content early in the request (even a single added space).
• Modifying or reordering the tools array instead of restricting via allowed_tools.
• Changing instructions or system prompts between requests.
• Tool or response-format schema changes (including schema key modifications).
• Changing reasoning effort settings between requests.
• Naive truncation when hitting the context window limit — shifts the prefix every turn.
• Using Chat Completions with reasoning models (drops chain-of-thought reuse between turns).
• Exceeding ~15 RPM on one prefix+prompt_cache_key combination without bucketing.
• Ignoring cache expiration from time passage — use prompt_cache_retention: "24h" for long-lived prefixes.
• Shipping layout changes without watching cached_tokens — you cannot manage a hit rate you don't measure.

Source

• Prompt Caching 201 — https://developers.openai.com/cookbook/examples/prompt_caching_201 — 2026-02-18

Distilled from the official document(s) above on 2026-08-12. If this skill and the source disagree, trust the source.