Useful Cheatsheets
Build / AI /

Prompt Caching

Slash AI latency and token costs by up to 90 percent using Anthropic and OpenAI prompt caching breakpoints.

TL;DR

  1. Place static instructions at request start to enable prompt-caching.
  2. Mark cache breakpoints in Anthropic using the cache_control property.
  3. Monitor cache_read_input_tokens metrics to verify ninety percent cost savings.
Jump to section

Anthropic Cache Breakpoints

    System Cache Control

    Attach ephemeral cache breakpoint to static system instructions.

    const system = [
  {
    type: 'text',
    text: largeStaticDocumentation,
    cache_control: { type: 'ephemeral' },
  },
];
    Tool Cache Breakpoints

    Cache extensive tool definition schemas to avoid re-billing.

    const cachedTools = tools.map((t, i) => ({
  ...t,
  ...(i === tools.length - 1
    ? { cache_control: { type: 'ephemeral' } }
    : {}),
}));
    Multiple Breakpoint Limits

    Configure up to four distinct cache checkpoints per request.

    // Breakpoint 1: System prompt
// Breakpoint 2: Static reference book
// Breakpoint 3: Conversation history
// Dynamic: Final user question

OpenAI Automatic Caching

    Prefix Alignment

    Order prompt components strictly from static to dynamic.

    // 1. Static system prompt (> 1024 tokens)
// 2. Static documents
// 3. Dynamic runtime user query
// OpenAI caches shared prefixes automatically
    Inspect Cached Tokens

    Verify cache hits through prompt_tokens_details usage field.

    const u = res.usage as any;
const cached =
  u.prompt_tokens_details?.cached_tokens ?? 0;
console.log(`Cached: ${cached}`);
    Exact Prefix Match

    Ensure identical character prefix to achieve cache hits.

    const match = prev.startsWith(prefix);
// Any change invalidates subsequent cache

Cache Hit Verification

    Anthropic Cache Usage

    Read cache creation and cache read metrics from response.

    const u = res.usage as any;
const w = u.cache_creation_input_tokens;
const r = u.cache_read_input_tokens;
console.log(`Write: ${w}, Read: ${r}`);
    Calculate Cost Savings

    Compute monetary savings realized through cached tokens.

    function calcSavings(
  readTokens: number,
  rate: number
) {
  const full = (readTokens / 1e6) * rate;
  return full * 0.90; // 90% cache discount
}
    Cache TTL Monitoring

    Track 5-minute ephemeral cache window expiration.

    const isWarm = Date.now() - lastRequestTime < 300000;
// Keep cache warm with recurring queries

Architectural Best Practices

    Document Pinning

    Pin massive background context above conversation turns.

    const messages = [
  { role: 'user', content: cachedDocBlock },
  { role: 'user', content: dynamicUserQuestion },
];
    Remove Dynamic Variables

    Strip dynamic timestamps and request IDs from cached blocks.

    // Bad: `Current time: ${new Date().toISOString()}`
// Good: Place timestamp in dynamic user prompt only
    Multi-Turn Dialogue Caching

    Cache conversation history up to the second-to-last turn.

    const turns = history.map((h, i) => ({
  ...h,
  ...(i === history.length - 2
    ? { cache_control: { type: 'ephemeral' } }
    : {}),
}));

Tips

  1. Structure prompts with static context first and dynamic user questions last to maximize cache-hit-rate metrics.
  2. Maintain cache warmth by sending periodic keep-alive ping requests within the five-minute ephemeral-cache window to preserve fast response times.

Warnings

  1. Avoid dynamic timestamps or random IDs in system prompts because changing even one token invalidates all subsequent cached-prefixes.
  2. Note that Anthropic enforces a minimum prompt size of 1024 tokens for claude-3-5-sonnet before caching activates.

In Practice

FAQ