Useful Cheatsheets
Build / AI /

Context Window Management

Manage token budgets, rolling conversation history, semantic compaction, and prompt summarization in production AI systems.

TL;DR

  1. Enforce maxTokens boundaries before dispatching conversational history arrays.
  2. Prune oldest dialogue turns using sliding window slice() mechanics.
  3. Summarize evicted context into dense system message briefs dynamically.
Jump to section

Context Budgeting & Token Limits

    Model Context Allocation Table

    Allocate safe budget margins for system prompts, history, and outputs.

    const BUDGETS = {
  'gpt-4o': { max: 128000, reserved: 4096 },
  'claude-3-5': { max: 200000, reserved: 8192 },
  'gemini-1.5': { max: 1000000, reserved: 8192 },
} as const;

function getAvailableTokens(m: keyof typeof BUDGETS) {
  const config = BUDGETS[m];
  return config.max - config.reserved;
}
    Context Ceiling Threshold Check

    Detect when conversation history nears hard context boundaries.

    function isContextNearLimit(
  currentTokens: number,
  limit = 128000,
  marginRatio = 0.85
): boolean {
  return currentTokens >= limit * marginRatio;
}
    Needle-In-Haystack Warning

    Position critical facts at edges where model recall peaks.

    // Attention degrades in middle 40-70% of context.
const optimizedMessages = [
  systemPrompt,     // Primacy: High attention weight
  ...retrievedDocs, // Middle: Background evidence
  userFinalQuery,   // Recency: High attention weight
];

Sliding Window Strategies

    Turn-Based Sliding Window

    Retain system prompt and most recent N dialogue messages.

    type Msg = { role: string; content: string };
function sliceRecent(history: Msg[], maxTurns = 10) {
  const sys = history.find(m => m.role === 'system');
  const nonSys = history.filter(
    m => m.role !== 'system'
  );
  const recent = nonSys.slice(-maxTurns);
  return sys ? [sys, ...recent] : recent;
}
    Token-Aware Message Pruning

    Evict oldest user/assistant turns until within token budget.

    function pruneToBudget(
  msgs: Array<{ role: string; content: string }>,
  budget: number,
  countFn: (t: string) => number
) {
  const out = [...msgs];
  let cur = out.reduce(
    (s, m) => s + countFn(m.content), 0
  );
  while (out.length > 2 && cur > budget) {
    const [removed] = out.splice(1, 1); // Keep system
    cur -= countFn(removed.content);
  }
  return out;
}
    Preserve Multi-Turn Tool Call Pairs

    Avoid separating tool_call blocks from corresponding tool_results.

    function safeEvictTurn(messages: any[]) {
  // Evict entire tool call + tool response pairs
  const idx = messages.findIndex(
    m => m.role === 'tool'
  );
  if (idx > 0 && messages[idx - 1].tool_calls) {
    messages.splice(idx - 1, 2);
  }
}

Compaction & Summarization

    Incremental Conversation Compaction

    Compress evicted dialogue turns into a persistent running summary.

    async function compactHistory(
  summary: string,
  evicted: string[],
  client: any
): Promise<string> {
  const p = `Update summary:\nOld: ${summary}` +
    `\nTurns: ${evicted.join('\n')}`;
  const r = await client.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: p }],
  });
  return r.choices[0]?.message?.content ?? summary;
}
    Structured Scratchpad Injector

    Inject synthesized state into system prompt instead of full logs.

    function buildCompactedPrompt(
  stateSummary: string,
  userGoal: string
) {
  const txt = `State: ${stateSummary}`;
  return [
    { role: 'system', content: txt },
    { role: 'user', content: userGoal },
  ];
}
    Observation Payload Pruner

    Truncate oversized raw API responses before feeding into context.

    function truncatePayload(raw: string, maxLen = 1500) {
  if (raw.length <= maxLen) return raw;
  const head = raw.slice(0, maxLen);
  const diff = raw.length - maxLen;
  return `${head}\n[... ${diff} chars omitted]`;
}

Attention Architecture & Positioning

    Front-Loading Primacy Placement

    Anchor fundamental operating rules at the beginning of prompt context.

    const promptStructure = [
  { role: 'system', content: 'Never hallucinate' },
  ...middleDocuments,
  { role: 'user', content: 'Query: revenue?' },
];
    Recency Bias Reinforcement

    Re-state critical constraints immediately prior to user instruction.

    const reinforcedQuery = `${userQuestion}\n\n` +
  'REMINDER: Base answer on provided excerpts.';
const finalTurn = {
  role: 'user', content: reinforcedQuery,
};
    Context Window Degradation Monitor

    Log warning when token payload enters mid-context attention dip.

    function checkMiddleAttention(
  tokenCount: number, max: number
) {
  const ratio = tokenCount / max;
  if (ratio > 0.4 && ratio < 0.75) {
    logger.warn('Query in mid-attention valley');
  }
}

Tips

  1. Calculate token usage with tiktoken to prune dialogue before incurring unexpected provider context overflow exceptions.
  2. Place essential instructions and system rules at both ends of messages to overcome model middle-context degradation.

Warnings

  1. Never pass unbounded chat histories directly into create() requests without implementing hard token budget thresholds.
  2. Avoid blind substring slicing that cuts multi-byte characters or breaks structural JSON markers mid-stream during message trimming.

In Practice

FAQ