Useful Cheatsheets
Build / Code / TypeScript

TypeScript AI Multimodal Resilience

An advanced guide to processing multimodal images and files, generating vector embeddings, and establishing fault-tolerant model fallback architectures in TypeScript.

TL;DR

  1. Convert local image Buffer objects into Base64 strings for vision inputs.
  2. Generate semantic vector embeddings using standard embeddings.create endpoints.
  3. Implement exponential backoff retry loops to handle HTTP 429 rate limits.
Jump to section

Multimodal Payloads and Vision

    Base64 image encoder

    Convert local file buffer to base64 data URI format.

    import fs from 'fs';
function toBase64(path: string) {
  const b = fs.readFileSync(path).toString('base64');
  return `data:image/jpeg;base64,${b}`;
}
    OpenAI vision payload

    Send image data URI inside OpenAI chat message content.

    const res = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{
    role: 'user',
    content: [
      { type: 'image_url', image_url: { url } },
    ],
  }],
});
    Claude vision block

    Construct Anthropic image block using raw base64 data.

    const msg = await anthropic.messages.create({
  model: 'claude-3-5-sonnet-20241022',
  max_tokens: 500,
  messages: [{
    role: 'user',
    content: [{
      type: 'image',
      source: { type: 'base64', data: raw },
    }],
  }],
});

Vector Embeddings Generation

    openai.embeddings.create

    Generate dense vector embeddings for semantic search.

    const emb = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: 'TypeScript search index',
});
const vector: number[] = emb.data[0].embedding;
console.log('Dimensions:', vector.length);
    Cosine similarity math

    Compute similarity score between two vector arrays.

    function cosine(a: number[], b: number[]): number {
  let dot = 0, ma = 0, mb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    ma += a[i] * a[i];
    mb += b[i] * b[i];
  }
  return dot / (Math.sqrt(ma) * Math.sqrt(mb));
}
    Batch embeddings creation

    Embed multiple text strings in a single batch request.

    const batch = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: ['First chunk', 'Second chunk'],
});
const vectors = batch.data.map(d => d.embedding);

Exponential Backoff and Jitter

    Backoff with jitter helper

    Calculate randomized backoff delay in milliseconds.

    function getBackoff(attempt: number): number {
  const base = 500;
  const cap = 10000;
  const temp = Math.min(cap, base * 2 ** attempt);
  return Math.floor(Math.random() * temp);
}
    Retryable error checker

    Determine if status code justifies network retry attempt.

    function isRetryable(err: any): boolean {
  const code = err?.status || err?.statusCode;
  const is5xx = code >= 500 && code < 600;
  return code === 429 || is5xx;
}
    Robust retry wrapper

    Execute promise with automatic backoff retry cycles.

    async function retry<T>(fn: () => Promise<T>, n = 3) {
  for (let i = 0; i < n; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i === n - 1) throw err;
      const ms = 1000 * 2 ** i;
      await new Promise(r => setTimeout(r, ms));
    }
  }
}

Provider Failover Architecture

    Failover chain pattern

    Attempt secondary model when primary provider fails.

    async function resilientCall(
  prompt: string
): Promise<string> {
  try {
    return await callPrimary(prompt);
  } catch (err) {
    console.warn('Primary failed, falling back:', err);
    return await callSecondary(prompt);
  }
}
    interface FallbackConfig

    Structure prioritized list of model configurations.

    interface FallbackConfig {
  providers: Array<{
    name: string;
    call: (text: string) => Promise<string>;
  }>;
}
    Pipeline circuit breaker

    Bypass degraded providers after repeated failures.

    let failureCount = 0;
function recordFailure() {
  failureCount++;
  if (failureCount > 5) switchActiveProvider();
}

Tips

  1. Always add randomized jitter to your Math.random() backoff calculations to avoid synchronized thundering-herd retry storms against provider APIs.
  2. Chain diverse model providers using an AIProvider fallback pipeline so requests automatically fail over when a primary provider experiences an outage.

Warnings

  1. Never send high-resolution image files directly to image_url payloads without downscaling them first to prevent excessive token consumption.
  2. Avoid retrying client errors like 400 Bad Request or 401 Unauthorized because invalid credentials or malformed payloads will never succeed on retries.

In Practice

FAQ