Useful Cheatsheets
Build / Code / TypeScript

TypeScript AI Multimodal Resilience

An advanced guide to processing multimodal images and files, generating vector embeddings, and establishing fault-tolerant model fallback architectures in TypeScript.

TL;DR

  1. Convert local image Buffer objects into Base64 strings for vision inputs.
  2. Generate semantic vector embeddings using standard embeddings.create endpoints.
  3. Implement exponential backoff retry loops to handle HTTP 429 rate limits.
Jump to section

Multimodal Payloads and Vision

    Base64 image encoder

    Turn a local file into a data URI.

    import fs from 'fs';
function toBase64(path: string) {
  const b = fs.readFileSync(path).toString('base64');
  return `data:image/jpeg;base64,${b}`;
}
    OpenAI vision payload

    Send image_url inside the message content.

    const res = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{
    role: 'user',
    content: [{ type: 'image_url', image_url: { url } }],
  }],
});
    Claude vision block

    Claude takes a base64 image block.

    const msg = await anthropic.messages.create({
  model: 'claude-3-5-sonnet-20241022',
  max_tokens: 500,
  messages: [{
    role: 'user',
    content: [{ type: 'image', source: { type: 'base64', data: raw } }],
  }],
});

Vector Embeddings Generation

    openai.embeddings.create

    Get a dense vector for a string.

    const emb = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: 'TypeScript search index',
});
const v: number[] = emb.data[0].embedding;
    Cosine similarity math

    Compare two vectors.

    function cosine(a: number[], b: number[]): number {
  let dot = 0, ma = 0, mb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    ma += a[i] ** 2;
    mb += b[i] ** 2;
  }
  return dot / (Math.sqrt(ma) * Math.sqrt(mb));
}
    Batch embeddings creation

    Pass an array of strings for one request.

    const batch = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: ['First', 'Second'],
});
const vectors = batch.data.map(d => d.embedding);

Exponential Backoff and Jitter

    Backoff with jitter helper

    Randomized backoff delay in ms.

    function backoff(attempt: number): number {
  const cap = Math.min(10000, 500 * 2 ** attempt);
  return Math.floor(Math.random() * cap);
}
    Retryable error checker

    429 or any 5xx is worth retrying.

    function retryable(err: any): boolean {
  const code = err?.status ?? err?.statusCode;
  return code === 429 || (code >= 500 && code < 600);
}
    Robust retry wrapper

    Run a promise with backoff retries.

    async function retry<T>(fn: () => Promise<T>, n = 3) {
  for (let i = 0; i < n; i++) {
    try { return await fn(); }
    catch (err) {
      if (i === n - 1) throw err;
      await new Promise(r => setTimeout(r, 1000 * 2 ** i));
    }
  }
}

Provider Failover Architecture

    Failover chain pattern

    Attempt secondary model when primary provider fails.

    async function resilientCall(
  prompt: string
): Promise<string> {
  try {
    return await callPrimary(prompt);
  } catch (err) {
    console.warn('Primary failed, falling back:', err);
    return await callSecondary(prompt);
  }
}
    interface FallbackConfig

    Structure prioritized list of model configurations.

    interface FallbackConfig {
  providers: Array<{
    name: string;
    call: (text: string) => Promise<string>;
  }>;
}
    Pipeline circuit breaker

    Bypass degraded providers after repeated failures.

    let failureCount = 0;
function recordFailure() {
  failureCount++;
  if (failureCount > 5) switchActiveProvider();
}

Tips

  1. Always add randomized jitter to your Math.random() backoff calculations to avoid synchronized thundering-herd retry storms against provider APIs.
  2. Chain diverse model providers using an AIProvider fallback pipeline so requests automatically fail over when a primary provider experiences an outage.

Warnings

  1. Never send high-resolution image files directly to image_url payloads without downscaling them first to prevent excessive token consumption.
  2. Avoid retrying client errors like 400 Bad Request or 401 Unauthorized because invalid credentials or malformed payloads will never succeed on retries.

In Practice

FAQ