Useful Cheatsheets
Build / AI /

Vision and Multimodal AI

Process images, extract document data, and analyze visual diagrams using OpenAI and Claude multimodal Vision APIs.

TL;DR

  1. Encode local raster images into base64 data:image/jpeg payload strings.
  2. Pass multi-image comparisons into messages arrays using content blocks.
  3. Extract structured JSON records directly from scanned documents with zod.
Jump to section

Base64 & URL Image Payloads

    Remote Image URL Input

    Send public HTTPS image URL to OpenAI Vision API.

    import OpenAI from 'openai';
const client = new OpenAI();

const res = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'Describe this chart.' },
      {
        type: 'image_url',
        image_url: {
          url: 'https://example.com/c.png',
        },
      },
    ],
  }],
});
    Local File Base64 Encoder

    Read local image buffer and format as base64 data URI.

    import fs from 'node:fs';

function encodeImage(filePath: string) {
  const buf = fs.readFileSync(filePath);
  const b64 = buf.toString('base64');
  return `data:image/jpeg;base64,${b64}`;
}
    Low vs High Detail Mode

    Toggle image resolution mode to optimize cost and latency.

    // low: 85 tokens flat cost, fast triage
// high: detailed tiles, reads dense text
const imageBlock = {
  type: 'image_url' as const,
  image_url: { url: dataUri, detail: 'low' as const },
};

Claude Multimodal Messages

    Claude Image Content Block

    Pass raw base64 data to Anthropic Messages API.

    import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();

const r = await anthropic.messages.create({
  model: 'claude-3-5-sonnet-20241022',
  max_tokens: 1000,
  messages: [{
    role: 'user',
    content: [
      {
        type: 'image',
        source: {
          type: 'base64',
          media_type: 'image/jpeg',
          data: base64Data,
        },
      },
      { type: 'text', text: 'Extract invoice total.' },
    ],
  }],
});
    Multi-Image Comparison

    Compare two consecutive UI screenshots in a single prompt.

    const blocks = [
  { type: 'image', source: img1 },
  { type: 'image', source: img2 },
  { type: 'text', text: 'List visual diffs.' },
];
    Supported Image Media Types

    Allowed mime types for Anthropic Vision requests.

    type AllowedMime =
  | 'image/jpeg'
  | 'image/png'
  | 'image/gif'
  | 'image/webp';

Visual Document & OCR Extraction

    Structured Receipt Extraction

    Extract typed JSON schema fields from receipt photos.

    const receiptPrompt = [
  { type: 'text', text: 'Extract receipt into JSON.' },
  { type: 'image_url', image_url: { url: imgUri } },
];
const out = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{
    role: 'user',
    content: receiptPrompt as any,
  }],
  response_format: { type: 'json_object' },
});
    Chart & Diagram Data Table Parser

    Convert visual line chart into structured CSV rows.

    const chartPrompt = 'Extract all series data points ' +
  'from this graph as CSV format with columns: x,y';
const payload = [
  { type: 'text', text: chartPrompt },
  { type: 'image_url', image_url: { url: chartUri } },
];
    Image Sanitization & Dimension Check

    Ensure image dimensions fit within provider bounds.

    function validateImageBounds(w: number, h: number) {
  const maxDim = 8000;
  if (w > maxDim || h > maxDim) {
    throw new Error('Image exceeds 8000px maximum');
  }
}

Multimodal Performance & Optimization

    Dynamic Image Downscaling

    Resize large images using Sharp prior to base64 encoding.

    import sharp from 'sharp';

async function prepImage(buffer: Buffer) {
  return await sharp(buffer)
    .resize(1600, 1600, { fit: 'inside' })
    .jpeg({ quality: 80 })
    .toBuffer();
}
    Vision Token Calculation Formula

    Estimate token footprint for high-resolution images.

    function estimateVisionTokens(w: number, h: number) {
  const tilesX = Math.ceil(w / 512);
  const tilesY = Math.ceil(h / 512);
  return tilesX * tilesY * 170 + 85;
}
    Caching Multimodal Prompts

    Mark static reference images with prompt cache breakpoints.

    const cachedImg = {
  type: 'image',
  source: imgSource,
  cache_control: { type: 'ephemeral' },
};

Tips

  1. Downscale high-resolution images to under 2000 pixels with sharp before base64 encoding to reduce token billing by up to sixty percent.
  2. Combine explicit OCR text instructions with structured response_format schemas to guarantee flawless table extraction from image invoices.

Warnings

  1. Never send raw uncompressed PNG image buffers directly over the wire without validating upload size under provider 20MB limits.
  2. Avoid using multimodal models for fine-grained coordinate bounding without running specialized post-processing object detection algorithms.

In Practice

FAQ