Useful Cheatsheets
Build / AI /

Observability and Tracing

Monitor LLM latency, track token expenses, trace multi-agent execution spans, and instrument OpenTelemetry telemetry.

TL;DR

  1. Instrument multi-step agent reasoning loops using OpenTelemetry startSpan() traces.
  2. Track prompt and completion token counts using structured logger.info() metadata.
  3. Aggregate dollar expenses in real time utilizing centralized cost accounting formulas.
Jump to section

OpenTelemetry Span Instrumentation

    Span Wrapper for LLM Calls

    Wrap API requests in OpenTelemetry trace spans.

    import { trace } from '@opentelemetry/api';
import { SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('ai-service');

async function tracedChat(
  msgs: any[], model = 'gpt-4o'
) {
  return tracer.startActiveSpan('chat', async s => {
    s.setAttributes({ 'gen_ai.model': model });
    try {
      const res = await callLLM(msgs, model);
      s.setStatus({ code: SpanStatusCode.OK });
      return res;
    } catch (err: any) {
      s.recordException(err);
      s.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      s.end();
    }
  });
}
    Nested Agent Tool Span Hierarchy

    Create child spans for nested agent tool executions.

    async function traceTool(name: string, fn: () => any) {
  const sName = `tool.${name}`;
  return tracer.startActiveSpan(sName, async s => {
    s.setAttribute('tool.name', name);
    try {
      return await fn();
    } finally {
      s.end();
    }
  });
}
    Context Propagation Across Services

    Propagate traceparent headers across distributed microservices.

    import { propagation } from '@opentelemetry/api';
import { context } from '@opentelemetry/api';
const carrier: Record<string, string> = {};
propagation.inject(context.active(), carrier);

Token Metrics & Cost Accounting

    Real-Time Dollar Cost Calculator

    Calculate financial cost of completion turn from usage data.

    const RATES = {
  'gpt-4o': { inM: 2.50, outM: 10.00 },
  'gpt-4o-mini': { inM: 0.15, outM: 0.60 },
} as const;
type MKey = keyof typeof RATES;

function calcTurnCost(model: MKey, u: any) {
  const r = RATES[model];
  const inCost = (u.prompt_tokens / 1e6) * r.inM;
  const outCost = (u.completion_tokens / 1e6) * r.outM;
  return inCost + outCost;
}
    Token Spend Telemetry Emitter

    Record token metrics to monitoring backend.

    function recordUsage(
  model: string, usage: any, cost: number
) {
  metrics.increment('ai.requests', 1, { model });
  const inTok = usage.prompt_tokens;
  metrics.gauge('ai.in', inTok, { model });
  const outTok = usage.completion_tokens;
  metrics.gauge('ai.out', outTok, { model });
  metrics.gauge('ai.cost.usd', cost, { model });
}
    User Budget Ceiling Guard

    Halt user agent execution when monthly budget cap is reached.

    async function checkBudget(
  userId: string, addedCost: number
) {
  const cur = await getMonthlySpend(userId);
  if (cur + addedCost > 50.00) {
    throw new Error('Monthly AI spend cap exceeded');
  }
}

Langfuse Integration

    Langfuse Client Initialization

    Connect to Langfuse observability platform.

    import { Langfuse } from 'langfuse';
const langfuse = new Langfuse({
  publicKey: process.env.LANGFUSE_PUBLIC_KEY!,
  secretKey: process.env.LANGFUSE_SECRET_KEY!,
  baseUrl: 'https://cloud.langfuse.com',
});
    Traced Generation Execution

    Record generation event with full inputs and metadata.

    const trace = langfuse.trace({
  name: 'rag-query', userId: 'usr_1',
});
const gen = trace.generation({
  name: 'generate-answer',
  model: 'gpt-4o',
  input: query,
});
gen.end({ output: answer, usage: { total: 420 } });
    User Feedback Score Hook

    Attach thumbs-up user feedback scores directly to trace.

    await langfuse.score({
  traceId: trace.id,
  name: 'user-feedback',
  value: 1, // 1 for thumbs up, 0 for thumbs down
});

Latency & Anomaly Alerts

    Time-to-First-Token (TTFT) Stopwatch

    Measure latency until initial streaming token reaches client.

    async function measureTTFT(stream: any) {
  const t0 = performance.now();
  for await (const chunk of stream) {
    const ttft = performance.now() - t0;
    logger.info('streaming_ttft_ms', { ttft });
    break;
  }
}
    Repeated Tool Call Spike Detector

    Detect infinite agent loops before budget exhaustion.

    function detectToolLoop(
  toolCalls: string[], threshold = 5
) {
  const lastN = toolCalls.slice(-threshold);
  const allSame = lastN.every(t => t === lastN[0]);
  if (toolCalls.length >= threshold && allSame) {
    const err = 'Agent loop anomaly: identical calls';
    throw new Error(err);
  }
}
    Observability Flush on Exit

    Ensure pending telemetry spans flush before process exit.

    process.on('SIGTERM', async () => {
  await langfuse.flushAsync();
  await tracerProvider.shutdown();
  process.exit(0);
});

Tips

  1. Attach unique sessionId and userId attributes to every trace span to isolate aberrant customer usage patterns instantly.
  2. Export spans asynchronously via BatchSpanProcessor to prevent observability network overhead from degrading user-facing request latency across applications.

Warnings

  1. Never record raw unmasked customer PII or confidential API keys inside telemetry span attributes payloads.
  2. Avoid high-cardinality unbounded metric tags that overwhelm time-series databases and inflate external observability cloud bills.

In Practice

FAQ