Useful Cheatsheets
Build / AI /

AI Evaluation and Benchmarks

Build automated test suites, implement LLM-as-a-judge rubrics, and track accuracy benchmarks for AI pipelines.

TL;DR

  1. Construct quantitative regression evaluation datasets using typed EvalSample records.
  2. Score qualitative output performance utilizing structured LLM-as-a-judge rubrics.
  3. Automate continuous evaluation passes inside GitHub Actions CI/CD deployment workflows.
Jump to section

LLM-as-a-Judge Rubrics

    Structured Scoring Rubric Prompt

    Prompt frontier model to evaluate answer quality on 1-5 scale.

    function buildJudgePrompt(
  query: string, answer: string, expected: string
) {
  return 'Score answer against expected (1 to 5).\n' +
    `Q: ${query}\nExp: ${expected}\nAns: ${answer}\n` +
    'Return JSON: { score, reasoning }';
}
    Judge Score Parser & Zod Validator

    Validate that judge model outputs typed numeric score.

    import { z } from 'zod';
const JudgeSchema = z.object({
  score: z.number().min(1).max(5),
  reasoning: z.string().min(10),
});
const json = JSON.parse(judgeOutput);
const result = JudgeSchema.parse(json);
    Pairwise Comparison with Bias Swap

    Evaluate two model candidates swapping order to negate bias.

    async function comparePairwise(
  q: string, a1: string, a2: string
) {
  const [s1, s2] = await Promise.all([
    judgePair(q, a1, a2),
    judgePair(q, a2, a1),
  ]);
  const pickA = s1.winner === 'A' && s2.winner === 'B';
  return pickA ? a1 : a2;
}

Deterministic Evaluation Metrics

    Exact Match & Normalized Equivalence

    Fast zero-cost accuracy check for factual outputs.

    function exactMatch(actual: string, expected: string) {
  const norm = (s: string) => {
    return s.trim().toLowerCase().replace(/\s+/g, ' ');
  };
  return norm(actual) === norm(expected);
}
    Token F1 Score Calculation

    Calculate precision, recall, and harmonic F1 across tokens.

    function tokenF1(actual: string, expected: string) {
  const a = actual.toLowerCase().split(/\s+/);
  const e = expected.toLowerCase().split(/\s+/);
  const aTokens = new Set(a);
  const eTokens = new Set(e);
  const list = [...aTokens];
  const match = (t: string) => eTokens.has(t);
  const overlap = list.filter(match).length;
  if (overlap === 0) return 0;
  const p = overlap / aTokens.size;
  const r = overlap / eTokens.size;
  return (2 * p * r) / (p + r);
}
    Regex & Schema Assertions

    Assert output satisfies expected format constraints.

    function assertFormat(val: string, re: RegExp) {
  if (!re.test(val)) {
    throw new Error(`Failed pattern: ${re}`);
  }
}

Golden Dataset & Test Suite Runner

    Eval Sample Interface Contract

    Structure test case records with inputs and expectations.

    type EvalSample = {
  id: string;
  query: string;
  expected: string;
  category: 'rag' | 'math' | 'formatting';
  minScore: number;
};
    Continuous Evaluation Suite Runner

    Iterate across golden test dataset and aggregate metrics.

    async function runTestSuite(
  samples: EvalSample[],
  pipelineFn: (q: string) => Promise<string>
) {
  const results = [];
  for (const s of samples) {
    const answer = await pipelineFn(s.query);
    const pass = exactMatch(answer, s.expected);
    results.push({ id: s.id, pass });
  }
  const passed = results.filter(r => r.pass).length;
  const passRate = passed / samples.length;
  return { passRate, results };
}
    CI/CD Regression Threshold Check

    Fail pull request build if test pass rate regresses.

    const MIN_PASS_RATE = 0.95;
if (report.passRate < MIN_PASS_RATE) {
  const r = report.passRate;
  console.error(`Rate ${r} < ${MIN_PASS_RATE}`);
  process.exit(1);
}

Synthetic Dataset Generation

    Q&A Pair Generation from Documents

    Generate synthetic test queries from knowledge documents.

    async function makeQA(doc: string, client: any) {
  const p = 'Read document and output 3 queries ' +
    'and answers as JSON:\n' + doc;
  const res = await client.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: p }],
    response_format: { type: 'json_object' },
  });
  return JSON.parse(res.choices[0].message.content!);
}
    Adversarial Edge Case Generator

    Synthesize edge cases to test system resilience.

    const edgePrompt = 'Generate 5 tricky edge cases ' +
  'testing ambiguous dates and null fields.';
    Dataset Deduplication & Cleaning

    Remove duplicate synthetic queries using embedding distance.

    function dedupeSamples(samples: EvalSample[]) {
  const seen = new Set<string>();
  return samples.filter(s => {
    const k = s.query.trim().toLowerCase();
    if (seen.has(k)) return false;
    seen.add(k);
    return true;
  });
}

Tips

  1. Swap candidate answer positions during pairwise llmJudge scoring passes to eliminate model order-bias preferences completely across comparative runs.
  2. Measure fast deterministic metrics like exact match, regex assertions, and zod Schema validity before executing expensive LLM judge evaluations.

Warnings

  1. Never ship prompt modifications or model migrations to production without verifying pass rates across your reference golden-dataset.
  2. Avoid using weak lightweight models as judges because grading nuanced reasoning requires frontier-class gpt-4o or Claude evaluation.

In Practice

FAQ