Evaluating & Testing AI Features

You can't unit-test a paragraph with assertEqual — build a small eval set, score outputs, and treat prompt changes like code changes.

AI features fail differently from ordinary code: the same input can produce different words, and "correct" is often a judgement call. That does not mean you cannot test them — it means you test them like a machine-learning system, with an eval set.

Build a small, real eval set

Collect 20–50 representative inputs with the outcome you expect. Real user messages beat invented ones. This is your regression suite: every prompt or model change gets run against it before it ships.

Score what actually matters

Structured features (classification, extraction) score exactly — did it pick the right label, extract the right fields? Assert on those.
Open-ended features (summaries, chat) need a rubric. Check for must-haves (mentions the refund policy), must-not-haves (no invented prices), and format. You can even use Claude as a judge to score outputs against your rubric.

Make prompts diff-able

Keep prompts in version control, not scattered string literals. A prompt tweak is a code change: it goes through review and re-runs the eval set. Pin the exact model ID in tests so an upgrade is a deliberate, measured event — not a silent behaviour shift.

Example

// A tiny eval harness for a classification feature (Vitest-style).
import { describe, it, expect } from "vitest";
import { classifyTicket } from "../src/classify.js";

// Real inputs + expected labels — your regression suite.
const EVAL_SET = [
  { input: "I was charged twice this month",        expected: "billing" },
  { input: "The export button does nothing on Safari", expected: "bug" },
  { input: "Can you add dark mode?",                 expected: "feature" },
  // ...20-50 cases total, harvested from real traffic
];

describe("ticket classifier", () => {
  it("meets the accuracy bar on the eval set", async () => {
    let correct = 0;
    for (const { input, expected } of EVAL_SET) {
      const label = await classifyTicket(input);   // calls Claude, pinned model ID
      if (label === expected) correct++;
    }
    const accuracy = correct / EVAL_SET.length;
    // Fail CI if a prompt or model change regresses accuracy below the bar.
    expect(accuracy).toBeGreaterThanOrEqual(0.9);
  });
});

When to use it

Build a 20-item golden dataset of (input, expected_output) pairs and run it after every prompt change to catch regressions.
Use Claude itself as a judge to score the quality of another Claude response on a 1-5 scale for automated eval.
Track pass-rate over time in a CI step so a prompt degradation blocks the pull request from merging.

More examples

Run a golden dataset eval

Runs the active prompt against a fixed labelled dataset and reports accuracy as a percentage.

const goldenSet = [
  { input: "great product!",        expected: "positive" },
  { input: "completely broken",     expected: "negative" },
  { input: "it is okay I guess",    expected: "neutral"  },
];

let correct = 0;
for (const { input, expected } of goldenSet) {
  const msg = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 8,
    messages: [{ role: "user", content: `Classify: ${input}` }],
  });
  if (msg.content[0].text.trim().toLowerCase() === expected) correct++;
}
console.log(`Accuracy: ${(correct/goldenSet.length*100).toFixed(0)}%`);

LLM-as-judge for open-ended output

Uses a second Claude call as an automated judge to score open-ended answers where exact-match comparison fails.

async function judgeQuality(question, answer) {
  const msg = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 16,
    system: "Score the answer 1-5 for accuracy and helpfulness. Respond with a single digit.",
    messages: [{ role: "user", content: `Question: ${question}\nAnswer: ${answer}` }],
  });
  return parseInt(msg.content[0].text.trim(), 10);
}

const score = await judgeQuality("What is HTTPS?", candidateAnswer);

CI eval script that fails on regression

Exits with code 1 when pass rate drops below the threshold so the CI pipeline blocks the merge automatically.

// eval.mjs — run in CI
const THRESHOLD = 0.80; // 80% pass rate required

let passed = 0;
for (const tc of testCases) {
  const result = await runPrompt(tc.input);
  if (matches(result, tc.expected)) passed++;
}
const rate = passed / testCases.length;
console.log(`Pass rate: ${(rate*100).toFixed(0)}%`);
if (rate < THRESHOLD) {
  console.error("Eval failed — prompt regression detected");
  process.exit(1); // non-zero exit fails the CI step
}

Discussion

Be the first to comment on this lesson.