Evaluating Agents: Golden Sets, Trajectories and Judges

Evaluation is how you find out whether an agent is right, as opposed to merely fast and cheap. It exists because agents have no compiler and no type system: a prompt edit that looks harmless can change behaviour on a class of inputs nobody thought about, and nothing in your build will notice. Evaluation is the missing test suite — but a statistical one.

Start with a golden task set

A golden set is a versioned file of real tasks with known good outcomes, kept in the repository next to the prompts. Aim for a few dozen to a few hundred cases, not thousands, and make them earn their place: the common happy paths, the awkward inputs, every past production incident, and the cases where the correct behaviour is to refuse or escalate. Each case records the input, the expected outcome, and — for an agent — the tool calls that must or must not happen. Feed it continuously from sampled production sessions, so the suite tracks reality rather than what you imagined at launch.

Final-answer evaluation versus trajectory evaluation

Final-answer evaluation scores only the output: is it correct, grounded in the retrieved sources, complete, appropriately refusing? It measures what the user experiences and it is the score you report to the business. Trajectory evaluation scores the path: did the agent call the right tools, with the right arguments, in a sensible order, without redundant loops or dangerous side effects? It catches the right answer reached for the wrong reason — the agent that got the refund amount correct by guessing rather than by calling get_order, which will be wrong on the next input. Run both. Trajectory checks are also the cheapest checks you have, because most are deterministic assertions: was issue_refund called exactly once, with this order id, after an eligibility check? Do the deterministic assertions first, and only send what survives to a judge.

LLM-as-a-judge and its limits

For anything free-form you need a model to grade the output against a rubric. Amazon Bedrock Evaluations provides this as a managed capability — automatic evaluation with an evaluator model, human evaluation workflows, RAG evaluation with metrics such as correctness, completeness, faithfulness and context relevance, and the ability to bring your own inference responses so you can score outputs an agent has already produced rather than re-running it inside the service.

Judges are useful and biased. Known failure modes: verbosity bias, where longer answers score higher; position bias in pairwise comparisons, where the first option wins more often; self-preference, where a judge favours text from its own model family; and drift, where the score moves because the judge model version changed and nothing about your agent did. Mitigate by pinning the judge model id and version, writing rubrics with concrete criteria rather than "rate helpfulness 1-5", swapping positions in pairwise tests, and calibrating against a set of human-labelled examples so you know how far the judge is from a person. Treat judge scores as relative — good for comparing version 7 against version 6 — not as absolute truth.

Regression testing in CI

Prompts, model ids, tool schemas and knowledge base configurations are all versioned artefacts, and any of them changing should trigger the suite. Run it in CodeBuild on pull requests, gate the merge on two conditions — the aggregate score must not drop beyond a tolerance, and a designated critical subset must stay at 100% — and publish the per-case diff so a reviewer can see which behaviour changed. Because model responses vary, run each case a few times and compare distributions rather than demanding exact equality.

What evaluation is not

It is not monitoring. Evaluation runs against a fixed set before release; monitoring watches live traffic afterwards. Nor is a passing suite proof of safety — golden sets contain the problems you already know about. Keep guardrails and human escalation regardless of the score.