What Is AI Agent Evaluation?
AI agent evaluation is the practice of measuring whether an AI agent does its job, using a repeatable set of test cases and scoring rules rather than impressions. It scores the path the agent took as well as the answer it produced, because an agent chooses its own steps, and it runs each case more than once, because the same agent given the same task does not behave identically twice.
Why agents are harder to measure than models#
A model returns an answer and you grade the answer. An agent decides what to do next, calls tools, reacts to what comes back and can reach the same result by several routes or the wrong result by a plausible one. Two things follow. The trajectory has to be graded alongside the outcome, and a single run tells you very little.
Run-to-run variance is the part teams underestimate. Researchers at KTH collected 60,000 agent trajectories on a standard coding benchmark and found that single-run scores "vary by 2.2 to 6.0 percentage points depending on which run is selected", with standard deviations above 1.5 points even at temperature zero. Their conclusion is the practical one: a reported improvement of two or three points may be noise. If an eval runs each case once, it is not yet a measurement.
A score is a property of the instrument#
Benchmarks are easy to quote and easy to misread. When OpenAI had 1,699 samples of the SWE-bench coding benchmark annotated by developers, it found 38.3% had underspecified problem statements and 61.1% had tests that could fail a valid solution. On the cleaned 500-sample subset the same model scored 33.2% against 16% on the original. Same model, same work, twice the score, because the ruler changed.
Cleaning it did not settle the matter either. A 2025 study found SWE-bench still counts 7.8% of patches as correct while they fail the developer's own tests, inflating reported resolution rates by 6.2 percentage points. Treat any single benchmark number as evidence about the benchmark first and the agent second.
What to measure, and who does the grading#
Generic metrics are the common failure. Hamel Husain, whose writing is the reference point for most teams doing this seriously, puts it bluntly: "Generic metrics are worse than useless", because they create a false sense of progress and split attention across numbers nobody acts on. What works is a small set of cases drawn from your own failures, scored on criteria you wrote.
Most teams grade with another model, and that model is an instrument with a known error rate. The original study of the technique put agreement between a strong judge and humans at 80 to 85%, against humans agreeing with each other 81% of the time, and documented position, verbosity and self-preference biases. Berkeley's follow-up found something more awkward: criteria cannot reliably be written in advance, because grading outputs is how people work out what their criteria are. A judge needs validating against human labels before it is trusted, and re-validating when the criteria move.
Frequently asked questions#
How do you measure the performance of an AI agent?
Define the tasks it is supposed to complete, write cases with a known good outcome, and score each run on both the result and the route taken. Run every case several times and report a distribution rather than a single pass or fail. Add the operational numbers that matter to whoever pays for it: cost and tokens per completed task, how often a human had to step in, and how often the agent failed loudly rather than quietly.
How do you write evals for AI agents?
Start from failures you have actually seen rather than a framework's built-in metrics. Collect real traces, read them, group what went wrong, and turn each recurring failure into a case with a pass condition you can defend to a colleague. Keep the set small enough to run often, because an eval suite that takes a day to run stops being part of the loop. Expect the criteria to change as you read more output; that is normal and is why the set needs revisiting rather than freezing.
What is the difference between AI agent evaluation and LLM evaluation?
LLM evaluation grades a single response to a single prompt. Agent evaluation grades a sequence of decisions: which tools were called, in what order, with what arguments, and whether the agent recovered when something returned an error. An agent can produce an acceptable final answer through a route you would never approve, which a response-level score cannot see.
Is there a standard for evaluating AI agents?
Not yet, and be careful what gets called one. NIST's ARIA programme is the most substantial institutional work, and its published pilot covered seven applications from five organisations at version 0.1, making it a methodology demonstration rather than a standard. An Internet-Draft on agent security benchmarking is sometimes cited as an IETF standard; it is an individual submission from four engineers at one company, carrying the IETF's own notice that such drafts are "not endorsed by the IETF" and have "no formal standing". Benchmarks, frameworks and vendor scorecards are what exist today.