How do you measure the performance of an AI agent?

Define the tasks it is supposed to complete, write cases with a known good outcome, and score each run on both the result and the route taken. Run every case several times and report a distribution rather than a single pass or fail. Add the operational numbers that matter to whoever pays for it: cost and tokens per completed task, how often a human had to step in, and how often the agent failed loudly rather than quietly.

How do you write evals for AI agents?

Start from failures you have actually seen rather than a framework's built-in metrics. Collect real traces, read them, group what went wrong, and turn each recurring failure into a case with a pass condition you can defend to a colleague. Keep the set small enough to run often, because an eval suite that takes a day to run stops being part of the loop. Expect the criteria to change as you read more output; that is normal and is why the set needs revisiting rather than freezing.

What is the difference between AI agent evaluation and LLM evaluation?

LLM evaluation grades a single response to a single prompt. Agent evaluation grades a sequence of decisions: which tools were called, in what order, with what arguments, and whether the agent recovered when something returned an error. An agent can produce an acceptable final answer through a route you would never approve, which a response-level score cannot see.

Is there a standard for evaluating AI agents?

Not yet, and be careful what gets called one. NIST's ARIA programme is the most substantial institutional work, and its published pilot covered seven applications from five organisations at version 0.1, making it a methodology demonstration rather than a standard. An Internet-Draft on agent security benchmarking is sometimes cited as an IETF standard; it is an individual submission from four engineers at one company, carrying the IETF's own notice that such drafts are "not endorsed by the IETF" and have "no formal standing". Benchmarks, frameworks and vendor scorecards are what exist today.

AI agent evaluation

What Is AI Agent Evaluation?

In short

AI agent evaluation is the practice of measuring whether an AI agent does its job, using a repeatable set of test cases and scoring rules rather than impressions. It scores the path the agent took as well as the answer it produced, because an agent chooses its own steps, and it runs each case more than once, because the same agent given the same task does not behave identically twice.

Updated 16 September 20263 min read

Why are AI agents harder to evaluate than models?#

A model returns an answer and you grade the answer. An agent decides what to do next, calls tools, reacts to what comes back and can reach the same result by several routes or the wrong result by a plausible one. Two things follow. The trajectory has to be graded alongside the outcome, and a single run tells you very little.

Run-to-run variance is the part teams underestimate. Researchers at KTH collected 60,000 agent trajectories on a standard coding benchmark and found that single-run scores "vary by 2.2 to 6.0 percentage points depending on which run is selected", with standard deviations above 1.5 points even at temperature zero. Their conclusion is the practical one: a reported improvement of two or three points may be noise. If an eval runs each case once, it is not yet a measurement.

How reliable are AI agent benchmarks?#

A score is a property of the instrument. Benchmarks are easy to quote and easy to misread. When OpenAI had 1,699 samples of the SWE-bench coding benchmark annotated by developers, it found 38.3% had underspecified problem statements and 61.1% had tests that could fail a valid solution. On the cleaned 500-sample subset the same model scored 33.2% against 16% on the original. Same model, same work, twice the score, because the ruler changed.

Cleaning it did not settle the matter either. A 2025 study found SWE-bench still counts 7.8% of patches as correct while they fail the developer's own tests, inflating reported resolution rates by 6.2 percentage points. Treat any single benchmark number as evidence about the benchmark first and the agent second.

One structural warning applies to any eval that uses a model to grade a model. When the grader shares a base model, a prompt or a context with the thing it is grading, its mistakes are not independent of the mistakes it is meant to catch — see correlated failure. A deterministic test, a different method or a person with different information is a stronger check than a second agent, however capable.

Evals also answer a narrower question than teams often assume. They test whether the agent did the job as specified; whether the job was worth doing is an outcome check, declared before the work starts and measured after release.

Keep up with the Knowledge BaseEvery month, get new terms and updated definitions straight to your inbox.

How do you build evals for an AI agent?#

Generic metrics are the common failure. Hamel Husain, whose writing is the reference point for most teams doing this seriously, puts it bluntly: "Generic metrics are worse than useless", because they create a false sense of progress and split attention across numbers nobody acts on. What works is a small set of cases drawn from your own failures, scored on criteria you wrote.

Most teams grade with another model, and that model is an instrument with a known error rate. The original study of the technique put agreement between a strong judge and humans at 80 to 85%, against humans agreeing with each other 81% of the time, and documented position, verbosity and self-preference biases. Berkeley's follow-up found something more awkward: criteria cannot reliably be written in advance, because grading outputs is how people work out what their criteria are. A judge needs validating against human labels before it is trusted, and re-validating when the criteria move.

Frequently asked questions#

How do you measure the performance of an AI agent?: Define the tasks it is supposed to complete, write cases with a known good outcome, and score each run on both the result and the route taken. Run every case several times and report a distribution rather than a single pass or fail. Add the operational numbers that matter to whoever pays for it: cost and tokens per completed task, how often a human had to step in, and how often the agent failed loudly rather than quietly.
How do you write evals for AI agents?: Start from failures you have actually seen rather than a framework's built-in metrics. Collect real traces, read them, group what went wrong, and turn each recurring failure into a case with a pass condition you can defend to a colleague. Keep the set small enough to run often, because an eval suite that takes a day to run stops being part of the loop. Expect the criteria to change as you read more output; that is normal and is why the set needs revisiting rather than freezing.
What is the difference between AI agent evaluation and LLM evaluation?: LLM evaluation grades a single response to a single prompt. Agent evaluation grades a sequence of decisions: which tools were called, in what order, with what arguments, and whether the agent recovered when something returned an error. An agent can produce an acceptable final answer through a route you would never approve, which a response-level score cannot see.
Is there a standard for evaluating AI agents?: Not yet, and be careful what gets called one. NIST's ARIA programme is the most substantial institutional work, and its published pilot covered seven applications from five organisations at version 0.1, making it a methodology demonstration rather than a standard. An Internet-Draft on agent security benchmarking is sometimes cited as an IETF standard; it is an individual submission from four engineers at one company, carrying the IETF's own notice that such drafts are "not endorsed by the IETF" and have "no formal standing". Benchmarks, frameworks and vendor scorecards are what exist today.

Sources

Go deeper

Measuring AI Agent Quality When You Can't Freeze the Data →

Work with re:cinq on AI Native transformation

See the service →

Found something wrong, missing or badly explained? Tell us. We credit every correction we publish.