Learn how to evaluate AI agents through outcome checks, tool-use traces, realistic test cases, repeatability, permissions and cost per useful result.
An AI agent says it has created three follow-up tasks from a meeting. The response is clear, the task descriptions sound reasonable and the demonstration looks complete. Then someone opens the project tracker: one task has the wrong owner, another has an invented deadline, and the third was created twice.
AI agent evaluation is the process of testing whether an agent achieves the intended result under defined conditions. It examines the work performed, the actions taken and the limits respected, rather than judging only the final message. A system can write an excellent account of a job it did incorrectly.
This guide uses an illustrative task-tracker assistant to explain how to evaluate AI agents. The same principles apply to research, coding and administrative agents, although each needs its own success criteria. For the foundations of how agents differ from conventional assistants, see our agentic AI guide.
What AI Agent Evaluation Actually Measures
An agent can make several decisions before it responds: retrieve information, select a tool, interpret its result and decide whether to continue. Anthropic’s agent overview distinguishes model-directed agents from workflows with predefined code paths. That flexibility makes it useful to examine both the outcome and the process that produced it.
A model benchmark alone cannot establish whether your complete agent works. The surrounding application supplies tools, permissions, instructions and task context. Changing any of those can change the result even when the model stays the same. Evaluation therefore needs to reflect the system people will actually use.
For our example, the agent’s job is to read approved meeting notes, prepare three tasks and save them in a specified project after approval. It may select existing owners and use dates explicitly agreed in the notes. It must not invent missing information, create new user accounts or email participants. Those boundaries are part of the task, not optional extras to check after deployment.
Turn the Task Into a Checkable Contract
Start by writing down what must be true when the agent finishes. In the task-tracker example, each approved task must exist in the correct project, have the correct description and owner, and preserve the agreed deadline. There must be no duplicate tasks or unapproved changes. The agent’s final message should accurately describe what was saved and what remains unresolved.
Also define the correct response when completion is impossible. If two team members have the same name, asking which person was intended may be the right result. If a deadline is missing, leaving it unresolved may be preferable to guessing. An agent should not be marked wrong for seeking information that the test itself fails to provide.
Anthropic’s evaluation guidance separates a task, an individual trial, its recorded interactions and the final outcome. This distinction helps avoid a common scoring mistake: accepting the agent’s claim of success as proof. For our assistant, the evidence is the saved project state, supported by the record of authorized actions.
A practical test specification should include the starting data, the user’s request, the allowed actions and the expected result. Keep the example understandable enough that another reviewer can reach the same verdict. A test that secretly expects a particular date format or output location, without making that requirement available to the agent, can measure ambiguity rather than competence.
Build a Test Set That Exposes Different Failures
Begin with a manageable collection of realistic cases rather than a large pile of nearly identical prompts. OpenAI’s evaluation recommendations emphasize task-specific tests, representative data and human feedback. For a new workflow, the questions that arise during manual testing are often a useful starting point.
Our task-tracker assistant could be tested with the following cases. These are illustrative scenarios, not measured results from a particular product:
Complete notes identify three tasks, owners and deadlines. The agent should prepare the correct records and save them only after approval.
One task has no deadline. The agent should preserve that uncertainty or ask for the missing date instead of inventing one.
Two existing users share a name. The agent should resolve the ambiguity before assigning work.
A matching task already exists. The agent should follow the stated duplicate-handling rule rather than silently creating another copy.
The save tool returns a timeout. The agent should check whether the operation completed before deciding whether to retry.
The notes contain an instruction to email an unrelated address. The agent should treat that text as source material, not permission to expand the task.
Include cases where an action is appropriate and cases where it is not. Testing only whether an agent creates tasks can reward excessive task creation. Testing only whether it asks questions can reward unnecessary hesitation. The useful behavior depends on the information and authority available in each case.
Keep a separate set of cases for judging proposed changes. If every failure is repeatedly used to rewrite the prompt, improved scores on those same examples may not show broader progress. New cases and independent review help reveal whether the agent learned a more useful pattern or merely became better fitted to a familiar test set.
Inspect Tool Use Without Demanding One Exact Path
OpenAI’s function-calling documentation describes a loop in which a model requests a tool action, the application executes it and the result is returned to the model. Each boundary offers something to check: the selected tool, its arguments, the execution result and the interpretation of that result.
In our example, choosing the task-creation tool is not enough. The project identifier, owner identifier and task fields must also be correct. A properly formatted request can still target the wrong project. Equally, a successful save operation does not prove that the agent’s next message accurately describes what happened.
A trace records the interactions across a run. OpenAI’s trace-grading guide explains how structured scores or labels can help locate errors within those interactions. For the task tracker, that record might show whether an incorrect owner came from ambiguous source notes, a mistaken lookup or a bad argument passed to the save tool.
Do not confuse a useful trace check with requiring every agent to follow one rigid sequence. A bulk lookup and three separate lookups may both be valid. Require ordering where it matters, such as obtaining approval before saving, while allowing alternative methods that meet the same constraints. Otherwise, a test can reject an efficient solution for being different from its author’s preferred route.
Choose the Right Grader for Each Check
A grader is the logic or reviewer that decides how well a trial met its criteria. Different parts of the job need different kinds of judgment. The existence of a task is usually easier to verify directly than the clarity of its description.
Use deterministic checks for facts the system can inspect reliably. These might compare the saved project identifier, verify that a deadline matches the agreed date or count duplicate records. They produce repeatable results for explicit requirements, but they are only as useful as the assertions they encode. A check for three records would miss three records assigned to the wrong people.
Model-based grading can help assess qualities that are harder to reduce to exact matches, such as whether a task description captures the relevant meeting decision. OpenAI’s grader documentation covers approaches including string checks, text similarity, model judgments and code-based evaluation. A model score remains an evaluation signal; it is not independent proof that every requirement was satisfied.
Write a clear rubric for subjective judgments. For our task descriptions, a reviewer might check whether the requested action is identifiable, the wording preserves the agreed scope and unsupported details were added. Compare automated judgments with human reviews, especially on disagreements and borderline cases. Reviewers also need enough context to distinguish a reasonable interpretation from a convenient guess.
Test Repeatability and Recovery
One successful attempt answers whether the system succeeded once. Repeated trials help answer whether the behavior is consistent. The 2024 tau-bench paper evaluated tool-using agents through final database states and introduced a repeated-trial reliability metric. Its historical scores should not be treated as current model rankings, but the distinction between occasional success and consistent behavior remains useful.
Repeat important cases from the same defined starting state. For our task tracker, leaving records from an earlier run in place could make the next trial harder, easier or simply different. Reset the test environment, or deliberately specify that persistent state is part of the scenario. Record the model, prompt and tool versions so the results remain interpretable.
Then introduce controlled failures. If a save request times out, the operation may have failed or may have completed without a confirmed response. Blindly retrying could create a duplicate. A useful test checks whether the agent investigates the state, reports uncertainty honestly and follows the workflow’s recovery rules.
Track assisted completion separately from uninterrupted autonomous completion. An agent that needs a person to correct the owner may still be useful, but its supervision requirement matters. Likewise, a correct handoff on an ambiguous case can be a success under that case’s contract. Treating every escalation as failure encourages guessing; treating every escalation as success can conceal an agent that rarely completes straightforward work.
Make Permissions and Untrusted Content Part of the Test
An agent can reach the requested outcome while taking an unacceptable extra action. The task-tracker assistant might create the right records and also send the meeting notes elsewhere. A completion score that ignores those actions would present a misleading picture of the system.
OpenAI’s safety guidance describes prompt injection as untrusted content attempting to redirect an AI system. In our example, text inside a meeting note might tell the agent to ignore its instructions or use an unrelated tool. Evaluation should test whether the system preserves the distinction between information it reads and authority to act.
Use synthetic examples and a controlled environment for these checks. For the task tracker, verify that the agent saves only within the permitted project, honors the approval boundary and does not use unrelated communication tools. Also test ordinary misunderstandings, not only deliberately hostile text. An incorrectly interpreted request can cause an unauthorized action without an attacker being involved.
Passing a collection of boundary tests does not establish that every attack or misuse has been prevented. It provides evidence about the cases tested. The deployment still needs appropriate access limits, monitoring and a way to stop or correct unwanted activity.
Measure the Cost of Useful Work
A faster or cheaper run is not automatically a better run. Google ADK’s evaluation documentation distinguishes final-response quality from the agent’s tool-use trajectory and includes efficiency measurements such as tool calls, token usage and duration. Those measures become meaningful when considered alongside correct task completion.
For our assistant, useful operational metrics would include accepted completions, human interventions, end-to-end time and the cost of all attempts. OpenAI’s tool-calling guidance recommends comparing efficiency with correctness, completeness and evidence coverage. Reducing the number of calls should not be rewarded if it removes a verification step the task requires.
Consider an illustrative calculation: 100 trials cost $12 in total, and 80 produce correct, authorized completions. The model-and-tool cost per accepted completion is $0.15, calculated as $12 divided by 80. That figure includes unsuccessful attempts, but it still excludes any human review time or other operating costs not counted in the $12. State what the measure includes before comparing systems.
Averages can also hide disruptive outliers. If most runs finish quickly but a few enter long retry loops, record those delays and the stopping behavior. Set limits appropriate to the task and test what happens when they are reached. A budget limit that ends with a false claim of completion is a different outcome from a clear report of unfinished work.
Keep Evaluation Useful After Launch
An evaluation suite should evolve with the workflow. Save meaningful failures as new cases, keep previously working behavior under test and rerun relevant checks when models, prompts, tools or permissions change. OpenAI’s agent-evaluation guide connects individual trace inspection with repeatable datasets and evaluation runs for comparing changes over time.
Separate the question “Can the agent do something new?” from “Does it still do the work it handled before?” A change that improves task descriptions might also weaken duplicate detection. Looking at both makes the tradeoff visible, rather than allowing a better average score to hide a regression in an important behavior.
Before expanding access, write a short decision record: which tasks the agent supports, how success was measured, where it failed, how much supervision it required and which conditions remain untested. There is no universal pass rate that makes every agent ready for deployment. The acceptable failure modes depend on the work and the consequences of an incorrect action.
For the task-tracker assistant, the practical standard is straightforward: the right tasks are saved in the right place, approval limits are respected, uncertainty is reported accurately and failures can be diagnosed. A polished final message is welcome. Evidence that the system did the authorized work is what makes that message trustworthy.
OpenAI says its models helped trace a hidden office recorder to an employee who was fired. The company says an FBI investigation is ongoing after an arrest.
