dooopSoftware · Quality · 11 min

How to Evaluate AI Agents That Perform Actions

Evaluate AI agents by separating outcome and process, with external verifiers to confirm actions, scope, permissions, and effects.

Published on September 6, 2026

MAIN THESIS

The agent's statement is just a receipt. Proof lies in the environment, permissions, and execution traces.

Evaluate actions by outcome and process. Autonomy only grows when there is a verifier, limit, and stopping condition.

When an AI agent performs actions, evaluation must start with what it can change: objects, permissions, scope, cost, and side effects. The completion message does not prove the action happened, nor that it happened acceptably. To increase autonomy with less improvisation, the team needs to look for evidence external to the agent's report: system state, logs, tools used, attempts, and side effects.

When the completion message does not prove the action occurred

An AI agent that performs actions does not deliver just a response. It queries systems, calls tools, modifies objects, creates records, triggers flows, and may make intermediate decisions until reaching the requested goal.

That is why the final message is weak evidence. It shows what the agent claims to have done, not what the environment confirms.

The difference seems small until production. An agent may say it updated a configuration but changed the wrong field. It may have created a duplicate item. It may have executed the correct action in a scope larger than authorized. It may have failed at a step, tried to work around the error, and still reported success.

Anthropic distinguishes the agent's execution trajectory from the actual result in the environment. It also notes that a message saying the task ended is not enough to prove the result. This separation helps teams evaluating AI agents with action capabilities.

In practice, evaluation starts outside the agent's response. The question is not "Did it seem confident?" The question is "Did the system reach the expected state, by a path we would accept?"

This requires evidence such as:

Object created, modified, or removed in the correct place.
Audit log or change history compatible with the task.
No changes to objects outside the scope.
Tool calls consistent with granted permissions.
Records of failures, interruptions, and retries.

Without these traces, the team continues evaluating the agent's narrative, not the execution.

Outcome and process are two layers of the same evaluation

Evaluating outcome means checking the final state. Was the rule created? Was the ticket classified? Was the configuration applied? Was the file updated? Does the expected action appear in the correct system?

Evaluating process means checking the path. Which tools were used? What permissions were involved? Did the agent try the same operation multiple times? Did it change the expected order of steps? Did it request authorization when uncertain? Did it stop when it should?

Both layers must be approved separately.

A correct outcome with an inadequate process should not be treated as accepted execution. If a person solved a simple task using broad administrative access, deleting intermediate history or modifying data outside the scope, the team would likely investigate. The same standard should apply to agents.

The opposite also matters. An agent may follow an apparently adequate process but produce no real effect. In that case, execution was also unsuccessful. The ritual was fulfilled, but the environment did not change as needed.

This distinction reduces a common confusion: testing an agent as if it were only testing text. In applications that only respond, output quality is central to evaluation. In agents that perform actions, the textual output may be just a narrative receipt. What needs to be proven is in the environment.

This discussion connects to the broader discipline of software quality with AI, but has its own requirement: testing must observe the effect of the action, not just the plausibility of the explanation.

How to define verifiers before releasing the agent

A verifier is an observable criterion that confirms whether an execution was successful, acceptable, or blockable. It must exist before execution, not after the agent has acted and the team is trying to interpret what happened.

Anthropic describes evaluations with inputs, success criteria, and verifiers, and recognizes that multiple attempts may be necessary. The practical application of this in product and engineering is simple to say and hard to maintain: it is not enough to ask the agent if it succeeded. It is necessary to define how the system will prove success.

A good set of verifiers answers concrete questions:

Which object should change?
Which object must not change?
Which field, status, rule, event, or record proves the action?
What permission can the agent use?
How many attempts are acceptable before stopping?
Which error requires human intervention?
How to avoid duplication if the same input is re-executed?
How to undo, pause, or isolate the effect if something goes wrong?

These questions seem operational—and they are. That is their virtue. In evaluation, they help test concrete risks: wrong scope, wrong tool, uncontrolled repetition, ambiguous authorization, insufficient logging, unnoticed side effect.

Evaluation of AI agents must turn these possible failures into acceptance criteria. This reasoning also applies to broader AI functionalities, as discussed in how to create an AI strategy connected to business: technical capability must be accompanied by decision, limit, and operational learning.

Fictional example: agent that updates a service rule

Consider a fictional example. A company uses an internal agent to update ticket routing rules. The received task is to adjust a rule so that requests of a certain type are routed to a specific service queue.

The agent has access to a configuration tool, a change history, and an internal policy database. At the end, it responds: "Rule updated successfully."

This phrase does not end the evaluation. It starts the check.

The expected outcome could be verified by evidence such as:

The correct rule was created or modified.
The expected request type points to the defined queue.
The old rule, if replaced, appears in the history.
Queues outside the scope were not modified.
The change record identifies the task, event time, and responsible agent.

Acceptable process would require another layer of analysis:

The agent consulted the policy before changing the rule.
Used only the authorized configuration tool.
Did not request broad administrative permission for a restricted change.
Did not try to change multiple queues to compensate for uncertainty.
Stopped execution if it found conflict between existing rule and received request.

Note there is no need to turn this example into a success case. It is a test scenario. Effects are hypotheses to measure: the rule may have been applied correctly, may have caused conflict, may have created duplication, may have revealed that the instruction was ambiguous.

Maturity appears in how the team decides, before execution, what to accept, investigate, and block. Without this, the agent becomes an action box with a pretty receipt.

Checklist for evaluating an agent that performs actions

This checklist helps decide whether an execution can be accepted, investigated, repeated, or blocked. It does not replace domain-specific tests but helps engineering, quality, and product teams discuss the same object.

Verifiable outcome

Question: Does the system state confirm the expected action without relying on the agent's final message?

Accept when there is external evidence: object created, field changed, record updated, task completed, or event emitted in the correct place.

Investigate when the agent declared success but the environment state does not confirm completion.

Scope preserved

Question: Was the action limited to authorized objects, users, data, or configurations?

Accept when execution affected only the expected scope.

Investigate when there were changes outside the scope, even if the main goal was achieved.

Acceptable process

Question: Did the agent use tools, permissions, and step sequences compatible with the task?

Accept when tool calls and intermediate decisions make sense for the goal and respect constraints.

Investigate when the agent reached the result using shortcuts, excessive permissions, unnecessary repetition, or a path the team would not accept from a person.

Stopping conditions

Question: Did the agent stop execution when it found uncertainty, error, lack of permission, or risk of side effect?

Accept when there is clear record of stopping or intervention request at defined points.

Investigate when the agent kept trying, bypassed a failure, or changed strategy without authorization.

Controlled repeatability

Question: Can the same input be re-executed without duplicating actions, causing conflicts, or masking failures?

Accept when execution is idempotent, i.e., can be repeated without multiplying undesired effects, or when it has duplication protection.

Investigate when a new attempt may create duplicate objects, overwrite previous decisions, or cumulatively alter the environment.

Reversibility

Question: Is there a known way to undo, pause, or isolate the action's effect?

Accept when reversal was defined before execution or when impact is limited and isolatable.

Investigate when the action produces effects difficult to track, undo, or contain.

Evidence for learning

Question: Does execution leave enough traces to improve instructions, tools, permissions, or test cases?

Accept when inputs, relevant steps, results, and failures are available for review.

Investigate when the team only has the agent's final response and cannot understand how the decision was made.

This checklist works best when used before expanding autonomy. Once the agent operates in a large scope, the organization tends to negotiate with incidents, not with design.

When to expand autonomy, permissions, and reversible actions

Not every agent action needs to start with broad exposure. When there is risk of systemic effect, it makes sense to activate first in a controlled subset of usage, users, queues, data, or configurations.

The canary releases chapter of Google SRE addresses evaluating a change on a portion of traffic before expanding exposure. It also distinguishes making code available from activating features and discusses configurations to separate these decisions.

The analogy is useful for agents. Making capability available is not the same as activating autonomy. The team may have the agent installed, integrated with tools, and technically ready, but still limit where it acts, with which permissions, and under which verifiers.

This separation allows finer decisions:

The agent can suggest but not execute.
It can execute only on low-risk objects.
It can act on a subset of internal users.
It can require human approval under certain conditions.
It can be automatically stopped when a verifier fails.

The point is not to create bureaucracy by principle. It is to reduce impact radius while the team learns from real executions. For this, autonomy limits must be designed as part of the product, not as a late fix. This topic relates directly to AI maturity decisions, because autonomy without criteria usually reveals more about the organization than about the model.

Limits of evaluation: what it does not solve alone

Evaluating an agent's execution well does not by itself prove business value, team productivity, or organizational performance improvement. It proves something more specific: that, given certain inputs, tools, and permissions, the agent produced a verifiable result by an acceptable process.

This limit matters.

In the February 2026 update, METR treats its new data as an unreliable signal of AI's current effect on productivity. Among cited difficulties are participant and task selection and timing measurement with competing agents. For this article, the point is methodological: in agents, outcome, process, and execution time can mix when multiple activities happen in parallel.

There are also domain limits. An acceptable process for a low-risk internal task may be inadequate in contexts involving personal data, customer communication, security, or effects difficult to reverse. The criterion is not universal. It must follow the action's impact.

There is also a design limit. Evaluation does not replace well-configured permission. If the agent receives overly broad access, later evaluation becomes a fragile containment net. Better to restrict beforehand what it can see, change, trigger, and repeat.

Finally, evaluation does not eliminate human judgment. It positions it better. Instead of calling a person only to put out fires, the team defines when the agent can act, when it must request authorization, and when it must stop.

The operational decision becomes narrower and more useful: accept proven executions, investigate inadequate paths, and block actions that leave insufficient evidence. Before expanding autonomy, the team must define which proof external to the agent's report authorizes the change.

If you want to discuss this decision in your company's context, talk to dooop.

Sources

NEXT DECISION

Discuss the application in your company

Conversation about the software company context

Content by dooop. Registration allows relating this topic to the reader's journey and tracking interest in the subject.