dooopSoftware · Quality · 12 min

How to Evaluate an AI Application with Clear Criteria

Evaluate AI applications by real cases, verifiable criteria, and failure severity before comparing models or expanding exposure.

Published on September 6, 2026

MAIN THESIS

A high score can mask a blocking failure. Case-based criteria show where the application can improve.

Evaluation starts from the decision to be made. Cases, severity, and verifiers come before comparison.

An evaluation of artificial intelligence applications begins when you define which situations represent the real work, the acceptable risk, and the decision the application needs to support. The model’s average score can help later, but it does not replace a clear evaluative decision: which failure is tolerable, which requires mitigation, and which blocks progress. Before comparing prompts, models, agents, or versions, the team needs to specify what counts as an acceptable result, a recoverable failure, and a blocking failure for each type of use.

What Exactly Will Be Evaluated in the AI Application

The first decision is to delimit the evaluation unit. This seems simple, but it is where much discussion about quality in AI applications becomes confused.

You may be evaluating a textual response, a recommendation, a classification, an action executed in another system, or a complete flow combining interface, model, business rules, and human review. Each of these units requires different criteria.

Evaluating the model is not the same as evaluating the user experience. Evaluating the user experience is not the same as evaluating the operational outcome. An application can generate a clear response yet still make an inappropriate decision. It can also use a competent model within a poorly designed flow, with insufficient context, absent confirmation, or ambiguous policy.

Therefore, the initial question must be concrete: what needs to work for this application to be considered safe enough to proceed?

For example, in a customer service feature, the evaluation unit may be the summary produced by the AI. In an application that suggests actions, the unit may be the recommendation and its justification. In an AI agent—that is, a system that plans and executes steps in tools or environments—the unit may be the entire flow: input received, relevant steps, result in the environment, and final verification.

This separation avoids a common trap: turning everything into a single score. The score can be useful to track trends but should not hide the nature of the failure. An incomplete response, a fabricated response, and an inappropriate action are not the same type of problem.

For a broader view on software quality with AI, it is worth connecting this evaluation to the cluster’s reference guide: Software Quality with AI: Testing, Evaluation, and Responsibility. Here, the focus is narrower: creating an evaluation applicable to a specific feature or application.

How to Choose Representative Cases Without Trying to Cover Everything

A good evaluation does not try to anticipate all possible uses. It deliberately chooses a small set of cases that represent frequency, impact, risk, ambiguity, and relevant exceptions.

The goal is not to prove the application will never fail. It is to gather evidence to decide release, mitigation, human review, or blocking.

An initial set of AI evaluation cases may combine:

Common cases, representing the expected and frequent use of the application.
Difficult cases, where the input is incomplete, ambiguous, or contradictory.
High-impact cases, where a wrong response causes significant rework, user frustration, or operational exposure.
Refusal, escalation, or confirmation cases, where the application should not respond alone.
Known exceptions, which do not always occur but change the decision when they do.

Fictional example: an AI application helps a customer service team summarize refund requests. A common case evaluates whether the AI identifies the amount, date, and reason for the request. An ambiguous case evaluates whether it asks for confirmation when the receipt is illegible. A blocking case evaluates whether it avoids automatically approving a refund outside policy.

The criterion is not “good summary.” The criterion is: contains the three mandatory fields, signals uncertainty when evidence is missing, and does not execute approval without authorization.

This example is fictional, but the logic applies: each case needs to exist for a reason. If no one can explain why a case is included in the evaluation, it is probably there out of habit, anxiety, or personal preference.

A good test for choosing cases is to ask:

Does this case represent a real usage situation?
Is it common, critical, ambiguous, rare with high impact, or a known exception?
Is there any product, engineering, or operational decision associated with the result?
Would failure in this case change the feature’s release?

If the answer is “no” to all, the case probably should not be in the first version of the evaluation.

How to Write Success Criteria That a Team Can Apply

Success criteria in AI need to be observable. “Good response,” “seems correct,” and “appropriate tone” can be useful perceptions in conversation but are weak as evaluation criteria.

An applicable criterion states what must be present, what cannot happen, and how someone or some verifier will identify the acceptable result.

In the fictional refund example, the criteria could be written as follows:

Completeness: the response identifies amount, date, and reason when these data are available in the request.
Correctness: the response does not invent missing data nor alter values provided by the user.
Policy adherence: the application does not automatically approve requests outside defined conditions.
Clarity: the response separates extracted information, uncertainty, and suggested next action.
Human confirmation: when the receipt is illegible, the application asks for confirmation instead of concluding the case.
Absence of inappropriate action: the application does not execute approval without explicit authorization.

These criteria allow product, engineering, quality, and operations to discuss the same thing. The conversation shifts from “I liked it” or “I don’t trust it” to “in this case, the confirmation criterion failed.”

The DORA recommendation on test automation helps situate this practice within development: tests should accompany the work, combining automation and manual activities such as exploration and usability, and suites need maintenance and review. In AI applications, this means evaluation should not be treated as an isolated step after the product is already finished.

Automation can repeat checks. Human judgment remains necessary when the criterion involves ambiguity, contextual sensitivity, or still poorly defined policy.

How Criteria Cover Trajectory, Response, and Outcome

In AI applications, especially agents, the team needs to separate three things: the trajectory, the final response, and the effective outcome.

Trajectory is the path taken: which information was used, which tools were called, which intermediate decisions occurred. The final response is what the AI communicates to the user or system. The outcome is what actually changed in the environment.

This distinction matters because a message saying the task is complete does not prove the task was completed.

Anthropic, in a text about agent evaluations, distinguishes the agent’s execution trajectory from the effective outcome in the environment. It also describes evaluations with inputs, success criteria, and verifiers, possibly requiring multiple attempts.

In an application that only summarizes text, trajectory may be less relevant. But in an agent that queries data, fills fields, or triggers tools, it can reveal errors that the final response hides.

Imagine, still in the fictional customer service example, that the AI responds: “Request successfully registered.” This phrase does not prove the record was created in the correct system, with the right fields, and without improperly approving the refund. The evaluation needs to verify the final state, not just the message.

A simple evaluation design for this type of case includes:

Input: original request, available attachments, and allowed context.
Relevant behavior: data consulted, confirmation requested, action executed or avoided.
Outcome in the environment: record created, fields filled, status maintained or changed.
Verifier: rule, human inspection, or automated check that confirms whether the outcome meets the criterion.

This is a practical difference between testing a response and evaluating an AI feature. When there is an action, the criterion needs to reach the consequence of the action.

How to Score the Evaluation Without Hiding Risk

Scoring helps compare versions, but scoring poorly creates a false sense of control. The problem is not using a score. The problem is letting the average hide critical cases.

A simple matrix can work better than a sophisticated scale:

Approved: meets the case criteria without relevant reservations.
Approved with reservation: meets the main objective but requires adjustment of clarity, format, or experience.
Recoverable failure: errs in a way correctable by human review, repetition, confirmation, or instruction improvement.
Blocking failure: generates relevant risk to the user, business, operation, or internal compliance defined by the organization.

The central point is to separate severity from frequency. A rare failure can prevent production if it is blocking. Likewise, several small failures can indicate low perceived quality, even if none is individually severe.

In practice, each evaluation case should carry an associated decision. If the common case fails, the team may correct instruction, context, or interface. If the ambiguous case fails, it may require human confirmation. If the blocking case fails, the feature may not have autonomy at that point.

This approach also helps avoid rushed comparisons between alternatives. One model may perform better in fluency and worse in refusal. Another may be less elegant but more conservative in risk cases. Without case-by-case criteria, the team tends to choose the most convincing demonstration.

AI application evaluation needs to make the decision auditable, not reward the most seductive response.

This evaluation adds to other quality checks. Testing AI-generated code, reviewing dependencies, or calibrating evaluations with experts are separate issues. For the specific product acceptance decision, the question is different: which cases prevent progress if they fail?

How to Use the Evaluation Before Expanding Exposure

Offline evaluation, done with controlled cases, does not end the decision. It creates evidence to decide whether the feature can be activated for a limited group, observed, and then expanded.

Here a technical and managerial distinction is worth noting: making code available is not the same as activating a feature. The Google SRE chapter on gradual releases addresses evaluating changes on a portion of traffic before expanding exposure and discusses using configurations to separate code availability from feature activation.

For AI applications, this separation is useful because it reduces confusion between technical delivery and operational exposure. The team can deliver the technical base without releasing broad autonomy. It can activate for a limited group. It can maintain human review in certain cases. It can stop expansion when a blocking case appears.

At this point, the evaluation ceases to be a document and becomes a release ruler.

Before expanding exposure, the team should be able to answer:

Which cases were evaluated and why were they chosen?
Which criteria defined success, reservation, recoverable failure, and blocking failure?
Which failures were consciously accepted and with what mitigation?
Which types of cases still require human review?
What will be monitored when the feature is activated for more users?
What outcome stops the expansion?

These questions do not eliminate uncertainty. They make uncertainty manageable.

There are clear limits. The evaluation proposed here does not prove the application will be correct in all future uses. Representative cases need to be reviewed when the product, users, policies, or model change. Automation helps repeat checks but does not replace human judgment in ambiguous criteria. And an aggregated score can hide serious failures.

It is also worth separating this topic from broad productivity measurement. The METR update from February 2026 considers its new data an unreliable signal of AI’s current effect on productivity and points out difficulties measuring time with competing agents. For this article, the lesson is more limited: explicitly define the measured unit before interpreting any result.

Decision Roadmap for Cases and Criteria

Before comparing models, prompts, agents, or application versions, record the decisions below.

Does the case represent a real usage situation? Describe who makes the request, the objective, available data, and which decision depends on the response.
Does the case have a clear reason to be in the evaluation? Classify as common, critical, ambiguous, rare with high impact, known exception, or a situation where the AI should refuse, escalate, or ask for confirmation.
Is success verifiable? Write what needs to be correct, what cannot happen, and how someone or some verifier will identify the acceptable result.
Does the failure have defined severity? Separate cosmetic failures, utility failures, failures requiring rework, and blocking failures due to risk to the user, business, or internal organizational rules.
Does the criterion avoid personal taste? Replace “good response” with observable signals such as “includes the three mandatory pieces of information,” “does not invent missing data,” or “asks for confirmation before executing the action.”
Does the case allow comparison between versions? Keep input, context, expectation, and criterion stable enough to compare prompt, model, tool, or application version.
Is there a decision associated with the result? Define before execution what happens if the case fails: correct instruction, adjust product, limit autonomy, require human review, or prevent usage expansion.

Building an AI application evaluation is choosing which cases represent real use, which criteria make success verifiable, and which failures prevent expansion. From there, the team does not decide by the most comfortable average. It decides by the error it can still manage and the error that requires containing the application.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss the Application in the Company

Conversation about the software company context

Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.