dooopSoftware · Quality · 12 min

How to Organize Evaluation Cases for AI

An AI evaluation dataset requires separating cases, labels, and review so that the score reflects real risks, not just comfortable averages.

Published on September 6, 2026

MAIN THESIS

A high score can hide a poorly maintained dataset. The risk appears in the origin, label, and review of cases.

Evaluation cases need function, status, and evidence. Without these, the score measures a mix, not a decision.

An AI evaluation dataset is a living governance artifact. It helps decide when to record where cases came from, what role they play, which labels support the evaluation, and when they need to be reviewed. Without this discipline, the team can improve the score while remaining blind to relevant failures, ambiguous criteria, or outdated examples that no longer describe the product.

When the Case Base Stops Measuring What Matters

In many teams, the dataset grows cumulatively. The team starts with some real examples, adds situations recalled by experts, includes fixed bugs, copies old responses, creates difficult questions to test system limits, and soon calls all this the evaluation dataset.

The problem is not gathering varied cases. The problem is treating cases with different functions as if they were the same.

A frequent case shows common use. A critical exception shows risk. An exploratory case tests an immature hypothesis. An old bug may serve for regression but might no longer represent the expected product behavior. When all are averaged without identification, the final score looks objective but loses decision power.

This article does not address how to design an entire evaluation, a topic close to how to create an evaluation for an AI application. The focus here is more specific: organizing evaluation cases so leadership can understand what the score measures, what it does not measure, and what kind of review is still missing.

A well-organized dataset does not eliminate human judgment. It records that judgment so it can be compared and reviewed.

Separate Representative, Critical, and Exploratory Cases

The first decision is to separate the origin and function of each case. Without this, the dataset mixes volume, risk, and technical curiosity.

Representative cases reflect recurring product uses. They help answer: does the functionality work well in scenarios people actually encounter frequently? For a customer service assistant, for example, they may include questions about delivery time, exchange policy, and order status.

Critical cases cover rare but high-impact situations. They do not need to appear in large numbers to deserve attention. A response that authorizes improper cancellation, exposes inappropriate information, or advises a wrong action can block a decision even if common cases are satisfactory.

Exploratory cases test new hypotheses. They are useful for learning but dangerous when included too early in the main metric. If every newly created hypothesis changes the official score, the evaluation becomes unstable. The team ends up discussing score fluctuations, not decision quality.

A practical way to organize the dataset is to tag each case with its function:

Recurring use: measures expected behavior in common situations.
Critical risk: measures failures that may prevent exposure or increase review requirements.
Regression: measures if a fixed problem has reoccurred.
Complaint or field evidence: records something observed in real use, when such evidence exists.
Exploratory hypothesis: investigates a question before becoming an official criterion.

This separation also reduces a common confusion: thinking representativeness means just having many examples. Representativeness here means the dataset covers the uses and risks that matter for that product decision.

Record Labels as Decisions, Not Loose Opinions

An evaluation label is not just the “correct answer.” It is the recorded decision about what counts as success, failure, incomplete response, or situation requiring review.

If a case has only one input and an ideal response, it can serve as an initial reference. But alone, it does not explain why a future execution should pass or fail. Two people can reasonably disagree about a partially correct response. The label exists to make this disagreement visible and reduce unnecessary ambiguity.

Each case should record at least:

Input: the request, context, or event used in the evaluation.
Expected result: what the functionality should produce or trigger.
Success criterion: how to decide if the response passes, fails, or requires review.
Reason for the label: why that judgment was adopted.
Case origin: recurring use, risk, regression, complaint, product decision, or hypothesis.
Review responsible: person or role who can update the case.
Status: new, validated, under observation, obsolete, or retired.

This structure does not need to start sophisticated. It can start simple. The point is to prevent the dataset from relying on oral memory. When a label does not explain its reason, the evaluation is vulnerable to two distortions: repeating old decisions without context or adjusting criteria to fit the desired result.

In AI functionalities, this is especially relevant because the response can vary. The evaluation should not depend solely on literal comparison with a model answer. In many cases, the correct question is: is this response acceptable to the user, respects product limits, and avoids dangerous actions?

This distinction connects to broader discussions about AI software quality, but the practical decision here is smaller and more operational: every label needs to carry the judgment that will allow comparing future executions.

Record What Type of Evidence Each Case Requires

In simple functionalities, it may suffice to check the final response. In flows with agents, external tools, or multiple steps, this may be insufficient.

Anthropic distinguishes the agent’s execution trajectory from the effective result in the environment. It also notes that a message stating the task is finished is not enough to prove the result. This distinction is useful for organizing evaluation cases because result and trajectory answer different questions.

The final result asks: was the task completed in the expected environment? Was the information recorded? Is the response delivered to the user correct? Was the requested action actually executed?

The trajectory asks: did the agent follow acceptable steps? Did it consult the correct source? Avoid a prohibited tool? Ask for confirmation when it should? Interrupt execution upon finding an unsafe condition?

Not every case needs to evaluate trajectory. If the user experience depends only on the final response and there is no relevant risk in the path, recording steps can become noise. But when product design makes the path affect safety, cost, privacy, traceability, or rollback, trajectory becomes part of the criterion.

The same Anthropic source describes evaluations with inputs, success criteria, and checkers, noting that multiple attempts may be necessary in agent evaluations. As a practical criterion, the case dataset should specify what will be checked rather than assume a confident response equals success.

A case may have, for example:

Response checker: compares if the final guidance complies with product policy.
Environment checker: confirms if the intended change actually occurred.
Trajectory checker: identifies if the agent used a permitted source or asked for confirmation before acting.
Human checker: requires review when judgment depends on context not yet automated.

This avoids false comfort. The AI may declare it solved. The evaluation dataset needs to say what evidence proves this.

Create a Routine to Promote, Review, and Retire Cases

A case dataset changes along with the product. Cases gain relevance, lose function, or need reclassification.

DORA recommends testing throughout development, combining automation and manual activities such as exploration and usability. It also recommends maintaining and reviewing test suites rather than treating quality as a post-development step. This guidance supports a simple point for AI evaluations: the dataset needs maintenance, not just expansion.

Adding cases increases apparent coverage. Deciding what each case means increases dataset usefulness.

A healthy routine can treat new cases as observation before promoting them to the main metric. This allows time to review the label, understand if the case represents recurring use or critical exception, define the appropriate checker, and decide if it should influence the aggregate score.

It is also necessary to retire cases. An example may become obsolete because product policy changed, the functionality no longer exists, the risk was redesigned, or the case was an exploratory hypothesis that did not prove relevant. Keeping everything forever does not increase rigor. Often, it just preserves old decisions no one wants to defend anymore.

A simple review routine can answer:

Does this case still represent current product use, risk, or decision?
Is the label still understandable to someone who did not participate in the original discussion?
Is the success criterion clearer than in the last review?
Should this case enter the main metric, remain under observation, or be retired?
Has there been a product change requiring a new checker?

There is a useful analogy with gradual releases. The Google SRE chapter on canaries addresses evaluating a change on a portion of traffic before broadening exposure and distinguishes deploying code from activating features. For an evaluation dataset, the implication is not to copy the canary process but to preserve the same discipline: separate preparation, activation, and confidence expansion.

A case may be available in the dataset but not yet activated in the main metric. This separation helps learn without turning every discovery into an immediate block or statistical noise.

Fictional Example: Case Dataset for a Customer Service Assistant

Consider a fictional example: a company creates a customer service assistant to answer questions about orders, exchanges, and cancellations. The dataset’s goal is not to prove the assistant “is good.” It is to support decisions about release, review, and autonomy limits.

A representative case could be a question about the exchange period for a recently purchased product. The input records the user’s question and allowed context. The label defines that the response must explain the current exchange policy, inform the appropriate channel, and avoid promising exceptions. The success criterion accepts language variations but fails responses that invent deadlines, ignore restrictions, or direct to a nonexistent channel.

A critical case could involve order cancellation. The input simulates a user requesting immediate cancellation of a purchase made by another account holder. The label should not just say “do not cancel.” It should explain that the assistant needs to request proper confirmation or forward for review, according to product design. The checker in this case may look at both the response and the action executed. If the assistant only says it canceled, that proves nothing. If it executes cancellation without the necessary condition, the case fails.

An exploratory case could test intention ambiguity: “I don’t want this order like this anymore.” The team does not yet know if this phrase should trigger exchange, cancellation, human service, or clarification request. Instead of placing the case in the main metric, it enters as an exploratory hypothesis. The label remains under observation until product, service, and quality decide the expected behavior.

A regression case could come from a fixed bug: the assistant confused exchange policy with warranty policy. This case enters marked as regression. It may be part of the main metric if the risk remains relevant or stay in a specific suite to prevent problem recurrence.

The same case, therefore, is not just a line in a spreadsheet. It carries input, label, success criterion, risk, origin, checker, and review status. This is the difference between a collection of examples and an AI evaluation dataset that supports decision-making.

Readiness Criterion: Does the Dataset Support Decision or Only Scoring?

The final question is not whether the dataset is large. It is whether it allows an honest decision.

An AI evaluation dataset is more ready when leadership can look at the score and simultaneously understand its composition. Which cases measure common use? Which cover critical exceptions? Which are still exploratory? Which labels have been reviewed? Which checkers prove results in the environment, not just a model statement?

Review these fields before treating the dataset as a decision reference:

Origin and function: does each case indicate where it came from and what role it plays in the evaluation?
Weight in decision: are rare and serious failures identified without being diluted in the overall average?
Label: does the case record expected result and reason for judgment?
Success criterion: does the dataset differentiate acceptable, ideal, dangerous, and incomplete responses?
Required evidence: is it clear whether the final result, execution trajectory, environment evidence, or a combination will be evaluated?
Review: does each case have status, responsible party, or reason for next review?
Metric usage: does the team separate main metric cases from those used only for investigation?

If the answer is negative on many points, the dataset may still serve for learning but not for deciding alone. This limit needs to appear in the dataset’s own status to separate learning, investigation, and decision reference.

The concrete decision is to organize the dataset in three layers: representativeness, labels, and review. With this, the score stops being an isolated scoreboard and starts showing which cases support the decision, which are still under observation, and which should already be retired.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss Application in Your Company

Conversation about your software company’s context

Content by dooop. Registration allows relating this topic to the reader’s journey and tracking interest in the subject.