dooopSoftware · Quality · 12 min

Acceptance Criteria for AI Features

Define success, blocking failures and autonomy limits before releasing AI, with verifiers for content, process and environment.

Published on September 6, 2026

CORE THESIS

An AI feature is not ready just because it seems good. It is ready when success, failure and limits can be verified.

Acceptance criteria set the boundary for the AI product. The focus is deciding what can act, pause or reject.

Acceptance criteria for artificial intelligence are the way to decide, before release, what a feature needs to demonstrate to be considered ready. They turn three things into verification: what counts as success, what counts as failure and which limits cannot be exceeded even when the experience seems good.

Without this, feature acceptance becomes a discussion of impressions about the product: the response seems useful, the flow seems acceptable, the demonstration seems convincing.

AI acceptance starts with the decision the feature can make

An AI feature rarely delivers just "one answer." It suggests, classifies, prioritizes, drafts, recommends, fills fields, triggers flows or prepares a decision for someone. The acceptance criterion needs to start with this concrete action.

The initial question is not "Is the AI good?" It is: what decision, suggestion or action is this feature authorized to produce within the product?

This formulation removes evaluation from the abstract. The team stops discussing intelligence in general and starts verifying the specific behavior that affects the user in the flow. A well-written text can be wrong. A plausible classification can move an item to the wrong place. A recommendation useful in most cases can be unacceptable if it exceeds the defined autonomy limit.

Therefore, acceptance of AI features must describe the unit of responsibility. For example:

Does the AI only suggest a response?
Does it fill fields that a person reviews?
Does it execute an action in the system?
Can it reject a request?
Should it ask for more information when there is uncertainty?

This definition also connects software quality with business decision. If the organization is still structuring its adoption, it is worth relating this acceptance level with broader choices of priority, risk and capacity, as in a business-connected artificial intelligence strategy or in an AI roadmap. The acceptance criterion does not replace these decisions. It prevents them from disappearing in implementation.

Describe success as an observable result, not as a convincing response

The success of an AI feature needs to be verifiable in the flow where it operates. "Responding well," "being useful" or "seeming correct" are initial assessments, not sufficient acceptance criteria.

A good criterion describes three components:

Input: which data, requests or contexts are within scope.
Expected behavior: what the feature must produce or do.
Minimum evidence: how the team proves the result was achieved.

Anthropic, when discussing agent evaluations, distinguishes the agent's execution trajectory from the actual result in the environment. A message saying the task is finished is not enough to prove the result, and the evaluation uses inputs, success criteria and verifiers Anthropic. Even when the feature is not an autonomous agent, this separation is useful: do not accept the AI's narrative as proof of the result.

Compare two criteria:

Weak: the AI must generate an appropriate response for the user.
Better: for requests classified as product usage questions, the AI must respond using only the provided base, cite the internal reference section shown to the reviewer and indicate when there is insufficient information.

The second criterion may still require technical refinement but already allows behavior verification. It defines domain, source, evidence and reaction to gaps. The first depends on loose judgment and tends to produce endless discussions in review.

When content requires human interpretation, the criterion can include manual review. This is not a failure of automation. DORA recommends tests throughout development, combining automation and manual activities such as exploration and usability, and also recommends maintaining and reviewing test suites instead of treating quality as a later step DORA. In this guide, this combination is the starting point: automated criteria when possible, human review when judgment about context, sufficiency or use is still necessary.

Turn predictable failures into blocking criteria

Acceptance is not just proving something works. It is deciding which failures prevent release.

In AI features, averages and positive examples can hide relevant risks. A team may observe several useful responses and still need to block the feature if it executes an unauthorized action, invents critical information, omits uncertainty in a sensitive case or produces output in a format incompatible with the system.

Negative criteria must be written before delivery pressure. Some common types of blocking failure:

Response without basis in available information.
Instruction that contradicts a defined business rule.
Action executed without confirmation when confirmation was mandatory.
Omission of uncertainty when data are incomplete.
Treating an exception as if it were a routine case.
Output outside the required format for flow continuity.
Improper alteration of data in a connected system.

The practical formulation is: "the presence of this failure rejects acceptance until correction, scope restriction or explicit design change."

This phrase changes the conversation. Instead of asking if the feature "was good enough," the team asks if any failure appeared that the organization has already decided not to accept. Confidence does not arise from enthusiasm about successes. It arises from the ability to name errors that cannot pass.

This is also where leadership, product, engineering and quality need to be in the same room. Engineering can automate some verifiers. Product can define value and usage context. Quality can make edge cases and regressions explicit. Leadership must decide which risks are incompatible with user exposure. If the organization does not yet know how to have this conversation, the problem is not only technical. It is a maturity sign to develop, as discussed in AI maturity.

Define operational limits before measuring performance

A feature can perform well in common cases and still be unacceptable. This happens when it exceeds the allowed autonomy level.

Autonomy limit is the boundary between suggesting, preparing, executing, approving, rejecting or interrupting an action. For acceptance criteria, this boundary needs to be written operationally.

Instead of "the AI helps the user solve requests," prefer something like:

The AI can suggest a category.
The AI can fill a preliminary justification.
The AI cannot approve the request.
The AI must forward to human review when the receipt is illegible, incomplete or contradictory.

This type of limit reduces ambiguities. It also prevents a good demonstration experience from being confused with authorization to operate at scale.

Autonomy does not need to be the same for all actions. A feature can fill a low-risk field, only suggest text in another point and block any execution in cases outside the domain. Mature decision-making is not always to restrict. It is to calibrate autonomy according to impact, reversibility, available evidence and monitoring capacity.

Third-party models can be part of mature products. The point is not owning the model but designing the allowed behavior, result verifiers and interruption conditions. Maturity appears less in vendor choice and more in clarity about what the system can do when it is right, uncertain or wrong.

Use different verifiers for content, process and environment

An acceptance criterion for AI can fail for different reasons. The response may be wrong. The path used to reach the response may violate a rule. The effect on the environment may not have occurred.

Therefore, it is worth separating three verifiers:

Content verifier: evaluates if the output is correct, sufficient, grounded or in the expected format.
Process verifier: evaluates if the feature respected steps, sources, permissions and autonomy limits.
Environment verifier: evaluates if the expected effect occurred in the system, record, flow or interface.

This separation is especially useful when the feature calls tools, queries databases, writes in systems or prepares actions for people. The AI may say it updated information, but the environment may not reflect this. It may produce the right response using a prohibited source. It may follow the correct process and still generate insufficient output.

Not every verifier needs to be automated. Some can be deterministic rules, such as checking format, presence of fields or absence of prohibited action. Others may involve human review, especially when there is usability, contextual interpretation or judgment about sufficiency. The acceptance criterion must say which evidence will be accepted, not just who had a good impression.

Here is a useful check for the team: if two people review the same output, can they use the criterion to reach a similar decision? If the answer is no, perhaps the criterion is still describing preference, not acceptance.

Include gradual acceptance when user exposure increases risk

Accepting a feature in a controlled environment does not mean activating it for all users, flows and volumes. In AI, this distinction is relevant because context changes can alter perceived behavior and operational risk.

The Google SRE chapter on gradual rollouts addresses evaluating a change on a portion of traffic before expanding exposure. It also distinguishes releasing code from activating features and discusses configurations to separate these decisions Google SRE. The application to AI acceptance is direct: releasing code, activating the feature and expanding use do not need to be the same decision.

Acceptance criteria can foresee expansion and interruption conditions. For example:

The feature starts active only in a limited set of contexts.
Expansion depends on absence of previously defined blocking failures.
Qualified complaints, flow degradation or increased manual intervention may require pause.
Certain input types remain out of scope even after initial release.

The function of gradual acceptance is to make expansion an observable decision. The team defines beforehand which signals allow advancing, pausing or reducing exposure, instead of discovering these limits only in broad use.

It also helps avoid a trap: treating the launch as an endpoint. In AI features, initial acceptance should open a monitoring cycle. If context changes, the base changes, the model changes or user behavior changes, criteria need to be revisited.

Acceptance matrix for AI features

This guide proposes using the matrix below before construction, in quality review and in release decision. It does not replace a complete technical, legal, regulatory, information security or governance assessment when context requires, but helps make success, failure and limits verifiable.

Covered input

Question: which inputs should the feature accept and which should be rejected or routed to another flow?

Expected evidence: list of input types with valid, invalid and ambiguous examples.

Acceptance rule: the feature only passes if it correctly handles the defined input types and rejects or escalates out-of-scope cases.

Verifiable result

Question: how will the team know the output produced the expected result, not just a well-written response?

Expected evidence: comparison criterion, business rule, automated verifier or predefined human review.

Acceptance rule: the feature only passes if evidence confirms the expected result in the defined environment or flow.

Blocking failure

Question: which errors prevent release even if the average of tests seems good?

Expected evidence: list of failures that block acceptance, such as unauthorized action, response with dangerous instruction, omission of uncertainty when it must be declared or improper data alteration.

Acceptance rule: presence of any blocking failure rejects acceptance until correction or scope change.

Autonomy limit

Question: can the AI suggest, fill, execute, approve or only prepare an action for review?

Expected evidence: simple matrix of allowed actions, prohibited actions and actions requiring human confirmation.

Acceptance rule: the feature only passes if it remains within the defined autonomy level for each action.

Behavior under uncertainty

Question: what should the feature do when it lacks sufficient evidence to respond or act?

Expected evidence: test cases with incomplete, contradictory or out-of-domain data.

Acceptance rule: the feature only passes if it declares limitation, requests additional information, routes for review or interrupts the action as appropriate.

Expansion or interruption condition

Question: which signals authorize expanding exposure and which require pausing, reversing or restricting use?

Expected evidence: monitoring criteria defined before activation, including critical failures, qualified complaints, flow degradation or increased manual intervention.

Acceptance rule: the feature only advances if defined signals remain within accepted limits. Otherwise, exposure must be interrupted or reduced.

Fictional example: acceptance of a refund suggestion

Fictional example: an AI feature suggests responses for refund requests in an internal system.

A weak criterion would be: "the AI must respond correctly to most refund requests." It does not say what is correct, which risk blocks release, which evidence will be used or who decides when there is uncertainty.

A better criterion would be: for expenses below a limit defined by the organization and with a legible receipt, the AI can suggest the category and a justification. Final approval remains with a person. Acceptance requires that the suggestion uses only data present in the receipt and internal rules provided to the system. If the receipt is illegible, if there is contradiction between data or if the expense is outside the expected categories, the AI must declare limitation and route for review.

This criterion allows verifying input, result, autonomy, blocking failure and behavior under uncertainty. It does not promise to eliminate errors. It creates a more honest basis to decide if the feature can enter controlled use, needs adjustment or should have scope reduced.

An AI feature should only advance to implementation or release when the team can write beforehand input, expected behavior, minimum evidence, blocking failures, autonomy limits and interruption condition. If this still does not fit on paper, the product does not yet have sufficient acceptance criteria for user exposure.

If you want to discuss this decision in your company context, talk to dooop.

Sources

NEXT DECISION

Discuss application in the company

Conversation about the software company context

Content by dooop. Registration allows linking this topic to the reader's journey and tracking interest in the subject.