dooopSoftware · Quality · 12 min

How to Review Quality with AI Between Acceptance and Operation

Use AI to cross-check acceptance criteria, tests, and operational signals to locate gaps by evidence, risk, and responsible party.

Published on September 6, 2026

CENTRAL THESIS

Tests can pass while operation accumulates invisible corrections.

AI helps when it cross-references evidence, not when it declares quality.

A feature can meet acceptance criteria, pass existing tests, and still generate manual corrections, recurring doubts, or exceptions in operation. Reviewing the quality system with artificial intelligence starts by comparing three layers many teams treat separately: what was accepted, what was tested, and what appeared in operation.

When Acceptance, Testing, and Operation Tell Different Stories

A feature meets acceptance criteria, passes existing tests, and reaches production appearing ready. Later, operation shows another version: users repeat the same question, support performs manual corrections, exceptions appear in real cases, or the team realizes that behavior acceptable in the test environment causes confusion in use.

The problem is not necessarily a poorly written test. It may lie in how specification, verification, and operational feedback connect.

Acceptance criteria describe what must be true for the team to consider a delivery acceptable. Tests verify part of that behavior. Operation reveals how the feature encounters real use, with data, habits, exceptions, and ambiguities rarely fully visible before release.

When these three layers do not communicate, the team gains a false sense of control. Acceptance says one thing, the test suite protects another, and operation signals a risk that does not return to the process.

Artificial intelligence can help because it reads, groups, and compares large volumes of textual artifacts quickly. It can cross-reference stories, criteria, test cases, support records, incident reports, manual evaluations, and product notes. But AI should not become the final authority on quality. It organizes evidence so product, engineering, quality, and operation can decide better.

This distinction helps avoid two bad extremes: treating AI as a quality oracle or ignoring its usefulness because it does not replace human judgment.

Build the Minimum Evidence Map Before Using AI

Before requesting any analysis, the team needs to gather traceable evidence. Without this, AI tends to fill silence with plausibility. And plausibility is not evidence.

The minimum map should contain three sets.

Defined acceptance criteria: business rules, expected behavior, known limits, relevant messages, permissions, exceptions, and definitions of done.
Existing tests: automated tests, manual tests, exploratory cases, AI evaluation checkers, regression scripts, and execution evidence when available.
Operational signals: recurring questions, manual corrections, incidents, complaints, logs interpreted by the team, service records, qualitative feedback, and usage observations.

This map does not need to be perfect to start. It needs to be honest. If an acceptance criterion exists only in someone's memory, it should be marked as absence of evidence, not as an inferred requirement. If a test seems relevant but no one knows which risk it protects, that is also information.

The guidance from DORA on test automation supports this view: tests should accompany development, combining automation and manual activities such as exploration and usability, and suites need maintenance and review. Quality does not work well when treated only as a later stage.

For an organization structuring broader AI adoption, this work connects to capability diagnosis. A good starting point is also to look at the organization's AI maturity, because quality review depends on process, roles, data, and governance, not just tools.

Use AI to Locate Gaps, Not to Declare Quality

The most useful AI application in this review is to ask comparative questions about already existing materials.

For example:

Which acceptance criteria do not appear in any test, evaluation, or checker?
Which tests do not clearly point to a criterion, risk, or product behavior?
Which operational signals appear recurrently but have not generated changes in acceptance, testing, monitoring, or documentation?
In AI features, is there verification of the actual result or only the success message?
Are there differences between what product considers acceptable and what support needs to correct manually?

AI's response should come with references to the artifacts used. If it says a criterion lacks coverage, it must point to which documents, tests, or records were compared. If it suggests a risk, it should separate inferred risk from evidenced risk.

With artifact references, the conversation stops depending on a generic question like “is the quality good?” and starts addressing protected behaviors, existing verifications, and operational signals that have not yet returned to the system.

In products with agents or features that perform actions, this caution is even more relevant. Anthropic, discussing agent evaluations, distinguishes the agent's execution trajectory from the actual result in the environment. A message saying the task finished is not enough to prove the expected result occurred. Evaluations need inputs, success criteria, and checkers, possibly requiring multiple attempts.

Even when the feature is not an agent, the principle is useful: execution evidence is not always evidence of effect.

Separate Specification, Verification, and Operation Gaps

Not every gap has the same cause. Treating everything as “missing test” impoverishes the review.

A specification gap appears when the acceptance criterion does not describe relevant behavior. The team often discovers late that everyone agreed on delivery because each person imagined a different version of the expected result.

A verification gap appears when the behavior is described but there is no test, evaluation, manual review, or checker covering it traceably. Here, the team knows what it wanted but does not know if it protected it.

An operation gap appears when a real signal does not return to the quality system. Support corrects manually, users repeat the same question, an operational team creates a parallel spreadsheet, or a recurring exception becomes routine. None of this changes criteria, tests, or monitoring.

There is also a proof gap, common in AI features. The system reports that it completed a task or produces a convincing response, but the team does not verify if the expected result occurred in the environment or if the response is appropriate for use.

These categories help decide the next action. Specification gap calls for product conversation. Verification gap calls for test, evaluation, checker, or manual review. Operation gap calls for feedback mechanism. Proof gap calls for evidence of result, not just message inspection.

This separation also reduces waste. Not every gap should become test automation. Some require exploratory testing, usability adjustment, longer observation, or acceptance criterion change.

Decision Matrix to Find Gaps Between Acceptance, Testing, and Operation

The matrix below is a practical proposal for quality management with AI. It does not depend on a specific tool. AI can support cross-reading artifacts, but final classification should remain with the responsible team.

For each analyzed item, assign a simple score to the evidence:

0: no traceable evidence.
1: partial, ambiguous, or outdated evidence.
2: clear, linked, and reviewable evidence.

Then classify the gap.

Does every relevant acceptance criterion have at least one form of verification?

Mark as verification gap when a product-defined behavior does not appear in automated test, manual test, evaluation, or operational checker.

The concrete question is: “if this behavior breaks, how will we know?”. If the answer depends on someone noticing by chance, the evidence is weak.

Does every relevant test point to an explicit acceptance criterion or risk?

Mark as test without clear link when the team maintains tests that do not protect a product decision, business rule, or known operational risk.

This does not mean automatically deleting tests. It means asking for context. A test may protect a historical regression, a sensitive integration, or a rare exception. But this reason must be visible.

Do operational signals return to acceptance or testing?

Mark as operation gap when manual corrections, recurring complaints, exceptions, or incidents appear in use but do not change criteria, tests, or checkers.

Absence of complaints does not prove quality. It may mean low exposure, poor feedback channel, low usage, or instrumentation failure. Therefore, operational evidence must be interpreted carefully.

Does the team distinguish success message from actual result?

In AI features, verify if the system only reports task completion or if there is evidence the expected result occurred in the environment.

This question applies to agents, recommendations, classifications, summaries, assisted searches, and automated flows. A confident answer may be inadequate. An apparently correct execution may not solve the user’s problem.

Does each gap have an owner and next decision?

Classify the action as review acceptance, create test, adjust checker, observe operation, or temporarily accept risk with justification.

The prioritization criterion is simple: items scoring 0 in critical criteria or recurring operational signals should enter the conversation first. They indicate the team does not know if it is protecting the behavior it claims to protect.

Fictional Example: An Intelligent Search That Passes Tests but Fails in Use

Consider a fictional example. A company creates an AI-assisted search to help users find information in an internal product database. The acceptance criterion states the search must return a relevant answer, cite the consulted source, and allow question refinement.

Automated tests verify if the search responds, if the interface loads, if there is a source citation, and if the user can ask a new question. In a controlled environment, all pass.

After activation for some users, operation begins recording qualitative signals. Some people copy the answer to another channel and then make manual corrections. Others ask again because the answer seems complete but omits an exception. Support starts receiving questions about “correct but incomplete” answers. These effects are hypotheses to measure in this fictional scenario, not actual results.

Applying the matrix with AI support, the team finds three gaps.

The first is specification. The criterion “relevant answer” is too broad. It does not say if the answer should express uncertainty, cover known exceptions, or warn when the consulted database is insufficient.

The second is verification. The test confirms a citation exists but does not verify if the citation supports the answer. Presence of a source does not prove adequacy.

The third is operation. Manual corrections made by users appear in usage records but do not return to evaluation cases. The team sees operational effort but has not yet transformed this signal into criterion, test, or checker.

In this case, AI does not decide if the search is ready or not. It helps build the contrast between acceptance, testing, and operation. The decision may be to revise the acceptance criterion, create evaluation cases with incomplete answers, include a source support checker, and observe exposure before expanding use.

This logic also connects to release management. The Google SRE chapter on canary releases addresses evaluating a change on a portion of traffic before expanding exposure and distinguishes deploying code from activating features. For AI features, separating technical delivery from gradual activation helps observe operational signals before expanding exposure.

If the team is still designing the adoption plan, it is worth connecting this decision to the AI roadmap. In this plan, quality must enter before the final choice of use cases, with owners, expected evidence, and review criteria.

What Changes in the Team’s Quality Ritual

Reviewing the quality system with AI should change the ritual, not just generate a report.

In a planning review, critical acceptance criteria should be marked with the expected form of verification. This does not require automating everything. It requires clarifying how the team will know a behavior was protected.

In a test review, cases without links to criteria, risk, or previous incidents should be discussed. Some will be kept. Others adjusted. Others possibly removed. The point is not to reduce the suite for vanity but to maintain a clear relationship between testing effort and protected risk.

In an operational review, recurring signals need an owner. A repeatedly appearing question may indicate interface, documentation, acceptance criterion, AI evaluation, or product expectation problems. If no one is responsible for turning this signal into system learning, operation becomes a permanent buffer for low quality.

It is also necessary to recognize limits. AI can compare artifacts, summarize evidence, and suggest gaps. It should not invent product intent, risk severity, or business impact without provided material. The matrix organizes the conversation but does not replace judgment from product, engineering, quality, security, support, or leadership.

Caution with metrics also applies here. The METR update from February 2026 considers its new data an unreliable signal of AI’s current effect on productivity and points out difficulties measuring time with competing agents, participant selection, and tasks. This source does not decide the merit of quality review but reminds that measuring AI effects requires care with context, task, and method.

For leadership organizing broader strategy, quality review should align with the business-connected artificial intelligence strategy. An intelligent feature only gains space with operational trust, and trust does not arise from discourse. It arises from evidence, responsibility, and the ability to correct the system.

At the review’s close, each gap between acceptance, testing, and operation must have available evidence, discussed risk, and defined owner. Recurring operational signals should not remain without someone responsible for deciding whether to change criterion, test, checker, monitoring, or exposure.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss Application in Your Company

Conversation about the software company context

Content from dooop. Registration allows relating this topic to the reader’s journey and tracking interest in the theme.