dooopPillar page · 16 min

Software Quality with AI Requires Dual Evaluation

Separating software testing and AI evaluation helps teams validate functionality, acceptable behavior, and responsibility before exposure.

Published on September 6, 2026

MAIN THESIS

Passing tests alone is not enough. AI requires evidence of the effect produced.

Quality changes when behavior is variable. Decisions involve risk, evaluation, and authority.

In software quality with artificial intelligence, it is important to verify both predictable components and the suitability of responses and actions for the expected use. In this guide, we call this second front AI evaluation. Both are part of software quality: a feature may compile and pass integration tests but still produce an inadequate response, an incomplete action, or a decision outside the limits accepted by the organization.

How to Combine Software Testing and AI Evaluation

Component, integration, and flow tests continue to cover a decisive part of the product: business rules, integrations, permissions, predictable flows, data persistence, observability, and error handling. If a screen does not load, if an API returns incorrect data, if a user without permission accesses restricted information, or if an error is not logged, the problem is not "intelligent behavior." It is software failure.

AI evaluation begins when the question stops being just "did the system execute the rule?" and becomes "is the response, recommendation, or action produced acceptable for this context?" This distinction seems simple but changes the design of the quality process.

Imagine a fictional example: a company uses an internal assistant to summarize customer requests before the daily operations meeting. The technical flow works. The user authenticates, the assistant accesses authorized messages, generates the summary, records the history, and logs events. Integration tests pass.

Even so, the summary omits a relevant exception: a customer had conditioned the continuation of a delivery on receiving a response that day. The assistant produced a clear, grammatically correct, and apparently useful text. From a technical perspective, nothing broke. From an operational decision perspective, the response may be insufficient.

The two fronts complement each other. Component and integration tests verify known contracts; evaluation cases examine the adequacy of AI outputs for the expected use.

In AI products, trust does not arise from a single piece of evidence. It depends on layers. Automated tests help detect failures in the behaviors they cover. Behavior evaluations, which can also be automated, examine inconsistent responses, relevant omissions, and incomplete actions. Responsibility defines who accepts, limits, or interrupts exposure when uncertainty arises.

This separation also avoids a common management error: turning a green test suite into broad authorization for use. Passing tests is not synonymous with approved behavior. It is a necessary but partial piece of evidence.

For organizations structuring a business-connected artificial intelligence strategy, this distinction helps move the discussion from generic enthusiasm to a more useful question: what kind of evidence do we need before exposing this capability to people, customers, or operations?

Three Questions to Decide Exposure of an AI Feature

Before discussing tools, it is worth classifying the risk of the feature. Not as a bureaucratic exercise, but as a way to decide how much evaluation is reasonable before exposure.

The first question is: does the AI only suggest or does it execute something?

A feature that rewrites text for human review has a different risk profile than an agent that changes a record, opens a ticket, sends a message, or triggers an operational flow. The closer to execution, the higher the requirement for verifying the result in the environment. It is not enough to evaluate if the response seems good. It is necessary to confirm if the expected effect occurred, if it occurred in the right place, and if it did not exceed defined limits.

The second question is: is the error reversible or does it generate relevant operational consequences?

Not all errors have the same weight. A bad title suggestion can be discarded. A wrong classification that moves a request to the wrong queue can delay service, confuse responsible parties, or require rework. The evaluation should reflect this difference. Rare cases with high impact cannot be lost within a comfortable average.

The third question is: does validation depend on an objective rule, human judgment, or observation in controlled production?

Some behaviors can be verified by rule. A ticket was created or not. The category is among the allowed or not. The record contains the minimum fields or not. Other behaviors require human judgment, such as evaluating whether a summary preserves enough nuances for a decision. There are also situations where the team needs to observe signals in controlled exposure before expanding use.

These three questions create an initial screening:

If the AI only informs, evaluation can prioritize accuracy, completeness, and clarity.
If the AI recommends, evaluation must include acceptance criteria for the recommendation and confidence limits.
If the AI prepares an action, the team must verify if the proposed action respects rules, permissions, and context.
If the AI executes an action, evaluation must measure the real effect, anticipate containment, and define authority for interruption.

This screening does not replace a deeper maturity analysis. It is a first cut to avoid treating very different features with the same approval ritual. In an organization still diagnosing its starting point, this type of criterion aligns directly with a reading of AI maturity: capability is not just having models available but knowing where uncertainty enters the process.

Code Tests Remain the First Barrier of Trust

The arrival of AI does not make automated tests obsolete. On the contrary: when intelligent behavior adds variability, the deterministic part of the system needs to be even more reliable.

Unit tests continue to verify isolated functions. Integration tests continue to protect communication between services. API contract tests help identify incompatible changes between systems. Authentication and permission validations continue to prevent unauthorized access. Logs and error handling continue to allow investigation when something fails.

None of this alone evaluates whether an AI response was good. But without this foundation, the team tries to interpret intelligent behavior on unstable ground.

DORA recommends testing throughout development, combining automation and manual activities such as exploration and usability. It also recommends maintaining and reviewing test suites rather than treating quality as a post-development step. This point is especially relevant when a feature begins to depend on prompts, models, external tools, knowledge bases, or new decision paths.

When the feature changes, the test suite also needs to be reviewed. Not because all old tests lose value, but because part of the risk has shifted.

In a traditional feature, it might be enough to check if the form validated fields, saved data, and displayed confirmation. In an AI feature, in addition, it may be necessary to test if the system sends the correct context to the model, removes data that should not be sent, records the version of the configuration used, handles provider unavailability, and prevents actions outside the allowed scope.

These are software tests, not semantic evaluations. They verify if the system is prepared to operate the intelligent capability in a controlled manner.

There is a practical consequence for leadership: do not use AI evaluation to compensate for fragile engineering. If authentication fails, if integration is unstable, or if logs do not allow tracing a decision, the problem must be resolved as software quality. Behavior evaluation comes afterward to answer a different question.

It is also useful to separate productivity from quality. The February 2026 update of METR considers its new data an unreliable signal of AI's current effect on productivity, pointing out participant and task selection and difficulties measuring time with competing agents. This source does not conclude on product quality. The operational caution here is that development speed, when present, should not be used as proof that the delivered behavior is acceptable.

Behavior Evaluations Must Measure Result, Not Intention

Evaluating AI is not asking if the response seems convincing. Nor is it accepting the agent's final message as proof of completion. In intelligent features, the appearance of competence can hide a simple failure: the expected result did not happen.

Anthropic distinguishes the agent's execution trajectory from the effective result in the environment. A message saying the task is finished is not enough to prove the result. Evaluation uses inputs, success criteria, and checkers, and may require multiple attempts.

This distinction is very useful for product and quality teams. An AI evaluation needs at least three elements:

an input or initial situation;
a success criterion;
a manual or automated checker.

Returning to a fictional example, with no real case behind it. An internal agent receives the task of opening a support ticket from a conversation with a customer. At the end, it responds: "Ticket opened successfully." This phrase is not sufficient evidence.

A stronger evaluation would verify if the ticket actually exists in the system, if it was associated with the correct customer, if it received the appropriate category, and if it contains minimum data for service. Depending on the risk, it could also verify if the agent did not include inappropriate information, did not duplicate requests, and did not forward the case to an incompatible queue.

Note that the evaluation does not need to be sophisticated to be useful. In many cases, a simple checker is better than a subjective and diffuse review. The question is not "was the AI intelligent?" The question is "can the effect we expected be observed by a defined criterion?"

When behavior involves language, the checker can combine objective criteria and human review. For example: does the summary mention all mandatory items? Does it avoid inventing commitments? Does it preserve signaled exceptions? Is it clear enough for the next person to act? Some of these questions can be evaluated automatically. Others may require human sampling, especially when the cost of an omission is relevant.

The limit also needs to be clear. Evaluations do not eliminate AI error. They reduce operational blindness. They help find failure patterns, compare changes, block fragile exposures, and decide when human review should remain in the flow.

This is an important difference: human judgment should not appear only as late correction after the technology erred. It can be part of the system design. In some features, AI prepares a suggestion and the person decides. In others, AI executes only reversible actions. In others, autonomous action simply should not be allowed without more robust evidence.

For those building an AI roadmap, this point helps prioritize. The cost of evaluating behavior must enter the sequencing decision. An apparently simple opportunity may require a larger evaluation operation than the team can sustain at that moment.

Responsibility Defines Who Can Accept Ambiguity

Responsibility in AI is not about finding culprits after failure. It is about designing beforehand who decides, based on what evidence, how far the feature can go, and when it must stop.

Every feature with intelligent behavior carries some degree of ambiguity. The management question is: who has the authority to accept this ambiguity?

There are at least four responsibilities that should not be mixed:

Quality criterion: defining what counts as an acceptable response, correct action, intolerable omission, and relevant exception. This responsibility usually requires proximity to product, operation, and user, not just engineering.
Exposure approval: deciding if the available evidence authorizes advancing, limiting, or blocking. Passing tests and evaluations should not automatically activate the feature for everyone.
Exception review: analyzing, classifying, and deciding if a difficult case becomes a new criterion, product adjustment, usage limitation, or conscious acceptance.
Interruption: defining who can reduce exposure, disable a configuration, or stop a capability when signals indicate inappropriate behavior. If the answer is "everyone monitors," probably no one decides.

A practical way to organize this conversation is to separate behaviors into three groups.

Behaviors accepted by metric: can advance when objective criteria and monitored signals remain within limits defined by the team.
Behaviors requiring human review: can be used as support, but the final decision needs a person at specific points in the flow.
Behaviors blocking release: should not be exposed while the team lacks control, evidence, or sufficient containment.

The value of this separation is not to create heavy governance. It is to prevent risk decisions from being made by technical inertia. A feature should not advance just because "it is ready." Ready for deploy does not mean ready for impact.

This reasoning also avoids a cultural trap: treating responsibility as distrust of the team. In practice, it is the opposite. When criteria and authorities are clear, the team can experiment more honestly because it knows which signals matter and which limits cannot be crossed.

In leadership action, this is a natural extension of the theme of leadership and augmented human: technology expands capacity when the process clarifies where the machine suggests, where it executes, and where the person must sustain the decision.

Launching with AI Requires Separating Code Deployment and Behavior Activation

Deploying code to production does not need to mean releasing an intelligent capability to everyone. This separation is one of the most useful decisions to reduce risk without paralyzing delivery.

The Google SRE chapter on gradual releases addresses evaluating a change on a portion of traffic before expanding exposure. It also distinguishes making code available from activating features and discusses using configurations to separate these decisions.

Applied to AI, this means a team can publish the technical infrastructure, keep the feature disabled for most users, activate it for a controlled group, and observe signals before expanding. These signals may include technical failures, response inconsistencies, need for human intervention, flow abandonment, manual corrections, or occurrences of expected exceptions.

This is not a promise that a gradual release will find all problems. It will not. What it does is create a controlled way to observe real behavior without turning the first exposure into an all-or-nothing bet.

The separation between deploy and activation also changes the conversation with leadership. Instead of approving "the AI release," the organization approves exposure stages:

code available, feature disabled;
internal activation or for a restricted group;
activation for a controlled usage segment;
expansion conditioned on defined signals;
interruption or exposure reduction when a limit is exceeded.

These stages need to be technically feasible. If the only way to contain risk is to remove code hastily, the release strategy is fragile. Configuration, user segmentation, usage limits, and controlled shutdown are not operational details. They are part of software quality with AI.

Here too, it is worth separating observation from absolute validation. Controlled production helps see behavior in context but does not replace software tests or prior evaluations. It complements previous layers with signals from the real environment.

Checklist for Separating Testing, Evaluation, and Responsibility

The checklist below is a practical proposal to guide exposure decisions. It is not a certification, does not eliminate judgment, and does not turn probabilistic behavior into a guarantee. Its usefulness is to force the team to separate software correctness, behavior evaluation, and operational responsibility.

Has the feature risk been classified by action type?

Mark if the AI only informs, recommends, prepares an action, or executes an action. The closer to execution, the higher the requirement for evaluation and exposure control.

If the team cannot say what the AI can do, the feature should not advance to broad exposure. Before evaluating quality, it is necessary to delimit capability.

Do software tests cover the system's predictable functioning?

Check authentication, permissions, integration, persistence, error handling, observability, and known flows before discussing if the AI response is good.

If the system fails deterministic rules, it makes no sense to use AI evaluation to compensate for an unstable base. Fix engineering before interpreting intelligent behavior.

Does the evaluation measure a verifiable result, not just a plausible response?

Define input, success criterion, and checker. In an agent task, confirm if the expected effect occurred in the environment, not just if the final message seems convincing.

If the only evidence is the AI itself saying it finished, the evaluation is weak. A good evaluation must look at the result, not the system's self-declaration.

Are there cases representing normal use, relevant exceptions, and intolerable failures?

Separate common cases from rare cases with high impact. The goal is not to cover the entire possible universe but to prevent the average from hiding risks the organization does not accept.

If a rare failure can cause significant operational damage, it needs its own treatment. It should not be buried in an aggregate score that seems acceptable.

Is there a responsible party to accept, limit, or interrupt exposure?

Define who approves the stage change, who monitors quality signals, and who has authority to reduce or disable the feature.

If everyone monitors but no one decides, responsibility is poorly designed. AI ambiguity needs an owner in the process, not just good intentions.

Can activation be controlled without a new deploy?

Check if the team can make code available and activate the feature separately, by configuration, user group, or controlled traffic slice.

If the only way to contain risk is to remove code hastily, the release strategy is fragile. Exposure control is part of quality, not an operational accessory.

Quality Decisions Change When Software Starts to Behave

The central question is not whether the organization should trust or distrust AI. The question is what evidence it requires for each type of behavior.

A system can be technically correct and still deliver an inadequate response. An agent can declare success and not have produced the expected effect. A feature can be ready for deploy and still not ready for broad exposure. These distinctions seem small but prevent a false sense of security.

To govern this exposure, the quality process needs to operate on three layers: tests for software correctness, evaluations for intelligent behavior, and responsibility to authorize, limit, or interrupt exposure. When these layers have their own criteria and responsible parties, the organization stops approving AI by appearance of functioning and starts deciding by the level of evidence the risk requires.

The next decision is not to choose another testing tool. It is to define, for each AI capability, what evidence authorizes exposure, what signal requires human review, and what limit interrupts use.

To bring this separation to your product, engineering, or leadership agenda, talk to dooop.

Sources

NEXT DECISION

Discuss application in your company

Conversation about the software company context

Content by dooop. Registration allows relating this topic to the reader's journey and tracking interest in the subject.

Deepen this topic

Calibrate AI evaluations with domain expertsLearn to confront automatic scores with domain judgment, adjusting criteria, weights, evidence, and limits before approving AI. How to evaluate AI with human judgment without turning into opinionHuman AI evaluation requires risk-based samples, observable criteria, and separation between style, utility, and operational failure. How to evaluate AI agents that execute actionsEvaluate AI agents separating result and trajectory, with external checkers to confirm actions, scope, permissions, and effects. How to evaluate inconsistencies in AI responsesNot all variation is defect: compare AI responses by requirements, rules, facts, and practical effects before treating inconsistency as failure. How to evaluate an AI application with clear criteriaEvaluate AI applications by real cases, verifiable criteria, and failure severity before comparing models or expanding exposure. How to classify manual corrections in AILearn to separate style adjustments, lack of context, and real failures in manual corrections made by users in AI features. How to create a baseline before adopting AIRecord tasks, time with context, quality, and completion before the pilot to compare AI gains without confusing speed with rework. How to identify fragile tests generated by AIAI-generated tests may seem solid and still fail to protect rules. See how to review assertions, negative cases, and limits. How to investigate regression after changing AI modelQuality drop after model change requires comparing complete versions, isolating variables, and deciding between adjustment, containment, or rollback. How to read AI evaluation scores without falling for the averageHigh AI evaluation scores do not alone authorize advancement: understand criteria, weights, checkers, and failures the average may hide. How to limit autonomy of AI agentsDefine autonomy by action, impact, and reversibility, with specific permissions, approval when necessary, and objective checkers. How to measure developer productivity with AICompare equivalent tasks, record time until reviewable delivery, review, and rework, and only count gain when acceptance becomes cheaper. How to organize evaluation cases for AIAI evaluation bases require separating cases, labels, and review so scores show real risks, not just comfortable averages. How to document AI failures to avoid recurrenceSeparate event, cause hypothesis, and verifiable action to turn AI failures into operational learning, not dispute or noise. How to review AI-generated code with less riskReview AI-generated pull requests by verifiable intention, evidence proportional to risk, and explicit decision before merge. How to review dependencies suggested by code agentsLibraries suggested by AI must undergo review of necessity, origin, technical fit, tests, and removal plan. How to review quality with AI between acceptance and operationUse AI to cross acceptance criteria, tests, and operation signals, locate gaps by evidence, risk, and responsibility. How to safely test AI-generated codeAI-generated tests help but are not enough: approve code only after verifying behavior, risks, and real effects in the environment. How to test rare exceptions in AI applicationsPrioritize damage, reversal, visibility, and exposure to turn rare scenarios into verifiable tests before releasing AI actions. How to test failures in AI productsTest AI failures by defining triggers, safe behavior, verifiable evidence, user recovery, and limits before production. Acceptance criteria for AI featuresDefine success, blocking failures, and autonomy limits before releasing AI, with checkers for content, process, and environment. Rollback plan for AI featuresDefine observable signals, responsible parties, and technical and operational paths to reduce or stop AI without improvisation when it fails. When an AI feature can go to productionBefore launching AI to real users, gather evidence, operation limits, and a responsible party to advance, pause, or revert. Regression testing in AI: preserve behaviorsLearn to build a regression base for AI focused on accepted behaviors, clear checkers, and gradual exposure of changes.