dooopSoftware · Quality · 10 min

Regression Testing in AI: Preserve Behaviors

Learn how to build a regression base for AI focused on accepted behaviors, clear verifiers and gradual exposure of changes.

Published on September 6, 2026

CORE THESIS

Regression in AI rarely appears as a syntax error. It arises when a change erases an accepted commitment.

Good testing does not freeze phrases. It protects constraints, formats, refusals and actions that support the product.

AI regression tests serve to answer a simple and difficult question: after changing prompt, model, context, tool or business rule, does the product still preserve the behaviors that were already accepted?

The goal is not to get the same phrase. It is to prevent a technically correct change from erasing product commitments such as action limits, output formats, necessary refusals, acceptance criteria and user expectations.

When a Change in AI Becomes a Regression Risk

An AI regression occurs when a feature that already met an accepted behavior stops meeting it after a change. This change may seem small: shortening a prompt, swapping a third-party model, changing the context policy, adjusting a business rule or connecting a new tool to the flow.

The problem is that AI’s linguistic variation can hide the loss. The response remains fluent. The deploy passes. Technical tests of API, authentication, latency and availability do not break. Yet something the product promised to do stops happening.

Therefore, AI regression tests should not try to freeze the surface of the response. They must preserve the accepted behavior. If a feature summarized support tickets, it is not enough to check if it still writes a plausible summary. It is necessary to verify if it maintains the commitments that made that summary usable in the real flow.

This difference changes the conversation between engineering, quality and product. The question stops being “Did the AI respond well?” and becomes “Which product commitment needed to remain intact after the change?”.

What Should Be Preserved: Behavior, Not Exact Phrase

In AI features, requiring identical text usually generates fragile tests. Two responses can use different words and fulfill the same role. They can also use similar words and break a relevant limit.

The focus of AI regression tests should be on preservation categories. Some are simple to observe:

Intention met: the response solves the main request the case represents.
Constraint respected: the AI does not execute, recommend or assert something that was prohibited or conditioned.
Useful format: the output maintains fields, structure, order, level of detail or language necessary for the flow.
Correct action: when the feature triggers a tool, the expected final state is produced.
Appropriate refusal: the AI stops or asks for confirmation when the case requires a limit.
Preserved autonomy: the new version does not start deciding alone something that previously depended on human confirmation.

This list does not replace acceptance criteria. It helps translate them into a regression base. If the product has not yet defined acceptance criteria for AI, regression will be too subjective. In that case, the first step is to separate what is a writing preference from what is an operational commitment.

In regression, this separation prevents a writing preference from blocking change and an operational break from passing as simple linguistic variation.

How to Build a Minimal AI Regression Base

An AI regression base is a revisable set of cases that represent already accepted behaviors and that should not disappear silently. It does not need to start large. It needs to start relevant.

Selection should prioritize cases that protect product commitments:

Frequent cases, because they affect the common user experience.
High-risk cases, because a failure changes the decision, confidence or operational flow.
Known exceptions, because they tend to be ignored when the team optimizes for the average case.
Recent fixes, because corrected defects tend to return when prompts, models or tools change.
Examples that represent already approved acceptance criteria, because they reduce subjective debate when deciding.

DORA recommends testing throughout development, combining automation and manual activities such as exploration and usability, as well as maintaining and reviewing test suites. This guidance is especially useful in AI: a regression base should not become dead weight. When the product changes, cases also need to be reviewed.

A good minimal base has cases with input, expected behavior, success criterion and verification method. The expected behavior should not be “respond appropriately.” That is too broad. Better: “separate summary from recommendation,” “do not promise automatic action,” “ask for confirmation before triggering the tool,” “include required fields” or “refuse when context is missing.”

Decision Checklist to Test Regressions in AI Features

Before changing prompt, model or tool, the team can go through these points to decide if the new version preserves accepted behaviors.

Behavior preserved: does the new version maintain the main intention already accepted for this case? If the intention is no longer met, treat as blocking regression, even if the response seems well written.
Constraint preserved: does the new version continue respecting defined limits, such as not executing a certain action, not asserting something without basis or asking for confirmation before proceeding? If an explicit constraint is broken, block or escalate for product decision and risk.
Format preserved: does the output remain useful for the flow where it will be consumed, including structure, required fields, level of detail and expected language? If variation prevents operational use, record as regression even if content is semantically close.
Verifiable result: is there a clear verifier to confirm the task was completed in the environment, not just an AI message saying it finished? If the test depends only on the agent’s statement, this case should not approve regression alone.
Previous fix protected: did a previously fixed problem reappear after the change? If yes, the case should enter the permanent regression base or have its permanence explicitly reviewed.
Acceptable variation: does the observed difference change the product commitment or only the form of expression? If it changes the commitment, it is regression. If it changes only the form, it can be accepted when it does not harm use.
Controlled exposure: can the change be activated for a smaller portion before expanding use? If potential impact is high and separation between delivering and activating is possible, prefer gradual exposure.

The value of the checklist lies less in standardizing responses and more in forcing an explicit decision. A break can be accepted. But when that happens, it must be assumed as a product decision, not hidden as natural AI variation.

Fictional Example: Support Ticket Summary

Fictional example: an AI feature summarizes support tickets and suggests the next action to the analyst.

The accepted behavior is this: when the ticket involves subscription cancellation, the AI must summarize the request, indicate pending documents and not promise automatic refund. The team swaps the model to reduce cost and adjusts the prompt for shorter responses.

The regression test does not need to require the same paragraph as the previous version. It needs to run old cancellation cases and verify three commitments:

The AI still separates summary from recommendation.
The AI maintains the list of pending documents when they appear in the history.
The AI avoids promising automatic refund.

If the new response is shorter but preserves these three commitments, the change can proceed to a controlled stage. If it omits documents or promises refund, the version regressed. The hypothesis of cost reduction or improved conciseness alone does not compensate for the loss of accepted behavior. This effect must be measured in the appropriate environment, not presumed as a result.

This example also shows a leadership point: not every local improvement improves the product. A shorter response may be desirable, but not if it removes information that supported the analyst’s decision.

When the Trajectory Changes Without Breaking Accepted Behavior

In features with agents, regression can be even less visible. The agent may narrate coherent steps, say it checked data, declare it opened a request or affirm it completed the task. None of this proves the result.

Anthropic distinguishes the agent’s execution trajectory from the actual result in the environment. A message saying the task ended is not enough to prove the result. Evaluations need inputs, success criteria and verifiers, and may require multiple attempts.

For regression, this distinction is decisive. If the feature should update a field, query a database, create a task or maintain a lock, the test needs to verify the final state or the output consumed by the user. The apparent reasoning chain is not sufficient criteria.

This also applies to features that are not full agents. A feature may explain very well why it decided something and still fail the expected format, forget a constraint or ignore relevant context. Trust does not come from fluency. It comes from verifiers appropriate to the product commitment.

When to Accept a Break and When to Treat as a Defect

Not every difference between versions is regression. The challenge is to classify the change without turning personal taste into a block, nor treating relevant break as detail.

A practical classification can separate five situations:

Blocking regression: an accepted behavior stops happening, an explicit constraint is broken or an improper action becomes allowed.
Acceptable regression with explicit decision: the product decides to abandon or change a previous commitment, records the reason and adjusts criteria, communication and tests.
Improvement with side effect: the new version improves one aspect but harms another. The decision depends on the relative weight of each commitment.
Irrelevant variation: changes wording, order of explanation or style without harming use.
Case requiring human judgment: involves ambiguity, user experience, reputational risk or a business decision that should not be automated without review.

In the February 2026 update, METR considers new data an unreliable signal of AI’s current effect on productivity and points out difficulties measuring time with competing agents. This point does not prove quality or lack of quality in AI features. In the context of this article, it serves as a methodological alert: indirect metrics such as time or productivity do not replace verification of the behavior the product promised to deliver.

If the regression discussion becomes only “the team got faster” or “the response seems better,” the organization is measuring around the problem. Regression testing needs to return to the accepted commitment.

How to Use Gradual Rollout to Reduce Regression Impact

Even with a good regression base, tests do not eliminate unexpected behavior in AI. They reduce known losses and make breaks more visible. Therefore, the way to activate the change is also part of the quality strategy.

The canarying chapter of Google SRE addresses evaluating a change on a portion of traffic before expanding exposure. It also distinguishes delivering code from activating features, using configurations to separate these decisions.

Applied to AI, this allows separating technical delivery from change activation when this separation is possible. When possible, the change of prompt, model, tool or rule can be activated for a smaller group, accompanied by verifiers, review of critical cases and support signals.

This separation helps avoid a common trap: treating successful deploy as authorization of behavior. Delivering is not the same as expanding. For AI features, this difference can be the boundary between learning safely and discovering a regression too late.

For the next change, the team needs to leave with three definitions: which accepted behaviors enter the revisable base, which verifiers prove preservation and which breaks can only proceed if assumed as product decisions.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss Application in Your Company

Discussion about the software company context

Content from dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.