dooopSoftware · Quality · 11 min
How to Investigate Regression After Changing an AI Model
Quality drops after model changes require comparing complete versions, isolating variables, and deciding between adjustment, containment, or rollback.
Published on September 6, 2026
CORE THESIS
Changing the model alone does not explain regression. The investigation must treat the version as a system.
Compare the previous and new versions in layers. Isolate model, prompt, context, tool, and criteria before deciding.
When quality declines immediately after switching an AI model, the worst response is to instinctively blame the model. The drop may be due to the model, but it can also stem from the prompt, context, tools, routing, evaluation criteria, or how the change was activated.
Investigating regression after an AI model change requires comparing versions as systems and isolating one variable at a time before deciding to adjust, contain, or revert.
Model Change as a System Change
The sequence seems convincing: the team changes the model, observes worse responses, and concludes the new model reduced quality. This hypothesis is legitimate but not sufficient evidence.
In practice, a product version with AI can change more than just the model. The investigation must consider new prompts, new system instructions, temperature adjustments, context window changes, different search engines, new routing rules, or revised evaluation criteria. Sometimes the interface also changes, leading users to input different queries.
Therefore, the first leadership decision is not to pick another model. It is to formulate the right investigation: what exactly changed between the previous and new version?
This distinction avoids two costly errors. The first is discarding a model that could perform well with a different orchestration. The second is keeping the change and trying to compensate for the drop with superficial fixes without understanding the likely cause.
In the investigation, the final text is only one layer of evidence. The comparison must also consider input, context, instruction, execution, verification, and actual use. Ignoring this set leads the team to discuss impressions, not regression.
Rebuild the Previous Version Before Comparing
Every investigation starts with a baseline. Baseline here means a reproducible snapshot of the version considered acceptable before the change.
This snapshot should record at least:
- model used in the previous version;
- system prompt and additional instructions;
- generation parameters available to the team;
- context sources, consulted databases, and retrieval rules;
- tools or actions the AI could trigger;
- security policies, filters, and restrictions;
- success criteria used to evaluate responses;
- set of reference cases.
Without this reconstruction, the team compares the new version against collective memory. Collective memory tends to be generous with the past and impatient with the present.
DORA recommends testing throughout development, combining automation with manual activities like exploration and usability. It also recommends maintaining and reviewing test suites rather than treating quality as a post-development step. Applied to AI, this means the comparison baseline must exist before the crisis, not only after complaints.
When the previous version cannot be reconstructed, the team should declare uncertainty. Investigation is still possible but less precise. In this case, it is worth reconstructing an approximate reference with available records, usage examples, support cases, and human evaluations. The point is not to pretend precision where there is a gap.
A good sign of maturity is not immediate certainty but knowing what can be stated, what is hypothesis, and what still needs testing.
Compare Results, Trajectory, and Environmental Effect
After rebuilding the baseline, compare three different layers.
The first is the produced result. Was the response correct, complete, useful, safe, and appropriate to the expected format? This is the most visible layer but should not be the only one.
The second is the trajectory. Trajectory is the path the AI used to reach the result: which sources it consulted, which tool it triggered, which steps it executed, which instructions it prioritized, and where it abandoned the flow. Two similar responses can have very different trajectories. Two similar trajectories can produce different final results.
The third is the effect on the environment. In an agent or action flow, it is not enough for the AI to say it completed a task. It is necessary to verify whether the expected effect occurred in the product, system, or process. Anthropic distinguishes the agent’s execution trajectory from the effective result in the environment and notes that a completion message alone does not prove the task was performed.
This separation changes the quality conversation. A response may seem elegant yet consult the wrong source. An agent may declare success but not have completed the action. A new version may write better but decide worse.
Therefore, each evaluation case must have inputs, success criteria, and verifiers. A verifier is any mechanism that confirms the expected result: an automatic check, a comparison with the current procedure, a structured human review, or confirmation that an action actually changed the correct state.
In cases requiring judgment, ambiguity, operational risk, or relevant user impact, the team must define verifiers compatible with the risk and success criteria.
Isolate One Change at a Time in the Investigation Matrix
The core of the investigation is simple to explain and laborious to execute: change one variable per round.
If the new version combines a new model, new prompt, and revised context base, the regression may be in any of these points or their combination. Comparing only old versus new version shows something changed but not what.
An investigation matrix should create controlled combinations, for example:
- old model with old prompt;
- new model with old prompt;
- old model with new prompt;
- new model with new prompt;
- same model and prompt but old or new context;
- same configuration but old or new tool;
- same results evaluated by old and new criteria.
The order depends on the main suspicion. If complaints involve vague answers, start with prompt and context. If incomplete actions, look at tools, permissions, and verifiers. If failures occur in cases that previously passed, also review evaluation criteria because the standard may have changed.
This matrix does not identify the cause with certainty. It reduces ambiguity. An honest conclusion may be: the likely cause lies in the combination of new model and new instruction, or between new context and old verifier.
This is already progress. A well-defined likely cause guides better decisions than a strong opinion about the model’s name.
Regression Isolation Checklist After Model Change
Use this checklist to conduct the investigation without turning the analysis into an abstract debate.
- Can the previous version be reproduced? Confirm if old model, prompt, parameters, context, tools, and criteria are recorded. Without this, comparison becomes team memory.
- Is the sample of cases the same in both versions? Run old and new versions on the same cases, with the same input data and success criteria.
- Is the investigated change isolated? Change only one component per round: model, prompt, context, tool, policy, or verifier. If two variables change together, the cause is ambiguous.
- Was the result verified in the environment, not just in the response? In agents or action flows, confirm the task produced the expected effect, not just that the AI declared completion.
- Does regression appear in critical cases or only in style preferences? Separate failures affecting safety, decision, operation, or trust from acceptable differences in wording, format, or tone.
- Is there a stop criterion for the new version? Define before scaling which signals require containment or rollback, such as critical failures, recurring manual corrections, or failure in reference cases.
- Is the final decision linked to the likely cause? Avoid changing the model again if evidence points to prompt, context base, tool, routing, or evaluation criteria.
This checklist connects to a broader quality discipline. If the organization lacks reference cases, acceptance criteria, and minimum evaluations, it is worth reviewing the foundational guide on software quality with AI and deepening the creation of an AI application evaluation. Regression investigation is much more fragile when it starts only after an incident.
Use Gradual Exposure to Test Hypotheses Without Increasing Damage
After formulating the likely cause, resist the temptation to release the fix to everyone at once. In AI, gradual exposure is a way to learn with limited damage.
The canary chapter of Google SRE covers evaluating a change on a portion of traffic before expanding exposure. It also distinguishes releasing code from activating features and discusses using configurations to separate these decisions.
This distinction is very useful in AI products. The team may have the code ready but keep the new configuration disabled for most users. It can activate it only for a controlled group, for less sensitive question types, or for scenarios where rollback is simple.
Before scaling, define stop criteria. They do not need to be numerous but must be clear. Examples: failure in critical reference cases, recurring manual corrections, severe divergence from current procedure, or verifier indicating actions were not completed.
Aggregate metrics help but can hide rare and relevant problems. An acceptable average can coexist with severe failures in specific cases. Therefore, combine usage signals with reviewed samples, reference cases, and failure-type analysis.
If the organization is structuring broader AI adoption, this type of control should appear in the AI roadmap, not only as an emergency reaction.
Decide Between Adjustment, Containment, or Rollback
The investigation must end with an operational decision. There are three main paths.
Adjust when the likely cause is located and the fix can be safely tested. Example: regression appears when the new model receives an instruction too short to consult the procedure before answering. The response is not to abandon the model but to revise the instruction, test the combination, and expose gradually.
Contain when regression concentrates in specific scenarios, especially if critical. Containment may involve routing rules, blocking certain request types, requiring human review, limiting autonomy, or temporarily returning to the previous flow for a usage category. In agents, containment must consider environmental effect, not just textual response quality.
Rollback when the cause remains uncertain and impact is relevant. Rollback is not a technical failure. It is a trust preservation decision while the team learns. The organization can continue investigating outside main exposure, with controlled cases and clearer criteria.
The worst decision is insisting on a new version because it seems more modern, even when evidence shows quality loss in important situations. Trust does not rest on novelty preference. It rests on verification capability.
Fictional Example: Internal Assistant Worsens After Model Change
Imagine a fictional example: a company uses an internal assistant to answer operational questions about support procedures. The team changes the AI model and, in the first weeks, users report longer, more confident, and less procedure-adherent answers.
Initial suspicion falls on the new model. However, the team reconstructs the previous version and notices the change included three variables: new model, revised prompt, and new instruction to synthesize answers in more natural language.
The investigation uses the same question sample in both versions. In each case, evaluators check if the response cites the correct procedure, preserves relevant exceptions, and avoids recommending actions outside the current rule. When an action is suggested, the team verifies if it matches the procedure rather than accepting the AI’s statement as evidence.
Isolating variables, a hypothesis emerges: the new model responds adequately with the old prompt, and the new prompt works acceptably with the old model. The drop appears strongest in the combination of new model and synthesis instruction. The formulation asks for natural, short answers but does not require preserving procedure exceptions before summarizing.
This example does not prove a universal result. It shows the type of reasoning needed. The likely cause was not simply “new model worse.” It was the interaction between new model, synthesis instruction, and insufficient criteria to verify exceptions.
The prudent decision would be to contain the new configuration, revise the prompt, include reference cases with operational exceptions, and only expand exposure after a new comparison. If the company already uses an AI strategy connected to business, this decision should also consider impact on user trust, manual correction cost, and operational risk.
The next decision is not to change the model again. It is to compare the previous and new versions in a controlled matrix, isolate one change at a time, and decide based on the likely cause between adjustment, containment, or rollback.
If you want to discuss this decision in your company’s context, talk to dooop.
Further Reading
Sources
To Continue This Reading
NEXT DECISION
Discuss Application in Your Company
Conversation about the software company context
Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.