dooopSoftware · Learning · 12 min

How to Plan Experiments in AI Features

Define hypothesis, success criteria, protection metrics, and segments before exposing an AI feature to users.

Published on September 6, 2026

CORE THESIS

Without a falsifiable hypothesis, an experiment becomes organized opinion. In AI, the design determines what the data can prove.

Predefined criteria prevent local wins with systemic harm. The decision is made before user exposure.

Planning experiments in products with artificial intelligence starts before exposing the user. The team needs to write down which hypothesis will be tested, what evidence counts as success, which signals block progress, and which segments will be analyzed separately.

Without this design, an apparent improvement may hide quality degradation, loss of trust, or operational cost shifted to another area.

The experiment must delimit exactly what will be tested in the AI feature

A common mistake is treating the experiment as a generic AI evaluation. The question becomes too broad: "Does AI help?" This formulation does not guide design, measurement, or decision.

An AI feature combines model behavior, interface, business rules, available context, operational process, and user expectation. If everything changes at once, the team may observe a difference but will struggle to explain what caused it.

The testable unit needs to be smaller. Instead of "testing an intelligent assistant in support," the team can test "changing the initial ticket category suggestion using ticket history context before human triage." This phrase separates the AI behavior from the operational decision to be improved.

This delimitation also avoids confusing demonstration with product. A compelling demo shows the feature can produce a plausible response. A product experiment needs to verify if that response improves a real task, in a real flow, with criteria defined beforehand.

Microsoft describes its experimentation platform ExP as a way to incorporate experimentation into the development cycle, validate hypotheses, measure impact, and iterate products Microsoft Research. The practical lesson for an AI feature is not to copy a platform but to adopt discipline: an experiment starts with an operational hypothesis, not enthusiasm for technical capability.

If your organization is still defining broader AI priorities, it is worth separating this exercise from strategic planning. An AI roadmap helps choose where to invest. The experiment discussed here helps decide if a specific change deserves to advance.

A useful hypothesis must be falsifiable

A product hypothesis is not a vague bet. It links an observable change to an expected outcome, for a defined audience, with quality boundaries. The simplest form is written like this:

A useful formulation is: for a given user group or task type, if we change a certain AI behavior, we expect to observe a certain effect without worsening certain limits.

This phrase forces the team to make decisions before the test. Who is the group? Which AI behavior changes? What effect will be observed? Which limits cannot worsen?

A weak hypothesis would be: "using AI will improve support." It cannot be clearly rejected because any favorable signal can be reinterpreted as proof. A more useful hypothesis would be: "for configuration tickets, if AI suggests an initial category before human triage, we expect to reduce time to first classification without increasing subsequent reclassifications or complaints about incorrect routing."

Note that the hypothesis does not promise a result. It defines what will be observed. It also does not assume user feedback automatically retrains a model. Feedback can generate evaluation cases, context review, interface adjustment, process change, or, in some products, model revision. These are different decisions.

To deepen this path after the test, reading about how to turn feedback into AI evaluation cases is complementary but does not replace experiment design.

The phrase to remember is this: a good hypothesis protects the team from their own desire to be right.

Success criteria must separate apparent improvement, sufficient evidence, and doubt

Before starting the experiment, the team needs to define what will happen if the signal is positive, negative, or ambiguous. This does not require turning the article into a statistical manual but demands decision discipline.

There are at least three categories:

Success: the main criterion improves and protection metrics remain within agreed limits.
Failure: the main criterion does not improve or a protection limit is violated.
Doubt: the data are insufficient, inconsistent, or incompatible with the test design.

The most delicate point in AI products is choosing a criterion linked to the real task outcome, not just feature usage. Clicks, adoption, screen time, or completion messages can be useful as auxiliary signals but do not prove operational value.

Anthropic, when discussing agent evaluations, distinguishes execution trajectory from effective outcome in the environment. A message stating the task ended is not enough to prove the expected result occurred Anthropic. Even when the feature is not an autonomous agent, the principle applies: the product must verify the effect of the action, not just the appearance of completion.

In a feature suggesting responses, for example, "response generated" is not success. "Response accepted without editing" may also be insufficient because the user might accept due to overconfidence, haste, or lack of expertise. The criterion needs to approach the outcome: problem solved, step correctly completed, rework avoided, appropriate routing, or decision revised when necessary.

Adverse effects must become protection metrics or qualitative checks

Experiments in AI products can improve a main metric and worsen the surrounding system. This is the easiest risk to underestimate because the feature seems productive at the measured point.

Some adverse effects deserve anticipation:

Incorrect responses accepted without review.
Increased rework in support or operations.
Worsening for specific user segments, tasks, or contexts.
Loss of trust when AI errs in cases sensitive to experience.
Higher operational costs to review, correct, or explain the feature.
Unstable behavior in rare but relevant cases.

Planning must turn each risk into a protection metric or qualitative check. If AI suggests ticket categories, a protection metric could be the rate of subsequent reclassification. If AI summarizes interactions, a qualitative check might review samples of summaries in long or ambiguous cases. If AI recommends next steps, the team can monitor signals of rework or improper escalation.

Protection metrics prevent a local win from becoming systemic loss. An experiment can increase adoption yet not deserve launch if it induces errors, shifts work to another area, or harms a specific group.

This discussion connects to AI strategy but on a more concrete scale. A business-connected artificial intelligence strategy defines principles and priorities. The experiment translates these principles into observable limits.

Monitoring must observe broad signals without premature reaction

During the test, the challenge is balancing vigilance and patience. Ignoring adverse signals is irresponsible. Reacting to every initial fluctuation is also bad because the team may stop or celebrate an experiment based on noise.

Monitoring should combine three readings:

Main metric: the outcome the hypothesis intends to improve.
Protection metrics: signals that block progress even with main improvement.
Segments: user groups, task types, or contexts that may react differently.

Microsoft's article on experiment monitoring recommends observing a broad set of metrics and segments to identify regressions and avoid premature interpretations during the test Microsoft Research. This is especially relevant in AI because averages can be too comfortable.

A general average may suggest stability while a smaller group suffers relevant worsening. Google SRE, addressing monitoring, explains that averages can hide problematic behavior and that different views serve different audiences Google SRE. In product, this means leadership may need a synthetic view, while engineering, support, and data need sufficient slices to detect localized harm.

The caution here is not to turn monitoring into hunting for any variation. Segmenting does not mean looking for a convenient narrative after the result. It means identifying beforehand which slices are critical for the decision: new and returning users, ticket types, plans, operational regions, entry channels, or task complexity levels.

The post-experiment decision must respect test design and data quality

After the experiment, the team should not ask only "did it work?" The more honest question is: "what do the data allow us to decide, considering the test design?"

There are four possible paths:

Launch the change if the main criterion was met, protections stayed within limits, and data support the interpretation.
Redesign and test again if the direction seems promising but there is a clear problem with design, interface, context, or segment.
Repeat the test if the signal is relevant but still insufficient for a launch decision.
Discard the change if the hypothesis was not confirmed or adverse effects make the main improvement undesirable.

Microsoft's article on post-experiment analysis recommends verifying if metric changes are compatible with test design and if data quality issues compromise interpretation before deciding to launch Microsoft Research. This step reduces two common temptations in AI features: launching a change because a number improved or discarding a test because an incomplete reading disappointed.

Data quality is not a technical detail separate from the decision. If events were not recorded correctly, if the sample does not represent intended use, if verifiers do not capture the real outcome, or if there was a parallel process change, the conclusion must be limited. Sometimes the most valuable learning from the experiment is discovering the organization does not yet measure well enough to decide.

In that case, the next decision is not to launch or discard the feature but to fix measurement and repeat the test.

Fictional example: triage assistant in support

Consider a fictional example. A company wants to test an AI feature that suggests the initial category of support tickets. Today, a triage person reads the ticket, chooses the category, and forwards it to the responsible team. The new feature presents a suggestion before the human decision.

The isolated change is not "deploy AI in support." It is "show an AI-suggested category at triage using the ticket's initial text and information already available in the system." The process continues with human decision. The experiment evaluates if the suggestion helps or hinders this step.

The hypothesis could be written like this: for configuration tickets, if AI suggests an initial category before human triage, we expect to reduce time to first classification without increasing subsequent reclassifications, complaints about incorrect routing, or escalations caused by inappropriate category.

The main criterion would be linked to time to first classification. But that alone would not suffice. If classification becomes faster and more incorrect, the product only pushed the problem forward. Therefore, protection metrics could include subsequent reclassification, ticket return to triage, dissatisfaction signals related to routing, and qualitative review of samples in categories with higher ambiguity.

Critical segments also need to be defined beforehand. Simple tickets may benefit from the suggestion, while ambiguous tickets may suffer more errors. New users may write less complete descriptions. Certain products or modules may have similar vocabulary and cause confusion. The overall average would not answer these differences alone.

The decision rule could be like this, without invented numbers: launch only if time to first classification improves according to the criterion defined by the team, without violating reclassification, complaint, and escalation limits. Redesign and test again if improvement appears only in simple tickets but worsens ambiguous tickets. Repeat the test if category or event data are incomplete. Discard if the suggestion induces recurring error in segments considered critical.

This example shows why planning must come before exposure. If the team only discusses criteria afterward, each area tends to defend the metric confirming its impression. Product may look at adoption. Support may look at rework. Engineering may look at stability. Leadership may look at speed. All these readings can be useful but must be organized into a common decision rule.

AI feature experiment planning checklist

Before starting the test, use this checklist as a practical review. It does not eliminate risk but reduces the chance of starting an experiment that cannot sustain a decision.

Isolated change: can the team explain which AI feature behavior is being changed without mixing model, interface, process, and business rule in the same test? If not, redesign the experiment before starting.
Falsifiable hypothesis: does the hypothesis allow saying, at the end, that it was not confirmed? If any result can be interpreted as success, the hypothesis is still weak.
Main criterion: is there a metric or check linked to the real task outcome, not just feature usage? If the metric measures only clicks, adoption, or completion message, include a result check.
Protection metrics: has the team defined which signals block launch even if the main criterion improves? Without protection limits, the experiment may reward a local improvement with systemic harm.
Critical segments: are there user groups, task types, or contexts where the effect may differ from the average? If so, these segments must be monitored separately.
Data quality: are the events, records, or verifiers used to interpret the experiment reliable enough to support the decision? If fragile, the conclusion should be treated as inconclusive or limited.
Decision rule: before starting, is it clear what will lead to launch, redesign, repeat, or discard the experiment? If the rule will only be discussed afterward, the team risks adapting interpretation to the desired result.

The experiment is only ready when hypothesis, main criterion, protection metrics, critical segments, and decision rule fit on a clear page. If that page cannot be written, the feature is not yet ready for users.

If you want to discuss this decision in your company's context, talk to dooop.

Sources

NEXT STEP

Discuss application in your company

Conversation about the software company context

Content by dooop. Registration allows linking this topic to the reader's journey and tracking interest in the subject.