dooopSoftware · Quality · 11 min

When an AI Feature Can Go to Production

Before releasing AI to real users, gather evidence, operational limits, and assign someone responsible for advancing, pausing, or reverting.

Published on September 6, 2026

CENTRAL THESIS

A convincing prototype does not authorize production. The decision requires evidence, limits, and the possibility to pause.

AI release demands proof in the environment, explicit scope, and someone able to stop it.

The decision to put an artificial intelligence feature into production usually comes after a convincing demonstration. At this point, the discussion needs to change tone: the release should only proceed when there is evidence of functionality, operational limits, and clear authority to advance, pause, or revert. Without this, the question stops being “does the AI look good?” and becomes “who authorized operating with this uncertainty?”.

What Needs to Be Proven Before Exposing the Feature

The demonstration usually shows the feature at its best moment. The flow is clean, the input carefully chosen, and the narrative favors the idea that the AI already understands the task. This has value to align vision but is not enough for production.

In production, authorization changes consequence. The feature faces real variations in language, incomplete data, urgency, exceptions, frustrated users, unstable integrations, and ambiguous interpretations. Therefore, the quality criterion for AI cannot be just “answered well in some examples.” It must show what was tested, how it was judged, where it failed, and what limit prevents failure from causing operational damage.

DORA, when addressing test automation, recommends testing throughout development, combining automation and manual activities such as exploration and usability, as well as maintaining and reviewing test suites instead of treating quality as a post-development step DORA. For this guide, the inference is limited: quality must accompany construction, not appear only as a stamp at the end.

For product, engineering, or quality leadership, the practical question is: what evidence would make a prudent person authorize limited exposure? If the answer is vague, the feature is still in internal experimentation territory.

Here it is worth separating three situations many teams mix:

Demonstration: proves that an idea can be understood and presented.
Internal testing: verifies behavior in cases chosen by the team.
Authorization for production: decides real exposure with evidence, limits, and responsibility.

The third situation requires more discipline because it changes the consequence of the decision. The feature does not need to be perfect. It needs to be sufficiently delimited to operate without relying on improvisation.

What Evidence Is Included in the Decision Package

The decision package should answer a simple question: what do we know, based on observation, about the feature’s behavior?

For AI features, this includes more than recording prompts and responses. If the AI performs a task, suggests an action, or alters a flow, the team needs to verify the actual result in the environment. Anthropic distinguishes the execution trajectory of an agent from the actual result obtained in the environment: a message saying the task is finished does not prove by itself that the task was completed Anthropic. The same source describes evaluations with inputs, success criteria, and verifiers, possibly requiring multiple attempts.

Without turning this article into a complete evaluation guide, a minimum package should contain:

Tested inputs: what types of requests, data, or cases were used.
Success criteria: what counts as an acceptable, incomplete, or incorrect response.
Verifiers: how the team checked if the result was true, useful, or safe within the flow.
Attempts made: whether the behavior was observed once or in different variations.
Result in the environment: what happened beyond the AI’s textual response.
Failure records: what errors appeared and how they were classified.

This set does not eliminate uncertainty. It makes the decision auditable: someone can see what was tested, which criteria were used, and which failures are known.

It also helps avoid a common bias: choosing only examples where the AI shines. A feature candidate for production needs to be observed in normal cases, ambiguous cases, and predictable exceptions. Not to prove it never fails, but to know if failure stays within a manageable area.

If the team has not yet defined acceptance criteria for expected behavior, it is worth resolving this before discussing scale. There is a difference between “I liked the answer” and “the answer meets the agreed criteria.” This is the kind of distinction that supports a broader practice of software quality with AI, without turning every release into an informal bet.

How to Declare AI Operational Limits

Production does not mean unrestricted freedom. An AI feature can be in production and still operate in a narrow, well-defined, and observable space.

This limit needs to be written so product, engineering, quality, and support can use it. It is not enough to say “the AI helps the user.” It is necessary to declare where it helps, where it must stop, and when it needs to transfer the decision to a person.

A good operational limit answers questions such as:

Can the AI only suggest or also execute actions?
What data can it consult?
What data should it not use?
In which situations should it refuse a request?
When should it ask for human confirmation?
What actions are outside its scope?
What signals indicate unexpected behavior?

This document does not need to be long. It needs to be operational. If no one can verify whether the limit was respected, it is just an intention.

There are cases where the best decision is not to release. For example, when the feature makes decisions the team cannot audit, when it depends on data with unclear origin, or when there is no way to interrupt exposure without affecting a larger flow. The operational limit is a maturity condition, not a bureaucratic detail.

This conversation also avoids a frequent confusion: using a third-party model does not make the product immature. The problem is not the model’s origin. The problem is placing a probabilistic capability inside a process without declaring what it can do, who observes, and who responds when it exceeds expectations.

Who Decides to Advance, Pause, or Revert

Responsibility for AI in production cannot be hidden in a tool, a vendor, or a generic phrase like “the team monitors.” If the feature affects real users, someone needs authority to advance, pause, or revert.

This responsibility can be distributed among roles, but the final exposure decision cannot be diffuse. In a practical design:

Product is responsible for expected value, user experience, and functional scope.
Engineering is responsible for operation, observability, configuration, and technical rollback.
Quality is responsible for evidence, acceptance criteria, and known gaps.
A decision owner authorizes exposure, defines conditions, and interrupts when necessary.

The point is not to create a heavy hierarchy. It is to avoid the feature going live because no one clearly felt responsible to say “not yet.”

Pressure to release usually appears when the demonstration impresses, the schedule tightens, or leadership wants to show progress. In these moments, explicit responsibility protects the product’s ambition itself. It allows conditioning the launch on scope, monitoring signals, and pause rules.

The February 2026 METR update is useful as a humility alert about AI measurements. The organization considered new data an unreliable signal of AI’s current effect on productivity and pointed out difficulties such as participant and task selection, as well as measuring time with competing agents METR. This does not say whether a specific feature has value for your client. But it reminds that decisions involving AI may seem more measurable than they really are when the metric is not well designed.

Therefore, before releasing, the team must know which signals to observe and who interprets them. Without a decision owner, metrics become decoration.

How to Release in Production Without Confusing Available Code with Active Feature

One of the most useful decisions is to separate available code from active feature. Code can be deployed in the production environment without the AI being enabled for all users. This separation gives the team a more careful way to learn from real use.

The Google SRE chapter on canarying releases addresses evaluating a change on a portion of traffic before expanding exposure. It also distinguishes making code available from activating features and discusses using configurations to separate these decisions Google SRE.

Applied to an AI feature release, this means the team can start with a restricted group, observe combined signals, and expand only if conditions are met. Gradual activation should not be treated as a safety guarantee. It only helps decide better when there is advancement criteria and interruption condition.

A minimum gradual exposure plan should clarify:

Who will see the feature first.
What actions the AI can perform in this phase.
Which signals authorize expanding exposure.
Which signals require pausing or reducing.
How the feature will be deactivated.
When the team will review evidence and failures.

This separation is especially relevant when AI interferes with existing flows. If the release can only be turned on or off for everyone, the organization loses the ability to learn with control. If the feature can be activated by configuration, group, or context, the decision stops being too binary.

Fictional Example: Internal Support Ticket Triage Assistant

Consider a fictional example. A company creates a triage assistant for internal support tickets. The feature reads the ticket description and suggests category, priority, and next step. It does not close tickets, send external messages, or change user permissions.

The demonstration impresses because the assistant understands common requests and suggests plausible routing. Still, going to production depends on the decision package.

Evidence could include evaluation cases with frequent tickets, incomplete descriptions, ambiguous requests, and known exceptions. For each case, the team defines what counts as acceptable classification, incomplete suggestion, or error. The verifier can be a review by support and quality staff comparing the AI suggestion with the expected procedure.

Operational limits would be explicit. The assistant can suggest category, priority, and next step. It must refuse requests outside the internal catalog. It must escalate to a person when the ticket involves sensitive access, ambiguous description, or conflict between apparent priority and operational rule. It cannot close tickets. It cannot respond on behalf of the team. It cannot consult data outside authorized sources.

Responsibility also needs to be named. Product is responsible for scope and experience. Engineering is responsible for activation, technical logs, and rollback. Quality is responsible for the case set and failure records. The exposure decision owner authorizes initial activation and can pause the feature.

Release starts with a limited group of internal users, separated from code deployment. The pause rule may include increased manual reclassifications, recurring complaints about inadequate suggestions, or occurrence of suggestions outside documented limits. These effects are hypotheses to measure, not presumed results.

This example shows what changes when the decision moves from demonstration to release. The assistant does not need to solve everything. It needs to operate in a defined space, with sufficient evidence and possible interruption.

AI Production Readiness Matrix

This matrix is a proposal from this guide to support the launch decision. The consulted sources help ground parts of the reasoning but do not validate this matrix as an external method.

Proven Result in the Environment

Has the team verified the actual result of the feature, not just the AI’s message saying the task is complete?

If the answer is no, block production.

Defined Evaluation Cases

Are there sufficient inputs, success criteria, and verifiers to judge expected behavior?

If the answer is partial, release only for controlled testing or expand evaluation before production.

Explicit Operational Limits

Is it documented what the AI can do, what it must refuse, and when it needs to escalate to a person?

If limits are not written, postpone release.

Named Responsible Party

Is there a person or role responsible for advancing, pausing, or reverting the feature?

If responsibility is vaguely distributed, do not release.

Gradual Exposure

Can activation start with a limited group, separated from mere code availability?

If there is no exposure control, reduce scope or prepare a configurable activation method.

Pause Criteria

Has the team defined which signals require interrupting or reducing exposure?

If pause depends on improvised interpretation, do not expand the launch.

Review After Real Use

Is there a defined moment to review evidence, failures, and corrections after initial exposure?

If no review is planned, treat the release as incomplete.

The concrete decision is to name who is responsible for the release and which conditions authorize advancing, pausing, or reverting. If evidence, operational limits, or responsible party are missing, the feature is not yet ready for real exposure.

In the end, production is not the natural continuation of a good demonstration. It is an operational authorization to observe, limit, and interrupt the feature when it encounters real use.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

To Continue This Reading

How to Evaluate Inconsistent AI Responses

NEXT DECISION

Discuss Application in Your Company

Conversation about the software company context

Content from dooop. Registration allows relating this topic to the reader’s journey and tracking interest in the subject.