dooopSoftware · Learning · 13 min

How to Choose Metrics for AI Products

Separate usage, quality, and customer outcome metrics to evaluate if the AI was adopted, worked well, and helped advance the task.

Published on September 6, 2026

CENTRAL THESIS

A full dashboard is not sufficient evidence. The decision depends on separating signals before cross-referencing them.

Usage shows entry into the flow. Quality and outcome show if the task progressed.

A product dashboard with artificial intelligence can show growth and still leave the main decision open. Clicks, activations, and messages indicate the feature entered the flow but do not prove the response was good or that the customer advanced in the task. To choose metrics for AI products, separate three layers: usage, quality, and customer outcome. Only then cross these signals to decide whether to expand, adjust, limit, or stop.

When a Usage Metric Does Not Answer if the AI Worked

The most common mistake when measuring an AI feature is treating usage as synonymous with success. Usage is a necessary but narrow signal. It indicates exposure, frequency, curiosity, dependency, or friction. Without context, it does not say which interpretation is correct.

An assistant may receive many messages because it helps. It may also receive many messages because the first response does not solve the issue, because the user needs to reformulate the question several times, or because the manual flow is too hidden. The same number allows opposite readings.

This does not diminish the value of usage metrics. They remain the first place to understand if the feature was seen, triggered, and incorporated into the flow. The point is different: a usage metric should only guide product decisions when linked to a clear step of the task.

"Users who triggered the assistant" is an exposure metric. "Users who triggered the assistant during ticket triage and proceeded to classification without returning to the manual process" already starts to describe a relevant interaction. It still does not prove quality or outcome but positions the signal in the flow.

This separation avoids a dangerous meeting: one where the dashboard shows adoption, support reports doubts, engineering sees unstable behavior, and product tries to decide with a single metric. In AI products, trust does not come from a higher number. It comes from knowing what each number can and cannot answer.

Separate Three Questions Before Choosing Any Metric

Before listing indicators, ask three questions in order:

Did the customer use the feature?
Did the AI perform the task with acceptable quality?
Did the customer achieve the expected outcome in the task that justified the feature?

Each question requires a different type of evidence.

The first question calls for usage metrics. They show exposure, frequency, repetition, abandonment, and return to the previous path. The second calls for quality metrics. They observe whether the AI behaved within scope, used the correct context, avoided inappropriate responses, and required human correction or not. The third calls for outcome metrics. They verify if the customer advanced in the observable task: resolved, decided, classified, found, continued, or completed.

The order matters because it prevents one layer from stealing the function of another. If the question is quality, usage volume does not answer. If the question is customer outcome, a technically correct response may not be enough. If the question is exposure, a perfect evaluation in a few cases also does not prove adoption.

This logic aligns with a broader product management decision: treat AI as part of a learning cycle, not as a magic box that improves just because it received interactions. The guide on learning cycles in AI products deepens this view. Here, the focus is more specific: choosing metrics without mixing signals.

Choose Usage Metrics to Measure Exposure, Frequency, and Abandonment

Usage metrics answer whether the feature appeared in the customer's routine. They help identify if the problem is discovery, flow fit, recurrence, or abandonment.

In an intelligent feature, good examples of usage metrics include:

users exposed to the feature;
triggers per flow step;
repeated use in similar tasks;
abandonment after the first response;
return to the manual process;
time until the user ignores, accepts, or edits a suggestion.

The caution is not to call these metrics value prematurely.

Fictional example: imagine an internal search assistant on a support platform. It suggests knowledge base articles for the team to answer customer questions. In the first weeks, query volume grows. This signal may indicate the assistant was discovered and seems useful. But it may also indicate agents are making several attempts because the search does not find the right article.

In this case, "queries to the assistant" measures usage. To improve interpretation, product could cross this signal with abandonment, question reformulation, click on suggested article, and return to manual search. Still, the decision to expand would depend on other layers: was the suggestion correct? Did the support advance? Was there rework?

The usage metric should be designed close to the task. The farther it is from the actual flow, the greater the risk of vanity. "Messages sent" is weaker than "messages sent during an attempt to classify a ticket." "Panel opening" is weaker than "use of the suggestion before completing the step."

Choose Quality Metrics to Evaluate AI Behavior

Quality in an AI feature should not be measured only by the final message. In many cases, it is necessary to observe how the response was produced, what context was used, which limits were respected, and whether the execution actually reached the expected environment.

Anthropic distinguishes the execution trajectory of an agent from the effective result in the environment. A message saying the task ended is not enough to prove the result happened. The same logic applies to AI products that are not full agents: the appearance of completion does not replace verification.

Quality metrics may include:

response within the defined scope;
correct use of available information;
absence of instructions incompatible with product policy;
need for human correction;
repeated failures in similar cases;
classification by criteria defined before analysis;
verification when there is an observable condition.

The most sensitive point is the phrase "criteria defined before analysis." If the team looks at results first and only then decides what counts as good, the evaluation becomes vulnerable to convenient justifications. AI evaluations must specify inputs, success criteria, and verifiers. This verification must be compatible with the risk and the task.

In the fictional example of the internal search assistant, a quality metric could classify suggestions into four categories: correct article, partially useful article, out-of-scope article, and absence of an adequate suggestion. Another metric could record when the agent edits the suggested response because it does not correctly represent the knowledge base content.

Note that these metrics still do not prove customer outcome. They show whether the AI is behaving acceptably for the assigned function. A suggestion may be correct and still not solve the user's task. Therefore, the third layer cannot disappear.

Choose Outcome Metrics to Know if the Customer Advanced

Customer outcome is the layer that connects the feature to the reason it exists. The question stops being "Did the AI respond?" and becomes "Did the person manage to advance in the task?"

This difference seems small but changes metric choice. Instead of measuring only suggestion acceptance, you look for observable signs of flow advancement.

Possible outcome metrics include:

task completed without visible rework;
reduction in item reopening;
flow continuation after the suggestion;
decision made with less support dependency;
less return to the manual path;
classification completed without later correction;
resolution perceived by the user, when structured collection exists.

None of these metrics is universal. The choice depends on the task, risk, flow, and verification capability. A feature suggesting ticket categories requires different signals than an AI summarizing internal documents or recommending next steps in an operations tool.

The practical criterion is this: describe the task in an observable sentence before choosing the metric. For example: "the agent needs to classify the ticket to forward it to the correct team." From there, the outcome is not "use AI." The possible outcome is "classify the ticket without reopening due to category error" or "proceed to forwarding without returning to the previous step."

This also helps avoid easy promises. The metric should not declare that AI improved the operation. It should indicate whether, in that flow, there was sufficient signal for a decision. In some cases, the decision will be to expand. In others, adjust. In others, limit use until behavior is better understood.

Build a Combined Interpretation Before Deciding

The three layers only gain strength when read together. Separating does not mean isolating. It means preventing one signal from dominating interpretation.

Some combinations help guide the conversation:

high usage with low quality indicates scaling risk;
low usage with high quality may indicate discovery, trust, or flow fit problems;
high quality with low outcome suggests the AI may be correct but not very useful for the task;
positive outcome in one segment and negative in another calls for investigation before a general decision;
increasing usage with more human corrections suggests adoption may be pushing work to another step.

Microsoft recommends, when monitoring experiments, observing a broad set of metrics and segments to identify regressions and avoid premature interpretations while the test runs. Outside a formal experiment, the allowed statement is narrower: this care helps prevent a single metric from carrying the entire decision.

Google SRE also warns that averages can hide problematic behavior and that different visualizations serve different audiences. In AI products, this is especially relevant because an acceptable average may hide failures concentrated in more complex task types, customers with less context, or higher-risk flows.

In the fictional example of the internal search assistant, average satisfaction with the suggestion may seem acceptable. But segmenting by ticket type, the team may discover a hypothesis to investigate: the AI helps with frequent questions and fails in cases requiring recent context. This finding should not be treated as an automatic conclusion. It becomes a better question for evaluation, monitoring, or experiment.

If there is a formal experiment, Microsoft recommends checking whether metric changes are compatible with the test design and whether data quality issues compromise interpretation before deciding on release. In other words: good decisions do not arise only from the dashboard. They also arise from confidence in design and data.

Checklist to Choose AI Product Metrics Without Mixing Signals

Use this checklist to review a planned feature, start an experiment, or improve an existing dashboard.

Does the metric answer usage, quality, or customer outcome?

If the same metric tries to answer all three, it should be divided or rewritten.

Number of messages sent to the assistant is usage. It is not, by itself, response quality or task success. Suggestion acceptance can be an intermediate signal but still needs context: did the user accept because it was correct, because it was fast, or because there was no clear alternative?

Is the metric linked to an observable step of the task?

Avoid metrics that do not indicate where the feature interfered in the customer flow.

In a fictional example of a ticket triage assistant, measuring suggestion opening is less useful than measuring if the ticket was classified and did not return due to category error. The second metric is closer to the task. The first measures only contact with the feature.

Is there a quality criterion defined before analysis?

The evaluation must have inputs, success criteria, and verifiers compatible with the risk and task.

For an AI recommending support responses, the criterion can separate correct response, incomplete response, out-of-scope response, and response requiring correction. Without this prior classification, the team tends to discuss case by case without accumulating learning.

Can the metric hide problematic behavior in averages?

Whenever there are groups, task types, or different risk levels, the reading should consider segments.

An acceptable average satisfaction may hide a drop among customers using the feature in more complex cases. An overall acceptance rate may hide differences between novice and experienced users, including how each group accepts poor suggestions. The segment is not to complicate the dashboard. It is to prevent an overly optimistic decision.

Does the metric have an associated decision?

Before measuring, define whether the signal can lead to expansion, adjustment, limitation, new evaluation, or interruption.

If usage rises but the human correction rate also rises, the likely decision is not to expand. It is to review quality, restrict the use case, or investigate why the feature is transferring effort to another step.

Is there a protection metric to avoid overly optimistic reading?

Every main metric should have at least one signal that alerts regression, flow damage, or fragile interpretation.

If the main metric is task completion, a protection metric can be reopening, support complaint, or return to the manual process. This topic connects to the discussion about protection metrics in AI experiments, when the question stops being just "did it work?" and includes "what can get worse while it seems to work?"

Define Decision Limits to Expand, Adjust, or Stop

A metric without an associated action becomes dashboard decoration. For AI products, this is even riskier because the novelty of the feature can create excessive tolerance for confusing signals.

Before reading data, define decision limits. They do not need to be universal. They need to be explicit for that use case.

Expansion makes sense when usage, quality, and outcome move in the same direction and protection metrics do not indicate relevant regression. Adjustment makes sense when there is explainable divergence: good quality with low usage, high usage with abandonment, weak outcome in a specific segment. Limitation or interruption come when quality is unstable, protection indicates risk, or data does not allow reliable interpretation.

This last option requires analytical discipline. Stopping a poorly understood feature can preserve trust, contain rework, and open space for a better hypothesis. The next step may be better instrumentation of usage, review of evaluation criteria, case segmentation, or redefinition of the observable task.

If you already have a broader AI strategy, it is worth connecting this dashboard to portfolio decisions. The article on how to create an AI strategy connected to the business addresses this bridge between opportunity, capability, and priority. And if the organization is still understanding its starting point, the AI maturity guide helps separate ambition from operational capability.

The dashboard becomes more useful when each metric has a layer, question, and possible action: usage to understand flow entry, quality to evaluate AI behavior, and outcome to verify task progress. The final question for the dashboard is simple: does this number help decide to expand, adjust, limit, or stop, or does it just make the feature seem better understood than it really is?

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss Application in Your Company

Conversation about the software company context

Content by dooop. Registration allows relating this topic to the reader’s journey and tracking interest in the subject.