dooopSoftware · Quality · 11 min

Calibrating AI Evaluations with Domain Experts

Learn how to confront automatic scores with domain judgment by adjusting criteria, weights, evidence, and thresholds before approving AI.

Published on September 6, 2026

MAIN THESIS

A high score can hide domain errors. Experts enter to calibrate the signal, not to replace the metric.

Confront automatic evaluation with domain criteria before turning a score into a product decision.

An automatic score can approve a response that a domain expert would reject. AI evaluation calibration exists to address this friction before the score becomes a signal for approval, blocking, or prioritization. An evaluation may pass format, clarity, and completeness but fail in the real use context. The point is not to replace the metric with opinion but to adjust what automation considers quality in use.

The Hidden Risk Behind a Good Automatic Score

A team launches an AI feature because the automatic evaluation indicates good performance. Responses follow the expected format, cover requested topics, and seem useful on a quick read. In expert review, discomfort arises: the AI responds well to simple cases but fails on exceptions that truly change user guidance.

This is a calibration problem, not just a testing issue.

Automatic AI evaluation usually transforms criteria into repeatable checks. This is necessary to scale quality, compare versions, and avoid each decision depending on subjective conversation. But in AI products, the result may seem correct without being domain-appropriate. A polite, complete, and well-structured response can still guide a wrong action, omit an exception, or overstate confidence.

DORA recommends testing throughout development, combining automation and manual activities such as exploration and usability, and maintaining and reviewing test suites instead of treating quality as a post-development step DORA. This combination is especially relevant when automatic evaluation measures part of quality but does not see all usage risks.

The practical decision is simple to state and hard to execute: before an automatic score becomes authorization, blocking, or correction ranking, it must be confronted with domain judgment.

What Needs to Be Calibrated: Criterion, Weight, Evidence, and Threshold

When experts disagree with automatic evaluation, the conversation usually starts with "the score is wrong." This is not very helpful. To truly calibrate, the team needs to discover which evaluation component failed.

Criterion defines success. For example: the response must follow product policy, cite an applicable restriction, request additional information when ambiguous, or refuse an out-of-scope request.

Weight is the relative importance of each criterion. Clarity may count less than adherence to a domain rule. Format may be mandatory but not sufficient. A response can be pleasant yet unacceptable.

Evidence proves the result occurred. In systems that perform actions, it is not enough for AI to say the task was completed. Anthropic distinguishes the agent's execution trajectory from the effective result in the environment: a message saying the task ended does not prove the result; evaluations use inputs, success criteria, and verifiers Anthropic. Even when the product is not an autonomous agent, the lesson applies: evaluation must observe evidence, not just the appearance of completion.

Threshold is the line separating approval, blocking, human review, or gradual exposure. Two versions may have close scores but require different decisions if one fails on rare but relevant cases.

Experts should not enter the process just to give a parallel score. They need to indicate if the problem lies in criterion, weight, evidence, or threshold. Without this separation, calibration becomes a matter of preference.

How to Compare Automatic Evaluation with Expert Judgment

A comparison ritual helps reduce noise. It does not need to start sophisticated. It needs to start controlled.

First, select representative cases of expected use and known exceptions. The base should not contain only frequent and happy questions. It should include ambiguities, incomplete inputs, out-of-scope requests, and situations where the correct response depends on internal policy.

Then, run the automatic evaluation always with the same input, model version, configuration, and usage context. If the team changes prompt, model, temperature, or reference base during comparison, it loses the ability to interpret divergence.

Next, ask the expert to judge the same cases independently. Ideally, avoid showing the automatic score first, as it may anchor the review. The expert should record the expected decision, perceived risk, and domain justification.

Only then does the team compare results.

Fictional example: a company uses an internal assistant to guide support teams on service procedures. The automatic evaluation approves a response because it is polite, complete, follows the expected format, and cites general policy. A domain expert rejects the case because the question contained an operational exception that changes guidance. The response is not absurd. It is plausible. Precisely for that reason, it is dangerous.

Calibration should not just lower the score for this case. The most useful adjustment may be to create a specific criterion for policy exceptions, require evidence that the exception was considered, and separate similar cases for human review until the automatic evaluation shows sufficient consistency to support exposure decisions.

Calibration addresses a specific operational need. An organization may have a good business-connected AI strategy and still need calibration if operational quality depends on specific rules the initial metric does not capture.

Classifying Divergences Without Turning Everything into Opinion

Every divergence needs a name before receiving a solution. This changes the conversation.

The most useful categories are:

False positive: automatic evaluation approved a response the domain considers unacceptable.
False negative: automatic evaluation rejected a response that meets domain criteria.
Missing criterion: evaluation did not check something necessary for real use.
Ambiguous criterion: human or automatic evaluators interpret the rule differently.
Disproportionate weight: evaluation gives too much importance to a secondary aspect or too little to a relevant risk.
Insufficient evidence: the response seems to fulfill the task, but there is no observable proof the expected result occurred.

The most dangerous category is usually false positive. That is when the score creates confidence where review should be triggered. But not every divergence calls for more strictness. A recurring false negative may indicate the evaluation punishes acceptable language variations. An ambiguous criterion may reveal even experts disagree on the rule. Insufficient evidence may show the team is evaluating response text when it should verify an action, state, or consequence in the environment.

Experts also disagree. When this happens, the team should not rush to adjust evaluation based on the loudest voice. They should record the reason for divergence, separate explicit rule from individual preference, and decide if the product needs clearer policy before requiring AI compliance.

This record makes divergence actionable. Diagnostics like an AI maturity exercise help only when they reach the level of decisions the team really needs to make: approve, block, review, or monitor.

How to Adjust Evaluation After Expert Review

After classifying divergences, the team must turn learning into verifiable evaluation change. Otherwise, expert review becomes a ceremony.

Possible adjustments are concrete:

Rewrite criteria to remove ambiguity.
Add evaluation verifiers when the response must prove a result.
Create exception cases in the evaluation base.
Change weights so domain risks have proportional influence.
Raise or segment approval thresholds.
Separate case classes requiring human review.
Record expected decision and justification for future comparison.

Care must be taken not to turn every divergent case into an overly narrow rule. If the team adds cases only as literal memory, evaluation may become good at repeating past examples but fragile recognizing equivalent variations. The expert’s role is to help extract the domain rule behind the case.

It is also worth separating evaluation from release. A version may be available in code without enabling the feature for all. Google SRE discusses gradual releases as a way to evaluate a change on a portion of traffic before expanding exposure, distinguishing code availability from feature activation and using configurations to separate decisions Google SRE. In AI products, this separation helps when calibrated evaluation improves confidence but does not yet eliminate the need to observe behavior in controlled use.

Calibration does not prove productivity gain, risk reduction, or commercial value by itself. Calibration improves the quality of the signal used to decide. This distinction matters because AI metrics can be difficult to interpret outside their context. METR, in a February 2026 update, treated new data as an unreliable signal about AI’s current effect on productivity, pointing to participant and task selection and difficulties measuring time with competing agents METR. The parallel here is prudence: a metric must be understood by what it measures, not by what we wish it meant.

When to Keep Human Judgment in the Flow, Even with Automation

A calibrated evaluation does not eliminate experts. It helps decide where automation is reliable enough and where human control remains part of product design.

Situations where keeping human judgment in the flow is the most responsible choice include:

High-impact cases for the user.
Rare but relevant situations for product reputation.
Domain ambiguity not yet turned into clear policy.
Recent changes in internal rules, positioning, or product scope.
Recurring divergence between automatic evaluation and experts.
Need to interpret context not present in test input.

The common mistake is treating human review only as delay. In the evaluation flow, it allows discovering missing criteria, updating the case base, and preventing an aggregate score from hiding failures concentrated in the most important scenarios.

The question is not "when can we remove all experts?" The better question is: "in which decisions is automatic evaluation already sufficient evidence, and in which does it still need to be accompanied by domain judgment?"

This difference changes process design. For reversible, low-impact decisions, a well-calibrated automatic evaluation can support progress with monitoring. For sensitive decisions, frequent exceptions, or unstable policies, automation can filter, organize, and prioritize but should not approve alone.

Calibrated Decision Checklist Between AI and Experts

Use this checklist when an automatic score and expert evaluation point to different decisions.

Did the automatic evaluation and expert judge exactly the same case? Use the same input, model version, configuration, and usage context.
Was the divergence classified before solution discussion? Mark each difference as false positive, false negative, missing criterion, ambiguous criterion, disproportionate weight, or insufficient evidence.
Did the expert explain which domain risk the score missed? Record the risk in observable terms such as inadequate guidance, exception omission, out-of-policy response, low practical utility, or weak evidence of completion.
Did the adjustment change a specific evaluation component? Alter criterion, weight, evidence, threshold, test case, or human review requirement. Avoid generic adjustments like "increase rigor" without specifying where.
Was the divergent case added to the future evaluation base? Add the case or a representative variation, with expected decision and domain justification.
Did calibration change an operational decision? Check if approval, blocking, correction priority, monitoring, or human review requirement changed.

This checklist works best when connected to product decisions. If the team is building an AI roadmap, calibrating evaluations helps separate initiatives that seem ready in demonstration from those needing stronger criteria before scaling.

The Final Criterion: Did the Evaluation Change a Decision?

Calibration that does not change a decision may be just ritual. After confronting automatic evaluation with experts, the result must appear in at least one decision: approval changed, blocking removed or maintained, correction prioritized, monitoring adjusted, or case sent for human review.

If nothing changes, perhaps the evaluation was already good for that scope. It may also be that the team is discussing quality without giving operational consequence to learning. The difference appears when calibrated evaluation begins to guide decisions that previously depended on vague trust, deadline pressure, or convincing demonstrations.

The concrete decision is to calibrate automatic criteria with domain judgment before using the score as a signal for approval, blocking, or priority. If this conversation needs to move beyond improvisation, talk to dooop.

Sources

NEXT DECISION

Discussing Application in Your Company

Conversation about the software company context

Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.