dooopSoftware · Quality · 13 min

How to Evaluate AI with Human Judgment Without Turning It into Opinion

Human evaluation of AI requires sampling by risk, observable criteria, and separation between style, utility, and operational failure.

Published on September 6, 2026

MAIN THESIS

Human judgment helps when it separates preference from failure. Without criteria, it only organizes opinions.

Sampling by risk and observable criteria make human review actionable without promising full coverage.

Human evaluation of artificial intelligence should not be a vote on pretty answers. It works best when the team selects samples based on decision risk, defines observable criteria before review, and separates style preference from operational failure.

Human judgment in AI products is useful when it clearly states what is being evaluated: response quality, style preference, or need for further verification.

When Human Evaluation Becomes Over-Organized Opinion

A team puts an AI feature into internal testing. The response seems polite, well-written, and convincing. One person approves because it "felt right." Another disapproves because they found the tone risky. A third is unsure whether the problem lies in the response, the user’s expectation, or the evaluation form itself.

This scenario does not show that human evaluation is useless. It shows it was designed as overly organized opinion.

AI human evaluation needs to separate at least three layers. The first is taste: tone, style, conciseness, word choice. The second is user adequacy: does the response help that person with that task? The third is operational quality: did the AI complete the task, respect constraints, avoid fabricating information, and not guide an inappropriate action?

Mixing these layers into a single score creates false precision. The spreadsheet looks clean, but the decision remains unclear.

This distinction aligns with a broader quality practice. DORA recommends treating tests throughout development, combining automation and manual activities like exploration and usability, rather than pushing quality to a final stage. In AI products, human evaluation is one of these manual activities. It does not replace automated tests, user research, security review, or production decisions. It helps identify cases where human criteria are necessary.

If leadership does not define what will be judged, the evaluator tries to guess. When each evaluator guesses differently, divergence seems like a people problem. Often, it is a design problem.

Choose Samples by Decision Risk, Not by What Is Easy to Review

Sampling for AI evaluation should not start with the most available examples. It should start with the decisions the team needs to make.

If the product uses AI to answer questions about software configuration, for example, a sample composed only of simple questions about password, language, and notifications may show fluency but will not reveal if the feature handles ambiguous situations, technical constraints, or incomplete instructions.

A useful sample set usually combines four types of cases:

Common cases, representing frequent uses and helping verify consistency in everyday experience.
High-impact cases, where a wrong answer can cause rework, operational blockage, or loss of trust.
Ambiguous cases, where the user’s intent is not fully clear and the AI needs to ask for clarification or limit the response.
Cases where the response may seem convincing even when it is wrong, incomplete, or misaligned with product policy.

The point is not to cover everything. That would be a false promise. The point is to choose samples that inform a decision: release to more users, adjust instructions, restrict scope, expand testing, or block a behavior.

Discussion about measurement also requires care. In the February 2026 update, METR considers its new data an unreliable signal of AI’s current effect on productivity, pointing out difficulties such as participant and task selection and time measurement with competing agents. This source does not authorize concluding whether a specific feature generates value for users or commercial differentiation. But it helps remind that measuring AI requires attention to measurement design, not just aggregated results.

The same applies to human evaluation. A biased sample can produce confidence without quality. If the team reviews only easy cases, the product looks better than it is. If it reviews only extreme cases, it may look worse than it will be in daily use. Leadership needs to declare which decision that sample supports.

To connect this practice to product management, it is worth treating evaluation as part of a maturity system, not as an isolated ceremony. This reasoning aligns with the idea of diagnosing capabilities before scaling initiatives, as discussed in AI Maturity: How to Diagnose the Organization’s Starting Point.

Turn Taste into a Separate Field, Not a Quality Score

Taste is not irrelevant. It can influence perception of the response. But taste should not alone decide if a response is correct, safe, or useful.

A response can be dry and still complete the task accurately. Another can be elegant and fail by omitting a constraint. A third can be friendly but guide the user to do something outside the product’s scope.

Therefore, the evaluation form should separate dimensions. Instead of asking only "Was the response good?", it is better to ask:

Did the response fulfill the user’s main intent?
Did the response fabricate information or state something without available basis?
Did the response omit a relevant constraint?
Did the response guide an inappropriate action for that context?
Was the tone appropriate for the type of interaction?
Did style preference affect utility or was it just an evaluator’s preference?

The last question is more powerful than it seems. It allows recording style discomfort without automatically turning it into a quality failure.

This separation also reduces political noise. When more senior people influence evaluation by personal taste, the form design needs to prevent preference from becoming an implicit criterion. If the criterion is not explicit, "I didn’t like it" can become "this lacks quality." The consequence is dangerous: the product reflects internal preferences, not criteria of use, risk, and outcome.

Quality criteria in AI need operational consequence. If the tone is bad, the team can adjust instructions, examples, or response guidelines. If the response fabricates information, the problem is of another nature. It may require limiting sources, changing context retrieval, modifying verifiers, or preventing the AI from responding when there is insufficient evidence.

Factual error, omission of constraint, inappropriate tone, and style preference should not receive the same label.

Use Observable Criteria Before Asking for an Overall Score

The overall score should be a synthesis, not the sole evidence.

When evaluation starts with an open score, the evaluator compresses several perceptions into a number or scale. The problem appears later: did a low score come from factual error, inappropriate tone, lack of completeness, evaluator’s fear, or personal preference?

Observable criteria help reduce this ambiguity. They do not eliminate human judgment but make it debatable.

A good criterion does not need to be perfect. It needs to be applicable. "Reliable response" is vague. "Does not state a configuration that does not appear in the documentation available for that product" is more observable. "Good experience" is broad. "Explains the next possible action without promising the problem is solved" is more useful.

Fictional example: imagine an internal support product for operations teams at a retail company. The AI answers questions about how to open tickets, check order status, and guide simple internal procedures. The team decides to evaluate human responses before expanding use.

A sample question says: "The order shows as delivered, but the store says it did not receive it. What do I do?"

An elegant response might say: "Don’t worry, the order will be regularized. Open a request and wait for confirmation." It sounds calm but may fail if it promises regularization without evidence or omits the need to check order data.

A less elegant response might say: "Check the status in the order system, confirm the store code, and open a ticket with delivery data. If there is a discrepancy, do not confirm regularization before analysis." It is less polished. But depending on product policy, it may be operationally better.

In this fictional example, evaluation should separate:

Correctness: does the response follow the available procedure?
Completeness: does it include the minimum information for the next action?
Operational safety: does it avoid promising a result not yet verified?
Utility: does it help the user act without creating unnecessary dependency?
Tone: is it clear and appropriate to the context?

The effects of this evaluation would be hypotheses to measure, not presumed results. The team could hypothesize that separate criteria reduce evaluator divergence or make corrections more actionable. But it would need to monitor this in the process itself.

This type of design also connects to a business-connected artificial intelligence strategy: evaluation does not exist to admire technology but to decide where it can operate responsibly.

Compare Evaluators to Find Poor Criteria

Divergence among evaluators is not only a problem to fix. It can be a source of learning.

If two competent people disagree on the same case, there are some possible explanations. One is that one person erred. But this should not be the first hypothesis. Perhaps the criterion is vague. Perhaps the sample is ambiguous. Perhaps the product does not yet have a clear policy for that type of situation.

Leadership should treat disagreement as a calibration signal.

Calibration does not mean forcing everyone to think alike. It means aligning what counts as evidence. In a calibration round, the team can select some cases, request independent evaluations, and then discuss only points of divergence. The central question is not "who is right?" but "which rule would allow better decisions next time?"

If divergence appears in tone preference, perhaps separating that field suffices. If it appears in correctness, perhaps a reliable reference is missing. If it appears in safety, perhaps the risk policy is incomplete. If it appears in utility, perhaps the team needs to better understand the user’s task.

There is an important limit: human evaluation does not guarantee AI quality. It reduces blindness to relevant cases, provided samples, criteria, and calibration are well designed. Without this, it only records opinions on a scale.

This is the practical difference between recording human review and designing human evaluation. The value lies in knowing what these humans judge, with which criteria, on which cases, and with what consequence.

Separate Final Result from AI’s Apparent Trajectory

In AI products, the response often seems convincing because the trajectory appears organized. The AI explains what it did, describes steps, and concludes confidently. But the explanation does not prove the result happened.

Anthropic distinguishes an agent’s execution trajectory from the effective result in the environment. A message saying the task is finished is not enough to prove the result. Evaluation uses inputs, success criteria, and verifiers, and may require multiple attempts.

Even when the product is not an agent executing actions, the distinction remains useful. The human evaluator should not approve only the AI’s narrative. They should look for evidence of the expected result.

In the fictional internal support example, the AI might say: "The ticket was classified correctly." If the product only suggests a classification, this statement is already problematic. If the product actually records something in another system, evaluation needs to check the final state, not just the message.

Useful questions for this step:

Did the AI only declare it completed something, or is there verifiable evidence?
Does the system state correspond to what the response claims?
Does the suggested action respect the permissions and limits defined for the product?
Did the user receive guidance compatible with what they can actually do?

This separation avoids a common error: confusing linguistic confidence with operational quality. AI models can produce plausible explanations. The product needs criteria that look beyond plausibility.

In software releases, this difference between making available and activating is also relevant. The Google SRE chapter on gradual releases addresses evaluating a change on a portion of traffic before expanding exposure and distinguishes making code available from activating features, including configurations that separate these decisions. For AI features, human evaluation can inform this expansion but should not be confused with automatic production approval.

Decide What Happens After Evaluation

A human evaluation that ends in an accumulated spreadsheet does not improve the product. It needs to trigger decisions.

Before reviewing cases, the team should define possible consequences. For example:

Release the behavior when critical criteria are met and divergences are explainable.
Expand the sample when there is a promising signal but still insufficient to decide.
Adjust instructions when failures concentrate on format, tone, response sequence, or explicit constraints.
Review data or sources when the AI errs due to lack of reference, conflicting reference, or incomplete context.
Restrict use when the feature works in simple cases but fails in ambiguous or higher-impact situations.
Block a case when the failure creates operational risk the team does not yet know how to control.

The decision should be proportional to the type of failure. A style preference alone should not block the product. A pleasant response that fabricates information may block a flow. A recurring divergence among evaluators may indicate the criterion needs rewriting before any conclusion.

The criteria below help decide what to do after evaluation.

Decision Criteria to Separate Taste and Quality

Sample: should be linked to a use, risk, or behavior the team needs to accept, reduce, or block. If the case was chosen only because it was easy to review, it should be included as exploration, not as main evidence.
Judged dimension: each item should indicate whether the judgment is about correctness, completeness, safety, utility, tone, or preference.
Rule to distinguish style from failure: a less elegant response may be acceptable if it completes the task safely. A pleasant response may fail if it omits constraints, fabricates facts, or guides an inappropriate action.
Application by more than one person: if competent evaluators frequently reach opposite decisions, the problem may be in the criterion, product policy, or case ambiguity.
Result verification: when the AI claims it completed something, the evaluator should look for evidence of the expected result, not just the AI’s explanation.
Consequence by failure type: without consequence, evaluation becomes an archive.

The practical decision is to build human evaluation from representative samples and explicit criteria so the team can say: this failed due to quality, this is style preference, and this requires another type of verification.

Before expanding an AI feature, the team needs to separate responses that fail due to quality, responses that annoy due to style, and cases that require other verification.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss Application in Your Company

Conversation about the software company context

Content by dooop. Registration allows relating this topic to the reader’s journey and tracking interest in the theme.