dooopSoftware · Quality · 13 min

How to Measure Developer Productivity with AI

Compare equivalent tasks, record time until reviewable delivery, review and rework, and only count gains when acceptance becomes cheaper.

Published on September 6, 2026

MAIN THESIS

AI can speed up the initial delivery and increase the cost of acceptance. The metric must follow the work through review and rework.

Compare equivalent tasks. Count gains only when total time, review, and rework improve without shifting costs.

Measuring developer productivity with artificial intelligence requires comparing similar tasks and looking beyond the first delivery. The question is not whether AI wrote code faster, but whether the work reached acceptance with less total time, without pushing costs into review, rework, or support. A good start is to record three dimensions per task type: time until reviewable delivery, review effort, and changes needed until meeting the agreed criterion.

Why Productivity with AI Seems Higher Before Being Measured

The initial perception is often strong. The developer describes the change, the tool suggests a solution, files appear modified, and the first version arrives sooner. For those observing the flow from afar, this seems like productivity.

But productivity in engineering does not end when code exists. It ends when the change advances the system in a verifiable way, within a standard accepted by the team. Between the first version and this point come questions that generation speed does not answer: Does the code cover the right case? Do the tests protect the expected behavior? Does the solution respect previous architectural decisions? Was the review easier or harder?

This is where measurement needs to be less seduced by demonstration and more useful for decision-making. Lines of code, number of commits, and perceived speed can show activity but do not show validated progress. A flow with AI can reduce initial implementation and increase review effort. It can also do the opposite. Without measuring the complete path to acceptance, leadership ends up debating impressions.

This distinction aligns with a recurring idea in software quality: quality should not appear only at the end. The DORA recommends testing throughout development, combining automation and manual activities such as exploration and usability, and also recommends maintaining and reviewing test suites. To measure developer productivity with AI, this matters because real gain must survive the verification process, not just the generation step.

What to Compare: Equivalent Tasks, Not Generic Averages

The fastest way to measure poorly is to mix different tasks and calculate a comfortable average. A simple interface fix should not be compared with a domain rule change full of dependencies. A test adjustment is not the same nature as a product evolution. A change in a stable system area is not equivalent to modifying a little-known module.

Useful comparison starts with groups of equivalent tasks. They do not need to be identical because real development rarely offers perfect clones. But they must be similar enough to support a practical decision.

Some criteria help:

Type of work: bug fix with bug fix, small evolution with small evolution, test adjustment with test adjustment.
System area: compare tasks within similar parts of the product, with known patterns and dependencies.
Perceived complexity: separate simple, moderate, and uncertain changes without mixing everything into a single average.
External dependencies: consider whether the task required product decision, response from another team, environment change, or API adjustment.
Acceptance criterion: only compare tasks that had a minimum definition of what would be considered complete.

This separation avoids a common trap: attributing to AI a difference that came from the task type. If activities with AI were smaller, more repetitive, or better specified, the comparison is biased. If tasks without AI were more ambiguous, the result also says little.

The February 2026 update of METR is a good reminder of caution. The organization considers the new data an unreliable signal of AI's current effect on productivity and points out difficulties such as participant and task selection, as well as problems measuring time with concurrent agents. This does not decide what will happen in your team. But it reinforces that measuring productivity with AI requires careful design, especially in choosing compared tasks.

Three Minimum Measures: Time, Review, and Rework

For engineering, quality, or product leadership, measurement must fit into the flow. If it depends on heavy apparatus, the team abandons recording or starts working for the metric. The minimum set must answer one question: did AI reduce the total cost of taking comparable tasks to reliable acceptance?

The first measure is time until reviewable delivery. It goes from the effective start of the task until the moment another person could review the change. This metric captures initial speed. It is useful but incomplete. A reviewable delivery can still be wrong, fragile, or hard to understand.

The second measure is review time. This includes reading, comments, questions, manual tests, local validations, and discussions necessary to accept or reject the change. If AI speeds up the author but makes review longer, the gain may have just shifted to someone else. In pressured teams, this cost appears as queue, interruption, or superficial review.

The third measure is rework until acceptance. Rework is everything that needs to change after the first delivery to meet the agreed criterion. It can be functional correction, test adjustment, code simplification, removal of unnecessary dependency, forgotten error handling, or alignment with technical standard.

These three measures avoid shallow reading. One task may have a fast initial delivery and high rework. Another may start slower and go through review with fewer adjustments. The decision does not come from preferring one of these stages in isolation but from understanding where effort shifts.

This approach also relates to the boundary between measuring productivity and discussing quality. To deepen validation mechanisms, consult the guide on software quality with AI, without turning every productivity measurement into a full discussion about testing, review, or agent evaluation.

How to Record Data Without Turning the Team into a Laboratory

Recording must be light enough to happen in real work. The team does not need to pretend scientific precision if the goal is operational decision. It needs consistency, honesty, and context.

A good record per task can contain:

Task type.
System area.
Brief scope.
AI use: yes or no.
Approximate time until reviewable delivery.
Number of review cycles.
Main causes of rework.
Final decision: accepted, returned, split, suspended, or redone.

The most delicate point is recording cause, not just volume. If rework came from ambiguity in the request, blaming AI is analytical laziness. If it came from insufficient testing suggested by the tool, ignoring AI's role is also lazy. If review grew because the reviewer did not know the area, the metric is capturing a distributed knowledge problem, not necessarily assisted generation.

Leadership should treat these data as signals for conversation, not as individual scoreboards. Measuring developer productivity with AI per person, without task context, encourages defensive behavior. Measuring by task type helps decide where AI enters the standard flow, where it remains optional, and where it should be restricted.

There is a phrase worth keeping on the wall, even if it does not become a formal rule: if AI reduces visible effort of the deliverer and increases invisible effort of the validator, system productivity may not have improved.

Fictional Example: Two Similar Fixes, Different Results

Imagine a fictional example in an e-commerce product. The team needs to fix two form validations in similar system areas. Both tasks have small scope, depend on known components, and have similar acceptance criteria: prevent invalid entries, preserve existing error messages, and keep relevant tests passing.

In the first task, the developer uses AI to generate the initial change. The reviewable delivery appears quickly. In review, however, doubts arise: one error case was not covered, a message changed unnecessarily, and the created test verifies behavior in a fragile way. The task returns for adjustment. After a new round, another point appears: the solution handled the common case but did not preserve an existing variation in the component.

In the second task, without AI, the first delivery takes longer. The developer consults the pattern used in previous validations, changes fewer files, and creates a test closer to expected behavior. Review still finds adjustments, but they are smaller and do not change the solution's direction.

This example does not prove AI hinders. It also does not prove working without AI is better. It shows why the right question is not "which first version came out sooner?" The question is: in comparable tasks, which path led to acceptance with lower total cost and acceptable risk?

The opposite could also happen. In a repetitive, well-specified task with a clear pattern, AI can speed up the first version without increasing review or rework. The criterion is the same: measure the path to acceptance, not defend a prior preference.

When Apparent Gain Should Not Guide the Decision

There are situations where measurement should be read with caution. Not because the data are useless, but because they may be answering another question.

The first is task mixing. If the AI group concentrates simple adjustments and the non-AI group concentrates ambiguous changes, the comparison is compromised. The second is the learning curve. In the first weeks, part of the time measures adaptation to the tool, not stabilized flow. This can artificially worsen or improve perception.

The third appears when agents perform concurrent actions. If a tool opens files, changes sections, runs commands, and reports completion, measuring time becomes harder. Also, a message saying it finished does not prove the result. Anthropic distinguishes the agent's execution trajectory from the effective result in the environment and describes evaluations with inputs, success criteria, and verifiers. For productivity, the practical implication is simple: only count as completed the task whose effect was verified.

The fourth situation is weak acceptance criteria. When no one knows exactly what would be accepted, any measurement becomes a narrative dispute. AI can produce a plausible solution for a poor request, and review can become late scope discovery. This is not a good productivity measure. It is a symptom of insufficient definition.

The fifth is simultaneous change of tool, process, or deadline pressure. If the team changed library, altered review policy, and adopted AI in the same window, isolating AI's effect becomes unlikely. In this case, the best decision may be to reduce ambition of conclusion: observe signals by task type, not declare overall gain or loss.

There is also a boundary with feature rollout and exposure. The Google SRE chapter on canarying deals with evaluating a change on a portion of traffic before expanding exposure and distinguishes making code available from activating features. Although this topic is not an individual productivity metric, it reminds that delivering code is not the same as releasing impact to all users.

Checklist to Measure Developer Productivity with AI

This checklist is a practical proposal for engineering decision. It is not a conclusion of consulted sources.

Task Comparability

Does the compared task have similar work type, system area, dependency level, and acceptance criterion? If not, do not use the comparison to conclude productivity gain or loss.

Time Until Reviewable Delivery

How much time passed between starting the task and delivering something another person could review? Use this measure to assess initial speed, not complete productivity.

Review Time

How much effort was spent understanding, questioning, testing, or correcting the delivery? If review grows significantly, AI may be shifting work from author to reviewer.

Rework Until Acceptance

How many changes were needed after the first delivery to meet the agreed criterion? Recurring rework indicates generation sped up the first version but not necessarily the final delivery.

Result in the Environment

Did the task actually produce the expected effect in the system, not just a message or description of completion? Only count as completed when the result can be verified.

Cause of Adjustment

Did rework come from functional error, insufficient test coverage, product decision, technical standard, or ambiguity in the request? Separating cause avoids blaming or celebrating AI for scope, review, or process problems.

Decision About the Flow

Did AI reduce total time without significantly increasing risk, review, or rework? If yes, consider expanding for this task type. If not, restrict, redesign instructions, or keep use as a controlled experiment.

Criterion to Decide if AI Should Enter the Standard Flow

The most useful decision is not to authorize or prohibit AI generically. It is to define in which task types it enters as standard practice, where it remains optional support, and where it requires additional care.

Advance when comparable tasks show reduction or stability of total time until acceptance, without relevant increase in review and rework. Restrict when savings in implementation migrate to reviewers, quality, or support. Reassess when data are contaminated by different tasks, weak criteria, or simultaneous process changes.

Leadership should turn this result into usage rules by task type, with periodic review. For organizations structuring AI adoption beyond engineering, the article on how to create an AI strategy connected to business helps link technical choices to organizational priorities. To diagnose the starting point, the AI maturity guide can support conversation without replacing local measurement.

For the next cycle, choose a comparable task type, record time until reviewable delivery, review time, and rework until acceptance, and only expand use when gain is not just effort displacement.

If you want to discuss this decision in your company context, talk to dooop.

Sources

To Continue This Reading

NEXT DECISION

Discuss Application in the Company

Conversation about the software company context

Content by dooop. Registration allows relating this topic to the reader's journey and tracking interest in the subject.