dooopSoftware · Quality · 11 min

How to Create a Baseline Before Adopting AI

Record tasks, time with context, quality, and completion before the pilot to compare AI gains without confusing speed with rework.

Published on September 6, 2026

CORE THESIS

Without a baseline, gains become impressions. The comparison must exist before the tool changes the workflow.

The AI pilot begins by measuring current work. Only then do speed, rework, and quality enter the same decision.

A productivity baseline with artificial intelligence serves to record how work happens before the change. It does not prove that AI will work. It creates a comparison. Before releasing a tool, a copilot, or an agent in the development flow, the team needs to record current work in a comparable way. Without this, any perceived improvement may be real gain, noise, risk shifting, or displaced rework.

Why measuring afterward does not answer what changed

The scene is common. The team starts using AI to write tests, generate code snippets, summarize issues, or propose fixes. Some people say they are faster. Others notice longer reviews, doubts about acceptance, or defects that slipped through. Leadership asks: did it improve?

If no one recorded the previous state, the answer becomes an opinion organized in a meeting.

A baseline before AI is a comparable snapshot of work before intervention. It needs to show the type of task, execution conditions, evidence of completion, and quality signals that will be observed afterward. The goal is not to freeze the team in an old process. It is to prevent the comparison from mixing different things.

Measuring only after adoption usually creates three confusions:

The team compares tasks of different complexity as if they were equivalent.
Time saved in implementation reappears in review, testing, or manual correction.
The tool declares a task completed, but the result has not yet been verified in the real environment.

The baseline reduces this ambiguity. It does not eliminate judgment. On the contrary, it gives human judgment a better foundation.

This care aligns with a larger quality decision. In the guide Software Quality with AI: Testing, Evaluation, and Responsibility, the central question is how to maintain responsibility when systems start generating, suggesting, or executing parts of the work. Here, the focus is narrower: how to compare before and after without self-deception.

Minimum variables for comparison before AI

A productivity baseline with AI does not need to become an exhaustive engineering inventory. If data collection is too heavy, it disrupts the very flow it should observe. The best starting point is to record only what will be compared later.

The minimum set should include:

Task type, such as bug fix, interface adjustment, test creation, small refactoring, or endpoint improvement.
Task input, such as issue, specification, bug report, previous pull request, or operational alert.
Expected output, such as reviewed code, passing test, accepted behavior, or updated documentation.
Approximate complexity, described by simple criteria used by the team itself.
People involved, mainly who executes, reviews, and validates.
Execution time with context, separating execution, waiting for review, relevant interruptions, and identified rework.
Quality signals, including reviews, tests performed, defects found, regressions, and subsequent manual corrections.
Verifiable completion criteria, such as passing automated test, recorded acceptance, or reviewed change.

The most sensitive point is separating gross time from useful time for decision. A task may seem fast because implementation was accelerated but require more questions in review. Another may seem slow because it was waiting for validation, not because the technical execution was difficult.

The February 2026 update of METR is a useful reference for caution without extrapolating its findings. The organization treated new data as an unreliable signal about AI's current effect on productivity and pointed out difficulties such as participant and task selection, as well as measuring time with competing agents. For leadership, the applicable lesson is simple: how tasks are chosen and time is measured can distort the reading.

How to choose comparable tasks without biasing the reading

Task selection defines the honesty of the comparison. If the team selects only activities where AI tends to shine, the pilot starts overly optimistic. If it chooses only complex, urgent, or poorly specified exceptions, it starts overly pessimistic.

The first step is not to formulate a hypothesis. It is to discover work patterns.

Discovery means observing which tasks appear frequently, which have minimally similar input, and which generate clear evidence of completion. Only then comes the hypothesis, for example: AI can reduce implementation effort in repetitive fixes without increasing review rework.

Good tasks for the baseline usually have some characteristics:

They are recurring in the team's flow.
They have relatively recognizable input and output.
They do not depend on a single person with excessive tacit knowledge.
They allow verifying results by review, test, acceptance, or behavior observation.
They represent real work, not an isolated demonstration.

Bad tasks to start with are those that change nature with each occurrence, depend on strategic decisions, require negotiation with multiple areas, or have quality difficult to observe in the short term. They can be evaluated later but tend to generate noise in the first comparison.

After recording the previous state, the next step is to decide how to measure developer productivity with AI. Measuring productivity is a broader discussion. Creating the baseline is the prior step: choosing what will be comparable before the tool alters the team's behavior.

Observable quality beyond speed

AI adoption in software development is often sold on the feeling of speed. Leadership, however, needs to ask where the effort went.

If a tool generates code faster but increases review time, creates fragile tests, or shifts fixes to after deployment, apparent productivity is not enough. The baseline needs to record observable quality before the change.

Observable quality does not mean measuring everything. It means choosing signals that allow comparing side effects. Some examples:

Quantity and type of comments in code review.
Tests executed before completion.
Defects found during review or validation.
Task reopenings due to incorrect behavior.
Manual corrections made after automatic suggestion or generation.
Recurring doubts about acceptance criteria.

DORA recommends testing throughout development, combining automation and manual activities such as exploration and usability. It also recommends maintaining and reviewing test suites instead of treating quality as a post-development step. For a baseline, this reinforces that quality should not appear only at the end, when the cost of understanding what changed has already increased.

A fictional example helps. A team decides to observe low and medium complexity bug fixes before allowing AI in patch generation. For several weeks, it records diagnosis time, implementation time, reviews, tests executed, reopened defects, and manual corrections after review. In the pilot, the comparison only considers bugs of the same type and maintains the same done criteria.

Nothing in this example states that AI improved or worsened the flow. The value is in making the hypothesis measurable. If implementation seems faster, the team will be able to verify if the gain came with more review, more rework, or stable quality.

How to compare human work, assistive tools, and agents

Not all AI adoption carries the same risk. Putting all tools in the same comparison is a quick way to lose clarity.

There are at least three different usage modes:

AI as a point suggestion, when the person consults the tool and decides whether to use or discard the response.
AI as an execution copilot, when the tool generates relevant artifacts such as code, tests, documentation, or queries.
Agent, when the system executes a sequence of actions and can modify files, trigger tools, or interact with the environment.

The baseline should record the degree of autonomy because the trust criterion changes. In a point suggestion, the person remains the direct executor. In an agent, evaluation needs to confirm the result in the environment, not just the tool's narrative.

Anthropic distinguishes the agent's execution trajectory from the effective result in the environment. A message saying the task is finished is not enough to prove the objective was achieved. Evaluation uses inputs, success criteria, and verifiers, and may require multiple attempts.

For engineering leadership, this changes baseline recording. It is not enough to note that the agent responded well. It is necessary to record which evidence confirms completion: passing test, reviewed pull request, correctly changed file, validated behavior, or recorded acceptance.

When AI triggers real changes, the comparison should separate making code available from activating functionality. The canary releases chapter of Google SRE addresses evaluating a change on a portion of traffic before expanding exposure and distinguishes making code available from activating features by configuration. This separation is useful when the team wants to observe a change before expanding its impact.

Done criteria to start the pilot

Use this checklist as done or not done criteria to start an AI pilot in a development flow.

Comparable tasks

Done when there are at least some recurring task types described with input, expected output, approximate complexity, and responsible for validation.

If missing, the comparison after AI may mix easy, urgent, or incomparable tasks.

Time measured with context

Done when the record separates execution time, waiting for review, relevant interruptions, and identified rework.

If missing, the team may attribute gain or loss to AI when variation came from workflow.

Observable quality

Done when the baseline includes code review, tests executed, defects found, regressions, and subsequent manual corrections.

If missing, adoption may seem faster while shifting cost to review, support, or correction.

Completion criteria

Done when each task has verifiable evidence of completion in the work environment, such as passing test, recorded acceptance, or reviewed change.

If missing, a completion statement by the tool may be confused with effective result.

Degree of AI autonomy

Done when the record indicates whether AI only suggested, generated artifacts, executed steps, or triggered environment changes.

If missing, tools with different risk levels enter the same comparison.

Stable comparison conditions

Done when the team identifies relevant simultaneous changes, such as process change, team change, new testing standard, or exceptional deadline pressure.

If missing, pilot results may reflect organizational changes, not AI adoption.

Decision before data collection

Done when it is defined what will happen if the result is positive, inconclusive, or negative: expand, repeat, restrict, or stop.

If missing, measurement becomes a retrospective argument for an already made decision.

This last item is uncomfortable because it forces leadership to declare beforehand what evidence it will accept. But this is precisely where the baseline stops being bureaucracy and becomes practical governance.

When the baseline is still not reliable to decide scale

A baseline improves comparison but does not turn a small pilot into a general truth. It helps decide the next step with less noise.

There are situations where the best decision is not to scale yet:

The observed sample is too small to support broad change.
Tasks were chosen for convenience and do not represent the real flow.
The team changed process, tool, people, and goal simultaneously.
Time was measured without separating wait, interruption, and rework.
Quality was assessed only by perception.
The agent declared completion without external verification of the result.
The pilot had a positive result in speed but inconclusive in quality.

In these cases, leadership does not need to reject AI. It can restrict use, repeat comparison, choose another task type, or redesign evaluation. The useful decision is knowing what kind of evidence is missing to expand, restrict, repeat, or stop use.

A team can allow assistive use in low-risk tasks and keep agents restricted to controlled environments. It can release test generation but require explicit human review before accepting changes in production code.

Before adopting AI in a development flow, the question is not only which tool seems better. It is what evidence leadership will accept to expand, restrict, repeat, or stop use. If comparable tasks, time with context, observable quality, degree of autonomy, and evidence of completion do not yet exist, the next decision is to create the baseline and run a limited comparison.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss application in the company

Conversation about the software company context

Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.