dooopSoftware · Process · 11 min

How to Conduct a Process Pilot with AI

Compare the current and AI-assisted flows using the same standards, applying small changes, predefined criteria, human review, and testing.

Published on September 6, 2026

CORE THESIS

A good pilot does not measure speed alone. It measures where costs appear when AI enters the flow.

Compare the same change before and after. Only scale if review, testing, and understanding remain strong.

An amplified development pilot starts with a verifiable hypothesis: the same unit of work can go through both the current flow and the AI-assisted flow without losing quality. The comparison must observe whether the entire process delivers a small, reviewable, testable, and understandable change, without shifting risk to review, integration, or those who will maintain the code later.

Choose a Small, Independent, and Comparable Change

The first mistake in an AI-amplified development pilot is choosing a task that works well in demonstration but poorly in comparison. If the task is too large, mixes objectives, or depends on open-ended decisions, any conclusion will be fragile. AI may seem fast in code generation while the team wastes time understanding scope, reviewing side effects, or fixing something that should not have been included in the same batch.

The pilot’s unit of work must fit within the team’s normal flow. It should have expected behavior, acceptance criteria, possible tests, and limited impact. A small change in a calculation rule, an administrative interface validation, or an adjustment to a known integration usually teaches more about the process than a broad refactoring.

The DORA guidance on small batches is a good support here: small, independent, and testable changes allow earlier feedback and hypothesis review with less coupling. The same source warns about the difficulty of reviewing and integrating large AI-generated changes. This point is decisive for the pilot. If AI increases the change size, it may be accelerating writing but worsening review.

A practical rule: choose a change that another developer can review without mentally reconstructing the entire system. If the task only works because a specific person knows shortcuts, dependencies, and undocumented exceptions, the pilot will measure improvisation, not process.

This scope also avoids overlapping this pilot’s decision with broader process organization discussions, addressed in the amplified development by AI guide. Here, the question is narrower: does the AI-assisted flow preserve quality when compared to the current flow by the same standard?

Document How the Flow Works Before AI

Before introducing AI, document the current flow with enough precision to compare. It is not necessary to map the entire company’s engineering. It is necessary to understand how that change goes from a request to integrated code.

The documentation should capture steps, responsible parties, and artifacts used. Where does the request originate? What specification exists? Who decides the scope? What files or documents does the developer consult? How is the pull request opened? Who reviews? What tests run? What blocks integration into the main code?

This map prevents unfair comparison. Without a baseline, the team tends to compare an AI-assisted execution with an optimistic memory of the previous process. Or the opposite: blame AI for problems that already existed in the flow, such as ambiguous acceptance criteria, fragile tests, or review concentrated in few people.

The documentation must also include wait times. Often, the bottleneck is not implementation but scope authorization, context discovery, review queue, or a broken build that no one prioritizes. If the pilot measures only the time between “started coding” and “opened the pull request,” it may ignore the real cost of the change.

A good map of the flow before AI answers simple questions:

What was the intention of the change?
What information was available before implementation?
What decisions were made by people and where were they recorded?
What tests were executed before integration?
What type of comments appeared in the review?
What rework arose after review or integration?

This documentation does not need to become bureaucracy. It needs to be comparable.

Design the Amplified Flow Without Removing Human Judgment

In the amplified flow, AI must have an explicit role. It can help raise hypotheses, locate relevant files, suggest tests, draft implementation, explain code snippets, or propose alternatives. But generation is not authorization. Suggestion is not decision.

The GitHub documentation on Copilot agents describes agent features with distinct environments and permissions and emphasizes human supervision and output review. The implication for the pilot is direct: define what AI can do, what it can only suggest, and what requires human decision before proceeding.

It is also worth separating AI as a support tool from AI as a flow substitute. A poor pilot delivers an ambiguous request to a tool, accepts a large change, and expects human review to fix everything at the end. A better pilot divides responsibilities:

The person responsible for the request defines the expected behavior.
The developer selects context, boundaries, and relevant files.
AI supports investigation, generation, or preliminary review.
Human review evaluates intention, readability, risk, and adherence to tests.
Continuous integration validates build and automated tests before entry into the main code.

This design connects to the theme of human review in AI-assisted flow but does not depend on a sophisticated framework to start. The pilot only needs to clarify who decides, who reviews, and which AI outputs cannot be accepted automatically.

In the pilot, record which suggestions were accepted, which were rejected, and who was responsible for the decision.

Define Quality Criteria Before Measuring Speed

Speed is seductive because it seems objective. But isolated implementation time does not measure process quality. A change can be written quickly and cost dearly in review. It can pass insufficient tests and break domain understanding. It can save minutes of coding and consume hours of explanation afterward.

Therefore, quality criteria must come before measurement. For a short pilot, use a simple standard:

Unit of work: does the change remain small, independent, and testable?
Provided context: what instructions, files, decisions, and history were made available to AI?
Human review: was the review clearer, equivalent, or more difficult?
Tests and integration: did the same applicable tests continue running and did failures block progress?
Rework: did corrections arise before or after integration?
Understanding of the change: can another developer explain the change and its limits?

Anthropic defines context engineering as selecting and maintaining information available to the model during inference, including instructions, tools, external data, and history, within a limited window. For the pilot, this means the context used by AI must be treated as part of the process, not as an invisible detail.

If a person selects files, pastes instructions, omits decisions, and then mentally corrects the output, the result may depend more on individual skill than on the flow. The pilot only becomes useful when the team can observe what context was provided, what hypothesis was accepted, and what decision remained human.

This care relates to context engineering practices for development teams, but here the criterion is operational: if the result depends on implicit knowledge that only one person had, do not scale the pilot yet.

Compare Before and After Using the Same Standard

Imagine a fictional example. A team maintains an internal order management system and chooses a small change in a discount calculation rule. The rule already exists, acceptance criteria are known, and there are automated tests for part of the behavior.

In the current flow, a developer reads the task, consults the existing rule, changes the implementation, adds or adjusts tests, opens a pull request, and waits for review. In the amplified flow, the team keeps the same task and uses AI to suggest test cases, locate probable files, and draft an implementation. The developer reviews the context, accepts or rejects suggestions, adjusts the code, and opens the pull request. In both flows, human review and continuous integration remain mandatory.

The comparison should not only ask “which was faster?”. Ask:

Did the assisted change remain the same size or grow because AI suggested out-of-scope improvements?
Did the suggested tests cover the expected behavior or only confirm the happy path?
Could human review understand the intention without long explanations from the author?
Did build and automated tests have the same blocking weight in both flows?
Did rework appear before integration or was it shifted afterward?
Could another developer explain why the rule changed, where it changed, and what was not altered?

The DORA definition of continuous integration involves frequent integration into the main code with automated build and tests. The guidance also states that fixing a broken build should have priority over new changes. In the pilot, this is a red line: if the team ignores a broken build to preserve the feeling of speed, the experiment stops measuring quality.

The same standard protects against two biases. The first is giving a discount to the AI flow because it is new. The second is demanding perfection from AI that the current flow never delivered. The pilot does not need to prove superiority. It needs to reveal where there is real amplification, where cost is shifted, and where new risk exists.

Stop the Pilot When Quality Depends on Heroism

A useful pilot also defines stop signals. Without this, the team tends to continue because the tool impresses, because someone invested energy in adoption, or because the initial demonstration was good. But a seductive demonstration is not operational value.

Stop or redesign the pilot when the change becomes too large for common review. This is a strong sign of loss of control. AI may suggest related adjustments, style improvements, name changes, reorganizations, and additional tests. Some suggestions may be good. Even so, if they enter the same batch without explicit decision, the pilot begins mixing objectives.

Another sign is inconsistent context. If AI receives different instructions each attempt, uses outdated files, or ignores relevant architectural decisions, the problem is not only the tool. It is the flow that cannot offer verifiable context. Before scaling, adjust specification, minimal documentation, and task boundaries.

Also stop when fragile tests become an excuse for heroic review. Human review is indispensable but should not be the only quality mechanism. When the reviewer must discover expected behavior, validate scope, interpret opaque code, and compensate for missing tests, the pilot is only transferring cost.

Other clear signs:

Review becomes formality because “it was generated by AI.”
Broken build stops blocking new changes.
Gains appear in implementation but rework increases afterward.
The team cannot explain the change without consulting the original author.
AI begins deciding architecture, security, product behavior, or use of sensitive data without explicit human authorization.

In these cases, insisting on the pilot does not increase maturity. It only normalizes exceptions.

Turn the Result into a Process Decision

The pilot’s final product is not a declaration of love or rejection of AI. It is a process decision. At the end of the comparison, technical leadership should be able to choose among three options:

Maintain the current flow and not expand AI use in that task class. This may be the correct decision if implementation gain is small, review becomes harder, or rework only shifts location.
Expand AI use in specific steps. For example, use AI to suggest tests, explain legacy code, or raise impact hypotheses, keeping implementation and scope decisions under closer human guidance. This decision makes sense when AI improves part of the flow without degrading review, integration, and understanding.
Redesign part of the process. Perhaps the pilot reveals that the problem is not AI but lack of specification, unrecorded decisions, slow tests, or concentrated review. In this case, the next step is not buying more tools or banning tools. It is adjusting the work system.

To conclude, use this checklist as a decision criterion:

If AI increases the change size or mixes objectives, do not scale.
If the result depends on implicit context, improve context before repeating.
If review becomes opaque, reduce batch size and make decisions explicit.
If tests and integration lose blocking strength, invalidate the comparison.
If rework appears afterward, do not count fast implementation as gain.
If another developer cannot sustain the change, the flow has not yet preserved quality.

An amplified development pilot should only advance when the comparison shows where gains occurred, where cost shifted, and which conditions must remain. Start with a small change class, document the current flow, run the AI-assisted flow with explicit human judgment, and compare quality before talking about scale.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

To Continue This Reading

How to Coordinate Parallel Tasks of Code Agents

NEXT DECISION

Discuss Application in the Company

Discussion about the software company context

Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.