dooopSoftware · Process · 12 min

Retrospective with AI: How to Create Small Experiments

Use the retrospective to transform development frictions with AI into testable hypotheses, clear boundaries and review criteria.

Published on September 6, 2026

CENTRAL THESIS

The retrospective does not judge AI as a whole. It selects a friction in the flow to test in the next cycle.

Recurring problems become small hypotheses. Boundaries and review prevent local learning from becoming dogma.

An AI development retrospective should not decide whether the tool "worked." That question is too broad to guide the next sprint. The best use of the meeting is to turn observed frictions in the flow into a small experiment: a concrete problem, a testable hypothesis, a limited process change and a review criterion.

Start with flow signals, not opinions about AI

The sprint ends and the signals seem contradictory. One task progressed faster because AI helped explore alternatives. Another got stuck in review because the generated code spread changes across unexpected files. In a third case, the agent followed an old instruction because the context was incomplete. The conversation easily turns into a generic judgment: "AI helped," "AI hindered," "the review got worse."

This type of evaluation produces little learning. The AI retrospective needs to replace tool opinion with work evidence. Where was there waiting? Where was there rework? What was difficult to review? What broke in integration? Which scope decision arrived late? Which context had to be reconstructed by someone?

The question changes quality when it moves from preference to flow. Instead of asking whether the team should use more or less AI, ask at which step the assisted work lost predictability.

Some useful signals to start the conversation:

Pull requests with changes outside the agreed scope.
Reviews that required reconstructing the task's intent.
Builds broken by changes that could have been isolated.
Tasks started without a clear stopping criterion.
Context scattered among issue, chat, documentation and code comments.
Recurring doubts about who authorizes exceptions.

This focus relates to a larger process decision. In the guide on AI-augmented development, the central question is organizing work so technology amplifies capacity, not just volume. In the retrospective, this idea appears more practically: choose a real friction and learn from it in the next cycle.

Separate incident, pattern and hypothesis before proposing a solution

A weak retrospective jumps too quickly to solutions. Someone reports a problem, another suggests a rule, the team agrees to close the discussion and in the next sprint no one knows if the rule solved anything. With AI, this risk increases because the tools are new enough to be blamed for problems that already existed in the process.

A simple way to avoid this is to separate three layers:

Incident: an isolated case in a specific task. It may have occurred due to haste, ambiguity, communication failure or a rare combination of conditions. It deserves recording but not always an experiment.
Pattern: repetition of the same type of friction in similar tasks, similar modules or common flow steps. Here there is material for investigation.
Hypothesis: a testable explanation for the pattern. It is not certainty. It is a small bet enough to be verified in the next cycle.

Fictitious example: in an internal product team, three AI-assisted tasks reach review with changes in unexpected files. The first reaction is to say "the agent changes too much." As an incident, this is just a complaint. As a pattern, the team notices the three tasks started without an explicit list of allowed and forbidden files. As a hypothesis, they formulate: when an AI-assisted task starts without file limits, the chance of out-of-scope changes increases and review becomes harder.

Notice the difference. The team is not saying all AI use causes rework. Nor are they creating a broad policy. They are formulating a hypothesis about part of the flow.

This care is especially relevant when the team works with programming agents. GitHub's documentation on Copilot agents describes features with distinct environments and permissions and emphasizes human supervision and output review. This statement does not solve your process design but reinforces an operational point: assisted work must remain understandable to those responsible for decisions.

Turn recurring problems into small experiments

An AI development retrospective gains value when it ends with a process experiment, not a list of intentions. An experiment is a limited change in how work is prepared, executed, reviewed or integrated. It must fit in the next iteration and be small enough to be abandoned without drama.

DORA recommends small, independent and testable work units to get feedback on changes and review hypotheses earlier. The same guidance warns about the difficulty of reviewing and integrating large AI-generated changes. In the retrospective, this suggests a practical criterion: if the proposed improvement requires changing the entire process, it is probably too large to become a sprint experiment.

A good experiment has seven components:

Observed problem: what happened in the flow, with a concrete example.
Pattern: why the team believes it was not just an isolated case.
Hypothesis: which probable cause will be tested.
Small change: which part of the process will be altered.
Application boundary: where the test applies and where it does not.
Success criterion: which signals will be observed.
Responsible and review: who monitors and when the team decides to keep, adjust or discard.

Fictitious example: a team notices AI-generated code reviews are tiring because pull requests arrive with auxiliary changes mixed with the main fix. The hypothesis is that assisted tasks without stopping criteria encourage opportunistic changes. The experiment for the next sprint is to require, before code generation, a short statement with the task objective, allowed files, forbidden files and stopping condition. The test applies only to small fixes in a known module. Refactorings and critical areas are excluded. The success criterion is qualitative and operational: reviewers should identify the change intent without reconstructing reasoning in side conversations, and the team should observe if out-of-scope changes decrease.

None of this proves in advance that the practice will work. That is exactly why it is an experiment.

Define what is included and excluded from the experiment

Many process improvements fail due to excessive ambition. The team identifies a real problem and tries to solve it for all repositories, all task types and all people at once. The result is a rule hard to apply and easy to ignore.

In AI-assisted development, boundaries matter even more because tasks vary greatly. The team can treat different change types with different boundaries, making explicit where the experiment applies and where it does not. Agent flows should also specify environment, permissions and authorization points. GitHub documentation describes agents with distinct environments and permissions and emphasizes human supervision and output review.

Therefore, every experiment must explicitly state what is inside and outside.

Useful scoping criteria:

Task type: fix, small improvement, test adjustment, documentation, investigation or refactoring.
System area: known module, peripheral service, shared library or critical component.
Reversibility degree: change easy to undo or change with broad dependencies.
Execution permission: local suggestion, patch generation, pull request opening or automated environment action.
Exception authority: who can authorize going beyond the agreed boundary.

If the experiment applies to everything, any conclusion becomes confusing. If it applies to a clear set of tasks, the team can discuss concrete signals in the next retrospective.

It is also worth recognizing human limits. Not every friction should become automation. Some tasks still require human guidance from the start because they involve product ambiguity, architectural risk, operational impact or cross-area negotiation. In these cases, the retrospective may conclude the best process change is to restrict AI use at a certain step, not expand it.

Include human review and integration in the success criterion

A bad experiment measures only the feeling of speed. "It seemed faster" can be an initial signal but is not enough. If the team saves time generating but spends more on review, integration or later correction, the flow has not improved. It just shifted the cost.

The success criterion should include human review, testing and integration. DORA describes continuous integration as frequent integration into the main codebase, accompanied by automated build and tests, and states that fixing a broken build should have priority over new changes. For the retrospective, the practical implication is simple: an AI-assisted practice should not encourage accumulation of changes difficult to integrate.

Human review is not a stamp at the end either. It must be designed into the flow. Reviewers must understand the change intent, the limits given to the agent, the files that should remain untouched and the reason for any exceptions. If review depends on guessing the original prompt or reconstructing scattered conversations, the problem may not be the reviewer. It may be task preparation.

Context engineering helps name this issue. Anthropic defines context engineering as selecting and maintaining information available to the model during inference, including instructions, tools, external data and history, within a limited window. In retrospective terms, "context was missing" should not remain a generic phrase. The team needs to ask which context was missing, where it should have been and who was responsible for maintaining it.

Good success criteria may observe:

Whether the change intent was clear to the reviewer.
Whether there was less scope discussion during review.
Whether auxiliary changes were separated from the main change.
Whether relevant automated tests continued running.
Whether integration avoided accumulating batches too large to review safely.
Whether exceptions were recorded instead of treated as invisible improvisation.

These criteria should fit into the ritual the team already uses and be visible enough to guide a decision in the next cycle.

Record learning as a provisional rule, not dogma

The retrospective closing is where many teams waste learning. They discuss good points, choose vague actions and leave without a review mechanism. In AI development, this creates two opposite risks: abandoning a promising practice too early or turning a local solution into permanent dogma.

The record should be short and specific. It does not need to become an extensive document. It must allow someone to understand what was tested, in which context, with what boundaries and which signals appeared.

A good record answers:

What flow problem motivated the experiment?
What hypothesis was tested?
What process change was applied?
In which tasks or modules did it apply?
What did the team observe in preparation, review and integration?
Will the practice be kept, adjusted or discarded?
What situation remains outside this rule's scope?

This last question is the most neglected. Every provisional rule has a boundary. If the experiment worked on small fixes, that does not authorize automatic application to extensive refactorings. If it helped a team with a well-documented repository, it does not prove it will serve for legacy with scattered context. If it reduced scope doubts, it does not mean it solved architectural risk.

The record is useful when the team can say: it worked here, by these signals, with these boundaries. This phrase is worth more than a grand policy no one knows how to apply.

To connect the retrospective with the broader process mesh, it is worth relating learnings with practices already discussed in related topics, such as AI strategy connected to business, AI maturity and leadership and augmented human. The retrospective does not replace these decisions. It creates a local learning mechanism so adoption does not depend only on enthusiasm, fear or individual preference.

Retrospective script to test a flow improvement

Use this script when the team is about to turn a complaint into action. It helps keep the conversation at the right level: neither too abstract nor too operational to forget learning.

Observed problem

Was the problem described as a flow event, with a concrete example, and not as a general impression about AI?

Fictitious example: in four sprint pull requests, review required more effort because generated code changed files outside the agreed scope. This example does not state a real result. It serves to show the type of evidence the team should seek.

Pattern or isolated case

Does the team know if there is enough repetition to justify an experiment?

If it happened once, it may be just a record. If it appeared in similar task types, it can become a hypothesis. The difference avoids creating rules for exceptions.

Testable hypothesis

Can the team formulate a probable cause that can be tested in the next iteration?

Fictitious hypothesis: when an AI-assisted task starts without an explicit list of allowed files, the chance of out-of-scope changes increases. The hypothesis may be wrong. The value is in being able to verify it.

Small process change

Does the experiment change a specific part of the flow without redesigning the entire development process?

Fictitious example: during a sprint, every AI-assisted task in small fixes must declare allowed files, forbidden files and stopping criteria before code generation.

Application boundary

Is it clear in which tasks, modules or repositories the experiment applies?

Applying only to small fixes in a known module may be sufficient. Refactorings, critical areas and architectural decisions may be excluded until there is more solid learning.

Success criterion

Has the team defined signals that allow deciding to keep, adjust or discard the practice?

Signals may include out-of-scope changes, more straightforward review and maintenance of test and integration controls. The point is not to declare victory. It is to have a basis for the next adjustment.

Responsible and review

Is there a person responsible for monitoring the experiment and a review moment in the next cycle?

Without a responsible person, the experiment becomes intention. Without review, it becomes a rule by inertia.

An AI development retrospective works best when it ends with a small, verifiable choice: which observed problem will be tested in the next cycle, with which hypothesis, within which boundary and with which review criterion.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss application in the company

Conversation about the software company context

Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.