dooopSoftware · Strategy · 12 min

How to Choose an AI Pilot Project in Software

Choose AI pilots based on testable hypotheses, reversible scope, observable metrics, and criteria to continue, adjust, or stop.

Published on September 6, 2026

CORE THESIS

A good pilot is not the flashiest. It is the one that can fail early without causing operational damage.

The choice starts with the uncertainty to reduce. Reversible scope and exit criteria come before the tool.

Choosing an artificial intelligence pilot project in software means separating a test from a bet. Before the most interesting tool, comes the uncertainty that leadership needs to reduce now. A good pilot is a testable hypothesis, linked to a real product or operational choice, with reversible scope, observable metric, and clear criteria to continue, adjust, or stop.

If the idea cannot fail without breaking the product, it is probably not a pilot yet. It is a bet.

The AI pilot starts with the right uncertainty

The pressure to start using AI often produces a long list of possibilities: automated customer service, report generation, ticket analysis, roadmap prioritization, action recommendations for clients, internal copilots, new product interfaces.

The list can be useful but does not solve the choice.

A better question is: which uncertainty will be reduced after the pilot?

In a software company, an AI pilot should help decide something concrete. For example: is it worth incorporating an assisted experience into the product? Does it make sense to automate part of an internal workflow? Can the team supervise AI-generated suggestions without creating a new rework queue? Do the available data support a limited intelligent feature?

This separates a pilot from a broad initiative. A broad initiative tries to capture operational or commercial value at scale. A pilot tries to produce reliable learning with contained risk.

The DORA 2025 report describes AI as an amplifier of existing organizational strengths and weaknesses and highlights the importance of the organizational system for return on investment. This reading matters for choosing a pilot because technology does not operate in a vacuum. It enters existing processes, responsibilities, data, decision rituals, and trust boundaries.

If the current process is confusing, the pilot can amplify the confusion. If responsibility for review is unclear, AI tends to make this gap more visible. If the chosen metric does not relate to a decision, the pilot may produce a nice demonstration but no useful learning.

Therefore, the first choice is not technical. It is managerial: which uncertainty deserves to be reduced before spending energy on broad implementation?

Turn the idea into a hypothesis that can fail

An idea like “using AI in customer service” is too broad to be a good pilot. It mixes user, task, technology, process, expectation, and outcome in one sentence.

A testable hypothesis needs to be more precise. It should indicate:

who uses or is affected by the solution;
which task will be modified;
what change is expected to be observed;
what minimum evidence will help decide the next step;
which risks need to be contained during the test.

The formulation changes significantly when you move from intention to hypothesis.

Instead of “using AI in support,” a better hypothesis would be: “an assistant that suggests answers for recurring questions can reduce rework by the support team without increasing manual review effort.”

Note that this hypothesis can fail. Perhaps the suggestions are superficial. Perhaps the team spends more time reviewing than writing from scratch. Perhaps recurring questions are not well classified. Perhaps the gain is in standardizing answers, not speeding up service.

That is the point. A pilot that cannot fail on paper is probably being sold as a certainty before being tested.

Microsoft Research describes its ExP platform as a way to incorporate experimentation into the development cycle, validate hypotheses, measure impact, and iterate products. The applicable lesson here is not to copy a specific platform. It is to treat the pilot as a product or operation experiment: a hypothesis goes in, evidence is recorded, and the next step is decided based on it.

For companies still organizing their AI agenda, this reasoning aligns with building an AI roadmap, but on a smaller scale. The roadmap organizes fronts. The pilot chooses a hypothesis that can be tested without compromising the entire system.

Choose a reversible scope before choosing the tool

Reversibility is the ability to turn off, replace, limit, or revert to the previous process without affecting the product core or the customer relationship.

This criterion may seem conservative but it allows learning with more freedom. When the pilot is reversible, leadership can authorize the test without turning any error into a crisis. When it is not reversible, the organization tends to discuss the tool as if deciding the entire product’s future.

A reversible scope usually has some characteristics:

low dependence on sensitive data;
limited impact on customers or end users;
human alternative or current process available;
simple integration to remove or isolate;
defined human review for exceptions;
clear responsible person to monitor risk and quality;
explicit exposure limit during the test.

This does not mean every pilot needs to be internal. A pilot can involve real users if the error is contained, expectations are well designed, and review responsibility is clear. The problem is exposing a sensitive decision directly to the customer without containment and calling it learning.

It also does not mean AI maturity requires owning the model. Mature products can integrate third-party models, provided process design, governance, measurement, and usage limits are clear. The issue is not owning the technology. It is knowing where it fits, who is responsible for it, and how the organization learns from its use.

To diagnose whether the company has a basis for this type of decision, it is worth separating the pilot from the broader assessment of AI maturity. Maturity looks at organizational capabilities. The pilot tests a delimited hypothesis.

Separate convincing demonstration from operational learning

An AI demonstration can impress in minutes. It can summarize a ticket, generate a plausible response, classify a request, or produce an apparently sophisticated analysis.

A good demonstration can still hide review costs, exceptions, and impact on the real workflow.

Real use includes ambiguous cases, incomplete data, exceptions, review, correction, team confidence, responsibilities, and workflow impact. The question is not just “did AI get this example right?” It is “did the process improve when AI was introduced, considering supervision, error, review, and decision?”

The February 2026 update from METR considers new data an unreliable signal of AI’s current effect on productivity and points out difficulties such as participant and task selection and measuring time with competing agents. This source does not allow concluding that AI increases or decreases productivity in any company. It reinforces caution in managing the pilot: measuring productivity in AI scenarios can be difficult, and the pilot should not become a general proof of efficiency.

In an AI pilot in software, observe smaller things more directly linked to the hypothesis:

how many suggestions were accepted without significant change;
how much review effort appeared;
which types of errors required intervention;
which cases were out of scope;
when the team preferred to ignore the suggestion;
what workflow changes emerged to accommodate AI;
which decisions became easier or harder after the test.

These observations do not need to become a huge dashboard. They need to help leadership decide if the hypothesis deserves continuation.

Define small but useful metrics to decide

An AI pilot should not start with an overly ambitious dashboard. The more metrics unrelated to the hypothesis, the greater the chance the discussion gets lost in contradictory signals.

The metric should show whether the intervention improved, worsened, or complicated the chosen task.

If the hypothesis is about support suggestions, it makes sense to observe suggestion acceptance rate, time to review, number of human interventions, occurrences of critical errors, internal user perception of the specific task, and minimum case volume for responsible decision-making.

If the hypothesis is about generating draft release notes for a SaaS product, the metrics may differ: completeness of information, need for correction by product or engineering, adherence to editorial standards, review time, and most frequent types of changes.

The point is not to measure “AI” generically. The intervention in a task is measured.

It is also better to define beforehand what will not be concluded. A draft generation pilot does not prove the company should automate communication with clients. A ticket classification pilot does not prove support should be automated. An internal recommendation pilot does not prove the product gained a new value proposition.

This care avoids inflating small learning until it seems like an entire strategy. Artificial intelligence strategy requires other decisions, such as business priorities, governance, capabilities, and investment sequence. This topic appears at another level in the guide on business-connected artificial intelligence strategy.

Use a checklist to compare pilot candidates

When there are many ideas, discussing in the abstract favors the most seductive proposal or the area with more internal power. A simple checklist helps compare candidates with the same criteria.

Use scoring only to support the conversation. It does not approve broad implementation and does not replace technical, legal, operational, or commercial judgment when risk requires it.

AI Pilot Selection Checklist with Reversible Scope

Evaluate each idea from zero to two on each criterion: zero for weak signal or high risk, one for relevant gap, two for clear test condition.

Testable hypothesis. Can the idea be written with user, task, expected change, and success evidence? Zero: it is just a broad intention to use AI. One: has a defined task but no evidence or success criteria yet. Two: has a clear hypothesis, observable metric, and decision condition.
Reversibility. Is it possible to turn off or remove the pilot without breaking the product or main operation? Zero: the change affects a central flow without alternative. One: there is an alternative, but reversal would require significant rework. Two: reversal is simple, planned, and has a defined responsible.
Risk to the customer. Would AI error be contained before causing damage to the customer or product reputation? Zero: error reaches the customer directly without review. One: there is some review, but exceptions are still ambiguous. Two: there is containment, human review, or well-defined exposure limit.
Available data. Does the pilot have sufficient, accessible, and adequate data to test the hypothesis without broad reorganization? Zero: data do not exist or require major preparation. One: data exist but have relevant gaps. Two: necessary data are available for a limited test.
Supervision cost. Does the effort to review, correct, and monitor AI fit the team’s routine during the pilot? Zero: supervision consumes more energy than the original task. One: supervision is possible but not yet sized. Two: supervision has defined frequency, responsible, and limit.
Reusable learning. Even if the pilot does not advance, does it teach something useful for future product, operation, or architecture decisions? Zero: learning would be too specific or irrelevant. One: may generate learning but not linked to a future decision. Two: learning informs a clear decision about product, process, or capability.

Practically: ideas with high scores tend to be better candidates, provided specific risks are documented. Intermediate ideas may require reducing ambiguity before starting. Low-scoring ideas are probably too large to be the first pilot.

Fictional example: a B2B software company compares three ideas. The first is to generate executive reports automatically for all clients. The second is to suggest internal answers for recurring support questions, with human review before sending. The third is to predict customer churn based on usage events.

By the checklist, the second tends to be the better initial pilot. It has a delimited task, controlled exposure, simple reversal, and direct learning about suggestion quality. This does not mean it will succeed. It means it can fail with less damage and teach something useful: whether suggestions help, if the knowledge base is adequate, if review fits the routine, and if the team trusts the generated support.

The first idea may be flashier but exposes executive communication to all clients. The third may be valuable but may depend on more complex data, interpretation, and commercial responsibility. They can come back later. They do not need to be the first test.

Decide beforehand what continuing, adjusting, or stopping means

A pilot without exit criteria becomes a project that continues by inertia. The team invests time, leadership avoids ending it, and any positive sign becomes justification to continue.

Before starting, define three possible paths.

Continuing means the hypothesis gained enough evidence to expand the test with control. It is not scaling to the entire product. It is authorizing the next level of exposure, integration, or investment.

Adjusting means the learning was useful but the design failed. Perhaps the chosen task is inadequate, data are poorly structured, the interface causes confusion, or human review is too heavy. In this case, the pilot did not fail. It revealed where the design needs to change.

Stopping means the pilot depends too much on exceptions, creates disproportionate risk, requires more supervision than the original task, or does not help decide the next step. Ending early can be a sign of good governance, not lack of ambition.

In practice, choose an AI hypothesis that can be written in one sentence, tested in a delimited task, turned off without trauma, and evaluated by a few metrics linked to the next step. If the idea does not pass this filter, reduce the scope before choosing the tool.

If you want to discuss this decision in your company’s context, talk to dooop.

Sources

NEXT DECISION

Discuss application in your company

Conversation about the software company context

Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.