dooopSoftware · Strategy · 12 min

How to Organize an AI Experiment Portfolio

Limit AI experiments, connect each test to a concrete decision, and use entry, continuation, and stopping criteria to learn better.

Published on September 6, 2026

CORE THESIS

Many AI ideas compete for the same attention. The portfolio must protect the capacity to learn.

Fewer active fronts, clearer decisions. Each experiment enters, continues, or exits by explicit criteria.

An artificial intelligence experiment portfolio should organize a short queue of active initiatives, with real capacity to review data, involve experts, and make decisions at the end of each test. The central point is not how many AI ideas fit in the quarter, but which uncertainties leadership can reduce without diluting attention, governance, and stopping criteria. When everything enters at once, the company produces movement but loses clarity about what should continue, wait, or leave the portfolio.

Why Many AI Initiatives Advance Without Decision

Artificial intelligence often enters the agenda through a seductive path: someone saw a demonstration, a client asked for a feature, a competitor announced something similar, or a technical team noticed a possible automation. The initiative seems small. The risk seems manageable. The initial cost seems low.

The problem appears when several of these ideas are approved simultaneously. Each one requires data, expert review, technical evaluation, product discussion, commercial alignment, and some form of governance. Nothing seems large individually, but the whole competes for the same executive attention.

This is how an AI experiment portfolio becomes full of movement and poor in choice.

Exploring possibilities is legitimate, as long as exploration does not occupy the space reserved for active experiments. An experiment needs to test a hypothesis and inform a decision. If at the end of the test no one knows whether to continue, stop, reduce scope, change the technical approach, involve operations, or move to controlled production, the organization did not execute an experiment. It only produced more material for discussion.

For software companies, this distinction changes how to prioritize the portfolio. It affects value proposition, data, support, operations, commercial promise, and responsibility after delivery. Therefore, the topic connects to the broader debate about AI strategy connected to business and the design of an AI roadmap, but here the decision is more specific: what enters, what waits, and what leaves the experiment portfolio.

Define the Decision Before Defining the Experiment

The first question should not be which model to use, which tool to test, or which flow to automate. The first question is: which decision will be better after this experiment?

Some possible decisions are simple to write:

Authorize an intelligent feature for an operational pilot.
Discard an automation because it requires too much human review for the expected gain.
Reduce the scope of an idea to test only one step of the process.
Switch an approach based on text generation to assisted classification.
Validate if the available data are sufficient to continue.
Involve customer service, implementation, or support before any technical advance.
Reposition the commercial promise to avoid selling autonomy where there is only assistance.

This decision statement needs to exist before the experiment occupies portfolio capacity. Without it, the team tends to measure what is easy, not what decides. They may measure informal satisfaction, apparent response quality, execution time in an isolated test, or enthusiasm from a presentation. These signals can be useful but do not replace the decision.

A well-formulated AI experiment does not need to solve everything. It needs to reduce a chosen uncertainty. If the doubt is technical, the test should show technical feasibility. If the doubt is adoption, it should observe use in a realistic situation. If the doubt is operational, it should verify impact on review, exceptions, support, and maintenance. Mixing all doubts in a single test usually creates a comfortable conclusion: “promising, but we need to study more.”

That is an expensive phrase.

Classify Each Experiment by Type of Uncertainty

AI portfolio management improves when each initiative is classified by its main type of uncertainty. This classification is not bureaucracy. It avoids comparing incomparable tests and helps choose coherent metrics.

A practical taxonomy can separate five uncertainties:

Customer value uncertainty: does the customer perceive the feature as useful, reliable, and relevant to their work?
Technical uncertainty: can the solution produce answers, classifications, or recommendations with sufficient quality for the intended use?
Operational uncertainty: can the company operate the resource without creating rework, review queues, or excessive dependence on experts?
Risk uncertainty: does the experiment expose data, decisions, or promises to risks that require restriction, human review, or abandonment of the approach?
Commercial uncertainty: does the initiative help sustain a proposal, expansion, or sales conversation without exaggerating what AI actually does?

Each type requires a different criterion. A value test is not resolved with elegant architecture. A technical test is not resolved with commercial enthusiasm. An operational test is not resolved with a nice demonstration. And a risk test should not be postponed as an implementation detail.

Classification also helps avoid repetition. If five different ideas depend on the same doubt about data quality, perhaps the organization does not need five experiments. It needs one well-designed data test, with reusable learning for the five initiatives.

Limit Simultaneous Initiatives by Learning Capacity

There is a common confusion: treating experimentation capacity as development capacity. If the team can build several proofs of concept, it seems the company can conduct several initiatives in parallel. But the bottleneck of an AI experiment portfolio is not only writing code or connecting services.

The bottleneck is learning with quality.

Learning requires accessible data, domain context, review time, shared criteria, participation of decision-makers, and courage to stop what did not pass the test. When these conditions are missing, AI does not eliminate disorganization. It can amplify it.

The presentation of the DORA 2025 report describes AI as an amplifier of existing organizational strengths and weaknesses and highlights the importance of the organizational system for return on investment. This observation does not prove that every AI initiative will have gain or loss, but serves as a warning: a dispersed portfolio tends to expose the quality of the system that leads it.

It is also advisable to be cautious with easy conclusions about productivity. In the February 2026 update, METR considers new data an unreliable signal of AI’s current effect on productivity and points out measurement difficulties, including participant and task selection and time measurement with competing agents. For an executive portfolio, this reinforces a prudent stance: do not turn productivity expectation into an automatic justification to open more fronts.

Limiting simultaneous initiatives is not lack of ambition. It is learning design. A portfolio with fewer active experiments can generate better decisions if each experiment has an owner, main doubt, appropriate metric, and stopping condition.

Use Entry, Continuation, and Stopping Criteria

Before approving a new experiment, leadership can apply a simple checklist. If two or more answers are negative, the practical recommendation is not to start yet. Not because the idea is bad, but because it is not ready to compete for attention.

Checklist to Decide if an AI Experiment Enters the Portfolio

Explicit decision: what decision will be made at the end of the experiment? Passes when the decision can be written in a sentence, such as continue, stop, change scope, move to operational pilot, or discard the hypothesis.
Testable hypothesis: does the experiment test a hypothesis or just demonstrate a technology? Passes when there is a verifiable hypothesis about value, feasibility, operation, risk, or adoption.
Duration limit: is there a short enough deadline to prevent the initiative from becoming a permanent project? Passes when the deadline is defined before the start and is compatible with the decision that needs to be made.
Available capacity: is there real time from technical people, domain experts, and decision-makers to follow the test? Passes when the experiment does not depend on attention the organization cannot offer.
Metric appropriate to the doubt: does the metric measure the main uncertainty of the experiment? Passes when it relates to the doubt, whether value, accuracy, operation time, rework, risk, or acceptance.
Stopping criterion: under what condition will the initiative be interrupted without becoming a political discussion? Passes when the condition is written before the test and accepted by those responsible.
Reusable learning: will the result help other portfolio decisions? Passes when, even if the experiment does not advance, it reduces a relevant uncertainty for product, operation, data, or positioning.

This checklist does not replace judgment. It forces making decision, uncertainty, available capacity, and stopping rule explicit before the team falls in love with the solution. It is easier to stop a hypothesis when everyone agreed on the stopping rule before seeing the demonstration working.

There is a useful parallel with product experimentation platforms. Microsoft describes ExP as a platform to incorporate experimentation into the development cycle, validate hypotheses, measure impact, and iterate products. The applicable lesson here is not to copy a specific platform or assume feedback automatically retrains models. It is to treat experimentation as a decision discipline, not as a collection of technical tests.

Record Learning in Decision Language, Not Tool Language

Many AI experiment reports record what was used: model, prompt, flow, architecture, implementation time, limitations found. This is useful for the technical team but insufficient for the portfolio.

Leadership needs another type of record:

What hypothesis was tested?
What main uncertainty was reduced?
What became clearer for product, operation, risk, or commercial?
What decision was made?
What future decisions became easier?
What premise should be abandoned or reformulated?

This record prevents the company from repeating the same learning in several fronts. It also helps connect experiments with capacity diagnosis. If one initiative fails because of lack of data, another because of lack of expert review, and another because operations cannot absorb exceptions, perhaps the problem is not choosing the tool better. Perhaps it is recognizing the starting point, as in an AI maturity assessment.

Learning needs to fit in decision language. “The model responded well” is weak. “The approach is technically viable for assisted triage, but not for autonomous response without review” is better. “The client liked it” is weak. “The proposal should be presented as support for prioritization, not as a substitute for human analysis” is better.

The difference seems semantic but changes the portfolio.

Fictional Example: How to Reduce Eight AI Ideas to Three Active Experiments

Imagine a B2B software company with eight AI ideas raised by product, sales, and technology. The example is fictional and serves only to show the reasoning.

The ideas include: assistant to configure system rules, automatic ticket summarization, classification of customer messages, recommendation of next action for users, report generation, search in internal documents, deployment risk analysis, and commercial suggestions for account expansion.

Leadership decides not to approve all eight. First, they ask each proposal to be rewritten as an expected decision.

Two ideas fall immediately. Report generation and commercial suggestions seemed interesting, but no one can say what decision the experiment would unlock at that moment. They return to exploration, outside the active portfolio.

Three ideas depend on the same doubt: the quality and organization of internal documents. Instead of opening three fronts, the company creates a single experiment on document search, with a clear decision: continue or not using the current base as a source for intelligent features. The metric is defined around the ability to retrieve useful information for a specific task, reviewed by domain experts. The stopping condition is recorded before the start: if the base does not allow verifiable answers for the chosen use, the decision will be to invest in information organization before new features.

Two other ideas deal with operation: ticket summarization and message classification. Leadership chooses only one for the active cycle because both require support team review. The chosen one is message classification, as it can inform a broader decision about assisted triage. The experiment does not assume workload reduction. It tests whether classification helps route messages for human treatment with less ambiguity, a hypothesis that still needs to be measured.

The last approved idea is recommendation of next action for users. The main doubt is customer value, not technical. The expected decision is to proceed to a demonstration with selected users or reformulate the proposal. The criterion is not “AI generated recommendations.” The criterion is whether the recommendations make sense in the task context and help better explain the product proposal.

The active portfolio has three experiments: document search, message classification, and next action recommendation. The other ideas were not discarded for lack of imagination. They were removed because they would compete for data, review, and attention without a sufficiently clear decision.

That is the point. Organizing an AI experiment portfolio does not mean killing ideas. It means protecting the capacity to decide which ideas deserve to occupy the active queue now.

Before opening another experiment, write which decision it should inform, choose the main uncertainty, and define the stopping condition. If this cannot be done, the idea should not yet enter the active portfolio.

If you want to discuss the design of the experiment portfolio in your company’s context, talk to dooop.

Sources

DORA 2025: presentation describes AI as an amplifier of existing organizational strengths and weaknesses and highlights the importance of the organizational system for return on investment.
METR: Limits of Productivity Measurement: February 2026 update treats new data as an unreliable signal about AI’s current effect on productivity and points out measurement limitations.
Microsoft: Experimentation Platform: page describes ExP as a platform to incorporate experimentation into the development cycle, validate hypotheses, measure impact, and iterate products.

NEXT DECISION

Discuss Application in Your Company

Conversation about your software company’s context

Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.