dooopSoftware · Organization 12 min

AI Competency Matrix for Development

Learn how to create a matrix that evaluates autonomy, judgment, and responsibility in the use of AI throughout the development cycle.

Published on September 6, 2026

CENTRAL THESIS

Using AI does not demonstrate maturity if the person does not know how to validate, limit, and sustain the technical decision.

The matrix should observe behaviors by responsibility, not familiarity with tools.

A development competency matrix with artificial intelligence should show what each person can decide, review, document, and sustain when AI participates in the work. The useful question is not "who knows how to use AI?" but "in which responsibility does this person demonstrate autonomy, judgment, and limits?" For this, the matrix needs to start from the real development cycle and transform abstract competencies into observable behaviors.

Why the matrix should start with responsibilities, not tools

Tools change quickly. Engineering responsibilities change more slowly.

When leadership builds an AI competency matrix based on tool names, commands, or prompt types, it creates a portrait that ages before guiding decisions. A person may be fluent in an interface and still not know when to reject a suggestion, request a review, document a choice, or stop a risky path.

A more useful matrix starts with software development responsibilities: understanding requirements, analyzing legacy, proposing solutions, implementing, testing, reviewing code, documenting decisions, monitoring product effects, and guiding others. From there, AI enters as part of the work, not as a separate skill category.

This avoids a common mistake: treating "using AI" as a single competency. Using AI to explain a legacy code snippet does not require the same level of judgment as using AI to change a critical rule, suggest an architectural migration, or generate a battery of tests for a sensitive feature.

The presentation of the DORA 2025 report describes AI as an amplifier of existing organizational strengths and weaknesses and highlights the importance of the organizational system for return on investment. This is a good reason to look less at the tool’s shine and more at the responsibility system where it will be used.

If the company is already discussing AI maturity, strategy, or governance, the competency matrix should connect to that diagnosis. The starting point may be a broader reading on AI maturity, but the matrix needs to descend to evidence from daily work.

A good competency does not just say "uses code assistant." It says something like: "uses AI to raise implementation alternatives, compares options with service constraints, and records why suggestions were accepted, adapted, or rejected."

This phrase allows observation of work. The previous one only declares familiarity.

How to separate technical competence, judgment, and responsibility

Three things are often mixed when talking about AI development competencies.

The first is the technical competence to operate the tool. The person knows how to formulate a question, provide context, request variations, ask for explanations, and explore alternatives.

The second is technical judgment. The person can evaluate if the response makes sense given the architecture, design patterns, security constraints, existing tests, expected product behavior, and the team’s ability to maintain that decision afterward.

The third is responsibility. The person has authorization and maturity to accept or reject the AI suggestion in a given context. In some situations, they can decide alone. In others, they need to call for technical review, involve product, consult operations, or record a more formal decision.

The matrix needs to make these three dimensions visible. Prompt fluency is not the same as technical autonomy. Speed in generating code is not the same as the ability to sustain a change.

An observable behavior combines the three dimensions. For example: "uses AI to propose regression tests, checks if tests cover the changed flows, removes irrelevant cases, and requests review when the change affects behavior shared by other modules."

This formulation shows operation, validation, and autonomy limits.

It also helps leadership avoid a common injustice. A person who uses AI little but validates well may be more mature than someone who uses AI all the time without leaving a decision trail. The matrix should not measure usage quantity. Using more does not mean deciding better.

This separation aligns with a larger leadership decision: creating a business-connected AI strategy without turning adoption into tool theater. In software development, capability appears when the team improves decision quality, not when everyone uses the same shortcut.

Which behaviors to observe in each development responsibility

The simplest way to build the matrix is to choose recurring responsibilities and write verifiable behaviors for each. The goal is not to cover everything. It is to start with points where AI already appears in work or tends to alter risks and decisions.

In requirement analysis, observable behaviors may include:

uses AI to identify ambiguities in a story or specification;
compares generated questions with product context before bringing them to the team;
records relevant doubts for discussion with product or business;
distinguishes plausible suggestions from validated requirements.

In implementation, the matrix may observe if the person:

provides sufficient context about design patterns before requesting a solution;
adapts suggested code instead of copying without reading;
checks compatibility with conventions, dependencies, and existing constraints;
knows when a local change requires broader review.

In code review, the competency changes nature. AI can help explain snippets, point out risks, or suggest tests, but responsibility remains human and technical. Observable behaviors include:

compares AI explanation with the diff and expected behavior;
identifies risks AI did not mention;
records relevant decisions when accepting a non-obvious approach;
guides the team on AI-assisted review patterns.

Fictional example: in a digital product team, a developer uses AI to review a change in a subscription module. At an initial level, she asks for a diff explanation and checks if the response seems compatible with the changed code. At a more mature level, she notices AI did not comment on a duplication risk in an asynchronous flow, creates tests to explore the hypothesis, records the decision in the pull request, and shares a review example with the team for similar changes. The expected effect, which would need to be measured in the real context, would be reducing implicit decisions and improving future review quality.

In technical documentation with AI, the matrix should evaluate clarity, location, and reliability. The DORA on documentation quality assesses documentation by attributes such as clarity, ease of location, and reliability, and recommends active creation and maintenance. This supports a practical rule: AI can help draft, but competency lies in keeping documentation useful for decision-makers and operators.

Observable behaviors in this responsibility may be:

uses AI to transform a technical decision into a clear draft;
checks if the text describes the system’s real state;
removes generalizations that do not help maintenance;
updates documentation when the decision changes.

How to define maturity levels without creating artificial roles

The matrix does not need to invent parallel roles like "junior AI developer" or "senior prompt specialist." This tends to confuse more than help.

A better alternative is to describe levels of autonomy and impact by responsibility. Names may vary, but four levels usually work well as management language: assisted, operational, discerning, and reference.

At the assisted level, the person uses AI with supervision. They can ask for explanations, generate simple alternatives, and compare responses with instructions received, but still need support to validate impact, scope, and risk.

At the operational level, the person applies AI in known tasks. They provide context, review output, perform basic validations, and know when the suggestion does not fit design patterns.

At the discerning level, the person decides autonomously in a relevant part of the responsibility. They identify non-obvious risks, document choices, reject seductive answers when they do not support the problem, and request review when the change exceeds their decision limit.

At the reference level, the person guides others. They transform good uses and relevant errors into internal examples, improve review patterns, help the team calibrate risks, and participate in evolving practices.

The difference between these levels is not the amount of tool use. It is the quality of the question, validation, decision, and shared learning.

Therefore, each competency should have evidence. Evidence may be in a pull request, a decision document, a created test, a commented review, an internal example, or a pairing conversation appropriately recorded for the team.

If the company is designing an AI roadmap, these levels help separate ambition from capability. It is not enough to decide an area will use AI. It is necessary to know who can review, who can guide, and where supervision dependency remains.

Checklist to write an observable AI development competency

Before including a phrase in the matrix, it is worth submitting it to a simple test. If the competency does not pass these criteria, it is probably too generic.

Explicit responsibility

Is the competency linked to a real responsibility, such as analyzing requirements, writing code, reviewing, testing, documenting, or deciding architecture?

Weak: "knows how to use AI to program."

Strong: "uses AI to raise implementation alternatives and compares options with the service’s technical constraints."

Observable behavior

Could leadership observe or review evidence of this behavior in a work artifact?

Weak: "has good sense when using AI."

Strong: "records which AI suggestions were accepted, rejected, or adapted in a relevant technical decision."

Validation criterion

Does the competency clarify how the person checks if AI output is adequate?

Weak: "trusts AI when the answer seems good."

Strong: "validates AI output with tests, context reading, and comparison with design patterns before incorporating the change."

Autonomy limit

Does the matrix indicate when the person can decide alone and when they must request review?

Weak: "uses AI with autonomy."

Strong: "can use AI for local refactorings but requests technical review before changing integration contracts or critical rules."

Shared learning

Does the behavior generate learning for the team or remain restricted to the individual?

Weak: "learns new ways to use AI."

Strong: "transforms a successful use or a relevant error into an internal example reusable by the team."

This checklist does not replace leadership judgment. It reduces ambiguity. The matrix improves when a competency can be observed, discussed, and reviewed without depending on the vague impression that someone "is good with AI."

How to use the matrix in team development conversations

The competency matrix should not become a punitive instrument. If it is born as a ranking, the team learns to perform maturity. If it is born as a development tool, it helps distribute responsibility more clearly.

In individual conversations, the matrix can guide concrete feedback. Instead of saying "you need to use AI better," leadership can say: "in reviews, you already ask good explanations from AI, but you still do not record why you accepted or rejected suggestions in relevant changes."

In pairing, the matrix helps match people with different autonomy levels. Someone discerning in testing can pair with someone operational in implementation. Someone a reference in documentation can transform recurring decisions into internal examples.

In planning, the matrix helps decide who should review a change, who can take on an investigation, and where the team needs support before expanding AI use. This is particularly useful when the organization tries to balance productivity, quality, and learning without treating automation as a universal shortcut.

The DORA on learning culture relates learning culture to software delivery performance and proposes treating learning as an organizational investment. The matrix can support this investment when it transforms individual use into shared learning. It does not prove by itself that the team will perform better.

There is also a connection with leadership. Working with AI requires trust as infrastructure, not as rhetoric. If the person does not know when they can decide, when they need to request review, and how to document a choice, trust turns into anxiety or permissiveness. Neither sustains mature engineering.

How to keep the matrix updated as AI use matures

A useful matrix is not written once and stored. It should be reviewed as the team finds new usage patterns, risks, doubts, and responsibilities.

The review does not need to be heavy. It can start from work evidence:

technical decisions that required much discussion;
errors or near errors in suggestions accepted without sufficient validation;
incidents avoided by careful review;
recurring doubts about when to use AI;
documentation gaps;
changes in product, data, or operations practices;
internal examples that began to guide the team.

When the team experiments with new ways of working, it is advisable to separate discovery, hypothesis, and experiment. Discovery is understanding where there is opportunity or problem. Hypothesis is formulating a testable assumption. Experiment is changing a practice, measuring impact, and deciding whether to continue, change, or stop it.

Microsoft describes ExP as a platform to incorporate experimentation into the development cycle, validate hypotheses, measure impact, and iterate products. The applicable lesson here is discipline: AI practices also need hypothesis, observation, and iteration. This does not mean any feedback automatically retrains a model, nor that every team needs its own platform to learn.

It is also worth making limits explicit in the matrix itself. Not every activity should be AI-assisted. In critical decisions, the matrix should indicate when automation is inappropriate or when human review is mandatory. Third-party models can be part of mature products and flows, provided responsibility for validation, documentation, and decision is clear.

A good AI development competency matrix does not try to predict all tools that will come. It creates language to observe work, distribute autonomy, and learn from evidence. The next decision is to choose three or four real responsibilities from the development cycle, write verifiable behaviors for each, and review the matrix when work reveals new risks or new capabilities.

If this conversation needs to move from abstract to concrete in your team, talk to dooop.

Sources

NEXT DECISION

Discuss application in the company

Conversation about the software company context

Content by dooop. Registration allows linking this topic to the reader’s journey and tracking interest in the subject.