How to Measure a Team's Ability to Work with AI: Practical Tasks, 4 Levels, and 5 Competencies
By Pedro Sonda, dooop
To measure a team’s ability to work with AI, observe individuals performing two or three real AI-related job tasks and evaluate five competencies: choosing where AI helps, instructing the tool with context, verifying results, protecting data, and improving the work process. Rate each person across four levels (beginner, user, practitioner, and multiplier) and consolidate the team profile. Self-assessment measures confidence; practical tasks measure capability. Adoption shows how much the team uses AI; capability shows how well they use it.
Updated October 2026.
Why Self-Assessment and Usage Counts Are Not Enough
| Data | Source |
|---|---|
| In tasks outside the "frontier" of what AI does well, consultants using AI had 19 percentage points fewer correct answers than those who did not use AI | Dell'Acqua et al., 2023 |
| 66% of employees trust AI results without evaluating accuracy, and 56% have made work errors because of AI | KPMG and University of Melbourne, 2025 |
| Only 42% of employees say they know where AI can improve their own work | Gartner, 2025 |
| Only 36% of workers feel adequately trained; those receiving more than five hours of training, with in-person format and follow-up, have higher regular use | BCG, 2025 |
Interpretation: Frequent AI use does not mean effective use. The competency most lacking—verifying results and knowing where AI fails—only appears when a person is observed performing a real task.
dooop Framework: The 5 Competencies
| Competency | Meaning | What to Observe in the Task |
|---|---|---|
| 1. Choosing Where to Use | Distinguishing tasks where AI helps from those where it hinders | The person decides to use (or not) and explains why |
| 2. Instructing with Context | Providing objective, context, format, and examples | Quality of instruction and number of attempts until a good result |
| 3. Verifying the Result | Checking facts, numbers, logic, and appropriateness | Errors left in the final delivery |
| 4. Protecting Data | Following rules about what can be used and where | Sensitive data entered into unapproved tools |
| 5. Improving the Process | Proposing changes to the workflow, not just speeding up the task | Suggestions for redesign after the task |
The 4 Levels
| Level | Description | Typical Competencies |
|---|---|---|
| Beginner | Uses AI little or generically | 1 to 2 weak; verification absent |
| User | Uses AI in known tasks with reasonable results | 2 and 4 adequate; 3 irregular |
| Practitioner | Uses AI in the right tasks, verifies and protects data | 1 to 4 consistent |
| Multiplier | Improves processes and helps colleagues | 1 to 5 consistent; teaches others |
How to Apply the Assessment
- Choose two or three real tasks from the role, with verifiable results (generic example: summarize a contract and point out risks; prepare a client response; analyze a spreadsheet).
- Include a task with a trap, where AI tends to err, to observe verification.
- Define criteria beforehand, with the manager and someone expert in the task.
- Observe and record, without turning the assessment into a punitive test.
- Consolidate the team profile: how many people at each level and which competencies are weakest.
From Profile to Action
| Team Profile | Likely Action |
|---|---|
| Majority beginner | Applied literacy with role-specific tasks |
| Majority user, weak verification | Specific training on verification with real errors |
| Practitioners without multipliers | Develop two or three multipliers |
| Multipliers present | Involve the team in process redesign |
To define what each team needs to learn, see How to Define What Each Team Needs to Learn About AI.
Capability, Adoption, and Training
| Measure | Question | Where to Read More |
|---|---|---|
| Adoption | How much does the team use AI? | How to Measure AI Adoption Within Teams |
| Capability | How well does the team use AI? | This article |
| Training Effect | Did training change behavior and results? | How to Measure if AI Training Really Worked |
What to Do Now
- Choose a team and two real tasks with verifiable results.
- Define criteria for the five competencies.
- Apply the assessment, consolidate the profile, and decide on actions for the next 90 days.
To measure team capability and maturity methodically, learn about dooop’s work at dooop.com.br/maturidade-de-times-em-ia.
Frequently Asked Questions
Does assessing capability create fear in teams? It can if presented as a test. Present it as a diagnostic to guide support, do not link results to performance evaluation in the first round, and share the team profile rather than individual scores.
Can operational teams be assessed the same way? Yes, using tasks from their routine, such as recording an incident, consulting a procedure, or preparing a shift report. The five competencies remain valid.
How often should the assessment be repeated? Before training and three to six months afterward to measure progress.
Also Read
About dooop
dooop is a consultancy based in Caxias do Sul (RS) that prepares leaders and organizations to transform artificial intelligence into business capability through strategy, governance, and training.