How to Assess the Risk of an Artificial Intelligence Solution: 7 Dimensions, Tests for Each, and the Evaluation Sheet
By Pedro Sonda, dooop
To assess the risk of an AI solution, whether purchased or developed, gather evidence across seven dimensions: task performance (using company data), robustness (behavior in difficult cases and unexpected inputs), biases (unequal results among groups), security (resistance to manipulation and misuse), privacy (data handling), transparency (ability to explain and log what the solution does), and vendor dependency (continuity, cost, and exit). Each dimension can be tested before deployment. The result is an evaluation sheet that supports the decision to approve, approve with conditions, or reject.
Updated in October 2026.
Why Evaluate the Solution, Not Just the Use Case
| Fact | Source |
|---|---|
| Amazon abandoned an experimental recruiting tool after discovering it penalized women’s resumes; the issue was the solution’s behavior learned from historical data | CNBC, 2018 |
| Air Canada was held liable for incorrect information provided by the chatbot on its website | ABA Business Law Today, 2024 |
| AI performance is inconsistent: it excels in some tasks and makes more errors in others that appear similar | Dell'Acqua et al., 2023 |
| OWASP lists specific risks for applications using language models, such as prompt injection, sensitive information disclosure, misinformation, and excessive agency | OWASP, 2025 |
Interpretation: A moderate-risk use case can become problematic if the chosen solution is fragile. Risk assessment of the use case (see How to Assess Risk Before Developing an AI Use Case) must be complemented by evaluation of the solution.
dooop Framework: The 7 Dimensions
The dimensions are inspired by the trustworthy AI characteristics described in the NIST AI Risk Management Framework, adapted for the decision to approve a solution.
| Dimension | Question | Test or Evidence |
|---|---|---|
| 1. Performance | Does the solution perform well enough on our tasks? | Test with 50 to 200 real cases with known answers; accuracy rate and error types |
| 2. Robustness | How does it behave in difficult, incomplete, or out-of-pattern cases? | Set of difficult cases; out-of-scope questions; check if it recognizes what it does not know |
| 3. Biases | Do results vary unfairly among groups? | Compare results by relevant groups (region, gender, age group), when applicable |
| 4. Security | Does it resist manipulation and misuse? | Prompt injection tests; attempts to extract information; action limits |
| 5. Privacy | Are data handled appropriately? | Retention and use conditions for training; access controls; LGPD compliance |
| 6. Transparency | Can we explain and log what the solution does? | Logs of inputs and outputs; explanation of recommendations; vendor documentation |
| 7. Dependency | What happens if the vendor changes price, model, or ceases to exist? | Contract terms; data portability; exit plan; cost predictability |
How Much to Test
The depth depends on the risk level of the use; see How to Define Risk Levels for Different AI Uses.
| Use Level | Mandatory Dimensions |
|---|---|
| Low | Privacy (5) and dependency (7), in a simplified manner |
| Moderate | Performance (1), robustness (2), privacy (5), dependency (7) |
| High | All seven, with documented tests |
The Evaluation Sheet
| Field | Content |
|---|---|
| Solution and Version | Name, vendor, model version when known |
| Use and Risk Level | What it will do and the classified risk level |
| Result by Dimension | Adequate, adequate with conditions, or inadequate, with evidence |
| Approval Conditions | For example: mandatory human review; limited use to internal data |
| Decision | Approve, approve with conditions, or reject; who decided |
| Reassessment Date | Especially after model version changes |
Attention to Version Changes
Solutions based on language models change when the vendor updates the model. A saved test set allows repeating the evaluation with each relevant change instead of starting from scratch.
What to Do Now
- Assemble a set of real cases for the next solution under evaluation, including difficult cases.
- Use the evaluation sheet to record the assessment and conditions.
- Define when to reassess: version change, use change, or incident.
To structure AI solution evaluation, explore dooop’s AI Governance practice.
Frequently Asked Questions
Do solutions from major vendors need to be evaluated? Yes. The vendor evaluates the model generally; only the company can assess performance on its tasks, with its data and context.
Who should conduct the evaluation? IT and security lead technical tests; the use owner defines cases and what constitutes a good result; risk and legal assess privacy, biases, and contracts.
What if there is no data to test biases? Record the limitation and compensate with human review and monitoring after deployment, especially in uses affecting people.
Also Read
About dooop
dooop is a consultancy based in Caxias do Sul (RS) that prepares leaders and organizations to transform artificial intelligence into business capability through strategy, governance, and training.