dooopSoftware · Product · 11 min

Rules, RAG or Agents: How to Choose the Architecture

The choice depends on the task: rules serve stable decisions, RAG retrieves knowledge, and agents require autonomy with control.

Published on September 6, 2026

CENTRAL THESIS

Choosing architecture too early increases cost, latency, and risk. The smallest sufficient layer tends to be the strongest.

Rules, RAG, and agents do not compete for the same problem. The choice depends on autonomy, context, and control.

To choose between rules, retrieval-augmented generation (RAG), and AI agents, start with the work the product needs to solve. Rules work best for stable, auditable decisions. RAG helps when the answer depends on retrievable knowledge. Agents make sense when the product needs to plan steps, use tools, and adapt execution. The greater the autonomy, the greater the need for control over risk, measurement, and review.

The common mistake is choosing the architecture before the type of work

The question "should we use RAG or AI agents?" often arises too early. Before that, there is a more useful question: what decision, answer, or execution does the product need to support?

In an intelligent feature, different tasks may be mixed on the same screen. One part may be a binary decision, such as blocking or allowing an action. Another may be a response based on internal policy. Another may require consulting data, triggering a system, requesting confirmation, and completing an operation. Calling all this "AI" impoverishes the technical decision and increases the risk of building an architecture larger than the problem.

Anthropic distinguishes flows with predefined paths from agents that dynamically decide their process and tool usage, and recommends starting with the simplest solution and adding complexity when necessary, in their text on agent patterns. This distinction is useful because it removes the agent from being the inevitable destination. Sometimes, the best intelligent product is a well-designed rule. Sometimes, it is a search with response generation. Sometimes, it is an agent with limited autonomy.

The architecture decision needs to fit the product operation. If the team cannot explain why a response was given, measure exceptions, review errors, and limit actions, increasing autonomy only shifts complexity to production.

When rules solve better than generative AI

Business rules are explicit instructions that determine system behavior under known conditions. They work well when the logic is stable, criteria are clear, and the organization needs to audit the decision without interpreting a probabilistic response.

Use rules when the task looks like this:

If status is A and condition B is present, apply C.
If the request does not meet a minimum criterion, block the action.
If there is a known exception, forward for review.
If a priority depends on structured fields, order by a defined rule.

The value of rules lies in predictability. They are easier to test, explain, and maintain when the domain does not change constantly. They can also be the best choice when an operational error has high impact and the organization prefers a deterministic decision over a fluid response.

The limit appears when the number of exceptions grows so much that maintenance becomes the real product. If each new situation requires another rule, another exception, and another chained condition, perhaps the task is not as stable as it seemed. Still, this does not automatically lead to agents. It may lead to a process review, a better knowledge base, or a more explicit experience design.

A simple criterion: if a product leadership can describe the decision in stable conditions and the team can test the main paths without relying on semantic interpretation, rules are probably sufficient.

When RAG or AI agents enter different decisions

RAG means retrieval-augmented generation: the model generates a response using information retrieved from an external source. The article by Lewis et al. presents this combination of generation and passage retrieval. For the product decision discussed here, RAG is an option when the answer depends on available knowledge. The user’s question may vary, but the answer source needs to be anchored in retrievable information.

Anthropic defines context engineering as the selection and maintenance of information available to the model during inference, including instructions, tools, external data, and history, within a limited window, in their material on context engineering. This definition helps separate two things the market mixes: retrieving context is not the same as granting autonomy.

A search with RAG can answer: "which policy applies to this case?", "how does this feature work?" or "which steps should the user follow?" It does not need to decide alone which systems to trigger, which exceptions to negotiate, or which operation to complete.

The limit of RAG is equally important. It does not transform disorganized knowledge into reliable decision by itself. If the base is outdated, ambiguous, or contradictory, the response may be well written and still be poor. RAG increases the ability to bring context to the model but does not replace curation, content governance, and boundary design.

Therefore, before using RAG, it is worth asking: does the product have retrievable knowledge, with source, scope, and update minimally controlled? If the answer is no, the first improvement may be in domain knowledge, not the model. This point connects directly to the work of organizing domain knowledge for AI in the product.

When AI agents make sense in the product

AI agents make sense when the task requires more than responding. The system needs to dynamically decide which steps to follow, when to consult a tool, how to adapt the path based on an intermediate response, and when to stop.

This can include investigating a request, consulting different systems, comparing information, proposing an action, and executing it within defined limits. The point is not the agent "thinking" abstractly. The point is the product delegating part of the process choice to the system.

This autonomy changes the type of risk. An incorrect response in assisted search can already be problematic. An incorrect action executed by an agent can generate rework, loss of trust, or an operation difficult to reverse. Therefore, agents require permission limits, decision trails, observability, testing by task type, and interruption criteria.

Mature choice separates automatable execution, necessary review, and decisions that should not be delegated.

A good sign to consider agents is the simultaneous presence of path variation, tool use, and need for adaptation during execution. If the task always follows the same path, a predefined flow can solve it. If the task only requires consulting knowledge, RAG may suffice. If the task requires choosing different paths and triggering resources to complete a goal, agents enter the conversation.

Criteria to choose the smallest sufficient layer

Before choosing rules, RAG, or AI agents, answer the questions below and choose the smallest layer that passes the criteria. The goal is not to reduce ambition. It is to avoid technical sophistication hiding a poorly defined task.

Stable conditions and few exceptions: rules tend to be sufficient.
Consulting documents, policies, history, or knowledge base before the response: RAG tends to be more proportional than pure rules.
Choosing different steps during execution: agents may be considered.
Triggering tools or external systems to complete the task: agents may make sense, provided there are limits, permissions, and review.
Error with potential financial loss, trust breach, or irreversible action: reduce autonomy, make human approval explicit, or maintain deterministic rules.
Measurement of quality, latency, cost, and exceptions by task type: without minimum measurement, avoid increasing autonomy.
Frequent change or incomplete context: prefer an approach that exposes limits and allows review before automating execution.

The practical comparison is as follows: rules are better for stable, auditable decisions with explicit criteria. RAG is better for answers that need to retrieve knowledge before generating a synthesis. Agents are better for tasks with multiple steps, tool use, and adaptation during execution.

The warning differs in each case. Rules become fragile when exceptions dominate. RAG becomes fragile when knowledge is poor. Agents become fragile when autonomy grows without operational control.

Fictional example: billing support in a SaaS

Imagine a SaaS with a billing support area. The example is fictional, created only to apply the criteria.

The team wants to reduce effort on recurring tickets and monitor if the user experience remains acceptable. The first temptation is to create an agent that "solves billing." But "billing" is not a single decision unit. It is an area with classifications, responses, and flows of different natures.

To classify payment status, rules may solve better. If payment is confirmed, the subscription is active. If the attempt failed, the product shows the available reason and guides the next action. If there is divergence between internal fields, it forwards for review. The decision depends on structured data and explicit criteria.

To answer questions about refund policy, deadlines, or reissue, RAG may be more proportional. The user asks in natural language, the product retrieves policy excerpts, synthesizes the answer, and shows limits. Here, the difficulty is finding and using the correct knowledge, not planning an entire operation.

If the sequence of queries and actions is always the same, a predefined flow can check the subscription, consult the payment attempt, and forward the request. An agent becomes an alternative when it needs to dynamically choose the next step according to the information found. Even in this case, actions like resending a bill may require explicit user confirmation or forwarding to a person.

The expected effects of this design would be hypotheses to measure, not presumed results. The hypothesis may be that rules reduce ambiguities in simple classifications, that RAG improves consistency of policy-based answers, and that agents help in flows with many steps. Each hypothesis needs observation in production or controlled experiment before becoming conviction.

Microsoft describes its ExP platform as a way to incorporate experimentation into the development cycle, validate hypotheses, measure impact, and iterate products, on its page about experimentation. This does not mean user feedback automatically retrains a model. It means product decisions can be treated as verifiable hypotheses.

How to measure if the choice remains appropriate

The right architecture at launch may cease to be right later. Volume changes, exceptions appear, knowledge becomes outdated, and user behavior reveals tasks the team had not named.

Monitor the choice by task type, not just by the entire feature. The same product area can combine rules, RAG, and agents. If measurement mixes everything, the team does not know which layer is working and which is increasing cost or risk.

Some practical signs deserve review:

The exception rate in rules grows.
RAG responses depend on ambiguous or outdated documents.
The agent needs to ask for help in steps that should be well defined.
Perceived latency harms the user flow.
Cost per task grows without proportional quality gain.
Human intervention appears too late, only after a bad action.
The team cannot explain why a decision was made.

Reliability enters the product decision when autonomy, review, and scale limits change. Google SRE defines service level objectives as reliability goals that guide engineering decisions, with agreement on goals, use of error budget for prioritization, and review process, in the guide on SLOs. For intelligent features, this logic helps discuss the acceptable level of failure, delay, exception, and review before scaling autonomy.

This connects to the broader design of intelligence in the product: software with AI does not improve just because it gained a model. It improves when the organization learns to separate task, context, boundary, metric, and responsibility.

The minimally sufficient architecture is the smallest layer capable of solving the task with acceptable quality: rules for stable decisions, RAG for responses based on retrievable knowledge, and agents for dynamic execution with tools and clear limits. Before increasing autonomy, the useful question is simple: which part of the task really needs to decide the next step?

If you want to discuss this decision in the context of your company, talk to dooop.

Sources

NEXT DECISION

Discuss application in the company

Conversation about the software company context

Content by dooop. Registration allows relating this topic to the reader’s journey and tracking interest in the subject.