Part 5 of 5 — RAG Explained

The wrong question is “which AI should we buy?” The right question is “which loop fits the job?”

Part 1 argued for an approved corpus. Parts 2–4 showed the pipe and product instances. This part is the decision table: four technologies, four best fits — plus the question teams skip until production: when should you not use an AI agent?

Pick the tool

Technology Best for
Traditional automation Predictable, rule-bound workflows with fixed steps
RAG Knowledge retrieval and grounded answers over an approved corpus
LLM Generation and reasoning when the needed context is already in the prompt
AI agent Dynamic, multi-step workflows that call tools and re-plan from observations

Read down the table as increasing flexibility — and increasing cost, risk, and ops surface.

Traditional automation wins when the path is known: if state A and event B, run script C. Cron, ETL, approval chains with explicit branches. No model required. Adding an LLM “to be smart” usually adds variance without benefit.

RAG wins when the job is find the right internal knowledge, then answer or draft. Support lookup, policy Q&A, research over uploaded docs. You need index, retrieve, generate — not a twenty-step tool loop. Part 2 is the reference.

LLM alone wins when context fits in the prompt and does not live in a large corpus: rewrite this paragraph, classify this ticket from the text shown, summarize these three bullets you pasted. You are paying for generation, not retrieval infrastructure.

AI agent wins when the goal requires multiple steps, tool calls, and re-planning after the environment talks back: run tests, open a ticket, query an API, try again if the first path failed. How AI Agents Actually Work Part 7 is the loop diagram. From Models to Agents Part 6 is where agents actually earn their keep — checkable goals, narrow tools, human gates.

When should you NOT use an AI agent?

An agent is the wrong default when:

  1. The workflow is fully predictable. Invoice export every night at 02:00 with the same SQL does not need a planner. Use automation.
  2. The job is read-only Q&A over docs. RAG plus human review beats a tool loop that could post, send, or delete.
  3. You cannot define “done.” If no test or human can verify success, an agent will optimize for sounding finished. Part 5 of From Models to Agents is the cold-water read.
  4. Side effects are irreversible and un gated. Payments, production config, mass email — without allowlists and approvals, a loop is a liability multiplier.
  5. Latency and cost must be minimal. Each agent step is another model call plus tool round trip. A single RAG query is cheaper and easier to trace.
  6. Compliance requires exact replay. Agents are non-deterministic. Regulators and auditors prefer fixed pipelines with logs you can replay byte-for-byte.

None of this is anti-agent. It is fit. Agents shine when the path unfolds — investigate, hypothesize, act, observe, revise — and when you can cap steps, tools, and spend.

Common misfits (quick examples)

Situation Better fit
“Answer HR questions from the handbook” RAG
“Rewrite marketing copy in our tone” LLM (prompt includes examples)
“Sync CRM to warehouse nightly” Traditional automation
“Triage L1 tickets: search KB, draft reply, suggest category” RAG + human; agent only if you truly need multi-system writes with gates
“Implement feature from issue, run CI, open PR” Agent-shaped — with tests as scoreboard (FMA Part 6)

Conclusion

Start with the smallest loop that can finish the job. Automate the fixed paths. Add RAG when answers must cite your corpus. Use an LLM when the prompt already contains the facts. Reach for an agent when the task is multi-step, tool-shaped, and you can still say what “done” looks like — and when you are willing to operate traces, eval, and gates.

Takeaway: RAG is for grounded knowledge; agents are for dynamic workflows. Most enterprise “AI projects” are RAG or automation with a chat UI — and that is often correct.

Part 4: Product instances
Hub: RAG Explained
Related: Where agents actually make sense · RAG on the agent stack
Next series: Why AI Agents Fail