Part 2 of 5 — RAG Explained

RAG is a pipe.

Retrieval-augmented generation (Lewis et al., 2020) named the pattern: index a corpus, retrieve relevant passages, condition generation on those passages. Every vendor implementations — search box, copilot, notebook — still runs some version of that pipe. How AI Agents Actually Work Part 6 shows where the pipe sits inside an agent stack. Here we stay at the pipe itself.

The architecture

┌─────────────┐     ┌─────────────┐     ┌──────────────┐
│   Corpus    │────▶│    Index    │     │  (embeddings │
│  you approve│     │  build job  │     │  or keyword) │
└─────────────┘     └──────┬──────┘     └──────────────┘
                           │
User question ──▶ Query ──▶ Retrieve top-k chunks
                           │
                           ▼
                    ┌──────────────┐
                    │ chunks +     │
                    │ question ──▶ │ LLM ──▶ answer
                    └──────────────┘

Index. Ingest documents, chunk them, store vectors or inverted indexes. Run on a schedule or on upload.

Retrieve. Turn the user question into a search query; fetch k chunks. k is a budget — too small misses the answer paragraph; too large crowds the context window.

Generate. The LLM reads the question plus retrieved chunks and produces the reply. Grounding quality depends on what retrieval returned.

That is the whole mechanism. Orchestration, tools, and re-planning are optional layers on top — not requirements for basic RAG.

Operational example: internal wiki Q&A

Corpus: Confluence export for Engineering + Support (approved namespaces only).

Index job: Nightly pipeline chunks pages (~512 tokens), embeds, writes to a vector store. ACL (Access Control List) metadata copied from source so retrieval respects group membership.

Ask-time: Employee types “How do we rotate API keys for staging?” Query embeds; top-5 chunks return from pages tagged security and staging. Prompt template: system instructions + chunks + question.

Success: Answer quotes the runbook section; UI shows source links.

Failure modes you will see in production: page renamed but index stale; duplicate chunks from copy-paste; retrieval returns a deprecated page because keywords matched but date did not. Part 3 is how teams live with those failures.

Retrieve, then generate — not “stuff the wiki”

Lewis et al. are explicit: retrieval selects a small set of passages. Pasting entire manuals into the prompt is not RAG — it is an expensive search miss. Chunking strategy, metadata filters, and k tuning are part of the architecture, not polish.

Conclusion

Draw RAG as three boxes and one LLM call. If you cannot name your corpus, index cadence, and k, you do not have an architecture yet — you have a demo with a search plugin.

Takeaway: RAG architecture is index → retrieve k → generate over a corpus you approve. Everything else is ops and policy.

Sources

Part 1: Why your data
Part 3: Enterprise ops