Part 3 of 5 — RAG Explained

The demo indexed ten PDFs. Production indexes ten thousand — with ACLs.

Part 2 drew the pipe. Enterprise RAG is what happens when that pipe touches real permissions, stale content, and tickets that use words your handbook never uses. The architecture does not change. The failure modes do.

Permissions on the corpus

Retrieval must respect the same rules as the source system. If an engineer should not read HR docs, their query must not surface HR chunks — even if the embedding match is perfect.

document ACL  →  stored on chunk metadata
user identity →  filter at retrieve time

Indexing without ACL (Access Control List) metadata is a data leak waiting for a clever question. Treat the index as sensitive storage, not a cache of “public internal text.”

Freshness and chunking

Freshness. Policies change. Indexes that run once a quarter teach the model last year’s refund rules. Define SLAs: how fast a published doc appears in search, and how you invalidate deleted content.

Chunking. Split too small: you lose context across section boundaries. Split too large: retrieval dilutes signal and wastes the context window. Operational teams tune chunk size and overlap per corpus type — runbooks vs contracts vs ticket threads.

Metadata. Title, product line, effective date, and language belong on chunks. Filters (“only 2026 policies”) beat praying the ranker guesses.

When retrieval returns the wrong chunk

Silent failures look like success:

  • Wrong product — chunk mentions “Pro” when the question was about “Enterprise.”
  • Outdated page — old wiki version outranks the new one.
  • Near duplicate — two chunks say slightly different things; the model averages them into fiction.

Citations in the UI help humans catch errors. They do not fix bad retrieval. You need eval: a set of real questions with expected source IDs, re-run when the index changes. From Models to Agents Part 8 is the eval mindset; this part is the RAG-specific test cases (did we retrieve the right doc?).

Operational example: support ticket assist

Corpus: Last 90 days of resolved tickets (PII scrubbed) plus current product FAQ.

Flow: Agent pastes customer email. System retrieves similar tickets + FAQ chunks. Draft reply generated for human edit — no auto-send.

What ops watches:

Signal Action
Retrieval score below threshold Show “no good match”; do not generate from thin air
FAQ chunk older than 30 days Warn in UI; queue re-index
Draft cites ticket from wrong product line Human rejects; log for chunk metadata fix

Architecture sketch:

ticket text ──▶ retrieve (FAQ + tickets, product filter)
              ──▶ generate draft
              ──▶ human review ──▶ send

No agent loop required. If you add tools that post replies without review, you have crossed into agent territory — see Part 5.

Conclusion

Enterprise RAG is the same three-step pipe with governance: who may retrieve what, how fast the index updates, and how you detect wrong chunks before a customer sees them.

Takeaway: Production RAG fails on permissions, freshness, and retrieval quality — not on missing a fancier model name.

Part 2: Architecture
Part 4: Product instances