The 6-Layer RAG Architecture Every Enterprise System Actually Needs

A support engineer types one question into your internal assistant: "Does our enterprise plan cover SSO for contractor accounts?" In the demo, the system answers in two seconds and looks perfect. Six weeks into production, the same question returns a contract clause that was superseded in March, shows it to someone in a region that should never see that contract, and leaves you with no way to explain how either thing happened.

Three failures, three different places in the system. The stale clause came from ingestion. The wrong reader came from indexing. The inability to explain either one came from missing evaluation. A production RAG system has six layers of responsibility, and each one breaks in a way you can recognize from the symptom. This part covers what each layer does, what it costs you to skip it, and how to work backwards from a broken answer to the layer that caused it.

The loop and its assumptions

Every production RAG system runs the same five steps. Four of them carry an assumption that can fail silently. The fifth is how you find out which one did.

• Embed the query. Assumes the query and the documents were embedded by the same model, the same version, with the same preprocessing.
• Retrieve candidates. Assumes this user is allowed to see everything that entered the candidate pool.
• Rerank and compress. Assumes what survived the cut is the evidence the answer needs.
• Generate. Assumes the model will answer from the evidence instead of filling gaps from what it already knows.
• Log everything. Carries no assumption. This is how you check the other four.

Most production RAG failures come down to one of those four assumptions being false, and most teams cannot say which one because they skipped the fifth step. The six layers below are what it takes to hold each assumption up.

Layer 1: Ingestion and preprocessing

The failure it prevents: the answer is right there in the source document, and the system never finds it.

Retrieval operates on chunks. If the answer got split across a chunk boundary during ingestion, no reranker, bigger model, or better prompt can recover it, because the complete answer no longer exists anywhere in your index. Ingestion sets the quality ceiling for the whole system. Every layer after it can lose information, and none can add it back.

• Chunk size and overlap (512-1024 tokens, 10-15% overlap). Above that range the embedding averages several ideas into one vector and represents none of them precisely. Below it, a chunk stops containing a complete answer. Overlap exists so that an answer straddling a boundary still appears whole in at least one chunk.
• Chunking strategy. Recursive character splitting respects paragraph and sentence boundaries before falling back to hard splits, and it is the default worth starting from. Semantic chunking splits where consecutive sentence embeddings diverge past a threshold, producing more coherent units for heterogeneous documents at the cost of an extra embedding pass at ingestion.
• Parser fidelity, which matters more than chunking strategy. Naive text extraction turns a PDF table into a stream of numbers with the column headers gone and multi-column layouts interleaved. The chunk stays retrievable and becomes dangerous, because the model will read it confidently and get it wrong. Layout-aware parsers like unstructured.io or Azure Document Intelligence preserve table structure and reading order.
• Metadata written at ingest, never backfilled: doc_id, source_uri, acl_tags, version_hash, ingested_at. The version_hash would have caught the superseded March clause. The acl_tags are what make the security pre-filter in Layer 2 possible, since you cannot filter on a field you never stored.
• Near-duplicate detection (SimHash or MinHash) at write time. Enterprise corpora contain the same policy in nine slightly different decks. Without dedup, your top five results are the same passage five times, and the model sees one idea where it should have seen five.

Layer 2: Embedding and indexing

The failures it prevents: exact identifiers are never retrieved, and users see passages they should not.

Two decisions dominate this layer. The second one is a security decision that often gets treated as a filtering detail.

• Embedding model. You are choosing along three axes: dimensionality (recall against storage and latency), domain fit (a general-purpose model underperforms badly on clinical or legal vocabulary), and where the data is legally allowed to be processed. text-embedding-3-large gives 3072 dimensions with truncation support. voyage-3 encodes queries and documents differently, since the two are asymmetric objects. Open-weight options like BGE-M3 cover the case where data residency rules out an API call. The model names turn over every few months; the three axes stay put.
• Index structure. HNSW is the default below roughly 10M vectors, tuned with M (graph connectivity, 16-64) and ef_construction (build-time search depth, 100-200). Both buy recall with memory and build time. IVF-PQ compresses vectors for billion-scale corpora and pays for it in recall. Approximate nearest neighbor search is approximate by design, and these knobs set how approximate.
• Hybrid retrieval, because dense search has a specific blind spot. Embeddings generalize, which is why they miss TS-4471, a SKU, a clause number, or an unusual surname. BM25 finds those because it matches literally. Fuse the two rankings with Reciprocal Rank Fusion. Averaging the scores breaks down because BM25 scores are unbounded while cosine similarity sits in a fixed range, so you would be adding two different units.
• Access control belongs in the search call itself. Pinecone, Weaviate, and Qdrant all support metadata pre-filtering natively. Applying ACLs to the results afterwards fails two ways. The restricted passage still entered the candidate pool, so a reranker may have read it and a summary may carry its influence even after the text is stripped. And the result set silently shrinks: you asked for ten, seven were removed, the model answered from three, and nothing in the logs says so.

Layer 3: Retrieval

The failure it prevents: the right passage was retrieved, ranked fourteenth, and you only sent five.

This layer splits into two jobs with different cost profiles. Vector similarity is cheap and approximate, so it handles recall: over-fetch 20 to 25 candidates. Cross-encoder reranking is expensive and accurate, so it handles precision: narrow to the 5 to 8 that go in the prompt. Skip the over-fetch and your ceiling becomes whatever the approximate ranking happened to put on top. Skip the narrowing and send all 25, and you dilute attention across mostly irrelevant text while burning the token budget.

The similarity metric follows from your embeddings: cosine for normalized vectors, dot product when magnitude carries signal, L2 for certain clustering-derived spaces. Most commercial embeddings are normalized, in which case cosine and dot product produce identical rankings.

The one that bites teams in production is embedding consistency. Query and document embeddings must match exactly on model, version, and preprocessing, including instruction prefixes like query: that some models require. A mismatch throws no error and quietly degrades recall, which makes it the first thing to check when recall drops after a deploy.

Layer 4: Context orchestration

The failure it prevents: the model had the answer in front of it and answered from the wrong passage anyway.

This is where most of the engineering effort in a mature system concentrates, because this layer decides what the model is allowed to see.

• Diversity via Maximal Marginal Relevance. Pure relevance ranking hands you five near-identical passages. MMR (lambda around 0.5-0.7) re-scores candidates to penalize redundancy, so a compound question gets coverage across its parts.
• Cross-encoder reranking (Cohere Rerank, bge-reranker-large). A bi-encoder embeds query and passage separately and compares the two vectors, so it can tell they are about the same topic. A cross-encoder reads the pair together in one forward pass, so it can tell whether the passage answers the question. That forward pass runs once per candidate, which is why you rerank 25 items and not 25,000.
• Compression with LLMLingua or extractive sentence selection reduces tokens while keeping the spans the answer depends on. Truncating by position is the common shortcut, and it drops conclusions, which tend to sit at the end of a passage.
• Ordering. Attention across a long context is uneven and the middle gets the least of it. Put the strongest evidence at the start and end of the context block.

Across all four, the target is the smallest evidence set that answers the question, enforced against an explicit token budget.

Layer 5: Generation

The failure it prevents: a fluent, confident, correctly formatted answer that the retrieved evidence does not support.

The model already knows things about SSO and contractor accounts from pretraining. Given retrieved context and no instruction on how to treat it, it blends what it just read with what it already knew, and the seam is invisible in the output. Grounding removes that ambiguity, and it lives in the prompt as an explicit contract.

Three things have to be enforced: answer only from the provided context, attach a passage ID to each claim, and say plainly when the context is insufficient instead of inferring. The third one is a product decision as much as a technical one. A system that says "I do not have this" beats one that guesses correctly 85% of the time, because with the guesser you cannot tell which 15% you are looking at.

Citation tagging is also what makes the next layer possible. Without a claim-to-passage link, measuring faithfulness reduces to a human reading two paragraphs and forming an opinion.

Layer 6: Evaluation, memory, and feedback

The failure it prevents: the system got worse and nobody noticed, and when someone finally complains, nobody can say which layer is at fault.

The four standard RAGAS-style metrics each isolate a different part of the stack, which turns a vague complaint into a bounded investigation.

• Context recall (did retrieval surface what was needed) points at Layers 1 to 3.
• Context precision (is what we retrieved relevant and well ranked) points at Layers 3 and 4.
• Faithfulness (is the answer supported by what was retrieved) points at Layer 5.
• Answer relevancy (did it address the question actually asked) points at generation or at query understanding upstream.

Alongside the scores, log per query: which candidates were retrieved, their scores, which survived reranking, and which were ultimately cited. Sampled logging misses the failures, which live in the tail. User behavior gives you free labels here. A thumbs-down is useful, and a user rephrasing the same question is the cheapest negative signal available.

Reading the symptom backwards

"Our RAG gives wrong answers, how would you debug it?" comes up in most RAG interviews. The answer works better as a diagnosis than as a list of techniques. Start at the symptom, name the layer, name the check.

The opening scenario maps onto three rows of that table. The stale clause was Layer 1, because no version_hash was stored. The wrong region seeing it was Layer 2, because ACLs were applied after retrieval. The inability to explain either one was Layer 6. None of the three was a model problem, which is typical.

Six questions and how to answer them

"Walk me through a RAG system." Give the six responsibilities rather than the five runtime steps, and point out that four of those steps carry assumptions the layers exist to protect. It shows you have thought about where the system breaks.

"Your RAG returns a wrong answer. How do you debug it?" Split it in two. Did the right evidence reach the prompt, or did the model fail to use evidence that was already there? Context recall answers the first question, faithfulness the second. Then walk the table above.

"Isn't semantic search strictly better than keyword search?" No. Embeddings generalize, so they are weakest where you need literal matching: identifiers, codes, rare proper nouns. BM25 handles those well. That complementarity is the argument for hybrid retrieval, and RRF is how you combine two rankings whose scores are not comparable.

"How do you stop it from hallucinating?" Push back on the framing first. Most "hallucinations" in RAG are retrieval failures, so check whether the evidence ever reached the prompt. For genuine generation failures the fix is structural: answer only from context, a passage ID per claim, an explicit abstain path. Then measure faithfulness to confirm it worked.

"How do you handle permissions?" ACL tags written as indexed metadata at ingest, applied as a pre-filter on the search call. Then explain the two failure modes of post-filtering: the passage still entered the candidate pool, and the result set shrinks without anyone noticing.

"When would you not use RAG?" When the corpus fits in the context window, retrieval only adds a failure mode. When the question requires reasoning over the whole corpus rather than a handful of passages, graph or agentic patterns fit better (Parts 4 and 5). When the answer requires computation or a live system of record, you want tools and APIs.

How the series is structured

Each subsequent part introduces a capability that addresses a specific, observable failure, following the same pattern: start with the simplest system that could work, measure it against real questions, identify the specific gap, and add exactly the capability that closes it.

Part 2 covers naive RAG, the simplest useful baseline and the right starting point for almost every system. Part 3 introduces advanced retrieval techniques for when the basic pipeline retrieves the wrong evidence. Part 4 covers routing and graph-based retrieval for questions that require different sources or relationship traversal. Part 5 covers agentic and iterative retrieval for questions that require multiple dependent steps. Part 6 brings the patterns together as a practical decision guide.

The more advanced patterns do not replace this core architecture. They add components inside these six layers, and every addition has an obvious home once the foundation is clear.