Some questions cannot be solved by improving similarity search alone. They require the system to choose the correct source, or to understand the relationships between entities.
Advanced retrieval improves how well a single pipeline finds evidence within a single index. Routing and graph-based retrieval address a different class of problem: questions where the right answer depends on knowing where to look, or on how pieces of information connect.
Click to enlarge
Routing RAG: choosing where to look
Routing sends different questions to different retrievers, systems, or tools based on query intent. A query about a product goes to the product documentation index. A revenue question goes to a financial database. An employee-policy question goes to the HR knowledge base. A question about current operational status goes to an API rather than a document index at all.
The router itself is usually lightweight - a small classifier or a single LLM call - relative to the retrieval it is directing. The complexity it earns is proportional to how genuinely different your data domains are from each other. If all your content lives in one coherent corpus, routing adds overhead without benefit. If you have multiple distinct domains with different content types, access rules, and retrieval strategies, routing is what makes unified search possible.
The router's misclassification rate must be measured explicitly. A question routed to the wrong domain returns plausible-looking but wrong evidence - harder to detect than a retrieval miss because the system still generates a confident answer from the incorrect source. Monitor routing accuracy as a first-class metric, not as an afterthought.
Graph-based RAG: following how information connects
Graph retrieval follows relationships between entities rather than matching on semantic similarity. At ingestion time, an LLM extracts entities and relationships from documents and builds a knowledge graph. At query time, retrieval can traverse those relationships explicitly.
The quality of the graph is determined entirely by the quality of entity extraction at ingestion. An LLM that misidentifies or incorrectly links entities builds a graph that traverses confidently to wrong answers. Graph construction quality must be evaluated against a labeled set of known entity relationships before the index serves live traffic - not after users report incorrect answers.
This matters for questions that vector search structurally cannot answer. Consider a supply chain question: a supplier provides a component, the component appears in a product, the product is affected by a recall, and the recall applies to a specific region. Vector search can find documents that mention any of these entities. Only graph traversal can follow the chain of relationships that connects them into an answer.
Graph RAG's ingestion cost is not one-time - it recurs every time the corpus changes. It is justified when relationship-aware queries represent a meaningful share of real traffic, evidenced by evaluation data, not because relationship reasoning sounds like a useful capability.
When to use each
Use routing when the problem is deciding where to look - when different question types require fundamentally different retrieval strategies, data sources, or tools. Use graph retrieval when the answer depends on how entities connect - when the information exists in the index but the relationship between pieces is what the question is actually about. They solve different problems and can be combined: routing to choose the right source, graph traversal to answer relationship questions within it.
Do you need Routing, Graph RAG, or neither?
Answer both questions based on your actual traffic, not anticipated future use cases.
Remember: routing chooses the evidence source. Graph retrieval follows relationships inside the evidence. Improving similarity search will not solve either problem.