When a Graph Earns Its Place in RAG

Benchmark work through 2025 and 2026 keeps returning the same awkward result: GraphRAG underperforms ordinary vector RAG on a large share of real tasks, while clearly beating it on a narrow set. That pattern is more useful than either the enthusiasm or the dismissal, because it turns the question from whether graphs help retrieval into which questions they help with, and whether your corpus contains enough of them to justify the cost.

This piece covers what the graph adds, how it gets built, the two query shapes where it pays, what it costs, and how to decide without building one first.

What the graph actually is

A graph is a set of nodes (things) connected by edges (the stated relations between them). The same structure sits behind a family tree or an org chart; here it is built from text rather than drawn by hand.

Standard RAG treats a corpus as independent chunks. Embed each chunk, embed the query, return the nearest few. The retrieval unit is a passage, and the relationships between passages are invisible to the system.

GraphRAG, in the form Microsoft Research published in 2024, runs an extraction pass over the corpus, pulling out entities and the relations between them, and assembles the result into a knowledge graph. It clusters the graph into communities and generates a summary of each one ahead of any query arriving. At query time it retrieves graph fragments (nodes, triples, paths, subgraphs) or community summaries rather than raw passages, and generates from those.

Two things change. The retrieval unit becomes structured, so the model receives “A supplies B, B acquired C in 2019” instead of three paragraphs that each mention one of those facts. And the system holds a representation of the whole corpus, in the community summaries, which plain RAG has no equivalent of.

Building the graph is a separable choice

Microsoft’s original pipeline extracts entities and relations with an LLM prompt, and most write-ups treat that as the method. It is one option among several, and the choice affects cost and coverage more than it affects the resulting graph’s shape.

Classical named-entity recognition paired with a relation-extraction model (or an OpenIE-style triple extractor) produces the same nodes and edges at a fraction of the token cost, without an LLM call per chunk. The trade-off is the one that affects any NER pipeline: a model trained on general text misses the relation types specific to a narrow domain, and a contracts corpus or a clinical record set will need extraction tuned to it rather than the off-the-shelf kind.

Entailment models offer a different edge type again. Instead of “supplies” or “acquired”, the edge becomes “confirms” or “contradicts” between two claims, which suits a corpus built from assertions that agree or disagree with each other (audit findings, conflicting witness statements, versions of a policy) rather than one built from entities acting on each other.

Where the relations already exist in structured form (an ownership register, a citation index, a CRM’s account hierarchy) no extraction step is needed at all. The graph is built directly from data already held, which is the case covered from the embedding side in Embedding-Based Relationship Discovery.

The two shapes where it pays

Multi-hop questions. “Which of our suppliers are exposed to the sanctions affecting our second-tier vendors” requires joining facts that appear in different documents and are never stated together. Vector retrieval returns chunks similar to the question, and no chunk resembles a question whose answer is a chain. The graph traverses the chain because the chain is what it stores.

Whole-corpus questions. “What are the recurring themes in three years of incident reports” has no answer in any single passage. Top-k retrieval returns five reports and the model summarises those five. Community summaries were built for exactly this case: the system answers from a hierarchical view of the entire corpus rather than from whatever the nearest-neighbour search surfaced.

Everything else is the majority case. When the answer sits in one passage, and the question resembles that passage, plain vector retrieval finds it, and the graph adds cost without adding accuracy. Ordinary document question-answering, policy lookup, support deflection, and most internal search fall here.

Where the embeddings go

The graph does not replace vector search; it usually sits downstream of it. In Microsoft’s design, entities and community summaries are themselves embedded, and a query’s nearest neighbours among those embeddings decide where in the graph to start: which node, which community. Traversal happens after that, across explicit edges, to pull in connected facts the query embedding alone would not have surfaced. Vector search finds the foothold; graph structure walks outward from it.

What it costs

Three costs, and the third is the one that catches deployments after they ship.

Indexing. Extraction runs over every chunk to pull entities and relations, then again to generate community summaries. On a large corpus this is a substantial one-off token bill and a slow build, and none of it answers a user’s question directly.

Query latency. Traversing the graph and summarising what comes back adds work at inference time. Reported figures put end-to-end latency at roughly two to three times a comparable vector pipeline, which matters for anything user-facing and matters less for batch analysis.

Incremental updates. The graph index and its summaries grow super-linearly with corpus size, and adding documents can shift community boundaries, invalidating the summaries built from them. A corpus that changes daily either gets rebuilt repeatedly or drifts out of date. This is the cost that decides most deployments, and the one least visible in a proof of concept run against a frozen document set.

Traversal, or folding relations into the index

The research response to the traversal cost has been to keep the relational signal and drop the walk. LightRAG folds entities and relations directly into a standard dense index and retrieves in two rounds, a fine-grained pass over specific entities and a coarser pass over thematic keys. There is no explicit path-walking step; “ROCm” and the frameworks it supports sit near each other in the same embedding space, found by nearest-neighbour search rather than by following an edge. This removes the traversal cost, allows the index to be patched incrementally, cuts indexing token cost by roughly 60% and roughly halves median query latency against full GraphRAG, without a clear quality penalty on multi-hop benchmarks.

If the reason for wanting a graph is multi-hop retrieval rather than whole-corpus summarisation, the folded-relations approach is the sensible starting point. Community summaries are the expensive component, and they exist to serve the global-question case specifically.

Deciding without building one

Take a sample of fifty real questions from whoever will use the system, before any implementation exists. Classify each one: does the answer sit in a single passage, does it require joining facts across documents, or does it require a view of the whole corpus. If the first category holds forty of the fifty, build vector RAG and spend the saved effort on chunking and reranking, which will move accuracy further than a graph would.

If the second and third categories carry real weight, the follow-up question is how often the corpus changes. A static corpus (regulations, published research, completed case files) absorbs the indexing cost once. A corpus with daily additions pays it repeatedly, and that recurring cost is what the decision turns on.

The general point holds beyond this choice: the retrieval architecture is a consequence of the question distribution, and the question distribution is knowable before anything is built. Most RAG projects skip that step and discover the distribution afterwards, from complaints.

A worked example

Using our article on GPU selection: a paragraph stating that the RX 7900 XTX runs on ROCm, and a separate table, further down the document, listing which frameworks ROCm supports. “Which frameworks run on the RX 7900 XTX” has no single-passage answer; a flat vector index retrieves whichever chunk is closest to “RX 7900 XTX” and stops there.

The script below extracts a handful of triples from that article by hand (standing in for an LLM or NER pass), builds the graph, picks an entry node with a toy embedding, and walks the graph to answer the question.

import re
from collections import defaultdict
from math import sqrt

import networkx as nx

# --- 1. Extraction --------------------------------------------------------
# Normally an LLM pass or a NER + relation-extraction model over the
# source article. Written out by hand here so the graph stays legible.

triples = [
    ("RTX 5090",        "uses_driver", "CUDA"),
    ("RTX 4090",        "uses_driver", "CUDA"),
    ("RTX 3090",        "uses_driver", "CUDA"),
    ("Jetson AGX Thor",  "uses_driver", "CUDA"),
    ("DGX Spark",        "uses_driver", "CUDA"),
    ("RX 7900 XTX",      "uses_driver", "ROCm"),
    ("W7900",            "uses_driver", "ROCm"),
    ("MI300X",           "uses_driver", "ROCm"),
    ("M3 Ultra",         "uses_driver", "Metal"),
    ("M4 Max",           "uses_driver", "Metal"),
    ("CUDA",  "supports", "PyTorch"),
    ("CUDA",  "supports", "vLLM"),
    ("CUDA",  "supports", "Ollama"),
    ("CUDA",  "supports", "llama.cpp"),
    ("CUDA",  "supports", "ComfyUI"),
    ("ROCm",  "supports", "PyTorch"),
    ("ROCm",  "supports", "vLLM"),
    ("ROCm",  "supports", "Ollama"),
    ("ROCm",  "supports", "llama.cpp"),
    ("ROCm",  "supports", "ComfyUI"),
    ("Metal", "supports", "Ollama"),
    ("Metal", "supports", "llama.cpp"),
    ("Metal", "supports", "ComfyUI"),
]

G = nx.DiGraph()
for subject, relation, obj in triples:
    G.add_edge(subject, obj, relation=relation)

# --- 2. A toy embedding, standing in for a real one ------------------------
# Bag-of-words cosine similarity over each node's name plus its immediate
# neighbours: enough signal to pick an entry point, no LLM call required.

def bow(text):
    return re.findall(r"[a-z0-9.]+", text.lower())

def node_text(node):
    neighbours = list(G.successors(node)) + list(G.predecessors(node))
    return " ".join([node] + neighbours)

def cosine(a, b):
    va, vb = defaultdict(int), defaultdict(int)
    for w in a: va[w] += 1
    for w in b: vb[w] += 1
    dot = sum(va[w] * vb.get(w, 0) for w in va)
    norm_a = sqrt(sum(v * v for v in va.values()))
    norm_b = sqrt(sum(v * v for v in vb.values()))
    return dot / (norm_a * norm_b) if norm_a and norm_b else 0.0

node_vectors = {n: bow(node_text(n)) for n in G.nodes}

def entry_point(query):
    query_vec = bow(query)
    scored = [(cosine(query_vec, vec), n) for n, vec in node_vectors.items()]
    return max(scored)[1]

# --- 3. Traversal: the step a flat vector index cannot do ------------------

def frameworks_for(device):
    driver = next(iter(G.successors(device)))
    frameworks = [n for n in G.successors(driver) if G[driver][n]["relation"] == "supports"]
    return driver, frameworks

query = "Which frameworks run on the RX 7900 XTX?"
device = entry_point(query)
driver, frameworks = frameworks_for(device)

print(f"Entry point: {device}")
print(f"Driver stack: {driver}")
print(f"Supported frameworks: {frameworks}")

This prints:

Entry point: RX 7900 XTX
Driver stack: ROCm
Supported frameworks: ['PyTorch', 'vLLM', 'Ollama', 'llama.cpp', 'ComfyUI']

The query names only the card, yet the answer joins a fact stated in one paragraph (device to driver) with a fact stated in a separate table (driver to frameworks), which is what “multi-hop” means in practice. Replace the hand-written triples with an LLM or NER extraction pass, replace the bag-of-words vector with a real sentence embedding, and the shape of the pipeline does not change. Fold the supports edges directly into node_vectors instead of walking them at query time, and this becomes the LightRAG.

(If you are working out whether your question set justifies a graph, get in touch.)