Prathap

rag, from first principles

most rag tutorials teach you an api. embed your documents, push them into a vector database, search, paste the results into a prompt. it works on the demo, and then it quietly falls apart on your actual data, and you have no idea which part broke.

this post is the other thing. i want to build rag up from the problem it solves, one piece at a time, and for every piece ask the same five questions: what problem does this solve, how does it work, why does it work, when does it fail, and what would i measure to know?

by the end you should be able to look at a rag system and say which component is costing you, rather than swapping frameworks and hoping.

#the problem, stated honestly

a language model knows an enormous amount about the world in general and nothing at all about your world. it has never seen your company's handbook, last tuesday's incident report, or the contract you signed in march.

the knowledge it does have is parametric — baked into its weights during training. it is frozen, it is unattributable, and you can't update it without retraining.

so the question is: how do we get the model to answer using knowledge that lives outside its weights?

that's it. that's the whole problem. every technique below is a different answer to that one question.

the original paper that named it — lewis et al., 2020 — framed it exactly this way: combine the model's parametric memory with a non-parametric memory it can look things up in.

the shape of the whole thing. everything in this post is a detail inside one of these boxes.

#first principle: why not just paste everything in

the obvious answer is: don't retrieve anything. context windows are huge now. dump the whole handbook into the prompt and let the model sort it out.

for small collections, you should actually do this. if your entire knowledge base fits in a prompt and the cost is acceptable, rag is over-engineering. i mean that seriously — a lot of rag systems exist because somebody assumed they needed one.

it stops working for three reasons, and they're worth knowing precisely because they tell you when the assumption flips.

1. it doesn't scale. a million documents don't fit in any context window, and they never will, because your document pile grows faster than the window does.

2. it costs money and time on every single request. you pay for the whole handbook on every question, even when the answer is one paragraph. latency scales with input length too. retrieval is a way of paying only for the part you need.

3. models get worse as the context gets longer. this is the unintuitive one. liu et al. found that models answer best when the relevant fact sits near the beginning or the end of a long context, and measurably worse when it's buried in the middle — the now-famous "lost in the middle" result.

so the context window is not a bucket you fill. anthropic's context engineering writeup puts it well: context is a finite resource with diminishing returns, and your job is to decide what the model sees at this exact moment.

retrieval, then, is not a storage trick. it's an attention budget decision.

#searching, family one: matching words

before embeddings, before transformers, search worked by matching words. it still does, and it's still very good, which surprises people.

start from scratch. i give you a question and a pile of documents. how do you score which document is most relevant?

your first instinct is to count how many of the question's words appear in each document. that's the right instinct. it's just naive in three specific ways, and fixing each one gives you a real scoring function.

bm25 is these three corrections, multiplied together and summed over the query's words.

#fix one: saturation

if the word "invoice" appears once in a document, that's a strong signal. if it appears fifty times, the document is probably about invoices — but it is not fifty times more about invoices. the payoff should flatten out.

 score
   │              ..........----------------
   │         ..---
   │      .--
   │    .-
   │  .-
   │ -
   └──────────────────────────────────────── occurrences
     1    2    3    5    10   20   50

bm25 controls how fast that curve flattens with a parameter called k1, usually somewhere around 1.2 to 2.0. small k1 means the curve flattens almost immediately — one mention is nearly as good as twenty.

#fix two: rare words matter more

a document matching "the" tells you nothing, because every document matches "the". a document matching "clopidogrel" tells you a lot.

so weight each word by how rare it is across the whole collection. that's inverse document frequency: words in few documents get a big weight, words in most documents get almost none.

this is the part that makes keyword search feel smart without any semantics involved. it isn't understanding your query — it's noticing which of your words are unusual.

#fix three: long documents cheat

a 400-page document will contain almost any word you search for, purely by being long. without a correction, length alone wins. bm25 divides by document length relative to the average, with a knob b (usually 0.75) controlling how aggressively.

put the three together and you get the scoring function:

              f(q,D) · (k1 + 1)
BM25 = Σ IDF(q) · ─────────────────────────────────
       q∈Q        f(q,D) + k1 · (1 - b + b · |D|/avgdl)
  • f(q,D) — how often the word appears in the document
  • IDF(q) — how rare the word is overall
  • |D|/avgdl — this document's length against the average
  • k1 — how quickly repeats stop helping
  • b — how hard to punish long documents

that's it. no neural network, no training, no gpu. if you want the full derivation, robertson and zaragoza's probabilistic relevance framework is the source, and it's more readable than its length suggests.

where it fails: bm25 has no idea that "car" and "automobile" are related. if the user's words aren't the document's words, it scores zero. this is the vocabulary mismatch problem, and it's the entire reason the second family exists.

#searching, family two: matching meaning

the fix for vocabulary mismatch is to stop comparing words and start comparing meanings.

turn every chunk of text into a vector — a point in a few-hundred-dimensional space — arranged so that text with similar meaning lands in nearby positions. then turn the question into a vector the same way and find the nearest points. (i wrote a whole post on what embeddings actually are if that sentence went by too fast.)

the key question is: how do you train a model so that "how do i reset my password" and "steps for credential recovery" land near each other?

you show it pairs that should be close, and punish it when they aren't. dense passage retrieval — dpr — is the clean version of this idea: two encoders, one for questions and one for passages, trained so a real question–passage pair scores higher than the same question against unrelated passages sitting in the same training batch.

the results were the wake-up call. on natural questions, dpr found the right passage in the top 20 about 78% of the time, against roughly 59% for bm25. the gap was too big to ignore, and dense retrieval became the default.

where it fails — and this is the part the vector-database marketing skips:

  • exact strings. error code E_4432, a part number, a specific function name. the embedding smooths these into the neighbourhood of "things that look like error codes". bm25 matches them exactly and instantly.
  • rare proper nouns. an internal project codename the embedding model has never seen becomes noise.
  • negation and small differences. "contract renewals before 2024" and "contract renewals after 2024" are very close in embedding space and mean opposite things.
  • domain drift. an embedding model trained on web text doesn't know that in your company, "closing" means something very specific.

notice the failure modes are almost exactly the inverse of bm25's. that's not a coincidence — it's the setup for the next step.

#hybrid: use both, then fuse

one method is great at exact words and terrible at paraphrase. the other is great at paraphrase and terrible at exact words. so run both.

the interesting problem is merging the two result lists. you can't just compare the scores: bm25 returns something unbounded that depends on your corpus, cosine similarity returns something between -1 and 1. they aren't on the same scale, and normalising them is fragile.

the trick is to throw the scores away and keep only the ranks.

reciprocal rank fusion: a document that both methods like moderately beats a document only one method loves.

this is reciprocal rank fusion, and the whole thing is one line of code. the constant 60 is the standard default; it damps the influence of the very top positions so one method can't dominate on its own.

why it works: agreement is evidence. a document that two independent methods both rank reasonably well is more likely to be right than one that a single method ranks first. you're ensembling.

hybrid search is the highest value-per-effort change in most rag systems, and databricks puts it in exactly that bucket in their retrieval quality guide — highest impact, lowest effort, do it first.

#chunking: the failure nobody looks at

here's the part that quietly destroys more rag systems than any retrieval algorithm.

you can't embed a 100-page document as one vector — you'd average away everything specific. so you cut it into chunks, a few hundred tokens each, and embed those.

and in doing so, you cut each chunk off from the context that made it meaningful.

the chunk is accurate, indexed, and useless. no query about acme corp will ever find it.

that chunk is now unfindable. someone asks "what was acme's revenue growth in q2 2024" and neither bm25 nor the embedding can connect the question to it, because the words and the meaning both left with the surrounding pages.

anthropic's contextual retrieval is the cleanest fix i've seen, and it is almost stupidly direct: before indexing a chunk, ask a cheap model to write one or two sentences explaining what this chunk is, given the document it came from. prepend that. index the result.

original chunk:
  "revenue grew by 3% over the previous quarter"

contextualized chunk:
  "This chunk is from ACME Corp's Q2 2024 SEC filing, in the
   section comparing Q2 2024 against Q1 2024 revenue.
   Revenue grew by 3% over the previous quarter."

the chunk now carries its own context. the embedding lands somewhere sensible, and bm25 can match on "acme" and "q2 2024" because those words are physically present.

the numbers from their experiments are the reason to take it seriously. measuring how often the right chunk failed to appear in the top 20:

  • baseline embeddings: 5.7% failure
  • contextual embeddings: 3.7% — a 35% reduction
  • plus contextual bm25: 2.9% — a 49% reduction
  • plus a reranker: 1.9% — a 67% reduction

the obvious objection is cost: you're making a model call for every chunk in your corpus. prompt caching kills most of it, since the document stays the same across all of its chunks. anthropic put it around a dollar per million document tokens, one time, at ingestion.

that's the shape of the tradeoff to notice — you can move work from query time to ingestion time. you pay once per chunk instead of once per question, and every question afterwards is faster. that's a pattern worth reaching for beyond this specific technique.

on chunk size itself, there is no universal right answer, and anyone who gives you one hasn't measured. databricks suggests 256–1024 tokens as the useful range to explore. small chunks are precise but lose context; large chunks carry context but dilute the embedding and waste your context budget. try three and measure — which brings us to the part of this post that actually matters, in a moment.

#reranking: the two-stage trick

your retriever returns twenty chunks. some are genuinely relevant and some just look relevant. can we do better?

yes, but first understand why the retriever is imperfect, because the reason points directly at the fix.

when you embed a document, you do it before you have seen the question. the vector has to be a single summary that works for every possible future query. that's a real compression loss, and it's the price of being able to search millions of documents in milliseconds.

a cross-encoder doesn't accept that compromise. it takes the question and one document together, in one forward pass, and lets every word of the question attend to every word of the document. it's far more accurate — and far too slow to run over your whole corpus, because it needs a full model pass per document.

cheap and approximate to narrow the field, expensive and accurate to order what's left.

so you use each where it's strong. stage one casts a wide net cheaply; stage two reads carefully within the net. pinecone's rerankers and two-stage retrieval is the best explanation of this architecture i know of, and their whole rag series is worth working through in order.

the thing that clicked for me: reranking lets you retrieve wide without paying for it downstream. if you were sending chunks straight to the model, you'd have to keep k small — every extra chunk costs tokens and dilutes attention. with a reranker in between, you can pull 100 candidates and still hand over only the best 5. recall goes up, and the context stays clean.

the cost is latency — an extra model call on the request path. databricks puts the quality gain from their built-in reranker at around 15% for rag agents, with the honest caveat that high-throughput applications may not be able to afford the round trip. measure both numbers, not just the quality one.

#the smaller knobs

these matter less than the above, but they're cheap and they compound.

top-k. how many chunks you finally pass to the model. too few and the answer isn't there; too many and you're paying tokens to bury the right chunk in the middle of the context. don't guess — sweep k = 1, 3, 5, 10, 20, 50 against your evaluation set and look at where quality stops improving. it plateaus earlier than most people expect.

query rewriting. users write "what about the second one" and "refund??". neither is a good search query. have a cheap model rewrite the question into something searchable first — expand the pronouns, add the implied context from the conversation, spell out the acronym. for ambiguous questions, generate two or three variants, search with all of them, and fuse the results with rrf.

metadata filtering. if the user asks about 2024, don't make the vector space figure that out — filter on a date field first and search the remainder. this is the single most underused trick in rag. a hard filter is exact, free, and removes entire categories of wrong answer that no amount of embedding tuning will fix.

#evaluation: where this stops being vibes

everything above is a menu of options. the thing that turns the menu into engineering is being able to tell whether a change helped.

without measurement you will end up saying things like "hybrid search feels better." that sentence has no information in it. the alternative sounds like:

"on our 80-question evaluation set, hybrid search took Recall@10 from 0.71 to 0.84 and added 40ms to p95."

now you can make a decision. you know what you gained, what it cost, and whether the trade is worth it for your application.

databricks is blunt about the ordering, and they're right: if you don't have evaluation in place, stop and set it up first. optimising retrieval without measurement isn't optimisation, it's redecorating.

#the evaluation set

it's less work than you think. 30 to 50 real questions with the chunk (or document) that should answer each one. pull the questions from your actual support tickets, search logs, or the people who will use the thing. handwritten questions are biased toward the words already in your documents, which is exactly the failure you're trying to catch.

#the metrics, in plain english

  • Recall@k — of the documents that should have been found, what fraction showed up in the top k? this is the one that matters most for rag, because a chunk the retriever misses is gone forever. the model cannot reason about something it never received.
  • Precision@k — of the k you returned, what fraction were actually relevant? this is about wasted context, not lost answers.
  • MRR — mean reciprocal rank. if the first correct result is at position 3, that question scores 1/3. averages over all questions. good when only the top hit really matters.
  • NDCG — like MRR but handles degrees of relevance, and discounts results further down the list. use it when "relevant" isn't binary.

if you track one number, track Recall@k at the stage before your reranker, because it's your ceiling. the reranker can reorder what it's given; it can't conjure a chunk the retriever never returned.

#failure analysis: the highest-value skill here

this is the section i'd keep if i had to delete the rest.

when a rag system gives a wrong answer, "the rag is bad" is not a diagnosis. there are five distinct places it can break, they have completely different fixes, and you can tell them apart in about two minutes by walking the pipeline backwards.

run this every time. the fix you reach for is almost never the one you'd have guessed.

a few things this tree will teach you fast:

ingestion failures are more common than anyone admits. a shocking amount of "bad retrieval" is a pdf that parsed into column-scrambled soup, or a table that became a run of unlabelled numbers, or a folder that was never synced. check this first. it's the cheapest check and it's wrong more often than the clever parts.

the last two look identical from the outside. the answer is wrong, the right chunk was in the context — so is the model ignoring it, or did the chunk never contain the fact in the first place? read the chunk. actually read it. this is the single most useful two minutes in rag debugging, and it's the one people skip in favour of changing the embedding model.

generation failures have retrieval-shaped fixes. if the fact was in the context and the model missed it, remember "lost in the middle" — try putting the highest-ranked chunk last, or cutting k so there's less to wade through. the bug is in the generator, but the lever is in retrieval.

and the reason to be careful about which one you're looking at: rag components interact. making chunks smaller improves precision and hurts context. adding a reranker helps ranking and hurts latency. you cannot debug this pipeline by staring at the end of it.

#so, is rag dead?

you'll see the claim regularly. it's worth taking seriously rather than dismissing, because each version of it is pointing at something real.

"long context killed it." partly true, at small scale — and i said the same thing earlier. but cost, latency and lost-in-the-middle all still apply, and none of them go away by waiting. retrieval is a way of choosing what the model sees, and that decision doesn't become unnecessary just because the window grew.

"agents just search, they don't need a vector index." this one is more interesting. give a model a search tool and let it decide when to call it, what to look for, and whether to look again — this is the react loop, and for code especially, a plain grep in an agent loop often beats an embedding index. but look at what that agent is doing: it's still retrieving. it's still ranking. it still has a chunking problem, a top-k problem, and an evaluation problem. the pipeline didn't disappear, it moved inside a loop and became harder to measure. everything in this post still applies, and failure analysis applies more.

"memory replaces it." nexxel's take is the sharpest version of this argument, and it's worth reading. the claim is that stateless similarity matching is the wrong primitive for personalised agents: it can't express that a fact was true until march, it can't walk from a person to their team to their manager, and it treats every question as if it were the first one. a structured memory — entities, relationships, validity windows — handles those, and it's a genuinely different primitive rather than a better retriever.

what i take from all three: "rag" was always a bad name for a real problem. the problem is getting the right information in front of the model at the right moment, and it isn't going anywhere. what's dying is the specific 2023 implementation — one embedding model, one vector database, fixed top-k, no evaluation. that thing deserves to die.

anthropic's building effective agents makes the same point from the other direction: the base unit is an llm augmented with retrieval, tools and memory. retrieval is a component of the system, not the architecture of it.

#the order i'd actually build it in

if you're starting today, resist doing all of it. build in this order, and measure after each step so you know what each one bought you:

  1. write 30 evaluation questions with known answers. before any code. this is the step everyone skips and it is the step that makes the others meaningful.
  2. bm25 only. yes, really. it's a strong baseline, it takes an afternoon, and it's the number every later improvement has to beat. plenty of systems never need more.
  3. add vector search and fuse with rrf. measure Recall@10 against step 2.
  4. look at the failures. walk the tree. the answer will surprise you, and it will tell you which of the next steps is worth doing.
  5. add a reranker — retrieve 100, keep 5. measure quality and p95 latency together.
  6. fix chunking, with context prepended if your documents have the structure that needs it.
  7. then query rewriting, metadata filters, a k sweep.

steps 1 and 4 are the ones that make you an engineer instead of a framework user. the rest is configuration.

#what to read, in what order

a short list, deliberately. read four things properly rather than forty badly.

start here

the papers, one per idea

don't collect papers. read one per concept, understand the mechanism, come back to the engineering writeups.

  • rag — parametric plus non-parametric memory, the original formulation.
  • bm25 and beyond — why lexical retrieval works. you don't need every equation, just tf, idf and length normalisation.
  • dpr — the jump from keywords to learned semantic representations.
  • lost in the middle — the result that justifies caring about context order at all.
  • self-rag — read it after the basics. it asks should i retrieve, is this useful, is my answer grounded, which is where retrieval starts turning into an agent.

when you get to agents

and the loop that makes it stick

reading doesn't produce intuition. this does:

read the explanation
       ↓
implement the smallest version
       ↓
run it against your 30 questions
       ↓
measure Recall@10
       ↓
look at what failed, and why
       ↓
next component

implement bm25 yourself once — it's maybe forty lines and it will teach you more about retrieval than any amount of reading. then add embeddings and compare. then fuse them. then rerank. at every step you'll have a number, and the number is the thing that turns "this seems better" into knowing.

that's the whole difference. everyone can assemble a rag pipeline now. very few people can tell you which part of theirs is broken.