Vector Databases & (Agentic) RAG¶

Agentic AI · L-RAG

Knowledge the model was not trained on: what retrieval repairs and what it breaks, the three stages of the pipeline, how the pipeline is measured, and the decision to let the agent retrieve for itself.

Prerequisite: the recap Linear Algebra.

Learning Objectives¶

1. State which failures of a language model retrieval repairs — arbitrary fabrication, stale knowledge, the reversal curse, knowledge that was never in the weights — and which failure it creates.
2. Describe the three stages of a retrieval pipeline — indexing, retrieval, generation — and name the decisions each stage fixes.
3. Decide the ingestion questions: where a document is cut, what the indexed string contains, and which embedding model produces the vector.
4. Explain approximate nearest-neighbour search, run HNSW by hand, and read the parameters a vector database exposes.
5. Compare lexical, dense and hybrid retrieval, fuse two rankings, and derive why the most accurate scorer can only ever be a re-ranker.
6. Choose a synthesis strategy over $k$ passages, and measure the pipeline: recall@$k$, precision@$k$, the three quality scores, and the difference between reference-based and reference-free evaluation.
7. Decide from those measurements whether retrieval should become a tool the agent calls.

Overview¶

Retrieval-augmented generation (RAG) supplies a language model with passages retrieved from an external document collection, so that an answer rests on knowledge that is not stored in the weights. Part I states which failures of a language model retrieval repairs and which failure it creates, Part II how documents are cut, embedded and indexed, Part III how a query is matched against a chunk, Part IV how $k$ retrieved passages become one answer, Part V how the pipeline is measured, and Part VI whether the agent itself decides when to retrieve. Ingestion, retrieval and synthesis are the three stages of the pipeline, in the order in which a document passes through them:

Part I — Knowledge outside the Weights¶

The question this part answers: why retrieve at all?

A language model's knowledge is parametric — it lives in the weights, and it is fixed at the moment training ends. Four consequences follow from that single fact. The model cannot hold what was private to an organisation. It cannot hold what happened after training stopped. It holds part of what it was taught in one direction only. And whatever it does hold, it holds without a source: no part of the weights records which document a fact came from.

Retrieval answers all four in the same way. The knowledge is moved out of the weights and into the context, where changing it costs an edit to a file rather than a training run, where the documents never leave the organisation that owns them, and where every sentence of the answer has a passage it can be attributed to. What that buys, what it does not buy, and what it breaks are the subject of this part.

The Limits of Parametric Knowledge¶

The weights are a store of finite capacity, and the training objective fills it in proportion to how often a fact appears. A fact met once contributes one term to the loss and competes for the same parameters as everything else, so the rare, the recent and the private are exactly what is held weakly or not at all.

Which of a model's mistakes can a document fix? Hallucination is one word for several different failures, and the answer differs for each, so the word has to be split before the question can be asked properly. The literature splits it along two independent axes.

Axis 1 — what the output conflicts with (Huang et al., 2025).

Axis 2 — whether the error is stable. Farquhar et al. (2024) reserve the term confabulation for a claim that is "both wrong and arbitrary — by which we mean that the answer is sensitive to irrelevant details such as random seed", and separate it from an error the model repeats.

The axes are independent — a factuality error may be arbitrary or stable — and it is the second that decides whether an index helps. Where the model is arbitrary it holds no belief, and the retrieved passage fills a vacuum. Where the model is consistent the passage must overrule what the weights already contain: a contest between what was trained and what was supplied in the prompt, and one whose winner is measured rather than assumed.

Which repair applies is decided by the pair of values, not by either axis alone.

Supplying a document is a repair by itself only where a factuality error is arbitrary. Where it is consistent, the passage has to win a contest against the weights; and the faithfulness row has no entries at all until a passage has been supplied.

What Retrieval Repairs, and What It Creates¶

Retrieval repairs what the weights do not hold; it does not settle what they hold wrongly, and it creates one failure an ungrounded model cannot have.

Knowledge and reasoning are separated, and only one of them is expensive to change. A new document is indexed in minutes; the same fact written into the weights costs a training run. The model is then kept for what it is good at — reading and reasoning over text placed in front of it — and the facts are kept where facts can be edited.

Retrieval-Augmented Generation¶

RAG is two steps: retrieve, then generate. A retriever selects passages from a corpus of your choosing, those passages are placed into the prompt, and the model answers from them instead of from its training data alone. Embeddings (→ L-Embed) make the first step possible: they place meaning in $\mathbb{R}^d$, so relevance becomes nearest neighbour by cosine.

RAG was named and formalized by Lewis et al. (2020) through a core framing that continues to define modern architecture: a system operating with two distinct memory types, only one of which is cheap and fast to update.

"We can update RAG's world knowledge by simply replacing its non-parametric memory." > — Lewis et al. (2020), §4.5

What this lecture builds is the simpler, later form. Lewis et al. fine-tuned the retriever and the generator together; what is built today prepends the retrieved documents to the input and changes nothing about the model. Ram et al. (2023) name that form the in-context RALM — retrieval-augmented language model — and measure what it buys: with an off-the-shelf BM25 retriever a 345 M-parameter model outperforms a 762 M one, and with a trained re-ranker a 1.5 B one. Retrieval buys the equivalent of a two- to fourfold increase in parameters, and it costs no training at all.

The Context Budget and the Retrieval Threshold¶

Retrieval is not the only answer to "the model does not know this." Where the knowledge fits comfortably into the context window, it is placed there directly and no index is built. What decides the threshold is not the size of the corpus but the context budget, which has four terms:

• Fit — a large corpus simply exceeds the window.
• Cost and latency — the context is re-sent on every call, so a large stuffed context is paid turn after turn.
• Attention — even where it fits, a long context buries the relevant part and accuracy falls (Liu et al., 2024).
• Freshness — an index is updated without rewriting the prompt and without retraining (Lewis et al., 2020).

A concrete anchor. A forty-page product handbook is roughly 25,000 tokens and fits in a 128,000-token window with room to spare, so it is placed in the context and no index is built. Fifty thousand support tickets are roughly 25 million tokens and will never fit. Between those two extremes lies the interesting case, and it is decided by the four terms above rather than by the word large.

The consequence is a spectrum rather than a switch:

• a small corpus is placed in the context;
• a medium one keeps an index and summaries in the context and loads whole documents on demand;
• a large one is served by retrieval over a vector index.

A second boundary, independent of size. The four terms above decide whether the corpus fits. They say nothing about a class of question that top-$k$ retrieval cannot serve at any $k$: "what are the main themes in this collection?". Similarity retrieval assumes the answer is localised — contained in a few records, with the rest irrelevant. Here every record contributes, so a retrieved subset is not a shorter answer but a different one, and enlarging $k$ does not repair it. Edge et al. (2025) classify such a question as query-focused summarization rather than retrieval, and answer it by summarising the corpus ahead of the query instead of ranking passages after it. The failure when a pipeline does not notice the difference is worth naming: every sentence of the answer may be supported by a retrieved passage, and the answer still misrepresents the corpus.

The Three Stages: Indexing, Retrieval, Generation¶

Retrieval is justified; what remains is its shape. A retrieval pipeline is three stages, see e.g. (Gao et al., 2024).

1. Indexing (Ingestion)
2. Retrieval
3. Generation (Synthesis)

Stages 1 and 2 together are the retriever, which maps a query to relevant chunks; the model in stage 3 is the generator. Retriever + generator = RAG (Lewis et al., 2020). Retrievers are dense (embeddings and approximate search), sparse (term matching) or hybrid, and are often refined by a re-ranker.

The two halves of the diagram are paid at different rates, and that governs every design choice in this lecture.

• Stage 1 is paid once per document, so an expensive chunker or a model that rewrites each chunk is affordable there.
• Stages 2 and 3 are paid once per query, so everything in them is measured in milliseconds and tokens. Work that can be moved from the right half to the left half is usually worth moving.

The third stage joins the retrieved text to the model through the prompt — the only one of several possible joining points that requires no access to the weights, which is why this course teaches it.

Standard Hybrid RAG¶

A production pipeline rarely retrieves by embedding alone. The same chunk is written into two indexes — a vector index over its embedding and a term index over its words — both are searched with the query, and the two rankings are combined into one.

The second retriever is lexical, also called sparse or term-based: it scores a passage by the words it shares with the query, weighting each word by how rare that word is in the corpus. TF-IDF is the classical weighting and BM25 its standard refinement; both are derived later in Lexical Retrieval: TF-IDF and BM25. Two properties matter for the diagram. The lexical side needs no model at all — an inverted index and a formula — and it fails on different queries than the dense side does: it matches a part number exactly and misses a paraphrase, where the embedding does the reverse.

The rest of the lecture takes this diagram apart: what a chunk is and where it is cut, what is embedded, what the two indexes are as data structures, how one ranking is made from two, and what the generator does with the passages once it has them.

Part II — Indexing/Ingestion: From Documents to an Index¶

The index decides what retrieval can possibly return, and every choice that fixes it is made before a single query arrives. The stage decomposes into seven steps:

1. load the documents
2. transform them into text
3. chunk them
4. extract metadata
5. embed each chunk
6. build an index over the vectors
7. store it.

Three of these seven steps involve permanent, unrecoverable decisions:

• where the document is chunked (step 3)
• what text goes into the embedded string (step 4)
• which model generates the embedding (step 5)

A mistake in any of these three stages creates a silent failure downstream: the correct answer simply disappears from the search space rather than returning an incorrect result, making it far harder to detect. Steps 6 and 7 (indexing and storage) shift to a classic data-structures problem, which brings this section to a close.

Chunking¶

A chunk is a smaller, self-contained snippet of text extracted from a larger document. Instead of feeding a 100-page PDF into a system all at once, the system breaks the document down into bite-sized pieces—typically ranging from a few sentences to a couple of paragraphs.

Why Chunking Matters¶

• Precision Retrieval: When a user asks a question, the search system finds the exact chunk containing the answer rather than retrieving an entire document filled with irrelevant information.
• Context Window Limits: AI models have limits on how much text they can read at one time. Chunks keep the context lean and focused.
• Embedding Accuracy: Search in RAG relies on converting text into numerical vectors (embeddings). Short, single-topic chunks generate clear vector meanings, whereas a 50-page document converted to a single vector becomes blurry and hard to match against user queries.

A chunk is two things at once, and they want opposite sizes:

• As the retrieval unit, it is what a single vector must summarize, where smaller is better because one vector can accurately represent one distinct idea.
• As the reading unit, it is what the generator receives, where larger is better because an accurate answer requires surrounding context.

One example, both directions. A one-sentence chunk reading "It must be filed within that period." is a precise retrieval unit and a useless reading unit: retrieved on its own it tells the generator neither what must be filed nor which period. A whole-chapter chunk is the reverse — it certainly contains the answer, but its single vector is an average over thirty pages, so it sits near nothing in particular. Every named chunking policy is an answer to that tension, and most of them work by making the two units different.

Chunking Policies in RAG¶

Citability. Only the last two rows have a reachable primary source. Fixed size, structure-aware, sentence-window and parent–child are described in the practitioner literature without a citation, an experiment or a dataset, and the survey that names them is a pointer source rather than a citable one.

In the common case a chunk is a few hundred tokens of contiguous text with a small overlap into its neighbours, so that a sentence cut by a boundary still appears whole in one of the two pieces.

This section nevertheless recommends no chunk size, and the reason is measurable. In the standard document-question-answering collections the evidence for a question occupies only 1.7 to 10.5 sentences of a very long document, so any chunker that does not cut through that one sentence returns it and the end-to-end score barely moves: 64.50 for fixed-size chunking against 65.53 for the best method the same authors could build (Lu et al., 2025). A benchmark can only measure a component whose failure changes the answer.

Metadata Extraction¶

A chunk is stored together with the fields that say where it came from — title, author, date, source URL or file path, section heading, page number, and whatever the corpus itself carries: product, language, customer, access level. They are extracted once the document has been cut and before the chunk is embedded, and they are kept beside the vector rather than inside it. Two uses follow, and only one of them touches the score.

Filtering restricts the candidate set before anything is ranked. A question about the current price list is run only against the chunks whose validity range covers today; a question from a user without clearance is run only against the chunks that user may read. Passages that would mislead, or that must not be shown, are then not candidates at all — a guarantee no amount of ranking quality provides. Gao et al. (2024) count metadata among the pre-retrieval optimisations, beside index granularity and query rewriting.

Chunk Augmentation¶

Every policy above answers the question what is returned once a chunk matches? Chunk augmentation answers a different one: what is matched against? Before indexing, a model that has been shown the whole document writes fifty to a hundred tokens situating each chunk, and the augmented text — not the original — is what is embedded and what enters the term index (Contextual Retrieval = Contextual Embeddings + Contextual BM25; Ford, 2024). The indexed string is then no longer a substring of the document.

The model is called once per chunk and is shown the whole document each time, so the cost falls entirely in the offline half of the pipeline, where it is paid once per document instead of once per query. Both indexes are then built over the augmented text — which is why one name covers two methods, contextual embeddings and contextual BM25.

It is the member of the family that reaches the chunk that is unidentifiable rather than too small, and one example carries the whole idea:

The italic half is what the model wrote; the rest is the chunk, unchanged. On its own the chunk names neither the company nor the quarter, so no term index matches a query about ACME in 2023 and no embedder places the chunk near such a query. Returning its parent would not help either: a parent is returned only if the child was retrieved, and it was not. The repair has to happen on the indexed string.

Reported on one corpus, the proportion of questions whose evidence is missing from the top twenty falls from 5.7 % to 2.9 % when both indexes are contextualised, and to 1.9 % with a re-ranker over 150 candidates. ⚠ Anthropic's own experiment (Ford, 2024), whose methodology is not published; the mechanism is established, the magnitudes are indicative.

How to implement Contextual Retrieval in practice, see e.g. in the Enhancing RAG with contextual retrieval cookbook from Antrophic.

Choosing the Embedding Model¶

The index can only rank what the embedder separated, so retrieval quality is decided one step before the index is built. A text embedder is trained contrastively — the matching document must out-score the others in the batch — so what the model calls similar is whatever its training pairs called similar, and none of that is visible from the vector store afterwards.

Contrastive training, concretely. A training example is a triple: an anchor, a positive that should land near it, and one or more negatives that should not.

• anchor — "How do I cancel my subscription?"
• positive — "Steps to terminate your account membership." — the same intent in different words
• negative — "How do I change my delivery address?" — the same vocabulary, a different subject

The loss is InfoNCE (van den Oord et al., 2018): a softmax over the scores of all candidates, in which the positive is the class to be predicted. Minimising it raises the anchor's score against the positive relative to every negative — so similar means what the training pairs called similar, and nothing else. The negatives are usually the other examples of the same batch, which makes the batch size part of the objective rather than a property of the hardware.

What that means concretely. Two embedders may disagree about whether "How do I cancel my subscription?" is nearer to "Termination of contract"—the same intent, different words—or to "How do I cancel my newspaper delivery?"—the same words, a different subject. Neither answer is wrong in general; each reflects its training data, and only one will fit a given corpus.

The case in one table. On short texts, models perform similarly, but context length shifts relative model performance:

On MTEB (truncated to 512 tokens), text-embedding-ada-002 scores 60.99—a respectable mid-field result. On LoCo (evaluated at 8192 tokens), its score drops to 52.7, while Nomic Embed—a 137M open model—outperforms Ada on MTEB at 62.4 and reaches 85.5 on LoCo (Nussbaum et al., 2025). Model rankings established on short-context benchmarks do not survive when documents get long.

Asymmetric Retrieval and Task Prefixes¶

One embedder places two kinds of text, and nearest does not mean the same thing for both. Asked which text is closest to "What is the capital of France?", a similarity task answers "What is the name of the capital city of France?" — a rephrasal — and a question-answering task answers "Paris is the capital of France." Both are right, and a single model trained on both kinds of pair therefore "receives conflicting reward signal" (Nussbaum et al., 2025, §4.2.2): one batch pushes questions toward questions, the next pushes questions toward answers.

The repair is to make the task an input. A short string is prepended to the text before it is embedded, telling the model which relation to encode:

The first two are asymmetric — the sides are structurally unlike, a short interrogative against a long declarative passage, and are tagged differently. The last two are symmetric. The strings themselves are one model's interface and not a standard: another embedder has other prefixes, a free-text instruction, or none.

Omitting a prefix is a silent failure. It is ordinary text, so nothing raises an error: a vector comes back, the search returns its nearest neighbours, and they are the neighbours of a point placed under the wrong relation. The loss shows up only as recall the corpus never had. And because the prefix is part of every stored string, changing it means re-embedding the corpus rather than re-running the query.

HyDE¶

Removing the asymmetry at query time — hypothetical document embeddings (HyDE). A task prefix tells the embedder which relation to encode. Gao, L. et al. (2023) remove the need for a query-side representation instead. An instruction-following language model is asked to "write a passage to answer the question"; the same encoder $f$ that embedded the corpus embeds that hypothetical passage, and the mean over $N$ sampled passages $\hat d_1, \dots, \hat d_N$ becomes the query vector:

$$\hat{\mathbf{v}}_q = \frac{1}{N}\sum_{k=1}^{N} f(\hat d_k).$$

Retrieval has become passage-to-passage similarity — a symmetric task, which an encoder trained without relevance labels already performs.

The hypothetical passage may be factually wrong, and the design does not depend on it being right: it serves only as a search key and is discarded after the search. The fixed-size vector keeps the vocabulary and the shape of an answer and loses most of its details; the authors describe the encoder as "a lossy compressor" in exactly this role.

Measured on the TREC Deep Learning 2019 web-search queries with nDCG@10, a rank-weighted score over the top ten results: the unsupervised encoder alone reaches 44.5, with HyDE 61.3 — level with the same encoder fine-tuned on labelled pairs (62.1). The gain depends on the generator: 48.9 with an 11-billion-parameter model, 61.3 with InstructGPT. The cost is one generation call before every retrieval.

Practical Takeaway¶

1. Ignore Generic Leaderboards: Standard benchmark ranks (like MTEB) do not guarantee performance on your real-world text.
2. Test on Your Own Data: You must evaluate embedders empirically on your specific documents, chunk lengths, and user queries by measuring Recall@$k$ (how often the correct source text appears in the top $k$ results).
3. The Hidden Failure: Getting embedding configuration wrong doesn't just produce bad answers—it prevents the search engine from finding the correct material entirely. The generator model can only answer using the documents it is actually given.

Vector Indexing & Approximate Nearest Neighbors (ANN)¶

Exact search (FLAT) compares a query against all $N$ vectors ($O(Nd)$ cost), which only scales up to a few thousand items. To scale to millions of vectors, Approximate Nearest Neighbor (ANN) algorithms trade a small amount of recall for drastically reduced search latency.

Every ANN method works by searching only a target subset of the dataset:

• IVF (Inverted File): Partitions space into Voronoi cells via $k$-means; checks only the nearest cells (nprobe: how many of those cells a single query actually searches).
• HNSW (Graph): Navigates a multi-layer graph from sparse top layers down to dense local neighborhoods (efSearch the size of the candidate list HNSW keeps while it searches ).
• PQ (Product Quantization): Compresses high-dimensional vectors into short byte codes to speed up distance calculations and reduce RAM usage.
• LSH (Locally Sensitive Hashing): Hashes vectors so similar items collide in the same bucket.

Part III — Retrieval: Matching a Query to a Chunk¶

Retrieval is the selection of a few passages from a large collection, given a query. Two components perform it.

• A scorer is a function $s(q,p)$ mapping a query $q$ and a passage $p$ to a number meant to grow with relevance;
• A ranking sorts the passages by that number and keeps the top $k$.

Three practical principles follow from how scoring and ranking work in practice:

1. Query and document encoders are decoupled (Bi-encoders): To make search fast over millions of documents, passages are embedded into vectors in advance. At search time, the query is encoded independently, allowing the system to compare vectors using fast approximate nearest-neighbor search.
2. Heavy cross-encoders only re-rank (Two-stage retrieval): The most accurate scoring models (cross-encoders that process query and document text together) are too slow to run over an entire database. They are only used as a second stage to re-order a small candidate pool fetched by a fast initial retriever.
3. Hybrid search beats single-method search: Combining rankings from two complementary retrieval methods (e.g., keyword-based sparse search + vector-based dense search) performs better than relying on a single scoring function.

Semantic Retrieval / Dense Retrieval¶

Semantic retrieval uses dense embeddings at query time to fetch contextually relevant passages. This dense half of the pipeline consists of three distinct operations, though only the first runs in real time once a user query arrives:

1. Encode the query: The query encoder maps the incoming prompt to a vector $\mathbf{q} \in \mathbb{R}^d$ via a single forward pass of the embedding model (taking only a few milliseconds).
2. Search the index: The vector index locates stored document vectors nearest to $\mathbf{q}$ based on cosine similarity $S_C(\mathbf{q}, \mathbf{d})$, typically using one of the approximate nearest neighbor (ANN) families described above.
3. Map back to text: The $k$ nearest vectors are resolved back to their original text chunks, passing the raw passage text—not the vector representations—to the generator.

Notice what is absent at inference time: no passage computation happens on the fly. Passage embeddings are computed once during offline ingestion, imposing the core constraint of scalable semantic search: a dual-encoder architecture can only scale if the corpus is fully embedded before the query ever exists.

The bi-encoder, also called a dual encoder: query and passage pass through the same embedding model separately, a pooling step (by default the mean over the token vectors) reduces each to one vector in $\mathbb{R}^d$, and the two meet only in the cosine (Reimers & Gurevych, 2019). The passage branch is computed offline, once per chunk; only the query branch is computed when the query arrives.

No query token ever attends to a passage token: everything the model knows about the pair is contained in two vectors computed without knowledge of each other.

Lexical Retrieval: TF-IDF and BM25¶

Where dense retrieval embeds text into continuous vector spaces, lexical (or sparse) retrieval scores passages purely by exact word overlap, requiring no neural models (Jurafsky & Martin, 2026, §11.1).

TF-IDF Mechanics¶

TF-IDF measures the importance of a term $t$ within a document $d$:

$$\text{tf-idf}(t,d) = \underbrace{\text{tf}(t,d)}_{\text{frequent in }d} \times \underbrace{\log\frac{N}{\text{df}(t)}}_{\text{rare across corpus (idf)}}$$

where $\text{df}(t)$ is the number of documents containing $t$, and $N$ is the total document count. A term scores high when it is frequent in a specific document but rare across the corpus. Both queries and documents are represented as high-dimensional, sparse vectors across the vocabulary and ranked by their dot product (Spärck Jones, 1972).

The Mathematical Weight of IDF¶

In a corpus of $N = 1{,}000{,}000$ documents using natural logarithms:

• A term appearing in only 10 documents yields $\text{idf} = \ln(10^5) \approx 11.5$.
• A term appearing in 500,000 documents yields $\text{idf} = \ln(2) \approx 0.69$.

Ubiquitous words like "the" receive $\text{idf} \approx 0$, effectively filtering them out automatically without needing manual stop-word lists.

BM25: The Modern Baseline¶

The High-Level Idea: While standard TF-IDF dampens term frequency logarithmically, its score can still grow indefinitely with high word counts. BM25 replaces unbounded growth with a strict mathematical ceiling: no matter how many times a keyword appears in a document, its term frequency score can never exceed a hard upper limit ($k_1 + 1$).

BM25 (Robertson & Zaragoza, 2009) refines standard TF-IDF through two critical adjustments:

1. Asymptotic Term-Frequency Saturation: Prevents keyword repetition from driving scores to infinity by capping the term's maximum contribution.
2. Dynamic Document-Length Normalization: Adjusts expectations based on average corpus length, ensuring longer documents aren't penalized or favored disproportionately.

It remains the standard lexical baseline in production search architectures.

The BM25 ranking function. For a query $q$ and a document $d$, BM25 adds one weight for every query term $t$ (Robertson & Zaragoza, 2009, Eq. (3.15), in the common variant with $(k_1+1)$ in the numerator, which multiplies every term alike and therefore leaves the ranking unchanged):

$$\mathrm{BM25}(q,d)\;=\;\sum_{t\,\in\,q}\ \mathrm{idf}(t)\cdot \frac{f(t,d)\,\bigl(k_1+1\bigr)}{f(t,d)\;+\;k_1\!\left(1-b+b\,\dfrac{|d|}{\overline{|d|}}\right)}$$

where $f(t,d)$ is the raw count of term $t$ in $d$, $|d|$ the length of $d$ in terms, $\overline{|d|}$ the average document length in the corpus, and $k_1>0$, $b\in[0,1]$ the two parameters (commonly $k_1\approx1.2$–$2.0$ and $b\approx0.75$ — conventions of the implementations, not constants of the model). The $\mathrm{idf}$ factor is the Robertson/Spärck Jones weight in the case of no relevance judgements, "a close approximation to classical idf":

$$\mathrm{idf}(t)\;=\;\log\frac{N-\mathrm{df}(t)+0.5}{\mathrm{df}(t)+0.5}$$

with $N$ and $\mathrm{df}(t)$ as in TF-IDF above.

Where the two refinements act. Saturation: with $b=0$ the fraction is $f(k_1+1)/(f+k_1)$, which increases with $f$ and tends to $k_1+1$ as $f\to\infty$ — the ceiling named on the slide. Length normalisation: the term $b\,|d|/\overline{|d|}$ enlarges the denominator for a document longer than average, so the same count $f$ earns a smaller weight; $b=0$ ignores length, $b=1$ normalises it fully. A query term absent from $d$ has $f(t,d)=0$ and contributes nothing, so the sum only needs the documents listed under the query's terms — the postings an inverted index stores.

Tokenization Mismatch¶

Lexical retrievers tokenize text into words or stems (applying lowercasing, stop-word removal, and stemming/lemmatization), rather than using the generator's sub-word Byte-Pair Encoding (BPE) tokens. Meanwhile, dense retrievers rely on their own specific embedding model's vocabulary. Consequently, the term "token" carries entirely different structural definitions across different stages of the retrieval pipeline.

Sparse and Dense Indexes: Cost and Quality¶

Measured over the same 21 M Wikipedia passages (Karpukhin et al., 2020): top-20 retrieval accuracy 78.4 % (dense) against 59.1 % (BM25) on Natural Questions — real, anonymised Google queries paired with a Wikipedia page, annotated with a long answer, a short answer, or null where the page answers neither (Kwiatkowski et al., 2019) — at 995 against 23.7 queries per second — but 8.5 h to build the vector index against about 30 min for the inverted one. Dense is the faster to query and by far the slower to build.

Sparse gives high precision where the overlap is lexical, dense higher recall on meaning, and hybrid retrieval — fusing the two rankings, optionally followed by a re-ranker — tends to beat either alone.

Hybrid Retrieval and Rank Fusion¶

A hybrid retriever runs a lexical and a dense retriever over the same corpus and combines the two rankings. It combines the ranks, not the scores, because the two numbers are calibrated against different things: a BM25 score moves with the corpus and with the rarity of the query's terms, while a cosine similarity lies in $[-1,1]$ and, for a trained embedder, is concentrated in a narrow band. Adding them requires a normalisation, and every normalisation is a parameter that does not transfer between corpora, query types or embedder versions.

Reciprocal rank fusion (Cormack et al., 2009) needs none. For a document $d$ and a set $R$ of rankings, each assigning $d$ the rank $r(d)$:

$$\mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + r(d)}, \qquad k = 60 .$$

Documents are then sorted by that score: no training, no calibration, no per-corpus tuning. The reciprocal shape keeps lower ranks from vanishing as an exponential would, and the constant $k$ "mitigates the impact of high rankings by outlier systems" (Cormack et al., 2009).

Two documents, worked out. Document $A$ is placed 3rd by the lexical retriever and 3rd by the dense one. Document $B$ is placed 1st by the dense retriever and is missing from the lexical ranking entirely:

$$\mathrm{RRF}(A) = \tfrac{1}{63} + \tfrac{1}{63} \approx 0.0317, \qquad \mathrm{RRF}(B) = \tfrac{1}{61} \approx 0.0164 .$$

$A$ wins, although no retriever ever ranked it first. That is the point of fusing at all: agreement between two different failure modes is evidence, and one confident system is not.

Retrieval Variants¶

The returned unit — what is indexed and scored need not be what the generator reads:

The ranking — which candidates are considered, and in which order:

Once the returned units differ in size, top-$k$ stops being a comparable setting. The retrieval parameter is then a token budget, of which top-$k$ is the special case in which every chunk has the same length. It is also why merging pays only where there is more than one granularity to choose between: over a single-level hierarchy the merged unit is the whole section, and the system degrades.

Re-ranking and the Cross-Encoder¶

A cross-encoder reads the query and the passage in one sequence, so attention runs across both and every passage token is represented in the presence of the query. It can therefore encode this passage answers this question rather than merely this passage is about this topic, which is why it scores better than any retriever above — and why nothing about the passage can be computed before the query is known. It is exactly the model the decomposability constraint excludes. So it is used wherever a small candidate set makes it affordable: as a re-ranker over the shortlist a fast retriever returned, and offline as the teacher from which fast retrievers are distilled.

The cross-encoder The cross-encoder processes the query and passage together as a single concatenated input sequence (Devlin et al., 2019). A classification head scores full cross-attention interaction directly. Because representations cannot be pre-computed offline, the model must run real-time inference across every candidate pair on the shortlist.

The contrast with the bi-encoder is the position at which query and passage meet: there in one cosine between two finished vectors, here in every attention layer.

A re-ranker cannot exceed the pool it is given, and that ceiling is measurable. Re-ranking the first thousand results of a lexical retriever, a far stronger neural retriever reaches recall@1000 of 81.4 — which is exactly the lexical retriever's own recall@1000, since re-ranking only re-orders that pool. Removing the lexical stage and searching the whole collection lifts the same model to 96.8, and improves the top of the ranking as well, because it reaches documents "missed entirely" by the first stage (Khattab & Zaharia, 2020). Two stages buy latency, and they cost recall.

Part IV — Synthesis¶

The passages have been retrieved. They are not yet an answer.

Synthesis is the downstream half of RAG: taking the retrieved passages and transforming them into a single coherent, grounded response. Perfect retrieval only supplies raw materials; synthesis governs how context is ordered inside the prompt, how conflicting evidence is resolved, and how strict generation boundaries prevent hallucinations.

Chunk Order and Grounding¶

Where a passage sits in the prompt changes whether it is used. Holding content fixed and varying only its position, accuracy follows a U-shaped curve: performance peaks when key information is placed at the very beginning or end, but plummets in the middle—sometimes dropping below closed-book baselines where providing documents performs worse than providing none at all Liu et al., 2024. Context layout is therefore an explicit ordering decision, and top-ranked passages belong at the edges rather than in the center.

Grounding is not verification, and the two are easy to confuse.

• Grounded means the answer was produced from the supplied passage.
• Verified would mean something checked that the passage is true and no stage of the pipeline does that.

Given documents whose answers had been deliberately falsified, and a system instruction warning that documents may contain errors, accuracy falls from 91 % without documents to 17 % with them, with the error detected in 1–8 % of cases (Chen, J. et al., 2024). The pipeline contains no step that compares the corpus against anything.

RAG models follow retrieved context blindly rather than fact-checking it.

Standard retrieval and generation do not automatically verify citations: Citation precision asks whether every claim in the answer is supported by the passage it cites. Retrieval-augmented instruction-tuned models left to themselves reach 2–6 % on long-form answers, where a system that explicitly judges support for each segment reaches 66.9–70.3 % (Asai et al., 2024). The judgement is a separate step; retrieval does not perform it.

To ensure reliable attribution, your RAG pipeline must treat attribution evaluation as an explicit, post-generation judging step.

Response Synthesis over $k$ Passages¶

Retrieval returns passages; the answer is built from them, and how it is built is a decision of its own with three standard forms. Write $k$ for the number of passages and $b$ for how many of them fit in one prompt.

What each strategy carries forward is what separates them. The single pass carries nothing and is the only form in which attention can compare two passages directly. Refinement carries the answer: every passage after the first is shown the answer so far and rewrites it, so the result is order-dependent and the last passage has the most influence over the wording. Merging carries intermediate summaries: each group is compressed once and the compressions are combined, so no passage ever sees another passage's text.

And the arithmetic, for $k = 20$ passages and $b = 5$ per prompt. The single pass makes 1 call. Refinement makes 20, strictly one after another. Merging makes $\lceil 20/5 \rceil = 4$ calls in the first round and $\lceil 4/5 \rceil = 1$ in the second — 5 calls in 2 rounds, and the four calls of a round run in parallel. The total work of merging is therefore linear in $k$ while its depth is logarithmic, and only depth is unavoidably sequential.

The recursion may be paid once instead of once per query. RAPTOR (Sarthi et al., 2024) runs the same cluster-and-summarise recursion over the corpus at indexing time — the summary tree of the chunking table — so that the merged summaries are retrievable chunks rather than work done per query. Restricting retrieval to any single layer of that tree scores about 57.9 on one document and allowing three layers 73.68: the summaries answer thematic questions no leaf contains, and the leaves answer the detail the summaries have compressed away.

Part V — Evaluation¶

Three stages of design decisions have now been made, and none of them can be settled by argument.

Retrieval is measured separately from generation, and it is measured first: a better generator or an agentic loop placed around a retriever that never returned the evidence inherits the miss, and no amount of work on the second stage recovers it. Separate measurement is what tells the two apart — and it is also what decides the question of the final part.

Retrieval Metrics and the Three Quality Scores¶

Measure retrieval before improving it. The retrieval stage is measured on its own, against a set $R$ of gold passages marked relevant for a query by hand. Write $T_k$ for the top $k$ the retriever returned:

$$\text{recall@}k = \frac{|R \cap T_k|}{|R|}, \qquad \text{precision@}k = \frac{|R \cap T_k|}{k}.$$

Worked. Four passages in the corpus answer the query, and the retriever's top 10 contains three of them. Then $\text{recall@}10 = 3/4 = 0.75$ and $\text{precision@}10 = 3/10 = 0.30$.

The two answer different questions, and recall@$k$ is the one that bounds everything downstream: it asks whether the evidence reached the generator at all, and nothing later in the pipeline can recover a passage that was never returned. Precision@$k$ asks how much of the context window was spent on passages that were not needed.

Three further scores measure the pipeline end to end — the triad of Es et al. (2024), collected in Gao et al., 2024, §VI.C. A query produces three objects — the query itself, the retrieved context, the response — and the three scores are the three edges between them:

Faithfulness and relevance are not the same score, and an example separates them. Asked "how long is the refund window?" over a context stating "14 days", the answer "30 days" is perfectly relevant and flatly unfaithful; the answer "our support team is available 24/7" is faithful to nothing in particular and not relevant. Only faithfulness is checkable against something outside the model, which is why citation precision — is every claim supported by the passage it cites? — is the form in which faithfulness is usually made concrete.

Faithfulness¶

Faithfulness is the share of the answer's statements that the retrieved context supports. Es et al. (2024) compute it, like the two scores that follow, by prompting a model, and all three are reference-free: they use the question $q$, the retrieved context $c(q)$ and the answer $a_s(q)$, never a gold answer. Faithfulness takes two prompts:

1. Decomposition. The model rewrites the answer as a set $S$ of short statements, each of which makes a single assertion.
2. Verification. For each statement $s_i \in S$, the model returns the verdict yes or no to the question whether $s_i$ can be inferred from $c(q)$. The statements that receive yes form the subset $V \subseteq S$ of verified statements.

$$F = \frac{|V|}{|S|} \in [0, 1].$$

Worked. The context states "Refunds are accepted within 14 days of delivery." The answer "Refunds are accepted within 14 days, and the customer pays the return shipping." decomposes into $s_1$ = refunds are accepted within 14 days and $s_2$ = the customer pays the return shipping. The context supports $s_1$ and says nothing about $s_2$, so $V = \{s_1\}$ and $F = 1/2$.

An unsupported statement lowers the score because it enters the denominator $|S|$ without entering the numerator $|V|$ — whether $s_2$ is true does not matter, only whether the context supports it. Since the decomposition is itself a model call, a different split of the same answer changes $|S|$, and $F$ with it: had $s_1$ been split into two statements, $F$ would be $2/3$.

Answer Relevance¶

Answer relevance measures whether the answer addresses the question that was asked, and it does so in reverse. From the answer alone, the model generates $n$ questions $q_1, \dots, q_n$ that the answer would answer. Each generated question and the real question $q$ are embedded, and $\mathrm{sim}(q, q_i)$ is the cosine similarity of the two embedding vectors:

$$AR = \frac{1}{n}\sum_{i=1}^{n}\mathrm{sim}(q, q_i).$$

An answer to a different question generates questions close to that other question, so each $\mathrm{sim}(q, q_i)$ falls, and $AR$ with it. The authors state that the score also penalises an answer that is incomplete or contains redundant information; the route is the same, since questions generated from such an answer cover only part of $q$, or cover topics that $q$ does not contain.

The score is blind to factuality by construction, as its authors state. In the refund example, the wrong answer "30 days" generates "how long is the refund window?" as readily as the correct "14 days" does: the generated question depends on what the answer is about, not on whether it is true.

Context Relevance¶

Context relevance is the share of the retrieved context that is needed to answer $q$. The model extracts from $c(q)$, without changing them, the sentences needed to answer $q$ — or returns "Insufficient Information" when there are none — and the score divides their number by the length of the context:

$$CR = \frac{|S_{\text{ext}}|}{\text{number of sentences in } c(q)}.$$

Worked. The retriever returns three passages of 20 sentences in total, and one sentence states the 14-day refund window. Then $|S_{\text{ext}}| = 1$ and $CR = 1/20 = 0.05$, although the context contains the correct answer.

The score is therefore a penalty on length, not on error: every retrieved sentence enters the denominator, and only a needed sentence enters the numerator. The authors motivate the penalty by the cost of long contexts and by the weaker use a model makes of information placed in the middle of a long context (Liu et al., 2024).

All three scores are judgements of a model, so each inherits the biases of a judge; and answer relevance is a cosine between embeddings, which cannot become a factuality score however well it correlates.

Reference-Based and Reference-Free Evaluation¶

The three scores above share one property, and naming it divides evaluation in two.

Why the left column is not enough. It measures word overlap, not correctness: against the gold answer "14 days", the perfectly correct "two weeks" scores zero on exact match and close to zero on ROUGE. The effect is not a curiosity of that one example — re-running 23 summarization systems under a single protocol against expert human annotations, ROUGE and BERTScore correlate poorly with human judgement (Fabbri et al., 2021). And a corpus that changes weekly never has a current labelled set at all.

Why the right column is not enough either. The score is produced by a model, so it inherits that model's biases. G-Eval gives the judge the criterion, has it write its own evaluation steps, and takes the probability-weighted sum over the score tokens rather than the integer the model emits, reaching a Spearman Correlation Coefficient of 0.514 with human judgement on that same benchmark — ahead of every reference-based metric in the table there (Liu, Y. et al., 2023). Its own authors nevertheless report a "bias towards the LLM-generated texts."

So the two are used together. Report a judged score with a second judge and the disagreement between them, keep a reference-based metric beside it, and treat a difference that only one of the two can see as a hypothesis rather than as a result.

Part VI — Agentic RAG¶

The pipeline built so far fixes three decisions before it has read a single passage: it retrieves for every query, it retrieves with the query as the user wrote it, and it accepts whatever the top $k$ contains. Agentic RAG returns these decisions to run time, where they can depend on what retrieval has already returned.

The survey literature arranges RAG systems on four rungs, which differ in control flow rather than in quality (Gao et al., 2024):

Most production systems stand on the second rung. It improves every step of the chain, but the steps and their order are still fixed before anything has been read. Whether the fourth rung repays its additional calls is decided by measurement, not by its position on the ladder.

The same four rungs, drawn as the control flow each one executes for a single query (the first three after Singh et al., 2026, Figs. 3–5):

In the first three the path is fixed by the designer of the system, even where it branches or repeats; in the last it is chosen by the model at run time, and the number of retrievals is unknown before the query arrives.

Agentic RAG in Brief¶

Agentic RAG is retrieval-augmented generation in which an agent, and not a fixed pipeline, decides how to retrieve (Singh et al., 2026).

In the agentic RAG retrieval is a tool inside the agent loop (→ L-ReAct): the model reasons, calls a tool, observes the result, and repeats until it decides to answer. Every retrieval is therefore preceded by a decision, and every result is read before the next decision is made.

The loop makes five decisions that the static pipeline makes once or not at all:

1. Analyse the query — is retrieval needed, and how much? A question the weights can answer needs none; a question whose second fact depends on the first needs several retrievals.
2. Select the source — a vector index over documents, a relational database queried in SQL, a web search or another API. The agent acts as a router over knowledge sources (Singh et al., 2026, §5).
3. Formulate the query — rewrite the question, divide it into sub-questions, or use a term that appeared only in an earlier result.
4. Judge the evidence — are the returned passages relevant, and do they suffice? If not, the loop returns to the second or third decision.
5. Generate and check — write the answer from the evidence and test whether every claim is supported by a passage. An unsupported claim is a further reason to retrieve.

Each return edge in the diagram costs one more retrieval and at least one more model call, so a production loop also carries a stopping criterion — a maximum number of iterations or a token budget — after which it answers from the evidence it holds.

Agentic Patterns in the Loop¶

Each of the five decisions is an instance of a pattern the agent literature already names. Singh et al. (2026), §3, build agentic RAG from four such patterns:

The diagram is the reflection pattern (after Singh et al., 2026, Fig. 8).

Retrieval as a Tool¶

What an agent can do after a failed retrieval is bounded by the operations its retrieval tool offers. A tool with one operation, search(query), permits one response to a poor result: rephrase the query and search again. An agent that can additionally search lexically, restrict a search to one named entity, and include or exclude documents from its own candidate set scores 22.5 % higher in exact match, relative, than the strongest agent limited to issuing queries (Hui et al., 2026). The tool interface is therefore a design decision of the same weight as the embedder.

The agent knows its retrieval options only from their definitions in its context. Each tool is presented to the model as a name, a natural-language description and a parameter schema; the model selects a source by emitting a call to one of these names with arguments that satisfy the schema, and nothing outside the definitions is visible to it. Whether a question about an order reaches the SQL database rather than the index over the manuals is therefore decided by what the two descriptions say. The definitions belong to what Yang et al. (2024) call the agent–computer interface (ACI): the commands available to the model and the form in which their results are returned to it.

Self-RAG (Asai et al., 2024) is a language model trained to emit reflection tokens — special tokens with which it requests a retrieval, grades a retrieved passage and grades its own output. They encode the three judgements the static pipeline never makes: retrieve at all? · is this passage relevant? · is the answer supported by it? Agentic RAG makes the same judgements with an unmodified model, through prompts and tool calls in the agent loop; Self-RAG moves them into the weights, which requires training the generator.

One trace that the static pipeline cannot produce. The question is "How long is the refund window?", asked of a corpus in which the answer is divided between two documents:

search("refund policy")           →  §4: "Refunds are governed by the terms in Annex B."
search("Annex B refund window")   →  Annex B: "14 days from delivery."
answer: 14 days from delivery.

The first result does not answer the question; it names where the answer is. The second query could not have been written in advance, because it contains a term — Annex B — that appears only in the first result. A static pipeline embeds the original question once, retrieves once, and stops.

The Retriever Ceiling¶

A fixed-context reader cannot outperform its retriever. It sees exactly the top $k$ chunks it was handed, so "if the embedding search retriever fails to surface the gold article using the provided question, the fixed-context baselines are guaranteed to never see the gold article" (Packer et al., 2024). That is a ceiling, not a tendency: whatever the reader's quality, the evidence it can see is bounded above by recall@$k$, and no amount of prompt work on the reader raises it.

An agent that can query again and page onwards is not bounded that way. A second query with different wording, or the next page of the same ranking, reaches documents the first call never showed. Agentic retrieval therefore converts a one-shot top-$k$ into a search over the index, and the ceiling moves from recall@$k$ to recall over the whole ranking.

Where it matters most is the multi-hop question, in which the value retrieved is itself the key to the next lookup, so a single retrieval cannot succeed by construction. Packer et al. (2024) measure it on a synthetic nested key–value task: 140 pairs of random identifiers (UUIDs), about 8,000 tokens, in which a value may itself be a key, so that $n$ nesting levels require $n+1$ lookups. The identifiers are random, so no model holds them in its weights, and the fixed-context baselines receive all 140 pairs in their context window: the only difficulty is chaining the lookups. GPT-3.5 falls to 0 % accuracy at one nesting level, GPT-4 by three. MemGPT with GPT-4, which looks up each key with a search function call, is unaffected by the nesting depth; with GPT-4 Turbo or GPT-3.5 it improves on the corresponding baseline but still declines from two levels, "as a result of failing to perform enough lookups".

The gain is theoretical until the agent uses it: the same authors observe that their agent "will often stop paging through retriever results before exhausting the retriever database". Agentic RAG buys a higher ceiling, not a guarantee of reaching it.

Agentic RAG Architectures¶

The architectures differ in where the retrieval decisions are made and how many contexts hold them. One agent may make every decision in one context, or the decisions may be distributed over several agents, each of which sees only its own part (Singh et al., 2026, §5):

The last row changes the task rather than the control flow: the input is a document to be processed, not a question to be answered, and the output is a structured record rather than an answer. The term was introduced by a framework vendor (LlamaIndex) and is not yet established in the research literature; the survey's own diagram of it is taken from that vendor's blog.

Agentic versus Static RAG¶

Agentic RAG buys recall on questions that one retrieval cannot answer, and pays for it in calls, latency and predictability; on questions that one retrieval does answer, it buys nothing.

In Practice / This Week's Lab¶

lecture-RAG-exercises.ipynb — by hand: recall@$k$ and precision@$k$; TF-IDF; reciprocal rank fusion; the curse of dimensionality; an HNSW search and the size of the index it needs.

lecture-RAG-lab.ipynb — build it: a tiny RAG index from scratch, then a vector database behind a search tool your agent decides to call, an evaluation set of your own, and a sparse/dense/fused comparison scored against it.

Summary¶

The six questions, answered.

• Why retrieve? Retrieval repairs the failures whose cause is knowledge missing from the weights — stale, private and one-directional facts — by editing a corpus instead of a model. It does not repair a belief the weights hold confidently, and it creates one new failure: an answer unfaithful to the passage it cites.
• What goes into the index? Where the document is cut, what the indexed string says, and which embedder produced the vector. None of it is visible afterwards, and an error there costs a missing answer. The index itself is a data structure: ANN makes search sublinear, HNSW is a skip list over proximity graphs searched best-first, and the curse of dimensionality is why the approximation is needed at all.
• How is a query matched? By a similarity function whose passage side must be computable in advance — which is why a retriever is two encoders, why the cross-encoder that answers no to that becomes a re-ranker, and why a re-ranker can never exceed the pool it is given. Hybrid retrieval fuses ranks, not scores.
• How do $k$ passages become one answer? Order them, ground the answer in them, make each claim traceable — then choose a single pass, iterative refinement, or a recursion that may be paid once at index time instead of once per query.
• How is it measured? Recall@$k$ and precision@$k$ for retrieval, the three quality scores for the pipeline, and a reference-free judge read beside a reference-based metric, never alone.
• Should the agent retrieve for itself? Agentic RAG returns the retrieval decisions to run time — whether to retrieve, from which source, with which query, and whether the evidence and the answer suffice. It raises the retriever ceiling on questions one retrieval cannot answer, at the price of calls, latency and a path that differs per query; it does not repair the index — so fix the index first, and let the measurements decide.

References¶

Peer-reviewed papers

• Asai, Akari, et al. "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." International Conference on Learning Representations. Vol. 2024. 2024. arXiv:2310.11511.
• Berglund, Lukas, et al. "The Reversal Curse: LLMs trained on 'A is B' fail to learn 'B is A'." International Conference on Learning Representations. Vol. 2024. 2024. arXiv:2309.12288. — a fact fine-tuned in one direction is answered 96.7 % that way and 0.1 % in reverse, with no detectable likelihood difference from a random name, and no training-side repair; the same sentence in context is usable in both directions.
• Chen, Jiawei, et al. "Benchmarking Large Language Models in Retrieval-Augmented Generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. 2024, pp. 17754–17762. arXiv:2309.01431. — the RGB benchmark: counterfactual robustness 91 % → 17 %, negative rejection at ≤ 45 %, and the near-miss definition of noise. (Cited as Chen, J. et al.)
• Cormack, Gordon V., Charles L. A. Clarke, and Stefan Büttcher. "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods." Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2009, pp. 758–759. — RRF: $\sum_r 1/(k+r(d))$ with $k=60$, and the measured flatness of that constant.
• Cuconasu, Florin, et al. "The Power of Noise: Redefining Retrieval for RAG Systems." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024, pp. 719–729. arXiv:2401.14887. — distracting passages (the retriever’s own top scorers, without the answer) cost up to 0.24 accuracy; random passages help by up to 35 %.
• Devlin, Jacob, et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019, pp. 4171–4186. arXiv:1810.04805. — the [CLS]/[SEP] sentence-pair input and the classification layer on the final [CLS] vector: the cross-encoder.
• Es, Shahul, et al. "RAGAs: Automated Evaluation of Retrieval Augmented Generation." Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 2024, pp. 150–158. arXiv:2309.15217. — the three reference-free scores and their definitions: faithfulness as the supported share of extracted statements, answer relevance as the mean similarity of questions generated back from the answer, context relevance as extracted sentences over total.
• Fabbri, Alexander R., et al. "SummEval: Re-evaluating Summarization Evaluation." Transactions of the Association for Computational Linguistics, vol. 9, 2021, pp. 391–409. arXiv:2007.12626. — 23 summarization systems re-run under one protocol with expert and crowd human annotations: ROUGE and BERTScore correlate poorly with human judgement, which is the case against reference-based evaluation alone.
• Farquhar, Sebastian, et al. "Detecting hallucinations in large language models using semantic entropy." Nature, vol. 630, 2024, pp. 625–630. — confabulation defined as the arbitrary subset of hallucination, and entropy computed over meanings rather than over token sequences.
• Gao, Luyu, et al. "Precise Zero-Shot Dense Retrieval without Relevance Labels." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023, pp. 1762–1777. arXiv:2212.10496. — HyDE: a hypothetical passage as the query vector, which turns retrieval into a symmetric task. (Cited as Gao, L. et al. to separate it from Gao, Y. et al.)
• Huang, Lei, et al. "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions." ACM Transactions on Information Systems, vol. 43, no. 2, 2025, art. 42. arXiv:2311.05232. — the factuality / faithfulness taxonomy this week's first slide uses. (A survey: cited for its own taxonomy, not for a measurement.)
• Hui, Yulong, et al. "Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrieval." International Conference on Learning Representations. Vol. 2026. 2026. arXiv:2510.27566. — the retrieval tool's action space as a design decision: lexical search, entity anchoring and candidate-set primitives beside search.
• Indyk, Piotr, and Rajeev Motwani. "Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality." Proceedings of the 30th Annual ACM Symposium on Theory of Computing (STOC 1998), 1998, pp. 604–613. — LSH.
• Jégou, Hervé, et al. "Product Quantization for Nearest Neighbor Search." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 1, 2011, pp. 117–128. — IVF / PQ.
• Jeong, Soyeong, et al. "Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024, pp. 7036–7050. arXiv:2403.14403. — routing each query to no, single-step or multi-step retrieval before the first call; the labels built from the simplest strategy that succeeded.
• Jiang, Huiqiang, et al. "LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, pp. 1658–1677. arXiv:2310.06839. — context compression by the shift in perplexity the question causes, i.e. conditional pointwise mutual information.
• Johnson, Jeff, et al. "Billion-scale similarity search with GPUs." IEEE Transactions on Big Data, vol. 7, no. 3, 2021, pp. 535–547. arXiv:1702.08734. — FAISS.
• Karpukhin, Vladimir, et al. "Dense Passage Retrieval for Open-Domain Question Answering." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020. arXiv:2004.04906. — the decomposability constraint that forces two encoders, and the sparse/dense quality and cost comparison over 21 M passages.
• Khattab, Omar, and Matei Zaharia. "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT." Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020, pp. 39–48. arXiv:2004.12832. — the measured ceiling a re-ranker inherits from its candidate generator (recall@1000 81.4 → 96.8).
• Kwiatkowski, Tom, et al. "Natural Questions: A Benchmark for Question Answering Research." Transactions of the Association for Computational Linguistics, vol. 7, 2019, pp. 453–466. — real search queries, a Wikipedia page from the top five results, and long / short / null answers; the null option is what lets it score abstention.
• Lewis, Patrick, et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Advances in Neural Information Processing Systems. Vol. 33. 2020. arXiv:2005.11401. — parametric vs. non-parametric memory, and the index swap that changes what the system knows.
• Liu, Nelson F., et al. "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the Association for Computational Linguistics, vol. 12, 2024. arXiv:2307.03172. — the U-shaped accuracy curve over the position of the relevant passage, and mid-context accuracy falling below the model's own closed-book 56.1 %.
• Liu, Yang, et al. "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023, pp. 2511–2522. — the reference-free judged metric: an automatically generated chain of evaluation steps, a form-filling prompt, and the probability-weighted score; Spearman 0.514 with human judgement, and its authors' own caution that such evaluators are biased toward model-generated text.
• Longpre, Shayne, et al. "Entity-Based Knowledge Conflicts in Question Answering." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021, pp. 7052–7063. arXiv:2109.05052. — knowledge conflict: substituting the answer entity throughout the retrieved passage measures how often the reader answers from its weights anyway, and the five factors that move it — question familiarity, model size, the retrieval quality the reader was trained with, the answer's entity type, and entity popularity.
• Ma, Xinbei, et al. "Query Rewriting in Retrieval-Augmented Large Language Models." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023, pp. 5303–5315. arXiv:2305.14283. — Rewrite-Retrieve-Read: the rewrite step before the retriever, and the HotpotQA result that retrieval with the raw question can lower accuracy.
• Malkov, Yu A., and D. A. Yashunin. "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 4, 2020, pp. 824–836. arXiv:1603.09320. — HNSW.
• Nussbaum, Zach, et al. "Nomic Embed: Training a Reproducible Long Context Text Embedder." Transactions on Machine Learning Research, 2025. arXiv:2402.01613. — retrieval quality decided before the index, and the MTEB→LoCo reversal (text-embedding-ada-002 60.99 → 52.7) that argues for evaluating the embedder on your own documents.
• Ram, Ori, et al. "In-Context Retrieval-Augmented Language Models." Transactions of the Association for Computational Linguistics, vol. 11, 2023, pp. 1316–1331. — the primary for the pipeline this lecture builds: "leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM." A 345 M model with BM25 beats a 762 M one, and with a trained re-ranker a 1.5 B one.
• Reimers, Nils, and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019, pp. 3982–3992. arXiv:1908.10084. — the bi-encoder: two weight-tied towers, mean pooling, cosine similarity.
• Robertson, Stephen, and Hugo Zaragoza. "The Probabilistic Relevance Framework: BM25 and Beyond." Foundations and Trends in Information Retrieval, vol. 3, no. 4, 2009, pp. 333–389. — BM25.
• Ru, Dongyu, et al. "RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation." Advances in Neural Information Processing Systems. Vol. 37. 2024, pp. 21999–22027. arXiv:2408.08067. — claim-level metrics that attribute a failure to the retriever or the generator, and the relevant/irrelevant noise split.
• Sarthi, Parth, et al. "RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval." International Conference on Learning Representations. Vol. 2024. 2024. arXiv:2401.18059. — recursive cluster-and-summarise at index time: a tree whose upper nodes are text absent from the corpus, and the layer ablation (any single layer ≈ 57.9, three layers 73.68).
• Spärck Jones, Karen. "A statistical interpretation of term specificity and its application in retrieval." Journal of Documentation, vol. 28, no. 1, 1972, pp. 11–21. — IDF.
• Yang, John, et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." Advances in Neural Information Processing Systems. Vol. 37. 2024. arXiv:2405.15793. — the agent–computer interface (ACI): the commands available to a model and the form in which their results return to it.

Preprints & other non-peer-reviewed papers

• Edge, Darren, et al. "From Local to Global: A GraphRAG Approach to Query-Focused Summarization." arXiv preprint arXiv:2404.16130 (2025). — the question no top-$k$ answers at any $k$, and query-focused summarization as the task it really is.
• Gao, Yunfan, et al. "Retrieval-Augmented Generation for Large Language Models: A Survey." arXiv preprint arXiv:2312.10997 (2024). — the three stages; the Naive / Advanced / Modular / Agentic paradigms; the three quality scores.
• Lu, Wensheng, et al. "HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking." arXiv preprint arXiv:2509.11552 (2025). — evidence sparsity: the standard document-QA collections carry 1.7–10.5 sentences of evidence per document, so they cannot separate one chunker from another; and the auto-merge retrieval rule.
• Oord, Aaron van den, et al. "Representation Learning with Contrastive Predictive Coding." arXiv preprint arXiv:1807.03748 (2018). — InfoNCE, the loss a text embedder is trained with: the cross-entropy of picking the positive out of $N$ candidates, whose optimum is the density ratio $p(x\mid c)/p(x)$ and which bounds the mutual information from below, $I(x;c) \ge \log N - \mathcal{L}_N$ — tighter the more negatives the batch holds.
• Packer, Charles, et al. "MemGPT: Towards LLMs as Operating Systems." arXiv preprint arXiv:2310.08560 (2024). — the retriever ceiling on a fixed-context reader; paging and re-querying as the escape; the nested key-value multi-hop task.
• Rackauckas, Zackary. "RAG-Fusion: a New Take on Retrieval-Augmented Generation." arXiv preprint arXiv:2402.03367 (2024). — multi-query retrieval fused by reciprocal rank fusion; the drift of generated queries and the latency cost. (A case study: cited for the mechanism, not for a size of effect.)
• Singh, Aditi, et al. "Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG." arXiv preprint arXiv:2501.09136 (2026). — §10's cautions: agentic reasoning does not compensate for poor retrieval. (Judgement, not measurement.)
• Yan, Shi-Qi, et al. "Corrective Retrieval Augmented Generation." arXiv preprint arXiv:2401.15884 (2024). — the retrieval evaluator, the three actions and the web-search fallback; the control showing that the gain is the decision, not the web.

Textbooks & monographs

• Jurafsky, Daniel, and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. 3rd ed., online manuscript released August 19, 2026. https://web.stanford.edu/~jurafsky/slp3/ — tf-idf and BM25 term weighting, document scoring and the inverted index (ch. 11, §11.1); the evaluation of ranked retrieval (§11.2); retrieval with dense vectors (§11.3).

Blogs, documentation & other web sources

• Ford, Daniel. "Introducing Contextual Retrieval." Anthropic Engineering, 2024. https://www.anthropic.com/engineering/contextual-retrieval — chunk augmentation: a model that has seen the whole document writes 50–100 tokens situating each chunk, and the augmented text enters both indexes. ⚠ One experiment, one vendor, methodology not published. Cite the mechanism freely; carry the numbers with that label.
• Hadfield, Jeremy, et al. "How we built our multi-agent research system." Anthropic Engineering, 2025. https://www.anthropic.com/engineering/multi-agent-research-system — subagents with their own context windows as compression for the lead agent; about fifteen times the tokens of a chat interaction for the multi-agent system.
• LlamaIndex. "Response Synthesizer Modes." LlamaIndex documentation, 2026. https://developers.llamaindex.ai/python/framework/module_guides/querying/response_synthesizers/ — the framework names for the three synthesis strategies (compact, refine, tree_summarize). (Cited for the vocabulary only; it reports no measurement.)