Agentic AI · L-Embed
Text as a vector: the geometry that turns "similar in meaning" into "close in $\mathbb{R}^d$", how that geometry is trained rather than found, and the nearest-neighbour search an agent runs over it.
Prerequisite: the recap Linear Algebra.
After this week you can:
An agent compares text by meaning rather than by exact words: to retrieve a relevant document, to recall a stored fact, or to select one tool out of many. The construction that makes such a comparison computable is the embedding — a map from text to a vector in which closeness is similarity.
An embedding represents a text as a vector in $\mathbb{R}^d$, so that a comparison of two meanings becomes a comparison of two vectors. Part I states the measures by which two vectors are compared, Part II how one vector per word type is learned from co-occurrence, Part III how a vector is computed for a word token and for a whole text, and Part IV how an agent searches a collection of text vectors:
Every construction in this lecture reduces to a comparison of two vectors in $\mathbb{R}^d$. Vectors, the dot product, norms and the angle between two vectors are a prerequisite of the lecture, recapped in Linear Algebra; this part adds the two measures by which embeddings are compared.
$$ S_C(\mathbf{u},\mathbf{v}) = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|_2\,\|\mathbf{v}\|_2} = \cos\theta \;\in\; [-1, 1] $$
$$ \|\mathbf{u}-\mathbf{v}\|_2 = \sqrt{\sum_i (u_i - v_i)^2} $$
Straight-line distance. Useful identity: $$ \|\mathbf{u}-\mathbf{v}\|_2^2 = \|\mathbf{u}\|_2^2 + \|\mathbf{v}\|_2^2 - 2\,\mathbf{u}\cdot\mathbf{v}. $$ For unit vectors this is $2 - 2\cos\theta$ — so on normalized embeddings, Euclidean distance and cosine similarity rank neighbours identically.
Example — why magnitude can mislead. A 5-page document about cats and a one-sentence tweet about cats point in nearly the same direction ($\theta \approx 0^\circ \Rightarrow S_C \approx 1$), but the long document's vector has a much larger magnitude $\|\mathbf{u}\|_2$. Raw Euclidean distance would call them far apart; cosine (angle only) correctly sees them as similar. (After normalizing, the two agree — the identity above.)
import numpy as np
u = np.array([2.0, 1.0, 0.0])
v = np.array([1.0, 1.0, 1.0])
dot = u @ v
cos = dot / (np.linalg.norm(u) * np.linalg.norm(v))
euc = np.linalg.norm(u - v)
print(f"dot={dot:.3f} cosine={cos:.3f} euclidean={euc:.3f}")
A word embedding assigns one vector to each word type of a fixed vocabulary; the vector is learned from the co-occurrence statistics of a corpus, so that words which occur in similar contexts receive nearby vectors.
An embedding is a learned function that maps text to a vector in $\mathbb{R}^d$ so that semantically similar text lands nearby. Contrast one-hot coding, in which each word receives an axis of its own: every pair of words is then orthogonal, and cat is as dissimilar from kitten as it is from car. An embedding instead compresses meaning into a dense vector, in which cat and kitten are nearly aligned.
A central difficulty of natural-language processing (NLP) is that a word form need not carry one meaning:
Three answers have been given to the question what receives a vector, and they arrived in this order:
| Unit | What Receives a Vector | How Context Is Used | Example Models |
|---|---|---|---|
| Word type (Static) | A word independently of its sentence | Context is used only during training to create a single static lookup vector. It cannot resolve polysemy or homonymy at inference time. | word2vec (Mikolov et al. 2013), GloVe (Pennington et al. 2014), fastText (Bojanowski et al. 2017) |
| Word token (Dynamic) | A word in a given sentence | Context is processed at inference time, generating a unique vector for play in a theatre sentence vs. a baseball sentence. | ELMo (Peters et al. 2018), BERT (Devlin et al. 2019) |
| Text (Sequence) | A whole sentence, passage, or document | The entire sequence is compressed into one single vector representing holistic semantic meaning. | Sentence-BERT (Reimers & Gurevych 2019), nomic-embed-text (Nussbaum et al. 2025) |
The word-level static models are where the geometry is easiest to derive; the text-level models are what a modern retrieval system calls.
Because similar meanings map to nearby vectors, geometry encodes semantics:
This is why we can search by meaning instead of by keywords.
A word vector is learned from co-occurrence alone. No model in this section reads a dictionary, a parse tree or a morphological analysis; each one is a loss function built on a single assumption, the distributional hypothesis (Harris 1954; Jurafsky & Martin, 2026, ch. 5): a word is characterised by the words it occurs with.
Three training signals implement that assumption:
All three produce a space that is searched with the cosine similarity.
word2vec (Mikolov et al. 2013) proposes two architectures, both log-linear: an input layer, a projection, and an output layer, with no non-linear hidden layer.
The word vector is the projection matrix: training is a by-product, and the matrix is the result.
Why the hidden layer had to go. The paper measures a model by the number of parameters accessed per training example, written $Q$, with total cost $O = E \times T \times Q$ for $E$ epochs over $T$ training words. With vocabulary size $V$, vector dimension $D$, context size $N$ and hidden layer size $H$:
| Architecture | $Q$ per example |
|---|---|
| Feed-forward neural language model | $N D + \underbrace{N D H}_{\text{dominant}} + H V$ |
| Recurrent neural language model | $H H + H V$ |
| CBOW | $N D + D \log_2 V$ |
| Skip-gram | $C\,(D + D \log_2 V)$ |
The letters are the paper's own: its $D$ is the dimension written $d$ elsewhere in this lecture, its $V$ is the vocabulary size (written $|V|$ in the course notation), and its $T$ counts training words rather than denoting a temperature.
Deleting the non-linear hidden layer deletes the $N D H$ term; the $V$-sized output is replaced by a hierarchical softmax over a Huffman tree of the vocabulary, which evaluates about $\log_2 V$ units instead of $V$. The trade is stated plainly in the paper: models "that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently" — and that is what made a 6-billion-word corpus with a one-million-word vocabulary reachable in 2013.
When evaluating word embeddings, semantics and syntax capture two fundamental aspects of word meaning:
| Dimension | Focus | Core Question | Vector Space Behavior | Example Analogy |
|---|---|---|---|---|
| Semantics | Meaning & Concepts | What does the word refer to? | Maps words with shared topics, categories, or real-world relationships close together. | $\text{king} - \text{man} + \text{woman} \approx \text{queen}$ $\text{Paris} - \text{France} + \text{Japan} \approx \text{Tokyo}$ |
| Syntax | Grammar & Form | How does the word function grammatically? | Maps words with identical grammatical inflections, tenses, or parts of speech close together. | $\text{walking} - \text{walk} + \text{swim} \approx \text{swimming}$ $\text{quick} - \text{quickly} + \text{slow} \approx \text{slowly}$ |
The context window selects between the two. A large window records broad thematic association and therefore semantics; a small or one-sided window records immediate local structure and therefore syntax.
The two properties occupy different directions. A high-quality space separates them, which is why one and the same vector addition and subtraction answers both kinds of analogy.
Which of the two, and what the numbers say. Trained on the same 320 M-word corpus with 640-dimensional vectors, the two architectures separate by question type (Mikolov et al. 2013, Table 3; accuracy in per cent on the paper's own Semantic-Syntactic Word Relationship set):
| Model | Semantic | Syntactic |
|---|---|---|
| Recurrent neural language model | 9 | 36 |
| Feed-forward neural language model | 23 | 53 |
| CBOW | 24 | 64 |
| Skip-gram | 55 | 59 |
Skip-gram is much stronger on semantics, CBOW slightly stronger on syntax and about three times cheaper (one day against three for 300-dimensional vectors on 783 M words). That is the actual choice between them.
Two further results are worth carrying:
One point of vocabulary. This paper trains its output layer with the hierarchical softmax described above. The negative sampling objective usually associated with word2vec — the normalised softmax replaced by a set of binary decisions against sampled non-neighbours — is introduced in the same authors' follow-up paper, and it is the objective fastText builds on.
word2vec (Mikolov et al. 2013) produces vectors whose differences encode analogies, but does not explain why. GloVe (Pennington et al. 2014) supplies the argument, and it begins with a co-occurrence table rather than a predictive model.
Let $X_{ij}$ count how often word $j$ occurs in the context of word $i$ — throughout this section $X$ is a matrix of counts, not a random variable — and let:
$$P_{ij} = P(j \mid i) = \frac{X_{ij}}{X_i}$$
where $X_i = \sum_k X_{ik}$ is the total number of times any word appears in the context of word $i$.
The relationship between target words $i = \textit{ice}$ and $j = \textit{steam}$ can be examined by studying the ratio of their co-occurrence probabilities with various probe words $k$ over a 6-billion token corpus:
| Probability / Ratio | $k=\textit{solid}$ | $k=\textit{gas}$ | $k=\textit{water}$ | $k=\textit{fashion}$ |
|---|---|---|---|---|
| $P(k \mid \textit{ice})$ | $1.9 \times 10^{-4}$ | $6.6 \times 10^{-5}$ | $3.0 \times 10^{-3}$ | $1.7 \times 10^{-5}$ |
| $P(k \mid \textit{steam})$ | $2.2 \times 10^{-5}$ | $7.8 \times 10^{-4}$ | $2.2 \times 10^{-3}$ | $1.8 \times 10^{-5}$ |
| Ratio $\frac{P(k \mid \textit{ice})}{P(k \mid \textit{steam})}$ | 8.9 | 0.085 | 1.36 | 0.96 |
The ratio $\frac{P(\textit{solid} \mid \textit{ice})}{P(\textit{solid} \mid \textit{steam})}$ is much larger than 1, revealing that $\textit{ice}$ has a much stronger association with $\textit{solid}$ than $\textit{steam}$ does. Conversely, for $k = \textit{gas}$, the ratio is much smaller than 1. Ratios for non-discriminative context words ($\textit{water}$ or $\textit{fashion}$) remain close to $1$.
Compared to raw probabilities, the ratio isolates relevant words ($\textit{solid}$ and $\textit{gas}$) from noise words ($\textit{water}$ and $\textit{fashion}$). The key quantity word vectors should encode is therefore the ratio of co-occurrence probabilities, not the raw probabilities themselves.
Three constraints leave the model only one form: the relation between two words is carried by their difference $(\mathbf{w}_i - \mathbf{w}_j)$, because a vector space is a linear structure; that difference must reduce to a scalar, because the ratio it has to reproduce is one; and exchanging a word with a context word must change nothing, because the count matrix is symmetric. What survives is a dot-product relationship
$$\mathbf{w}_i^\top \tilde{\mathbf{w}}_k + b_i + \tilde{b}_k = \log X_{ik},$$
with $\tilde{\mathbf{w}}_k$ the separate vector of the context word, and $b_i$ and $\tilde b_k$ absorbing the terms that depend on one side alone.
Because $75\%\text{--}95\%$ of the co-occurrence matrix is zero, GloVe optimizes a weighted least-squares loss strictly over non-zero entries ($X_{ij} > 0$):
$$J = \sum_{i,j=1}^{V} f(X_{ij}) \left( \mathbf{w}_i^\top \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2$$
where $f(X_{ij})$ downweights rare, noisy co-occurrences and caps frequent stop words. Crucially, $f(0) = 0$, so zero entries are ignored entirely and the fit costs $\mathcal{O}(|C|^{0.8})$ in the corpus length $|C|$ rather than $\mathcal{O}(|C|)$.
Counting (GloVe) and predicting (word2vec) are usually presented as opposing paradigms, and they probe the same underlying corpus statistics:
| Dimension | word2vec (Skip-gram / CBOW) | GloVe |
|---|---|---|
| Primary Input | Local sliding windows (streaming tokens) | Global co-occurrence matrix $X$ |
| Training Loss | Cross-entropy (Hierarchical Softmax / Neg. Sampling) | Weighted least-squares on log-counts |
| Efficiency Driver | Subsampling frequent words & negative sampling | $f(0)=0$ skips zero-count cells in matrix |
| Core Advantage | Faster online updating on new data streams | Direct statistical efficiency; avoids repeated scans |
The two differ in their distance measure and their weighting, and recover the same linear structure from the same global co-occurrence distribution.
Both word2vec and GloVe assign a single vector to each word in a fixed vocabulary. A word absent from the training corpus (Out-of-Vocabulary / OOV) receives no vector at all, and morphologically related words like walking and walked are treated as entirely unrelated tokens. For morphologically rich or compounding languages (e.g., German Tischtennis, Finnish noun cases), this vocabulary bottleneck is severe.
fastText (Bojanowski et al. 2017) modifies the scoring function while retaining the underlying Skip-gram architecture. A target word is represented as a bag of character $n$-grams alongside the special boundary-wrapped word itself.
For example, with $n = 3$, the word where yields: $$\mathcal{G}_{\text{where}} = \{\texttt{<wh}, \texttt{whe}, \texttt{her}, \texttt{ere}, \texttt{re>}, \texttt{<where>}\}$$
where < and > mark word boundaries to distinguish prefixes/suffixes from internal character sequences.
Crucially, context words are not decomposed into $n$-grams—a context word $c$ is represented by a single, standard word vector $\mathbf{v}_c$.
Given $\mathbf{z}_g$ as the vector representation for subword $g \in \mathcal{G}_w$, the scoring function between target word $w$ and context word $c$ becomes:
$$s(w,c) = \sum_{g \in \mathcal{G}_w} \mathbf{z}_g^\top \mathbf{v}_c$$
Thus, a target word's representation is the sum of its constituent subwords, evaluated against a standard context word vector.
Note: Standard fastText evaluations extract $n$-grams for $3 \le n \le 6$, adding roughly $1.5\times$ computational overhead compared to standard Skip-gram training.
Character $n$-grams encode orthographic form — the surface shape of a word — and form correlates much more strongly with grammar than with meaning. The evaluation splits accordingly (accuracy in per cent, Bojanowski et al. 2017, Table 2; sisg, subword information skip-gram, is the paper's own name for the model):
| Language | Skip-gram — semantic / syntactic | fastText (sisg) — semantic / syntactic |
|---|---|---|
| Czech | 25.7 / 52.8 | 27.5 / 77.8 |
| German | 66.5 / 44.5 | 62.3 / 56.4 |
| English | 78.5 / 70.1 | 77.8 / 74.9 |
Syntax improves, semantics does not. Czech syntactic accuracy rises by 25 points; German semantic accuracy falls, from 66.5 to 62.3. A shared character sequence is strong evidence of a shared grammatical form and weak evidence of a shared meaning.
Subwords buy data efficiency. Every word containing a gram trains that gram's vector, so the parameters are shared: a model trained on 5 % of German Wikipedia surpasses a CBOW baseline trained on all of it.
A contextual embedding assigns a vector to each word token, computed from the sentence in which the token occurs; a text embedding assigns one vector to a whole text, and contrastive training arranges these vectors so that cosine similarity measures the similarity of the texts.
A word-type vector must average every use of the word. Consider play. Its nearest neighbours under GloVe are playing, game, games, played, players, plays, player, football, multiplayer — several parts of speech at once, and all of them sporting. The theatre sense is present in the training corpus and absent from the vector, because one vector cannot hold two senses.
A contextual embedding assigns a vector to a word token — a word in a particular sentence — so that the vector may differ from occurrence to occurrence. ELMo, Embeddings from Language Models (Peters et al. 2018), obtains one from a bidirectional language model: a recurrent language model reading left to right and a second one reading right to left, trained jointly on the same corpus. The trained pair is then frozen and used as a feature extractor — the embedding of a token is a weighted sum of the model's own internal states at that position, one vector per layer. When looking up the nearest neighbors for a word in context, we extract its vector from the model and find the most similar word vectors across a corpus. For play in "made a spectacular play on Alusik's grounder", the nearest matching vector comes from a baseball sentence; for "signed to do a Broadway play", the nearest match comes from a theatre sentence.
The layers of such a model are not interchangeable. Used directly as features, the lower layer tags parts of speech at 97.3 % accuracy against the top layer's 96.8, while the top layer disambiguates word senses at 69.0 $F_1$ — the harmonic mean of precision and recall — against the lower layer's 67.4. Syntax is represented low, meaning high, which is why ELMo hands the downstream task a learned weighting over all layers rather than the topmost one.
ELMo concatenates two models that each read in one direction. (BERT — Bidirectional Encoder Representations from Transformers (Devlin et al. 2019) — replaces them by a single transformer encoder in which every layer attends over the whole input in both directions at once. Most of the text embedders in this lecture are built from it, so three of its properties are fixed here. The attention mechanism itself is not needed for that: the only property an embedder uses is that the attention is bidirectional.
[CLS] token. Every input sequence begins with a special reserved token, [CLS] (short for classification). Rather than representing a specific word, its final-layer output vector aggregates contextual information from the entire input sequence. Downstream classification tasks (such as sentiment analysis or topic labeling) use this pooled vector as the input to a final linear prediction layer.Retrieval compares a query with a passage, not a word with a word, so what is needed is one vector
per text. The obvious construction is to run a contextual encoder over the text and reduce its
token vectors to one — by taking the mean, or by taking the [CLS] vector. Measured against human
similarity judgements on seven standard similarity sets, that construction fails (Reimers & Gurevych
2019; rank correlation $\times 100$):
| Representation | Correlation |
|---|---|
BERT [CLS] vector |
29.19 |
| Mean of BERT token vectors | 54.81 |
| Mean of GloVe word vectors | 61.32 |
| Sentence-BERT | 74.89 |
An untuned contextual encoder is beaten by averaged word vectors, and the [CLS] vector is far
behind both. BERT's own authors state the reason in a footnote:
"The vector $C$ is not a meaningful sentence representation without fine-tuning, since it was trained with NSP." — Devlin et al. (2019), footnote 6
Here $C$ is the [CLS] output and NSP the next-sentence prediction objective: the vector is an
aggregate for that task, and for no other.
The information is present; the geometry is not arranged for cosine. Fitting a supervised classifier on top of mean-pooled contextual vectors works well because the linear layer learns task-specific weights to extract salient features from noisy dimensions.
Cosine similarity, by contrast, treats all vector dimensions equally. It cannot filter out uninformative features or scale up critical semantic signals on its own. Unless the model was explicitly trained to cluster semantically similar sentences closely in angular space, isotropic metrics like cosine similarity merely measure non-metric background noise.
Cosine similarity reads only a geometry explicitly constructed for it. An embedding model is a model whose geometry has been built for metric retrieval, and contrastive training is the primary method used to build it.
The need to arrange vector space geometry for metric search exposes a fundamental distinction in how we compare texts using Transformers:
A cross-encoder feeds both texts through the network together as a single concatenated sequence ([CLS] Text A [SEP] Text B).
Because every Transformer layer applies cross-attention between all tokens of both texts, it yields extremely accurate relevance scores. However, a cross-encoder produces no sentence embeddings whatsoever. The score is indecomposable—no representation is ever computed for one text in isolation. Every pair requires a full forward pass through the model:
$$\text{Cost} = \frac{n(n-1)}{2} \text{ forward passes}$$
For ranking pairs among $n = 10{,}000$ documents, a cross-encoder requires 49,995,000 forward passes (~65 hours on standard GPU hardware per Reimers & Gurevych, 2019).
A bi-encoder routes each text through the Transformer separately to generate independent vector embeddings $f(a)$ and $f(b)$, then compares them using a fixed, parameter-free metric like cosine similarity ($S_C$):
$$\text{score}(a,b) = S_C\big(f(a), f(b)\big)$$
By factorizing the calculation into distinct vector representations, each text is encoded once and cached:
$$\text{Cost} = n \text{ forward passes} + \text{vector comparisons}$$
For $n = 10{,}000$ documents, this requires 10,000 encodings (~5 seconds) plus 10,000 cosine dot-products (~0.01 seconds). 65 hours drop to 5 seconds because the system shifts from runtime inference to vector search over cached embeddings.
Sentence-BERT builds such an encoder out of BERT (Reimers & Gurevych 2019), by fine-tuning it in a siamese structure: two copies of the network with tied weights — one set of parameters, applied twice — a pooling layer reducing each copy's token outputs to one vector (the mean, by default), and a training objective comparing the two vectors. At inference one copy and the cosine remain, and the second copy was never a second model.
Because a Bi-Encoder delegates the similarity decision to a simple metric function $S_C$, the model's single job during training is to shape its vector space geometry. Bi-Encoders must be trained using contrastive loss functions (e.g., InfoNCE) to explicitly pull positive pairs together and push negative pairs apart in vector space. In contrast, standard language modeling losses (like Cross-Entropy) do not shape vector distances.
Contrastive Learning (used in models like Sentence-BERT, SimCSE, and OpenAI's cpt-text (Neelakantan et al., 2022)) explicitly forces the geometry to conform to cosine metrics using objectives like InfoNCE loss:
The geometry is not discovered — it is optimised for. The training data are pairs $(q_i, d_i)$: a query and a document that belongs to it. Write $\mathbf{q}_i,\mathbf{d}_i \in \mathbb{R}^d$ for their embeddings. Over a batch of $n$ such pairs the model minimises the InfoNCE loss — the name is from noise-contrastive estimation, learning by distinguishing a true pair from sampled noise:
$$\mathcal{L}_C = -\frac{1}{n}\sum_{i=1}^{n} \log \frac{\exp\left(S_C(\mathbf{q}_i, \mathbf{d}_i) / T\right)}{\sum_{j=1}^{n} \exp\left(S_C(\mathbf{q}_i, \mathbf{d}_j) / T\right)}$$
with $S_C$ the cosine similarity and $T > 0$ a temperature — the same quantity that sharpens a sampling distribution, and written $\tau$ in this literature.
Reading the loss. The fraction is a softmax over cosine similarities, so the expression as a whole is the cross-entropy of an $n$-way classification: which of these $n$ documents belongs to this query? Three details decide what it does.
Key Takeaway: Cosine similarity is the search metric at inference time because it was the explicit training score during optimization. "Similarity" in vector space reflects whichever relationships were defined by the training pairs (Nussbaum et al. 2024).
The models named here are trained by the same contrastive principle and differ in two places: the encoder they start from, and what counts as a positive pair. The constant is not BERT — it is bidirectional attention together with contrastive training.
| Model | Encoder it starts from | What supplies the positive pair |
|---|---|---|
| Sentence-BERT (Reimers & Gurevych 2019) | BERT | annotated natural-language-inference pairs |
| DPR (Karpukhin et al. 2020) | two BERT towers, weights not tied | question–passage pairs from question-answering data |
| SimCSE (Gao, Yao & Chen 2021) | BERT | the same sentence encoded twice, differing only by the dropout mask |
| E5 (Wang et al. 2024) | bert-base-uncased |
structure in web documents — (post, comment), (title, abstract) |
| INSTRUCTOR (Su et al. 2023) | GTR, an encoder initialised from T5 | 330 datasets, each side carrying a written instruction |
nomic-embed-text (Nussbaum et al. 2025) |
a BERT variant with rotary position embeddings, 2048 tokens of context, 137 M parameters | web structure, then 1.6 M labelled pairs with mined hard negatives |
| NV-Embed (Lee et al. 2025) | Mistral-7B, a decoder, with its causal mask removed | labelled retrieval and classification data |
The last row locates the property. A decoder-only language model is a poor embedder as it stands, because a causal mask lets a token attend only to its predecessors and the earlier tokens of a passage therefore never see the later ones. Deleting that mask makes the model bidirectional, and the ablation reports that "the bidirectional mask consistently outperforms the causal mask" for every pooling choice (Lee et al. 2025). What an embedder needs is bidirectional attention, not a particular architecture family.
One entry in the table deviates in the score rather than in the encoder: DPR compares its two vectors by the plain inner product and not by the cosine, so a long vector can win on magnitude alone. Every other model in the table normalises and compares by cosine.
An embedder reads neither characters nor words: its input is a sequence of subword tokens, produced by the same kind of learned tokenizer a generative model uses — byte-pair encoding (Sennrich et al. 2016), or the WordPiece vocabulary of 30,000 token types introduced with BERT (Devlin et al. 2019). How such a vocabulary is constructed is covered in → L-Att; three of its consequences belong here.
nomic-embed-text is trained on text tokenized with BERT's
bert-base-uncased vocabulary (Nussbaum et al. 2025); an embedder initialised from a decoder-only
model inherits that model's vocabulary together with its weights, as NV-Embed does from Mistral-7B
(Lee et al. 2025).nomic-embed-text is trained at 2048 tokens, and its own evaluation truncates every text
to 512 (Nussbaum et al. 2025). Truncation raises no error — the part beyond the limit is simply not
represented — so the chunk length of a retrieval corpus is measured in the embedder's tokens. A
task prefix is ordinary text and is counted in the same budget.Subword tokenization is the second answer to the unseen word. fastText composes an out-of-vocabulary word from the character $n$-grams it contains; a subword tokenizer splits that word into pieces of a fixed vocabulary before the encoder sees it, so every input is representable and no word needs an entry of its own.
Semantic search returns the stored texts whose vectors are nearest to the vector of a query; an agent uses it for retrieval, for long-term memory and for tool selection.
To find text relevant to a query:
Brute force costs $O(n\,d)$ per query over $n$ documents. That is affordable for thousands of documents; for millions it is replaced by approximate nearest-neighbour (ANN) search over a vector index (→ L-RAG).
Four qualifications.
search_query: and search_document:, so the two sides are not mapped by
the identical function. The section Symmetric and Asymmetric Tasks treats this case.ERR_2043);
the remedy is hybrid search, a sparse keyword score (BM25, Robertson & Zaragoza 2009)
+ the dense one.if cos > 0.8 discards good matches for one query and
admits noise for the next. Rank first, then cut by $k$ (Nussbaum et al. 2025).Not every use of a text embedding compares two things of the same kind, and the difference decides how the model must be called.
In a symmetric task — detecting duplicate questions, measuring semantic similarity, clustering — the two sides share structure, length and intent: two questions, two captions, two summaries. In retrieval, and therefore in RAG, they do not. A query is short, interrogative and states an information need; a passage is long, declarative and states facts. Nothing about the two texts is alike except the answer relation between them.
One encoder serving both is required to satisfy two different meanings of belongs together at once. The remedy is to tell the model which side it is encoding, and the marker has grown more expressive over time:
| Model | What marks the side |
|---|---|
| E5 (Wang et al. 2024) | two prefixes, query: and passage: |
| INSTRUCTOR (Su et al. 2023) | a written instruction per side — "Represent the Wikipedia question for retrieving supporting documents:" against "Represent the Wikipedia document for retrieval:" |
nomic-embed-text (Nussbaum et al. 2025) |
four fixed prefixes — search_query, search_document, classification, clustering |
For a symmetric task the two sides receive the same marker; for an asymmetric one they do not (Su et al. 2023).
A marker is not the only mechanism, and it is not an architectural one. Two alternatives bracket it.
nomic-embed-text share a single set of weights and
break the symmetry with a string prepended to the text. The encoder has no notion that the string
is a marker: it is ordinary input, and the behaviour comes entirely from training pairs that
carried the same strings.The second is what is in general use, and its two failure modes are therefore silent. The corpus and the queries must follow the same convention — a wrongly marked query raises no error, it only ranks worse — and changing the convention means re-embedding the corpus, not re-running the query.
# Semantic search with real embeddings via Ollama (once: `ollama pull nomic-embed-text`)
import numpy as np
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
def embed(texts, prefix): # nomic-embed-text expects a task prefix
r = client.embeddings.create(model="nomic-embed-text",
input=[f"{prefix}: {t}" for t in texts])
return np.array([d.embedding for d in r.data])
docs = ["The cat sat on the mat.",
"A dog barked at the mailman.",
"Python is a popular programming language."]
try:
D = embed(docs, "search_document") # stored documents
q = embed(["a feline resting on a rug"], "search_query")[0] # the query
sims = (D @ q) / (np.linalg.norm(D, axis=1) * np.linalg.norm(q))
best = int(sims.argmax())
print(f"best match ({sims[best]:.3f}): {docs[best]}") # -> the cat sentence
except Exception as e:
print("Error connecting to Ollama. Ensure Ollama is running and 'nomic-embed-text' is pulled.")
print("Error details:", e)
All three are nearest-neighbour search over embeddings.
lecture-Embed-exercises.ipynb — calculations: cosine similarity & Euclidean distance by hand.
lecture-Embed-lab.ipynb: implement the distance functions and a brute-force nearest-neighbour
search from scratch, then do semantic search over tool descriptions with real embeddings
(Ollama nomic-embed-text).
[CLS] vector is a worse
similarity representation than averaged word vectors. The geometry is trained, by the
InfoNCE objective over cosine similarity with in-batch negatives.Peer-reviewed papers
[CLS] vector is not a sentence representation without fine-tuning.search_query / search_document / classification / clustering) with the training-signal conflict that motivates them.Preprints & other non-peer-reviewed papers
query: / passage: prefixes that break a shared encoder's symmetry.Textbooks & monographs