Agentic AI · Fundamentals
Vectors in $\mathbb{R}^d$, the dot product, norms and normalization, and the angle between two vectors.
A vector is an ordered list of $d$ real numbers, read as a point or an arrow in $d$-dimensional space. Two vectors can be compared by their length and by the angle between them. The course needs this because a language model represents tokens and texts as vectors, and because retrieval ranks documents by the angle between their vectors and the vector of a query.
What this recap provides: the dot product in its algebraic and its geometric form · the $\ell_1$ and $\ell_2$ norms and the unit vector · the angle between two vectors and its cosine.
What it builds on: the vector geometry of school and the linear-algebra course of the first semesters. The definitions follow Deisenroth et al. (2020), Ch. 3.
A vector $\mathbf{u} = (u_1,\dots,u_d) \in \mathbb{R}^d$ is a point (or arrow) in $d$-dimensional space; $u_i$ is its $i$-th component and $d$ its dimension. Vectors are written in bold lowercase.
Example. An embedding is such a vector: a token or a piece of text represented as a point in $\mathbb{R}^d$, with $d$ typically between 384 and 4096.
$$ \mathbf{u}\cdot\mathbf{v} \;=\; \sum_{i=1}^{d} u_i v_i \;=\; \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta $$
The algebraic form (sum of products, Deisenroth et al., 2020, §3.2, eq. 3.5) and the geometric form ($\|\cdot\|\,\|\cdot\|\cos\theta$, with $\theta$ the angle between the two vectors, §3.4, eq. 3.25) are equal. The dot product therefore measures how aligned two vectors are: it is positive for an angle below $90°$, zero at $90°$ and negative above.
Example. $\mathbf{u}=(1,2,0)$, $\mathbf{v}=(3,-1,4)$: $\mathbf{u}\cdot\mathbf{v} = 3 - 2 + 0 = 1$.
A norm assigns to each vector a length $\|\mathbf{u}\|\ge 0$ that is $0$ only for the zero vector, scales with a constant factor, $\|a\mathbf{u}\| = |a|\,\|\mathbf{u}\|$, and satisfies the triangle inequality $\|\mathbf{u}+\mathbf{v}\|\le\|\mathbf{u}\|+\|\mathbf{v}\|$ (Deisenroth et al., 2020, Def. 3.1).
Where no subscript is written, $\|\cdot\|$ means the $\ell_2$ norm.
Normalizing discards the magnitude of a vector and keeps only its direction.
Example. $\mathbf{u}=(3,4)$: $\|\mathbf{u}\|_2 = 5$, $\|\mathbf{u}\|_1 = 7$, $\hat{\mathbf{u}} = (0.6,\ 0.8)$.
The cosine of the angle between two non-zero vectors is their dot product divided by the product of their lengths:
$$\cos\theta = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|_2\,\|\mathbf{v}\|_2} \;\in\; [-1,1], \qquad \theta\in[0,\pi] .$$
The value lies in $[-1,1]$ by the Cauchy–Schwarz inequality $|\mathbf{u}\cdot\mathbf{v}|\le\|\mathbf{u}\|_2\,\|\mathbf{v}\|_2$, and $\theta$ is the unique angle in $[0,\pi]$ with this cosine (Deisenroth et al., 2020, §3.4, eq. 3.24–3.25). The cosine is the dot product of the two unit vectors, $\cos\theta = \hat{\mathbf{u}}\cdot\hat{\mathbf{v}}$, so it does not change when either vector is scaled by a positive factor.
| $\cos\theta$ | $\theta$ | the two vectors |
|---|---|---|
| $1$ | $0$ | point in the same direction |
| $0$ | $90°$ | are orthogonal |
| $-1$ | $180°$ | point in opposite directions |
Example. $\mathbf{u}=(1,0)$, $\mathbf{v}=(1,1)$: $\cos\theta = 1/(1\cdot\sqrt2) \approx 0.707$, $\theta = 45°$.
Deisenroth, Marc Peter, et al. Mathematics for Machine Learning. Cambridge University Press, 2020. doi:10.1017/9781108679930. — norm (Def. 3.1, Ex. 3.1, Ex. 3.2), dot product (§3.2, eq. 3.5), angle between two vectors and the Cauchy–Schwarz inequality (§3.3–§3.4, eq. 3.24–3.25).