Eigen-Decomposition¶

on the example Covariance Matrix

From Data to the Covariance Matrix¶

Synthetic Data Generation

We start by generating a 2D Gaussian distribution with a known mean $\boldsymbol \mu$ and covariance $\boldsymbol \Sigma$. This simulates noisy sensor measurements.

Estimating $\mathbf{C}$ from the Samples (Recap)¶

In reality we don't know the true $\boldsymbol \mu$ or $\boldsymbol \Sigma$; we estimate them from the $m$ samples. As derived in the empirical covariance matrix in vector form notebook, we first center the data by subtracting the empirical mean,

$$\bar{x}_i = \frac{1}{m} \sum_{k=1}^m x_i^{(k)}, \qquad \tilde{\mathbf{X}} = \mathbf{X} - \mathbf{1}\bar{\mathbf{x}}^T,$$

so that each row of the $m \times n$ matrix $\tilde{\mathbf{X}}$ is one centered sample, and then read off the empirical covariance as the inner product

$$\mathbf{C} = \frac{1}{m-1}\, \tilde{\mathbf{X}}^T \tilde{\mathbf{X}}.$$

The factor $m-1$ (Bessel's correction) makes the estimate unbiased. The code cell below evaluates exactly this for our sampled data; the resulting $\mathbf{C}$ is the matrix we eigen-decompose throughout the rest of the notebook.

Recap: Symmetry¶

Definition: A matrix $\mathbf{C}$ is symmetric if it is equal to its transpose ($\mathbf{C} = \mathbf{C}^T$).

For a covariance matrix, symmetry is guaranteed by its very definition. The entry $C_{ij}$ is the covariance between the scalar components $x_i$ and $x_j$ of the random vector $\mathbf{x}$, and covariance does not depend on the order of its two arguments:

$$C_{ij} = \operatorname{cov}(x_i, x_j) = \operatorname{cov}(x_j, x_i) = C_{ji}.$$

Since every entry equals its mirror image across the diagonal, $\mathbf{C} = \mathbf{C}^T$.

Positive definite¶

Definition: A square matrix $\mathbf{C} \in \mathbb{R}^{n \times n}$ is positive-definite if for all $\mathbf v \in \mathbb{R}^{n} $ and $\mathbf v \neq 0$ holds $$ \mathbf{v}^T \mathbf{C} \mathbf{v} > 0 $$

The covariance matrix is positive (semi-)definite¶

Let $\mathbf{y} = \mathbf{x} - \boldsymbol \mu $ be a random vector (centered at zero) drawn from our data distribution. The covariance is defined as the expected value:$$\mathbf{C} = \mathbb{E}[\mathbf{y}\mathbf{y}^T]$$

with an arbitrary vector $\mathbf{v}$:

$$\mathbf{v}^T \mathbf{C} \mathbf{v} = \mathbf{v}^T \mathbb{E}[\mathbf{y}\mathbf{y}^T] \mathbf{v}$$

Since $\mathbf{v}$ is a constant vector, we can move it inside the expectation: $$\mathbb{E}[\mathbf{v}^T \mathbf{y}\mathbf{y}^T \mathbf{v}]$$

Notice that $\mathbf{v}^T \mathbf{y}$ is a scalar (let's call it $a$). Then $\mathbf{y}^T \mathbf{v}$ is also $a$. So the equation becomes:

$$\mathbb{E}[a \cdot a] = \mathbb{E}[a^2]$$

Because $a^2$ is the square of a number, it is always $\ge 0$. Therefore its average (Expectation) is also $\ge 0$:

$$\mathbf{v}^T \mathbf{C} \mathbf{v} = \mathbb{E}[a^2] \ge 0 \qquad \text{for all } \mathbf{v}.$$

This proves that a covariance matrix is always positive semi-definite. The expression $\mathbf{v}^T \mathbf{C} \mathbf{v}$ is the variance of the data projected onto the direction $\mathbf{v}$, and a variance can never be negative.

It is positive definite (strictly $> 0$) exactly when the data has spread in every direction—that is, when it does not lie entirely within a lower-dimensional subspace (a line, a plane, …). If some direction $\mathbf{v}$ carries zero variance, then $\mathbf{v}^T \mathbf{C} \mathbf{v} = 0$ and the matrix is only semi-definite. For the remainder we assume the data has full-dimensional spread, so $\mathbf{C}$ is positive definite—this is what guarantees the strictly positive eigenvalues and the invertibility we use below.

The Fundamental Eigen-Equation¶

For a square matrix $\mathbf{C}$, a non-zero vector $\mathbf{q}$ is an eigenvector if the transformation of $\mathbf{q}$ by $\mathbf{C}$ results only in a change of scale, not direction:

$$\mathbf{C}\mathbf{q} = \lambda\mathbf{q}$$

• Eigenvector ($\mathbf{q}$): The "characteristic direction." It defines an axis that remains invariant (un-rotated) under the transformation.
• Eigenvalue ($\lambda$): The "characteristic scale." It represents the factor by which the eigenvector is stretched or shrunk.

Remember: The Eigenvectors $\bf{q}$ are not rotated by $C$.

Eigenvalues of positive-definite matricies are strictly positive¶

$$\mathbf{C}\mathbf{q} = \lambda \mathbf{q} \implies \mathbf{q}^T \mathbf{C} \mathbf{q} = \lambda (\mathbf{q}^T \mathbf{q})$$ Since $\mathbf{q}^T \mathbf{C} \mathbf{q} > 0$ and $\mathbf{q}^T \mathbf{q} > 0$, it follows that $\lambda > 0$.

Solving the Eigen-problem¶

To find $\lambda$, we rearrange the equation:$$(\mathbf{C} - \lambda \mathbf{I}) \mathbf{q} = \mathbf{0}$$

This equation has a non-zero solution for $\mathbf{q}$ only if the matrix $(\mathbf{C} - \lambda \mathbf{I})$ is singular, meaning its determinant is zero:

$$\det(\mathbf{C} - \lambda \mathbf{I}) = 0$$

Mathematical Example:

To illustrate the mechanics on plain numbers, take a general matrix $\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 0 & 2 \end{pmatrix}$. We deliberately call it $\mathbf{A}$ and not $\mathbf{C}$: a covariance matrix is symmetric, so it would never look like this.

$$\det \begin{pmatrix} 3-\lambda & 1 \\ 0 & 2-\lambda \end{pmatrix} = (3-\lambda)(2-\lambda) - 0 = 0$$

Results: $\lambda_1 = 3, \lambda_2 = 2$.

• For $\lambda_1=3$, solving $(\mathbf{A}-3\mathbf{I})\mathbf{q} = 0$ gives $\mathbf{q}_1 = \begin{pmatrix} \alpha \\ 0 \end{pmatrix}$.
• For $\lambda_2=2$, solving $(\mathbf{A}-2\mathbf{I})\mathbf{q} = 0$ gives $\mathbf{q}_2 = \begin{pmatrix} \beta \\ -\beta \end{pmatrix}$.

Note that these two eigenvectors are not orthogonal ($\mathbf{q}_1^T \mathbf{q}_2 = \alpha\beta \neq 0$). This is expected, because $\mathbf{A}$ is not symmetric. For a symmetric matrix—such as any covariance matrix—the eigenvectors are orthogonal, as we prove further below.

Geometric Intuition: The "Power" of the Eigenvector¶

What happens if we repeatedly apply the covariance matrix $\mathbf{C}$ to random vector (not an eigenvector) and rescale it to length 1?

The result:

No matter where you start (unless you are perfectly aligned with a smaller eigenvector), repeated transformation by $C$ will "pull" the vector toward the eigenvector with the largest eigenvalue.

The faint blue crosses are the centered data $\tilde{\mathbf{X}} = \mathbf{X} - \bar{\mathbf{x}}$ that $\mathbf{C}$ was estimated from. Since the raw cloud is far larger than the unit-length vectors, it is shrunk by a single isotropic factor $s = 1.1 / \max_{i,j}|\tilde X_{ij}|$ (the same scale on both axes) so that its most extreme coordinate just reaches the plot border. Because the scaling is isotropic, only the size changes—the shape and orientation of the cloud are preserved—so you can see that the vectors converge along the cloud's long axis, i.e. the direction of maximum variance.

In the context of our data, this direction is the axis of maximum variance.

Orthogonality of Eigenvectors¶

Premise: Let $\mathbf{C} \in \mathbb{R}^{n \times n}$ be a symmetric matrix ($\mathbf{C} = \mathbf{C}^T$). Let $\mathbf{q}_1, \mathbf{q}_2$ be eigenvectors with corresponding distinct eigenvalues $\lambda_1, \lambda_2$ (where $\lambda_1 \neq \lambda_2$).

Starting from the fundamental eigen-equations:

1. $\mathbf{C}\mathbf{q}_1 = \lambda_1 \mathbf{q}_1$
2. $\mathbf{C}\mathbf{q}_2 = \lambda_2 \mathbf{q}_2$

From the first eigen-equation:

$$(\lambda_1 \mathbf{q}_1)^T \cdot \mathbf{q}_2 = (\mathbf{C}\mathbf{q}_1)^T \cdot \mathbf{q}_2$$ By the property of transposes $(AB)^T = B^T A^T$: $$\lambda_1 \mathbf{q}_1^T \mathbf{q}_2 = \mathbf{q}_1^T \mathbf{C}^T \mathbf{q}_2$$

Apply the symmetry property. Since $\mathbf{C} = \mathbf{C}^T$: $$\lambda_1 \mathbf{q}_1^T \mathbf{q}_2 = \mathbf{q}_1^T \mathbf{C} \mathbf{q}_2$$

From the second eigen-equation: $$\mathbf{q}_1^T \cdot (\lambda_2 \mathbf{q}_2) = \mathbf{q}_1^T \cdot (\mathbf{C} \mathbf{q}_2) $$ $$\lambda_2 \mathbf{q}_1^T \mathbf{q}_2 = \mathbf{q}_1^T \mathbf{C} \mathbf{q}_2$$

The difference of the two result: $$(\lambda_1 - \lambda_2) (\mathbf{q}_1^T \mathbf{q}_2) = 0$$

Conclusion: Since we assumed the eigenvalues are distinct ($\lambda_1 - \lambda_2 \neq 0$), the scalar product must be zero:$$\mathbf{q}_1^T \mathbf{q}_2 = 0$$This proves that the eigenvectors $\mathbf{q}_1$ and $\mathbf{q}_2$ are orthogonal.

Eigen-decomposition¶

Eigen-decomposition is the process of "deconstructing" a square matrix into a set of characteristic directions and scales. In the context of data science and robotics, it reveals the hidden geometric structure of a linear transformation or a covariance matrix.

Factorization of a matrix $\mathbf{C}$¶

1. The Starting Point: The Vector Level

By definition, for an individual eigenvector $\mathbf{q}_i$ and its eigenvalue $\lambda_i$, we have:

$$\mathbf{C}\mathbf{q}_i = \lambda_i \mathbf{q}_i$$

This tells us that $\mathbf{C}$ acts on $\mathbf{q}_i$ just like a simple scalar multiplication (no rotation).

2. Scaling to the Matrix Level

If our matrix $\mathbf{C}$ is $n \times n$ and has $n$ linearly independent eigenvectors, we can "stack" these individual equations side-by-side.

Let $\mathbf{Q}$ be the matrix where each column is one of these eigenvectors:

$$\mathbf{Q} = \begin{bmatrix} | & | & & | \\ \mathbf{q}_1 & \mathbf{q}_2 & \dots & \mathbf{q}_n \\ | & | & & | \end{bmatrix}$$

If we multiply $\mathbf{C}$ by this entire matrix $\mathbf{Q}$, the rules of matrix multiplication tell us:

$$\mathbf{C}\mathbf{Q} = \begin{bmatrix} | & | & & | \\ \mathbf{C}\mathbf{q}_1 & \mathbf{C}\mathbf{q}_2 & \dots & \mathbf{C}\mathbf{q}_n \\ | & | & & | \end{bmatrix}$$

Using our definition from Step 1, we can replace each $\mathbf{C}\mathbf{q}_i$ with $\lambda_i\mathbf{q}_i$:

$$\mathbf{C}\mathbf{Q} = \begin{bmatrix} | & | & & | \\ \lambda_1\mathbf{q}_1 & \lambda_2\mathbf{q}_2 & \dots & \lambda_n\mathbf{q}_n \\ | & | & & | \end{bmatrix}$$

3. Factoring out the Eigenvalues

Notice that the right side can be rewritten as the product of $\mathbf{Q}$ and a diagonal matrix $\mathbf{\Lambda}$:

$$\mathbf{C}\mathbf{Q} = \mathbf{Q}\mathbf{\Lambda}$$

Where $\mathbf{\Lambda} = \text{diag}(\lambda_1, \lambda_2, \dots, \lambda_n)$.

4. Completing the Factorization

Since we assumed the eigenvectors are linearly independent, the matrix $\mathbf{Q}$ is guaranteed to be invertible. We can therefore multiply both sides by $\mathbf{Q}^{-1}$ from the right:

$$\mathbf{C}\mathbf{Q}\mathbf{Q}^{-1} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}$$$$\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}$$

Key Requirements (The "Catch")

Not every matrix can be factorized this way. For this math to work, two conditions must be met:

1. Square Matrix: $\mathbf{C}$ must be $n \times n$.
2. Eigenbasis existence: you must be able to find $n$ linearly independent eigenvectors. A general "defective" matrix has too few, so $\mathbf{Q}$ is not invertible and the factorization fails. In our setting this never happens: a symmetric matrix—and every covariance matrix is symmetric—is guaranteed by the spectral theorem to possess a full set of $n$ orthonormal eigenvectors. Hence $\mathbf{Q}$ is always invertible, and in fact orthogonal ($\mathbf{Q}^{-1} = \mathbf{Q}^T$).

Note that "$\mathbf{Q}$ is invertible" (which makes the factorization possible) is a different statement from "$\mathbf{C}$ is invertible." The invertibility of $\mathbf{C}$ itself follows separately from positive-definiteness: the determinant equals the product of the eigenvalues, $\det\mathbf{C} = \prod_i \lambda_i$, and since every $\lambda_i > 0$ (see above), $\det\mathbf{C} > 0$, so $\mathbf{C}$ is non-singular. We use this fact later for $\mathbf{C}^{-1}$.

Matrix Factorization (Summary)¶

If a matrix $\mathbf{C} \in \mathbb{R}^{n \times n}$ has $n$ linearly independent eigenvectors, we can group them into a matrix $\mathbf{Q}$ and the eigenvalues into a diagonal matrix $\mathbf{\Lambda}$ (Lambda):

$$\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}$$

• $\mathbf{Q}$: A matrix where each column is an eigenvector of $\mathbf{C}$. We can normalize these eigenvectors.
• $\mathbf{\Lambda}$: A diagonal matrix where $\Lambda_{ii} = \lambda_i$.
• $\mathbf{Q}^{-1}$: The inverse transformation that brings the data back to the original basis.

The Connection to the Sandwich Product:

If $\mathbf{C}$ is a symmetric matrix (like a covariance matrix), its eigenvectors are orthogonal (see above), meaning $\mathbf{Q}^{-1} = \mathbf{Q}^T$. This reveals that:

$$\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$$

Notice the structure!
This is the exact same sandwich product we used for rotations. This tells us that any covariance matrix can be viewed as a simple diagonal matrix $\mathbf{\Lambda}$ (where variances are independent) that has been actively rotated into its current position by the eigenvector matrix $\mathbf{Q}$.

Is $\mathbf{Q}$ really a rotation? Orthogonality ($\mathbf{Q}^T\mathbf{Q}=\mathbf{I}$) only forces $\det\mathbf{Q}=\pm 1$, which also allows a reflection. But eigenvectors are fixed only up to sign, and flipping the sign of one column leaves $\mathbf{C}=\mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$ unchanged while flipping $\det\mathbf{Q}$. So we can always choose $\det\mathbf{Q}=+1$ and treat $\mathbf{Q}$ as a genuine rotation. (NumPy's eig/eigh make no such guarantee, so fix the sign yourself if it matters.)

The Spectral Decomposition: $\mathbf{C}$ as a Sum of Rank-1 Projectors¶

The sandwich product $\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$ can be written equivalently as an explicit sum over the eigen-pairs. The key is a general identity for a matrix product: if $\mathbf{A}$ has columns $\mathbf{a}_i$ and $\mathbf{B}$ has rows $\mathbf{b}_i^T$, then

$$\mathbf{A}\mathbf{B} = \sum_i \mathbf{a}_i\, \mathbf{b}_i^T \qquad \text{(sum of outer products).}$$

Apply this with $\mathbf{A} = \mathbf{Q}\mathbf{\Lambda}$ and $\mathbf{B} = \mathbf{Q}^T$. The columns of $\mathbf{Q}\mathbf{\Lambda}$ are the scaled eigenvectors $\lambda_i \mathbf{q}_i$, and the rows of $\mathbf{Q}^T$ are the eigenvectors $\mathbf{q}_i^T$, so

$$\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T = \sum_{i=1}^{n} \lambda_i\, \mathbf{q}_i \mathbf{q}_i^T.$$

Each term $\mathbf{q}_i \mathbf{q}_i^T$ is an $n \times n$ matrix of rank 1: it is the orthogonal projector onto the line spanned by the unit eigenvector $\mathbf{q}_i$ (for a unit vector, $\mathbf{q}_i \mathbf{q}_i^T$ applied to any $\mathbf{v}$ returns its component along $\mathbf{q}_i$). So the eigen-decomposition presents $\mathbf{C}$ as a weighted sum of projectors onto the principal axes, each weighted by its eigenvalue (variance) $\lambda_i$.

This "sum" form is the one we truncate in PCA: keeping only the largest few $\lambda_i \mathbf{q}_i \mathbf{q}_i^T$ terms gives the best low-rank approximation of $\mathbf{C}$.

Summary Table:

The Trace Revisited: Total Variance = Sum of Eigenvalues¶

In the tensors notebook we saw that the trace of a covariance matrix is the total variance—the sum of the per-axis variances, $\operatorname{tr}(\mathbf{C}) = \sum_i C_{ii}$. The eigen-decomposition now gives this quantity a second, basis-free reading.

Using $\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$, the cyclic property of the trace, $\operatorname{tr}(\mathbf{A}\mathbf{B}) = \operatorname{tr}(\mathbf{B}\mathbf{A})$, and the orthogonality $\mathbf{Q}^T\mathbf{Q} = \mathbf{I}$:

$$\operatorname{tr}(\mathbf{C}) = \operatorname{tr}(\mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T) = \operatorname{tr}(\mathbf{\Lambda}\mathbf{Q}^T\mathbf{Q}) = \operatorname{tr}(\mathbf{\Lambda}) = \sum_i \lambda_i$$

So the total variance can be read in two equivalent ways:

• as the sum of variances along the coordinate axes (the diagonal entries $C_{ii}$), or
• as the sum of variances along the principal axes (the eigenvalues $\lambda_i$).

Each eigenvalue is exactly the variance captured along its eigenvector: evaluating the quadratic form in the direction of a unit eigenvector $\mathbf{q}_i$ gives

$$\mathbf{q}_i^T \mathbf{C}\, \mathbf{q}_i = \mathbf{q}_i^T (\lambda_i \mathbf{q}_i) = \lambda_i\, (\mathbf{q}_i^T \mathbf{q}_i) = \lambda_i.$$

A rotation of the data reshuffles the spread among the coordinate-axis variances $C_{ii}$ but leaves the eigenvalues $\lambda_i$—and therefore the total $\operatorname{tr}(\mathbf{C})$—unchanged.

Why Factorize? The Change of Basis Perspective

To understand why we write $\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}$, think of it as a three-step pipeline. Instead of applying the transformation $\mathbf{C}$ directly, we "simplify" the space first.

1. $\mathbf{Q}^{-1}$ (Coordinate Change): We move from the standard basis (the world’s $x, y, z$) into the eigenbasis (the matrix's natural axes). In this new coordinate system, the variables are no longer "tangled" or correlated.
2. $\mathbf{\Lambda}$ (Independent Scaling): Since $\mathbf{\Lambda}$ is a diagonal matrix, it performs the simplest possible operation: it just stretches or shrinks each axis independently by its eigenvalue $\lambda_i$. No rotation or shearing happens here.
3. $\mathbf{Q}$ (Return to World): We rotate the result back into the original world coordinates.

Computational Advantage: The same factorization makes repeated application of a matrix cheap. This is rarely needed for a covariance matrix, but it is exactly what we need for a transformation that is applied over and over—for example a dynamics matrix $\mathbf{M}$ that advances a system one step at a time (we meet precisely this in the stability section below). Computing $\mathbf{M}^{100}$ directly is expensive and accumulates rounding errors. Using the factorized form $\mathbf{M} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}$, however, the inner $\mathbf{Q}^{-1}\mathbf{Q}$ pairs all cancel:

$$\mathbf{M}^{100} = (\mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1})^{100} = \mathbf{Q}\mathbf{\Lambda}^{100}\mathbf{Q}^{-1}$$

Since $\mathbf{\Lambda}$ is diagonal, $\mathbf{\Lambda}^{100}$ is just each diagonal element $\lambda_i$ raised to the power of 100. This turns a massive matrix-matrix multiplication problem into a simple element-wise power problem.

The inverse of $C$¶

For a symmetric positive-definite covariance matrix $\mathbf{C} = \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^T$, the inverse $\mathbf{C}^{-1}$ is calculated by applying the inverse property of matrix products $(ABC)^{-1} = C^{-1}B^{-1}A^{-1}$:

$$\mathbf{C}^{-1} = \left( \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T \right)^{-1} \\ = (\mathbf{Q}^T)^{-1} \mathbf{\Lambda}^{-1} \mathbf{Q}^{-1} = \mathbf{Q}\mathbf{\Lambda}^{-1}\mathbf{Q}^T$$

Where:

• $\mathbf{\Lambda}^{-1}$: A diagonal matrix where each diagonal element is the reciprocal of the eigenvalue, $\Lambda_{ii}^{-1} = 1/\lambda_i$.
• Structure: The resulting matrix $\mathbf{C}^{-1}$ remains symmetric and positive-definite, as all diagonal elements of $\mathbf{\Lambda}^{-1}$ are positive ($1/\lambda_i > 0$).

The square root of $C$¶

with normalized eigenvectors $ \mathbf{q}_i^T \mathbf{q}_j = \delta_{ij}$ in $\mathbf{Q}$ is the square root of $C$:

$$\mathbf{C}^{\frac{1}{2}} = \mathbf{Q} \mathbf{\Lambda}^{\frac{1}{2}} \mathbf{Q}^T$$

where $\mathbf{\Lambda}^{\frac{1}{2}}$ is the diagonal matrix containing the square roots of the (positive) eigenvalues $\sqrt{\lambda_i}$.

This definition is consistent because multiplying the matrix by itself retrieves the original covariance matrix $\mathbf{C}$:

$$\mathbf{C} = \mathbf{C}^{\frac{1}{2}} \mathbf{C}^{\frac{1}{2}} = \mathbf{Q} \mathbf{\Lambda}^{\frac{1}{2}} \mathbf{Q}^T \mathbf{Q} \mathbf{\Lambda}^{\frac{1}{2}} \mathbf{Q}^T = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$$

Application: Stability Analysis in Dynamic Systems¶

Beyond data science, eigen-decomposition is the primary tool used to determine if a system—such as a self-balancing robot or an autonomous vehicle—will remain under control or spiral into chaos.

The Mathematical Intuition

Imagine a robot’s state (position and velocity) is described by a vector $\mathbf{x}$. The way this state changes over time is often modeled by a system of linear equations:

$$\mathbf{x}_{k+1} = \mathbf{F}\mathbf{x}_k$$

If we apply this repeatedly, the state after $k$ steps is $\mathbf{x}_k = \mathbf{F}^k \mathbf{x}_0$. This is where eigen-decomposition ($\mathbf{F} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}$) becomes powerful. Using the same cancellation of inner $\mathbf{Q}^{-1}\mathbf{Q}$ pairs as above, it lets us view the system not as a tangled mess of variables, but as a set of independent "modes":$$\mathbf{x}_k = \mathbf{Q}\, \mathbf{\Lambda}^k\, \mathbf{Q}^{-1} \mathbf{x}_0$$

Two notes on the symbols. (1) A dynamics matrix $\mathbf{F}$ is generally not symmetric, so here $\mathbf{Q}^{-1} \neq \mathbf{Q}^T$ and the eigenvalues may even be complex—unlike the symmetric covariance matrix $\mathbf{C}$ studied above. (2) This $\mathbf{Q}$ is the matrix of eigenvectors of $\mathbf{F}$. It is not the process-noise covariance $\mathbf{Q}_{k-1}$ from the Kalman filter, even though the prediction equation we use below, $\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{G}\mathbf{u}_{k-1}$, is the very same one used there ($\mathbf{F}, \mathbf{G}, \mathbf{u}$ all carry their Kalman-filter meaning).

Why the Eigenvalues ($\lambda$) Dictate Reality

The diagonal matrix $\mathbf{\Lambda}^k$ contains the eigenvalues raised to the power of $k$ ($\lambda_1^k, \lambda_2^k, \dots$). This reveals the "long-term fate" of the robot:

• Stability ($|\lambda| < 1$): If the magnitude of every eigenvalue is less than 1, then as $k$ increases, $\lambda^k$ shrinks toward zero. Physically, this means if you push the robot, the "error" eventually dies out, and the robot returns to its desired state.
• Instability ($|\lambda| > 1$): If even a single eigenvalue has a magnitude greater than 1, then $\lambda^k$ grows exponentially. Even a tiny gust of wind will cause the robot's tilt or speed to increase until it physically fails or hits a limit.
• Oscillation (Complex $\lambda$): If the eigenvalues are complex numbers, the robot won't just fall; it will oscillate. The imaginary part determines the frequency of the "wobble".

Example: The "Segway" Problem

Consider a self-balancing robot. The matrix $\mathbf{F}$ captures the physics of gravity. Previously, we used $\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{G}\mathbf{u}_{k-1}$ for passive prediction. However, if $\mathbf{F}$ has an eigenvalue $\lambda > 1$ (e.g., $\lambda = 1.05$), our model simply predicts a crash as errors grow by 5% every step.

To prevent this, the computer scientist implements a feedback loop by defining the control input as $\mathbf{u}_k = -\mathbf{L}\mathbf{x}_k$.

Here, $\mathbf{L}$ is the state-feedback (control) gain—the software logic that determines how strongly the robot reacts to its current state.

Not the Kalman gain. Do not confuse this control gain $\mathbf{L}$ with the Kalman gain $\mathbf{K}_k$ from the Kalman-Filter. They serve opposite purposes: the Kalman gain blends a noisy measurement into the state estimate ($\hat{\mathbf{x}}_k^+ = \hat{\mathbf{x}}_k^- + \mathbf{K}_k \mathbf{y}_k$), whereas $\mathbf{L}$ turns the current state into a control action that we feed back into the physics. We use a different letter here precisely to keep the two apart.

By substituting the feedback law back into our prediction model, we "rewrite" the physics:

$$\mathbf{x}_k = (\mathbf{F} - \mathbf{G}\mathbf{L})\mathbf{x}_{k-1}$$

By choosing an $\mathbf{L}$ that shifts all eigenvalues of this new matrix $(\mathbf{F} - \mathbf{G}\mathbf{L})$ into the unit circle ($|\lambda| < 1$), we turn an unstable "falling" system into a stable "self-correcting" one.

Key Takeaway: In AI and Robotics, eigen-decomposition isn't just about "summarizing" data (like PCA); it is about designing the future. It allows us to verify if our software's logic (the feedback gain $\mathbf{L}$) will result in a smooth trajectory or a catastrophic crash.