Principal Component Analysis (PCA)¶

What is PCA?¶

Principal Component Analysis (PCA) answers a simple question: if we are allowed to rotate our coordinate axes, which orientation describes the data most simply?

The idea in three steps:

1. Measure the shape of the data. The covariance matrix $\mathbf{C}$ captures how the data spreads and how its variables are correlated. A correlated cloud looks like a tilted ellipse—its axes are not aligned with $x_1, x_2, \dots$.
2. Rotate to the natural axes. We eigen-decompose $\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$. The eigenvectors (columns of $\mathbf{Q}$) point along the ellipse's own axes—the principal components—and, being orthonormal, they form a new orthonormal coordinate system. Expressing the data in this basis is a pure rotation (a change of basis); it does not move or distort the points, it only re-labels them. In these new coordinates the variables are uncorrelated.
3. Rank and (optionally) truncate. Each eigenvalue $\lambda_i$ is the variance along its principal component, so the components come with a built-in ranking from "most informative" to "least." Keeping only the top few axes and discarding the rest gives dimensionality reduction: fewer numbers per data point, with the smallest possible loss of variance.

So PCA is, in essence, a rotation into the data's own natural axes, ordered by how much variance each axis carries. Everything below makes this precise and shows it in code.

Why Eigen-decomposition works for PCA¶

When we decompose $\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$, we are essentially "un-sandwiching" the matrix to find its core properties:

• Orthogonality: Because $\mathbf{C}$ is a symmetric matrix, its eigenvectors are always perpendicular (orthogonal) to each other. This is a massive advantage: it means our new Principal Components ($\text{PC}_1, \text{PC}_2, \dots$) form a perfect, rigid coordinate grid—just like the original $X$ and $Y$ axes, but rotated.

Why the eigenvectors are the new coordinate system. A set of $n$ orthonormal vectors in $\mathbb{R}^n$ is, by definition, an orthonormal basis—so the eigenvector matrix $\mathbf{Q} = [\mathbf{v}_1\,\cdots\,\mathbf{v}_n]$ is not just a bundle of directions, it is a complete new set of axes. We met this already in eigen-decomposition, where $\mathbf{Q}$ turned out to be orthogonal ($\mathbf{Q}^{-1}=\mathbf{Q}^T$), i.e. a pure rotation. Re-expressing a point in this basis is exactly the passive rotation / change of basis from tensors and transformation: the coordinate of a centered point $\tilde{\mathbf{x}}$ along $\text{PC}_i$ is simply its projection $\tilde{\mathbf{x}}\cdot\mathbf{v}_i$ (stacked together, $\mathbf{Q}^T\tilde{\mathbf{x}}$). That is why the arrows below can be read as genuine axes and not merely as "interesting directions."

• Variance Mapping: The eigenvalues $\lambda_i$ correspond exactly to the amount of variance in the data along the direction of their respective eigenvectors.The largest eigenvalue $\lambda_1$ points to the direction of maximum spread.The smallest eigenvalue points to the direction of least information.

$\mathbf{\Lambda}$ is the covariance matrix in the new coordinates. A covariance matrix transforms under a change of basis by the law $\mathbf{C}' = \mathbf{A}\,\mathbf{C}\,\mathbf{A}^T$ (see tensors and transformation). Choosing the eigenvector basis, i.e. $\mathbf{A} = \mathbf{Q}^T$, gives $$\mathbf{C}' = \mathbf{Q}^T \mathbf{C}\, \mathbf{Q} = \mathbf{Q}^T (\mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T)\, \mathbf{Q} = \mathbf{\Lambda}.$$ So $\mathbf{\Lambda}$ is literally the covariance matrix of the data expressed in the principal-component basis: the off-diagonal entries are zero (the principal components are uncorrelated), and the diagonal entries are the variances $\lambda_i$ along each new axis. This is the precise statement behind "the variables become uncorrelated"—PCA is the rotation that diagonalizes the covariance.

• Total Variance (The Trace): One of the most beautiful properties of this transformation is that the Trace is invariant under rotation. The sum of the diagonal of $\mathbf{C}$ (original variances) is equal to the sum of the eigenvalues (variances in the new system):$$\sum \text{Var}(\text{Original Axes}) = \sum \lambda_i$$This proves that PCA doesn't lose any information; it just "repackages" it.

The eigenvectors are the columns of the rotation matrix¶

Where do the entries of $\mathbf{Q}$ come from, and why do we call it a rotation? Compare with the ordinary 2D rotation by an angle $\theta$:

$$\mathbf{R}(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} = \big[\; \mathbf{e}_1' \;\big|\; \mathbf{e}_2' \;\big].$$

The two columns of $\mathbf{R}(\theta)$ are exactly the rotated axes written in the old coordinates: the first column $\mathbf{e}_1'=(\cos\theta,\sin\theta)$ is the new first axis expressed in the original frame, and the second column $\mathbf{e}_2'=(-\sin\theta,\cos\theta)$ is the new second axis.

The eigen-equation $\mathbf{C}\mathbf{q}_i=\lambda_i\mathbf{q}_i$ produces the same kind of object. Each eigenvector $\mathbf{q}_i$ is a unit vector whose entries are the components of the $i$-th principal axis measured in the original $(x_1,x_2)$ coordinates. Stacking them as columns,

$$\mathbf{Q} = \big[\; \mathbf{q}_1 \;\big|\; \mathbf{q}_2 \;\big],$$

fills $\mathbf{Q}$ column-by-column with the new axes in the old system—precisely how $\mathbf{R}(\theta)$ is built. For a 2D covariance (symmetric, with the signs chosen so $\det\mathbf{Q}=+1$) this makes $\mathbf{Q}$ literally a rotation matrix: there is a single angle $\theta$—the tilt of the data ellipse—such that

$$\mathbf{q}_1 = \begin{bmatrix}\cos\theta\\ \sin\theta\end{bmatrix}, \qquad \mathbf{q}_2 = \begin{bmatrix}-\sin\theta\\ \cos\theta\end{bmatrix}, \qquad \mathbf{Q} = \mathbf{R}(\theta).$$

Reading off coordinates mirrors rotation too: the new coordinates of a centered point are $\mathbf{Q}^\top\tilde{\mathbf{x}} = \mathbf{R}(-\theta)\,\tilde{\mathbf{x}}$—we rotate the point by $-\theta$ (equivalently, rotate the axes by $+\theta$), and the $i$-th new coordinate is the projection $\tilde{\mathbf{x}}\cdot\mathbf{q}_i$.

The one subtlety versus a textbook rotation: eigenvectors are fixed only up to sign, so a solver may return a column pointing the opposite way (giving $\det\mathbf{Q}=-1$, a reflection). Flipping that column's sign restores a proper rotation and leaves $\mathbf{C}=\mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^\top$ unchanged (see eigen-decomposition).

• The Data Cloud (Blue Crosses): These are your $m=100$ measurements. Their elongated shape shows correlation: when $x_1$ increases, $x_2$ usually does too. PCA aims to "unmix" this relationship.
• The Ground Truth (Dashed Ellipse): This is the 2-sigma contour ($2\sigma$) of the true Gaussian distribution. It represents the "hidden" model that generated the data. About 95% of all potential points fall within this boundary.
• The Principal Components (Arrows): These are the Eigenvectors of the covariance matrix, centered at the empirical mean:
  • Red Arrow ($PC_1$): Points in the direction of maximum variance. This is the "main axis" of your data.
  • Green Arrow ($PC_2$): Points in the direction of minimum variance, perpendicular to the first.
  • Length: The arrows are scaled by $\sqrt{\lambda}$ (standard deviation). A longer arrow means more "information" or spread is captured along that axis.

The visualization summarizes the Eigen-decomposition of a Covariance Matrix:

• Direction: The eigenvectors (the arrows) define the orientation of the data cloud.
• Magnitude: The eigenvalues (captured here as standard deviations via sqrt) define the "stretch" or width of the distribution.
• Orthogonality: Because the covariance matrix is symmetric, the eigenvectors are guaranteed to be orthogonal (perpendicular), creating a new, natural coordinate system for your data.

The Takeaway

The plot shows that PCA has found a new coordinate system. Instead of using $x_1$ and $x_2$, we can describe the data more efficiently using the red and green axes. In dimensionality reduction, we would keep the red axis ($PC_1$) and discard the green one ($PC_2$) because it contains less variance.

Dimensionality Reduction¶

To project the data onto the first Principal Component, we mathematically "collapse" every 2D point onto the line defined by the first eigenvector.

The Projection Step

If $\mathbf{q}_1$ is our first eigenvector (the red arrow), the 1D projection $z^{(k)}$ of a centered data point $\tilde{\mathbf{x}}^{(k)}$ is calculated via the dot product:

$$z^{(k)} = \tilde{\mathbf{x}}^{(k)} \cdot \mathbf{q}_1$$

This gives us a single coordinate representing the point's position along the axis of maximum variance.

Why this matters

• Compression: We now only need one number ($z_1$) per data point instead of two, yet we've preserved the maximum possible variance.
• Information Loss: The distance of the black dashed lines represents the "error" or information lost. Because we chose the eigenvector with the largest eigenvalue, this error is mathematically minimized.

What is happening mathematically?

We are performing a change of basis.

• The original data uses the standard basis $\mathbf{e}_1 = [1,0]$ and $\mathbf{e}_2 = [0,1]$.
• The projection uses the Eigenvector basis. By calculating $\tilde{\mathbf{X}}\mathbf{q}_1$, we find the coordinates of our data points relative to that red arrow.

PCA as a Dimensionality Reduction Tool¶

The "magic" of PCA happens when we decide to ignore the eigenvectors with small eigenvalues.

If we have a 100-dimensional dataset, but the first 3 eigenvalues capture 95% of the total variance, we can "project" our data onto those 3 eigenvectors and discard the rest. We effectively simplify our "world" from 100 dimensions down to 3, while keeping almost all the important structural information.

The Scree Plot: Deciding What to Keep¶

A Scree Plot is a diagnostic tool used to visualize the importance of each Principal Component. It plots the eigenvalues (variance) against the component number.

In a tensor context, the scree plot tells us which "dimensions" of the Rank 2 tensor are physically significant and which are just noise. We look for the "elbow" of the plot—the point where the variance drops off sharply.

Python Code: Generating a Scree Plot This script calculates PCA on a synthetic dataset and visualizes how much "information" each eigenvalue captures.

Interpretation¶

• The Bars: Show how much of the "Total Variance" (the Trace) is captured by each eigenvector.

• The Line: Shows the cumulative total. If the first two components reach 90%, it means you can describe the 5D dataset using only 2 dimensions with minimal information loss.

• The "Elbow": The point where the curve flattens out. In a tensor interpretation, this is where we distinguish the Signal (the primary axes of the tensor) from the Noise (the tiny, insignificant fluctuations).

Dimensionality Reduction: Truncating the Tensor¶

Dimensionality reduction is the ultimate goal of PCA. From a tensor perspective, we are identifying which dimensions of the space are physically meaningful (the signal) and which are just background fluctuations (the noise).

By looking at the Scree Plot, we can decide to keep only the top $k$ eigenvectors. We then form a projection matrix $\mathbf{Q}_k$ using these $k$ vectors. When we project our data:

$$\mathbf{\tilde X}_{reduced} = \mathbf{\tilde{X}} \mathbf{Q}_k$$

We are performing a passive rotation into a new coordinate system and then simply "ignoring" the axes that don't carry enough variance.

The Geometry of Truncation When we perform PCA, we aren't just "deleting" columns of data; we are choosing a new subspace that best represents the "energy" or "mass" of our data cloud.

Constructing the Projection Matrix ($\mathbf{Q}_k$)

After sorting our eigenvalues ($\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_n$), we select the first $k$ eigenvectors to form our projection matrix $\mathbf{Q}_k$. This matrix is an $n \times k$ operator.

• Each column of $\mathbf{Q}_k$ is a unit vector (a direction) in the original $n$-dimensional space.
• These $k$ columns are mutually orthonormal, so they span a $k$-dimensional linear subspace of the original $n$-dimensional space. Because we work with the centered data $\mathbf{\tilde X}$, this subspace passes through the origin of the centered coordinates—i.e. through the mean of the original data cloud. (Only in the special case $k = n-1$ is this subspace a "hyperplane" in the strict sense; for general $k$ it is simply a $k$-dimensional flat.)

The Projection Equation: $\mathbf{\tilde X}_{reduced} = \mathbf{\tilde X} \mathbf{Q}_k$

This multiplication is a Passive Transformation. For every data point (row) in $\mathbf{\tilde X}$:

• The dot product between the data point and the first eigenvector gives its new "PC1 coordinate."
• The dot product with the second eigenvector gives the "PC2 coordinate," and so on.

By doing this, we are effectively looking at the data from the perspective of the "best" $k$ axes and ignoring the remaining $d-k$ axes.

Why it is "Truncating the Tensor"

Recall from eigen-decomposition that the sandwich product $\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$ can be written equivalently as a sum of rank-1 projectors, one per principal axis. In this full form the covariance matrix (our Rank 2 tensor) is:

$$\mathbf{C} = \sum_{i=1}^{n} \lambda_i \mathbf{q}_i \mathbf{q}_i^T$$

When we reduce dimensions, we are approximating the tensor by cutting off the end of this sum, keeping only the $k$ terms with the largest eigenvalues:

$$\mathbf{C}_{approx} \approx \sum_{i=1}^{k} \lambda_i \mathbf{q}_i \mathbf{q}_i^T$$

We are claiming that the "physical relationship" described by the tensor is almost entirely contained within those first $k$ directions. The remaining variance (the sum of the ignored eigenvalues) is treated as isotropic noise—shaking that doesn't follow a meaningful pattern.

Summary of Information Loss

The reduction $\mathbf{\tilde X}_{reduced} = \mathbf{\tilde X}\,\mathbf{Q}_k$ maps each $n$-dimensional point to just $k$ coordinates. To compare it against the original we map it back into the full $n$-dimensional space using the same axes, giving the reconstruction

$$\mathbf{\tilde X}_{reconstructed} = \mathbf{\tilde X}_{reduced}\,\mathbf{Q}_k^T = \mathbf{\tilde X}\,\mathbf{Q}_k\mathbf{Q}_k^T.$$

Geometrically, $\mathbf{Q}_k\mathbf{Q}_k^T$ is the orthogonal projector onto the $k$-dimensional subspace: $\mathbf{\tilde X}_{reconstructed}$ is the shadow of each (centered) point on that subspace, still expressed in the original coordinates. (Adding back the mean, $\mathbf{\tilde X}_{reconstructed} + \bar{\mathbf{x}}^T$, returns it to the original, uncentered frame.)

The difference between $\mathbf{\tilde X}$ and $\mathbf{\tilde X}_{reconstructed}$ is the Reconstruction Error.

If the eigenvalues we discarded are near zero, the error is negligible. In tensor terms, we have successfully removed the "dimensions of least resistance," keeping only the axes where the data actually lives.

Measuring Information Retention

In the tensor perspective, the Trace of the covariance matrix represents the "Total Energy" or "Total Variance" of the system. Since the Trace is invariant under rotation, the sum of our eigenvalues must equal the total variance of the original data.When we reduce the dimensions from $n$ to $k$, we can calculate the percentage of information kept using this formula:$$\text{Explained Variance} = \frac{\sum_{i=1}^{k} \lambda_i}{\sum_{j=1}^{n} \lambda_j}$$

PCA: Summary¶