determin.ant — Linear Algebra from Scratch

Where you'll see this: every shadow is a projection. More practically: in computer graphics, projecting 3D scenes onto a 2D screen is literally a matrix projection. In signal processing, projecting onto a frequency basis decomposes a signal. In ML, PCA projects data onto its principal components.

Projection answers a simple question: given a vector a and a direction b, what is the component of a along b?

The formula

The projection of a onto b is:

\text{proj}_{\mathbf{b}}(\mathbf{a}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}} \mathbf{b}

Read this as: "how much of a points in the b direction, times the b direction." The scalar $\frac{\mathbf{a} \cdot \mathbf{b}}{|\mathbf{b}|^2}$ is how far along b the foot of the perpendicular falls.

The perpendicular component

Every vector splits cleanly into two perpendicular parts:

\mathbf{a} = \underbrace{\text{proj}_{\mathbf{b}}(\mathbf{a})}_{\text{along } \mathbf{b}} + \underbrace{(\mathbf{a} - \text{proj}_{\mathbf{b}}(\mathbf{a}))}_{\perp \text{ to } \mathbf{b}}

The remainder $\mathbf{a} - \text{proj}_{\mathbf{b}}(\mathbf{a})$ is always perpendicular to b. You can verify: take the dot product — it's zero.

Projection decomposes a vector into "what's in this direction" and "what's left over." The left-over part is always perpendicular to the projection direction.

Projection onto a subspace

This generalizes: you can project onto a plane, or any subspace. The projection matrix is $P = A(A^T A)^{-1} A^T$ . Applying P to any vector gives its component in the subspace.

Try it

Drag vector a (blue) and direction b (pink). The green arrow shows the projection. The dashed purple line is the perpendicular remainder. Notice that as a approaches the b direction, the projection equals a; when a is perpendicular to b, the projection is zero.

In NumPy:

import numpy as np
a = np.array([3.0, 3.0])
b = np.array([4.0, 1.0])

scalar = np.dot(a, b) / np.dot(b, b)
proj = scalar * b
perp = a - proj

print(proj)               # projection along b
print(np.dot(perp, b))    # ~0 — perpendicular check