determin.ant
05-vector-spaces / 5.4

Module 5: Vector Spaces

Rank-Nullity Theorem

Where you'll see this: in neural networks, the number of linearly independent features a layer can distinguish is its effective rank. If rank drops below the number of classes, no amount of training can separate them. Rank-Nullity tells you the ceiling on what a linear layer can learn.

The Rank-Nullity theorem is one of the most fundamental results in linear algebra. It says that inputs to a matrix split cleanly into two complementary parts.

The theorem

For any m×n matrix A:

rank(A)+nullity(A)=n\text{rank}(A) + \text{nullity}(A) = n

where n is the number of columns (the dimension of the input space), rank(A) = dim(column space), and nullity(A) = dim(null space).

What it means

Every input vector splits into two perpendicular components:

  • The row space component — this part "survives" the transformation
  • The null space component — this part gets annihilated (sent to zero)

Rank counts the surviving dimensions. Nullity counts the lost ones. Together they always add up to the total input dimension n.

Think of it as a conservation law: dimensions don't appear or disappear. They either make it through the transformation (rank) or get zeroed out (nullity). rank + nullity = total input dimensions. Always.

Consequences

For a square n×n matrix:

  • Rank n → nullity 0 → the matrix is invertible, null space is just zero
  • Rank n−1 → nullity 1 → one direction collapses, infinite solutions or no solution
  • Rank 0 → everything goes to zero (only possible for the zero matrix)

Try it

Drag the column vectors to change the rank. The bar shows how the two dimensions are split between column space (green) and null space (amber). When vectors are parallel, rank drops to 1 and nullity rises to 1 — they always sum to 2.

Verifying in NumPy:
import numpy as np
from scipy.linalg import null_space

A = np.array([[1., 2.], [2., 4.]])  # rank 1
n = A.shape[1]  # number of columns = 2

rank = np.linalg.matrix_rank(A)      # 1
nullity = n - rank                    # 1
ns = null_space(A)                    # the null space vector(s)

print(f"rank={rank}, nullity={nullity}, sum={rank+nullity}")  # 1 1 2
RankChange of Basis
-6-6-5-5-4-4-3-3-2-2-1-1112233445566col₁col₂

rank + nullity = n  →  2 + 0 = 2

rank = 2
column space (dim = 2) null space (dim = 0)

Drag the column vectors until they're parallel. The rank drops to 1, nullity rises to 1 — they always sum to 2.