determin.ant — Linear Algebra from Scratch

Where you'll see this: gradient clipping in deep learning uses the Frobenius norm to limit weight updates. Regularization terms (L2, weight decay) penalize the Frobenius norm of weight matrices. The spectral norm of a layer controls its Lipschitz constant — critical for GANs and robust models.

A norm measures the "size" of a mathematical object. For vectors we have the Euclidean norm. For matrices, there are several useful choices, each emphasizing different aspects of size.

Frobenius norm

The simplest matrix norm: treat the matrix as a long vector and take the Euclidean norm of its entries:

\|A\|_F = \sqrt{\sum_{i,j} a_{ij}^2} = \sqrt{\text{tr}(A^T A)} = \sqrt{\sigma_1^2 + \sigma_2^2 + \cdots}

The last equality links it to SVD: the Frobenius norm is the Euclidean norm of the singular values. It's easy to compute and differentiable everywhere.

Spectral norm (operator 2-norm)

The spectral norm is the largest factor by which A can stretch any vector:

\|A\|_2 = \max_{\mathbf{x} \neq 0} \frac{\|A\mathbf{x}\|}{\|\mathbf{x}\|} = \sigma_1

It equals the largest singular value. Geometrically, it's the semi-major axis of the ellipse that A maps the unit circle to.

The spectral norm is the "worst case" amplification. If ‖A‖₂ = 5, some input gets stretched by a factor of 5. No input gets stretched more.

Induced 1-norm and ∞-norm

$\|A\|_1$ = maximum absolute column sum. $\|A\|_\infty$ = maximum absolute row sum. These are cheaper to compute than the spectral norm but less geometrically meaningful.

The submultiplicative property

All matrix norms satisfy $\|AB\| \leq \|A\| \cdot \|B\|$ . This is crucial for bounding errors in matrix computations — errors don't compound faster than the product of norms.

Try it

Edit the matrix. The green ellipse shows where A maps the unit circle. The blue dashed circle has radius ‖A‖₂ — the spectral norm. The table shows all four norms updating live.

In NumPy:

import numpy as np
A = np.array([[2., 1.], [0.5, 1.5]])

print(np.linalg.norm(A, 'fro'))  # Frobenius
print(np.linalg.norm(A, 2))      # Spectral (= largest singular value)
print(np.linalg.norm(A, 1))      # Max column sum
print(np.linalg.norm(A, np.inf)) # Max row sum

# Spectral norm via SVD
s = np.linalg.svd(A, compute_uv=False)
print(s[0])  # same as norm(A, 2)

Matrix Norms

Frobenius norm

Spectral norm (operator 2-norm)

Induced 1-norm and ∞-norm

The submultiplicative property

Try it