determin.ant — Linear Algebra from Scratch

Where you'll see this: gradient descent converges when the Hessian matrix (second derivatives of the loss function) is positive definite — the loss landscape is bowl-shaped everywhere. Covariance matrices in statistics and the Gram matrix in kernel methods are always positive semi-definite. It's one of the most checked properties in optimization.

Positive definiteness is eigenvalues meeting a sign condition — and it has profound geometric consequences.

The definition

A symmetric matrix A is positive definite (PD) if for every non-zero vector x:

\mathbf{x}^T A \mathbf{x} > 0

The expression $\mathbf{x}^T A \mathbf{x}$ is called a quadratic form. It assigns a scalar to every point in space.

The eigenvalue test

A symmetric matrix is positive definite if and only if all its eigenvalues are strictly positive. This gives the full classification:

PD: all λ > 0 — quadratic form is always positive (bowl)
PSD: all λ ≥ 0 — non-negative (flat bowl, touches zero)
Indefinite: mixed signs — saddle shape (some positive, some negative)
ND: all λ < 0 — always negative (inverted bowl)

PD = bowl-shaped = unique minimum. This is why gradient descent works: if the Hessian is PD, the loss has exactly one minimum and descent always converges.

Connection to Cholesky decomposition

Every positive definite matrix has a Cholesky decomposition: $A = LL^T$ where L is lower triangular. This is the "square root" of a matrix and is used extensively in statistics (sampling from a Gaussian) and numerical solvers (it's twice as fast as LU for symmetric systems).

Try it

The heatmap shows the quadratic form xᵀAx — how positive or negative the value is at each point. Blue/green = positive, red = negative. Use the presets or edit the matrix. A positive definite matrix produces a pure green bowl. An indefinite matrix produces a saddle.

Checking in NumPy:

import numpy as np

A = np.array([[3., 1.], [1., 2.]])
eigenvalues = np.linalg.eigvalsh(A)  # for symmetric matrices
print(all(eigenvalues > 0))  # True — positive definite

# Cholesky decomposition (only works for PD)
L = np.linalg.cholesky(A)
print(np.allclose(L @ L.T, A))  # True

Positive Definite Matrices

The definition

The eigenvalue test

Connection to Cholesky decomposition

Try it