determin.ant — Linear Algebra from Scratch

Where you'll see this: linear regression is least squares. Every time you fit a line (or curve, or neural network layer) to data, you're minimizing a sum of squared errors. The closed-form solution involves a matrix equation derived directly from projection theory.

Most real-world systems are overdetermined — more data points than unknowns. There's no exact solution. Least squares finds the best approximate one.

The problem

We have n data points and want to fit $y = mx + b$ . For each point (xᵢ, yᵢ), the error is $y_i - (mx_i + b)$ . We want to minimize the total squared error:

\min_{m,b} \sum_i (y_i - mx_i - b)^2

The matrix view

Stack the equations into a matrix: $A\mathbf{x} = \mathbf{b}$ where each row is one data point. When there's no exact solution, the least squares solution is:

\mathbf{x} = (A^T A)^{-1} A^T \mathbf{b}

This is called the normal equation. The term $A(A^T A)^{-1} A^T$ is the projection matrix onto the column space of A.

Least squares is projection. You're finding the point in the column space of A that's closest to b. The residuals are perpendicular to the column space.

The residuals

The vertical distances from each data point to the fitted line are the residuals. Least squares minimizes the sum of their squares — not their absolute values, not their maximum. Squaring penalizes large errors more heavily.

Try it

Click on the canvas to add points. Click an existing point to remove it. The green line updates in real-time. The yellow dashed lines are the residuals. Move points far off the line and watch the residual sum grow.

Linear regression via normal equations:

import numpy as np
# x values, y values
x = np.array([-3, -2, -1, 0, 1, 2, 3])
y = np.array([-2, -1.5, -0.5, 0.5, 1, 2.5, 2])

# Build design matrix: [x, 1] for each point
A = np.column_stack([x, np.ones_like(x)])

# Normal equation: x = (AᵀA)⁻¹Aᵀb
m, b = np.linalg.lstsq(A, y, rcond=None)[0]
print(f"y = {m:.2f}x + {b:.2f}")

Least Squares

The problem

The matrix view

The residuals

Try it