Free lesson · Linear algebra

Regression, conditioning, and regularisation

Open interactive lesson Practice calculations Explore labs

Start with the idea

Least squares selects coefficients that minimise squared residuals. If predictors nearly duplicate one another, many coefficient combinations fit almost equally well; regularisation makes extreme combinations less attractive.

Symbols, units & horizon

y: n-vector of observed targets
X: n-by-k design matrix of known predictors
β: k-vector of coefficients
ε (epsilon): residual vector y−Xβ
β̂: estimated coefficients
T: transpose
−1: matrix inverse when it exists
λ: nonnegative ridge penalty, not an eigenvalue here
I: identity matrix
|| ||²: sum of squared vector entries
∇: gradient, the vector of partial derivatives

When and why to use this

Use regression for hedge ratios, factor exposure and simple forecasting baselines. Use ridge when correlated predictors make coefficients unstable.

y = X β + ε, \hat{β} = (X^{𝖳} X)^{- 1} X^{𝖳} y

Differentiation + linear algebra

Obtain the normal equations

Minimise $L = (y - X β)^{T} (y - X β)$ . Expand: $y^{T} y - 2 β^{T} X^{T} y + β^{T} X^{T} X β$ .
Differentiate with respect to β: $- 2 X^{T} y + 2 X^{T} X β = 0$ , so $X^{T} X β = X^{T} y$ . If full rank, solve to obtain the displayed formula; numerical code should solve rather than form an inverse.
For one predictor with an intercept, centring gives $b = \frac{\sum (x - \overline{x}) (y - \overline{y})}{\sum (x - \overline{x})^{2}}$ , then $a = \overline{y} - b \overline{x}$ .

Work it by hand

x=1,2,3 and y=2,3,5 give b=3/2=1.5 and a=1/3. Prediction at x=4 is 6⅓.

Rows of X are observations, columns are features; include an intercept column if the model requires one. This closed form assumes full column rank. Numerical software should generally use QR or SVD rather than explicitly invert the matrix.

{\hat{β}}_{ridge} = (X^{𝖳} X + λ I)^{- 1} X^{𝖳} y

Differentiation + linear algebra

Add a ridge penalty

Add $λ β^{T} β$ to squared-error loss. Its gradient adds $2 λ β$ , giving $(X^{T} X + λ I) β = X^{T} y$ .
For a centred single predictor, $b = \frac{S_{x y}}{(S_{x x} + λ)}$ . Usually replace the intercept penalty entry by zero.

Work it by hand

With Sxy=3, Sxx=2 and λ=1, ridge slope=1 instead of 1.5. Predictors need consistent scaling before λ has a meaningful comparison.

Ridge penalises squared coefficient size and stabilises collinear features. Standardise predictors using training-only statistics and usually leave the intercept unpenalised. λ is chosen using validation, not the final test. A smaller training residual is not the same as a better forecast.

Research sources, review dates and limitations

Extend the research question

Represent each traded position by its payoff in each scenario. Check feasibility and hedge units before solving for a portfolio.

Continue with the connected research module →

Connect the ideas: Dependence and diversification

Retrieve: Joint behavior matters when combining uncertain outcomes.

Check the change: Correlation, cointegration, covariance and event dependence answer different questions.

Statistics → Time series → Portfolio construction → Portfolio management theory → Prediction foundations

Explain it yourself: Why does a highly correlated pair not automatically provide a converging spread?

Self-assessed. Write your explanation before opening this comparison.

Correlation measures co-movement under a chosen sample and horizon. It does not establish a stationary combination or contractual convergence; those require separate definitions, tests and implementation checks.

Python implementation

Self-contained teaching example. Python 3.10+; dependencies and input conventions are shown in the code and notation. Run in your own Python environment.

import numpy as np  # dependency: numpy

def ols(features, targets):
    """Include a column of ones in X if an intercept is wanted."""
    x, y = np.asarray(features, float), np.asarray(targets, float)
    return np.linalg.lstsq(x, y, rcond=None)[0]  # avoid explicit inverse

def ridge(features, targets, penalty):
    """This version penalises ALL columns, including any intercept."""
    x, y = np.asarray(features, float), np.asarray(targets, float)
    if penalty <= 0:
        raise ValueError("Use OLS for zero penalty")
    return np.linalg.solve(x.T@x + penalty*np.eye(x.shape[1]), x.T@y)

print(ols([[1,0],[1,1],[1,2]], [1,3,5]))

Continue learning

Linear Algebra — all lessons

Quantitative finance and development glossary · Python resources and libraries · Research sources and limitations