Free lesson · Alternative data

Measure improvement against a frozen baseline

Open interactive lesson Practice calculations Explore labs

Start with the idea

A new data source matters only if it adds information beyond what the existing model already knew.

Symbols, units & horizon

y_i: realized target for test observation i
ŷ_b,i,ŷ_a,i: previously issued baseline and augmented predictions
n: matched test observations
ΔMSE: reduction in squared target units, positive favors augmented model
all forecasts use identical decision horizons

When and why to use this

Decide whether text or alternative data adds forecasting value beyond a simple incumbent.

A new data source matters only if it adds information beyond what the existing model already knew.

Compare baseline and augmented predictions on identical future observations. Fit both using permitted history, count all tuning trials and preserve a final test. Paired errors make it easier to see whether gains occur on the same examples or only in a selected subgroup.

A lower mean-squared error can be concentrated in economically irrelevant states. Also compare forecast calibration, ranking at the trading threshold, turnover and cost-adjusted decisions. Statistical uncertainty needs dependence-aware analysis, not just a better point estimate.

Δ M S E = \frac{1}{n} \sum_{i = 1}^{n} [(y_{i} - {\hat{y}}_{b, i})^{2} - (y_{i} - {\hat{y}}_{a, i})^{2}]

Paired out-of-sample loss comparison

Measure improvement against a frozen baseline

Compute baseline and augmented squared errors for every identical test observation.
Subtract augmented error from baseline error observation by observation.
Average the paired differences; keep model selection separate from this evaluation.

Work it by hand

Targets [0,2], baseline [1,1], augmented [0,1.5]. Baseline MSE=1; augmented MSE=.125. Improvement=.875 target units squared.

Apply it in a strategy

Freeze inputs at the stated decision time and record their units.
Decide whether text or alternative data adds forecasting value beyond a simple incumbent.
Recompute the example, then change the material assumption and explain the difference.

Research deliverable

Measure improvement against a frozen baseline: produce the worked calculation, a timestamped input record and a written decision addressing this limitation: Repeatedly selecting features on the same test window makes the reported improvement optimistic.

These are synthetic mechanics examples, not historical performance or paper replications. Module evidence and research boundaries record the 12 September 2026 review.

Python implementation

Self-contained teaching example. Python 3.10+; dependencies and input conventions are shown in the code and notation. Run in your own Python environment.

# Python 3.10+; standard library and NumPy only.
# Synthetic teaching inputs; conventions and units are defined in the notation above.
import numpy as np

def incremental_mse(actual,baseline,augmented):
    y=np.asarray(actual,float); b=np.asarray(baseline,float); a=np.asarray(augmented,float)
    if y.ndim!=1 or len(y)==0 or y.shape!=b.shape or y.shape!=a.shape: raise ValueError('Matched nonempty vectors required')
    return float(np.mean((y-b)**2-(y-a)**2))

assert incremental_mse([0,2],[1,1],[0,1.5])==.875
print(incremental_mse([0,2],[1,1],[0,1.5]))

Continue learning

Alternative Data: Measurement, Text & Incremental Value — all lessons

Quantitative finance and development glossary · Python resources and libraries · Research sources and limitations