Free lesson · Research & robust tuning

Hyperparameter optimisation without an unrestricted search

Open interactive lesson Practice calculations Explore labs

Start with the idea

Hyperparameters control how the model is fitted or how the strategy acts. Optimising them uses information from validation results. That information consumption must be counted even when an automated optimiser performs the search.

Symbols, units & horizon

θ: hyperparameter configuration
R̄_val: mean validation score in a declared return unit
s_fold: dispersion of scores across chronological folds in matching units
TO: turnover under a fixed convention
λ: dimensionless stability penalty
η: cost or penalty per unit turnover
J: selection objective, not a probability of success

When and why to use this

Use a bounded selection objective to connect tuning to stability and implementation cost, while reserving independent evidence for the selection procedure.

Use an inner chronological loop to select lookbacks, thresholds, regularisation, learning rate or network width. The outer future loop evaluates the entire selection procedure. Keep the final reserved period untouched until the model family, metric and promotion rule are fixed.

Random search can cover a few influential dimensions efficiently. Bayesian search prioritises configurations based on earlier trials; successive halving stops weak candidates early. Neither prevents overfitting by itself. Trials, manual revisions, feature additions and random-seed selection all consume the research budget.

Prefer broad stable parameter regions over isolated peaks. Inspect nearby parameters and performance by market, year and cost scenario. Early stopping uses validation information too; do not then report that same validation score as an independent result. Allocate equal compute and realistic retraining cost to competing model families.

J (θ) = {\overline{R}}_{v a l} (θ) - λ s_{f o l d} (θ) - η {T O}_{v a l} (θ)

Model assumptions, derivation and arithmetic

Hyperparameter optimisation without an unrestricted search

Choose a common validation return statistic for each candidate, then compute its dispersion across the same folds.
Subtract a prespecified instability penalty and turnover penalty, both converted into matching objective units.
Choose the candidate with the largest J using inner folds only. Evaluate the selected procedure on the outer fold without changing λ, η or the search space.

Work it by hand

Candidate A: mean .010, fold SD .008, turnover 2. Candidate B: mean .009, SD .002, turnover 1. With λ=.5 and η=.001, A scores .004 and B .007, so B is selected.

Apply it in a strategy

Declare search dimensions, ranges, budget, seeds and early-stopping rule before inspecting outer results.
Run chronological inner folds; save all trials and neighbourhood sensitivity, not only the best run.
Refit the selected configuration on permitted history and evaluate the next outer block once.

Research deliverable

Publish a trial ledger, parameter sensitivity surface, compute budget and outer-fold performance for the complete tuning policy.

Python implementation

Self-contained teaching example. Python 3.10+; dependencies and input conventions are shown in the code and notation. Run in your own Python environment.

def robust_selection_score(mean_return,fold_sd,turnover,stability_penalty=.5,cost=.001):
    if min(fold_sd,turnover,stability_penalty,cost)<0: raise ValueError("Nonnegative penalties required")
    return mean_return-stability_penalty*fold_sd-cost*turnover

print(robust_selection_score(.010,.008,2),robust_selection_score(.009,.002,1))

Continue learning

Strategy Research, Backtesting & Robust Optimisation — all lessons

Quantitative finance and development glossary · Python resources and libraries · Research sources and limitations