Free lesson · Research & robust tuning

Fine-tuning, retraining and the research-to-production decision

Open interactive lessonPractice calculationsExplore labs

Decide whether retraining is scheduled or triggered by a measurable event, and what data it may use. Keep label maturity, preprocessing and embargo rules identical during retraining. Fine-tuning a pretrained model also requires a documented training cutoff: historical evaluation can be contaminated if pretraining already saw the test period.

Compare a frozen incumbent, a freshly trained model and a fine-tuned model at the same decision times. Evaluate their incremental net value after turnover, compute and operational costs. A complex candidate should justify its additional failure modes and maintenance burden.

Promotion needs versioned artifacts, reproducible signals, paper reconciliation and independent portfolio constraints. Rollback criteria should refer to observed data or operating conditions, rather than allowing unlimited discretionary rescue tuning when performance deteriorates.

Apply it in a strategy

• Lock candidate training cutoff, retraining schedule and promotion metric before the final period.
• Compare paired results under equal costs and exposure limits; inspect failures as well as averages.
• Promote only after reproducibility and paper reconciliation, with independent limits and a reversible model-version switch.

Research deliverable

Create a promotion memo containing incremental net results, uncertainty, operating budget, training lineage and rollback criteria.

Research checkpoint · reviewed 11 September 2026

These sources inform the questions to test. A result is conditional on its data, simulator and evaluation design. The examples in this module are teaching calculations, not reproductions of the reported experiments.

Validation-method comparison. This recent journal study compares out-of-sample testing approaches in a synthetic controlled environment and reports favourable results for combinatorial purged validation. Publisher abstract only reviewed. The DOI is dated 2024; it is not a 2026 release. The synthetic setting does not establish that one validation layout is best for every deployment schedule. Our chronological outer evaluation remains tied to the actual retraining policy.

Further reading: Backtest overfitting in the machine learning era · Knowledge-Based Systems ↗

Foundational selection diagnostic. The author-hosted paper develops a framework and combinatorially symmetric cross-validation examples for assessing selection overfitting. The framework and examples were reviewed; the author bibliography identifies the journal publication as 2017. Its diagnostic is distinct from a new chronological deployment test. The illustrative independent-test calculation in this module is not the paper’s PBO estimator.

Further reading: Bailey, Borwein, López de Prado & Zhu · The Probability of Backtest Overfitting ↗

Research sources, review dates and limitations

Extend the research question

Audit pretrained models and text timestamps alongside the train/test split. A causal prediction pipeline can still inherit future information from upstream processing.

Continue with the connected research module →

Connect the ideas: Information and decision time

Retrieve: Use only information available when the decision is made.

Check the change: Observation dates, release delays, revisions and label maturity require different availability checks.

Statistics → Research & backtests → Point-in-time data → Time series → Financial machine learning → Putting it all together

def paired_improvement(candidate,incumbent):
    if not candidate or len(candidate)!=len(incumbent): raise ValueError("Nonempty aligned returns required")
    differences=[a-b for a,b in zip(candidate,incumbent)]
    return differences,sum(differences)/len(differences)

print(paired_improvement([.01,-.005,.004],[.008,-.004,.003]))