Free lesson · Prediction foundations

Log loss and overconfident mistakes

Open interactive lessonPractice calculationsExplore labs

Log loss heavily penalizes assigning almost no chance to an event that happens. This rewards honest uncertainty.

Use natural logarithms and report log loss in nats per forecast. Probabilities exactly zero or one create infinite loss when contradicted.

Numerical clipping prevents undefined floating-point operations but changes the score. Record the clipping threshold and never use it to conceal unjustified certainty. Compare forecasts on the same outcome set and horizon.

Apply it in a strategy

• Detect overconfident forecast systems and compare full predictive distributions.
• Record the input timestamp, executable quantity, currency and horizon. Reconcile the result with a cash-flow or state table.
• Stress this failure condition: A rare mislabeled or voided event can dominate log loss; resolve data quality before interpreting model failure.

Research deliverable

Build and explain a log loss and overconfident mistakes worksheet. Detect overconfident forecast systems and compare full predictive distributions.

Evidence boundary: Synthetic arithmetic and scenarios illustrate mechanics. They are not historical returns, a paper replication, or evidence of an executable edge. Research sources and their access limitations are recorded at the end of this module.

# Python 3.10+; standard library unless NumPy is imported below.
# Inputs and outputs use the units defined in this lesson. Synthetic teaching example.
from math import log
def log_loss(probabilities,outcomes):
    if not probabilities or len(probabilities)!=len(outcomes): raise ValueError("Aligned forecasts required")
    if any(not 0<p<1 for p in probabilities) or any(y not in (0,1) for y in outcomes): raise ValueError("Use probabilities strictly inside (0,1)")
    return -sum(y*log(p)+(1-y)*log(1-p) for p,y in zip(probabilities,outcomes))/len(outcomes)

print(log_loss([.8],[1]))