Your model says 0.73. The event happens. Were you right?

There is no honest answer at the level of a single forecast — and so many systems are still graded as if there were, which is why calibration stays one of the least-audited properties of deployed models. A probability of 0.73 is not a claim that “yes” will happen; it is a claim that outcomes like this one occur about 73% of the time. To check it, you need many forecasts and a rule that rewards the right thing.

That is what proper scoring rules are for. Most practitioners have met Brier score and log loss in passing, then reached for accuracy or AUC when it actually mattered. This piece is about using scoring rules the way a forecaster does: to grade probabilities honestly, decompose why a number is good or bad, and avoid the pitfalls that quietly make your leaderboard meaningless.

## What “proper” actually means

Proper is a technical guarantee, not a compliment. A scoring rule takes a forecast probability and an observed outcome and returns a number. It is proper if a forecaster minimizes their expected score by reporting their true beliefs, and strictly proper if that honest report is the unique optimum. The rule is incentive-compatible: you cannot game it by shading probabilities toward 0 or 1 to look confident, or toward 0.5 to look safe.

This disqualifies most metrics people reach for. Accuracy depends on a threshold and collapses your probability to a single bit. AUC only cares about ranking — it is invariant to any monotone transform of your scores, so a model that prints 0.9 when it means 0.55 can still post a perfect AUC. Those metrics answer real questions, but not “are these probabilities any good?” Only a proper scoring rule does.

## The two you will actually use

For binary outcomes, two proper rules dominate practice.

Brier score is mean squared error on probabilities: BS = (1/N) Σ (pᵢ − oᵢ)², with oᵢ ∈ {0,1}. It is bounded in [0,1], symmetric, and forgiving — a confident miss costs at most 1.

Log loss (the logarithmic scoring rule) is −(1/N) Σ [oᵢ log pᵢ + (1−oᵢ) log(1−pᵢ)]. It is unbounded: predict 0.001 on something that happens and you eat an enormous penalty. Log loss is the rule to reach for when overconfidence is the expensive failure mode — which, in trading and risk, it usually is.

Both are strictly proper, so they agree on who is honest and disagree only on how harshly to punish confident errors. For ordinal outcomes (score buckets, price ranges) use the Ranked Probability Score, which respects distance between categories. For a full predictive distribution over a continuous target — a crypto price 24 hours out — use CRPS, which reduces to mean absolute error when the forecast is a point.

## Decompose the score, don’t just rank on it

A single Brier number tells you who won, not why. Murphy’s decomposition splits it into three interpretable terms:

BS = Reliability − Resolution + Uncertainty

Uncertainty is the base-rate variance p̄(1−p̄); it is fixed by the data and identical for every model scored on the same events. Reliability is miscalibration — the gap between your stated probabilities and the empirical frequencies at those levels — and you want it near zero. Resolution rewards sharpness: issuing probabilities far from the base rate and being right to do so.

That is the whole game in one line, and it maps to Gneiting’s principle: maximize sharpness subject to calibration. A model that always predicts the base rate is perfectly calibrated and useless (zero resolution); a model that is confidently wrong has high resolution and terrible reliability. It is also why post-hoc calibration (Platt scaling, isotonic regression) improves log loss and Brier without touching AUC — it fixes reliability without changing the ranking that drives resolution.

## A raw score is unreadable without a reference

Is a Brier of 0.18 good? Unanswerable. On events with a 50/50 base rate, the uninformative forecast scores 0.25; on rare events it can score far lower while knowing nothing at all. Absolute scores conflate skill with the difficulty of the question.

The fix is a skill score against an explicit baseline: BSS = 1 − BS_model / BS_reference. Zero means you matched the reference, positive means you beat it, negative means you would have done better with the baseline. Choose the reference honestly — climatology (the long-run base rate) is the standard, and for market forecasting the current market price is a brutal one, because no method reliably beats a liquid market and the price already encodes the crowd’s information. Report the reference alongside every score, or the score means nothing.

## Pitfalls that bite in production

– Infinite log loss. A single p = 0 or 1 on the wrong side sends log loss to infinity. Clip to [1e-15, 1 − 1e-15], but log the clip rate — a high one means your model is emitting overconfident extremes.
– Cross-set comparison. Brier and log loss are only comparable across models scored on the same events. Different event sets carry different uncertainty terms; a leaderboard that mixes them is noise.
– Silent leakage. If a feature encodes the outcome — a line that moved after the event, a label that bled in — your score looks brilliant and means nothing.
– Averaging away the tails. A great mean can hide systematic miscalibration in the confident bin. Plot a reliability diagram; the mean will not show you the hook at the ends.

## The part the leaderboard can’t see: commitment

Here is the failure no scoring rule detects on its own: a forecast graded after the outcome is known is not a forecast. Proper scoring rules assume the probability was fixed before the event. Nothing in the arithmetic stops someone from adjusting a stored prediction, re-running with hindsight, or quietly dropping the losing bets before publishing. The rule stays proper; the evaluation becomes fiction.

The only durable fix is to make commitment verifiable rather than trusted. In our own work at NeuPortal (neuportal.ai) we treat this as the primary problem: each forecast is locked before the event, SHA-256 hashed and Bitcoin-timestamped via OpenTimestamps, then scored in public with Brier and log loss — losses included. The benchmark we care about is not accuracy, which is easy to cherry-pick, but verifiability, which is not. A proper scoring rule is only as honest as the timestamp underneath it.

Grading a probabilistic forecast well is three disciplines stacked: pick a proper rule so honesty is optimal, decompose it so you know whether you are winning on calibration or sharpness, and anchor it to a reference and a timestamp so the number cannot be gamed. Do that and your leaderboard starts measuring skill instead of storytelling.

Educational content — not financial or betting advice.