Brier Score

Developed by Glenn Brier in 1950, the Brier score measures the mean squared error between a forecaster's stated probability and the actual binary outcome. For each prediction, you subtract the outcome (1 or 0) from the forecast probability, square the result, then average across all predictions. A forecast of 0.9 that resolves YES contributes only 0.01 to the total, while a forecast of 0.9 that resolves NO contributes 0.81. This asymmetry strongly penalizes miscalibrated overconfidence. In the context of prediction markets, the Brier score can assess how well market prices predict binary events compared to individual forecasters.

The Brier score is one of two dominant strictly proper scoring rules alongside the Log Score. Both reward honest probability reports, but they differ in sensitivity: the log score penalizes confident wrong predictions far more severely (approaching infinity as confidence approaches 1 on a wrong outcome), while the Brier score applies a bounded quadratic penalty. This makes the Brier score more forgiving of extreme errors and preferred in settings where robust comparison across forecasters matters. Platforms like Metaculus use variants of the Brier score to maintain their Leaderboard, and the base rate of events affects how to interpret raw scores. A Brier skill score normalizes against a naive prior baseline.