Brier Score and Log Loss: Judging a Sports Prediction Model

Como won 3-1 at Torino on 27 October. The Betting Tips AI tip, written on 25 October, backed Como. Pinnacle's match-day prices made Como the favourite too. Same pick, same result, one tick in the win column each.

The scores differ anyway. On the Brier scale the tip gets 0.39 and Pinnacle's prices get 0.31. On log loss it is 0.71 against 0.60. Lower is better in both. A win rate never sees that gap, because it grades the pick and ignores how much confidence stood behind it. (The prices were captured about 58 hours after the tip was written, a detail that matters further down.)

Same pick, different score

The tip gave Como 49%, Torino 27% and a draw 24%. Pinnacle's prices, with the bookmaker's margin taken out, imply about 55% for Como, 20% for Torino and 25% for the draw. A probability forecast makes a bigger claim than a pick does. It says how often the forecaster expects to be right, and Brier score and log loss are two standard ways to grade that claim.

Take ten home games in which the home side wins six. One forecaster gives the home side 60% every time, another gives it 90% every time. Both back the home side and both are right six times in ten, so a win rate cannot separate them. For a yes-or-no question, the Brier score squares the gap between forecast and result, with 1 for yes and 0 for no, then averages. The 60% forecaster scores 0.24. The 90% forecaster scores 0.33. The second one claimed a certainty the games did not deliver, and the score charges for it. (Those ten games are an invented illustration, not data.)

A hit rate has the same blind spot. The break-even math for AI betting tips shows what a win rate must clear to pay. It cannot show whether the confidence behind each pick was earned, which is what calibration checks.

Two rulers for the same probabilities

Football has three outcomes. The Brier score adds up the squared gaps for home, draw and away, then averages across matches. scikit-learn's documentation writes it that way and gives the range as 0 to 2. Glenn Brier introduced it in a 1950 paper in Monthly Weather Review, to check probability forecasts of the weather.

Log loss takes a different route. It ignores the outcomes that did not happen. It takes the probability the forecast gave to the one that did and averages its negative natural log. The scikit-learn log_loss page has the definition.

Both belong to the family of proper scoring rules. A forecaster scores best on average by reporting what they believe. Gneiting and Raftery's 2007 review is the standard reference. For an easier way in, Niall MacKay's Who has the best probabilities? Luck versus skill in prediction tournaments is an informal introduction aimed at less mathematical readers.

Here is a pair of invented forecasts, small enough to check by hand. Forecast A says 50% home, 30% draw, 20% away. Forecast B says 80%, 10%, 10%. If the home side wins, A's Brier score is (0.5 - 1)² + 0.3² + 0.2², which is 0.25 + 0.09 + 0.04 = 0.38. B's is 0.04 + 0.01 + 0.01 = 0.06.

Forecast	Result	Brier	Log loss
A: 50 / 30 / 20	Home win	0.38	0.693
B: 80 / 10 / 10	Home win	0.06	0.223
A: 50 / 30 / 20	Away win	0.98	1.609
B: 80 / 10 / 10	Away win	1.46	2.303

The bold forecast wins big when it is right and loses big when it is wrong. Log loss has no ceiling and punishes confident misses without limit: put 1% on the result that happens and it reaches 4.6. Brier cannot pass 2 with three outcomes, however wrong the forecast.

A score needs something to stand next to

A Brier score of 0.58 means nothing alone. It needs a reference, and three are useful.

• Know nothing: 1/3 for each outcome. Brier 0.667, log loss 1.099. A forecast that cannot beat this adds nothing.
• League base rates: how often home, draw and away happen in each league. Across the 454 matches below they came to 196 home wins, 109 draws and 149 away wins, roughly 43%, 24% and 33%. They are computed on the same matches they are scored on, so this baseline gets a small head start.
• Bookmaker prices with the margin removed. Take 1 divided by each decimal price, then divide each of the three by their total so they add to 100%. The amount by which the total exceeded 100% before that step is the margin. In Pinnacle's Torino prices of 4.85, 3.86 and 1.76 it is 3.34%. Then average the books for each match.

454 matches and one awkward footnote

First, what is being scored. The probabilities come from a prompt, not a fitted statistical model. Betting Tips AI, the tips site run by Collatio Media, asks Google's Gemini (gemini-2.5-flash-lite) with web search switched on to write each card. The three outcome probabilities are numbers the model writes and the system rescales to total 100%. The prompt carries fixture details such as teams, league, kickoff and venue. It does not carry bookmaker odds.

Matches scored

454

Kickoffs

19 Sep to 27 Oct 2026

Bookmakers per match

20.4 on average

Prices captured

20 hours before kickoff

Forecast	Brier	Log loss
Know nothing (1/3 each)	0.667	1.099
League base rates	0.629	1.039
Betting Tips AI tip probabilities	0.578	0.974
Bookmaker average, margin removed	0.544	0.921

The table covers finished football matches with a tip written before kickoff and a stored bookmaker snapshot. Lower is better.

Two things follow. The tip probabilities beat the league base rates: 0.578 against 0.629 on Brier, a gap of 0.051 with a standard error of about 0.015, and 0.974 against 1.039 on log loss. The base rates were measured on the same matches, which favours them, so the model's probabilities carry some information beyond how often each result happens in a league. Six weeks of matches and one model version limit how far that goes.

The tips also trail the bookmaker average, by 0.034 on Brier (standard error about 0.010) and 0.054 on log loss. That gap is too large to put down to chance on this sample.

Timing is the obvious suspect, so the same scores were re-run on the matches where the prices were captured closer to the tip. Within 48 hours (83 matches, 29 hours apart on average) the tips scored 0.580 on Brier against 0.541 for the books, and 0.967 against 0.913 on log loss. Within 24 hours (23 matches, 12 hours apart) the Brier scores were 0.630 and 0.584. The gap did not shrink as the lag shrank. The samples are small, with a standard error of about 0.024 on the 48-hour difference and 0.042 on the 24-hour one, so neither subset settles anything. They do show that the 106-hour lag is not enough on its own to explain the shortfall. Without prices captured at the moment each tip is written, the question stays open, and the prices that exist are still later than the tips and are not closing prices.

What a fair test would look like

The fix is to capture prices at the moment the tip is written and score both on that snapshot. A second run against the closing line is the sharper benchmark, which is the same reason closing line value matters to bettors. Pinnacle alone would be a stricter yardstick than an average of 20 books. It quotes 397 of the 496 matches with a stored snapshot, enough for a run of its own.

Noise needs its own line. The tips run at a temperature of 0.35, so a second pass over the same fixture can return different probabilities, and one scored probability per match is one draw from a spread. Language models vary between runs in other settings too, a pattern the piece on why AI gives different answers every time measured for search queries.

The same test is a handy filter for paid services. A tipster who publishes only picks and a win rate cannot be scored this way. One who publishes the probability behind each pick can, so ask for it when you compare free and paid AI betting picks. Until prices are captured when the tip is written, the table above is a sanity check against baselines. It is not a ranking of model against market.