Model validation and limitations
The historical test evaluates the rating component. It does not establish the accuracy of the complete conference appearance forecast.
Historical test design
The rating uncertainty fit uses 19 pre-championship CFP polls from 2018, 2019, 2021 and 2022. Evaluation uses 15 polls from the untouched 2023–2025 seasons. Pandemic 2020 and final selection polls are excluded. Only completed FBS-versus-FBS games available before each historical cutoff enter its rating.
All 375 ranked labels in the held-out polls were checked against corroborated official CFP records. Unranked teams were not assigned invented relative rankings.
Held-out results
| Measure | Colley + uncertainty | Win percentage + uncertainty |
|---|---|---|
| Correct ordering | 86.8% | 81.1% |
| Brier score (lower is better) | 0.0936 | 0.1151 |
| Log loss (lower is better) | 0.2944 | 0.3743 |
The subset has 273 dependent pair comparisons across 15 polls in three seasons. They are not independent observations, and these figures do not establish statistical significance. Across all ranked-team pairs, ordering accuracy is 86.0%; deterministic top-25 overlap is 86.7%, with mean absolute ranked-position error of 3.35 places.
What remains unvalidated
No complete historical SportSource rating archive was available. Applying CFP-fitted uncertainty to SportSource remains an unvalidated transfer. This test also does not validate upstream game forecasts, the SEC future-score distribution, correlated rating assumptions, or end-to-end championship appearance probabilities. Calibration is imperfect: comparisons assigned 80–90% confidence were correct about 78% of the time.
Simulation sensitivity
In the published snapshot check, the largest difference between 50,000 and 200,000 simulations was 0.426 percentage points. Removing rating uncertainty changed one ACC team’s estimate by up to 3.386 points. These are snapshot-specific sensitivity checks, not confidence bounds or promises about future accuracy.
Inspect the evidence
- Historical backtest and reliability bins (JSON)
- Official-source cross-check (JSON)
- Appearance-model sensitivity results (JSON)
Return to the full methodology. Report a problem with these results to [email protected].