Our public leaderboard reads ai_models.hit_rate, sorts descending, renders the top. Standard design, structurally broken. The failure is not the sort comparator; it is a property of sorting point estimates computed from wildly unequal denominators.

Here is the board as it renders today:

Four rows, four perfect records, none above seven graded calls. Across the table, 327 models sit at exactly 100.0%, and 204 of those have a single graded call. Of the 584 models with at least 20 graded calls, zero are at 100%. The first hundred rows of the board contain no model with ten or more calls.

A leaderboard topped by 204 coin flips is not a leaderboard. It is a sampling artifact rendered as a ranking.

Two populations, one table

The first thing an audit has to separate is how many models exist versus how many have enough evidence to be compared.

-- population split: board cache vs. the fact table
SELECT count(*)                                        AS board_rows,
       count(*) FILTER (WHERE total_predictions > 0)   AS with_evidence,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY total_predictions) AS median_n
FROM ai_models;

WITH t AS (
  SELECT model_id, count(*) AS n, count(*) FILTER (WHERE hit) AS h
  FROM ai_predictions
  WHERE hit IS NOT NULL AND model_id IS NOT NULL
  GROUP BY 1
)
SELECT count(*)                                        AS models,
       sum(n) AS rows, sum(h) AS hits,
       count(*) FILTER (WHERE n >= 20)                 AS n20,
       count(*) FILTER (WHERE n >= 50)                 AS n50,
       count(*) FILTER (WHERE n >= 200)                AS n200,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY n)  AS median_n,
       max(n)                                          AS max_n
FROM t;

Output, unedited:

board_rows 8942 | with_evidence 3229 | median_n 7.0
models 3736 | rows 46284 | hits 26415 | n20 661 | n50 171 | n200 7 | median_n 6.0 | max_n 381

Three numbers matter. The median model has six graded calls, 171 models clear fifty, seven clear two hundred. Sorting 3,229 estimates where the median sample is 7 is not ranking models — it is ranking the draws.

The test: split each model's history in half

If a hit rate reflects a durable property of the model, the rate on the first half of its own history must predict the rate on the second half. If it reflects noise, the halves are independent and the ranking carries no forward information.

The split is by sequence position, not calendar time, so a model that started last week is treated like one that started in June. Both halves need at least ten graded calls, restricting the test to the 661 models with n ≥ 20.

WITH ordered AS (
  SELECT model_id, hit,
         row_number() OVER (PARTITION BY model_id ORDER BY id) AS rn,
         count(*)     OVER (PARTITION BY model_id)             AS n
  FROM ai_predictions
  WHERE hit IS NOT NULL AND model_id IS NOT NULL
),
halves AS (
  SELECT model_id, max(n) AS n,
         avg(CASE WHEN rn <= n/2 THEN hit::int END) AS h1,
         avg(CASE WHEN rn >  n/2 THEN hit::int END) AS h2,
         count(*) FILTER (WHERE rn <= n/2 AND hit)  AS k1,
         count(*) FILTER (WHERE rn <= n/2)          AS n1
  FROM ordered WHERE n >= 20 GROUP BY model_id
)
SELECT count(*) AS models, corr(h1, h2) AS pearson,
       avg(h1) AS mean_h1, stddev_samp(h1) AS sd_h1,
       avg(h2) AS mean_h2, stddev_samp(h2) AS sd_h2
FROM halves;

Result over 661 models:

models 661 | pearson 0.0443 | mean_h1 0.6291 | sd_h1 0.0991 | mean_h2 0.5656 | sd_h2 0.1135

A correlation of 0.0443. Spearman on the same pairs is 0.0261. For practical purposes these are zero — the first half of a model's record tells you almost nothing about its second half.

Note the drift, not just the flatness: first half averages 62.91%, second half 56.56%. Instances are not stationary across their own lifetime. One tuned against recent fixtures starts hot and mean-reverts, and a board built on the early record is systematically reading the hot streak.

What the top of the board is worth

Take the 661 pairs, sort by first-half rate, cut the top 66, look at what that group did next.

The top decile displays 80.85% and delivers 60.36%. The surviving edge over the rest of the population is 3.8 points, not the 24.7 the display implies. The bottom decile displays 46.14% and delivers 56.29% — the global rate within noise. Both tails converge on 56–57% because that is where the population lives; the spread at the tails was estimation error, not skill.

This is what rewarding luck means numerically. A 100.0% display on seven calls is not a stronger signal than a 60% display on 380 calls — it is a weaker one wearing a better font.

The multiplicity check

Under the null that every model draws from the same 57.07% process, the probability of a clean sweep at n calls is 0.5707^n:

n=5   6.05e-02
n=7   1.97e-02
n=10  3.67e-03
n=20  1.34e-05

At n=7 a perfect record happens once in fifty-one. With 3,229 board rows and thousands of small-n models, dozens of perfect records are expected, not remarkable. Summing 0.5707^n over every model's actual n gives an expected 661.75 perfect records. We observe 401.

Observed is below expected, and the reason is informative: the smallest buckets underperform. Models with 1–9 graded calls hit 49.02% across 8,606 rows; the 100–199 bucket hits 61.01%. Fresh instances start worse than the population they are compared against, so perfect records are not over-represented relative to chance — they are the surviving right tail of many short, slow-starting records.

Correct for multiplicity and the picture gets worse. With 3,229 candidates at α = 0.05 the per-test threshold is 1.55e-5, which requires n ≥ 20 with a clean sweep. We have 584 models at n ≥ 20 and none is perfect. No defensible 100% exists on this board.

Trying to fix the ranking function

The obvious next move is replacing the point estimate with something that penalises small samples — a Wilson lower bound, or a Beta posterior shrunk toward the population rate. Implement both, then re-run the same split-half evaluation at the same 66-model cut.

import math

P0 = 0.570716        # 26,415 hits / 46,284 graded rows
STRENGTH = 20.0      # pseudo-count mass pulling toward P0

def wilson_lb(k, n, z=1.96):
    if n == 0:
        return 0.0
    p = k / n
    denom  = 1 + z * z / n
    centre = p + z * z / (2 * n)
    margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (centre - margin) / denom

def shrunk(k, n, prior=P0, strength=STRENGTH):
    return (k + prior * strength) / (n + strength)

def top_slice(rows, key, k):
    """rows: dicts with h1, k1, n1, n, h2"""
    chosen = sorted(rows, key=key, reverse=True)[:k]
    return {"second_half": round(sum(r["h2"] for r in chosen) / len(chosen), 4),
            "mean_n":      round(sum(r["n"]  for r in chosen) / len(chosen), 1)}

Run over the same 661 pairs, k = 66:

The honest result: none of them fixes the forward rate. Every variant lands within 3.8 points of the population mean, and the raw sort is nominally best — not because raw sorting is good, but because the top decile of a positively-correlated noisy signal still concentrates some genuine signal, and shrinkage trades that sliver for conservatism.

The problem is not the ranking function. It is the object being ranked. With a median of six graded calls, no monotone transform of (k, n) can recover forward skill, because there is no forward skill in the estimator to recover.

What actually works

Never render a point estimate without its denominator and interval. 60.4% (n=62, 95% CI 47.9–71.7) is honest. 100.0% is a lie of omission when n=4. The interval is one line of Python; the bare number has no excuse.

Freeze the board as a historical artifact. A live query whose ordering reshuffles as rows arrive can be gamed by waiting. Materialise a weekly snapshot of (as_of, model_id, k, n, rank) and publish that. A model falling from rank 1 to rank 90 then becomes a visible fact instead of a silent overwrite.

Report the panel, not the total. The 171 models at n ≥ 50 average 59.66% with a standard deviation of 6.17 points. That spread is the real content: the credible population sits in a band roughly ±12 points wide at two sigma. A ranked list of 3,229 rows destroys that information; a distribution preserves it.

Set the threshold by what the estimate can support. At n = 50 the standard error of a 0.6 estimate is 0.069, so a 95% interval spans ~27 points. Nobody should compare two models across a 27-point interval. Either label the board as a discovery surface below fifty calls, or restrict ranking to the 171 models where the interval is narrower than the effect being claimed.

The generalisable lesson

Any system ranking entities by a ratio of two counters fails this way once the denominators vary by more than an order of magnitude — recommender A/B tests, creative scoring, per-shard error rates, per-endpoint latency percentiles, analyst dashboards. The tell is always the same: the top of the list is held by the smallest denominators, and the list collapses into ties at the boundary value (100%, 0%, perfect).

The diagnostic is cheap: split each entity's own history in half, correlate the halves, and check what the top decile delivers next. Near-zero correlation plus a twenty-point mean reversion means the ranking is measuring sample size, not the entity.

We still show the board. We now show it with n, with intervals, with a weekly frozen snapshot, and with a hard label that below fifty graded calls the ordering is not a comparison. The models did not get worse. The display stopped flattering the ones that had barely played.

Analytics and research content. Not betting advice. 18+

本内容仅供数据分析与研究参考，不构成任何决策建议。

46,160 graded predictions carry a boolean hit. The all-cohort rate is 57.05% (26,334 correct). The rate over models still in a live lifecycle state is 61.17% (19,281 of 31,520). Nothing about the models changed between those queries — only the WHERE clause did. A 4.12-point swing produced entirely by bookkeeping. In a system that retires models automatically, survivorship bias is not an edge case; it is the default output. What follows is the audit protocol we now run before any retirement decision.

Where a model actually goes when it dies

There is no single "deleted" state. A model leaves the visible cohort through six different doors, and the aggregate rate treats each differently.

Three things stand out. First, the worst cohorts are also the smallest. demoted holds 2,937 model rows but only 1,014 graded predictions — 0.35 per model. pending_retrain holds 56 models and 318 predictions at 28.30%. These are not measured failures; they are barely-measured ones. A rule that fires on 0.35 observations per model is not selecting on quality, it is selecting on noise.

Second, eliminated is a real signal. 801 models, 7,950 graded predictions, 45.11% — 11.98 points below the evolving cohort on a sample worth reading.

Third: 4,854 graded predictions reference 525 model_id values that no longer exist in ai_models at all. Their score, 56.61%, is close to the pool average, and any metric written as JOIN ai_models silently drops every one of them. We also keep an ai_models_archive table with 4,054 rows, while archived_at is NULL on all 8,789 live rows — the primary table cannot tell you when anything was retired.

The audit query: publish two numbers, never one

The protocol: publish two numbers from one query so they cannot drift apart — the headline, and its complement over the full historical cohort including rows whose model row is missing.

-- Cohort audit: the same graded rows, seen through two different filters.
WITH graded AS (
    SELECT p.id,
           p.model_id,
           p.hit,
           p.created_at,
           COALESCE(m.lifecycle_status, 'hard_deleted') AS lc
    FROM ai_predictions p
    LEFT JOIN ai_models m ON m.id = p.model_id
    WHERE p.settled = TRUE
      AND p.hit IS NOT NULL
)
SELECT 'all_cohort'      AS view,
       COUNT(*)          AS n,
       SUM(hit::int)     AS hits,
       ROUND(100.0 * SUM(hit::int) / COUNT(*), 2) AS hit_rate
FROM graded
UNION ALL
SELECT 'live_only', COUNT(*), SUM(hit::int),
       ROUND(100.0 * SUM(hit::int) / COUNT(*), 2)
FROM graded
WHERE lc IN ('evolving', 'active', 'incubating', 'validating')
UNION ALL
SELECT 'retired_cohort', COUNT(*), SUM(hit::int),
       ROUND(100.0 * SUM(hit::int) / COUNT(*), 2)
FROM graded
WHERE lc IN ('eliminated', 'demoted', 'archived');

     view      |   n   | hits  | hit_rate
---------------+-------+-------+----------
 all_cohort    | 46160 | 26334 |    57.05
 live_only     | 31520 | 19281 |    61.17
 retired_cohort|  8964 |  3931 |    43.85

The rule we now enforce: the delta between all_cohort and live_only is itself a monitored metric. Call it the retirement gap. When it grows, the pipeline is deleting measured failures faster than unmeasured models — flattering the published number without improving the system. Month by month:

The gap is not constant. It is smallest in the month when almost nothing had been retired and largest when a quarter of the pool was already gone. The full series runs between 50% and 62%, and part of the recent stability is the deletion of 23% of the evidence.

Why the survivors look better than they are

Hard deletion destroys the denominator. 525 model ids vanished from ai_models while their predictions stayed, so every JOIN-based metric silently excludes 4,854 rows — worse than a wrong number, because nothing in the output declares it.

Status filters are applied at read time, not write time. The same physical rows produce 57.05% or 61.17% depending on a WHERE clause chosen by a dashboard author, and no schema-level signal marks that the choice exists.

The leaderboard adds a minimum-sample filter on top. Of 4,681 models with any graded prediction, only 660 reach 20 and 170 reach 50; the median is 6. A "top models by hit rate" view is therefore almost entirely models whose whole record is under ten games, and sorting it descending returns 100% records: 388 with fewer than five graded predictions, 18 more between five and nine.

Here is the part that should end the argument. If every model were exchangeable at the pool rate of 0.5705, the expected number of perfect records is Σ p^n over the actual sample sizes:

models with n < 5   : expected 634.5 perfect records, observed 388
models with n >= 5  : expected  25.3 perfect records, observed  18

Observed is below the coin-flip expectation in both bands. A 4-for-4 record is not evidence of skill here — it is less impressive than chance would produce. Any rule that ranks on raw rate ranks a mixture of noise and sample size, and the models it rewards are the ones that guessed few times.

Make retirement append-only and auditable

The fix is not "stop filtering" — some views legitimately want the live cohort. The fix is to make the filter explicit, permanent and reversible, so excluded rows stay addressable. Retirement becomes an event, never a delete.

"""Retirement ledger: retirement is an event, never a delete."""
import hashlib, json, datetime as dt

def _h(prev: str, payload: dict) -> str:
    body = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256((prev + body).encode()).hexdigest()

def retire(cur, model_id: int, new_state: str, reason: str, evidence: dict):
    """Append one row. Never UPDATE ai_models.status, never DELETE."""
    cur.execute(
        "SELECT state, entry_hash FROM model_lifecycle_ledger "
        "WHERE model_id = %s ORDER BY id DESC LIMIT 1", (model_id,))
    row = cur.fetchone()
    prev_state, prev_hash = (row if row else ("born", "0" * 64))

    payload = {
        "model_id": model_id, "from_state": prev_state,
        "to_state": new_state, "reason": reason,     # 'eliminated_low_n'
        "evidence": evidence,                        # n, hits, rate, window
        "at": dt.datetime.utcnow().isoformat(timespec="seconds"),
    }
    cur.execute(
        "INSERT INTO model_lifecycle_ledger "
        "(model_id, from_state, to_state, reason, evidence, entry_hash, prev_hash) "
        "VALUES (%s,%s,%s,%s,%s,%s,%s)",
        (model_id, prev_state, new_state, reason,
         json.dumps(evidence), _h(prev_hash, payload), prev_hash))
    # projections may change; history may not
    cur.execute("UPDATE ai_models SET lifecycle_status = %s WHERE id = %s",
                (new_state, model_id))

With that ledger, the cohort view stops being a guess:

CREATE OR REPLACE VIEW v_model_cohort AS
SELECT m.id AS model_id,
       m.lifecycle_status AS current_state,
       COALESCE(f.first_retired_at, TIMESTAMPTZ 'infinity') AS retired_at,
       f.retirement_reason
FROM ai_models m
LEFT JOIN LATERAL (
    SELECT MIN(at) AS first_retired_at,
           (ARRAY_AGG(reason ORDER BY at))[1] AS retirement_reason
    FROM model_lifecycle_ledger l
    WHERE l.model_id = m.id
      AND l.to_state IN ('eliminated', 'demoted', 'archived')
) f ON TRUE;

Now "as of date D" is a temporal join instead of a status filter: a model retired in September still contributes its September predictions to every point-in-time metric, and the historical series can no longer be rewritten by a later decision. Three invariants belong in CI, each of which our bad state violated:

1. COUNT(*) FROM ai_predictions p WHERE NOT EXISTS (SELECT 1 FROM ai_models m WHERE m.id = p.model_id) must be zero. Ours was 4,854.
2. archived_at must be non-NULL for every row whose state is archived. Ours was 0 of 170.
3. Every state change needs a ledger row, and the chain must verify from genesis. Ours had 1,185 rows with reactivated_at set and no recorded reason.

What a retirement decision actually needs

Judge a retirement rule on the predictions it would have removed, not on the leaderboard it produces afterwards. Report four numbers together for every candidate cohort: n, hits, rate, and the rate of the cohort that survives. A rule removing 8,964 rows at 43.85% from a pool running at 57.05% raises the published number by 3.18 points while deleting 19.4% of the evidence. Whether that trade is defensible depends on whether the rule fired on measured failure — and with demoted averaging 0.35 graded predictions per model, ours mostly did not.

The honest summary of our pool: 63.6% of model rows (5,587 of 8,789) have never produced a single graded prediction, the median model has six, only 170 clear fifty, and 253 models were fathered by parents that were later eliminated. A system that kills a parent but keeps its descendants runs a selection loop it cannot audit, because the descendants inherit a training word whose measured performance was negative and nothing records the inheritance.

Retirement is a data operation, not a UI operation. If deleting a row changes the headline metric, that metric is a function of your bookkeeping rather than your model quality — and the cheapest way to improve it is always to delete more.

Analytics and research content. Not betting advice. 18+

Disclaimer: This content is provided for data analysis and research purposes only and does not constitute any decision-making advice.

We run a football analytics platform with 209 models and 46,233 settled predictions. Hunting for time leakage, I found it in three places, none a model bug — all three schema problems: tables physically able to store a value that could not have existed at decision time. The audit came down to one question per input table: what stops a row from being written after the event it describes? Mostly, nothing.

Lead time is the cheapest detector

The gap between row creation and the event timestamp is the first thing to plot. Join to fixtures, bucket the delta.

WITH lead AS (
  SELECT p.id, p.hit,
         EXTRACT(EPOCH FROM (n.match_time - p.created_at)) / 60.0 AS lead_min
  FROM ai_predictions p
  JOIN nami_matches n ON n.id = p.match_id
  WHERE p.settled
)
SELECT CASE WHEN lead_min < 0    THEN 'after kickoff'
            WHEN lead_min < 10   THEN '0-10 min'
            WHEN lead_min < 60   THEN '10-60 min'
            WHEN lead_min < 360  THEN '1-6 h'
            WHEN lead_min < 1440 THEN '6-24 h'
            ELSE 'over 24 h' END AS bucket,
       COUNT(*) AS n,
       ROUND(AVG(CASE WHEN hit THEN 1 ELSE 0 END) * 100, 2) AS hit_pct
FROM lead GROUP BY 1 ORDER BY 2 DESC;

Eighty-one rows were written after the fixture had started, settling at 69.14% against 56.9% for the rest. At n=81 that is a symptom, not a result: those rows come from a different code path, one close enough to the outcome to matter. The reading is not "late rows are better" but "we cannot tell what they saw, and should not have been able to write them".

Percentiles: p5 100.4 | p25 413.1 | p50 1,074.1 | p75 1,840.5 | p95 4,183 minutes. A 17.9-hour median is healthy; a p5 of 100 puts one row in twenty inside the window where the market is most informative — and most likely quoted from the wrong tick.

A change log with no as-of boundary is a time machine

The price store is a change log: 15,378,162 rows over 44,933 fixtures, 22.1 sources per fixture (median 19, max 87), median 140 ticks per fixture (p90 983). No is_prematch flag, no write guard — so the worker keeps appending after kickoff:

SELECT COUNT(*) FILTER (WHERE e.changed_at > n.match_time) AS post_kickoff_ticks,
       COUNT(*)                                            AS total_ticks,
       COUNT(DISTINCT e.match_id)
         FILTER (WHERE e.changed_at > n.match_time)        AS affected_fixtures
FROM nami_europe_odds e JOIN nami_matches n ON n.id = e.match_id;

post_kickoff_ticks | total_ticks | affected_fixtures
             62,922 |  15,378,162 |            15,259

0.41% of rows across 15,259 fixtures — a third of the store, small per fixture (mean 4.1 ticks, max 443) and short (median worst-case lag 2.4 minutes, max 1,439.6). The issue is that ORDER BY changed_at DESC LIMIT 1 — the pattern in every dashboard, prompt and backfill — silently returns an in-play value and cannot know it.

The consequence: of 44,625 fixtures with a pre-kickoff tick, the last lands a median 3.0 minutes before kickoff but the mean is 444.9; 13,540 (30.3%) have nothing newer than six hours, 2,201 (4.9%) nothing newer than 24. Bimodal — the feed either tracks to the whistle or stopped last evening, and no column records which.

The stored price is not the decision-time price

ai_predictions carries opening_eu_home/draw/away; the name promises the fixture's first quote. Two reference sets, tolerance 0.005, same 45,819 settled rows carrying a stored price:

A column called opening reconciles with a pre-kickoff price more often than with the opening. Backwards. So I restricted to the 15,487 rows whose fixture has a declared opening, giving both tests one denominator:

WITH declared AS (
  SELECT match_id, MAX(initial_eu_home) AS ini
  FROM nami_europe_odds WHERE initial_eu_home IS NOT NULL GROUP BY 1
), prekick AS (
  SELECT DISTINCT e.match_id, e.eu_home
  FROM nami_europe_odds e JOIN nami_matches n ON n.id = e.match_id
  WHERE e.changed_at <= n.match_time
    AND e.changed_at >  n.match_time - INTERVAL '3 hours'
)
SELECT COUNT(*) AS n,
       COUNT(*) FILTER (WHERE EXISTS (SELECT 1 FROM declared d
           WHERE d.match_id = p.match_id
             AND abs(d.ini - p.opening_eu_home) < 0.005)) AS hit_opening,
       COUNT(*) FILTER (WHERE EXISTS (SELECT 1 FROM prekick k
           WHERE k.match_id = p.match_id
             AND abs(k.eu_home - p.opening_eu_home) < 0.005)) AS hit_prekick
FROM ai_predictions p
WHERE p.settled AND p.opening_eu_home IS NOT NULL
  AND EXISTS (SELECT 1 FROM declared d WHERE d.match_id = p.match_id);

n = 15,487 | hit_opening 849 (5.5%) | hit_prekick 7,942 (51.3%)

The innocent explanation is that prices rarely move. Control: of the 13,882 fixtures holding both a declared opening and a pre-kickoff price, only 1,273 (9.2%) are unchanged. With 90.8% moving, 51.3% against the pre-kickoff price and 5.5% against the opening cannot both come from stability.

A placebo run on the 3,000 most recent rows — matching each stored price against the next fixture's pre-kickoff quote — returns 8.5%, the base rate of a 19-source grid; the same-fixture rate is 49.7%.

So opening_eu_home is filled at write time from whatever the feed currently says — usually the freshest quote. A timestamp-of-write price wearing a timestamp-of-event name:

Across 13,876 measured fixtures, the drift between those two ticks is a median 3.08 pp of implied probability, p90 10.93 pp, with 4,649 fixtures (33.5%) moving more than five points.

The reference column itself decays

Worse than a wrong name is a disappearing reference. Coverage of initial_eu_home by month:

Backfilled for two months, then abandoned — no migration, no alert. The store's semantics moved under its consumers: the most expensive kind of data debt, invisible to NOT NULL, and it turns every historical comparison into a comparison of different things.

Fixtures move too

match_time_audit exists because kickoff times are not stable: 42,378 fixtures, 141,111 changes, 13,500 of them carrying settled predictions. 2,056 predictions were created inside the window between the old and new kickoff — already running by the original schedule, not yet started by the revised one. Each has an unanswerable "was this pre-kickoff?" until the row records which copy of the schedule it used.

The fix: watermark, freeze, hash

1. A watermark, not a clock. Every price row carries the event time it describes (as_of) plus its fixture. Queries use as_of <= :decision_time, never changed_at. "Latest price" becomes "latest price as of a timestamp someone chose deliberately".

2. Freeze once, at kickoff, with a hash. One row per fixture, rejected at or after kickoff by the writer and by a constraint:

import hashlib, json
from datetime import datetime, timezone

def freeze(cur, match_id: int, as_of: datetime, prices: dict, source: str) -> str:
    """Insert the one and only frozen price for a fixture. Returns the hash."""
    cur.execute("SELECT match_time FROM nami_matches WHERE id = %s", (match_id,))
    kickoff = cur.fetchone()[0]
    if as_of >= kickoff:
        raise ValueError(f"{match_id}: as_of {as_of} is not before kickoff {kickoff}")

    body = {"match_id": match_id,
            "as_of": as_of.astimezone(timezone.utc).isoformat(),
            "source": source, "prices": prices}
    digest = hashlib.sha256(
        json.dumps(body, sort_keys=True, separators=(",", ":")).encode()).hexdigest()

    # the first freeze wins, forever
    cur.execute("""INSERT INTO frozen_prices (match_id, as_of, source, payload, sha256)
                   VALUES (%s, %s, %s, %s, %s)
                   ON CONFLICT (match_id) DO NOTHING""",
                (match_id, as_of, source, json.dumps(body, sort_keys=True), digest))

    cur.execute("SELECT sha256 FROM frozen_prices WHERE match_id = %s", (match_id,))
    if cur.fetchone()[0] != digest:
        raise RuntimeError(f"{match_id}: frozen row exists with a different payload")
    return digest

3. Reference the freeze from the prediction. Store frozen_price_id and kickoff_at instead of three copied numbers; "which price did the model see" becomes a foreign key that survives every backfill.

One table already gets this right: odds_snapshot_ledger, 15.4M rows over 20,065 fixtures, write lag median 6 s and p95 34 s against the bucket it records, and zero rows in the last seven days whose live-tick timestamp postdates kickoff (2,083,478 checked). Same vendor, same worker — the difference is the schema, not the ingest code.

Invariants worth testing in CI

1. Predictions with created_at >= kickoff_at = 0, and frozen prices with as_of >= kickoff_at = 0. Alert, not warn.
2. Every settled prediction resolves through frozen_price_id to exactly one row, and the hash verifies.
3. Stored price reconciles with at least one source quote at as_of — 100%, not 30%.
4. Reference-column coverage measured monthly, alerted on any drop over 10 pp. A column that silently stops being populated is worse than one never added.
5. Kickoff changes recorded before the window they invalidate; predictions carry the kickoff used.

None of these are modelling decisions. They are the arithmetic that makes a backtest mean anything — and all of them were failing on a system that produced 46,233 settled predictions without an alarm. Our 56.9% settled rate was computed against a price column that, in half the rows, holds a later price than the model saw. The model was not cheating; the schema left the door open and nobody checked who walked through it.

If you take one thing from this: data freshness is not a cron schedule, it is a column. A pipeline that refreshes every 60 seconds is still a time machine if the destination is not recorded on the row.

Analytics and research content. Not betting advice. 18+

本内容仅供数据分析与研究参考，不构成任何决策建议。

Every number below comes from the live production table, queried today, not estimated.

Two distributions that are required to match, and don't

The base table stores one row per model per match with the realised outcome attached. Two distributions matter: what happened, and what the model said would happen. If the second is a faithful compression of the first, they should be close.

-- D0: outcome distribution vs emitted-label distribution
WITH actual AS (
  SELECT actual_result AS cls, count(*) AS n
  FROM ai_predictions
  WHERE settled AND actual_result IS NOT NULL
  GROUP BY 1
),
emitted AS (
  SELECT prediction_direction AS cls, count(*) AS n
  FROM ai_predictions
  WHERE settled
  GROUP BY 1
)
SELECT 'actual' AS side, cls, n,
       round(100.0 * n / sum(n) OVER (), 2) AS pct
FROM actual
UNION ALL
SELECT 'emitted', cls, n,
       round(100.0 * n / sum(n) OVER (), 2)
FROM emitted
ORDER BY side, n DESC;

The draw happens 24.20% of the time and is named as the sole answer 0.78% of the time — a 31× under-representation. 16,925 rows name it as one of two outcomes; 27,647 rows chose strictly between home and away and could not be correct if the match finished level.

That last set is the interesting one. Of those 27,647 binary picks, 6,269 (22.68%) finished level. The ceiling was set by the label vocabulary, not the analysis.

Why the headline accuracy hides it

Reported accuracy by output arity on rows with a settled outcome:

58.52% on single-label rows looks like a margin over chance. But chance for a 44/31/24 class split under a majority-class strategy is 44.26% — and the "single" bucket is not homogeneous: 4,490 of those rows had a compound single label (home/draw, away/draw) and 27,647 were strictly two-class. Averaging them reports neither.

This is a binary classifier wearing a three-class schema: any binary classifier over football outcomes pays a fixed 22.68% tax on the subset it is applied to. A genuine three-class system moves that tax into calibrated probability on the draw.

The audit query that should be a unit test

Aggregate accuracy is what lets this bug live for 46,000 rows. Per-class precision and recall make it visible in one screen:

-- D1: per-class precision / recall on the collapsed label set
WITH base AS (
  SELECT actual_result, string_to_array(prediction_direction, '/') AS picked
  FROM ai_predictions
  WHERE settled AND actual_result IS NOT NULL
),
exploded AS (
  -- a compound label is a set; a single label is a set of one
  SELECT actual_result, unnest(picked) AS cls FROM base
),
per_class AS (
  SELECT cls,
         count(*)                                    AS emitted_n,
         count(*) FILTER (WHERE actual_result = cls) AS correct
  FROM exploded
  GROUP BY cls
),
totals AS (
  SELECT actual_result AS cls, count(*) AS actual_n FROM base GROUP BY 1
)
SELECT c.cls, c.emitted_n, c.correct,
       round(100.0 * c.correct / c.emitted_n, 2) AS precision_pct,
       round(100.0 * c.correct / t.actual_n,  2) AS recall_pct
FROM per_class c
JOIN totals t USING (cls)
ORDER BY c.emitted_n DESC;

Result on the 45,860 rows with a non-null outcome:

Draw precision of 26.81% hides the shape of the failure: draw alone hit 78 of 360 (21.67%), while away/draw hit 5,920 of 10,997 (53.83%) and home/draw hit 3,272 of 5,928 (55.20%). The compound labels are not predicting a draw — they express uncertainty between two outcomes, and the draw is an escape hatch. Recall on the draw is 41.16%: 6,576 draws were named by no label at all.

The most damning row: 283 distinct models emitted draw at least once across 46,189 predictions, and the busiest did it 4 times. No model here has learned to identify draws. Some occasionally guess.

Scoring both systems on one yardstick

Accuracy cannot compare a one-hot label with a distribution. Log loss can. The emitted label is scored as a probability vector (1.0 on the named class, split evenly for compounds) against the market-implied probability from the quoted three-way price, normalised to sum to one. 45,110 of 46,189 rows survive the price sanity filter.

import math

def implied(prices):
    inv = [1.0 / p for p in prices]
    s = sum(inv)
    return [x / s for x in inv]

LL_MARKET = LL_MODEL = BRIER_DRAW = 0.0
n = 0
for actual, ph, pd, pa, direction, arity in rows:
    mh, md, ma = implied((ph, pd, pa))
    LL_MARKET += -math.log(max({"home": mh, "draw": md, "away": ma}[actual], 1e-12))
    BRIER_DRAW += (md - (1.0 if actual == "draw" else 0.0)) ** 2

    if arity == "single":
        q_true = 1.0 if direction == actual else 0.0
    else:
        parts = direction.split("/")
        q_true = 1.0 / len(parts) if actual in parts else 0.0
    LL_MODEL += -math.log(max(q_true, 1e-12))
    n += 1

print(n, round(LL_MARKET / n, 4), round(LL_MODEL / n, 4), round(BRIER_DRAW / n, 4))
# 45110  0.9683  13.4625  0.1812

Log loss 0.9683 for the market-implied probabilities, 13.4625 for the emitted label — a factor of 13.9. That is what a zero looks like inside a logarithm: every row where the model picked a compound and the outcome was the excluded class contributes -log(0), clamped at 1e-12, or 27.6 nats.

This is not a claim that market probabilities are a good model. It is a claim about representation: a probability vector can be scored and improved; a one-hot label can only be counted, and counting rewards the majority class. The draw Brier of 0.1812 is what a real three-class output would have to beat.

What actually carries draw information

If confidence carried draw-risk signal, high-confidence binary picks would convert to draws less often. They don't. Draw rate among single picks restricted to home or away (27,647 rows, 6,269 draws):

Flat, and mildly inverted at the top: a model that says 9 is no less likely to be level than one that says 4. That is the expected failure when confidence is attached to a binary comparison and the third class was pruned before scoring — the pruning discards the evidence.

The market-implied draw probability, on the same 45,110 rows, is calibrated in the ordinary sense:

Mean implied 0.2423 against an observed 24.20%. The information is already in a column we store, and it is thrown away at the label step.

The imbalance is structural, not random

Draw rates are not uniform across competitions. On 78,553 finished matches the overall draw rate is 24.69% (19,392 draws), and competition buckets with at least 500 settled rows span 15.49% to 30.42%:

A single global threshold for "is this a draw?" is therefore wrong by about 15 percentage points. A per-class correction must be conditioned on the competition, and that requires a calibrated probability rather than a relabelled decision — you cannot attach a competition prior to a one-hot output.

Six rules that would have prevented this

1. Store the distribution, not the decision. A row should carry p_home, p_draw, p_away; the label is a rendering derived at read time. Ours stores prediction_direction and a 1–10 integer, so every metric above had to reconstruct intent from a string.
2. Never score a three-class problem by accuracy. Log loss and per-class recall, in the same query as the headline number.
3. Assert a per-class floor in CI. If the draw is emitted in under 10% of rows while the observed base rate is 24%, fail the build.
4. Measure compound labels separately. home/draw at 55.20% precision is not three-class capability; it inflates the "single" hit rate whenever the draw lands.
5. Condition on competition before claiming a threshold. A 15.49%–30.42% spread makes one global cut miscalibrated for every league it touches.
6. Score against market-implied probability. It is the only genuine distribution in the schema and it is already calibrated on our own rows — without it, a 0.78% draw rate looks like a modelling choice rather than a defect.

Two integrity facts

The 46,189 settled predictions cover 16,724 distinct matches, and only 3,736 (8.09%) join to matches on id; the rest carry a match_id from an older identifier space. Every per-competition number above was computed from matches directly — joining predictions to competitions would have dropped 92% of rows and produced a confident, wrong table.

Second: 329 settled rows have actual_result IS NULL and 121 have hit IS NULL while settled is true. They are excluded explicitly with FILTER.

The point

A 24% versus 0.8% gap is not a dataset problem. Twenty-four percent is an ordinary share for the third class of a football outcome. The imbalance was manufactured by the output format: the model emitted a label, the vocabulary had three slots, and the scoring loop only rewarded the slot that appeared most. Fixing it needs no better model — three numbers instead of one string, and a refusal to ship a three-class system whose per-class recall is never measured for one of its classes.

The draw was never hard to predict. It was hard to record.

Analytics and research content. Not betting advice. 18+

本内容仅供数据分析与研究参考，不构成任何决策建议。

The symptom

The dashboard, the model profile page, and the leaderboard each showed a different rate for the same model — one a confident 100% while the model's own detail page showed 43%. Both pages rendered cached columns: neither was wrong at the SQL level, both were wrong as statements about reality. Four writers each owned a slightly different definition of hit, denominator, and in-scope rows, and nobody had written down which was authoritative, so all four survived.

What the source of truth actually says

The only table that holds one row per prediction, with the outcome attached, is ai_predictions. Everything else is a derived cache. Query the base table, count settled rows, and count hits:

-- S0: the only honest denominator
SELECT
  count(*)                                                   AS settled_rows,
  count(*) FILTER (WHERE hit IS TRUE)                        AS hits,
  count(*) FILTER (WHERE hit IS NULL)                        AS unsettled_flag_nulls,
  round(100.0 * count(*) FILTER (WHERE hit IS TRUE) / count(*), 2) AS hit_rate_incl_nulls,
  round(100.0 * count(*) FILTER (WHERE hit IS TRUE)
        / nullif(count(*) FILTER (WHERE hit IS NOT NULL), 0), 2)  AS hit_rate_excl_nulls
FROM ai_predictions
WHERE settled = true;

Result:

settled_rows | hits  | unsettled_flag_nulls | hit_rate_incl_nulls | hit_rate_excl_nulls
     46,139  | 26,232|          121         |        56.85        |        57.01

121 rows are flagged settled = true but carry a NULL hit, which alone moves the headline by 0.16 points. And the same table holds a second hit column, hit_1x2, disagreeing with hit on 162 rows (26,070 vs 26,232). A third: exact directional agreement, prediction_direction = actual_result, yields 17,022 — 36.89%. Same rows, same settlement timestamp, three defensible definitions, three numbers twenty points apart.

Part of that gap is by design. The platform emits single-direction predictions (32,423 rows, 58.12% hit) and double-direction leans (13,687 rows, 53.85%). Exact-match scoring marks every double a miss unless one specific side lands; any-covered-side scoring credits either. A metric that does not declare its shape is not a metric.

The four pipelines

Read "rows in scope" before "rate". S1 and S2 are each missing ~5,100 settled predictions. S3 covers 802 rows — 1.74% of the corpus — and S5's "global" row covers 3,916 rows (8.49%). A metric describing 1.74% of the platform is a sample without a sample size.

The four gap sources, in order of impact:

1. Definition drift. S4 lands exactly on 26,070 — the hit_1x2 count, not the hit count of 26,232. The rollup and the base table never measured the same event, and it stayed invisible because both round to "about 57%".

2. Denominator drift. S1 and S2 were backfilled at different moments, so 5,129 and 5,188 settled rows never entered their counters. A counter incremented on write and never reconciled always lags by the rows that bypass the increment: batch settlement, retry, manual re-grade.

3. Coverage drift. S3 keeps the top 150 models; S5 keeps whatever the cron reached. Neither states coverage, so a consumer reads "59.48%" as a platform claim.

4. Survivorship. 2,708 of 8,392 model rows are retired and own 7,843 settled predictions — 17.0% of the corpus — and the leaderboard excludes them. Dropping losers before averaging is the fastest way to manufacture a better hit rate, and it happens by filtering, not lying.

What the leaderboard looks like from the base table

-- S3 vs. truth, per model
WITH truth AS (
  SELECT model_id, count(*) AS n, count(*) FILTER (WHERE hit IS TRUE) AS h
  FROM ai_predictions WHERE settled = true GROUP BY model_id
)
SELECT l.rank, m.name, l.total_predictions AS cached_n, l.hit_rate AS cached_rate,
       t.n AS true_n, round(100.0 * t.h / t.n, 2) AS true_rate
FROM leaderboard l JOIN ai_models m ON m.id = l.model_id
JOIN truth t ON t.model_id = l.model_id
ORDER BY l.rank LIMIT 10;

Top of the board:

rank | model_id | cached_n | cached_rate | true_n | true_rate
  1  |  25183   |    7     |   100.0     |   21   |   42.86
  2  |  25954   |   10     |    80.0     |   77   |   49.35
  4  |  25915   |    5     |   100.0     |   18   |   61.11
  6  |  25148   |    8     |    75.0     |   57   |   64.91

Rank 1 holds seven predictions in the cache and twenty-one in reality: displayed at 100%, scores 42.86%. Across all 132 leaderboard models that join to settled data, 131 have a cached total_predictions that disagrees with the base table, and the mean absolute deviation between cached and true rate is 17.08 percentage points. That is not rounding. It is a ranking replaced by a memory of a ranking.

Per-model divergence, measured

WITH truth AS (
  SELECT model_id, count(*) AS n, count(*) FILTER (WHERE hit IS TRUE) AS h
  FROM ai_predictions WHERE settled = true GROUP BY model_id
)
SELECT count(*)                                                             AS models_compared,
       count(*) FILTER (WHERE abs(100.0 * t.h / t.n - m.dir_hit_rate) > 0.5) AS off_by_half_point,
       round(avg(abs(100.0 * t.h / t.n - m.dir_hit_rate)), 2)               AS mean_abs_error_pp,
       round(max(abs(100.0 * t.h / t.n - m.dir_hit_rate)), 2)               AS max_abs_error_pp
FROM ai_models m JOIN truth t ON t.model_id = m.id
WHERE m.dir_total_predictions > 0 AND t.n >= 50;

models_compared | off_by_half_point | mean_abs_error_pp | max_abs_error_pp
      147       |        54          |       0.86        |      13.33

A third of models are off by more than half a point; the worst by 13.33. Against prediction_hit_stats it degrades further: of 111 comparable model rows, 68 are off by more than half a point and 89 carry a total_predictions that does not equal the true count — 80% wrong denominators, worst-case rate error 73.33 points.

Worst individual case, 美洲国家联赛·凌统: cached 60 predictions at 60.00%, actually 60 predictions at 73.33%. Same denominator, eight missing hits in the numerator — what a settlement path writing hit_1x2 while the counter reads hit looks like from outside.

The reconciliation job

The fix is not a migration: declare one source authoritative, make every other number a checked derivative.

import psycopg2

TRUTH_SQL = """
WITH truth AS (
  SELECT model_id, count(*) AS n, count(*) FILTER (WHERE hit) AS h
  FROM ai_predictions WHERE settled = true AND hit IS NOT NULL GROUP BY model_id
)
SELECT m.name, m.dir_total_predictions AS cached_n, m.dir_hit_rate AS cached_rate,
       t.n AS true_n, round(100.0 * t.h / t.n, 2) AS true_rate,
       abs(COALESCE(m.dir_hit_rate, 0) - round(100.0 * t.h / t.n, 2)) AS drift_pp
FROM ai_models m JOIN truth t ON t.model_id = m.id
WHERE m.dir_total_predictions > 0 AND t.n >= 50
ORDER BY drift_pp DESC;
"""

with psycopg2.connect(DSN) as conn, conn.cursor() as cur:
    cur.execute(TRUTH_SQL)
    rows = cur.fetchall()

hard = [r for r in rows if float(r[5]) > 1.0]
print(f"compared={len(rows)} drift>1.0pp={len(hard)} "
      f"coverage={sum(r[1] for r in rows) / sum(r[3] for r in rows):.2%}")

# gate: refuse to publish the leaderboard when the cache has drifted
assert not hard, f"{len(hard)} models exceed 1.0pp drift; rebuild blocked"

The invariant that makes this enforceable is not "the cache equals the base table" — it is monotonic agreement: for every model, dir_total_predictions may only grow, may never exceed the base-table count, and must reach equality after each settlement batch. A counter behind is lagging; a counter ahead is double-counting. Both are detectable, which is the whole point — before this invariant existed, neither was.

Governance rules that actually stopped the drift

One table owns the event. ai_predictions is the only place a hit is decided. Every downstream number must be reconstructible from it in one statement, by someone who has never read the application code. A metric that cannot be reconstructed is not a metric — it is an opinion with a percentage sign.

Every derived table declares its coverage. 57.00% over 3,916 rows is fine as long as the row says 3,916 and the UI renders "n=3,916". It is not fine as a headline. Coverage is part of the value, not metadata.

Every metric names its definition and arity policy. Scoring a double-direction lean under exact-match rules silently penalizes 29.7% of the corpus. hit_direction_exact and hit_direction_covered are two columns, not one column with two histories.

Retired models stay in the denominator. Report the all-model rate and the active-only rate side by side, always both, always labeled. Removing 7,843 settled predictions changes the story more than any model improvement shipped this quarter. And every cached rate renders its age: a 100% hit rate from seven predictions computed three weeks ago should look as fragile as it is.

What this cost, honestly

No prediction was wrong, no settlement was wrong. The damage was that four numbers existed — each individually traceable — and the platform was willing to show whichever one was convenient. That is more dangerous than a bug: a bug gets fixed, a comfortable number gets defended.

The state: 46,139 settled predictions, 26,232 directionally correct under hit — 56.85% inclusive of 121 NULL-flagged rows, 57.01% excluding them. 8,392 model rows: 2,231 active holding 31,247 of those predictions, 2,708 retired holding 7,843. Cached counters disagree with the base table on 131 of 132 leaderboard rows and 54 of 147 model rows above the 50-prediction threshold.

Those are the numbers this platform will defend. Every other number now carries a label saying where it came from.

Analytics and research content. Not betting advice. 18+

本内容仅供数据分析与研究参考，不构成任何决策建议。

That inference is where audits die. In our production table, confidence is an integer from 1 to 10 assigned at call time. It is not a probability, it never was, and the table below is the proof.

The dataset

The numbers come from one PostgreSQL table, ai_predictions, joined to matches for settlement fields. Nothing is simulated; every figure is recomputable below.

• 46,196 prediction rows across 16,732 distinct matches, created 2026-06-21 → 2026-11-09
• 45,722 rows are settled and carry a binary outcome (hit_1x2) plus a confidence
• 3,715 distinct model_id values produced them
• Predictions per match: mean 2.77, median 1, p90 7, max 358

That last line matters. The unit of analysis is the call, not the row, and one match with 358 rows will silently dominate any bucket it lands in.

What the score actually predicts

Here is the honest bucket table. n is row count, hit% is empirical accuracy on the binary outcome, and the interval is a 95% Wilson score interval — not a normal approximation, because at 57 rows the normal approximation is a work of fiction.

SELECT confidence,
       count(*)                                        AS n,
       sum(CASE WHEN hit_1x2 THEN 1 ELSE 0 END)        AS hits,
       round(100.0 * avg(CASE WHEN hit_1x2 THEN 1 ELSE 0 END)::numeric, 2) AS hit_pct
FROM ai_predictions
WHERE settled = true
  AND hit_1x2 IS NOT NULL
  AND confidence IS NOT NULL
GROUP BY confidence
ORDER BY confidence;

conf    n      hits   hit%    95% Wilson CI
 1     251     118    47.01    40.93 – 53.19
 2    1108     604    54.51    51.57 – 57.42
 3    8043    4327    53.80    52.71 – 54.89
 4    8750    5255    60.06    59.03 – 61.08
 5   11375    6305    55.43    54.51 – 56.34
 6    9260    5063    54.68    53.66 – 55.69
 7    4292    2575    60.00    58.52 – 61.45
 8    2037    1335    65.54    63.45 – 67.57
 9     548     404    73.72    69.88 – 77.23
10      57      33    57.89    44.98 – 69.81

There are three separate failures in it.

One: the middle of the scale is not monotonic. A call labelled 5 hits 55.43%; a call labelled 4 hits 60.06%, and the intervals do not overlap (59.03–61.08 vs 54.51–56.34). Claiming more confidence produced a worse result by 4.6 points across 20,125 rows, and the inversion recurs at 6 vs 5 (54.68% vs 55.43%). Whatever the model emits when it says "5", it is not a probability — it is a label that merely looks ordinal.

Two: the top of the scale collapses. Confidence 9 gives 73.72% on 548 rows; confidence 10 gives 57.89% on 57 rows. The loudest tier is both the worst and the thinnest — 0.12% of the data, with an interval of 44.98–69.81. The claim you can least verify is the claim published loudest.

Three: the aggregate hides all of it. Mean confidence is 4.94 (49.4%), overall accuracy 56.91%. So the system is under-confident by 7.5 points at the mean while its top tier is over-confident by 42. A single averaged "calibration" number reports a timid herd and hides a reckless tail.

Expected Calibration Error, and why it is not enough

The standard summary is ECE: bucket the predictions by stated probability, average the absolute gap between stated and observed, weight by bucket size.

WITH s AS (
  SELECT confidence::numeric / 10 AS p,
         CASE WHEN hit_1x2 THEN 1 ELSE 0 END::numeric AS y
  FROM ai_predictions
  WHERE settled = true AND hit_1x2 IS NOT NULL AND confidence IS NOT NULL
), b AS (
  SELECT width_bucket(p, 0, 1.0001, 10) AS k,
         count(*) AS n, avg(p) AS mp, avg(y) AS my
  FROM s GROUP BY 1
)
SELECT round(sum(n * abs(mp - my)) / sum(n), 4) AS ece,
       round(sum(n * my) / sum(n), 4)           AS overall_hit,
       sum(n)                                   AS n
FROM b;

Result: ece = 0.1333, overall_hit = 0.5691, n = 45722.

13.3 points of average miscalibration, a Brier score of 0.2689, and a mean signed bias of −0.0751. Now the baseline nobody computes: a constant forecast at the base rate, 0.5691, scores 0.5691 × 0.4309 = 0.2453. The raw score's 0.2689 is worse than predicting the same number every time. That is the whole argument in one line — a column that cannot beat a constant is not carrying probability information, it is carrying formatting.

Note also that ECE is weighted by bucket size, so 45% of it comes from confidence 5 and 6. The two tiers anyone would actually audit — 9 and 10 — contribute 1.3% of the weight. A metric weighted by frequency will always under-report the failures that matter most.

The fix we shipped, and the trap inside it

We added a fitted calibration layer. Each settled row now carries confidence_calibrated, a continuous probability. Re-running the same two queries against it:

• ECE: 0.0017 (down from 0.1333)
• Brier: 0.2439 (down from 0.2689)
• Mean bias: +0.0011 (down from −0.0751)
• Range across all 45,722 settled rows: 0.4741 to 0.7286, mean 0.5701

That last bullet is the one to stare at. A near-zero ECE looks like a triumph. What it actually means is that the calibrated map refuses to output anything outside a 25-point band, and then the data obliges — because the map was fitted on the data it is being scored on. An ECE of 0.0017 in-sample is a statement about the fit, not the skill. The honest reading is the range: this system's real probability band is 47% to 73%, and nothing it says should be read as an 85% claim. Even the improved Brier, 0.2439, is only 0.6% better than the constant-forecast baseline of 0.2453 — all the gain is in the level, none of it in the ordering.

If you take one engineering lesson from this post: fitting a calibration layer always makes your calibration metric look good. What survives is the width of the output band plus an out-of-sample check. We report the band, not the ECE.

Two contamination sources that quietly inflate everything

Two-way calls. 32,147 settled rows are single-direction calls and hit 58.35%; 13,546 are "either of two outcomes" calls and hit 53.50%. Counter-intuitively the easier category scores worse — conditioning on two of three outcomes still loses ground. Any hit-rate table mixing the two arities measures the mix, not the skill.

Per-match multiplicity. With a mean of 2.77 predictions per match and a maximum of 358, one fixture can contribute dozens of correlated rows. A confidence-9 bucket can read "548 rows" while holding far fewer independent calls. Deduplicate to one row per match before computing any interval, or your Wilson bounds are narrower than reality — the one direction of error an audit must never make.

Per-model, the audit is brutal: of 3,715 model ids, only 7 have 200+ settled rows, and the worst of those claims 63.6% average confidence over 306 rows while delivering 55.23% — an 8.4-point overstatement no leaderboard sorted by raw hit rate would surface.

What an automated audit actually looks like

Calibration decays quietly. A single check is worthless; the useful artifact is a function that runs daily and fails loudly on inversion. This is the shape we use — the interesting part is not the ECE calculation, it is the INVERSION flag and the minimum-sample guard:

import math
from collections import defaultdict

MIN_N, INVERSION_GAP = 30, 0.04   # no claim below 30 rows; 4 pts back = failure

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p, d = k / n, 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (c - h, c + h)

def audit(rows):
    """rows: (match_id, model_id, stated_p, outcome_0_1).
    Buckets hold [n, hits, sum_p] so ECE uses the real mean, not the midpoint --
    midpoints hide exactly the skew we are hunting."""
    seen, buckets = set(), defaultdict(lambda: [0, 0, 0.0])
    for mid, _model, p, y in rows:
        if mid in seen:               # one independent observation per match
            continue
        seen.add(mid)
        b = buckets[round(p * 10)]
        b[0] += 1
        b[1] += int(y)
        b[2] += p

    total = sum(b[0] for b in buckets.values())
    ece = sum(b[0] * abs(b[2] / b[0] - b[1] / b[0]) for b in buckets.values())
    report = {"n_independent": len(seen), "buckets": {}, "inversions": [],
              "ece": round(ece / total, 4) if total else None}

    prev = None
    for decile, (n, k, _sp) in sorted(buckets.items()):
        if n < MIN_N:
            report["buckets"][decile] = {"n": n, "skipped": "n < MIN_N"}
            continue                  # no claim, not a 50% guess
        hit = k / n
        lo, hi = wilson(k, n)
        report["buckets"][decile] = {"n": n, "hit": round(hit, 4),
                                     "ci95": [round(lo, 4), round(hi, 4)]}
        if prev and prev[1] - hit > INVERSION_GAP:
            report["inversions"].append((prev[0], decile))
        prev = (decile, hit)

    if report["inversions"]:
        report["status"] = "FAIL: non-monotonic confidence"
    return report

Two design choices there matter. MIN_N = 30 with an explicit skipped marker means a bucket that cannot support a claim is reported as having no claim, not as a 50% guess. And inversion detection is a hard failure, not a warning: monotonicity is the minimum contract between a model and everyone reading its output.

The checklist

1. Never publish an integer confidence as a probability. Store it as a label; derive a probability in a separate, versioned column with a fit date in its name.
2. Bucket-then-diff, never average-then-report. Under-confident at the mean and 42 points over-confident at the top is not "slightly miscalibrated".
3. Weight the audit by cost, not frequency. ECE's weighting makes the tiers that matter invisible — report the tail separately, always.
4. Deduplicate to one observation per match, and state the deduplication in the report.
5. Guard thin buckets explicitly. 57 rows is not evidence; n=57 with a 45–70% interval is a press release, not a measurement.

None of this requires new modelling — just admitting that the number the model hands you is a label needing a schema around it, the same conclusion we reached about immutability and leaderboards, from a third direction. Every number that leaves a prediction system needs a contract, and confidence is the one everyone forgets.

The table above is reproducible from the queries in this post against the public dataset snapshot; where they disagree, the dataset is the authority.

Analytics and research content. Not betting advice. 18+. 本内容仅供数据分析与研究参考，不构成任何决策建议。

That definition breaks the moment the prediction is a business artifact — a number someone acts on, a number you publish, a number you will later claim was right. Under that definition, reproducibility has nothing to do with seeds. It has everything to do with whether the row you are looking at today is the row that existed when the call was made.

I run a football analytics platform with a few thousand model variants writing predictions into one Postgres table. We measured our own table last week. The result was uncomfortable enough that I want to publish it.

The measurement

The table is called ai_predictions. At the time of writing it holds 46,192 rows, produced by 3,712 distinct model variants across 16,731 fixtures, spanning 2026-06-21 to 2026-11-09. 46,041 of thMost teams treat "reproducibility" as a machine learning problem. Fix the seed, pin the library versions, snapshot the model weights, and you are done.

That definition breaks the moment the prediction is a business artifact — a number someone acts on, a number you publish, a number you will later claim was right. Under that definition, reproducibility has nothing to do with seeds. It has everything to do with whether the row you are looking at today is the row that existed when the call was made.

I run a football analytics platform with a few thousand model variants writing predictions into one Postgres table. We measured our own table last week. The result was uncomfortable enough that I want to publish it.

The measurement

The table is called ai_predictions. At the time of writing it holds 46,192 rows, produced by 3,712 distinct model variants across 16,731 fixtures, spanning 2026-06-21 to 2026-11-09. 46,041 of those rows are settled — meaning the fixture finished and the outcome is known.

We have a primary key on id. That is the only uniqueness constraint on the table. There is no unique constraint on (model_id, fixture_id).

Let me measure three things: whether a row is rewritten after creation, whether it is rewritten after settlement, and whether we are able to tell what changed.

-- 1. rows modified more than 60 seconds after creation
SELECT count(*) FROM ai_predictions
WHERE updated_at > created_at + interval '60 seconds';

-- 2. rows modified AFTER the outcome was already graded
SELECT count(*) FROM ai_predictions
WHERE settled = true
  AND settled_at IS NOT NULL
  AND updated_at > settled_at;

-- 3. is there any table holding a before-image?
SELECT count(*) FROM information_schema.tables
WHERE table_name LIKE '%prediction%histor%'
   OR table_name LIKE '%prediction%audit%';

The answers:

46181   -- rows rewritten >60s after creation (99.98% of the table)
35013   -- rows rewritten AFTER settlement (75.8% of settled rows)
0       -- audit tables holding a before-image

Read the second number again. Three quarters of our settled predictions have been modified since they were graded.

The two interpretations, and why you cannot choose between them

There is a benign reading and a damning reading of 35013.

The benign reading: prediction rows carry a lot of envelope fields besides the call itself — confidence calibration, market-snapshot injection timestamps, secondary-market settlement, dictionary status. A row can be touched dozens of times after grading while direction and confidence never move. In our case this is almost certainly what happened: those patch columns are written by separate jobs that run on their own schedules.

The damning reading: someone changed the call after seeing the result.

Here is the problem. With no before-image, we cannot distinguish these two cases. Not "we believe it's the benign one" — we literally cannot produce the evidence. The updated_at column tells you a write happened. It does not tell you which columns changed, from what, or by whom.

That is what a missing audit trail actually costs. It is not that you have no audit trail. It is that you have no way to prove you did nothing wrong, on any row, ever. Every number you publish rests on an assertion nobody can verify.

And note the first number: 99.98%. Almost every row is mutable. That is not a discipline problem, it is a schema problem — the table was designed to permit it.

Mutation is a design decision, not an accident

Postgres will happily let you UPDATE any column you did not explicitly protect. If your schema is:

CREATE TABLE ai_predictions (
    id          bigserial PRIMARY KEY,
    model_id    bigint NOT NULL,
    fixture_id  bigint NOT NULL,
    direction   text,
    confidence  numeric(5,4),
    created_at  timestamptz DEFAULT now(),
    updated_at  timestamptz DEFAULT now()
);

then "predictions are immutable" is a convention, and conventions do not survive a deadline. The table's permissions are the real specification. Everything else is documentation.

There is a cheap structural fix. Make the graded fields generated or trigger-guarded, so the database itself rejects the write:

-- a write that changes a settled call is rejected by the database,
-- not by a code review
CREATE OR REPLACE FUNCTION reject_settled_mutation() RETURNS trigger AS $$
BEGIN
    IF OLD.settled = true THEN
        IF NEW.p_home  <> OLD.p_home
        OR NEW.p_draw  <> OLD.p_draw
        OR NEW.p_away  <> OLD.p_away
        OR NEW.direction <> OLD.direction THEN
            RAISE EXCEPTION
              'settled prediction % is immutable (attempted change to graded fields)', OLD.id;
        END IF;
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER ai_predictions_immutable
BEFORE UPDATE ON ai_predictions
FOR EACH ROW EXECUTE FUNCTION reject_settled_mutation();

Envelope columns still move. The part that constitutes a claim cannot. That single trigger converts "we promise" into "the database refuses" — which is the only version of a promise that a third party can check.

If you must mutate, mutate with a diff

The trigger above is a hard stop. Sometimes you genuinely need a legal path for corrections — a data correction after a bad fixture mapping, say. Then the answer is not to allow silent mutation; it is to require a companion write that records the transition.

CREATE TABLE prediction_mutations (
    id            bigserial PRIMARY KEY,
    prediction_id bigint NOT NULL REFERENCES ai_predictions(id),
    changed_at    timestamptz NOT NULL DEFAULT now(),
    changed_by    text NOT NULL,          -- job name, never "human"
    reason        text NOT NULL,          -- enum-checked in practice
    before_row    jsonb NOT NULL,
    after_row     jsonb NOT NULL,
    CHECK (before_row <> after_row)
);

CREATE OR REPLACE FUNCTION audit_prediction_update() RETURNS trigger AS $$
BEGIN
    INSERT INTO prediction_mutations (prediction_id, changed_by, reason, before_row, after_row)
    VALUES (NEW.id,
            current_setting('app.job_name', true),
            COALESCE(current_setting('app.change_reason', true), 'unspecified'),
            to_jsonb(OLD),
            to_jsonb(NEW));
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER ai_predictions_audit
AFTER UPDATE ON ai_predictions
FOR EACH ROW EXECUTE FUNCTION audit_prediction_update();

Now the benign reading is provable. When someone asks why row 8,412,003 changed after settlement, you open prediction_mutations, look at the two jsonb blobs, and show that exactly one key moved and it was ah_settled_at. Or you show that direction moved, and you have a much harder conversation — but at least it is a conversation grounded in evidence.

reason being NOT NULL with a changed_by that names a job rather than a person is deliberate. "Someone fixed it" is not an audit entry. "calibration_worker updated confidence because CALIBRATION_V2 backfill ran" is.

Reproducibility, restated

Once the record is immutable and diffs are recorded, an actual reproducibility check becomes possible. Not "run it twice and eyeball the output" — a stored, replayable assertion:

def verify_reproducible(conn, prediction_id, tolerance=1e-6):
    """Re-derive the stored prediction from its recorded inputs.

    Asserts that: the row is unchanged since grading, the inputs it
    was computed from are still the inputs on record, and recomputing
    from those inputs lands on the stored numbers.
    """
    row, mutations = conn.execute("""
        SELECT p.*, count(m.id) AS mutation_count
        FROM ai_predictions p
        LEFT JOIN prediction_mutations m ON m.prediction_id = p.id
        WHERE p.id = %s
        GROUP BY p.id
    """, (prediction_id,)).fetchone()

    graded_mutations = conn.execute("""
        SELECT count(*) FROM prediction_mutations
        WHERE prediction_id = %s AND after_row ? 'direction'
          AND (before_row->>'direction') IS DISTINCT FROM (after_row->>'direction')
    """, (prediction_id,)).fetchone()[0]

    if graded_mutations:
        return False, "graded field changed post-settlement (%d times)" % graded_mutations

    recomputed = rederive(row["model_id"], row["fixture_id"], row["price_snapshot_id"])
    for k in ("p_home", "p_draw", "p_away"):
        if abs(float(recomputed[k]) - float(row[k])) > tolerance:
            return False, "recompute drifted on %s" % k
    return True, "reproduced"

The interesting failure mode is the third one, and it is the one nobody instruments: rederive returns a different number because the inputs changed underneath the row — the price snapshot was refreshed in place, the fixture was remapped, the model's prompt template was edited. A prediction can be perfectly immutable and still not reproducible, because its inputs were not. Which is why the snapshot has to be frozen before kickoff and referenced by id, not by timestamp.

That is the whole argument compressed: immutability is about the row, reproducibility is about the row plus its inputs, and neither is a property of your code — both are properties of your schema.

What we are changing, in this order

Being precise about status, since the whole post is about not overstating things — at the time of writing the numbers above, the fix is specified and not yet deployed. The plan, in dependency order:

1. prediction_mutations companion table first, with the audit trigger. This is additive and cannot break a writer, so it goes out before anything that can reject a write.
2. Backfill nothing. There is no before-image for the existing 35,013 rows, and inventing one would be worse than admitting the gap. The audit trail starts at the timestamp it starts.
3. (model_id, fixture_id) uniqueness, added NOT VALID first, validated after a duplicate scan — duplicates are sitting at zero right now by luck, not by constraint.
4. Only then the BEFORE UPDATE hard stop on graded fields, after auditing which writers actually touch them. A trigger that rejects a write some nightly job depends on is an outage with extra steps.
5. Grading moved into a single SECURITY DEFINER function that refuses to run twice for the same row.

Rough effort: about a day of work, spread across four deploys specifically so that no single step can take the writers down. The expensive part is not the SQL. It is admitting that a table with 46,192 rows and zero constraints was never immutable — it was merely unchallenged.

If you take one thing from this: go and run the three queries at the top against your own predictions table. The number you get for "rows modified after settlement" is the size of the trust gap you are currently asking your users to close on faith.

Analytics and research content. Not betting advice. 18+Most teams treat "reproducibility" as a machine learning problem. Fix the seed, pin the library versions, snapshot the model weights, and you are done.

That definition breaks the moment the prediction is a business artifact — a number someone acts on, a number you publish, a number you will later claim was right. Under that definition, reproducibility has nothing to do with seeds. It has everything to do with whether the row you are looking at today is the row that existed when the call was made.

I run a football analytics platform with a few thousand model variants writing predictions into one Postgres table. We measured our own table last week. The result was uncomfortable enough that I want to publish it.

The measurement

The table is called ai_predictions. At the time of writing it holds 46,192 rows, produced by 3,712 distinct model variants across 16,731 fixtures, spanning 2026-06-21 to 2026-11-09. 46,041 of those rows are settled — meaning the fixture finished and the outcome is known.

We have a primary key on id. That is the only uniqueness constraint on the table. There is no unique constraint on (model_id, fixture_id).

Let me measure three things: whether a row is rewritten after creation, whether it is rewritten after settlement, and whether we are able to tell what changed.

-- 1. rows modified more than 60 seconds after creation
SELECT count(*) FROM ai_predictions
WHERE updated_at > created_at + interval '60 seconds';

-- 2. rows modified AFTER the outcome was already graded
SELECT count(*) FROM ai_predictions
WHERE settled = true
  AND settled_at IS NOT NULL
  AND updated_at > settled_at;

-- 3. is there any table holding a before-image?
SELECT count(*) FROM information_schema.tables
WHERE table_name LIKE '%prediction%histor%'
   OR table_name LIKE '%prediction%audit%';

The answers:

46181   -- rows rewritten >60s after creation (99.98% of the table)
35013   -- rows rewritten AFTER settlement (75.8% of settled rows)
0       -- audit tables holding a before-image

Read the second number again. Three quarters of our settled predictions have been modified since they were graded.

The two interpretations, and why you cannot choose between them

There is a benign reading and a damning reading of 35013.

The benign reading: prediction rows carry a lot of envelope fields besides the call itself — confidence calibration, market-snapshot injection timestamps, secondary-market settlement, dictionary status. A row can be touched dozens of times after grading while direction and confidence never move. In our case this is almost certainly what happened: those patch columns are written by separate jobs that run on their own schedules.

The damning reading: someone changed the call after seeing the result.

Here is the problem. With no before-image, we cannot distinguish these two cases. Not "we believe it's the benign one" — we literally cannot produce the evidence. The updated_at column tells you a write happened. It does not tell you which columns changed, from what, or by whom.

That is what a missing audit trail actually costs. It is not that you have no audit trail. It is that you have no way to prove you did nothing wrong, on any row, ever. Every number you publish rests on an assertion nobody can verify.

And note the first number: 99.98%. Almost every row is mutable. That is not a discipline problem, it is a schema problem — the table was designed to permit it.

Mutation is a design decision, not an accident

Postgres will happily let you UPDATE any column you did not explicitly protect. If your schema is:

CREATE TABLE ai_predictions (
    id          bigserial PRIMARY KEY,
    model_id    bigint NOT NULL,
    fixture_id  bigint NOT NULL,
    direction   text,
    confidence  numeric(5,4),
    created_at  timestamptz DEFAULT now(),
    updated_at  timestamptz DEFAULT now()
);

then "predictions are immutable" is a convention, and conventions do not survive a deadline. The table's permissions are the real specification. Everything else is documentation.

There is a cheap structural fix. Make the graded fields generated or trigger-guarded, so the database itself rejects the write:

-- a write that changes a settled call is rejected by the database,
-- not by a code review
CREATE OR REPLACE FUNCTION reject_settled_mutation() RETURNS trigger AS $$
BEGIN
    IF OLD.settled = true THEN
        IF NEW.p_home  <> OLD.p_home
        OR NEW.p_draw  <> OLD.p_draw
        OR NEW.p_away  <> OLD.p_away
        OR NEW.direction <> OLD.direction THEN
            RAISE EXCEPTION
              'settled prediction % is immutable (attempted change to graded fields)', OLD.id;
        END IF;
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER ai_predictions_immutable
BEFORE UPDATE ON ai_predictions
FOR EACH ROW EXECUTE FUNCTION reject_settled_mutation();

Envelope columns still move. The part that constitutes a claim cannot. That single trigger converts "we promise" into "the database refuses" — which is the only version of a promise that a third party can check.

If you must mutate, mutate with a diff

The trigger above is a hard stop. Sometimes you genuinely need a legal path for corrections — a data correction after a bad fixture mapping, say. Then the answer is not to allow silent mutation; it is to require a companion write that records the transition.

CREATE TABLE prediction_mutations (
    id            bigserial PRIMARY KEY,
    prediction_id bigint NOT NULL REFERENCES ai_predictions(id),
    changed_at    timestamptz NOT NULL DEFAULT now(),
    changed_by    text NOT NULL,          -- job name, never "human"
    reason        text NOT NULL,          -- enum-checked in practice
    before_row    jsonb NOT NULL,
    after_row     jsonb NOT NULL,
    CHECK (before_row <> after_row)
);

CREATE OR REPLACE FUNCTION audit_prediction_update() RETURNS trigger AS $$
BEGIN
    INSERT INTO prediction_mutations (prediction_id, changed_by, reason, before_row, after_row)
    VALUES (NEW.id,
            current_setting('app.job_name', true),
            COALESCE(current_setting('app.change_reason', true), 'unspecified'),
            to_jsonb(OLD),
            to_jsonb(NEW));
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER ai_predictions_audit
AFTER UPDATE ON ai_predictions
FOR EACH ROW EXECUTE FUNCTION audit_prediction_update();

Now the benign reading is provable. When someone asks why row 8,412,003 changed after settlement, you open prediction_mutations, look at the two jsonb blobs, and show that exactly one key moved and it was ah_settled_at. Or you show that direction moved, and you have a much harder conversation — but at least it is a conversation grounded in evidence.

reason being NOT NULL with a changed_by that names a job rather than a person is deliberate. "Someone fixed it" is not an audit entry. "calibration_worker updated confidence because CALIBRATION_V2 backfill ran" is.

Reproducibility, restated

Once the record is immutable and diffs are recorded, an actual reproducibility check becomes possible. Not "run it twice and eyeball the output" — a stored, replayable assertion:

def verify_reproducible(conn, prediction_id, tolerance=1e-6):
    """Re-derive the stored prediction from its recorded inputs.

    Asserts that: the row is unchanged since grading, the inputs it
    was computed from are still the inputs on record, and recomputing
    from those inputs lands on the stored numbers.
    """
    row, mutations = conn.execute("""
        SELECT p.*, count(m.id) AS mutation_count
        FROM ai_predictions p
        LEFT JOIN prediction_mutations m ON m.prediction_id = p.id
        WHERE p.id = %s
        GROUP BY p.id
    """, (prediction_id,)).fetchone()

    graded_mutations = conn.execute("""
        SELECT count(*) FROM prediction_mutations
        WHERE prediction_id = %s AND after_row ? 'direction'
          AND (before_row->>'direction') IS DISTINCT FROM (after_row->>'direction')
    """, (prediction_id,)).fetchone()[0]

    if graded_mutations:
        return False, "graded field changed post-settlement (%d times)" % graded_mutations

    recomputed = rederive(row["model_id"], row["fixture_id"], row["price_snapshot_id"])
    for k in ("p_home", "p_draw", "p_away"):
        if abs(float(recomputed[k]) - float(row[k])) > tolerance:
            return False, "recompute drifted on %s" % k
    return True, "reproduced"

The interesting failure mode is the third one, and it is the one nobody instruments: rederive returns a different number because the inputs changed underneath the row — the price snapshot was refreshed in place, the fixture was remapped, the model's prompt template was edited. A prediction can be perfectly immutable and still not reproducible, because its inputs were not. Which is why the snapshot has to be frozen before kickoff and referenced by id, not by timestamp.

That is the whole argument compressed: immutability is about the row, reproducibility is about the row plus its inputs, and neither is a property of your code — both are properties of your schema.

What we are changing, in this order

Being precise about status, since the whole post is about not overstating things — at the time of writing the numbers above, the fix is specified and not yet deployed. The plan, in dependency order:

1. prediction_mutations companion table first, with the audit trigger. This is additive and cannot break a writer, so it goes out before anything that can reject a write.
2. Backfill nothing. There is no before-image for the existing 35,013 rows, and inventing one would be worse than admitting the gap. The audit trail starts at the timestamp it starts.
3. (model_id, fixture_id) uniqueness, added NOT VALID first, validated after a duplicate scan — duplicates are sitting at zero right now by luck, not by constraint.
4. Only then the BEFORE UPDATE hard stop on graded fields, after auditing which writers actually touch them. A trigger that rejects a write some nightly job depends on is an outage with extra steps.
5. Grading moved into a single SECURITY DEFINER function that refuses to run twice for the same row.

Rough effort: about a day of work, spread across four deploys specifically so that no single step can take the writers down. The expensive part is not the SQL. It is admitting that a table with 46,192 rows and zero constraints was never immutable — it was merely unchallenged.

If you take one thing from this: go and run the three queries at the top against your own predictions table. The number you get for "rows modified after settlement" is the size of the trust gap you are currently asking your users to close on faith.

Analytics and research content. Not betting advice. 18+ose rows are settled — meaning the fixture finished and the outcome is known.

We have a primary key on id. That is the only uniqueness constraint on the table. There is no unique constraint on (model_id, fixture_id).

Let me measure three things: whether a row is rewritten after creation, whether it is rewritten after settlement, and whether we are able to tell what changed.

-- 1. rows modified more than 60 seconds after creation
SELECT count(*) FROM ai_predictions
WHERE updated_at > created_at + interval '60 seconds';

-- 2. rows modified AFTER the outcome was already graded
SELECT count(*) FROM ai_predictions
WHERE settled = true
  AND settled_at IS NOT NULL
  AND updated_at > settled_at;

-- 3. is there any table holding a before-image?
SELECT count(*) FROM information_schema.tables
WHERE table_name LIKE '%prediction%histor%'
   OR table_name LIKE '%prediction%audit%';

The answers:

46181   -- rows rewritten >60s after creation (99.98% of the table)
35013   -- rows rewritten AFTER settlement (75.8% of settled rows)
0       -- audit tables holding a before-image

Read the second number again. Three quarters of our settled predictions have been modified since they were graded.

The two interpretations, and why you cannot choose between them

There is a benign reading and a damning reading of 35013.

The benign reading: prediction rows carry a lot of envelope fields besides the call itself — confidence calibration, market-snapshot injection timestamps, secondary-market settlement, dictionary status. A row can be touched dozens of times after grading while direction and confidence never move. In our case this is almost certainly what happened: those patch columns are written by separate jobs that run on their own schedules.

The damning reading: someone changed the call after seeing the result.

Here is the problem. With no before-image, we cannot distinguish these two cases. Not "we believe it's the benign one" — we literally cannot produce the evidence. The updated_at column tells you a write happened. It does not tell you which columns changed, from what, or by whom.

That is what a missing audit trail actually costs. It is not that you have no audit trail. It is that you have no way to prove you did nothing wrong, on any row, ever. Every number you publish rests on an assertion nobody can verify.

And note the first number: 99.98%. Almost every row is mutable. That is not a discipline problem, it is a schema problem — the table was designed to permit it.

Mutation is a design decision, not an accident

Postgres will happily let you UPDATE any column you did not explicitly protect. If your schema is:

CREATE TABLE ai_predictions (
    id          bigserial PRIMARY KEY,
    model_id    bigint NOT NULL,
    fixture_id  bigint NOT NULL,
    direction   text,
    confidence  numeric(5,4),
    created_at  timestamptz DEFAULT now(),
    updated_at  timestamptz DEFAULT now()
);

then "predictions are immutable" is a convention, and conventions do not survive a deadline. The table's permissions are the real specification. Everything else is documentation.

There is a cheap structural fix. Make the graded fields generated or trigger-guarded, so the database itself rejects the write:

-- a write that changes a settled call is rejected by the database,
-- not by a code review
CREATE OR REPLACE FUNCTION reject_settled_mutation() RETURNS trigger AS $$
BEGIN
    IF OLD.settled = true THEN
        IF NEW.p_home  <> OLD.p_home
        OR NEW.p_draw  <> OLD.p_draw
        OR NEW.p_away  <> OLD.p_away
        OR NEW.direction <> OLD.direction THEN
            RAISE EXCEPTION
              'settled prediction % is immutable (attempted change to graded fields)', OLD.id;
        END IF;
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER ai_predictions_immutable
BEFORE UPDATE ON ai_predictions
FOR EACH ROW EXECUTE FUNCTION reject_settled_mutation();

Envelope columns still move. The part that constitutes a claim cannot. That single trigger converts "we promise" into "the database refuses" — which is the only version of a promise that a third party can check.

If you must mutate, mutate with a diff

The trigger above is a hard stop. Sometimes you genuinely need a legal path for corrections — a data correction after a bad fixture mapping, say. Then the answer is not to allow silent mutation; it is to require a companion write that records the transition.

CREATE TABLE prediction_mutations (
    id            bigserial PRIMARY KEY,
    prediction_id bigint NOT NULL REFERENCES ai_predictions(id),
    changed_at    timestamptz NOT NULL DEFAULT now(),
    changed_by    text NOT NULL,          -- job name, never "human"
    reason        text NOT NULL,          -- enum-checked in practice
    before_row    jsonb NOT NULL,
    after_row     jsonb NOT NULL,
    CHECK (before_row <> after_row)
);

CREATE OR REPLACE FUNCTION audit_prediction_update() RETURNS trigger AS $$
BEGIN
    INSERT INTO prediction_mutations (prediction_id, changed_by, reason, before_row, after_row)
    VALUES (NEW.id,
            current_setting('app.job_name', true),
            COALESCE(current_setting('app.change_reason', true), 'unspecified'),
            to_jsonb(OLD),
            to_jsonb(NEW));
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER ai_predictions_audit
AFTER UPDATE ON ai_predictions
FOR EACH ROW EXECUTE FUNCTION audit_prediction_update();

Now the benign reading is provable. When someone asks why row 8,412,003 changed after settlement, you open prediction_mutations, look at the two jsonb blobs, and show that exactly one key moved and it was ah_settled_at. Or you show that direction moved, and you have a much harder conversation — but at least it is a conversation grounded in evidence.

reason being NOT NULL with a changed_by that names a job rather than a person is deliberate. "Someone fixed it" is not an audit entry. "calibration_worker updated confidence because CALIBRATION_V2 backfill ran" is.

Reproducibility, restated

Once the record is immutable and diffs are recorded, an actual reproducibility check becomes possible. Not "run it twice and eyeball the output" — a stored, replayable assertion:

def verify_reproducible(conn, prediction_id, tolerance=1e-6):
    """Re-derive the stored prediction from its recorded inputs.

    Asserts that: the row is unchanged since grading, the inputs it
    was computed from are still the inputs on record, and recomputing
    from those inputs lands on the stored numbers.
    """
    row, mutations = conn.execute("""
        SELECT p.*, count(m.id) AS mutation_count
        FROM ai_predictions p
        LEFT JOIN prediction_mutations m ON m.prediction_id = p.id
        WHERE p.id = %s
        GROUP BY p.id
    """, (prediction_id,)).fetchone()

    graded_mutations = conn.execute("""
        SELECT count(*) FROM prediction_mutations
        WHERE prediction_id = %s AND after_row ? 'direction'
          AND (before_row->>'direction') IS DISTINCT FROM (after_row->>'direction')
    """, (prediction_id,)).fetchone()[0]

    if graded_mutations:
        return False, "graded field changed post-settlement (%d times)" % graded_mutations

    recomputed = rederive(row["model_id"], row["fixture_id"], row["price_snapshot_id"])
    for k in ("p_home", "p_draw", "p_away"):
        if abs(float(recomputed[k]) - float(row[k])) > tolerance:
            return False, "recompute drifted on %s" % k
    return True, "reproduced"

The interesting failure mode is the third one, and it is the one nobody instruments: rederive returns a different number because the inputs changed underneath the row — the price snapshot was refreshed in place, the fixture was remapped, the model's prompt template was edited. A prediction can be perfectly immutable and still not reproducible, because its inputs were not. Which is why the snapshot has to be frozen before kickoff and referenced by id, not by timestamp.

That is the whole argument compressed: immutability is about the row, reproducibility is about the row plus its inputs, and neither is a property of your code — both are properties of your schema.

What we are changing, in this order

Being precise about status, since the whole post is about not overstating things — at the time of writing the numbers above, the fix is specified and not yet deployed. The plan, in dependency order:

1. prediction_mutations companion table first, with the audit trigger. This is additive and cannot break a writer, so it goes out before anything that can reject a write.
2. Backfill nothing. There is no before-image for the existing 35,013 rows, and inventing one would be worse than admitting the gap. The audit trail starts at the timestamp it starts.
3. (model_id, fixture_id) uniqueness, added NOT VALID first, validated after a duplicate scan — duplicates are sitting at zero right now by luck, not by constraint.
4. Only then the BEFORE UPDATE hard stop on graded fields, after auditing which writers actually touch them. A trigger that rejects a write some nightly job depends on is an outage with extra steps.
5. Grading moved into a single SECURITY DEFINER function that refuses to run twice for the same row.

Rough effort: about a day of work, spread across four deploys specifically so that no single step can take the writers down. The expensive part is not the SQL. It is admitting that a table with 46,192 rows and zero constraints was never immutable — it was merely unchallenged.

If you take one thing from this: go and run the three queries at the top against your own predictions table. The number you get for "rows modified after settlement" is the size of the trust gap you are currently asking your users to close on faith.

Analytics and research content. Not betting advice. 18+

The first version of the admin panel had a leaderboard. Sort by accuracy, descending, take the top one. It took about three weeks to learn that a leaderboard over a population that large is not a ranking — it's a multiple-comparisons machine with a UI.

Here's what replaced it, and the four bugs that forced each piece.

The problem with sorting by accuracy

Two hundred models, dozens of fixtures a day. Somebody will hit a hot streak. It is not a hypothesis that one of them will look excellent over a 40-fixture window; it is arithmetic. A leaderboard has no memory of how many candidates were tried, so a 40-fixture window of luck is indistinguishable from a 40-fixture window of skill — and the leaderboard will promote it to the top of the page, where it gets attention, which is the only reward the system offers.

The failure isn't subtle. In the early build the "top model" changed almost weekly, always upward, never stable. When I pulled the histories, the models that topped the board were systematically the ones with the fewest graded predictions. Small sample, large variance, and a ranking that treats variance as merit.

So the ranking had to stop being a ranking and start being a life cycle.

Four states, one direction

probation ──▶ active ──▶ degraded ──▶ retired
     │            │           │
     └────────────┴───────────┘
        (demotion allowed, promotion re-earned)

• probation — new model, publishing publicly but with a visible "unproven" badge. It exists to accumulate a sample.
• active — passed the promotion gate. Eligible for prominence.
• degraded — was active, drifted. Still publishing, no longer promoted. This state exists because the alternative is a binary that oscillates, which I'll get to.
• retired — stopped. Terminal in practice, but see the section on why the rows stay.

One rule I'd defend hard: state changes are events, not columns.

The obvious schema is models.status plus a cron job that reassigns it. That's what I wrote first, and it cost me every bit of history. Once the cron overwrites the field, you cannot answer "when did this model die, and on what evidence" — which is the only question worth asking when you're deciding whether a retirement was a mistake. A status column is a snapshot pretending to be a record.

Append-only transitions

CREATE TABLE model_transitions (
    id         bigserial PRIMARY KEY,
    model_id   bigint      NOT NULL REFERENCES models(id),
    from_state text        NOT NULL CHECK (from_state IN ('probation','active','degraded','retired')),
    to_state   text        NOT NULL CHECK (to_state   IN ('probation','active','degraded','retired')),
    reason     text        NOT NULL,
    evidence   jsonb       NOT NULL,   -- the actual numbers that justified it
    decided_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX ON model_transitions (model_id, decided_at DESC);

Current state is a view, never stored:

CREATE VIEW model_state AS
SELECT DISTINCT ON (model_id)
       model_id, to_state AS state, decided_at AS state_since
FROM model_transitions
ORDER BY model_id, decided_at DESC;

And the promotion guard lives in the writer, not the reader:

def transition(conn, model_id, to_state, reason, evidence, min_dwell_days=7):
    cur = conn.execute(
        "SELECT state, state_since FROM model_state WHERE model_id = %s", (model_id,)
    ).fetchone()
    if cur and cur.state == to_state:
        return None                       # no-op, not an error
    if cur and (now() - cur.state_since) < timedelta(days=min_dwell_days):
        return None                       # too soon, by design
    conn.execute(
        "INSERT INTO model_transitions (model_id, from_state, to_state, reason, evidence) "
        "VALUES (%s, %s, %s, %s, %s)",
        (model_id, cur.state if cur else "probation", to_state, reason, json.dumps(evidence)),
    )

evidence is the part people skip and then regret. It stores the window size, the metric value, and the comparison baseline at the moment of the decision. Six months later, "why was this model retired" has an answer that isn't my memory.

Flapping, and why degraded exists

The first demotion rule was a single threshold. Below it, demote. Above it, promote. A model sitting near the line generates a state change every single evaluation tick, and each change fires notifications, rewrites the public badge, and generally makes the system look broken.

Three fixes, all necessary:

1. Two thresholds, not one. Promote at the upper band, demote only below the lower band. The gap is dead space where nothing happens. This is just hysteresis and I should have started there.
2. Minimum dwell time. Even inside the dead space, refuse any transition within 7 days of the last one. Short-lived states are almost never information.
3. degraded as a buffer. A model coming off active goes to degraded first, not straight to retired. Retirement requires it to fail again from degraded. One bad window shouldn't be terminal; two consecutive bad windows in different states is a pattern.

The counter I keep is the number of transitions per model per month. Anything above ~2 is either a threshold sitting exactly at the population mean or a genuinely unstable model — both worth knowing.

Retiring means stopping, not deleting

This is the one that actually corrupted a number I had published.

When a model failed, the cleanup job deleted its predictions. Reasonable-sounding: dead models don't need rows. The effect was that the platform-level accuracy figure — computed over whatever predictions still existed — silently excluded every model that performed badly enough to be retired. The number went from 54% to 61% and I briefly believed it, because the computation was correct and the denominator was a lie.

The platform-level metric is only meaningful if the population includes the failures:

-- every model that ever published, including the dead
SELECT count(*) FILTER (WHERE g.outcome = argmax_of_prediction)::float / count(*) AS accuracy
FROM predictions p
JOIN models m       ON m.id = p.model_id
LEFT JOIN grades g  ON g.prediction_id = p.id
WHERE p.created_at BETWEEN %s AND %s;

No WHERE m.state = 'active'. If you filter to survivors and call the result platform performance, you're not measuring the system, you're measuring the system after it selected for the outcome you wanted.

Retired models keep their predictions, their grades, and their transition history. 102 of them at the moment. They are the control group.

Promotion gates that survive contact with reality

Absorbing everything above, the gate for probation → active is:

• at least 50 graded predictions in the current window (below that, the interval on any rate is wider than the differences we're trying to detect)
• performance measured within the same competition and compared against that competition's contemporaneous median, never against a global figure
• no promotion within 7 days of the last transition

The competition scoping is not pedantry. A model covering one evenly-matched league and a model covering forty leagues of wildly different predictability are answering different questions. Blending them produces a number that describes neither, and the model that happened to pick the easy competition wins the leaderboard for reasons that have nothing to do with modelling.

I also stopped showing a single "platform accuracy" number on the site. The honest answer to "how accurate is the platform" is a distribution across models and competitions, and the moment you collapse it to one figure you've started marketing instead of measuring.

Failure modes, honestly

• The cron that ate the history. Described above. Append-only transitions, view for current state. Cost: one extra join on read, worth it every time.
• Thresholds set at the population mean. Guarantees flapping, because half the population is always on the wrong side of the mean. Set bands from the distribution's interquartile range, not its centre.
• Comparing a model to itself across regimes. A model's own history includes retraining. Comparing this month to six months ago compares two different objects wearing the same model_id. Any retraining that changes weights is a new model row.
• retired used as a soft delete. If retirement hides a model from every query, you've rebuilt survivorship bias with a nicer name.
• No cap on promotions. Without a dwell time, a noisy window promotes, the next window demotes, and the public page flickers. Users read flicker as unreliability, correctly.

What this buys

The leaderboard answered "which model looks best right now," which is a question about noise. The life cycle answers "which models have earned attention, which are being watched, and which are gone" — and every answer has a timestamp, a reason, and the numbers that produced it.

The schema is thirty lines. The hard part is accepting that a model's status is a claim you have to defend later, with evidence you kept at the time.

The platform this comes from is SoccerDream AI — independent models publishing timestamped probability estimates on football fixtures, graded in the open, retired models included in the record. Per-model pages show the full life cycle rather than a current score.

Analytics and research content. Not betting advice. 18+