The Model That Aced the BacktestQuality · hard · ~13 minThe new fraud model was spectacular offline: AUC 0.94 against 0.81 for the incumbent. It shipped two weeks ago. Online, it's performing *below* the incumbent — and its precision decays a little more each day.The training data was built fast: the team joined the current feature-store snapshot (each user's chargebacks_90d, orders_30d, account_age…) onto 18 months of labeled historical transactions and trained on the result.The serving path reads the same feature store live. Same features, same store, same model. The data scientist insists nothing is inconsistent. The evaluation harness agrees. Production disagrees.Evidence

Offline vs online

How the training set was built

Team context

Checkpoints

Why did the model ace the backtest and fail online?

The example row: a first-offense fraud on 2024-01-15 trained with chargebacks_90d = 4 (its later consequences). At serving time, what value does a comparable first-offense fraudster show for this feature?

Which rebuild produces correct training data without waiting months?

Write the remediation plan: the backfill design (what bounds each window, what the left side is), why proposal A is a losing strategy here, and the standing safeguard that catches this class of bug before the next launch.