The Replay That Double-BilledKafka · hard · ~13 minYour ads pipeline attributes orders to ad clicks: a Flink attribution job consumes committed orders, looks up matching ad events in a document store, and emits attribution records — each with a UUID — into Kafka. The OLAP store dedupes on that UUID via upsert. Kafka transactions + checkpoint two-phase commit protect the pipeline hops. The design doc says exactly-once, end to end.Last Tuesday the attribution job was down for 4 hours (a dependency outage). The on-call rewound its consumer offset and replayed the window — standard runbook.Today, finance reports advertiser invoices for Tuesday are ~6% high, concentrated in the replayed window. The pipeline dashboards show zero errors and exactly-once intact.Evidence

Attribution records (Tuesday)

Code + runbook excerpts

Design-doc claims

Checkpoints

Transactions held, sinks deduped on UUID, dashboards show exactly-once intact — yet the replayed window double-billed. Where is the hole?

The 16:00 hour emitted 88K records; the store holds 86K distinct UUIDs for it. Roughly how many duplicate attributions (records that represent an already-attributed order under a fresh UUID) landed in that hour, assuming the true workload was ~44K?

Why does the UPSTREAM aggregation job get away with random UUIDs while attribution cannot?

Write the fix and the remediation: the key-derivation change, what makes Tuesday's data repairable, the enrichment caveat that remains even after the fix, and the design-review rule you'd institute.