The Compaction Job That Couldn't CommitLakehouse · medium · ~14 minThe events_iceberg table has two writers: a streaming job upserting micro-batches every 60 seconds, and a nightly compaction job that rewrites small files.For the past week, compaction has failed almost every night — burning 25 minutes of compute per attempt, retrying four times, then giving up. The table keeps degrading. The on-call suggestion: "just add a lock so only one job can write."Explain what's actually happening in the commit protocol, and design a fix that keeps BOTH jobs running.Evidence

Logs

Config

Metrics

Checkpoints

Why does compaction keep failing while the streaming job never does?

With streaming commits every 60 seconds and a 25-minute compaction attempt, roughly how many streaming commits land during ONE compaction attempt? (whole number)

What's the highest-leverage structural fix?

Write the co-existence design: job scoping, commit/retry strategy, what you monitor, and why the 'just add a lock' suggestion loses more than it gains.