Ten Million Tiny FilesSpark · medium · ~12 minThe events lake ingests hourly. Each hourly Spark job writes with 2,000 tasks, producing 2,000 files — most under 300KB. Eighteen months later the daily full-table model spends 70 of its 95 minutes just LISTING and OPENING files, and the metastore groans. You inherit the mess: fix today's pain and stop tomorrow's.Evidence

Metrics

Config

Checkpoints

Where do the tiny files come from, mechanically?

You compact the 3.7TB table targeting ~512MB files. Roughly how many files should result? (round number)

Stopping the bleeding for FUTURE writes?

Why exactly are small files so expensive on object storage, and how do you narrate the whole cleanup? Interview-style.