A Million Small FilesLakehouse · medium · ~13 minThree months ago the clickstream_iceberg table moved from hourly batch loads to micro-batch commits every minute — product wanted fresher dashboards, and they got them.Since then, the same dashboard queries have slowed from 8 seconds to nearly 3 minutes. Nothing about the queries changed. Nobody set up anything else — that's the problem.Evidence

Metrics

Config

Logs

Checkpoints

Same queries, same data volume — why did performance collapse?

At ~20 files per commit and one commit per minute, how many new data files does the table accumulate per day without compaction? (whole number)

Whose job was compaction?

Design the recovery and the steady state: immediate cleanup, the ongoing maintenance loop, and the commit-cadence conversation with product.