Partition Pruning, BrokenSpark · medium · ~12 minThe orders fact table is partitioned by order_date. The BI extract filters to the last 7 days — but not with literals: the date list comes from joining a small reporting_windows control table. Since that 'config-driven' refactor, the job scans all 6 years of partitions. Static pruning can't see through a join... but Spark has a tool for exactly this.Evidence

Scan Evidence

Config

Checkpoints

Why doesn't ordinary (static) partition pruning apply here?

What is Dynamic Partition Pruning and why is it built for this exact shape?

Fix and hard-verify — plus: when would you ALSO keep a literal-pushdown path? Interview-style.