The Two-Minute IcebergLakehouse · medium · ~14 minToday, orders flows from Kafka into the Iceberg table via an hourly Spark batch. The ops team just got a new requirement: "fraud review needs orders visible in the lakehouse within 2 minutes."Three proposals are on the table:

A: Kafka Connect's Iceberg sink, committing every 30 seconds

B: A Flink job writing upserts to Iceberg, checkpointing every minute

C: Keep the hourly batch and tell fraud review to query Kafka directly

Every option is someone's favorite. Pick with your eyes open — the freshness-files-semantics triangle has no free corner.Evidence

Requirements

Notes

PoC metrics

Checkpoints

What disqualifies option A (Connect sink at 30s) for THIS table, regardless of file counts?

Option B works correctly in the PoC — and read amplification is climbing toward 4×. What's the mechanism?

The buried lede in the notes changes the whole decision. Which is it?

Make the call and defend it: which ingestion path, what commit cadence, who owns compaction/expiry, and what you tell fraud review and finance.