Lakehouse & Table Formats in ProductionIceberg, Hudi, Delta, and Paimon at company scale — compaction ops, format wars, catalogs, and the S3-native future.The freshest slice of the corpus: Adobe's Iceberg arc, Ancestry's 100-billion-row table, Kakao's operational takeaways, Whoop's schema-migration tooling — plus the deepest expert vein in the archive (Vanlightly's consistency-model series). This is also the natural partner-track cohort.Week 1: Why tables grew a brainWhat table formats actually add over files — transactions, snapshots, and optimistic concurrency — and the consistency models worth knowing cold.

Figure 1 — what a table format adds over a pile of files. By the end of this week you can read this as a prediction: which failures are structurally impossible once the metadata layer exists, and which races remain yours to manage

Your nightly backfill writes 900 Parquet files to a table's S3 prefix. At 2 a.m., on file 400, the Spark driver dies. Nothing cleans up. At 7 a.m. the revenue dashboard runs, lists the prefix, and happily reads 400 files' worth of "revenue" — down 55%, apparently. No alert fired, because nothing failed *after* the crash: every system did exactly what it was built to do. Who, exactly, let half a dataset become the truth?This week is about the one missing primitive that answer hides behind — and the machinery every table format (Iceberg, Delta, Hudi, Paimon) builds to supply it.After this week you can

Explain why plain object storage cannot give you a consistent table — and name the missing primitive

Walk the write-data-then-flip-one-pointer commit trick, and say precisely what a snapshot is

Narrate two writers racing to commit and how Delta, Hudi, and Iceberg each resolve the race

Answer "which format?" with a workload analysis, and read a vendor benchmark without being had

The missing primitiveStart with what object storage actually promises: you can PUT one object atomically — it appears whole or not at all — and that is where the guarantees end. There is no transaction spanning two objects, no rename-directory, no lock. A "table" on S3 is thousands of files plus a *convention* that they belong together. Readers discover the table by listing a prefix; writers add files to the same prefix; and between a writer's first file and its last, every reader sees a partially written table. That's the 2 a.m. incident: the missing primitive is atomic multi-file commit — the ability to make N files appear as one all-or-nothing event.Teams first patch this with discipline: _SUCCESS marker files, staging directories renamed at the end, "don't query before 8 a.m." norms. Every patch shares a weakness — nothing *enforces* it. One consumer who lists the prefix directly defeats the whole scheme. Consistency by discipline fails open.A table format closes the gap by inverting authority: a metadata layer, not the file listing, defines which files are the table. Files on disk are just bytes; the metadata *is* the table.🏭 In production at Adobe: their lake ingests on the order of a million batches — tens of billions of events — a day, and before Iceberg, failed Spark jobs leaving partial results readable downstream was a standing class of incident. After: that class of bug disappeared *structurally*, not through better pipeline hygiene. Consistency by construction beats consistency by discipline precisely because it doesn't depend on everyone behaving.🤔 Check yourself: A reader lists the table's prefix while your 900-file write is at file 400. What does it see with plain Parquet-on-S3, and what does it see once a table format is in charge — and why?Plain: 400 files that look exactly like the table, because listing *is* membership. With a format: the previous complete version of the table, because the 400 new files aren't referenced by any committed metadata yet — unreferenced files are invisible bytes, not data.The trick: write data, then flip one pointerHere's the move every format plays, in two beats. Beat one: write all your data files. They're durable but *invisible* — no metadata references them, so no reader will ever find them. Beat two: perform ONE atomic metadata write that creates a new snapshot — a complete, versioned description of exactly which files constitute the table right now. Readers resolve the current snapshot once at query start and then work from that frozen file list, unbothered by anything committed after — which is snapshot isolation, and it means a crash between beat one and beat two strands some harmless invisible files and corrupts nothing.

Figure 2 — the two-beat commit: data files land invisible, then one atomic metadata write flips the table to snapshot N+1; a crash between the beats strands bytes, never truth

The formats differ in what that one metadata write is:

Delta appends a sequentially numbered JSON entry to the _delta_log directory — the log is the table. To avoid replaying history forever, a Parquet checkpoint summarizing state is written every ~10 commits: reading a table with 10,000 commits means one checkpoint plus at most ten JSON tail files, not ten thousand parses. Do that arithmetic once and checkpoint lag stops being an abstract concern — every checkpoint you're behind adds files to *every reader's* startup.

Iceberg builds a manifest tree (files → manifests → manifest list → snapshot) and swaps a single table pointer in a catalog. The tree structure is why planning scales: statistics live at every level, so pruning happens before file listings are ever touched.

Hudi writes timeline "instants" — files named [timestamp].[action].[state] progressing requested → inflight → completed — and only the *completed* instant exposes new file slices.

Time travel, incidentally, costs nothing extra: old snapshots are just older versions of the metadata, still pointing at still-existing files. You met the bill for that in week 2 — snapshots pin files until expired.⚠️ Gotcha: the metadata being the table cuts both ways. Corrupt or tamper with _delta_log and the table is gone even though every data byte survives. And in multi-region setups, replicating the log ahead of its data files produces readers that resolve a snapshot pointing at files that don't exist yet at their site — the exact bug the ordering was designed to prevent, reintroduced by the replication layer.🤔 Check yourself: Why must the data files land *before* the commit record — what breaks if a format wrote the metadata first and the data files second?The commit record is the moment of visibility. Written first, it advertises files that don't exist yet, so readers resolve the new snapshot and hit missing-file errors — a *corrupt-looking* table from a healthy writer. Data-first means the failure mode is invisible garbage; metadata-first means the failure mode is visible lies.Writers race, and that's the designTwo jobs commit to one table with no lock in sight. Every format's answer is optimistic concurrency control (OCC) — assume you won't conflict, do all your work against a base snapshot, and validate at the moment of commit. The mechanism has three steps you should be able to narrate cold. Writer A and writer B both read snapshot N and prepare their commits. A publishes first: the table is now at N+1. B now attempts to publish its own N+1 — and this step *must be exclusive*, via a conditional put (write-if-absent, so the second write of the same slot fails) or an external lock service. B's attempt fails cleanly, and B re-reads what A committed and asks: do we actually conflict?

Figure 3 — two writers race under OCC: both prepare against snapshot N, one wins the atomic publish, the loser validates overlap and either re-commits or aborts

That conflict check is where the formats genuinely diverge, and the granularity decides production behavior:

Delta checks at partition granularity: A and B touched disjoint partitions → B simply re-commits as N+2; overlapping partitions → B aborts and retries its work. Coarse, simple, more false conflicts on same-partition writes.

Hudi checks per file group — every primary key maps to exactly one file group, so disjoint *keys* commit concurrently even inside one partition. Finer, but correctness leans on timestamps being issued monotonically (in practice: a lock provider issuing them).

Iceberg validates at the snapshot level with file-overlap checks — the week-1 scenario puts you inside exactly this machinery, adjudicating a compaction job racing ingest.

⚠️ Gotcha: Vanlightly's model-checking of these protocols found the sharpest production edge: multi-writer Delta on plain S3 *without* a lock provider — S3 historically lacked conditional puts — can silently lose committed transactions. Two writers overwrite the same log slot; one commit simply vanishes; nothing errors. The conditional put isn't an optimization. It's the keystone holding the arch up.🤔 Check yourself: Two hourly Delta jobs write to disjoint partition ranges and occasionally overlap in time. A third job periodically rewrites the whole table. Predict each pairing's behavior under partition-granularity OCC.The two hourly jobs race for the log slot; the loser's conflict check finds disjoint partitions and it re-commits at the next version — invisible, automatic, fine. Either hourly job racing the full-table rewrite finds overlapping partitions and must abort and retry — and if the rewrite is long-running, it may be the one that keeps losing. Whole-table operations under OCC are conflict magnets; schedule them like you mean it.Choosing a format is choosing a workloadFeature matrices won't make this call for you, because the real fork is mechanical: where do primary keys live? Hudi binds every key to a file group in the format itself; Paimon goes further with an LSM design — write-optimized levels merged in the background — where every row carries a change marker. In both, a row-level upsert commits *without scanning the table*: the format knows where the row lives. Iceberg and Delta made the opposite bet — simpler specs, no native keys, broadest engine ecosystem — so row-level work is the engine's job: a MERGE that scans and joins to locate rows, or an engine-maintained key index. Update-heavy CDC mirrors lean toward the native-key camp; append-heavy analytics lean toward the simple-spec camp; week 4 prices this fork in detail.The second skill in this section is reading the comparisons themselves. The loudest format shootout in the corpus is authored by a vendor founded by one format's creators — and even it concedes that feature tables and benchmarks rarely represent real workloads. Before believing any number, check three things: were both sides in the same write mode (a CoW-vs-MoR mismatch flips rankings by itself)? Was maintenance state comparable, or was one table freshly compacted? And who signs the author's paycheck?⚖️ Tradeoff: merge-on-read vs copy-on-write is this same who-pays choice at file scale — week 2 priced it: MoR cheapens writes by deferring merge work to every reader and making compaction a standing tax; CoW pays at write time and reads clean. Formats don't eliminate costs. They relocate them, and the relocation is the product.🤔 Check yourself: You're mirroring an orders table via CDC — thousands of row updates a second — into a lakehouse. Which format camp fits, and what's the first critical question if a vendor benchmark says otherwise?Native-primary-key camp (Hudi/Paimon): at that update rate, per-batch MERGE scans are ruinous, and the format locating rows by key is the whole game. If a benchmark claims otherwise, first check write-mode parity — an Iceberg table benchmarked CoW-vs-MoR-mismatched, or measured fresh while the other side carried maintenance debt, proves nothing.The mental model| Concept | Mechanism in one line | Number to remember | Production proof ||---|---|---|---|| The missing primitive | Object stores commit one object at a time; listings ≠ membership | 400 of 900 files readable mid-crash | Adobe: partial-result bugs ended structurally || The commit trick | Data files invisible until one atomic metadata write flips the snapshot | ~10 commits per Delta checkpoint | ~1M batches/day riding snapshot isolation || Writers racing | OCC: prepare against base snapshot, exclusive publish, validate on loss | 1 conditional put = the keystone | Model checking: no lock on S3 → silent lost commits || Format choice | Native primary keys (Hudi/Paimon) vs engine-side MERGE/indexes (Iceberg/Delta) | — | Vendor shootouts concede benchmarks ≠ workloads |Interview lensEach objective maps to an interview move:

"Why do lakehouses need table formats at all?" — derive it from the missing primitive: one-object atomicity, listing-as-membership, the 2 a.m. partial-write story, then formats as metadata-owns-membership. Mechanism first, Adobe as confirmation.

"How does a commit actually work?" — the two beats, in one breath: invisible data files, then one atomic metadata write; name what your chosen format's metadata write is (log entry, pointer swap, completed instant), and why the ordering is non-negotiable.

"Two writers hit the table at once — what happens?" — narrate the OCC race: prepare, exclusive publish, loser validates at *partition / file-group / snapshot* granularity depending on format, then re-commit or abort. Cite the no-lock-on-S3 silent-loss edge as the senior flourish.

"Which format would you pick?" — answer with workload shape (CDC-upsert vs append-scan) and where keys live, then demonstrate benchmark skepticism: write-mode parity, maintenance state, authorship.

Case studies

Capital One — Delta Lake transaction logs explained

Jack Vanlightly — Understanding Apache Hudi's Consistency Model

Jack Vanlightly — Understanding Delta Lake's consistency model

Kyle Weller — Apache Iceberg™ vs Delta Lake vs Apache Hudi™ - Feature Comparison Deep Dive

Alireza Sadeghi — The History and Evolution of Open Table Formats

Jack Vanlightly — Understanding Apache Paimon's Consistency Model Part 1

Exercises

Week 1 check: Formats & Internals

The Compaction Job That Couldn't Commit

The Pushdown That Wasn't

Partition Count: Too Few, Too Many