Feature stores and platforms — the DoorDash arc, Airbnb's Chronon, Uber's Michelangelo — plus training-data pipelines and the AI-era stack.
MLOps is mostly data engineering. The proof is the feature-platform canon: DoorDash's Riviera→Fabricator→Workbench arc, Airbnb's Chronon (run at Stripe too), LinkedIn's Feathr, Uber's three Michelangelo generations, and the embedding-era stack — where interviews are heading as DE and ML infra converge.
One declarative definition compiled twice: point-in-time joins that kill leakage, the lambda split behind 10ms serving of long windows, and consistency measured — not assumed.
The model crushed it offline: AUC up four points, backtest immaculate. In production it's mediocre — and worse, *slowly getting worse*. The investigation finds the quiet crime: the training features were computed one way in the warehouse (clean SQL over settled data) and reimplemented another way for serving (a Java service over live streams). "Average order value, last 30 days" means something subtly different in each. The model learned one world and is being asked about another. Half the team wants to rewrite the serving path; the other half wants to log the online features and retrain in six months, once enough accumulate. Both fixes are miserable. What removes the choice?
The cold open's dilemma is the founding problem of feature platforms. Strategy one — reimplement offline features online — creates *training-serving skew*: two codebases, two interpretations, drift forever. Strategy two — log-and-wait — serves only features computed online and logs them until enough history accumulates to train on: consistent by construction, but every new feature idea costs *months* of waiting, and seasonal effects need a year. Both strategies also flirt with the subtler crime, label leakage: if training rows see data from *after* the moment of prediction, the model trains on the future and dies on contact with the present.
The mechanism that removes the choice: one declarative definition, compiled twice. You declare a feature — say, a windowed aggregation (COUNT of orders, last 30 days, keyed by user) — and the platform compiles it into *both* a historical backfill and an online serving pipeline. The correctness core is the point-in-time join: training data is built from a "left side" of timestamped query events (every checkout, with its timestamp), and each feature attached to a row is computed *as of that row's timestamp* — the left timestamp bounds every aggregation window, so no future data can leak into any training example. Backfill replays history with this discipline; serving computes the same definition live. One logic, two executions — skew and leakage die together, and new features backfill *years* of training data in days instead of logging forward for months.
COUNT of orders, last 30 days, keyed by user
🤔 Check yourself: A teammate builds training data by joining yesterday's feature-table snapshot to all of last year's labeled checkout events. The model looks great offline and flops online. Name both defects in one sentence each.
Leakage: every training row — even January's — sees *yesterday's* feature values, i.e., the future relative to its label. Skew-adjacent staleness inversion: training saw perfectly fresh features for old events, while serving sees features at real freshness — the model learned from a world that never exists at prediction time. The point-in-time join fixes both by making each row's timestamp the boundary.
How does the same definition serve at ~10 milliseconds *and* backfill years? The reference implementation's mechanics generalize.
Primitives: a *GroupBy* — a windowed, keyed aggregation maintained as partial aggregates, so huge windows and long backfills combine cheaply instead of recomputing per-day; a *Join* — the point-in-time attachment of many GroupBys to a left side; and typed sources: event sources (timestamped logs — a Kafka topic paired with its warehouse table), entity sources (dimension tables via daily snapshots — upgraded to millisecond precision *only if CDC exists*), and cumulative sources (insert-only histories where the latest partition suffices).
The accuracy dial: each feature declares SNAPSHOT (daily refresh, midnight cutoffs) or TEMPORAL — and TEMPORAL compiles to a lambda pipeline: a batch job seeds the KV store with the compressed "middle" of long windows from warehouse partitions, a streaming job tails the event topic writing incremental "heads," and the *fetcher reconciles batch tail plus streaming head at read time*. That reconciliation is the load-bearing trick: long windows stay accurate without unbounded streaming state, at ~10ms reads.
And the honesty layer: consistency between what was served and what backfills would say is *measured, not assumed* — serving responses are logged, the same rows later backfilled offline, and the diff quantifies online/offline skew continuously.
🏭 In production at Airbnb: this is Chronon — ten thousand+ features, new feature sets shipping in under a week, open-sourced and running at Stripe too. The two constraint-shaped details worth quoting: temporal accuracy on *dimension* data requires CDC infrastructure (no mutation stream, no millisecond entity features — you're stuck at daily snapshots), and un-windowed "lifetime" aggregations are supported but discouraged, because their distributions drift forever and quietly degrade models.
⚠️ Gotcha: the platform's defaults encode temporal semantics people miss in review: a training join whose left side is an *entity* (not event) source defaults to midnight-snapshot timestamps — silent day-level granularity when you expected event-level. Temporal semantics are the feature store's real API; the YAML is just syntax.
Strip the specifics and a repeatable pattern emerges — registry in, generated infrastructure out — and it shows up independently across the industry's platforms.
*The package-manager variant:* features registered once by name — a definition over raw sources or over other registered features — and consumers *import by name*; the platform replays definitions over history for point-in-time training joins and pre-materializes them into online stores. The unit of reuse becomes the feature name: a feature built by Search is one import away for Feed or Ads. Measured effect at LinkedIn: iteration cut from weeks to days, and the centralized replay engine ran up to 50% *faster* than the bespoke pipelines it replaced — a generic engine, optimized once, beats N hand-rolled ones.
*The streaming variant:* a YAML spec — source topic, a SQL transformation with windows and interval joins, a sink with TTL and delivery semantics — instantiates a generic streaming job per feature: isolation by construction (one feature's failure or resource spike touches nothing else), protobuf events flattened into SQL-addressable columns, connections and auth invisible to the author. DoorDash's Riviera: iteration from weeks to hours, ~70% less code than hand-written Flink.
*The batch variant:* the registry drives code generation — declarative definitions validated at registration, orchestration DAGs generated (the author never writes one), and materialization to the online store auto-triggered when upstream data lands. DoorDash's Fabricator: 100+ billion feature values daily from ~500 declared features, with year-long backfills in hours.
⚖️ Tradeoff: every variant standardizes the 80% case and taxes the tail — SQL-level DSLs can't express arbitrary logic, generated DAGs put debugging one level away from what the user wrote, and auto-materialization ties online freshness to upstream table SLAs the feature author doesn't control. The platform bet is that uniform operations for the many beats maximum expressiveness for the few — and the adoption numbers keep vindicating it.
🤔 Check yourself: Your platform generates pipelines from a registry. A data scientist's new feature spec is valid YAML but references a source table with a 6-hour landing SLA, feeding a model that assumes 15-minute freshness. Where should this fail, and why is "at 2am in orchestration" the wrong answer?
At *registration* — the registry knows the source's SLA and the sink's freshness declaration, so the mismatch is statically checkable the moment the spec is submitted. Failing at runtime means the feature ships, silently serves 6-hour-stale values as "real-time," and the model quietly underperforms — the exact class of invisible failure declarative platforms exist to make visible.
Theory meets a latency budget in the newest reference build: a document-ranking feature store inside a sub-100ms query budget, on split infrastructure (on-prem serving, cloud batch) that disqualified off-the-shelf cloud feature stores outright — network topology is an architecture input, not a detail.
The design: keep the framework's *definition layer* (Feast) so training and serving share feature definitions, but rebuild the data plane where the budget demands. Batch features flow through a medallion pipeline with change detection — diff against prior state, write only changed records: hundreds of millions of candidate rows shrink to ~1M actual writes, batch cycle from over an hour to under five minutes. Streaming signals ingest in seconds-to-minutes. And the serving hot path: the framework's Python server couldn't hold latency under concurrency (GIL plus JSON parsing — serialization, not model math, was the bottleneck), so a custom Go service replaced it — adding only ~5–10ms over the ~20ms store read, for ~25–35ms p95 against the 100ms budget.
💡 Note: the pattern to carry forward — *keep the control plane, replace the data plane where the budget bites* — recurs all week. Frameworks earn their keep in definitions and consistency; hot paths get rebuilt in whatever holds the SLO.
| Concept | One-line mechanism | Number to remember | Production proof |
|---|---|---|---|
| Point-in-time join | left row's timestamp bounds every feature window | leakage = training on the future | Chronon's Join |
| Lambda serving | batch seeds the window's middle; streaming patches the head; fetch reconciles | ~10ms serving on long windows | Chronon TEMPORAL mode |
| Measured consistency | log served values, backfill the same rows, diff | skew quantified, not assumed | Chronon consistency pipeline |
| Registry pattern | declare once → generated pipelines, import-by-name reuse | weeks → days; 50% faster than bespoke | Feathr; Riviera; Fabricator |
| Budget decomposition | keep definitions, rebuild the hot path | 25–35ms p95 in a 100ms budget | Dropbox Dash |
Feature-store questions test temporal reasoning more than tooling. Expect: *"Why did your model degrade between offline and online?"* (skew and leakage, mechanically — then the one-definition-two-compilations fix). *"How do you build training data for a new feature without waiting months?"* (point-in-time backfill; what the left side is; what bounds each window). *"How can a 90-day windowed feature serve in 10ms?"* (the lambda split and fetch-time reconciliation — and why pure streaming can't hold the tail). *"Would you trust that online and offline agree?"* (no — measure it; describe the log-and-diff pipeline). *"Design the platform"* (registry in, generated pipelines out, with the registration-time SLA checks that make invisible failures visible). The senior tell: you treat timestamps as the API — everything else is syntax.
Open week 1 →
The serving stack, budgeted: layout beats engine choice, a client cache absorbs the repetition, cold tails route to disk by cardinality × read rate, and streaming state stays bounded by design.
The feature store bill doubled two quarters running, and the capacity review is grim: the Redis fleet crossed one hundred nodes, every resize is a two-day blue-green ceremony, and feature count grew 10× in a year with no ceiling in sight. Meanwhile the serving graphs show something odd: tens of millions of reads per second, but the *same* entities — the same restaurants, the same busy users — fetched over and over, milliseconds apart, at full price each time. Storage priced by memory, reads priced by volume, and a workload that's mostly repetition. Three different inefficiencies are hiding in that one bill. Can you name them?
The full week 2 brief is part of LeetData Pro.
Open week 2 →
The generations law (rewrite on workload-class change, extend when API-shaped), adoption as an org variable, the build/buy boundary migrating toward abstractions, and UX as the last bottleneck.
The ML platform team presents its third roadmap in four years. The first platform standardized everyone onto shared pipelines — then deep learning arrived and none of it fit. The second rebuilt for GPUs — adoption stalled at a handful of teams for a year, then mysteriously hit 95% the next. Now generative AI is here, product teams are calling external APIs from laptops, and the platform lead proposes rewrite number three. The CTO asks the only question that matters: *why will this one be different?* The honest answer requires knowing why the last two went the way they did. Do you?
The full week 3 brief is part of LeetData Pro.
Open week 3 →
Two-tower retrieval with its three silent-failure decisions, RAG stages named by the failure each fixes, vectors beside your metadata, and the debugging funnel that keeps any of it honest.
The demo was magic: ask the corpus a question, get a cited answer. Six weeks into production the complaints have a pattern — exact product codes return nothing (the embedding "understood" them into mush), a rephrased question retrieves different documents than its twin, and one answer confidently cited a paragraph that says the opposite. The team's fix list reads: bigger context window, better LLM, more chunks. None of those is the fix, because each failure lives in a *specific stage* of a pipeline the demo never made visible. Can you name the stage for each?
The full week 4 brief is part of LeetData Pro.
Open week 4 →
