How do you version data for ML, and why is 'code + model versioning' not enough for reproducibility?
Quick Answer
A model is a function of code AND data — reproducing or auditing it requires pinning both. Version data with immutable snapshots or content-addressed references (DVC, lakehouse table versions/time travel, s3 object versions + manifest), and record the exact data reference in the training run's metadata so any model maps back to the bytes that trained it.
Detailed Answer
The failure that motivates it: a model regresses, you retrain from 'the same' code, and get a different model — because the feature table mutated in place. Approaches: (1) content-addressed pipelines (DVC/lakeFS) — data files hashed, referenced by pointer files that live in git, so git checkout reconstructs the exact dataset; (2) lakehouse time travel (Delta/Iceberg table versions) — training reads a pinned snapshot ID, cheap because storage is shared across versions; (3) discipline-level minimum: immutable dated partitions plus a manifest (paths + checksums + row counts) logged with the run. Whichever mechanism, the contract is the same: the training run's metadata records an immutable data reference, and re-running with that reference is byte-identical. This also unlocks: auditability (regulators ask 'what data trained this?'), safe backfills (new dataset version, not in-place mutation), and debugging skew (diff data versions between good and bad models). The trap to name: versioning raw data but not the feature-engineering output — if features are computed by a mutable pipeline at train time, pin the feature table, not just the source.
Code Example
# DVC-style
dvc add data/train.parquet # .dvc pointer file goes in git
git commit -m 'train data v14'
# lakehouse-style
df = spark.read.option('versionAsOf', 412).table('features.churn')
mlflow.log_params({'data_ref': 'features.churn@v412', 'data_rows': df.count()})Interview Tip
Frame it as 'model = f(code, data) — pinning one of two inputs is not reproducibility'. Mentioning the feature-table-vs-raw-data distinction shows you've been burned properly.