Data Science and Machine Learning 101: Programming and Data Engineering Version Control and ML Experiment Tracking Git DVC MLflow

What This Is

Version control & experiment tracking keep every piece of a machine‑learning project—code, data, model artifacts, and hyper‑parameters—reproducible and searchable. In a churn‑prediction pipeline, for example, you might iterate over dozens of feature‑engineering scripts, data snapshots, and model versions; Git, DVC, and MLflow let you snap each change, compare results, and roll back to the exact state that produced the best AUC without guessing which notebook you ran last.

Key Terms & Formulas

• Git – Distributed version‑control system that snapshots text files (code, notebooks, config) as commits; each commit is a SHA‑1 hash.
• DVC (Data Version Control) – Extension of Git that tracks large data files, model binaries, and pipelines via .dvc metafiles; stores heavy artifacts in remote storage (S3, GCS, Azure).
• MLflow Tracking – Server‑client API that logs runs (parameters, metrics, artifacts) to a central store (SQLite, MySQL, or S3).
• Run ID – Unique UUID generated by MLflow for each experiment execution; used to retrieve logged artifacts.
• Metric = f(y_true, y_pred) – Any scalar you log (e.g., RMSE = √(1/n Σ (y_i – ŷ_i)²)).
• Parameter = hyper‑parameter value – Anything you pass to a training function (e.g., max_depth=6).
• Pipeline Stage – DVC concept: a command line that transforms inputs → outputs; defined in dvc.yaml and reproducible via dvc repro.
• Git LFS (Large File Storage) – Optional Git extension that stores binary blobs (e.g., .h5 models) outside the repo while keeping pointers in Git.
• Artifact – Any file produced by a run (model checkpoint, plot, CSV of predictions).
• Reproducibility Equation: Result = f(Code, Data, Params, Env) – To guarantee the same Result, you must version‑control all four components.
• Experiment Comparison – In MLflow UI, runs are sorted by a chosen metric; you can query with mlflow.search_runs(filter_string="params.max_depth='6'").

Step‑by‑Step / Process Flow

1. Initialize Git & DVC
   bash
git init
dvc init
git commit -m "baseline repo"
2. Add raw data to DVC (remote storage optional)
   bash
dvc add data/raw/churn.csv
git add data/raw/churn.csv.dvc .gitignore
git commit -m "track raw data"
3. Create an MLflow experiment & log a run
   python
import mlflow, mlflow.sklearn
mlflow.set_experiment("churn-prediction")
with mlflow.start_run():
mlflow.log_param("model", "RandomForest")
mlflow.log_param("n_estimators", 200)
model.fit(X_train, y_train)
preds = model.predict_proba(X_test)[:,1]
mlflow.log_metric("AUC", roc_auc_score(y_test, preds))
mlflow.sklearn.log_model(model, "model")
4. Define a DVC pipeline stage (dvc.yaml) that runs the training script, declares inputs (data, code) and outputs (model artifact, metrics JSON).
   yaml
stages:
train:
cmd: python src/train.py
deps:
- src/train.py
- data/processed/train.parquet
outs:
- models/rf.pkl
- metrics/metrics.json
5. Reproduce & iterate – After tweaking features or hyper‑parameters, run dvc repro to rebuild only the affected stages; Git/DVC will automatically version the new data/model files.
6. Review & compare – Open the MLflow UI (mlflow ui) to see a table of runs, pick the best AUC, and use git checkout <commit> + dvc checkout to retrieve the exact data/model that produced it.

Common Mistakes

• Mistake: Committing large CSVs or model binaries directly to Git.
  Correction: Use DVC (or Git LFS) to store heavy files; Git should only contain lightweight code and .dvc pointers.

• Mistake: Logging the same metric under different names (e.g., “accuracy” vs “acc”).
  Correction: Standardize metric keys across runs; MLflow UI groups runs by exact key strings, so inconsistency hides comparisons.

• Mistake: Relying on a single local mlruns folder and forgetting to push it to a remote.
  Correction: Configure a remote backend store (SQL or S3) early; mlflow server --backend-store-uri sqlite:///mlflow.db or mlflow.set_tracking_uri("s3://my-bucket/mlflow").

• Mistake: Changing data preprocessing code without updating the DVC stage dependencies.
  Correction: List every script, config, and data file in deps; DVC will detect changes and re‑run the stage automatically.

• Mistake: Assuming the latest Git commit always matches the best MLflow run.
  Correction: Tag the commit that produced a given run (git tag -a v1.2 -m "best AUC"), or store the Git SHA as an MLflow tag (mlflow.set_tag("git_sha", repo.head.commit.hexsha)).

Data Science Interview / Practical Insights

1. Why combine Git with DVC? – Interviewers expect you to explain that Git handles code versioning, while DVC adds data and pipeline versioning, keeping the repo lightweight and enabling reproducible pipelines.
2. MLflow vs. DVC tracking: – Be ready to contrast: MLflow is a run‑centric server for metrics/parameters; DVC is pipeline‑centric and focuses on data lineage. Both can coexist; a typical answer: “MLflow logs experiment metadata, DVC guarantees that the exact data used in that experiment is versioned.”
3. Remote storage choices: – Discuss trade‑offs: S3 offers durability and parallel download; Azure Blob may be cheaper for EU regions; local NFS is fine for small teams but lacks versioning guarantees.
4. Handling secret credentials: – Mention using .env files, Git‑ignored, and configuring DVC remote with dvc remote modify --local myremote auth basic or MLflow with environment variables (MLFLOW_S3_ENDPOINT_URL).

Quick Check Questions

1. Scenario: You added a new feature engineering script but the model performance didn’t change.
   Answer: Check the DVC pipeline – the stage may be cached because DVC thinks inputs are unchanged. Run dvc repro -f to force re‑execution.

2. Scenario: Two MLflow runs have identical AUC scores, but one uses a different random seed.
   Answer: Log the seed as a tag (mlflow.set_tag("seed", 42)) and compare the full parameter set; reproducibility requires the seed to be part of the run metadata.

3. Scenario: Your Git history is clean, but mlflow ui shows 30 runs with the same parameters.
   Answer: You likely logged multiple runs without committing the code changes; each run points to the same Git SHA, so differentiate runs by adding a unique identifier (e.g., timestamp) as a tag.

Last‑Minute Cram Sheet (10 one‑liners)

1. Git + DVC = code + data reproducibility – never store >100 KB files directly in Git.
2. dvc add creates a .dvc file; commit that, not the data itself.
3. MLflow run → mlflow.start_run() → log_params, log_metrics, log_artifact.
4. mlflow.log_metric("loss", val, step=epoch) records a time‑series for curve plotting.
5. ⚠️ DVC caches data by content hash; moving a file without dvc move breaks the cache.
6. mlflow.set_tag("git_sha", repo.head.commit.hexsha) ties a run to a Git commit.
7. dvc repro only re‑runs stages whose deps changed – great for fast iteration.
8. Remote back‑ends: mlflow server --backend-store-uri sqlite:///mlflow.db for local, postgresql://... for production.
9. ⚠️ Forgetting .gitignore for .dvc/tmp leads to noisy diffs and large repo pushes.
10. mlflow.search_runs(filter_string="metrics.AUC > 0.85") quickly finds top models.