Data Science and Machine Learning 101: Model Deployment and MLOps CICD for ML Pipeline Automation Testing Models Model Registry

What This Is

CI/CD for ML (sometimes called MLOps) is the practice of automatically building, testing, version‑controlling, and deploying machine‑learning code and models. It turns a manual “train‑then‑hand‑off” workflow into a repeatable pipeline so that new data, code changes, or model improvements can be pushed to production with the same confidence you have for a web‑app release.

Real‑world example: A telecom company predicts customer churn daily. Every night a pipeline pulls the latest usage logs, retrains a Gradient‑Boosted Tree, runs a suite of validation tests, registers the new model, and rolls it out to the scoring service without a human touching the code.

Key Terms & Formulas

• CI (Continuous Integration) – Automatically merge and test code changes (e.g., a new feature engineering script) on every push to main.
• CD (Continuous Delivery / Deployment) – Automatically move a validated model artifact from staging to production (or just make it ready for production).
• Pipeline DAG – Directed acyclic graph that orders steps (e.g., extract → transform → train → evaluate → register). Tools: Apache Airflow, Prefect, Dagster.
• Model Registry – Central store (e.g., MLflow, Sage‑Maker Model Registry) that tracks model versions, signatures, and lineage.
• Test‑set Leakage – Accidentally using test data in training; Metric = (TP + TN) / (TP + FP + FN + TN) is meaningless if leakage occurs.
• Canary Deployment – Deploy a new model to a small traffic slice (e.g., 5 %) and compare live metrics before full rollout.
• Drift Detection (Population Stability Index):

[ \text{PSI} = \sum_{i=1}^{k} (p_i - q_i) \cdot \ln!\left(\frac{p_i}{q_i}\right) ]

where (p_i) = proportion of feature values in reference data, (q_i) = proportion in current data. PSI > 0.2 signals significant drift.
- ML‑specific Unit Test – Assert that a model’s AUC‑ROC on a hold‑out set stays above a threshold (e.g., assert auc > 0.78).
- Versioned Data Artifacts – Store raw, cleaned, and feature‑engineered datasets with immutable hashes (e.g., DVC dvc add data/train.parquet).
- Infrastructure as Code (IaC) – Declare compute resources (Dockerfile, Terraform) so the same environment runs locally and in CI.
- Rollback Strategy – Keep the previous model version ID; on failure, mlflow models revert <prev_version> restores service instantly.

Step‑by‑Step / Process Flow

1. Create a reproducible repo – git init; add requirements.txt, Dockerfile, and a Makefile that runs pytest and flake8.
2. Define the pipeline as code – e.g., with Airflow:

python
@dag(schedule_interval='@daily')
def churn_pipeline():
raw = extract()
clean = transform(raw)
model = train(clean)
eval = evaluate(model, clean)
register_if_ok(eval, model)

1. Add automated tests
2. Unit: assert isinstance(df, pd.DataFrame) for each step.
3. Integration: after train, compute np.mean(y_pred == y_true).
4. Performance: assert roc_auc_score(y_test, prob) > 0.80.
5. Push to CI – GitHub Actions runs the test suite inside the Docker image; on success, builds a model artifact (model.pkl) and pushes it to the MLflow Registry.
6. Deploy via CD – A CD job pulls the latest approved model, spins up a TensorFlow Serving container, runs a canary on 5 % of traffic, monitors drift (PSI) and latency; if metrics stay green, promote to 100 % traffic.

Common Mistakes

• Mistake: Skipping data versioning – treating raw CSVs as mutable.
  Correction: Use DVC or Git‑LFS to lock each dataset with a hash; this guarantees reproducibility when a model is rebuilt.

• Mistake: Testing only on training data (e.g., model.score(X_train, y_train)).
  Correction: Reserve a hold‑out or cross‑validation set that the CI pipeline evaluates; training‑set metrics are optimistic.

• Mistake: Hard‑coding paths or credentials inside the pipeline code.
  Correction: Pull configuration from environment variables or a secret manager (e.g., os.getenv("S3_BUCKET")).

• Mistake: Deploying a model that passes offline tests but never monitors live performance.
  Correction: Add a post‑deployment monitor (A/B test, drift alerts) as part of the CD stage; auto‑rollback if live KPI degrades.

• Mistake: Using a single large Docker image for all steps, leading to long build times.
  Correction: Split the pipeline into lightweight stages (e.g., separate builder and runtime images) and cache layers in CI to speed up builds.

Data Science Interview / Practical Insights

1. “Explain the difference between CI and CD in an ML context.” – Interviewers expect you to discuss code integration vs. model delivery (including registry, canary, and rollback).
2. “How would you test for data drift before redeploying a model?” – Mention statistical tests (PSI, KS test) on feature distributions and a monitoring dashboard that triggers a retrain.
3. “What’s the role of a model registry, and why is it better than just saving a pickle file?” – Emphasize lineage (code + data + hyper‑params), stage transitions (Staging → Production), and reproducibility.

Quick Check Questions

1. Scenario: Your nightly CI run fails because mlflow.log_metric("auc", auc) raises a ValueError.
   Answer: The failure is likely due to logging a metric before the run is started; wrap logging inside with mlflow.start_run(): ….

2. Scenario: After a canary rollout, the latency spikes from 30 ms to 120 ms.
   Answer: Trigger an automatic rollback to the previous model version and investigate whether the new model introduced heavier preprocessing (e.g., high‑dimensional embeddings).

Last‑Minute Cram Sheet (10 one‑liners)

1. CI = lint + unit + integration tests on every push; CD = automated packaging + deployment.
2. MLflow Model Registry stages: None → Staging → Production → Archived.
3. PSI > 0.2 ⇒ feature drift; PSI < 0.1 ⇒ stable.
4. Canary deployment: Deploy to a small traffic slice, compare live KPI, then promote.
5. Docker best practice: COPY . /app && pip install -r /app/requirements.txt → cache layers for faster CI builds.
6. ⚠️ Storing raw data in Git: Bad idea; use DVC or object storage instead.
7. Rollback command (MLflow): mlflow models revert -v <prev_version>.
8. Test‑set leakage kills AUC; always split before any preprocessing.
9. IaC tools (Terraform, CloudFormation) let you spin up identical GPU nodes for training in CI.
10. ⚠️ MinMaxScaler vs. StandardScaler: MinMax preserves zero‑inflated features; StandardScaler assumes Gaussian distribution.