Data Science and Machine Learning 101: Model Deployment and MLOps Model Serving REST APIs FlaskFastAPI gRPC Batch vs Online Inference

What This Is

Model serving is the bridge that takes a trained machine‑learning model and makes it available to other software (web apps, mobile apps, downstream pipelines) via an API. In production you expose the model as a REST endpoint, a FastAPI/Flask service, or a gRPC method, then send new data to get predictions on‑demand (online) or in bulk (batch).
Why it matters: without a reliable serving layer, a model that scores 0.95 AUC in notebooks never reaches users.
Real‑world example: a SaaS churn‑prediction model is trained nightly, saved as churn.pkl, and a Flask micro‑service returns {"prob":0.82} for each incoming customer record, enabling the sales team to prioritize outreach in real time.

Key Terms & Formulas

• REST (Representational State Transfer) – HTTP‑based protocol; each request is stateless and identified by a URL (e.g., POST /predict).
• gRPC (Google Remote Procedure Call) – Binary‑protocol using Protocol Buffers; lower latency than REST, ideal for high‑throughput inference.
• Online Inference – Predict on a single request as it arrives (e.g., a user uploads an image → immediate classification).
• Batch Inference – Predict on a large dataset at once, usually scheduled (e.g., nightly scoring of all active users).
• Latency = t_response – t_request – Time from request arrival to response; critical for real‑time UI.
• Throughput = #predictions / second – Number of predictions the service can sustain; drives scaling decisions.
• Docker Image = FROM python:3.11 + COPY model.pkl /app/ + CMD ["uvicorn","app:app","--host","0.0.0.0"] – Container that packages code, model, and dependencies for reproducible deployment.
• Cold Start – First request after a service spin‑up; often slower because the model must be loaded into memory.
• A/B Testing (Model Versioning) – Serve two model versions (v1, v2) to subsets of traffic, compare KPI lift (e.g., conversion rate).
• Circuit Breaker Pattern – Stops forwarding requests to a failing model service, returns fallback response, and prevents cascade failures.
• Model Serialization – joblib.dump(model, "model.pkl") or tf.saved_model.save(model, "saved_model/"); stores the trained weights and metadata for later loading.

Step‑by‑Step / Process Flow

1. Export the trained model
   python
# sklearn example
import joblib
joblib.dump(best_clf, "model.pkl")
# TensorFlow example
tf.saved_model.save(tf_model, "saved_model/")
2. Create a lightweight API (Flask or FastAPI) that loads the model once at startup and defines a /predict endpoint.
   
   ```python
   from fastapi import FastAPI
   import joblib, pandas as pd

app = FastAPI()
model = joblib.load("model.pkl") # cold‑start load

@app.post("/predict")
def predict(payload: dict):
df = pd.DataFrame([payload])
prob = model.predict_proba(df)[:, 1][0]
return {"probability": float(prob)}
3. Containerize the service with Docker, pinning Python and library versions.dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "80"]
`` 4. Deploy to a cloud platform (Kubernetes, AWS ECS, GCP Cloud Run). Set horizontal pod autoscaling on CPU or request latency. 5. Monitor & Log – expose Prometheus metrics (/metrics`), log request IDs, latency, and error rates. Hook alerts to PagerDuty for > 95th‑percentile latency breaches.
6. Iterate – when a new model version is ready, push a new Docker image, roll out via a canary deployment, and compare KPI (e.g., churn lift) before full promotion.

Common Mistakes

Data Science Interview / Practical Insights

1. “When would you choose gRPC over a REST endpoint?” – Expect discussion of binary payload size, streaming, and sub‑millisecond latency (e.g., real‑time video inference).
2. “Explain the trade‑off between batch and online inference.” – Highlight latency vs. resource utilization, cost (spot instances for batch), and data freshness.
3. “How do you handle model drift in a serving pipeline?” – Talk about monitoring feature distributions, automated retraining triggers, and versioned APIs.
4. “What is a ‘cold‑start’ problem and how can you mitigate it?” – Mention warm‑up requests, pre‑loading models in a sidecar container, or using serverless platforms with provisioned concurrency.

Quick Check Questions

1. Scenario: Your churn API latency spikes to 2 seconds after a weekend of heavy traffic.
   Answer: Scale out the service (add more pods) and enable a warm‑up request to keep the model in memory.
   Why: More replicas share the load; warm‑up avoids repeated cold starts.

2. Scenario: You need to serve a 10‑GB recommendation model to 5 K RPS with < 50 ms latency.
   Answer: Use gRPC with a GPU‑enabled pod and enable request batching (e.g., batch size = 32).
   Why: Binary protocol reduces overhead; batching amortizes GPU kernel launch cost.

3. Scenario: A client sends a JSON payload with a missing feature column.
   Answer: Return a 400 error with a clear validation message; do not fall back to default predictions.
   Why: Silent failures corrupt downstream decisions and hide data‑quality issues.

Last‑Minute Cram Sheet (10 one‑liners)

1. REST = stateless HTTP; gRPC = protobuf + HTTP/2, ~10× lower latency.
2. Cold start = model load time; mitigate with warm‑up requests or provisioned concurrency.
3. Latency = response – request; keep < 100 ms for UI‑critical apps.
4. Throughput = predictions / second; scale horizontally on CPU or GPU metrics.
5. Docker = reproducible environment; always pin exact library versions.
6. FastAPI auto‑generates OpenAPI docs → easy client SDK generation.
7. Batch inference → schedule (cron) or Spark job; reduces per‑record overhead.
8. Circuit breaker → stop sending traffic to a failing model service (fallback to previous version).
9. Model versioning → tag Docker images (churn:v1.2.3) and route traffic via canary releases.
10. ⚠️ Never expose raw model objects (e.g., sklearn estimator) over the network; always serialize predictions, not the model itself.