Cloud ML - Google Cloud Professional Machine Learning Engineer: A/B Testing and Canary Deployments

GCP_ML – A/B Testing and Canary Deployments

Google Cloud Professional Machine Learning Engineer: A/B Testing & Canary Deployments – Exam-Ready Study Guide

What This Is

A/B testing and canary deployments are controlled rollout strategies for ML models in production. Instead of replacing an old model with a new one instantly (risking performance drops or outages), you gradually expose users to the new model while monitoring key metrics (latency, accuracy, business KPIs). This is critical in ML pipelines where model behavior is probabilistic—even a "better" model in training may fail in production due to data drift, edge cases, or unintended biases.

Real-world scenario:
A ride-hailing app wants to deploy a new dynamic pricing model (predicting surge pricing based on demand, weather, and traffic). Instead of rolling it out to all users at once, they canary-deploy the model to 5% of users in a single city, monitor ride acceptance rates and driver earnings, and compare against the baseline model. If metrics improve, they A/B test the model in 50% of another region before full deployment.

Key Terms & Services

• Vertex AI Model Deployment (Endpoints):
  GCP’s managed service for hosting ML models (TensorFlow, PyTorch, scikit-learn, custom containers). Supports traffic splitting for A/B testing and canary deployments. Best for low-latency, scalable inference.

• Vertex AI Model Monitoring:
  Tracks prediction drift, feature skew, and latency in real time. Critical for detecting if a canary model is underperforming before full rollout.

• Cloud Load Balancing (GCP):
  Distributes traffic between model versions (e.g., 90% to v1, 10% to v2). Works with Vertex AI Endpoints to route requests based on rules (e.g., user ID, geography).

• Cloud Logging & Cloud Monitoring:
  Captures prediction logs, latency, and errors for both model versions. Used to compare performance during A/B tests.

• BigQuery ML:
  Runs statistical significance tests (t-tests, chi-square) on A/B test results stored in BigQuery. Helps determine if observed differences are real or due to randomness.

• Traffic Splitting (Vertex AI):
  Configures % of requests sent to each model version (e.g., 5% to canary, 95% to baseline). Adjustable in real time via API or console.

• Shadow Mode (Dark Launch):
  Deploys a new model alongside the old one but discards its predictions (or logs them for comparison). Low-risk way to test model behavior without affecting users.

• Multi-Armed Bandit (MAB):
  An adaptive A/B testing algorithm that dynamically shifts traffic to better-performing models (e.g., Vertex AI’s Optimization AI). Reduces regret (lost revenue) compared to fixed-split A/B tests.

• Feature Store (Vertex AI Feature Store):
  Ensures consistent feature serving across model versions during A/B tests. Prevents skew from different preprocessing logic.

• Dataflow (Apache Beam):
  Processes real-time prediction logs for A/B test analysis (e.g., aggregating conversion rates by model version).

• Statistical Significance (p-value):
  In A/B testing, a p-value < 0.05 means the observed difference is likely real (not due to chance). BigQuery ML can compute this automatically.

• Canary vs. A/B Testing:

• Canary: Small % of users, focused on stability (latency, errors).
• A/B: Larger % of users, focused on business metrics (conversion, revenue).

Step-by-Step / Process Flow

1. Deploy Baseline & Canary Models to Vertex AI

• Train & upload both models to Vertex AI Model Registry.
• Create a Vertex AI Endpoint and deploy:
• Baseline model (v1) with 100% traffic initially.
• Canary model (v2) with 0% traffic (or shadow mode).
• Enable Model Monitoring for both versions (drift, skew, latency).

2. Configure Traffic Splitting

• Use Vertex AI’s traffic splitting to route:
• 5% of requests to the canary model (v2).
• 95% to baseline (v1).
• (Optional) Use Cloud Load Balancing for more granular routing (e.g., by user ID, region).

3. Log Predictions & Monitor Metrics

• Stream prediction logs to BigQuery (via Vertex AI’s built-in logging or Dataflow).
• Track key metrics in Cloud Monitoring:
• Latency (p99, p50).
• Error rates (4xx/5xx responses).
• Business KPIs (e.g., conversion rate, revenue per user).
• Set up alerts for anomalies (e.g., canary model latency > 200ms).

4. Run Statistical Analysis

• Query BigQuery to compare metrics between v1 and v2.
• Use BigQuery ML to run a t-test (for continuous metrics) or chi-square test (for categorical metrics).
  sql -- Example: Compare conversion rates between v1 and v2 SELECT
model_version,
AVG(conversion_rate) as avg_conversion,
COUNT(*) as sample_size FROM `project.dataset.prediction_logs` GROUP BY model_version
• Check p-value: If p < 0.05, the difference is statistically significant.

5. Gradually Increase Traffic (Canary → A/B → Full Rollout)

• If canary model performs well:
• Increase traffic to 20% (A/B test phase).
• Monitor for 1-2 weeks (longer for low-traffic systems).
• If metrics improve:
• Shift 100% traffic to v2.
• Retire v1 (or keep as fallback).

6. Automate with Vertex AI Pipelines (Optional)

• Use Vertex AI Pipelines to:
• Automate model deployment (e.g., trigger canary on new model version).
• Run A/B test analysis (BigQuery ML + Cloud Functions).
• Auto-roll back if metrics degrade (e.g., latency > threshold).

Common Mistakes

Mistake 1: Not Logging Enough Data for A/B Tests

• Problem: Only logging predictions, not user actions (e.g., clicks, purchases).
• Correction:
• Log both predictions and downstream events (e.g., "user saw recommendation → clicked → purchased").
• Use BigQuery’s nested/repeated fields to link predictions to outcomes.
• Why? Without this, you can’t measure business impact (e.g., revenue lift).

Mistake 2: Ignoring Statistical Significance

• Problem: Declaring a model "better" after seeing a 2% lift in 100 users.
• Correction:
• Run tests until p < 0.05 (or use sequential testing to stop early if results are clear).
• Use BigQuery ML’s ML.TTEST to automate significance testing.
• Why? Small sample sizes lead to false positives (Type I errors).

Mistake 3: Canary Testing Without Monitoring Latency

• Problem: Only monitoring accuracy, not p99 latency.
• Correction:
• Set up Cloud Monitoring alerts for latency spikes.
• Test under load (e.g., 10x normal traffic) before canary.
• Why? A model with 99% accuracy but 10s latency is unusable.

Mistake 4: Using Fixed Traffic Splits for Long-Running Tests

• Problem: Keeping 5% traffic on canary for months, even if it’s clearly worse.
• Correction:
• Use Multi-Armed Bandit (MAB) (Vertex AI Optimization AI) to dynamically shift traffic to better models.
• Why? Fixed splits waste traffic on bad models (high "regret").

Mistake 5: Not Having a Rollback Plan

• Problem: Deploying a canary model without a way to quickly revert if it fails.
• Correction:
• Keep the old model deployed (even at 0% traffic) for quick rollback.
• Automate rollback (e.g., Cloud Function triggered by latency alerts).
• Why? Manual rollbacks take time—automation prevents outages.

Certification Exam Insights

1. Traffic Splitting Service Selection

• Trap: Candidates confuse Vertex AI Endpoints (for model hosting) with Cloud Load Balancing (for traffic routing).
• Vertex AI Endpoints supports built-in traffic splitting (simpler, managed).
• Cloud Load Balancing is needed for advanced routing (e.g., by user ID, geography).
• Exam Tip: If the question mentions model versions and % traffic, pick Vertex AI Endpoints. If it mentions header-based routing, pick Cloud Load Balancing.

2. A/B Testing vs. Shadow Mode

• Trap: Candidates think shadow mode is the same as A/B testing.
• Shadow mode: New model runs alongside old one, but predictions are discarded (or logged for comparison). No user impact.
• A/B testing: New model serves real users (5-50% traffic). User impact exists.
• Exam Tip: If the question asks for "zero-risk testing", pick shadow mode. If it asks for "measuring business impact", pick A/B testing.

3. Statistical Significance in BigQuery

• Trap: Candidates forget that BigQuery ML can run t-tests and chi-square tests directly in SQL.
• Correct Approach: Use ML.TTEST or ML.CHI_SQUARE_TEST instead of exporting data to Python.
• Why? The exam tests GCP-native tools, not external libraries.

4. Multi-Armed Bandit (MAB) vs. Fixed A/B Tests

• Trap: Candidates default to fixed A/B tests even when the question mentions minimizing regret.
• MAB (Vertex AI Optimization AI): Dynamically shifts traffic to better models. Lower regret.
• Fixed A/B: Static traffic split. Higher regret if one model is clearly worse.
• Exam Tip: If the question mentions "adaptive traffic allocation", pick MAB.

Quick Check Questions

1. A fintech company wants to deploy a new fraud detection model. They need to test it on 1% of transactions without affecting users. Which GCP service should they use?

• Answer: Vertex AI Endpoints with shadow mode.
• Why? Shadow mode runs the new model alongside the old one but discards its predictions (or logs them for comparison), ensuring zero user impact.

2. An e-commerce site is A/B testing two recommendation models. They want to automatically shift more traffic to the better-performing model. Which GCP service should they use?

• Answer: Vertex AI Optimization AI (Multi-Armed Bandit).
• Why? MAB dynamically allocates traffic to better models, reducing regret compared to fixed A/B tests.

3. A team is running an A/B test on a new pricing model. They need to determine if the 3% revenue lift is statistically significant. Which GCP service should they use?

• Answer: BigQuery ML (ML.TTEST).
• Why? BigQuery ML can run t-tests directly on A/B test data stored in BigQuery, avoiding the need to export data to Python.

Last-Minute Cram Sheet

1. Vertex AI Endpoints = Managed model hosting + traffic splitting (A/B, canary).
2. Shadow mode = Test new model without user impact (predictions discarded/logged).
3. Canary deployment = 5-10% traffic, focus on stability (latency, errors).
4. A/B testing = 20-50% traffic, focus on business metrics (conversion, revenue).
5. Multi-Armed Bandit (MAB) = Adaptive A/B testing, shifts traffic to better models.
6. BigQuery ML = Run t-tests/chi-square tests directly in SQL for A/B analysis.
7. Cloud Monitoring = Track latency, errors, drift during canary/A/B tests.
8. ⚠️ Vertex AI Endpoints traffic splitting = Not for header-based routing (use Cloud Load Balancing instead).
9. ⚠️ Statistical significance = p < 0.05 (or use sequential testing to stop early).
10. ⚠️ Rollback plan = Always keep the old model deployed (even at 0% traffic) for quick reversion.