Cloud ML - Google Cloud Professional Machine Learning Engineer: Deployment Strategies (Vertex AI Endpoints, Batch Predictions, Edge TPU)

GCP_ML – Deployment Strategies (Vertex AI Endpoints, Batch Predictions, Edge TPU)

Google Cloud Professional Machine Learning Engineer Study Guide: Deployment Strategies

Topic: Vertex AI Endpoints, Batch Predictions, Edge TPU

What This Is

Deployment strategies determine how your trained ML model serves predictions—whether in real-time (e.g., fraud detection in banking transactions), batch (e.g., nightly credit risk scoring for millions of customers), or on edge devices (e.g., a retail store’s camera detecting shelf stockouts without cloud dependency). Google Cloud’s Vertex AI provides managed endpoints for low-latency inference, batch prediction jobs for offline processing, and Edge TPU for ultra-low-latency, offline inference on IoT devices. Mastering these ensures your model meets business SLAs (e.g., <100ms latency for ad recommendations) while optimizing cost and scalability.

Key Terms & Services

• Vertex AI Endpoints: GCP’s fully managed service for deploying ML models as REST APIs. Handles autoscaling, A/B testing, and traffic splitting (e.g., canary deployments). Best for real-time inference (e.g., chatbots, fraud detection).
• Vertex AI Batch Predictions: GCP’s serverless service for running predictions on large datasets (e.g., CSV/JSON files in Cloud Storage) without managing infrastructure. Ideal for offline tasks like monthly churn prediction reports.
• Edge TPU: Google’s custom ASIC (Application-Specific Integrated Circuit) for running TensorFlow Lite models on edge devices (e.g., Coral Dev Board, USB Accelerator). Optimized for ultra-low latency (<1ms) and offline inference (e.g., industrial quality control cameras).
• Model Serving: The process of hosting a trained model to serve predictions. Vertex AI Endpoints handle this for online serving; Batch Predictions for offline.
• Traffic Splitting: A Vertex AI Endpoint feature to route a percentage of traffic to different model versions (e.g., 90% to v1, 10% to v2 for A/B testing).
• Quantization: Reducing model precision (e.g., float32 → int8) to speed up inference and reduce memory usage. Critical for Edge TPU deployments.
• TF Lite: TensorFlow’s lightweight framework for mobile/edge devices. Required for Edge TPU compatibility.
• Autoscaling: Vertex AI Endpoints automatically scale based on request volume (e.g., 0 to 1000+ instances during peak traffic). Configured via minReplicaCount and maxReplicaCount.
• Prediction Logs: Vertex AI can log prediction inputs/outputs to BigQuery for monitoring and compliance (e.g., auditing fraud detection decisions).
• Custom Containers: Vertex AI supports custom Docker containers for models not in its pre-built frameworks (e.g., PyTorch, XGBoost). Useful for legacy or proprietary models.
• Cold Start: The delay when a Vertex AI Endpoint scales from 0 to 1 replica. Mitigate by setting minReplicaCount=1 for critical low-latency apps.
• VPC-SC (VPC Service Controls): GCP’s security feature to restrict Vertex AI access to a private network (e.g., prevent data exfiltration to public endpoints).

Step-by-Step / Process Flow

1. Deploying a Model to Vertex AI Endpoints (Real-Time)

Scenario: Deploy a fraud detection model to serve predictions in <50ms for a banking app.
1. Train & Export Model:
- Train your model (e.g., XGBoost, TensorFlow) and save it in a supported format (e.g., model.joblib, saved_model.pb).
- Upload to Google Cloud Storage (GCS) in a bucket (e.g., gs://my-bucket/models/fraud-detection-v1).
2. Create a Model in Vertex AI:
- In the GCP Console, navigate to Vertex AI > Models > Create.
- Select the model artifact from GCS and specify the framework (e.g., "Scikit-learn" or "TensorFlow").
- Configure explanation settings (optional) for feature attribution (e.g., SHAP values).
3. Deploy to an Endpoint:
- Create an Endpoint (Vertex AI > Endpoints > Create).
- Deploy the model to the endpoint with:
- Machine type (e.g., n1-standard-4 for CPU, n1-standard-16 for GPU).
- Traffic split (e.g., 100% to v1 initially).
- Autoscaling (e.g., minReplicaCount=1, maxReplicaCount=10).
4. Test & Monitor:
- Send a test request via the REST API or Python SDK:
python
from google.cloud import aiplatform
endpoint = aiplatform.Endpoint("projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID")
prediction = endpoint.predict(instances=[{"feature1": 0.5, "feature2": 1.2}])
- Monitor latency, errors, and traffic in Cloud Monitoring and Vertex AI Dashboard.

2. Running Batch Predictions

Scenario: Score 10M customer records nightly for credit risk.
1. Prepare Input Data:
- Store input data in GCS (e.g., gs://my-bucket/batch-input/2023-10-01.csv).
- Ensure the file format matches the model’s expected schema (e.g., CSV with columns age, income, credit_score).
2. Create a Batch Prediction Job:
- In Vertex AI, navigate to Batch Predictions > Create.
- Select the model and input/output GCS paths.
- Configure:
- Machine type (e.g., n1-standard-4 for small jobs, n1-highmem-16 for large datasets).
- Output format (e.g., JSONL, CSV).
- Explanations (optional, for interpretability).
3. Run & Monitor:
- The job runs asynchronously. Monitor progress in the Vertex AI Jobs tab.
- Output is saved to GCS (e.g., gs://my-bucket/batch-output/2023-10-01/).
4. Post-Processing:
- Load results into BigQuery for analysis or trigger downstream workflows (e.g., email alerts for high-risk customers).

3. Deploying to Edge TPU

Scenario: Deploy a retail shelf-monitoring model to a Coral Dev Board for real-time stockout detection.
1. Convert Model to TF Lite:
- Use TensorFlow’s TFLiteConverter to quantize the model (e.g., float32 → int8):
python
converter = tf.lite.TFLiteConverter.from_saved_model("saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
with open("model_quant.tflite", "wb") as f:
f.write(tflite_model) 2. Compile for Edge TPU:
- Use the Edge TPU Compiler to optimize the model:
bash
edgetpu_compiler model_quant.tflite -o output_dir
- Output: model_quant_edgetpu.tflite.
3. Deploy to Device:
- Flash the Coral Dev Board with the latest Mendel OS (Google’s Linux distro for Edge TPU).
- Copy the compiled model to the device:
bash
scp model_quant_edgetpu.tflite mendel@coral-dev:/home/mendel/ 4. Run Inference:
- Use the Coral Python API to load and run the model:
```python
from pycoral.adapters import common
from pycoral.adapters import classify
from pycoral.utils.edgetpu import make_interpreter

 interpreter = make_interpreter("model_quant_edgetpu.tflite")
 interpreter.allocate_tensors()
 common.set_input(interpreter, image)  # Preprocessed input
 interpreter.invoke()
 classes = classify.get_classes(interpreter, top_k=1)
 ```

Common Mistakes

Certification Exam Insights

1. Service Selection Traps:
2. Vertex AI Endpoints vs. Batch Predictions: Know that Endpoints are for real-time (<1s latency), while Batch Predictions are for offline (hours/days) jobs. The exam may ask, "Which service for a nightly report on 5M records?" → Batch Predictions.

3. Edge TPU vs. Vertex AI Endpoints: Edge TPU is for offline/edge (e.g., IoT cameras), while Endpoints are for cloud-based real-time. The exam may ask, "Which for a factory floor quality control camera?" → Edge TPU.

4. Key Constraints:

5. Edge TPU Model Size: Max 8MB for compiled models. Larger models won’t fit.
6. Batch Prediction Limits: Max 100K rows per file (split larger datasets into multiple files).

7. Endpoint Autoscaling: Default cooldown period is 5 minutes (adjust for bursty traffic).

8. Tricky Scenarios:

9. Canary Deployments: The exam may ask how to roll out a new model version to 10% of users. Answer: Traffic splitting in Vertex AI Endpoints.

10. Cost Optimization: For sporadic traffic, use Vertex AI Endpoints with minReplicaCount=0 (but accept cold starts). For predictable traffic, set minReplicaCount=1.

11. Security Questions:

12. VPC-SC: If the exam asks how to restrict Vertex AI access to a private network, the answer is VPC Service Controls.
13. Prediction Logging: To audit model decisions, enable prediction logs to BigQuery in the Endpoint config.

Quick Check Questions

1. A fintech company needs to deploy a fraud detection model with <100ms latency for 10K requests/second. Which GCP service should they use?
2. Answer: Vertex AI Endpoints (with autoscaling and minReplicaCount=1 to avoid cold starts).

3. Explanation: Endpoints are designed for real-time, low-latency inference with autoscaling.

4. A retail chain wants to run a one-time batch inference job on 50M customer records stored in BigQuery. What’s the most cost-effective approach?

5. Answer: Export data from BigQuery to GCS, then use Vertex AI Batch Predictions.

6. Explanation: Batch Predictions are serverless and cheaper than running a VM for hours.

7. An industrial IoT company needs to deploy a TensorFlow model to 10K edge devices with <1ms latency. Which GCP service should they use?

8. Answer: Edge TPU (with a quantized TF Lite model).
9. Explanation: Edge TPU is optimized for ultra-low-latency, offline inference on edge devices.

Last-Minute Cram Sheet

1. Vertex AI Endpoints: Real-time inference, autoscaling, traffic splitting, REST API. ⚠️ Cold starts if minReplicaCount=0.
2. Batch Predictions: Offline inference, reads from GCS, writes to GCS. Max 100K rows/file.
3. Edge TPU: For edge devices, requires int8-quantized TF Lite models. Max 8MB model size.
4. Traffic Splitting: Use for A/B testing (e.g., 90% v1, 10% v2).
5. Autoscaling: Configure minReplicaCount (0 for cost savings, 1 for low latency).
6. Prediction Logs: Enable in Endpoint config to log to BigQuery for auditing.
7. Custom Containers: Supported in Vertex AI for non-standard frameworks (e.g., PyTorch).
8. VPC-SC: Restrict Vertex AI access to private networks.
9. Cold Start Mitigation: Set minReplicaCount=1 for critical endpoints.
10. Edge TPU Compiler: Required to optimize models for Edge TPU. ⚠️ Forgetting this causes runtime errors.