Cloud ML - Google Cloud Professional Machine Learning Engineer: Scaling and Latency (Autoscaling, Node Pools, Nvidia Triton)

GCP_ML – Scaling and Latency (Autoscaling, Node Pools, Nvidia Triton)

Google Cloud Professional Machine Learning Engineer Study Guide: Scaling and Latency (Autoscaling, Node Pools, Nvidia Triton)

What This Is

Scaling and latency optimization ensure ML models handle variable workloads efficiently while meeting real-time inference demands. This is critical in real-world scenarios like: - Fraud detection in banking (low-latency predictions on millions of daily transactions).
- Recommendation engines (scaling to handle Black Friday traffic spikes).
- LLM-powered chatbots (serving thousands of concurrent users with sub-100ms response times).

Google Cloud provides autoscaling, custom node pools, and Nvidia Triton to balance cost, performance, and reliability in production ML systems.

Key Terms & Services

• Vertex AI Prediction (Online Prediction):
  GCP’s managed service for deploying ML models as REST endpoints. Supports autoscaling (CPU/GPU) and custom containers (e.g., Triton). Best for low-latency, high-throughput inference.

• Autoscaling in Vertex AI:
  Dynamically adjusts the number of prediction nodes based on CPU/GPU utilization, QPS (queries per second), or custom metrics. Reduces cost during low traffic and prevents throttling during spikes.

• Node Pools (GKE & Vertex AI):
  A group of VMs with identical configurations (e.g., GPU type, machine type). Used in Vertex AI Prediction and GKE to isolate workloads (e.g., CPU for preprocessing, GPU for inference).

• Nvidia Triton Inference Server:
  Open-source model-serving framework optimized for GPU acceleration, dynamic batching, and multi-model serving. Deployed in Vertex AI custom containers or GKE. Best for high-performance, low-latency inference (e.g., LLMs, CV models).

• GPU Types in GCP:

• NVIDIA T4: Cost-effective for batch inference (e.g., nightly predictions).
• NVIDIA A100: High-performance for real-time inference (e.g., fraud detection, LLMs).

• NVIDIA L4: Balanced option for moderate workloads (e.g., recommendation systems).

• Vertex AI Pipelines:
  Orchestrates ML workflows (training, deployment, scaling). Can trigger autoscaling based on pipeline metrics (e.g., "Scale up if inference latency > 50ms").

• Cloud Load Balancing (GCLB):
  Distributes traffic across multiple regions for global low-latency inference. Works with Vertex AI endpoints to route requests to the nearest healthy node.

• Batch Prediction vs. Online Prediction:

• Batch: Process large datasets offline (e.g., nightly recommendations). Uses Vertex AI Batch Prediction (no autoscaling).

• Online: Real-time predictions (e.g., fraud detection). Uses Vertex AI Online Prediction (with autoscaling).

• Custom Containers in Vertex AI:
  Allows deploying custom inference code (e.g., Triton, FastAPI). Required for non-standard models (e.g., PyTorch, ONNX, TensorRT).

• Model Warm-Up:
  Pre-loading a model into memory before traffic spikes to avoid cold-start latency. Configured in Vertex AI or Triton.

• Dynamic Batching (Triton):
  Groups multiple inference requests into a single batch to maximize GPU utilization. Reduces latency for high-throughput workloads.

• Vertex AI Model Monitoring:
  Tracks prediction drift, latency, and errors to trigger autoscaling or retraining.

Step-by-Step / Process Flow

1. Deploying a Scalable Model with Vertex AI Online Prediction

Scenario: A fintech company needs to deploy a fraud detection model with <100ms latency and autoscaling for traffic spikes.

1. Train & Export Model
2. Train a model (e.g., XGBoost, TensorFlow) and export it in a supported format (SavedModel, ONNX, or custom container).

3. Example:
   bash
gcloud ai models upload --region=us-central1 --display-name=fraud-detection --container-image-uri=us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-12:latest

4. Create a Vertex AI Endpoint

5. Define an endpoint (logical container for models).

6. Example:
   bash
gcloud ai endpoints create --region=us-central1 --display-name=fraud-endpoint

7. Deploy Model to Endpoint with Autoscaling

8. Configure autoscaling (min/max nodes, CPU/GPU, traffic-based scaling).
9. Example (CLI):
   bash
gcloud ai endpoints deploy-model ENDPOINT_ID \
--region=us-central1 \
--model=MODEL_ID \
--machine-type=n1-standard-4 \
--min-replica-count=1 \
--max-replica-count=10 \
--traffic-split=0=100

10. Key Parameters:

    
    
    • --min-replica-count: Minimum nodes (cost control).
    • --max-replica-count: Maximum nodes (prevents runaway costs).
    • --machine-type: n1-standard-4 (CPU) or n1-standard-4-gpu (GPU).
    • --accelerator: type=nvidia-tesla-t4,count=1 (for GPU).

11. Test & Monitor

12. Send a test request:
    bash
gcloud ai endpoints predict ENDPOINT_ID --region=us-central1 --json-request=request.json

13. Monitor latency, QPS, and errors in Vertex AI Model Monitoring.

14. Optimize with Nvidia Triton (Optional)

15. If using custom models (e.g., PyTorch, TensorRT), deploy with Triton in a custom container.
16. Example Dockerfile:
    dockerfile
FROM nvcr.io/nvidia/tritonserver:23.10-py3
COPY model_repository /models
17. Deploy to Vertex AI:
    bash
gcloud ai endpoints deploy-model ENDPOINT_ID \
--region=us-central1 \
--model=MODEL_ID \
--container-image-uri=gcr.io/PROJECT_ID/triton-server:latest \
--machine-type=n1-standard-4-gpu \
--accelerator=type=nvidia-tesla-t4,count=1

2. Setting Up Node Pools for Multi-Model Serving

Scenario: A retail company runs two models (recommendations + fraud detection) with different hardware needs.

1. Create a GKE Cluster with Node Pools
2. Example:
   bash
gcloud container clusters create ml-cluster \
--region=us-central1 \
--node-pools=pool-cpu,pool-gpu \
--machine-type=e2-standard-4 \
--num-nodes=1

3. Add a GPU node pool:
   bash
gcloud container node-pools create pool-gpu \
--cluster=ml-cluster \
--region=us-central1 \
--machine-type=n1-standard-4 \
--accelerator=type=nvidia-tesla-t4,count=1 \
--num-nodes=1

4. Deploy Models to Separate Node Pools

5. Use Kubernetes labels to route traffic:
   yaml
# recommendation-deployment.yaml
spec:
nodeSelector:
cloud.google.com/gke-nodepool: pool-cpu
yaml
# fraud-deployment.yaml
spec:
nodeSelector:
cloud.google.com/gke-nodepool: pool-gpu

6. Autoscale Node Pools

7. Enable cluster autoscaler:
   bash
gcloud container clusters update ml-cluster \
--enable-autoscaling \
--min-nodes=1 \
--max-nodes=10 \
--region=us-central1

Common Mistakes

Certification Exam Insights

What the Exam Tests

1. Autoscaling Triggers
2. Know the difference between CPU-based, QPS-based, and custom metric scaling.

3. Example: "A model’s latency spikes during peak hours. Which autoscaling metric should you use?" → Answer: Custom metric (latency).

4. GPU Selection

5. T4 (cost-effective) vs. A100 (high-performance).

6. Example: "A company needs to serve a BERT model with <50ms latency. Which GPU should they use?" → Answer: A100.

7. Vertex AI vs. GKE for Inference

8. Vertex AI (managed, simpler) vs. GKE (customizable, complex).

9. Example: "A team needs to deploy a Triton server with custom CUDA kernels. Should they use Vertex AI or GKE?" → Answer: GKE (Vertex AI doesn’t support custom CUDA).

10. Multi-Model Serving

11. Triton (supports multiple models in one container) vs. Vertex AI (one model per endpoint).

12. Example: "A company wants to serve 10 models from a single endpoint. Which service should they use?" → Answer: Triton in GKE.

13. Cost Optimization

14. Spot VMs (cheaper, preemptible) vs. On-Demand (reliable).
15. Example: "A batch inference job runs nightly. Which VM type is most cost-effective?" → Answer: Spot VMs.

Quick Check Questions

Question 1

A fintech company deploys a fraud detection model on Vertex AI. During peak hours, latency spikes to 500ms. They want to maintain <100ms latency. What should they do? Answer: Enable autoscaling based on latency and increase --max-replica-count.
Why? Autoscaling adjusts nodes dynamically to handle traffic spikes.

Question 2

A retail company runs a recommendation model (CPU) and a fraud model (GPU) on the same GKE cluster. The fraud model is starved for GPU resources. What’s the fix? Answer: Create separate node pools for CPU and GPU workloads.
Why? Isolating workloads prevents resource contention.

Question 3

A team deploys a PyTorch model using Vertex AI’s default container. Inference latency is 300ms. They want to reduce it to <50ms. What should they do? Answer: Deploy the model in Nvidia Triton with dynamic batching.
Why? Triton optimizes GPU inference for low latency.

Last-Minute Cram Sheet

1. Vertex AI Online Prediction = Managed, autoscaling, low-latency inference.
2. Autoscaling metrics: CPU, QPS, or custom metrics (e.g., latency).
3. T4 GPU = Cost-effective for batch; A100 = High-performance for real-time.
4. Triton = Best for multi-model serving, dynamic batching, GPU optimization.
5. Node pools = Isolate CPU/GPU workloads to avoid contention.
6. Cold starts = Mitigate with model warm-up or --min-replica-count=1.
7. Batch vs. Online Prediction = Batch = offline; Online = real-time.
8. Spot VMs = Cheaper but preemptible (use for non-critical batch jobs).
9. ⚠️ Vertex AI doesn’t support custom CUDA kernels → Use GKE + Triton.
10. ⚠️ --min-replica-count=0 saves cost but adds cold-start latency.