Cloud ML - Google Cloud Professional Machine Learning Engineer: Choosing Infrastructure (Vertex AI, Compute Engine, GKE, Cloud Run)

GCP_ML – Choosing Infrastructure (Vertex AI, Compute Engine, GKE, Cloud Run)

Google Cloud Professional Machine Learning Engineer Study Guide: Choosing Infrastructure (Vertex AI, Compute Engine, GKE, Cloud Run)

What This Is

This topic covers how to select the right Google Cloud infrastructure for ML workloads—whether training models, serving predictions, or running batch jobs. The right choice impacts cost, scalability, latency, and operational overhead. Real-world scenario: A fintech company needs to deploy a fraud detection model that processes 10,000 transactions per second with <100ms latency. Should they use Vertex AI Endpoints, Cloud Run, or GKE? The answer depends on model size, traffic patterns, and team expertise.

Key Terms & Services

• Vertex AI: GCP’s unified ML platform for training, deploying, and monitoring models. Best for managed ML workflows (e.g., AutoML, custom training, batch/online predictions).
• Compute Engine (GCE): GCP’s virtual machines (VMs). Best for custom, long-running workloads (e.g., training large models, running Jupyter notebooks).
• Google Kubernetes Engine (GKE): GCP’s managed Kubernetes service. Best for scalable, containerized ML workloads (e.g., microservices, distributed training, A/B testing).
• Cloud Run: GCP’s serverless container platform. Best for stateless, event-driven ML services (e.g., REST APIs, lightweight inference).
• Vertex AI Pipelines: GCP’s managed ML pipeline orchestrator (based on Kubeflow). Best for automating ML workflows (e.g., data prep → training → deployment).
• Vertex AI Training: GCP’s managed training service (supports custom containers). Best for scaling training jobs without managing VMs.
• Vertex AI Prediction: GCP’s managed model serving (supports online/batch predictions). Best for low-latency inference with auto-scaling.
• Preemptible VMs: Short-lived, low-cost VMs (up to 80% cheaper). Best for fault-tolerant workloads (e.g., batch training, hyperparameter tuning).
• Spot VMs: GCP’s preemptible VMs with a 30-second warning (vs. AWS Spot Instances). Best for interruptible workloads (e.g., distributed training).
• GPU/TPU Acceleration: Hardware accelerators for deep learning. Best for training large models (e.g., LLMs, CNNs).
• Cold Start: Latency when scaling from zero (common in serverless). Critical for real-time inference (e.g., Cloud Run vs. GKE).
• Model Serving Tradeoff: Latency vs. cost vs. scalability (e.g., Vertex AI Endpoints for managed serving vs. GKE for custom control).

Step-by-Step / Process Flow

1. Assess Workload Requirements

• Is it training or inference?
• Training → Vertex AI Training, GKE, or Compute Engine (GPU/TPU).
• Inference → Vertex AI Prediction, Cloud Run, or GKE.
• What’s the traffic pattern?
• Spiky/burst traffic → Cloud Run or Vertex AI Prediction (auto-scaling).
• Steady traffic → GKE or Compute Engine (fixed capacity).
• What’s the latency requirement?
• <100ms → Vertex AI Prediction or GKE (warm containers).
• >1s → Cloud Run or batch predictions.

2. Choose the Right Service

3. Configure for Cost & Performance

• Training:
• Use Spot VMs for fault-tolerant jobs (e.g., hyperparameter tuning).
• Use Vertex AI Training for managed jobs (avoid VM overhead).
• Inference:
• Cloud Run for low-cost, sporadic traffic (scales to zero).
• Vertex AI Prediction for high traffic (auto-scaling, GPUs).
• GKE for custom serving (e.g., Triton, KServe).
• Storage:
• Cloud Storage for training data (cheap, durable).
• Vertex AI Feature Store for real-time features (low-latency lookups).

4. Deploy & Monitor

• Vertex AI:
• Deploy model → create Endpoint → test with gcloud ai endpoints predict.
• Monitor with Vertex AI Model Monitoring (drift, latency).
• GKE:
• Deploy with KServe or Triton Inference Server.
• Use Horizontal Pod Autoscaler (HPA) for traffic spikes.
• Cloud Run:
• Deploy container → set min/max instances (avoid cold starts).
• Use Cloud Logging/Monitoring for observability.

Common Mistakes

1. Mistake: Using Cloud Run for High-Traffic Inference

• Why it’s wrong: Cloud Run has cold starts and concurrency limits (default: 80 requests per container).
• Correction: Use Vertex AI Prediction or GKE for high-throughput, low-latency workloads.

2. Mistake: Using Compute Engine for Real-Time Inference

• Why it’s wrong: VMs are not auto-scaling and require manual management.
• Correction: Use Vertex AI Prediction or Cloud Run for serverless, auto-scaling inference.

3. Mistake: Overprovisioning GKE for Simple Models

• Why it’s wrong: GKE has high operational overhead (cluster management, networking).
• Correction: Use Cloud Run for lightweight models (e.g., scikit-learn, small PyTorch).

4. Mistake: Ignoring Spot VMs for Training

• Why it’s wrong: On-demand VMs are 3-5x more expensive than Spot VMs.
• Correction: Use Spot VMs for fault-tolerant training (e.g., hyperparameter tuning).

5. Mistake: Not Using Vertex AI for Managed Workflows

• Why it’s wrong: Building custom pipelines on Compute Engine/GKE adds complexity.
• Correction: Use Vertex AI Pipelines for automated ML workflows (data → train → deploy).

Certification Exam Insights

1. Service Selection Traps

• Vertex AI vs. GKE:
• Vertex AI = managed, easy (best for most use cases).
• GKE = custom, complex (best for advanced serving, e.g., Triton, KServe).
• Cloud Run vs. Vertex AI Prediction:
• Cloud Run = cheap, scales to zero (best for sporadic traffic).
• Vertex AI Prediction = low-latency, auto-scaling (best for high traffic).
• Compute Engine vs. Vertex AI Training:
• Compute Engine = cheaper for long jobs (but manual management).
• Vertex AI Training = managed, auto-scaling (best for most training jobs).

2. Key Constraints to Know

• Vertex AI Prediction:
• Max model size: 10GB (for online predictions).
• Cold start: ~5-10s (mitigate with min instances).
• Cloud Run:
• Max request timeout: 60 minutes (default: 5 minutes).
• Concurrency: Default 80 requests per container (adjustable).
• GKE:
• GPU quotas: Must request GPU quota in GCP.
• Networking: Requires VPC-native clusters for private IPs.

3. "Which Service?" Scenarios

Quick Check Questions

1. A startup needs to deploy a fraud detection model that processes 500 requests per second with <50ms latency. They have a small team and want minimal DevOps overhead. Which service should they use?

✅ Answer: Vertex AI Prediction
Why? Managed, auto-scaling, low-latency, and requires minimal setup.

2. A data science team wants to train a large language model (LLM) on 100TB of text data. They need full control over the training environment and want to minimize costs. Which service should they use?

✅ Answer: Compute Engine with Spot VMs (GPU/TPU)
Why? Full OS access, cheaper than managed services, and Spot VMs reduce costs.

3. A company wants to deploy a recommendation model as a REST API. Traffic is unpredictable, with peaks of 10,000 requests per minute. They want to avoid cold starts. Which service should they use?

✅ Answer: Vertex AI Prediction (with min instances set to 1)
Why? Auto-scaling, low latency, and min instances prevent cold starts.

Last-Minute Cram Sheet

1. Vertex AI Prediction = managed, low-latency inference (best for most real-time use cases).
2. Cloud Run = serverless, scales to zero (best for sporadic traffic).
3. GKE = custom serving (Triton, KServe) (best for advanced use cases).
4. Compute Engine = cheap for long training jobs (but manual management).
5. Vertex AI Training = managed training (no VMs to manage).
6. Spot VMs = 80% cheaper (but can be preempted).
7. Vertex AI Batch Prediction = no infrastructure (best for one-time jobs).
8. Cold starts = Cloud Run/Vertex AI (mitigate with min instances).
9. ⚠️ Vertex AI Prediction max model size = 10GB (for online predictions).
10. ⚠️ Cloud Run max timeout = 60 minutes (default: 5 minutes).