Cloud ML - Google Cloud Professional Machine Learning Engineer: Cost Optimization (Preemptible VMs, Spot VMs, Commitments, Reducing Serving Costs)

GCP_ML – Cost Optimization (Preemptible VMs, Spot VMs, Commitments, Reducing Serving Costs)

Google Cloud Professional Machine Learning Engineer: Cost Optimization Study Guide

Topic: Preemptible VMs, Spot VMs, Commitments, and Reducing Serving Costs

What This Is

Cost optimization in GCP ML pipelines is about balancing performance, reliability, and expense—especially for compute-heavy workloads like training deep learning models, batch inference, or real-time serving. For example, a fintech company running real-time fraud detection on Vertex AI might use Spot VMs for training models (saving 80% on compute) but switch to reserved commitments for their always-on prediction endpoints to avoid cold starts. Poor cost choices can balloon cloud bills (e.g., leaving GPUs idle) or degrade user experience (e.g., preemptible VMs killing a long-running job).

Key Terms & Services

• Preemptible VMs (GCP):
  Short-lived (max 24h) Compute Engine VMs that can be terminated by GCP with 30-second notice. Best for: Fault-tolerant workloads like batch training, hyperparameter tuning, or data preprocessing. Cost: Up to 80% cheaper than regular VMs.

• Spot VMs (GCP):
  Similar to preemptible VMs but with no 24-hour limit and a dynamic pricing model (billed per second). Best for: Longer-running jobs (e.g., distributed training) where interruptions are acceptable. Key difference: Spot VMs use a market-based pricing model (like AWS Spot Instances).

• Committed Use Discounts (CUDs):
  1- or 3-year commitments for Compute Engine or GPU resources in exchange for up to 70% discount. Best for: Predictable workloads (e.g., always-on Vertex AI endpoints, production inference). Trap: Must commit to specific machine types/regions.

• Sustained Use Discounts (SUDs):
  Automatic discounts (up to 30%) for VMs running >25% of a month. Best for: Workloads with variable but consistent usage (e.g., nightly batch jobs). No commitment required.

• Vertex AI Prediction Endpoints:
  Managed serving infrastructure for ML models. Cost drivers: Instance type (CPU/GPU), autoscaling settings, and request volume. Optimization: Use autoscaling to 0 for dev/test, batch predictions for offline jobs, and custom containers for cost-efficient runtimes.

• Batch Predictions (Vertex AI):
  Offline inference for large datasets (e.g., processing millions of images overnight). Cost: Pay per prediction (no idle compute). Best for: Non-real-time use cases (e.g., monthly customer churn analysis).

• Custom Containers (Vertex AI):
  Bring your own Docker image for prediction endpoints. Optimization: Use lightweight runtimes (e.g., FastAPI + ONNX) to reduce memory/CPU needs. Trap: Over-provisioning resources (e.g., requesting 8 vCPUs when 2 suffice).

• Cloud TPUs:
  Google’s custom hardware for training deep learning models (e.g., transformers, CNNs). Cost: Pay per TPU-hour (cheaper than GPUs for large-scale training). Best for: Models with heavy matrix operations (e.g., LLMs, image segmentation).

• Cloud GPUs (NVIDIA A100/T4):
  Accelerators for training/inference. Cost: Expensive (e.g., A100 ~$2.93/hour). Optimization: Use preemptible GPUs for training, reserved commitments for production.

• BigQuery ML:
  Train ML models directly in BigQuery (no separate VMs needed). Cost: Pay per query + model storage. Best for: Simple models (linear regression, k-means) on structured data.

• Dataflow (Apache Beam):
  Serverless batch/streaming data processing. Cost: Pay per vCPU/memory/hour. Optimization: Use FlexRS for batch jobs (cheaper than streaming) and autoscaling.

• Cloud Functions / Cloud Run:
  Serverless options for lightweight inference (e.g., REST APIs). Cost: Pay per request + compute time. Best for: Low-latency, sporadic traffic (e.g., chatbot responses).

Step-by-Step / Process Flow

1. Choosing the Right Compute for Training

Scenario: You’re training a PyTorch model on 100GB of image data. The job takes ~6 hours but can restart if interrupted.
Steps:
1. Estimate job duration: Use gcloud compute instances list to check historical runtimes.
2. Select a preemptible/Spot VM:
- For short jobs (<24h): gcloud compute instances create my-trainer --preemptible --machine-type=n2-standard-8 --accelerator=type=nvidia-tesla-t4,count=1
- For longer jobs: gcloud compute instances create my-trainer --provisioning-model=SPOT --machine-type=n2-standard-8 3. Enable checkpointing: Save model weights every 30 minutes to Cloud Storage (gs://my-bucket/checkpoints/).
4. Use a managed service (if possible):
- For Vertex AI Training, set training_job.preemptible=True in the Python SDK.
5. Monitor costs: Use Cloud Billing reports to compare preemptible vs. on-demand spend.

2. Reducing Serving Costs for Vertex AI Endpoints

Scenario: Your production endpoint serves 100K predictions/day with 90% traffic during business hours.
Steps:
1. Right-size the instance:
- Test with n1-standard-2 (2 vCPUs) vs. n1-standard-4 using Vertex AI’s online prediction latency metrics.
- Use custom containers to optimize runtime (e.g., FastAPI + ONNX instead of TensorFlow Serving).
2. Autoscale efficiently:
- Set min_replica_count=1 (to avoid cold starts) and max_replica_count=10.
- Use CPU utilization (not request count) as the scaling metric (e.g., scale at 60% CPU).
3. Use batch predictions for offline jobs:
- For monthly reports, run gcloud ai batch-prediction-jobs create instead of hitting the endpoint.
4. Commit to CUDs for production:
- Purchase a 1-year commitment for the n1-standard-4 machine type in your region.
5. Monitor and adjust:
- Use Cloud Monitoring to track aiplatform.googleapis.com/prediction/latency and cost_per_prediction.

3. Optimizing Data Processing Costs

Scenario: You’re preprocessing 1TB of text data for an NLP model using Dataflow.
Steps:
1. Choose the right runner:
- For batch jobs: Use Dataflow Prime (cheaper than streaming) with runner=DataflowRunner.
- For streaming: Use FlexRS (autoscaling) with flexRSGoal=COST_OPTIMIZED.
2. Optimize worker settings:
- Set worker_machine_type=n1-standard-4 (balance cost/performance).
- Use num_workers=10 and autoscalingAlgorithm=THROUGHPUT_BASED.
3. Leverage Spot VMs for workers:
- Add --worker_harness_container_image=gcr.io/cloud-dataflow/v1beta3/spot-worker-harness to use Spot VMs.
4. Cache intermediate data:
- Store processed data in BigQuery or Cloud Storage to avoid reprocessing.
5. Monitor and kill idle jobs:
- Set a Cloud Scheduler to cancel jobs running >24h.

Common Mistakes

Mistake 1: Using Preemptible VMs for Production Serving

• Why it’s wrong: Preemptible VMs can terminate anytime, causing downtime for user-facing endpoints.
• Correction: Use reserved commitments (CUDs) or autoscaling Vertex AI endpoints for production. Preemptible VMs are only for fault-tolerant training jobs.

Mistake 2: Over-Provisioning Vertex AI Endpoints

• Why it’s wrong: Requesting 8 vCPUs when the model only needs 2 wastes money (e.g., $0.50/hour vs. $0.10/hour).
• Correction: Benchmark latency with different machine types (e.g., n1-standard-2 vs. n1-standard-4) and choose the smallest instance that meets SLA.

Mistake 3: Ignoring Batch Predictions for Offline Jobs

• Why it’s wrong: Running batch jobs via online endpoints (e.g., REST API) is 10x more expensive than Vertex AI Batch Predictions.
• Correction: Use gcloud ai batch-prediction-jobs create for offline inference (pay per prediction, no idle compute).

Mistake 4: Not Using Custom Containers for Lightweight Models

• Why it’s wrong: Default Vertex AI runtimes (e.g., TensorFlow Serving) are bloated and expensive for simple models.
• Correction: Deploy models in FastAPI + ONNX or Triton Inference Server to reduce memory/CPU usage.

Mistake 5: Forgetting to Shut Down Idle Resources

• Why it’s wrong: Leaving GPUs or VMs running 24/7 can cost thousands/month (e.g., A100 GPU ~$2,100/month if idle).
• Correction: Use Cloud Scheduler + Cloud Functions to shut down non-production resources outside business hours.

Certification Exam Insights

1. Preemptible vs. Spot VMs: Know the Difference

• Exam trap: The exam may ask when to use preemptible vs. Spot VMs.
• Preemptible: Max 24h runtime, 30s termination notice, fixed discount (~80%).
• Spot: No time limit, dynamic pricing, terminated with 30s notice.
• Answer: Use preemptible for short jobs (<24h), Spot for longer jobs (e.g., distributed training).

2. CUDs vs. SUDs: When to Commit

• Exam trap: "Should you use Committed Use Discounts (CUDs) or Sustained Use Discounts (SUDs) for a production endpoint?"
• CUDs: Best for predictable, always-on workloads (e.g., Vertex AI endpoints, production databases).
• SUDs: Best for variable but consistent workloads (e.g., nightly batch jobs).
• Answer: Use CUDs for production endpoints (1- or 3-year commitment).

3. Vertex AI Endpoint Autoscaling: CPU vs. Requests

• Exam trap: "Should you scale Vertex AI endpoints based on CPU utilization or request count?"
• CPU utilization: Better for compute-heavy models (e.g., LLMs, CNNs).
• Request count: Better for lightweight models (e.g., linear regression).
• Answer: Use CPU utilization for most ML models (avoids over-provisioning).

4. Batch vs. Online Predictions: Cost Tradeoffs

• Exam trap: "A company needs to process 1M images monthly for a report. Should they use Vertex AI Batch Predictions or an online endpoint?"
• Batch Predictions: Cheaper (pay per prediction, no idle compute), but not real-time.
• Online Endpoint: More expensive (idle compute), but low-latency.
• Answer: Use Batch Predictions for offline jobs (e.g., monthly reports).

Quick Check Questions

Question 1

A startup is training a PyTorch model on 500GB of image data. The job takes ~8 hours but can restart if interrupted. Which GCP service offers the lowest cost with minimal management overhead? Answer: Vertex AI Training with preemptible VMs.
Why: Preemptible VMs are 80% cheaper than on-demand, and Vertex AI Training handles job orchestration.

Question 2

A production Vertex AI endpoint serves 50K predictions/day with 90% traffic between 9 AM–5 PM. How can you reduce costs without affecting latency? Answer: Use autoscaling with min_replica_count=1 and max_replica_count=10, and purchase a 1-year CUD for the machine type.
Why: Autoscaling reduces idle costs, and CUDs provide a 70% discount for predictable workloads.

Question 3

A data team runs a weekly Dataflow job to preprocess 1TB of text data. The job takes ~6 hours. How can they reduce costs? Answer: Use Dataflow Prime with FlexRS and Spot VMs for workers.
Why: FlexRS optimizes for cost, and Spot VMs reduce worker costs by up to 80%.

Last-Minute Cram Sheet

1. Preemptible VMs: Max 24h runtime, 30s termination notice, 80% discount. ⚠️ Not for production serving.
2. Spot VMs: No time limit, dynamic pricing, 30s termination notice. ⚠️ Use for long-running training jobs.
3. CUDs: 1- or 3-year commitments, up to 70% discount. ⚠️ Must commit to specific machine types/regions.
4. SUDs: Automatic discounts (up to 30%) for VMs running >25% of a month. ⚠️ No commitment, but unpredictable.
5. Vertex AI Batch Predictions: Pay per prediction, no idle compute. ⚠️ Not real-time.
6. Custom Containers: Reduce serving costs by using lightweight runtimes (e.g., FastAPI + ONNX). ⚠️ Benchmark latency before deploying.
7. Autoscaling Vertex AI Endpoints: Scale based on CPU utilization (not request count) for ML models. ⚠️ Set min_replica_count=1 to avoid cold starts.
8. Cloud TPUs: Cheaper than GPUs for large-scale training (e.g., LLMs). ⚠️ Not all frameworks support TPUs (e.g., PyTorch requires XLA).
9. Dataflow FlexRS: Cost-optimized batch processing. ⚠️ Use Spot VMs for workers.
10. BigQuery ML: Train simple models (e.g., linear regression) without VMs. ⚠️ Limited to structured data.