Cloud ML - Google Cloud Professional Machine Learning Engineer: Designing ML Pipelines (Training, Evaluation, Deployment)

GCP_ML – Designing ML Pipelines (Training, Evaluation, Deployment)

Google Cloud Professional Machine Learning Engineer Study Guide: Designing ML Pipelines (Training, Evaluation, Deployment)

What This Is

Designing an end-to-end ML pipeline in Google Cloud means orchestrating data ingestion, feature engineering, model training, evaluation, deployment, and monitoring in a scalable, reproducible, and automated way. A real-world scenario: A retail company wants to deploy a real-time recommendation system that personalizes product suggestions based on user behavior. The pipeline must: - Ingest streaming clickstream data (Pub/Sub → Dataflow → BigQuery).
- Compute and store features (Vertex AI Feature Store).
- Train and evaluate models (Vertex AI Training with custom containers).
- Deploy for low-latency inference (Vertex AI Endpoints with autoscaling).
- Monitor for drift and performance decay (Vertex AI Model Monitoring).

A poorly designed pipeline leads to data leakage, training-serving skew, high latency, or cost overruns—all of which the exam tests.

Key Terms & Services

• Vertex AI Pipelines
  GCP’s serverless orchestration service for ML workflows (built on Kubeflow Pipelines). Best for automating training, evaluation, and deployment with pre-built components (e.g., AutoMLTrainingJobOp, ModelDeployOp).

• Vertex AI Training
  GCP’s managed training service for custom models (TensorFlow, PyTorch, XGBoost) or AutoML. Supports custom containers, distributed training (TPUs/GPUs), and hyperparameter tuning (Vertex AI Vizier).

• Vertex AI Feature Store
  GCP’s centralized repository for storing, sharing, and serving ML features. Online store (low-latency for inference) + offline store (batch training). Prevents training-serving skew by ensuring consistent feature computation.

• Vertex AI Model Registry
  A versioned repository for trained models. Tracks metadata (metrics, artifacts, lineage) and enables A/B testing via model aliases (e.g., champion, challenger).

• Vertex AI Endpoints
  GCP’s managed inference service for deploying models. Supports online prediction (REST API), batch prediction (async), and autoscaling. Can deploy multiple models per endpoint (traffic splitting for A/B tests).

• Vertex AI Model Monitoring
  GCP’s drift detection service for deployed models. Monitors feature skew, prediction drift, and custom metrics. Alerts trigger retraining pipelines.

• Cloud Dataflow
  GCP’s serverless stream/batch processing (Apache Beam). Used for ETL, feature engineering, and real-time data prep before training/inference.

• BigQuery ML
  GCP’s SQL-based ML service for training models directly in BigQuery (e.g., CREATE MODEL with linear regression, XGBoost). Best for quick prototyping on structured data.

• TensorFlow Extended (TFX) on Vertex AI
  GCP’s managed TFX service for production-grade pipelines. Includes data validation (TFDV), feature engineering (TF Transform), and model analysis (TFMA).

• Kubeflow Pipelines (KFP)
  Open-source ML orchestration framework (alternative to Vertex AI Pipelines). More flexible but requires GKE cluster management.

• Training-Serving Skew
  When features differ between training and inference (e.g., computed differently, stale data). Vertex AI Feature Store mitigates this by ensuring consistent feature computation.

• Data Leakage
  When future data "leaks" into training (e.g., using test-set statistics for feature scaling). TFX Data Validation or custom Dataflow jobs can detect this.

Step-by-Step / Process Flow

1. Design the Pipeline Architecture

• Scenario: Build a real-time fraud detection system for a fintech app.
• Steps:
• Ingest data:
  • Streaming: Pub/Sub → Dataflow (Apache Beam) → BigQuery (for batch training) + Vertex AI Feature Store (for online features).
  • Batch: Cloud Storage → BigQuery (for historical training).
• Feature engineering:
  • Dataflow computes real-time features (e.g., "transaction velocity in last 5 mins").
  • Vertex AI Feature Store stores features for consistent training/inference.
• Train & evaluate:
  • Vertex AI Training (custom container with XGBoost) + Vertex AI Vizier (hyperparameter tuning).
  • TFX/TFMA for model validation (e.g., check for data drift).
• Deploy:
  • Vertex AI Endpoint (autoscaled, traffic splitting for A/B tests).
• Monitor:
  • Vertex AI Model Monitoring (alerts on feature/prediction drift).

2. Implement a Vertex AI Pipeline

• Steps:

• Define components (Python SDK):
  ```python
  from google.cloud import aiplatform
  from kfp.v2 import dsl

  @dsl.component def train_model(data_path: str, model_dir: str):
  # Custom training code (e.g., TensorFlow)
  pass

  @dsl.pipeline(name="fraud-detection-pipeline") def pipeline(project: str, region: str):
  train_task = train_model(data_path="gs://bucket/data", model_dir="gs://bucket/model")
  deploy_task = aiplatform.ModelDeployOp(
  model=train_task.outputs["model"],
  endpoint_name="fraud-endpoint"
  ) 2. Compile & run:bash pipeline.compile("pipeline.json") aiplatform.PipelineJob.from_json("pipeline.json").run() ``` 3. Schedule runs (Cloud Scheduler + Cloud Functions for triggers).

3. Deploy for Low-Latency Inference

• Steps:
• Upload model to Vertex AI Model Registry:
  python
model = aiplatform.Model.upload(
display_name="fraud-model",
artifact_uri="gs://bucket/model",
serving_container_image_uri="us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-12:latest"
)
• Create an endpoint:
  python
endpoint = model.deploy(
machine_type="n1-standard-4",
min_replica_count=1,
max_replica_count=10 # Autoscaling
)
• Test predictions:
  python
endpoint.predict(instances=[{"feature1": 0.5, "feature2": 1.2}])

4. Monitor for Drift

• Steps:
• Enable monitoring in Vertex AI:
  python
endpoint = aiplatform.Endpoint("projects/123/locations/us-central1/endpoints/456")
endpoint.deployed_model_monitoring_job.create(
display_name="fraud-drift-monitor",
objective_configs=[
aiplatform.ObjectiveConfig(
feature_skew_detection_config=aiplatform.FeatureSkewDetectionConfig(
skew_thresholds={"feature1": 0.1}
)
)
]
)
• Set up alerts (Cloud Monitoring → Pub/Sub → Cloud Functions for retraining).

Common Mistakes

Certification Exam Insights

1. Service Selection Traps:
2. Vertex AI Pipelines vs. Kubeflow Pipelines:
   • Use Vertex AI Pipelines for serverless, managed orchestration (exam favorite).
   • Use Kubeflow Pipelines if you need custom GKE clusters (rare in exam).

3. Vertex AI Training vs. BigQuery ML:

   
   
   • Vertex AI Training for custom models (PyTorch, TensorFlow, XGBoost).
   • BigQuery ML for quick SQL-based models (linear regression, XGBoost) on structured data.

4. Key Constraints:

5. Vertex AI Endpoints have a default quota of 100 requests per second (RPS). Request increases via GCP support.

6. Vertex AI Feature Store has a max online store size of 100K features per entity type.

7. Tricky Scenarios:

8. "Which service ensures consistent features between training and inference?"
   → Vertex AI Feature Store (not BigQuery or Cloud Storage).

9. "How do you automate retraining when drift is detected?"
   → Vertex AI Model Monitoring → Pub/Sub → Cloud Functions → Vertex AI Pipeline trigger.

10. Cost Optimization:

11. Batch predictions (Vertex AI Batch Prediction) are cheaper than online endpoints for large-scale inference.
12. Preemptible VMs in Vertex AI Training reduce costs by ~80% (but may fail).

Quick Check Questions

1. A retail company wants to deploy a recommendation model that updates hourly with new user behavior data. Which GCP service ensures features are computed consistently for both training and real-time inference?
2. Answer: Vertex AI Feature Store.

3. Why: It provides a centralized feature repository with online/offline stores to prevent training-serving skew.

4. Your team needs to automate a pipeline that trains a model weekly, evaluates it, and deploys only if accuracy improves. Which GCP service should you use for orchestration?

5. Answer: Vertex AI Pipelines.

6. Why: It’s serverless, integrates with Vertex AI services, and supports conditional logic (e.g., if accuracy > threshold: deploy).

7. A fraud detection model deployed on Vertex AI Endpoints is experiencing high latency during traffic spikes. What’s the most cost-effective way to handle this?

8. Answer: Enable autoscaling (set min_replica_count=1, max_replica_count=10).
9. Why: Autoscaling dynamically adjusts replicas based on traffic, avoiding over-provisioning.

Last-Minute Cram Sheet

1. Vertex AI Pipelines = serverless Kubeflow (use for automation, not custom GKE clusters).
2. Vertex AI Feature Store = online + offline features (prevents training-serving skew).
3. Vertex AI Endpoints = autoscaling, traffic splitting, A/B testing.
4. Vertex AI Model Monitoring = feature skew + prediction drift alerts.
5. BigQuery ML = SQL-based training (not for real-time inference).
6. TFX on Vertex AI = production-grade pipelines (data validation, feature engineering).
7. ⚠️ Vertex AI Training vs. Vertex AI AutoML:
8. Training = custom code (PyTorch, TensorFlow).
9. AutoML = no-code (tables, vision, NLP).
10. ⚠️ Vertex AI Endpoints max RPS = 100 (request quota increase if needed).
11. Batch predictions = cheaper than online endpoints (use for large-scale async inference).
12. Preemptible VMs = 80% cheaper for training (but may fail—use for non-critical jobs).