Cloud ML - Google Cloud Professional Machine Learning Engineer: Data Ingestion (Cloud Storage, BigQuery, Pub/Sub, Dataflow)

GCP_ML – Data Ingestion (Cloud Storage, BigQuery, Pub/Sub, Dataflow)

Google Cloud Professional Machine Learning Engineer – Data Ingestion Study Guide

Topic: Data Ingestion (Cloud Storage, BigQuery, Pub/Sub, Dataflow)

What This Is

Data ingestion is the first step in any ML pipeline—getting raw data from sources (databases, APIs, IoT devices, logs) into a format and location where it can be cleaned, transformed, and used for training or inference. In GCP, this involves Cloud Storage (for batch files), BigQuery (for structured analytics), Pub/Sub (for real-time streaming), and Dataflow (for scalable ETL). Real-world scenario: A ride-sharing app ingests real-time GPS data (Pub/Sub) to predict surge pricing, while historical trip data (BigQuery) trains demand-forecasting models. Poor ingestion leads to slow pipelines, high costs, or stale features.

Key Terms & Services

• Cloud Storage (GCS): GCP’s object storage (like AWS S3 or Azure Blob Storage). Best for batch files (CSV, Parquet, images), model artifacts, and intermediate pipeline data. Supports lifecycle rules (auto-delete old data) and regional/multi-regional buckets.
• BigQuery: GCP’s serverless data warehouse (like AWS Redshift or Azure Synapse). Best for SQL-based analytics, feature engineering, and training data storage. Uses columnar storage (cheap for queries, expensive for frequent writes).
• Pub/Sub: GCP’s real-time messaging service (like AWS SNS/SQS or Azure Event Hubs). Best for streaming data (e.g., IoT telemetry, clickstreams). Decouples producers (devices/apps) from consumers (Dataflow, BigQuery).
• Dataflow: GCP’s managed Apache Beam service (like AWS Glue or Azure Data Factory). Best for ETL/ELT pipelines (batch or streaming). Uses autoscaling workers and integrates with Pub/Sub, BigQuery, and GCS.
• BigQuery Storage API: Lets Dataflow/Vertex AI read BigQuery tables directly (faster than exporting to GCS first). Critical for large-scale ML training.
• Data Fusion: GCP’s no-code ETL tool (like AWS Glue Studio or Azure Data Factory). Best for drag-and-drop pipelines, but less flexible than Dataflow for ML workloads.
• Dataproc: GCP’s managed Spark/Hadoop service (like AWS EMR or Azure HDInsight). Best for legacy Spark jobs or custom PySpark transformations. Overkill for simple ETL.
• Streaming vs. Batch Ingestion:
• Streaming (Pub/Sub + Dataflow): Low-latency (seconds), higher cost, used for real-time features (e.g., fraud detection).
• Batch (GCS/BigQuery): High-latency (minutes/hours), lower cost, used for training data or offline inference.
• Schema Drift: When incoming data’s structure changes (e.g., new columns). Dataflow can auto-detect this; BigQuery requires schema updates.
• Dead-Letter Queues (DLQ): Pub/Sub/Cloud Functions can route failed messages to a DLQ for reprocessing (critical for fault tolerance).
• Dataflow Templates: Pre-built pipelines (e.g., "Pub/Sub to BigQuery") to avoid writing Beam code from scratch. Saves time but limits customization.
• Vertex AI Feature Store: Stores pre-computed features for low-latency serving (e.g., user embeddings for recommendations). Requires data to be ingested first (via Dataflow or BigQuery).

Step-by-Step / Process Flow

1. Batch Ingestion (GCS → BigQuery)

Use case: Historical data for training (e.g., 10 years of customer transactions).
1. Upload data to GCS:
- Use gsutil cp or Cloud Storage Transfer Service (for large datasets).
- Store in Parquet/ORC (columnar = faster queries) or Avro (schema evolution).
2. Load into BigQuery:
- Use bq load CLI or the BigQuery UI.
- Choose partitioning (e.g., by date) and clustering (e.g., by customer_id) to reduce query costs.
3. Validate data:
- Run SELECT COUNT(*), MIN(date), MAX(date) FROM dataset.table to check for gaps.
- Use BigQuery ML to detect anomalies (e.g., CREATE MODEL ... OPTIONS(model_type='logistic_reg')).

2. Streaming Ingestion (Pub/Sub → Dataflow → BigQuery)

Use case: Real-time fraud detection (e.g., credit card transactions).
1. Set up Pub/Sub:
- Create a topic (e.g., transactions) and a subscription (e.g., fraud-detection-sub).
- Configure message retention (default: 7 days) and acknowledgement deadline (default: 10s).
2. Deploy Dataflow pipeline:
- Use a template (e.g., "Pub/Sub to BigQuery") or write a Beam pipeline (Python/Java).
- Add windowing (e.g., 5-minute tumbling windows) and triggers (e.g., emit results every 30s).
3. Write to BigQuery:
- Use beam.io.WriteToBigQuery with a schema (define columns like transaction_id, amount, timestamp).
- Enable streaming inserts (higher cost) or batch writes (lower cost, higher latency).
4. Monitor and scale:
- Check Dataflow job metrics (e.g., "Elements Added," "System Lag").
- Set autoscaling (default: 1–1,000 workers) or fixed workers (for predictable costs).

3. Hybrid Ingestion (GCS + Pub/Sub → Dataflow → Vertex AI Feature Store)

Use case: Recommendation system with real-time and batch features.
1. Batch features (GCS → BigQuery):
- Load historical user-item interactions (e.g., clicks, purchases) into BigQuery.
- Pre-compute features (e.g., avg_purchase_value) using SQL.
2. Streaming features (Pub/Sub → Dataflow):
- Ingest real-time events (e.g., user_clicked_item) via Pub/Sub.
- Use Dataflow to compute rolling features (e.g., clicks_last_5_minutes).
3. Write to Vertex AI Feature Store:
- Create a feature group (e.g., user_features) with a schema (e.g., user_id, avg_purchase_value).
- Use Dataflow to write batch features (from BigQuery) and streaming features (from Pub/Sub) to the feature store.
4. Serve features:
- Call the feature store from your Vertex AI endpoint for low-latency inference.

Common Mistakes

Certification Exam Insights

1. Service Selection Traps:
2. "When to use Dataflow vs. Dataproc?"
   • Dataflow: Serverless, autoscaling, best for Beam pipelines (ETL, streaming).
   • Dataproc: For Spark/Hadoop jobs (e.g., legacy PySpark MLlib code).

3. "BigQuery vs. Cloud SQL for ML data?"

   
   
   • BigQuery: Petabyte-scale analytics, SQL-based feature engineering.
   • Cloud SQL: Transactional data (e.g., user profiles), not for large-scale ML.

4. Key Constraints:

5. BigQuery streaming inserts have a 100k rows/sec limit per table (use batch loads for higher throughput).
6. Dataflow autoscaling can take 5–10 minutes to adjust workers (set a higher initial worker count for latency-sensitive jobs).

7. Pub/Sub message size limit: 10MB (compress large payloads or split into chunks).

8. "Which Service?" Scenarios:

9. Q: "A team needs to process 1TB of CSV files daily with minimal code. Which GCP service?"
   • A: Data Fusion (no-code ETL) or Dataflow with a template (e.g., "GCS to BigQuery").

10. Q: "A fraud detection model needs sub-second feature updates. Which ingestion pipeline?"

    
    
    • A: Pub/Sub → Dataflow → Vertex AI Feature Store (streaming + low-latency serving).

11. Cost Traps:

12. BigQuery: Charges for storage (cheap) and query bytes scanned (expensive). Use partitioning/clustering to reduce costs.
13. Dataflow: Charges for vCPU/hour and GB processed. Use batch mode (cheaper) for non-real-time jobs.

Quick Check Questions

1. A retail company wants to analyze 5 years of sales data (10TB) for demand forecasting. The data is in CSV files on GCS. Which GCP service should they use to transform and query this data efficiently?
2. Answer: BigQuery (serverless, SQL-based, cost-effective for large-scale analytics).

3. Why? BigQuery is optimized for petabyte-scale queries, and CSV files can be loaded directly with partitioning/clustering.

4. A gaming app streams player events (e.g., "level_completed") to a Pub/Sub topic. The ML team needs to compute rolling features (e.g., "games_played_last_hour") and store them for real-time inference. Which GCP services should they use?

5. Answer: Pub/Sub → Dataflow → Vertex AI Feature Store.

6. Why? Pub/Sub for streaming, Dataflow for windowed aggregations, and Feature Store for low-latency serving.

7. A data engineer notices that their Dataflow job fails when new columns are added to the Pub/Sub messages. How can they make the pipeline resilient to schema changes?

8. Answer: Use schema auto-detection in Dataflow or write a custom Beam pipeline with dynamic schema handling.
9. Why? Default templates assume a fixed schema; auto-detection or custom code can adapt to changes.

Last-Minute Cram Sheet

1. Cloud Storage (GCS): Object storage for batch files; use Parquet/ORC for ML data. ⚠️ Not for real-time access (use Firestore or Bigtable instead).
2. BigQuery: Serverless data warehouse; partition by date, cluster by ID to reduce costs. ⚠️ Streaming inserts cost 10x more than batch loads.
3. Pub/Sub: Real-time messaging; 10MB message limit, no ordering by default. ⚠️ Use ordering keys for sequential data (e.g., stock prices).
4. Dataflow: Managed Beam; autoscaling workers (1–1,000), supports batch/streaming. ⚠️ Templates fail on schema drift.
5. BigQuery Storage API: Lets Dataflow/Vertex AI read BigQuery tables without exporting to GCS first.
6. Data Fusion: No-code ETL; good for simple pipelines, but not for ML-specific transformations.
7. Dataproc: Managed Spark/Hadoop; use for PySpark MLlib, not for simple ETL.
8. Vertex AI Feature Store: Stores pre-computed features; requires Dataflow/BigQuery for ingestion.
9. Streaming vs. Batch:
10. Streaming (Pub/Sub + Dataflow): Low-latency, higher cost.
11. Batch (GCS/BigQuery): High-latency, lower cost.
12. Cost Traps:
    • BigQuery: Pay per query (bytes scanned). ⚠️ Unpartitioned tables = $$$.
    • Dataflow: Pay per vCPU/hour. ⚠️ Autoscaling can spike costs.