Cloud ML - Google Cloud Professional Machine Learning Engineer: Data Transformation (Dataprep, Dataflow, Spark on Dataproc)

GCP_ML – Data Transformation (Dataprep, Dataflow, Spark on Dataproc)

Google Cloud Professional Machine Learning Engineer Study Guide: Data Transformation (Dataprep, Dataflow, Spark on Dataproc)

What This Is

Data transformation is the backbone of any ML pipeline—cleaning, enriching, and structuring raw data into features ready for training or inference. In GCP, this means choosing between Cloud Dataprep (no-code ETL), Dataflow (serverless stream/batch processing), and Spark on Dataproc (managed big data clusters) to handle everything from real-time fraud detection (streaming) to large-scale feature engineering for recommendation systems (batch). For example, a retail company might use Dataflow to process clickstream data in real time, Dataproc to join historical purchase records with user profiles, and Dataprep to clean and normalize the final dataset before training a demand-forecasting model.

Key Terms & Services

• Cloud Dataprep by Trifacta: GCP’s no-code, browser-based tool for visually cleaning, shaping, and enriching structured/semi-structured data (CSV, JSON, Parquet). Best for analysts or ML engineers who need quick, interactive transformations without writing code.
• Dataflow: GCP’s fully managed, serverless Apache Beam service for batch and streaming data processing. Scales automatically, integrates with BigQuery/PubSub, and is ideal for real-time pipelines (e.g., fraud detection) or large-scale batch jobs (e.g., feature engineering).
• Dataproc: GCP’s managed Hadoop/Spark service for running big data workloads (PySpark, Spark SQL, Hive). Best for complex, long-running jobs (e.g., joining terabytes of data) or when you need fine-grained control over cluster configuration (e.g., GPU acceleration for ML preprocessing).
• Apache Beam: Open-source unified programming model for batch/streaming pipelines. Dataflow is the managed Beam runner on GCP; other runners include Flink (on-prem) or Spark (via Dataproc).
• Pub/Sub: GCP’s real-time messaging service for ingesting streaming data (e.g., IoT sensor data, user clicks). Often the source for Dataflow streaming pipelines.
• BigQuery: GCP’s serverless data warehouse. Frequently the sink (destination) for transformed data, especially for training datasets or feature stores.
• Feature Store (Vertex AI Feature Store): Centralized repository for storing, sharing, and serving ML features. Reduces duplication and ensures consistency between training and inference (e.g., same feature logic for batch and real-time predictions).
• Data Fusion: GCP’s GUI-based ETL/ELT tool (like Dataprep but more enterprise-focused). Less common for ML pipelines but useful for hybrid cloud data integration.
• TensorFlow Transform (TFT): Library for preprocessing data at scale (e.g., normalizing features, encoding categoricals) within a TFX pipeline. Runs on Dataflow or Dataproc.
• Schema Drift: When the structure of incoming data changes (e.g., new columns, missing fields), breaking downstream pipelines. Dataflow/Beam can handle this with dynamic schemas; Dataprep can detect it visually.
• Windowing (in Dataflow): Splitting streaming data into time-based chunks (e.g., 5-minute windows for aggregating user sessions). Critical for real-time ML (e.g., fraud detection, anomaly scoring).
• Shuffle (in Spark/Dataproc): The expensive process of redistributing data across nodes during operations like groupBy or join. Minimize shuffles to reduce costs and latency.

Step-by-Step / Process Flow

Scenario: Building a Real-Time Fraud Detection Pipeline

Goal: Process streaming transactions, enrich with user profiles, and generate features for a fraud model.

1. Ingest Streaming Data
2. Set up a Pub/Sub topic to receive transaction events (e.g., JSON messages with user_id, amount, timestamp).

3. Configure a Dataflow streaming job (Apache Beam) to read from Pub/Sub:
   python
pipeline_options = PipelineOptions(streaming=True)
with beam.Pipeline(options=pipeline_options) as p:
transactions = (p
| "Read from PubSub" >> beam.io.ReadFromPubSub(topic="projects/my-project/topics/transactions")
| "Parse JSON" >> beam.Map(lambda x: json.loads(x))
)

4. Enrich with Batch Data

5. Join streaming transactions with BigQuery (user profiles, historical fraud rates) using beam.io.ReadFromBigQuery:
   python
user_profiles = (p
| "Read User Profiles" >> beam.io.ReadFromBigQuery(
query="SELECT user_id, risk_score FROM `my-project.users.profiles`",
use_standard_sql=True)
)
enriched = ({'transactions': transactions, 'profiles': user_profiles}
| "Join" >> beam.CoGroupByKey()
)

6. Transform and Window Data

7. Apply windowing (e.g., 1-minute fixed windows) to aggregate transactions per user:
   python
windowed = (enriched
| "Window" >> beam.WindowInto(beam.window.FixedWindows(60))
| "Aggregate" >> beam.CombinePerKey(CombineFn())
)

8. Use TensorFlow Transform (TFT) for scaling/normalization if preparing data for a TF model:
   python
| "TFT Preprocess" >> beam.ParDo(PreprocessFn())

9. Write to Feature Store or BigQuery

10. For real-time serving, write features to Vertex AI Feature Store:
    python
| "Write to Feature Store" >> beam.io.WriteToFeatureStore(
project="my-project",
location="us-central1",
feature_group="fraud_features"
)

11. For batch training, write to BigQuery or Cloud Storage (Parquet/CSV).

12. Deploy and Monitor

13. Deploy the Dataflow job with autoscaling (adjusts workers based on throughput).
14. Set up Cloud Monitoring alerts for pipeline latency or errors.

Common Mistakes

Certification Exam Insights

1. Service Selection Traps
2. Dataprep vs. Dataflow vs. Dataproc:
   • Use Dataprep for interactive, no-code transformations (e.g., cleaning a CSV for a quick prototype).
   • Use Dataflow for serverless, scalable batch/streaming (e.g., real-time feature engineering).
   • Use Dataproc for complex Spark jobs (e.g., joining terabytes of data with PySpark).

3. When to use TFT: Only if you’re using TensorFlow and need consistent preprocessing between training and serving. For PyTorch or scikit-learn, use Dataflow/Spark.

4. Key Constraints

5. Dataflow quotas: Default quotas limit concurrent jobs (e.g., 25 streaming jobs per region). Request increases if scaling.
6. Dataproc cluster costs: Clusters are billed per second (even when idle). Use preemptible VMs for cost savings (but jobs may fail).

7. BigQuery ML: Can train simple models (linear regression, k-means) inside BigQuery, but for deep learning, export data to Vertex AI.

8. Tricky Scenarios

9. "Which service for a one-time batch job?"
   • Dataflow (serverless, no cluster management) or Dataproc (if you need Spark).
   • Avoid Dataprep (not designed for large-scale batch).
10. "How to handle late data in streaming?"
    • Use allowed lateness in Dataflow (e.g., beam.WindowInto(..., allowed_lateness=Duration(minutes=5))).
11. "How to ensure feature consistency?"
    • Use Vertex AI Feature Store or TFT to avoid training-serving skew.

Quick Check Questions

1. A fintech company needs to process 100K transactions per second in real time, enrich them with user profiles from BigQuery, and generate features for a fraud model. Which GCP service should they use for the transformation pipeline?
2. Answer: Dataflow (serverless, scales to streaming volume, integrates with Pub/Sub and BigQuery).

3. Why: Dataproc isn’t designed for streaming, and Dataprep is for interactive use.

4. A data scientist wants to preprocess a 500GB dataset for a TensorFlow model. The preprocessing includes normalizing numerical features and one-hot encoding categoricals. Which GCP service should they use to ensure the same transforms are applied during inference?

5. Answer: TensorFlow Transform (TFT) on Dataflow.

6. Why: TFT ensures preprocessing logic is consistent between training and serving.

7. A team is running a PySpark job on Dataproc to join two large tables (1TB and 500GB). The job is slow and fails due to memory errors. What’s the most likely cause, and how can they fix it?

8. Answer: Shuffle overhead. Fix by:
   • Using broadcast join for the smaller table.
   • Increasing executor memory (spark.executor.memory).
   • Partitioning data by a key to reduce shuffle size.

Last-Minute Cram Sheet

1. Dataprep: No-code, interactive data cleaning (CSV/JSON/Parquet). ⚠️ Not for large-scale batch/streaming.
2. Dataflow: Serverless Beam for batch/streaming. Autoscaling, integrates with Pub/Sub/BigQuery.
3. Dataproc: Managed Spark/Hadoop. Use for complex PySpark jobs or GPU acceleration. ⚠️ Not for streaming.
4. TFT: TensorFlow preprocessing at scale. Runs on Dataflow/Dataproc. Ensures training-serving consistency.
5. Vertex AI Feature Store: Centralized feature storage. Reduces duplication and drift.
6. Windowing: Fixed/sliding/session windows in Dataflow for streaming aggregations.
7. Shuffle: Expensive operation in Spark. Minimize with broadcast joins or partitioning.
8. Schema drift: Handle with dynamic schemas in Beam or Dataprep’s schema detection.
9. Cost traps:
10. Dataproc clusters bill per second (even idle). Use preemptible VMs.
11. Dataflow streaming jobs have higher costs than batch.
12. Exam traps:
    • ⚠️ Dataprep ≠ Dataflow: Dataprep is for interactive cleaning, not pipelines.
    • ⚠️ BigQuery ≠ training data: Export to Cloud Storage for large ML datasets.
    • ⚠️ TFT only for TF: Not needed for PyTorch/scikit-learn.