Cloud ML - AWS Certified Machine Learning Engineer – Associate (MLA-C01): Data Transformation (AWS Glue ETL, EMR, SageMaker Data Wrangler, Lambda)

AWS_ML – Data Transformation (AWS Glue ETL, EMR, SageMaker Data Wrangler, Lambda)

AWS Certified Machine Learning – Specialty: Data Transformation Study Guide

Topic: AWS Glue ETL, EMR, SageMaker Data Wrangler, Lambda

What This Is

Data transformation is the backbone of any ML pipeline—cleaning, enriching, and structuring raw data into features ready for training or inference. In AWS, this means choosing between serverless ETL (Glue), big data processing (EMR), interactive data prep (SageMaker Data Wrangler), or lightweight event-driven transformations (Lambda). Real-world scenario: A fintech company ingests 10M daily transactions from Kafka, needs to deduplicate records, normalize amounts, and join with customer metadata before training a fraud detection model. The wrong tool (e.g., Lambda for 10TB joins) could cost $10K/month in failed jobs or timeouts.

Key Terms & Services

• AWS Glue: Serverless ETL service that crawls data (S3, RDS, DynamoDB), generates schemas, and runs Spark/Python jobs. Best for batch transformations (e.g., aggregating logs, converting CSV to Parquet).
• Amazon EMR: Managed Hadoop/Spark cluster for large-scale, complex transformations (e.g., 100TB joins, graph processing). Requires cluster management but offers fine-grained control.
• SageMaker Data Wrangler: GUI-based tool for interactive data prep (cleaning, feature engineering, bias detection). Exports to SageMaker Pipelines or Jupyter notebooks. Ideal for exploratory analysis before training.
• AWS Lambda: Event-driven compute for lightweight transformations (e.g., filtering S3 uploads, calling APIs). Hard limit of 15-minute runtime and 10GB memory—not for big data.
• Glue DataBrew: Visual data prep tool (like Data Wrangler but standalone). Best for non-ML use cases (e.g., BI dashboards) or teams without SageMaker access.
• SageMaker Processing: Managed containers for custom Python/R scripts (e.g., scikit-learn feature engineering). Scales like EMR but integrates with SageMaker Pipelines.
• Spark on Glue vs. EMR:
• Glue Spark: Serverless, auto-scaling, but limited to Glue’s runtime (no custom Spark versions).
• EMR Spark: Full Spark control (e.g., GPU clusters, custom libraries), but requires cluster management.
• Data Lake vs. Data Warehouse:
• Data Lake (S3 + Glue): Schema-on-read, cheap storage for raw data.
• Data Warehouse (Redshift): Schema-on-write, optimized for SQL analytics.
• Feature Store (SageMaker Feature Store): Centralized repository for reusable features (e.g., customer churn risk scores). Avoids recomputing features for training/inference.
• Partitioning (S3/Glue): Organizing data by date/customer (e.g., s3://bucket/year=2023/month=01/) to reduce scan costs and speed up queries.
• Schema Evolution (Glue): Handling changes in data structure (e.g., adding a column) without breaking pipelines. Use Glue Schema Registry for streaming data.

Step-by-Step / Process Flow

1. Batch Transformation with AWS Glue (ETL)

Scenario: Clean and aggregate 500GB of raw CSV logs in S3 for a recommendation model.
1. Crawl the data: Create a Glue Crawler to scan S3 and populate the Glue Data Catalog (schema + metadata).
2. Define the job: Use Glue Studio (visual editor) or Python/Scala scripts to:
- Filter null values.
- Convert timestamps to UTC.
- Join with a DynamoDB table (e.g., user profiles).
3. Optimize performance:
- Use Glue DynamicFrames (handles schema evolution better than Spark DataFrames).
- Enable bookmarks to track processed files (avoid reprocessing).
- Set DPUs (Data Processing Units) based on data size (1 DPU = 4 vCPUs, 16GB RAM).
4. Schedule the job: Trigger via EventBridge (e.g., "run at 2 AM daily") or Glue Workflows (for multi-step pipelines).
5. Output: Write to S3 in Parquet (columnar format for ML) or Redshift (for BI).

2. Large-Scale Processing with EMR

Scenario: Join 10TB of clickstream data with user profiles for a personalization model.
1. Launch an EMR cluster:
- Choose Spark as the framework.
- Select instance types (e.g., r5.2xlarge for memory-heavy joins, g4dn.xlarge for GPU-accelerated ML).
- Enable auto-scaling to handle variable workloads.
2. Submit a Spark job:
- Use EMR Notebooks or Step Functions to run PySpark scripts.
- Optimize with:
- Partitioning (e.g., df.repartition("user_id")).
- Broadcast joins for small tables (e.g., user metadata).
3. Store results: Write to S3 (for ML) or Redshift (for analytics).
4. Terminate the cluster: Use EMR Managed Scaling or auto-termination to avoid idle costs.

3. Interactive Data Prep with SageMaker Data Wrangler

Scenario: Clean a dataset for a churn prediction model before training.
1. Import data: Connect to S3, Redshift, or Athena (no need to move data).
2. Explore & transform:
- Use the visual editor to:
- Handle missing values (impute or drop).
- Encode categorical variables (one-hot or target encoding).
- Detect bias (e.g., class imbalance in target variable).
3. Export:
- To a notebook: Generate a Python script for reproducibility.
- To SageMaker Pipelines: Automate the workflow.
4. Train: Use the cleaned data in a SageMaker Training Job.

4. Event-Driven Transformation with Lambda

Scenario: Filter and enrich IoT sensor data before storing in DynamoDB.
1. Trigger Lambda: Set up an S3 Event Notification (e.g., "run when a new file is uploaded").
2. Write the function:
- Use boto3 to read the S3 file.
- Apply transformations (e.g., convert units, filter outliers).
- Write to DynamoDB or Kinesis (for streaming).
3. Optimize:
- Increase memory (scales CPU proportionally).
- Use provisioned concurrency to avoid cold starts.
4. Monitor: Check CloudWatch Logs for errors (e.g., timeouts, throttling).

Common Mistakes

Certification Exam Insights

1. Glue vs. EMR:
2. Use Glue for serverless, simple ETL (e.g., CSV → Parquet, small joins).

3. Use EMR for complex transformations (e.g., 10TB joins, graph algorithms, custom Spark versions).

4. Data Wrangler vs. Glue DataBrew:

5. Data Wrangler integrates with SageMaker (ML-focused).

6. DataBrew is standalone (BI/analytics-focused).

7. Lambda for ML?

8. Only for lightweight tasks (e.g., filtering S3 uploads, calling APIs).

9. Not for training/inference (use SageMaker instead).

10. Feature Store vs. S3:

11. SageMaker Feature Store is for reusable features (e.g., customer risk scores).

12. S3 is for raw/processed data (e.g., training datasets).

13. Tricky Question Types:

14. "Which service scales automatically for a 1TB join?" → Glue (serverless) or EMR with auto-scaling.
15. "Which tool detects bias in a dataset?" → SageMaker Data Wrangler (not Glue or EMR).
16. "How to handle schema changes in a Glue job?" → DynamicFrames or Schema Registry.

Quick Check Questions

1. A team needs to process 50GB of JSON files daily, apply transformations, and store results in Redshift. The pipeline must run at 3 AM and scale automatically. Which AWS service is the best fit?
2. Answer: AWS Glue.

3. Why: Glue is serverless, scales automatically, and integrates with Redshift.

4. A data scientist wants to clean a dataset interactively and detect bias before training a model. Which tool should they use?

5. Answer: SageMaker Data Wrangler.

6. Why: Data Wrangler provides a GUI for cleaning and bias detection, and integrates with SageMaker.

7. A company receives 10,000 IoT sensor readings per second. They need to filter outliers and enrich the data with metadata before storing it in DynamoDB. Which service should they use?

8. Answer: AWS Lambda (for lightweight filtering) + Kinesis (for streaming).
9. Why: Lambda can handle the filtering, but Kinesis is better for high-throughput streaming.

Last-Minute Cram Sheet

1. Glue vs. EMR: Glue = serverless, EMR = full Spark control. ⚠️ Glue can’t run custom Spark versions.
2. Data Wrangler: GUI for ML data prep. Exports to SageMaker Pipelines. ⚠️ Not for production ETL.
3. Lambda limits: 15-minute timeout, 10GB memory. ⚠️ Not for big data.
4. Glue DPUs: 1 DPU = 4 vCPUs, 16GB RAM. Scale based on data size.
5. EMR cost savings: Use spot instances and auto-scaling.
6. Partitioning: Always partition S3 data (e.g., year=2023/month=01/).
7. Schema evolution: Use Glue DynamicFrames or Schema Registry.
8. Feature Store: SageMaker Feature Store for reusable features. ⚠️ Not for raw data.
9. Parquet vs. CSV: Parquet is columnar, faster for ML, and cheaper to query.
10. Exam trap: ⚠️ "Use Lambda for training" → No! Use SageMaker Training Jobs.