Cloud ML - AWS Certified Machine Learning Engineer – Associate (MLA-C01): Data Quality Checks (Missing Values, Outliers, Class Imbalance)

AWS_ML – Data Quality Checks (Missing Values, Outliers, Class Imbalance)

AWS Certified Machine Learning – Specialty: Data Quality Checks (Missing Values, Outliers, Class Imbalance) – Exam-Ready Study Guide

What This Is

Data quality checks are the first line of defense in any ML pipeline—garbage in, garbage out. This topic covers how to detect and handle missing values, outliers, and class imbalance in training datasets. These issues distort model performance, increase bias, and lead to unreliable predictions.

Real-world scenario:
A fintech company builds a fraud detection model using transaction logs. The dataset has: - Missing values (e.g., null merchant_category in 15% of records).
- Outliers (e.g., a single $10M transaction skewing the distribution).
- Class imbalance (only 0.1% of transactions are fraudulent).
Without proper checks, the model may overfit to noise, ignore rare fraud cases, or fail in production due to unseen edge cases.

Key Terms & Services

AWS-Specific Services

• Amazon SageMaker Data Wrangler
  AWS’s no-code/low-code tool for exploratory data analysis (EDA), cleaning, and feature engineering. Integrates with SageMaker Pipelines for automated workflows. Best for quick prototyping and visualizing data quality issues (e.g., missing values, outliers).

• AWS Glue DataBrew
  A serverless data preparation tool for cleaning and normalizing datasets. Unlike Data Wrangler, it’s batch-focused and integrates with Glue ETL jobs. Best for large-scale, scheduled data cleaning (e.g., nightly fraud dataset updates).

• Amazon SageMaker Clarify
  Detects bias and data quality issues (e.g., class imbalance, feature skew) before training. Generates bias reports and feature importance metrics. Critical for compliance-heavy use cases (e.g., lending, hiring models).

• Amazon SageMaker Processing
  A managed compute environment for running custom data processing scripts (Python/R) at scale. Use for custom outlier detection (e.g., IQR, Z-score) or synthetic minority oversampling (SMOTE) for class imbalance.

• AWS Lambda + Amazon EventBridge
  For real-time data quality checks in streaming pipelines (e.g., Kinesis). Example: Trigger a Lambda function to flag missing values in incoming transaction data before it hits a fraud model.

General ML Concepts

• Missing Values (Nulls/NAs)
  Gaps in data that can bias models or break pipelines. Common fixes:
• Imputation (mean/median/mode, KNN, regression).
• Dropping (if <5% of data and not critical).

• Flagging (add a binary column indicating missingness).

• Outliers
  Data points far from the rest (e.g., a $10M transaction in a dataset where 99% are <$1K). Can skew models (e.g., linear regression) or hide fraud patterns. Detection methods:

• Statistical (Z-score, IQR).
• Visual (box plots, scatter plots).

• ML-based (Isolation Forest, DBSCAN).

• Class Imbalance
  When one class dominates the dataset (e.g., 99.9% non-fraud, 0.1% fraud). Causes models to ignore the minority class. Fixes:

• Resampling (oversampling minority class, undersampling majority class).
• Synthetic data (SMOTE, ADASYN).

• Algorithm-level (class weights, anomaly detection models like XGBoost with scale_pos_weight).

• Data Drift
  When statistical properties of data change over time (e.g., transaction amounts increase due to inflation). Detected via SageMaker Model Monitor or Evidently AI. Requires retraining or adjusting thresholds.

• Feature Store (SageMaker Feature Store)
  A centralized repository for storing and serving features. Ensures consistency between training and inference (e.g., same imputation logic for missing values). Critical for real-time ML (e.g., recommendation systems).

Step-by-Step / Process Flow

1. Detect Data Quality Issues (EDA)

Goal: Identify missing values, outliers, and class imbalance before training.
Tools: SageMaker Data Wrangler, Pandas, SageMaker Clarify.

Steps:
1. Load data into SageMaker Data Wrangler (from S3, Redshift, or Athena).
2. Run automated EDA (Data Wrangler generates histograms, missing value heatmaps, and outlier plots).
3. Use SageMaker Clarify to generate a bias report (checks for class imbalance, feature skew).
4. Manually inspect with Pandas:
```python
# Missing values
df.isnull().sum()

# Outliers (IQR method)
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))]

# Class imbalance
df['target'].value_counts(normalize=True)
```

2. Handle Missing Values

Goal: Decide whether to impute, drop, or flag missing data.
Tools: Data Wrangler, Glue DataBrew, SageMaker Processing.

Steps:
1. For numerical columns:
- Impute with mean/median (if <10% missing).
- Use KNN imputation (if missingness is correlated with other features).
- Flag missingness (add a binary column, e.g., is_missing_merchant_category).
2. For categorical columns:
- Impute with mode (most frequent category).
- Create a "Missing" category (if missingness is meaningful).
3. Automate in Data Wrangler or Glue DataBrew:
- Apply pre-built transformations (e.g., "Fill missing values with median").
- Export the cleaned dataset to S3.

3. Detect and Handle Outliers

Goal: Decide whether to remove, cap, or keep outliers.
Tools: SageMaker Processing, Pandas, Scikit-learn.

Steps:
1. Detect outliers using:
- Z-score (for normally distributed data).
- IQR (for skewed data).
- Isolation Forest (for high-dimensional data).
2. Decide on treatment:
- Remove (if outliers are errors, e.g., negative age).
- Cap (e.g., set all values above 99th percentile to the 99th percentile).
- Keep (if outliers are meaningful, e.g., fraud transactions).
3. Implement in SageMaker Processing:
python
from sklearn.ensemble import IsolationForest
clf = IsolationForest(contamination=0.01) # Adjust contamination
outliers = clf.fit_predict(df[['amount']])
df = df[outliers == 1] # Keep inliers

4. Fix Class Imbalance

Goal: Ensure the model doesn’t ignore the minority class.
Tools: SageMaker Processing, Imbalanced-learn (SMOTE), XGBoost.

Steps:
1. Check imbalance ratio (e.g., 99:1 non-fraud to fraud).
2. Choose a method:
- Resampling:
- Oversample minority class (duplicate fraud cases).
- Undersample majority class (randomly drop non-fraud cases).
- Synthetic data:
- SMOTE (generate synthetic fraud cases).
- ADASYN (focus on "hard" minority cases).
- Algorithm-level:
- Class weights (e.g., scale_pos_weight=100 in XGBoost).
- Anomaly detection (Isolation Forest, One-Class SVM).
3. Implement in SageMaker Processing:
python
from imblearn.over_sampling import SMOTE
smote = SMOTE(sampling_strategy=0.5) # Balance to 50:50
X_res, y_res = smote.fit_resample(X, y)

5. Automate Data Quality Checks in Pipelines

Goal: Ensure consistency between training and inference.
Tools: SageMaker Pipelines, Lambda, EventBridge.

Steps:
1. Create a SageMaker Pipeline with a Processing Step for data quality checks.
2. Add a Lambda function to validate incoming data in real-time (e.g., check for missing values in Kinesis streams).
3. Use SageMaker Model Monitor to detect data drift in production.
4. Store cleaned data in SageMaker Feature Store for reuse.

Common Mistakes

Mistake 1: Ignoring Missing Values in Categorical Features

• What happens: Models treat NaN as a separate category, leading to unexpected splits in decision trees.
• Correction: Impute with mode or create a "Missing" category (if missingness is meaningful).

Mistake 2: Removing Outliers Without Domain Knowledge

• What happens: Removing legitimate outliers (e.g., fraud transactions) hides critical patterns.
• Correction: Consult domain experts before removing outliers. Use capping instead of removal if outliers are meaningful.

Mistake 3: Using SMOTE on High-Dimensional Data

• What happens: SMOTE generates noisy synthetic samples in high-dimensional space, hurting model performance.
• Correction: Reduce dimensionality first (PCA, feature selection) or use algorithm-level fixes (class weights).

Mistake 4: Not Monitoring Data Drift in Production

• What happens: Model performance degrades silently as real-world data changes (e.g., transaction amounts increase due to inflation).
• Correction: Set up SageMaker Model Monitor to alert on feature drift and label drift.

Mistake 5: Assuming All Missing Values Are Random

• What happens: Missingness is often not random (e.g., high-income users skip optional survey questions). Imputing blindly introduces bias.
• Correction: Analyze missingness patterns (e.g., is income missing more for a certain demographic?). Use multiple imputation (MICE) for non-random missingness.

Certification Exam Insights

1. Service Selection Traps

• SageMaker Data Wrangler vs. Glue DataBrew:
• Data Wrangler = Interactive, SageMaker-native, best for exploratory data cleaning.
• Glue DataBrew = Serverless, batch-focused, best for scheduled ETL jobs.

• Exam trap: If the question mentions real-time data cleaning, neither is ideal—use Lambda + Kinesis.

• SageMaker Clarify vs. SageMaker Model Monitor:

• Clarify = Pre-training (bias, data quality).
• Model Monitor = Post-deployment (drift, performance).
• Exam trap: Clarify does not monitor production models.

2. Key Constraints

• SageMaker Processing has a default timeout of 24 hours. For longer jobs, use EMR or Glue.
• Data Wrangler supports up to 100K rows in the UI. For larger datasets, use Glue DataBrew or SageMaker Processing.
• SMOTE requires numerical features. For categorical data, use ADASYN or class weights.

3. "Which Service?" Scenarios

• Scenario: A team needs to automate data quality checks in a CI/CD pipeline for a fraud model.
• Answer: SageMaker Pipelines + Processing Step (not Data Wrangler, which is interactive).
• Scenario: A company wants to detect bias in a hiring model before deployment.
• Answer: SageMaker Clarify (not Model Monitor, which is for post-deployment).
• Scenario: A streaming pipeline needs to flag missing values in real-time before inference.
• Answer: Lambda + Kinesis (not Data Wrangler or Glue DataBrew, which are batch-only).

4. Tricky Concepts

• Class imbalance ≠ bias: Imbalance is a data issue; bias is a fairness issue. Fix imbalance with resampling; fix bias with SageMaker Clarify.
• Outliers ≠ errors: Not all outliers are bad (e.g., fraud). Domain knowledge is key.
• Missing values ≠ always impute: Sometimes missingness is informative (e.g., users skipping optional fields). Flag it instead of imputing.

Quick Check Questions

Question 1

A fintech company trains a fraud detection model on a dataset where 0.1% of transactions are fraudulent. The model performs well on the majority class but fails to detect fraud. What is the most likely cause, and what two fixes would you recommend?

Answer:
- Cause: Class imbalance (model ignores the minority fraud class).
- Fixes:
1. Resampling (oversample fraud cases or undersample non-fraud cases).
2. Algorithm-level (use scale_pos_weight in XGBoost or class weights in logistic regression).

Question 2

A data scientist uses SageMaker Data Wrangler to clean a dataset with missing values in a categorical column. They impute with the mode, but the model’s performance degrades in production. What is the most likely reason, and what should they do instead?

Answer:
- Reason: Missingness was not random (e.g., certain demographics were more likely to skip the field). Imputing with mode introduced bias.
- Fix: Create a "Missing" category or use multiple imputation (MICE) to account for non-random missingness.

Question 3

A retail company deploys a recommendation model that uses customer purchase history. After 6 months, the model’s accuracy drops significantly. SageMaker Model Monitor detects feature drift in average_order_value. What two actions should the team take?

Answer:
1. Retrain the model with recent data to adapt to the new distribution.
2. Investigate the root cause (e.g., inflation, new product lines) and adjust business rules if needed (e.g., cap extreme values).

Last-Minute Cram Sheet

1. Missing values:
2. Impute (mean/median/mode, KNN) or flag (add binary column).

3. ⚠️ Never drop >5% of data without justification.

4. Outliers:

5. Remove (if errors), cap (if meaningful), or keep (if critical, e.g., fraud).

6. Z-score (normal data), IQR (skewed data), Isolation Forest (high-dimensional).

7. Class imbalance:

8. Resampling (SMOTE, ADASYN), algorithm-level (class weights), or anomaly detection.

9. ⚠️ SMOTE fails on high-dimensional data—reduce dimensions first.

10. SageMaker Data Wrangler:

11. Interactive EDA, no-code cleaning, max 100K rows.

12. ⚠️ Not for production pipelines—use SageMaker Processing instead.

13. Glue DataBrew:

14. Serverless, batch-focused, scheduled jobs.

15. ⚠️ No real-time support—use Lambda + Kinesis for streaming.

16. SageMaker Clarify:

17. Pre-training bias detection, data quality reports.

18. ⚠️ Does not monitor production models—use Model Monitor.

19. SageMaker Model Monitor:

20. Post-deployment drift detection, alerts on feature/label drift.

21. ⚠️ Requires a baseline dataset (e.g., training data).

22. Feature Store:

23. Consistency between training and inference (e.g., same imputation logic).

24. ⚠️ Not a replacement for ETL—clean data before storing.

25. Data drift:

26. Statistical changes in data over time (e.g., transaction amounts increase).

27. Fix: Retrain model or adjust thresholds.

28. Exam traps:

    
    
    • ⚠️ Data Wrangler ≠ production tool (use Processing or Glue).
    • ⚠️ SMOTE ≠ always the answer (fails on high-dimensional data).
    • ⚠️ Missing values ≠ always impute (sometimes flagging is better).