Cloud ML - AWS Certified Machine Learning Engineer – Associate (MLA-C01): Data Storage and Management (S3, Redshift, RDS, DynamoDB, Lake Formation)

AWS_ML – Data Storage and Management (S3, Redshift, RDS, DynamoDB, Lake Formation)

AWS Certified Machine Learning – Specialty: Data Storage and Management (S3, Redshift, RDS, DynamoDB, Lake Formation) – Exam-Ready Study Guide

What This Is

Data storage and management are the backbone of any ML pipeline. Whether you're training a fraud detection model on streaming transactions, building a recommendation engine with petabytes of user behavior data, or deploying a real-time inference endpoint, how and where you store data determines performance, cost, and scalability. For example: - A retail recommendation system might use S3 for raw clickstream data, Redshift for aggregated user profiles, and DynamoDB for low-latency feature lookups during inference.
- A fraud detection pipeline could ingest transactions into Kinesis Data Streams, store processed features in SageMaker Feature Store, and use Lake Formation to enforce fine-grained access controls.

This guide covers AWS’s core storage services, their trade-offs, and how to choose the right one for ML workloads.

Key Terms & Services

AWS-Specific Storage Services

• Amazon S3 (Simple Storage Service):
  AWS’s object storage for raw, processed, and model artifacts. Best for scalability, durability, and cost-efficiency (e.g., storing training datasets, model checkpoints, or logs). Supports versioning, lifecycle policies, and event triggers (e.g., triggering a Lambda when new data arrives).

• ML Use Case: Storing training data, model artifacts, and batch inference outputs.

• Amazon Redshift:
  Columnar data warehouse optimized for analytical queries (OLAP). Best for aggregating structured data (e.g., user behavior, sales records) before training. Supports Redshift Spectrum to query data directly in S3 without loading it into the warehouse.

• ML Use Case: Preprocessing large datasets for batch training (e.g., aggregating customer purchase history).

• Amazon RDS (Relational Database Service):
  Managed SQL database (PostgreSQL, MySQL, etc.) for transactional data (OLTP). Best for structured data with frequent reads/writes (e.g., user profiles, product catalogs).

• ML Use Case: Storing labeled training data or metadata (e.g., customer IDs, timestamps).

• Amazon DynamoDB:
  NoSQL key-value/document database for low-latency, high-throughput access. Best for real-time feature stores, session data, or metadata (e.g., user preferences, device states).

• ML Use Case: Storing online features for real-time inference (e.g., user embeddings for recommendations).

• AWS Lake Formation:
  Governance layer for S3-based data lakes. Simplifies data cataloging, access control, and fine-grained permissions (e.g., restricting access to PII). Works with Glue Data Catalog for schema management.

• ML Use Case: Enforcing security policies on training data (e.g., masking sensitive fields).

ML-Specific Data Concepts

• Feature Store:
  A centralized repository for storing, sharing, and serving ML features. SageMaker Feature Store (AWS) and Vertex AI Feature Store (GCP) reduce feature drift and duplication by ensuring consistent features for training and inference.

• Example: Storing user embeddings in DynamoDB for real-time recommendations.

• Data Lake vs. Data Warehouse:

• Data Lake (S3 + Lake Formation): Stores raw, unstructured, or semi-structured data (e.g., logs, images, JSON). Best for exploratory analysis and ML training.

• Data Warehouse (Redshift): Stores structured, processed data for analytics and reporting. Best for aggregated features.

• Batch vs. Streaming Data:

• Batch (S3, Redshift): Processed in large chunks (e.g., daily model retraining).

• Streaming (Kinesis, DynamoDB): Processed in real-time (e.g., fraud detection).

• ACID vs. BASE:

• ACID (RDS, Redshift): Ensures strong consistency (e.g., financial transactions).
• BASE (DynamoDB, S3): Prioritizes availability and partition tolerance (e.g., user session data).

Step-by-Step / Process Flow

Scenario: Building a Feature Store for a Recommendation System

1. Store Raw Data in S3
2. Ingest clickstream data (JSON/Parquet) into an S3 bucket with versioning enabled.

3. Use Lake Formation to define data access policies (e.g., restrict PII access).

4. Process Data with Glue/EMR

5. Use AWS Glue (serverless) or EMR (managed Spark) to clean, transform, and aggregate data.

6. Write processed features to S3 (offline store) and DynamoDB (online store).

7. Register Features in SageMaker Feature Store

8. Create a Feature Group in SageMaker Feature Store with:
   • Offline store (S3): For batch training.
   • Online store (DynamoDB): For real-time inference.

9. Define feature definitions (e.g., user_id, last_purchase_category).

10. Query Features for Training & Inference

11. Training: Use SageMaker Processing to pull features from the offline store (S3).

12. Inference: Query the online store (DynamoDB) via SageMaker Endpoints for low-latency predictions.

13. Monitor Feature Drift

14. Use SageMaker Model Monitor to detect feature distribution shifts between training and inference.

Common Mistakes

Certification Exam Insights

What the Exam Tests

1. Service Selection Traps
2. "When to use Redshift vs. RDS vs. DynamoDB?"
   • Redshift: Analytics (OLAP), aggregations, batch training.
   • RDS: Transactional data (OLTP), structured metadata.
   • DynamoDB: Low-latency feature serving, session data.

3. "S3 vs. EFS vs. FSx for Lustre?"

   
   
   • S3: Cheap, durable, but not for frequent small file access.
   • EFS: Shared file system for EC2/SageMaker notebooks (e.g., Jupyter).
   • FSx for Lustre: High-performance distributed file system for training large models.

4. Cost Optimization

5. S3 Storage Classes: Use Intelligent-Tiering for unpredictable access patterns, Glacier for archival.

6. Redshift Costs: Concurrency scaling adds cost; use WLM (Workload Management) to prioritize queries.

7. Security & Compliance

8. Lake Formation: Know how to grant fine-grained access (e.g., SELECT on specific columns).

9. KMS Encryption: Required for HIPAA/GDPR compliance; enabled by default in SageMaker Feature Store.

10. Performance Pitfalls

11. DynamoDB Throttling: Use on-demand capacity for unpredictable workloads, provisioned for steady traffic.
12. Redshift Spectrum: Slower than Redshift native tables (use for infrequent queries on S3 data).

Quick Check Questions

Question 1

A fintech company needs to store real-time transaction data for fraud detection. The system must support millions of reads/writes per second with single-digit millisecond latency. Which AWS service should they use? ✅ Answer: DynamoDB
Explanation: DynamoDB is optimized for high-throughput, low-latency access, making it ideal for real-time feature serving.

Question 2

A data scientist wants to query a 10TB dataset in S3 without loading it into a database. They need SQL support and cost efficiency. Which AWS service should they use? ✅ Answer: Amazon Athena (or Redshift Spectrum) Explanation: Athena allows serverless SQL queries on S3 data, while Redshift Spectrum extends Redshift to query S3 directly.

Question 3

A retail company is building a recommendation system and needs to store user embeddings for real-time inference. The embeddings are updated hourly and must be accessed with <10ms latency. Which storage solution is best? ✅ Answer: SageMaker Feature Store (online) + DynamoDB
Explanation: The online store in SageMaker Feature Store uses DynamoDB for low-latency access, while the offline store (S3) handles batch updates.

Last-Minute Cram Sheet

1. S3 is the default for ML data storage – cheap, durable, but not a database.
2. Redshift = analytics (OLAP), RDS = transactions (OLTP), DynamoDB = real-time features.
3. Lake Formation = governance for S3 data lakes (fine-grained access control).
4. SageMaker Feature Store = offline (S3) + online (DynamoDB) features.
5. Athena = SQL on S3, Redshift Spectrum = Redshift querying S3.
6. DynamoDB on-demand = unpredictable workloads, provisioned = steady traffic.
7. ⚠️ S3 is not a file system – use EFS/FSx for Lustre for frequent small file access.
8. ⚠️ Redshift is not for real-time serving – use DynamoDB or Feature Store.
9. S3 lifecycle policies – move old data to Glacier/Deep Archive to save costs.
10. KMS encryption is mandatory for compliance (HIPAA, GDPR) in ML pipelines.