Cloud ML - AWS Certified Machine Learning Engineer – Associate (MLA-C01): Service Limits and Quotas (SageMaker instance types, S3 partitions)

AWS_ML – Service Limits and Quotas (SageMaker instance types, S3 partitions)

AWS Certified Machine Learning – Specialty: Study Guide

Topic: Service Limits and Quotas (SageMaker Instance Types, S3 Partitions)

What This Is

Service limits and quotas in AWS are hard boundaries (e.g., max instances, API calls, or storage partitions) that can break your ML pipeline if ignored. For example: - A real-time fraud detection system using SageMaker endpoints hits a concurrent instance limit, causing throttling during Black Friday traffic.
- A batch inference job fails because the S3 bucket exceeds 5,000 GET requests per second per prefix, stalling feature retrieval.
- A distributed training job crashes because the team didn’t request a quota increase for ml.p4d.24xlarge instances, which are capped at 2 per account by default.

Understanding these limits ensures your ML workloads scale without surprises.

Key Terms & Services

AWS-Specific Limits & Quotas

• SageMaker Instance Quotas:
  AWS-imposed limits on the number of concurrent instances (e.g., ml.m5.2xlarge = 20 per account by default). Request increases via AWS Service Quotas console.
  Why it matters: Training jobs fail if you exceed the limit, even if your code is correct.

• S3 Request Rate Limits:

• 5,000 GET/HEAD requests per second per prefix (e.g., s3://bucket/prefix/).

• 3,500 PUT/COPY/POST/DELETE requests per second per prefix.
  Workaround: Distribute objects across prefixes (e.g., s3://bucket/2023-01-01/, s3://bucket/2023-01-02/).

• SageMaker Endpoint Concurrency Limits:

• Default: 5 concurrent invocations per endpoint (soft limit).

• Hard limit: 200 concurrent invocations per endpoint (after quota increase).
  Use case: Auto-scaling endpoints for high-traffic inference (e.g., recommendation engines).

• SageMaker Training Job Limits:

• Max 20 concurrent training jobs per account (default).

• Max 100 GB input data per training job (unless using Pipe Mode to stream from S3).
  Workaround: Use SageMaker Pipelines to orchestrate sequential jobs.

• S3 Partitioning (Prefix Design):

• Prefix = folder path (e.g., s3://bucket/data/year=2023/month=01/).

• Partitioning improves query performance in Athena, Glue, and SageMaker Feature Store by reducing the data scanned.
  Example: A feature store with 1B records partitioned by user_id and date speeds up training data retrieval.

• SageMaker Notebook Instance Limits:

• Default: 10 notebook instances per account.

• Max volume size: 16 TB (EBS gp3).
  Trap: Notebooks are not for production inference—use SageMaker Endpoints instead.

• SageMaker Processing Job Limits:

• Max 20 concurrent processing jobs per account.

• Max 100 GB input data per job (unless using distributed processing with ml.m5.24xlarge).
  Use case: Preprocessing large datasets (e.g., 100M rows) before training.

• S3 Transfer Acceleration Limits:

• Disabled by default (must enable per bucket).

• No request rate limits, but costs extra ($0.04/GB for transfers > 1 GB).
  Use case: Speeding up global data uploads (e.g., training data from EU to US).

• SageMaker Ground Truth Limits:

• Max 100,000 objects per labeling job.

• Max 10 concurrent labeling jobs per account.
  Workaround: Use private workforce for large-scale labeling.

• SageMaker Feature Store Online Store Limits:

• Max 10,000 records per second per feature group (for OnlineStore).
• Max 100 feature groups per account.
  Use case: Low-latency feature retrieval for real-time inference (e.g., fraud detection).

Step-by-Step / Process Flow

1. Check and Increase SageMaker Instance Quotas

Scenario: You need to train a distributed PyTorch model on 8 ml.p4d.24xlarge instances, but the default quota is 2.
Steps:
1. Check current quotas:
- Go to AWS Service Quotas console → Search for "SageMaker" → Filter for ml.p4d.24xlarge.
- Note the applied quota value (e.g., 2).
2. Request a quota increase:
- Click "Request quota increase" → Enter the new value (e.g., 10) → Submit.
- AWS typically approves within 24 hours (or immediately for small increases).
3. Verify the increase:
- After approval, confirm the new quota in the console.
- Run your training job with instance_count=8.

2. Optimize S3 Request Rates for High-Throughput ML

Scenario: Your batch inference job fails because it exceeds 5,000 GET requests/sec on a single S3 prefix.
Steps:
1. Analyze the prefix bottleneck:
- Use S3 Server Access Logs or CloudWatch Metrics (GetRequests, 4xxErrors) to identify hot prefixes.
2. Redesign the prefix structure:
- Bad: s3://bucket/data/ (all objects in one prefix).
- Good: s3://bucket/data/year=2023/month=01/day=01/ (distributed across 365+ prefixes).
3. Use S3 List API efficiently:
- Instead of list_objects_v2() (slow for large buckets), use S3 Inventory or Glue Crawlers to catalog objects.
4. Test with S3 Benchmarking Tools:
- Use AWS S3 Performance Tool (s3-benchmark) to simulate load and verify improvements.

3. Scale SageMaker Endpoints for High Traffic

Scenario: Your real-time recommendation endpoint throttles during peak traffic (1,000 RPS).
Steps:
1. Check current concurrency limits:
- Go to SageMaker → Endpoints → [Your Endpoint] → Monitoring → Invocations.
- Look for ThrottledRequests in CloudWatch.
2. Increase endpoint concurrency:
- Option 1: Request a quota increase (up to 200 concurrent invocations).
- Option 2: Enable auto-scaling (recommended):
- Set min capacity = 2, max capacity = 20.
- Use CloudWatch Alarms (e.g., CPUUtilization > 70% for 5 minutes → scale out).
3. Optimize endpoint configuration:
- Use ml.m5.2xlarge for low-latency inference (vs. ml.c5.xlarge for cost savings).
- Enable data capture to log requests/responses for debugging.

Common Mistakes

Certification Exam Insights

What the Exam Tests

1. SageMaker Instance Selection Traps:

2. Trick: The exam may ask, "Which instance is best for distributed training of a 100M-parameter PyTorch model?"

   
   
   • Answer: ml.p4d.24xlarge (GPU-optimized) or ml.trn1.32xlarge (Trainium).
   • Trap: ml.m5.2xlarge (CPU-only) is cheaper but too slow for large models.

3. S3 Partitioning for Performance:

4. Scenario: "A Glue job scans 10 TB of data daily. How can you reduce costs?"

   
   
   • Answer: Partition data by date (e.g., s3://bucket/data/year=2023/month=01/) to limit the data scanned.
   • Trap: Using a single prefix (e.g., s3://bucket/data/) forces Glue to scan all files.

5. SageMaker Endpoint Scaling:

6. Scenario: "A fraud detection endpoint gets 10,000 RPS. What’s the most cost-effective solution?"

   
   
   • Answer: Auto-scaling with ml.m5.2xlarge (vs. over-provisioning ml.c5.4xlarge).
   • Trap: Manually setting a fixed instance count (e.g., 10) wastes money during low traffic.

7. Quota Increase Workflow:

8. Scenario: "Your training job fails with ResourceLimitExceeded. What’s the first step?"
   • Answer: Check Service Quotas and request an increase for the specific instance type.
   • Trap: Re-running the job without checking quotas (it will fail again).

Quick Check Questions

Question 1

A data science team is training a distributed XGBoost model on 50 GB of data. The job fails with ResourceLimitExceeded. What’s the most likely cause? Answer: The team hit the default SageMaker training job concurrency limit (20 jobs) or the instance quota limit (e.g., only 2 ml.m5.2xlarge allowed).
Explanation: Always check Service Quotas before launching large jobs.

Question 2

A batch inference job reads 1M files from s3://bucket/data/ and fails with SlowDown errors. What’s the fastest fix? Answer: Distribute files across multiple prefixes (e.g., s3://bucket/data/part-0001/, s3://bucket/data/part-0002/).
Explanation: S3 has a 5,000 RPS limit per prefix; partitioning avoids throttling.

Question 3

A real-time recommendation endpoint gets 1,000 requests per second but throttles after 500 RPS. What’s the issue? Answer: The endpoint’s default concurrency limit is 5 (soft limit). Request a quota increase or enable auto-scaling.
Explanation: SageMaker endpoints have hard and soft limits; auto-scaling is the best long-term fix.

Last-Minute Cram Sheet

1. SageMaker instance quotas are per-account and per-region. ⚠️ Request increases before launching large jobs.
2. S3 GET limit = 5,000 RPS per prefix. Partition data (e.g., by date) to avoid throttling.
3. SageMaker endpoint default concurrency = 5. Use auto-scaling for high-traffic endpoints.
4. SageMaker training job default concurrency = 20. Use Pipelines to orchestrate sequential jobs.
5. S3 Transfer Acceleration costs extra ($0.04/GB) but speeds up global uploads.
6. SageMaker Notebooks are for exploration only. ⚠️ Never use them for production inference.
7. SageMaker Processing Jobs have a 100 GB input limit. Use Pipe Mode for larger datasets.
8. S3 Inventory is cheaper than list_objects_v2() for large buckets.
9. SageMaker Feature Store Online Store = 10,000 RPS per feature group. Partition features if needed.
10. ⚠️ ml.p4d.24xlarge is for large-scale training, not inference. Use ml.g5.2xlarge for GPU inference.