Cloud ML - AWS Certified Machine Learning Engineer – Associate (MLA-C01): CI/CD for ML (SageMaker Pipelines, CodePipeline, CloudFormation)

AWS_ML – CI/CD for ML (SageMaker Pipelines, CodePipeline, CloudFormation)

AWS Certified Machine Learning – Specialty: CI/CD for ML (SageMaker Pipelines, CodePipeline, CloudFormation) – Exam-Ready Study Guide

What This Is

CI/CD for ML automates the build, test, and deploy stages of machine learning models—just like software CI/CD, but with extra steps for data validation, model training, evaluation, and inference deployment. In AWS, this is done using SageMaker Pipelines (ML-specific workflows), CodePipeline (orchestration), and CloudFormation (infrastructure-as-code).

Real-world scenario:
A fintech company retrains its fraud detection model weekly using new transaction data. They use SageMaker Pipelines to automate data preprocessing, model training, and A/B testing of the new model against the old one. CodePipeline triggers the pipeline when new data lands in S3, and CloudFormation deploys the updated endpoint with zero downtime.

Key Terms & Services

• SageMaker Pipelines
  AWS’s ML-specific workflow orchestrator (like Airflow but native to SageMaker). Defines steps for data prep, training, evaluation, and deployment. Uses Python SDK or JSON/YAML for pipeline definitions.

• CodePipeline
  AWS’s CI/CD service for automating software (and ML) release workflows. Triggers pipelines on code commits, S3 uploads, or manual approvals. Integrates with CodeBuild (for running tests) and CodeDeploy (for deployments).

• CloudFormation
  AWS’s infrastructure-as-code (IaC) service. Defines AWS resources (SageMaker endpoints, IAM roles, S3 buckets) in YAML/JSON templates. Enables repeatable, version-controlled deployments.

• SageMaker Projects
  A pre-built CI/CD template for ML in SageMaker. Combines SageMaker Pipelines, CodePipeline, and CloudFormation into a single, reusable project. Includes model registry, approval gates, and automated rollbacks.

• Model Registry
  A central catalog in SageMaker to track model versions, metadata, and approval status. Used in pipelines to promote models from "staging" to "production" after validation.

• Processing Jobs
  SageMaker’s managed compute for data preprocessing, feature engineering, or model evaluation. Runs in isolated containers (no need to manage EC2 instances).

• Approval Gates
  Manual or automated checkpoints in CodePipeline (e.g., "Approve model before deployment"). Critical for governance and compliance in regulated industries (e.g., healthcare, finance).

• Canary Deployment
  A gradual rollout strategy where a new model version serves a small % of traffic before full deployment. Reduces risk of bad deployments. Supported in SageMaker endpoints via traffic splitting.

• Infrastructure-as-Code (IaC)
  The practice of defining cloud resources in code (e.g., CloudFormation, Terraform). Ensures consistent, reproducible environments for ML pipelines.

• Artifact Store
  An S3 bucket where SageMaker Pipelines store outputs (e.g., trained models, evaluation reports). Required for reproducibility and auditing.

• EventBridge (CloudWatch Events)
  AWS’s event bus for triggering pipelines based on events (e.g., "New data in S3," "Model drift detected"). Used to automate retraining.

• SageMaker Model Monitor
  Detects data drift, feature skew, or prediction anomalies in production. Can trigger retraining pipelines via EventBridge.

Step-by-Step / Process Flow

How to Build a CI/CD Pipeline for ML in AWS

1. Define the Pipeline in SageMaker
2. Write a Python script using the sagemaker.workflow SDK to define steps:
   • Data processing (SageMaker Processing Job)
   • Training (SageMaker Training Job)
   • Evaluation (Processing Job to compare metrics)
   • Conditional deployment (if metrics pass, deploy to endpoint)

3. Example:
   python
from sagemaker.workflow.pipeline import Pipeline
pipeline = Pipeline(
name="FraudDetectionPipeline",
steps=[processing_step, training_step, evaluation_step, condition_step]
)

4. Store Pipeline Definition in CodeCommit/GitHub

5. Commit the pipeline script and CloudFormation template (for infrastructure) to a CodeCommit repository (or GitHub).

6. Example CloudFormation snippet for a SageMaker endpoint:
   yaml
Resources:
FraudDetectionEndpoint:
Type: AWS::SageMaker::Endpoint
Properties:
EndpointConfigName: !Ref EndpointConfig

7. Set Up CodePipeline

8. Create a CodePipeline with 3 stages:
   1. Source (CodeCommit/GitHub) → Triggers on code changes.
   2. Build (CodeBuild) → Runs tests (e.g., pytest) and packages the pipeline.
   3. Deploy → Executes the SageMaker Pipeline via AWS CLI or SDK.

9. Add an approval gate before deployment (e.g., "Approve model for production").

10. Automate Triggers

11. Use EventBridge to trigger the pipeline when:

    
    
    • New data lands in S3 (e.g., PutObject event).
    • Model Monitor detects drift (e.g., ModelPackageStateChange event).

12. Deploy with CloudFormation

13. Use CloudFormation StackSets to deploy the same pipeline across multiple AWS accounts/regions (e.g., dev, staging, prod).

14. Example CLI command:
    bash
aws cloudformation deploy --template-file pipeline.yaml --stack-name FraudDetectionStack

15. Monitor and Roll Back

16. Use CloudWatch Alarms to detect endpoint failures (e.g., high latency, errors).
17. Configure automatic rollback in CloudFormation if the deployment fails.

Common Mistakes

Certification Exam Insights

1. Service Selection Traps

2. SageMaker Pipelines vs. Step Functions vs. Airflow

   
   
   • Use SageMaker Pipelines for ML-specific workflows (training, evaluation, deployment).
   • Use Step Functions for general-purpose orchestration (e.g., Lambda, Glue, EMR).
   • Use Airflow (MWAA) if you need complex DAGs or multi-cloud support.

3. Key Constraints

4. SageMaker Pipelines can only run SageMaker jobs (Training, Processing, Transform). For non-SageMaker steps (e.g., Glue, Lambda), use Step Functions.

5. CodePipeline has a 1-year execution limit for pipelines. For long-running ML jobs (e.g., training LLMs), use SageMaker Pipelines with manual approvals.

6. Tricky Scenarios

7. Question: "A team needs to automate model retraining when new data arrives in S3. Which AWS service should trigger the pipeline?"

   
   
   • Answer: EventBridge (CloudWatch Events). It can listen for PutObject events in S3 and trigger CodePipeline or SageMaker Pipelines.
   • Trap: Candidates often pick S3 Event Notifications, but EventBridge is more flexible (e.g., filtering, multiple targets).

8. Cost Optimization

9. SageMaker Pipelines are free (you only pay for the underlying jobs: Training, Processing, etc.).
10. CodePipeline costs $1 per active pipeline per month (first pipeline is free). Use one pipeline per project to minimize costs.

Quick Check Questions

1. Question:
   "A company wants to automate its ML pipeline, including data preprocessing, model training, and deployment. The pipeline must support manual approvals before production deployment. Which AWS services should they use?"

2. Answer: SageMaker Pipelines + CodePipeline + CloudFormation.

   
   
   • SageMaker Pipelines handles ML steps (preprocessing, training, deployment).
   • CodePipeline orchestrates the pipeline and adds approval gates.
   • CloudFormation deploys the infrastructure (endpoints, IAM roles).

3. Question:
   "A data scientist accidentally deployed a faulty model to production. The team wants to prevent this in the future. What should they implement?"

4. Answer: Approval gates in CodePipeline + Model Registry.

   
   
   • Approval gates require manual sign-off before deployment.
   • Model Registry tracks model versions and approval status.

5. Question:
   "A team uses SageMaker Pipelines for training but needs to run a custom Python script for data validation before training. Which SageMaker feature should they use?"

6. Answer: SageMaker Processing Job.
   • Processing Jobs run custom scripts in managed containers (no EC2 management).

Last-Minute Cram Sheet

1. SageMaker Pipelines = ML-specific workflows (training, evaluation, deployment). CodePipeline = CI/CD orchestrator.
2. Model Registry = Central catalog for model versions, approvals, and metadata.
3. CloudFormation = IaC for SageMaker endpoints, IAM roles, and S3 buckets.
4. Approval gates in CodePipeline = Required for compliance (e.g., HIPAA, GDPR).
5. EventBridge triggers pipelines on S3 uploads or Model Monitor alerts.
6. SageMaker Projects = Pre-built CI/CD templates for ML (Pipelines + CodePipeline + CloudFormation).
7. Processing Jobs = Managed compute for data preprocessing or model evaluation.
8. Canary deployment = Gradual rollout of new models to reduce risk.
9. ⚠️ SageMaker Pipelines cannot run non-SageMaker jobs (e.g., Glue, Lambda). Use Step Functions instead.
10. ⚠️ CodePipeline has a 1-year execution limit—not suitable for long-running ML jobs. Use SageMaker Pipelines with manual approvals.