TECH **AWS Batch & Step Functions: Zero-Fluff, Hands-On Study Guide**

AWS Batch & Step Functions: Zero-Fluff, Hands-On Study Guide

(For AWS Solutions Architect – Associate)

1. What This Is & Why It Matters

You’re a cloud engineer at a genomics startup. Your team runs thousands of parallel DNA sequence analyses—some take 5 minutes, others 12 hours. Some need GPUs, others just CPU. Some fail and need retries. Your boss says: “We need this to run automatically, scale to zero when idle, and cost less than our old on-prem cluster.”

AWS Batch is your auto-scaling, job-scheduling superpower. It dynamically provisions EC2 or Fargate resources, runs Docker containers, and shuts them down when done. No idle costs. No manual scaling.

Step Functions is your orchestration glue. It chains Batch jobs (or Lambda, ECS, Glue, etc.) into reliable workflows with retries, error handling, and human approvals. Think of it as a visual, serverless state machine for your entire pipeline.

Why this matters in production:
- Without Batch: You’d over-provision EC2 instances (costly) or manually babysit jobs (error-prone).
- Without Step Functions: Your workflows become brittle scripts with no retries, no visibility, and no rollback.
- Together: You get fully automated, observable, cost-optimized pipelines that scale from 1 job to 10,000.

Real-world scenario:
You inherit a legacy Python script that processes customer uploads. It runs on a single EC2 instance, crashes often, and costs $500/month. You rewrite it as a Batch job (auto-scaling, spot instances) and wrap it in a Step Function (retries, notifications, S3 cleanup). Now it costs $50/month, never crashes, and your boss promotes you.

2. Core Concepts & Components

AWS Batch

• Job Definition
• Definition: A template for your job (Docker image, vCPUs, memory, environment variables, retry attempts).

• Production insight: ⚠️ Always set retryAttempts (default: 1)—jobs fail. Use timeout to avoid runaway costs.

• Job Queue

• Definition: A queue where jobs wait for compute resources. Can prioritize jobs (e.g., "urgent" vs. "batch").

• Production insight: Use multiple queues for different workloads (e.g., GPU vs. CPU jobs). Avoid mixing them.

• Compute Environment

• Definition: The EC2 or Fargate resources that run your jobs. Can be managed (AWS scales for you) or unmanaged (you control the cluster).

• Production insight: Use Fargate for short jobs (<15 min)—no EC2 management. Use EC2 Spot Instances for long jobs to save 70%+.

• Job

• Definition: A single unit of work (e.g., "process file X"). Submitted to a queue, runs on a compute environment.

• Production insight: Tag jobs with jobName and parameters for debugging. Logs go to CloudWatch.

• Array Job

• Definition: A job that spawns N identical sub-jobs (e.g., process 1,000 files in parallel).
• Production insight: Use size (1–10,000) to control parallelism. Set attempts for retries.

Step Functions

• State Machine
• Definition: A JSON-defined workflow (e.g., "Run Batch job → Wait → Notify Slack").

• Production insight: Use Standard workflows for long-running jobs (up to 1 year). Use Express for short jobs (<5 min).

• States

• Definition: Steps in your workflow (e.g., Task, Choice, Wait, Parallel).

• Production insight: Always add Catch and Retry blocks—Batch jobs fail. Use Choice to route errors.

• Task State

• Definition: A step that calls an AWS service (e.g., Batch:SubmitJob, Lambda:Invoke).

• Production insight: Use ResultPath to store outputs (e.g., "ResultPath": "$.batchOutput").

• Execution

• Definition: A single run of a state machine (e.g., "Process file X").

• Production insight: Enable CloudWatch Logs for debugging. Use executionId to track jobs.

• Error Handling

• Definition: Built-in retries, fallbacks, and timeouts.
• Production insight: Set MaxAttempts: 3 and BackoffRate: 2 for exponential backoff.

3. Step-by-Step Hands-On: Deploy a Batch + Step Functions Pipeline

Prerequisites

• AWS account with admin IAM permissions (or at least Batch, StepFunctions, ECS, IAM, CloudWatch access).
• AWS CLI installed (aws --version).
• Docker installed (for building images).

Goal

Deploy a Batch job that processes a CSV file (e.g., "calculate average sales per region") and wrap it in a Step Function that: 1. Submits the Batch job.
2. Waits for completion.
3. Sends a Slack notification on success/failure.

Step 1: Create a Docker Image for Your Job

Your Batch job runs in a container. Let’s create a simple Python script that processes a CSV.

1.1 Write the Python Script (process_csv.py)

import pandas as pd
import os
import sys

def process_csv(input_path, output_path):
    df = pd.read_csv(input_path)
    result = df.groupby('region')['sales'].mean().reset_index()
    result.to_csv(output_path, index=False)
    print(f"Processed {len(df)} rows. Output: {output_path}")

if __name__ == "__main__":
    input_path = os.environ.get("INPUT_PATH", "/data/input.csv")
    output_path = os.environ.get("OUTPUT_PATH", "/data/output.csv")
    process_csv(input_path, output_path)

1.2 Create a Dockerfile

FROM python:3.9-slim
WORKDIR /app
COPY process_csv.py .
RUN pip install pandas
CMD ["python", "process_csv.py"]

1.3 Build and Push to ECR

# Authenticate Docker to ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com

# Create ECR repo
aws ecr create-repository --repository-name batch-processor --region us-east-1

# Build and push
docker build -t batch-processor .
docker tag batch-processor:latest 123456789012.dkr.ecr.us-east-1.amazonaws.com/batch-processor:latest
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/batch-processor:latest

Step 2: Set Up AWS Batch

2.1 Create an IAM Role for Batch

Batch needs permissions to run jobs and access ECR.

# Create a trust policy (batch-trust-policy.json)
cat > batch-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "batch.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create the role
aws iam create-role --role-name AWSBatchServiceRole --assume-role-policy-document file://batch-trust-policy.json

# Attach policies
aws iam attach-role-policy --role-name AWSBatchServiceRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole
aws iam attach-role-policy --role-name AWSBatchServiceRole --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

2.2 Create a Compute Environment

aws batch create-compute-environment \
  --compute-environment-name my-fargate-env \
  --type MANAGED \
  --state ENABLED \
  --compute-resources type=FARGATE,maxvCpus=100,subnets=subnet-12345678,securityGroupIds=sg-12345678

⚠️ Replace subnet-12345678 and sg-12345678 with your VPC subnet and security group.

2.3 Create a Job Queue

aws batch create-job-queue \
  --job-queue-name my-fargate-queue \
  --state ENABLED \
  --priority 1 \
  --compute-environment-order order=0,computeEnvironment=my-fargate-env

2.4 Create a Job Definition

aws batch register-job-definition \
  --job-definition-name process-csv \
  --type container \
  --container-properties '{
    "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/batch-processor:latest",
    "vcpus": 1,
    "memory": 2048,
    "environment": [
      {"name": "INPUT_PATH", "value": "s3://my-bucket/input.csv"},
      {"name": "OUTPUT_PATH", "value": "s3://my-bucket/output.csv"}
    ],
    "jobRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
  }'

⚠️ Replace 123456789012 with your AWS account ID.

Step 3: Set Up Step Functions

3.1 Create an IAM Role for Step Functions

# Create a trust policy (stepfunctions-trust-policy.json)
cat > stepfunctions-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "states.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create the role
aws iam create-role --role-name StepFunctionsBatchRole --assume-role-policy-document file://stepfunctions-trust-policy.json

# Attach policies
aws iam attach-role-policy --role-name StepFunctionsBatchRole --policy-arn arn:aws:iam::aws:policy/AWSBatchFullAccess
aws iam attach-role-policy --role-name StepFunctionsBatchRole --policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess

3.2 Define the State Machine (JSON)

{
  "Comment": "Process CSV with Batch and notify Slack",
  "StartAt": "SubmitBatchJob",
  "States": {
    "SubmitBatchJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobName": "process-csv-job",
        "JobQueue": "arn:aws:batch:us-east-1:123456789012:job-queue/my-fargate-queue",
        "JobDefinition": "arn:aws:batch:us-east-1:123456789012:job-definition/process-csv:1"
      },
      "Next": "NotifySuccess",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "NotifyFailure"
        }
      ]
    },
    "NotifySuccess": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:slack-notifier",
        "Payload": {
          "status": "SUCCESS",
          "message": "CSV processed successfully!"
        }
      },
      "End": true
    },
    "NotifyFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:slack-notifier",
        "Payload": {
          "status": "FAILURE",
          "message": "CSV processing failed!"
        }
      },
      "End": true
    }
  }
}

⚠️ Replace 123456789012 with your AWS account ID.

3.3 Deploy the State Machine

aws stepfunctions create-state-machine \
  --name ProcessCSVWorkflow \
  --definition file://state-machine.json \
  --role-arn arn:aws:iam::123456789012:role/StepFunctionsBatchRole

Step 4: Test the Pipeline

4.1 Upload a Test CSV to S3

echo "region,sales\nnorth,100\nsouth,200\neast,150\nwest,300" > input.csv
aws s3 cp input.csv s3://my-bucket/input.csv

4.2 Start the Step Function Execution

aws stepfunctions start-execution \
  --state-machine-arn arn:aws:states:us-east-1:123456789012:stateMachine:ProcessCSVWorkflow

4.3 Verify Success

1. Check Batch:
   bash
aws batch describe-jobs --job-queue my-fargate-queue --job-status RUNNING
2. Check Step Functions:
3. Go to AWS Console → Step Functions → Executions.
4. Click on your execution to see the visual workflow.
5. Check S3 for Output:
   bash
aws s3 ls s3://my-bucket/
# Should see output.csv
aws s3 cp s3://my-bucket/output.csv -
# Output: region,sales
# north,100.0
# south,200.0
# east,150.0
# west,300.0

4. ? Production-Ready Best Practices

Security

• IAM Least Privilege:
• Batch job roles should only access the S3 buckets they need.
• Step Functions roles should only have permissions for the services they call.
• VPC Isolation:
• Run Batch jobs in a private subnet with a NAT gateway for S3 access.
• Use security groups to restrict traffic (e.g., only allow outbound to S3).
• Secrets Management:
• Never hardcode secrets in job definitions. Use AWS Secrets Manager or SSM Parameter Store.

Cost Optimization

• Spot Instances for Long Jobs:
• Use type=SPOT in your compute environment for 70% cost savings.
• Set bidPercentage (e.g., 80) to control spot pricing.
• Fargate for Short Jobs:
• Fargate is cheaper for jobs <15 min (no EC2 management).
• Job Sizing:
• Right-size vCPUs/memory (e.g., vcpus=1, memory=2048 for Python jobs).
• Use AWS Compute Optimizer to analyze usage.

Reliability & Maintainability

• Idempotency:
• Design jobs to be re-runnable (e.g., check if output exists before processing).
• Tagging:
• Tag all resources (Environment=prod, Project=genomics) for cost tracking.
• Naming Conventions:
• Use project-env-resource (e.g., genomics-prod-batch-queue).
• Versioning:
• Version your job definitions (e.g., process-csv:2). Use revision in Step Functions.

Observability

• CloudWatch Logs:
• Enable Batch job logs ("logConfiguration": {"logDriver": "awslogs"}).
• Set up CloudWatch Alarms for failed jobs.
• Step Functions Metrics:
• Monitor ExecutionsFailed, ExecutionsTimedOut.
• Use CloudWatch Dashboards for visibility.
• X-Ray Tracing:
• Enable AWS X-Ray for Step Functions to debug latency.

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. Batch vs. ECS/Fargate:
2. "You need to run 1,000 parallel jobs with varying CPU/memory. Which service?"

3. Answer: AWS Batch (auto-scaling, job queues, cost-optimized).

4. Step Functions Workflow Design:

5. "You need to run a Lambda, then a Batch job, then notify Slack. How?"

6. Answer: Step Functions with Task states for each step + Catch for errors.

7. Cost Optimization:

8. "You have a 1-hour job that runs nightly. How to minimize cost?"

9. Answer: Use Fargate Spot or EC2 Spot Instances in Batch.

10. Error Handling:

11. "A Batch job fails occasionally. How to retry automatically?"
12. Answer: Set retryAttempts in job definition + Retry in Step Functions.

Key ⚠️ Trap Distinctions

• Batch vs. Lambda:
• Lambda: <15 min, stateless, event-driven.
• Batch: Long-running, containerized, auto-scaling.
• Step Functions Standard vs. Express:
• Standard: Long-running (up to 1 year), $0.025 per 1,000 state transitions.
• Express: Short (<5 min), $1 per million requests.
• Fargate vs. EC2 in Batch:
• Fargate: No EC2 management, good for short jobs.
• EC2: Cheaper for long jobs,