TECH **AWS EC2 Placement Groups & HPC: Zero-Fluff, Hands-On Study Guide**

AWS EC2 Placement Groups & HPC: Zero-Fluff, Hands-On Study Guide

(For AWS Solutions Architect – Associate & Real-World Deployments)

1. What This Is & Why It Matters

You’re a cloud engineer at a genomics startup. Your team runs high-performance computing (HPC) workloads—think DNA sequencing, financial risk modeling, or 3D rendering—that demand ultra-low latency and high network throughput between servers. If you deploy these workloads on random EC2 instances across AWS’s data centers, your jobs will crawl (or fail) due to network jitter and packet loss.

Enter EC2 Placement Groups: A feature that lets you control where your instances physically run in AWS’s data centers. Think of it like reserving seats on a plane: - Cluster Placement Group: All instances sit in the same rack (lowest latency, highest throughput).
- Spread Placement Group: Instances are isolated on separate hardware (for fault tolerance).
- Partition Placement Group: Instances are grouped into logical partitions (for large-scale distributed systems like Hadoop).

Why this matters in production:
- HPC workloads (e.g., MPI-based simulations) fail or slow to a crawl without cluster placement groups.
- Fault-tolerant systems (e.g., multi-AZ databases) break if instances aren’t spread across hardware.
- Costly mistakes: Deploying a cluster placement group across AZs (it won’t work) or mixing instance types (performance tanks).

Real-world scenario:
You inherit a legacy financial risk modeling app that runs on 20 c5n.18xlarge instances. The app uses MPI (Message Passing Interface) to parallelize calculations. Right now, it takes 4 hours to run—but if you deploy it in a cluster placement group, you can cut that to 30 minutes (and save $1,200/month in compute costs).

2. Core Concepts & Components

? Placement Group Types

? Key AWS Services for HPC

? Networking & Performance

3. Step-by-Step: Deploy an HPC Cluster with Placement Groups & EFA

Prerequisites

✅ AWS account with admin IAM permissions.
✅ VPC with a public subnet (for bastion host) and private subnet (for HPC nodes).
✅ Key pair for SSH access.
✅ Budget alert (HPC instances are expensive!).

Step 1: Create a Cluster Placement Group

aws ec2 create-placement-group \
  --group-name "hpc-cluster" \
  --strategy "cluster" \
  --region us-east-1

Verify:

aws ec2 describe-placement-groups --group-names "hpc-cluster"

Expected output:

{
  "PlacementGroups": [
    {
      "GroupName": "hpc-cluster",
      "State": "available",
      "Strategy": "cluster",
      "GroupId": "pg-1234567890abcdef0"
    }
  ]
}

⚠️ Trap: If you try to launch instances in multiple AZs, the placement group fails silently (instances launch but aren’t clustered).

Step 2: Launch EFA-Enabled Instances in the Placement Group

We’ll use c5n.18xlarge (100 Gbps networking) with EFA.

aws ec2 run-instances \
  --image-id ami-0abcdef1234567890 \  # Amazon Linux 2 with EFA (us-east-1)
  --instance-type c5n.18xlarge \
  --key-name "your-key-pair" \
  --security-group-ids "sg-1234567890abcdef0" \
  --subnet-id "subnet-1234567890abcdef0" \
  --placement "GroupName=hpc-cluster" \
  --count 4 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=hpc-node}]' \
  --ena-support \
  --region us-east-1

Key flags:
- --placement "GroupName=hpc-cluster" → Forces instances into the placement group.
- --ena-support → Enables Enhanced Networking (required for EFA).
- --count 4 → Launches 4 instances (adjust based on workload).

Verify EFA is enabled:

ssh -i "your-key-pair.pem" ec2-user@<public-ip-of-bastion>
sudo lspci | grep "Elastic Fabric Adapter"

Expected output:

1a:00.0 Ethernet controller: Amazon.com, Inc. Elastic Fabric Adapter (EFA)

Step 3: Configure Jumbo Frames (MTU 9001)

On each instance:

sudo ip link set dev eth0 mtu 9001

Verify:

ip link show eth0 | grep mtu

Expected output:

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000

⚠️ Trap: If VPC or subnet doesn’t support Jumbo Frames, this fails silently (packets fragment, performance drops).

Step 4: Install & Test MPI (OpenMPI)

On one instance (head node):

sudo yum install -y openmpi-devel
mpirun --version

Expected output:

Open MPI v4.1.1, package: Open MPI ...

Run a test MPI job across all nodes:
1. Create a hostfile (/home/ec2-user/hostfile) with private IPs of all instances:
10.0.1.10 slots=36
10.0.1.11 slots=36
10.0.1.12 slots=36
10.0.1.13 slots=36
(Each c5n.18xlarge has 36 vCPUs.)

1. Run a test job:
   bash
mpirun --hostfile hostfile -np 144 hostname
   Expected output:
   ip-10-0-1-10
ip-10-0-1-10
... (144 lines, distributed across nodes)
   ⚠️ If this fails:
2. Check security groups (allow all traffic between instances).
3. Verify EFA is enabled (lspci | grep EFA).
4. Ensure Jumbo Frames are set (ip link show eth0).

Step 5: (Optional) Deploy FSx for Lustre for High-Speed Storage

aws fsx create-file-system \
  --file-system-type LUSTRE \
  --storage-capacity 1200 \
  --subnet-ids "subnet-1234567890abcdef0" \
  --security-group-ids "sg-1234567890abcdef0" \
  --lustre-configuration DeploymentType="SCRATCH_2" \
  --region us-east-1

Mount on each instance:

sudo mkdir /fsx
sudo mount -t lustre <fsx-dns-name>@tcp:/fsx /fsx

⚠️ Trap: FSx for Lustre is expensive (~$0.14/GB-month). Use S3 as a backend to reduce costs:

sudo mount -t lustre <fsx-dns-name>@tcp:/fsx /fsx -o s3import=<s3-bucket-name>

4. ? Production-Ready Best Practices

? Security

• Least privilege IAM roles: HPC nodes shouldn’t have admin access.
• Security groups: Allow only necessary ports (e.g., MPI: 1024-65535).
• EFA traffic: No encryption (performance hit)—use private subnets instead.
• Bastion host: Never expose HPC nodes to the internet—use a jump box.

? Cost Optimization

• Spot Instances: Use for fault-tolerant HPC workloads (up to 90% discount).
• Auto Scaling: Scale down when idle (e.g., aws autoscaling suspend-processes).
• FSx for Lustre: Tier to S3 when not in use.
• Instance types: c5n for CPU-bound, p4d for GPU-bound, i3en for storage-bound.

?️ Reliability & Maintainability

• Tag everything: Environment=hpc, Project=genomics, Owner=team-bio.
• Immutable infrastructure: Rebuild clusters instead of patching.
• CloudWatch Alarms: Monitor CPU, network, and EFA errors.
• ParallelCluster: Automate cluster deployment (saves hours of manual work).

? Observability

• CloudWatch Metrics:
• NetworkPacketsIn/Out (spikes = MPI traffic).
• EFAErrors (high = misconfiguration).
• CPUUtilization (should be >90% for HPC).
• Logs:
• MPI logs (/var/log/mpirun.log).
• EFA driver logs (/var/log/efa/).
• Alarms:
• EFAErrors > 0 for 5 minutes → Page the team.
• CPU < 50% for 1 hour → Scale down.

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus (AWS SAA-C03)

Typical Question Patterns

1. "You need the lowest latency between EC2 instances for an MPI workload. Which placement group?"
2. ✅ Cluster (correct).
3. ❌ Spread (isolates instances).

4. ❌ Partition (for distributed systems, not HPC).

5. "A financial modeling app runs on 10 c5n.18xlarge instances. It’s slow. What’s the issue?"

6. ✅ Not using a cluster placement group (correct).
7. ❌ "Use a larger instance type" (already 18xlarge).

8. ❌ "Enable EBS optimization" (irrelevant for MPI).

9. "You need to deploy a fault-tolerant database with 3 nodes. Which placement group?"

10. ✅ Spread (correct—isolates nodes on separate hardware).
11. ❌ Cluster (all nodes fail if rack fails).
12. ❌ Partition (overkill for 3 nodes).

Key ⚠️ Trap Distinctions

Scenario-Based Question

"You’re running a genomics workload that processes 10 TB of data daily. It uses MPI and requires <1ms latency between nodes. Cost is a concern. What’s the best setup?"
- ✅ Cluster placement group + c5n.18xlarge + EFA + Spot Instances (correct).
- ❌ "Use spread placement group" (not for HPC).
- ❌ "Use m5.large instances" (too slow for MPI).
- ❌ "Use On-Demand only" (Spot saves 90%).

7. ? Hands-On Challenge (With Solution)

Challenge

Deploy a 2-node HPC cluster with: - Cluster placement group.
- EFA-enabled instances (c5n.4xlarge).
- Jumbo Frames (MTU 9001).
- Run a test MPI job (hostname across both nodes).

Constraints:
- Use only the AWS CLI (no console).
- Free tier eligible (c5n.4xlarge is not free, but you can use t3.micro for testing—just know EFA won’t work).

Solution

1. Create placement group:
   bash
aws ec2 create-placement-group --group-name "test-cluster" --strategy "cluster"
2. Launch 2 c5n.4xlarge instances:
   bash
aws ec2 run-instances \
--image-id ami-0abcdef1234567890 \ # Amazon Linux 2 with EFA
--instance-type c5n.4xlarge \
--key-name "your-key" \
--security-group-ids "sg-1234567890abcdef0" \
--subnet-id "subnet-1234567890abcdef0" \
--placement "GroupName=test-cluster" \
--count 2 \
--ena-support
3. SSH into one instance and set MTU:
   bash
sudo ip link set dev eth0 mtu 9001
4. Create a hostfile with private IPs:
   10.0.1.10 slots=8
10.0.1.11 slots=8
5. Run MPI test:
   bash
mpirun --hostfile hostfile -np 16 hostname
   Expected output: 16 lines (8 from each node).

Why it works:
- Cluster placement group ensures low-latency networking.
- EFA enables OS-bypass for MPI.
- Jumbo Frames prevent packet fragmentation.

8. ? Rapid-Reference Crib Sheet