TECH **Horizontal Pod Autoscaler (HPA) & Cluster Autoscaler: Zero-Fluff, Hands-On Guide**

Horizontal Pod Autoscaler (HPA) & Cluster Autoscaler: Zero-Fluff, Hands-On Guide

For engineers who need to scale Kubernetes workloads in production—without the theory overload.

1. What This Is & Why It Matters

You’re running a Kubernetes cluster, and your app suddenly gets a traffic spike (e.g., Black Friday, a viral blog post, or a DDoS attack). Without autoscaling: - Pods crash (OOMKilled) because they can’t handle the load.
- Nodes run out of CPU/memory, causing scheduling failures.
- Your pager explodes at 3 AM because the on-call engineer forgot to manually scale.

HPA (Horizontal Pod Autoscaler) automatically adjusts the number of pod replicas based on CPU/memory usage or custom metrics (e.g., requests per second).
Cluster Autoscaler adds/removes nodes when pods can’t be scheduled due to resource constraints.

Real-world scenario:
You deploy a microservice that processes user uploads. During peak hours, CPU usage spikes to 90%. HPA scales from 2 to 10 pods in minutes. But if your cluster has only 2 nodes, Cluster Autoscaler spins up 3 more nodes to accommodate the new pods. When traffic drops, both scale down—saving you money.

If you ignore this:
- Over-provisioning → Wasted cloud spend (e.g., $5K/month for idle nodes).
- Under-provisioning → Downtime, SLA violations, and angry customers.
- Manual scaling → Human error, slow response, and burnout.

2. Core Concepts & Components

? Horizontal Pod Autoscaler (HPA)

• Definition: Kubernetes controller that scales pod replicas based on observed CPU/memory or custom metrics (e.g., QPS, latency).
• Production insight: HPA reacts to current metrics, not future load. Use predictive scaling (e.g., KEDA) for time-based traffic (e.g., "scale up at 8 AM").
• Key metric: targetCPUUtilizationPercentage (e.g., "scale when CPU > 70%").

? Cluster Autoscaler

• Definition: Kubernetes add-on that adjusts the number of nodes in a cluster based on pod scheduling failures.
• Production insight: Works with cloud providers (AWS, GCP, Azure) to provision/terminate nodes. Does not work with bare-metal clusters.
• Key setting: --scale-down-delay-after-add (how long to wait before scaling down nodes).

? Metrics Server

• Definition: Cluster-wide aggregator of resource metrics (CPU, memory) used by HPA.
• Production insight: If Metrics Server is down, HPA fails silently. Monitor its health (kubectl top nodes should work).
• Default install: Not enabled in some managed Kubernetes services (e.g., EKS). You must install it manually.

? Custom Metrics Adapter

• Definition: Extends HPA to scale based on non-CPU/memory metrics (e.g., Prometheus queries, Kafka lag).
• Production insight: Required for scaling based on business metrics (e.g., "scale when Redis queue length > 1000").
• Example: kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1".

? Resource Requests/Limits

• Definition: Pod-level CPU/memory constraints (requests = guaranteed, limits = max allowed).
• Production insight: HPA uses requests to calculate utilization. If requests are too low, HPA scales too aggressively.
• Example:
  yaml resources:
requests:
cpu: "500m" # 0.5 CPU
memory: "512Mi"
limits:
cpu: "1" # 1 CPU
memory: "1Gi"

? Pod Disruption Budget (PDB)

• Definition: Ensures a minimum number of pods stay available during voluntary disruptions (e.g., node drains).
• Production insight: Prevents Cluster Autoscaler from evicting all pods during scale-down. Always set PDBs for critical apps.
• Example:
  yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata:
name: my-app-pdb spec:
minAvailable: 2 # At least 2 pods must stay running
selector:
matchLabels:
app: my-app

? Cooldown Periods

• Definition: Delays before HPA/Cluster Autoscaler takes another action (e.g., "don’t scale down for 5 minutes after scaling up").
• Production insight: Prevents thrashing (e.g., scaling up/down every 30 seconds). Defaults are often too short for production.
• HPA defaults:
• --horizontal-pod-autoscaler-downscale-stabilization: 5 minutes
• --horizontal-pod-autoscaler-upscale-stabilization: 3 minutes

? Vertical Pod Autoscaler (VPA)

• Definition: Adjusts pod CPU/memory requests/limits (not replica count). Complements HPA.
• Production insight: Use VPA for apps with variable resource needs (e.g., batch jobs). Do not use with HPA on the same resource (conflicts).
• Example:
  yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata:
name: my-app-vpa spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: my-app
updatePolicy:
updateMode: "Auto" # Automatically adjust requests/limits

3. Step-by-Step Hands-On: Deploy HPA + Cluster Autoscaler

Prerequisites

1. A Kubernetes cluster (EKS, GKE, AKS, or Minikube for local testing).
2. kubectl configured to access the cluster.
3. Metrics Server installed (if not already present).
4. For Cluster Autoscaler: A cloud provider with autoscaling groups (e.g., AWS ASG, GCP MIG).

Step 1: Install Metrics Server

HPA needs Metrics Server to fetch CPU/memory data.

# For EKS/GKE/AKS (if not pre-installed)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# For Minikube
minikube addons enable metrics-server

# Verify
kubectl top nodes
# Output should show CPU/memory usage

Troubleshooting:
- If kubectl top nodes fails, check Metrics Server logs: bash kubectl logs -n kube-system deployment/metrics-server - Common issue: x509: certificate signed by unknown authority. Fix: bash kubectl patch deployment metrics-server -n kube-system --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]'

Step 2: Deploy a Sample App

We’ll use a CPU-intensive app to trigger HPA.

# Create a deployment
kubectl create deployment php-apache --image=k8s.gcr.io/hpa-example
kubectl set resources deployment php-apache --requests=cpu=200m
kubectl expose deployment php-apache --port=80

# Verify
kubectl get pods
kubectl get svc php-apache

Step 3: Deploy HPA

Scale the php-apache deployment when CPU > 50%.

kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

# Verify
kubectl get hpa
# Output:
# NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
# php-apache   Deployment/php-apache   0%/50%    1         10        1          5s

Check HPA details:

kubectl describe hpa php-apache

Step 4: Generate Load to Trigger HPA

Open a new terminal and run a load generator:

kubectl run -it --rm load-generator --image=busybox --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://php-apache; done"

Watch HPA scale up:

kubectl get hpa -w
# Output:
# NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
# php-apache   Deployment/php-apache   0%/50%     1         10        1          1m
# php-apache   Deployment/php-apache   250%/50%   1         10        1          2m
# php-apache   Deployment/php-apache   250%/50%   1         10        4          2m

Stop the load generator (Ctrl+C) and watch HPA scale down:

kubectl get hpa -w
# After ~5 minutes, replicas will drop back to 1.

Step 5: Set Up Cluster Autoscaler (AWS Example)

Cluster Autoscaler adds/removes nodes when pods can’t be scheduled.

1. Create an IAM Policy for Cluster Autoscaler

# Save this as cluster-autoscaler-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeTags",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "ec2:DescribeLaunchTemplateVersions"
      ],
      "Resource": "*"
    }
  ]
}

# Create the policy
aws iam create-policy --policy-name AmazonEKSClusterAutoscalerPolicy --policy-document file://cluster-autoscaler-policy.json

2. Attach the Policy to the EKS Node Role

# Get the node instance role name
kubectl -n kube-system describe configmap aws-auth | grep rolearn

# Attach the policy (replace <ROLE_NAME>)
aws iam attach-role-policy --role-name <ROLE_NAME> --policy-arn arn:aws:iam::<ACCOUNT_ID>:policy/AmazonEKSClusterAutoscalerPolicy

3. Deploy Cluster Autoscaler

# Get your EKS cluster name
eksctl get cluster

# Deploy Cluster Autoscaler (replace <CLUSTER_NAME>)
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/download/cluster-autoscaler-1.27.3/cluster-autoscaler-autodiscover.yaml

# Edit the deployment to add your cluster name
kubectl -n kube-system edit deployment cluster-autoscaler

# Add these flags to the container args:
# - --balance-similar-node-groups
# - --skip-nodes-with-system-pods=false
# - --cluster-name=<CLUSTER_NAME>

4. Verify Cluster Autoscaler

# Check logs
kubectl -n kube-system logs -f deployment/cluster-autoscaler

# Scale up a deployment to trigger node addition
kubectl scale deployment php-apache --replicas=20

# Watch nodes being added
kubectl get nodes -w

4. ? Production-Ready Best Practices

? Security

• Least privilege: Cluster Autoscaler’s IAM role should only have permissions for ASGs, not full EC2 access.
• Network policies: Restrict pod-to-pod communication to prevent HPA from being manipulated by malicious pods.
• Secrets: If using custom metrics (e.g., Prometheus), ensure credentials are stored in Kubernetes Secrets, not ConfigMaps.

? Cost Optimization

• Right-size requests/limits: HPA scales based on requests, not limits. Set requests close to actual usage.
• Scale-down delays: Increase --scale-down-delay-after-add (default: 10m) to avoid thrashing.
• Spot instances: Use spot nodes for fault-tolerant workloads (e.g., batch jobs) to save 70%+ on costs.
• HPA cooldown: Set --horizontal-pod-autoscaler-downscale-stabilization to 10–15 minutes to avoid rapid scale-downs.

?️ Reliability & Maintainability

• PDBs for critical apps: Always set minAvailable to prevent Cluster Autoscaler from evicting all pods during scale-down.
• Label selectors: Use consistent labels (e.g., app: my-service) for HPA and PDBs to avoid misconfigurations.
• Idempotent deployments: Ensure your app can handle sudden pod terminations (e.g., graceful shutdowns).
• Rollback strategy: Test HPA behavior during rollouts (e.g., kubectl rollout undo).

? Observability

• Monitor HPA metrics:
• kube_hpa_status_current_replicas (current pod count)
• kube_hpa_status_desired_replicas (target pod count)
• kube_hpa_spec_max_replicas (max allowed pods)
• Alerts:
• HPA unable to scale (e.g., kube_hpa_status_condition{condition="AbleToScale"} == 0)
• Cluster Autoscaler failing to add nodes (check logs for Failed to scale up).
• Logging:
• HPA events: kubectl get events --sort-by=.metadata.creationTimestamp
• Cluster Autoscaler logs: kubectl -n kube-system logs -f deployment/cluster-autoscaler

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. HPA vs. VPA:
2. Question: "Your app’s memory usage varies wildly. Should you use HPA or VPA?"

3. Answer: VPA (adjusts memory/CPU requests), but not both on the same resource (conflicts).

4. Cluster Autoscaler Requirements:

5. Question: "Which of these is required for Cluster Autoscaler to work?"
   • A) Bare-metal nodes
   • B) Cloud provider autoscaling groups
   • C) Kubernetes Metrics Server

6. Answer: B (Cluster Autoscaler needs cloud provider integration).

7. HPA Metrics:

8. Question: "HPA is not scaling your deployment. What’s the most likely cause?"
   • A) Missing requests in the pod spec
   • B) No limits set
   • C) Metrics Server not installed

9. Answer: A (HPA uses requests to calculate utilization).

10. PDB and HPA:

11. Question: "You set minAvailable: 2 in a PDB for a deployment with 3 replicas. HPA scales it to 1 replica. What happens?"
12. Answer: HPA will not scale below 2 replicas (PDB takes precedence).

⚠️ Trap Distinctions

• HPA scales pods, Cluster Autoscaler scales nodes. Don’t confuse them!
• HPA uses requests, not limits, to calculate utilization.
• Cluster Autoscaler only works with cloud providers, not bare metal.
• PDBs prevent voluntary disruptions (e.g., node drains), not involuntary (e.g., node failures).

7. ? Hands-On Challenge

Scenario:
You deploy a web app with HPA configured to scale at 70% CPU. During load testing, HPA scales from 2 to 10 pods, but the app still crashes. The cluster has 3 nodes, and kubectl get nodes shows all nodes at 100% CPU.

Task:
1. Identify why the app is crashing.
2. Fix the issue without adding more nodes manually.

Solution:
1. Problem: HPA scales pods, but the cluster has no capacity left (nodes are full). Cluster Autoscaler isn’t configured.
2. Fix:
- Deploy Cluster Autoscaler (as shown in Step 5).
- Ensure the node group has autoscaling enabled (e.g., AWS ASG with min/max nodes).
- Verify with:
bash
kubectl get nodes -w # Watch new nodes being added
kubectl describe hpa # Check HPA status

Why it works:
Cluster Autoscaler adds nodes when pods can’t be scheduled, allowing HPA to scale further.

8. ? Rapid-Reference Crib Sheet

HPA Commands

# Create HPA
kubectl autoscale deployment <deployment> --cpu-percent=50 --min=1 --max=10

# Check HPA status
kubectl get hpa
kubectl describe hpa <name>

# Delete HPA
kubectl delete hpa <name>

# Scale based on custom metrics (e.g., Prometheus)
kubectl autoscale deployment <deployment> --min=1 --max=10 --metric-name=my_metric --target-value=100

Cluster Autoscaler Commands

# Check logs
kubectl -n kube-system logs -f deployment/cluster-autoscaler

# Force scale-up (for testing)
kubectl scale deployment <deployment> --replicas=100

# Watch nodes being added
kubectl get nodes -w

Metrics Server Commands

```bash

Check if Metrics Server