TECH **Docker & Kubernetes: StorageClasses, Dynamic Provisioning, and CSI Drivers**

Docker & Kubernetes: StorageClasses, Dynamic Provisioning, and CSI Drivers

A Hyper-Practical, Zero-Fluff Study Guide

1. What This Is & Why It Matters

You’re deploying a stateful app (PostgreSQL, Redis, or a custom microservice) in Kubernetes. Your cluster runs on AWS, GCP, or bare metal. Without StorageClasses, Dynamic Provisioning, and CSI Drivers, you’re manually creating and attaching storage volumes—like a sysadmin from 2005.

Here’s the problem: - Manual provisioning = Slow, error-prone, and unscalable. You’d have to: 1. Log into your cloud provider’s console.
2. Create a disk (EBS, Persistent Disk, etc.).
3. Attach it to a node.
4. Manually reference it in a PersistentVolume (PV) YAML.
5. Hope no one deletes it accidentally.
- Static provisioning = Wastes money (you pay for unused storage) and blocks auto-scaling.

Dynamic Provisioning + StorageClasses + CSI Drivers = Kubernetes automates storage on-demand.
- A PersistentVolumeClaim (PVC) requests storage → Kubernetes creates a PersistentVolume (PV) automatically using the right StorageClass.
- CSI Drivers let Kubernetes talk to cloud providers (AWS EBS, GCP PD, Azure Disk) or on-prem storage (Ceph, NFS, Longhorn) without vendor lock-in.
- Production impact:
- No downtime when scaling stateful apps (e.g., databases, message queues).
- Cost control (e.g., use gp3 instead of gp2 for EBS, or standard vs. ssd in GCP).
- Disaster recovery (snapshots, cross-zone replication).
- Security (encryption at rest, IAM policies for storage).

Real-world scenario:
You’re migrating a monolithic app to Kubernetes. The app uses a 100GB PostgreSQL database. Your team wants: - Zero manual storage management (no more "Bob, can you create a disk for me?").
- Fast failover (if a node dies, the pod reschedules with the same data).
- Cost-efficient storage (don’t pay for 100GB if the app only uses 20GB).
- Compliance (data must be encrypted at rest).

This guide shows you how to set this up in 30 minutes.

2. Core Concepts & Components

1. PersistentVolume (PV)

• Definition: A piece of storage in the cluster (e.g., an EBS volume, GCP Persistent Disk, or NFS share).
• Production insight:
• PVs are cluster-wide resources (not namespaced).
• If you delete a PV, data is gone (unless you have backups).
• ⚠️ Trap: PVs don’t auto-delete when their PVC is deleted (unless persistentVolumeReclaimPolicy: Delete).

2. PersistentVolumeClaim (PVC)

• Definition: A request for storage by a pod (e.g., "I need 10Gi of fast SSD").
• Production insight:
• PVCs are namespaced (a pod in dev can’t use a PVC from prod).
• If no PV matches the PVC, dynamic provisioning kicks in (if a StorageClass is set).
• ⚠️ Trap: If you delete a PVC, the PV may or may not be deleted (depends on reclaimPolicy).

3. StorageClass (SC)

• Definition: A template for dynamic provisioning (e.g., "Create EBS volumes with gp3 and encryption").
• Production insight:
• Default StorageClass: If you don’t specify one, Kubernetes uses the cluster’s default (e.g., standard in GKE, gp2 in EKS).
• Parameters: Define performance (IOPS, throughput), encryption, and replication.
• ⚠️ Trap: If you change a StorageClass’s parameters, existing PVs aren’t updated (only new ones).

4. Dynamic Provisioning

• Definition: Kubernetes automatically creates PVs when a PVC is requested (no manual steps).
• Production insight:
• Enabled by default in most managed Kubernetes services (EKS, GKE, AKS).
• Disabled by default in kubeadm clusters (you must install a CSI driver).
• ⚠️ Trap: If no StorageClass matches the PVC, the PVC stays in Pending forever.

5. CSI Driver (Container Storage Interface)

• Definition: A plugin that lets Kubernetes talk to storage providers (AWS EBS, GCP PD, Azure Disk, Ceph, etc.).
• Production insight:
• Replaces in-tree volume plugins (e.g., kubernetes.io/aws-ebs is deprecated).
• Must be installed (e.g., ebs.csi.aws.com for AWS, pd.csi.storage.gke.io for GCP).
• ⚠️ Trap: If the CSI driver crashes, new PVCs fail (existing PVs keep working).

6. Volume Modes

• Filesystem (default): Mounts as a directory (e.g., /var/lib/postgresql).
• Block: Exposes raw block storage (e.g., for databases like Cassandra).
• Production insight:
• Block mode is faster (no filesystem overhead) but harder to manage.
• ⚠️ Trap: Not all CSI drivers support block mode (check docs).

7. Access Modes

• ReadWriteOnce (RWO): Single pod can read/write (most common).
• ReadOnlyMany (ROX): Multiple pods can read (e.g., config files).
• ReadWriteMany (RWX): Multiple pods can read/write (e.g., shared storage like NFS).
• Production insight:
• RWX is rare (most cloud providers don’t support it natively; use NFS or CephFS).
• ⚠️ Trap: If you request RWX but the StorageClass doesn’t support it, the PVC stays Pending.

8. Reclaim Policy

• Delete: PV is deleted when PVC is deleted (default for dynamic provisioning).
• Retain: PV is kept (you must manually delete it).
• Production insight:
• Use Retain for critical data (e.g., databases).
• ⚠️ Trap: If you delete a PVC with Delete policy, data is gone forever.

9. Volume Expansion

• Definition: Resize a PVC without downtime (e.g., grow a 10Gi volume to 20Gi).
• Production insight:
• Supported by most CSI drivers (but not all; check docs).
• ⚠️ Trap: Some filesystems (e.g., XFS) don’t support shrinking.

10. Snapshots

• Definition: Point-in-time backups of PVs (e.g., for disaster recovery).
• Production insight:
• Requires VolumeSnapshotClass (similar to StorageClass).
• ⚠️ Trap: Snapshots are not backups (if the underlying storage fails, snapshots may be lost).

3. Step-by-Step Hands-On: Dynamic Provisioning with AWS EBS

Prerequisites

• An EKS cluster (or any Kubernetes cluster with AWS access).
• kubectl configured to talk to the cluster.
• AWS CLI installed and configured (aws configure).
• IAM permissions to create EBS volumes (e.g., AmazonEBSCSIDriverPolicy).

Step 1: Install the AWS EBS CSI Driver

The CSI driver lets Kubernetes create EBS volumes dynamically.

# Add the AWS EBS CSI driver Helm repo
helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm repo update

# Install the driver (replace <CLUSTER_NAME> with your EKS cluster name)
helm upgrade --install aws-ebs-csi-driver \
  aws-ebs-csi-driver/aws-ebs-csi-driver \
  --namespace kube-system \
  --set controller.serviceAccount.create=true \
  --set controller.serviceAccount.name=ebs-csi-controller-sa \
  --set controller.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<ACCOUNT_ID>:role/AmazonEKS_EBS_CSI_DriverRole

Verify installation:

kubectl get pods -n kube-system | grep ebs-csi

Expected output:

ebs-csi-controller-7d6b5c8d8f-abc12   4/4     Running   0   2m
ebs-csi-node-abc12                    3/3     Running   0   2m

Step 2: Create a StorageClass for Fast SSD (gp3)

AWS EBS has multiple volume types: - gp2 (old default, burstable) - gp3 (new default, cheaper, configurable IOPS) - io1 (high performance, expensive)

We’ll use gp3 with 3000 IOPS and 125 MiB/s throughput.

# storageclass-gp3.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer  # Delay binding until pod is scheduled
parameters:
  type: gp3
  fsType: ext4
  encrypted: "true"  # Enable encryption at rest
  iops: "3000"       # Default is 3000 for gp3
  throughput: "125"  # Default is 125 MiB/s
reclaimPolicy: Delete  # Delete PV when PVC is deleted
allowVolumeExpansion: true  # Allow resizing

Apply it:

kubectl apply -f storageclass-gp3.yaml

Verify:

kubectl get storageclass

Expected output:

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          Immediate           false                  1d
gp3             ebs.csi.aws.com         Delete          WaitForFirstConsumer true                 10s

Step 3: Set gp3 as the Default StorageClass

Kubernetes uses gp2 by default (slow and expensive). Let’s change it.

# Remove the default annotation from gp2
kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

# Set gp3 as default
kubectl patch storageclass gp3 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Verify:

kubectl get storageclass

Expected output:

NAME             PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2              kubernetes.io/aws-ebs   Delete          Immediate              false                  1d
gp3 (default)    ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                 2m

Step 4: Deploy a Stateful App with Dynamic Provisioning

We’ll deploy PostgreSQL with a PVC that triggers dynamic provisioning.

# postgres-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3  # Explicitly use gp3 (optional if gp3 is default)
  resources:
    requests:
      storage: 10Gi

# postgres-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:13
        env:
        - name: POSTGRES_PASSWORD
          value: "mysecretpassword"
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: postgres-storage
        persistentVolumeClaim:
          claimName: postgres-pvc

Apply both:

kubectl apply -f postgres-pvc.yaml
kubectl apply -f postgres-deployment.yaml

Step 5: Verify Dynamic Provisioning

Check the PVC status:

kubectl get pvc

Expected output:

NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-pvc   Bound    pvc-12345678-1234-1234-1234-1234567890ab   10Gi       RWO            gp3            30s

Check the PV:

kubectl get pv

Expected output:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pvc-12345678-1234-1234-1234-1234567890ab   10Gi       RWO            Delete           Bound    default/postgres-pvc   gp3                     1m

Check the EBS volume in AWS:

aws ec2 describe-volumes --filters "Name=tag:kubernetes.io/created-for/pvc/name,Values=postgres-pvc"

Expected output:

{
  "Volumes": [
    {
      "VolumeId": "vol-1234567890abcdef0",
      "Size": 10,
      "VolumeType": "gp3",
      "Iops": 3000,
      "Throughput": 125,
      "Encrypted": true,
      "State": "in-use",
      "Tags": [
        {
          "Key": "kubernetes.io/created-for/pvc/name",
          "Value": "postgres-pvc"
        }
      ]
    }
  ]
}

Step 6: Test Volume Expansion

Let’s resize the PVC from 10Gi to 20Gi.

kubectl patch pvc postgres-pvc -p '{"spec": {"resources": {"requests": {"storage": "20Gi"}}}}'

Check the PVC:

kubectl get pvc postgres-pvc

Expected output (may take a few minutes):

NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-pvc   Bound    pvc-12345678-1234-1234-1234-1234567890ab   20Gi       RWO            gp3            5m

Check the EBS volume in AWS:

aws ec2 describe-volumes --volume-ids vol-1234567890abcdef0

Expected output:

{
  "Volumes": [
    {
      "VolumeId": "vol-1234567890abcdef0",
      "Size": 20,  # Resized from 10Gi to 20Gi
      "VolumeType": "gp3",
      "Iops": 3000,
      "Throughput": 125,
      "Encrypted": true,
      "State": "in-use"
    }
  ]
}

4. ? Production-Ready Best Practices

Security

• Encrypt all volumes at rest (use encrypted: "true" in StorageClass).
• Use IAM roles for service accounts (IRSA) for CSI drivers (don’t use static AWS keys).
• Restrict PVC access with PodSecurityPolicy or NetworkPolicy.
• Audit storage usage with tools like kube-state-metrics + Prometheus.

Cost Optimization

• Use gp3 instead of gp2 (30% cheaper, configurable IOPS).
• Set reclaimPolicy: Retain for critical data (prevents accidental deletion).
• Monitor unused PVs and delete them (use kubectl get pv + kubectl delete pv <name>).
• Use VolumeSnapshotClass for backups (cheaper than full EBS snapshots).

Reliability & Maintainability

• Use WaitForFirstConsumer (avoids creating volumes in the wrong AZ).
• Label PVs/PVCs (e.g., app: postgres, env: prod).
• Test volume expansion before using in production (not all filesystems support it).
• Document StorageClass parameters (e.g., "This SC uses gp3 with 3000 IOPS").

Observability

• Monitor PVC/PV status (alert if Pending for >5 minutes).
• Track storage usage (use kubelet_volume_stats_* metrics in Prometheus).
• Log CSI driver errors (check kubectl logs -n kube-system <csi-pod>).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. StorageClass Parameters

2. "Which StorageClass parameter controls IOPS for gp3 volumes?"

   
   
   • Answer: iops (e.g., iops: "3000").

3. Dynamic Provisioning

4. "A PVC is stuck in Pending. What’s the most likely cause?"

   
   
   • Answer: No StorageClass matches the PVC, or the CSI driver isn’t installed.

5. Reclaim Policy

6. "You delete a PVC with reclaimPolicy: Delete. What happens to the PV?"