TECH **Headless Services & DNS in Kubernetes (CoreDNS) – Zero-Fluff Study Guide**

Headless Services & DNS in Kubernetes (CoreDNS) – Zero-Fluff Study Guide

1. What This Is & Why It Matters

A headless Service in Kubernetes is a Service without a ClusterIP. Instead of load-balancing traffic, it returns the Pod IPs directly via DNS. This is critical when: - You need direct Pod-to-Pod communication (e.g., stateful apps like databases, Kafka, or Redis clusters).
- You want custom discovery logic (e.g., a leader election system where clients need to know which Pod is the leader).
- You’re running StatefulSets (where each Pod has a stable identity, like db-0, db-1).

Real-world scenario:
You’re deploying a PostgreSQL cluster with Patroni (a high-availability tool). Each Pod (pg-0, pg-1, pg-2) needs to: 1. Discover its peers by name (not just a load-balanced IP).
2. Communicate directly (e.g., pg-1 needs to talk to pg-0 for replication).
3. Survive restarts (Pod IPs change, but DNS names like pg-0.postgres.default.svc.cluster.local must stay stable).

What breaks if you ignore this?
- Your StatefulSet apps fail (e.g., databases can’t replicate).
- Service meshes (Istio, Linkerd) misbehave (they rely on DNS for sidecar injection).
- Debugging becomes a nightmare (you can’t nslookup individual Pods).

2. Core Concepts & Components

3. Step-by-Step Hands-On: Deploy a Stateful Redis Cluster with Headless Service

Prerequisites

• A running Kubernetes cluster (Minikube, Kind, or cloud-based).
• kubectl configured to access the cluster.
• Basic familiarity with kubectl and YAML.

Goal

Deploy a 3-node Redis cluster where: - Each Pod (redis-0, redis-1, redis-2) can discover its peers via DNS.
- Clients can connect to any Pod directly (e.g., redis-0.redis.default.svc.cluster.local:6379).

Step 1: Create a Headless Service for Redis

# redis-headless-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  clusterIP: None  # ⚠️ This makes it headless!
  ports:
    - port: 6379
      name: redis
  selector:
    app: redis

Apply it:

kubectl apply -f redis-headless-service.yaml

Verify:

kubectl get svc redis

Expected output:

NAME    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
redis   ClusterIP   None         <none>        6379/TCP   5s

✅ Key check: CLUSTER-IP must be None.

Step 2: Deploy Redis as a StatefulSet

# redis-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis  # ⚠️ Must match the headless Service name!
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7
        ports:
        - containerPort: 6379
          name: redis
        command: ["redis-server", "--cluster-enabled", "yes", "--cluster-config-file", "/data/nodes.conf"]
        volumeMounts:
        - name: redis-data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

Apply it:

kubectl apply -f redis-statefulset.yaml

Verify:

kubectl get pods -w

Wait until all Pods are Running:

NAME      READY   STATUS    RESTARTS   AGE
redis-0   1/1     Running   0          30s
redis-1   1/1     Running   0          20s
redis-2   1/1     Running   0          10s

Step 3: Test DNS Resolution Inside the Cluster

Launch a debug Pod to test DNS:

kubectl run dns-test --image=busybox:1.28 --rm -it --restart=Never -- sh

Inside the Pod, run:

nslookup redis

Expected output:

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      redis
Address 1: 10.244.1.5 redis-0.redis.default.svc.cluster.local
Address 2: 10.244.2.4 redis-1.redis.default.svc.cluster.local
Address 3: 10.244.1.6 redis-2.redis.default.svc.cluster.local

✅ Key check: You see all 3 Pod IPs (not a single ClusterIP).

Now test individual Pod DNS:

nslookup redis-0.redis

Expected output:

Name:      redis-0.redis
Address 1: 10.244.1.5 redis-0.redis.default.svc.cluster.local

✅ Key check: Each Pod has a stable DNS name (redis-0.redis, redis-1.redis, etc.).

Step 4: Connect to Redis and Verify Cluster Formation

Exec into redis-0:

kubectl exec -it redis-0 -- redis-cli

Inside Redis, run:

CLUSTER NODES

Expected output (shows all 3 nodes):

3e3a6cb0d9a9a87168e266b0a0b24026c0aae377 10.244.1.5:6379@16379 myself,master - 0 1650982452000 1 connected 0-5460
a211e24273170ff94509829de567aece3340430a 10.244.2.4:6379@16379 master - 0 1650982451000 2 connected 5461-10922
292f8b365bb7edb5e285caf0b7e6ddc7265d2f4f 10.244.1.6:6379@16379 master - 0 1650982450000 3 connected 10923-16383

✅ Key check: All 3 Pods are connected in a cluster.

4. ? Production-Ready Best Practices

Security

• Network Policies: Restrict Pod-to-Pod traffic (e.g., only allow redis Pods to talk to each other).
  ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata:
  name: redis-allow-cluster spec:
  podSelector:
  matchLabels:
  app: redis
  ingress:
  • from:
  • podSelector:
    matchLabels:
    app: redis ports:
  • protocol: TCP
    port: 6379 ```
• TLS: Use mTLS (e.g., Istio, Linkerd) for Pod-to-Pod encryption.
• RBAC: Limit who can modify headless Services (they’re critical for stateful apps).

Cost Optimization

• Resource Requests/Limits: Set requests and limits for StatefulSet Pods to avoid over-provisioning.
  yaml resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "1"
memory: "2Gi"
• StorageClass: Use cheap storage (e.g., gp2 on AWS, standard on GKE) for non-critical data.

Reliability & Maintainability

• Naming Conventions: Use consistent names (e.g., redis-headless, postgres-headless).
• Labels & Annotations: Tag Services with app: redis, env: prod for easier filtering.
• Readiness Probes: Ensure Pods are ready before DNS resolves them.
  yaml readinessProbe:
exec:
command: ["redis-cli", "ping"]
initialDelaySeconds: 5
periodSeconds: 5

Observability

• CoreDNS Metrics: Monitor coredns_dns_requests_total (Prometheus) to detect DNS failures.
• Pod Logs: Check kubectl logs -l app=redis for cluster formation errors.
• Alerts: Set up alerts for:
• kube_pod_status_ready (Pods not ready).
• coredns_dns_requests_failed_total (DNS failures).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. "What’s the difference between a headless Service and a regular Service?"
2. Regular Service: Returns a ClusterIP (load-balanced).

3. Headless Service: Returns Pod IPs directly (no load balancing).

4. "How do you make a StatefulSet Pod discoverable by DNS?"

5. Create a headless Service with clusterIP: None.

6. Pods get DNS names like pod-0.service.namespace.svc.cluster.local.

7. "What happens if you delete a headless Service?"

8. DNS resolution fails for all Pods in the StatefulSet.

9. StatefulSet Pods keep running, but they can’t discover each other.

10. "How do you debug DNS resolution in Kubernetes?"

11. Run a debug Pod:
    bash
kubectl run dns-test --image=busybox:1.28 --rm -it --restart=Never -- sh
12. Check /etc/resolv.conf:
    bash
cat /etc/resolv.conf
13. Test DNS:
    bash
nslookup redis-0.redis

Key ⚠️ Trap Distinctions

7. ? Hands-On Challenge (with Solution)

Challenge

Deploy a 2-node MongoDB replica set where: - Each Pod (mongo-0, mongo-1) can discover its peer via DNS.
- The replica set initializes automatically.

Solution

1. Headless Service:
   ```yaml
   apiVersion: v1
   kind: Service
   metadata:
   name: mongo
   spec:
   clusterIP: None
   ports:
   • port: 27017 selector:
     app: mongo
     ```
2. StatefulSet:
   ```yaml
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
   name: mongo
   spec:
   serviceName: mongo
   replicas: 2
   selector:
   matchLabels:
   app: mongo
   template:
   metadata:
   labels:
   app: mongo
   spec:
   containers:
   - name: mongo
   image: mongo:5
   command: ["mongod", "--replSet", "rs0", "--bind_ip_all"]
   ports:
   - containerPort: 27017
   volumeMounts:
   - name: mongo-data
   mountPath: /data/db
   volumeClaimTemplates:
   • metadata:
     name: mongo-data
     spec:
     accessModes: [ "ReadWriteOnce" ]
     resources:
     requests:
     storage: 1Gi
     ```
3. Initialize the Replica Set:
   bash
kubectl exec -it mongo-0 -- mongosh --eval "rs.initiate({_id: 'rs0', members: [{_id: 0, host: 'mongo-0.mongo:27017'}, {_id: 1, host: 'mongo-1.mongo:27017'}]})" Why it works:
4. The headless Service ensures mongo-0.mongo and mongo-1.mongo resolve to Pod IPs.
5. The StatefulSet guarantees stable DNS names (mongo-0, mongo-1).
6. MongoDB’s --replSet flag enables replication, and the rs.initiate() command configures the replica set.

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. Kubernetes Docs – Services (Official headless Service docs).
2. CoreDNS GitHub (Advanced DNS tuning).
3. Kubernetes StatefulSets (Deep dive on stateful apps).
4. Debugging DNS in Kubernetes (Troubleshooting guide).