TECH **Docker & Kubernetes: Readiness and Liveness Probes – Zero-Fluff Study Guide**

Docker & Kubernetes: Readiness and Liveness Probes – Zero-Fluff Study Guide

1. What This Is & Why It Matters

Readiness and liveness probes are Kubernetes’ way of answering two critical questions about your containers: 1. Is this container ready to serve traffic? (Readiness Probe) 2. Is this container still alive and functioning correctly? (Liveness Probe)

Why this matters in production:
- Without probes, Kubernetes assumes your container is healthy as soon as it starts. If your app crashes or hangs, Kubernetes won’t know—it’ll keep sending traffic to a broken pod, causing cascading failures.
- With probes, Kubernetes can: - Stop sending traffic to a pod that’s not ready (e.g., still initializing, waiting for a database connection).
- Automatically restart a pod that’s stuck in a deadlock or infinite loop.
- Improve uptime by failing fast and recovering automatically.

Real-world scenario:
You’re deploying a microservice that depends on a database. The database takes 30 seconds to initialize, but your app starts in 5 seconds. Without a readiness probe, Kubernetes will send traffic to your app before it can connect to the database, causing 500 errors. With a readiness probe, Kubernetes waits until the app signals it’s ready before routing traffic.

2. Core Concepts & Components

1. Liveness Probe

• Definition: Checks if a container is alive (not stuck in a crash loop or deadlock).
• Production insight: If a liveness probe fails, Kubernetes kills the container and restarts it. Use this for critical failures (e.g., app crashed, deadlock, infinite loop).
• Example: A web server that stops responding to HTTP requests.

2. Readiness Probe

• Definition: Checks if a container is ready to serve traffic (e.g., finished initializing, connected to dependencies).
• Production insight: If a readiness probe fails, Kubernetes stops sending traffic to the pod (but doesn’t kill it). Use this for temporary unavailability (e.g., waiting for a database, warming up caches).
• Example: A backend service that needs 10 seconds to load configs before accepting requests.

3. Probe Types

• HTTP Probe (httpGet): Sends an HTTP request to a specific endpoint (e.g., /health). Success = HTTP 2xx/3xx.
• TCP Probe (tcpSocket): Checks if a TCP port is open (e.g., database port 3306). Success = port accepts connection.
• Command Probe (exec): Runs a command inside the container (e.g., cat /tmp/healthy). Success = exit code 0.

4. Probe Configuration Fields

5. Pod Restart Policies

• Always (default): Restart the container if it crashes or liveness probe fails.
• OnFailure: Restart only if the container exits with a non-zero status.
• Never: Never restart the container (use for debugging).
• Production insight: Always use Always for production workloads unless you have a very good reason not to.

6. Probe Endpoints (HTTP)

• Liveness endpoint: Should return 200 OK if the app is alive (e.g., /healthz).
• Readiness endpoint: Should return 200 OK only if the app can serve traffic (e.g., /ready).
• Production insight: Keep these endpoints lightweight—don’t query databases or external services.

7. Probe vs. Startup Probe (Kubernetes 1.16+)

• Startup Probe: Used for slow-starting containers (e.g., legacy apps). Once the startup probe succeeds, liveness/readiness probes take over.
• Production insight: If your app takes >30s to start, use a startup probe to avoid false liveness failures.

3. Step-by-Step Hands-On: Deploying a Pod with Probes

Prerequisites

• A running Kubernetes cluster (Minikube, Kind, or cloud-based).
• kubectl installed and configured.
• A simple web app (we’ll use a Python Flask app for this example).

Step 1: Create a Simple Flask App with Health Endpoints

Save this as app.py:

from flask import Flask
import time

app = Flask(__name__)

# Simulate a slow startup (e.g., waiting for DB)
startup_time = time.time()
is_ready = False

@app.route('/')
def home():
    return "Hello, Kubernetes!"

@app.route('/healthz')
def healthz():
    # Liveness probe: Always return 200 if the app is running
    return "OK", 200

@app.route('/ready')
def ready():
    # Readiness probe: Return 200 only after 10 seconds (simulate DB connection)
    if time.time() - startup_time > 10:
        return "Ready", 200
    else:
        return "Not Ready", 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Step 2: Create a Dockerfile

FROM python:3.9-slim
WORKDIR /app
COPY app.py .
RUN pip install flask
CMD ["python", "app.py"]

Step 3: Build and Push the Image

docker build -t yourusername/flask-probes:latest .
docker push yourusername/flask-probes:latest

Step 4: Deploy the Pod with Probes

Save this as pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: flask-app
spec:
  containers:
  - name: flask-app
    image: yourusername/flask-probes:latest
    ports:
    - containerPort: 5000
    livenessProbe:
      httpGet:
        path: /healthz
        port: 5000
      initialDelaySeconds: 5
      periodSeconds: 5
    readinessProbe:
      httpGet:
        path: /ready
        port: 5000
      initialDelaySeconds: 5
      periodSeconds: 5

Step 5: Apply the Pod and Verify

kubectl apply -f pod.yaml
kubectl get pods -w

Expected output:

NAME        READY   STATUS    RESTARTS   AGE
flask-app   0/1     Running   0          5s
flask-app   0/1     Running   0          10s
flask-app   1/1     Running   0          15s  # Ready after 10s

Step 6: Test the Probes

1. Check liveness probe:
   bash
kubectl describe pod flask-app | grep -A 10 "Liveness"

2. If the probe fails, Kubernetes will restart the pod.

3. Check readiness probe:
   bash
kubectl describe pod flask-app | grep -A 10 "Readiness"

4. If the probe fails, the pod won’t receive traffic (check with kubectl get endpoints).

5. Force a failure (for testing):

6. Modify app.py to return 500 for /healthz and redeploy.
7. Kubernetes will restart the pod after failureThreshold (default: 3) failures.

4. ? Production-Ready Best Practices

Security

• Never expose probe endpoints publicly. Use internal-only ports or network policies.
• Avoid sensitive data in probe responses (e.g., don’t return DB connection status).
• Use HTTPS for probes if your app supports it (but don’t fail if certs are self-signed).

Cost Optimization

• Tune periodSeconds and timeoutSeconds to balance overhead and recovery speed.
• Too frequent = higher CPU/memory usage.
• Too slow = delayed recovery.
• Use startupProbe for slow-starting apps to avoid unnecessary restarts.

Reliability & Maintainability

• Separate liveness and readiness endpoints. Liveness should be lightweight (e.g., ping), while readiness can check dependencies.
• Set initialDelaySeconds to avoid false negatives during startup.
• Use successThreshold and failureThreshold to avoid flaky probes.
• Log probe failures for debugging (e.g., kubectl logs -p <pod> for previous instance).

Observability

• Monitor probe failures in Prometheus/Grafana: promql kube_pod_container_status_last_terminated_reason{reason="Error"} # Liveness failures kube_pod_status_ready{condition="false"} # Readiness failures
• Set alerts for probe failures (e.g., Slack/PagerDuty).
• Include probe metrics in SLOs (e.g., "99.9% of probes succeed over 30 days").

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. Scenario-based:
2. "Your app takes 30 seconds to initialize. What probe should you use?"
   • Answer: startupProbe (with initialDelaySeconds: 30).

3. "A pod is crashing repeatedly. What’s the most likely cause?"

   
   
   • Answer: Liveness probe failing (check kubectl describe pod).

4. Probe type selection:

5. "Which probe type should you use for a database connection check?"

   
   
   • Answer: tcpSocket (for port check) or exec (for a custom script).

6. Configuration traps:

7. "What happens if initialDelaySeconds is set to 0?"
   • Answer: The probe runs immediately, likely failing if the app isn’t ready.
8. "What’s the default failureThreshold?"
   • Answer: 3 (⚠️ easy to forget).

Key Distinctions

7. ? Hands-On Challenge

Challenge:
Deploy a pod with a liveness probe that fails after 20 seconds (simulating a crash). Observe how Kubernetes restarts the pod. Then, add a readiness probe that succeeds after 10 seconds and verify traffic is only sent after readiness.

Solution:
1. Modify app.py to fail the liveness probe after 20s:
python
@app.route('/healthz')
def healthz():
if time.time() - startup_time > 20:
return "Dead", 500
return "OK", 200 2. Deploy the pod with both probes (as in Step 4).
3. Watch the pod restart:
bash
kubectl get pods -w 4. Verify readiness:
bash
kubectl describe pod flask-app | grep -A 5 "Readiness"

Why it works:
- The liveness probe fails after 20s, triggering a restart.
- The readiness probe succeeds after 10s, allowing traffic.

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. Kubernetes Docs: Configure Liveness, Readiness, and Startup Probes
2. Google SRE Book: Monitoring Distributed Systems (See "Black-Box vs. White-Box Monitoring")
3. Kubernetes Best Practices: Probes
4. Debugging Kubernetes Probes (GitHub) (Common pitfalls)