TECH **Docker & Kubernetes Monitoring & Debugging: Zero-Fluff, Hands-On Guide**

Docker & Kubernetes Monitoring & Debugging: Zero-Fluff, Hands-On Guide

(kubectl describe, logs, exec, Telepresence)

1. What This Is & Why It Matters

You’re on-call at 3 AM. A critical microservice in your Kubernetes cluster is returning 503 errors. The dashboard shows CPU spiking, but the logs are a firehose of noise. How do you find the root cause in minutes, not hours?

This guide is your battle-tested toolkit for debugging Kubernetes (and Docker) in production. You’ll learn: - kubectl describe: The "X-ray" for pods, nodes, and services—showing events, conditions, and configuration in real time.
- kubectl logs: How to filter, tail, and aggregate logs from multiple containers (without drowning in them).
- kubectl exec: Your "SSH into a container" lifeline—when you need to run commands inside a running pod.
- Telepresence: The "wormhole" that lets you debug a remote Kubernetes service locally (as if it were running on your laptop).

Why this matters in production:
- Downtime = lost revenue. Every minute a service is broken costs money (and sleep).
- Logs lie. Raw logs are useless if you can’t filter, correlate, or access them when pods crash.
- Debugging blind is slow. Without exec or Telepresence, you’re guessing instead of diagnosing.
- Incidents escalate. If you can’t quickly isolate a failing pod, you’ll waste time redeploying or rolling back blindly.

Real-world scenario:
You inherit a legacy Kubernetes cluster where a payment service is failing intermittently. The logs show Connection refused errors, but the service’s pod looks healthy. How do you:
1. Check if the pod is actually receiving traffic? 2. Verify if its dependencies (like a Redis cache) are reachable? 3. Debug the service locally without redeploying?

This guide gives you the exact steps.

2. Core Concepts & Components

kubectl describe

• What it is: A CLI command that dumps human-readable metadata about Kubernetes objects (pods, nodes, services, etc.).
• Production insight: If a pod is CrashLoopBackOff, describe shows the last 5 exit codes and events—saving you from digging through logs.
• Key fields to watch:
• Events: (e.g., FailedScheduling: 0/3 nodes are available)
• Conditions: (e.g., Ready=False with a reason like ContainersNotReady)
• Last State: (exit code and signal of the last crash)

kubectl logs

• What it is: Streams logs from a container in a pod (like docker logs but for Kubernetes).
• Production insight: Use --previous to see logs from a crashed container (critical for debugging CrashLoopBackOff).
• Key flags:
• -f (follow/tail logs)
• --tail=100 (show last 100 lines)
• --since=5m (show logs from the last 5 minutes)
• -c <container> (if a pod has multiple containers)

kubectl exec

• What it is: Runs a command inside a running container (like docker exec).
• Production insight: If a pod is misbehaving, exec lets you inspect files, run curl, or check environment variables without redeploying.
• Security risk: Anyone with exec access can run arbitrary commands (e.g., kubectl exec -it pod -- sh gives a shell). Restrict RBAC permissions!

Telepresence

• What it is: A tool that swaps a remote Kubernetes service with a local process, letting you debug as if the service were running in the cluster.
• Production insight: When a service works locally but fails in Kubernetes, Telepresence lets you test real cluster dependencies (e.g., Redis, databases) without deploying.
• How it works:
• You run telepresence connect to link your local machine to the cluster.
• You run telepresence intercept <service> to redirect traffic from the remote service to your local process.
• Your local code talks to the real cluster (e.g., real Redis, real DB).

Sidecar Containers

• What it is: A secondary container in the same pod as your main app (e.g., for logging, monitoring, or proxies).
• Production insight: If your app logs to stdout but a sidecar (like Fluentd) is misconfigured, logs disappear. Use kubectl logs -c <sidecar> to debug.

Init Containers

• What it is: Containers that run before your main app starts (e.g., to download configs or wait for dependencies).
• Production insight: If an init container fails, your pod stays in Init:Error or Init:CrashLoopBackOff. Check kubectl describe pod for the init container’s exit code.

Liveness & Readiness Probes

• What it is: Kubernetes checks if your app is alive (livenessProbe) and ready to serve traffic (readinessProbe).
• Production insight: A misconfigured livenessProbe can kill healthy pods (e.g., if it checks /health but the endpoint is slow). A missing readinessProbe can send traffic to pods that aren’t ready.

Node Conditions

• What it is: Metadata about a node’s health (e.g., MemoryPressure, DiskPressure, PIDPressure).
• Production insight: If a node is in MemoryPressure, pods may be evicted. Check kubectl describe node to see why.

3. Step-by-Step Hands-On Debugging

Prerequisites

• A running Kubernetes cluster (Minikube, Kind, or cloud-based like EKS/GKE).
• kubectl installed and configured (kubectl get nodes should work).
• A sample app deployed (we’ll use this simple Nginx deployment).

Task: Debug a Failing Pod

Scenario: You deploy an Nginx pod, but it’s stuck in CrashLoopBackOff. Let’s debug it.

Step 1: Check Pod Status

kubectl get pods

Expected output:

NAME                     READY   STATUS             RESTARTS   AGE
nginx-7cdbd8cdc9-abc12   0/1     CrashLoopBackOff   3          2m

What this tells you:
- The pod is not Running.
- It’s crashing and restarting (CrashLoopBackOff).
- It’s been restarted 3 times.

Step 2: Inspect the Pod with kubectl describe

kubectl describe pod nginx-7cdbd8cdc9-abc12

Key things to look for:
- Events: Shows scheduling failures, image pull errors, etc.
- Last State: Exit code and reason for the crash.
- Containers: Command, args, and environment variables.

Example output (truncated):

Name:         nginx-7cdbd8cdc9-abc12
...
Containers:
  nginx:
    ...
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 01 Jan 2023 00:00:00 +0000
      Finished:     Wed, 01 Jan 2023 00:00:01 +0000
...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  2m                 default-scheduler  Successfully assigned default/nginx-7cdbd8cdc9-abc12 to node-1
  Normal   Pulled     2m                 kubelet            Successfully pulled image "nginx:1.25.0"
  Warning  BackOff    1m (x3 over 2m)    kubelet            Back-off restarting failed container

What this tells you:
- The container exited with code 1 (generic error).
- The image was pulled successfully, so it’s not an image issue.
- The BackOff event confirms the crash loop.

Step 3: Check Logs from the Crashed Container

kubectl logs nginx-7cdbd8cdc9-abc12 --previous

Why --previous? Because the current container is already dead, but the previous instance’s logs might show why it crashed.

Example output:

2023/01/01 00:00:00 [emerg] 1#1: no "events" section in configuration
nginx: [emerg] no "events" section in configuration

What this tells you:
- Nginx failed to start because the config file is invalid (missing events section).

Step 4: Fix the Config and Redeploy

1. Edit the ConfigMap or deployment to fix the Nginx config.
2. Redeploy:
   bash
kubectl apply -f nginx-deployment.yaml
3. Verify:
   bash
kubectl get pods -w
   (Wait for Running status.)

Task: Debug a Pod with kubectl exec

Scenario: Your app is running, but it’s returning 500 errors. You suspect a misconfigured environment variable.

Step 1: Get a Shell Inside the Pod

kubectl exec -it nginx-7cdbd8cdc9-abc12 -- sh

What this does:
- Opens an interactive shell (sh) inside the container.
- -it = interactive terminal.

Step 2: Inspect Environment Variables

env | grep DB_

Example output:

DB_HOST=postgres.default.svc.cluster.local
DB_PORT=5432
DB_USER=admin
DB_PASSWORD=  # <-- Empty! This is the bug.

What this tells you:
- The DB_PASSWORD environment variable is missing or empty.

Step 3: Check Files and Run Commands

# Check if a config file exists
ls /etc/nginx/nginx.conf

# Test a connection to a dependency
curl http://redis:6379/ping

Why this matters:
- You can verify if files exist, permissions are correct, or dependencies are reachable.

Step 4: Exit the Shell

exit

Task: Debug a Service with Telepresence

Scenario: Your app works locally but fails in Kubernetes. You suspect it’s because it can’t reach a Redis cache in the cluster.

Step 1: Install Telepresence

# On macOS/Linux
curl -fL https://app.getambassador.io/download/tel2oss/releases/download/v2.14.0/telepresence-darwin-amd64 -o telepresence
chmod +x telepresence
sudo mv telepresence /usr/local/bin/

# On Windows (PowerShell)
Invoke-WebRequest https://app.getambassador.io/download/tel2oss/releases/download/v2.14.0/telepresence-windows-amd64.exe -OutFile telepresence.exe

Verify:

telepresence version

Step 2: Connect to the Cluster

telepresence connect

What this does:
- Creates a two-way network tunnel between your local machine and the Kubernetes cluster.

Step 3: Intercept the Service

telepresence intercept my-service --port 8080

What this does:
- Redirects traffic from my-service in the cluster to your local process (running on port 8080).

Step 4: Run Your App Locally

# Example: Run a Python app locally
python app.py

What this lets you do:
- Your local app can now talk to real cluster dependencies (e.g., Redis, databases).
- You can debug with your local IDE, set breakpoints, etc.

Step 5: Test the Fix

1. Make changes to your local code.
2. Verify it works with the real cluster dependencies.
3. Deploy the fixed version.

Step 6: Clean Up

telepresence leave my-service
telepresence quit

4. ? Production-Ready Best Practices

Security

• Restrict kubectl exec with RBAC:
  ```yaml # Only allow exec for specific pods rules:
• apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
  resourceNames: ["my-app-*"] # Only allow exec on pods with this prefix ```
• Never log secrets: Use kubectl logs --ignore-errors to avoid accidentally printing sensitive data.
• Telepresence security:
• Only intercept services you own.
• Use --mechanism=vpn-tcp (default) for better security than --mechanism=inject-tcp.

Cost Optimization

• Limit log retention: Use kubectl logs --since=1h to avoid downloading gigabytes of logs.
• Telepresence efficiency:
• Only intercept what you need (don’t leave connections open).
• Use --docker-run for local testing without deploying.

Reliability & Maintainability

• Label your pods: Always include app, env, and version labels for easier debugging.
  yaml metadata:
labels:
app: my-service
env: prod
version: v1.2.3
• Use kubectl debug for ephemeral containers:
  bash kubectl debug -it my-pod --image=busybox --target=my-container (Creates a temporary container in the same pod for debugging.)

Observability

• Log aggregation: Use Fluentd/Fluent Bit + Elasticsearch or Loki to centralize logs.
• Metrics: Monitor pod restarts (kube_pod_container_status_restarts_total in Prometheus).
• Alerts: Set up alerts for:
• CrashLoopBackOff (pods restarting too often).
• Pending pods (scheduling failures).
• High 5xx error rates.

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. kubectl logs vs kubectl describe:
2. "A pod is in CrashLoopBackOff. What’s the first command you run?"
   • Answer: kubectl describe pod (to see exit codes and events).

3. "How do you see logs from the previous instance of a crashed pod?"

   
   
   • Answer: kubectl logs --previous.

4. Debugging probes:

5. "A pod is Running but not receiving traffic. What’s likely misconfigured?"

   
   
   • Answer: readinessProbe (pod is alive but not ready).

6. Telepresence use cases:

7. "Your app works locally but fails in Kubernetes. How do you debug it without deploying?"

   
   
   • Answer: Use Telepresence to intercept the service and test locally.

8. Sidecar debugging:

9. "A pod has two containers, and logs aren’t showing up. What’s wrong?"
   • Answer: You forgot to specify the container with -c <sidecar>.

Key ⚠️ Trap Distinctions

• kubectl logs vs kubectl exec:
• logs = read-only logs.
• exec = run commands inside the container (dangerous in prod).
• livenessProbe vs readinessProbe:
• livenessProbe = "Is the app alive?" (restarts pod if failed).
• readinessProbe = "Can the app serve traffic?" (removes pod from service if failed).
• kubectl describe vs kubectl get:
• get = quick status.
• describe = detailed metadata (events, conditions, etc.).

Common Scenario-Based Question

"A pod is stuck in Pending state. What are the most likely causes, and how do you debug them?" Answer:
1. Insufficient resources: Check kubectl describe pod for FailedScheduling events.
2. NodeSelector/tolerations mismatch: Verify the pod’s nodeSelector matches node labels.
3. PersistentVolumeClaim issues: Check kubectl get pvc for Pending status.
4. Taints/tolerations: Run kubectl describe node to see if the node has taints the pod doesn’t tolerate.

7. ? Hands-On Challenge (with Solution)

Challenge

You deploy a pod with the following YAML:

apiVersion: v1
kind: Pod
metadata:
  name: debug-me
spec:
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "echo 'Hello' && sleep 3600"]
  - name: sidecar
    image: busybox
    command: ["sh", "-c", "while true; do echo 'Sidecar log'; sleep 5; done"]

The pod is Running, but when you run kubectl logs debug-me, you only see logs from the sidecar container. How do you see logs from the app container?

Solution

kubectl logs debug-me -c app

Why it works:
- By default, kubectl logs shows logs from the first container in the pod.
- -c app explicitly specifies the app container.

8. ? Rapid-Reference Crib Sheet