TECH **Kubernetes Control Plane & Worker Node Components: Zero-Fluff, Hands-On Guide**

Kubernetes Control Plane & Worker Node Components: Zero-Fluff, Hands-On Guide

(For Engineers Who Need to Debug, Optimize, or Certify—Fast)

1. What This Is & Why It Matters

You’re a DevOps engineer inheriting a Kubernetes cluster that’s slow, unstable, or costing too much. Logs show kube-apiserver timeouts, pods stuck in Pending, and nodes failing health checks. Or worse: you’re prepping for the CKA/CKAD exam, and you know the control plane will be tested—but you’ve only ever deployed apps, not debugged the cluster itself.

This guide is your cheat sheet to:
- Debug why pods aren’t scheduling (hint: it’s often the kube-scheduler or kubelet).
- Optimize cluster performance by tuning control plane components (e.g., kube-apiserver rate limits).
- Secure the cluster by locking down worker node components (e.g., kubelet TLS bootstrapping).
- Pass certifications by nailing the "why" behind each component (e.g., "Why does kube-proxy use iptables vs. ipvs?").

Real-world scenario:
Your team deploys a new microservice, but pods stay in Pending for 10+ minutes. The kubectl describe pod shows:

Warning  FailedScheduling  2m   default-scheduler  0/3 nodes are available: 3 Insufficient cpu.

Root cause? The kube-scheduler isn’t aware of node resources because the kubelet on worker nodes isn’t reporting metrics correctly. Fixing this requires understanding how the control plane and worker nodes actually communicate.

2. Core Concepts & Components

Control Plane Components (Master Node)

1. kube-apiserver

• Definition: The front door to Kubernetes. Validates and processes REST requests (e.g., kubectl apply, kubectl get).
• Production insight:
• If this crashes, your cluster is unusable (no kubectl, no CI/CD pipelines).
• Tune --max-requests-inflight to prevent DDoS from misconfigured clients (default: 400).
• Audit logs (--audit-log-path) are critical for compliance (e.g., SOC2, HIPAA).

2. etcd

• Definition: The cluster’s "source of truth" (key-value store for all cluster state).
• Production insight:
• Backup etcd daily—losing it = losing your cluster.
• Monitor disk I/O—slow etcd = slow kube-apiserver (latency spikes).
• Avoid colocating with noisy neighbors (e.g., databases).

3. kube-scheduler

• Definition: Assigns pods to nodes based on resource requests, taints/tolerations, and affinity rules.
• Production insight:
• Misconfigured predicates (e.g., NodeSelector) can cause pods to stay Pending forever.
• Custom schedulers (e.g., for GPU workloads) require --scheduler-name in pod specs.
• Monitor scheduling_attempts_total metric to detect bottlenecks.

4. kube-controller-manager

• Definition: Runs "control loops" (e.g., ReplicaSet, Deployment, Node controllers) to reconcile desired vs. actual state.
• Production insight:
• Crashes here cause "zombie" resources (e.g., pods stuck in Terminating).
• Tune --concurrent-deployment-syncs (default: 5) for large clusters.
• Watch workqueue_depth—high values mean controllers are falling behind.

Worker Node Components

5. kubelet

• Definition: The "node agent" that registers the node with the cluster and manages pod lifecycle (e.g., starts/stops containers).
• Production insight:
• If kubelet crashes, the node is "NotReady"—pods won’t schedule or restart.
• Enable --rotate-certificates to avoid expired TLS certs (common in long-running clusters).
• Monitor kubelet_volume_stats_* for disk pressure alerts.

6. kube-proxy

• Definition: Maintains network rules (e.g., iptables/ipvs) to route traffic to services.
• Production insight:
• iptables mode is slow for large clusters (use ipvs instead).
• Misconfigured kube-proxy = broken ClusterIP services (pods can’t talk to each other).
• Watch sync_proxy_rules_duration_seconds for latency spikes.

7. Container Runtime (e.g., containerd, cri-o)

• Definition: Pulls images and runs containers (e.g., Docker, but Kubernetes deprecates Docker as a runtime).
• Production insight:
• containerd is the default in modern clusters (faster, lighter than Docker).
• Image pull errors (e.g., ErrImagePull) often trace back to misconfigured imagePullSecrets.

3. Step-by-Step Hands-On: Debug a Broken Cluster

Prerequisites:
- A Kubernetes cluster (local minikube/kind or cloud-based).
- kubectl configured to access the cluster.
- ssh access to a worker node (for kubelet debugging).

Task: Fix a Pod Stuck in Pending

Symptoms:
- kubectl get pods shows a pod stuck in Pending.
- kubectl describe pod <pod-name> shows: Warning FailedScheduling 1m default-scheduler 0/3 nodes are available: 3 Insufficient cpu.

Step 1: Check Node Resources

kubectl top nodes  # Requires metrics-server
kubectl describe nodes | grep -A 5 "Allocatable"

Expected output:

Allocatable:
  cpu:                2
  memory:             4Gi
  pods:               110

If Allocatable is low:
- The kubelet isn’t reporting correct resources (e.g., --max-pods is too low).
- Fix: Edit /var/lib/kubelet/config.yaml on the node and set: yaml maxPods: 110 # Default is 110, but some distros set it lower Then restart kubelet: bash sudo systemctl restart kubelet

Step 2: Verify kube-scheduler is Running

kubectl get pods -n kube-system | grep scheduler

Expected output:

kube-scheduler-control-plane   1/1     Running   0   5d

If missing/crashed:
- Check logs: bash kubectl logs -n kube-system kube-scheduler-control-plane - Common issue: Misconfigured --leader-elect (multiple schedulers fighting for leadership).
Fix: Ensure only one scheduler is running (or set --leader-elect=true).

Step 3: Check kubelet Status on Worker Nodes

SSH into a worker node:

ssh <worker-node-ip>

Check kubelet status:

sudo systemctl status kubelet
journalctl -u kubelet -n 50 --no-pager

Common issues:
1. Expired certificates:
- Error: x509: certificate has expired or is not yet valid
- Fix: Rotate certs:
bash
sudo kubeadm certs renew all
sudo systemctl restart kubelet 2. Disk pressure:
- Error: DiskPressure in kubectl describe node
- Fix: Clean up unused images:
bash
sudo crictl rmi --prune

Step 4: Verify kube-proxy is Working

kubectl get pods -n kube-system | grep kube-proxy
kubectl logs -n kube-system kube-proxy-<pod-id>

Expected output:

I1010 12:00:00.123456       1 proxier.go:314] "Syncing iptables rules"

If logs show errors:
- Issue: iptables rules are corrupted.
- Fix: Restart kube-proxy: bash kubectl delete pod -n kube-system kube-proxy-<pod-id> Or switch to ipvs (faster for large clusters): bash kubectl edit cm -n kube-system kube-proxy Add: yaml mode: "ipvs" Then restart kube-proxy.

Step 5: Test Service Connectivity

Deploy a test pod:

kubectl run test-pod --image=nginx --restart=Never
kubectl expose pod test-pod --port=80 --name=test-service

Check if kube-proxy is routing traffic:

kubectl run curl --image=curlimages/curl -it --rm -- sh
curl test-service

Expected output:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

If it fails:
- Issue: kube-proxy isn’t updating iptables/ipvs rules.
- Fix: Restart kube-proxy (as above) or check node network policies.

4. ? Production-Ready Best Practices

Security

• kube-apiserver:
• Enable --enable-admission-plugins=NodeRestriction to limit kubelet permissions.
• Use --audit-log-path and rotate logs (e.g., with logrotate).
• kubelet:
• Disable anonymous auth: --anonymous-auth=false.
• Enable TLS bootstrapping: --rotate-certificates.
• kube-proxy:
• Use ipvs instead of iptables for large clusters (better performance).

Cost Optimization

• kube-scheduler:
• Use PodTopologySpread to distribute pods across AZs (avoid single-AZ costs).
• Set requests and limits to avoid over-provisioning.
• kubelet:
• Set --eviction-hard to reclaim resources before nodes run out of memory.
• Use --max-pods to limit node density (e.g., --max-pods=50 for small nodes).

Reliability & Maintainability

• etcd:
• Backup daily: ETCDCTL_API=3 etcdctl snapshot save backup.db.
• Monitor etcd_server_leader_changes_seen_total (frequent leader changes = instability).
• kube-controller-manager:
• Set --concurrent-deployment-syncs=10 for large clusters.
• Monitor workqueue_depth (high values = controllers falling behind).

Observability

• Metrics to monitor:
• kube_apiserver_request_duration_seconds (latency).
• kubelet_running_pods (node capacity).
• kube_proxy_sync_proxy_rules_duration_seconds (network performance).
• Logs to watch:
• kube-apiserver: W0101 00:00:00.123456 1 authentication.go:60] "Unable to authenticate the request"
• kubelet: E0101 00:00:00.123456 1 eviction_manager.go:256] "Eviction manager: failed to get summary stats"
• Alerts:
• kube_node_status_condition{condition="Ready",status="false"} (node not ready).
• kube_pod_status_phase{phase="Pending"} > 0 (pods stuck scheduling).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

CKA/CKAD Question Patterns:
1. "Why is my pod stuck in Pending?"
- Answer: Check kube-scheduler logs, node resources (kubectl describe node), and kubelet status.
- Trap: "The pod has nodeSelector but no matching nodes" (forgetting taints/tolerations).

1. "How do you debug kube-apiserver timeouts?"
2. Answer: Check etcd health (etcdctl endpoint status), kube-apiserver logs, and --max-requests-inflight.

3. Trap: "Restart kube-apiserver" (wrong—it’s usually etcd or network issues).

4. "What’s the difference between iptables and ipvs in kube-proxy?"

5. Answer:
   • iptables: Simple, but slow for large clusters (O(n) complexity).
   • ipvs: Faster (O(1) complexity), supports load balancing algorithms (e.g., rr, lc).

6. Trap: "Use iptables for all clusters" (wrong—ipvs is better for >1000 services).

7. "How do you rotate kubelet certificates?"

8. Answer: kubeadm certs renew all + systemctl restart kubelet.
9. Trap: "Delete the certs and restart" (wrong—this breaks the node).

7. ? Hands-On Challenge

Challenge:
Your cluster has 3 nodes, but pods are only scheduling on 2 of them. The 3rd node shows Ready but has no pods. Debug and fix it.

Solution:
1. Check for taints:
bash
kubectl describe node <node-name> | grep Taints
If you see NoSchedule, remove it:
bash
kubectl taint nodes <node-name> key:NoSchedule- 2. Check kubelet logs for registration errors:
bash
journalctl -u kubelet -n 50 --no-pager | grep "failed to register" 3. Verify node resources:
bash
kubectl describe node <node-name> | grep -A 5 "Allocatable"
If Allocatable is 0, check kubelet config (/var/lib/kubelet/config.yaml).

Why it works:
- Taints prevent scheduling unless pods have matching tolerations.
- kubelet registration issues (e.g., wrong --node-ip) can make nodes appear Ready but unusable.

8. ? Rapid-Reference Crib Sheet

Essential Commands:

# Check control plane health
kubectl get pods -n kube-system

# Check node status
kubectl describe node <node-name>

# View kubelet logs
journalctl -u kubelet -n 50 --no-pager

# Check etcd health
ETCDCTL_API=3 etcdctl endpoint status --write-out=table

# Restart kube-proxy
kubectl delete pod -n kube-system kube-proxy-<pod-id>

9. ? Where to Go Next

1. Kubernetes Official Docs: Control Plane
2. Debugging Kubernetes Nodes (Official guide)
3. Kubelet TLS Bootstrapping (Critical for security)
4. Kubernetes the Hard Way (Hands-on cluster setup)
5. CKA Exam Curriculum (What you’ll be tested on)