TECH **Docker & Kubernetes: Images vs Containers – Layered Filesystem & Copy-on-Write**

Docker & Kubernetes: Images vs Containers – Layered Filesystem & Copy-on-Write

Hyper-practical, zero-fluff guide for engineers in production

1. What This Is & Why It Matters

You’re debugging a Kubernetes pod that’s mysteriously filling up disk space, or you’re trying to optimize a Docker image that’s ballooned to 2GB. Images and containers aren’t just "files" – they’re layered filesystems with copy-on-write (CoW) mechanics. Ignore this, and you’ll waste storage, slow down deployments, or break applications when they least expect it.

Real-world scenario:
- You deploy a new version of your app, but the container fails to start because /var/log is full. The logs should be ephemeral, but they’re persisting across restarts. Why? Because you wrote to a layer that wasn’t supposed to exist.
- Your CI/CD pipeline takes 10 minutes to build an image because it’s re-downloading 1.5GB of dependencies every time. You could cut that to 30 seconds if you understood layers.

This guide gives you:
✅ The mental model to debug storage issues in seconds.
✅ The power to shrink images by 90% and speed up builds.
✅ The confidence to explain to your team why docker commit is a terrible idea.

2. Core Concepts & Components

? Docker Image

• Definition: A read-only template with instructions for creating a container (like an ISO for a VM).
• Production insight: Every layer in an image is cached. If you change a line in your Dockerfile, only layers after that change rebuild. Order your Dockerfile from least to most volatile.

? Container Layer (R/W Layer)

• Definition: A thin, writable layer on top of an image that captures changes made by the container (e.g., logs, temp files).
• Production insight: This layer is ephemeral. If the container dies, the layer is gone. Never store persistent data here.

? Layered Filesystem (UnionFS, OverlayFS, AUFS)

• Definition: A filesystem that stacks multiple read-only layers (image) + one writable layer (container) into a single virtual filesystem.
• Production insight: OverlayFS (default in Docker) is fast but has a 128-layer limit. If your image has 129+ layers, it won’t run.

? Copy-on-Write (CoW)

• Definition: When a container modifies a file, the filesystem copies the file from the read-only layer to the writable layer only at the moment of change.
• Production insight: CoW saves space but can cause performance spikes if you modify large files (e.g., databases). Use volumes for databases, not the container layer.

? Dockerfile Instructions & Layers

? Volume vs Bind Mount

• Volume: Managed by Docker (stored in /var/lib/docker/volumes/). Use for databases, persistent data.
• Bind Mount: Maps a host directory into the container. Use for development (e.g., mounting code).
• Production insight: Volumes are portable across hosts; bind mounts are not. Never use bind mounts in production for persistent data.

? .dockerignore

• Definition: A file that tells Docker which files/dirs to exclude from the build context (like .gitignore).
• Production insight: A missing .dockerignore can bloat your build context, slowing down docker build. Always exclude node_modules, .git, and logs.

3. Step-by-Step Hands-On: Debugging & Optimizing Layers

Prerequisites

• Docker installed (docker --version should work).
• A simple app (we’ll use a Node.js "Hello World" for this example).

Task 1: Inspect Layers in an Image

Goal: See how layers work in a real image.

1. Pull a sample image:
   bash
docker pull node:18-alpine
2. Inspect its layers:
   bash
docker inspect node:18-alpine --format='{{.RootFS.Layers}}'
   Output:
   [sha256:abc123... sha256:def456...]

3. Count the layers:
   bash
docker inspect node:18-alpine --format='{{len .RootFS.Layers}}'
   Output: 5 (or similar).

4. Why this matters: If you see 100+ layers, your image is bloated. Each layer adds overhead.

Task 2: Build an Image and Observe CoW in Action

Goal: See how CoW works when a container modifies a file.

1. Create a Dockerfile:
   dockerfile
FROM alpine:3.18
RUN echo "Original content" > /data.txt
CMD ["cat", "/data.txt"]

2. Build and run:
   bash
docker build -t cow-demo .
docker run --name cow-test cow-demo
   Output: Original content

3. Modify the file inside the container:
   bash
docker exec -it cow-test sh -c "echo 'Modified content' > /data.txt && cat /data.txt"
   Output: Modified content

4. Check the container layer:
   bash
docker diff cow-test
   Output:
   C /data.txt

5. C = Changed (CoW in action! The file was copied to the writable layer).

6. Delete the container and restart:
   bash
docker rm -f cow-test
docker run --name cow-test2 cow-demo
   Output: Original content (the modification is gone because it was in the container layer).

Task 3: Optimize a Bloated Image

Goal: Reduce a 1.2GB Node.js image to 150MB.

1. Start with a naive Dockerfile:
   dockerfile
FROM node:18
WORKDIR /app
COPY . .
RUN npm install
CMD ["node", "index.js"]

2. Build it:
   bash
docker build -t naive-node-app .
docker images | grep naive-node-app
   Output: naive-node-app latest abc123... 1.2GB

3. Optimize it:
   dockerfile
FROM node:18-alpine # 1. Use Alpine (smaller base)
WORKDIR /app
COPY package*.json ./ # 2. Copy package files first (cache npm install)
RUN npm install --production # 3. Only install prod deps
COPY . . # 4. Copy app code last (changes most often)
CMD ["node", "index.js"]

4. Rebuild:
   bash
docker build -t optimized-node-app .
docker images | grep optimized-node-app
   Output: optimized-node-app latest def456... 150MB

5. Why this works:

6. Alpine is smaller than Debian-based images.
7. npm install is cached until package.json changes.
8. --production skips dev dependencies.

4. ? Production-Ready Best Practices

Security

• Never run containers as root. Use USER in your Dockerfile: dockerfile RUN adduser -D appuser && chown -R appuser /app USER appuser
• Scan images for vulnerabilities. Use docker scan or tools like Trivy: bash docker scan optimized-node-app

Cost Optimization

• Use multi-stage builds to discard build-time dependencies: ```dockerfile FROM node:18 AS builder WORKDIR /app COPY . .
  RUN npm install && npm run build

FROM node:18-alpine COPY --from=builder /app/dist ./dist CMD ["node", "dist/index.js"] - Set resource limits in Kubernetes to avoid noisy neighbors:yaml resources:
limits:
cpu: "1"
memory: "512Mi" ```

Reliability & Maintainability

• Tag images with Git SHAs or semantic versions (never latest in prod): bash docker build -t myapp:v1.2.3 .
• Use .dockerignore to exclude unnecessary files: node_modules .git *.log

Observability

• Monitor disk usage in containers:
  bash docker stats --no-stream
• Log to stdout/stderr (not files) so Docker/Kubernetes can capture them: javascript console.log("This goes to stdout");

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical questions:
1. "You need to reduce the size of a Docker image. What’s the most effective step?"
- ✅ Use multi-stage builds (discards build-time dependencies).
- ❌ "Use a smaller base image" (helps, but not as much as multi-stage).

1. "A container’s disk usage grows unexpectedly. What’s the likely cause?"
2. ✅ Writing to the container layer instead of a volume.

3. ❌ "The image is too large" (image size is fixed; container layer grows).

4. "What happens when a container modifies a file from a read-only layer?"

5. ✅ The file is copied to the writable layer (CoW).
6. ❌ "The file is modified in-place" (this would break immutability).

Key trap distinctions:
- Image vs Container: Images are read-only; containers are ephemeral instances.
- Layer vs Volume: Layers are for the filesystem; volumes are for persistent data.
- OverlayFS vs AUFS: OverlayFS is the default (faster, fewer layers); AUFS is older (slower, more layers).

7. ? Hands-On Challenge

Challenge:
You have a Dockerfile that installs Python and a bunch of dependencies. The image is 1.5GB. Optimize it to under 300MB without removing any functionality.

Solution:

# Stage 1: Build
FROM python:3.9-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --user -r requirements.txt

# Stage 2: Runtime
FROM python:3.9-alpine
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["python", "app.py"]

Why it works:
- Uses python:3.9-alpine (smaller base).
- Multi-stage build discards build-time dependencies.
- --user installs Python packages locally (avoids sudo).

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. Docker Docs: Image Layers
2. Kubernetes: Volumes
3. Best Practices for Writing Dockerfiles
4. Trivy: Vulnerability Scanner