Forward Deployed Engineer 101: Containerization (Docker, Docker Compose) and Orchestration (Kubernetes Basics)

Containerization (Docker, Docker Compose) and Orchestration (Kubernetes Basics)

Containerization & Orchestration: Field-Ready Study Guide for Forward Deployed Engineers (FDEs)

What This Is

Containerization and orchestration are the backbone of modern deployments—especially in high-stakes environments where uptime, security, and reproducibility matter. As an FDE, you’ll use Docker to package applications (ML models, data pipelines, APIs) into portable, isolated units and Kubernetes to manage them at scale. Real-world example: You’re on-site at a military base deploying a computer vision model for drone footage analysis. The network is air-gapped, the hardware is outdated, and the customer’s security team won’t let you use standard Docker images. You need to: 1. Build a minimal, hardened container image offline.
2. Deploy it to a Kubernetes cluster running on bare metal with no internet access.
3. Debug why the model’s latency spikes under load—while the customer watches over your shoulder.

This guide gives you the practical, battle-tested steps to handle these scenarios.

Key Terms & Concepts

• Docker Image: A lightweight, standalone, executable package that includes everything needed to run a piece of software (code, runtime, system tools, libraries). Think of it as a "snapshot" of your application.

• FDE use case: Shipping a PyTorch model to a classified network where you can’t install Python dependencies manually.

• Dockerfile: A text file with instructions for building a Docker image (e.g., FROM python:3.9, COPY . /app, RUN pip install -r requirements.txt).

• FDE tip: Always pin versions (python:3.9-slim instead of python:latest) to avoid breaking changes in the field.

• Docker Compose: A tool for defining and running multi-container applications (e.g., a Flask app + Redis + Postgres). Uses a docker-compose.yml file.

• FDE use case: Spinning up a local replica of a customer’s environment to debug a data pipeline before deploying to their air-gapped cluster.

• Kubernetes (K8s): An open-source system for automating deployment, scaling, and management of containerized applications. Runs "pods" (groups of containers) on "nodes" (servers).

• FDE tools: kubectl (CLI), helm (package manager), k9s (terminal UI for debugging).

• Pod: The smallest deployable unit in Kubernetes. A pod can run one or more containers (e.g., your app + a sidecar logging container).

• FDE trap: Never run multiple unrelated containers in one pod—it violates the "single responsibility" principle and makes debugging harder.

• Deployment: A Kubernetes object that manages a set of identical pods. Handles rolling updates, rollbacks, and scaling.

• FDE example: Updating a model serving API without downtime during a live mission.

• Service: Exposes a set of pods as a network service (e.g., ClusterIP for internal traffic, NodePort/LoadBalancer for external access).

• FDE tip: Use kubectl port-forward to debug a service locally when you can’t expose it publicly.

• ConfigMap/Secret: Kubernetes objects for storing configuration data (ConfigMap) or sensitive data (Secrets, e.g., API keys, database passwords).

• FDE security: Never hardcode secrets in Dockerfiles or Git. Use kubectl create secret generic or a tool like Vault.

• PersistentVolume (PV) / PersistentVolumeClaim (PVC): Kubernetes objects for managing storage (e.g., databases, model weights). A PV is the actual storage; a PVC is a request for storage.

• FDE use case: Deploying a PostgreSQL database in a Kubernetes cluster where the customer requires data to persist across pod restarts.

• Helm: A package manager for Kubernetes. Lets you define, install, and upgrade complex applications (e.g., helm install my-app ./charts).

• FDE tip: Use Helm to templatize deployments for different customers (e.g., one chart for AWS, another for on-prem).

• Air-Gapped Deployment: Deploying to a network with no internet access. Requires:

• Pre-downloaded Docker images (saved as .tar files).
• Offline dependency caches (e.g., pip download for Python).
• Manual approval chains for every change.

• FDE tools: docker save, docker load, skopeo (for copying images between registries).

• Immutable Infrastructure: The principle that containers (and infrastructure) should never be modified after deployment. If something breaks, you replace the container instead of fixing it in place.

• FDE benefit: Makes rollbacks trivial and debugging reproducible.

Step-by-Step / Field Process

1. Containerize an Application (Docker)

Scenario: You’re deploying a Python Flask API to a customer’s on-prem server. They won’t let you install Python or dependencies manually.

Steps:
1. Write a minimal Dockerfile:
```dockerfile
# Use a slim, pinned base image
FROM python:3.9-slim

# Set working directory
WORKDIR /app

# Copy requirements first (for caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the app
COPY . .

# Run as non-root user (security)
RUN useradd -m appuser && chown -R appuser /app
USER appuser

# Expose port and set entrypoint
EXPOSE 5000
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "app:app"]
2. Build the image:bash
docker build -t my-flask-app:1.0.0 .
3. Test locally:bash
docker run -p 5000:5000 my-flask-app:1.0.0
curl http://localhost:5000/health
4. Save the image for air-gapped deployment:bash
docker save my-flask-app:1.0.0 > flask-app.tar
``
- Transferflask-app.tar` to the customer’s network via USB or approved file transfer.

1. Load the image on the customer’s machine:
   bash
docker load < flask-app.tar

2. Deploy a Multi-Container App (Docker Compose)

Scenario: Your Flask API needs Redis for caching. The customer wants to test it locally before deploying to Kubernetes.

Steps:
1. Write docker-compose.yml:
yaml
version: "3.8"
services:
web:
build: .
ports:
- "5000:5000"
depends_on:
- redis
redis:
image: redis:6.2-alpine
ports:
- "6379:6379" 2. Start the stack:
bash
docker-compose up -d 3. Test the API:
bash
curl http://localhost:5000/cache/set?key=foo&value=bar
curl http://localhost:5000/cache/get?key=foo # Should return "bar" 4. Debug if something breaks:
bash
docker-compose logs web # Check Flask logs
docker-compose exec web sh # Shell into the container

3. Deploy to Kubernetes (Basics)

Scenario: The customer wants to run your Flask + Redis app in their on-prem Kubernetes cluster.

Steps:
1. Create Kubernetes manifests:
- deployment.yaml:
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: flask-app
spec:
replicas: 2
selector:
matchLabels:
app: flask-app
template:
metadata:
labels:
app: flask-app
spec:
containers:
- name: flask-app
image: my-flask-app:1.0.0
ports:
- containerPort: 5000
envFrom:
- configMapRef:
name: flask-config
- name: redis
image: redis:6.2-alpine
ports:
- containerPort: 6379
- service.yaml:
yaml
apiVersion: v1
kind: Service
metadata:
name: flask-service
spec:
selector:
app: flask-app
ports:
- protocol: TCP
port: 80
targetPort: 5000
type: ClusterIP # Internal-only access
- configmap.yaml (for non-sensitive config):
yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: flask-config
data:
REDIS_HOST: "localhost"
LOG_LEVEL: "info" 2. Apply the manifests:
bash
kubectl apply -f configmap.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml 3. Verify the deployment:
bash
kubectl get pods # Check pod status
kubectl logs -l app=flask-app # View logs
kubectl port-forward svc/flask-service 8080:80 # Test locally
curl http://localhost:8080/health 4. Debug common issues:
- Pods stuck in Pending: Check kubectl describe pod <pod-name> for resource constraints or node issues.
- CrashLoopBackOff: Check logs (kubectl logs <pod-name>) or exec into the container (kubectl exec -it <pod-name> -- sh).
- Service not reachable: Verify the selector in the Service matches the pod labels.

4. Handle Air-Gapped Kubernetes Deployments

Scenario: The customer’s cluster has no internet access. You need to deploy your app and its dependencies.

Steps:
1. Pre-download all images:
bash
# On a machine with internet:
docker pull my-flask-app:1.0.0
docker pull redis:6.2-alpine
docker save my-flask-app:1.0.0 > flask-app.tar
docker save redis:6.2-alpine > redis.tar 2. Transfer images to the air-gapped network:
- Use approved media (USB, secure file transfer).
3. Load images on the customer’s nodes:
bash
docker load < flask-app.tar
docker load < redis.tar 4. Push images to a local registry (if available):
bash
docker tag my-flask-app:1.0.0 local-registry:5000/my-flask-app:1.0.0
docker push local-registry:5000/my-flask-app:1.0.0 5. Update Kubernetes manifests to use the local registry:
yaml
image: local-registry:5000/my-flask-app:1.0.0 6. Deploy:
bash
kubectl apply -f .

Common Mistakes

Mistake 1: Using latest Tags in Production

• What happens: Your app breaks when the base image updates unexpectedly.
• Correction: Always pin versions (e.g., python:3.9-slim, redis:6.2-alpine).
• Why: Reproducibility is critical in the field. The customer’s environment won’t change, so your deployments shouldn’t either.

Mistake 2: Running as root in Containers

• What happens: A security audit fails, or an attacker exploits a vulnerability to gain root access on the host.
• Correction: Use a non-root user in your Dockerfile: dockerfile RUN useradd -m appuser && chown -R appuser /app USER appuser
• Why: Principle of least privilege. If the container is compromised, the attacker doesn’t get root on the host.

Mistake 3: Not Testing in the Customer’s Environment

• What happens: Your app works in your lab but fails behind the customer’s firewall (e.g., due to proxy settings, missing CA certificates, or network policies).
• Correction:
• Get a VM or staging environment that mirrors the customer’s setup.
• Test with the exact Docker images and Kubernetes manifests you’ll deploy.
• Why: "Works on my machine" is not an excuse in the field.

Mistake 4: Hardcoding Secrets in Dockerfiles or Git

• What happens: You accidentally leak API keys or database passwords.
• Correction:
• Use Kubernetes Secrets or a tool like Vault.
• Never commit secrets to Git. Use .gitignore and environment variables.
• Why: Security incidents can derail a deployment and damage trust.

Mistake 5: Ignoring Resource Limits

• What happens: Your app consumes all CPU/memory on a node, crashing other critical services.
• Correction: Set resource requests and limits in Kubernetes: yaml resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
• Why: The customer’s cluster is shared. Your app must play nicely with others.

FDE Interview / War Story Insights

1. "How would you deploy a model to an air-gapped environment?"

• What they’re probing: Can you handle offline deployments, security constraints, and manual approval chains?
• How to answer:
• "First, I’d containerize the model with a minimal base image (e.g., python:3.9-slim). I’d pre-download all dependencies (e.g., pip download) and save the Docker image as a .tar file. On-site, I’d load the image onto the air-gapped network, push it to a local registry if available, and deploy it to Kubernetes using manifests that reference the local registry. I’d also bring a USB drive with all dependencies and a backup plan for manual installation if containers aren’t allowed."
• Bonus: Mention tools like skopeo for copying images between registries.

2. "The customer’s security team won’t let us use standard Docker images. What do you do?"

• What they’re probing: Can you adapt to security constraints without sacrificing functionality?
• How to answer:
• "I’d ask for their approved base images (e.g., a hardened RHEL or Ubuntu image). If they don’t have one, I’d work with them to build a minimal, approved image using their build system (e.g., podman or buildah). I’d also ask for a list of allowed packages and dependencies to ensure compliance. If containers are banned entirely, I’d fall back to a manual deployment with a Python virtual environment or a statically compiled binary."
• Key point: Show that you’re flexible and can work within constraints.

3. "You’re on-site and the customer demands a last-minute feature that violates the original scope. How do you respond?"

• What they’re probing: Can you balance customer demands with technical reality?
• How to answer:
• "First, I’d clarify the urgency: Is this blocking the mission, or is it a nice-to-have? If it’s critical, I’d assess the technical feasibility and security implications. If it’s low-risk (e.g., a UI tweak), I might implement it on the spot. If it’s high-risk (e.g., a new data pipeline), I’d push back and explain the trade-offs (e.g., 'This will delay the deployment by 2 days and require a new security review'). I’d also document the change and get written approval from the customer’s technical lead."
• Key point: Show that you’re customer-focused but not a pushover.

4. "How do you debug a Kubernetes pod that’s crashing in production?"

• What they’re probing: Can you troubleshoot under pressure?
• How to answer:
• "First, I’d check the pod status (kubectl get pods) and describe the pod (kubectl describe pod <pod-name>) to see events and resource usage. If it’s crashing, I’d check the logs (kubectl logs <pod-name> --previous for the last crash). If the logs aren’t helpful, I’d exec into the pod (kubectl exec -it <pod-name> -- sh) to inspect the filesystem or run commands manually. If the issue is intermittent, I’d enable debug logging or add a sidecar container for monitoring. If it’s a resource issue, I’d check kubectl top pod and adjust limits."
• Bonus: Mention tools like k9s for faster debugging.

Quick Check Questions

1. You’re deploying to an environment where you can’t run standard Docker images due to security restrictions. What’s your first step?

• Answer: Ask the customer’s security team for their approved base images or build system (e.g., podman, buildah).
• Why: You can’t assume standard tools will work—always ask for constraints upfront.

2. A Kubernetes pod is stuck in Pending state. What’s the most likely cause, and how do you debug it?

• Answer: The most likely cause is insufficient resources (CPU/memory) or node constraints (e.g., taints, node selectors). Debug with kubectl describe pod <pod-name> to see events.
• Why: Pending usually means the scheduler can’t place the pod on a node.

3. You need to deploy a database (PostgreSQL) in Kubernetes. What Kubernetes objects do you use, and why?

• Answer: Use a StatefulSet (for stable network identities and persistent storage), a PersistentVolumeClaim (for data persistence), and a Service (for network access).
• Why: StatefulSet is designed for stateful apps like databases, and PVCs ensure data isn’t lost when pods restart.

Last-Minute Cram Sheet

1. Docker commands:
2. docker build -t my-app:1.0.0 . → Build an image.
3. docker run -p 5000:5000 my-app → Run a container.
4. docker save my-app > app.tar → Save an image for air-gapped deployments.
5. docker load < app.tar → Load an image on an air-gapped machine.

6. docker-compose up -d → Start a multi-container app.

7. Kubernetes commands:

8. kubectl apply -f deployment.yaml → Deploy a manifest.
9. kubectl get pods → List pods.
10. kubectl logs <pod-name> → View pod logs.
11. kubectl describe pod <pod-name> → Debug pod issues.

12. kubectl port-forward svc/my-service 8080:80 → Access a service locally.

13. Key ports:

14. 5000: Default Flask port.
15. 6379: Redis.
16. 5432: PostgreSQL.

17. 80/443: HTTP/HTTPS.

18. Deployment checklist:

19. ✅ Pin all versions (Docker images, dependencies).
20. ✅ Run as non-root user.
21. ✅ Set resource requests/limits.
22. ✅ Use ConfigMaps/Secrets for configuration.
23. ✅ Test in the customer’s environment.

24. ✅ Have a rollback plan.

25. Field traps:

26. ⚠️ Never assume internet access. Always prepare for air-gapped deployments.
27. ⚠️ Test with the exact customer setup. What works in your lab will break behind their firewall.
28. ⚠️ Document everything. The customer will ask for the Dockerfile, Kubernetes manifests, and dependency list.

29. ⚠️ Security reviews take time. Start early and involve the customer’s security team from day one.

30. Acronyms:

31. ATO: Authority to Operate (security approval).
32. ACO: Approval Chain of Operations (who signs off on changes).
33. IAM: Identity and Access Management (who can access what).
34. PVC: PersistentVolumeClaim (Kubernetes storage).
35. PV: PersistentVolume (actual storage in Kubernetes).