TECH **Docker & Kubernetes: Managing ConfigMap-Immutable & Secrets via Sealed-Secrets**

Docker & Kubernetes: Managing ConfigMap-Immutable & Secrets via Sealed-Secrets

A hyper-practical, zero-fluff guide for engineers in production

1. What This Is & Why It Matters

You’re deploying a microservice in Kubernetes, and your app needs: - Database credentials (username/password) - API keys (Stripe, Twilio, AWS) - Environment-specific configs (dev vs. prod URLs, feature flags)

Problem: If you hardcode these in your Docker image or YAML manifests, you violate security best practices (secrets in Git = bad). If you use plain Kubernetes ConfigMaps or Secrets, they’re: - Base64-encoded, not encrypted (anyone with kubectl get secret can decode them).
- Mutable by default (accidental changes can break prod).
- Stored in etcd (a single point of failure if compromised).

Solution:
- ConfigMap-immutable → Lock down configs to prevent drift.
- Sealed-Secrets → Encrypt secrets so they can be safely committed to Git.

Real-world scenario:
You inherit a legacy Kubernetes cluster where: - Secrets are stored in plaintext in Helm charts.
- A junior dev accidentally overwrites a ConfigMap in prod, causing an outage.
- Your security team flags Git history for exposed credentials.

This guide fixes all three.

2. Core Concepts & Components

1. ConfigMap

• Definition: Key-value store for non-sensitive configuration data (e.g., app settings, URLs).
• Production insight: If you don’t make it immutable, a misplaced kubectl edit can break prod.

2. ConfigMap-immutable

• Definition: A ConfigMap with immutable: true, preventing all modifications.
• Production insight: Reduces etcd load (Kubernetes doesn’t watch for changes) and prevents config drift.

3. Secret

• Definition: Like a ConfigMap, but for sensitive data (passwords, tokens). Stored base64-encoded in etcd.
• Production insight: Base64 is not encryption—anyone with kubectl get secret -o yaml can decode it.

4. SealedSecret (via sealed-secrets controller)

• Definition: A Kubernetes custom resource that encrypts a Secret so it can be safely stored in Git.
• Production insight: Only the cluster’s SealedSecret controller can decrypt it—even if someone steals your Git repo, they can’t access the secret.

5. kubeseal CLI

• Definition: CLI tool to encrypt a Secret into a SealedSecret.
• Production insight: You run this locally to generate encrypted YAML files for Git.

6. SealedSecret Controller

• Definition: A Kubernetes operator that decrypts SealedSecret resources into regular Secrets.
• Production insight: Must be installed in the cluster before deploying SealedSecrets.

7. etcd

• Definition: Kubernetes’ key-value store where Secrets and ConfigMaps are stored.
• Production insight: If etcd is compromised, all Secrets are exposed. SealedSecrets mitigate this risk.

8. RBAC (Role-Based Access Control)

• Definition: Controls who can read/modify Secrets and ConfigMaps.
• Production insight: Even with SealedSecrets, restrict get access to Secrets to only necessary service accounts.

3. Step-by-Step Hands-On

Prerequisites

• A running Kubernetes cluster (Minikube, Kind, or cloud-based).
• kubectl installed and configured.
• kubeseal CLI installed (brew install kubeseal or download here).

Task: Deploy an Immutable ConfigMap + SealedSecret in 10 Minutes

Step 1: Install the Sealed-Secrets Controller

# Install the controller in the cluster
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/controller.yaml

# Verify it's running
kubectl get pods -n kube-system | grep sealed-secrets

Expected output:

sealed-secrets-controller-abc123   1/1     Running   0   1m

Step 2: Create a Local Key Pair (for kubeseal)

# Fetch the public key from the controller
kubeseal --fetch-cert > pub-cert.pem

Why? kubeseal needs this to encrypt secrets locally.

Step 3: Create an Immutable ConfigMap

# configmap-immutable.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  API_URL: "https://api.example.com"
immutable: true  # <-- This locks it down

kubectl apply -f configmap-immutable.yaml

Verify:

kubectl get configmap app-config -o yaml | grep immutable

Expected output:

immutable: true

Step 4: Create a Secret (to be sealed)

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=  # echo -n "admin" | base64
  password: cGFzc3dvcmQxMjM=  # echo -n "password123" | base64

⚠️ Never commit this to Git!

Step 5: Seal the Secret

kubeseal --format yaml --cert pub-cert.pem < secret.yaml > sealed-secret.yaml

What just happened?
- kubeseal encrypted the Secret into a SealedSecret.
- The output (sealed-secret.yaml) is safe to commit to Git.

Step 6: Deploy the SealedSecret

kubectl apply -f sealed-secret.yaml

Verify:

kubectl get sealedsecret
kubectl get secret db-credentials -o yaml

Expected output:
- The SealedSecret is created.
- The Secret is automatically decrypted by the controller.

Step 7: Use the ConfigMap and Secret in a Pod

# pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app
    image: nginx
    envFrom:
    - configMapRef:
        name: app-config
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

kubectl apply -f pod.yaml

Verify:

kubectl exec -it app-pod -- env | grep DB_

Expected output:

DB_USERNAME=admin
DB_PASSWORD=password123

4. ? Production-Ready Best Practices

Security

• Never commit plain Secrets to Git—always use SealedSecrets.
• Rotate SealedSecret keys periodically (the controller’s private key is a single point of failure).
• Restrict RBAC for Secrets: ```yaml # Example: Only allow the "app" service account to read secrets apiVersion: rbac.authorization.k8s.io/v1 kind: Role rules:
• apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["db-credentials"]
  verbs: ["get"] ```
• Use immutable: true for all ConfigMaps unless you explicitly need mutability.

Cost Optimization

• Immutable ConfigMaps reduce etcd load (Kubernetes doesn’t watch for changes).
• Avoid storing large files in ConfigMaps (etcd has a 1MB limit per object).

Reliability & Maintainability

• Name SealedSecrets clearly (e.g., prod-db-credentials-sealed).
• Tag resources with environment: prod for easier filtering.
• Test SealedSecrets in a staging cluster before deploying to prod.

Observability

• Monitor SealedSecret controller logs for decryption failures: bash kubectl logs -n kube-system sealed-secrets-controller-abc123
• Alert on ConfigMap/Secret changes (use tools like Prometheus + Alertmanager).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. "How do you prevent a ConfigMap from being modified?"
2. ✅ Answer: Set immutable: true.

3. ❌ Trap: "Use RBAC" (RBAC restricts access but doesn’t prevent edits).

4. "What’s the difference between a Secret and a SealedSecret?"

5. ✅ Answer: Secret is base64-encoded; SealedSecret is encrypted and safe for Git.

6. ❌ Trap: "They’re the same" (they’re not—SealedSecret is encrypted).

7. "How do you rotate a SealedSecret key?"

8. ✅ Answer: Re-encrypt the Secret with a new cert and redeploy.
9. ❌ Trap: "Delete the old SealedSecret first" (this breaks existing deployments).

Key ⚠️ Trap Distinctions

7. ? Hands-On Challenge

Challenge:
You have a Secret with an API key (api-key: c3VwZXItc2VjcmV0). Seal it, deploy it, and verify the pod can access it.

Solution:

# 1. Create secret.yaml
echo "apiVersion: v1
kind: Secret
metadata:
  name: api-key
type: Opaque
data:
  key: c3VwZXItc2VjcmV0" > secret.yaml

# 2. Seal it
kubeseal --format yaml --cert pub-cert.pem < secret.yaml > sealed-secret.yaml

# 3. Deploy
kubectl apply -f sealed-secret.yaml

# 4. Verify
kubectl get secret api-key -o yaml | grep key

Why it works:
- kubeseal encrypts the Secret into a SealedSecret.
- The controller decrypts it into a Secret in the cluster.

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. Sealed-Secrets GitHub – Official docs.
2. Kubernetes Secrets Docs – Best practices.
3. Immutable ConfigMaps – Official guide.
4. Kubernetes Security Best Practices – Hardening guide.

Final Tip:
Always test SealedSecrets in a non-prod cluster first. If the controller isn’t running, your SealedSecret won’t decrypt—and your app will fail silently. ?