Forward Deployed Engineer 101: CI/CD Pipelines (GitHub Actions, GitLab CI, Jenkins – Building, Testing, Deploying)

CI/CD Pipelines (GitHub Actions, GitLab CI, Jenkins – Building, Testing, Deploying)

CI/CD Pipelines (GitHub Actions, GitLab CI, Jenkins) – Field-Ready Study Guide

What This Is

CI/CD pipelines automate the build, test, and deployment of software—critical for FDEs who must deliver solutions in high-stakes, constrained environments. Example: You’re deploying a real-time satellite imagery analysis tool to a classified network with no internet access. The pipeline must run offline, validate security compliance, and deploy via air-gapped media. Or, during a disaster response, you’re patching a data pipeline mid-mission when a customer reports a critical failure—your CI/CD system must roll back, test, and redeploy in minutes without downtime.

Key Terms & Concepts

• CI (Continuous Integration): Automatically build and test code on every commit (e.g., GitHub Actions running pytest on a Python repo).
• CD (Continuous Deployment/Delivery): Automate deployment to staging/production (e.g., GitLab CI deploying a Docker container to Kubernetes).
• Pipeline as Code: Define CI/CD workflows in YAML/JSON (e.g., .github/workflows/deploy.yml).
• Artifact: A deployable output (e.g., a Docker image, Python wheel, or compiled binary).
• Runner: The machine executing pipeline jobs (e.g., GitHub-hosted runners, self-hosted GitLab runners on-prem).
• Secrets Management: Securely inject credentials (e.g., AWS keys, SSH keys) into pipelines (GitHub Secrets, HashiCorp Vault).
• Approval Gates: Manual approvals for deployments (e.g., GitLab’s when: manual or Jenkins’ "Promote" stage).
• Blue/Green Deployment: Run two identical environments; switch traffic to the new one after validation (common in enterprise).
• Canary Deployment: Roll out changes to a small subset of users first (e.g., 5% of traffic).
• Air-Gapped Pipeline: A CI/CD system with no internet access (e.g., Jenkins on a classified network, using local mirrors for dependencies).
• Immutable Infrastructure: Deployments replace entire systems (e.g., Terraform + Kubernetes) rather than patching in-place.
• Shift-Left Security: Integrate security checks early (e.g., trivy scanning Docker images in CI).

Step-by-Step / Field Process

1. Assess the Environment

• Action: SSH into the customer’s bastion host → check network constraints (firewalls, proxies, air-gap) → verify tooling (Docker? Kubernetes? Bare metal?).
• Example: Run docker info to check if Docker is installed and configured. If not, fall back to scp + systemd for deployments.

2. Design the Pipeline

• Action: Sketch the pipeline stages (build → test → scan → deploy) in a whiteboard or .md file.
• Example: For a Python app: yaml # .github/workflows/deploy.yml jobs:
build:
runs-on: self-hosted # Customer-provided runner
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements.txt
- run: pytest
deploy:
needs: build
runs-on: self-hosted
steps:
- run: scp app.py user@prod-server:/opt/app/
- run: ssh user@prod-server "systemctl restart myapp"

3. Handle Secrets Securely

• Action: Never hardcode secrets. Use:
• GitHub/GitLab Secrets (for cloud runners).
• HashiCorp Vault (for on-prem).
• Environment files (for air-gapped systems, encrypted with gpg).
• Example: Inject an SSH key in GitHub Actions: ```yaml
• name: Deploy
  env:
  SSH_KEY: ${{ secrets.PROD_SSH_KEY }}
  run: |
  echo "$SSH_KEY" > key.pem
  chmod 600 key.pem
  ssh -i key.pem user@prod-server "uptime" ```

4. Test in the Customer’s Environment

• Action: Run a dry-run deployment first. Tail logs (journalctl -u myapp -f) and validate with a quick script: python # test_deployment.py import requests resp = requests.get("http://localhost:8000/health") assert resp.status_code == 200, "Deployment failed!"

5. Roll Back on Failure

• Action: Always define a rollback step. For Kubernetes: ```yaml
• name: Rollback
  if: failure()
  run: kubectl rollout undo deployment/myapp For bare metal:bash ssh user@prod-server "systemctl stop myapp && cp /opt/app/backup/app.py /opt/app/ && systemctl start myapp" ```

6. Document the Pipeline

• Action: Write a 1-pager for the customer with:
• Pipeline stages.
• How to trigger deployments (e.g., git tag v1.0.0).
• Rollback procedure.
• Contact info for emergencies.

Common Mistakes

• Mistake: Assuming the customer’s environment matches your lab.
  Correction: Always test in the exact target environment. Use a "staging" runner that mirrors production (same OS, network rules, etc.).

• Mistake: Hardcoding secrets in pipeline files.
  Correction: Use secrets management tools. For air-gapped systems, encrypt secrets with gpg and store them on a USB drive.

• Mistake: Skipping rollback tests.
  Correction: Simulate a failed deployment (e.g., exit 1 in a script) and verify rollback works before go-live.

• Mistake: Ignoring pipeline flakiness.
  Correction: Add retry logic (retry: 3 in GitLab CI) and log failures to a dashboard (e.g., Grafana).

• Mistake: Not validating artifacts.
  Correction: Scan Docker images with trivy or grype in CI. For air-gapped systems, use offline vulnerability databases.

FDE Interview / War Story Insights

• Interview Question: "How would you design a CI/CD pipeline for a classified network with no internet access?" Answer: Use a self-hosted runner, pre-download dependencies (e.g., pip download + docker save), and deploy via physical media (USB/SSD). Validate artifacts with offline tools like syft for SBOMs.

• War Story: A customer demanded a last-minute feature during a go-live week. The FDE:

• Paused: "Let’s validate the impact first."
• Tested: Wrote a quick script to simulate the change in staging.
• Negotiated: "This adds risk. Can we deploy it in Phase 2?"

• Documented: Added the request to the backlog with a risk assessment.

• Tricky Situation: The pipeline fails in production but works in staging. The FDE:

• Checks DNS resolution (nslookup prod-db vs nslookup staging-db).
• Compares environment variables (env | grep DB).
• Reproduces the issue with a minimal test case (e.g., curl -v http://prod-api/health).

Quick Check Questions

1. Scenario: You’re deploying to an environment where you can’t run standard Docker images due to security restrictions. What’s your first step?
   Answer: Build a minimal base image (e.g., alpine) and scan it with trivy to identify vulnerabilities. Then, work with the customer’s security team to whitelist approved images.
   Why: Security constraints often require custom images or non-Docker deployments (e.g., systemd services).

2. Scenario: The customer’s Jenkins pipeline fails with "Permission denied" when pulling from Git. What do you check?
   Answer: Verify the SSH key or PAT (Personal Access Token) is correctly injected into the Jenkins credential store. Test manually with ssh -T [email protected].
   Why: Jenkins often runs as a service account with restricted permissions.

3. Scenario: A deployment to Kubernetes fails with ImagePullBackOff. What’s the most likely cause?
   Answer: The image registry credentials are missing or incorrect. Check the Kubernetes secret (kubectl get secrets) and verify the image exists in the registry.
   Why: Kubernetes needs explicit credentials to pull private images.

Last-Minute Cram Sheet

1. GitHub Actions: on: push triggers on commits; on: workflow_dispatch for manual runs.
2. GitLab CI: rules: for conditional jobs; artifacts: to pass files between stages.
3. Jenkins: Jenkinsfile defines pipelines; sh 'command' for shell steps.
4. Secrets: Never log secrets (echo ${{ secrets.TOKEN }} fails in GitHub Actions).
5. Docker: docker build --no-cache to avoid layer caching issues.
6. Kubernetes: kubectl rollout status deployment/myapp to check deployments.
7. Air-Gap: Use docker save + docker load to move images offline.
8. Security: Scan images with trivy image myapp:latest.
9. Rollback: kubectl rollout undo or systemctl revert.
10. ⚠️ Field Trap: Always test rollback procedures—customers will ask for them during outages.