Forward Deployed Engineer 101: Secure Development Practices (OWASP, Secrets Management, Encryption at Rest/In Transit)

Secure Development Practices (OWASP, Secrets Management, Encryption at Rest/In Transit)

Secure Development Practices for Forward Deployed Engineers (FDEs)

A field-ready study guide for real-world, high-stakes environments

What This Is

Secure development isn’t just about writing "secure code"—it’s about shipping resilient systems in constrained, high-risk environments where a single misconfiguration can derail a mission. As an FDE, you’ll deploy ML models to air-gapped networks, build data pipelines for disaster response, or debug a live system during a customer escalation where downtime = lives lost. Example: You’re on-site at a defense contractor, deploying a fraud-detection model to a classified network. The customer’s security team rejects your Docker image because it pulls from docker.io (internet access is forbidden). You now have 4 hours to rebuild the image with offline dependencies, sign it with their internal CA, and deploy it via sneakernet (USB drives). This guide gives you the practical, battle-tested tools to handle these scenarios.

Key Terms & Concepts

• OWASP Top 10: The "cheat sheet" for web app security risks (e.g., injection, broken auth, sensitive data exposure). FDEs use this to prioritize fixes when time is limited. Tools: OWASP ZAP (for scanning), bandit (Python static analysis).
• Secrets Management: Storing API keys, DB passwords, and certs outside code (no hardcoded secrets!). Tools: Vault (HashiCorp), AWS Secrets Manager, sops (Mozilla’s encrypted YAML/JSON). ⚠️ Never commit .env files to Git—even "private" repos leak.
• Encryption at Rest: Data encrypted on disk (e.g., AES-256). Tools: LUKS (Linux full-disk), AWS KMS, TDE (Transparent Data Encryption for SQL Server). Field use: Deploying a DB to a customer’s on-prem server where physical theft is a risk.
• Encryption in Transit: Data encrypted while moving (e.g., TLS 1.2+). Tools: certbot (Let’s Encrypt), OpenSSL, Istio (service mesh for mTLS). Field use: Debugging a "connection refused" error in a customer’s network where TLS 1.0 is blocked.
• Zero Trust: "Never trust, always verify." Assume the network is compromised. Tools: SPIFFE/SPIRE (identity), OPA (policy enforcement). Field use: Deploying a microservice in a DMZ where lateral movement is a threat.
• Immutable Infrastructure: Servers/containers are never modified after deployment—only replaced. Tools: Terraform, Packer, Kubernetes. Field use: Rolling back a failed deployment in a classified environment where SSH access is restricted.
• SBOM (Software Bill of Materials): A manifest of all dependencies (e.g., pip freeze, npm ls). Tools: syft (generates SBOMs), grype (scans for vulnerabilities). Field use: Customer’s security team demands a full dependency list before approving your container.
• Short-Lived Credentials: Temporary tokens (e.g., AWS STS, Vault dynamic secrets). Field use: Generating a 1-hour DB password for a contractor’s script instead of sharing a long-lived key.
• Air-Gapped Deployment: No internet access. Tools: apt-offline, docker save/load, Nexus (local artifact repo). Field use: Deploying to a submarine or a forward operating base.
• ATO (Authority to Operate): Government approval to deploy. Process: STIGs (DISA’s security checklists) → ACAS (vulnerability scans) → eMASS (documentation). Field use: Your customer’s ATO is expiring in 2 weeks—you must patch all CVEs before then.
• IAM (Identity and Access Management): Who can do what. Tools: AWS IAM, Keycloak, OpenID Connect. Field use: Debugging a "403 Forbidden" error where the customer’s IAM roles are misconfigured.
• FIPS 140-2: U.S. government standard for cryptographic modules. Tools: OpenSSL FIPS, AWS KMS FIPS. Field use: Customer rejects your app because it uses MD5 (not FIPS-approved).

Step-by-Step / Field Process

1. Pre-Deployment: Secure by Default

• Generate an SBOM (e.g., syft scan dir:. -o spdx-json=sbom.json).
• Scan for vulnerabilities (e.g., grype sbom:sbom.json).
• Encrypt secrets (e.g., sops --encrypt --kms "arn:aws:kms:..." secrets.yaml > secrets.enc.yaml).
• Build an air-gapped-compatible image (e.g., docker build --no-cache -t myapp:offline . then docker save myapp:offline > myapp.tar).

2. Deployment: Assume the Worst

• SSH into the bastion host (e.g., ssh -J bastion-user@bastion-ip app-user@app-ip).
• Verify TLS certs (e.g., openssl s_client -connect app:443 -showcerts).
• Deploy secrets securely (e.g., kubectl create secret generic db-pass --from-literal=password=$(vault read -field=value secret/db-pass)).
• Test connectivity (e.g., curl -v https://app:443/health—check for TLS errors).

3. Post-Deployment: Monitor & Respond

• Tail logs (e.g., kubectl logs -f pod/myapp --tail=100).
• Check for exposed secrets (e.g., grep -r "password\|secret\|token" /var/log/).
• Rotate credentials (e.g., vault lease revoke -prefix auth/token/lookup/).
• Document for ATO (e.g., update eMASS with SBOM, scan reports, and deployment steps).

Common Mistakes

FDE Interview / War Story Insights

Interview Questions They’ll Ask

1. "You’re deploying to a classified network and the customer’s security team says your container has a CVE. How do you respond?"

2. Answer: "I’d first verify the CVE (e.g., grype scan), then check if it’s exploitable in their environment (e.g., is the vulnerable code even reachable?). If it is, I’d patch it, rebuild the image, and redeploy—documenting every step for their ATO process."

3. "A customer demands a feature that violates the original scope (e.g., storing PII in plaintext). How do you handle it?"

4. Answer: "I’d push back politely but firmly: ‘This violates our security requirements and could put your ATO at risk. Here’s a secure alternative [propose one].’ If they insist, I’d escalate to my manager and document the risk in writing."

War Stories from the Field

• The "Harmless" Log File: A customer’s app was logging API keys in plaintext. Their security team found it during an audit and revoked our ATO. Lesson: Always grep logs for secrets before deployment.
• The Air-Gapped Nightmare: Deploying to a submarine, we forgot to include a dependency in our offline bundle. Spent 6 hours debugging a ModuleNotFoundError before realizing we needed to bring a USB drive with the missing .whl file.
• The TLS Time Bomb: A customer’s network only allowed TLS 1.2, but our app defaulted to TLS 1.3. Spent a day debugging "connection reset" errors before checking openssl s_client.

Quick Check Questions

1. You’re deploying to an environment where you can’t run standard Docker images due to security restrictions. What’s your first step?
2. Answer: Build a distroless or scratch image (e.g., FROM gcr.io/distroless/python3.9) and scan it with grype to ensure no CVEs.

3. Why: Distroless images have no shell or package manager, reducing attack surface.

4. A customer’s security team rejects your app because it uses SHA-1 for hashing. What do you do?

5. Answer: Replace SHA-1 with SHA-256 or SHA-3 (e.g., hashlib.sha256() in Python) and document the change for their ATO.

6. Why: SHA-1 is cryptographically broken and fails FIPS 140-2 compliance.

7. You’re on-site and the customer’s DB admin asks for your root password to "debug an issue." How do you respond?

8. Answer: "I’ll generate a short-lived credential for you using our secrets manager (e.g., vault read -field=value secret/db-debug-pass). Here’s the password—it’ll expire in 1 hour."
9. Why: Never share long-lived credentials. Short-lived tokens limit exposure.

Last-Minute Cram Sheet

1. OWASP Top 3: Injection, Broken Auth, Sensitive Data Exposure.
2. Secrets Tools: Vault, AWS Secrets Manager, sops.
3. Encryption at Rest: LUKS (Linux), AWS KMS, TDE (SQL Server).
4. Encryption in Transit: TLS 1.2+, certbot, OpenSSL.
5. SBOM Tools: syft (generate), grype (scan).
6. Air-Gapped Tools: docker save/load, apt-offline, Nexus.
7. FIPS 140-2: No MD5, SHA-1, or weak ciphers.
8. IAM Principle: Least privilege (e.g., AWS IAM policies).
9. ATO Process: STIGs → ACAS scan → eMASS documentation.
10. ⚠️ Field Traps:
11. Always test in the exact customer environment.
12. Never assume "it works on my machine" = "it works in production."
13. Logs will contain secrets—audit them before deployment.
14. Air-gapped deployments always miss a dependency—bring extras.