Forward Deployed Engineer 101: Working in Air‑Gapped and Classified Environments (Data Handling, Clearances)

Working in Air‑Gapped and Classified Environments (Data Handling, Clearances)

Field-Ready Study Guide: Working in Air-Gapped & Classified Environments

What This Is

As a Forward Deployed Engineer (FDE), you’ll often work in air-gapped (no internet) or classified (restricted access) environments—think military bases, intelligence agencies, or secure enterprise data centers. Your job isn’t just coding; it’s deploying, debugging, and delivering under constraints (no downloads, strict approvals, limited tooling). Example: You’re on-site at a DoD facility, deploying a real-time object detection model for drone footage, but the network blocks PyPI, Docker Hub, and even pip install. You must pre-package dependencies, get them approved, and deploy manually—while ensuring the model meets accuracy and latency SLAs. If you mess up, the mission fails, and the customer loses trust.

Key Terms & Concepts

• Air-Gapped Deployment: Installing software on a network with no internet access—requires physical media (USB, DVD), offline dependency bundles, and manual approval chains (e.g., burning a DVD with a Python wheelhouse for a classified system).
• Classified Environment (SCIF, SAP, NOFORN): A Sensitive Compartmented Information Facility (SCIF) or Special Access Program (SAP) where data handling is restricted by clearance level (e.g., Top Secret/SCI). NOFORN = "No Foreign Nationals" (data can’t leave the country).
• Data Diode / One-Way Transfer: A hardware device that allows data to flow in only one direction (e.g., from unclassified → classified, but not back). Used to prevent leaks.
• Offline Dependency Management: Pre-downloading all dependencies (Python wheels, Docker images, OS packages) in a clean room (internet-connected but isolated) and transferring them via approved media (e.g., pip download -r requirements.txt → burn to DVD).
• Cross-Domain Solution (CDS): A hardware/software system that securely transfers data between networks of different classification levels (e.g., NIPRNet → SIPRNet). Often requires manual approval per transfer.
• Authority to Operate (ATO): A formal approval from the customer’s security team (e.g., DoD’s Risk Management Framework (RMF)) that your software is allowed to run in their environment. No ATO = no deployment.
• Bastion Host / Jump Box: A single hardened server that acts as the entry point to a classified network. You SSH into it first, then access other machines.
• FIPS 140-2 / Common Criteria: Cryptographic standards required for classified systems. Your software must use approved algorithms (e.g., AES-256, SHA-256) and validated libraries (e.g., OpenSSL FIPS module).
• Hardened OS / STIG Compliance: A locked-down operating system (e.g., RHEL with DISA STIGs) where root access is restricted, services are disabled, and auditing is enforced. You’ll need sudo privileges just to install a package.
• Data Tagging / Marking: Classified data must be labeled (e.g., SECRET//NOFORN) and handled according to its dissemination controls. Mishandling = security violation.
• Zero Trust in Air-Gapped: Even without internet, assume breach. Use mutual TLS (mTLS), short-lived certs, and strict IAM (e.g., SPIFFE/SPIRE for service identity).
• Physical Security (Two-Person Rule, Escorts): In some environments, you can’t be alone with classified systems—a cleared escort must accompany you at all times.

Step-by-Step / Field Process

1. Pre-Deployment: Dependency Packaging & ATO Prep

• Action: Before stepping on-site, pre-package everything in a clean room (internet-connected but isolated).
• For Python: pip download -r requirements.txt --dest ./wheels
• For Docker: docker save my_image:latest > my_image.tar
• For OS packages: yum --downloadonly --downloaddir=./rpms install <package>
• Action: Burn dependencies to approved media (DVD, encrypted USB) and get them scanned by the customer’s security team.
• Action: Submit ATO package (System Security Plan, STIG checklist, network diagrams) to the customer’s Information System Security Manager (ISSM).

2. On-Site: Transfer & Validate Dependencies

• Action: Physically transfer media to the classified environment (e.g., hand-deliver a DVD to the SCIF).
• Action: Verify hashes (SHA-256) of all files to ensure no tampering: bash sha256sum -c checksums.txt
• Action: Install dependencies offline: bash pip install --no-index --find-links=./wheels -r requirements.txt docker load < my_image.tar

3. Deployment: Manual & Scripted Installs

• Action: SSH into the bastion host (if remote access is allowed): bash ssh -J bastion-user@bastion-host target-user@target-host
• Action: Deploy manually (no kubectl apply or terraform apply—often blocked):
• Copy files via scp or rsync (if allowed).
• Run scripts step-by-step (no CI/CD pipelines).
• Action: Validate deployment: bash curl -v http://localhost:8080/health # Check app status journalctl -u my-service -f # Tail logs

4. Debugging: No Stack Overflow, No Google

• Action: Reproduce the issue locally (if possible) in a mirrored environment (same OS, same restrictions).
• Action: Write quick scripts to validate data: python # Check if a file has the right format with open("data.csv") as f:
print(f.readline()) # First line should be header
• Action: Log everything (no remote monitoring): bash script -a debug_session.log # Record terminal session

5. Post-Deployment: ATO & Documentation

• Action: Update ATO documentation (e.g., "Added new dependency X, verified FIPS compliance").
• Action: Train the customer on manual updates (no auto-updates in air-gapped).
• Action: Leave a runbook with:
• Exact commands to restart services.
• Troubleshooting steps (e.g., "If logs show Permission denied, run sudo ausearch -m avc -ts recent to check SELinux denials").

Common Mistakes

FDE Interview / War Story Insights

1. "You’re on-site and the customer demands a last-minute feature that wasn’t in the original scope. How do you respond?"

• Field Answer:
• Clarify the ask: "What’s the mission impact if we don’t deliver this?"
• Assess risk: "Does this require new dependencies? New ATO approvals?"
• Propose a phased approach: "We can deliver a minimal version in 2 weeks, but full functionality will take 4 weeks and require a new ATO."
• Escalate if needed: "Let me check with my team on dependency risks."
• Why? Customers in classified environments often change requirements—your job is to balance speed with security.

2. "You’re deploying to a system where you can’t use Docker. What do you do?"

• Field Answer:
• Ask for constraints: "Is it a kernel restriction? A security policy?"
• Fall back to systemd/native installs: Package your app as an RPM/DEB or run it as a systemd service.
• Use Podman (if allowed): It’s Docker-compatible but rootless and more secure.
• Why? Many classified systems block Docker due to security concerns (e.g., container escapes).

3. "The customer’s security team rejects your ATO package. What’s your next step?"

• Field Answer:
• Ask for specifics: "Which controls failed? Can we see the scan results?"
• Work with their ISSM: "What’s the minimal change needed to pass?"
• Propose a waiver: "Can we get a temporary ATO with mitigations?"
• Why? ATO rejections are common—your job is to negotiate a path forward.

Quick Check Questions

1. You’re deploying a Python app to an air-gapped system, but pip install fails with "No module named '_ssl'". What’s the issue, and how do you fix it?

• Answer: The system is missing OpenSSL development headers (required for Python’s SSL module). Pre-package openssl-devel (RHEL) or libssl-dev (Debian) in your RPM/DEB bundle.
• Explanation: Air-gapped systems often strip development packages to reduce attack surface.

2. You need to transfer a 10GB dataset from an unclassified to a classified network. What’s your first step?

• Answer: Check if a Cross-Domain Solution (CDS) is available. If not, request approval for a manual transfer (e.g., burning to DVD, using a data diode).
• Explanation: Never assume data can move freely—classified networks have strict transfer rules.

3. Your app works in staging but fails in production with "Permission denied" errors. The customer says SELinux is enforcing. What do you do?

• Answer: Check SELinux denials:
  bash sudo ausearch -m avc -ts recent sudo audit2allow -a # Generate a policy to allow the denied action Then request a policy update from the customer’s security team.
• Explanation: SELinux is common in classified systems—you must work within its rules.

Last-Minute Cram Sheet

1. ⚠️ Always test in the exact customer environment—what works in your lab will break behind their firewall.
2. Pre-package everything: pip download, docker save, yum --downloadonly.
3. FIPS compliance: Use openssl list -digest-algorithms to check approved hashes.
4. ATO = Authority to Operate—no ATO, no deployment.
5. Bastion host: ssh -J user@bastion target-host.
6. Data diodes: One-way data transfer (unclassified → classified only).
7. STIGs: Security Technical Implementation Guides (e.g., DISA STIGs for RHEL).
8. Two-person rule: Some environments require an escort for classified systems.
9. NOFORN: "No Foreign Nationals"—data can’t leave the country.
10. ⚠️ Never assume you can debug interactively—write scripts instead.