Forward Deployed Engineer 101: On‑Premise Deployment (Air‑gapped Environments, Physical Hardware, Networking)

On‑Premise Deployment (Air‑gapped Environments, Physical Hardware, Networking)

On-Premise Deployment (Air-Gapped, Physical Hardware, Networking) – Field-Ready Study Guide

What This Is

On-premise deployment means installing, running, and maintaining software inside a customer’s physical or virtual infrastructure—often with no internet access, strict security controls, and hardware constraints. As an FDE, you’ll deploy ML models in classified networks, build data pipelines for disaster response, or debug a critical system during a customer’s go-live week. Example: You’re at a military base deploying a computer vision model on air-gapped servers. The customer’s firewall blocks all outbound traffic, their hardware is 5+ years old, and the model must run in real-time for a mission-critical operation. You have 48 hours to make it work—no cloud, no external dependencies, and no room for error.

Key Terms & Concepts

• Air-gapped Deployment: Installing software on a network with no internet access. Requires offline dependency bundles (e.g., .deb/.rpm files, Docker images on USB drives), manual approval chains (e.g., ATO, ACO), and physical media (DVDs, encrypted drives).
• Bastion Host: A jump server (often Linux) that acts as the single entry point into a secure network. You SSH into it first, then hop to internal machines. Example: ssh -J bastion-user@bastion-host internal-user@internal-server.
• Offline Dependency Management: Pre-downloading all libraries, binaries, and containers (e.g., pip download -r requirements.txt, docker save > model.tar). Tools: Nexus Repository, Artifactory, dpkg/apt-offline.
• Hardware Constraints: Deploying on old/embedded systems (e.g., Intel NUCs, Raspberry Pis, or servers with no GPU). You’ll need optimized builds (e.g., TensorFlow Lite, ONNX runtime) and manual kernel tuning.
• Network Segmentation: The customer’s network is split into VLANs, DMZs, or classified enclaves. You’ll need firewall rules, proxy configs, and static IPs to move data between zones.
• Configuration as Code (CaC): Managing infrastructure with Terraform, Ansible, or Kubernetes manifests—but in air-gapped environments, you’ll often hand-edit configs or use USB drives to transfer files.
• Immutable Infrastructure: Deploying pre-built, read-only artifacts (e.g., Docker containers, VM images) to ensure consistency. In air-gapped environments, you’ll sign and verify these artifacts manually.
• Zero-Trust Networking: No implicit trust—every request is authenticated. You’ll work with mTLS, SPIFFE/SPIRE, or customer-specific IAM (e.g., DoD’s PKI certificates).
• Data Gravity: The idea that data stays where it is (e.g., in a classified network). You’ll bring the code to the data, not the other way around.
• Deployment Checklist (ACO/ATO): Authority to Operate (ATO) and Approval to Connect (ACO) are mandatory for government/defense deployments. You’ll need STIG compliance, vulnerability scans, and customer sign-off before go-live.
• Hotfix vs. Patch: In air-gapped environments, hotfixes (quick, manual fixes) are common because patches (automated updates) are impossible. Example: scp hotfix.py user@server:/opt/app/ && ssh user@server "sudo systemctl restart app".
• Customer-Specific Constraints: Every environment is unique. You’ll encounter custom firewalls (e.g., Palo Alto), proprietary protocols (e.g., DDS, ZeroMQ), or hardware dongles for licensing.

Step-by-Step / Field Process

1. Discovery & Requirements Gathering (The "Ask vs. Infer" Phase)

• Action: Meet the customer (sysadmins, security teams, end-users) to understand:
• Network topology (VLANs, firewalls, proxies).
• Hardware specs (CPU, RAM, GPU, storage).
• Security policies (ACO/ATO, STIGs, allowed ports).
• Data flow (where data lives, how it moves, who owns it).
• Field Tip: Bring a network diagram template (e.g., draw.io) and ask: "Can you walk me through how data gets from Point A to Point B?"
• Example Command: nmap -sP 192.168.1.0/24 (if allowed) to scan the network.

2. Build an Offline Dependency Bundle

• Action: Pre-download everything your software needs:
• Python: pip download -r requirements.txt --dest ./deps
• Docker: docker save my-image:latest > my-image.tar
• System packages: apt-offline set offline.sig --install-packages python3,nginx
• Field Tip: Use a USB 3.0 drive (encrypted with VeraCrypt) to transfer files. Label it with version, date, and contents.
• Example: For a Python app: bash mkdir -p offline_deps && cd offline_deps pip download -r ../requirements.txt --python-version 3.8 --platform manylinux1_x86_64 --only-binary=:all: tar -czvf python_deps.tar.gz .

3. Deploy to a Staging Environment (Mirror the Customer’s Setup)

• Action: Replicate the customer’s environment exactly (same OS, same hardware if possible).
• Use Vagrant or Packer to build VMs.
• For air-gapped testing, use a local Nexus/Artifactory mirror.
• Field Tip: If the customer uses RHEL 7.9, don’t test on Ubuntu 22.04—it will break.
• Example: Deploy a Docker container in an air-gapped VM: ```bash # On your machine (with internet) docker save my-app:latest > my-app.tar scp my-app.tar user@staging-server:/tmp/

# On the staging server (air-gapped) docker load < /tmp/my-app.tar docker run -d --name my-app -p 8080:8080 my-app:latest ```

4. Transfer & Install in the Customer Environment

• Action:
• Physically transport the dependency bundle (USB, DVD, or secure transfer).
• Verify checksums (sha256sum bundle.tar.gz).
• Install dependencies (e.g., dpkg -i *.deb, pip install --no-index --find-links=./deps -r requirements.txt).
• Deploy the app (e.g., docker load < my-app.tar, systemctl start my-service).
• Field Tip: Never assume the customer’s environment matches your staging. Always:
• Check /etc/os-release (OS version).
• Run lscpu (CPU info).
• Test network connectivity (ping, nc -zv, curl).

5. Debug & Hotfix (When Things Break)

• Action:
• Reproduce the issue (e.g., tail -f /var/log/app.log).
• Check network/firewall (iptables -L, ss -tulnp).
• Write a quick script to validate data (e.g., Python script to check API responses).
• Push a hotfix (e.g., scp fix.py user@server:/opt/app/ && ssh user@server "sudo systemctl restart app").
• Field Tip: Always have a rollback plan. Example: bash # Before deploying, back up the current version cp /opt/app/app.py /opt/app/app.py.bak # If the hotfix fails, revert cp /opt/app/app.py.bak /opt/app/app.py

6. Document & Hand Off

• Action:
• Write a runbook (how to deploy, troubleshoot, and roll back).
• Train the customer (e.g., "Here’s how to restart the service if it crashes").
• Leave behind a USB drive with all dependencies (labeled, encrypted).
• Field Tip: Assume you won’t be there next week. Document like your life depends on it.

Common Mistakes

FDE Interview / War Story Insights

1. "The Customer Demands a Last-Minute Feature That Violates Scope"

• Interviewer’s Goal: Test your ability to push back while keeping the customer happy.
• How to Respond:
• Acknowledge the request: "I understand why this is important for your mission."
• Explain the risk: "Adding this feature now could delay the deployment and introduce security vulnerabilities."
• Offer alternatives: "What if we deliver this in Phase 2, after we’ve stabilized the current release?"
• Escalate if needed: "Let me check with my team to see if we can safely accommodate this."

2. "The Firewall Blocks Your App—Now What?"

• Interviewer’s Goal: Test your debugging skills in constrained environments.
• How to Respond:
• Check logs first: journalctl -u my-service -f
• Test connectivity: nc -zv <host> <port> or curl -v http://localhost:8080
• Ask for firewall rules: "Can you show me the Palo Alto/iptables rules for this VLAN?"
• Workaround: "Can we temporarily whitelist this IP/port for testing?"

3. "You’re Deploying to a Classified Network—How Do You Handle Secrets?"

• Interviewer’s Goal: Test your security awareness.
• How to Respond:
• Never hardcode secrets. Use customer-provided HSMs (Hardware Security Modules) or PKI certificates.
• For local testing: Use Vault in dev mode (but never in production).
• For air-gapped: Use encrypted USB drives (e.g., VeraCrypt) or customer-approved key management.

Quick Check Questions

1. You’re deploying to an environment where you can’t run standard Docker images due to security restrictions. What’s your first step?

Answer: Check if the customer allows rootless Docker, Podman, or custom container runtimes (e.g., Kata Containers). If not, build a static binary (e.g., with PyInstaller or Go) or deploy as a systemd service.
Why: Docker requires root privileges and kernel modules, which are often blocked in secure environments.

2. The customer’s network has no DNS resolution. How do you ensure your app can communicate with other services?

Answer: Use static IPs in configs and hardcode hostnames in /etc/hosts. Example:

echo "192.168.1.10 db-server" | sudo tee -a /etc/hosts

Why: Without DNS, your app will fail to resolve hostnames unless you manually map them.

3. You’re deploying a Python app, but the customer’s servers have no internet access. How do you install dependencies?

Answer: Pre-download all Python wheels (pip download -r requirements.txt) and install them offline:

pip install --no-index --find-links=./deps -r requirements.txt

Why: pip install requires internet access—you must bring the dependencies with you.

Last-Minute Cram Sheet

1. ⚠️ Always test in the exact customer environment—what works in your lab will break behind their firewall.
2. Offline dependency tools: pip download, docker save, apt-offline, yumdownloader.
3. Bastion host command: ssh -J bastion-user@bastion-host internal-user@internal-server.
4. Check OS version: cat /etc/os-release.
5. Check CPU/RAM: lscpu, free -h.
6. Test network connectivity: nc -zv <host> <port>, curl -v http://localhost:8080.
7. Firewall rules: iptables -L, ss -tulnp.
8. ACO/ATO: Authority to Connect / Authority to Operate—mandatory for government deployments.
9. STIG: Security Technical Implementation Guide—compliance checklist for DoD systems.
10. Data gravity: Bring the code to the data, not the data to the code.