Forward Deployed Engineer 101: Networking Essentials (DNS, Firewalls, VPNs, Proxies – Debugging Connectivity)

Networking Essentials (DNS, Firewalls, VPNs, Proxies – Debugging Connectivity)

Networking Essentials for Forward Deployed Engineers

DNS, Firewalls, VPNs, Proxies – Debugging Connectivity

What This Is

As an FDE, you’re often dropped into high-stakes environments where networking is the invisible force breaking your deployment. Maybe you’re deploying an ML model on a classified network where DNS resolution fails silently, or debugging a data pipeline in a disaster zone where the VPN drops every 10 minutes. Unlike lab environments, real-world networks have firewalls with undocumented rules, proxies that strip headers, and DNS servers that lie. This guide gives you the field-tested playbook to diagnose, fix, and work around networking issues—fast.

Field Example:
You’re on-site at a defense contractor deploying a real-time object detection system. The model works in staging, but in production, it fails to fetch updates from an internal API. The logs show ConnectionTimeout. The customer insists it’s your code. You SSH into the bastion host, run dig, and discover their DNS server is misconfigured—it’s resolving the API hostname to a decommissioned IP. You bypass DNS with a /etc/hosts entry, push a hotfix, and buy time to escalate the real issue (while the customer thinks you’re a wizard).

Key Terms & Concepts

• DNS (Domain Name System): The phonebook of the internet. Translates human-readable names (api.example.com) to IPs (10.20.30.40). Tools: dig, nslookup, host, systemd-resolved.
• Firewall: A security barrier that filters traffic based on rules (IP, port, protocol). Real-world: Often misconfigured or undocumented. Tools: iptables, nftables, ufw, firewalld, AWS Security Groups, Azure NSGs.
• VPN (Virtual Private Network): Encrypted tunnel to a private network. Field trap: VPNs can drop, split-tunnel, or throttle traffic. Tools: OpenVPN, WireGuard, Cisco AnyConnect, openconnect.
• Proxy: Middleman that forwards requests (often for caching, filtering, or anonymity). Types: Forward (outbound), reverse (inbound), transparent (invisible). Tools: Squid, Nginx, Envoy, mitmproxy.
• Bastion Host: A hardened jump server to access internal networks. Why it matters: Often the only way into a customer’s VPC. Tools: SSH (ssh -J), AWS Session Manager, Teleport.
• Port Forwarding: Redirecting traffic from one port to another (e.g., localhost:8080 → remote:80). Tools: ssh -L, ssh -R, kubectl port-forward.
• MTU (Maximum Transmission Unit): Largest packet size allowed on a network. Field trap: VPNs or proxies can fragment packets, breaking large payloads. Fix: ping -M do -s 1472 <host> to test.
• Split Horizon DNS: Different DNS responses based on the requester’s network.
  Example: internal.example.com resolves to 10.0.0.1 internally but 203.0.113.1 externally.
• TLS/SSL Inspection: Firewalls decrypting HTTPS traffic for inspection. Field trap: Breaks pinned certificates or custom CAs. Fix: Add the customer’s CA to your trust store (update-ca-certificates).
• Network Namespace: Isolated network stack (used in containers). Tools: ip netns, nsenter, Docker --network.
• Zero Trust: "Never trust, always verify" security model. Implication: Even internal traffic may require authentication (e.g., mutual TLS).
• ACL (Access Control List): Rules defining who can access what.
  Example: AWS S3 bucket policies, Kubernetes NetworkPolicies.

Step-by-Step / Field Process

Debugging Connectivity: The FDE Playbook

Scenario: Your app can’t reach an API. The customer says, "It works for us."

1. Reproduce the Issue Locally
2. SSH into the bastion host (or customer-provided jump box):
   bash
ssh -J user@bastion user@internal-host

3. Test basic connectivity:
   bash
ping api.example.com # Is the host reachable?
curl -v https://api.example.com # Verbose HTTP request
nc -zv api.example.com 443 # Is the port open? (Netcat)

4. Check DNS Resolution

5. Query DNS directly (bypass /etc/hosts):
   bash
dig api.example.com +short # Quick IP lookup
nslookup api.example.com # Cross-check with another tool

6. Field trap: If DNS fails, try hardcoding the IP in /etc/hosts as a temporary fix.

7. Inspect Firewall Rules

8. Check local firewall:
   bash
sudo iptables -L -n -v # Linux
sudo ufw status # Ubuntu
9. Ask the customer for their firewall rules (they may not know—check their docs or ask their network team).

10. Test if a port is blocked:
    bash
telnet api.example.com 443 # If it hangs, the port is blocked

11. Debug Proxies

12. Check if a proxy is intercepting traffic:
    bash
env | grep -i proxy # Check environment variables
curl -x http://proxy.example.com:8080 https://api.example.com

13. Field trap: Some proxies strip headers (e.g., Host). Use curl -H "Host: api.example.com" to test.

14. Validate TLS/SSL

15. Check certificate validity:
    bash
openssl s_client -connect api.example.com:443 -showcerts

16. Field trap: If the customer uses a custom CA, add it to your trust store:
    bash
sudo cp customer-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

17. Escalate or Work Around

18. If the issue is on the customer’s side (e.g., misconfigured firewall), document the fix and escalate.
19. If you need a quick workaround:
    • Use a different port (e.g., 8443 instead of 443).
    • Bypass DNS with /etc/hosts.
    • Use a VPN or SSH tunnel to route traffic.

Common Mistakes

FDE Interview / War Story Insights

Interview Questions They’ll Ask

1. "You’re deploying to a classified network and your app can’t reach an external API. What do you do?"

2. Answer: First, confirm if external access is allowed (ATO/ACO restrictions). If not, work with the customer to set up an internal mirror or offline dependency cache. If allowed, debug step-by-step (DNS → firewall → proxy → TLS).

3. "The customer’s VPN keeps dropping, and they blame your app. How do you prove it’s not your code?"

4. Answer: Reproduce the issue with a simple tool (e.g., curl or ping). Show that the VPN drops even for basic requests. Escalate to their network team with logs.

5. "You’re on-site and the customer demands a feature that violates the original scope. How do you respond?"

6. Answer: Acknowledge the request, but clarify the trade-offs (e.g., "This will delay the current milestone"). Document the ask and escalate to your PM/lead. Never say "no" outright—frame it as a prioritization question.

War Story: The Case of the Vanishing Packets

Scenario: You’re deploying a data pipeline in a disaster response zone. The pipeline pulls data from a satellite feed, but packets keep disappearing. The customer insists it’s your code.

What Happened:
- The satellite feed was behind a transparent proxy that dropped packets larger than 1400 bytes.
- The customer’s network team didn’t document this.
- Your app was sending 1500-byte packets (default MTU).

How You Fixed It:
1. Ran ping -M do -s 1472 <satellite-feed> to confirm MTU issues.
2. Adjusted the app’s MTU (ip link set mtu 1400 dev eth0).
3. Added a retry mechanism for dropped packets.
4. Lesson: Always test MTU in constrained networks.

Quick Check Questions

1. You’re deploying to an air-gapped network, and your app fails to start because it can’t reach pypi.org. What’s your first step?
2. Answer: Check if the app has offline dependencies (e.g., a requirements.txt mirror). If not, work with the customer to set up an internal PyPI mirror or use a pre-built container with all dependencies.

3. Why: Air-gapped networks have no internet access—you must bring dependencies with you.

4. Your app works in staging but fails in production with ConnectionRefused. The customer says their firewall allows all traffic. What do you check next?

5. Answer: Test if the port is actually open (nc -zv <host> <port>). If it’s open, check if the app is binding to the correct IP (e.g., 0.0.0.0 vs 127.0.0.1).

6. Why: ConnectionRefused means the port is reachable but nothing is listening (or the firewall is silently dropping traffic).

7. You’re debugging a slow API call in a customer’s network. curl takes 5 seconds, but ping is instant. What’s likely the issue?

8. Answer: Check for a proxy or TLS inspection. Proxies add latency, and TLS inspection can slow down HTTPS traffic.
9. Why: ping uses ICMP (no proxy/TLS), while curl uses HTTP/HTTPS (subject to proxies and TLS overhead).

Last-Minute Cram Sheet

1. DNS Debugging:
2. dig +short example.com → Quick IP lookup.
3. nslookup example.com → Cross-check DNS.

4. host example.com → Simple DNS query.

5. Firewall Checks:

6. sudo iptables -L -n -v → List Linux firewall rules.
7. sudo ufw status → Ubuntu firewall status.

8. telnet example.com 443 → Test if a port is open.

9. Proxy Debugging:

10. env | grep -i proxy → Check proxy env vars.

11. curl -x http://proxy:8080 https://example.com → Test with proxy.

12. TLS Debugging:

13. openssl s_client -connect example.com:443 -showcerts → Inspect certs.

14. update-ca-certificates → Add a custom CA.

15. Port Forwarding:

16. ssh -L 8080:localhost:80 user@remote → Local port forward.

17. ssh -R 8080:localhost:80 user@remote → Remote port forward.

18. MTU Testing:

19. ping -M do -s 1472 example.com → Test MTU.

20. ip link set mtu 1400 dev eth0 → Adjust MTU.

21. VPN Checks:

22. openconnect --protocol=anyconnect vpn.example.com → Connect to Cisco VPN.

23. ip a → Check VPN interface.

24. Network Namespace:

25. ip netns list → List namespaces.

26. nsenter -t <PID> -n → Enter a namespace.

27. ⚠️ Field Traps:

28. ⚠️ Always test in the exact customer environment—what works in your lab will break behind their firewall.
29. ⚠️ Proxies strip headers—use curl -H "Host: example.com" to test.
30. ⚠️ DNS lies—hardcode IPs in /etc/hosts as a temporary fix.

31. ⚠️ TLS inspection breaks pinned certs—ask the customer if they do it.

32. Port Numbers to Memorize:

    
    
    • 22 → SSH
    • 53 → DNS
    • 80 → HTTP
    • 443 → HTTPS
    • 3306 → MySQL
    • 5432 → PostgreSQL
    • 6379 → Redis
    • 27017 → MongoDB

Final Tip: When in doubt, reproduce the issue with the simplest tool possible (ping, curl, nc). Complex apps fail for simple reasons—don’t overcomplicate debugging.