Forward Deployed Engineer 101: Speed and Pragmatism (MVP, 80/20, Iteration in the Field)

Speed and Pragmatism (MVP, 80/20, Iteration in the Field)

Speed and Pragmatism (MVP, 80/20, Iteration in the Field) – Field-Ready Study Guide

What This Is

Speed and pragmatism are the difference between a solution that ships and one that dies in PowerPoint. As a Forward Deployed Engineer (FDE), you’ll often work in high-pressure environments—deploying ML models in classified networks, building data pipelines for disaster response, or de-escalating a customer crisis during a go-live. The goal isn’t perfection; it’s shipping something that works today while leaving room to improve. Example: A defense customer needs a real-time object detection model running on edge devices in a denied comms environment. You don’t have time for a 6-month R&D cycle—you take an off-the-shelf YOLO model, quantize it for their hardware, wrap it in a Flask API, and deploy it on a ruggedized NVIDIA Jetson with a USB camera. It’s not "production-grade" by Silicon Valley standards, but it saves lives in the field.

Key Terms & Concepts

• MVP (Minimum Viable Product): The smallest working version of a solution that delivers just enough value to unblock the mission. Example: A disaster response team needs a dashboard to track supply drops—your MVP is a Python script that scrapes a shared spreadsheet, plots points on a Leaflet map, and updates every 5 minutes. No auth, no fancy UI, just a URL they can refresh.
• 80/20 Rule (Pareto Principle): 80% of the impact comes from 20% of the effort. Focus on the 20% that matters. Example: A customer wants a "fully automated" data pipeline, but 80% of their pain comes from a single manual CSV upload step. Automate just that first.
• Iteration in the Field: Shipping small, frequent updates based on real-world feedback. Example: Deploy a model with a /feedback endpoint that logs misclassifications, then retrain nightly with the new data.
• Ask vs. Infer: The customer’s stated "ask" (e.g., "We need a blockchain-based audit log") vs. the actual problem (e.g., "We need to prove no one tampered with the data"). Always dig deeper.
• Technical Debt (Field Edition): Not all debt is bad. Sometimes you need to hardcode a config or skip tests to meet a deadline—but document it and plan to fix it in the next iteration. Example: TODO: Replace this hardcoded API key with Vault integration before ATO review.
• Bastion Host: A secure jump server used to access internal networks. You’ll SSH into this first, then pivot to other machines. Example: ssh -J bastion-user@bastion-host internal-user@internal-server.
• Offline Dependencies: In air-gapped environments, you can’t pip install or docker pull. Pre-download wheels, containers, and models onto a USB drive or internal mirror. Tools: pip download, docker save, apt-offline.
• Hotfix vs. Patch: A hotfix is an immediate, minimal change to fix a critical issue (e.g., a one-line Python script to filter bad data). A patch is a planned update with tests and docs. Hotfixes are for fires; patches are for maintenance.
• Customer Proxy: Someone on the customer’s team who can unblock you (e.g., get firewall rules changed, approve a deployment). Cultivate this relationship early.
• Shadow IT: When the customer’s "real" IT team is too slow, and users set up their own solutions (e.g., a Raspberry Pi running a critical script under someone’s desk). Document it, then either kill it or make it official.
• ATO (Authorization to Operate): The formal approval to deploy a system in a government environment. Without it, your code isn’t running. Example: A 6-month ATO process might require you to freeze your codebase 3 months before deployment—plan accordingly.
• ACO (Authority to Connect): Permission to plug a device into a network. Example: "We can’t deploy the sensor until the ACO is signed—it’s stuck in legal."

Step-by-Step / Field Process

1. Discovery: Separate the Ask from the Infer

• Action: Run a 30-minute "pain interview" with the customer. Ask:
• "What’s the one thing that, if fixed today, would make your life 10x easier?"
• "What’s the manual workaround you’re using now?" (This is your MVP.)
• "What’s the worst-case scenario if this breaks?" (This tells you how much testing you need.)
• Output: A one-pager with:
• The ask (what they say they want).
• The infer (what you think they actually need).
• The MVP (the smallest thing you can build to test the infer).

2. Build the MVP (Fast, Ugly, Functional)

• Action:
• Use the simplest tools possible. Example: Need a dashboard? Start with a Python script + streamlit or plotly dash. Need a model? Use fastai or scikit-learn before jumping to PyTorch.
• Hardcode configs, skip tests, and use mock data if needed. Example:
  python
# TODO: Replace with real API call after ATO
def get_customer_data():
return pd.read_csv("mock_data.csv")
• Deploy locally first. Example: python app.py --host 0.0.0.0 --port 8080.
• Output: A working prototype you can demo in <48 hours.

3. Deploy to the Customer’s Environment (The Hard Part)

• Action:
• Step 1: Get access. SSH into the bastion host, check network connectivity, and verify you can reach dependencies (e.g., curl http://internal-db:5432).
• Step 2: Test in isolation. Run your code in a container or VM that mimics the customer’s environment. Example:
  bash
docker run --rm -it -v $(pwd):/app -p 8080:8080 python:3.9-slim bash
pip install -r requirements.txt
python app.py
• Step 3: Deploy incrementally. Start with a read-only version (e.g., a dashboard that only displays data) before adding write functionality.
• Step 4: Monitor like your job depends on it (because it does). Example:
  bash
# Tail logs in real-time
tail -f /var/log/app.log | grep -i "error\|exception"
• Output: A deployed MVP with basic monitoring.

4. Iterate Based on Real-World Feedback

• Action:
• Step 1: Set up a feedback loop. Example: Add a /feedback endpoint that logs user input:
  python
@app.route("/feedback", methods=["POST"])
def feedback():
with open("feedback.log", "a") as f:
f.write(f"{datetime.now()}: {request.json}\n")
return {"status": "thanks!"}
• Step 2: Prioritize fixes based on impact. Example: If users keep asking for a CSV export, add it in the next iteration.
• Step 3: Automate the boring stuff. Example: If you’re manually running a script every day, schedule it with cron or Airflow.
• Output: A v2 that actually solves the customer’s problem.

5. Harden for Production (If Time Permits)

• Action:
• Step 1: Add tests. Start with the critical path (e.g., "Does the model return predictions?").
• Step 2: Replace hardcoded values with configs. Example: Move API keys to environment variables or a secrets manager.
• Step 3: Document the "runbook" for the customer. Example:
  # How to restart the app
ssh user@bastion
ssh user@internal-server
sudo systemctl restart myapp
• Output: A system that won’t collapse when you leave.

Common Mistakes

Mistake 1: Over-Engineering the MVP

• What Happens: You spend a week setting up Kubernetes, CI/CD, and monitoring for a prototype that the customer might scrap.
• Correction: Start with a single Python script or a Jupyter notebook. Example: Need a data pipeline? Write a pandas script first, then refactor into Airflow later.

Mistake 2: Ignoring the Customer’s Environment

• What Happens: Your code works in your lab but fails in the customer’s network due to firewalls, outdated Python versions, or missing dependencies.
• Correction: Always test in the exact customer environment. Example: If they’re on RHEL 7, test on RHEL 7. If they’re air-gapped, test offline.

Mistake 3: Not Documenting Technical Debt

• What Happens: You hardcode an API key, skip tests, or use a deprecated library. Six months later, no one remembers why, and the system breaks.
• Correction: Document debt in the code. Example: python # TODO: Replace this with OAuth2 before ATO review (due 2024-06-01) API_KEY = "12345"

Mistake 4: Assuming the Customer Knows What They Want

• What Happens: You build exactly what the customer asked for, but it doesn’t solve their problem.
• Correction: Always ask "why" 5 times. Example:
• Customer: "We need a blockchain-based audit log."
• You: "Why?"
• Customer: "Because we need to prove no one tampered with the data."
• You: "Why not just use a cryptographic hash + timestamp?"
• Customer: "Oh, that would work too."

Mistake 5: Not Having a Rollback Plan

• What Happens: You deploy a hotfix, it breaks something, and now the system is down.
• Correction: Always have a rollback plan. Example:
• Before deploying, take a snapshot of the database.
• Keep the old version running in parallel (e.g., app-v1 and app-v2).
• Know how to revert (e.g., git checkout main && git pull && sudo systemctl restart app).

FDE Interview / War Story Insights

Interview Question: "How do you handle scope creep in the field?"

• What They’re Testing: Can you balance speed with discipline?
• How to Answer:
• "First, I clarify the ask vs. the infer. If the new request aligns with the infer, I’ll scope it into the next iteration. If it’s a distraction, I’ll push back with data—e.g., ‘This will take 2 weeks and delay the MVP by 30%. Is that worth it?’"
• "I also use the ‘parking lot’ technique—write down the request and revisit it later."

War Story: The Customer Demands a Feature That Violates Scope

• Situation: You’re on-site deploying a model for a defense customer. The colonel in charge demands a new feature that wasn’t in the original contract—and it would require a 6-month ATO re-review.
• How to Handle:
• Step 1: Acknowledge the request. "I understand why this is important. Let me check with my team on the feasibility."
• Step 2: Assess the impact. "Adding this feature would require an ATO re-review, which could delay the deployment by 6 months. Is that acceptable?"
• Step 3: Propose a workaround. "What if we deliver the MVP first, then add this feature in the next iteration? That way, you get value sooner."
• Step 4: Escalate if needed. If the customer insists, loop in your program manager and their contracting officer.

Interview Question: "How do you prioritize when everything is on fire?"

• What They’re Testing: Can you triage under pressure?
• How to Answer:
• "I use the Eisenhower Matrix: urgent vs. important. If a system is down, that’s urgent and important—I’ll drop everything to fix it. If it’s a ‘nice to have,’ I’ll defer it."
• "I also ask: ‘What’s the worst that happens if we don’t do this?’ If the answer is ‘people die,’ it gets top priority. If it’s ‘the dashboard looks ugly,’ it can wait."

Quick Check Questions

1. You’re deploying to an environment where you can’t run standard Docker images due to security restrictions. What’s your first step?

• Answer: Check if the customer has an approved base image (e.g., a hardened RHEL image). If not, build a minimal container from scratch using podman or buildah and submit it for security review.
• Why: Security restrictions often allow custom-built containers if they meet compliance requirements.

2. The customer’s data pipeline keeps failing because their input data is malformed. They want you to "fix the data." What do you do first?

• Answer: Write a quick Python script to validate the data and log the errors. Example: python def validate_data(df):
errors = []
if "timestamp" not in df.columns:
errors.append("Missing timestamp column")
if df["value"].isna().any():
errors.append("Null values in 'value' column")
return errors Then, share the errors with the customer and ask: "Should we fix the data at the source, or add validation in the pipeline?"
• Why: You need to diagnose the problem before proposing a solution.

3. You’re on-site, and the customer’s IT team says your code can’t run because it requires Python 3.9, but they only have Python 3.6. What’s your next move?

• Answer: Check if you can downgrade your dependencies to work with Python 3.6. If not, ask if they can install a newer Python version in a virtual environment (e.g., pyenv). If all else fails, rewrite the critical parts in a language they support (e.g., Bash, Go).
• Why: You’re there to solve their problem, not force them to upgrade.

Last-Minute Cram Sheet

1. MVP = Minimum Viable Product. Ship something that works today, even if it’s ugly.
2. 80/20 Rule: Focus on the 20% of effort that delivers 80% of the impact.
3. Always test in the customer’s environment. ⚠️ What works in your lab will break behind their firewall.
4. Ask vs. Infer: The customer’s "ask" is often not the real problem.
5. Hotfix > Patch. Fix the fire first, then document and test later.
6. Bastion host: ssh -J user@bastion internal-user@internal-server.
7. Air-gapped deployments: Pre-download dependencies with pip download or docker save.
8. ATO (Authorization to Operate): No ATO = no deployment. Plan for it early.
9. Rollback plan: Always know how to revert. Example: git checkout main && sudo systemctl restart app.
10. Document technical debt. Example: # TODO: Replace hardcoded API key before ATO review.