Forward Deployed Engineer 101: APIs and Integration Patterns (REST, gRPC, Webhooks, Auth – OAuth/JWT)

APIs and Integration Patterns (REST, gRPC, Webhooks, Auth – OAuth/JWT)

APIs and Integration Patterns (REST, gRPC, Webhooks, Auth – OAuth/JWT)

What This Is

As an FDE, you’ll spend 80% of your time integrating systems—whether it’s stitching together a classified sensor network, piping real-time disaster response data into a dashboard, or hotfixing an OAuth flow during a customer’s go-live week. APIs are the glue, but the real challenge isn’t the code—it’s operating under constraints: air-gapped networks, strict auth policies, or a customer who insists their "REST API" is actually a SOAP endpoint with a JSON wrapper. Example: You’re on-site at a military base, and their "REST API" for drone telemetry is gRPC over a custom protocol. Your job is to make it work today, not debate architecture.

Key Terms & Concepts

• REST (Representational State Transfer): Stateless, resource-based APIs using HTTP methods (GET, POST, etc.). Field tools: curl, httpie, Postman, or Python’s requests library. ⚠️ Customers often call any HTTP API "REST" even if it’s RPC-style.
• gRPC: High-performance RPC framework using Protocol Buffers (protobuf) and HTTP/2. Used in low-latency systems (e.g., drone control, trading platforms). Field tools: grpcurl, protoc, buf (for schema management).
• Webhooks: "Reverse APIs" where the server pushes events to a customer-provided URL. Common in SaaS integrations (e.g., Slack, Stripe). ⚠️ Always validate payloads—customers will send malformed data.
• OAuth 2.0: Delegated authorization framework (e.g., "Log in with Google"). Field tools: oauth2-proxy, hydra (Ory), or Python’s authlib. Key flows:
• Authorization Code: For web apps (most secure).
• Client Credentials: For machine-to-machine (e.g., internal services).
• ⚠️ Implicit Flow: Deprecated—never use it.
• JWT (JSON Web Tokens): Self-contained tokens for stateless auth. Field tools: jwt.io (debugger), PyJWT (Python). ⚠️ Always validate exp, iss, and aud claims.
• API Gateway: Reverse proxy for routing, auth, and rate limiting. Field tools: Kong, Apigee, AWS API Gateway, or Envoy. Critical for on-prem deployments where you can’t modify the backend.
• Service Mesh: Decentralized API management (e.g., Istio, Linkerd). Used in Kubernetes-heavy environments. ⚠️ Overkill for simple integrations—start with an API gateway.
• Idempotency: Ensuring repeated API calls don’t cause side effects (e.g., duplicate payments). Field pattern: Use idempotency keys (e.g., Idempotency-Key: abc123).
• Rate Limiting: Protecting APIs from abuse. Field tools: nginx rate limiting, Redis + token bucket. ⚠️ Customers will hit limits—always log and alert.
• Schema Registry: Centralized schema management (e.g., for protobuf, Avro). Field tools: Confluent Schema Registry, buf (for gRPC). Critical for long-lived integrations.
• mTLS (Mutual TLS): Both client and server authenticate with certificates. Used in zero-trust environments (e.g., defense, finance). Field tools: openssl, cert-manager (K8s), or step-ca.
• Postman/Newman: API testing tools. Newman runs Postman collections in CI/CD. ⚠️ Always export collections as code (not just GUI clicks).

Step-by-Step / Field Process

1. Discovery: Map the Terrain

• Action: SSH into the customer’s bastion host (or use their VPN) and run: bash curl -v https://api.customer.com/health # Check if the API is up nslookup api.customer.com # Verify DNS resolution traceroute api.customer.com # Check network path
• Ask the customer:
• "What’s the exact error you’re seeing?" (Get logs, not interpretations.)
• "What’s the auth flow?" (OAuth? API keys? mTLS?)
• "Are there rate limits or IP whitelisting?" (Common in enterprise.)
• Infer: If they say "REST," test with: bash curl -X POST https://api.customer.com/endpoint -H "Content-Type: application/json" -d '{}' If it returns 415 Unsupported Media Type, it’s likely gRPC or SOAP.

2. Auth: Get a Token (Without Breaking Anything)

• OAuth 2.0 (Client Credentials):
  bash curl -X POST https://auth.customer.com/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=YOUR_ID&client_secret=YOUR_SECRET" ⚠️ Never hardcode secrets—use environment variables or a secrets manager.
• JWT:
  python import jwt token = jwt.encode({"sub": "user123"}, "secret", algorithm="HS256") print(token) # Use in Authorization: Bearer <token>
• mTLS:
  bash curl --cert client.crt --key client.key --cacert ca.crt https://api.customer.com

3. Test the Integration (Locally First)

• REST:
  bash http POST https://api.customer.com/data id=123 # httpie (better than curl for JSON)
• gRPC:
  bash grpcurl -plaintext -d '{"id": "123"}' api.customer.com:50051 package.Service/Method
• Webhooks:
  bash ngrok http 8080 # Expose local server for testing Then send a test payload: bash curl -X POST http://your-ngrok-url/webhook -H "Content-Type: application/json" -d '{"event": "test"}'

4. Deploy to Customer Environment

• Air-gapped:
• Build a Docker image with all dependencies:
  dockerfile
FROM python:3.9-slim
COPY . /app
RUN pip install -r requirements.txt --no-cache-dir
• Export and transfer:
  bash
docker save my-image > my-image.tar
scp my-image.tar user@customer-server:/tmp
ssh user@customer-server "docker load < /tmp/my-image.tar"
• On-prem with strict auth:
• Use an API gateway (e.g., Kong) to handle auth before forwarding to your service.
• Example Kong route:
  ```yaml
  routes:
  • name: my-route
    paths: ["/my-api"]
    plugins:
    • name: jwt
      config:
      key_claim_name: iss ```

5. Monitor and Debug

• Logs:
  bash kubectl logs -f pod/my-pod # If K8s journalctl -u my-service -f # If systemd
• Metrics:
• Use Prometheus + Grafana for latency, error rates, and throughput.
• Example Prometheus query:
  promql
rate(http_requests_total{status=~"5.."}[5m]) # 5xx errors
• Tracing:
• Jaeger or OpenTelemetry for distributed tracing. Critical for debugging "it’s slow" complaints.

6. Handle the Inevitable Fire Drill

• Scenario: Customer says, "The API is down!" but their monitoring shows it’s up.
• Action:
• Reproduce locally:
  bash
curl -v https://api.customer.com/endpoint -H "Authorization: Bearer $TOKEN"
• Check their firewall rules (ask for iptables -L or netstat -tulnp).
• If it’s auth-related, validate the token:
  bash
jwt decode $TOKEN # Using jwt-cli
• If it’s rate limiting, check headers:
  bash
curl -I https://api.customer.com/endpoint
# Look for X-RateLimit-Remaining

• Push a hotfix:

  
  
  • Add retry logic (exponential backoff):
    ```python
    from tenacity import retry, stop_after_attempt, wait_exponential

  @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
  def call_api():
  response = requests.get("https://api.customer.com")
  response.raise_for_status()
  return response.json()
  `` - Or add a circuit breaker (e.g.,pybreaker`).

Common Mistakes

FDE Interview / War Story Insights

Interview Questions

1. "The customer’s API returns 500 errors, but their logs show no issues. What do you do?"
2. Answer: Check network-level issues (firewall, DNS, TLS handshake), validate the payload format, and test with curl -v to see headers. Often, it’s a missing Content-Type or a malformed JWT.
3. "You’re integrating with a legacy SOAP API, but the customer insists it’s REST. How do you proceed?"
4. Answer: Don’t argue—build a thin adapter layer (e.g., Flask app) that translates REST calls to SOAP. Example:
   python
from zeep import Client
client = Client('https://api.customer.com/soap?wsdl')
result = client.service.SomeMethod(param1="value")
return {"data": result} # REST response
5. "The customer’s OAuth server is down, but they need the integration live today. What’s your workaround?"
6. Answer: Use a fallback auth method (e.g., API keys) or implement a local JWT issuer with short-lived tokens. Document the risk and escalate.

War Stories

• The "REST" API That Wasn’t:
• Scenario: A defense customer said their drone telemetry API was REST. It was actually gRPC over a custom protocol with mTLS.
• Fix: Used grpcurl to reverse-engineer the protobuf schema, then wrote a Python adapter to translate gRPC to REST for their dashboard.
• The OAuth Meltdown:
• Scenario: During a go-live, the customer’s OAuth server started returning invalid_grant errors. Their team was offline.
• Fix: Implemented a local JWT issuer with a 1-hour expiry and a hardcoded secret (temporary measure). Documented the risk and added monitoring for token failures.

Quick Check Questions

1. You’re deploying to an air-gapped network, and the customer’s API requires a JWT signed by their OAuth server. How do you get a token?
2. Answer: Ask for an offline token (e.g., a long-lived JWT or a service account key). If unavailable, set up a local JWT issuer with a shared secret (document the risk).

3. Why: Air-gapped networks can’t reach external auth servers.

4. The customer’s webhook payloads are malformed (missing fields, wrong types). How do you handle it?

5. Answer: Add schema validation (e.g., Pydantic, JSON Schema) and return 400 Bad Request with error details. Log all invalid payloads for debugging.

6. Why: Webhooks are often "fire and forget"—you must validate to avoid silent failures.

7. You’re integrating with a rate-limited API, and the customer’s traffic is causing 429 errors. What’s your first step?

8. Answer: Implement client-side rate limiting (e.g., token bucket algorithm) and add exponential backoff for retries.
9. Why: Server-side rate limits are often strict—you must throttle your own requests.

Last-Minute Cram Sheet

1. REST: GET /resources, POST /resources – use httpie for testing.
2. gRPC: grpcurl -plaintext -d '{}' host:port package.Service/Method.
3. OAuth 2.0: Always use client_credentials or authorization_code (never implicit).
4. JWT: Validate exp, iss, aud – use jwt.io to debug.
5. mTLS: curl --cert client.crt --key client.key --cacert ca.crt.
6. Webhooks: Always validate payloads – customers will send garbage.
7. Rate limiting: Log X-RateLimit-Remaining and implement retries with jitter.
8. Air-gapped: docker save > image.tar and scp to transfer.
9. ⚠️ Always test in the customer’s environment – what works in your lab will break behind their firewall.
10. Ports:
    • REST: 80 (HTTP), 443 (HTTPS)
    • gRPC: 50051 (default), 443 (TLS)
    • OAuth: 443 (HTTPS)