**Monitoring: A Practical Guide**

Monitoring: A Practical Guide

What Is This?

Monitoring is the continuous observation and measurement of systems, applications, or infrastructure to detect issues, track performance, and ensure reliability. You use it to catch failures before they impact users, optimize resource usage, and maintain uptime in production environments.

Why It Matters

Without monitoring, systems fail silently. Downtime costs businesses thousands per minute (e.g., AWS outages cost ~$100K/hour). Monitoring: - Prevents outages by alerting teams before small issues become critical.
- Improves performance by identifying bottlenecks (e.g., slow database queries).
- Reduces debugging time by providing logs, metrics, and traces.
- Ensures compliance (e.g., GDPR, HIPAA) by tracking access and data flows.

Core Concepts

1. Metrics

Numerical data points collected over time (e.g., CPU usage, request latency, error rates). Metrics answer "What’s happening?" - Types: - Counters: Cumulative values (e.g., total HTTP requests).
- Gauges: Current values (e.g., memory usage).
- Histograms: Distribution of values (e.g., request duration percentiles).

2. Logs

Structured or unstructured text records of events (e.g., "User X logged in at 10:00 AM"). Logs answer "Why did this happen?" - Key attributes: Timestamp, severity (INFO/WARN/ERROR), source, message.

3. Traces

End-to-end records of a single request’s journey through a distributed system (e.g., a user’s API call hitting 5 microservices). Traces answer "Where is the bottleneck?" - Components: - Span: A single operation (e.g., a database query).
- Trace ID: Links all spans for one request.

4. Alerts

Automated notifications triggered when metrics/logs meet predefined conditions (e.g., "CPU > 90% for 5 minutes"). Alerts answer "When should I act?" - Best practices: - Avoid "alert fatigue" (too many false positives).
- Use severity levels (P0/P1/P2).

5. Observability

The ability to infer a system’s internal state from external outputs (metrics, logs, traces). Observability is the goal; monitoring is the tool.

How It Works (Architecture)

A typical monitoring pipeline has 4 stages:

1. Instrumentation
2. Add code to emit metrics/logs/traces (e.g., Prometheus client libraries, OpenTelemetry SDKs).

3. Example: Wrap a function to log its execution time.

4. Collection

5. Agents (e.g., Prometheus Node Exporter, Fluentd) scrape/pull data from sources.

6. Push vs. pull:

   
   
   • Push: Application sends data to a collector (e.g., StatsD).
   • Pull: Collector fetches data (e.g., Prometheus scraping /metrics endpoints).

7. Storage

8. Time-series databases (TSDBs) for metrics (e.g., Prometheus, InfluxDB).
9. Log storage (e.g., Elasticsearch, Loki).

10. Trace storage (e.g., Jaeger, Zipkin).

11. Visualization & Alerting

12. Dashboards (e.g., Grafana) display metrics.
13. Alert managers (e.g., Alertmanager, PagerDuty) notify teams.

Simple Diagram:

[App] → (Metrics/Logs/Traces) → [Collector] → [Storage] → [Dashboard/Alerts]

Hands-On / Getting Started

Prerequisites

• Basic Linux command line.
• A system to monitor (e.g., a local machine, Docker container, or cloud VM).
• Docker (for running tools like Prometheus/Grafana).

Minimal Example: Monitor CPU Usage with Prometheus + Grafana

1. Run Prometheus (collects metrics):
   bash
docker run -d --name=prometheus -p 9090:9090 prom/prometheus

2. Prometheus scrapes its own metrics by default (visit http://localhost:9090).

3. Add Node Exporter (collects system metrics):
   bash
docker run -d --name=node-exporter -p 9100:9100 prom/node-exporter

4. Configure Prometheus to scrape it by editing prometheus.yml:
   ```yaml
   scrape_configs:
   • job_name: 'node'
     static_configs:
     • targets: ['node-exporter:9100'] ```

5. Restart Prometheus:
   bash
docker restart prometheus

6. Run Grafana (visualize metrics):
   bash
docker run -d --name=grafana -p 3000:3000 grafana/grafana

7. Access Grafana at http://localhost:3000 (default creds: admin/admin).
8. Add Prometheus as a data source (http://prometheus:9090).
9. Import the "Node Exporter Full" dashboard (ID: 1860).

Expected Outcome: - A Grafana dashboard showing CPU, memory, disk, and network metrics.
- Alerts can be added later (e.g., "CPU > 80% for 5m").

Common Pitfalls & Mistakes

1. Monitoring Too Much (or Too Little)
2. Problem: Collecting every metric drowns you in noise; missing key metrics hides problems.

3. Fix: Focus on the "Four Golden Signals":

   
   
   • Latency (time to serve a request).
   • Traffic (requests per second).
   • Errors (failed requests).
   • Saturation (resource usage, e.g., CPU).

4. Alert Fatigue

5. Problem: Too many alerts → teams ignore them.

6. Fix:

   
   
   • Use severity levels (e.g., P0 = page someone, P2 = email).
   • Set thresholds carefully (e.g., "CPU > 95% for 10m" instead of "CPU > 80%").

7. Ignoring Logs and Traces

8. Problem: Metrics alone don’t explain why something failed.

9. Fix: Correlate metrics with logs/traces (e.g., "High latency? Check the trace for slow DB queries").

10. Not Testing Alerts

11. Problem: Alerts fail silently (e.g., email misconfigured).

12. Fix: Trigger test alerts (e.g., curl -X POST http://alertmanager/api/v1/alerts).

13. Storing Data Forever

14. Problem: Logs/metrics eat up storage and slow queries.
15. Fix: Set retention policies (e.g., "Keep metrics for 30 days, logs for 7 days").

Best Practices

For Metrics

• Use labels wisely: Too many labels (e.g., user_id) explode cardinality and slow queries.
• Prefer counters over gauges: Counters (e.g., http_requests_total) are easier to aggregate.
• Sample at consistent intervals: Irregular sampling (e.g., 1s then 10s) skews graphs.

For Logs

• Structure logs (e.g., JSON) for easier querying: json {"level": "ERROR", "service": "auth", "message": "Login failed", "user_id": 123}
• Avoid logging sensitive data (e.g., passwords, PII).
• Use log levels appropriately:
• DEBUG: Verbose, for development.
• INFO: Normal operation.
• WARN: Recoverable issues.
• ERROR: Failures needing action.

For Traces

• Instrument key paths: Focus on critical user journeys (e.g., checkout flow).
• Use context propagation: Ensure trace IDs carry across service boundaries.
• Sample traces: Don’t trace every request (e.g., sample 1% of low-priority traffic).

For Alerts

• Alert on symptoms, not causes: "Error rate > 1%" is better than "Database is down" (the latter is a cause).
• Use multi-window alerts: "CPU > 90% for 5m" reduces flapping.
• Escalate alerts: If no one acknowledges in 10m, page the next person.

For Dashboards

• Keep it simple: One dashboard per team/service.
• Use templates: Grafana’s $__interval or $__rate_interval auto-adjusts time ranges.
• Avoid "wall of graphs": Highlight key metrics (e.g., SLOs) at the top.

Tools & Frameworks

Real-World Use Cases

1. E-Commerce Site (Black Friday)

• Problem: During Black Friday, traffic spikes cause slow checkout flows.
• Solution:
• Monitor latency (P99 checkout time) and error rates (failed payments).
• Set alerts for "Checkout latency > 2s for 5m" or "Payment errors > 1%".
• Use traces to identify slow database queries in the payment service.

2. Kubernetes Cluster (Auto-Scaling)

• Problem: Pods crash or run out of memory unexpectedly.
• Solution:
• Monitor pod restarts, memory usage, and CPU throttling.
• Set alerts for "Pod restart rate > 3 in 5m" or "Memory usage > 90%".
• Use Prometheus + Grafana to visualize cluster health.

3. SaaS Application (SLO Compliance)

• Problem: Users complain about slow API responses, but the team doesn’t know how often it happens.
• Solution:
• Define an SLO (e.g., "99% of requests < 500ms over 30d").
• Monitor request latency and error rates.
• Use Grafana to track SLO burn rate (e.g., "10% of error budget used in 1h").

Check Your Understanding (MCQs)

Question 1

You’re monitoring a web app and notice a sudden spike in 5xx errors. Which tool would best help you identify the root cause? - A: Prometheus (metrics) - B: Grafana (dashboards) - C: Jaeger (traces) - D: Loki (logs)

Correct Answer: D (Loki) Explanation: While Prometheus (A) and Grafana (B) can show that errors are happening, Loki lets you query logs to see why (e.g., "NullPointerException in line 42"). Jaeger (C) is for tracing, which is overkill for this scenario.
Why the Distractors Are Tempting: - A: Prometheus shows error rates, but not the cause.
- B: Grafana visualizes metrics, but doesn’t explain errors.
- C: Traces are useful for latency, not error debugging.

Question 2

You’re setting up alerts for a database. Which threshold is least likely to cause alert fatigue? - A: "CPU > 80% for 1m" - B: "CPU > 95% for 5m" - C: "Disk usage > 90%" - D: "Memory usage > 85% for 10m"

Correct Answer: B ("CPU > 95% for 5m") Explanation: A high threshold (95%) with a long duration (5m) reduces false positives. Short windows (A) or static thresholds (C) trigger too often.
Why the Distractors Are Tempting: - A: Short windows catch spikes but cause flapping.
- C: Disk usage rarely drops, so this alert will fire constantly.
- D: Memory usage can fluctuate, leading to false alarms.

Question 3

You’re instrumenting a microservice with OpenTelemetry. What’s the minimum you need to add to a request to enable tracing? - A: A unique trace ID - B: A trace ID and a span ID - C: A trace ID, span ID, and parent span ID - D: A trace ID, span ID, and baggage

Correct Answer: B (A trace ID and a span ID) Explanation: A trace ID links spans across services, and a span ID identifies the current operation. Parent span ID (C) is optional (for nested spans). Baggage (D) is for propagating context (e.g., user ID), not tracing.
Why the Distractors Are Tempting: - A: Without a span ID, you can’t distinguish operations in the same trace.
- C: Parent span ID is only needed for child spans.
- D: Baggage is useful but not required for basic tracing.

Learning Path

1. Beginner
2. Learn the "Four Golden Signals" (latency, traffic, errors, saturation).
3. Set up Prometheus + Grafana to monitor a local app.

4. Write a simple alert rule (e.g., "CPU > 80%").

5. Intermediate

6. Instrument an app with OpenTelemetry (metrics + traces).
7. Set up Loki for log aggregation.

8. Define SLOs and error budgets for a service.

9. Advanced

10. Build a distributed tracing pipeline (Jaeger + OpenTelemetry).
11. Implement anomaly detection (e.g., Prometheus + ML).
12. Optimize query performance (e.g., PromQL, Elasticsearch indexing).

Further Resources

Books

• Site Reliability Engineering (Google) – Chapters on monitoring and alerting.
• Observability Engineering (Charity Majors) – Practical guide to metrics/logs/traces.

Courses

• Prometheus Official Docs – Hands-on tutorials.
• OpenTelemetry Bootcamp – Free course on instrumentation.

Tools

• Grafana Play – Interactive dashboards.
• Jaeger HotROD Demo – Trace a sample app.

Communities

• r/Observability – Reddit community.
• CNCF Slack – #prometheus, #opentelemetry channels.

30-Second Cheat Sheet

1. Monitor the "Four Golden Signals": Latency, traffic, errors, saturation.
2. Alert on symptoms, not causes: "Error rate > 1%" vs. "Database is down."
3. Correlate metrics + logs + traces: Metrics show what, logs show why, traces show where.
4. Avoid alert fatigue: Use severity levels and multi-window thresholds.
5. Start small: Instrument one service, then expand.

Related Topics

1. Incident Response – How to act on monitoring alerts.
2. SRE (Site Reliability Engineering) – Building reliable systems with monitoring.
3. Distributed Tracing – Deep dive into tracing microservices.