TECH **AWS CloudWatch Metrics, Logs, and Alarms: Zero-Fluff Study Guide**

AWS CloudWatch Metrics, Logs, and Alarms: Zero-Fluff Study Guide

(For AWS Solutions Architect – Associate & Real-World Production)

1. What This Is & Why It Matters

You’re a cloud engineer at a fintech startup. Your team deploys a new payment processing microservice to ECS Fargate. Two days later, the service starts throttling under load, but no one notices—until customers complain. Your CTO asks: "Why didn’t we get an alert?"

CloudWatch is your eyes and ears in AWS. It collects metrics (CPU, latency, errors), stores logs (application, API, system), and triggers alarms (e.g., "CPU > 80% for 5 minutes → send SNS alert"). Without it, you’re flying blind. With it, you can: - Detect issues before users do (e.g., high latency, failed Lambda invocations).
- Debug production problems (e.g., "Why did this API call fail at 2:17 AM?").
- Optimize costs (e.g., "This EC2 instance is 90% idle—downsize it").
- Automate responses (e.g., "If DynamoDB throttles, trigger a Lambda to scale up").

Ignoring CloudWatch is like driving a car without a dashboard. You won’t know you’re running out of gas until the engine sputters.

2. Core Concepts & Components

? CloudWatch Metrics

• Definition: Time-ordered data points (e.g., CPU utilization, request count) for AWS resources or custom applications.
• Production insight: Metrics are retention-limited (15 months for most, 3 hours for high-resolution). If you need long-term storage, export to S3 or a third-party tool.
• Key detail: Metrics are region-specific—a metric in us-east-1 won’t appear in eu-west-1.

? CloudWatch Logs

• Definition: Centralized storage for logs from AWS services (e.g., Lambda, EC2, API Gateway) or your applications (via the CloudWatch Logs Agent).
• Production insight: Logs are not free—pricing is based on ingestion, storage, and retrieval. Use log filters to avoid storing unnecessary data.
• Key detail: Logs are organized into log groups (e.g., /aws/lambda/my-function) and log streams (individual instances of a log source).

⏰ CloudWatch Alarms

• Definition: Rules that trigger actions (e.g., SNS notifications, Auto Scaling, Lambda) when a metric crosses a threshold.
• Production insight: Alarms have three states: OK, ALARM, and INSUFFICIENT_DATA. INSUFFICIENT_DATA often means the metric isn’t being published (e.g., an EC2 instance is stopped).
• Key detail: Alarms can only monitor one metric at a time (e.g., "CPU > 80%"). For multi-metric logic (e.g., "CPU > 80% AND memory > 90%"), use CloudWatch Composite Alarms or EventBridge.

? CloudWatch Logs Insights

• Definition: A query language to search and analyze logs in real time (like SQL for logs).
• Production insight: Logs Insights is not enabled by default—you must explicitly query logs. Use it to debug issues (e.g., "Show me all 5XX errors from my API Gateway in the last hour").
• Key detail: Queries are charged per GB scanned, so filter early (e.g., filter @message like /ERROR/).

? CloudWatch Agent

• Definition: A lightweight daemon to collect custom metrics (e.g., memory usage, disk I/O) and logs from EC2 instances, on-prem servers, or containers.
• Production insight: The agent is required for non-AWS metrics (e.g., "How much RAM is my EC2 instance using?"). AWS services (e.g., Lambda, RDS) publish metrics automatically.
• Key detail: The agent uses an IAM role (for EC2) or credentials file (for on-prem). Misconfigured permissions = no data.

? CloudWatch Events / EventBridge

• Definition: A near-real-time event bus that triggers actions (e.g., Lambda, SNS) based on AWS events (e.g., "EC2 instance state changed to stopped").
• Production insight: EventBridge is the evolved version of CloudWatch Events (same API, more features). Use it for automated remediation (e.g., "If an EC2 instance fails, restart it").
• Key detail: Events are JSON payloads—you can parse them in Lambda to take custom actions.

? CloudWatch Dashboards

• Definition: Customizable visualizations of metrics and logs (e.g., "Show me CPU, memory, and error rates for my microservices").
• Production insight: Dashboards are region-specific—if you have resources in multiple regions, you’ll need separate dashboards.
• Key detail: Dashboards can embed Logs Insights queries for real-time debugging.

? CloudWatch Pricing

• Definition: You pay for:
• Metrics (custom metrics cost extra).
• Logs (ingestion, storage, retrieval).
• Alarms (per alarm).
• Logs Insights (per GB scanned).
• Production insight: Costs add up fast—monitor your CloudWatch usage in the Billing Dashboard. Use log retention policies to delete old logs.
• Key detail: Free tier includes:
• 10 custom metrics.
• 5 GB of log ingestion.
• 3 dashboards (up to 50 metrics each).

3. Step-by-Step Hands-On: Monitor an EC2 Instance with CloudWatch

Prerequisites

• AWS account with admin IAM permissions (or at least CloudWatchFullAccess and EC2FullAccess).
• An EC2 instance (Amazon Linux 2 or Ubuntu) running in us-east-1.
• AWS CLI installed and configured (aws configure).

Step 1: Install the CloudWatch Agent on EC2

The agent collects custom metrics (e.g., memory, disk) and logs from your instance.

1. SSH into your EC2 instance:
   bash
ssh -i "your-key.pem" ec2-user@<EC2_PUBLIC_IP>
   (Replace ec2-user with ubuntu for Ubuntu instances.)

2. Download and install the agent:
   bash
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
sudo rpm -U ./amazon-cloudwatch-agent.rpm

3. Create a config file for the agent:
   bash
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
   (This downloads a default config. We’ll customize it next.)

4. Edit the config file (/opt/aws/amazon-cloudwatch-agent/bin/config.json):
   json
{
"metrics": {
"metrics_collected": {
"mem": {
"measurement": ["mem_used_percent"],
"metrics_collection_interval": 60
},
"disk": {
"measurement": ["used_percent"],
"metrics_collection_interval": 60,
"resources": ["/"]
}
}
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/messages",
"log_group_name": "ec2-instance-logs",
"log_stream_name": "{instance_id}-messages"
}
]
}
}
}
}
   (This collects memory/disk metrics and /var/log/messages logs.)

5. Start the agent:
   bash
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start

6. Verify the agent is running:
   bash
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
   (Output should show status: running.)

Step 2: View Metrics in CloudWatch

1. Go to the CloudWatch console → Metrics → CWAgent.
2. You should see:
3. mem_used_percent (memory usage).
4. disk_used_percent (disk usage).
5. Create a graph:
6. Select mem_used_percent → Graphed metrics → Add to dashboard.
7. Name the dashboard EC2-Monitoring.

Step 3: Create a CloudWatch Alarm for High CPU

1. Go to CloudWatch → Alarms → Create alarm.
2. Select metric → EC2 → Per-Instance Metrics → CPUUtilization.
3. Configure the alarm:
4. Threshold: Static, Greater than 80.
5. Period: 5 minutes.
6. Evaluation periods: 1 (trigger after 1 period).
7. Actions: Create new topic → SNS topic name: HighCPUAlert → Email: [email protected].
8. Name the alarm: EC2-High-CPU.

9. Create the alarm.

10. Test the alarm:

11. SSH into your EC2 instance.
12. Run a CPU stress test:
    bash
sudo yum install -y stress
stress --cpu 2 --timeout 300
13. Wait 5 minutes. You should receive an SNS email alert.

Step 4: Query Logs with Logs Insights

1. Go to CloudWatch → Logs → Log groups → Select ec2-instance-logs.

2. Click "Logs Insights" → Run query:
   sql
fields @timestamp, @message
| filter @message like /error|fail|warn/i
| sort @timestamp desc
| limit 20
   (This shows the last 20 error/warning messages.)

3. Save the query for future debugging.

Step 5: Automate Remediation with EventBridge

1. Go to EventBridge → Rules → Create rule.
2. Name: EC2-High-CPU-Remediation.
3. Rule type: Rule with an event pattern.
4. Event source: AWS events or EventBridge partner events.

5. Event pattern:
   json
{
"source": ["aws.cloudwatch"],
"detail-type": ["CloudWatch Alarm State Change"],
"resources": ["arn:aws:cloudwatch:us-east-1:123456789012:alarm:EC2-High-CPU"],
"detail": {
"state": {
"value": ["ALARM"]
}
}
}
   (Replace 123456789012 with your AWS account ID.)

6. Select target: Lambda function → Create new function:
   ```python
   import boto3

def lambda_handler(event, context):
ec2 = boto3.client('ec2')
instance_id = event['detail']['trigger']['Dimensions'][0]['value']

   # Reboot the instance (or take other actions)
   ec2.reboot_instances(InstanceIds=[instance_id])
   return {
       'statusCode': 200,
       'body': f'Rebooted instance {instance_id}'
   }

`` 7. Deploy the Lambda function and attach an IAM role withec2:RebootInstances` permissions.
8. Create the rule.

Now, when CPU > 80%, the alarm triggers → EventBridge invokes Lambda → Lambda reboots the instance.

4. ? Production-Ready Best Practices

Security

• Least privilege IAM roles: The CloudWatch agent needs cloudwatch:PutMetricData and logs:PutLogEvents. Don’t use admin roles.
• Encrypt logs: Enable KMS encryption for log groups (especially for compliance-sensitive logs).
• VPC endpoints: If your EC2 instances are in a private subnet, use a CloudWatch VPC endpoint to avoid sending logs over the public internet.

Cost Optimization

• Log retention policies: Set log groups to expire after 30 days (or less) unless required for compliance.
  bash aws logs put-retention-policy --log-group-name "ec2-instance-logs" --retention-in-days 30
• Filter logs before ingestion: Use the CloudWatch agent’s filters to exclude noisy logs (e.g., debug logs).
• Use metric math: Instead of creating multiple alarms, use metric math (e.g., CPUUtilization + MemoryUtilization) to reduce costs.

Reliability & Maintainability

• Tag everything: Tag alarms, log groups, and dashboards with Environment=prod, Team=backend, etc.
• Standardize log formats: Use JSON logs for easier parsing in Logs Insights.
• Test alarms: Manually trigger alarms (e.g., aws cloudwatch set-alarm-state --alarm-name "EC2-High-CPU" --state-value ALARM --state-reason "Testing") to ensure they work.

Observability

• Key metrics to monitor:
• EC2: CPU, memory, disk, network.
• Lambda: Invocations, errors, duration, throttles.
• RDS: CPU, connections, read/write latency.
• API Gateway: 4XX/5XX errors, latency.
• Log patterns to watch:
• ERROR, FAIL, timeout, throttle.
• Unusual spikes in log volume (could indicate an attack or misconfiguration).
• Alert thresholds:
• CPU > 80% for 5 minutes (not 1 minute—avoid flapping).
• 5XX errors > 1% of requests (not 0—some errors are normal).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus (AWS SAA)

Typical Question Patterns

1. "Which service collects custom metrics from EC2?"
2. ✅ CloudWatch Agent (not the default CloudWatch metrics).

3. ❌ CloudTrail (audit logs), X-Ray (distributed tracing).

4. "How do you monitor memory usage on an EC2 instance?"

5. ✅ Install the CloudWatch Agent and configure it to collect mem_used_percent.

6. ❌ AWS doesn’t collect memory metrics by default.

7. "You need to trigger a Lambda when an EC2 instance’s CPU > 90%. What’s the most cost-effective way?"

8. ✅ CloudWatch Alarm → SNS → Lambda (or EventBridge for more complex logic).

9. ❌ Polling the metric with a Lambda (inefficient).

10. "How do you retain CloudWatch logs for 1 year for compliance?"

11. ✅ Set a retention policy on the log group (aws logs put-retention-policy).
12. ❌ Export to S3 manually (not automated).

Key ⚠️ Trap Distinctions

• CloudWatch vs. CloudTrail:
• CloudWatch: Metrics, logs, alarms (operational monitoring).
• CloudTrail: API calls (audit logs).
• Logs Insights vs. Log Groups:
• Log Groups: Storage for logs.
• Logs Insights: Query tool for logs (charged per GB scanned).
• Alarms vs. Events:
• Alarms: Monitor metrics (e.g., "CPU > 80%").
• Events/EventBridge: Monitor AWS events (e.g., "EC2 instance stopped").

Scenario-Based Question

"You need to monitor a Lambda function’s errors and latency. What’s the most cost-effective solution?"
- ✅ Enable CloudWatch Lambda Insights (pre-built dashboards for Lambda).
- ✅ Create a CloudWatch Alarm for Errors > 0 and Duration > 1s.
- ❌ X-Ray (more expensive, better for distributed tracing).

7. ? Hands-On Challenge (with Solution)

Challenge

You have a Lambda function that processes orders. You want to: 1. Count how many times it fails (and alert if > 5 failures in 5 minutes).
2. Log the error details for debugging.

Write the CloudWatch Alarm and Logs Insights query to achieve this.

Solution

1. Create a CloudWatch Alarm:
   bash
aws cloudwatch put-metric-alarm \
--alarm-name "Lambda-OrderProcessor-Errors" \
--metric-name "Errors" \
--namespace "AWS/Lambda" \
--statistic "Sum" \
--period 300 \
--threshold 5 \
--comparison-operator "GreaterThanThreshold" \
--evaluation-periods 1 \
--alarm-actions "arn:aws:sns:us-east-1:123456789012:LambdaErrorAlerts" \
--dimensions "Name=FunctionName,Value=OrderProcessor"
   (Replace 123456789012 with your account ID.)

2. Logs Insights Query (to debug errors):
   sql
fields @timestamp, @message, @logStream
| filter @message like /ERROR|Task timed out|Unhandled exception/i
| sort @timestamp desc
| limit 50

Why it works:
- The alarm triggers if the Lambda’s Errors metric exceeds 5 in 5 minutes.
- The Logs Insights query filters for error messages in the Lambda logs.

8. ? Rapid-Reference Crib Sheet

CloudWatch Metrics

• Default metrics retention: 15 months (high-resolution: 3 hours).
• Custom metrics cost: $0.30 per metric/month (first 10 free).
• Publish a custom metric:
  ```bash aws cloudwatch put-metric-data --namespace "MyApp" --metric-name "OrdersProcessed" --value