TECH **Amazon MSK & Kinesis (Data Streams, Firehose, Analytics) – Zero-Fluff Study Guide**

Amazon MSK & Kinesis (Data Streams, Firehose, Analytics) – Zero-Fluff Study Guide

For AWS Solutions Architects who need to build, debug, and optimize real-time data pipelines in production.

1. What This Is & Why It Matters

You’re a cloud engineer at a fintech startup. Your team just built a fraud-detection system that processes 10,000 transactions per second. The legacy batch pipeline (dumping CSVs into S3 and running nightly Spark jobs) is too slow—by the time fraud is detected, the money is already gone.

Enter real-time streaming:
- Amazon Kinesis (AWS’s managed streaming service) lets you ingest, process, and analyze data in milliseconds.
- Amazon MSK (Managed Streaming for Kafka) gives you Apache Kafka as a service, if you need open-source compatibility or advanced Kafka features.

Why this matters in production:
- If you ignore streaming, you’re stuck with batch processing (slow, high latency).
- If you misconfigure it, you’ll either lose data (no durability) or blow up costs (over-provisioned shards).
- If you don’t monitor it, you won’t know when your pipeline is silently failing (e.g., a producer stuck retrying).

Real-world scenario:
You inherit a Kinesis Data Stream that’s dropping records during peak traffic. Your boss asks: - "Why is our fraud detection missing transactions?" - "Can we scale this without downtime?" - "How do we debug this without losing data?"

This guide gives you the exact steps to answer those questions.

2. Core Concepts & Components

Amazon Kinesis Data Streams

• What it is: A scalable, durable real-time data ingestion service. Think of it like a conveyor belt where producers (apps, IoT devices) put data, and consumers (Lambda, EC2, ECS) process it.
• Production insight: ⚠️ Shards = throughput units. Each shard handles 1MB/s write, 2MB/s read. If you under-provision, records get throttled. If you over-provision, you waste money.
• Key terms:
• Record: A single data unit (e.g., a JSON payload from a mobile app).
• Partition Key: Determines which shard a record goes to (e.g., user_id). Bad keys cause hot shards (one shard overloaded, others idle).
• Sequence Number: Unique ID for each record in a shard.
• Retention Period: How long data stays in the stream (default 24h, max 365d). After this, data is deleted.

Amazon Kinesis Data Firehose

• What it is: A fully managed delivery service that loads streaming data into S3, Redshift, OpenSearch, or HTTP endpoints. Think of it like a pipeline with built-in transformations (e.g., convert JSON to Parquet).
• Production insight: ⚠️ No shards to manage (unlike Data Streams), but no replayability—once data is delivered, it’s gone. Use this for append-only use cases (e.g., logs, clickstreams).
• Key features:
• Buffering: Data is batched before delivery (configurable size/time).
• Transformations: Lambda functions can modify data before delivery.
• Compression: Supports GZIP, ZIP, SNAPPY, or Parquet.

Amazon Kinesis Data Analytics

• What it is: A SQL or Flink engine to process streaming data in real time. Think of it like running a SQL query on a firehose of data.
• Production insight: ⚠️ Flink apps cost more than SQL (you pay for the underlying EC2). Use SQL for simple aggregations, Flink for complex stateful processing.
• Key use cases:
• Anomaly detection (e.g., "Alert if a user makes 10 transactions in 5 seconds").
• Real-time dashboards (e.g., "Show live sales by region").
• Data enrichment (e.g., "Join streaming orders with a DynamoDB customer table").

Amazon MSK (Managed Streaming for Kafka)

• What it is: AWS’s fully managed Apache Kafka service. Use this if:
• You need Kafka’s ecosystem (e.g., Kafka Connect, Kafka Streams).
• You’re migrating an existing Kafka cluster to AWS.
• You need longer retention (Kinesis maxes at 365d; Kafka can go to years).
• Production insight: ⚠️ More complex than Kinesis (you manage brokers, topics, partitions). Not serverless—you pay for broker instances even when idle.
• Key terms:
• Broker: A Kafka server (MSK manages these for you).
• Topic: A category/feed name (e.g., transactions).
• Partition: A subset of a topic (like a Kinesis shard). More partitions = higher throughput.
• Producer/Consumer: Apps that write/read data.
• Consumer Group: A set of consumers that share the load of reading from a topic.

3. Step-by-Step Hands-On: Build a Fraud Detection Pipeline with Kinesis

Prerequisites

• AWS account with admin IAM permissions.
• AWS CLI installed (aws --version).
• Basic Python knowledge (we’ll use boto3).

Goal

Build a pipeline that: 1. Ingests fake transaction data into Kinesis Data Streams.
2. Processes it with Kinesis Data Analytics (SQL) to detect fraud.
3. Delivers results to S3 via Firehose.

Step 1: Create a Kinesis Data Stream

# Create a stream with 2 shards (handles ~2MB/s write, 4MB/s read)
aws kinesis create-stream --stream-name fraud-detection-stream --shard-count 2

# Verify it's active (takes ~1 min)
aws kinesis describe-stream --stream-name fraud-detection-stream

Expected output:

{
  "StreamDescription": {
    "StreamName": "fraud-detection-stream",
    "StreamARN": "arn:aws:kinesis:us-east-1:123456789012:stream/fraud-detection-stream",
    "StreamStatus": "ACTIVE",
    "Shards": [...]
  }
}

Production insight: If StreamStatus is CREATING, wait. If it’s DELETING, you have a problem.

Step 2: Simulate Transactions (Producer)

Create a Python script (producer.py) to send fake transactions:

import boto3
import json
import random
import time

kinesis = boto3.client('kinesis', region_name='us-east-1')

def generate_transaction():
    return {
        "transaction_id": str(random.randint(100000, 999999)),
        "user_id": f"user_{random.randint(1, 100)}",
        "amount": round(random.uniform(1.0, 1000.0), 2),
        "timestamp": int(time.time())
    }

while True:
    transaction = generate_transaction()
    kinesis.put_record(
        StreamName="fraud-detection-stream",
        Data=json.dumps(transaction),
        PartitionKey=transaction["user_id"]  # Distributes records by user
    )
    print(f"Sent: {transaction}")
    time.sleep(0.1)  # ~10 records/sec

Run it:

python3 producer.py

Production insight: If you see ProvisionedThroughputExceededException, you need more shards.

Step 3: Create a Kinesis Data Analytics App (SQL)

1. Go to AWS Console > Kinesis > Data Analytics > Create application.
2. Name: fraud-detection-analytics.
3. Source: Select fraud-detection-stream.
4. Schema discovery: Click "Discover schema" (AWS auto-detects JSON fields).
5. SQL query:
   ```sql
   -- Detect users making >3 transactions in 10 seconds
   CREATE OR REPLACE STREAM "FRAUD_ALERTS" (
   user_id VARCHAR(16),
   transaction_count INTEGER,
   window_start TIMESTAMP
   );

CREATE OR REPLACE PUMP "STREAM_PUMP" AS
INSERT INTO "FRAUD_ALERTS"
SELECT STREAM
user_id,
COUNT() AS transaction_count,
FLOOR("APPROXIMATE_ARRIVAL_TIME" TO SECOND) AS window_start
FROM "SOURCE_SQL_STREAM_001"
WINDOWED BY STAGGER (
PARTITION BY user_id RANGE INTERVAL '10' SECOND
)
GROUP BY user_id, FLOOR("APPROXIMATE_ARRIVAL_TIME" TO SECOND)
HAVING COUNT() > 3;
``` 6. Destination: Create a new Kinesis Firehose delivery stream (we’ll set this up next).
7. Start the app.

Production insight: If the app fails, check CloudWatch Logs for SQL errors.

Step 4: Create a Kinesis Firehose Delivery Stream

aws firehose create-delivery-stream \
  --delivery-stream-name fraud-alerts-firehose \
  --s3-destination-configuration '{
    "RoleARN": "arn:aws:iam::123456789012:role/FirehoseDeliveryRole",
    "BucketARN": "arn:aws:s3:::your-fraud-alerts-bucket",
    "Prefix": "fraud-alerts/",
    "BufferingHints": {
      "SizeInMBs": 5,
      "IntervalInSeconds": 300
    },
    "CompressionFormat": "GZIP"
  }'

Production insight:
- Buffering: Firehose waits until 5MB or 5 minutes before delivering to S3. Adjust based on your latency needs.
- IAM Role: You need a role with permissions for Firehose to write to S3. Create one if missing.

Step 5: Connect Data Analytics to Firehose

1. Go back to your Kinesis Data Analytics app.
2. Under Destinations, add the Firehose stream (fraud-alerts-firehose).
3. Save and restart the app.

Step 6: Verify the Pipeline

1. Check S3: After ~5 minutes, look in s3://your-fraud-alerts-bucket/fraud-alerts/. You should see .gz files with fraud alerts.
2. Check CloudWatch Metrics:
3. Kinesis Data Stream: IncomingBytes, IncomingRecords, IteratorAgeMilliseconds (if this spikes, consumers are falling behind).
4. Firehose: DeliveryToS3.Success, DeliveryToS3.Records.

Production insight: If IteratorAgeMilliseconds keeps rising, your consumers can’t keep up—scale your shards or optimize your app.

4. ? Production-Ready Best Practices

Security

• IAM Least Privilege:
• Producers: kinesis:PutRecord, kinesis:PutRecords.
• Consumers: kinesis:GetRecords, kinesis:GetShardIterator, kinesis:DescribeStream.
• Firehose: firehose:PutRecordBatch (if using it as a destination).
• Encryption:
• At rest: Enable KMS encryption for Kinesis streams.
• In transit: Use HTTPS/TLS (enabled by default).
• VPC Endpoints:
• If your producers/consumers are in a VPC, use VPC endpoints to avoid public internet.

Cost Optimization

• Kinesis Data Streams:
• Right-size shards: Use the Kinesis Shard Calculator.
• On-demand mode: If traffic is spiky, use on-demand (pay per GB, no shard management).
• Kinesis Firehose:
• Compress data (GZIP, Parquet) to reduce S3 costs.
• Buffering: Increase IntervalInSeconds to reduce S3 PUT requests.
• Kinesis Data Analytics:
• SQL apps are cheaper than Flink (Flink runs on EC2).
• Stop apps when not in use (you pay for runtime).

Reliability & Maintainability

• Naming conventions:
• Streams: app-name-environment-stream (e.g., fraud-detection-prod-stream).
• Firehose: app-name-environment-firehose (e.g., fraud-detection-prod-firehose).
• Tagging:
• Tag all resources with Environment, Owner, Project.
• Idempotency:
• Use sequence numbers to avoid duplicate processing.
• Rollback:
• For Kinesis Data Analytics, save SQL queries in version control.

Observability

• CloudWatch Metrics to Monitor:
  | Metric | Threshold | Action | |--------|-----------|--------| | IncomingBytes | >80% of shard limit | Scale shards | | IteratorAgeMilliseconds | >10,000 | Optimize consumers | | DeliveryToS3.Success | <100% | Check Firehose errors |
• CloudWatch Alarms:
• Alert on ProvisionedThroughputExceededException.
• Alert on DeliveryToS3.FailedRecords.
• Logging:
• Enable CloudWatch Logs for Kinesis Data Analytics.
• Log producer errors (e.g., throttling).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus (AWS SAA)

Typical Question Patterns

1. Kinesis vs. MSK:
2. "You need to migrate an existing Kafka cluster to AWS. Which service?" → MSK.
3. "You need a fully managed, serverless streaming service with no Kafka expertise." → Kinesis Data Streams.
4. Shard Scaling:
5. "Your Kinesis stream is throttling. What’s the cheapest fix?" → Increase shards (or switch to on-demand).
6. Firehose Use Cases:
7. "You need to deliver streaming data to S3 with minimal code. Which service?" → Kinesis Firehose.
8. Data Analytics:
9. "You need to detect anomalies in a stream with SQL. Which service?" → Kinesis Data Analytics (SQL).

Key ⚠️ Trap Distinctions

Common Scenario Question

"A gaming company wants to analyze player behavior in real time. They need to process 5,000 events/sec, store raw data in S3, and run SQL queries on the stream. Which services should they use?"

Answer:
1. Kinesis Data Streams (ingest 5,000 events/sec).
2. Kinesis Data Analytics (SQL) (run real-time queries).
3. Kinesis Firehose (deliver raw data to S3).

Why not MSK?
- Overkill if they don’t need Kafka’s ecosystem.
- More complex to manage.

7. ? Hands-On Challenge (with Solution)

Challenge:
You have a Kinesis Data Stream with 1 shard. Your producer is sending 1,500 records/sec, but CloudWatch shows IncomingRecords at 1,000 records/sec and ProvisionedThroughputExceededException. Fix the throttling without losing data.

Solution:

# Scale to 2 shards (each shard handles 1,000 records/sec)
aws kinesis update-shard-count \
  --stream-name your-stream \
  --target-shard-count 2 \
  --scaling-type UNIFORM_SCALING

Why it works:
- Each shard supports 1,000 records/sec write.
- Scaling to 2 shards gives 2,000 records/sec capacity.
- UNIFORM_SCALING splits existing shards evenly.

8. ? Rapid-Reference Crib Sheet

Kinesis Data Streams

• Shard limits: 1MB/s write, 2MB/s read.
• Retention: Default 24h, max 365d.
• Partition key: Determines shard. Use high-cardinality keys.
• CLI commands:
  bash aws kinesis create-stream --stream-name my-stream --shard-count 2 aws kinesis put-record --stream-name my-stream --data "data" --partition-key "key" aws kinesis get-shard-iterator --stream-name my-stream --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON
• ⚠️ On-demand mode is cheaper for spiky traffic.

Kinesis Firehose

• Buffering: Default 5MB or 300s.
• Transformations: Use Lambda for JSON → Parquet.
• CLI command:
  bash aws firehose create-delivery-stream --delivery-stream-name my-firehose --s3-destination-configuration '{"BucketARN": "arn:aws:s3:::my-bucket", "RoleARN": "arn:aws:iam::123456789012:role/FirehoseRole"}'
• ⚠️ No replayability—data is gone after delivery.

Kinesis Data Analytics

• SQL vs. Flink: SQL is cheaper, Flink is more powerful.
• Window types:
• Tumbling: Fixed-size, non-overlapping