Principles of Information Security: OSI Model and TCP/IP – Security Implications

OSI Model and TCP/IP – Security Implications

OSI Model & TCP/IP – Security Implications: Exam-Ready Study Guide

What This Is

The OSI (Open Systems Interconnection) model and TCP/IP (Transmission Control Protocol/Internet Protocol) are foundational networking frameworks that define how data moves across networks. Understanding their security implications is critical because attackers exploit weaknesses at different layers (e.g., ARP spoofing at Layer 2, SQL injection at Layer 7, or TCP SYN floods at Layer 4). A real-world example: The 2017 Equifax breach exploited an unpatched Apache Struts vulnerability (Layer 7 – Application), allowing attackers to execute remote code and steal 147 million records of PII (Personally Identifiable Information). Security professionals must secure each layer to prevent such attacks.

Key Terms & Concepts

• OSI Model (Open Systems Interconnection): A 7-layer conceptual framework (Physical → Application) that standardizes network communication. Used for troubleshooting, protocol design, and security analysis (e.g., firewalls operate at Layer 3/4, WAFs at Layer 7).

• Mnemonic: Please Do Not Throw Sausage Pizza Away (Physical → Data Link → Network → Transport → Session → Presentation → Application).

• TCP/IP Model: A 4-layer practical model (Network Interface → Internet → Transport → Application) used in real-world networking (e.g., the Internet). Simpler than OSI but maps roughly as:

• Network Interface (OSI Layers 1-2)
• Internet (OSI Layer 3)
• Transport (OSI Layer 4)

• Application (OSI Layers 5-7)

• Encapsulation/Decapsulation: Data is wrapped with headers/trailers as it moves down the OSI stack (encapsulation) and unwrapped as it moves up (decapsulation). Attackers exploit this (e.g., VLAN hopping by manipulating Layer 2 tags).

• PDU (Protocol Data Unit): The "name" for data at each OSI layer:

• Layer 1 (Physical): Bits
• Layer 2 (Data Link): Frames (e.g., Ethernet)
• Layer 3 (Network): Packets (e.g., IP)
• Layer 4 (Transport): Segments (TCP) / Datagrams (UDP)

• Layers 5-7: Data (e.g., HTTP requests)

• TCP (Transmission Control Protocol): A connection-oriented, reliable protocol (Layer 4) that uses 3-way handshake (SYN → SYN-ACK → ACK). Vulnerable to SYN floods (DoS) or session hijacking.

• Ports: Well-known (0-1023), registered (1024-49151), dynamic (49152-65535).

• UDP (User Datagram Protocol): A connectionless, fast protocol (Layer 4) used for DNS, VoIP, and streaming. No handshake = no built-in reliability or security (e.g., DNS amplification attacks).

• IP (Internet Protocol): Layer 3 protocol for addressing/routing. IPv4 (32-bit, e.g., 192.168.1.1) and IPv6 (128-bit, e.g., 2001:db8::1). Vulnerabilities:

• IP spoofing (forging source IP in attacks like Smurf attacks).

• Fragmentation attacks (e.g., Teardrop – overlapping fragments crash systems).

• ARP (Address Resolution Protocol): Layer 2 protocol that maps IP → MAC addresses. Vulnerable to ARP poisoning (MITRE T1557), where attackers redirect traffic to their machine (e.g., Man-in-the-Middle (MitM) attacks).

• Defense: Static ARP entries, ARP inspection (Cisco), port security.

• ICMP (Internet Control Message Protocol): Layer 3 protocol for error messages (e.g., ping, traceroute). Exploited in ICMP floods (DoS) or ping sweeps (reconnaissance).

• Defense: Rate limiting, blocking ICMP echo requests (but breaks ping).

• Ports & Services:

• Well-known ports (0-1023): 22 (SSH), 80 (HTTP), 443 (HTTPS), 53 (DNS).
• Registered ports (1024-49151): 3389 (RDP), 1433 (MSSQL).
• Attack example: Port scanning (Nmap) to find open services (MITRE T1046).

• Defense: Firewalls, port knocking, service hardening.

• Firewalls & Layered Security:

• Packet-filtering firewalls (Layer 3/4): Filter by IP/port (e.g., iptables, Windows Firewall).
• Stateful firewalls (Layer 4): Track connections (e.g., Cisco ASA).
• Next-gen firewalls (Layer 7): Deep packet inspection (e.g., Palo Alto, FortiGate).

• WAF (Web Application Firewall): Protects Layer 7 (e.g., ModSecurity, Cloudflare).

• TLS/SSL (Transport Layer Security / Secure Sockets Layer): Layer 6 (Presentation) encryption for secure communication (e.g., HTTPS). Vulnerabilities:

• Heartbleed (CVE-2014-0160): Memory leak in OpenSSL.
• POODLE (CVE-2014-3566): Downgrade attack forcing SSL 3.0.

• Defense: Disable weak ciphers, TLS 1.2+, HSTS (HTTP Strict Transport Security).

• DNS (Domain Name System): Layer 7 protocol that translates domains (e.g., google.com) to IPs. Attack vectors:

• DNS spoofing/cache poisoning (MITRE T1557.002).
• DNS tunneling (exfiltrating data via DNS queries).
• Defense: DNSSEC, Response Policy Zones (RPZ), logging/monitoring.

Step-by-Step: Applying OSI/TCP/IP Security

1. Map the Attack Surface (Layer-by-Layer)

• Layer 1 (Physical): Secure cabling, TEMPEST shielding (preventing electromagnetic eavesdropping), lock server rooms.
• Layer 2 (Data Link): Enable port security, DHCP snooping, ARP inspection, VLAN segmentation.
• Layer 3 (Network): Configure ACLs (Access Control Lists), IPsec VPNs, route filtering, anti-spoofing.
• Layer 4 (Transport): Use stateful firewalls, SYN cookies (to mitigate SYN floods), UDP rate limiting.
• Layer 5 (Session): Implement session timeouts, secure cookies, Kerberos (for authentication).
• Layer 6 (Presentation): Enforce TLS 1.2+, data encryption (AES-256), secure file formats (e.g., PDF/A).
• Layer 7 (Application): Deploy WAFs, input validation, OWASP Top 10 mitigations, API gateways.

2. Harden Protocols & Services

• Disable unnecessary services/ports (e.g., Telnet (23), FTP (21), RDP (3389) if unused).
• Patch known vulnerabilities (e.g., EternalBlue (CVE-2017-0144) for SMB, Log4j (CVE-2021-44228)).
• Use secure alternatives:
• SSH (22) instead of Telnet (23)
• SFTP/SCP instead of FTP
• HTTPS (443) instead of HTTP (80)

3. Monitor & Detect Anomalies

• SIEM (Security Information and Event Management): Correlate logs from firewalls (Layer 3/4), IDS/IPS (Layer 7), and endpoints.
• Tools: Splunk, IBM QRadar, Elastic SIEM.
• Network Traffic Analysis (NTA): Detect lateral movement, C2 (Command & Control) traffic, data exfiltration.
• Tools: Zeek (Bro), Darktrace, Cisco Stealthwatch.
• Baseline normal traffic and alert on deviations (e.g., unexpected ICMP traffic, DNS tunneling).

4. Respond to Layer-Specific Attacks

5. Validate with Penetration Testing

• Layer 2: Test for VLAN hopping, MAC flooding (e.g., Yersinia, Scapy).
• Layer 3/4: Test for IP spoofing, port scanning (e.g., Nmap, Masscan).
• Layer 7: Test for OWASP Top 10 (e.g., Burp Suite, OWASP ZAP).
• Report findings and remediate (e.g., disable unused ports, patch vulnerabilities).

Common Mistakes

Mistake 1: Confusing OSI and TCP/IP Layers

• Mistake: Thinking TCP/IP’s "Application" layer = OSI’s "Application" layer only.
• Correction: TCP/IP’s Application layer includes OSI Layers 5-7 (Session, Presentation, Application). Example:
• HTTP (Layer 7) + SSL/TLS (Layer 6) + Session cookies (Layer 5) all map to TCP/IP’s Application layer.

Mistake 2: Assuming All Firewalls Operate at the Same Layer

• Mistake: Believing all firewalls filter at Layer 3/4.
• Correction:
• Packet-filtering firewalls: Layer 3/4 (IP/port).
• Stateful firewalls: Layer 4 (track connections).
• Next-gen firewalls (NGFW): Layer 7 (deep packet inspection).
• WAFs: Only Layer 7 (HTTP/HTTPS).

Mistake 3: Ignoring Layer 2 Security

• Mistake: Focusing only on Layer 3+ (e.g., firewalls) and neglecting Layer 2 (e.g., ARP, VLANs).
• Correction: Layer 2 attacks (ARP poisoning, MAC flooding) can bypass higher-layer security. Mitigate with:
• Port security (limit MAC addresses per port).
• DHCP snooping (prevent rogue DHCP servers).
• VLAN segmentation (isolate traffic).

Mistake 4: Overlooking UDP Security

• Mistake: Assuming UDP is "secure" because it’s connectionless.
• Correction: UDP is stateless and unauthenticated, making it ideal for:
• DoS attacks (DNS amplification, NTP floods).
• Data exfiltration (DNS tunneling).
• Defense: Rate limiting, UDP flood protection, monitoring.

Mistake 5: Misconfiguring TLS/SSL

• Mistake: Enabling weak ciphers (e.g., SSLv3, RC4) or self-signed certificates.
• Correction:
• Disable SSLv3, TLS 1.0/1.1 (use TLS 1.2+).
• Use strong ciphers (AES-256-GCM, ChaCha20).
• Enforce HSTS (prevents downgrade attacks).
• Use certificates from trusted CAs (e.g., Let’s Encrypt, DigiCert).

Certification Exam Tips

1. CISSP: Management vs. Technical Depth

• Expect questions on:
• Risk management at each layer (e.g., "Which layer is most vulnerable to MitM attacks?" → Layer 2 (ARP poisoning)).
• Compliance implications (e.g., "Which layer does PCI DSS require encryption for?" → Layer 6 (TLS for cardholder data)).
• Security architecture (e.g., "Where should a WAF be placed?" → Layer 7, in front of web apps).
• ⚠️ Trap: CISSP may ask about OSI vs. TCP/IP in a governance context (e.g., "Which model is used for regulatory compliance?" → OSI is conceptual, TCP/IP is practical).

2. Security+: Memorize Ports & Layer Functions

• Ports to know:
• 20/21 (FTP), 22 (SSH), 23 (Telnet), 25 (SMTP), 53 (DNS), 80 (HTTP), 110 (POP3), 143 (IMAP), 443 (HTTPS), 3389 (RDP).
• Layer functions:
• Layer 2: MAC addresses, switches, VLANs.
• Layer 3: IP addresses, routers, ACLs.
• Layer 4: TCP/UDP, ports, firewalls.
• Layer 7: HTTP, DNS, APIs, WAFs.
• ⚠️ Trap: Questions may ask, "Which layer does a stateful firewall operate at?" → Layer 4 (Transport).

3. CEH: Attack Vectors & Tools

• Know the tools for each layer:
• Layer 2: Yersinia (VLAN hopping), Ettercap (ARP poisoning).
• Layer 3/4: Nmap (port scanning), Hping3 (SYN floods).
• Layer 7: Burp Suite (SQLi, XSS), SQLmap (automated SQLi).
• Attack examples:
• ARP poisoning (Layer 2) → MitM (e.g., stealing credentials).
• SYN flood (Layer 4) → DoS (crash a server).
• SQL injection (Layer 7) → Data breach (e.g., Equifax).
• ⚠️ Trap: CEH may ask, "Which tool is used for DNS spoofing?" → Dnsspoof (part of dsniff).

4. All Certs: OSI vs. TCP/IP Mapping

Quick Check Questions

1. A security analyst detects unusual ARP traffic on the network. Which OSI layer is most likely being targeted, and what is the primary risk?

• A) Layer 3 – IP spoofing leading to DoS
• B) Layer 2 – ARP poisoning leading to MitM attacks
• C) Layer 4 – TCP session hijacking
• D) Layer 7 – SQL injection

✅ Correct Answer: B
Explanation: ARP operates at Layer 2 (Data Link) and is vulnerable to ARP poisoning, which enables Man-in-the-Middle (MitM) attacks (e.g., intercepting traffic).

2. An organization wants to prevent DNS-based data exfiltration. Which of the following controls is MOST effective?

• A) Deploying a stateful firewall to block UDP port 53
• B) Implementing DNSSEC to validate responses
• C) Using a WAF to inspect HTTP traffic
• D) Enabling Response Policy Zones (RPZ) and monitoring DNS queries

✅ Correct Answer: D
Explanation: DNS tunneling (data exfiltration via DNS queries) is best mitigated by RPZ (blocking malicious domains) and monitoring/logging DNS traffic. DNSSEC (B) only validates responses, not content.

3. During a penetration test, an attacker sends malformed TCP packets to crash a server. Which OSI layer is being exploited, and what is the BEST defense?

• A) Layer 3 – IP fragmentation attack; use a packet-filtering firewall
• B) Layer 4 – SYN flood; enable SYN cookies
• C) Layer 7 – HTTP request smuggling; deploy a WAF
• D) Layer 2 – MAC flooding; enable port security

✅ Correct Answer: B
Explanation: SYN floods exploit Layer 4 (Transport) by overwhelming a server with half-open TCP connections. SYN cookies (a TCP stack feature) mitigate this by not allocating resources until the ACK is received.

Last-Minute Cram Sheet

1. OSI Layers (7 → 1): Application, Presentation, Session, Transport, Network, Data Link, Physical.
2. TCP/IP Layers (4 → 1): Application, Transport, Internet, Network Interface.
3. Layer 2 Attacks: ARP poisoning, VLAN hopping, MAC flooding. Defense: Port security, ARP inspection.
4. Layer 3 Attacks: IP spoofing, ICMP floods. Defense: ACLs, uRPF, rate limiting.
5. Layer 4 Attacks: SYN floods, UDP floods. Defense: SYN cookies, stateful firewalls.
6. Layer 7 Attacks: SQLi, XSS, CSRF. Defense: WAF, input validation, OWASP Top 10.
7. TCP 3-Way Handshake: SYN → SYN-ACK → ACK. Vulnerability: SYN floods.
8. UDP is stateless → No handshake = faster but less secure (used for DNS, VoIP).
9. TLS/SSL (Layer 6): Encrypts data. Weaknesses: Heartbleed, POODLE. Fix: TLS 1.2+, HSTS.
10. ⚠️ Exam Trap: "Which layer does a WAF operate at?" → Layer 7 (Application). "Which layer does a stateful firewall operate at?" → Layer 4 (Transport).