Skip to content

Chapter 10: Network Protocol Security

This chapter examines security mechanisms built into foundational Internet protocols. We analyze where the original protocol designers made security trade-offs, how attackers exploit those gaps, and how modern extensions restore the integrity and confidentiality properties that were missing from early designs.

The pattern repeats across protocols: DNS, BGP, ARP, and NDP were all designed before adversarial internet-scale deployment was a realistic concern. Each was later retrofitted with authentication extensions (DNSSEC, RPKI, DAI, SEND) that preserve backward compatibility at the cost of complexity. Understanding the original vulnerability and the retrofit’s limitations prepares practitioners to reason about which deployments actually close the attack surface versus which create a false sense of security.

10.1 DNSSEC: Authenticating the Domain Name System

The Domain Name System was designed in the early 1980s with no authentication whatsoever. A resolver that receives a DNS response cannot distinguish a legitimate answer from a forged one injected by an on-path attacker—a vulnerability exploited in the Kaminsky attack of 2008, which could poison a caching resolver in under a minute.

The Kaminsky attack works because DNS uses 16-bit transaction IDs (TXID) and UDP source port randomization as the only anti-spoofing mechanism. An attacker who can send forged responses faster than the legitimate nameserver can race to answer the resolver’s query. With a 16-bit TXID and a fixed source port, there are only 65,535 possible values: a fast attacker floods the resolver with all 65,535 forged responses, betting one will arrive and match the TXID before the legitimate answer. Kaminsky’s insight was that an attacker could cause the resolver to make many queries in rapid succession by querying random subdomains (each generates a new TXID), giving thousands of shots at the race without waiting for cache TTLs to expire. Modern resolvers use both source port randomization (extending the space to \(65535 \times 65535 \approx 2^{32}\)) and 0x20 encoding (randomizing the case of query letters, since DNS is case-insensitive for resolution but case-preserving in the response, so a forged response must match the exact capitalization the resolver used). Even so, a 32-bit entropy space is vulnerable to off-path spoofing attacks from fast networks. DNSSEC is the only complete fix.

DNSSEC (DNS Security Extensions, RFC 4033–4035) retrofits public-key signatures onto DNS. Every zone operator generates a key pair; the public key is published as a DNSKEY record, and every resource record set (RRset) is signed, producing an RRSIG record.

10.1.1 ZSK/KSK Split and the Chain of Trust

DNSSEC Key Hierarchy

KSK (Key Signing Key)
Long-lived, high-value key. Used only to sign the DNSKEY RRset. Typically 2048-bit RSA or P-256 ECDSA.
ZSK (Zone Signing Key)
Shorter-lived key used to sign all other RRsets. Rolling ZSKs every 30–90 days limits exposure.
DS record
A hash of the child zone’s KSK, published by the parent zone, forming the chain of trust up to the root.

The root zone’s KSK is the Trust Anchor. ICANN maintains an offline ceremony to sign the root KSK; the 2010 and 2018 key rollovers were broadcast publicly. A validator begins at a locally configured trust anchor (often the root), walks the DS chain downward, and verifies each RRSIG with the corresponding DNSKEY.

The 2018 root KSK rollover highlighted an operational risk in DNSSEC: about 25 million resolvers worldwide had their trust anchor configured to the old KSK only and had not fetched the new one via RFC 5011 automatic rollover. Had ICANN not delayed the rollover, those resolvers would have started returning SERVFAIL for all DNSSEC-signed domains globally. A trust anchor is a hard dependency — if the validator’s configured trust anchor doesn’t match the current signed key at the root, everything under that zone appears forged. Operators who deploy DNSSEC validation must implement automated trust anchor updates and monitor for rollover events.

DNSSEC also doesn’t protect the last mile between the recursive resolver and the client application. A validating recursive resolver at 8.8.8.8 performs full DNSSEC validation, but if your application sends its DNS query over plain UDP to the resolver, an attacker on your local network can still inject a forged response before it arrives. DNS-over-TLS or DNS-over-HTTPS between the client and the validating resolver closes this gap.

DNSSEC chain of trust from root to resource record.
Figure 10.1. DNSSEC chain of trust from root to resource record.

10.1.2 NSEC3 Authenticated Denial of Existence

When a queried name does not exist, DNSSEC must prove that fact without leaking the full zone contents. NSEC records expose the sorted list of all names; NSEC3 (RFC 5155) uses salted SHA-1 hashes of owner names to prevent zone enumeration while still allowing authenticated denial.

10.1.3 Key Rollover

RFC 7583 defines the recommended ZSK pre-publish rollover: (1) publish new ZSK in DNSKEY RRset; (2) wait one TTL for caches to pick it up; (3) switch all signing to new ZSK; (4) wait one TTL; (5) remove old ZSK. KSK rollover additionally requires updating the DS record at the parent.

10.2 DNS Privacy: DoH, DoT, and ODoH

Plain DNS leaks query contents to every hop between the client and resolver. Three standards address this:

Table 10.1. DNS Privacy Protocol Comparison
Protocol Port RFC Notes
DNS-over-TLS (DoT) 853 TCP RFC 7858 Easy to block; resolver sees client IP
DNS-over-HTTPS (DoH) 443 TCP RFC 8484 Blends with web traffic; resolver sees IP
Oblivious DoH (ODoH) 443 TCP RFC 9230 Proxy decouples IP from query content
DNS-over-QUIC (DoQ) 853 UDP RFC 9250 Low latency; nascent deployment

ODoH introduces an Oblivious Proxy between client and target resolver. The client encrypts the DNS query for the target using HPKE; the proxy cannot read the query; the target cannot see the client IP. Combined with DNSSEC, ODoH provides both confidentiality and integrity.

The threat model for DNS privacy protocols differs from DNSSEC’s. DNSSEC protects the content of DNS answers against forgery. DoT, DoH, and ODoH protect the confidentiality of DNS queries against eavesdroppers. These are orthogonal: you can have forged answers delivered confidentially (DoH without DNSSEC) or authentic answers delivered in plaintext (DNSSEC without DoT/DoH). A complete DNS security posture requires both. DoH also has an important operational impact: because it uses port 443 and looks like HTTPS traffic, enterprise firewalls can’t easily block it without also blocking legitimate web traffic, which means employees can bypass corporate DNS filtering using a hard-coded DoH resolver. Enterprise deployments therefore typically deploy canary DNS domains or TLS inspection to detect DoH usage that bypasses the corporate resolver.

10.3 BGP Security and RPKI

The Border Gateway Protocol (BGP) was designed in 1989 with no authentication. Any autonomous system (AS) can announce any IP prefix, enabling prefix hijacking and route leaks.

Pakistan Telecom BGP Hijack, 2008

On 24 February 2008, Pakistan Telecom (AS17557) announced a more-specific 208.65.153.0/24 prefix that covered YouTube’s 208.65.152.0/22. The announcement propagated globally within minutes, black-holing YouTube traffic for approximately 2 hours. No cryptographic authentication was required to make the announcement.

RPKI (Resource Public Key Infrastructure, RFC 6480) binds IP address blocks to their legitimate holder via X.509 certificates rooted at the five Regional Internet Registries (RIRs). An operator creates a Route Origin Authorization (ROA):

Route Origin Authorization:
  AS:        64496
  Prefix:    192.0.2.0/24
  Max-Length: 24
  Not Before: 2024-01-01
  Not After:  2025-01-01

Routers running an RPKI-to-Router (RTR) daemon obtain validated prefix-origin pairs and can set route preference: Valid, Invalid, or NotFound. BGPsec (RFC 8205) adds per-hop AS-path signatures but has seen limited deployment due to performance cost.

10.3.1 What RPKI Validates and What It Doesn’t

RPKI with ROAs validates one thing precisely: that the AS announcing a prefix is authorized to originate it. It does not validate the AS path — the list of intermediate autonomous systems through which the announcement traveled. An attacker can take a legitimate ROA-valid origin announcement and prepend their own AS to the path (a route leak), and RPKI won’t flag it. They can also announce a more-specific prefix that overlaps with a ROA-covered block; if the ROA’s max-length field permits the more-specific announcement, RPKI marks it valid even though it causes traffic to divert.

The max-length field is a common misconfiguration. Operators sometimes set it too permissively (e.g., max-length: 32 for a /24 allocation), which means any more-specific prefix up to a /32 from that AS is RPKI-valid. An attacker who hijacks a host route inside that block gets a “Valid” RPKI result because the origin AS matches and the prefix length is within bounds. Correct ROA hygiene requires max-length to equal the actual prefix length of all announced prefixes, with separate ROAs for each aggregate and more-specific announcement that legitimately originates.

Even with perfect ROA coverage, RPKI enforcement is uneven. Routers that receive an “Invalid” route can drop it (route origin validation in strict mode), prefer valid routes, or ignore the status entirely. As of 2024, roughly 50% of IPv4 prefixes have ROAs but enforcement by receiving ASes is far less consistent. A route marked Invalid by one AS’s RPKI validator may still be accepted by an AS that hasn’t enabled validation. BGPsec would close the AS-path gap but requires all ASes along a path to sign their announcements, and a single non-participating AS breaks the chain — which is why BGPsec deployment remains near zero despite years of standardization effort.

10.4 ARP Spoofing and Dynamic ARP Inspection

ARP has no authentication. An attacker on the same Layer-2 segment can send gratuitous ARP replies, poisoning the ARP cache of victims and enabling man-in-the-middle attacks. ARP caches accept unsolicited replies — a host can send a gratuitous ARP (a reply with no corresponding request) and every device on the segment that receives it will update its cache. Tools like arpspoof and ettercap automate this: the attacker sends gratuitous ARPs claiming the gateway IP maps to the attacker’s MAC, then sends the same for the victim, positioning themselves between all gateway-bound traffic. Because ARP operates below the transport layer, TLS provides no protection against the man-in-the-middle position itself — but TLS certificate verification prevents the attacker from decrypting intercepted HTTPS traffic (assuming the victim doesn’t click through a certificate warning).

Dynamic ARP Inspection (DAI) is a Cisco IOS feature that validates ARP packets against the DHCP snooping binding table (IP \(\leftrightarrow\) MAC \(\leftrightarrow\) port):

! Enable DHCP snooping globally
ip dhcp snooping
ip dhcp snooping vlan 10,20

! Mark uplinks and server ports as trusted
interface GigabitEthernet0/1
 ip dhcp snooping trust
 ip arp inspection trust

! Enable DAI per VLAN
ip arp inspection vlan 10,20

! Rate-limit ARP on untrusted ports
interface GigabitEthernet0/2
 ip arp inspection limit rate 100

Packets with MAC-to-IP bindings not found in the snooping table are dropped and the port may be error-disabled.

10.5 IPv6 Neighbor Discovery Security

IPv6 Neighbor Discovery Protocol (NDP) replaces ARP and adds router discovery, but inherits the same lack of authentication. Key threats include rogue Router Advertisements (RAs) that redirect all traffic through an attacker.

10.5.1 RA Guard (RFC 6105)

Switches inspect incoming ICMPv6 packets and drop RA messages received on ports configured as host-facing. Only uplink ports designated as router-facing are permitted to forward RAs.

10.5.2 SEND (RFC 3971)

Secure Neighbor Discovery uses Cryptographically Generated Addresses (CGA): the lower 64 bits of an IPv6 address are derived from the node’s public key via SHA-1. Every NDP message is signed with the corresponding private key, preventing spoofing. Deployment is rare due to high computational overhead.

10.5.3 Privacy Extensions (RFC 8981)

Nodes generate temporary, randomized interface identifiers for outbound connections, renewed periodically (default: 24 h preferred, 48 h valid). The stable address derived from EUI-64 or stable-opaque (RFC 7217) is used for inbound services.

10.6 WPA3-Personal: SAE/Dragonfly Key Exchange

WPA2-Personal used the 4-way handshake with a Pre-Shared Key (PSK) that is vulnerable to offline dictionary attacks if the handshake is captured. WPA3-Personal replaces PSK with Simultaneous Authentication of Equals (SAE), also known as Dragonfly (RFC 7664).

SAE Handshake Steps

  1. Both parties independently derive a Password Element (PE)—a point on an elliptic curve—from the password and their MAC addresses, using a deterministic hunting-and-pecking or hash-to-curve algorithm.

  2. Each party generates a random scalar \(r\) and element \(E\), then sends a Commit message: \((scalar, element) = (r + mask \bmod q,\; -(mask \cdot PE))\).

  3. After receiving the peer’s Commit, each derives the shared secret \(K = (r_{self} + mask_{self})(r_{peer} \cdot PE + E_{peer})\).

  4. A Confirm message (HMAC of the transcript) proves knowledge of \(K\) without revealing it.

SAE provides forward secrecy: capturing the exchange does not allow offline password guessing because no verifier of the password is transmitted. WPA3 also mandates PMF (Protected Management Frames, 802.11w) to prevent deauthentication attacks.

WPA2’s vulnerability to offline dictionary attacks was severe in practice. An attacker who captures the 4-way handshake (which airodump-ng and a deauthentication frame make trivially easy to collect) can test billions of password candidates offline using GPU-accelerated tools like hashcat. For an 8-character password from the printable ASCII alphabet, the full keyspace is about \(95^8 \approx 6 \times 10^{15}\); a modern GPU cluster running at \(10^{11}\) tests per second searches it in under a day. SAE eliminates this attack entirely: every SAE exchange produces a unique shared secret derived from the ephemeral randomness, so captured SAE transcripts contain no offline-testable verifier. The only way to test a password guess is to attempt a new SAE handshake, which the access point rate-limits and can lock out after repeated failures.

A known limitation of SAE is the Dragonblood vulnerability family (2019): timing and cache side-channel attacks against the hunting-and-pecking algorithm allowed offline password recovery from observed handshake timing. The fix was to replace hunting-and-pecking with a hash-to-curve algorithm that runs in constant time; updated implementations (WPA3 R2) are immune, but older devices that received no firmware update remain vulnerable.

10.7 ICMP Security Policy

Table 10.2. Recommended ICMP Filtering Policy
Type Name Direction Action
0 Echo Reply In Allow (if stateful)
3 Destination Unreachable In Allow (needed for PMTUD)
8 Echo Request In Allow (selective) / Drop
11 Time Exceeded In Allow (traceroute)
3/4 Fragmentation Needed In Allow (PMTUD critical)
5 Redirect In Drop (potential attack)
17 Address Mask Request Both Drop (obsolete)

10.8 Protocol Security Deployment Status

Table 10.3. Protocol Security Extension Deployment Status (2024)
Protocol Mechanism Deployment
DNS DNSSEC \(\sim\)35% of TLDs fully signed; patchy recursive validation
DNS DoT / DoH Growing; major resolvers (1.1.1.1, 8.8.8.8) support both
BGP RPKI \(\sim\)50% of IPv4 prefixes have ROAs; enforcement varies
BGP BGPsec Experimental only
IPv6 SEND Minimal; academic
Wi-Fi WPA3 Mandatory for Wi-Fi 6 (802.11ax) certification
Email DNSSEC+DANE \(<\)5% of mail servers

10.9 Exercises

Exercise 10.1

A DNSSEC validator receives an RRSIG with a signature expiry timestamp in the past. The resolver’s system clock is accurate. Describe what the validator should do and explain the security consequence of accepting expired signatures.

Exercise 10.2

You are the network administrator for AS64497. You discover that a neighboring AS is announcing a more-specific prefix that overlaps with your legitimate 203.0.113.0/24 allocation. Describe two immediate technical actions and one long-term structural remedy.

Exercise 10.3

A penetration tester on a wired LAN segment captures an ARP exchange and observes that the victim’s ARP cache now maps the gateway IP to the attacker’s MAC. List the prerequisites for this attack, three artifacts a forensic analyst could find to prove it occurred, and the exact DAI configuration needed to prevent it.

Exercise 10.4

Compare the forward secrecy properties of WPA2-Personal (4-way handshake with PSK) and WPA3-Personal (SAE). Assume an attacker recorded 1,000 WPA2-Personal handshakes and 1,000 WPA3-Personal SAE exchanges using a password-length-8 dictionary. How many offline operations does each scenario require, and why?