Core protocols and their security

The internet protocol suite was designed in the 1970s and 1980s for a small community of researchers who largely trusted each other. Most of its core protocols therefore authenticate nothing and encrypt nothing. ARP believes any reply, DNS originally believed any answer that matched its query, DHCP hands configuration to whoever asks, and TCP trusts that the address on a packet is real. Security has since been added in layers, extensions and replacements, but the original weaknesses are still widely exploitable on networks that do not deploy the fixes. This lesson walks through the core protocols, how each is attacked and which controls a security manager should insist on. Cryptographic protocols such as TLS 1.3 are covered in depth in CISSP-112; here we focus on how they protect the protocols that lack security of their own.

TCP and UDP

The Transmission Control Protocol, TCP, provides a reliable byte stream. It numbers every byte with sequence numbers, acknowledges what arrives, retransmits what is lost, reorders out-of-order segments and controls the sending rate with a sliding window. A connection begins with the three-way handshake: the client sends a segment with the SYN flag and an initial sequence number; the server replies with SYN and ACK, acknowledging the client's number and choosing its own; and the client sends ACK. Connections end with a FIN exchange, or abruptly with a RST. The flags in the TCP header, SYN, ACK, FIN, RST, PSH, URG and the congestion flags, are what firewalls and intrusion detection systems examine to track connection state.

Weaknesses follow from this design. In a SYN flood the attacker sends many SYN segments, often from spoofed addresses, and never completes the handshake, filling the server's table of half-open connections. SYN cookies defend by encoding the connection state in the server's initial sequence number so that no memory is used until the final ACK arrives; firewalls and load balancers can also proxy the handshake. Session hijacking takes over an established connection by predicting sequence numbers and injecting segments; modern operating systems randomize initial sequence numbers, and encryption with integrity protection, such as TLS or IPsec, makes injected data detectable. A forged RST can tear down a connection, which is why long-lived routing sessions such as BGP use TCP authentication options.

The User Datagram Protocol, UDP, sends independent datagrams with no handshake, no acknowledgement, no ordering and no flow control. It is used where speed matters more than reliability, or where the application handles reliability itself: DNS queries, DHCP, SNMP, syslog, NTP, voice and video, and QUIC, the transport under HTTP/3. Because UDP has no handshake, the source address is trivially spoofed, which makes UDP services ideal for reflection and amplification attacks: a small spoofed request produces a large reply sent to the victim. Lesson 10 quantifies this.

Port numbers identify services. Well-known ports run from 0 to 1023, registered ports from 1024 to 49151 and dynamic or ephemeral ports from 49152 to 65535. A port scan probes which ports respond. In TCP scanning, an open port answers a SYN with SYN-ACK, a closed port answers with RST and a filtered port, behind a firewall that silently drops packets, gives no answer at all. UDP scanning is slower and less certain because open UDP ports often send nothing back. Scanning your own networks is a normal control, covered in CISSP-115; scanning others without permission may be an offence under computer misuse laws.

# Illustrative capture of a TCP handshake and an HTTP request (addresses from documentation ranges)
10:15:02.100 IP 192.0.2.10.51544 > 198.51.100.20.80: Flags [S], seq 1849102, win 64240, length 0
10:15:02.130 IP 198.51.100.20.80 > 192.0.2.10.51544: Flags [S.], seq 3911877, ack 1849103, win 65160, length 0
10:15:02.130 IP 192.0.2.10.51544 > 198.51.100.20.80: Flags [.], ack 3911878, win 502, length 0
10:15:02.131 IP 192.0.2.10.51544 > 198.51.100.20.80: Flags [P.], seq 1849103:1849190, ack 3911878, length 87: HTTP: GET /login?user=amira&pass=Spring2026 HTTP/1.1
10:15:02.162 IP 198.51.100.20.80 > 192.0.2.10.51544: Flags [.], ack 1849190, length 0

ARP

The Address Resolution Protocol, ARP, maps IPv4 addresses to MAC addresses on a local segment. A host that wants to reach 10.20.0.1 broadcasts "who has 10.20.0.1?" and the owner replies with its MAC address, which the host caches. ARP has no authentication, and hosts accept unsolicited replies, called gratuitous ARP. In ARP poisoning or spoofing, an attacker on the segment sends forged replies claiming that the gateway's IP address belongs to the attacker's MAC address. Victims then send their traffic to the attacker, who can read, modify or drop it before forwarding it on: a classic on-path, or man-in-the-middle, attack. Defences include dynamic ARP inspection on switches, which checks ARP replies against the bindings that DHCP snooping recorded; static ARP entries for critical devices; segmentation that keeps untrusted devices off sensitive segments; and, above all, encryption, which keeps an on-path attacker from reading or silently altering data even if they intercept it.

DHCP

The Dynamic Host Configuration Protocol, DHCP, gives hosts an IP address, subnet mask, default gateway and DNS servers through a four-step exchange: discover, offer, request and acknowledge, often remembered as DORA. DHCP has no authentication in normal use. A rogue DHCP server, whether a misconfigured home router plugged in by an employee or a deliberate attacker, can hand out its own address as the gateway or DNS server, steering victims' traffic. DHCP starvation exhausts the legitimate server's pool by requesting leases with many fake MAC addresses, often as a prelude to a rogue server. DHCP snooping on switches marks only uplink ports towards real servers as trusted and drops server messages arriving on any other port; port security limits how many MAC addresses a port can use. DHCP logs are also vital for investigations, because they record which device held which address and when.

DNS

The Domain Name System translates names into addresses and much more: mail exchanger records, text records used for email authentication, service records and certificate authority authorization records. Resolution starts at a client's stub resolver, which asks a recursive resolver; the recursive resolver walks down the hierarchy from the root servers to the top-level domain servers to the authoritative servers for the zone and caches the answers for the time to live each record specifies. Because DNS underpins nearly every connection, it is both a prime target and a valuable control point.

Attacks on DNS include cache poisoning, where an attacker tricks a recursive resolver into caching a forged answer so that everyone using it is sent to the wrong address; the 2008 Kaminsky attack showed how fast this could be done, and resolvers responded by randomizing source ports and query identifiers. Domain hijacking takes over a domain's registration or its records through stolen registrar credentials. Typosquatting and look-alike domains trick users. DNS tunnelling hides data in queries and responses to exfiltrate information or carry command and control traffic past firewalls that allow DNS. DNS servers can be abused for amplification attacks, and open resolvers that answer anyone are especially attractive. Zone transfers that are not restricted leak a full map of an organization's names.

DNS Security Extensions, DNSSEC, add origin authentication and integrity to DNS answers. Each zone signs its records; the parent zone publishes a delegation signer, DS, record that vouches for the child's key; and the chain runs up to the root, whose key is the trust anchor configured in validating resolvers. A validating resolver that receives a forged answer finds that the signature does not verify and refuses it. DNSSEC does not encrypt anything, so it provides no confidentiality. Privacy comes from DNS over TLS, DoT, on port 853, and DNS over HTTPS, DoH, which hides queries inside ordinary HTTPS on port 443. DoH protects users on hostile networks but can also bypass an organization's DNS filtering and monitoring if browsers use an outside resolver, so enterprises usually configure managed devices to use the corporate resolver and block unknown DoH services.

Protective DNS is now a common government service. In Canada, the Canadian Centre for Cyber Security, part of the Communications Security Establishment, operates the Canadian Shield service with the Canadian Internet Registration Authority, which blocks known malicious domains for organizations and individuals that point their resolvers at it. Treat any such service as one layer in a defence, not a complete solution.

ICMP

The Internet Control Message Protocol, ICMP, carries error and diagnostic messages for IP: echo request and reply used by ping, destination unreachable, time exceeded used by traceroute, and redirect. ICMP has been abused in ping floods, the old ping of death that used oversized fragmented packets, smurf attacks that sent spoofed echo requests to broadcast addresses so every host replied to the victim, ICMP redirects that altered hosts' routing, and ICMP tunnelling to hide data. Blanket blocking is tempting, but it breaks path MTU discovery, which relies on "fragmentation needed" messages, and harms troubleshooting. The mature approach is to allow the ICMP types the network needs, rate-limit echo, drop redirects at hosts and routers, disable directed broadcasts and monitor for unusual volumes. As lesson 2 explained, IPv6 needs ICMPv6 even more.

Other protocols to know

Several other protocols appear often in Domain 4 questions. The Simple Network Management Protocol, SNMP, monitors and configures devices; versions 1 and 2c use community strings sent in clear text, often left at defaults such as "public", while SNMPv3 adds user authentication and encryption and should be the only version permitted. The Network Time Protocol, NTP, keeps clocks synchronized, which matters for logs, certificates and Kerberos; Network Time Security, NTS, adds authentication. Syslog carries log messages, traditionally over UDP 514 without protection; syslog over TLS fixes this. Lightweight Directory Access Protocol, LDAP, should run over TLS. Server Message Block, SMB, used for Windows file sharing, has been central to major worms such as WannaCry in 2017 and should never be reachable from the internet. Remote Desktop Protocol exposed directly to the internet remains one of the commonest initial access routes for ransomware gangs.

Thinking like a manager

When a question describes a clear-text protocol in use, the best answer is usually to replace it with its secure counterpart, not to add monitoring around it. When it describes a local attack such as ARP poisoning or a rogue DHCP server, look for switch-level controls and segmentation, and remember that encryption limits the damage of any on-path position. When DNS is involved, think in two directions: integrity, which DNSSEC provides, and confidentiality, which DoT and DoH provide. And whenever a protocol has no built-in authentication, ask what compensating control the organization relies on and whether anyone owns it.