Chapter 4: System and Operating-System Attacks¶
System and operating-system attacks go after the foundation under every application. If an attacker controls the operating system, the attacker controls everything that runs on it. This chapter covers buffer overflows, privilege escalation, rootkits, zero-day exploits, DLL hijacking and race conditions. Several of these are about getting more power than you were given, and several are about hiding once you have it.
Layman analogy
If the network is the road system and web applications are storefronts, the operating system is the building's foundation and structural frame. System attacks go after the foundation itself.
4.1 Buffer Overflow¶
A program allocates a fixed-size block of memory called a "buffer" (like a box of a specific size). If an attacker sends more data than the buffer can hold, the overflow writes into adjacent memory — including the program's execution instructions — allowing the attacker to redirect what the program does next.
Layman analogy
Imagine a form that says "Enter your name (maximum 10 characters)." You type 50 characters. The extra 40 spill over into the "next instruction" box and change what the programme does — now it runs YOUR instructions instead.
- Stack-based overflow — overwrites the return address on the call stack, redirecting execution
- Heap-based overflow — overwrites dynamically allocated memory structures
- Return-to-libc — redirects execution to existing system library functions to avoid writing new shellcode
Example 4.1: Buffer Overflow
The Morris Worm (1988) — the first internet worm — exploited a buffer overflow in the Unix "fingerd" daemon. It self-replicated across the early internet, infecting ~6,000 machines (roughly 10% of all internet-connected computers at the time).
4.2 Privilege Escalation¶
An attacker who starts with limited, low-privilege access uses vulnerabilities to gain higher permissions — becoming an administrator or root user (the highest privilege level on a system).
- Vertical Escalation — regular user gains administrator/root access
- Horizontal Escalation — regular user accesses another user's data (same privilege, different account)
Common techniques:
- Kernel exploits (e.g., "Dirty COW" CVE-2016-5195 — any Linux user could become root)
- Misconfigured SUID binaries — programs that always run as root, regardless of who launches them
- Unquoted service paths — Windows searches for executables in incorrect locations, allowing substitution
Example 4.2: Privilege Escalation
CVE-2021-4034 "PwnKit": A vulnerability in the polkit package present in virtually every Linux distribution for 12 years. Any local user could run a single command and instantly gain full root (administrator) access. Patched in January 2022.
4.3 Rootkits¶
Malware specifically engineered to hide its own presence — and the presence of other malware — from the operating system, antivirus software, and system administrators. Operates at the deepest levels of the system.
Layman analogy
Imagine hiring a corrupt building inspector who falsifies the inspection report to hide illegal construction. When you look at the blueprints (Task Manager, File Explorer), everything appears clean. The illegal rooms (malware) are there but completely hidden from official view.
- Kernel-mode rootkits — run at the highest privilege level, modifying the OS kernel itself
- User-mode rootkits — intercept API calls between applications and the kernel
- Bootkits — load before the operating system even starts, invisible to any OS-level detection
Example 4.3: Rootkits
2005 — Sony BMG Music: Sony secretly installed a rootkit on millions of Windows PCs through commercially purchased music CDs. The rootkit hid itself and other files from the OS, created security vulnerabilities, and was installed without user consent. Sony faced massive legal action in multiple countries.
4.4 Zero-Day Exploits¶
A "zero-day" is a software vulnerability that is unknown to the vendor. Because it's unpatched, there is no defence available. Defenders have had "zero days" to fix it.
Layman analogy
Your house has a secret hidden entrance that even YOU don't know about. Burglars discover it first. Until you find it and seal it, your house is completely vulnerable no matter how many other locks and alarms you install.
- Browser zero-days — attack through Chrome, Firefox, Safari while users simply browse the web
- OS kernel zero-days — gain kernel-level control without any user interaction
- Hardware firmware zero-days — persist even after complete OS reinstallation
Example 4.4: Zero-Day Exploits
Stuxnet (2010): A nation-state cyberweapon (attributed to the US and Israel) that used FOUR simultaneous Windows zero-days. It infiltrated Iranian nuclear facilities, destroyed centrifuges by causing them to spin at destructive speeds, while reporting completely normal operation to human operators. One of the most sophisticated cyber weapons ever publicly documented.
4.5 DLL Hijacking¶
Windows applications load Dynamic Link Libraries (DLLs) — shared code modules — by searching through folders in a specific order. If an attacker places a malicious DLL in a folder that is searched before the legitimate location, the malicious code runs instead.
Layman analogy
You ask your assistant to fetch "the Johnson contract." They check your desk first, then the filing cabinet. An attacker places a fake "Johnson contract" on your desk. Your assistant finds and acts on the fake one before ever reaching the legitimate filing cabinet.
- Search order hijacking — place malicious DLL in the current directory or early in the PATH
- DLL side-loading — place malicious DLL next to a legitimate, digitally signed application
- Phantom DLL hijacking — create a DLL that an application tries to load but that doesn't yet exist
4.6 Race Condition (TOCTOU)¶
TOCTOU stands for Time-of-Check to Time-of-Use. An attacker exploits the tiny time gap between when a program checks a security condition and when it actually uses the resource — substituting something malicious in between.
Layman analogy
A nightclub bouncer checks your ID (Time of Check) and waves you forward to walk to the door (Time of Use). In the 2 seconds you walk, someone swaps your genuine wristband for a fake one. The door scanner sees the (now fake) wristband, not what the bouncer verified.
4.7 Defence tips¶
- Keep operating systems and all software fully updated with security patches
- Apply the principle of least privilege — users and applications get only the minimum access they need
- Use application whitelisting to prevent unauthorised programs from running
- Deploy Endpoint Detection and Response (EDR) tools for behavioural monitoring
- Enable secure boot to protect against bootkits loading before the OS
- Monitor system logs for unusual privilege escalation events
4.8 Summary¶
- A buffer overflow writes beyond a fixed block of memory and can redirect execution; privilege escalation gains more power than granted, vertically or horizontally.
- Rootkits hide malware from the operating system, down to bootkits that load before it; zero-days exploit flaws the vendor does not yet know.
- DLL hijacking abuses the order in which Windows searches for libraries; a race condition (TOCTOU) exploits the gap between a check and a use.
- Defences: patching, least privilege, application whitelisting, EDR, secure boot and log monitoring.
4.9 Exercises¶
-
Explain how a stack-based buffer overflow can redirect a program.
Answer
The program copies input into a fixed-size buffer on the call stack. Too much input overruns the buffer and overwrites the saved return address next to it. When the function returns, execution jumps to the address the attacker placed there.
-
Give an example of vertical and of horizontal privilege escalation.
Answer
Vertical: a regular user becomes root or administrator, for example through a kernel exploit such as Dirty COW or a misconfigured SUID program. Horizontal: a regular user reads or changes another regular user's data at the same privilege level.
-
Why can antivirus running inside the operating system not find a bootkit?
Answer
A bootkit loads before the operating system starts, so it can alter what the operating system sees. Any scan that runs inside the compromised system relies on that altered view. Secure boot, which verifies the boot chain, is the countermeasure.
-
Why is a zero-day vulnerability called "zero-day", and why is defence so hard?
Answer
The vendor has had zero days to fix it, because the flaw is unknown to the vendor. No patch exists, so patching cannot help until one is released. Defence relies on layers: least privilege, monitoring and segmentation.
-
Describe a TOCTOU attack using a symbolic link and the principle that prevents it.
Answer
A privileged program checks that a file is harmless and then opens it. In the gap between check and use the attacker replaces the file with a symlink to a sensitive file, and the program acts on that. The principle is to make the check and the use a single atomic operation, so nothing can change in between.