A newly disclosed vulnerability in OpenSSL, dubbed “HollowByte,” allows a remote, unauthenticated attacker to trigger a denial-of-service (DoS) condition using a malicious payload as small as 11 bytes.
Discovered by the Okta Red Team, the flaw exploits how OpenSSL pre-allocates memory during the TLS handshake, forcing servers to reserve massive memory chunks before any authentication ever occurs.
OpenSSL “HollowByte” Vulnerability
Every TLS handshake begins with a ClientHello message wrapped in a record carrying a 4-byte header that declares the size of the incoming message body. In older OpenSSL versions, the library allocates a receive buffer based purely on that attacker-declared length before any actual data arrives.
When a crafted 11-byte payload lands, the TLS state machine reads the header and triggers the following unvalidated allocation chain:
- Read Header → grow_init_buf() → OPENSSL_clear_realloc() → malloc(attacker_size)
Since no validation occurs at this stage, a single malicious packet can force malloc() to allocate up to 131 KB based solely on the attacker’s claim. The worker thread then blocks indefinitely, waiting for data that never arrives.
Classic connection-exhaustion attacks like Slowloris rely on holding connections open to starve server threads. HollowByte compounds this with a memory-fragmentation problem rooted in how glibc manages freed memory.
When an attacking connection drops, OpenSSL frees the buffer, but glibc does not immediately return small-to-medium allocations to the operating system; it retains them for potential reuse.
By launching waves of connections with randomized claimed sizes, attackers prevent the allocator from reusing these freed chunks, causing the server’s Resident Set Size (RSS) to climb continuously and permanently, even after the attacker disconnects. The only remedy is killing the process itself.
Okta Red Team testing on unpatched OpenSSL instances running NGINX revealed severe consequences:
- In a 1 GB RAM environment, the server was OOM-killed after accumulating 547 MB of frozen, fragmented memory.
- In a 16 GB RAM environment, the attack locked up 25% of total system memory while staying under standard connection ceilings — meaning typical connection-limiting defenses fail to stop it.
Because OpenSSL underpins a vast swath of internet infrastructure, the vulnerability’s blast radius extends to web servers like Apache and NGINX, language runtimes including Node.js, Python, Ruby, and PHP, and databases such as MySQL and PostgreSQL.
OpenSSL resolved the issue by shifting to incremental buffer growth, merged via pull requests #30792, #30793, and #30794. Instead of trusting the header’s claim outright, OpenSSL now expands the buffer only as bytes actually arrive on the wire — meaning an empty claim costs the server nothing.
The fix was quietly folded into OpenSSL v4.0.1, with silent backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21. Notably, OpenSSL treated this as a hardening improvement rather than issuing a formal CVE advisory, a pattern seen in other recent OpenSSL disclosures where DoS-class issues are addressed without high-profile CVE designations.
Because HollowByte received silent-patch treatment rather than a CVE, many organizations may not flag it through standard vulnerability scanning. Given OpenSSL’s ubiquity across web servers, runtimes, and databases, security teams should treat this as a priority patch regardless of CVE status.
Tip for Defenders
- Upgrade to OpenSSL 4.0.1 or the relevant backported version (3.6.3, 3.5.7, 3.4.6, or 3.0.21) immediately.
- Audit embedded OpenSSL versions in language runtimes (Node.js, Python, Ruby, PHP) since OS-level patches alone won’t fix bundled copies
- Monitor RSS memory trends on TLS-terminating servers for signs of gradual, unexplained memory bloat consistent with fragmentation attacks.
Strengthen Your SOC by Accelerating Threat Detection & Rapid Investigations. -> Integrate ANY.RUN With Your SOC Now.
