A denial-of-service vulnerability, dubbed "HollowByte," has been discovered in OpenSSL that allows remote, unauthenticated attackers to exhaust server memory and trigger denial-of-service attacks using a specially crafted 11-byte payload. The flaw was disclosed by Okta's Red Team on July 18, 2026.
The attack leverages a design issue where older versions of OpenSSL pre-allocate memory based on a declared message size in the TLS handshake header before any actual data arrives. When a malicious 11-byte payload is sent, the TLS state machine reads a 4-byte handshake header and triggers an unvalidated pre-allocation of up to 131 KB based on the header's 3-byte length declaration. Because there is no payload validation at this early stage, the `malloc()` function allocates memory solely based on the untrusted packet's claims.
After the memory allocation, the worker thread blocks indefinitely, waiting for data that never arrives. Even if the attacking connection drops, the allocated memory is not immediately returned to the operating system by `glibc`; instead, it is held for potential reuse. By launching waves of connections with randomized claimed sizes, an attacker can prevent the memory allocator from reusing these freed chunks, leading to progressive heap fragmentation and a continuous climb in the server's resident memory footprint. The only way to reclaim this memory is to restart the affected process.
Okta's testing demonstrated the severity of the vulnerability. In a standard 1GB RAM environment, an unpatched NGINX server was terminated by the operating system's out-of-memory (OOM) killer after accumulating 547MB of frozen, fragmented memory. In higher-spec environments with 16GB RAM, the attack successfully consumed 25% of the system's total memory while remaining below typical connection limits, indicating that standard defenses like rate limiting and connection caps are ineffective against this specific attack vector.
The scope of this vulnerability is broad due to the widespread use of OpenSSL. It impacts a variety of software, including popular web servers like Apache and NGINX, language runtimes such as Node.js, Python, Ruby, and PHP, and databases like MySQL and PostgreSQL.
The OpenSSL team addressed the issue by implementing incremental buffer growth. The server now only allocates memory as data actually arrives, meaning an empty claim no longer incurs a memory cost. The fix was incorporated through pull requests 30792, 30793, and 30794. While no CVE was assigned, as OpenSSL treated it as a hardening fix rather than a security advisory, the fix was included in OpenSSL v4.0.1 and silently backported to versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21. Users are advised to update their OpenSSL packages regardless.
