A denial-of-service vulnerability, dubbed HollowByte, has been identified in OpenSSL, allowing unauthenticated attackers to deplete server memory with minimal data transmission. The OpenSSL team has addressed the flaw in recent updates, though no CVE identifier has been assigned. Organizations are strongly advised to update their OpenSSL installations to a patched version.
The vulnerability was detailed in an advisory by Okta's Red Team. It exploits a weakness in how OpenSSL handles TLS handshake messages. Each TLS handshake message begins with a 4-byte header that indicates the size of the subsequent message body. Vulnerable OpenSSL versions would allocate memory based on this declared length *before* receiving and validating the actual payload.
An attacker can initiate a TLS connection and send an 11-byte malicious input. This input includes a header that falsely declares a significantly larger message body will follow. The server, trusting the header's claim, allocates a substantial amount of memory and then waits indefinitely for the promised data, effectively blocking the worker thread. By repeating this process across multiple connections, an attacker can force the server to allocate considerable amounts of memory using a relatively small volume of transmitted data.
While OpenSSL eventually frees these buffers when a connection terminates, the GNU C Library (glibc), which handles memory management, does not immediately return small-to-medium allocations to the operating system. Instead, glibc retains them for potential reuse. Okta researchers observed that by launching waves of connections with randomized claimed sizes, attackers can prevent the memory allocator from reusing these freed chunks. This leads to severe heap fragmentation and a continuous increase in the server's Resident Set Size (RSS). Even after an attacker disconnects, the server remains in a permanently bloated state, with the only way to fully reclaim the memory being a process restart.
The OpenSSL library is a fundamental component for secure internet communication, embedded in numerous popular software projects. These include web servers like NGINX and Apache, language runtimes such as Node.js, Python, Ruby, and PHP, and databases like MySQL and PostgreSQL. It is also pre-installed on most Linux distributions for TLS encryption and certificate handling.
Okta's tests on NGINX servers demonstrated that low-capacity environments could be easily depleted of memory using the HollowByte attack. Higher-specification servers could lose up to 25% of their memory, even while the attack bandwidth remained below typical security alerting thresholds. Although denial-of-service flaws are generally considered less severe than vulnerabilities enabling data theft or code execution, they can cause significant operational disruptions and reputational damage.
The HollowByte issue has been addressed in OpenSSL 4.0.1. The fix has also been backported to older releases, specifically versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21. These patched versions now grow the buffer only as data arrives, effectively ignoring the potentially false claims in the header. Despite being categorized as a "hardening fix" rather than a critical security vulnerability by OpenSSL, Okta strongly recommends immediate upgrades to distribution's OpenSSL packages.
