Okta Red Team has disclosed technical details of a newly named OpenSSL denial of service vulnerability called HollowByte, a flaw that can cause unpatched servers to reserve large amounts of memory using TLS requests as small as 11 bytes. According to the researchers, the issue was fixed by the OpenSSL project on June 9 through updated releases, but the security update was published without a CVE identifier, a dedicated security advisory, or release notes highlighting the vulnerability. The flaw affects OpenSSL releases prior to versions 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21. Because no CVE was assigned, organizations relying on automated vulnerability scanners or advisory feeds may not immediately identify systems requiring updates.
The vulnerability exists in the way older OpenSSL versions process TLS handshake messages. During the handshake, every message begins with a four byte header that specifies the expected size of the remaining data. Researchers explained that affected OpenSSL versions immediately allocate memory based on the size declared in that header before validating the message or receiving the actual payload. For an inbound ClientHello request, this allocation can reach approximately 131 KB per connection, even when the attacker never sends the remaining data. The worker thread then waits for a message body that never arrives. While this resembles traditional connection exhaustion attacks, Okta found that the behavior becomes significantly more serious on systems using the glibc memory allocator. Although OpenSSL releases the allocated memory after the connection closes, glibc retains many of those memory regions for future use instead of returning them to the operating system. By repeatedly opening connections with different declared message sizes, attackers can fragment the memory allocator, causing the server’s resident memory usage to continue increasing long after the attack has stopped. During testing with NGINX, Okta reported that a server with 1 GB of memory was terminated by the operating system after approximately 547 MB became trapped in fragmented memory, while a 16 GB server retained nearly one quarter of its memory without exceeding normal connection limits.
Despite releasing a fix, OpenSSL classified the issue as a bug or hardening improvement rather than a formal security vulnerability. As a result, HollowByte did not receive a CVE identifier, a vulnerability advisory, or an entry in the project’s official security documentation. This decision has raised questions because OpenSSL has previously assigned CVEs to other memory exhaustion issues that required more restrictive attack conditions. According to the published patch comments, the OpenSSL security team considered the allocation size relatively small for each connection, while Okta argued that the inability of memory to return to the operating system makes the issue significantly more impactful in real world deployments. Researchers have also noted that the same June 9 release addressed multiple documented vulnerabilities, including CVE 2026 34183 affecting QUIC memory usage and several additional CVEs, meaning many administrators may already have received the HollowByte fix without realizing it. Okta also stated that no public proof of concept exploit had been identified on GitHub as of July 18.
Security experts recommend upgrading OpenSSL installations to the latest supported versions and restarting all services that rely on the library to ensure the updated code is loaded into memory. Organizations using vendor supplied packages should also verify whether their operating system maintainers have backported the June 9 fixes because some Linux distributions retain older version numbers while applying security patches. In the absence of a CVE, administrators may need to review package changelogs or consult vendors directly to confirm whether the HollowByte patch has been included. Researchers further noted that the current fix only addresses TLS processing, while the equivalent DTLS code path remains unchanged. According to analysis of the updated source code, DTLS continues to allocate memory based on peer supplied message lengths, and OpenSSL has not publicly confirmed whether that behavior will be modified in a future release.
Follow the SPIN IDG WhatsApp Channel for updates across the Smart Pakistan Insights Network covering all of Pakistan’s technology ecosystem.