CVE-2026-100661

Executive Summary

CVE-2026-100661 is a denial-of-service vulnerability in Netty's HTTP/3 codec (io.netty:netty-codec-http3) affecting versions 4.2.0.Final through 4.2.17.Final. The flaw resides in the QPACK prefixed-integer decoder, which fails to bound the number of continuation bytes processed, allowing a remote unauthenticated attacker to exhaust server memory and CPU. With a CVSS score of 7.5 and no public PoC or exploit code, this is a high-impact availability threat to any internet-facing service built on Netty's HTTP/3 stack.

Technical Analysis

The vulnerability is a resource exhaustion issue (CWE-400) in QPACK — the header compression format used by HTTP/3 (RFC 9204). QPACK encodes integers using a variable-length scheme where a prefix of N bits holds an initial value and subsequent bytes use the high bit as a continuation flag. When the high bit is set, another byte follows; the value is multiplied by 128 with each continuation.

In Netty's QpackUtil.decodePrefixedInteger implementation, the decoder does not cap the number of continuation bytes it will consume. When it encounters a run of bytes with the continuation bit set (0x80), it keeps waiting for the terminating byte that never arrives. Critically, the function returns -1 to signal "need more bytes." The calling code interprets this as incomplete input and therefore does not advance the read index in the ByteToMessageDecoder cumulator. Because the input is never consumed, the cumulator's internal buffer grows unbounded, and every subsequent decode() invocation re-scans the entire accumulated buffer. This produces:

1. Unbounded per-connection heap growth — the cumulator accumulates every continuation byte, eventually triggering OutOfMemoryError.
2. O(N²) CPU cost — each decode() re-scans the whole buffer, so processing cost grows quadratically with the number of bytes received, starving the event loop.

The attack requires only a remote, unauthenticated peer capable of opening a QPACK unidirectional stream. QPACK defines two unidirectional stream types relevant here: type 0x02 (encoder stream) and type 0x03 (decoder stream). Either can be used to deliver the malicious payload. The trigger is a first byte with all prefix bits set — 0xFF for a 7-bit prefix or 0x3F for a 5-bit prefix — followed by an endless run of 0x80 continuation bytes. Because the decoder never returns a valid value, callers never consume the input.

The vulnerability is reachable "in every configuration," meaning no specific Netty settings or HTTP/3 tuning is required to expose it beyond enabling the HTTP/3 codec. This makes the attack surface equivalent to the entire HTTP/3 listener footprint.

Exploitation Scenarios

No public PoC or exploit code is available for CVE-2026-100661; the attack chain below is inferred from the vulnerability class (unbounded resource consumption) and the NVD description.

1. Reconnaissance / target identification. An attacker identifies internet-facing services negotiating HTTP/3. This is done passively by inspecting Alt-Svc response headers on HTTP/2 or HTTP/1.1 connections, or by probing common HTTP/3 ports (UDP 443). Any service exposing Netty's HTTP/3 codec in the affected version range is a candidate. The attacker does not need credentials or prior access.

2. Triggering the flaw. The attacker establishes an HTTP/3 connection and opens a QPACK unidirectional stream — either encoder (type 0x02) or decoder (type 0x03). On that stream, the attacker sends a first byte with all prefix bits set (0xFF for a 7-bit prefix, 0x3F for a 5-bit prefix). This signals to QpackUtil.decodePrefixedInteger that the encoded integer is maximal and more continuation bytes should follow.

3. Payload / execution method (vulnerability-class level). Immediately after the prefix byte, the attacker transmits an endless run of 0x80 bytes (continuation bytes with the high bit set). Each byte keeps the decoder in the "need more bytes" state, causing it to return -1. Because the return value indicates incomplete input, the caller never marks the bytes as consumed. The ByteToMessageDecoder cumulator buffer therefore retains all received bytes and continues to grow. The attacker can sustain this by keeping the connection open and feeding bytes continuously, or by rapidly opening many such connections.

4. Post-exploitation impact. Two compounding effects degrade the target: (a) the cumulator buffer expands until the JVM exhausts heap and throws OutOfMemoryError, and (b) every decode() call re-scans the entire accumulated buffer, producing O(N²) CPU cost that saturates the event-loop thread. The result is service unavailability — dropped connections, unresponsive HTTP/3 endpoints, and potential cascading failures for other services colocated on the same JVM. There is no confidentiality or integrity impact (C:N/I:N per the CVSS vector); the objective is purely denial of service.

Impact Assessment

CVE-2026-100661 carries a CVSS v3.1 base score of 7.5 (HIGH) with the vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H. The subscores indicate:

• Attack Vector (N): Exploitable remotely over the network.
• Attack Complexity (L): No special conditions; the flaw is reachable in every configuration.
• Privileges Required (N): No authentication needed.
• User Interaction (N): No victim action required.
• Scope (U): Impact is confined to the vulnerable component.
• Confidentiality (N) / Integrity (N): No data exposure or tampering.
• Availability (H): Total loss of availability is achievable.

Blast radius. Any application, gateway, proxy, or framework embedding Netty's netty-codec-http3 version 4.2.0.Final–4.2.17.Final is affected. Because Netty is a foundational networking library, the affected surface includes HTTP/3 endpoints in reactive frameworks, API gateways, service meshes, and custom Java services. A single unauthenticated attacker can consume per-connection heap proportional to the bytes they send; a single sustained connection may be sufficient to trigger OOM on memory-constrained instances, while many concurrent connections from one source amplify the effect. The O(N²) CPU cost means even modest byte volumes can starve the event loop and deny service to legitimate users. There is no known active exploitation, CISA KEV listing, or EPSS score at the time of writing.

Detection Guidance

Sigma Rule

title: Potential Netty HTTP/3 QPACK Prefixed-Integer DoS (CVE-2026-100661)
id: 9f2c4e1a-7b3d-4c8e-a1f5-2d6e8b0c9a41
status: experimental
description: Detects JVM OutOfMemoryError or repeated re-scan behavior associated with HTTP/3 QPACK stream abuse in Netty 4.2.0.Final-4.2.17.Final.
references:
  - https://nvd.nist.gov/vuln/detail/CVE-2026-100661
author: valtersit.com
date: 2026-01-01
logsource:
  category: application
  product: java
detection:
  keywords:
    - 'java.lang.OutOfMemoryError'
    - 'io.netty.handler.codec.http3'
    - 'QpackUtil'
    - 'decodePrefixedInteger'
  condition: selection
  selection:
    - 'QpackUtil'
    - 'decodePrefixedInteger'
    - 'Http3UnidirectionalStreamInboundHandler'
fields:
  - message
  - host
level: high
tags:
  - attack.t1499
  - cve.2026.100661

YARA Rule

rule CVE_2026_100661_Netty_QPACK_DoS
{
    meta:
        description = "Detects references to vulnerable Netty QPACK prefixed-integer decoder"
        author = "valtersit.com"
        date = "2026-01-01"
        reference = "https://nvd.nist.gov/vuln/detail/CVE-2026-100661"
        cve = "CVE-2026-100661"

    strings:
        $codec = "io.netty.handler.codec.http3" ascii wide
        $qp   = "QpackUtil" ascii wide
        $fn   = "decodePrefixedInteger" ascii wide
        $name = "netty-codec-http3" ascii wide

    condition:
        (uint16(0) == 0xCAFE or uint32(0) == 0x504B0304 or true) and
        (($qp and $fn) or ($codec and $name))
}

Suricata IDS Rule

alert udp any any -> any 443 (msg:"CVE-2026-100661 Netty HTTP/3 QPACK Prefixed-Integer DoS Attempt"; flow:to_server; content:"|FF|"; depth:0; content:"|80 80 80 80 80 80 80 80|"; within:64; threshold: type both, track by_src, count 5, seconds 10; classtype:attempted-dos; reference:cve,2026-100661; sid:2026100661; rev:1;)

Elastic Detection

process where process.name == "java"
  and message like~ "*OutOfMemoryError*"
  and (
    message like~ "*QpackUtil*" or
    message like~ "*decodePrefixedInteger*" or
    message like~ "*htt3*"
  )

Splunk SPL Query

index=app sourcetype=java_container
| search "QpackUtil.decodePrefixedInteger" OR ("decodePrefixedInteger" AND "Need more bytes") OR "java.lang.OutOfMemoryError: Java heap space"
| stats count, values(message) as samples by host, source
| where count > 10
| sort - count

Wazuh OSSEC Rule

<group name="netty,http3,cve-2026-100661,">
  <rule id="100661" level="12">
    <decoded_as>json</decoded_as>
    <field name="message">QpackUtil</field>
    <field name="message">decodePrefixedInteger</field>
    <description>CVE-2026-100661: Netty HTTP/3 QPACK prefixed-integer decoder DoS activity detected.</description>
    <mitre>
      <id>T1499</id>
    </mitre>
  </rule>
  <rule id="100662" level="14">
    <decoded_as>json</decoded_as>
    <field name="message">java.lang.OutOfMemoryError</field>
    <field name="message">netty</field>
    <description>CVE-2026-100661: Netty HTTP/3 heap exhaustion consistent with QPACK DoS.</description>
    <mitre>
      <id>T1499</id>
    </mitre>
  </rule>
</group>

Mitigation & Patch Guidance

Patch confidence: 0/4 (UNPATCHED — no patch evidence from any tracked source).

1) Immediate mitigations. Because the flaw is reachable in every configuration that enables the HTTP/3 codec, the most effective short-term control is to disable HTTP/3/QUIC listeners on any Netty-based service that does not strictly require them, falling back to HTTP/2 over TCP. If HTTP/3 must remain enabled, enforce strict connection and resource limits at the network edge: rate-limit inbound QPACK stream creation per source IP, cap per-connection buffered bytes where the surrounding stack allows it, and set conservative JVM heap limits with aggressive -XX:+ExitOnOutOfMemoryError restart policies. Deploy WAF or L4 filtering to drop UDP/443 traffic from untrusted, high-volume sources.

2) Patch/upgrade steps. Upgrade io.netty:netty-codec-http3 to 4.2.18.Final, which contains the fix bounding the number of continuation bytes processed in the prefixed-integer decoder. The absence of patch evidence in tracked sources at the time of writing means remediation must be verified directly against the upstream Netty 4.2.18.Final release and its changelog. After upgrading, confirm the deployed artifact version via dependency tree analysis (mvn dependency:tree / Gradle dependency report) to ensure no transitive dependency pins an older netty-codec-http3. Rebuild and redeploy all services that bundle Netty.

3) Long-term hardening. Maintain an inventory of HTTP/3-capable components and their Netty versions. Implement automated dependency scanning to catch future codec regressions. Harden JVM deployments with heap caps, golden-signal monitoring (heap utilization, GC pause, event-loop latency), and alerting on OutOfMemoryError events. Where feasible, terminate QUIC/HTTP-3 at a hardened edge proxy rather than in-process, so pathological streams are dropped before reaching application code. Establish a standing practice of tracking Netty security advisories given its pervasive role in JVM networking stacks.

References

• NVD — CVE-2026-100661: https://nvd.nist.gov/vuln/detail/CVE-2026-100661
• Netty project (official repository and release notes for 4.2.18.Final): https://github.com/netty/netty
• RFC 9204 — QPACK: Field Compression for HTTP/3: https://www.rfc-editor.org/rfc/rfc9204
• CWE-400 — Uncontrolled Resource Consumption: https://cwe.mitre.org/data/definitions/400.html
• CISA Known Exploited Vulnerabilities Catalog (no entry as of writing): https://www.cisa.gov/known-exploited-vulnerabilities-catalog

No threat actor campaigns or exploit-in-the-wild evidence are associated with CVE-2026-100661 at the time of writing. No public PoC, Nuclei template, or exploit code has been observed.