Description
In Bouncy Castle for Java before 1.86, the MLS implementation (org.bouncycastle.mls) holds RFC 9420's uint32 leaf_index in a signed int, so a wire value with the top bit set decodes to a negative number. That is a legitimate encoding rather than malformed input, and it must still decode, since the MLS interop test vectors round-trip the full range. GroupKeySet.SecretTree.hasLeaf and Group.validateRemove compared the decoded value directly against the tree's leaf count, and a signed comparison treats any negative int as less than a positive bound, so an out-of-range sender passed the membership check. In the hasLeaf case the SenderData of an unprotected PrivateMessage could then drive LeafIndex.directPath through NodeIndex.parent() arithmetic that never reaches the tree root, growing the resulting node list without bound until the JVM exhausted its heap. A single small message from any current group member could therefore deny service to every other member of the group. Both comparisons now interpret the value as unsigned via Integer.toUnsignedLong, rejecting an out-of-range sender however it was encoded; well-formed leaf indices are unaffected.
Published: 2026-10-02
Score: 8.7 High
EPSS: n/a
KEV: No
Impact: Denial of Service
Action: Patch Immediately
AI Analysis

Impact

In Bouncy Castle’s MLS implementation the IEEE‑754 conversion of a uint32 leaf_index into a signed integer allows a full 32‑bit range to be interpreted as a negative value. When the leaf_index exceeds the current group size, the negative comparison passes a membership test, permitting a malicious sender to construct an unprotected PrivateMessage that forces the tree construction algorithm to grow an unbounded node list until the JVM exhausts its heap. This results in a denial of service for all other group members. The flaw resides in CWE‑195, an improper signed‑to‑unsigned conversion.

Affected Systems

Legion of the Bouncy Castle Inc. provides the BC-JAVA library. Versions prior to 1.86 of the org.bouncycastle.mls package are vulnerable. The fix is included in release 1.86 onward, where leaf index comparisons use unsigned interpretation.

Risk and Exploitability

The CVSS score is 8.7, indicating a high‑severity vulnerability. EPSS is not available, and the issue is not listed in the CISA KEV catalog. The vulnerability is exploitable in any context where an attacker can send an MLS message to a group that uses an affected library instance. An attacker can craft a message with a leaf_index that has its most significant bit set. As a result, the victim will experience a denial of service. The attack vector is an in‑group message injection, requiring no external code execution.

Generated by OpenCVE AI on October 2, 2026 at 08:40 UTC.

Remediation

No solution or workaround provided in the CVE record.

OpenCVE Recommended Actions

  • Upgrade the BC-JAVA library to version 1.86 or newer, which performs unsigned comparisons for leaf_index and rejects out‑of‑range values.
  • If an upgrade is not feasible, modify the MLS implementation to validate that any received leaf_index is less than or equal to the current group size before decoding and processing the message.
  • After applying the fix, monitor JVM heap usage and incoming MLS messages to ensure that the denial‑of‑service condition no longer occurs, and run the full MLS interop test suite to confirm proper operation.

Generated by OpenCVE AI on October 2, 2026 at 08:40 UTC.

Tracking

Sign in to view the affected projects.

Advisories

No advisories yet.

History

Fri, 02 Oct 2026 09:30:00 +0000

Type Values Removed Values Added
First Time appeared Legion Of The Bouncy Castle Inc.
Legion Of The Bouncy Castle Inc. bc-java
Vendors & Products Legion Of The Bouncy Castle Inc.
Legion Of The Bouncy Castle Inc. bc-java

Fri, 02 Oct 2026 07:30:00 +0000

Type Values Removed Values Added
Description In Bouncy Castle for Java before 1.86, the MLS implementation (org.bouncycastle.mls) holds RFC 9420's uint32 leaf_index in a signed int, so a wire value with the top bit set decodes to a negative number. That is a legitimate encoding rather than malformed input, and it must still decode, since the MLS interop test vectors round-trip the full range. GroupKeySet.SecretTree.hasLeaf and Group.validateRemove compared the decoded value directly against the tree's leaf count, and a signed comparison treats any negative int as less than a positive bound, so an out-of-range sender passed the membership check. In the hasLeaf case the SenderData of an unprotected PrivateMessage could then drive LeafIndex.directPath through NodeIndex.parent() arithmetic that never reaches the tree root, growing the resulting node list without bound until the JVM exhausted its heap. A single small message from any current group member could therefore deny service to every other member of the group. Both comparisons now interpret the value as unsigned via Integer.toUnsignedLong, rejecting an out-of-range sender however it was encoded; well-formed leaf indices are unaffected.
Title MLS membership checks compare a uint32 leaf_index as signed, admitting an out-of-range sender
Weaknesses CWE-195
References
Metrics cvssV4_0

{'score': 8.7, 'vector': 'CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N/U:Amber'}

Subscriptions

Legion Of The Bouncy Castle Inc. Bc-java
cve-icon MITRE

Status: PUBLISHED

Assigner: bcorg

Published:

Updated: 2026-10-02T07:26:34.739Z

Reserved: 2026-07-26T22:40:20.567Z

Link: CVE-2026-17507

cve-icon Vulnrichment

No data.

cve-icon NVD

Status : Received

Published: 2026-10-02T08:17:01.310

Modified: 2026-10-02T08:17:01.310

Link: CVE-2026-17507

cve-icon Redhat

No data.

cve-icon OpenCVE Enrichment

Updated: 2026-10-02T09:15:08Z

Weaknesses
  • CWE-195

    Signed to Unsigned Conversion Error