The vulnerability is an out‑of‑bounds buffer access in the BpeBuilder::build function of Hugging Face’s tokenizers library. When a tokenizer.json file is loaded, the builder allocates a scratch buffer sized to the longest vocabulary key and then copies each merge token into it. If a merge token is longer than that key, the copy overruns the buffer, triggering a Rust panic that aborts the process. The defect is a classic out‑of‑bounds buffer overwrite (CWE‑787) that can be exploited by supplying a crafted tokenizer.json and causes a denial of service without any additional encoding or elevated privileges. The secondary defect at the same location can create an unsigned integer underflow when the flag “continuing_subword_prefix” is set and a merge token is shorter than the prefix. This can cause a panic in debug builds or potentially corrupt memory in release builds, but the primary impact is the denial of service. The vulnerability is exploitable at load time; an attacker who can influence the tokenizer file loaded by an application can trigger the overflow. The likelihood of exploitation is unknown due to the lack of EPSS data, but the CVSS score of 7.1 indicates a medium to high severity. The vulnerability is not yet listed in the CISA KEV catalog.

Hugging Face tokenizers 0.23.1 is confirmed affected. Any installation around that version should be verified for presence of the defect. Users of the Hugging Face tokenizers library should track releases and apply updates from the official repository, ensuring the latest version is in use.

The CVSS score of 7.1 classifies this as a substantial vulnerability that can be disabled by an attacker who can provide a custom tokenizer.json. The EPSS score is not available, but the absence of an EPSS rating indicates no published data on current exploitation prevalence. The KEV status shows it is not yet in the CISA catalog. The attack vector is inferred to be local file tampering or injection of malicious configuration via a trusted application, as no remote exploitation mechanisms are described in the input. Consequently, the risk is primarily higher in environments where applications dynamically load tokenizer configurations from untrusted sources or offer file upload capabilities.