Protego, a Python library that parses robots.txt files, contains a regex construction bug. Each asterisk in an Allow or Disallow directive is turned into a lazy wildcard, causing exponential backtracking when many asterisks are present. A specially crafted robots.txt file can therefore make the parser consume excessive CPU and memory during the can_fetch check, leading to a denial‑of‑service for the crawler. This weakness falls under CWE‑1333 and CWE‑400.

The flaw exists in all releases of Scrapy Protego prior to version 0.6.2. Vulnerable deployments include any system that imports protego._urlpattern._URLPattern and parses external robots.txt files. The patch was released in protego 0.6.2 and later versions.

The CVSS score of 7.1 indicates medium‑to‑high severity. Because the attack requires the crawler to download a malicious robots.txt file from a target domain, the vector is likely remote network (client‑side). The EPSS score is currently unavailable, and the vulnerability is not listed in the CISA KEV catalog. Nonetheless, a determined attacker can host a malicious robots.txt on their domain or compromise a site crawler visits, leading to resource exhaustion and potential service interruption.