CVE-2026-55520

Protego: Exponential backtracking ReDoS in robots.txt URL wildcard matching

Published Aug 28, 2026 CVSS 4.0 · 7.1 High EPSS < 1% KEV · Not listed
Description
Protego is a pure-Python robots.txt parser with support for modern conventions. Prior to 0.6.2, protego._urlpattern._URLPattern._prepare_pattern_for_regex translates every asterisk in an Allow or Disallow directive into a lazy regular-expression wildcard, so a directive containing many asterisks creates exponential backtracking. After protego.Protego.parse processes a crafted robots.txt file, protego.Protego.can_fetch can spend an attacker-controlled period matching a near-miss URL and deny service to the crawler. The vulnerable path is src/protego/_urlpattern.py in the _URLPattern match logic. This issue is fixed in version 0.6.2.

Analysis

AI-generated

Technical impact

Protego, a Python library that parses robots.txt files, contains a regex construction bug. Each asterisk in an Allow or Disallow directive is turned into a lazy wildcard, causing exponential backtracking when many asterisks are present. A specially crafted robots.txt file can therefore make the parser consume excessive CPU and memory during the can_fetch check, leading to a denial‑of‑service for the crawler. This weakness falls under CWE‑1333 and CWE‑400.

Affected Systems

The flaw exists in all releases of Scrapy Protego prior to version 0.6.2. Vulnerable deployments include any system that imports protego._urlpattern._URLPattern and parses external robots.txt files. The patch was released in protego 0.6.2 and later versions.

Risk and Exploitability

The CVSS score of 7.1 indicates medium‑to‑high severity. Because the attack requires the crawler to download a malicious robots.txt file from a target domain, the vector is likely remote network (client‑side). The EPSS score is currently unavailable, and the vulnerability is not listed in the CISA KEV catalog. Nonetheless, a determined attacker can host a malicious robots.txt on their domain or compromise a site crawler visits, leading to resource exhaustion and potential service interruption.

Remediation

No solution or workaround provided in the CVE record.

OpenCVE Recommended Actions

  • Upgrade to protego 0.6.2 or later, which replaces the wildcard translation with a non‑exponential pattern.
  • If an immediate update is not feasible, temporarily disable robots.txt parsing or limit the number of asterisks processed by the crawler.
  • After applying the patch, monitor CPU and memory usage during crawler runs to confirm that the denial‑of‑service condition has been mitigated.

Generated by OpenCVE AI on August 28, 2026 at 20:51 UTC.

Tracking

Sign in to view the affected projects.

Advisories

Source ID Title
Github GHSA Github GHSA GHSA-wjmf-p669-5m5p Protego has exponential backtracking ReDoS in robots.txt URL wildcard matching
History
Type Values Removed Values Added
References
Metrics threat_severity

None

cvssV3_1

{'score': 6.5, 'vector': 'CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H'}

threat_severity

Moderate

Type Values Removed Values Added
Metrics ssvc

{'options': {'Automatable': 'yes', 'Exploitation': 'none', 'Technical Impact': 'partial'}, 'version': '2.0.3'}

Type Values Removed Values Added
First Time appeared Scrapy
Scrapy protego
Vendors & Products Scrapy
Scrapy protego
Type Values Removed Values Added
Description Protego is a pure-Python robots.txt parser with support for modern conventions. Prior to 0.6.2, protego._urlpattern._URLPattern._prepare_pattern_for_regex translates every asterisk in an Allow or Disallow directive into a lazy regular-expression wildcard, so a directive containing many asterisks creates exponential backtracking. After protego.Protego.parse processes a crafted robots.txt file, protego.Protego.can_fetch can spend an attacker-controlled period matching a near-miss URL and deny service to the crawler. The vulnerable path is src/protego/_urlpattern.py in the _URLPattern match logic. This issue is fixed in version 0.6.2.
Title Protego: Exponential backtracking ReDoS in robots.txt URL wildcard matching
Weaknesses CWE-1333
CWE-400
References
Metrics cvssV4_0

{'score': 7.1, 'vector': 'CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N'}

Weaknesses
  • CWE-1333

    Inefficient Regular Expression Complexity

  • CWE-400

    Uncontrolled Resource Consumption

Sources

Status
PUBLISHED
Assigner
GitHub_M
Published
Updated
Aug 31, 2026, 6:54 PM
Reserved
Jun 16, 2026, 10:44 PM
Status
Deferred
Published
Aug 28, 2026, 8:18 PM
Modified
Sep 9, 2026, 9:09 PM
Severity
Moderate
Public date
Aug 28, 2026, 6:42 PM