Scrapy is a high-level web crawling and scraping framework for Python. If you use `HttpAuthMiddleware` (i.e. the `http_user` and `http_pass` spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as `robots.txt` requests sent by Scrapy when the `ROBOTSTXT_OBEY` setting is set to `True`, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new `http_auth_domain` spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the `w3lib.http.basic_auth_header` function to convert your credentials into a value that you can assign to the `Authorization` header of your request, instead of defining your credentials globally using `HttpAuthMiddleware`.
Metrics
Affected Vendors & Products
Advisories
| Source | ID | Title |
|---|---|---|
Debian DLA |
DLA-2950-1 | python-scrapy security update |
EUVD |
EUVD-2021-0239 | Scrapy is a high-level web crawling and scraping framework for Python. If you use `HttpAuthMiddleware` (i.e. the `http_user` and `http_pass` spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as `robots.txt` requests sent by Scrapy when the `ROBOTSTXT_OBEY` setting is set to `True`, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new `http_auth_domain` spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the `w3lib.http.basic_auth_header` function to convert your credentials into a value that you can assign to the `Authorization` header of your request, instead of defining your credentials globally using `HttpAuthMiddleware`. |
Github GHSA |
GHSA-jwqp-28gf-p498 | Scrapy HTTP authentication credentials potentially leaked to target websites |
Ubuntu USN |
USN-7476-1 | Scrapy vulnerabilities |
Fixes
Solution
No solution given by the vendor.
Workaround
No workaround given by the vendor.
References
History
Wed, 16 Jul 2025 13:45:00 +0000
| Type | Values Removed | Values Added |
|---|---|---|
| Metrics |
epss
|
epss
|
Status: PUBLISHED
Assigner: GitHub_M
Published:
Updated: 2024-08-04T02:59:31.432Z
Reserved: 2021-09-15T00:00:00
Link: CVE-2021-41125
No data.
Status : Modified
Published: 2021-10-06T18:15:10.953
Modified: 2024-11-21T06:25:31.600
Link: CVE-2021-41125
No data.
OpenCVE Enrichment
No data.
Debian DLA
EUVD
Github GHSA
Ubuntu USN