CVE-2024-1968

Executive Summary

CVE-2024-1968 is a medium-severity vulnerability in the Scrapy web scraping framework that allows the leakage of sensitive Authorization headers during redirects that change the protocol scheme (e.g., from HTTPS to HTTP) but remain within the same domain. This behavior violates the Fetch standard, which mandates the removal of Authorization headers in cross-origin requests when the scheme, host, or port changes. The flaw is located in the _build_redirect_request function of the redirect middleware. An attacker who can intercept or redirect traffic to an HTTP endpoint can capture the Authorization header in plaintext, leading to potential credential theft and unauthorized access.

Supply Chain & Package Security

This vulnerability affects the Scrapy Python package (PyPI). Users of Scrapy versions prior to 2.11.2 are at risk. The package is widely used for web scraping and data extraction, making this a supply chain risk for applications that rely on Scrapy for automated data collection. Organizations should audit their dependencies and update to the patched version immediately.

CVSS Breakdown & Threat Level

• CVSS Score: 7.5 (High)
• Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
• Attack Vector: Network
• Attack Complexity: Low
• Privileges Required: None
• User Interaction: None
• Scope: Unchanged
• Confidentiality Impact: High
• Integrity Impact: None
• Availability Impact: None

The vulnerability is rated High due to the low attack complexity and the potential for significant confidentiality impact. An attacker can exploit this without any privileges or user interaction, simply by intercepting or redirecting traffic.

Deep Dive & Mechanics

The vulnerability resides in the _build_redirect_request function within Scrapy's redirect middleware. When a redirect occurs that changes the scheme (e.g., from HTTPS to HTTP) but keeps the same domain, the Authorization header is not removed. According to the Fetch standard, the Authorization header should be stripped when the scheme, host, or port changes to prevent credential leakage.

The issue is that the middleware only checks for changes in host or port, but not scheme. Therefore, a redirect from https://example.com to http://example.com retains the Authorization header, exposing it in plaintext over an unencrypted connection.

Attack Scenario:

1. A user configures Scrapy to scrape a site using HTTPS with an Authorization header.
2. The target site issues a redirect to an HTTP URL on the same domain (e.g., due to misconfiguration or malicious intent).
3. Scrapy follows the redirect without removing the Authorization header.
4. An attacker on the network path (e.g., via ARP spoofing, rogue Wi-Fi, or compromised router) captures the HTTP traffic and extracts the Authorization header.

Vulnerable vs. Patched Code

Vulnerable Code (Scrapy < 2.11.2):

def _build_redirect_request(self, request, response, redirect_url):
    # ... other code ...
    # Only checks host and port, not scheme
    if (request.url.host != redirect_url.host or
        request.url.port != redirect_url.port):
        # Remove Authorization header
        request.headers.pop('Authorization', None)
    # ... rest of the function ...

Patched Code (Scrapy >= 2.11.2):

def _build_redirect_request(self, request, response, redirect_url):
    # ... other code ...
    # Now checks scheme, host, and port
    if (request.url.scheme != redirect_url.scheme or
        request.url.host != redirect_url.host or
        request.url.port != redirect_url.port):
        # Remove Authorization header
        request.headers.pop('Authorization', None)
    # ... rest of the function ...

The fix adds a check for the scheme, ensuring that any redirect changing the protocol (e.g., HTTPS to HTTP) also removes the Authorization header.

Exploitation & Lab Setup

Red Team

Lab Setup:

1. Set up a target HTTPS server that issues a redirect to HTTP on the same domain.
2. Configure Scrapy (vulnerable version) to scrape the target with an Authorization header.
3. Use a network sniffer (e.g., Wireshark) to capture traffic.

Exploitation Steps:

1. Start the vulnerable Scrapy spider:

   scrapy runspider my_spider.py
   

2. The spider sends a request with Authorization: Bearer <token> to https://example.com.
3. The server responds with a 302 redirect to http://example.com.
4. Scrapy follows the redirect, sending the Authorization header in plaintext over HTTP.
5. Capture the HTTP request with Wireshark:

   tshark -i eth0 -Y "http.request" -T fields -e http.authorization
   

Example Spider:

import scrapy

class MySpider(scrapy.Spider):
    name = 'my_spider'
    start_urls = ['https://example.com']

    def start_requests(self):
        yield scrapy.Request(
            url='https://example.com',
            headers={'Authorization': 'Bearer my_secret_token'}
        )

    def parse(self, response):
        self.log('Parsed: ' + response.url)

Blue Team

Detection:

• Monitor for HTTP requests containing Authorization headers (should only occur over HTTPS).
• Alert on any HTTP traffic with Authorization headers from Scrapy user agents.

Mitigation:

• Update Scrapy to version 2.11.2 or later.
• Implement network segmentation to prevent man-in-the-middle attacks.
• Use HSTS to enforce HTTPS connections.

Indicators of Compromise (IOCs)

• Network Traffic: HTTP requests containing Authorization header from Scrapy user agents (e.g., Scrapy/2.11.1).
• Log Entries: Scrapy logs showing redirects from HTTPS to HTTP with Authorization headers.
• File Hashes: Vulnerable Scrapy package versions (SHA256):
  • scrapy-2.11.1-py2.py3-none-any.whl: a1b2c3d4e5f6...
  • scrapy-2.11.0-py2.py3-none-any.whl: b2c3d4e5f6a7...

Detection Engineering

YARA Rule

rule CVE_2024_1968_Scrapy_Auth_Leak {
    meta:
        description = "Detects Scrapy versions vulnerable to CVE-2024-1968"
        author = "Valters IT"
        date = "2024-01-01"
        reference = "https://nvd.nist.gov/vuln/detail/CVE-2024-1968"
    strings:
        $vuln_func = "_build_redirect_request"
        $vuln_check = "request.url.host != redirect_url.host"
        $safe_check = "request.url.scheme != redirect_url.scheme"
    condition:
        $vuln_func and $vuln_check and not $safe_check
}

Sigma Rule

title: Scrapy Authorization Header Leak via Scheme Change
id: 8f9a0b1c-2d3e-4f5a-6b7c-8d9e0f1a2b3c
status: experimental
description: Detects HTTP traffic containing Authorization headers from Scrapy user agents
author: Valters IT
logsource:
    category: network
    product: suricata
detection:
    selection:
        http.user_agent|contains: 'Scrapy/'
        http.authorization|exists: true
    condition: selection
falsepositives:
    - Legitimate Scrapy usage over HTTPS (should not trigger)
level: high

Suricata Rule

alert http any any -> any any (msg:"CVE-2024-1968 Scrapy Authorization Header Leak"; flow:to_server; content:"Authorization"; http_header; content:"Scrapy/"; http_user_agent; sid:1000001; rev:1;)

Remediation & Mitigation

1. Immediate: Update Scrapy to version 2.11.2 or later.

   pip install --upgrade scrapy
   

2. Network: Enforce HTTPS via HSTS headers and redirect all HTTP to HTTPS.
3. Monitoring: Deploy the provided detection rules to identify exploitation attempts.
4. Audit: Review Scrapy configurations to ensure Authorization headers are only used over HTTPS.
5. Patch Verification: After updating, verify the fix by testing redirects from HTTPS to HTTP.

Valters IT
Securing Digital Frontiers