Sitemap Generator from Crawled URLs

Sitemap Generator from Crawled URLs

Problem Statement

Search engines rely on XML sitemaps to discover and index pages efficiently, but most sites have no reliable way to produce one — especially dynamic sites where URLs are generated at runtime and never appear in a static file. Manually maintaining sitemap.xml is error-prone and quickly goes stale as content is added, renamed, or removed. A better approach is to crawl the site from a known starting point, follow internal links, and emit the discovered URLs in the standard sitemaps.org format.

This script solves that problem by performing a breadth-first crawl of a single domain, respecting robots.txt, honoring a configurable page limit, and writing a valid <urlset> XML file. It's useful for SEO audits, static-site deployments, migration planning, and validating which pages are actually reachable from your homepage.

How It Works

The crawler uses a BFS queue seeded with the --start-url. For each URL it fetches the page with requests, parses the HTML with BeautifulSoup, extracts <a href> links, and normalizes them with urllib.parse.urljoin. Only URLs sharing the start URL's netloc (host) are enqueued, which keeps the crawl on-site. A visited set prevents infinite loops, and a robots.txt check via urllib.robotparser ensures polite crawling.

Design decisions worth noting: the crawler is single-threaded by default to avoid hammering servers (a --delay option adds politeness), the frontier is a collections.deque for O(1) popleft, and URL fragments (#section) are stripped before deduplication since they don't represent distinct pages. The output is streamed through xml.etree.ElementTree rather than string formatting, so all XML escaping is handled correctly. Errors during individual fetches are logged and skipped rather than aborting the whole run — a partial sitemap is more useful than none.

Full Script

#!/usr/bin/env python3
"""
Sitemap Generator from Crawled URLs

Crawls a website starting from a given URL, follows internal links,
and writes a sitemaps.org-compliant XML sitemap.

Usage:
    python3 sitemap_gen.py --start-url https://example.com --output sitemap.xml
"""

from __future__ import annotations

import argparse
import logging
import os
import sys
import time
import urllib.robotparser
from collections import deque
from datetime import date
from typing import Iterable
from urllib.parse import urljoin, urldefrag, urlparse

import requests
from bs4 import BeautifulSoup
from xml.etree import ElementTree as ET

# ---------------------------------------------------------------------------
# Configuration (overridable via environment variables)
# ---------------------------------------------------------------------------
DEFAULT_USER_AGENT = os.environ.get(
    "SITEMAP_USER_AGENT",
    "SitemapGenBot/1.0 (+https://example.com/bot)",
)
DEFAULT_TIMEOUT = float(os.environ.get("SITEMAP_TIMEOUT", "10"))
DEFAULT_DELAY = float(os.environ.get("SITEMAP_DELAY", "0.5"))
DEFAULT_MAX_PAGES = int(os.environ.get("SITEMAP_MAX_PAGES", "5000"))
DEFAULT_OUTPUT = os.environ.get("SITEMAP_OUTPUT", "sitemap.xml")

# Content types we're willing to parse for links.
HTML_CONTENT_TYPES = ("text/html", "application/xhtml+xml")

logging.basicConfig(
    level=os.environ.get("SITEMAP_LOG_LEVEL", "INFO").upper(),
    format="%(asctime)s [%(levelname)s] %(message)s",
)
log = logging.getLogger("sitemap_gen")


# ---------------------------------------------------------------------------
# Core helpers
# ---------------------------------------------------------------------------
def normalize(url: str) -> str:
    """Strip fragments and trailing whitespace; return canonical form."""
    url, _ = urldefrag(url.strip())
    return url


def same_host(a: str, b: str) -> bool:
    """Compare netloc (host[:port]) ignoring case."""
    return urlparse(a).netloc.lower() == urlparse(b).netloc.lower()


def load_robots(base_url: str, user_agent: str) -> urllib.robotparser.RobotFileParser:
    """Fetch and parse robots.txt; on failure, allow everything."""
    rp = urllib.robotparser.RobotFileParser()
    robots_url = urljoin(base_url, "/robots.txt")
    rp.set_url(robots_url)
    try:
        rp.read()
        log.info("Loaded robots.txt from %s", robots_url)
    except Exception as exc:  # noqa: BLE001 - any failure means "be permissive"
        log.warning("Could not load robots.txt (%s); proceeding without rules", exc)
        rp.allow_all = True  # type: ignore[attr-defined]
    return rp


def extract_links(html: str, page_url: str) -> Iterable[str]:
    """Yield absolute, normalized href values from a page's HTML."""
    soup = BeautifulSoup(html, "html.parser")
    for anchor in soup.find_all("a", href=True):
        href = anchor["href"].strip()
        # Skip non-navigational schemes and anchors.
        if not href or href.startswith(("mailto:", "tel:", "javascript:", "data:")):
            continue
        yield normalize(urljoin(page_url, href))


# ---------------------------------------------------------------------------
# Crawler
# ---------------------------------------------------------------------------
def crawl(
    start_url: str,
    max_pages: int,
    delay: float,
    timeout: float,
    user_agent: str,
) -> list[str]:
    """Breadth-first crawl returning a list of discovered same-host URLs."""
    start_url = normalize(start_url)
    session = requests.Session()
    session.headers.update({"User-Agent": user_agent, "Accept": "text/html,*/*"})

    robots = load_robots(start_url, user_agent)
    queue: deque[str] = deque([start_url])
    visited: set[str] = set()
    discovered: list[str] = []

    while queue and len(discovered) < max_pages:
        url = queue.popleft()
        if url in visited:
            continue
        visited.add(url)

        if not robots.can_fetch(user_agent, url):
            log.debug("Blocked by robots.txt: %s", url)
            continue

        try:
            resp = session.get(url, timeout=timeout, allow_redirects=True)
            resp.raise_for_status()
        except requests.RequestException as exc:
            log.warning("Fetch failed for %s: %s", url, exc)
            continue

        ctype = resp.headers.get("Content-Type", "").lower()
        if not any(ct in ctype for ct in HTML_CONTENT_TYPES):
            log.debug("Skipping non-HTML content at %s (%s)", url, ctype)
            continue

        # Record the final URL after redirects, not the pre-redirect one.
        final_url = normalize(resp.url)
        discovered.append(final_url)
        log.info("[%d/%d] %s", len(discovered), max_pages, final_url)

        for link in extract_links(resp.text, final_url):
            if not same_host(link, start_url):
                continue
            if link not in visited:
                queue.append(link)

        if delay > 0:
            time.sleep(delay)

    if queue:
        log.warning("Hit max-pages limit (%d); %d URLs left unvisited",
                    max_pages, len(queue))
    return discovered


# ---------------------------------------------------------------------------
# Sitemap writer
# ---------------------------------------------------------------------------
def write_sitemap(urls: list[str], output_path: str) -> None:
    """Write a sitemaps.org <urlset> XML file for the given URLs."""
    ET.register_namespace("", "http://www.sitemaps.org/schemas/sitemap/0.9")
    urlset = ET.Element(
        "{http://www.sitemaps.org/schemas/sitemap/0.9}urlset"
    )
    today = date.today().isoformat()
    for u in urls:
        url_el = ET.SubElement(
            urlset, "{http://www.sitemaps.org/schemas/sitemap/0.9}url"
        )
        ET.SubElement(
            url_el, "{http://www.sitemaps.org/schemas/sitemap/0.9}loc"
        ).text = u
        ET.SubElement(
            url_el, "{http://www.sitemaps.org/schemas/sitemap/0.9}lastmod"
        ).text = today

    tree = ET.ElementTree(urlset)
    ET.indent(tree, space="  ")  # Python 3.9+
    tree.write(output_path, encoding="utf-8", xml_declaration=True)
    log.info("Wrote %d URLs to %s", len(urls), output_path)


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Generate an XML sitemap by crawling a website.",
    )
    parser.add_argument("--start-url", required=True,
                        help="URL to begin crawling from")
    parser.add_argument("--output", default=DEFAULT_OUTPUT,
                        help=f"Output XML path (default: {DEFAULT_OUTPUT})")
    parser.add_argument("--max-pages", type=int, default=DEFAULT_MAX_PAGES,
                        help=f"Maximum pages to crawl (default: {DEFAULT_MAX_PAGES})")
    parser.add_argument("--delay", type=float, default=DEFAULT_DELAY,
                        help=f"Seconds between requests (default: {DEFAULT_DELAY})")
    parser.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT,
                        help=f"Per-request timeout in seconds (default: {DEFAULT_TIMEOUT})")
    parser.add_argument("--user-agent", default=DEFAULT_USER_AGENT,
                        help="User-Agent header to send")
    return parser.parse_args(argv)


def main(argv: list[str] | None = None) -> int:
    args = parse_args(argv)
    parsed = urlparse(args.start_url)
    if parsed.scheme not in ("http", "https") or not parsed.netloc:
        log.error("Invalid --start-url: %s", args.start_url)
        return 2

    try:
        urls = crawl(
            start_url=args.start_url,
            max_pages=args.max_pages,
            delay=args.delay,
            timeout=args.timeout,
            user_agent=args.user_agent,
        )
    except KeyboardInterrupt:
        log.warning("Interrupted by user")
        return 130

    if not urls:
        log.error("No pages discovered; not writing sitemap")
        return 1

    try:
        write_sitemap(urls, args.output)
    except OSError as exc:
        log.error("Failed to write sitemap: %s", exc)
        return 1

    return 0


if __name__ == "__main__":
    sys.exit(main())

Configuration

CLI flags override environment variables; environment variables override the hard-coded defaults. This lets you bake defaults into a container image while still allowing one-off overrides from the shell.

Usage Examples

1. Basic crawl of a small site:

$ python3 sitemap_gen.py --start-url https://example.com --output sitemap.xml
2024-05-12 10:14:02 [INFO] Loaded robots.txt from https://example.com/robots.txt
2024-05-12 10:14:02 [INFO] [1/5000] https://example.com/
2024-05-12 10:14:03 [INFO] [2/5000] https://example.com/about
2024-05-12 10:14:03 [INFO] [3/5000] https://example.com/blog
...
2024-05-12 10:15:41 [INFO] Wrote 47 URLs to sitemap.xml

2. Larger crawl with a stricter politeness delay and a cap:

$ SITEMAP_LOG_LEVEL=WARNING python3 sitemap_gen.py \
    --start-url https://docs.example.com \
    --max-pages 2000 \
    --delay 1.0 \
    --output docs-sitemap.xml

3. Debugging why a page was skipped:

$ SITEMAP_LOG_LEVEL=DEBUG python3 sitemap_gen.py \
    --start-url https://example.com --max-pages 20
2024-05-12 10:20:01 [DEBUG] Skipping non-HTML content at https://example.com/logo.png (image/png)
2024-05-12 10:20:01 [DEBUG] Blocked by robots.txt: https://example.com/admin

Error Handling

• robots.txt unreachable or malformed — logged as a warning and the crawl proceeds permissively (matching how most crawlers behave when robots.txt returns 404).
• Network failures per URL — requests.RequestException (timeouts, DNS errors, connection resets) is caught per-page; the URL is skipped and the crawl continues. A partial sitemap is still produced.
• Non-HTML responses — filtered by Content-Type before parsing, so PDFs, images, and JSON endpoints are excluded without raising a parser error.
• Redirects — allow_redirects=True follows them, and the final URL is what gets recorded, avoiding duplicate entries for redirect chains.
• Invalid --start-url — rejected early with exit code 2 before any network I/O.
• No pages discovered — returns exit code 1 without writing an empty sitemap (an empty <urlset> is technically valid but usually signals a bug).
• KeyboardInterrupt — caught, logged, and the process exits with code 130 so shell scripts can distinguish user cancellation from failure.
• Disk write failure — OSError is caught and reported; exit code 1.

Extending Ideas

• Concurrency with a bounded thread pool — swap the BFS loop for a concurrent.futures.ThreadPoolExecutor with a small worker count (e.g., 4–8) and a per-host semaphore. Keep the same visited set guarded by a threading.Lock, and always respect --delay per host to remain polite.
• Sitemap index splitting — the spec limits each sitemap to 50,000 URLs and 50 MB uncompressed. Chunk discovered into groups and emit a <sitemapindex> referencing individual sitemap-1.xml, sitemap-2.xml, etc.
• lastmod from Last-Modified headers — capture resp.headers.get("Last-Modified") per page and emit it as <lastmod> in W3C datetime format, which gives search engines a real freshness signal instead of today's date.
• Priority and changefreq heuristics — assign <priority> based on URL depth (homepage = 1.0, /blog/post = 0.6) and <changefreq> based on path patterns (/blog/* → weekly, /about → yearly). These hints are advisory but can influence crawl scheduling.
• Streaming output for huge sites — for crawls exceeding memory limits, write each <url> element to disk as it's discovered using ET.XMLPullParser or plain xml.sax.saxutils.escape, rather than accumulating the full list.