Markdown Table of Contents Generator

Markdown Table of Contents Generator

Problem Statement

Maintaining a table of contents (ToC) in long Markdown files like README.md is tedious and error-prone. Every time you add, rename, or reorder a section, you have to manually update the anchor links — and if you get a heading wrong, the links break silently. This is especially painful for open source projects where the README is the front door and stale anchors look unprofessional.

This script scans a Markdown file for ATX headings (#, ##, ###, etc.), generates a properly indented nested ToC with GitHub-compatible anchor slugs, and can either print the result or splice it back into the file between marker comments. It requires only the Python standard library, so it drops cleanly into any CI pipeline or pre-commit hook.

How It Works

The script is a small pipeline: read → parse → render → emit. The reader loads the file (or stdin) and normalizes line endings. The parser walks lines, skips fenced code blocks (so # inside a code sample isn't mistaken for a heading), and extracts (level, text) pairs from ATX headings. The renderer converts each heading into a GitHub-style anchor slug — lowercasing, stripping punctuation, replacing spaces with hyphens, and de-duplicating collisions by appending -1, -2, etc. — then emits a nested bullet list.

Two design decisions matter. First, fenced code block detection uses a simple toggle on triple-backtick lines, which is enough for real-world Markdown without pulling in a full parser. Second, in-place updates are guarded by HTML comment markers (<!-- toc --> / <!-- tocstop -->) so the script never clobbers user content — if the markers are missing, it refuses to modify the file and exits with a clear error. This makes the operation idempotent and safe to run repeatedly in CI.

Full Script

#!/usr/bin/env python3
"""
Markdown Table of Contents Generator.

Scans a Markdown file for ATX headings and either prints a nested
table of contents or splices it into the file between marker comments.

Usage:
    python3 toc_gen.py --file README.md
    python3 toc_gen.py --file README.md --update-inplace
    cat README.md | python3 toc_gen.py
"""

from __future__ import annotations

import argparse
import os
import re
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable

# ---------------------------------------------------------------------------
# Configuration (overridable via environment variables)
# ---------------------------------------------------------------------------
DEFAULT_START_MARKER = os.environ.get("TOC_START_MARKER", "<!-- toc -->")
DEFAULT_STOP_MARKER = os.environ.get("TOC_STOP_MARKER", "<!-- tocstop -->")
DEFAULT_MAX_LEVEL = int(os.environ.get("TOC_MAX_LEVEL", "4"))
DEFAULT_INDENT = os.environ.get("TOC_INDENT", "  ")

# ATX heading: 1-6 '#' followed by a space and text (trailing #'s stripped).
HEADING_RE = re.compile(r"^(#{1,6})\s+(.*?)\s*#*\s*$")
# Characters GitHub strips when generating an anchor from heading text.
SLUG_STRIP_RE = re.compile(r"[^\w\s-]", flags=re.UNICODE)
FENCE_RE = re.compile(r"^\s*(```|~~~)")


@dataclass(frozen=True)
class Heading:
    """A single Markdown heading with its level and raw text."""
    level: int
    text: str


def slugify(text: str) -> str:
    """Convert heading text to a GitHub-compatible anchor slug."""
    slug = text.strip().lower()
    slug = SLUG_STRIP_RE.sub("", slug)   # drop punctuation
    slug = slug.replace(" ", "-")        # spaces -> hyphens
    return slug


def parse_headings(lines: Iterable[str], max_level: int) -> list[Heading]:
    """Extract ATX headings, skipping anything inside fenced code blocks."""
    headings: list[Heading] = []
    in_fence = False
    for raw in lines:
        line = raw.rstrip("\n")
        if FENCE_RE.match(line):
            in_fence = not in_fence
            continue
        if in_fence:
            continue
        match = HEADING_RE.match(line)
        if not match:
            continue
        level = len(match.group(1))
        if level > max_level:
            continue
        headings.append(Heading(level=level, text=match.group(2).strip()))
    return headings


def build_toc(headings: list[Heading], indent: str) -> str:
    """Render headings as a nested Markdown bullet list with anchor links."""
    if not headings:
        return ""

    # Anchor de-duplication: GitHub appends -1, -2 on collisions.
    seen: dict[str, int] = {}
    lines: list[str] = []
    base_level = min(h.level for h in headings)

    for h in headings:
        slug = slugify(h.text)
        if slug in seen:
            seen[slug] += 1
            slug = f"{slug}-{seen[slug]}"
        else:
            seen[slug] = 0
        depth = max(0, h.level - base_level)
        lines.append(f"{indent * depth}- [{h.text}](#{slug})")
    return "\n".join(lines) + "\n"


def update_in_place(text: str, toc: str, start: str, stop: str) -> str:
    """Replace content between start/stop markers with the new ToC."""
    start_idx = text.find(start)
    stop_idx = text.find(stop)
    if start_idx == -1 or stop_idx == -1:
        raise ValueError(
            f"Markers not found. Add '{start}' and '{stop}' to the file."
        )
    if stop_idx < start_idx:
        raise ValueError("Stop marker appears before start marker.")
    before = text[: start_idx + len(start)]
    after = text[stop_idx:]
    return f"{before}\n{toc}{after}"


def read_source(path: Path | None) -> str:
    """Read Markdown from a file, or stdin when path is None."""
    if path is None:
        return sys.stdin.read()
    try:
        return path.read_text(encoding="utf-8")
    except FileNotFoundError:
        raise SystemExit(f"error: file not found: {path}")
    except OSError as exc:
        raise SystemExit(f"error: cannot read {path}: {exc}")


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description="Generate a Markdown ToC.")
    parser.add_argument("--file", "-f", type=Path, default=None,
                        help="Markdown file (defaults to stdin).")
    parser.add_argument("--update-inplace", action="store_true",
                        help="Splice ToC between marker comments in --file.")
    parser.add_argument("--max-level", type=int, default=DEFAULT_MAX_LEVEL,
                        help=f"Deepest heading level to include (default {DEFAULT_MAX_LEVEL}).")
    parser.add_argument("--indent", default=DEFAULT_INDENT,
                        help="Indent string per nesting level.")
    parser.add_argument("--start-marker", default=DEFAULT_START_MARKER)
    parser.add_argument("--stop-marker", default=DEFAULT_STOP_MARKER)
    args = parser.parse_args(argv)

    if args.update_inplace and args.file is None:
        parser.error("--update-inplace requires --file")

    text = read_source(args.file)
    headings = parse_headings(text.splitlines(), args.max_level)
    toc = build_toc(headings, args.indent)

    if args.update_inplace:
        assert args.file is not None
        try:
            new_text = update_in_place(text, toc, args.start_marker, args.stop_marker)
        except ValueError as exc:
            raise SystemExit(f"error: {exc}")
        args.file.write_text(new_text, encoding="utf-8")
        print(f"Updated ToC in {args.file} ({len(headings)} headings).")
    else:
        sys.stdout.write(toc)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Configuration

All knobs have sensible defaults; override them via flags or environment variables.

The --update-inplace flag is the only mode that mutates a file. Without it, output goes to stdout, which is safe for piping and diffing.

Usage Examples

1. Print the ToC for a README to stdout:

$ python3 toc_gen.py --file README.md
- [Installation](#installation)
- [Usage](#usage)
  - [CLI Flags](#cli-flags)
  - [Environment Variables](#environment-variables)
- [Contributing](#contributing)

2. Splice the ToC back into the file between markers:

$ python3 toc_gen.py --file README.md --update-inplace
Updated ToC in README.md (6 headings).

The file must contain:

<!-- toc -->
<!-- tocstop -->

Everything between those lines is replaced.

3. Read from stdin and limit depth to level 2:

$ cat docs/guide.md | python3 toc_gen.py --max-level 2
- [Getting Started](#getting-started)
- [Advanced Topics](#advanced-topics)

Error Handling

• Missing file → SystemExit("error: file not found: ...") with a non-zero exit code.
• Unreadable file (permissions, encoding) → caught OSError, clean message, exit code 1.
• Markers missing during --update-inplace → raises ValueError, converted to a friendly SystemExit. The file is left untouched.
• Stop marker before start marker → explicit ValueError rather than silent corruption.
• --update-inplace without --file → argparse rejects the invocation before any I/O.
• Empty document → build_toc returns an empty string; in-place mode simply clears the ToC region.
• Duplicate heading text → anchors are de-duplicated with -1, -2 suffixes, matching GitHub's behavior.

Extending Ideas

• Add a pre-commit hook: wrap the script in .pre-commit-config.yaml with --update-inplace so the ToC stays fresh automatically on every commit.
• Support Setext headings: extend parse_headings to recognize ===/--- underlines by looking at the previous non-blank line.
• Emit nested ordered lists: swap the - bullet for 1. and track numbering per level for docs that prefer numbered sections.
• Add --check mode: exit non-zero if the file's ToC differs from generated output — perfect for CI gates that fail on stale docs without modifying them.