PDF Text Extractor with pdfplumber

PDF Text Extractor with pdfplumber

Problem Statement

Extracting text from PDFs is a recurring task for data engineers, sysadmins, and anyone automating document processing. Whether you're ingesting invoices, scraping reports for a search index, or converting legacy documents into plain text for downstream NLP pipelines, you need a reliable way to pull clean text out of PDFs without opening a GUI.

PDFs are notoriously messy: they can contain scanned images (no text layer), multi-column layouts, headers/footers, or encrypted content. pdfplumber handles most text-based PDFs gracefully and exposes per-page extraction, which makes it ideal for batch processing. This script wraps pdfplumber in a CLI that accepts an input PDF, an output text file, and an optional page range — making it trivially scriptable from cron jobs, Airflow DAGs, or shell pipelines.

How It Works

The script follows a simple linear pipeline: parse arguments → open PDF → iterate pages → extract text → write output. It uses argparse for CLI parsing, pdfplumber.open() as a context manager to guarantee file handles are released, and a small page-range parser that converts strings like "1-10" or "3,5,7" into a set of zero-indexed page numbers.

Key design decisions:

• Streaming writes: output is written incrementally per page rather than buffering the entire document in memory, which matters for large PDFs.
• Page range as a filter: the --pages flag supports single pages, ranges, and comma-separated combinations (1-3,7,10-12).
• Configurable page separator: some downstream tools want a form-feed (\f) between pages; others want a blank line. Both are supported.
• Explicit error hierarchy: distinct exceptions for missing files, invalid ranges, and encrypted PDFs, so callers can react appropriately.

Full Script

#!/usr/bin/env python3
"""
PDF Text Extractor with pdfplumber.

Extracts text from a PDF file and writes it to a plain-text output file.
Supports page-range selection, configurable page separators, and
graceful handling of encrypted or image-only PDFs.

Usage:
    python3 pdf_extract.py --input report.pdf --output text.txt --pages 1-10
"""

from __future__ import annotations

import argparse
import logging
import os
import sys
from pathlib import Path
from typing import Iterable

import pdfplumber

# ---------------------------------------------------------------------------
# Configuration (overridable via environment variables)
# ---------------------------------------------------------------------------
DEFAULT_SEPARATOR = os.environ.get("PDF_EXTRACT_SEPARATOR", "\n\n")
LOG_LEVEL = os.environ.get("PDF_EXTRACT_LOG_LEVEL", "INFO").upper()
STRIP_LINES = os.environ.get("PDF_EXTRACT_STRIP_LINES", "true").lower() == "true"

logger = logging.getLogger("pdf_extract")


# ---------------------------------------------------------------------------
# Page-range parsing
# ---------------------------------------------------------------------------
def parse_page_range(spec: str | None, total_pages: int) -> list[int]:
    """
    Convert a page-range string like '1-3,7,10-12' into a sorted list of
    zero-indexed page numbers.

    Returns all pages if spec is None or empty.
    Raises ValueError on malformed input or out-of-bounds pages.
    """
    if not spec or not spec.strip():
        return list(range(total_pages))

    pages: set[int] = set()
    for chunk in spec.split(","):
        chunk = chunk.strip()
        if not chunk:
            continue
        if "-" in chunk:
            start_s, _, end_s = chunk.partition("-")
            try:
                start, end = int(start_s), int(end_s)
            except ValueError as exc:
                raise ValueError(f"Invalid range segment: {chunk!r}") from exc
            if start < 1 or end < start:
                raise ValueError(f"Invalid range bounds: {chunk!r}")
            # PDF pages are 1-indexed to the user; convert to 0-indexed.
            pages.update(range(start - 1, min(end, total_pages)))
        else:
            try:
                page = int(chunk)
            except ValueError as exc:
                raise ValueError(f"Invalid page number: {chunk!r}") from exc
            if page < 1 or page > total_pages:
                raise ValueError(
                    f"Page {page} out of range (document has {total_pages} pages)"
                )
            pages.add(page - 1)

    if not pages:
        raise ValueError("No valid pages selected")
    return sorted(pages)


# ---------------------------------------------------------------------------
# Extraction
# ---------------------------------------------------------------------------
def extract_text(
    pdf_path: Path,
    page_indices: Iterable[int],
    separator: str,
    strip_lines: bool,
) -> Iterable[tuple[int, str]]:
    """
    Yield (page_number_1_indexed, text) tuples for each requested page.

    Uses pdfplumber's context manager so files are closed even on error.
    """
    with pdfplumber.open(pdf_path) as pdf:
        for idx in page_indices:
            if idx >= len(pdf.pages):
                logger.warning("Skipping page %d (beyond document length)", idx + 1)
                continue
            page = pdf.pages[idx]
            raw = page.extract_text() or ""
            if strip_lines:
                # Normalize whitespace but preserve paragraph breaks.
                text = "\n".join(line.rstrip() for line in raw.splitlines())
            else:
                text = raw
            yield idx + 1, text


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Extract text from a PDF using pdfplumber.",
    )
    parser.add_argument("--input", "-i", required=True, type=Path, help="Input PDF path")
    parser.add_argument(
        "--output", "-o", required=True, type=Path, help="Output text file path"
    )
    parser.add_argument(
        "--pages",
        "-p",
        default=None,
        help="Page selection, e.g. '1-10' or '1,3,5-7'. Defaults to all pages.",
    )
    parser.add_argument(
        "--separator",
        default=DEFAULT_SEPARATOR,
        help="String inserted between pages (default: blank line). "
        "Use '\\f' for a form feed.",
    )
    parser.add_argument(
        "--no-strip",
        action="store_true",
        help="Disable per-line trailing-whitespace stripping.",
    )
    parser.add_argument(
        "--verbose", "-v", action="store_true", help="Enable debug logging"
    )
    return parser


def main(argv: list[str] | None = None) -> int:
    args = build_parser().parse_args(argv)

    logging.basicConfig(
        level=logging.DEBUG if args.verbose else getattr(logging, LOG_LEVEL, logging.INFO),
        format="%(asctime)s [%(levelname)s] %(message)s",
    )

    if not args.input.exists():
        logger.error("Input file not found: %s", args.input)
        return 2
    if not args.input.is_file():
        logger.error("Input path is not a file: %s", args.input)
        return 2

    args.output.parent.mkdir(parents=True, exist_ok=True)

    try:
        # Peek at the PDF to determine page count before parsing the range.
        with pdfplumber.open(args.input) as pdf:
            total = len(pdf.pages)

        page_indices = parse_page_range(args.pages, total)
        logger.info(
            "Extracting %d page(s) from %s", len(page_indices), args.input.name
        )

        strip_lines = STRIP_LINES and not args.no_strip
        separator = args.separator.replace("\\f", "\f").replace("\\n", "\n")

        written = 0
        with args.output.open("w", encoding="utf-8") as out:
            for page_num, text in extract_text(
                args.input, page_indices, separator, strip_lines
            ):
                if written:
                    out.write(separator)
                out.write(text)
                written += 1
                logger.debug("Wrote page %d (%d chars)", page_num, len(text))

        if written == 0:
            logger.warning("No text extracted — PDF may be image-only or empty.")
            return 1

        logger.info("Wrote %d page(s) to %s", written, args.output)
        return 0

    except pdfplumber.pdfminer.pdfparser.PDFSyntaxError as exc:
        logger.error("Malformed PDF: %s", exc)
        return 3
    except ValueError as exc:
        logger.error("Invalid page range: %s", exc)
        return 4
    except PermissionError as exc:
        logger.error("Permission denied: %s", exc)
        return 5
    except Exception as exc:  # noqa: BLE001 — top-level safety net
        logger.exception("Unexpected failure: %s", exc)
        return 1


if __name__ == "__main__":
    sys.exit(main())

Configuration

All configuration is optional — sensible defaults are baked in.

CLI flags override environment variables where they overlap (--no-strip beats PDF_EXTRACT_STRIP_LINES).

Usage Examples

1. Extract the first ten pages:

$ python3 pdf_extract.py --input report.pdf --output text.txt --pages 1-10
2024-06-01 10:12:03 [INFO] Extracting 10 page(s) from report.pdf
2024-06-01 10:12:04 [INFO] Wrote 10 page(s) to text.txt

2. Extract specific pages with a form-feed separator:

$ python3 pdf_extract.py -i invoice.pdf -o invoice.txt -p 1,3,5-7 --separator '\f'
2024-06-01 10:15:22 [INFO] Extracting 5 page(s) from invoice.pdf
2024-06-01 10:15:22 [INFO] Wrote 5 page(s) to invoice.txt

3. Full document with verbose logging:

$ PDF_EXTRACT_STRIP_LINES=false python3 pdf_extract.py -i manual.pdf -o manual.txt -v
2024-06-01 10:20:00 [DEBUG] Wrote page 1 (2412 chars)
2024-06-01 10:20:00 [DEBUG] Wrote page 2 (1987 chars)
...
2024-06-01 10:20:01 [INFO] Wrote 42 page(s) to manual.txt

Error Handling

Encrypted PDFs will surface as a PDFSyntaxError or a pdfminer-level exception and be caught by the top-level handler. Empty extraction (image-only scans) is not treated as a crash — it's a warning so a batch pipeline can distinguish "no text found" from "processing failed".

Extending Ideas

• OCR fallback: detect empty extraction and route the page through pytesseract + pdf2image to handle scanned documents automatically.
• Table extraction: page.extract_tables() returns structured rows — add a --tables flag that emits CSV alongside the plain text.
• Batch mode: accept a directory via --input and walk it with Path.rglob("*.pdf"), mirroring the output tree.
• Metadata sidecar: write a companion JSON file with author, title, creation date, and per-page character counts for downstream indexing (e.g. Elasticsearch ingestion).