This tutorial provides a production-ready Python script for extracting keywords from text documents using the RAKE (Rapid Automatic Keyword Extraction) alg
This tutorial provides a production-ready Python script for extracting keywords from text documents using the RAKE (Rapid Automatic Keyword Extraction) algorithm. RAKE is a highly effective, language-independent algorithm for extracting keywords from a document by analyzing the frequency of word co-occurrence and their positions relative to stop words.
In today's data-rich world, understanding the core topics and themes within large volumes of text is crucial for many applications. Manually reading and summarizing numerous documents is time-consuming and impractical. The problem this script solves is the automated identification of key phrases or keywords from a collection of text documents.
This automated keyword extraction is invaluable for various real-world scenarios: content summarization, document indexing and retrieval, search engine optimization (SEO) analysis, content tagging for recommendation systems, and competitive analysis. By distilling documents down to their most salient terms, we enable quicker insights and more efficient data processing.
This script implements a simple yet robust pipeline for keyword extraction. It takes an input directory containing text files, processes each file, extracts a specified number of top keywords using the RAKE algorithm, and then aggregates these results into a JSON output file.
The core of the pipeline is the rake-nltk library, which provides an implementation of the RAKE algorithm. RAKE operates by first identifying candidate keywords based on stop words (common words like "the", "is", "and") and punctuation. It then builds a graph of word co-occurrences within these candidate phrases. Each word receives a score based on its degree (how many times it co-occurs with other words) and its frequency. Finally, candidate phrases are scored by summing the scores of their constituent words, and the highest-scoring phrases are returned as keywords. The script handles file reading, text preprocessing (implicitly by RAKE), keyword extraction, and structured output generation, including robust error handling for file operations and NLTK data downloads.
To run this script, you'll need to install the dependencies:
pip install rake-nltk nltk
#!/usr/bin/env python3
"""
Keyword Extraction Pipeline with RAKE
This script processes text files from a specified input directory,
extracts a defined number of top keywords from each file using the RAKE algorithm,
and outputs the results into a JSON file.
"""
import argparse
import json
import logging
import os
import sys
from pathlib import Path
from typing import Any, Dict, List, Tuple
# External library imports
import nltk
from rake_nltk import Rake
# --- Configuration Section ---
# Environment variable for NLTK data path, defaults to user's home directory
NLTK_DATA_PATH: Path = Path(os.environ.get("NLTK_DATA_PATH", str(Path.home() / ".nltk_data")))
# Ensure NLTK_DATA_PATH exists
NLTK_DATA_PATH.mkdir(parents=True, exist_ok=True)
# Environment variable for logging level, defaults to INFO
LOG_LEVEL: str = os.environ.get("LOG_LEVEL", "INFO").upper()
# Default output filename if not specified via command line
DEFAULT_OUTPUT_FILENAME: str = "keywords.json"
# --- Logging Setup ---
def _configure_logging() -> None:
"""Configures the logging settings for the script."""
logging.basicConfig(
level=getattr(logging, LOG_LEVEL, logging.INFO),
format="%(asctime)s - %(levelname)s - %(message)s",
handlers=[logging.StreamHandler(sys.stdout)],
)
# --- NLTK Data Download Check ---
def _ensure_nltk_data() -> None:
"""
Ensures that necessary NLTK data (stopwords, punkt) are available.
Downloads them if they are missing.
"""
nltk.data.path.append(str(NLTK_DATA_PATH))
required_packages = ["stopwords", "punkt"]
for package in required_packages:
try:
nltk.data.find(f"tokenizers/{package}" if package == "punkt" else f"corpora/{package}")
logging.debug(f"NLTK package '{package}' already available.")
except nltk.downloader.DownloadError:
logging.info(f"NLTK package '{package}' not found. Attempting to download...")
try:
nltk.download(package, quiet=True, download_dir=str(NLTK_DATA_PATH))
logging.info(f"NLTK package '{package}' downloaded successfully to {NLTK_DATA_PATH}.")
except Exception as e:
logging.error(f"Failed to download NLTK package '{package}': {e}")
sys.exit(1)
logging.debug("All required NLTK data packages are available.")
# --- Keyword Extraction Logic ---
def extract_keywords_from_text(text: str, num_keywords: int = 10) -> List[Tuple[str, float]]:
"""
Extracts keywords from a given text using the RAKE algorithm.
Args:
text: The input text from which to extract keywords.
num_keywords: The maximum number of top keywords to return.
Returns:
A list of tuples, where each tuple contains a keyword phrase and its score.
The list is sorted by score in descending order.
"""
# Initialize RAKE with English stopwords.
# The stopwords list helps RAKE identify valid candidate phrases.
r = Rake(stopwords=nltk.corpus.stopwords.words('english'))
# Extract keywords from the text.
r.extract_keywords_from_text(text)
# Get ranked phrases with their scores.
# These are sorted by score in descending order by default.
ranked_phrases_with_scores: List[Tuple[str, float]] = r.get_ranked_phrases_with_scores()
# Return only the top N keywords as requested.
return ranked_phrases_with_scores[:num_keywords]
# --- Main Script Logic ---
def main() -> None:
"""
Main function to parse arguments, process files, and extract keywords.
"""
_configure_logging()
_ensure_nltk_data() # Ensure NLTK data is ready before initializing Rake
parser = argparse.ArgumentParser(
description="Extract keywords from text files using the RAKE algorithm.",
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
parser.add_argument(
"--input",
type=str,
required=True,
dest="input_path",
help="Path to the directory containing text files (.txt) for keyword extraction."
)
parser.add_argument(
"--top-keywords",
type=int,
default=15,
dest="top_keywords",
help="Number of top keywords to extract per document."
)
parser.add_argument(
"--output",
type=str,
default=DEFAULT_OUTPUT_FILENAME,
dest="output_file",
help="Path to the output JSON file where keywords will be saved."
)
args = parser.parse_args()
input_dir_path = Path(args.input_path)
output_file_path = Path(args.output_file)
# Validate input directory
if not input_dir_path.is_dir():
logging.error(f"Input path '{input_dir_path}' is not a valid directory or does not exist.")
sys.exit(1)
logging.info(f"Starting keyword extraction from directory: {input_dir_path}")
logging.info(f"Extracting top {args.top_keywords} keywords per document.")
all_document_keywords: Dict[str, List[Dict[str, Any]]] = {}
processed_count = 0
# Iterate over all .txt files in the input directory
for file_path in input_dir_path.glob("*.txt"):
try:
logging.info(f"Processing file: {file_path.name}")
# Read the content of the text file
file_content = file_path.read_text(encoding="utf-8")
# Extract keywords using the RAKE algorithm
keywords = extract_keywords_from_text(file_content, args.top_keywords)
# Store the extracted keywords, converting tuples to dictionaries for JSON output
all_document_keywords[file_path.name] = [
{"keyword": kw, "score": score} for kw, score in keywords
]
processed_count += 1
except FileNotFoundError:
logging.warning(f"File not found, skipping: {file_path.name}")
except PermissionError:
logging.warning(f"Permission denied to read file, skipping: {file_path.name}")
except UnicodeDecodeError as e:
logging.warning(f"Could not decode file '{file_path.name}' (encoding error: {e}), skipping.")
except Exception as e:
logging.error(f"An unexpected error occurred while processing '{file_path.name}': {e}")
if not all_document_keywords:
logging.warning("No keywords extracted from any document. Check input directory and file types.")
sys.exit(0)
# Ensure the output directory exists
output_file_path.parent.mkdir(parents=True, exist_ok=True)
# Write the aggregated keywords to the specified JSON file
try:
with open(output_file_path, "w", encoding="utf-8") as f:
json.dump(all_document_keywords, f, ensure_ascii=False, indent=4)
logging.info(f"Successfully extracted keywords from {processed_count} documents.")
logging.info(f"Results saved to: {output_file_path}")
except PermissionError:
logging.error(f"Permission denied to write to output file: {output_file_path}")
sys.exit(1)
except IOError as e:
logging.error(f"Error writing output file '{output_file_path}': {e}")
sys.exit(1)
if __name__ == "__main__":
main()
The script uses a few configuration parameters, some of which can be overridden via environment variables for greater flexibility in different deployment environments.
NLTK_DATA_PATH:~/.nltk_data (a .nltk_data directory in the user's home directory).NLTK_DATA_PATHexport NLTK_DATA_PATH=/opt/nltk_dataLOG_LEVEL:INFOLOG_LEVELDEBUG, INFO, WARNING, ERROR, CRITICALexport LOG_LEVEL=DEBUGDEFAULT_OUTPUT_FILENAME:--output is not specified.keywords.jsonFirst, let's create a dummy directory with some articles for demonstration:
mkdir -p articles
echo "Python is a high-level, interpreted programming language. It is widely used for web development, data analysis, artificial intelligence, and scientific computing. Its design philosophy emphasizes code readability with its use of significant indentation. Python supports multiple programming paradigms, including object-oriented, imperative, and functional programming." > articles/python.txt
echo "Artificial intelligence (AI) is intelligence demonstrated by machines, as opposed to the natural intelligence displayed by animals including humans. Leading AI textbooks define the field as the study of 'intelligent agents': any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals. Colloquially, the term 'artificial intelligence' is often used to describe machines that mimic 'cognitive' functions that humans associate with the human mind, such as 'learning' and 'problem-solving'." > articles/ai.txt
echo "Data science is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from noisy, structured and unstructured data, and apply knowledge and actionable insights from data across a broad range of application domains. Data science is related to data mining, machine learning and big data." > articles/data_science.txt
Basic Usage: Extract default top 15 keywords from articles/ directory, output to keywords.json
python3 keyword_extract.py --input articles/
Expected Output (console):
2023-10-27 10:00:01,123 - INFO - Starting keyword extraction from directory: articles
2023-10-27 10:00:01,123 - INFO - Extracting top 15 keywords per document.
2023-10-27 10:00:01,124 - INFO - Processing file: python.txt
2023-10-27 10:00:01,125 - INFO - Processing file: ai.txt
2023-10-27 10:00:01,126 - INFO - Processing file: data_science.txt
2023-10-27 10:00:01,127 - INFO - Successfully extracted keywords from 3 documents.
2023-10-27 10:00:01,128 - INFO - Results saved to: keywords.json
Expected keywords.json content (truncated for brevity):
{
"python.txt": [
{
"keyword": "significant indentation",
"score": 9.0
},
{
"keyword": "multiple programming paradigms",
"score": 9.0
},
{
"keyword": "scientific computing",
"score": 8.0
},
// ... more keywords
],
"ai.txt": [
{
"keyword": "artificial intelligence",
"score": 10.0
},
{
"keyword": "natural intelligence displayed",
"score": 9.0
},
// ... more keywords
],
"data_science.txt": [
{
"keyword": "actionable insights",
"score": 9.0
},
{
"keyword": "interdisciplinary field",
"score": 8.0
},
// ... more keywords
]
}
Custom Output File and Fewer Keywords: Extract top 5 keywords, save to my_analysis.json
python3 keyword_extract.py --input articles/ --top-keywords 5 --output my_analysis.json
Expected Output (console):
# ... (similar INFO messages as above) ...
2023-10-27 10:00:02,123 - INFO - Extracting top 5 keywords per document.
# ...
2023-10-27 10:00:02,128 - INFO - Results saved to: my_analysis.json
Expected my_analysis.json content (truncated):
{
"python.txt": [
{
"keyword": "significant indentation",
"score": 9.0
},
{
"keyword": "multiple programming paradigms",
"score": 9.0
},
{
"keyword": "scientific computing",
"score": 8.0
},
{
"keyword": "functional programming",
"score": 7.0
},
{
"keyword": "object-oriented",
"score": 6.0
}
],
// ... (similar for ai.txt, data_science.txt with 5 keywords each)
}
Error Handling Example: Invalid Input Directory
python3 keyword_extract.py --input non_existent_folder/
Expected Output (console):
2023-10-27 10:00:03,123 - ERROR - Input path 'non_existent_folder' is not a valid directory or does not exist.
The script is designed with several layers of error handling to ensure robustness:
--input path is a valid and existing directory. If not, it logs an error and exits gracefully.FileNotFoundError: If a file within the input directory disappears between listing and reading, it's caught and logged as a warning, allowing the script to continue processing other files.PermissionError: If the script lacks read permissions for a file, it's caught and logged as a warning, skipping that specific file.UnicodeDecodeError: If a text file cannot be decoded (e.g., it's not UTF-8), a warning is logged, and the file is skipped.Exception: Any other unexpected errors during file processing are caught and logged, preventing the script from crashing.stopwords and punkt NLTK data packages are present. If not, it attempts to download them to the NLTK_DATA_PATH. If the download fails (e.g., due to network issues or permission problems), an error is logged, and the script exits, as RAKE cannot function without these dependencies.PermissionError: If the script cannot write to the specified --output file (e.g., due to directory permissions), an error is logged, and the script exits.IOError: Other general I/O errors during file writing are caught and logged.