Process Monitor with Auto-Restart Policy

Process Monitor with Auto-Restart Policy

Problem Statement

Long-running services occasionally crash — a gunicorn worker hits an unhandled exception, a background job runs out of memory, or a daemon exits due to a transient dependency failure. Systemd and Docker restart policies handle this, but they require you to own the process lifecycle end-to-end. When you're running a service inside a container without an init system, on a dev box, or on a host where you don't control the supervisor, you need a lightweight watchdog that can watch any process by name and bring it back up if it dies.

This script solves that narrow problem: given a process name (or PID), it polls the system, detects when the target disappears, and re-launches it via a configurable command — up to a maximum number of restarts, with backoff, and with clean logging. It's small enough to read in one sitting and good enough to run in production for non-critical workloads.

How It Works

The monitor runs a single-threaded polling loop. On each tick it calls psutil.process_iter() and filters by the target name (or checks a specific PID with psutil.pid_exists). If the process is alive, the loop sleeps and repeats. If the process is gone, the script logs the event, applies an exponential backoff delay, and spawns the restart command using subprocess.Popen. Each successful restart increments a counter; when the counter reaches --max-restarts, the script exits with a non-zero status so an outer supervisor (systemd, Kubernetes, cron) can decide what to do next.

Two design decisions matter. First, restarts are counted cumulatively across the script's lifetime, not per-crash-window — this keeps the logic simple and predictable, and the outer supervisor handles the "too many crashes in a short window" case. Second, the script uses Popen with start_new_session=True so the child survives the monitor's own exit and doesn't receive the monitor's signals. The monitor never becomes the process's parent in a meaningful way; it's a fire-and-forget launcher with a health check loop.

Full Script

#!/usr/bin/env python3
"""
Process Monitor with Auto-Restart Policy

Watches a named process (or PID) and re-launches a restart command when it
disappears. Restarts are capped and spaced with exponential backoff. Exits
non-zero when the restart budget is exhausted so an outer supervisor can act.

Usage:
    python3 proc_watch.py --process gunicorn --max-restarts 5
"""

from __future__ import annotations

import argparse
import logging
import os
import shlex
import signal
import subprocess
import sys
import time
from dataclasses import dataclass
from typing import Optional

import psutil

# ---------------------------------------------------------------------------
# Configuration (env vars with sensible defaults)
# ---------------------------------------------------------------------------
DEFAULT_POLL_INTERVAL = float(os.environ.get("PROC_WATCH_POLL_INTERVAL", "5"))
DEFAULT_BACKOFF_BASE = float(os.environ.get("PROC_WATCH_BACKOFF_BASE", "2"))
DEFAULT_BACKOFF_MAX = float(os.environ.get("PROC_WATCH_BACKOFF_MAX", "60"))
DEFAULT_RESTART_CMD = os.environ.get("PROC_WATCH_RESTART_CMD", "")
LOG_LEVEL = os.environ.get("PROC_WATCH_LOG_LEVEL", "INFO").upper()

logger = logging.getLogger("proc_watch")


@dataclass
class MonitorState:
    """Mutable state carried across polling iterations."""

    restarts: int = 0
    last_restart_at: float = 0.0
    stopped: bool = False


def configure_logging(level: str) -> None:
    """Set up a single-line, timestamped log format to stderr."""
    logging.basicConfig(
        level=getattr(logging, level, logging.INFO),
        format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
        datefmt="%Y-%m-%dT%H:%M:%S%z",
        stream=sys.stderr,
    )


def find_process_by_name(name: str) -> Optional[psutil.Process]:
    """
    Return the first process whose name matches `name`, else None.

    Matching is case-insensitive and also checks the first token of the
    cmdline, which catches cases where the executable name differs from the
    display name (e.g. 'gunicorn' vs 'python3').
    """
    target = name.lower()
    for proc in psutil.process_iter(["name", "cmdline"]):
        try:
            pname = (proc.info.get("name") or "").lower()
            if pname == target:
                return proc
            cmdline = proc.info.get("cmdline") or []
            if cmdline:
                first = os.path.basename(cmdline[0]).lower()
                if first == target:
                    return proc
        except (psutil.NoSuchProcess, psutil.AccessDenied):
            # Process vanished or we can't inspect it; skip and continue.
            continue
    return None


def process_is_alive(name: Optional[str], pid: Optional[int]) -> bool:
    """Check liveness by PID if given, otherwise by process name."""
    if pid is not None:
        return psutil.pid_exists(pid) and psutil.Process(pid).is_running()
    if name is not None:
        return find_process_by_name(name) is not None
    return False


def compute_backoff(attempt: int, base: float, cap: float) -> float:
    """Exponential backoff: base * 2^(attempt-1), clamped to `cap` seconds."""
    delay = base * (2 ** max(0, attempt - 1))
    return min(delay, cap)


def launch_restart(command: str) -> Optional[int]:
    """
    Spawn the restart command detached from this process.

    Returns the child PID on success, or None if spawning failed. The child
    is placed in a new session so it survives the monitor's own exit and
    doesn't receive SIGHUP/SIGINT intended for the monitor.
    """
    try:
        # shell=True is intentional: users pass full command strings like
        # "gunicorn --bind 0.0.0.0:8000 app:app". shlex.quote is applied by
        # the caller if they want to build the string safely.
        child = subprocess.Popen(
            shlex.split(command),
            stdout=subprocess.DEVNULL,
            stderr=subprocess.DEVNULL,
            stdin=subprocess.DEVNULL,
            start_new_session=True,
            close_fds=True,
        )
        return child.pid
    except (OSError, ValueError) as exc:
        logger.error("Failed to launch restart command %r: %s", command, exc)
        return None


def install_signal_handlers(state: MonitorState) -> None:
    """SIGINT/SIGTERM set a flag so the loop exits cleanly after this tick."""

    def _handler(signum: int, _frame) -> None:
        logger.info("Received signal %s; shutting down after current tick", signum)
        state.stopped = True

    signal.signal(signal.SIGINT, _handler)
    signal.signal(signal.SIGTERM, _handler)


def parse_args(argv: Optional[list[str]] = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Monitor a process and restart it when it dies.",
    )
    target = parser.add_mutually_exclusive_group(required=True)
    target.add_argument("--process", help="Process name to watch (e.g. gunicorn)")
    target.add_argument("--pid", type=int, help="Specific PID to watch")

    parser.add_argument(
        "--restart-cmd",
        default=DEFAULT_RESTART_CMD,
        help="Shell command to run when the process is missing. "
        "Falls back to $PROC_WATCH_RESTART_CMD.",
    )
    parser.add_argument(
        "--max-restarts",
        type=int,
        default=5,
        help="Maximum number of restarts before exiting non-zero (default: 5).",
    )
    parser.add_argument(
        "--poll-interval",
        type=float,
        default=DEFAULT_POLL_INTERVAL,
        help=f"Seconds between liveness checks (default: {DEFAULT_POLL_INTERVAL}).",
    )
    parser.add_argument(
        "--backoff-base",
        type=float,
        default=DEFAULT_BACKOFF_BASE,
        help=f"Base seconds for exponential backoff (default: {DEFAULT_BACKOFF_BASE}).",
    )
    parser.add_argument(
        "--backoff-max",
        type=float,
        default=DEFAULT_BACKOFF_MAX,
        help=f"Maximum backoff in seconds (default: {DEFAULT_BACKOFF_MAX}).",
    )
    parser.add_argument(
        "--dry-run",
        action="store_true",
        help="Detect and log restarts but do not actually spawn commands.",
    )
    return parser.parse_args(argv)


def main(argv: Optional[list[str]] = None) -> int:
    args = parse_args(argv)
    configure_logging(LOG_LEVEL)

    if not args.restart_cmd and not args.dry_run:
        logger.error(
            "No restart command provided. Set --restart-cmd or "
            "$PROC_WATCH_RESTART_CMD, or use --dry-run."
        )
        return 2

    state = MonitorState()
    install_signal_handlers(state)

    target_label = args.process or f"pid:{args.pid}"
    logger.info(
        "Monitoring %s (poll=%.1fs, max_restarts=%d, dry_run=%s)",
        target_label,
        args.poll_interval,
        args.max_restarts,
        args.dry_run,
    )

    while not state.stopped:
        try:
            alive = process_is_alive(args.process, args.pid)
        except (psutil.NoSuchProcess, psutil.AccessDenied) as exc:
            # PID disappeared between pid_exists() and is_running(); treat as dead.
            logger.debug("Liveness check raised %s; treating as dead", exc)
            alive = False

        if alive:
            logger.debug("%s is alive", target_label)
            time.sleep(args.poll_interval)
            continue

        # --- Process is down -------------------------------------------------
        if state.restarts >= args.max_restarts:
            logger.error(
                "Restart budget exhausted (%d/%d). Exiting non-zero so an "
                "outer supervisor can intervene.",
                state.restarts,
                args.max_restarts,
            )
            return 1

        delay = compute_backoff(state.restarts + 1, args.backoff_base, args.backoff_max)
        logger.warning(
            "%s is down. Restart %d/%d in %.1fs",
            target_label,
            state.restarts + 1,
            args.max_restarts,
            delay,
        )

        # Interruptible sleep so signals are honored promptly.
        deadline = time.monotonic() + delay
        while not state.stopped and time.monotonic() < deadline:
            time.sleep(min(0.5, deadline - time.monotonic()))

        if state.stopped:
            break

        if args.dry_run:
            logger.info("[dry-run] would run: %s", args.restart_cmd)
        else:
            pid = launch_restart(args.restart_cmd)
            if pid is None:
                # Launch failed — don't count it as a used restart slot, but
                # do back off so we don't hammer a broken command.
                logger.error("Restart launch failed; will retry after backoff")
                time.sleep(delay)
                continue
            logger.info("Spawned restart command as PID %d", pid)

        state.restarts += 1
        state.last_restart_at = time.time()
        # Give the new process a moment to come up before re-checking.
        time.sleep(args.poll_interval)

    logger.info("Monitor stopped. Total restarts: %d", state.restarts)
    return 0


if __name__ == "__main__":
    sys.exit(main())

Configuration

All knobs are available as CLI flags, and several have environment-variable defaults so you can bake them into a container image or systemd unit:

Usage Examples

1. Watch gunicorn and restart it up to 5 times:

$ python3 proc_watch.py \
    --process gunicorn \
    --restart-cmd "gunicorn --bind 0.0.0.0:8000 app:app" \
    --max-restarts 5
2024-06-01T12:00:00+0000 INFO    proc_watch: Monitoring gunicorn (poll=5.0s, max_restarts=5, dry_run=False)
2024-06-01T12:00:35+0000 WARNING proc_watch: gunicorn is down. Restart 1/5 in 2.0s
2024-06-01T12:00:38+0000 INFO    proc_watch: Spawned restart command as PID 48211

2. Dry-run to verify detection before wiring it up:

$ PROC_WATCH_RESTART_CMD="systemctl start myapp" \
  python3 proc_watch.py --process myapp --dry-run
2024-06-01T12:01:00+0000 INFO    proc_watch: Monitoring myapp (poll=5.0s, max_restarts=5, dry_run=True)
2024-06-01T12:01:12+0000 WARNING proc_watch: myapp is down. Restart 1/5 in 2.0s
2024-06-01T12:01:14+0000 INFO    proc_watch: [dry-run] would run: systemctl start myapp

3. Pin a specific PID with a tight poll interval:

$ python3 proc_watch.py \
    --pid 12345 \
    --restart-cmd "./worker.sh" \
    --poll-interval 1 \
    --backoff-base 1 --backoff-max 10

Error Handling

• Process vanishes mid-check — psutil.NoSuchProcess and AccessDenied are caught inside find_process_by_name and around the liveness call. A process that disappears between pid_exists() and is_running() is treated as dead, not as an error.
• Restart command fails to spawn — subprocess.Popen can raise OSError (binary not found, permission denied) or ValueError (bad quoting). The failure is logged, the restart counter is not incremented, and the loop backs off before retrying so a broken command doesn't spin.
• Restart budget exhausted — the script exits with code 1. An outer supervisor (systemd Restart=on-failure, a Kubernetes liveness probe, or a cron job) can then take over. Exit code 2 means misconfiguration (no restart command supplied).
• Signals — SIGINT and SIGTERM set a stopped flag rather than raising, so the loop finishes its current tick and exits cleanly. The backoff sleep is chunked into 0.5s slices so shutdown is responsive even during a 60-second backoff.
• Unparseable command strings — shlex.split raises ValueError for unbalanced quotes; this is caught in launch_restart and logged rather than crashing the monitor.

Extending Ideas

• Health checks beyond liveness — replace the boolean process_is_alive with an HTTP probe (e.g. requests.get("http://127.0.0.1:8000/healthz", timeout=2)) so the monitor restarts on unhealthy rather than merely dead processes. This turns the script into a mini-Kubernetes liveness probe.
• Prometheus metrics endpoint — expose restarts_total, last_restart_timestamp, and up via a tiny http.server thread on --metrics-port. Scrape it from Prometheus and alert on restart storms.
• Sliding-window restart limit — swap the cumulative state.restarts counter for a collections.deque of timestamps and enforce "max N restarts in M minutes". This catches crash loops that the cumulative budget would miss.
• systemd unit generation — add a --write-unit flag that emits a .service file with Restart=always, RestartSec, and StartLimitBurst derived from the CLI arguments, so users can graduate from the script to a native supervisor once the policy is tuned.