Enforce Data Minimization with jq Redaction Pipeline for API Logs

This pipeline streams JSON API logs through jq to strip or hash fields that are not strictly necessary for the operational purpose, implementing GDPR storage limitation and data minimization at the ingestion boundary. It uses jq's walk and del functions to recursively remove keys matching a denylist (password, token, email, ssn, dob, address) and replaces them with either null or a salted SHA-256 hash computed via @base64 and openssl. The internal mechanism is a filter applied per JSON line with -c to keep NDJSON format, so downstream SIEM tools ingest clean records without schema change.

Production use cases include log shippers (Fluent Bit, Vector, Filebeat) invoking this as a filter processor, and ETL jobs that must persist request metadata without retaining user identifiers. Regulated environments often run a dual pipeline: full-fidelity logs go to a short-TTL encrypted store for incident response, while the minimized stream feeds analytics and long-term retention, satisfying Article 5(1)(c) and (e).

Security implications: the hashing salt must be stored in a secrets manager (Vault, AWS Secrets Manager) and rotated; an unrotated salt enables rainbow-table re-identification. Performance is dominated by per-line jq process spawn if invoked naively, so batch with jq -c '.' reading stdin instead of looping in bash. Common pitfalls: nested arrays of objects require walk, not top-level del; and regex key matching must be case-insensitive to catch Email vs email. Validate the filter with a golden test corpus before flipping production.

> System.Execute()

SALT=$(cat /etc/secrets/log-salt)
zcat /var/log/api/*.json.gz \
 | jq -c 'walk(if type=="object" then with_entries(select(.key|test("^(password|token|email|ssn|dob|address)$";"i")|not)) else . end)' \
 | while IFS= read -r line; do
     uid=$(printf '%s' "$line" | jq -r '.user_id // empty')
     if [ -n "$uid" ]; then
       h=$(printf '%s%s' "$SALT" "$uid" | sha256sum | cut -d' ' -f1)
       line=$(printf '%s' "$line" | jq -c --arg h "$h" '.user_id=$h')
     fi
     printf '%s\n' "$line"
   done > /var/log/api/minimized.ndjson

> Architecture & Security Context

Verified for Valters IT Hub. When deploying this configuration in a production homelab or enterprise environment, ensure that you monitor system resources and implement strict access controls. Administrator privileges are required.

ID: VALTERSIT-715817B9F666