Harden Alertmanager with inhibit rules and encrypted silence storage

This AlertmanagerConfig CRD defines inhibit rules that suppress lower-severity alerts when a higher-severity alert is firing for the same target, and references a Secret for SMTP and Slack credentials. The operator watches AlertmanagerConfig objects and merges them into the generated alertmanager.yaml secret consumed by the Alertmanager pod. Inhibit rules use equal label matching; when a critical alert matches, dependent warning alerts are muted, preventing alert storms during cascading failures.

In production, inhibit rules are essential during large incidents: without them, a single node failure can trigger hundreds of NodeDown, PodCrashLooping, and ServiceUnavailable alerts, overwhelming on-call. This config groups by cluster and namespace, so a critical ClusterDown suppresses all warnings in that cluster. AlertmanagerConfig also supports per-receiver routing trees, letting platform teams own their routing while the operator enforces cluster-wide defaults.

Security: credentials must live in Secrets, never inline in AlertmanagerConfig. The operator's webhook validates that referenced Secrets exist, but a missing Secret silently disables the receiver. Performance: overly broad inhibit rules can suppress legitimate alerts; always scope with equal and source_matchers. Common pitfall: AlertmanagerConfig requires the Prometheus Operator to be started with --alertmanager-config-namespaces and a matching alertmanagerConfigSelector on the Alertmanager CR, otherwise the config is ignored entirely. Also, mute_time_intervals must reference time_intervals defined in the same CR.

> System.Execute()

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: platform-inhibit
  namespace: monitoring
  labels:
    alertmanagerConfig: main
spec:
  route:
    receiver: slack-platform
    groupBy: [cluster, namespace, alertname]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
  inhibitRules:
    - sourceMatch:
        - name: severity
          value: critical
      targetMatch:
        - name: severity
          value: warning
      equal: [cluster, namespace]
  receivers:
    - name: slack-platform
      slackConfigs:
        - apiURL:
            name: alertmanager-secrets
            key: slack-webhook
          channel: '#platform-alerts'
          sendResolved: true

> Architecture & Security Context

Verified for Valters IT Hub. When deploying this configuration in a production homelab or enterprise environment, ensure that you monitor system resources and implement strict access controls. Administrator privileges are required.

ID: VALTERSIT-492B4F84D4A1