Tune Remote Write Shards and Queue for High-Throughput Ingest

This snippet sets the queue_config parameters that govern how Prometheus batches and ships samples over remote write. capacity is the number of samples buffered per shard, max_shards caps horizontal parallelism, max_samples_per_send controls batch size, and batch_send_deadline forces a flush even if the batch is not full. Prometheus dynamically increases shards when the queue is backing up and decreases them when idle, so max_shards effectively sets the ceiling on concurrent HTTP requests.

In production, these values are the difference between a stable pipeline and one that constantly drops samples under load. A common starting point for a node exporting 50k samples/s is capacity 10000, max_shards 200, max_samples_per_send 2000, batch_send_deadline 5s. High-latency links to a remote region benefit from larger batch_send_deadline and fewer, bigger batches.

Performance impact is direct: too many shards exhaust file descriptors and saturate the receiver; too few cause WAL lag and drops. Security-wise, larger batches mean larger TLS records, so ensure your proxy and receiver body limits accommodate them. Pitfalls include setting max_samples_per_send above the receiver's max body size (causing 413 responses), ignoring min_shards on cold start, and forgetting that capacity is per-shard, so total buffer equals capacity times active shards.

> System.Execute()

remote_write:
  - url: https://receiver.example.com/api/v1/write
    remote_timeout: 30s
    queue_config:
      capacity: 10000
      min_shards: 1
      max_shards: 200
      max_samples_per_send: 2000
      batch_send_deadline: 5s
      min_backoff: 30ms
      max_backoff: 5s
      retry_on_http_429: true
    metadata_config:
      send: true
      send_interval: 1m
      max_samples_per_send: 500

> Architecture & Security Context

Verified for Valters IT Hub. When deploying this configuration in a production homelab or enterprise environment, ensure that you monitor system resources and implement strict access controls. Administrator privileges are required.

ID: VALTERSIT-6A75E0D6D3BD