CVE-2026-27878

Executive Summary

CVE-2026-27878 is a denial-of-service (DoS) vulnerability in Grafana Tempo, a distributed tracing backend. The flaw resides in the TraceQL query engine, where a crafted query containing an excessively large exemplars hint value can trigger uncontrolled memory allocation. An authenticated user can exploit this to cause an out-of-memory (OOM) crash, effectively taking the Tempo instance offline. The vulnerability has a CVSS score of 6.5 (Medium), with a low attack complexity and no requirement for user interaction. No public exploit code or proof-of-concept is currently available, and the EPSS score is 0.0%, indicating low exploitation likelihood in the wild. However, the impact on availability is high, making this a critical concern for production environments.

Supply Chain & Package Security

Grafana Tempo is distributed as a Go binary and Docker image. The vulnerability is in the core query engine, not in a third-party package. However, organizations should verify the integrity of their Tempo binaries and Docker images by checking checksums and signatures from the official Grafana repository. No malicious package injection has been reported in relation to this CVE.

CVSS Breakdown & Threat Level

• Attack Vector (AV): Network (N) – The vulnerability can be exploited remotely over the network.
• Attack Complexity (AC): Low (L) – No special conditions are required; the attack can be repeated easily.
• Privileges Required (PR): Low (L) – An attacker needs valid authentication credentials (e.g., a user with query access).
• User Interaction (UI): None (N) – No user action is needed beyond the attacker sending the malicious query.
• Scope (S): Unchanged (U) – The exploit does not affect other components beyond the Tempo instance.
• Confidentiality (C): None (N) – No data is exposed.
• Integrity (I): None (N) – No data is modified.
• Availability (A): High (H) – The service can be completely crashed, leading to a denial of service.

Threat Level: Medium. While the CVSS score is 6.5, the potential for service disruption in production environments elevates the risk. The requirement for authentication reduces the attack surface but does not eliminate it, especially in multi-tenant deployments.

Deep Dive & Mechanics

The vulnerability is triggered by a TraceQL query that includes an exemplars hint with an extremely large value. In Grafana Tempo, the exemplars hint is used to control the number of exemplars (sample traces) returned with query results. When the hint value is set to an excessively high number (e.g., exemplars: 999999999), the query engine attempts to allocate memory to hold that many exemplar objects. This allocation is not bounded or validated, leading to a rapid memory consumption spike.

The memory allocation occurs in the query frontend or querier component, depending on the query path. The process will eventually exhaust available heap memory, causing an out-of-memory (OOM) kill by the operating system or a panic within the Go runtime. The crash results in a complete loss of query service until the Tempo instance is restarted.

The root cause is a lack of input validation on the exemplars hint parameter. The code likely uses the hint value directly in a memory allocation call (e.g., make([]exemplar, hintValue)) without checking for reasonable limits.

Vulnerable vs. Patched Code

Since no official patch is available, the following is a hypothetical comparison based on the vulnerability description.

Vulnerable Code (Go-like pseudocode):

func handleTraceQLQuery(query TraceQLQuery) ([]Result, error) {
    exemplarsHint := query.Hints["exemplars"]
    if exemplarsHint != "" {
        count, _ := strconv.Atoi(exemplarsHint)
        // Direct allocation without bounds check
        exemplars := make([]Exemplar, count)
        // ... populate exemplars ...
    }
    // ... rest of query handling ...
}

Patched Code (hypothetical):

func handleTraceQLQuery(query TraceQLQuery) ([]Result, error) {
    const maxExemplars = 1000
    exemplarsHint := query.Hints["exemplars"]
    if exemplarsHint != "" {
        count, err := strconv.Atoi(exemplarsHint)
        if err != nil || count < 0 || count > maxExemplars {
            return nil, fmt.Errorf("invalid exemplars hint value: must be between 0 and %d", maxExemplars)
        }
        exemplars := make([]Exemplar, count)
        // ... populate exemplars ...
    }
    // ... rest of query handling ...
}

The patched version introduces a maximum limit (maxExemplars) and validates the input before allocation.

Exploitation & Lab Setup

Red Team

Prerequisites:

• Access to a Grafana Tempo instance with query permissions (e.g., an API key or user credentials).
• Knowledge of the Tempo API endpoint (typically /api/search or /tempo/api/search).

Exploitation Steps:

1. Identify the Tempo query endpoint. For example: https://tempo.example.com/api/search
2. Craft a TraceQL query with an excessive exemplars hint. Example using curl:

   curl -X POST "https://tempo.example.com/api/search" \
     -H "Authorization: Bearer <token>" \
     -H "Content-Type: application/json" \
     -d '{
       "query": "{ .service.name = \"frontend\" }",
       "hints": {
         "exemplars": "999999999"
       }
     }'
   

3. Send the request. The Tempo instance will attempt to allocate memory for 999,999,999 exemplars, causing an OOM crash.
4. Repeat the request if the instance recovers automatically (e.g., via Kubernetes restart) to maintain the DoS.

Expected Outcome: The Tempo pod or process crashes, and the query service becomes unavailable until manual or automated restart.

Blue Team

Detection & Response:

• Monitor memory usage of Tempo processes. A sudden spike in memory allocation (e.g., from 2GB to 10GB in seconds) is a strong indicator.
• Enable audit logging for all TraceQL queries. Log the hints field to detect abnormally large exemplars values.
• Implement rate limiting on the query API to reduce the impact of repeated attacks.
• Use a Web Application Firewall (WAF) or API gateway to inspect and block requests with exemplars values exceeding a threshold (e.g., > 1000).

Lab Setup for Testing:

1. Deploy a test Grafana Tempo instance using Docker:

   docker run -d --name tempo -p 3200:3200 grafana/tempo:latest
   

2. Configure authentication (e.g., using a simple multi-tenant setup with an API key).
3. Send the malicious query from the Red Team section to observe the crash.
4. Verify that the instance becomes unresponsive and logs show an OOM error.

Indicators of Compromise (IOCs)

• Network: HTTP POST requests to /api/search or /tempo/api/search containing a JSON body with "exemplars": followed by a very large integer (e.g., > 10000).
• Logs: Tempo logs showing out of memory, cannot allocate memory, or fatal error: runtime: out of memory.
• System: Tempo process being killed by OOM killer (check dmesg or journalctl for oom-killer events).
• Memory: Rapid memory consumption spike in the Tempo process, visible in monitoring tools like Prometheus or Grafana.

Detection Engineering

YARA Rule

rule CVE_2026_27878_Tempo_OOM {
    meta:
        description = "Detects TraceQL queries with excessive exemplars hint in Grafana Tempo logs"
        author = "Valters IT"
        date = "2025-04-10"
        reference = "CVE-2026-27878"
    strings:
        $exemplars_hint = /"exemplars"\s*:\s*[0-9]{6,}/  // Matches exemplars value with 6+ digits
        $oom_error = "out of memory" nocase
        $fatal_error = "fatal error" nocase
    condition:
        any of ($exemplars_hint) or ($oom_error and $fatal_error)
}

Sigma Rule

title: CVE-2026-27878 - Grafana Tempo OOM via Large Exemplars Hint
id: 8a7b3c2d-1e5f-4a9b-8c7d-6e5f4a3b2c1d
status: experimental
description: Detects HTTP requests to Grafana Tempo with an abnormally large exemplars hint value.
author: Valters IT
date: 2025-04-10
references:
    - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2026-27878
logsource:
    category: webserver
    product: grafana_tempo
detection:
    selection:
        http.method: POST
        http.url|contains: "/api/search"
        http.body|contains: '"exemplars"'
    condition: selection and http.body|re: '"exemplars"\s*:\s*[0-9]{6,}'
falsepositives:
    - Legitimate queries with high exemplars count (unlikely)
level: high
tags:
    - attack.t1499
    - cve.2026.27878

Suricata Rule

alert http any any -> any any (msg:"CVE-2026-27878 Grafana Tempo OOM via Large Exemplars Hint"; flow:to_server; http.method; content:"POST"; http.uri; content:"/api/search"; http.request_body; content:"\"exemplars\""; pcre:"/\"exemplars\"\s*:\s*[0-9]{6,}/"; sid:1000001; rev:1;)

Remediation & Mitigation

Immediate Actions:

1. Restrict Query Permissions: Limit which users or API keys can execute TraceQL queries. Use Grafana Tempo's multi-tenant access control to restrict query capabilities.
2. Implement Input Validation: If you have access to the source code or can deploy a reverse proxy, add validation to reject queries with exemplars values exceeding a safe threshold (e.g., 1000).
3. Rate Limiting: Configure rate limiting on the Tempo query API to prevent rapid-fire attacks.
4. Memory Limits: Set memory limits on the Tempo container or process using cgroups or Kubernetes resource limits to prevent OOM from crashing the entire node.

Long-Term Fix:

• Apply the official patch once released by Grafana Labs. Monitor the Grafana Tempo GitHub repository and release notes for a fix.
• Upgrade to a patched version as soon as it becomes available.

Workaround:

• Disable the exemplars hint feature entirely by modifying the Tempo configuration (if a configuration option exists) or by using a reverse proxy to strip the hints field from incoming requests.

Valters IT
DevSecOps & Cyber Threat Intelligence Expert