Kubernetes CPU Throttling & OOMKilled: Requests vs Limits

A Java service on a node with 40% idle CPU. p99 latency quadrupled every six hours, like clockwork. kubectl top pods showed 200m usage against a 500m limit — “we have headroom.” The on-call engineer spent a week chasing GC logs and network retries before someone finally pulled the CFS throttling metrics. The container was being frozen for tens of milliseconds every scheduling period because of a burst pattern the average metric could never see. The node was idle. The pod was suffocating.

This is for platform engineers, SREs, and sysadmins who’ve been paged at 3am for a throttled pod or an OOMKilled container and are tired of guessing. By the end you’ll understand the actual kernel mechanisms behind requests and limits, know which metrics to alert on, and have a sizing playbook you can apply Monday morning without a meeting.

The core misunderstanding is this: requests and limits are two different mechanisms solving two different problems. Requests are a scheduling contract. Limits are a runtime enforcement boundary. Confusing them is how you get throttled pods on idle nodes and OOMKilled pods with 4Gi free on the host.

TL;DR

• Requests feed the scheduler’s bin-packing math. Limits feed the kernel’s CFS quota and cgroup memory controller. They are unrelated systems.
• CPU limits cause throttling via CFS quota — a hard 100ms window where your container simply stops running. Averages hide it; p99 latency exposes it.
• Memory limits trigger the cgroup OOM killer, not node eviction. Your pod can die with gigabytes free on the node.
• Alert on container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total > 25%, not on CPU usage percentage.
• CPU limits are optional on single-tenant latency-sensitive workloads. Memory limits are not optional anywhere.

Prerequisites

You’ll get the most out of this if you have:

• A Kubernetes cluster (1.24+ is fine; cgroup v2 behavior is noted where it differs)
• kubectl with cluster-admin or namespace-admin access
• Prometheus + cAdvisor scraping (standard in most distros via kubelet /metrics/cadvisor)
• Optional but recommended: Vertical Pod Autoscaler installed in recommendation mode
• Shell access to a node, or kubectl debug for ephemeral containers

Requests and Limits Are Not the Same Thing

The reason this matters: every sizing decision you make downstream depends on knowing which of these two knobs you’re actually turning. Most teams set them together out of habit and never question it.

What Requests Actually Do — The Scheduler Contract

A request is a promise made at scheduling time. The kubelet’s scheduler sums the requests of every pod on a node and refuses to place a new pod if the sum would exceed allocatable capacity. That’s it. The request does nothing at runtime. It is not a reservation, not a guarantee of available CPU, and not enforced by the kernel in any way.

apiVersion: v1
kind: Pod
metadata:
  name: scheduler-demo
spec:
  containers:
    - name: app
      image: nginx:1.27-alpine
      resources:
        requests:
          cpu: "250m"
          memory: "128Mi"

This pod requests 250 millicores and 128Mi. The scheduler will only place it on a node where 250m CPU and 128Mi memory are unallocated. Once running, the container can burst to the full node’s CPU — there is no runtime cap.

You can see how the scheduler is doing its math with:

kubectl describe node worker-03 | sed -n '/Allocated resources/,/Events/p'

The output shows requested vs. allocatable per resource. If that ratio is 95% but kubectl top node shows 30% actual usage, you’re overcommitting requests — the node is “full” from the scheduler’s point of view while sitting mostly idle. That’s a capacity planning problem, not a runtime one.

The QoS class is derived from how requests and limits relate:

kubectl get pod scheduler-demo -o jsonpath='{.status.qosClass}'
# Burstable

• BestEffort: no requests, no limits
• Burstable: requests set, limits differ (or only one set)
• Guaranteed: requests == limits for every container, every resource

What Limits Actually Do — The Runtime Enforcement

A limit is enforced by the Linux kernel via cgroups, not by Kubernetes, not by the kubelet. This distinction matters because it means the enforcement is fast, hard, and completely indifferent to what else is happening on the node.

For CPU, the kernel uses CFS (Completely Fair Scheduler) bandwidth control. The kubelet writes two values into the container’s cgroup: cpu.cfs_period_us (default 100000, i.e. 100ms) and cpu.cfs_quota_us (limit × period / 1000). A 500m limit becomes 50000 microseconds of quota per 100ms window.

You can read these from inside a running container:

kubectl exec -it scheduler-demo -- sh -c \
  'cat /sys/fs/cgroup/cpu.max 2>/dev/null || \
   cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpu.cfs_period_us'

On cgroup v2 (default on most modern distros), cpu.max returns 50000 100000 — quota and period on one line. On cgroup v1 you get two files. Either way, the semantics are identical: 50ms of CPU every 100ms, hard stop.

For memory, the limit is written to memory.max (v2) or memory.limit_in_bytes (v1). When the cgroup’s memory usage hits that ceiling, the kernel’s OOM killer picks a process inside the cgroup and kills it. The kubelet has no say in this — it just observes the exit code.

Table 1 — Requests vs Limits: Mechanism, Timing, Enforcement, Failure Mode

Aspect	Requests	Limits
Enforced by	Scheduler (userspace)	Kernel (cgroups)
When	Scheduling time	Runtime, every 100ms (CPU) or on allocation (memory)
Scope	Node-level bin packing	Per-container
CPU failure mode	Pod stays Pending	CFS throttling (latency spike)
Memory failure mode	Pod stays Pending	cgroup OOMKilled (exit 137)
QoS impact	Determines class with limits	Determines class with requests

Q: Should I always set requests equal to limits?

No. That gives you Guaranteed QoS, which buys you the highest eviction priority during node pressure — but it also means the scheduler reserves the full limit as capacity, wasting headroom on bursty workloads. Use it only for latency-critical, single-purpose pods where you’ve already measured steady-state usage. For everything else, set requests from p95 and limits with headroom.

CPU Throttling — The Silent Killer of p99 Latency

Throttling is invisible on every dashboard that shows averages. That’s why it takes days to find. Understanding the mechanism is the only way to build detection that actually fires.

CFS Quota Mechanics — Why Your 100ms Period Matters

The CFS period is a fixed window. Your container gets quota microseconds of CPU time inside each period. If it exhausts the quota, the kernel deschedules every thread in the cgroup until the next period begins. This is not a slowdown — it’s a hard freeze.

A 500m limit gives 50ms of CPU per 100ms. If your workload is a JVM that wakes up, spawns 20 threads, and burns 50ms of aggregate CPU in the first 20ms of a window, the remaining 80ms is a stop-the-world pause the JVM knows nothing about. Your latency histogram shows a spike. Your CPU usage shows 50%. Your throttle metric shows 100%.

You can watch this happen with a simple loop:

kubectl exec -it scheduler-demo -- sh -c '
  end=$((SECONDS+5))
  while [ $SECONDS -lt $end ]; do :; done
  cat /sys/fs/cgroup/cpu.stat 2>/dev/null | grep -E "throttled|nr_periods"
'

The cpu.stat file exposes nr_periods, nr_throttled, and throttled_usec. If nr_throttled is climbing, you’re being frozen. The Prometheus equivalent comes from cAdvisor:

rate(container_cpu_cfs_throttled_seconds_total{namespace="prod"}[5m])

This gives you throttled seconds per second. A value of 0.15 means 15% of wall-clock time is spent frozen.

Q: Why does my pod get throttled when the node has free CPU?

Because CFS quota is enforced per-container, not per-node. The scheduler’s decision to place your pod there is irrelevant to the kernel. If your container’s quota is exhausted inside the current 100ms window, it stops running regardless of how idle the rest of the machine is. Node-level CPU metrics will never show this.

Detecting Throttling Before Your Users Do

Three metrics matter. Everything else is noise:

• container_cpu_cfs_throttled_periods_total — how many 100ms windows ended with throttling
• container_cpu_cfs_periods_total — total windows elapsed
• container_cpu_usage_seconds_total — actual CPU consumed (for context, not alerting)

The ratio of the first two is your throttle ratio. Anything sustained above 25% is a problem. Above 50% and you’re leaving significant latency on the table.

sum by (pod, namespace) (
  rate(container_cpu_cfs_throttled_periods_total[5m])
)
/
sum by (pod, namespace) (
  rate(container_cpu_cfs_periods_total[5m])
)

Wrap that in a recording rule so your dashboards stay cheap:

groups:
  - name: cpu-throttle
    interval: 30s
    rules:
      - record: namespace_pod:container_cpu_throttle_ratio:rate5m
        expr: |
          sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
          /
          sum by (namespace, pod) (rate(container_cpu_cfs_periods_total[5m]))

Then alert on the recording rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cpu-throttling
  namespace: monitoring
spec:
  groups:
    - name: kubernetes.cpu.throttling
      rules:
        - alert: ContainerCpuThrottlingHigh
          expr: namespace_pod:container_cpu_throttle_ratio:rate5m > 0.25
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} throttling {{ $value | humanizePercentage }} of periods"

The 10-minute for clause filters out legitimate burst throttling. A pod that bursts once during a request spike is fine. A pod throttled continuously for 10 minutes is a sizing problem.

The “Just Remove CPU Limits” Debate

There’s a real argument here and it’s worth engaging with honestly. Google’s Borg paper and several Kubernetes maintainers have argued that CPU limits are an anti-pattern: they cause throttling, they’re poorly understood, and they don’t provide meaningful isolation on a well-sized cluster. The counter-argument is that without limits, a runaway process can starve every other pod on the node.

My take, after running both:

• Single-tenant, latency-sensitive workloads (your own services, dedicated node pools): remove CPU limits. Set requests from p95, add HPA, keep node headroom.
• Multi-tenant clusters, batch jobs, anything you don’t own: keep CPU limits. Noisy neighbors are real and the blast radius is the whole node.
• Mixed environments: separate node pools by tenancy class. This is the only clean answer.

Table 2 — With CPU Limits vs Without: Latency, Isolation, Cost, Eviction Risk

Dimension	With CPU Limits	Without CPU Limits
Tail latency	Throttling risk under burst	No CFS freeze
Isolation	Hard per-container cap	Soft — depends on node contention
Scheduler efficiency	Requests still drive packing	Same
QoS class	Burstable or Guaranteed	Burstable (unless requests == limits)
Eviction priority	Higher if Guaranteed	Lower — BestEffort is first to go
Blast radius of a runaway	Contained	Whole node

The gotcha nobody mentions: removing limits changes the QoS class. If you had requests == limits for Guaranteed, dropping the limit drops you to Burstable — and Burstable pods are evicted before Guaranteed under node pressure. You’ve traded throttling for eviction risk. Make that trade deliberately.

OOMKilled — When the Kernel Decides You’re Done

Memory is not compressible. You cannot freeze a process and resume it later. When a cgroup hits its memory ceiling, something has to die, and the kernel picks.

The cgroup OOM Killer vs Node Pressure Eviction

These are two entirely different mechanisms and people conflate them constantly. The cgroup OOM killer is a kernel behavior triggered when a container exceeds its own memory limit. Node pressure eviction is a kubelet behavior triggered when the node runs low on memory. They have different triggers, different scopes, and different signatures.

kubectl describe pod my-app-6d8f9c-abc12

A cgroup OOM looks like this in the output:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 27 Sep 2026 03:14:22 +0000
  Finished:     Mon, 27 Sep 2026 03:14:47 +0000

Exit code 137 is SIGKILL (128 + 9). Both cgroup OOM and node eviction produce 137, but evicted pods show up differently:

kubectl get events --field-selector reason=Evicted -A

The kubelet also logs the reason on the node. And the kernel itself logs cgroup OOM events:

dmesg -T | grep -i "oom-kill" | tail -20

You’ll see lines like Memory cgroup out of memory: Killed process 12345 (java) total-vm:8388608kB, anon-rss:6291456kB. That anon-rss number is what actually mattered — the resident set size, not virtual memory.

Table 3 — cgroup OOMKilled vs Node Eviction: Trigger, Scope, Exit Code, Recovery

Aspect	cgroup OOMKilled	Node Pressure Eviction
Trigger	Container exceeds memory.max	Node below eviction threshold
Enforced by	Linux kernel	kubelet
Scope	Single container	One or more pods on the node
Exit code	137 (SIGKILL)	137, plus Evicted reason
Pod status	Container restarts per policy	Pod deleted, rescheduled
Fix	Raise memory limit or fix leak	Reduce node pressure or add capacity

Q: Why is my pod OOMKilled when the node has free memory?

Because the cgroup memory limit is what triggered the kill, not node-level pressure. The kernel’s cgroup OOM killer fires when the container’s own memory usage hits memory.max — it doesn’t care how much memory is free elsewhere on the host. A pod with a 512Mi limit will be killed at 512Mi even on a 256Gi node with 200Gi free.

Why Your Java/Node/Python App OOMs at 80% of Its Limit

The classic trap: your runtime’s heap is not your container’s memory. RSS includes heap, Metaspace, thread stacks, JIT code cache, direct byte buffers, mmap’d files, and page cache. On the JVM, that overhead is routinely 25–40% above heap size.

Pre-Java 10, the JVM ignored cgroup limits entirely and sized the heap from host memory. Modern JVMs (11+) respect cgroups if you enable container support (default in recent versions, but explicitly set it to be safe):

env:
  - name: JAVA_TOOL_OPTIONS
    value: >-
      -XX:+UseContainerSupport
      -XX:MaxRAMPercentage=75.0
      -XX:InitialRAMPercentage=50.0
      -XX:+ExitOnOutOfMemoryError
resources:
  limits:
    memory: "2Gi"
  requests:
    memory: "2Gi"

MaxRAMPercentage=75.0 tells the JVM to cap heap at 75% of the container’s memory limit, leaving 25% for everything else. ExitOnOutOfMemoryError makes the JVM die cleanly on heap exhaustion instead of thrashing — you get a clean restart instead of a hang that eventually gets OOMKilled.

Node.js has the same problem. The default old-space size is derived from system memory, not cgroup limits, unless you tell it otherwise:

FROM node:22-alpine
ENV NODE_OPTIONS="--max-old-space-size=1536"
COPY . /app
WORKDIR /app
CMD ["node", "server.js"]

That 1536 is in MiB. If your container limit is 2Gi, this leaves ~500Mi for native modules, buffers, and V8’s own overhead. Tune it down if you see OOMKills.

Go is better behaved since 1.19 with GOMEMLIMIT, which sets a soft memory ceiling the GC will respect:

GOMEMLIMIT=1500MiB ./my-service

Note: GOMEMLIMIT is a soft limit. The GC will work harder to stay under it, but it can’t prevent a hard allocation spike. Pair it with a real cgroup limit for the hard stop.

Debugging OOMKills Without Losing Your Mind

Investigation order matters. Start with the pod, then the cgroup, then the application.

kubectl describe pod my-app-6d8f9c-abc12 | grep -A5 "Last State"
kubectl logs my-app-6d8f9c-abc12 --previous --tail=100

Then inspect the cgroup memory stats from inside the container (before it dies, or via an ephemeral debug container):

kubectl debug -it my-app-6d8f9c-abc12 --image=busybox --target=app -- sh -c '
  cat /sys/fs/cgroup/memory.stat 2>/dev/null || \
  cat /sys/fs/cgroup/memory/memory.stat
'

Key fields to look at:

• anon — anonymous memory (heap, stacks). This is what usually kills you.
• file — page cache. Counts against your limit but is reclaimable under pressure.
• slab — kernel structures. Rarely large, but can surprise you.
• pgmajfault — major page faults. High numbers mean you’re hitting swap or re-reading files.

If you see the pod OOMing only under load, it’s almost always a leak plus a spike — the leak raises the baseline, the spike pushes it over. Profile with your runtime’s tools (async-profiler for JVM, --inspect for Node, pprof for Go) and look at allocation rate, not just heap size.

One trap that catches people: a preStop hook that allocates memory. Graceful shutdown code that serializes state, flushes buffers, or opens new connections can push memory over the ceiling right as the pod is being terminated.

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15 && /app/flush-metrics.sh"]

That flush-metrics.sh might buffer 500Mi of data. Under a tight limit, this kills the pod during shutdown, which is confusing because the pod was healthy right up until it wasn’t. Test your preStop hooks under memory pressure.

The Production Playbook — Sizing Requests and Limits Without Guessing

Theory is cheap. Here’s what actually works.

Start With Data, Not Vibes

Run without limits in staging for at least two weeks, ideally 30 days for anything latency-sensitive. Capture p50, p95, p99 CPU and peak RSS memory. If you can’t run without limits (multi-tenant), use VPA in recommendation-only mode:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"

updateMode: "Off" means VPA only produces recommendations — it never mutates your pods. Read them:

kubectl get vpa my-app-vpa -o jsonpath='{.status.recommendation.containerRecommendations[*]}' | jq

You’ll get lowerBound, target, and upperBound per container. target is a reasonable starting point for requests. upperBound is where you’d want limits if you’re setting them.

The Sizing Rules I Actually Use

• CPU requests = p95 usage over 30 days
• CPU limits = 2–3× p95, or none on single-tenant latency-sensitive workloads
• Memory requests = peak RSS + 20%
• Memory limits = peak RSS + 40%
• JVM: limits.memory = heap_size × 1.5 as a floor, tune from there
• Never set memory limits below your runtime’s minimum viable footprint — you’ll get crash loops that look like bugs

A well-sized deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 6
  template:
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.4.2
          resources:
            requests:
              cpu: "500m"      # p95 usage measured over 30d
              memory: "768Mi"  # peak RSS 640Mi + 20%
            limits:
              memory: "900Mi"  # peak RSS + 40%
              # no CPU limit — single-tenant node pool

The bad version, for contrast:

resources:
  requests:
    cpu: "100m"      # "it's mostly idle"
    memory: "256Mi"  # "it looked fine in dev"
  limits:
    cpu: "200m"      # guarantees throttling under any load
    memory: "512Mi"  # OOMKills at 400Mi RSS

This pod will throttle constantly and OOM within hours of any real traffic. The requests are so low the scheduler packs dozens of them per node, then they all fight for CPU at runtime. This is how you get “the node is idle but everything is slow.”

Q: What should I set my requests and limits to?

Start with VPA recommendations in Off mode. Use the target value for requests. For limits, use upperBound for memory and either upperBound or nothing for CPU depending on tenancy. Measure for at least two weeks before you commit, and revisit quarterly.

Monitoring, Alerting, and the Dashboards You Actually Need

Node-level dashboards lie. A node at 30% CPU can be hosting ten pods that are all throttled to death. You need pod-level metrics or you’re flying blind.

The four signals that matter:

1. Throttle ratio — throttled_periods / total_periods per container
2. OOM events — kube_pod_container_status_last_terminated_reason\{reason="OOMKilled"\}
3. Memory pressure — container_memory_working_set_bytes / container_spec_memory_limit_bytes
4. CPU saturation — container_cpu_usage_seconds_total relative to request, not limit

The alert that matters is the throttle ratio. The alert that doesn’t is “CPU usage > 80%” — that’s a number without a denominator. 80% of what? Averages hide everything.

Table 5 — Metric Sources: cAdvisor vs kube-state-metrics vs node-exporter

Metric Source	Provides	Use For	Doesn’t Provide
cAdvisor (kubelet)	Per-container CPU, memory, throttling, cgroup stats	Runtime behavior, throttling, OOM forensics	Pod lifecycle, spec, status
kube-state-metrics	Pod/Deployment/Node spec and status	QoS class, requests/limits, restart reasons	Actual runtime usage
node-exporter	Host-level CPU, memory, disk, network	Capacity planning, node health	Per-pod attribution

You need all three. They answer different questions.

A complete alerting set:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: container-resources
  namespace: monitoring
spec:
  groups:
    - name: kubernetes.container.resources
      rules:
        - alert: ContainerCpuThrottlingHigh
          expr: namespace_pod:container_cpu_throttle_ratio:rate5m > 0.25
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.pod }} throttling {{ $value | humanizePercentage }}"

        - alert: ContainerOOMKilled
          expr: increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[15m]) > 0
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.pod }} OOMKilled in last 15m"

        - alert: ContainerMemoryNearLimit
          expr: |
            container_memory_working_set_bytes
            /
            container_spec_memory_limit_bytes > 0.9
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.pod }} at {{ $value | humanizePercentage }} of memory limit"

        - alert: ContainerCpuUsageAboveRequest
          expr: |
            sum by (namespace, pod) (rate(container_cpu_usage_seconds_total[5m]))
            /
            sum by (namespace, pod) (kube_pod_container_resource_requests{resource="cpu"}) > 1.5
          for: 30m
          labels:
            severity: info
          annotations:
            summary: "{{ $labels.pod }} using 1.5x its CPU request"

Q: What’s the single most important metric to alert on?

container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total > 25% sustained for 10 minutes. It’s the only metric that directly correlates with user-visible latency degradation on CPU-bound workloads, and it fires before your error rate moves. If you alert on one thing from this article, alert on this.

Common Mistakes and How to Avoid Them

1. Setting requests equal to limits “for safety.” This gives you Guaranteed QoS, which sounds safe but wastes scheduling capacity — the scheduler reserves the full limit as if it were always in use. Only do this for pods where you’ve measured steady-state usage and genuinely need the eviction priority.

2. Using kubectl top to diagnose throttling. It shows averages. Throttling is a per-period event. You will never see it. Use container_cpu_cfs_throttled_periods_total.

3. Setting memory limits below runtime overhead. A JVM with a 1Gi heap needs at least 1.4Gi of container memory. A Node process with --max-old-space-size=1536 needs at least 2Gi. Budget for the runtime, not just the heap.

4. Removing CPU limits without checking QoS class. Dropping a limit from a Guaranteed pod makes it Burstable, which changes its eviction priority. If you’re removing limits to fix throttling, verify you’re not creating an eviction problem.

5. Forgetting that page cache counts against memory limits. A container that reads large files will accumulate page cache, and that cache counts against memory.max. It’s reclaimable under pressure, but the kernel may OOM-kill before it gets around to reclaiming. If your workload does heavy file I/O, add headroom.

FAQ

Q: Should I set CPU limits?

Depends on tenancy. Single-tenant clusters or dedicated node pools: no, remove them and rely on requests plus HPA. Multi-tenant clusters or shared node pools: yes, keep them, because one runaway pod can degrade every neighbor. If you’re unsure, keep them and alert on throttling — you can always remove them later with data.

Q: Why is my pod throttled when the node has free CPU?

CFS quota is enforced per-container by the kernel, not per-node. Your container gets a fixed slice of CPU time per 100ms window regardless of what else is happening on the host. When that slice is exhausted, the container is frozen until the next window. The node’s idle CPU is irrelevant.

Q: Why is my pod OOMKilled when the node has free memory?

Same reason, different resource. The cgroup memory limit is enforced per-container. When your container’s memory usage hits memory.max, the kernel kills a process inside that cgroup. Node-level free memory doesn’t matter — the cgroup is a hard boundary.

Q: What’s the difference between OOMKilled and Evicted?

OOMKilled is a kernel action against a container that exceeded its cgroup memory limit. Evicted is a kubelet action against a pod when the node is under memory pressure. OOMKilled restarts the container in place (per restart policy). Evicted deletes the pod and the ReplicaSet schedules a replacement elsewhere.

Q: How do I know if I’m being throttled?

Query rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) for your pod. Anything above 0.25 sustained is a problem. Anything above 0.5 is a serious problem. If you don’t have this metric, install cAdvisor scraping — it’s enabled by default in most Kubernetes distributions via the kubelet’s /metrics/cadvisor endpoint.

Key Takeaways

Key Takeaways

• Requests are for the scheduler, limits are for the kernel. They solve different problems and are enforced by different systems. Stop treating them as a pair.
• Alert on throttle ratio, not CPU usage. throttled_periods / total_periods > 25% for 10 minutes is the signal. CPU usage percentage tells you nothing without context.
• Memory limits are mandatory; CPU limits are a tenancy decision. Remove CPU limits on single-tenant latency-sensitive workloads. Keep them on shared clusters. Never remove memory limits.
• Measure before you tune. VPA in recommendation-only mode costs nothing and gives you real numbers. Two weeks minimum, 30 days for anything latency-critical.
• Exit code 137 is ambiguous. Check kubectl describe pod for OOMKilled vs Evicted, and check dmesg on the node for the kernel’s view. They’re different failures with different fixes.

Find every pod without requests or limits in one shot:

kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.status.phase == "Running") |
  .metadata.namespace as $ns |
  .metadata.name as $pod |
  .spec.containers[] |
  select(.resources.requests == null or .resources.limits == null) |
  "\($ns)/\($pod) container=\(.name) requests=\(.resources.requests // "NONE") limits=\(.resources.limits // "NONE")"
'

Run that Monday morning. Whatever it prints is your backlog.