Kubernetes Requests and Limits: The Config That Saves Money

I once walked into a cluster running at 30% node utilization on a $41k/month GKE bill. The platform team blamed the app team for “inefficient services.” The app team blamed the platform team for “oversized nodes.” The actual culprit was requests: 500m / 512Mi copy-pasted from a 2019 blog post into 340 Deployments, never revisited. The apps were using 40m CPU and 90Mi at p95.

Requests and limits are a contract. Requests are your reservation with the scheduler. Limits are the kernel’s enforcement boundary. Most teams write that contract wrong, then pay for the mistake every single month until someone with a spreadsheet gets angry enough to fix it.

This guide is for platform engineers, SREs, and sysadmins who already run production clusters and are tired of watching the bill climb while utilization stays flat. By the end you’ll have the mental model, the right-sizing workflow, the namespace guardrails, and the tooling to cut real money without breaking anything.

Prerequisites

Before you start cutting, make sure you have:

A Kubernetes cluster (1.25+ recommended; cgroup v2 is the default on modern distros and changes some behavior)
kubectl with cluster-admin or at least namespace-scoped write access
Prometheus + kube-state-metrics already scraping (or a managed equivalent)
Helm 3 for installing VPA and Goldilocks
Access to your cloud provider’s cost explorer or a Kubecost/OpenCost deployment

If you don’t have metrics yet, stop reading and go install kube-state-metrics. You cannot right-size what you cannot measure.

The Mental Model: Requests vs Limits

This is the section everyone skips and then spends six months debugging. The scheduler and the kernel are two different systems with two different jobs, and requests and limits feed each of them separately.

Requests are summed per node and used for admission decisions. Full stop. The scheduler does not look at actual usage — it looks at the sum of requests on each node and refuses to place a pod if the sum would exceed allocatable capacity. Over-requesting means the node looks full while sitting at 15% CPU. That stranded capacity is money on fire.

Limits are enforced by cgroups v2 (or v1 on older clusters). CPU limits translate to CFS quota — the process gets throttled when it exceeds its share within a 100ms period. Memory limits translate to a cgroup memory ceiling — exceed it and the OOM killer sends SIGKILL. No graceful shutdown, no draining, just exit code 137.

Here’s the asymmetry that matters: CPU limits throttle, memory limits kill. Throttling degrades latency. Killing restarts your pod mid-request.

# Minimal Pod showing requests only — scheduler cares, kernel does not
apiVersion: v1
kind: Pod
metadata:
  name: api-requests-only
spec:
  containers:
    - name: api
      image: ghcr.io/example/api:1.14.2
      resources:
        requests:       # Scheduler reservation. Summed per node.
          cpu: "250m"   # 0.25 cores guaranteed to be schedulable
          memory: "256Mi"  # 256 MiB reserved in the node's allocatable pool

Now add limits. The diff is small; the behavioral difference is enormous.

# Same Pod, now with limits — cgroup enforcement kicks in
apiVersion: v1
kind: Pod
metadata:
  name: api-with-limits
spec:
  containers:
    - name: api
      image: ghcr.io/example/api:1.14.2
      resources:
        requests:
          cpu: "250m"
          memory: "256Mi"
        limits:
          cpu: "1000m"      # CFS quota: throttled above 1 core per period
          memory: "512Mi"   # cgroup ceiling: OOMKilled above 512 MiB

The QoS class your pod lands in is derived entirely from how requests and limits relate:

QoS Class	Config	Eviction Priority	Typical Use
Guaranteed	requests == limits for every container	Last to be evicted	Latency-critical, predictable workloads
Burstable	requests < limits (or limits unset)	Middle tier	Most stateless services
BestEffort	No requests, no limits	First to be shot	Batch jobs on dedicated nodes, CI runners

BestEffort pods are evicted first under node pressure. If your production API has no requests or limits, you are volunteering it as the sacrificial lamb during every memory-pressure event.

The Three Configuration Archetypes

There are exactly three ways to configure a container, and two of them are wrong for most workloads.

Archetype 1: Guaranteed (requests == limits). Correct for latency-sensitive services with predictable resource profiles — think a Go API that holds steady at 400m CPU and 300Mi memory. Wrong for anything that legitimately bursts: JVM services during GC, Go services during traffic spikes, anything with a working set that swings 3×.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 6
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: api
          image: ghcr.io/example/payments-api:2.7.1
          resources:
            requests:
              cpu: "500m"
              memory: "384Mi"
            limits:
              cpu: "500m"     # matched — Guaranteed QoS
              memory: "384Mi"

Archetype 2: Burstable (requests < limits). The default recommendation for stateless services. Requests sized to p95, limits set generously. This is where 80% of your workloads should live.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-api
spec:
  replicas: 8
  selector:
    matchLabels:
      app: catalog-api
  template:
    metadata:
      labels:
        app: catalog-api
    spec:
      containers:
        - name: api
          image: ghcr.io/example/catalog-api:3.2.0
          resources:
            requests:
              cpu: "250m"     # p95 usage from 30-day window
              memory: "256Mi"
            limits:
              cpu: "1000m"    # 4× headroom for bursts
              memory: "512Mi" # 2× for memory (kills, so be generous)

CPU limits cause throttling even when the node is idle. CFS quota is per-period, not per-node. A container with cpu: 200m that needs a 150ms burst in a 100ms window gets throttled, regardless of whether the node is at 5% or 95%. For latency-sensitive services, the position I take is: remove CPU limits, keep CPU requests, and rely on node capacity + QoS for isolation. This is the “no CPU limits” debate and it has been settled in favor of removing them for years. Take the flame.

Archetype 3: BestEffort (nothing set). Almost always wrong in production. Defensible only for batch jobs on dedicated tainted nodes or short-lived CI runners where you genuinely don’t care about eviction.

# DON'T ship this to a shared production namespace
apiVersion: v1
kind: Pod
metadata:
  name: bad-idea
spec:
  containers:
    - name: app
      image: ghcr.io/example/app:latest
      # No resources block = BestEffort = first to be evicted

Archetype	QoS Class	Cost Predictability	Blast Radius
Guaranteed	Guaranteed	High (fixed reservation)	Low (isolated)
Burstable	Burstable	Medium	Medium
BestEffort	BestEffort	Low	High (noisy neighbor)

How Over-Requesting Burns Money

Here’s the lie your monitoring tells you: kubectl top nodes shows 15% CPU utilization, so you think you have headroom. Meanwhile the scheduler refuses to place new pods because the sum of requests exceeds allocatable. Both are true.

# Real output from a node that "looks" empty
kubectl describe node gke-prod-pool-3-8f2a | sed -n '/Allocated resources/,/Events/p'

Typical output looks like this:

Allocated resources:
  Resource           Requests      Limits
  cpu                7450m (93%)   12000m (150%)
  memory             28Gi (87%)    42Gi (131%)

The node appears full to the scheduler while actual usage sits at 15%. The bin-packing tax: the cluster autoscaler adds nodes based on requests, not usage. You pay for reservations you never consume.

Concrete math: 100 pods × 500m CPU request = 50 cores reserved. Actual p95 usage across those pods: 12 cores. That’s 38 cores of pure waste, at roughly $25-30/core/month on most clouds = $950-1,140/month for one over-requested workload class.

Dump your per-namespace request totals:

kubectl get pods -A -o json | jq -r '
  .items[] as $p |
  $p.spec.containers[] |
  select(.resources.requests.cpu != null) |
  "\($p.metadata.namespace) \(.resources.requests.cpu)"
' | awk '{gsub("m","",$2); sum[$1]+=$2} END {for (ns in sum) print ns, sum[ns]"m"}'

That gives you a rough per-namespace CPU request total in millicores. Compare it against actual usage from Prometheus and the gap is your waste.

The “just add a node” reflex is the most expensive debugging technique in the industry. I’ve watched a team scale from 12 → 40 nodes over six months chasing a latency regression. The actual fix was removing cpu: 2 from a limit on a service that needed 200ms bursts — CFS throttling was the culprit, and no amount of extra capacity fixes a per-period quota violation.

Right-Sizing in Practice

The Vertical Pod Autoscaler in Off mode is your recommender. It observes actual usage and produces lowerBound / target / upperBound recommendations without touching running pods. Auto mode restarts pods to apply changes — do not run it in production unless you enjoy incident reviews.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: catalog-api-vpa
  namespace: prod-catalog
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: catalog-api
  updatePolicy:
    updateMode: "Off"   # Recommend only. Never mutates pods.
  resourcePolicy:
    containerPolicies:
      - containerName: api
        minAllowed:
          cpu: "100m"
          memory: "128Mi"
        maxAllowed:
          cpu: "2"
          memory: "2Gi"
        controlledResources: ["cpu", "memory"]

After a few days, inspect the recommendations:

kubectl describe vpa catalog-api-vpa -n prod-catalog

You’ll see a block like:

Recommendation:
  Container Recommendations:
    Container Name:  api
    Lower Bound:
      Cpu:     120m
      Memory:  180Mi
    Target:
      Cpu:     310m
      Memory:  240Mi
    Upper Bound:
      Cpu:     900m
      Memory:  620Mi

Use the Target for requests. Use Upper Bound + headroom for limits. Do not use the average — it will under-provision and you’ll OOM at 3am.

Read the histogram, not the average

p50 is useless for sizing. Use p95 for requests, p99.9 plus headroom for limits. PromQL against the kubelet’s cAdvisor metrics:

# p95 CPU usage per container over 30 days
quantile_over_time(0.95,
  rate(container_cpu_usage_seconds_total{container!="", namespace="prod-catalog"}[5m])[30d:5m]
)

# Memory high-water mark (working set, not RSS, not cache)
max_over_time(
  container_memory_working_set_bytes{container!="", namespace="prod-catalog"}[30d]
)

container_memory_usage_bytes includes page cache and will wildly overstate your real memory need. The OOM killer cares about container_memory_working_set_bytes. Use the working set metric. If you size from usage_bytes you’ll over-provision by 2-3× on any container that touches files.

The 30-day observation window

Do not right-size from a Tuesday afternoon sample. Cover a full billing cycle. Include the month-end batch job, the quarterly report generator, the Black Friday spike. If your workload has seasonal components, extend to 90 days.

A weekly CronJob that dumps p95/p99 per workload to CSV gives you a defensible data set:

#!/usr/bin/env bash
set -euo pipefail
OUT="/var/reports/rightsizing-$(date +%F).csv"
echo "namespace,workload,metric,p95,p99,p999" > "$OUT"

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  for wl in $(kubectl get deploy,sts -n "$ns" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do
    p95=$(curl -sG "$PROM/api/v1/query" \
      --data-urlencode "query=quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{namespace=\"$ns\"}[5m])[30d:5m])" \
      | jq -r '.data.result[0].value[1] // "0"')
    echo "$ns,$wl,cpu,$p95,," >> "$OUT"
  done
done

Namespace-Level Guardrails

Per-workload tuning is necessary but not sufficient. Without namespace guardrails, the next team that ships a Deployment with no resources block will silently break your capacity model.

LimitRange provides defaults and per-pod caps at admission time. Every namespace should have one.

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: prod-catalog
spec:
  limits:
    - type: Container
      default:            # applied when limits are unset
        cpu: "500m"
        memory: "512Mi"
      defaultRequest:     # applied when requests are unset
        cpu: "100m"
        memory: "128Mi"
      max:                # hard ceiling per container
        cpu: "4"
        memory: "4Gi"
      min:
        cpu: "10m"
        memory: "32Mi"

Gotcha: LimitRange applies only at admission. Existing pods are untouched. You must redeploy to pick up defaults.

ResourceQuota is the budget that actually enforces. It caps the sum of requests and limits in a namespace.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-catalog-budget
  namespace: prod-catalog
spec:
  hard:
    requests.cpu: "40"
    requests.memory: "80Gi"
    limits.cpu: "80"
    limits.memory: "160Gi"
    pods: "200"
    count/deployments.apps: "40"

I’ve watched a quota of requests.cpu: 40 catch a runaway HPA that was about to spawn 400 pods against a misconfigured metric. The quota turned a five-figure incident into a scheduling rejection.

Ownership: Platform team owns LimitRange and ResourceQuota defaults. App teams own per-workload tuning within those bounds. Blur that line and you get neither.

Concern	LimitRange	ResourceQuota
Scope	Per container/pod	Per namespace (aggregate)
Enforcement point	Admission	Admission + ongoing
Sets defaults	Yes	No
Caps totals	No	Yes
Blocks new pods if exceeded	No (caps individual)	Yes

Common Mistakes and How to Avoid Them

1. Copy-pasting requests from a blog post (including this one). Your workload is not someone else’s workload. Run VPA for two weeks before you touch a manifest.

2. Setting CPU limits on latency-sensitive services. CFS quota throttles per-period regardless of node capacity. Remove the limit, keep the request, and let the scheduler do its job. Fix: kubectl patch to drop limits.cpu on services with p99 latency SLOs.

3. Sizing memory from container_memory_usage_bytes. That metric includes page cache and will overstate by 2-3×. Use container_memory_working_set_bytes. Fix: audit your dashboards and alerts to use working set.

4. Forgetting -XX:MaxRAMPercentage on the JVM. Old JVMs ignore cgroup limits and size their heap from the host’s total memory. Set -XX:MaxRAMPercentage=75.0 and let the runtime respect the container. Fix: add to JAVA_TOOL_OPTIONS in the Deployment env.

5. Shipping a LimitRange without announcing it. The first pod that gets rejected by the new max cap will page someone at 2am. Fix: announce, provide a two-week grace period, then enforce.

FAQ

Q: Can I just set requests = limits everywhere and be done?

No. Guaranteed QoS is right for a narrow class of predictable, latency-critical workloads. It’s wrong for anything that legitimately bursts — you’ll either over-provision (wasting money) or under-provision (throttling and OOM kills). Use Guaranteed only when you have p99.9 data showing flat usage.

Q: My pod keeps getting OOMKilled but kubectl top shows it’s under the limit.

kubectl top reports working set, which excludes some kernel-accounted memory. The OOM killer fires on the cgroup’s memory.max including page cache under pressure. Check container_memory_working_set_bytes at high resolution right before the kill. If it’s within 5% of the limit, add headroom — memory limits are hard kills, not soft.

Q: What if VPA recommends something huge for a batch job?

VPA’s model assumes steady-state usage and will over-recommend for spiky batch workloads. Use resourcePolicy to exclude batch workloads from VPA control, and size them manually from max_over_time over the job’s natural run duration. Or move them to a dedicated node pool with BestEffort QoS — that’s one of the few legitimate uses.

Q: Should I remove CPU limits cluster-wide?

No — start with latency-sensitive services that have SLOs. Batch jobs, CI runners, and anything that can tolerate throttling should keep limits to prevent noisy-neighbor effects. The rule is: if a 100ms throttle window would violate your SLO, remove the limit. Otherwise keep it.

Q: How often should I re-run right-sizing?

Quarterly for steady-state services. Every time you change the workload’s resource profile (new dependency, new language runtime, architecture change), re-run. The 30-day window is a minimum, not a target.

Key Takeaways

Your cloud bill is a configuration file. Stop treating it like weather.