Are your CPU and memory requests inflated? An open source tool to find out

The Kubernetes scheduler reserves what a pod asks for (the request), not what it uses. A request four times the real usage keeps nodes half empty, and Karpenter cannot consolidate pods that say they need capacity they do not use. Before talking about Spot or Graviton, the first cost lever is usually this one: ask for what you use.

rightsizing-check is an open source command line tool that lists the requests to lower, how much that frees, and which workloads are asking for too little, which is a risk of its own. By default it works offline: it reads your manifests and a usage file that you exported, and never connects to the cluster. Optionally, it reads the usage straight from a Prometheus (--prometheus, with a --window such as 7d), with GET requests only.

Why another tool

Goldilocks and KRR already exist, and they are good. They run against a live cluster or a Prometheus. rightsizing-check is for when you cannot or do not want to: a CI job, reviewing a client's cluster from files, an isolated environment. You export the numbers once, and the analysis is a pure function of your files: the same files give the same result. When you have a Prometheus, --prometheus fetches the two queries for you.

Where the usage comes from

You pick one of four sources. The first two are the ones to trust.

Straight from Prometheus. One flag fetches the two queries below and reads the result, over the window you choose (it must include your busiest period; 7 days is a good start):

rightsizing-check scan ./k8s --prometheus https://prometheus.example.com --window 7d
rightsizing-check scan ./k8s --prometheus https://prometheus.example.com --window 1d \
  --prom-selector 'namespace=~"shop|nps"'     # a part of the cluster at a time

The address can be the one of the web UI (/query and /graph are removed). The tool only sends GET requests to /api/v1/query, one query at a time, the memory one first. A token goes through --prom-token-env, read from an environment variable.

A long window is heavy. The CPU query is a subquery over the whole window, so its cost grows with the number of series times the window. On a small Prometheus a 14-day window made the pod use enough memory to be evicted by the kubelet while the tool was being tested against a real server. A 7-day window over about 190 thousand series took 45 seconds and was fine. So a window over 7 days is refused unless you pass --prom-allow-long-window. Start with 1d or 7d, read a few namespaces at a time with --prom-selector, avoid peak hours, and never repeat a query that answered 504 or 503: the server may still be running it.

Prometheus, by hand. The same two instant queries, over the same window. Export each result as it comes:

PROM=http://prometheus:9090

# CPU: the 95th percentile of the per-container usage, in cores
curl -sG "$PROM/api/v1/query" --data-urlencode \
  'query=max by (namespace, pod, container) (quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[5m])[7d:5m]))' > cpu.json

# Memory: the maximum working set, in bytes
curl -sG "$PROM/api/v1/query" --data-urlencode \
  'query=max by (namespace, pod, container) (max_over_time(container_memory_working_set_bytes{container!="",container!="POD"}[7d]))' > mem.json

Use /api/v1/query, not query_range: the tool wants one value per container. The CPU query is a subquery and can be heavy on a large Prometheus, so shorten the window or query per namespace.

CSV. From Grafana, a notebook or anywhere else. The header is required, and a blank value means that number is not known:

namespace,pod,container,cpu_millis,memory_bytes
shop,api-5d8f7c9b6-x2k4j,app,120,629145600

`kubectl top`. Quick, and the weakest: one instant, not a percentile. The tool flags it when it is the source (RC008), because a workload that is idle now and busy at night looks oversized.

kubectl top pods -A --containers > top.txt

How pods are matched to workloads

Usage comes per pod and requests come per workload, so pods are matched by name, in the same namespace, the way controllers name them: name-<hash>-<suffix> for a Deployment, name-<ordinal> for a StatefulSet, name-<suffix> for a DaemonSet and the exact name for a bare Pod. The random parts use the alphabet Kubernetes uses, which has no vowels, so api-gateway-extra is never mistaken for a pod of api. Pods that match nothing are counted in the summary. Jobs and CronJobs are not matched, because their pods are short-lived.

With several replicas, the busiest one decides: one hot replica needs the reservation even if its siblings idle.

The recommendation

A request is inflated when it is above 1.5 times the recommendation (--slack), which is a utilization under about 58%. The headrooms are --cpu-headroom and --mem-headroom. Memory is not compressible, so the recommendation starts from the maximum over the window, not the average.

The rules

Each workload ends up with a status: oversized (a request far above usage and nothing at risk), undersized (a request or limit below usage, or a missing request), ok or no-data. undersized outranks oversized: a workload at risk is not a candidate to shrink. rightsizing-check explain RC001 documents any rule.

An example of the output

This is the real result on the repository's testdata, a made-up namespace:

STATUS      WORKLOAD                     CONTAINER  REPLICAS  CPU REQ  CPU USE  CPU REC  MEM REQ  MEM USE  MEM REC  FINDINGS
oversized   shop/deployment/api          app        3         1000m    120m     140m     2Gi      600Mi    720Mi    RC001,RC002
oversized   kube-system/daemonset/agent  agent      -         500m     40m      50m      512Mi    120Mi    144Mi    RC001,RC002
undersized  shop/deployment/worker       worker     2         100m     320m     370m     256Mi    576Mi    692Mi    RC003,RC004,RC005
undersized  shop/statefulset/db          db         1         -        450m     520m     -        1.5Gi    1844Mi   RC006
ok          shop/deployment/web          web        2         150m     120m     140m     200Mi    144Mi    176Mi    -
no-data     shop/cronjob/report          report     -         -        ?        ?        -        ?        ?        RC007

6 workloads: 2 oversized, 2 undersized, 1 ok, 1 without data
Reclaimable requests: 2.6 vCPU (73% of the 3.5 requested) and 3.9 GiB (56% of 6.9).

The reading, workload by workload:

• `api` requests 1000m and 2Gi, and uses 120m and 600Mi. The recommendation is 140m and 720Mi, which frees 860m and 1328Mi per pod, and with 3 replicas that is 2.6 vCPU in total.
• `worker` is the opposite: it uses 320m and requests 100m (RC003), uses 576Mi and requests 256Mi (RC004), and is at 90% of its 640Mi limit (RC005). It is not a candidate to shrink, it is a candidate to investigate.
• `db` has no request at all (RC006), so the scheduler treats it as if it needed nothing. The tool recommends 520m and 1844Mi from the usage.
• `report` is a CronJob, and CronJobs are not matched to pods, so it has no data (RC007).
• The DaemonSet shows in the table per pod, but stays out of the sums, because its replica count depends on the nodes.

VPAs and Argo Rollouts

A VerticalPodAutoscaler that is not in Off mode (Auto, Recreate, Initial or InPlaceOrRecreate; an unset mode is Auto) writes the requests of the pods it controls, so a request changed by hand in the manifest does not last. When the tool would change a request that a VPA controls, RC010 says so, and the summary adds how much of the reclaimable CPU sits on such workloads: change those through the VPA (minAllowed, maxAllowed or Off mode), not in the manifest. When the VPA object carries a status (a live cluster, or a manifest exported with kubectl get vpa -A -o yaml), RC011 shows its target next to the recommendation of the tool: the two can differ because each learns from a different window.

An Argo Rollout with a pod template is analyzed like a Deployment, and its pods are matched the same way. A Rollout that points at a Deployment (workloadRef) has no template of its own: the Deployment is the workload that is analyzed.

Live cluster, read-only

With --cluster, the tool reads workloads, HPAs, KEDA ScaledObjects, VPAs and Argo Rollouts straight from the cluster, and the usage of every container from the metrics API (metrics.k8s.io, what kubectl top reads), unless you give another usage source.

rightsizing-check scan --cluster
rightsizing-check scan --cluster --prometheus https://prometheus.example.com --window 7d

It is read-only: only list calls are sent (the tests check that no other verb is used), paged by 500 and with a few requests a second, one resource at a time. The usage from metrics-server is one instant and is flagged (RC008): with --prometheus, the metrics API is not read at all.

GitHub Action

The repository is also a GitHub Action: it installs the release with a checksum check, runs one scan that writes the report and the JSON (with --prometheus, the server is read once per run), comments on the pull request and fails the pipeline at the fail-on level only after the comment. The comment is found by a marker, so the Action never edits the comment of another tool.

- uses: robertasolimandonofreo/rightsizing-check@v0.4.1
  with:
    path: ./k8s
    prometheus: https://prometheus.example.com
    window: 7d
    comment: "true"
    fail-on: undersized

What the numbers mean

Reclaimable requests is what the recommendations free: for each inflated request, the request minus the recommendation, times the replicas. It covers Deployments, StatefulSets and Pods, and the percentages are over the requests of the workloads that have usage data.

The cost line (--cpu-price and --mem-price, per vCPU-month and per GiB-month) is your price times the reclaimable amount. The tool has no prices of its own. It is a figure in requests, not in your bill: it only becomes savings once nodes consolidate, which is what Karpenter does when pods stop asking for the capacity.

Cautions before lowering a request

• HPA. A HorizontalPodAutoscaler that scales on utilization uses the request as the denominator: lowering the request raises the utilization it sees, and it can scale out. The tool reads HPAs and KEDA ScaledObjects from the manifests and computes the utilization at the recommended request; RC009 only fires when it would pass the target. AverageValue targets and the other KEDA triggers (RabbitMQ, Prometheus, cron) do not depend on the request and are ignored. An HPA that exists only in the cluster is not seen in the manifests: use --cluster, or export it with kubectl get hpa -A -o yaml and pass the file.
• The window decides. A monthly job or a spike you did not capture makes a request look inflated. Use a window that covers your busiest period and lower one workload at a time, watching throttling and latency.
• Memory. Do not shrink a container flagged RC005 before you know why it is close to its limit. A limit below usage kills the container.
• CPU limits, init containers and sidecars injected at admission are not analyzed. An injected sidecar has no entry in the manifest.

What it does not do

• The recommendation is a starting point, as good as the window of your data.
• Pods are matched by name. A custom generateName, or a CRD that creates pods, is not recognized and shows up as unmatched.
• The Prometheus and `kubectl top` readers were tested against sample output, not against every version of them. If a file is rejected, --csv is the way out.
• It never edits your files.

Next steps for your cluster

1. Run rightsizing-check scan ./k8s --prometheus https://prometheus.example.com --window 7d (or export 7 days of usage and use --prom-cpu cpu.json --prom-mem mem.json).
2. Start with the undersized ones: a missing request, or one below usage, is a risk today.
3. Then lower the oversized ones with the most reclaimable vCPU, one at a time, checking the HPAs.
4. With honest requests, Karpenter consolidation has something to pack. Use the Karpenter NodePool generator and the Kubernetes savings calculator to estimate the gain.
5. Only then move workloads to Spot and Graviton. spot-readiness and graviton-readiness answer the equivalent question for each.

The tool's page, with install and example, is at /en/tools/rightsizing-check, and the code is at github.com/robertasolimandonofreo/rightsizing-check, under Apache-2.0. If you want me to review your cluster, validate the recommendations against your data and build a cost reduction plan, the efficiency assessment is the starting point.