Amazon EKS 1.37 upgrade guide: checklist, testing and safe rollback

Introduction

Upgrading an Amazon Elastic Kubernetes Service (Amazon EKS) cluster should not be treated as simply changing the version in Terraform or running aws eks update-cluster-version.

In production, the process spans several layers:

Applications
   ↓
Kubernetes APIs
   ↓
EKS Control Plane
   ↓
Controllers / Operators / Webhooks
   ↓
Worker Nodes
   ↓
CNI / DNS / CSI / kube-proxy
   ↓
AWS

An incompatibility in any of them can let the control plane finish the upgrade successfully while workloads start failing minutes or hours later.

For the migration:

EKS 1.36 → EKS 1.37

a safer strategy is:

Inventory
   ↓
Upgrade Insights
   ↓
Compatibility
   ↓
Backup
   ↓
Staging
   ↓
Control Plane 1.37
   ↓
Smoke Tests
   ↓
Bake Period
   ↓
Node Groups
   ↓
Add-ons
   ↓
Final validation

EKS also lets you return the control plane to the previous minor version during the seven days following the upgrade, as long as the rollback requirements are met. This reduces operational risk, but it does not automatically roll back Managed Node Groups, add-ons or workloads.

1. What upgrading EKS means

Amazon EKS splits the cluster's responsibility into two main layers.

Control plane

Managed by AWS, it contains components such as:

• kube-apiserver;
• scheduler;
• controller manager;
• managed etcd.

When we run:

aws eks update-cluster-version \
  --name production \
  --kubernetes-version 1.37 \
  --region us-east-1

we are mainly upgrading this layer.

Data plane

It remains the platform's responsibility and includes:

• Managed Node Groups;
• self-managed nodes;
• nodes provisioned by Karpenter;
• Fargate;
• EKS Auto Mode;
• applications;
• operators;
• add-ons.

Therefore:

Control Plane = 1.37

does not automatically mean:

Nodes = 1.37
CoreDNS = upgraded
VPC CNI = upgraded
CSI = upgraded
Controllers = compatible

This separation is central to planning both the upgrade and the rollback.

2. Version skew and upgrade order

Kubernetes allows a certain amount of version skew.

With kube-apiserver 1.37, the kubelet can be on:

1.37
1.36
1.35
1.34

as long as the other version skew rules are respected.

kubectl, in turn, should generally stay within one minor version above or below the API Server.

This enables an important strategy:

Control Plane: 1.37
Nodes:         1.36

for a controlled period.

This interval works as a bake period: the new control plane receives real traffic while the nodes remain on the previous version.

The upgrade must also follow consecutive versions:

1.35 → 1.36 → 1.37

and not:

1.35 → 1.37

3. What to review specifically for Kubernetes 1.37

Besides the EKS release notes, review the upstream Kubernetes changes and the dependencies installed in the cluster.

This mainly includes:

VPC CNI
CoreDNS
kube-proxy
EBS/EFS CSI
AWS Load Balancer Controller
Karpenter
KEDA
cert-manager
Kyverno
External Secrets
Prometheus Operator
StackGres
Argo CD
Traefik
service meshes
internal controllers

The question should not be only:

Does Kubernetes 1.37 work?

But:

Do the versions of the components installed in my cluster support Kubernetes 1.37?

List CRDs and admission webhooks:

kubectl get crd

kubectl get apiservices

kubectl get validatingwebhookconfigurations

kubectl get mutatingwebhookconfigurations

Old controllers are a common source of problems because they can keep working normally on the current Kubernetes and only fail after the API changes.

4. Pre-upgrade checklist

Before the change window, record the current state.

Cluster and nodes

kubectl version

kubectl get nodes \
  -o custom-columns='NODE:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,OS:.status.nodeInfo.osImage,RUNTIME:.status.nodeInfo.containerRuntimeVersion'

Check EKS:

aws eks describe-cluster \
  --name production \
  --region us-east-1 \
  --query 'cluster.{Version:version,Status:status,Platform:platformVersion}'

The cluster must be:

Status: ACTIVE

Upgrade Insights

Run:

aws eks list-insights \
  --cluster-name production \
  --region us-east-1

Upgrade Insights help identify conditions such as incompatible APIs and issues related to component versions. AWS keeps a history of deprecated API usage over a rolling window, so fixing an API does not necessarily mean the warning will disappear immediately.

After fixing issues:

aws eks start-insights-refresh \
  --cluster-name production \
  --region us-east-1

Add-ons

List them:

aws eks list-addons \
  --cluster-name production \
  --region us-east-1

Check compatibility individually:

aws eks describe-addon-versions \
  --kubernetes-version 1.37 \
  --addon-name vpc-cni

Repeat for:

vpc-cni
coredns
kube-proxy
aws-ebs-csi-driver

Always look up versions at the time of the change instead of pinning numbers taken from old articles.

Capacity

Make sure there is enough room for:

• new nodes to temporarily coexist with old ones;
• Pods to be rescheduled;
• DaemonSets to start;
• volumes to be remounted;
• autoscaling to respond.

Also review IP availability in the cluster subnets.

5. Backup and recovery strategy

Kubernetes rollback is not a backup.

Before the upgrade, confirm protection for:

Kubernetes objects
CRDs
ConfigMaps
Secrets
PVCs
EBS
EFS
PostgreSQL
RabbitMQ
other stateful data

Databases must have their own strategy.

For example:

PostgreSQL
   ↓
physical/logical backup
   ↓
external storage
   ↓
tested restore

The most important part is not knowing that a backup exists, but knowing that it can be restored.

A control plane rollback preserves the current etcd state; it does not automatically turn the cluster into a snapshot from before the change.

6. Preparing workloads for the upgrade

A production workload must tolerate gradual node replacement.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: production
spec:
  replicas: 3

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

  selector:
    matchLabels:
      app: payments-api

  template:
    metadata:
      labels:
        app: payments-api

    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: payments-api

      containers:
        - name: api
          image: example/payments-api:2.4.1

          ports:
            - containerPort: 8080

          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi

          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5

          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 20
            periodSeconds: 10

          startupProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5
            failureThreshold: 30

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payments-api

In this example:

• three replicas reduce the impact of Pod replacement;
• maxUnavailable: 0 keeps capacity during rollouts;
• topologySpreadConstraints spreads the replicas;
• probes keep traffic away from unavailable containers;
• the PodDisruptionBudget (PDB) keeps at least two replicas during voluntary disruptions.

The values must be tuned for each workload.

A common mistake is creating:

minAvailable: 100%

without considering that it can block node drains.

7. Testing before production

Staging must reproduce the relevant production behavior.

Validate at least:

Test the drain process:

kubectl drain <node> \
  --ignore-daemonsets \
  --delete-emptydir-data

Then:

kubectl uncordon <node>

This test usually reveals PDB problems, single-replica workloads, StatefulSets and node dependencies that go unnoticed in superficial checks.

8. Control plane upgrade and bake period

With the preflight complete:

aws eks update-cluster-version \
  --name production \
  --kubernetes-version 1.37 \
  --region us-east-1

The operation returns an update.id.

Monitor it:

aws eks describe-update \
  --name production \
  --region us-east-1 \
  --update-id <UPDATE_ID>

Wait for:

Successful

Then:

kubectl version
kubectl get nodes
kubectl get pods -A

At this point, temporarily keeping:

Control plane: 1.37
Nodes:         1.36

may be intentional.

Bake period

Do not move straight on to nodes and add-ons.

One possible flow:

T0       Control plane 1.37
T+15m    smoke tests
T+1h     metrics and logs
T+4h     behavior under real traffic
T+24h    data plane decision

The duration depends on how critical the platform is.

During this period, avoid introducing features that depend exclusively on Kubernetes 1.37, as that can make an eventual return to 1.36 harder.

9. Upgrading nodes and add-ons

After the bake period, upgrade the Managed Node Groups progressively.

aws eks list-nodegroups \
  --cluster-name production \
  --region us-east-1

Example:

aws eks update-nodegroup-version \
  --cluster-name production \
  --nodegroup-name prod-app \
  --kubernetes-version 1.37 \
  --region us-east-1

Do not upgrade them all at the same time.

One strategy could be:

canary pool
   ↓
less critical workloads
   ↓
prod-app
   ↓
special/stateful nodes

Then move the required add-ons forward:

aws eks describe-addon-versions \
  --kubernetes-version 1.37 \
  --addon-name coredns

and:

aws eks update-addon \
  --cluster-name production \
  --addon-name coredns \
  --addon-version <COMPATIBLE_VERSION> \
  --region us-east-1

Make one change at a time.

Avoid upgrading simultaneously:

Control Plane
+ Nodes
+ CNI
+ DNS
+ CSI
+ Ingress
+ Service Mesh

If there is a regression, finding its source becomes much harder.

10. Validation and observability

After each step:

kubectl get nodes
kubectl get pods -A
kubectl get deployments -A
kubectl get statefulsets -A
kubectl get daemonsets -A
kubectl get pdb -A
kubectl get apiservices

Events:

kubectl get events -A \
  --sort-by=.metadata.creationTimestamp

Pods not in the expected state:

kubectl get pods -A \
  --field-selector=status.phase!=Running,status.phase!=Succeeded

But Running does not mean the application is healthy.

Also track:

HTTP 5xx
p95/p99 latency
restarts
Pending Pods
CPU/memory
CPU throttling
DNS errors
queue backlog
database connections
controller errors
SLO burn rate

With Prometheus:

sum(rate(kube_pod_container_status_restarts_total[5m]))

count(kube_pod_status_phase{phase="Pending"} == 1)

With Loki, look for patterns such as:

timeout
connection refused
failed mount
failed scheduling
forbidden
admission webhook
context deadline exceeded

If you have OpenTelemetry, compare pre- and post-upgrade traces to find regressions in dependencies that don't show up immediately in the infrastructure.

The ideal is to work with:

baseline before the upgrade
          ↓
comparison
          ↓
post-upgrade data

and not just look at isolated values.

11. Rolling back to EKS 1.36

After an in-place upgrade:

1.36 → 1.37

EKS lets you request:

1.37 → 1.36

during a seven-day window.

The rollback is limited to one minor version and depends on the cluster's eligibility conditions.

What does not roll back automatically

The rollback does not revert:

etcd data
workloads
EKS add-ons
persistent volumes
self-managed nodes
Managed Node Groups
hybrid nodes

That is why there is a difference between a control plane rollback and a whole-platform rollback.

Rollback Readiness

After the upgrade:

aws eks list-insights \
  --cluster-name production \
  --region us-east-1 \
  --filter '{"categories":["ROLLBACK_READINESS"]}'

The checks analyze, among other things:

• API compatibility;
• cluster health;
• kubelet version skew;
• kube-proxy version skew;
• add-on compatibility.

The insights remain available during the rollback eligibility window.

12. Rollback strategies

Scenario A — nodes are still on 1.36

Control plane: 1.37
Nodes:         1.36

This is the simplest scenario.

After evaluating Rollback Readiness:

aws eks update-cluster-version \
  --name production \
  --kubernetes-version 1.36 \
  --region us-east-1

This is one of the main benefits of keeping a bake period before upgrading the data plane.

Scenario B — nodes have already moved to 1.37

Control plane: 1.37
Nodes:         1.37

Since the kubelet must not be newer than the API Server, first roll the Managed Node Groups back:

aws eks update-nodegroup-version \
  --cluster-name production \
  --nodegroup-name prod-app \
  --kubernetes-version 1.36 \
  --region us-east-1

Then validate:

kubectl get nodes

Review the add-ons as well.

If an upgraded add-on does not support 1.36, select a compatible version before returning the control plane.

Finally:

aws eks update-cluster-version \
  --name production \
  --kubernetes-version 1.36 \
  --region us-east-1

13. When to use --force

You can run a rollback using:

aws eks update-cluster-version \
  --name production \
  --kubernetes-version 1.36 \
  --force \
  --region us-east-1

But --force only bypasses certain Rollback Insights checks.

It does not bypass hard requirements, such as:

• the seven-day window having expired;
• the cluster having originally been created on the current version;
• rolling back more than one minor version;
• incompatibility with certain EKS features.

AWS explicitly states that it cannot guarantee rollback safety when checks are bypassed.

In production, --force should be used as an exceptional incident measure, not as a standard procedure.

14. Security and FinOps during the upgrade

The change is also an opportunity to review:

kubectl get clusterrolebindings
kubectl get rolebindings -A
kubectl get networkpolicy -A

Check:

• least-privilege RBAC;
• ServiceAccounts;
• IAM Roles for Service Accounts or EKS Pod Identity;
• NetworkPolicies;
• Secrets;
• Pod Security;
• seccomp;
• node role permissions;
• admission policies.

From a cost perspective, node upgrades usually create a coexistence period:

old nodes
+
new nodes

So account for temporary capacity and cost.

Oversized requests also amplify the impact:

requests:
  cpu: "4"
  memory: 8Gi

when the workload consumes, for example:

500m CPU
1.5 GiB RAM

can force the cluster to provision far more capacity than it really needs while nodes are being replaced.

The goal is to balance:

rollout speed
x
temporary capacity
x
risk of downtime

15. Troubleshooting

Node does not become Ready

kubectl describe node <node>

kubectl get events -A \
  --sort-by=.metadata.creationTimestamp

Check:

AMI
container runtime
kubelet
bootstrap
IAM
VPC CNI
security groups
DNS

Pods stuck in ContainerCreating

kubectl describe pod <pod> -n <namespace>

Look mainly for problems with:

volume mount
CSI
CNI
Secrets
ConfigMaps
SELinux

Node Group cannot drain nodes

kubectl get pdb -A
kubectl describe pdb <pdb> -n <namespace>

Look for overly restrictive PDBs.

DNS

kubectl get pods \
  -n kube-system \
  -l k8s-app=kube-dns

Test:

kubectl run dns-test \
  --image=busybox:1.36 \
  --restart=Never \
  -- nslookup kubernetes.default.svc.cluster.local

Admission Webhooks

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Controllers such as Kyverno, cert-manager and Gatekeeper can block changes in the cluster even when the API Server is healthy.

16. Development vs. production

In development it may be acceptable to have:

fewer replicas
short window
faster upgrade
simplified backup

In production, expect at least:

high availability
PodDisruptionBudget
readiness/liveness/startup probes
topology spread
capacity headroom
tested backup and restore
SLOs
smoke tests
bake period
Rollback Readiness
observability
rollback plan

The main difference is the need for diagnosis and reversibility.

17. In-place or blue/green?

An in-place upgrade works well when the cluster has:

• reliable IaC;
• a representative staging environment;
• good observability;
• validated controllers;
• test automation;
• backups;
• a rollback procedure.

On the other hand, consider creating a new cluster when the upgrade also involves deep changes such as:

new VPC
new CNI
Ingress replacement
Service Mesh replacement
new ARM/x86 architecture
operating system change
storage strategy change
multiple Kubernetes versions

In that scenario:

Old EKS
   ↓
New EKS 1.37
   ↓
Deploy
   ↓
Smoke tests
   ↓
Canary
   ↓
Traffic migration

Blue/green has a temporarily higher cost, but it reduces the coupling between the Kubernetes upgrade and structural changes.

18. Final operational checklist

Before

[ ] Current cluster validated
[ ] Nodes aligned with the current version
[ ] Cluster ACTIVE
[ ] Upgrade Insights reviewed
[ ] Deprecated APIs analyzed
[ ] CRDs and webhooks validated
[ ] Operators compatible with 1.37
[ ] Add-ons compatible
[ ] Backup executed
[ ] Restore validated
[ ] PDBs reviewed
[ ] Probes reviewed
[ ] Capacity/IPs available
[ ] Staging tested
[ ] Observability baseline recorded

Upgrade

[ ] Control plane → 1.37
[ ] Smoke tests
[ ] Metrics/logs/traces
[ ] Bake period
[ ] Rollback Readiness
[ ] Node Groups progressively → 1.37
[ ] Required add-ons

After

[ ] DNS validated
[ ] Storage validated
[ ] Autoscaling validated
[ ] Controllers reconciling
[ ] CI/CD working
[ ] SLOs within baseline
[ ] Backlog of outdated components recorded

Practical summary

The upgrade can be summarized as:

Preflight
   ↓
Backup
   ↓
Staging
   ↓
Control Plane 1.37
   ↓
Smoke Tests
   ↓
Bake Period
   ↓
Rollback Readiness
   ↓
Nodes
   ↓
Add-ons
   ↓
SLO Validation

If there is a regression:

Nodes still on 1.36?
      │
      ├── Yes
      │    ↓
      │ Roll back control plane
      │
      └── No
           ↓
       Roll back nodes
           ↓
       Review add-ons
           ↓
       Roll back control plane

Conclusion

An EKS upgrade does not end when the AWS API returns Successful.

The process can only be considered complete when applications, networking, storage, controllers, autoscaling, observability and SLOs keep working within the expected parameters.

Being able to roll back to the previous minor version for seven days significantly improves the reversibility of Amazon EKS upgrades, but it does not replace testing, backups or a gradual strategy. Managed Node Groups and add-ons still require independent planning.

In a mature Platform Engineering operation, the upgrade stops being a manual change and becomes a repeatable process:

Compatibility → Test → Upgrade → Observe → Expand → Validate
                           │
                           └── if needed → Rollback

This approach reduces blast radius, improves troubleshooting and turns Kubernetes upgrades into changes controlled by evidence, observability and reversibility.