Hidden EKS costs: NAT Gateway, cross-AZ traffic and forgotten EBS volumes

When an EKS bill goes up, the first reaction is to look at the EC2 instances. Often, though, a relevant part of the increase sits in three items that do not show up as "the cluster": NAT Gateway, cross-availability-zone traffic and EBS volumes nobody uses anymore.

1. NAT Gateway

A NAT Gateway charges in two ways: per hour it exists and per GB processed. In us-east-1 both list prices are $0.045, and in other Regions, such as sa-east-1, they are higher. With one NAT per zone, the fixed cost exists before any traffic, and the per-GB cost grows with everything pods send and receive over the internet and to AWS services.

What tends to surprise people is this: without endpoints, traffic from a pod to S3 or DynamoDB leaves through the NAT and is billed per GB, even though it stays inside AWS. The same goes for pulling images from ECR, because image layers live in S3.

What to fix:

• Gateway endpoint for S3 and DynamoDB. It has no cost and takes that traffic off the NAT. It is the best-return fix.
• Interface endpoints for ECR (api and dkr), STS and CloudWatch Logs, when volume justifies them. They charge per hour per zone plus a per-GB rate lower than the NAT's, so they only pay off above some volume. The calculator does that math and shows when an endpoint does not pay for itself.
• Image caching and fewer pulls: a large image pulled onto many new nodes, for example after a consolidation, becomes traffic.

2. Cross-availability-zone traffic

AWS charges for data that crosses zones within a Region, per GB and in each direction. In a cluster with nodes in three zones, a Service spreads requests across every replica, so a good share of service-to-service calls cross zones without anyone deciding that.

What to fix:

• Proximity routing. Kubernetes can prefer same-zone endpoints with the spec.trafficDistribution field, which became stable in version 1.33. The current value is PreferSameZone, and PreferClose is an older alias that the docs now mark as deprecated. On 1.33 clusters, use PreferClose and check whether PreferSameZone exists on your version. The service.kubernetes.io/topology-mode: Auto annotation does the same and, if set, takes precedence over the field.
• Well-spread replicas. Proximity routing only works if each zone has enough replicas. Use topologySpreadConstraints, or one zone ends up taking all the load from the others.
• Load balancers. An NLB with cross-zone enabled charges for inter-zone transfer, and an ALB does not. Check what is turned on.
• Databases and queues. Cross-zone replication may be required for availability. In that case the cost is the price of that guarantee, and should not be cut without understanding the risk.

Not all cross-zone traffic is avoidable, which is why the calculator assumes about half can stay local.

3. Forgotten EBS volumes

Unattached volumes keep charging per GB-month. In Kubernetes they come from PersistentVolumeClaims deleted with reclaimPolicy: Retain, from removed StatefulSets and from old nodes that left disks behind. Two adjustments usually pay off:

• Delete what is `available`, after confirming the owner and keeping a snapshot where it makes sense.
• Move gp2 volumes to gp3. gp3 costs less per GB and the migration can be done without downtime.

How to find each one

In Cost Explorer, group by usage type and look for:

• NatGateway-Bytes and NatGateway-Hours, for the NAT;
• DataTransfer-Regional-Bytes, for cross-zone traffic;
• EBS:VolumeUsage by volume type, and in EC2 the volumes in the available state.

VPC Flow Logs show which sources and destinations generate the NAT traffic.

A one-week plan

1. Run the query above and write down the three numbers.
2. Create the gateway endpoint for S3 and DynamoDB.
3. List the available volumes and delete the ones with a confirmed owner.
4. Turn on proximity routing on one high-traffic service, with topologySpreadConstraints, and measure.
5. Put the numbers into the hidden costs calculator to prioritize the rest.

These costs sit next to compute. To add up rightsizing, Spot and Graviton, use the savings calculator, and see how to combine Spot and Savings Plans. If you want me to look for these items in your environment, the efficiency assessment is the place to start.

Want the full guide, with commands, Terraform and YAML snippets, a worked scenario and a checklist? Download the free Hidden EKS costs e-book.