Upgrades Are a Feature: Designing AKS Clusters That Survive Day 2

Every AKS cluster has a clock on it. Kubernetes ships a new minor version every few months, Azure supports a rolling window of recent versions, and the moment you fall off the back of that window you're running unsupported infrastructure underneath your production traffic. The clock doesn't care that you're busy.

So the real question isn't whether you upgrade. It's whether upgrading is a Tuesday or a trauma. That outcome is decided long before the upgrade starts, in how the cluster was designed.

What an upgrade actually does to you

An AKS upgrade isn't a magic in-place swap. To upgrade a node pool, AKS brings up a surge node on the new version, cordons and drains an old node, evicting its pods so they reschedule elsewhere, then deletes the old node and repeats. The control-plane upgrade is Azure's problem. The node upgrade is a controlled, rolling eviction of your workloads, and your workloads get a vote on how that goes.

If your pods can be evicted and rescheduled without anyone noticing, the upgrade is a non-event. If they can't, the upgrade is an outage you scheduled yourself.

An upgrade is just your disaster-recovery design being tested on a calendar instead of by surprise. Most "upgrade outages" are really resilience bugs that were always there.

Designing for the drain

Everything that makes a rolling upgrade safe also makes a random node failure safe, which is the point.

• Run more than one replica, spread across nodes. A singleton pod cannot be drained without downtime. Multiple replicas with anti-affinity or topology spread can lose a node and shrug.
• Set PodDisruptionBudgets. A PDB tells the drain "you may take pods from this app, but never below N available." Without it, a drain can evict every replica at once. It's a handful of lines that converts a possible outage into a paced rollout. (There's a whole short post on just this. It earns its keep.)
• Handle SIGTERM like you mean it. When a pod is evicted it gets a termination signal and a grace period. Apps that ignore it, dropping in-flight requests instead of draining connections, turn every eviction into a fistful of errors.
• Keep your APIs current. Kubernetes removes deprecated APIs on a schedule. A manifest using a since-removed API version will fail after the upgrade, in a way that looks unrelated. Audit for deprecated APIs before you bump the version, not after.

Make it boring on purpose

• Upgrade a non-prod cluster on the same version path first. Surprises are cheaper there.
• Stay in the supported window deliberately. Falling behind turns one calm hop into several urgent ones, often during an incident you didn't choose.
• Decide your automation posture. Auto-upgrade channels keep you current with less toil; manual upgrades give you control at the cost of vigilance. Either is fine. Drifting with no posture at all is the failure mode.

The takeaway

The teams who dread AKS upgrades are usually running clusters that were never designed to lose a node gracefully, and an upgrade is just losing nodes, on purpose, in sequence. Build for the drain and you get two things for the price of one: upgrades that feel like maintenance, and a cluster that survives the node failures you didn't schedule. Upgradability isn't overhead. It's resilience you can see.