← Back to work

Making risky releases boringly routine on AKS.

A canary release strategy with percentage-based traffic splitting for containerized production workloads on Azure Kubernetes Service. The previous version stays deployed and warm, so rolling back is a traffic switch measured in seconds, not a redeploy.

Context

NextGen Technologies, production workloads

Stack

AKS · Kubernetes · Azure DevOps · ACR · Azure Monitor

Year

2025 - present

release vN+1 canary at 10% observe promote to 100% · vN kept on standby

The mission

Every release was all-or-nothing

Deploys to containerized production workloads replaced everything at once: a bad release meant every user felt it immediately, and rolling back meant a scramble to rebuild and redeploy the previous version while the incident clock ran.

I introduced canary releases on AKS. Each new version takes a small, configurable slice of live traffic first, and only takes over fully after proving itself against real users, with the outgoing version held warm the entire time.

What I built

The work

Challenges

Problems along the way

Full blast radius on day one

Rolling updates replaced every pod regardless of how the new version behaved under real traffic.

Fixed: weighted traffic splitting, so new versions serve ~10% of requests first, promoted only once error rates and latency hold within thresholds.

Rollbacks were slow and manual

Reverting meant redeploying the previous image, which cost minutes of degraded service every time.

Fixed: the prior version stays deployed and scaled; rollback became an immediate traffic-weight change measured in seconds.

Secrets and access sprawl

Long-lived credentials and broad access made security incidents slow to contain.

Fixed: RBAC governance and automated secret rotation via Key Vault, cutting security-incident response time by 30%.

Outcome

10%

initial blast radius for new releases

Seconds

to roll back via traffic switch

30%

faster security-incident response

18%

lower monthly spend, SLAs intact

← Back to work Next: End-to-end CI/CD pipelines