What Is Canary Deployment? A Dev's Practical Guide

· · Tech stack decisions
Summary

A canary deployment sends a new version to a small share of real traffic, often 1% to 5%, before everyone gets it. You compare error rate and latency against the stable version, widen the slice if it holds, and set the canary weight back to zero if it does not. It limits the damage of a bad release, but it needs per-version metrics and enough traffic to mean something.

A dark developer desk at night with a dashboard of traffic graphs and a small yellow canary figurine

What is canary deployment? It is a release where the new version of your app gets a small slice of real traffic first, say 1% to 5%, while everyone else stays on the old version. You watch error rates and latency on that slice. If it looks healthy you widen it. If it looks bad you send the traffic back and almost nobody noticed. That is the whole idea, and everything below is about making it work in practice.

An old miner's lantern and empty birdcage on a workbench, the origin of the canary name

Why is it called a canary?

Coal miners carried canaries underground because the birds reacted to toxic gas before people did. A canary that stopped singing was the signal to get out. The deployment version works the same way: a small group of users hits the new code first and acts as the early warning. The CNCF glossary describes it as two environments, one serving live traffic and one running the updated code, with traffic shifted gradually, starting at a small share such as 1%.

Here is what gets in the way in practice: the canary users are real users. Staging cannot give you that. Production has the weird browsers, the stale caches, the one customer with a 40 MB import file. A canary is how you meet them before all of them meet your bug.

How does a canary deployment actually work?

Four moving parts, nothing exotic.

A typical ramp looks like 1%, 5%, 25%, 50%, 100%, with a pause at each step. Google Cloud Deploy documents the same shape: you define a series of percentages and the tool apportions traffic, with an example using 25%, 50% and 75% phases before the final stable one. The Cloud Deploy canary docs also show that each phase can run a verify job and an analysis step to decide whether to advance.

It is not a perfect setup. It is a doable one. You do not need Kubernetes to do this. A weighted route in nginx, a Cloudflare worker that flips a cookie, or two Fly.io machines behind one proxy all count.

What does the traffic split look like in a config?

Here is the smallest useful example, an nginx upstream with weights. Nine requests in ten go to stable, one goes to the canary.

upstream app {
    server 10.0.0.10:3000 weight=9;   # stable
    server 10.0.0.11:3000 weight=1;   # canary
}

This has a flaw you should know about: the split is per request, so one user can bounce between versions on consecutive clicks. If your new version changes an API response shape or a database column, that user will have a bad time. Sticky routing by cookie or user ID fixes it, and it is the first thing worth adding once the basic split works.

Laptop showing two diverging metric lines comparing a stable and a canary release

What should you watch while the canary runs?

Compare the canary against the stable version at the same moment, not against last week. Last week had different traffic. The question is always: is the canary worse than the version serving the other 95% right now?

The mistake we see most with small teams is ramping on a timer. Five minutes at 1%, then 25%, regardless of what the graphs say. With low traffic, five minutes at 1% might be six requests. That tells you nothing. Wait for a minimum request count, not a minimum number of minutes.

Tools that make this less manual are worth a look once you have real traffic.

Canary vs blue-green: which one should you pick?

Blue-green keeps two full environments and flips all traffic at once. Rollback is flipping it back, which is fast and clean. The cost is that for a moment 100% of users are on the new version, and you pay for double the infrastructure during the switch.

Canary exposes only a slice, so the blast radius is smaller. The price is complexity: you need traffic weighting, per-version metrics and a decision rule. Rollback means setting the canary weight to zero, which is quick but not literally instant if connections need to drain.

Three reasons to go canary, one reason not to. Choose canary when a bug in production is expensive, when you have enough traffic for a small slice to mean something, and when you can compare metrics per version. Skip it when you have a few dozen users a day: the canary slice will be statistically empty and you will be guessing. At that scale, blue-green or even a plain deploy with a fast rollback script is the honest answer.

Rolling updates sit in the middle. They replace instances one by one, which is gentle, but they are not a controlled experiment: you do not choose who sees the new version or compare cohorts.

Where do feature flags fit in?

A canary deployment splits traffic between versions of your code. A feature flag splits behaviour inside one version. People mix these up, and the confusion is expensive.

Deploy the new code to everyone, hidden behind a flag that is off. Then turn the flag on for 1% of users, then 10%, then everyone. This is often called a canary release at the feature level. The advantage: rollback is flipping a flag, no redeploy. The disadvantage: the dead code path is in production for everyone, and flags left behind turn into technical debt.

For a solo dev, flags are usually the cheaper first step. A boolean in a config table and a hash of the user ID gets you 80% of the value. If you outgrow that, a managed service handles targeting, audit logs and gradual percentage rollouts.

A rotary dial turned up slightly, like shifting a small share of traffic to a new version

What goes wrong with canaries in real projects?

Tested. Not optimal. Here is why we still do it. These are the failures that actually bite.

Database migrations. Both versions hit the same database at once. If the canary drops or renames a column, stable breaks. The fix is the expand and contract pattern: add the new column first, ship code that handles both, and only remove the old one in a later release. Skip this and your canary takes down the stable version.

Stateful sessions. If sessions live in memory on one instance, a user bounced between versions gets logged out. Move session state to a shared store before you try canaries.

Cache poisoning. A new version that writes a different cache format can corrupt what the old version reads. Version your cache keys.

Background jobs. The canary also consumes queue messages. A buggy worker can chew through jobs at full speed, because the traffic percentage does not limit queue consumption. Give the canary its own queue or consumer weight.

Alerts nobody wired up. The canary can fail loudly in a dashboard nobody opens. Automatic abort on a threshold beats a human watching, especially at 2am in Taipei or Bangkok when the only person awake is the one who deployed.

How do you start with a canary on a side project?

Start smaller than you think you should. You are one dev, not a platform team.

  1. Label every metric with a version. Put the git SHA in a header and in your logs. Without this you cannot compare canary and stable at all.

  2. Pick one abort rule. For example: error rate on the canary above 2x stable for five minutes means roll back. One rule, written down.

  3. Add a weighted route. Nginx weights, a Cloudflare rule, or your host's traffic splitting feature. Most managed platforms now offer percentage-based routing without extra tooling.

  4. Ship something boring first. A copy change, not a pricing page rewrite. Practise the ramp and the rollback when nothing is at stake.

  5. Time the rollback. Actually run it once. If it takes ten minutes of fumbling, that is the thing to fix, not the ramp.

If your project is still at the stage of finding out who pays for it, this is probably premature. Ship plainly, keep a rollback script, and come back to canaries when an outage would cost you real customers. If it is already there, the weighted route plus one abort rule is a weekend of work and it pays back the first time a bad release dies at 1% instead of 100%.

Should you canary everything?

No. Config changes, copy tweaks and fixes to a typo in a footer do not need a ramp. Use canaries for the changes where being wrong costs money or trust: payments, auth, data models, anything touching the hot path. A pipeline that ramps every commit for an hour will get bypassed the first time someone is in a hurry, and a bypassed process is worse than no process.

So what is the riskiest change in your next release, and what would it take to send it to 1% of users first?

Frequently asked questions

What is a canary deployment in simple terms?
It is a release method where a new version first serves a small percentage of real users while the rest stay on the old version. If metrics stay healthy, the share grows until the new version serves everyone. If they degrade, traffic goes back to the stable version.
What percentage of traffic should a canary start with?
Many teams start around 1% to 5%. The right number depends on volume: the slice must produce enough requests to compare error rates and latency. With very low traffic, wait for a minimum request count rather than a fixed number of minutes.
What is the difference between canary and blue-green deployment?
Blue-green runs two full environments and switches all traffic at once, so rollback is flipping back. Canary shifts traffic gradually, so fewer users are exposed to a bad release, but it needs traffic weighting and per-version monitoring.
Is a canary deployment the same as a feature flag?
No. A canary splits traffic between two deployed versions of the code. A feature flag turns behaviour on or off inside one version. They combine well: deploy dark behind a flag, then enable it for a small percentage of users.
Do you need Kubernetes to run canary deployments?
No. Weighted upstreams in nginx, a CDN or edge rule, or traffic splitting on a managed host can all do it. Kubernetes tooling makes ramps and automatic analysis easier, but it is not required.
How do database migrations work with canary releases?
Both versions share one database, so use the expand and contract pattern: add new columns first, ship code that handles both shapes, and remove old columns only in a later release.
~/build
Generate an idea