Deployment Strategies: Rolling, Blue-Green, Canary and Rollbacks

An online grocery store is about to replace its checkout page. The old one works but converts poorly. The new one has been tested, reviewed and demoed. Is it safe to ship to everyone at 10 a.m. on a Saturday? Tests can say the code behaves as specified. They cannot say how it behaves under real traffic, real data and real customers' odd browsers. Deployment strategies are different ways of limiting the damage of being wrong.

Four ways to put new code in front of users

Choosing is a risk decision. A rolling update with readiness checks is enough for most internal services. Blue-green suits releases where you want to test the complete new environment before anyone sees it. Canary is best when a bad release would be expensive and you have the metrics to judge it quickly. For how rolling updates are configured on a cluster, see the Kubernetes course.

Blue-green on Kubernetes: switching the selector

A common way to do blue-green on Kubernetes is to run two Deployments labelled version: blue and version: green, and let a Service pick which one receives traffic through its label selector. The switch is a one-line change, and the rollback is the same change in reverse.

# 1. Deploy the new version next to the live one (not receiving traffic yet)
kubectl apply -f checkout-green.yaml
kubectl rollout status deployment/checkout-green

# 2. Smoke-test green directly, without touching users
kubectl port-forward deployment/checkout-green 9090:8080 &
curl -fsS http://localhost:9090/health

# 3. Switch the Service selector from blue to green
kubectl patch service checkout \
  -p '{"spec":{"selector":{"app":"checkout","version":"green"}}}'

# 4. Roll back instantly if metrics look wrong
kubectl patch service checkout \
  -p '{"spec":{"selector":{"app":"checkout","version":"blue"}}}'

Keep blue running until you are confident, then scale it down. The cost is the temporary double footprint. Note that a switch cannot hide an incompatible database change, which is the topic of the rollback section below.

Feature flags: deploy is not release

A feature flag is a runtime condition that decides whether a code path is active. With flags, you can merge and deploy unfinished or risky code switched off, then enable it for staff, then for a percentage of users, then for everyone, with no new deployment at each step. Turning it off is also instant. The percentage rollout needs one property above all: a given user must always land in the same bucket, otherwise customers see the old and new checkout alternately on every page load. Hashing the flag name and user id gives a stable bucket.

import { createHash } from "node:crypto";

// Deterministic bucket 0..99 from the flag name and the user id.
function bucket(flagName, userId) {
  const digest = createHash("sha256").update(`${flagName}:${userId}`).digest();
  return digest.readUInt32BE(0) % 100;
}

function isEnabled(flag, userId) {
  if (!flag.enabled) return false;
  if (flag.allowList?.includes(userId)) return true;
  return bucket(flag.name, userId) < flag.percent;
}

const newCheckout = { name: "new-checkout", enabled: true, percent: 25, allowList: ["qa-bot"] };
const users = ["qa-bot", "shopper-1", "shopper-2", "shopper-4", "shopper-6", "shopper-7", "shopper-8"];

for (const id of users) {
  const b = String(bucket(newCheckout.name, id)).padStart(2);
  console.log(id.padEnd(10), "bucket", b, "->", isEnabled(newCheckout, id));
}

// Raising the percentage only ever adds users: nobody flips back and forth.
const wider = { ...newCheckout, percent: 50 };
const at25 = users.filter((u) => isEnabled(newCheckout, u));
const at50 = users.filter((u) => isEnabled(wider, u));
console.log("at 25%:", at25.join(", "));
console.log("at 50%:", at50.join(", "));
console.log("superset:", at25.every((u) => at50.includes(u)));

qa-bot     bucket 50 -> true
shopper-1  bucket 83 -> false
shopper-2  bucket 30 -> false
shopper-4  bucket 46 -> false
shopper-6  bucket 15 -> true
shopper-7  bucket  2 -> true
shopper-8  bucket 97 -> false
at 25%: qa-bot, shopper-6, shopper-7
at 50%: qa-bot, shopper-2, shopper-4, shopper-6, shopper-7
superset: true

Save it as flag.mjs and run node flag.mjs. The QA account is enabled through the allow list despite its bucket of 50, and raising the rollout from 25 to 50 percent only adds shoppers. Real systems use a flag service or library rather than a hand-rolled one, but this is the mechanism inside them. The downside of flags is clutter: every flag is a branch in your code that must be tested both ways and removed once the rollout is complete.

Rollbacks: the part you plan before you need it

• Rolling back code is easy if artifacts are immutable. If you deployed image checkout:3.4.1 and the previous was 3.4.0, rollback is deploying 3.4.0 again. If you used latest, you do not actually know what to go back to. On Kubernetes, kubectl rollout undo deployment/checkout restores the previous template.
• Rolling back data is hard. If version 3.4.1 renamed a column and 3.4.0 expects the old name, the code rollback fails at the first query. The standard remedy is expand and contract: first add the new column while keeping the old one (expand), deploy code that writes both and reads the new one, and only in a later release remove the old column (contract). Every release is then compatible with the one before it.
• Prefer rolling forward for small bugs. A tiny fix through a fast pipeline can be quicker and safer than a rollback, if the pipeline is fast enough and the problem is understood. Roll back when customers are hurting and the cause is unclear.
• Define the trigger in advance. Decide which signals (error rate, latency, checkout completion rate) and which thresholds make you roll back, before the release. Otherwise a 3 a.m. debate decides it.

Common mistakes and the symptoms they cause

• Incompatible database and code versions in a rolling update. Symptom: errors only while some instances are old and some new, such as column does not exist from the old pods. Fix: expand/contract migrations.
• Canary with no metrics. Symptom: 5 percent of users see a bug and nobody knows for a day. Fix: define dashboards and automatic abort rules first.
• Sticky sessions or per-request randomness in canaries. Symptom: users flip between versions and report inconsistent behaviour. Fix: route by a stable key such as user id.
• Flags that never get removed. Symptom: code full of dead branches and a flag nobody dares touch. Fix: put a removal ticket and date on every flag.
• Rollback that depends on a rebuild. Symptom: the rollback pipeline takes twenty minutes or fails because a dependency changed. Fix: keep previous artifacts and redeploy them.

How this shows up in real projects and interviews

A standard interview question is "how would you release a risky change with minimal user impact?" A good answer combines a flag or canary with named metrics, a stated rollback trigger and a database compatibility plan. Be honest that each approach costs something: infrastructure, complexity or time. In practice, most teams use rolling updates by default, flags for large features, and reserve blue-green or canary for the services where an outage is expensive.