JS-260 · Lesson 11 of 12 · 25 min

Releases, rollbacks and versioning

▶ Video coming soon A video version of this lesson is on the way.

On Tuesday, November 7, Gareth starts by asking a question Theo could never answer quickly: what is running in production right now, and what changed since last week? With the pipeline, the first half has an answer, because every release folder and image is named after a version and a commit. The second half needs a habit. Today's lesson is about releases: giving each one a meaningful version, describing what changed, and being able to go back, or forward, safely when a release turns out to be wrong.

Versions that mean something

Most of the studio's code uses semantic versioning, MAJOR.MINOR.PATCH, which Mira met in lesson 2's version ranges. The rules come from semver.org: increase PATCH for fixes that change nothing else, MINOR for new features that do not break anything, and MAJOR for any change that could break something that depends on it. Before 1.0.0, anything may change. A pre-release such as 1.13.0-rc.1 comes before 1.13.0 and is used for trying a release out before calling it final.

For a library, whose users are other programs, the rules are essential, because ranges like ^5.2.0 depend on them. For a service, the version describes the deployable, and what counts as breaking is its API. The partner service's version is 1.12.0 and its API is v1, two separate things: JS-240 set the rule that v1 of the API stays compatible, so a breaking API change would mean adding v2 beside it, while the service's own version keeps counting minor releases. For the portal, which only people use, the version simply names each release.

JavaScript Example · Sorting semantic versions
// Compare and sort semantic versions, including pre-releases.
function parse(v) {
  const [core, pre] = v.split('-');
  return { nums: core.split('.').map(Number), pre: pre ? pre.split('.') : [] };
}
function compare(a, b) {
  const x = parse(a), y = parse(b);
  for (let i = 0; i < 3; i++) if (x.nums[i] !== y.nums[i]) return x.nums[i] - y.nums[i];
  if (!x.pre.length || !y.pre.length) return y.pre.length - x.pre.length;   // a release outranks its pre-releases
  for (let i = 0; i < Math.max(x.pre.length, y.pre.length); i++) {
    const p = x.pre[i], q = y.pre[i];
    if (p === undefined) return -1;
    if (q === undefined) return 1;
    if (p === q) continue;
    const pn = /^\d+$/.test(p), qn = /^\d+$/.test(q);
    if (pn && qn) return Number(p) - Number(q);
    if (pn !== qn) return pn ? -1 : 1;
    return p < q ? -1 : 1;
  }
  return 0;
}

const versions = ['1.13.0', '1.12.0', '1.13.0-rc.2', '1.10.4', '1.13.0-rc.10', '2.0.0', '1.13.0-beta.1', '1.9.12'];
console.log(versions.sort(compare).join('  '));
Output
1.9.12  1.10.4  1.12.0  1.13.0-beta.1  1.13.0-rc.2  1.13.0-rc.10  1.13.0  2.0.0

What this code does

The three numbers are compared one by one as numbers, so 1.10.4 comes after 1.9.12. When they are equal, a version without a pre-release part outranks one with it, which puts 1.13.0 after all its release candidates. Pre-release identifiers made of digits are compared numerically, so rc.10 comes after rc.2, and alphabetic ones in dictionary order, so beta comes before rc.

Commit messages that write the changelog

Deciding the next version by hand is easy to get wrong, and writing the changelog by hand is easy to skip. The studio adopts Conventional Commits, a convention for commit messages that makes both automatic. Each message starts with a type, an optional scope and a short description: feat(board): show the van loading order. The types used most are feat for a feature, fix for a bug fix, perf, docs, test, refactor and chore. An exclamation mark after the type, or a BREAKING CHANGE: line in the body, marks a breaking change.

From commit types to the next version
Commits since the last release includeNext version from 1.12.0
Only docs, test, refactor or choreNo release needed (or 1.12.1 if one is wanted)
At least one fix or perf1.12.1
At least one feat1.13.0
Any breaking change2.0.0

Mira writes a small script that reads the commits since the last tag, decides the bump and drafts the changelog. Tools such as release-please and semantic-release do the same at larger scale, opening a release pull request or publishing automatically. In a throwaway repository with the partner service's last four commits, the script and npm version do the whole release.

JavaScript Example · scripts/next-version.mjs
// scripts/next-version.mjs: suggest the next version from Conventional Commits since the last tag
import { execFileSync } from 'node:child_process';
import { readFileSync } from 'node:fs';

const git = (...args) => execFileSync('git', args, { encoding: 'utf8' }).trim();
const last = git('describe', '--tags', '--abbrev=0');
const log = git('log', `${last}..HEAD`, '--format=%s%x1f%b%x1e');
const commits = log.split('\x1e').map((c) => c.trim()).filter(Boolean).map((c) => {
  const [subject, body = ''] = c.split('\x1f');
  const m = /^(\w+)(\(([^)]+)\))?(!)?: (.+)$/.exec(subject);
  return m ? { type: m[1], scope: m[3], breaking: !!m[4] || /BREAKING CHANGE:/.test(body), text: m[5] } : null;
}).filter(Boolean);

const bump = commits.some((c) => c.breaking) ? 'major' : commits.some((c) => c.type === 'feat') ? 'minor'
  : commits.some((c) => c.type === 'fix' || c.type === 'perf') ? 'patch' : null;
const current = JSON.parse(readFileSync('package.json', 'utf8')).version;
const [maj, min, pat] = current.split('.').map(Number);
const next = bump === 'major' ? `${maj + 1}.0.0` : bump === 'minor' ? `${maj}.${min + 1}.0` : bump === 'patch' ? `${maj}.${min}.${pat + 1}` : current;

console.log(`last tag ${last}, ${commits.length} commits, bump: ${bump ?? 'none'}, next: ${next}`);
for (const [title, types] of [['Features', ['feat']], ['Fixes', ['fix', 'perf']]]) {
  const items = commits.filter((c) => types.includes(c.type));
  if (items.length) console.log(`\n## ${title}\n` + items.map((c) => `- ${c.scope ? `**${c.scope}:** ` : ''}${c.text}`).join('\n'));
}

What this code does

The script asks git for the latest tag and the subject and body of every commit since, separated by control characters that cannot appear in messages. Each subject is matched against the Conventional Commits pattern to find its type, scope and breaking marker. The highest bump wins, and features and fixes are listed under headings as the draft changelog.

Cutting release 1.13.0
$ git init -q && git checkout -q -b main
$ git log --oneline v1.12.0..HEAD
4285f82 perf(kitchen): load the day sheet with one query
5be0d45 docs: explain the fair-share rule in the README
850d89e fix(orders): reject notes longer than 500 characters with a clear error
8e2308a feat(board): show the van loading order on the day view
$ node scripts/next-version.mjs
last tag v1.12.0, 4 commits, bump: minor, next: 1.13.0

## Features
- **board:** show the van loading order on the day view

## Fixes
- **kitchen:** load the day sheet with one query
- **orders:** reject notes longer than 500 characters with a clear error
$ npm version minor -m "chore(release): %s"
v1.13.0
$ git tag
v1.12.0
v1.13.0

What this code does

Four commits follow the v1.12.0 tag: a feature, a fix, a documentation change and a performance improvement. The script finds the feature and suggests 1.13.0, with a changelog of the user-visible changes and the docs commit left out. npm version minor then updates package.json, commits the change and creates the v1.13.0 tag in one step.

The pushed tag starts the release workflow, which builds the artifact once, attaches it and its SBOM to a GitHub Release with the changelog as release notes, and runs the deployment from lesson 7. GitHub's immutable releases, enabled on the studio's repositories, prevent a published release's tag and files from being changed afterwards, so 1.13.0 always means the same code.

Developers' changelogs are not what Rosa and Lucy need. For them, Mira writes a few plain sentences when a release changes something they will see, posted in the Food Hub's staff channel: "From Wednesday the board shows the van loading order on each day. Nothing else changes." The changelog is generated; the note to people is written.

Rolling back

Lesson 7 made rollback of code a single step: point current at the previous release folder and restart. It takes seconds and should be the first thing tried when a release goes wrong, before anyone starts debugging, because restoring service matters more than understanding the problem. The understanding can come afterwards, with the old version running.

Not everything can be rolled back that way. Code can be swapped, but side effects cannot be unsent: emails that went out, notifications pushed to the board, orders created by agencies, calls made to GiveRight or Theo's API. And database changes are the hardest of all. If release 1.13.0 renamed a column, putting 1.12.0 back means old code running against the new schema, which fails. This is why the shipping script in lesson 7 runs migrations that must work with both the old and the new code.

Database changes: expand and contract

WEB-201 introduced the expand and contract pattern for changing a database without downtime, and it is what makes rollback safe. Instead of one release that changes the schema and the code together, a change is split so that every release works with the schema before and after it.

Four releases across a timeline. Release A expands the schema by adding a new column while keeping the old one. Release B writes both columns and reads the new one. A backfill copies old values. Release C stops using the old column. Release D contracts the schema by dropping it. At each step the previous release still works with the current schema.
Renaming a column with expand and contract
JavaScript Example · Is every step safe to roll back?
// Check that each release's code works with the schema before and after its migration.
const releases = [
  { name: '1.13.0', migration: { add: ['delivery_window'] }, reads: ['delivery_slot'], writes: ['delivery_slot'] },
  { name: '1.14.0', migration: {}, reads: ['delivery_window'], writes: ['delivery_slot', 'delivery_window'] },
  { name: '1.15.0', migration: {}, reads: ['delivery_window'], writes: ['delivery_window'] },
  { name: '1.16.0', migration: { drop: ['delivery_slot'] }, reads: ['delivery_window'], writes: ['delivery_window'] },
];
const risky = [{ name: '1.13.0-direct', migration: { add: ['delivery_window'], drop: ['delivery_slot'] }, reads: ['delivery_window'], writes: ['delivery_window'] }];

function check(list, columns) {
  let schema = new Set(columns);
  let previous = { name: 'running release', reads: ['delivery_slot'], writes: ['delivery_slot'] };
  for (const r of list) {
    for (const c of r.migration.add ?? []) schema.add(c);
    for (const c of r.migration.drop ?? []) schema.delete(c);
    const broken = [...previous.reads, ...previous.writes].filter((c) => !schema.has(c));
    console.log(r.name.padEnd(14), broken.length ? `rollback to ${previous.name} breaks: needs ${[...new Set(broken)].join(', ')}` : 'previous release still works');
    previous = r;
  }
}
check(releases, ['delivery_slot']);
check(risky, ['delivery_slot']);
Output
1.13.0         previous release still works
1.14.0         previous release still works
1.15.0         previous release still works
1.16.0         previous release still works
1.13.0-direct  rollback to running release breaks: needs delivery_slot

What this code does

After each release's migration, the check asks whether the release before it would still find every column it reads and writes. In the four-step plan the answer is always yes, so any one release can be rolled back without touching the database. Doing the rename in one release drops a column the running release still needs, so rolling back after it would fail.

Roll back or roll forward? If a release is broken and its migration only added things, rolling back is quick and safe. If a fix is small and obvious, shipping it as the next release through the normal pipeline, rolling forward, may be quicker than rolling back and then forward again. The studio's rule for December is to roll back first, always, and only roll forward with a reviewed change once the service is stable.

Deploying is not releasing

Lesson 3's feature flags make it possible to deploy code, putting it on the servers, without releasing it, turning it on for users. A new feature can go to production switched off, be turned on for the studio's test agency, then for Northside, then for everyone, and be switched off again in seconds if it misbehaves, without a deployment or a rollback. A flag used this way is a kill switch. The board's new layout ships like that in December: deployed in release 1.14.0, switched on for Rosa's tablet only, and for the other kitchen screens a week later.

Blue-green and canary releases

Two common patterns reduce the risk of switching versions all at once. In a blue-green deployment, two complete copies run side by side: blue serves traffic while green is deployed and checked, then the router switches all traffic to green. If green misbehaves, switching back to blue is instant, and blue is already warm. On Theo's server, this would mean running the new release on port 3101 beside the old one on 3100 and changing the web server's upstream, which also removes the brief gap of a restart; the studio plans to try it for the partner service in January.

In a canary release, the new version first receives a small share of real traffic, such as 5 in 100 requests, while its error rate and latency are compared with the old version's. If it does as well, its share grows step by step; if not, it is withdrawn automatically. Canaries need enough traffic to compare, which is the Food Hub's limit: with a few hundred partner requests a day, 5 in 100 is too few to see a difference within hours. For services that small, a flag that turns a feature on for one agency at a time does the same job better.

A bar showing traffic split in steps over time: 5 in 100 of requests to the new version, then 25, then 50, then all, with a check between each step comparing the new version's error rate with the old one's; a failed check sends all traffic back to the old version.
A canary release in steps
JavaScript Example · Promote or roll back a canary
// Canary analysis: compare the canary's error rate with the baseline's, with a minimum sample.
function decide(canary, baseline, { minRequests = 400, maxRatio = 1.5, minExtraErrors = 5 } = {}) {
  if (canary.total < minRequests) return `wait: only ${canary.total} canary requests`;
  const rate = (s) => s.errors / s.total;
  const expected = rate(baseline) * canary.total;
  const extra = canary.errors - expected;
  if (rate(canary) > rate(baseline) * maxRatio && extra >= minExtraErrors) {
    return `roll back: ${(rate(canary) * 100).toFixed(1)} in 100 vs ${(rate(baseline) * 100).toFixed(1)} in 100`;
  }
  return `promote: ${(rate(canary) * 100).toFixed(1)} in 100 vs ${(rate(baseline) * 100).toFixed(1)} in 100`;
}

const baseline = { errors: 38, total: 19000 };
console.log(decide({ errors: 1, total: 120 }, baseline));
console.log(decide({ errors: 3, total: 1000 }, baseline));
console.log(decide({ errors: 14, total: 1000 }, baseline));
Output
wait: only 120 canary requests
promote: 0.3 in 100 vs 0.2 in 100
roll back: 1.4 in 100 vs 0.2 in 100

What this code does

No decision is made until the canary has served a minimum number of requests, because a handful of requests says nothing reliable. After that, the canary is rolled back only if its error rate is clearly higher than the baseline's and the difference amounts to several actual errors, so one unlucky request cannot cause a rollback. The second canary is within normal variation and is promoted; the third has seven times the baseline's rate and is withdrawn.

Worked problem

How long does a rollback take now?

Recovering on September 15 took Theo 34 minutes after Rosa's call (from lesson 1's table). With the new process, an alert reaches Gareth about 2 minutes after a failure starts; he decides to roll back within 3 minutes of the alert; the rollback command switches current and restarts in about 5 seconds; and the smoke test confirms the old release in about 10 seconds. How long from the start of a failure to the old release serving again, and how much of the error budget from lesson 9 would a failure of that length use, at 8 board requests a minute?

Show solution
  1. Alert: 2 minutes after the failure starts.
  2. Decision: 3 more minutes, so 5 minutes.
  3. Rollback and smoke test: 5 + 10 = 15 seconds, so about 5.25 minutes in all.
  4. Failed board requests: 8 × 5.25 = 42.
  5. Budget used: 42 of 1,344, about 3 in 100 of the 28-day budget.

Answer: About five and a quarter minutes, using about 42 of the 1,344 failed requests the month allows, compared with 40 minutes and more than 300 failed requests on September 15.

Common mistakes

  • MistakeChanging the schema and the code that depends on it in one release.

    InsteadExpand first, migrate the code, then contract in a later release, so each release can be rolled back.

  • MistakeDebugging a broken release while it keeps failing.

    InsteadRoll back first to restore service, then investigate with the old version running.

  • MistakeDeciding version numbers by feel.

    InsteadFollow semantic versioning, and let Conventional Commits decide the bump.

  • MistakeRunning a canary on too little traffic.

    InsteadWait for a minimum sample, or use flags per customer when traffic is small.

  • MistakeSending developers' changelogs to staff.

    InsteadWrite a short plain note for the people a release affects.

Summary and key terms

  • Semantic versioning gives each release a meaningful version: patch for fixes, minor for compatible features, major for breaking changes, with pre-releases sorting before their release.
  • A service's version is separate from its API's version.
  • Conventional Commits mark each commit's type, so a script or tool can choose the next version and draft the changelog, and a tag starts the release workflow, with immutable releases keeping it fixed.
  • Code rollback is a quick switch and should come first, but side effects and database changes cannot be undone that way, so schema changes use expand and contract so every release works with the schema on either side.
  • Feature flags separate deploying from releasing and act as kill switches.
  • Blue-green deployments switch between two running copies, and canaries compare a new version on a share of traffic, which needs enough traffic to be meaningful.
Semantic versioning
MAJOR.MINOR.PATCH versions whose changes signal breaking changes, features and fixes.
Pre-release
A version such as 1.13.0-rc.1 that comes before its release in order.
Conventional Commits
A format for commit messages, such as feat(board): ..., that tools can read.
Rollback
Returning production to a previous release.
Roll forward
Fixing a broken release by deploying a new one rather than going back.
Expand and contract
Changing a schema in steps so each release works with the schema before and after it.
Blue-green deployment
Running two copies and switching all traffic from one to the other at once.
Canary release
Giving a new version a small share of traffic and comparing it before widening its share.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Why does 1.13.0 sort after 1.13.0-rc.10?

    Show answer

    A version without a pre-release outranks the same numbers with one.

  2. What version follows 1.12.0 if the commits include a fix and a feat?

    Show answer

    1.13.0, because the feature calls for a minor bump, which outranks the fix.

  3. Why can't release 1.13.0 both add delivery_window and drop delivery_slot?

    Show answer

    The running release still uses delivery_slot, so rolling back to it would fail.

  4. Why does the Food Hub use per-agency flags rather than canaries?

    Show answer

    With so little traffic, a small share of requests is too few to compare reliably.

Try it yourself

Bump a version

Complete bump(version, kind) for semantic versions without pre-release parts. major increases the first number and resets the others to 0, minor increases the second and resets the third, and patch increases the third.

For any other kind, throw an Error with the message unknown bump KIND.

function bump(version, kind) {
  return version;
}

for (const [v, k] of [['1.12.0', 'patch'], ['1.12.3', 'minor'], ['1.12.3', 'major'], ['0.9.9', 'minor']]) console.log(v, k, '->', bump(v, k));
try { bump('1.0.0', 'huge'); } catch (err) { console.log('error:', err.message); }

Try it yourself

Choose the bump from commits

Complete bumpFor(messages) so that it returns major, minor, patch or none for a list of Conventional Commit subjects. A ! before the colon means major; any feat means at least minor; any fix or perf means at least patch.

Subjects may have a scope in brackets, such as fix(orders): ....

function bumpFor(messages) {
  return 'none';
}

console.log(bumpFor(['docs: update README', 'test: add capacity tests']));
console.log(bumpFor(['fix(orders): round tray counts', 'docs: typo']));
console.log(bumpFor(['perf(kitchen): one query', 'feat(board): van order', 'fix: typo in message']));
console.log(bumpFor(['feat(api)!: remove the day sheet endpoint', 'fix: tidy']));

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz