JAVA-280 · Lesson 12 of 12 · 27 min

Capstone: a deployment pipeline for a Spring service

▶ Video coming soon A video version of this lesson is on the way.

On Wednesday, April 12, 2028, Leif deployed version 1.3.0 of the booking API to the shop server by hand: a JAR copied over SSH, an environment file with the secrets, and a service started by the operating system's service manager. It worked, and the review of the API in Ingrid's course a week later went well. But the review also produced a list of fixes, and Leif refused to ship them the same way. "The next version goes out through a pipeline," he said, "or it waits." From Thursday, April 20 to Wednesday, April 26, Hamza built that pipeline. Version 1.4.0 went to production through it on Thursday, April 27, Leif's release night, and the review was on Thursday, May 4, with Leif, Ingrid, Odile and Kofi in the back room.

Requirements

Leif and Hamza wrote the requirements as numbered lines, as Odile and Kofi had done for the API itself, so that the review could check each one against evidence. Most of them are lines of the checklist from lesson 1, now made precise.

Deployment requirements for the booking API.
No.RequirementLessons
D1Every push and pull request builds and tests the booking API on a clean runner4
D2A pull request with failing tests or a likely secret cannot be merged4, 5
D3Main produces one container image per commit, built from the tested JAR, run as a non-root user2, 3, 4
D4A release is a Git tag; the JAR, image, health endpoint and notes all carry its version10
D5Secrets reach the service only as files on the server, never through Git or the image5, 6
D6Production deployment needs an approval, deploys blue-green and switches only after readiness and smoke tests pass6, 11
D7Rolling back to the previous version takes one command and under a minute11
D8Heap, GC logging, heap dumps and metrics are configured, and an alert pages Leif when customers are affected7, 8, 9

The design

The pipeline has two workflows. The first, from lesson 4, runs on every push and pull request; on main it also publishes an image tagged with the commit and deploys it to staging, a second Compose project on the same server with its own database and the name staging.api.secondgear.org. The second, from lesson 10, runs on a version tag: it builds the release, publishes the image with its version, creates the GitHub release and then waits for approval in the production environment before deploying.

The complete pipeline. A pull request runs build and tests plus secret scan. A merge to main runs build, image push to ghcr.io and a staging deployment with smoke tests. A version tag runs the release build, publishes the versioned image and GitHub release, waits for approval, then deploys blue-green to production, runs smoke tests and either keeps the new color or switches back.
The booking API pipeline at the end of the capstone.

Gates are ordered so that the cheapest checks run first and nothing reaches production without passing all of them. The example runs the release's gates in order and stops at the first failure, reporting which requirement it protects.

Java Example
import java.util.*;
import java.util.function.BooleanSupplier;

public class Main {
    record Gate(String req, String name, BooleanSupplier passes) {}

    static boolean release(String version, List<Gate> gates) {
        System.out.println("release " + version);
        for (Gate g : gates) {
            if (!g.passes().getAsBoolean()) {
                System.out.println("  " + g.req() + " FAILED: " + g.name() + " -> release stopped");
                return false;
            }
            System.out.println("  " + g.req() + " ok: " + g.name());
        }
        System.out.println("  " + version + " is live");
        return true;
    }

    public static void main(String[] args) {
        Map<String, Boolean> state = new HashMap<>(Map.of("tests", true, "secrets", true, "versionMatches", true,
            "approved", true, "ready", true, "smoke", true));
        List<Gate> gates = List.of(
            new Gate("D1", "build and 165 tests", () -> state.get("tests")),
            new Gate("D2", "secret scan", () -> state.get("secrets")),
            new Gate("D4", "build version equals tag", () -> state.get("versionMatches")),
            new Gate("D6", "approval for production", () -> state.get("approved")),
            new Gate("D6", "new color ready", () -> state.get("ready")),
            new Gate("D6", "smoke tests on new color", () -> state.get("smoke")));
        release("1.4.0", gates);
        state.put("versionMatches", false);
        release("1.4.1", gates);
    }
}
Output
release 1.4.0
  D1 ok: build and 165 tests
  D2 ok: secret scan
  D4 ok: build version equals tag
  D6 ok: approval for production
  D6 ok: new color ready
  D6 ok: smoke tests on new color
  1.4.0 is live
release 1.4.1
  D1 ok: build and 165 tests
  D2 ok: secret scan
  D4 FAILED: build version equals tag -> release stopped

What this code does

Each gate names the requirement it serves, so a failure message tells the reader what was being protected, not just which step broke. The gates run from the cheapest to the most expensive, and the version check sits before the approval so that nobody is asked to approve a release that is already wrong. The second release stops at D4, before any image reaches production.

Building it

Most of the work was assembling pieces from the term. On Thursday, April 20, Hamza joined the two workflows into their final form and added the staging deployment. On Friday he wrote the blue-green scripts and moved the booking API from the service manager into Compose on staging, with the secrets as files under /srv/secondgear-staging/secrets. On the weekend, between Saturday shifts, he wrote the smoke tests. Monday and Tuesday were for failure: Leif broke things on purpose and Hamza made sure the pipeline noticed. Wednesday was the release checklist and the fixes from the API review.

The smoke tests are a handful of requests that prove a freshly started copy really works with its real configuration, run against the new color before the switch. They do not repeat the test suite; they check what the test suite cannot, such as the database password, the migrations and the version. The example runs the same checks against a small server standing in for the new copy.

Java Example
import com.sun.net.httpserver.HttpServer;
import java.net.*;
import java.net.http.*;
import java.time.Duration;
import java.util.*;

public class Main {
    static HttpServer fakeBookingApi(String version) throws Exception {
        HttpServer s = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
        Map<String, String> routes = Map.of(
            "/actuator/health/readiness", "{\"status\":\"UP\"}",
            "/actuator/info", "{\"build\":{\"version\":\"" + version + "\"}}",
            "/api/v1/repair-slots", "[{\"start\":\"2028-05-06T09:00\"},{\"start\":\"2028-05-06T09:30\"}]");
        s.createContext("/", ex -> {
            String body = routes.get(ex.getRequestURI().getPath());
            byte[] b = (body == null ? "{}" : body).getBytes();
            ex.sendResponseHeaders(body == null ? 404 : 200, b.length);
            try (var o = ex.getResponseBody()) { o.write(b); }
        });
        s.start();
        return s;
    }

    static boolean smoke(String base, String expectedVersion) throws Exception {
        HttpClient c = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(2)).build();
        String[][] checks = {
            {"/actuator/health/readiness", "\"UP\""},
            {"/actuator/info", "\"version\":\"" + expectedVersion + "\""},
            {"/api/v1/repair-slots?date=2028-05-06", "\"start\""}};
        boolean all = true;
        for (String[] ch : checks) {
            HttpResponse<String> r = c.send(HttpRequest.newBuilder(URI.create(base + ch[0])).build(), HttpResponse.BodyHandlers.ofString());
            boolean ok = r.statusCode() == 200 && r.body().contains(ch[1]);
            System.out.println("  " + (ok ? "pass" : "FAIL") + " " + ch[0] + " (" + r.statusCode() + ")");
            all &= ok;
        }
        return all;
    }

    public static void main(String[] args) throws Exception {
        HttpServer green = fakeBookingApi("1.4.0");
        String base = "http://127.0.0.1:" + green.getAddress().getPort();
        System.out.println("smoke tests expecting 1.4.0:");
        System.out.println(smoke(base, "1.4.0") ? "switch traffic" : "keep blue");
        System.out.println("smoke tests expecting 1.4.1:");
        System.out.println(smoke(base, "1.4.1") ? "switch traffic" : "keep blue");
        green.stop(0);
    }
}
Output
smoke tests expecting 1.4.0:
  pass /actuator/health/readiness (200)
  pass /actuator/info (200)
  pass /api/v1/repair-slots?date=2028-05-06 (200)
switch traffic
smoke tests expecting 1.4.1:
  pass /actuator/health/readiness (200)
  FAIL /actuator/info (200)
  pass /api/v1/repair-slots?date=2028-05-06 (200)
keep blue

What this code does

The stand-in server answers three paths the way the booking API does on its management and application ports, and a query string does not change the path it matches. The smoke test checks readiness, that the copy reports the expected version, and that a real endpoint returns slots, which exercises the database. When the expected version is wrong, as when an old image was pulled by mistake, the info check fails and traffic stays on blue.

The deployment script brought the pieces together. The example models its logic, with the steps simulated so that the outcomes of a good release, a broken one and a manual rollback can be compared.

Java Example
import java.util.*;

public class Main {
    static String live = "blue";
    static Map<String, String> running = new HashMap<>(Map.of("blue", "1.3.0"));
    static List<String> history = new ArrayList<>(List.of("1.3.0 on blue"));

    static void deploy(String version, boolean becomesReady, boolean smokePasses) {
        String target = live.equals("blue") ? "green" : "blue";
        System.out.println("deploy " + version + " to " + target);
        running.put(target, version);
        if (!becomesReady) { System.out.println("  not ready in 60 s; stop " + target + ", " + live + " stays live"); running.remove(target); return; }
        if (!smokePasses) { System.out.println("  smoke tests failed; stop " + target + ", " + live + " stays live"); running.remove(target); return; }
        live = target;
        history.add(version + " on " + target);
        System.out.println("  switched; " + version + " live on " + live + ", previous kept on " + (live.equals("blue") ? "green" : "blue"));
    }

    static void rollback() {
        String other = live.equals("blue") ? "green" : "blue";
        if (!running.containsKey(other)) { System.out.println("rollback impossible: nothing running on " + other); return; }
        live = other;
        history.add(running.get(live) + " on " + live + " (rollback)");
        System.out.println("rollback: " + running.get(live) + " live on " + live);
    }

    public static void main(String[] args) {
        deploy("1.4.0", true, true);
        rollback();
        rollback();
        deploy("1.4.1-broken", true, false);
        System.out.println("history: " + history);
    }
}
Output
deploy 1.4.0 to green
  switched; 1.4.0 live on green, previous kept on blue
rollback: 1.3.0 live on blue
rollback: 1.4.0 live on green
deploy 1.4.1-broken to blue
  smoke tests failed; stop blue, green stays live
history: [1.3.0 on blue, 1.4.0 on green, 1.3.0 on blue (rollback), 1.4.0 on green (rollback)]

What this code does

A deployment that fails readiness or the smoke tests stops the new copy and never touches the live one, so customers see nothing. A rollback is possible only while the previous copy is still running, which is why the script refuses rather than pretending when there is none. The history list plays the part of the deployed-version file and the release log, and the rehearsal's second rollback simply rolls forward again to 1.4.0.

Breaking it on purpose

On Monday, April 24, and Tuesday, April 25, Leif tried to get a bad version into staging. Each attempt was one requirement under test, and each was written down with what happened.

The failure rehearsal on staging.
Attempt (staging)What the pipeline didRequirement
A test that fails on purposeBuild job red; merge blocked by branch protectionD1, D2
A fake SMS key in application.ymlsecret-scan job flagged line 14; merge blockedD2, D5
A tag v1.4.0-test on a commit whose build said 1.4.0-SNAPSHOTVersion check failed; no image publishedD4
An image whose readiness never comes UP (bad database URL)Not ready after 60 s; blue kept liveD6
A version whose slots endpoint returns 500Smoke test failed; blue kept liveD6
A good version, then ./switch.sh blueBack on the previous version in 3 secondsD7
Container memory limit lowered to 300 MBKilled with exit code 137; Compose restarted it; NotReady alert after 2 minD8

The last row surprised Hamza, and it was Leif's favorite: the JVM never reported anything, because the kernel killed it, and only the alert and docker inspect told the story. It confirmed lesson 7's budget the hard way.

The release

After the API review, Hamza had four commits on main since v1.3.0. The limit of two active bookings per phone number was finished too, but Odile wanted to brief the counter staff first, so it was merged after the tag and went out later in 1.5.0 behind a feature flag. The session below is real, recorded on a copy of the repository with the same commits.

Tagging the release (real session)
$ git log --oneline v1.3.0..HEAD
91eca52 chore(ci): cache Gradle dependencies
6d37a7a docs: describe the release checklist
bc1992f feat(api): add /api/v1/repair-slots/next
b1b1228 fix: reject bookings on public holidays
$ git describe --tags
v1.3.0-4-g91eca52
$ git tag -a v1.4.0 -m "Booking API 1.4.0"
$ git describe --tags
v1.4.0
$ git tag --list
v1.3.0
v1.4.0

What this code does

git log with a range lists the commits that are in HEAD but not in v1.3.0, newest first, all in Conventional Commits form. git describe confirms the position, four commits after v1.3.0, and after the annotated tag it names the commit v1.4.0. One fix and one feature make this a minor release by SemVer, which is why the tag is 1.4.0 and not 1.3.1.

The release workflow did the rest. Its notes, generated from the commits as in lesson 10, had one line under Added and one under Fixed, and Hamza added a sentence for Tamsin saying that nothing had changed or been removed. At 19:30 on Thursday, April 27, Leif approved the production deployment on his phone in the back room.

Representative output
./deploy.sh 1.4.0
Output
pulling ghcr.io/second-gear/booking-api:1.4.0
  digest sha256:5b1e0c7a...e93f
starting booking-api-green (1.4.0)
waiting for readiness on green ... UP after 9 s
smoke tests on green:
  pass /actuator/health/readiness (200)
  pass /actuator/info (200)
  pass /api/v1/repair-slots?date=2028-05-06 (200)
switching api.secondgear.org to green
traffic now on green
recorded: 1.4.0 green sha256:5b1e0c7a...e93f (previous 1.3.0 blue)

What this code does

This is the release job's deployment step, which runs the server's script through the restricted deploy key. The first production run had to move from the service manager to Compose, so beforehand Leif had started 1.3.0 on blue from the image that main had published for that commit, and 1.4.0 went to green. The output is representative of the real run, which was not recorded from the course machine.

Kofi tested the public-holiday fix from the counter computer five minutes later: a booking for Independence Day, Tuesday, July 4 was refused with a clear message. The 1.3.0 copy on blue was stopped the next morning, and the old service manager unit was kept, disabled, for one more week.

Leif's last check before going home was that the digest recorded on the server matched the one the release workflow had published, so that what was running was provably the image built from the tag.

Java Example
import java.util.*;

public class Main {
    record Record(String version, String color, String digest) {}

    public static void main(String[] args) {
        Map<String, String> published = Map.of(
            "1.3.0", "sha256:9a07d2c4f1b8e6a3d5c2b19f0e7a4c6d8b3f5e1a2c9d7b6e4f0a8c3d5b7e9f1a",
            "1.4.0", "sha256:5b1e0c7a8d2f4e6b9c1a3d5f7e9b2c4a6d8f0e1b3c5a7d9f2e4b6c8a0d1fe93f");
        List<Record> server = List.of(
            new Record("1.3.0", "blue", "sha256:9a07d2c4f1b8e6a3d5c2b19f0e7a4c6d8b3f5e1a2c9d7b6e4f0a8c3d5b7e9f1a"),
            new Record("1.4.0", "green", "sha256:5b1e0c7a8d2f4e6b9c1a3d5f7e9b2c4a6d8f0e1b3c5a7d9f2e4b6c8a0d1fe93f"),
            new Record("1.4.0", "green", "sha256:0000c7a8d2f4e6b9c1a3d5f7e9b2c4a6d8f0e1b3c5a7d9f2e4b6c8a0d1fe93f"));
        for (Record r : server) {
            String expected = published.get(r.version());
            boolean ok = r.digest().equals(expected);
            System.out.println(r.version() + " on " + r.color() + ": " + r.digest().substring(7, 15) + "... "
                + (ok ? "matches the release" : "DOES NOT MATCH the release " + expected.substring(7, 15) + "..."));
        }
    }
}
Output
1.3.0 on blue: 9a07d2c4... matches the release
1.4.0 on green: 5b1e0c7a... matches the release
1.4.0 on green: 0000c7a8... DOES NOT MATCH the release 5b1e0c7a...

What this code does

The published map stands for the digests that the release workflow printed, and the server list for what deploy.sh recorded; the digests are invented for the example. A match proves that the running image has exactly the content built from the tag, because a digest is a hash of that content. The third record shows what a tampered or mistaken image would look like: same version tag, different digest.

The review

On Thursday, May 4, the review took an hour. Hamza went through D1 to D8, each with its evidence: a link to a red pull request for D1 and D2, the release page for D4, the Compose file and the empty secrets history for D5, the rehearsal table for D6 and D7, and the dashboard and the alert from the memory test for D8. Odile asked one question, whether she could see which version customers were using, and Hamza showed her the version on the info endpoint and the release notes beside it. Kofi asked to be told before each release, so Hamza added him to the production environment's approvers.

Ingrid's comment was about the term's shape. Every piece of the pipeline had first been built small, by hand, in plain Java or a shell command, before a tool did it automatically, and that was why Hamza could explain each failure in the rehearsal. Leif's comments were about what came next.

Findings from the review on May 4, 2028.
FindingFromAction
Start-up takes 9 s; blue-green hides it, but rollbacks of a stopped color would notLeifTry the Java 25 AOT cache from lesson 8 and measure
Images are not scanned for known vulnerabilitiesLeifAdd an image scan job to the release workflow
Staging data is a copy of production with real phone numbersIngridReplace with generated test data before the website launch
The NotReady alert fires after two minutes, too slow during opening hoursLeifOne minute on open days, as in the drill postmortem
Odile wants a short note in the shop channel after each releaseOdileThe release job posts the notes there
Kofi wants to approve releases that touch stockKofiDone: added to the production approvers

Exercises

Try it yourself

Check the deployment record

Each record is version color digest. Complete check so that it prints, for each record, ok when the digest equals the published digest for that version, unknown version when the version was never published, and MISMATCH otherwise, each followed by the version and color as shown in the expected output. Then print the number of problems.

import java.util.*;

public class Main {
    static void check(Map<String, String> published, List<String> records) {
        for (String r : records) System.out.println(r);
    }

    public static void main(String[] args) {
        Map<String, String> published = Map.of("1.3.0", "sha256:9a07", "1.4.0", "sha256:5b1e");
        check(published, List.of("1.3.0 blue sha256:9a07", "1.4.0 green sha256:5b1e",
                                 "1.4.0 green sha256:0000", "1.4.1 blue sha256:77aa"));
    }
}

Try it yourself

Which requirements are met?

Each piece of evidence names a requirement and whether it passed, as D3 pass or D6 fail. A requirement is met only if it has at least one piece of evidence and none of its evidence failed. Complete report so that it prints each requirement from D1 to D8 as D1 met, D6 not met or D8 no evidence, then the number met out of 8.

import java.util.*;

public class Main {
    static void report(List<String> evidence) {
        for (int i = 1; i <= 8; i++) System.out.println("D" + i + " met");
    }

    public static void main(String[] args) {
        report(List.of("D1 pass", "D2 pass", "D2 pass", "D3 pass", "D4 pass", "D5 pass",
                       "D6 pass", "D6 fail", "D6 pass", "D7 pass"));
    }
}

In January, deploying the booking API had been a JAR on a USB stick and a question about versions. By May it was a tag, a green check, an approval on Leif's phone and a switch that could be reversed in three seconds. Leif's prediction from the first evening had come true: on the night of April 27, the most interesting thing that happened in the back room was Kofi trying to book a repair on a public holiday.

Common mistakes

  • MistakeShipping review fixes by hand because the pipeline is not finished.

    InsteadHold the release until the pipeline can ship it, or the next manual deployment becomes the norm.

  • MistakeTrusting a pipeline nobody has tried to break.

    InsteadRehearse each failure on staging and write down what the pipeline did.

  • MistakeSmoke tests that repeat the unit tests.

    InsteadCheck what only the deployed copy can show: configuration, database, migrations, version.

  • MistakeChecking only the tag of the running image.

    InsteadRecord and compare the digest with the one the release published.

  • MistakeEnding the project at the first successful deployment.

    InsteadReview against the requirements and turn the findings into the next actions.

Summary and key terms

  • The capstone replaced a hand deployment of the booking API with a pipeline meeting eight numbered requirements.
  • Pull requests and pushes build and test on clean runners and scan for secrets, with branch protection blocking failures; main publishes one image per commit from the tested JAR and deploys it to staging.
  • A version tag builds the release, checks the version, publishes the image and GitHub release, and after an approval deploys blue-green to production, switching only after readiness and smoke tests pass and keeping the previous color for a one-command rollback.
  • Secrets are files on the server, and memory, GC logging, heap dumps, metrics and alerts are configured.
  • Each failure mode was rehearsed on staging.
  • Version 1.4.0 went live on April 27, 2028, its digest was checked against the release, and the review on May 4 produced findings on start-up time, image scanning, staging data and alert timing.
Pipeline
The automated sequence of gates from a commit to a production deployment.
Gate
A check that must pass before the pipeline continues.
Staging
An environment like production used to try a version before customers see it.
Smoke test
A few requests that prove a deployed copy works with its real configuration.
Approval
A required human decision before a protected deployment job runs.
Failure rehearsal
Deliberately introducing faults to check that the pipeline catches them.
Deployment record
The stored version, color and digest of what is running.
Digest check
Comparing the running image digest with the one the release published.
Requirement
A numbered, checkable statement of what the system must do.
Review finding
An observation from a review, with an owner and an action.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Why did Leif refuse to deploy the review fixes by hand?

    Show answer

    So that the next version would go out through the pipeline rather than make manual deployment the habit.

  2. What do the smoke tests check that the unit tests cannot?

    Show answer

    Readiness, the deployed version and an endpoint that uses the real database and configuration.

  3. What happened when the container limit was lowered to 300 MB?

    Show answer

    The kernel killed it with exit code 137, Compose restarted it and the NotReady alert fired.

  4. Why was 1.4.0 a minor release?

    Show answer

    It contained a new feature and a fix, and no breaking change.

  5. How did Leif prove that the running image came from the tag?

    Show answer

    By comparing the recorded digest with the one the release workflow published.

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz