JAVA-280 · Lesson 9 of 12 · 24 min

Monitoring and logging in production

▶ Video coming soon A video version of this lesson is on the way.

On Thursday, March 23, Leif opened his laptop on the bench and showed Hamza an empty Grafana dashboard on the shop server. He had installed Prometheus and Grafana there that afternoon, next to the health service, in two more containers. "In three weeks the booking API goes live," he said. "From that day, the only way we know what it is doing is what it tells us. Tonight we decide what it says." In Ingrid's course, the booking API had gained Actuator, Micrometer metrics and health groups; this lesson is about the other end, collecting those signals on the server, keeping logs and turning both into alerts that wake the right person for the right reasons.

Three kinds of signal

A running service can be observed through three kinds of signal. Logs are records of individual events, written as lines of text: a booking created, a supplier call that timed out. Metrics are numbers sampled over time: requests per second, the 95th percentile latency, heap used, bookings created. Traces follow one request through every component it touches, with the time spent in each. Metrics answer whether something is wrong and how much, logs and traces answer what and why. A small service needs all three, but in that order of importance: metrics first, because they are what alerts are built on.

The booking-api container emits metrics on port 9090, which Prometheus scrapes every 15 seconds, and writes JSON logs to standard output, which Docker stores in rotated files. Grafana reads from Prometheus and the logs, and an alert rule sends a message to Leif's phone.
Signals from the booking API on the shop server. Prometheus pulls metrics; logs go to standard output.
The signals and what they are for.
SignalQuestion it answersAt Second Gear
MetricsIs it healthy? How fast? How many?Actuator /actuator/prometheus, scraped by Prometheus
LogsWhat happened to this request?JSON lines on standard output, kept by Docker
TracesWhere did this slow request spend its time?Trace IDs in every log line; full tracing later
HealthShould it receive traffic? Should it be restarted?Readiness and liveness groups on port 9090

Logs as an event stream

The twelve-factor rule for logs is that a service never manages log files itself. It writes events to standard output, one per line, and the environment decides where they go. In a container that is the natural design: Docker captures standard output, Compose shows it with docker compose logs, and a log shipper can forward it. What the service controls is the content of each line. A line meant for a person, such as Booking 2041 created for Ana at 10:30, is easy to read and hard to search. A structured log line is a JSON object with named fields, easy to filter by booking ID or level and still readable.

Spring Boot 3.4 and later write structured logs without extra libraries: the property logging.structured.format.console set to ecs, logstash or gelf switches the console output to JSON in that format. Hamza used ecs, the Elastic Common Schema, for the prod profile. The example writes such a line by hand, with the fields the booking API's lines carry.

Java Example
import java.time.*;
import java.util.*;

public class Main {
    static String json(String s) {
        StringBuilder b = new StringBuilder("\"");
        for (char c : s.toCharArray()) {
            switch (c) {
                case '"' -> b.append("\\\"");
                case '\\' -> b.append("\\\\");
                case '\n' -> b.append("\\n");
                default -> b.append(c);
            }
        }
        return b.append('"').toString();
    }

    static String line(Instant at, String level, String logger, String message, Map<String, String> fields) {
        StringBuilder b = new StringBuilder("{");
        b.append("\"@timestamp\":").append(json(at.toString()));
        b.append(",\"log.level\":").append(json(level));
        b.append(",\"log.logger\":").append(json(logger));
        b.append(",\"message\":").append(json(message));
        new TreeMap<>(fields).forEach((k, v) -> b.append(",").append(json(k)).append(":").append(json(v)));
        return b.append("}").toString();
    }

    public static void main(String[] args) {
        Instant t = Instant.parse("2028-04-29T14:30:05.120Z");
        System.out.println(line(t, "INFO", "o.s.booking.BookingService", "booking created",
            Map.of("booking.id", "2041", "booking.start", "2028-05-06T10:30", "trace.id", "4bf92f3577b34da6")));
        System.out.println(line(t.plusSeconds(2), "WARN", "o.s.booking.SupplierStock", "supplier \"Velotek\" timed out\nafter 1000 ms",
            Map.of("supplier", "velotek", "trace.id", "a3ce929d0e0e4736")));
    }
}
Output
{"@timestamp":"2028-04-29T14:30:05.120Z","log.level":"INFO","log.logger":"o.s.booking.BookingService","message":"booking created","booking.id":"2041","booking.start":"2028-05-06T10:30","trace.id":"4bf92f3577b34da6"}
{"@timestamp":"2028-04-29T14:30:07.120Z","log.level":"WARN","log.logger":"o.s.booking.SupplierStock","message":"supplier \"Velotek\" timed out\nafter 1000 ms","supplier":"velotek","trace.id":"a3ce929d0e0e4736"}

What this code does

The json method escapes quotes, backslashes and newlines, so a message containing any of them still gives one valid line; without it, a newline would split one event into two broken ones. Extra fields are added in sorted order so that lines are easy to compare. The trace ID ties every line of one request together, and none of the fields holds a customer's name or phone number.

That last point was deliberate. A log line is copied to more places and kept longer than the database row it describes, so the booking API logs booking IDs, not customers. JAVA-260 added the redaction rules that enforce it.

Docker's default logging driver writes each container's output to a JSON file on the host, and by default that file grows without limit. Leif set rotation for every service in the Compose file: three files of twenty megabytes each, about a week of the booking API's logs at its expected volume.

Log rotation in compose.yaml
x-logging: &logging
  driver: json-file
  options:
    max-size: "20m"
    max-file: "3"

services:
  booking-api:
    logging: *logging
  proxy:
    logging: *logging

What this code does

The x-logging block is a YAML anchor that each service reuses with *logging, so the rule is written once. max-size and max-file bound each container's logs to sixty megabytes on disk. Lines older than that are gone, so anything that must be kept longer, such as booking history, belongs in the database, not the log.

Log levels in production

Levels decide what is written. Leif's policy was short. ERROR is something that needs a person, such as a failed database migration. WARN is something unusual that the service handled, such as a supplier timeout answered from the last known price. INFO is a business event, such as a booking created or canceled, at most a few lines per request. DEBUG is off in production, and switched on for one logger at a time through an environment variable when chasing a problem, as lesson 5 showed.

Metrics with Prometheus

Prometheus pulls metrics: every few seconds it requests a page of text from each service, the scrape, and stores each number with its time. Spring Boot's Actuator provides that page at /actuator/prometheus, on the management port 9090, in Prometheus's text exposition format: one line per series, a metric name, labels in braces and a value. The example writes the format for two of the booking API's metrics.

Java Example
import java.util.*;

public class Main {
    record Series(String name, Map<String, String> labels, double value) {}

    static String render(String name, String type, String help, List<Series> series) {
        StringBuilder b = new StringBuilder();
        b.append("# HELP ").append(name).append(" ").append(help).append("\n");
        b.append("# TYPE ").append(name).append(" ").append(type).append("\n");
        for (Series s : series) {
            StringJoiner lj = new StringJoiner(",", "{", "}");
            new TreeMap<>(s.labels()).forEach((k, v) -> lj.add(k + "=\"" + v + "\""));
            b.append(s.name()).append(lj).append(" ").append(s.value()).append("\n");
        }
        return b.toString();
    }

    public static void main(String[] args) {
        System.out.print(render("bookings_total", "counter", "Repair bookings by outcome", List.of(
            new Series("bookings_total", Map.of("outcome", "created"), 412),
            new Series("bookings_total", Map.of("outcome", "slot_taken"), 9))));
        System.out.print(render("http_server_requests_seconds", "summary", "HTTP request latency", List.of(
            new Series("http_server_requests_seconds_count", Map.of("uri", "/api/v1/repair-slots", "status", "200"), 5210),
            new Series("http_server_requests_seconds_sum", Map.of("uri", "/api/v1/repair-slots", "status", "200"), 312.6))));
    }
}
Output
# HELP bookings_total Repair bookings by outcome
# TYPE bookings_total counter
bookings_total{outcome="created"} 412.0
bookings_total{outcome="slot_taken"} 9.0
# HELP http_server_requests_seconds HTTP request latency
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{status="200",uri="/api/v1/repair-slots"} 5210.0
http_server_requests_seconds_sum{status="200",uri="/api/v1/repair-slots"} 312.6

What this code does

Each metric starts with HELP and TYPE comments, then one line per combination of labels. A **counter** only ever goes up, here the bookings created and refused since the service started. Latency is published as a count and a sum, from which Prometheus computes the average over any window, and with histogram buckets enabled it can compute percentiles as well.

prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: booking-api
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ["booking-api:9090"]
        labels: { env: production }
  - job_name: ops-health
    metrics_path: /metrics
    static_configs:
      - targets: ["ops:8080"]

What this code does

Prometheus runs in a container on the same Compose network, so it reaches the booking API by service name on the management port, which is never published to the internet. The env label is added to every series from this target, so the same dashboards can later show staging too. Fifteen seconds is frequent enough to notice a problem within a minute and cheap for a service of this size.

Counters are scraped as running totals, so a dashboard always works with their rate: the increase per second over a window. Prometheus's rate function also handles counter resets: when the service restarts, its counters start again from zero, and a naive subtraction would give a large negative number. The example computes a rate the same way from scraped samples.

Java Example
import java.util.*;

public class Main {
    record Sample(int second, double value) {}

    static double rate(List<Sample> s) {
        double increase = 0;
        for (int i = 1; i < s.size(); i++) {
            double d = s.get(i).value() - s.get(i - 1).value();
            increase += d >= 0 ? d : s.get(i).value();
        }
        int span = s.get(s.size() - 1).second() - s.get(0).second();
        return increase / span;
    }

    public static void main(String[] args) {
        List<Sample> normal = List.of(new Sample(0, 400), new Sample(15, 403), new Sample(30, 405), new Sample(45, 409), new Sample(60, 412));
        List<Sample> restart = List.of(new Sample(0, 400), new Sample(15, 403), new Sample(30, 1), new Sample(45, 4), new Sample(60, 7));
        System.out.printf("bookings per minute: %.1f%n", rate(normal) * 60);
        System.out.printf("with a restart at 20 s: %.1f%n", rate(restart) * 60);
        System.out.printf("naive (last - first): %.1f%n", (restart.get(4).value() - restart.get(0).value()) / 60 * 60);
    }
}
Output
bookings per minute: 12.0
with a restart at 20 s: 10.0
naive (last - first): -393.0

What this code does

rate adds the increases between consecutive samples, and when a value drops it assumes the counter restarted from zero and counts the new value as the increase. The normal series shows twelve bookings a minute, and the restart lowers the answer to ten, because the bookings between the last scrape and the restart are lost. The naive difference gives a nonsense negative rate, which is why dashboards always use rate on counters.

What to watch: RED and saturation

Leif's dashboard had four rows, following the RED method for request-driven services, plus one for the JVM. Rate: requests per second, by endpoint. Errors: the share of requests answered with a 5xx status. Duration: the 95th and 99th percentile latency. And saturation: how full the resources are, here the heap after GC, the database connection pool and the container's memory against its limit. Business metrics such as bookings created per hour sat at the top, because a drop in bookings on a Saturday morning is the most important alert of all, whatever the technical graphs say.

The booking API dashboard.
RowPrometheus query (sketch)Healthy looks like
Bookingsincrease(bookings_total{outcome="created"}[1h])Steady on open days, zero on Mondays
Ratesum by (uri) (rate(http_server_requests_seconds_count[5m]))A few requests per second
Errors5xx rate divided by total rateZero, or close to it
Durationhistogram_quantile(0.95, ...)p95 under 300 ms
Saturationhikaricp_connections_active, jvm_memory_used_bytesPool below 8 of 10; heap well below its maximum

Alerts that mean something

An alert should mean that a person needs to act, now. Alerts on causes, such as CPU above 80 for five minutes, fire often and mostly mean nothing; alerts on symptoms that customers feel, such as errors or slow answers, are rarer and real. Leif wrote the booking API's targets as service level objectives, SLOs: 99.5 of every 100 requests in a 30-day window succeed, and 95 of every 100 finish within 300 milliseconds. The 0.5 left over is the error budget, about 216 minutes of full failure a month, which lesson 6's exercise computed.

Alerting on every failed request would page someone for a single blip. Alerting when the monthly budget is gone would be far too late. The standard answer is the burn rate: how fast the budget is being spent compared with an even spend over the month. A burn rate of 1 uses the budget exactly in 30 days; a burn rate of 14.4 sustained for an hour would use 2 in every 100 of a month's budget in that hour. Alerting when both a long window and a short window burn fast catches real outages quickly without firing on short spikes.

Java Example
import java.util.*;

public class Main {
    static double burn(long errors, long requests, double slo) {
        double budget = 1 - slo / 100;
        return requests == 0 ? 0 : ((double) errors / requests) / budget;
    }

    public static void main(String[] args) {
        double slo = 99.5;
        long[][] cases = {
            {2, 4000, 0, 300},      // a normal hour: 2 errors; last 5 min clean
            {300, 4000, 28, 320},   // database failing for 20 minutes, still failing
            {350, 4000, 0, 330},    // an earlier spike, recovered now
            {0, 0, 0, 0}};          // shop closed
        String[] names = {"normal hour", "outage in progress", "recovered spike", "no traffic"};
        for (int i = 0; i < cases.length; i++) {
            long[] c = cases[i];
            double longBurn = burn(c[0], c[1], slo), shortBurn = burn(c[2], c[3], slo);
            boolean page = longBurn > 14.4 && shortBurn > 14.4;
            System.out.printf("%-18s 1h burn %5.1f, 5m burn %5.1f -> %s%n", names[i], longBurn, shortBurn, page ? "PAGE LEIF" : "no alert");
        }
    }
}
Output
normal hour        1h burn   0.1, 5m burn   0.0 -> no alert
outage in progress 1h burn  15.0, 5m burn  17.5 -> PAGE LEIF
recovered spike    1h burn  17.5, 5m burn   0.0 -> no alert
no traffic         1h burn   0.0, 5m burn   0.0 -> no alert

What this code does

burn divides the observed error rate by the rate the budget allows, which is 0.005 for an objective of 99.5. The outage fires because both the hour and the last five minutes are above 14.4. The recovered spike burned fast over the hour but is clean now, so nobody is woken for a problem that has already ended, and no traffic gives no alert rather than a division by zero.

alerts.yml (draft)
groups:
  - name: booking-api
    rules:
      - alert: BookingApiErrorBudgetBurn
        expr: |
          (sum(rate(http_server_requests_seconds_count{job="booking-api",status=~"5.."}[1h]))
            / sum(rate(http_server_requests_seconds_count{job="booking-api"}[1h]))) > (14.4 * 0.005)
          and
          (sum(rate(http_server_requests_seconds_count{job="booking-api",status=~"5.."}[5m]))
            / sum(rate(http_server_requests_seconds_count{job="booking-api"}[5m]))) > (14.4 * 0.005)
        labels: { severity: page }
        annotations:
          summary: "Booking API is failing requests"
          runbook: "https://github.com/second-gear/secondgear/blob/main/booking-api/docs/runbook.md#errors"
      - alert: BookingApiNotReady
        expr: up{job="booking-api"} == 0
        for: 2m
        labels: { severity: page }

What this code does

The first rule is the two-window burn-rate alert from the example, written in Prometheus's query language. The second fires when Prometheus has failed to scrape the service for two minutes, which covers a crashed or stuck container. Each alert links to a section of the runbook, so the person woken knows what to check first.

Leif left the rules as a draft that evening; he finished them with Hamza and Ingrid the following Wednesday, when the readiness alert and the slot alert were set up in Grafana.

Exercises

Try it yourself

Rate with a reset

Complete increase so that it returns the total increase of a counter over the samples, treating any drop as a restart from zero (the new value then counts as the increase).

public class Main {
    static double increase(double[] samples) {
        return samples[samples.length - 1] - samples[0];
    }

    public static void main(String[] args) {
        System.out.println(increase(new double[] {10, 12, 15, 15, 20}));
        System.out.println(increase(new double[] {10, 12, 2, 5, 9}));
        System.out.println(increase(new double[] {7}));
    }
}

Try it yourself

Should someone be paged?

Complete decide so that it returns page when the error rate (errors divided by requests) is above 14.4 times the budget in both windows, ticket when it is above 1 times the budget in the long window only, and ok otherwise. The budget is 1 - slo / 100, and a window with no requests has an error rate of 0.

public class Main {
    static String decide(double slo, long errLong, long reqLong, long errShort, long reqShort) {
        return "ok";
    }

    public static void main(String[] args) {
        System.out.println(decide(99.5, 300, 4000, 28, 320));
        System.out.println(decide(99.5, 40, 4000, 0, 330));
        System.out.println(decide(99.5, 2, 4000, 0, 300));
        System.out.println(decide(99.5, 0, 0, 0, 0));
    }
}

At ten o'clock the dashboard had one live row, the health service's request rate, a flat line at the four requests a minute that Docker's own health check produced. Leif said that a flat line nobody looks at is the best kind.

Common mistakes

  • MistakeWriting log files from inside the service.

    InsteadWrite one event per line to standard output and let the platform store and rotate it.

  • MistakeLeaving Docker's json-file logs unbounded.

    InsteadSet max-size and max-file for every service.

  • MistakeLogging customers' names and phone numbers.

    InsteadLog IDs and references; keep personal data in the database.

  • MistakeGraphing counters as raw values.

    InsteadUse rate or increase over a window, which also handles restarts.

  • MistakeAlerting on causes such as CPU or on every single error.

    InsteadAlert on customer-facing symptoms with SLOs and two-window burn rates.

Summary and key terms

  • Services are observed through metrics, logs, traces and health checks, and metrics carry the alerts.
  • Logs are events written one per line to standard output, structured as JSON (Spring Boot's logging.structured.format.console), carrying IDs and trace IDs but no personal data, and rotated by Docker's json-file options.
  • Prometheus scrapes Actuator's /actuator/prometheus on the private management port, storing counters, gauges and latency summaries, and dashboards use rate on counters to handle resets.
  • The RED method watches rate, errors and duration, plus saturation and business metrics.
  • Service level objectives define an error budget, and alerts page a person when both a long and a short window burn the budget fast, linking to a runbook.
Structured log
A log line made of named fields, usually JSON, rather than free text.
Metric
A number sampled over time, such as requests per second.
Trace
The record of one request across components, with timings.
Scrape
Prometheus fetching the current metric values from a service.
Counter
A metric that only increases, until the process restarts.
Rate
The per-second increase of a counter over a window.
RED method
Watching the rate, errors and duration of requests.
Service level objective
A target for reliability, such as 99.5 of every 100 requests succeeding.
Error budget
The amount of failure an objective allows over its window.
Burn rate
How fast the error budget is being used compared with an even spend.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Where should a container's service write its logs?

    Show answer

    To standard output, one event per line.

  2. Why does the booking API log booking IDs but not names?

    Show answer

    Logs are copied and kept widely, so personal data stays in the database.

  3. How does Prometheus get metrics from the booking API?

    Show answer

    It scrapes /actuator/prometheus on port 9090 every 15 seconds.

  4. What does rate do when a counter drops to zero?

    Show answer

    It treats the drop as a restart and keeps counting the increase.

  5. Why use two windows in a burn-rate alert?

    Show answer

    The long window shows a real problem and the short one shows it is still happening.

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz