JAVA-240 · Lesson 11 of 12 · 25 min

Testing and debugging concurrent code

▶ Video coming soon A video version of this lesson is on the way.

Monday, March 27, 2028, the last regular session. Leif Quarmby, who set up the shop's build in Hamza's first year, joins them, because he has spent more nights than he likes to remember chasing bugs that appear once a week in production and never on his laptop. "An ordinary test proves a method works for one input," he says. "A concurrent test has to prove it works for every way the threads could interleave, and you cannot try them all. So you cheat: you make the dangerous interleavings more likely, you make the test fail fast instead of hanging, and you learn to read a running program from the outside."

Why concurrent bugs hide

A race condition needs two threads to touch the same data within a tiny window of time. On a quiet machine that window is rarely hit, so a test passes a thousand times. Logging or a debugger breakpoint changes the timing and can make the bug disappear while you look for it. And the machine matters: a laptop with two cores and a server with forty interleave threads very differently. A concurrent test can never prove that code is correct; it can only fail to find a bug. The goal is to make tests that find bugs as often as possible and finish quickly either way.

For that reason, Leif's first advice is about design rather than testing. The less shared mutable state a program has, the less there is to test. Code that confines data to one thread, shares only immutable records and uses well-tested classes such as ConcurrentHashMap and BlockingQueue needs far fewer concurrent tests than code full of hand-written locks.

Making races more likely

Three tricks make a race show itself. First, repeat the test many times, since each run is another roll of the dice. Second, use a start gate, a CountDownLatch with a count of one, so that every thread waits at the gate and then all are released at the same instant, rather than starting one after another as they are created. Third, run more threads than cores so the scheduler switches between them in the middle of operations. The program below combines all three to test the unsafe reservation method from lesson 3. The Thread.yield call inside it is a hint to the scheduler that stands in for any slow step between the check and the act, and makes the window easier to hit.

Java Example · Main.java
import java.util.concurrent.*;

public class Main {
    static class Shelf {
        int left = 1;
        int confirmations = 0;
        boolean reserve() {
            if (left > 0) {
                Thread.yield();
                left--;
                confirmations++;
                return true;
            }
            return false;
        }
    }

    public static void main(String[] args) throws InterruptedException {
        int failures = 0;
        for (int trial = 0; trial < 200; trial++) {
            Shelf shelf = new Shelf();
            CountDownLatch go = new CountDownLatch(1);
            Thread[] ts = new Thread[4];
            for (int i = 0; i < 4; i++) {
                ts[i] = new Thread(() -> {
                    try { go.await(); } catch (InterruptedException e) { return; }
                    shelf.reserve();
                });
                ts[i].start();
            }
            go.countDown();
            for (Thread t : ts) t.join();
            if (shelf.confirmations != 1) failures++;
        }
        System.out.println("trials with more than one confirmation: " + failures + " of 200");
    }
}

What this code does

Each of 200 trials creates a fresh shelf with one item, starts four threads that wait at a start gate, then opens the gate so all four try to reserve at the same instant. A trial fails if more than one reservation was confirmed. The racy method fails in some trials and not others, and the number varies from run to run, which is why this is shown as a transcript.

Three runs
$ javac Main.java
$ java Main
trials with more than one confirmation: 157 of 200
$ java Main
trials with more than one confirmation: 141 of 200
$ java Main
trials with more than one confirmation: 155 of 200

What this code does

These are three real runs of the stress test above. Every run found the bug, but in a different number of trials, so a single passing trial would have proved nothing. A test like this should fail if even one trial fails.

Four threads drawn as horizontal arrows that all stop at a vertical gate labelled CountDownLatch of 1. Main is shown calling countDown, the gate opens, and all four arrows continue together into a box labelled shelf.reserve(), with a note saying all start at the same instant.
A start gate releases every thread at the same moment

The same harness, run on the synchronized version of the method, finds nothing, run after run. That is the shape of a good concurrent test: a fixed, generous number of trials, a check after every trial, and an answer that is the same every time for correct code.

Java Example · Main.java
import java.util.concurrent.*;

public class Main {
    static class Shelf {
        int left = 1;
        int confirmations = 0;
        synchronized boolean reserve() {
            if (left > 0) {
                Thread.yield();
                left--;
                confirmations++;
                return true;
            }
            return false;
        }
    }

    public static void main(String[] args) throws InterruptedException {
        int failures = 0;
        for (int trial = 0; trial < 200; trial++) {
            Shelf shelf = new Shelf();
            CountDownLatch go = new CountDownLatch(1);
            Thread[] ts = new Thread[4];
            for (int i = 0; i < 4; i++) {
                ts[i] = new Thread(() -> {
                    try { go.await(); } catch (InterruptedException e) { return; }
                    shelf.reserve();
                });
                ts[i].start();
            }
            go.countDown();
            for (Thread t : ts) t.join();
            if (shelf.confirmations != 1) failures++;
        }
        System.out.println("trials with more than one confirmation: " + failures + " of 200");
    }
}
Output
trials with more than one confirmation: 0 of 200

What this code does

This is the stress test from above with one change: reserve is synchronized. In every one of the 200 trials exactly one of the four threads gets a confirmation. Correct code passes the stress test every time, which is why its result can be printed as fixed output.

In a real project these checks belong in the test suite rather than in main. With JUnit, the testing framework from JAVA-220, the trial loop goes inside a test method, or the annotation @RepeatedTest(200) repeats the test, and an assertion replaces the println. Dedicated tools go further: OpenJDK's jcstress harness runs tiny concurrent snippets millions of times across many interleavings and reports every outcome it saw, and is what the JDK's own developers use to test the memory model.

Tests must not hang

A concurrent bug often shows itself not as a wrong answer but as a program that never finishes: a deadlock, a lost wake-up, a consumer waiting for a poison pill that never comes. A test that hangs is worse than one that fails, because it stops the whole test run and the continuous-integration server waits for hours. Every wait in a test needs a time limit: join(millis) followed by an isAlive check, await with a timeout, Future.get with a timeout. JUnit 5 also has the @Timeout annotation and assertTimeoutPreemptively for the same purpose.

Java Example · Main.java
import java.util.concurrent.*;

public class Main {
    static String bookWithTimeout(Callable<String> booking) throws InterruptedException {
        try (ExecutorService ex = Executors.newSingleThreadExecutor()) {
            Future<String> f = ex.submit(booking);
            try {
                return "passed: " + f.get(500, TimeUnit.MILLISECONDS);
            } catch (TimeoutException e) {
                f.cancel(true);
                return "failed: no answer within 500 ms";
            } catch (ExecutionException e) {
                return "failed: " + e.getCause().getMessage();
            }
        }
    }

    public static void main(String[] args) throws InterruptedException {
        System.out.println(bookWithTimeout(() -> "slot 10:00 booked"));
        System.out.println(bookWithTimeout(() -> {
            new CountDownLatch(1).await();
            return "never";
        }));
        System.out.println(bookWithTimeout(() -> { throw new IllegalStateException("rack full"); }));
    }
}
Output
passed: slot 10:00 booked
failed: no answer within 500 ms
failed: rack full

What this code does

Each check runs the code under test in an executor and waits for at most 500 milliseconds for its answer. The second booking waits on a latch that nobody will ever open, a stand-in for a deadlock, so the check reports a failure and cancels it instead of hanging forever. The third booking throws, and its exception is reported as a failure too.

Catching exceptions from other threads

An exception thrown inside a raw thread ends that thread and is printed to standard error, and nothing else notices, as lesson 2 explained. In a test that is dangerous: the test may pass even though a worker crashed. Install an uncaught exception handler that records problems, and check it at the end. Thread.setDefaultUncaughtExceptionHandler sets one for every thread, and setUncaughtExceptionHandler sets one for a single thread or builder. Executors are safer here, because exceptions come back through futures, but tasks started with execute rather than submit still use the handler.

Java Example · Main.java
import java.util.*;
import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        Queue<String> problems = new ConcurrentLinkedQueue<>();
        Thread.Builder builder = Thread.ofPlatform()
                .name("worker-", 1)
                .uncaughtExceptionHandler((t, e) -> problems.add(t.getName() + ": " + e.getMessage()));

        List<Thread> workers = new ArrayList<>();
        int[] codes = {1041, -1, 1043, -7};
        for (int code : codes) {
            workers.add(builder.start(() -> {
                if (code < 0) throw new IllegalArgumentException("bad order number " + code);
            }));
        }
        for (Thread t : workers) t.join();

        List<String> sorted = new ArrayList<>(problems);
        Collections.sort(sorted);
        System.out.println("problems found: " + sorted.size());
        sorted.forEach(System.out::println);
    }
}
Output
problems found: 2
worker-2: bad order number -1
worker-4: bad order number -7

What this code does

Four named workers each handle one order number, and the two with negative numbers throw. The builder gives every thread an uncaught exception handler that records the thread's name and the message in a thread-safe queue. After joining the workers, main prints the problems in sorted order, so a test could fail if the list is not empty.

Looking inside a running program

When a program misbehaves while running, a thread dump shows what every thread is doing. Lesson 9 used one to see a deadlock. It is just as useful for a program that is slow or stuck for other reasons. jcmd with no arguments lists the Java processes on the machine; jcmd followed by a process ID and Thread.print prints the dump. Here is a dump of a tiny program whose picker thread is waiting for orders:

Java Example · Main.java
import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        BlockingQueue<String> orders = new LinkedBlockingQueue<>();
        Thread picker = new Thread(() -> {
            try {
                while (true) System.out.println("picking " + orders.take());
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }, "picker");
        picker.start();
        orders.put("order 1041");
        picker.join();
    }
}

What this code does

A picker thread loops taking orders from a blocking queue and printing them, and main waits for the picker with join. After the one order has been picked, the picker waits in take for an order that never comes. The program is not broken, but if the inbox reader had crashed, it would look exactly like this.

A real thread dump from jcmd
$ java Main &
$ jcmd -l
24961 Main
$ jcmd 24961 Thread.print
24961:
2026-11-16 07:30:39
...
"main" #1 [24962] prio=5 os_prio=0 cpu=30.19ms elapsed=2.55s tid=0x00007f369c019ab0 nid=24962 in Object.wait()  [0x00007f36a17fe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait0(java.base@21.0.12.1/Native Method)
	- waiting on <0x00000000a8518018> (a java.lang.Thread)
	at java.lang.Object.wait(java.base@21.0.12.1/Object.java:366)
	at java.lang.Thread.join(java.base@21.0.12.1/Thread.java:2079)
	- locked <0x00000000a8518018> (a java.lang.Thread)
	at java.lang.Thread.join(java.base@21.0.12.1/Thread.java:2155)
	at Main.main(Main.java:15)

"picker" #18 [24979] prio=5 os_prio=0 cpu=3.87ms elapsed=2.52s tid=0x00007f369c123fe0 nid=24979 waiting on condition  [0x00007f368082d000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.12.1/Native Method)
	- parking to wait for  <0x00000000a8517e80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.12.1/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.12.1/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.12.1/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.12.1/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.12.1/AbstractQueuedSynchronizer.java:1746)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.12.1/LinkedBlockingQueue.java:435)
	at Main.lambda$main$0(Main.java:8)
	at Main$$Lambda/0x00007f362c000a08.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.12.1/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.12.1/Thread.java:1583)
...

What this code does

The dump shows main WAITING in Thread.join and the picker WAITING (parking) inside LinkedBlockingQueue.take, called from line 8 of Main.java. Reading from the bottom of each stack upwards shows how the thread got there. A waiting picker with an empty queue points to a producer that stopped, not to a bug in the picker.

Leif's tips for reading dumps: take two or three a few seconds apart, because a thread that is in the same place in all of them is stuck, not just busy. Name your threads, so that "picker" appears instead of "Thread-3". Look for BLOCKED threads and what they are waiting to lock, for many threads WAITING on the same object, and for one thread that is RUNNABLE in the same loop every time. For virtual threads, which can number in the millions, jcmd has a separate command, Thread.dump_to_file, that writes all of them to a file in plain text or JSON.

Java Example · Main.java
import java.util.*;
import java.util.concurrent.CountDownLatch;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        CountDownLatch release = new CountDownLatch(1);
        Thread.Builder builder = Thread.ofPlatform().name("orderdesk-", 1);
        List<Thread> mine = new ArrayList<>();
        for (int i = 0; i < 3; i++) {
            mine.add(builder.start(() -> {
                try { release.await(); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
            }));
        }
        for (Thread t : mine) while (t.getState() != Thread.State.WAITING) Thread.onSpinWait();

        List<String> report = new ArrayList<>();
        for (Map.Entry<Thread, StackTraceElement[]> e : Thread.getAllStackTraces().entrySet()) {
            Thread t = e.getKey();
            if (!t.getName().startsWith("orderdesk-")) continue;
            String where = e.getValue().length > 0 ? e.getValue()[0].getMethodName() : "?";
            report.add(t.getName() + " " + t.getState() + " in " + where);
        }
        Collections.sort(report);
        report.forEach(System.out::println);

        release.countDown();
        for (Thread t : mine) t.join();
    }
}
Output
orderdesk-1 WAITING in park
orderdesk-2 WAITING in park
orderdesk-3 WAITING in park

What this code does

Three named threads wait at a latch, and main builds its own small thread dump with Thread.getAllStackTraces, keeping only the program's own threads. For each it prints the name, the state and the method at the top of its stack, which is the low-level park call that every wait ends up in. A program can log a report like this when a watchdog notices that the work has stopped moving.

Other tools

The JDK ships more tools that help with concurrency. Java Flight Recorder, started with jcmd's JFR.start command or the -XX:StartFlightRecording option, records events such as lock contention, thread parks and pinned virtual threads with very low overhead, and JDK Mission Control displays them. IDE debuggers can pause one thread while others run and can show all threads at a breakpoint. And static analysis tools such as SpotBugs, from JAVA-120, flag common mistakes like fields written under a lock and read without one, or a wait called outside a loop.

From symptom to tool
SymptomFirst thing to tryWhat to look for
Program hangsTake a thread dump with jcmd or jstackA deadlock report, or threads WAITING for work that never comes
Wrong totals sometimesA stress test with a start gate and many trialsShared data changed without a lock or atomic method
Slow under loadSeveral dumps a few seconds apart, or Flight RecorderMany threads BLOCKED on one lock
A worker silently disappearedAn uncaught exception handler that logsExceptions in threads started with execute or new Thread
Test suite stallsTime limits on every wait in the testsjoin, await or get with no timeout

A checklist before shipping concurrent code

Before OrderDesk goes near a real Saturday, Leif and Ingrid give Hamza a checklist. Every piece of shared mutable data is either guarded by one documented lock or kept in a thread-safe class and changed only through its atomic methods. Every lock is released in a finally block, and the lock order is written down. Every thread and pool has a name. Every wait has a time limit or a clear reason it cannot hang. Every task's exceptions go somewhere visible. And there is at least one stress test, with a start gate and many trials, for every piece of code that two threads can reach at once.

Try it yourself

Report failed workers

The starter's threads swallow their exceptions, so the problem is never reported. Remove the try and catch so the second thread's IllegalStateException("supplier offline") escapes, and give the thread builder an uncaught exception handler that adds the thread's name and the exception message to the queue.

The program should print problems: 1 and check-2: supplier offline.

import java.util.*;
import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        Queue<String> problems = new ConcurrentLinkedQueue<>();
        Thread.Builder builder = Thread.ofPlatform().name("check-", 1);
        List<Thread> ts = new ArrayList<>();
        for (int i = 1; i <= 3; i++) {
            int n = i;
            ts.add(builder.start(() -> {
                try {
                    if (n == 2) throw new IllegalStateException("supplier offline");
                } catch (IllegalStateException e) {
                    // swallowed: nobody will ever know
                }
            }));
        }
        for (Thread t : ts) t.join();
        System.out.println("problems: " + problems.size());
        problems.forEach(System.out::println);
    }
}

Try it yourself

A stress test that passes

Write a stress test for an AtomicInteger-based counter: in each of 100 trials, start 4 threads that wait at a CountDownLatch start gate and then each increment the counter 1,000 times. A trial fails if the final count is not 4,000. Print the number of failed trials.

The program should print failed trials: 0 of 100.

import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        int failed = 0;
        for (int trial = 0; trial < 100; trial++) {
            AtomicInteger count = new AtomicInteger();
            // start 4 threads at a start gate, each adding 1,000, then check the count
        }
        System.out.println("failed trials: " + (failed + 100) + " of 100");
    }
}

Common mistakes

  • MistakeTrusting a concurrent test that passed once.

    InsteadRepeat it many times, release threads together with a start gate, and use more threads than cores.

  • MistakeWaiting in a test with no time limit.

    InsteadGive every join, await and get a timeout so a hang becomes a quick failure.

  • MistakeLetting worker exceptions vanish into standard error.

    InsteadUse futures, or an uncaught exception handler that records problems for the test to check.

  • MistakeDebugging a hang by adding print statements and rerunning.

    InsteadTake thread dumps of the hung process with jcmd or jstack, two or three seconds apart.

  • MistakeLeaving threads with default names.

    InsteadName every thread and pool so dumps and logs say what each thread is for.

Summary and key terms

  • Concurrent bugs hide because they need rare interleavings, and logging or debuggers change the timing; tests can find bugs but never prove their absence.
  • Designs with little shared mutable state need far less concurrent testing.
  • Stress tests repeat many trials, release threads together through a CountDownLatch start gate and use more threads than cores; correct code passes every trial, and a single failed trial means a bug.
  • Every wait in a test needs a time limit, so hangs become quick failures.
  • Uncaught exception handlers, set on a builder or as the default, collect exceptions from raw threads so tests can check them. jcmd lists Java processes and prints thread dumps with Thread.print, and jstack does the same; several dumps a few seconds apart show what is stuck.
  • Thread.getAllStackTraces gives a program its own dump.
  • Java Flight Recorder records lock contention and parking cheaply, and static analysis tools catch common mistakes.
Stress test
A test that runs concurrent code many times under heavy contention to expose races.
Start gate
A CountDownLatch of one that releases all waiting threads at the same moment.
Heisenbug
A bug that disappears or changes when you try to observe it.
Timeout
A limit on how long a wait may last before it gives up.
Uncaught exception handler
Code that receives exceptions that end a thread.
Thread dump
A snapshot of every thread's state and stack in a running JVM.
Thread.print
The jcmd command that prints a thread dump.
getAllStackTraces
A Thread method that returns the stack of every live thread.
Java Flight Recorder
A low-overhead recorder of JVM events such as lock contention and thread parking.
jcstress
An OpenJDK tool that runs small concurrent tests across many interleavings.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Why does a start gate make races more likely?

    Show answer

    All threads begin the risky operation at the same instant instead of one after another.

  2. A stress test passes 199 trials and fails one. Is the code correct?

    Show answer

    No; one failed trial proves there is a race.

  3. Which jcmd command prints a thread dump?

    Show answer

    jcmd followed by the process ID and Thread.print.

  4. Why take two or three thread dumps a few seconds apart?

    Show answer

    A thread in the same place in every dump is stuck rather than busy.

  5. Where do exceptions from tasks run with execute go?

    Show answer

    To the thread's uncaught exception handler, not to a future.

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz