The Case of the Blended Batches: A Race Hidden Inside a Replace

September 23, 2026

The jterm thread-safety suite exists to answer one question: can a widget survive being redrawn from one thread while another mutates it? It has run thousands of times across dozens of widgets, mostly to say yes. On the morning we cut the 0.1.1 release, it said no — and the test that failed wasn’t even the one that was broken.

Two failures, two days apart

The release gate runs the whole suite, 2,926 tests. The first full run failed one test: FireworksTest.renderFrameProducesOutput — the fireworks animation was supposed to paint visible pixels after five frames and, once out of 2,926, it painted nothing. We re-ran the class in isolation: 12/12 green. Read the render path: fresh instance, deterministic math, five calls that each draw a rocket or particles. The failure looked like noise under load — the kind of flake you shrug at and re-run.

So we re-ran the full suite. This time Fireworks passed, and a different test failed: DataGridThreadSafetyTest.concurrentSetRowsAndDraw, with the message

setRows replaces; final count should be 10 — expected: <10> but was: <20>

Two “random” failures in a row is no longer noise. It’s a signal that the suite, run under full load, is exercising interleavings that isolated runs never see. The Fireworks failure may well have been the same class of bug in a different costume. The DataGrid one, though, turned out to be a plain, provable, reproducible race — and it was in the model, not the widget.

The bug

DefaultGridModel backs the data grid with a CopyOnWriteArrayList and exposes the obvious mutation API:

public void setRows(Collection<T> newRows) {
    rows.clear();            // not atomic with the next line
    rows.addAll(newRows);
    fireGridChanged();
}

CopyOnWriteArrayList makes each individual call thread-safe — no ConcurrentModificationException, no torn reads. What it does not do is make a sequence of two calls atomic. So two threads calling setRows concurrently can interleave like this:

W1: rows.clear()
W2: rows.clear()
W1: rows.addAll(W1's 10 rows)
W2: rows.addAll(W2's 10 rows)   → 20 rows, half one caller's, half the other's

Every element-wise operation was thread-safe. The contract — “replace all rows” — was not, because the contract spans two operations. The final state isn’t one caller’s rows or the other’s; it’s a blend of both batches. Whoever reads the model next sees a grid that never existed.

This is the same lesson as the Ctrl-T case from a few weeks ago: the race doesn’t live in the dramatic code (the drawing loop with its repaints and virtual threads), it lives in the boring two-liner everyone assumes is too simple to break.

Why the test caught it

concurrentSetRowsAndDraw runs two virtual threads that each call setRows with a fresh 10-row batch, fifty times, while a reader loop draws the grid continuously. When the writers finish, the model must hold exactly one batch: 10 rows. The assertion final count should be 10 looks like a boring postcondition — it’s actually the only check that catches a blend, because a blended model is still internally consistent (a CopyOnWriteArrayList with 20 valid rows in it). Nothing throws. Nothing corrupts. The count is just wrong, and only if you know what the last writer intended.

That’s the shape of most concurrency bugs: not a crash, a plausible wrong answer.

The fix

Hold a lock across the clear+add pair, in every mutating method:

public void setRows(Collection<T> newRows) {
    synchronized (rows) {
        rows.clear();
        rows.addAll(newRows);
    }
    fireGridChanged();
}

addRow, addRows, and clear got the same treatment. Readers stay lock-free — CopyOnWriteArrayList iteration remains snapshot-cheap, and a reader now sees either the old rows or the new rows, never a mix. The cost is a lock acquire per mutation, which is nothing next to the cost of a grid that lies.

fireGridChanged() stays outside the lock on purpose: listener callbacks run user code, and running arbitrary code while holding a lock is how you invent deadlocks. The event may fire “late” by a few microseconds; that’s invisible. A deadlock would not be.

The part worth keeping

Two things made this bug findable, and both generalize:

The full-suite gate is worth its wall time. The race survived every targeted run and every widget test for the whole life of the class. It fell to a 10-second stress test the first time the entire suite ran on a loaded machine. Release gates don’t exist because machines are flaky; they exist because load is a feature.

“It’s flaky” is a hypothesis, not a verdict. One failed animation test reads like noise. The discipline that found the real bug was re-running the full suite instead of the failed test — and refusing to accept “passed this time” as an explanation for why it failed the first time. The second failure was the confession; the first was just the coincidence that opened the case.