The Case of the Very Random Dead Ctrl-T

September 20, 2026

Two weeks into the popup chat feature, users started reporting the same thing in different words: Ctrl-T is very random. Sometimes it opened the chat popup instantly. Sometimes it did nothing — not even an error. Same terminal, same screen, same key. First press of the session would work, three presses in a row would work, then a press after closing the popup with ESC would go dead. Then it would work again. There was no pattern, which was the most maddening part — a consistently broken key is a bug report; an inconsistently broken key is a haunting.

This post is the postmortem, because the debugging trail is more interesting than the fix. The fix, in the end, was deleting a second read. The finding took instrumentation, a thread dump, and a lucky observation about letters that typed fine.

What the key has to survive

Every keystroke you press in Phosphor over telnet crosses four layers:

The telnet handler strips the protocol framing (negotiation bytes, escape sequences in flight) and puts your actual bytes on a per-connection pipe.
The decoder turns bytes into keystrokes — here’s a T with the control flag set — and enqueues them.
The GUI event loop wakes up every few milliseconds, asks the queue “anything for me?”, and dispatches what it finds.
The screen — the focused widget finally receives the key.

Our popup needed a step 3½: a hotkey intercept between the event loop and the screen, so Ctrl-T could open the overlay from anywhere without any screen having to know about it. That intercept lives in the BBS’s subclass of the GUI’s event loop, right where the loop pulls its next keystroke.

The subclass polled the input queue with its own five-millisecond window. If a keystroke was sitting there, it grabbed it, checked for Ctrl-T, and either toggled the popup or handed the key up to the screen. If the poll came back empty, it did what looked like the obvious thing: fell through to the parent class’s input routine, which polled the same queue again.

The race

That fall-through was the bug.

Two readers were racing on one queue. The subclass read; if it missed by a hair — a keystroke arriving a few hundred microseconds after its five-millisecond window closed — the parent’s poll got first crack at the key. And a key the parent grabbed never touched the intercept. It went straight into the screen’s dispatch, where Ctrl-T became an ordinary control character with no handler, and was dropped on the floor.

Whether any given press worked depended on pure timing: which poller’s window happened to be open when the key arrived. That’s why it was “very random” — loop phase, repaint timing, and how long you’d been sitting on a screen all shifted the race. Sometimes the subclass won for minutes at a stretch and the popup felt flawless. Then the timing shifted — closing the popup with ESC repaints the screen and perturbs the loop — and suddenly presses died.

The reason it looked like a Ctrl-T-only bug is the good part. The race claimed keys indiscriminately — but for an ordinary key like o, the thief and the legitimate path both delivered it to the same focused text box. Who won didn’t matter; the letter appeared either way. Ctrl-T was the only key whose entire purpose lived in the stolen layer. Lose the race on an o, nobody notices. Lose the race on Ctrl-T, the key vanishes silently. Every other keystroke was being dropped at the same rate and landing anyway.

How it was found

Guessing doesn’t work on timing bugs, so the pipeline got instrumented — three trace markers behind the server’s debug flag, one at each stage: decoded (the keystroke left the decoder), dispatched (the event loop’s read picked it up), toggled (the intercept fired).

The traces told the story in one test session:

Ctrl-T #1, #2, #3: decoded → dispatched → toggled. Popup each time.
ESC to close: dispatched normally, popup closed.
Ctrl-T #4: decoded, then an endless run of empty dispatches. The keystroke existed in the queue. The loop polled thousands of times. The key never surfaced — and never toggled.

A keystroke sitting in a queue that a polling loop never returns — while later keystrokes flow past it — isn’t a stuck loop. It’s a key that was served to someone else. The only other poller was the parent class’s read. The race wasn’t a hypothesis anymore; it was arithmetic. Decode counts versus dispatch counts didn’t match, and the missing ones had all taken the parent’s path.

A deterministic regression test clinched it: a stub terminal that answers empty on the first poll and hands over the Ctrl-T on the second — exactly the interleaving the race produces. Old code: the parent’s poll wins the key, the intercept never fires, the popup never opens. That test went red, then the fix went in, then it went green.

The fix

Stop the second read. When the intercept layer’s own poll misses, it returns immediately — without letting the parent read — and the keystroke stays queued for the next spin of the loop, where the intercept will see it. One reader. The race is undefined behavior that can’t happen anymore, not a race with better odds.

The cost is a few hundred microseconds of latency in the worst case (the key waits one loop spin). The benefit is that the intercept is total: every keystroke crosses it, so any global hotkey that lives there — now or in the future — fires every time.

The takeaway

Two lessons worth keeping:

Intercepts must own their input. Any component that needs to see every keystroke can’t share the queue with a fallback reader — even one that “only” reads when you miss, because that’s precisely when the theft happens. If a key’s handler exists in exactly one layer, that layer must be the only consumer.

The absence of a symptom is not the absence of a bug. The race ate ordinary keys all day long, harmlessly, because the two paths converged. It’s worth asking of any intermittent failure: is this broken key special, or is it just the only one that can’t survive being dropped? Here it was the second — and the fix made the whole input path more honest for everything that comes next.