The elastic pool: one thread until a job has waited — 2026-09-06

A design note from the engineering record. It is the step after loop-user-space.md: with the event-loop thread at tokio's price, the zero-config handler pool — eight threads, what m0serve app.wsgi runs — was serving a trivial view at two thirds of the rate one thread served it, on 1.6x the cores. This note records what the eight threads were doing, the three rules that make the pool behave as one thread until a job has actually waited, the kernel fact that decided the fourth, and what the trivial and slow rows cost before and after.

Bare WSGI at one worker, apps/wsgi_bare, wrk -t2 with browser headers, Apple M4, CPython 3.13.6, granian 2.8.2, the same binary with the rules on and off in the same session, arms alternated. Zero-config went from 118.4k / 122.6k requests per second on 2.8 cores to 172.5k / 179.2k on the one-thread shape's cores at 16 connections, against 176.9k / 179.4k for --blocking-threads 1; at 256 connections from 0.90x of the one-thread shape to parity, one pool thread serving. A fast request behind two 1.5 s views is answered in 2 ms (smoke-blocking-threads, whose limit is 250 ms), and the mixed-workload table's pool rows are flat under slow load at the throughput they had — on a free-threaded interpreter by a fourth rule, recorded at the end of this note: there the pool is parallel, because a parked thread beside a queued job is an idle core rather than a GIL waiter.

What the eight threads were doing

Zero-config runs --blocking-threads min(cores, 8) so that one slow view cannot stall every connection out of the box (WSGI_PERFORMANCE.md, "A slow view strands the connections pinned behind it"). The isolation is real and the cost on a view that never waits had become large, because the loop got fast and the pool did not. Per-thread CPU by ps -M (scripts/probes/bench_threads.py), medians over 8 s, main at 3a365c4:

Every thread added cost throughput and CPU. The first reading was that a burst of jobs is taken by as many threads as are awake and they then serialize on the GIL with an OS wake per hand-off; that is true, and it is not where most of the cost was. Two counters on the pool (OffloadPool.wake_counts, printed at shutdown under M0_POOL_DEBUG=1) split the wakes the loop sent by site, over 11 s at 16 connections:

The one-thread shape parks and is woken 130,000 times in 11 s — once per burst, whenever the loop has nothing for it for longer than its 10 µs spin — and zero-config parks exactly as often. The difference was never how often a thread parks; it was which thread the wake lands on. With eight threads blocked in recv on one datagram socket, the kernel chooses, and a C ping-pong of that primitive (scripts/probes/herd.c, run by poe probe-herd: W receivers blocked on one SOCK_DGRAM pair, one datagram per round, whole-process CPU per round trip) says how it chooses:

macOS wakes every blocked receiver for every datagram — eight parked threads make one wake cost twenty times what it costs one — and they take turns serving. Linux wakes exactly one, at no extra cost, and it is the OLDEST waiter every time: strict round-robin. On both, every job after a park lands on a cold thread with a cold interpreter thread state, and on macOS twelve thousand wakes a second at 56 µs of extra CPU each is most of the core that zero-config was spending over the one-thread shape. The Linux VM's absolute figures are the virtualization's, not the kernel's; the shape is the finding.

The rules

All in packages/m0-http/lightbug_http/offload.mojo (next_job, submit, wake_aged, _park_on_own, _wake_registered), the loop's side in event_loop.mojo (_run_pass's bottom and _wait_for_events), and the registration in m0_wsgi/blocking_pool.mojo. M0_POOL_ELASTIC=0 turns all four off and is the A/B arm: every idle thread spins, every push into a parked lane pokes the lane socket, and a thread that takes a job pokes a sibling for the rest — the rules of the day before.

1. One idle spinner per lane. A thread that finds its ring empty spins only if no sibling is already spinning idle (spinners, a per-lane word beside parked); otherwise it parks at once. The spinner sees every push itself.
2. submit wakes nobody while any thread of the lane is busy or spinning. A busy thread with an empty ring behind it comes back in microseconds and takes the job sooner than a wake could land; a wake beside it is a second thread on the GIL for nothing. Only a lane whose every thread is parked gets a wake (_all_idle: spinners zero and threads <= parked, the thread count kept by register_thread), and exactly one, because the wake retires the woken thread's parked count before the next push can look — so a burst of sixteen into a parked lane wakes one thread, which takes the burst.
3. A ring that holds a job and has not been drained for T is behind a thread that is not coming back, and the LOOP wakes a parked sibling for it. Once per pass wake_aged reads each lane's pop counter (Ring.pops, the head position): moved since the last look, the lane is being drained, however deep its queue; unmoved, the head has waited since the later of its push and the last look that saw the ring move or empty, and past T of that one parked thread is woken. _wait_for_events caps its timeout at POOL_WAKE_WAIT_MS (1 ms) while any job is pending, so an idle loop looks within a millisecond. This replaced the chained wake (pool-ring-handoff.md, a thread that took a job poking a sibling for the rest): the hole the chain filled — a woken thread's socket poll consuming a sibling's wake, a hold registering 1.5 s late on Linux CI — is now closed every pass rather than once, because the loop re-examines the ring until it moves.
4. Every pool thread parks on a wake channel of its own, and the loop wakes the one that parked last. register_thread gives each thread a SOCK_DGRAM pair (BlockingPool.start reserves the records before spawning); a thread parks in recv on its own read end after announcing itself PARKED with a sequence number, and _wake_registered scans the lane's records for the parked thread with the highest sequence, moves it PARKED → WOKEN by compare-exchange, and pokes its write end. Whoever moves the state word off PARKED first owns the transition — a thread that finds a job on its own re-check and loses the race consumes the poke in flight, so no channel holds a stale wake. Pills go to those channels (stop), a WebSocket message on the lane socket is followed by a wake to a parked thread, which polls the socket first thing, and a thread that never registered (a test's, or every thread under the knob) parks on the lane socket under the old rule.

What did not change: announce, re-check, block on the pool side and push, read, poke on the loop side. Every transition into spinning or parking re-checks the ring after announcing itself, which is what lets submit read three counters one after another and skip the wake: a snapshot that straddles a thread's transition is safe because that thread's re-check follows its own announcement, and the push preceded the loop's reads. test_offload.mojo holds the rules — a burst into a parked lane sends one wake, a push beside a busy or a spinning sibling sends none, sixty gap-separated jobs land on one thread of two, a ring being drained is never stalled, a job behind a slow sibling is taken once the loop wakes a parked one — beside the lost-wakeup test and its no-pause variant, both with registered threads now.

Choosing T, and what the check measures

T (POOL_WAKE_AGE_NS, 200 µs; M0_POOL_WAKE_AGE_US overrides it for measurement) bounds how long a fast request waits behind a slow view before a sibling is woken: the smoke's limit is 250 ms against a 1.5 s hold and the mixed-workload p99 is about 2 ms, so 200 µs is well inside both and leaves the isolation where it was. What the check MEASURES mattered more than the number:

A ring 256 deep behind one thread taking 4 µs a job has a head that is a millisecond old and moving every 4 µs; waking for its age put eight threads on the GIL for a queue one thread was draining faster, and the cascade sustains itself (more threads, more hand-offs, a deeper queue, more wakes). Watching for the same job at the head across the loop's looks was no better there, because a pass at 256 connections is longer than T and every look finds a fresh head as old as the backlog. The pop counter is the signal that means "draining": it moves whenever any thread takes a job, whatever the depth, and stops only when nobody does. At 16 connections all three behaved alike (2 to 36 stall wakes a run), which is why the first two looked finished until the concurrency went up.

Measured

ps -M per thread, medians over 8 s of wrk -t2 with browser headers, apps/wsgi_bare, Apple M4, CPython 3.13.6, granian 2.8.2, one session, arms alternated. Absolute rates drift by several percent across the session; the ratios are the result.

The machine ran several percent slower by the final rounds (one thread at 168k against 177k an hour before); within a session the zero-config and one-thread arms are within 1 % of each other at both concurrencies, on the same cores, with one pool thread serving — which is the "done when" of the handoff that asked for this work.

Where the two rules landed before the fourth was built, for the record: rules 1–3 over the shared lane socket took zero-config from 118.4k / 122.6k to 159.3k / 163.5k at 16 connections, still 10 % under one thread on 0.4 more cores, with all eight threads at 14 % — which is the table in the first section, and what sent this note to the kernel.

Linux

The hole the chained wake closed reproduced on Linux and never on macOS, so the Linux reproducers ran first: the tree built in the m0lin container (Debian bookworm, aarch64 under colima, CPython 3.13.11), scripts/probes/phase5_probe.py — smoke-django-realtime phase 5 with a fresh server each round: two 1.5 s views in flight, a hold taken on a pool thread, the subscriber must be registered half a second later — passed 20 of 20 rounds (2 of 10 failed on the ring's first build, 0 of 20 after the chain), and scripts/probes/hold_race_probe.py registered 40 of 40 holds opened while the loop was kept busy. The pool smokes on both platforms and the fairness probe are in the pull request's gates.

The layer split, re-recorded

scripts/bench_layer_split.sh on the committed tree, three rounds, medians, process-level cores (bench/results/layer-split-20260906T215002Z.json, against layer-split-20260906T151035Z.json from the morning, main at a72e340):

Every row but the zero-config one is within 1 % of the morning's; that one is at the one-thread rate on the one-thread cores. A first recording of the same script an hour earlier landed the one-thread and zero-config rows 7 % lower with hello and Granian unmoved; an A/B of main's binary against this one at one thread (186.3k / 186.1k against 184.8k / 185.3k, the rules off 187.1k / 185.5k) found nothing, and ps found contactsd, knowledge-agent and AddressBookSourceSync at a core and a half between them. The artifact above was recorded after they went quiet; the depressed one was discarded, as the comparator rows exist to allow.

The fast-route tail, and why a free-threaded pool is parallel instead

The handoff's last condition was the mixed-workload row: the fast route's p99 with two 200 ms views in flight, unchanged at about 2 ms. The first recording of scripts/bench_mixed_workload.sh on this tree (3.14t, --workers 4 --blocking-threads 4, Django, wrk -c16) had that row's p99 at 8.6 / 9.0 ms with one slow view and 4.1 / 6.4 with two, against the previous artifact's 3.4 and 6.2 — and its throughput up from 55k to 80k requests per second, its p50 halved. Three things were needed to read that.

The tail on a GIL build is noise-shaped, on every binary. On the pinned 3.13, the same shape, fresh server per run, three and then four rounds alternated (scripts/probes/bench_slow.py):

main's own p99 with no slow view at all lands at 1.8 ms in one run and 7.6 in the next; the rules-off arm — code-identical to main on every path with the knob off — reads 7.5 three times and then 2 to 3 four times. The tail of four connections per worker on one interpreter is bimodal on this machine at 2–3 and 7–8 ms, for every binary, and an artifact's two-round median can land on either mode. What is not noise: the elastic arm halves the p50 and carries 55 % more throughput.

Without a GIL the tail is systematic, and the stall check is not the cause. The same A/B under the 3.14t swap, two rounds:

Here the rules-off arm is steadily better in the tail, and letting the loop wake a sibling for a head that has waited 200 µs since its push — the first hypothesis, that a queue one thread was draining needed a second — changed nothing. The difference is the shape: with a GIL, four threads on four connections serialize into one thread's worth of work and the elastic arm loses nothing by being one thread; without a GIL four threads are four cores, and a request queued behind an occasional slow one on the single hot thread pays that delay whole, while the rules-off arm's four threads absorb it. The stall check is for a thread that is not coming back; this is a thread that is coming back a few milliseconds late, and 200 µs of patience is the wrong instrument.

So the pool is parallel on a free-threaded interpreter (OffloadPool.parallel, set by the prefork worker from probe_free_threading and by the threaded mode unconditionally, M0_POOL_PARALLEL overriding either way): submit wakes a parked thread whenever there is one — a parked thread beside a queued job is an idle core — and the stall check counts from the push. The single spinner and the wake by name on each thread's own channel apply either way, which is what the GIL build's herd fix was. Measured under the swap, same shape, the arms alternated:

The parallel pool is the rules-off shape with the herd fix: the same throughput and percentiles as four threads on the old wake, a tail a third of the GIL rules'. What the GIL rules would have bought on a free-threaded build — 40 % more throughput, a p50 of 160 µs against 200 and a p90 of 350 against a millisecond — is the other side of a real trade, and M0_POOL_PARALLEL=0 takes it for a deployment that prefers the median to the tail. The default keeps the table's claim.

The mixed workload, re-recorded

scripts/bench_mixed_workload.sh under the 3.14t swap, two rounds, fast-route p99 medians (bench/results/mixed-workload-20260906T225838Z.json, against mixed-workload-20260905T204712Z.json):

The pool rows are flat under slow load, within the spread the section above measured, at the throughput they had. A recording made before the parallel rule existed — the GIL rules on 3.14t, kept in bench/results/mixed-workload-20260906T220102Z.json — has the --workers 4 +bt=4 row at 2.5 / 8.8 / 5.2 ms and 79k requests per second, which is the trade above in the table's own terms.