Inbound WebSocket flow control — shipped 2026-08-31

A design note from the engineering record, moved out of ROADMAP.md on 2026-09-03 and kept as written. The roadmap itself now holds only the project's state; the reasoning lives here.

The outbound direction was credit-gated (websocket.send awaits its window) and the inbound direction had no backpressure of any kind. Once the executor's submit channel filled, WSGIHandler.ws_message discarded each further message with a log line the client can never see -- the same "a full channel means skip this frame" mistake the chunk channel's own rule forbids, in the direction nobody had written a rule for. 2932 of 3000 lost at 4 KB.

The two directions were coupled, which is why the threshold was so low. The echo app awaits send inside its receive loop, so a client that stops reading blocks that send on its OUTBOUND window -- which stops the app calling receive, which stops the executor draining the channel that INBOUND messages ride. The outbound backpressure produced the inbound loss, and only one of the two directions was allowed to say "wait".

Both may now. The fix is the outbound design in mirror image, plus the one piece a mirror does not give you:

• A window, granted by the consumer. WS_IN_WINDOW (64 KB) bounds the unacked datagram bytes the loop may have in flight toward one socket's receive() queue. The shim acks CUMULATIVELY as the application actually consumes -- a running total, not a delta, so an ack frame the chunk channel has to drop heals at the next consume instead of costing credit for ever. That is why this is the one frame on the seam that may be dropped without an abort.
• Read suspension, which is what makes it real backpressure. A handler that cannot forward returns False from the new ws_message_take hook, and the loop takes the slot off the read set. The socket's receive buffer fills, TCP advertises a zero window, and the CLIENT stops sending. Nothing else in the design propagates past the process boundary.
• A parked queue, bounded by ONE recv. Parsing is all-or-nothing: a single 4 KB read can yield many complete messages and the window can run out partway through. The remainder is parked and replayed in order. It can never grow past what one recv_staging produced, because no further bytes are read -- bounded by the receive buffer, not by the client's send rate, which is the property that makes it safe.

take_ws_resumes names the drained slots once per pass and the loop re-arms their reads. Both new trait methods carry defaults (ws_message_take forwards to ws_message), so every handler that is not the WSGI one is untouched.

A pool-held socket gets the same treatment for free: its channel refusal parks and suspends identically. It has no window, because pool threads send no consumption acks -- the per-pass retry in take_ws_resumes is what drains it.

The gate's shape was chosen by measuring the wrong one first, and this is the part worth remembering. A client that reads CONCURRENTLY loses nothing on the BROKEN build -- 3000 of 3000 echoed, zero drop lines -- so the obvious "send a lot and count the echoes" test passes on the defect it was written for. scripts/ws_inbound_loss_probe.py therefore sends WITHOUT reading until the socket stops taking bytes (phase A), then reads and sends together and requires every message (phase B). Phase A stalls on BOTH builds -- socket buffers fill either way, 380 KB broken against 360 KB fixed -- so it is a precondition, not the assertion. Phase B is the discriminator:

Verified by running the gate against a rebuilt pre-fix binary: exit 1, naming the dropped messages. Holds at 8000 x 4 KB, 3000 x 16 KB and 20 000 x 64 B -- the last being the parked queue's hardest case, many complete messages out of one read.

And it uncovered an older bug, which is the part worth reading. The first Linux CI run of the new gate stalled -- 3 of 3000 echoed, and NO drops, so the backpressure was working and the socket simply never resumed. Two fixes were needed and only the first was mine:

• slot_read_armed is an INVARIANT every re-arm site in the loop consults, and suspending without updating it left the flag lying. On kqueue read and write are independent filters so nothing noticed; on epoll they share ONE registration, so the outbox drain's add_write_oneshot MODs the read interest away and nothing restores it. Setting the flag is not enough either: _after_send's WebSocket branch re-arms on exactly not armed, so the socket's own echo would undo the suspension it had just asked for -- and with it the parked queue's bound. Hence WSState.inbound_suspended: a DELIBERATE suspension, distinguishable from an incidental one.
• The WebSocket read path took ONE recv per event and never re-armed -- A13's defect, in the one path nothing had ever sent a large inbound burst to. The body path has carried the fix and the reason for a long time ("a body larger than the staging buffer leaves bytes pending that will never raise another edge on their own"); this path was simply never asked. kqueue's level trigger hides it entirely. On epoll the edge is spent, and the stall needs the client to STOP SENDING -- which is precisely what inbound backpressure is for, so the feature exposed the bug that had been waiting for it. Re-armed only on a FULL staging buffer, so an ordinary small-message socket pays no extra syscall. Row I19.

Verified by building Linux/aarch64 in a container and running the gate there: 3 of 3000 before, 3000 of 3000 after, same tree. The first hypothesis (the invariant alone) was wrong and the container said so -- macOS passed every version of this, including the two broken ones.

One consequence worth stating plainly. A client that sends without ever reading, against an app that echoes, now BLOCKS instead of losing data. That is the correct end of a deadlock every echo server has -- uvicorn included -- and the old behaviour only avoided it by discarding the client's messages.