Exercising the server against real applications

The server run against Django projects nobody here wrote: what broke, what was measured, what changed.

A record. Last run 2026-10-09, against m0serve 1.14.0 on the release branch at 57e9150, all four applications, six rows, in the newest section below. The full pass it re-runs was 2026-09-01 and 2026-09-02, against 0.16.0 (bin/m0serve from the tree at c823198), macOS 26 on an M4, CPython 3.13.6. The plan and the previous records are below and in the history of this file.

Every application this server had been tested against was written to test it — until the 2026-08-26 pass below, which put three real Django projects in front of it and found five defects. This pass re-runs those three against a server five minor versions on, adds a fourth application nobody here wrote at all, and replaces the pass's phase 5 — six thousand requests sampling RSS — with a driver that asserts bytes: every response compared, status and headers and body digest, against a capture recorded from a reference server (gunicorn, uvicorn, daphne), under five concurrent populations, with logins, and with the server SIGTERM'd and restarted underneath. Three of the previous six defects were silent — a clean status over a short or empty body — and a request loop passes every one.

The four applications

Clean clones served from their own directories, each with its own venv, scratch databases, their own settings unmodified except where the row says.

What was found

One server defect, since fixed and gated. And six things that were not the server, each measured rather than assumed.

Finding 1 could not have come from apps/: it needs a multi-megabyte upload in flight at the instant of the signal, which only a population that uploads continuously produces. Finding 2 could not have come from the previous pass either, whose textshelf row never abandoned a stream.

Phase 0 — will it even load?

m0serve --doctor ok: true on every entry point: transcript_manager.wsgi (--app-dir src), halftone_studio.wsgi and .asgi, config.wsgi and config.asgi, bakerydemo.wsgi and an asgi.py added for it (the startproject file; bakerydemo ships only wsgi.py). Protocol detection right every time; WSGI apps got the zero-config pool of 8, ASGI apps the executor.

Two applications did not load from a clean clone, and the doctor could see only one of them. transcripts' README installs .[dev], which does not contain django-watchfiles although settings.py lists it unconditionally (it is in the uv lockfile's dev group; uv sync works). color-separation keeps real source under output/ — composite.py, raster.py, vector.py — and .gitignore excludes output/, so a clone fails on its first request with No module named 'output'; the doctor passed, because Django imports its URLconf lazily. Phase 0 answers "does the callable import", not "does the first request work".

Phase 1 — parity, byte for byte

Each application captured from a reference server and every response of the soak held to it. The only differences that survived were named substitutions (findings 4 and 5), a framing choice — uvicorn frames an undeclared-length body as chunked where m0serve buffers it and sends Content-Length; the decoded bytes are identical — and the four security headers runserver's StaticFilesHandler bypasses, on which gunicorn agrees with m0serve byte for byte.

Phases 3 and 5 — topology, churn, and the soak

Every row: five populations at once (keep-alive bursts at 120 requests per connection, over the cap of 100; streams read to completion; uploads, slow views and logins; WebSocket echoes where the app has any; abandoners that vanish mid-body by FIN and by RST and reuse the slot at once), sampling RSS, descriptors, threads and the server's own /__metrics. "verified" is responses byte-identical to the reference capture.

Descriptors and threads were flat in every clean row. --threads was not run: none of the four dependency trees builds on free-threaded CPython (psycopg-binary ships no wheel; the record above says the same).

What this changes about the guards

• The driver itself (scripts/soak.py, poe soak-apps, poe soak-selftest): its comparator is a pure function with a self-test that found the driver's own first hole — a substitution greedy enough to absorb a truncation blinds the instrument — so a capture now refuses any route its patterns blind and carries a fingerprint of the rules it was recorded under. Sabotaging a manifest's byte count makes the task exit 1 naming the failure.
• scripts/drain_upload_probe.py, finding 1's reproducer and the gate the fix will land with, two-sided like its sibling: the upload must be answered whole, and the process must exit inside 3 s of the signal.
• Four manifests under scripts/soak_manifests/ are the re-runnable record of each application's shape, including every substitution and the reason for it.

FastHTML's and FastAPI's own examples, beside uvicorn — 2026-09-22

A separate pass, not the soak above. FastHTML's official examples — the examples/ of AnswerDotAI's fasthtml repository and the fasthtml-example repository, 37 applications — and FastAPI's documented chat room, served side by side under bin/m0serve from main at ef2811e and under uvicorn 0.53.0: python-fasthtml 0.14.14, Starlette 1.6.0, CPython 3.13.6, macOS 27 on an M4. Each application ran from its own copy per server, so no SQLite file or session key was shared; the same scripted scenario went to both; the responses were compared with only the port normalised, and both logs were read. The gates below are what the pass left in the tree.

Most of what the FastHTML tutorial teaches matched: sessions and Beforeware logins, fastlite writes, HTMX out-of-band swaps, toasts, cookies, multipart uploads, FileResponse with ranges, gzip, streamed and SSE responses under 40 concurrent clients, disconnect detection, lifespan, WebSockets, and live=True under --reload. Under --workers 2 both servers lost a worker to fastlite's schema race at import; m0serve's supervisor respawned it, and uvicorn carried on with one.

What did not match, most harmful first:

Three more failures were FastHTML's own and identical under both servers: setup_ws reads scope.client as an attribute of a dict, the terminal example's disconnect hook takes an argument FastHTML does not pass, and fastlite's db.create races across workers at import.

Re-run on the response heads, 2026-09-23 (A21, K12, L27; python-fasthtml 0.14.13, Starlette 1.6.0, uvicorn 0.53.0). The probe application's heads, read beside uvicorn: FastHTML's default 404, a RedirectResponse, a 204, a 304, HEAD of a FileResponse and of a 5 MB page, and HEAD of a sync and an async StreamingResponse, each HEAD followed by a second of listening for bytes after its head. All eight now match on status, Content-Type, Content-Length and Transfer-Encoding, and neither server sends a byte after a HEAD's head. Before, the 404, the redirect, the 204 and the 304 each carried application/octet-stream, the 204 and 304 a content-length: 0, and the FileResponse HEAD content-length: 0 where uvicorn sent 3145735. The streamed HEAD was found while fixing the rest, on apps/asgi_bare: all 10,000 bytes of a 10,000-byte stream followed its head (L27).

Re-run of every scenario, 2026-09-23 (after L25 and L26; the same versions). Sixteen scenarios, the heads probe and --workers 2, m0serve beside uvicorn. A GET's scope and a sized POST's now carry exactly the headers uvicorn's do. Before, m0serve's GET added a content-length: 0. A chunked POST's scope differs by design, content-length: 7 where uvicorn passes transfer-encoding: chunked, and so FastHTML's multipart check still accepts a chunked upload under m0serve that it refuses under uvicorn. 00_game_of_life, unmodified, now exits 1 naming what its module did and the lifespan fix, where it printed only no running event loop. Every other difference is the harness's own:

• file timestamps;
• the order of a Set Starlette joins into Allow;
• traceback paths;
• m0 in the lifespan state;
• the %2F recorded above.

FastHTML's own three failures are unchanged, and identical under both servers.

In production — 2026-09-12, textshelf's streams on the hold mount (m0serve 1.2.0)

Not a soak: a production record, the first use of the grant-verified hold mount (SPEC I21) by an application nobody wrote for this server. textshelf (Django 6.1, allauth, Postgres, one Fly machine, --realtime --blocking-threads 8) had run its five SSE streams as M0-Hold views since its cutover to m0serve; today it moved them onto --mount /_m0/rt=hold. Each view keeps its authentication and membership check and, asked with ?grant=1, signs its decision into the stream URL (m0serve.grant.stream_url, bound to the session cookie, an hour's TTL) instead of holding; the page renders that URL, and a Mojo pool thread verifies the grant and holds with no Django request per connection. Its side of the change is textshelf PR #570 (textshelf/utils/realtime.py, a template tag, one renewal script, the four process commands), with the M0_GRANT_KEY secret staged on each app before the command that needs it ran.

Local, under the tree's 1.2.0 binary, before the deploy. The view issued a grant into /_m0/rt bound to the session; production_stream_check.py followed it — 200, text/event-stream, both M0- headers stripped, : open as the mount's head, a heartbeat at 15 s; a notifications_read published from inside the process (mark-all-read) arrived on the mount-held stream; and the mount refused a missing cookie (invalid, no session cookie), a different session's cookie (invalid, session mismatch) and an expired grant (expired), each by name.

Staging (textshelf-staging.fly.dev), after the deploy. The two remote stream smokes passed through the suite's own Playwright login (the view holds; the URL the view issues is a held stream). The production check on the grant route: status 200 after 0.1s, held by the mount, : open, heartbeat at 15s, ok — through Fly's proxy, with the same Cache-Control: no-cache and X-Accel-Buffering: no the view-held stream carries. A malformed grant is refused as {"error":"invalid","reason":"malformed"}.

Production (textshelf.com). Promoted through textshelf's own workflow (staging validated, then promote-to-production.yml, then its post-deploy smoke, all green). The machine runs the 1.2.0 wheel's binary as m0serve --mount /=config.wsgi --mount /_m0/rt=hold --realtime --blocking-threads 8 ... with M0_GRANT_KEY in its environment; the mount refuses a malformed grant as {"error":"invalid","reason":"malformed"}, the view answers 401 without a session, and /_m0/health answers. Then the check with a real session, run by the owner from a signed-in browser the same day:

GET https://textshelf.com/notifications/stream/?grant=1
  status 200
  stream url /_m0/rt/stream (the hold mount)
  content-type 'text/event-stream'
  hold headers stripped, so m0serve took the hold
  `: open` from the mount
  heartbeat at 15s: : heartbeat
ok    stream is held

The view issued the grant, the mount took the hold, and the heartbeat only m0serve writes arrived on schedule through Fly's proxy: a stream held for a real user of a real application on a Mojo thread that never touched the interpreter.

Two things the pass found, neither in the server. The check script read a stream's head and then nothing against staging over TLS, on the view path as much as the mount: a fresh iter_content per frame marked the response consumed, and a single generator reading the same stream heartbeated on time — textshelf PR #571. And the first staging check's 24 s heartbeat window started before the connect, which a machine waking from auto-stop can consume; the window now starts at the response.

What this does and does not say. The mount holds real streams for a real application behind a real proxy, refuses what it should by name, and renews through the view when a grant expires — the browser side of that, held_stream.js, is exercised by every page that opens a stream and by no gate yet. The application layer's own soak was unchanged by it (NOT MET then; recorded below on 2026-10-02): this is the server holding for textshelf, not textshelf on Views and Fragment.

Re-soak — 2026-10-09, against 1.14.0 on the release branch at 57e9150, all four applications

The staleness rule, ahead of the release. The 1.11.0 pass is two minors behind 1.13.0, the limit, so 1.14.0 would have read it STALE. What moved under these applications since: the review of the fork's HTTP core, shipped in 1.12.1 and 1.13.0, changed where a request head ends, how pipelined and half-closed connections are answered, and how a stream open at a SIGTERM ends, which the rows' keep-alive bursts, abandoners and restarts all cross; and 1.14.0 closes a stream whose client has gone in the pass whose send failed, the path textshelf's abandoners take on its stream route. The access log is off in every row, as before.

/tmp/soak/ did not survive a reboot, so every checkout was made fresh: transcripts at 00dcb7a and bakerydemo at c8f8255, the 2026-10-06 commits; textshelf at 7479ee7, one commit on that touches only its editor settings, against its scratch database textshelf_soak, whose soak user had survived; and color-separation at 0f81a9c, thirteen commits on from 65c5445, with Django 6.1.1 and Pillow 12.3.0, its output/ package now tracked and m0serve now a dependency. The seeded fixtures and the processed archive came out at the sizes the manifests pin, and no manifest needed a change. transcripts' fresh venv had no gunicorn of its own, and without one its reference capture would have run the system Python's; gunicorn 26.2.0 was installed into the venv first. Every reference capture was re-recorded the same evening. bin/m0serve was built from the release branch and reports m0serve 1.14.0, and textshelf's venv, which imports m0serve.m0pub, carries the 1.14.0 wheel. CPython 3.13.6 (3.12.11 for color-separation and bakerydemo), macOS 27 on an M4. Raw driver outputs and the scripts that drove it are in bench/soak/2026-10-09/.

206,933 responses verified byte for byte, zero failures. No row's active connections rose above the nine its populations open, free slots mirroring them; the last sample is taken with traffic still in flight, as on 2026-10-06. Every upload landed: 286 of 286 in the WSGI row, whose restart cut none this time, and 214 of 214 under the executor.

The rows ran while a remote session's node jobs held a core at times, as the 2026-10-06 rows ran beside another session's, and the counts moved against that pass's both ways: bakerydemo 20,559 against 15,306 (306 logins against 199), textshelf's WSGI row 27,435 against 22,221, transcripts 26,041 against 30,159. The rows assert bytes rather than rate, so the figures are recorded and not explained.

Nothing was found on the wire. The one application that moved, color-separation, moved through its own dependency upgrades, and the server answered it byte for byte as gunicorn and uvicorn did.

Re-soak — 2026-10-06, against 1.11.0 on the release branch at 3e64f2a, all four applications

The staleness rule, ahead of the release. The 1.8.0 pass is two minors behind 1.10.0, the limit, so 1.11.0 would have read it STALE. What moved under these applications since: an ASGI application's lifespan startup is now waited for before the ready banner, and a failed one exits 1 (L33), which is the path both ASGI rows start on and the textshelf row's restart takes; the loop counts a response's body as sent, file bodies included, and --metrics, which every row sets, counts file bodies with it (F2); and --max-body, which the color-separation rows set to 64m, gained an environment form and a first-refusal notice beside it. The access log, whose line changed shape (F20, F21), is off in every row, as it was before.

The 2026-10-02 checkouts under /tmp/soak/ had survived and were used as they were: transcripts at 00dcb7a, color-separation at 65c5445, textshelf at 7981657 against its scratch database textshelf_soak, bakerydemo at c8f8255. Every reference capture was re-recorded first, and no manifest needed a change. bin/m0serve was rebuilt from the release branch and reports m0serve 1.11.0, and textshelf's venv, which imports m0serve.m0pub, carries the 1.11.0 wheel. CPython 3.13.6 (3.12.11 for color-separation and bakerydemo), macOS 27 on an M4. Raw driver outputs and the scripts that drove it are in bench/soak/2026-10-06/.

206,894 responses verified byte for byte, zero failures. Each row ends with its active connections within one of where they began and never above the nine its populations open, free slots mirroring them: the last sample is taken with traffic still in flight, as on 2026-10-02. One upload of 303 was cut by the WSGI row's restart and counted as tolerated, as one of 256 was then.

bakerydemo verified 15,306 against 21,490 in the same 180 s, signed in 199 times against 311, and its longest restart window was 2,055 ms against 1,177. The machine was not quiet: the rows began minutes after a stage of sabotage builds, while Spotlight was indexing the new release worktree, and another session's node process held a core. The row asserts bytes rather than rate, so the figure is recorded and not explained.

Nothing was found on the wire, and nothing moved with the applications: the four checkouts are the last pass's, and so is every manifest.

Re-soak — 2026-10-02, against 1.8.0 on main at d624ae7, all four applications

Run ahead of the staleness rule this time. At 1.8.0 the 1.6.0 pass is two minors behind a limit of two, so the milestone still read MET, and 1.9.0 would have read it STALE. Between the two passes the fork under every row was reviewed and cut: its typed socket errors became one SysError, the blocking accept loop and about 870 lines nothing used were removed, and 1.8.0's changelog lists the request-path fixes that came with it — a POST's leftover timer, a keep-alive request's second registration, a listener closed once by its drain, response heads that refuse CR and LF (G1, G2). So this pass asks the old question of a server whose loop was rewritten in place.

Fresh clones under /tmp/soak/ (the 2026-09-24 checkouts had not survived): transcripts at 00dcb7a, color-separation at 65c5445 with its gitignored output/ source copied in, textshelf at 7981657 against its scratch database textshelf_soak, bakerydemo at c8f8255. Every reference capture was re-recorded first. bin/m0serve was rebuilt from main and reports m0serve 1.8.0; CPython 3.13.6 (3.12.11 for color-separation and bakerydemo), macOS 27 on an M4. Raw driver outputs and the scripts that drove it are in bench/soak/2026-10-02/.

211,661 responses verified byte for byte, zero failures, and every row ends with its active connections and free slots where they began. transcripts verified a third fewer than on 2026-09-24 in the same 150 s (26,563 against 40,117), back at the 2026-09-13 figure (29,989); the build that preceded the rows ran on the same machine minutes earlier, and the row asserts bytes rather than rate, so the figure is recorded and not explained.

Nothing was found on the wire. Three things moved with the applications or the calendar, and the first cost a run of the two textshelf rows:

• textshelf at 7981657 renders a masked CSRF token into every signed-in page's hx-headers attribute, new bytes on each request. The first run of its rows failed 8,273 comparisons on /notifications/, /users/~update/ and /feeds/tokens/, each a body of the capture's length with one differing token. The manifest gained a body_sub rule for "X-CSRFToken": "…", daphne was re-captured under it (the rules fingerprint refuses the old capture), and both rows were run again; the table carries the second run.
• textshelf's components.css is 103,602 bytes at 7981657, where the manifest had 121,037. WhiteNoise under daphne serves the new size.
• color-separation files media by month, and the manifest names 2026/09. The checkout's media/uploads/2026/09 and media/outputs/2026/09 are symbolic links to 10 for this pass, so the manifest's two media routes read the files job 1 produced today.

Re-soak — 2026-09-24, against 1.6.0 on the release branch at 9b051fd, all four applications

The staleness rule again. At 1.6.0 the 1.3.0 pass is three minors behind a limit of two. Between the two passes the request path these applications take did change: a parsed request now carries only the headers its client sent, so a GET reaches Django with no CONTENT_LENGTH (L25); the ASGI executor's slot, disconnect and background-task rules moved for Starlette's sake (L20–L28), and textshelf and color-separation run on it; every descriptor the server creates is close-on-exec (G16); and a raw @ in a target is data (A22). So this pass asks the old question of a server whose request path shifted, not merely grew.

Fresh clones under /tmp/soak/ (the 2026-09-13 checkouts had not survived): transcripts at 00dcb7a, color-separation at 65c5445 with its gitignored output/ source copied in, textshelf at 1ef2bfa against its scratch database textshelf_soak, bakerydemo at c8f8255 with the asgi.py this record added in 2026-09. Every reference capture was re-recorded first. One manifest pin moved with an application rather than the server: textshelf's components.css is 121,037 bytes at 1ef2bfa, where the manifest had 125,113, and WhiteNoise under daphne serves exactly the new size. bin/m0serve reports m0serve 1.6.0; CPython 3.13.6 (3.12.11 for color-separation and bakerydemo, which pin it), macOS 27 on an M4. Raw driver outputs and the scripts that drove it are in bench/soak/2026-09-24/.

216,617 responses verified byte for byte, zero failures, and every row ends with its active connections and free slots where they began. transcripts verified a third more than on 2026-09-13 in the same 150 s (40,117 against 29,989). The rows otherwise read as that pass did: color-separation's two rows carry the application's own RSS (finding 6), its ASGI row's threads climb as asgiref's pool fills (25 → 31, max 36), and textshelf's WSGI row shows the startup transient falling away.

Nothing new was found on the wire. The run found three problems in the pre-release tooling and none in the server:

• sabotage-blobs reported a rule no gate guards. Its anchor still matched an import line that the host's command line had changed (cda3cd0), and it now matches the new one.
• The browser gates failed at launch. The Playwright that uv run --with resolved wanted a Chromium the cache lacked, and playwright install chromium fixed it.
• The first bench-linux-conclusions and autobahn runs came back contaminated, one with its spreads up to 23 % and the other with a thin section, while the host was busy. Both were discarded and rerun with the machine quiet.

Re-soak — 2026-09-13, against 1.3.0 on the release branch at 8c43979, all four applications

The staleness rule is why this pass exists, and this time the code under it moved a great deal. At 1.3.0 the 1.0.0 pass is three minors behind a limit of two, so poe milestones prints the soak STALE and the 1.0 milestone stops being MET. Between the two passes the tree gained m0-postgres and a Postgres NOTIFY door onto the broadcast bus (O6–O16, I22), a Mojo mount an application supplies as a module (N14), sessions and CSRF behind a login (N13), the Datastar form (N12), and a fix for a splitter that a request could trap (G14). Only the last is on the request path of these four applications; the rest is reached by configuration none of them sets. So this pass asks the same question the others did — does somebody else's Django project still work, byte for byte, under churn — against a server whose surface grew rather than shifted.

The checkouts under /tmp/soak/ survived this time, which is unusual. Every reference capture was re-recorded first regardless: they dated from 2026-09-10 and the rule is to re-baseline before each pass, because the soak's own traffic mutates the applications' databases and a stale capture fails on CONTENT, which reads as a server defect and is not one. textshelf ran against its scratch database textshelf_soak, never the adjacent dev database — the mistake a previous pass made, which put soak data into real developer data. bin/m0serve was built from the release branch and reports m0serve 1.3.0; CPython 3.13.6, macOS 26 on an M4. Raw driver outputs and the scripts that drove it are in bench/soak/2026-09-13/.

214,156 responses verified byte for byte, zero failures, every sample reporting failures=0 throughout. Descriptors and threads flat in every row but one: color-separation's ASGI row runs 23 → 32 threads (max 33), which is asgiref's thread pool filling under the executor, as the 2026-09-10 record says of the same row. The churn column reports the DRAIN, with the whole restart window beside it — 2026-09-10 reported drains alone, so reading this pass's window (1219 ms) against that pass's drain (0.26 s) would look like a regression and is not one: the drains here are 82–150 ms. The two color-separation rows carry the RSS the record has called the application's since 2026-09-10 (finding 6): gunicorn's two workers sum 1.5–1.8 GB on the same workload. textshelf's WSGI row again shows the startup transient falling away rather than growth.

Nothing new was found on the wire. The defect this release run did turn up was in the pre-release TOOLING rather than the server: the Linux container's build list never gained postgres after m0-wsgi began importing m0_postgres for --pg-listen, so build-wsgi could not build in the container at all and stress-pool failed before its first round. CI cannot see it — there is no such container there — which is the reason that tier exists. Fixed in the FOUR places that name the list: scripts/probes/linux_sync.sh, the stress-pool task, scripts/bench_linux_conclusions.py (which would have hit it next), and scripts/probes/linux_setup.sh — the last found in review, and the one that matters most, because it is what CREATES the container. Without it a fresh m0lin could not be built at all; this run escaped only because its container already existed.

Re-soak — 2026-09-10, against 1.0.0 as merged at a4f0aa5, all four applications

The version number is why this pass exists, and almost nothing else is. The 1.0.0 release bumped pyproject.toml and left the soak record naming 0.19.0, so poe milestones printed the soak STALE from the merge commit onward — the milestone's staleness rule allows two minors of lag and scores a major bump far past it. The release commit's own message says the soak is current against this version; it was true when written against 0.19.0 and false the moment the bump landed in the same commit. This pass is what makes it true.

The code under it barely moved. Between the 0.19.0 pass and this one, exactly one commit touches the request path: bf212a3, which splits the graceful drain into begin, step and finish so the loop inversion can drive it one pass at a time. The blocking composition every default topology runs is unchanged, and the stepped path is reachable only under M0_INVERTED=1. Everything else since is --doctor reporting (usable_cpus, topology.loop), the FastAPI gate, the row-status ratchet and the Linux benchmark work — none of it on the wire. So the interesting half of this pass is the three churn rows, which SIGTERM the server mid-load and are the only place the corpus exercises that refactor.

Fresh clones under /tmp/soak/ (the 2026-09-07 checkouts had not survived, as always), each with its own venv on PATH, bin/m0serve rebuilt from a4f0aa5, CPython 3.13.6, macOS 26 on an M4. Same driver, same five populations, same durations and churn cadences; every reference capture re-recorded first. Raw driver outputs and the scripts that drove it are in bench/soak/2026-09-10/.

215,214 requests verified byte for byte against the reference captures, zero failures in every sample of every row, every slot returned (the free-slot count back at its starting value at every sample), threads flat, and 5,682 abandoners across the run absorbed, half closing with FIN and half with RST. No server defect. The six drains all finished inside 0.26 s, which is the observation this pass was really after: the split left the composition alone, and a SIGTERM under load still ends in a quarter of a second.

Nothing moved against the 2026-09-07 table that is not already explained there. The color-separation gigabytes are its image renditions; textshelf's WSGI row again opens on the ~490 MB startup transient that finding 7 describes and settles inside the minute; the executor rows again carry more threads than the pooled ones, which is asgiref's thread pool for that app's synchronous views. One count differs by one on the color-separation churn row — 343 uploads, 342 landed — and it is the same shape 2026-09-07 recorded (344 and 343): the POST is counted when it answers, the follow-up GET of the page it redirects to is what marks it landed, and a restart between the two is a tolerated connection error rather than a failure.

Three things about the corpus changed, all setup rather than server.

• color-separation's fixtures are seeded now. Its two images are noise PNGs, generated fresh each pass and therefore a different size each pass, which is what forced the manifest's bytes pins to be re-measured by hand — the third of the three traps this record already lists. scripts/soak_fixtures.py generates them from seed 20260910; a second run reproduces both files byte for byte, so the pins are now a property of the manifest rather than of the run. They moved once more in the process, this seed's zip being 7.7 MB where the unseeded one was 9.6.
• The rows script no longer carries a database password. 2026-09-07's copy had textshelf's DATABASE_URL with the secret redacted by hand before it was committed, which is a step that only has to be forgotten once. libpq reads ~/.pgpass for the postgres role, so the URL in bench/soak/2026-09-10/soak_rows.sh names no password at all.
• textshelf now imports m0serve.m0pub at startup, so the subject cannot be served until the wheel is installed in its venv. This pass built the tree's own 1.0.0 wheel and installed that, which makes the application's realtime imports part of what the soak covers rather than something stubbed around.

What this pass did not do, unchanged from the records above: --threads (none of the four dependency trees builds on free-threaded CPython), NiceGUI, the proxy and worker shapes. It also did not exercise the stepped drain itself, the one thing that changed on the request path: that path needs M0_INVERTED=1, which no row here sets, and its gate is smoke-asgi's inverted arm on every pull request.

Re-soak — 2026-09-07, against 0.19.0 as merged at abf2016, all four applications

Three minor-version's worth of change landed between the pass above and this one, and two of them are on the request path of every keep-alive connection: the elastic pool (0.19.0 serves a trivial view on one handler thread until a job has waited, and wakes each thread by name on a channel of its own) and the keep-alive request cap, 1000 from 100, with --max-keepalive-requests to set it — the cap having turned out to be the whole of the fast-route tail the benchmark page had conceded to Granian (notes/pool-tail.md). So the corpus was run again on the release commit: fresh clones under /tmp/soak/ (the 2026-09-01 checkouts had not survived), each with its own venv on PATH, bin/m0serve built from abf2016, CPython 3.13.6 (the GIL build a pip install gets), macOS 26 on an M4. Same driver, same five populations, same durations and churn cadences; every reference capture re-recorded first, as the rule from the pass above says. Raw driver outputs and the three scripts that drove it are in bench/soak/2026-09-07/.

Every row: zero failures, every slot returned (the free-slot count back at its starting 1,015 at every sample), threads flat, every drain inside a fifth of a second, and 560–1,550 abandoners per run (half FIN, half RST) absorbed. No server defect. Two shapes moved against the 2026-09-04 table and neither is growth: the pooled rows hold 16–20 more descriptors from their first sample (the elastic pool parks each of the eight handler threads on a SOCK_DGRAM pair of its own — two descriptors per thread, allocated at start, flat for the run), and the executor rows carry more threads than before (29–33 on color-separation against 20), which is asgiref's thread pool for that app's synchronous views under a larger upload fixture, the application behaviour the 0.16.0 record already named. The color-separation gigabytes are its image renditions, as before, and textshelf's WSGI row again opens on the ~490 MB startup transient the record's finding 7 describes, settling inside the minute.

Three rows failed first, and none was the server. Recorded because each cost time and the next pass should not pay it again:

• transcripts and bakerydemo, "never became ready" — the runner had put <checkout>/bin rather than <checkout>/.venv/bin on PATH, so m0serve embedded the system interpreter, which has Django and none of the apps' packages. The driver's log named the missing module on the first line of the traceback.
• transcripts and textshelf came up on free-threaded 3.14t — the release worktree's benchmark swap had left that interpreter installed and uv sync preferred it, so textshelf's psycopg had no wheel to install. Both venvs were rebuilt with --python 3.13; color-separation pins 3.12 itself and kept it.
• color-separation 3,062 and textshelf 22,752 "failed" stream reads, each one shape: the manifests pin a bytes value for their large streams, and the pins were the 2026-09-01 fixtures' — a big.png twelve bytes larger than the regenerated one, job 1 processed from the big image (this pass had made it from the small one; rebuilt, and the zip is 9.6 MB where a photograph's was 4.7 — the fixture is noise, which does not compress), and a components.css that textshelf has since grown by 187 bytes. The pins now carry the fixture sizes with a comment saying so. The rule the pass above added — re-baseline before every pass — covers the captures; a manifest pin is a second expectation and rots the same way, so re-pin with the fixtures.

What this pass did not do, unchanged from the records above: --threads (none of the four dependency trees builds on free-threaded CPython), NiceGUI, the proxy and worker shapes.

Re-soak — 2026-09-04, against 0.17.1 as merged at 0766272, all four applications

The event loop stopped holding a thread state while it serves (PR #229, docs/notes/detached-loop.md): the default runtime shape of every pooled and executor deployment changed, and the smokes that gated it were all written here. So the corpus was run again the same night — the four checkouts of 2026-09-01 under /tmp/soak/, each with its own venv on PATH, bin/m0serve built from main at 0766272, CPython 3.13.6 (the GIL build a pip install gets), macOS 26 on an M4. Same driver, same five populations, same reference captures except where noted. Raw driver outputs are in bench/soak/2026-09-04/.

Every row: descriptors and threads flat, the server's free-slot count back at its starting 1,015 at every sample (every slot returned), zero 5xx, and 800–1,850 abandoners per run (half FIN, half RST) absorbed. The color-separation gigabytes are the application's image renditions, as in the 0.16.0 record; textshelf's executor RSS fell over the run, the sawtooth the record describes. No server defect.

Two rows failed first, and neither was the server. Recorded because each cost an hour and the next pass should not pay it again:

• transcripts, 2,798 "failed" stream reads — every read of the typst PDF. Its normalised digest no longer matched the 2026-09-01 capture; a fresh gunicorn from the same checkout produced the same digest as m0serve, four fetches out of four, so the capture was stale: the soak's own traffic had changed the database it renders from (db/transcript.db modified that evening). Re-captured from gunicorn, re-run: the row above. The rule this adds to the driver's page: re-baseline before every pass, not only after a normalisation change — and before blaming the server, fetch the route from the reference server and compare.
• textshelf, 3 of 3 logins answered 200 in both modes. The runs had been pointed at the database the checkout's .envs/.local/.django names — the development database — where no soak account exists; daphne answered the same 200. The soak's database is the scratch textshelf_soak (the 0.16.0 record's setup), and the rows above are against it. A soak account created in the development database while diagnosing this was removed the same minute.

One observation, not investigated. textshelf's churn drain took 0.19 s with two stream clients and an abandoner in flight, where the 0.16.0 record measured 5.0 s ("held streams never end, the drain waits its budget by design"). What changed between is D9 (2026-09-02): the drain now runs ordinary loop passes, so a stream whose client has gone is closed rather than waited for. Consistent, and not a defect either way; a pass that wants the claim can open a stream, hold it, and time the drain.

What this pass did not do, unchanged from the record above: --threads (none of the four dependency trees builds on free-threaded CPython), NiceGUI (phase 0 only, no manifest), the proxy and worker shapes.

What would make this worth repeating

The shapes these four still lack: a WebSocket application (NiceGUI passed phase 0 — its socket.io transport works in a real browser through the executor — and has no manifest yet), a real proxy in front (X-Forwarded-Proto → SECURE_PROXY_SSL_HEADER, which textshelf on Fly will supply), a background worker, and free-threaded --threads. And a week of real traffic, which is the plan.

The previous record. Run 2026-08-26, against 0.11.0 (the PyPI-shaped wheel, dist/wheels/m0serve-0.11.0-py3-none-macosx_13_0_arm64.whl), macOS 26 on an M4, CPython 3.13.6 — plus one row on free-threaded 3.14.7t. The plan this replaces is in the history of this file.

Every application this server had been tested against was written to test it. apps/ holds bare WSGI and ASGI apps that pin spec clauses, a Django demo built around the realtime feature, and a FastHTML row — all small, all ours, none carrying a dependency tree somebody else chose. That is the right shape for a smoke suite and the wrong shape for the question "would this serve my application?"

The gap was not hypothetical. The one time the wheel had met a real Django project it produced the --app-dir shadowing defect (Known issues). This pass produced four more, three of which no in-repo app could have shown, and every one of them is now fixed with a test that fails without the fix.

The three applications

Local Django projects, chosen because they stress different parts of the server rather than because they are convenient. Served from clean clones against scratch databases; their own settings, unmodified.

What was found

Five defects. Four are fixed in this repo with a guard; one was already known and stays open on purpose.

Findings 1, 3 and 4 could not have come from apps/: they need a real cookie policy, a real static-file middleware, and a real never-ending response respectively. Finding 2 needed a file somebody wanted to upload.

Phase 0 — will it even load?

m0serve --doctor against each entry point. All six passed ("ok": true, exit 0): transcript_manager.wsgi, halftone_studio.wsgi, halftone_studio.asgi, config.wsgi, config.asgi, and transcript_manager.wsgi again under --threads 4 on 3.14.7t, which reported free_threaded_build: true and mode: threads. Protocol detection was right every time; the WSGI apps got the zero-config pool of 8, the ASGI apps got the executor.

transcripts keeps its code in src/, so it is the shape the --app-dir defect targets. It loaded from both --app-dir src and --app-dir . (it is also installed as an editable package), and neither path was shadowed.

Phase 1 — parity against runserver

Same routes under m0serve and manage.py runserver, raw HTTP/1.1, no redirect following, diffed on status, headers and body after normalising what is per-process by design (Date, CSRF token, session id). Home, admin index, admin login, allauth login, a static asset, a 404, a 500, and a full login round trip on each app.

After the cookie fix, every remaining difference is one of three, and all three are the server being itself:

• connection: keep-alive — m0serve sends it explicitly; runserver does not.
• x-thread: N — which handler-pool thread served the request. Documented, and absent under --blocking-threads 0.
• Django's debug pages embed the request's own host:port, which differs because the two servers are on different ports.

Bodies were byte-identical on every route on all three apps, admin included. The login round trip — GET the form, POST credentials, GET the destination with the session — produced identical headers at all three steps on color-separation and textshelf. Before the fix the same comparison reported 15–24 differences per app, every one of them a cookie.

Two app-level failures appeared identically under both servers and are the applications', not ours: transcripts answers 403 on /accounts/login/ (its dev auto-login middleware logs you in, and allauth's login view forbids an authenticated user), and textshelf answers 500 on /accounts/signup/ (no SocialApp row in a scratch database).

Phase 2 — the feature matrix

Phase 3 — the topology matrix

Each app, each mode: a route sweep, a 200-request burst at concurrency 16, and 16 concurrent logins (a session write per request).

Every mode served every route correctly on all three apps, with zero 5xx and zero dropped connections in the bursts — --workers 1, --workers 4, --blocking-threads 4, --workers 4 --blocking-threads 4, and --threads 4 on free-threaded 3.14.7t (transcripts, which is the app whose dependency tree is thread-clean).

The 16-concurrent-login column found nothing about the server and one thing about the fixture: textshelf answers some of them 500 with sqlite3.OperationalError: database is locked. That is the scratch SQLite — the project runs PostgreSQL — and it reproduces at the same rate under gunicorn --workers 4 and under runserver, so it is attributed there rather than here. transcripts answers all 16 with the same app-level 403 as in Phase 1. textshelf also rate-limits its own auth endpoints (429), which is the application working as designed.

Phase 4 — the realtime retrofit (textshelf)

The claim is that a plain sync view can hold an SSE stream with two response headers and no added infrastructure. textshelf/notifications/ was the honest test: an async notification_stream view, an async_subscribe generator over PostgreSQL LISTEN/NOTIFY, a polling fallback for SQLite, a sync copy of both, and a hard dependency on running under Daphne.

The conversion is 20 lines of view:

@login_required
@require_http_methods(["GET"])
def notification_stream(request):
    response = HttpResponse(f"event: connected\ndata: {head}\n\n",
                            content_type="text/event-stream")
    response["M0-Hold"] = "stream"
    response["M0-Channel"] = NotificationPubSub.get_user_channel(request.user.id)
    return response

plus one m0pub.publish beside the existing pg_notify in the publish path, deferred to commit the same way.

Measured result: +52 / −293 across two files — a net 241 lines removed, because holding the connection in the server strands all four subscription implementations (sync, async, and a polling fallback for each: 236 of pubsub.py's 336 lines), and with them the psycopg import that existed only for LISTEN/NOTIFY.

It works: four EventSource clients held across two prefork workers, one POST to an ordinary synchronous Django view, and all four receive the same numbered event. No Daphne, no asyncio, no LISTEN/NOTIFY, and the database is SQLite — where the code being replaced had no delivery mechanism at all, only a 5-second poll.

The degradation is better than what it replaces, which was the surprise. Under gunicorn the converted view answers a short buffered text/event-stream that the browser's EventSource reconnects on — the GRIP property. The original async view under gunicorn hangs: 12 s with no bytes, and the worker never serves again. So this is not a change that trades portability for the feature.

The honest caveats: the app loses its SQLite polling fallback (nothing delivers under gunicorn now, where before it polled — at the cost above), and M0-Channel is one channel per connection, which suits a user-scoped notification stream and would not suit an endpoint that multiplexes several.

Verdict: a diff a maintainer would accept. The claim survives contact.

Phase 5 — soak

6,000 keep-alive requests per app over a mixed route set (home, admin, login, static, 404, an authenticated page), sampling RSS, open descriptors and threads.

Descriptors and threads are flat — nothing accumulates per request, and Django's CONN_MAX_AGE = 10 holds its connections steady rather than growing them. RSS rises 1.5–6.5 MB over the 5,000 requests after warm-up and then stops, which is Python's allocator reaching steady state rather than a leak; smoke-django's 10k-request guard still measures 0 KB against the bare app. Throughput was 363–819 rps single-worker on a laptop, which is not a benchmark and is not offered as one.

What this changes about the guards

Every fix landed with something that fails without it, and each was checked by sabotage rather than assumed:

• test_response_cookies.mojo — 4 of 6 cases fail when add_raw is reverted to parse-and-reserialise.
• test_config.mojo::test_recv_buffer_limit_covers_headers_plus_body — fails when recv_buffer_limit() returns the bare field.
• smoke-django — now reads the session cookie off the wire and requires expires, HttpOnly, Max-Age and SameSite=Lax, because curl's cookie jar stores name and value only and could never have seen this.
• smoke-serve — a 3 MB body at the default cap and 7 MB at --max-body 8m must be 200; 5 MB and 9 MB must be 413. Both sizes straddle the old ceiling.
• smoke-asgi — 32 concurrent 4 KB-piece streams, every body byte-exact. This one paid for itself on its first CI run, and is why the fix looks the way it does. It passed on macOS and failed on Linux: two streams short, because the budget had been sized against a 256 KB socket buffer and Linux clamps SO_RCVBUF to net.core.rmem_max and charges each datagram's whole skb against it. Raising the number would have made this platform pass and left the next to find out, so the number stopped being load-bearing instead — a full channel now waits (detached, so the loop can drain it) rather than dropping. Verified by removing the budget rather than trusting it: at _ASGI_TOTAL_WINDOW = 64 MB, effectively no budget at all, 64 concurrent streams still deliver every byte and log no drops.
• smoke-blocking-threads — two views that never return, SIGTERM, and the process must exit inside 20 s naming what it abandoned.

Revisited after stage 1 (2026-08-26)

--realtime composing with --blocking-threads changed the answer this record originally implied, so the question was re-measured rather than re-reasoned. Same laptop, same scratch SQLite, five events 0.3 s apart, arrival time of each event recorded by a client that reads incrementally — curl -w %{time_starttransfer} will not do, because several of these servers send an empty first body chunk that satisfies it.

Two things fall out, and the first is about their application rather than any server. textshelf's AI streaming endpoints do not stream, on their own production server. ai/views.py:summarize_submission is a sync view returning create_sse_response(...) over services.summarize_document_stream, whose annotation is Generator[str, None, None] — a sync generator, which Django's ASGI handler must consume before it can serve it (it says so in a warning). Every ASGI server buffers it identically; the token-by-token delivery the endpoint is written for arrives in one lump at the end. The rows above are a same-shaped instrument rather than their endpoint (which needs an API key), but the shape is what decides it, and the shape is theirs.

The second: m0serve's executor matches uvicorn and daphne to the millisecond in both rows. Whatever streams under them streams under it, and what buffers buffers everywhere.

That changes the recommendation this record's Phase 4 pointed at. Before stage 1, holds cost the pool, so the hybrid mount was the only shape with both isolation and working streams. Now:

The AI endpoints cost nothing to move, because they are already buffered wherever they run. So the choice was between the cheapest way to hold a stream and no application changes at all — and what forced it to be a choice was that --realtime and --mount could not be combined (measured the day this was written: exit 1, refused before the bind).

That is no longer true, and this pass is why. The ROADMAP entry it produced was implemented the same day: --realtime now composes with --blocking-threads, with --mount, and with WebSocket holds under both (ROADMAP, Hold on a pool thread, stages 1–3). So the row this table calls "convert 4 pub/sub endpoints" no longer costs the pool or the ASGI mount — an application can hold its pub/sub streams, run its generator streams on an executor mount, and keep pool isolation for its slow views, in one process. What is left for textshelf is application work: converting the four endpoints, and making ai/streaming.py's generator async so it streams at all.

Re-soak — 2026-08-31, against 0.16.0, textshelf only

Five minor versions landed in five days after the record above — --mount, per-mount lanes, the batched pump, ExecutorPort, the WebSocket credit gate, the close linger — all on the request path, none of it seen by a real application. textshelf was re-run first because it is the app with four SSE endpoints, so it stresses the seam that churned most.

It found one defect, and the defect was on the ASGI path the WSGI record above could never have reached. Clean clone, scratch SQLite, config.settings.local, the 0.16.0 wheel, CPython 3.13.6, macOS 26 on an M4.

The WSGI row matches the 0.11.0 shape: +4.3 MB over 5,000 requests, flattening to +0.4 MB across the last 2,000, with descriptors, database descriptors and threads all flat and the thread count identical at 13. 540 rps, inside the record's 363–819 range. The ASGI descriptor and RSS churn is asgiref's thread pool opening a connection per thread — application behaviour, not a leak.

The defect: the keep-alive cap destroyed the response it fired on

Nine of the 6,000 ASGI requests came back truncated, all of them on the same 124 KB WhiteNoise file, at intervals of exactly 700 requests — every hundredth time that route was hit. max_keepalive_requests was 100 (the default then; 1000 since notes/pool-tail.md, which found the cap itself to be the fast route's p99).

Isolated on one connection, and the two faces are the same bug:

_finish_response clears should_close for a stream and for a 101, because each owns its connection until it ends. The cap check below those branches was guarded by not should_close — precisely the state they had just established — so the two shapes that had opted out were the two it caught, and _after_send closed the slot as soon as the head drained, before any body frame arrived over the chunk channel. Both faces are silent: correct status line, nothing in the server log.

Gated by poe smoke-keepalive-cap (SPEC A3), whose third phase asserts the cap still fires for an ordinary response — without it the gate would pass on a build whose cap never fires. Four sabotages: reverting the fix, each guard alone, and raising the cap, each caught by the phase that should catch it.

The m0serve X.Y.Z marker in this file's opening line is the ONE the milestone gate reads, and it deliberately stays at the version the full three-application pass was run against. A second marker anywhere in the page would become a fallback for _real_app_version and silently defeat the sabotage that proves the gate can miss an unreadable record — which is why this heading says "against 0.16.0" rather than naming the binary.

This pass is not the 1.0 soak. transcripts and color-separation have not been run against 0.16.0, so the record above still stands at 0.11.0 for them and the milestone stays stale until they are.

What would make this worth repeating

It already was: four defects in one pass, three of them invisible to any application written to test this server. The next pass should use applications with shapes these three lack — a background worker, an app behind a proxy with SECURE_PROXY_SSL_HEADER, an upload-heavy API, and something that is not Django.

The application layer

Last run 2026-10-09, against m0-http 1.14.0 — unotes at 45a1cdd, built by the m0 0.11.0 wheel cut from the release branch at 57e9150, ahead of its upload, macOS 27 on an M4.

The application is unotes, a reader over a private notes corpus, written outside this repository on the documented path: m0 new from m0 0.1.0 as published, Views and Fragment[Htmx], a login on m0_http.login, the corpus read from SQLite through m0_sqlite, deployed as an image to Fly.io, and taken through four upgrades (0.2.0 to 0.5.0) with m0 doctor. Its SOAK_LOG.md is the running record. It was written to exercise the layer, by this repository's owner: it is the layer's first application outside apps/, not an independent adopter, and what it found is rows here already — Query (N36), push=True (N37), HEAD answered by GET (N38), SQLite opened at run time (O17, O18, N39, N40), the scaffold's login guidance, and the login module and auth template (N43–N45).

The synthetic soak. scripts/soak.py against a local bin/server on the real corpus, tools/soak_manifest.json its manifest, driven by bench/soak/2026-10-09/layer_unotes.sh. The reference is the same binary's answer to each route alone; every response under load is compared with it, status, headers and a digest of the body, the scan figure and the CSRF token normalised. Six keep-alive bursts over eleven routes, nine of them behind the session, four sessions signed in, two clients that abandon a response mid-body (FIN and RST alternating, paced at 50 ms) and at once reuse the slot, and a SIGTERM and restart every 40 s with everything in flight.

6,211 and 3,065 abandonments in the two rows. A request that met a restart and was refused or cut is counted apart and not as a failure (12,862 and 6,855), within a window of at most 524 ms a restart. The login population stays off (--bulk 0): the manifest sets no interval for it, and on 2026-10-02 it signed in about 170 times a second until the client ran out of local ports, every failure OSError 49 on a connect and no response wrong (unotes' SOAK_LOG, finding 15).

The application's source is its upgrade to m0 0.7.0; only the pin moved, in a scratch clone, to a wheel built from this release's tree. Nothing in it names a package export this release removes or a name it renames, so it built unchanged, in 15 s with no warning. m0 doctor read framework 1.14.0 and passed its six checks, m0 test passed 3 of 3 files (37 tests), and smoke.sh passed. The 2026-10-06 run, on 1.11.0, verified 1,409,040 and 814,210 in the same times, with the same zero; the rows assert bytes, not rate.

The deployed instance, unotes.fly.dev on a shared-cpu-1x, on m0 0.4.0 since 2026-09-30. Signed in on 2026-10-02: the list, a note and the themes page each answer 200, and the list signed out is a 303 to the login. The corpus scan on that CPU, x-scan-us, six requests of each:

The same figures as on 0.1.0. A deploy with a reader mid-session, and a phone that sleeps and reconnects, were taken on the deployed instance on 2026-09-24 with the phone emulated (the application's log, "the real use the log owed").

What this does not show. The framework as published: this run's is the release branch's, in a wheel not yet on the index. Traffic from anyone but its owner, an application with writes (unotes reads), a stream, or a second author's habits. The layer's write path is gated by apps/fragment_notes, apps/datastar_todo and apps/table_notes inside the tree and has no soak outside it.

textshelf, first named here, is not the soak: its streams moved onto the built-in hold mount in production on 2026-09-12 (above), and it writes no Mojo.

A record here opens with ONE line that poe milestones reads, the way the server's does:

**Last run DATE**, against m0-http X.Y.Z — on a single line; the version is the framework the application was built from, versions.framework in m0 doctor --json, not the m0 wheel's own number, which the reader does not accept.

It then names the application and the path it moved onto the layer, and carries the server soak's table: requests verified byte for byte against a reference, failures, churn, RSS. The milestone applies the server soak's staleness rule to that version.