Exercising the server against real applications

The server run against Django projects nobody here wrote, with what broke, what was measured and what changed as a result. A 1.0 requirement.

A record. Run 2026-09-01 and 2026-09-02, against m0serve 0.16.0 (bin/m0serve from the tree at c823198, the 0.16.0 wheel's source), macOS 26 on an M4, CPython 3.13.6. The plan and the previous records are below and in the history of this file.

Every application this server had been tested against was written to test it — until the 2026-08-26 pass below, which put three real Django projects in front of it and found five defects. This pass re-runs those three against a server five minor versions on, adds a fourth application nobody here wrote at all, and replaces the pass's phase 5 — six thousand requests sampling RSS — with a driver that asserts bytes: every response compared, status and headers and body digest, against a capture recorded from a reference server (gunicorn, uvicorn, daphne), under five concurrent populations, with logins, and with the server SIGTERM'd and restarted underneath. Three of the previous six defects were silent — a clean status over a short or empty body — and a request loop passes every one.

The four applications

Clean clones served from their own directories, each with its own venv, scratch databases, their own settings unmodified except where the row says.

What was found

One server defect, open, with its reproducer and its row. And six things that were not the server, each measured rather than assumed.

Finding 1 could not have come from apps/: it needs a multi-megabyte upload in flight at the instant of the signal, which only a population that uploads continuously produces. Finding 2 could not have come from the previous pass either, whose textshelf row never abandoned a stream.

Phase 0 — will it even load?

m0serve --doctor ok: true on every entry point: transcript_manager.wsgi (--app-dir src), halftone_studio.wsgi and .asgi, config.wsgi and config.asgi, bakerydemo.wsgi and an asgi.py added for it (the startproject file; bakerydemo ships only wsgi.py). Protocol detection right every time; WSGI apps got the zero-config pool of 8, ASGI apps the executor.

Two applications did not load from a clean clone, and the doctor could see only one of them. transcripts' README installs .[dev], which does not contain django-watchfiles although settings.py lists it unconditionally (it is in the uv lockfile's dev group; uv sync works). color-separation keeps real source under output/ — composite.py, raster.py, vector.py — and .gitignore excludes output/, so a clone fails on its first request with No module named 'output'; the doctor passed, because Django imports its URLconf lazily. Phase 0 answers "does the callable import", not "does the first request work".

Phase 1 — parity, byte for byte

Each application captured from a reference server and every response of the soak held to it. The only differences that survived were named substitutions (findings 4 and 5), a framing choice — uvicorn frames an undeclared-length body as chunked where m0serve buffers it and sends Content-Length; the decoded bytes are identical — and the four security headers runserver's StaticFilesHandler bypasses, on which gunicorn agrees with m0serve byte for byte.

Phases 3 and 5 — topology, churn, and the soak

Every row: five populations at once (keep-alive bursts at 120 requests per connection, over the cap of 100; streams read to completion; uploads, slow views and logins; WebSocket echoes where the app has any; abandoners that vanish mid-body by FIN and by RST and reuse the slot at once), sampling RSS, descriptors, threads and the server's own /__metrics. "verified" is responses byte-identical to the reference capture.

Descriptors and threads were flat in every clean row. --threads was not run: none of the four dependency trees builds on free-threaded CPython (psycopg-binary ships no wheel; the record above says the same).

What this changes about the guards

• The driver itself (scripts/soak.py, poe soak-apps, poe soak-selftest): its comparator is a pure function with a self-test that found the driver's own first hole — a substitution greedy enough to absorb a truncation blinds the instrument — so a capture now refuses any route its patterns blind and carries a fingerprint of the rules it was recorded under. Sabotaging a manifest's byte count makes the task exit 1 naming the failure.
• scripts/drain_upload_probe.py, finding 1's reproducer and the gate the fix will land with, two-sided like its sibling: the upload must be answered whole, and the process must exit inside 3 s of the signal.
• Four manifests under scripts/soak_manifests/ are the re-runnable record of each application's shape, including every substitution and the reason for it.

What would make this worth repeating

The shapes these four still lack: a WebSocket application (NiceGUI passed phase 0 — its socket.io transport works in a real browser through the executor — and has no manifest yet), a real proxy in front (X-Forwarded-Proto → SECURE_PROXY_SSL_HEADER, which textshelf on Fly will supply), a background worker, and free-threaded --threads. And a week of real traffic, which is the plan.

The previous record. Run 2026-08-26, against 0.11.0 (the PyPI-shaped wheel, dist/wheels/m0serve-0.11.0-py3-none-macosx_13_0_arm64.whl), macOS 26 on an M4, CPython 3.13.6 — plus one row on free-threaded 3.14.7t. The plan this replaces is in the history of this file.

Every application this server had been tested against was written to test it. apps/ holds bare WSGI and ASGI apps that pin spec clauses, a Django demo built around the realtime feature, and a FastHTML row — all small, all ours, none carrying a dependency tree somebody else chose. That is the right shape for a smoke suite and the wrong shape for the question "would this serve my application?"

The gap was not hypothetical. The one time the wheel had met a real Django project it produced the --app-dir shadowing defect (Known issues). This pass produced four more, three of which no in-repo app could have shown, and every one of them is now fixed with a test that fails without the fix.

The three applications

Local Django projects, chosen because they stress different parts of the server rather than because they are convenient. Served from clean clones against scratch databases; their own settings, unmodified.

What was found

Five defects. Four are fixed in this repo with a guard; one was already known and stays open on purpose.

Findings 1, 3 and 4 could not have come from apps/: they need a real cookie policy, a real static-file middleware, and a real never-ending response respectively. Finding 2 needed a file somebody wanted to upload.

Phase 0 — will it even load?

m0serve --doctor against each entry point. All six passed ("ok": true, exit 0): transcript_manager.wsgi, halftone_studio.wsgi, halftone_studio.asgi, config.wsgi, config.asgi, and transcript_manager.wsgi again under --threads 4 on 3.14.7t, which reported free_threaded_build: true and mode: threads. Protocol detection was right every time; the WSGI apps got the zero-config pool of 8, the ASGI apps got the executor.

transcripts keeps its code in src/, so it is the shape the --app-dir defect targets. It loaded from both --app-dir src and --app-dir . (it is also installed as an editable package), and neither path was shadowed.

Phase 1 — parity against runserver

Same routes under m0serve and manage.py runserver, raw HTTP/1.1, no redirect following, diffed on status, headers and body after normalising what is per-process by design (Date, CSRF token, session id). Home, admin index, admin login, allauth login, a static asset, a 404, a 500, and a full login round trip on each app.

After the cookie fix, every remaining difference is one of three, and all three are the server being itself:

• connection: keep-alive — m0serve sends it explicitly; runserver does not.
• x-thread: N — which handler-pool thread served the request. Documented, and absent under --blocking-threads 0.
• Django's debug pages embed the request's own host:port, which differs because the two servers are on different ports.

Bodies were byte-identical on every route on all three apps, admin included. The login round trip — GET the form, POST credentials, GET the destination with the session — produced identical headers at all three steps on color-separation and textshelf. Before the fix the same comparison reported 15–24 differences per app, every one of them a cookie.

Two app-level failures appeared identically under both servers and are the applications', not ours: transcripts answers 403 on /accounts/login/ (its dev auto-login middleware logs you in, and allauth's login view forbids an authenticated user), and textshelf answers 500 on /accounts/signup/ (no SocialApp row in a scratch database).

Phase 2 — the feature matrix

Phase 3 — the topology matrix

Each app, each mode: a route sweep, a 200-request burst at concurrency 16, and 16 concurrent logins (a session write per request).

Every mode served every route correctly on all three apps, with zero 5xx and zero dropped connections in the bursts — --workers 1, --workers 4, --blocking-threads 4, --workers 4 --blocking-threads 4, and --threads 4 on free-threaded 3.14.7t (transcripts, which is the app whose dependency tree is thread-clean).

The 16-concurrent-login column found nothing about the server and one thing about the fixture: textshelf answers some of them 500 with sqlite3.OperationalError: database is locked. That is the scratch SQLite — the project runs PostgreSQL — and it reproduces at the same rate under gunicorn --workers 4 and under runserver, so it is attributed there rather than here. transcripts answers all 16 with the same app-level 403 as in Phase 1. textshelf also rate-limits its own auth endpoints (429), which is the application working as designed.

Phase 4 — the realtime retrofit (textshelf)

The claim is that a plain sync view can hold an SSE stream with two response headers and no added infrastructure. textshelf/notifications/ was the honest test: an async notification_stream view, an async_subscribe generator over PostgreSQL LISTEN/NOTIFY, a polling fallback for SQLite, a sync copy of both, and a hard dependency on running under Daphne.

The conversion is 20 lines of view:

@login_required
@require_http_methods(["GET"])
def notification_stream(request):
    response = HttpResponse(f"event: connected\ndata: {head}\n\n",
                            content_type="text/event-stream")
    response["M0-Hold"] = "stream"
    response["M0-Channel"] = NotificationPubSub.get_user_channel(request.user.id)
    return response

plus one m0pub.publish beside the existing pg_notify in the publish path, deferred to commit the same way.

Measured result: +52 / −293 across two files — a net 241 lines removed, because holding the connection in the server strands all four subscription implementations (sync, async, and a polling fallback for each: 236 of pubsub.py's 336 lines), and with them the psycopg import that existed only for LISTEN/NOTIFY.

It works: four EventSource clients held across two prefork workers, one POST to an ordinary synchronous Django view, and all four receive the same numbered event. No Daphne, no asyncio, no LISTEN/NOTIFY, and the database is SQLite — where the code being replaced had no delivery mechanism at all, only a 5-second poll.

The degradation is better than what it replaces, which was the surprise. Under gunicorn the converted view answers a short buffered text/event-stream that the browser's EventSource reconnects on — the GRIP property. The original async view under gunicorn hangs: 12 s with no bytes, and the worker never serves again. So this is not a change that trades portability for the feature.

The honest caveats: the app loses its SQLite polling fallback (nothing delivers under gunicorn now, where before it polled — at the cost above), and M0-Channel is one channel per connection, which suits a user-scoped notification stream and would not suit an endpoint that multiplexes several.

Verdict: a diff a maintainer would accept. The claim survives contact.

Phase 5 — soak

6,000 keep-alive requests per app over a mixed route set (home, admin, login, static, 404, an authenticated page), sampling RSS, open descriptors and threads.

Descriptors and threads are flat — nothing accumulates per request, and Django's CONN_MAX_AGE = 10 holds its connections steady rather than growing them. RSS rises 1.5–6.5 MB over the 5,000 requests after warm-up and then stops, which is Python's allocator reaching steady state rather than a leak; smoke-django's 10k-request guard still measures 0 KB against the bare app. Throughput was 363–819 rps single-worker on a laptop, which is not a benchmark and is not offered as one.

What this changes about the guards

Every fix landed with something that fails without it, and each was checked by sabotage rather than assumed:

• test_response_cookies.mojo — 4 of 6 cases fail when add_raw is reverted to parse-and-reserialise.
• test_config.mojo::test_recv_buffer_limit_covers_headers_plus_body — fails when recv_buffer_limit() returns the bare field.
• smoke-django — now reads the session cookie off the wire and requires expires, HttpOnly, Max-Age and SameSite=Lax, because curl's cookie jar stores name and value only and could never have seen this.
• smoke-serve — a 3 MB body at the default cap and 7 MB at --max-body 8m must be 200; 5 MB and 9 MB must be 413. Both sizes straddle the old ceiling.
• smoke-asgi — 32 concurrent 4 KB-piece streams, every body byte-exact. This one paid for itself on its first CI run, and is why the fix looks the way it does. It passed on macOS and failed on Linux: two streams short, because the budget had been sized against a 256 KB socket buffer and Linux clamps SO_RCVBUF to net.core.rmem_max and charges each datagram's whole skb against it. Raising the number would have made this platform pass and left the next to find out, so the number stopped being load-bearing instead — a full channel now waits (detached, so the loop can drain it) rather than dropping. Verified by removing the budget rather than trusting it: at _ASGI_TOTAL_WINDOW = 64 MB, effectively no budget at all, 64 concurrent streams still deliver every byte and log no drops.
• smoke-blocking-threads — two views that never return, SIGTERM, and the process must exit inside 20 s naming what it abandoned.

Revisited after stage 1 (2026-08-26)

--realtime composing with --blocking-threads changed the answer this record originally implied, so the question was re-measured rather than re-reasoned. Same laptop, same scratch SQLite, five events 0.3 s apart, arrival time of each event recorded by a client that reads incrementally — curl -w %{time_starttransfer} will not do, because several of these servers send an empty first body chunk that satisfies it.

Two things fall out, and the first is about their application rather than any server. textshelf's AI streaming endpoints do not stream, on their own production server. ai/views.py:summarize_submission is a sync view returning create_sse_response(...) over services.summarize_document_stream, whose annotation is Generator[str, None, None] — a sync generator, which Django's ASGI handler must consume before it can serve it (it says so in a warning). Every ASGI server buffers it identically; the token-by-token delivery the endpoint is written for arrives in one lump at the end. The rows above are a same-shaped instrument rather than their endpoint (which needs an API key), but the shape is what decides it, and the shape is theirs.

The second: m0serve's executor matches uvicorn and daphne to the millisecond in both rows. Whatever streams under them streams under it, and what buffers buffers everywhere.

That changes the recommendation this record's Phase 4 pointed at. Before stage 1, holds cost the pool, so the hybrid mount was the only shape with both isolation and working streams. Now:

The AI endpoints cost nothing to move, because they are already buffered wherever they run. So the choice was between the cheapest way to hold a stream and no application changes at all — and what forced it to be a choice was that --realtime and --mount could not be combined (measured the day this was written: exit 1, refused before the bind).

That is no longer true, and this pass is why. The ROADMAP entry it produced was implemented the same day: --realtime now composes with --blocking-threads, with --mount, and with WebSocket holds under both (ROADMAP, Hold on a pool thread, stages 1–3). So the row this table calls "convert 4 pub/sub endpoints" no longer costs the pool or the ASGI mount — an application can hold its pub/sub streams, run its generator streams on an executor mount, and keep pool isolation for its slow views, in one process. What is left for textshelf is application work: converting the four endpoints, and making ai/streaming.py's generator async so it streams at all.

Re-soak — 2026-08-31, against 0.16.0, textshelf only

Five minor versions landed in five days after the record above — --mount, per-mount lanes, the batched pump, ExecutorPort, the WebSocket credit gate, the close linger — all on the request path, none of it seen by a real application. textshelf was re-run first because it is the app with four SSE endpoints, so it stresses the seam that churned most.

It found one defect, and the defect was on the ASGI path the WSGI record above could never have reached. Clean clone, scratch SQLite, config.settings.local, the 0.16.0 wheel, CPython 3.13.6, macOS 26 on an M4.

The WSGI row matches the 0.11.0 shape: +4.3 MB over 5,000 requests, flattening to +0.4 MB across the last 2,000, with descriptors, database descriptors and threads all flat and the thread count identical at 13. 540 rps, inside the record's 363–819 range. The ASGI descriptor and RSS churn is asgiref's thread pool opening a connection per thread — application behaviour, not a leak.

The defect: the keep-alive cap destroyed the response it fired on

Nine of the 6,000 ASGI requests came back truncated, all of them on the same 124 KB WhiteNoise file, at intervals of exactly 700 requests — every hundredth time that route was hit. max_keepalive_requests is 100.

Isolated on one connection, and the two faces are the same bug:

_finish_response clears should_close for a stream and for a 101, because each owns its connection until it ends. The cap check below those branches was guarded by not should_close — precisely the state they had just established — so the two shapes that had opted out were the two it caught, and _after_send closed the slot as soon as the head drained, before any body frame arrived over the chunk channel. Both faces are silent: correct status line, nothing in the server log.

Gated by poe smoke-keepalive-cap (SPEC A3), whose third phase asserts the cap still fires for an ordinary response — without it the gate would pass on a build whose cap never fires. Four sabotages: reverting the fix, each guard alone, and raising the cap, each caught by the phase that should catch it.

The m0serve X.Y.Z marker in this file's opening line is the ONE the milestone gate reads, and it deliberately stays at the version the full three-application pass was run against. A second marker anywhere in the page would become a fallback for _real_app_version and silently defeat the sabotage that proves the gate can miss an unreadable record — which is why this heading says "against 0.16.0" rather than naming the binary.

This pass is not the 1.0 soak. transcripts and color-separation have not been run against 0.16.0, so the record above still stands at 0.11.0 for them and the milestone stays stale until they are.

What would make this worth repeating

It already was: four defects in one pass, three of them invisible to any application written to test this server. The next pass should use applications with shapes these three lack — a background worker, an app behind a proxy with SECURE_PROXY_SSL_HEADER, an upload-heavy API, and something that is not Django.