jthomson04 · 2026-08-17T22:59:20Z

Motivation

axum::serve currently clones the shutdown watch::Sender for every accepted connection. Each connection polls Sender::closed(), so all connection shutdown waiters share synchronization state. This becomes expensive when many connections are active across many CPU cores.

Initial end-to-end benchmarks used Dynamo, a distributed inference serving framework that uses Axum for HTTP. Its frontend accepts OpenAI-compatible requests, performs tokenization and routing, and streams worker output to clients. These results were measured in a frontend-bound scenario. The only software difference between the two arms was the Axum build: unmodified Axum 0.8.9 versus Axum 0.8.9 with this patch.

Dynamo end-to-end arm	Throughput	First response latency p50	First response latency p95	First response latency p99
Unmodified Axum 0.8.9	1,000.3 req/s	63.08 ms	113.96 ms	262.87 ms
Axum 0.8.9 with receiver fix	1,897.9 req/s	42.47 ms	107.84 ms	263.93 ms

I also isolated this behavior in a standalone Axum benchmark. It used 4,096 persistent connections and finite responses with 32 256-byte SSE frames, one scheduler yield between frames, a stock rewrk client on another node, a 15-second warmup, and a 60-second measurement.

Arm	Throughput	p99 latency	CPU/request
Control	151,990 req/s	71.44 ms	0.312 ms
Receiver fix	180,091 req/s	65.79 ms	0.147 ms

In an on-CPU profile of this benchmark, Sender::closed used 16.57% inclusive CPU in the control. Its shared mutex used 15.32% inclusive CPU, including 7.12% in the contended lock path. Those stacks disappeared with this change, while Receiver::changed used 4.14% inclusive CPU.

Solution

Keep the shutdown sender in the server task and give each accepted connection its own cloned receiver. The accept loop and connection tasks wait on Receiver::changed(), and shutdown drops the sender to notify all receivers.

This keeps the public API and graceful shutdown behavior unchanged. Existing tests continue to cover graceful draining and custom executors.

Validation

cargo test -p axum --all-features serve::tests
cargo test --locked --workspace --all-features --all-targets
cargo clippy --locked --workspace --all-targets --all-features -- -D warnings
cargo fmt --all --check
git diff --check

mladedav · 2026-08-18T11:53:57Z

This makes sense, would you mind tying even with tokio::sync::Notify directly? That's what both of those end up using internally so we might as well do that too. Maybe it could still give a little more performance.

Anyway thanks, this looks good.

jthomson04 · 2026-08-18T22:50:56Z

Hey @mladedav. I tried out that tokio::sync::Notify approach. I found that it has significantly lower perf than both the baseline on main and the receiver patch. Under the hood, it looks like tokio::sync::watch::Receiver uses a BigNotify, which is internal to tokio. We could reproduce that behavior in Axum, but unless tokio exposes BigNotify, we'd need to implement and maintain our own sharded notifier here.

mladedav

I am a bit surprised that raw Notify would be slower than the current closed-based variant since that one does use it in the background and we would prevent spurious wakes which I suspect were cause of the contention. But I missed the big notify so you're right we might want to just use what you had originally.

Thanks.

Give each connection its own watch receiver instead of polling Sender::closed on cloned senders. This avoids synchronization through a shared waiter while preserving graceful shutdown behavior. Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

Document the receiver-side wait, preserve the channel-close invariant, and add the changelog entry. Keep pre-existing serve cleanup outside this change. Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

jthomson04 marked this pull request as ready for review August 18, 2026 02:27

mladedav reviewed Aug 19, 2026

View reviewed changes

Comment thread axum/CHANGELOG.md

Comment thread axum/src/serve/mod.rs Outdated

jthomson04 added 3 commits August 20, 2026 10:45

Avoid shared shutdown waiter contention

1f34d25

Give each connection its own watch receiver instead of polling Sender::closed on cloned senders. This avoids synchronization through a shared waiter while preserving graceful shutdown behavior. Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

Address shutdown waiter review feedback

bd4e2ae

Document the receiver-side wait, preserve the channel-close invariant, and add the changelog entry. Keep pre-existing serve cleanup outside this change. Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

chore(serve): remove shutdown notification comments

38ca471

Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

mladedav force-pushed the jthomson04/avoid-shared-shutdown-waiter branch from 4731943 to 38ca471 Compare August 20, 2026 08:45

mladedav approved these changes Aug 20, 2026

View reviewed changes

mladedav merged commit 3d78036 into tokio-rs:main Aug 20, 2026
17 checks passed

Navigation Menu

Uh oh!

Avoid shared shutdown waiter contention - #3867