Spindle vs the field
A linearized Matrix homeserver: an append-only log per room, materialized state, and no state resolution on the hot path. Every milestone, the same client operations are measured against Synapse and both Rust siblings — wins, noise and losses all published from the committed raw numbers.
0cells faster
0within noise
0slower — investigated
Why the shape of the storage shows up in every row
Watch the same six events arrive at both designs, then watch one read
ask "what is the state?" The loop below plays the whole
difference: on one side the answer must be computed; on the other it was
already written down.
A conventional homeserver
Rooms are a directed graph of events. Forks are normal, so answering
"what is the state?" means running state resolution over the
branches — work that grows with the room and sits on the hot path of
sync, send, and join.
Spindle
Each room is an append-only log with materialized state: every
entry carries the content address (BLAKE3 hash trie) of the state after
it. "The state now" is one read; history is a range; nothing resolves on
the hot path. Federation forks are collapsed at the door, bounded, and
never taxed on reads.
append-only log · Kafka, Raft
content-addressed state · git, Merkle trees
HAMT snapshots · Clojure, CHAMP
single-writer order · LMAX, TigerBeetle
losses published · every red cell links its investigation
That is the bet these pages test. The comparisons below measure the same
client operations against the same workloads on Synapse (the reference
implementation) and on Continuwuity and Tuwunel (the two Rust siblings,
both descended from Conduit) — and when a cell goes the wrong way, the
roadmap's rule is that it gets investigated, not explained away.
Latest sitting — m7-progress
continuwuity (m7-progress.continuwuity.r1.json) · continuwuity (m7-progress.continuwuity.r2.json) · continuwuity (m7-progress.continuwuity.r3.json) · spindle (m7-progress.spindle.r1.json) · spindle (m7-progress.spindle.r2.json) · spindle (m7-progress.spindle.r3.json) · synapse (m7-progress.synapse.r1.json) · synapse (m7-progress.synapse.r2.json) · synapse (m7-progress.synapse.r3.json) · tuwunel (m7-progress.tuwunel.r1.json) · tuwunel (m7-progress.tuwunel.r2.json) · tuwunel (m7-progress.tuwunel.r3.json)
spindle spindle spindle continuwuity continuwuity continuwuity synapse synapse synapse tuwunel tuwunel tuwunel — hover a server to isolate its line in every chart
The /context of an event far back in history. On a linear log an old event's neighbourhood is an index range, not a graph walk.
A local user joins: one membership event through the same append path as any other, authorized against materialized state.
A /messages page walking backwards. The linear log makes this a bounded range read — no topological sort at request time.
One event appended to a room. Spindle assigns the next index in the room's linear log and folds one state snapshot forward; a DAG server must pick extremities, and its cost tends to grow with the room.
The request Element X makes where classic clients call /sync: a sorted room-list window plus per-room state. The room-list question is one point read per room on Spindle's storage.
The full current state of a room. The log's head entry carries the state root; answering is one rehydration of a hash trie.
The same request when a room does have something waiting. It takes a different path from the empty poll -- the server skips rooms with no new events entirely -- so the two are measured separately rather than averaged into one misleading number.
A client's first sync: full state plus recent timeline for every room. Materialized state makes 'the state now' one content-addressed read instead of a resolution.
The request a running client makes over and over: since=<token>, and the answer is usually “nothing”. It is the most common request a homeserver ever serves, and until M5 no sitting measured it at all.
Every cell
Measured over 3 rounds per server. A cell is called only when the two servers' rounds separate: win our slowest round beat their fastest · overlapping the ranges cross, so the difference is not resolved by this many rounds, whatever the medians say · loss our fastest round lost to their slowest — every such cell links to its investigation, because the roadmap treats it as a defect until explained. Hover any cell for the median and the observed spread.
That rule bounds the false-call rate for one cell, and this table has 81. Two identical servers separate by luck 10.0% of the time at 3 rounds, so about 8.1 of these cells should be called by chance alone — against 75 actually called. The arithmetic cannot say which ones, so a call marked † is one that stands alone: the same operation is not called the same way at any other size. A cost that is real in a per-item measure normally shows across the size axis, so an isolated call is the shape a chance separation takes. Read those as unconfirmed rather than as results (#183).
Earlier sittings
The trail matters: each milestone re-measures the same operations, so a regression shows up as a cell that changed color between sittings.
m4-final — 44 faster · 19 within noise · 0 slower
1 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
m3-final — 51 faster · 12 within noise · 0 slower
1 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
Nothing in this sitting reads as a loss under the band above. The investigations below were written when it did, and are kept because they are the evidence for the conclusion, not a footnote to it.
Investigated — Sliding sync (MSC4186): 0.77× vs Continuwuity at 3,200 events, and 0.85× in the same day's discarded loaded run — repeatable by the two-sitting rule, so it got the full second look. Both servers were probed live minutes after the sitting, same client, same instant, two shapes: like-for-like the gap does not exist (creator shape 0.844 ms vs 0.854 ms, parity; the driver's exact observer shape 0.785 ms vs 0.874 ms, Spindle 1.11× faster). The two sitting legs caught opposite sides of the machine's same-day swing. No fix ships because no defect was found; the cell keeps its measured value and links here — the honest kind of red.
Investigated — Initial /sync: 0.89× vs Tuwunel at 200 events in this sitting — and 1.21× in the same day's discarded run, with 800 and 3,200 in noise both times. A cell that flips sign between sittings hours apart is run-to-run variance, published as measured.
m3-progress — 53 faster · 10 within noise · 0 slower
1 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
Nothing in this sitting reads as a loss under the band above. The investigations below were written when it did, and are kept because they are the evidence for the conclusion, not a footnote to it.
Investigated — Sliding sync (MSC4186): 0.90× and 0.88× vs Continuwuity at 800 and 3,200 events — and the same cell sat at 0.83× in the M2 close-out, so two sittings agreed this was repeatable, not noise. A component probe on the live bench server found the room list's recency sort reading each room's head event body from the store and parsing its JSON on every request, for one i64. #126 moved the sort key into memory, refreshed by the append that changes it; re-measured on the same idle machine the cells recover to 0.96× and 1.00×, and Spindle's own growth curve flattens from 1.28× to 1.13× across a 16× room-size increase.
m2-final — 51 faster · 12 within noise · 0 slower
1 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
Nothing in this sitting reads as a loss under the band above. The investigations below were written when it did, and are kept because they are the evidence for the conclusion, not a footnote to it.
Investigated — Sliding sync (MSC4186): 0.87× vs Continuwuity at 3,200 events was the one real loss of the M2 close-out, and the only curve growing with room size. Bisecting a live server pinned it in one probe: the unread counter was reading every event body after the receipt floor. #113 replaced that walk with two binary searches over a per-room sender index — 11.79 ms → 1.00 ms on the pathological case — and the M3 rows below show the cell recovered.
Investigated — Read room state: 0.85× vs Tuwunel at 200 events, and 0.93× again at M3 progress — the one cell Tuwunel held across two sittings, so the investigation started in their tree. Their /state serves each event through RocksDB's block cache; ours paid a room-lock acquisition, a body read and a JSON parse per state event, per request. Component probes showed their state machinery was never actually faster — their per-request pipeline is just leaner. #129 caches the rendered /state body under its BLAKE3 state root (content-addressed, so a hit is provably current and a root mismatch is the only invalidation); re-measured against the live Tuwunel binary the cell flips to 1.91× and 1.33× in Spindle's favour.
m2-progress — 36 faster · 6 within noise · 0 slower
1 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
m1-spindle-vs-synapse — 15 faster · 0 within noise · 0 slower
1 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
m1-spindle-vs-continuwuity — 15 faster · 3 within noise · 0 slower
1 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
concurrency-perroom — 0 faster · 0 within noise · 0 slower
2 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
concurrency — 8 faster · 0 within noise · 0 slower
1 round(s) per server — not resolved. Three rounds a side is the minimum that means anything: if two servers were identical, all of one side's rounds landing below all of the other's happens by chance 2/C(2n, n) of the time — one in three at two rounds, one in ten at three. With no spread to read, the cells below are coloured by the repeatability this host was measured to have: six rounds of the same binary moved the median cell 1.38×, and 21 of 21 cells varied by more than the ±10% band this page used to use. So: ≥1.38× · 0.72–1.38× · ≤0.72×. A grey cell is not a tie — it is a difference this sitting cannot see, and only more rounds can (#171). The large ratios are unaffected. Hover any cell for the raw milliseconds.
Method
- Same host, same sitting, same load. All servers in one
run on one idle machine — a leg once ran beside a compiler and read ~25%
slow across the board, so idleness is part of the method, and it is checked,
not assumed.
- Cold databases, verified binaries. Every leg starts
from an empty store, and the serving process is pgrep-verified before
measuring — a stale process once served an old binary on the right port.
- Curves, not points. Every operation is measured at
200, 800 and 3,200 events per room, because the failure mode worth catching
is the cost that grows with the room. Room size has a second axis —
joined members — and the sweep held it at two until M5, which hid
a sliding-window read that grew linearly with the member list. Membership
is now its own sweep, on its own chart, labelled by what it counts.
- Means over 25 samples after warmup, raw results
committed to the repository exactly as the driver wrote them
(
docs/benchmarks/data/); this page is regenerated from those
files and cannot change a measurement.
- Losses publish with the same prominence as wins.
Cells outside the ±10% band are printed as wins or losses, and a loss
links to its investigation.
- That ±10% band is narrower than this harness can currently
resolve, and saying so is part of the method. Six rounds of the
same binary on the same idle host move the median cell by 1.38× and the
worst by 2.80× — so every cell here varies more between runs of identical
code than the band that colours it. The large ratios clear that floor
comfortably and the conclusions rest on those; the cells near 1.0× should
be read as "not measured", whichever way they lean. Making a sitting
several rounds and deriving the band from the observed spread is
#171.
Versions measured, ports, registration quirks and the full narrative per
sitting:
docs/benchmarks.md.