The laptop that was closed all day

Here is the case that no amount of clever fetching fixes. Your laptop was closed all day. A busy site published forty items while it was shut. You open the lid, fetch the source, and get the current front page, which holds twenty. The other twenty are not slow to arrive, they are gone. They are not on the page any more.

Only a machine that stayed awake and wrote them down can give them back. That is the whole argument for nwf-sync/1, which ships in Neurowire 0.8.0.

Four read-only routes

The protocol is deliberately small. /journals lists what a peer will share, /head reports where it is, /since returns everything after a cursor, and /snapshot returns a full copy. All four are read-only, all four are versioned by an NWF-Sync-Version: 1 header, and the payload is NWFJ segments, the same journal format everything else already reads.

On the client side that is pullJournal and syncPeers, plus the CLI sync and peers commands.

Three reasons to run it, in order of honesty

Time. The closed-laptop case above. It is the only one of the three that direct fetching genuinely cannot solve, which is why it leads.

Bandwidth. Eight developers each following 200 sources make 1,600 requests a tick from eight IP addresses, which to the sites on the receiving end looks like a scraping operation. Run one hub and it makes 200; the other seven pull deltas. Taps get fixed once instead of eight times.

Reproducibility. Cite “journal ai, sequence 1 to 48210” and a colleague syncs the identical corpus. When retention has removed part of that range, the request fails loudly with a 410 rather than quietly handing back a shorter list that looks complete.

Measured on a real target, a full pull costs 1.2 KB over 4 requests, a delta 409 bytes over 2, and steady state settles at 52 bytes and one request thanks to a cursor short-circuit. Following a peer is cheaper than checking whether you need to.

One segment per response, never a concatenation

This is the decision I am most glad was reasoned out before it shipped rather than discovered in production.

A /since response serves one whole NWFJ segment, even when the client asked for a range spanning several. It would be easy to glue them together and hand back one big body. It would also be wrong. NWFJ dictionary indices are per-segment, and the hash chain reseeds at each segment header. Concatenated segments do not fail to parse, which is the dangerous part: they decode and verify incorrectly.

So the response carries NWF-Sync-Complete: 0 when more remains, and the client loops. Written into the spec, not into a postmortem.

The hash chain is integrity, not authenticity

Worth stating plainly, because chained hashes invite the wrong assumption. The chain detects corruption and truncation. It does not prove who wrote the entries. A peer that fabricates a journal from scratch produces a perfectly valid chain over its fabrication.

What that means in practice: you sync from peers you chose to trust, the same way you choose to trust a package registry. On the serving side, nothing is published without an explicit allowlist, and the auth token is checked before the publish list is consulted, so a 401 does not leak the names of journals you host. The trust model page says all of this on the record rather than in a footnote.

Sync is also the wrong tool in plenty of cases, and the page saying so is part of the documentation rather than something you find out by trying.

Where to start

Sync is the concept page and the full story. The wire protocol is at /formats/nwf-sync. The commands are sync and peers.

If you want to stand something up, the federation guide walks a three-node setup end to end, and the recipe share one archive across two machines is the smaller version: a desktop that stays awake, a laptop that does not, and one archive between them.

Neurowire is open source on GitHub, with a hosted reader at neurowire.app. More open-source work sits under projects.