The rig that sat dark all day

Here is the job no amount of slick fetching pulls off, choom. Your rig sat dark all day. A busy site pushed forty items while it was cold. You jack back in, fetch the source, and get the current front page, which holds twenty. The other twenty are not running late. They are gone. Off the page, off the net, flatlined out of reach. No amount of knocking brings them back.

Only a machine that stayed awake and wrote it all down can hand them over. That is the whole argument for nwf-sync/1, shipping in Neurowire 0.8.0.

Four read-only routes, lean chrome

The protocol is deliberately small, the kind of gear a fixer keeps because it never jams. /journals lists what a peer will share, /head says where that peer is, /since hands over everything past a cursor, and /snapshot hands over the whole stash. All four read-only, all four stamped with an NWF-Sync-Version: 1 header, and the payload is NWFJ segments, the same journal format every other piece of the rig already reads.

Client side that is pullJournal and syncPeers, plus the CLI sync and peers commands. No new format to learn, no new chrome to bolt on.

Three reasons to run it, most honest first

Time. The dark-rig case up top. It is the only one of the three that direct fetching genuinely cannot crack, no matter how nova your connection is, which is why it leads.

Bandwidth. Eight netrunners each riding 200 sources fire 1,600 requests a tick off eight different IPs. To the sites on the receiving end that reads as a scav crew casing the block, and they will ICE the lot of you for it. Stand up one hub and it makes 200; the other seven pull deltas off it. Taps get patched once instead of eight times by eight tired chooms doing the same work.

Reproducibility. Cite “journal ai, seq 1 to 48210” and a choom across town syncs the identical corpus, byte for byte. And when retention has already eaten part of that range, the request flatlines loud with a 410 instead of quietly handing back a short list dressed up as a complete one. A gap you can see beats a gap you cannot.

Measured on a real target: a full pull costs 1.2 KB over 4 requests, a delta 409 bytes over 2, and steady state settles at 52 bytes and one request on the cursor short-circuit. Riding a peer costs less than checking whether you need to. Cheap on eddies, cheap on bandwidth, cheap on everybody’s patience.

One segment per response, never a splice

This is the call I am gladdest got reasoned out at the bench instead of found in production at 3am with a cold cup of synth-coffee going colder.

A /since response serves one whole NWFJ segment, even when the client asked for a range covering several. Easy enough to glue them together and hand back one fat body. Also dead wrong. NWFJ dictionary indices are per-segment, and the hash chain reseeds at every segment header. Spliced segments do not fail to parse, and that is the part that will put a hole in you: they decode and verify wrong, quietly, and keep smiling while they do it.

So the response carries NWF-Sync-Complete: 0 when there is more in the stash, and the client loops until it is done. Written into the spec, not into a postmortem somebody writes after the corpus is already poisoned.

The hash chain is integrity, not authenticity

Worth saying flat out, choom, because chained hashes tempt you into trusting the wrong thing. The chain catches corruption and truncation. It does not prove who wrote the entries. A peer that fabricates a journal out of thin air produces a perfectly valid chain over its own fabrication, and the math will nod along.

What that means on the street: you sync from peers you picked to trust, same as you pick a ripperdoc. Somebody vouched, or you watched them work. On the serving side, nothing gets published without an explicit allowlist, and the token gets checked before the publish list is consulted, so a 401 leaks no journal names to whoever is knocking in the dark. The trust model page puts all of this on the record instead of burying it in a footnote where nobody reads it.

Sync is the wrong tool plenty of the time too, and the page saying so ships with the docs rather than getting learned the hard way.

Where to jack in

Sync is the concept page and the full story. The wire protocol sits at /formats/nwf-sync. Commands are sync and peers.

If you want to stand a mesh up, the federation guide walks a three-node setup end to end. If you want the small version first, the recipe share one archive across two machines is exactly that: a desktop that stays awake, a laptop that goes dark, one archive between them and nothing lost in the gap.

Neurowire is open source on GitHub, with a hosted reader at neurowire.app. More open-source work sits under projects.