Tail: riding a feed instead of re-fetching it

Until 0.8.0, riding a source in Neurowire meant re-fetching a document and diffing it yourself, choom. The engine could flip anything into a feed, but a feed was a snapshot: you knocked, you got the front page as it stood that second, and working out what was new since last time was your problem and your eddies.

pollFeed kills that. It is an async generator that yields new items as they land, and it is now the only polling loop breathing in the whole codebase.

Consolidation is the feature

Watch mode already had a loop of its own. Rebasing it onto pollFeed was not tidying the workshop, it was putting a whole class of bug in the ground.

Three surfaces poll: the CLI tail, the old --watch, and the API’s GET /tail over server-sent events. When each one runs its own loop, cadence, dedupe, backoff, and failure handling drift apart quiet as a pickpocket in a crowd. One gets conditional requests and the other two do not. One dedupes on link, another on GUID. Nobody clocks it until some choom reports a duplicate in watch mode that tail has never once shown.

The rewrite kept flag behaviour identical, which meant the old watch-mode tests were already the regression suite waiting for it. That is the cheap version of this job: if the tests do not flinch, the behaviour did not either.

A live stream is a journal getting written

tail -f nwf emits real NWFJ, the journal format itself, not some stream-only shape hacked together on the spot for the occasion.

Which means a live tail is a journal being written in real time, choom, and the same reader chews both. Pipe a tail into a file and you have a journal. Read a journal and it looks like a tail that already finished its shift. That is the payoff for building the format first instead of welding a stream onto chrome that was never cut to carry one.

Manners are built into the chrome

Polling is where well-meaning tools turn into a nuisance for the sites they read. Hammer a host every two seconds and your rig looks like a scav crew casing the block, and you get ICE’d for it, deserved. The defaults do that work so you do not have to think about it:

• A 30 second floor on the CLI and 60 seconds on the server. No config, no flag, no clever choom talking their way under them.
• Jittered waits, so a hundred tails kicked off by a hundred netrunners do not all land on the same second like a synchronised raid.
• Conditional requests, so an unchanged source costs a 304 and near enough zero bytes.
• On the API, one poll loop per target, shared. Fifty clients riding ai-news cost one upstream fetch a tick, not fifty. Fifty gonks knocking on one door helps nobody.

Reconnect with Last-Event-ID against a journaled target and you get replayed exactly what you missed while the link was down. A dropped connection stops being a hole in the record.

Which one do you want

Tail when you want to ride something for a while. Watch when you want an existing command re-run every time the source twitches. A plain fetch when you want the state once and then you delta. The decision table on the concept page settles it faster than any amount of prose.

Full story at Tail, including one engine, three surfaces and the politeness argument. The command lives under tail mode and the endpoint under GET /tail.

For something you can lift and run, the recipe pipe a live tail into other tools shows the Unix half of it. That is where tail stops being a feature and starts being plumbing in your rig.

Neurowire is open source on GitHub, with a hosted reader at neurowire.app. More open-source work sits under projects.