Tail: following a feed instead of re-fetching it

Until 0.8.0, following a source in Neurowire meant re-fetching a document and diffing it yourself. The engine could turn anything into a feed, but a feed was a snapshot: you asked, you got the front page as it stood, and working out what was new since last time was your problem.

pollFeed fixes that. It is an async generator that yields new items as they appear, and it is now the only polling loop in the codebase.

Consolidation is the feature

Watch mode already had a loop. Rebasing it onto pollFeed was not tidying, it was removing a class of bug.

Three surfaces poll: the CLI tail, the pre-existing --watch, and the API’s GET /tail over server-sent events. When each owns its own loop, cadence, deduplication, backoff, and failure handling drift apart quietly. One gets conditional requests and the others do not. One dedupes on link, another on GUID. Nobody notices until a user reports that watch mode shows a duplicate that tail does not.

The rewrite kept flag behaviour identical, which meant the old watch-mode tests were already the regression suite. That is the cheap version of this refactor: if the tests do not change, the behaviour did not either.

A live stream is a journal being written

tail -f nwf emits real NWFJ, the journal format, not a stream-only shape invented for the occasion.

That means a live tail is a journal being written in real time, and the same reader parses both. Pipe a tail into a file and you have a journal. Read a journal and it looks like a tail that already finished. This is the payoff of having built the format before the streaming, rather than bolting a stream onto a format that was never asked to carry one.

Politeness is designed in, not left to the caller

Polling is where well-meaning tools become a nuisance for the sites they read. The defaults do the work here:

• A 30 second floor on the CLI and 60 seconds on the server. You cannot configure your way under them.
• Jittered waits, so a hundred tails started by a hundred people do not converge on the same second.
• Conditional requests, so an unchanged source costs a 304 and almost no bytes.
• On the API, one poll loop per target, shared. Fifty clients following ai-news cost one upstream fetch per tick, not fifty.

Reconnect with Last-Event-ID against a journaled target and you get replayed exactly what you missed, so a dropped connection is not a gap.

Which one do you want

Tail when you want to follow something for a while. Watch when you want an existing command re-run on changes. A plain fetch when you want the current state once and then to stop. The decision table on the concept page settles it faster than prose does.

The full story is at Tail, including one engine, three surfaces and the politeness argument. The command is documented under tail mode and the endpoint under GET /tail.

For something runnable, the recipe pipe a live tail into other tools shows the Unix half of it, which is where tail stops being a feature and starts being plumbing.

Neurowire is open source on GitHub, with a hosted reader at neurowire.app. More open-source work sits under projects.