A feed catalogue, and a poller that can hold one

[ralyodio]

Imports the Kagi small-web OPML — 47,223 feeds — and adds a daemon that keeps them current without being a resource hog.

Why not feeds.json

FeedManager loads every feed (and up to 100 articles each) into one JSON file it reads whole on construction and rewrites whole on every change. Right for a few dozen hand-picked feeds; for a catalogue it is quadratic work against a file growing past a gigabyte, ending in an OOM crash.

Store is now SQLite via node:sqlite — built into Node 24, so no native module to compile (this box has no build toolchain, so better-sqlite3 was not an option).

Measured: full 47,223-feed import in 1.7s at 122 MB peak RSS.

The daemon

Eight feeds in flight, 2s between batches. Five things make almost every poll free:

• Conditional GET — ETag/Last-Modified per feed, so unchanged feeds cost a 304 and no parsing. Servers that ignore validators get the same path via content hash. Verified live: forced re-poll returned 176 unchanged of 188.
• Adaptive intervals — publishing feeds checked sooner, quiet ones back off to a day.
• Exponential backoff, then deactivation — dead domains are the biggest waste in a catalogue this size.
• Bounded everything — one feed per host per batch, request timeout, download size cap.
• Idle sleep when nothing is due.

The database is the queue (next_fetch_at is the cursor) — no Redis, no job server, and killing it mid-batch loses nothing.

Live run: 25 batches → 188 feeds polled, 3,418 articles, ~134 feeds/min. That is one full sweep per 6h interval.

Podcasts

feeds harvest-podcasts uses the iTunes Search API — the same keyless source media-streamer uses, and the one directory that returns a real feedUrl. No bulk export exists, so it is assembled from many paced searches: 2,642 unique feeds from the first 29 queries. Resumable; Ctrl-C keeps what it found.

Shaped for what comes next

Feeds are global rows and subscriptions keys a feed to an account, so a follow/unfollow product adds rows rather than needing the catalogue re-modelled. accounts.auth_kind leaves room for email or CoinPay. Both tables are empty and unused — this CLI is still single-user.

Commands

rssamp feeds import-opml --file ~/smallweb.opml
rssamp feeds harvest-podcasts [--all]
rssamp feeds stats
rssamp feeds recent --kind podcast
rssamp daemon start [--batch 8] [--pause 2]

Tests

38 new tests, all passing. Full suite is 192 passing / 11 failing — all 11 pre-existing in config-manager and feed-scheduler, neither of which this touches. (Note: the full suite does not terminate cleanly on its own, also pre-existing — feed-scheduler leaves real cron timers running.)

🤖 Generated with Claude Code

[github-actions]

ThreatCrush Security Scan

9 finding(s)

HIGH/CRITICAL: 8 | MEDIUM: 1

Severity	Rule	Location
HIGH	secret-generic-api-key	docs/CONFIGURATION.md:360
HIGH	secret-generic-credential	docs/CONFIGURATION.md:361
HIGH	secret-generic-credential	docs/CONFIGURATION.md:362
HIGH	secret-generic-credential	docs/CONFIGURATION.md:363
HIGH	secret-generic-credential	docs/CONFIGURATION.md:367
HIGH	secret-generic-api-key	docs/CONFIGURATION.md:368
HIGH	secret-generic-credential	docs/CONFIGURATION.md:368
HIGH	tls-verification-disabled	docs/TROUBLESHOOTING.md:221
MEDIUM	manifest-install-lifecycle-script	package.json:37

Snippets are redacted; ThreatCrush never prints matched credential material.