First step of reworking the firehose into owned, extensible code. We do it in phases — this issue is profiles only — but lay down a seam later phases (follows, relay lists, relay health) plug into without rewrites.
Background
Today src/indexer.js opens one subscription on kinds [0,3,10002] and blindly raw-upserts every event via upsertEvent. Two problems for profiles specifically:
- No validation — a malicious relay can inject a forged kind-0 and we'd store it as a
did:nostr identity. beacon is an identity/SSO substrate, so profile integrity matters.
- Indexes aren't owned by code —
connect() only creates {pubkey:1}. The content_text index that searchProfiles ($text) depends on exists only because it was created by hand; a fresh deploy silently breaks profile search.
(Legacy context: on nostr.social the old firehose.js/followshose.js/relayhose.js each do this per-kind and crash-loop; consolidating them is a later phase.)
Scope — phase 1 (profiles, kind 0)
- Hose seam: introduce a small hose contract —
{ name, kinds, ensureIndexes(db), ingest(event, db) } — and a src/hoses/ dir. Later phases add sibling modules.
src/hoses/profiles.js: the first concrete hose.
- Signature verification via
@noble/curves (schnorr) + @noble/hashes (sha256): recompute the NIP-01 event id, reject id/sig mismatches. Keeps the repo's "no nostr-tools" approach.
- Structural checks: kind 0, 64-hex pubkey, finite
created_at, JSON-parseable content (empty allowed).
- Owns its indexes:
pubkey, created_at, and the content_text text index (so search works on a fresh deploy).
- Latest-wins upsert into the
beacon collection (unchanged shape).
- Wire
indexer.js: route kind-0 through the profiles hose; kinds 3 and 10002 keep the existing upsertEvent path unchanged (migrated in their own phases). Call ensureIndexes at startup.
- Tests: sign events with a known key and assert accept/reject (valid, tampered content, bad sig, non-hex pubkey, wrong kind) + latest-wins.
Explicitly out of scope (later phases)
- Follows hose (kind 3) + relay-URL harvesting — phase 2
- Relay-lists hose (kind 10002) — phase 3
- Active relay-health prober (unfreezes
/relays) — phase 4
- Deploying / retiring the legacy server hoses — phase 5
Acceptance
npm test green, including new profiles-hose tests.
- Forged/tampered kind-0 events are rejected; valid ones upsert latest-wins.
- A fresh
runIndexer() creates the content_text index so /api/search works without manual setup.
First step of reworking the firehose into owned, extensible code. We do it in phases — this issue is profiles only — but lay down a seam later phases (follows, relay lists, relay health) plug into without rewrites.
Background
Today
src/indexer.jsopens one subscription on kinds[0,3,10002]and blindly raw-upserts every event viaupsertEvent. Two problems for profiles specifically:did:nostridentity. beacon is an identity/SSO substrate, so profile integrity matters.connect()only creates{pubkey:1}. Thecontent_textindex thatsearchProfiles($text) depends on exists only because it was created by hand; a fresh deploy silently breaks profile search.(Legacy context: on nostr.social the old
firehose.js/followshose.js/relayhose.jseach do this per-kind and crash-loop; consolidating them is a later phase.)Scope — phase 1 (profiles, kind 0)
{ name, kinds, ensureIndexes(db), ingest(event, db) }— and asrc/hoses/dir. Later phases add sibling modules.src/hoses/profiles.js: the first concrete hose.@noble/curves(schnorr) +@noble/hashes(sha256): recompute the NIP-01 event id, reject id/sig mismatches. Keeps the repo's "no nostr-tools" approach.created_at, JSON-parseablecontent(empty allowed).pubkey,created_at, and thecontent_texttext index (so search works on a fresh deploy).beaconcollection (unchanged shape).indexer.js: route kind-0 through the profiles hose; kinds 3 and 10002 keep the existingupsertEventpath unchanged (migrated in their own phases). CallensureIndexesat startup.Explicitly out of scope (later phases)
/relays) — phase 4Acceptance
npm testgreen, including new profiles-hose tests.runIndexer()creates thecontent_textindex so/api/searchworks without manual setup.