[FEAT] Webhook Delivery Queue with Retry, Dead Letter, and Ordering

[oomokaro1]

[FEAT] Webhook Delivery Queue with Retry, Dead Letter, and Ordering

Priority: High

Difficulty: Hard
Estimated Effort: 3-4 days
Relevant Packages: OrbitStream_backend/, orbitstream_docs/
Labels: enhancement, infrastructure, priority:high

Requirements

1. Job Queue Architecture

• Implement a Redis-backed job queue (use Bull from @nestjs/bull or a custom implementation)
• Queue name: webhook-delivery
• Jobs are enqueued when dispatchWebhook() is called
• A separate worker process (or same process with @Processor) consumes jobs from the queue

2. Priority Levels

• payment.confirmed → priority 1 (highest — merchants need this to fulfill orders)
• session.expired → priority 2
• session.created → priority 3
• session.cancelled → priority 3
• payment.failed → priority 2

3. Retry with Exponential Backoff + Jitter

• Retry schedule: 1min → 5min → 30min → 2hr → 12hr (max 5 attempts)
• Add random jitter (±20%) to each interval to prevent thundering herd
• Use Bull's built-in delay backoff or implement custom backoffDelay(attempt) function
• After 5 failed attempts, move to dead letter queue

4. Dead Letter Queue

• Queue name: webhook-dead-letter
• Dead letter jobs include: original payload, all delivery attempts with timestamps and error messages, merchant ID, event type
• Merchants can view dead letter entries via GET /v1/webhooks/dead-letter (JWT-authenticated)
• Merchants can manually retry dead letter entries via POST /v1/webhooks/dead-letter/:id/retry

5. Idempotency

• Each webhook delivery gets a unique X-OrbitStream-Delivery-Id header (UUID v4)
• Each delivery also gets X-OrbitStream-Timestamp (ISO 8601)
• The signature covers: delivery_id + timestamp + payload
• Merchants must store delivery IDs and reject duplicates
• Document idempotency in the integration guide

6. Webhook Ordering

• Webhooks for the same session must arrive in order
• Use a per-session sequence number: webhook_deliveries table gets a sequence column
• The worker processes jobs for the same session sequentially (not in parallel)
• If a job for session X is being processed, subsequent jobs for session X are delayed until the first completes

7. Network Handling

• HTTP timeout: 10 seconds
• 2xx response: success, mark delivered
• 4xx response: don't retry (merchant endpoint is broken), move to dead letter immediately
• 5xx response: retry with backoff
• Network error / timeout: retry with backoff
• Log all delivery attempts with full error details

8. API Endpoints

• GET /v1/webhooks/deliveries — list recent deliveries (JWT auth, per-merchant)
• GET /v1/webhooks/dead-letter — list dead letter entries (JWT auth, per-merchant)
• POST /v1/webhooks/dead-letter/:id/retry — manually retry a dead letter entry
• DELETE /v1/webhooks/dead-letter/:id — dismiss a dead letter entry

9. Testing

• Unit tests: retry backoff calculation, priority ordering, dead letter placement
• Integration tests: verify webhook delivery with mocked HTTP endpoint
• Test 4xx → dead letter (no retry), 5xx → retry
• Test ordering: verify webhooks for same session arrive in order
• Test idempotency: verify delivery IDs are unique and included in headers
• Load test: verify queue handles 100 concurrent webhook deliveries

[Joycejay17]

Hi, I'd like to apply for this issue.
This is a Redis-backed webhook delivery system with priority-aware retries, a dead letter queue, strict idempotency, and per-session ordering — replacing whatever currently calls dispatchWebhook() directly with a durable queue (webhook-delivery) consumed by a worker, with failures eventually routed to webhook-dead-letter.
Plan, by requirement:

Queue — @nestjs/bull queue named webhook-delivery, with a @processor consuming jobs. dispatchWebhook() becomes an enqueue call rather than a direct HTTP send.
Priority — map event type to Bull job priority at enqueue time (payment.confirmed = 1 down to session.created/session.cancelled = 3), so the queue naturally services payment-critical events first under load.
Backoff — custom backoffDelay(attempt) implementing the 1m/5m/30m/2h/12h schedule with ±20% jitter applied per attempt (not just at enqueue), so retries don't cluster. After attempt 5 fails, the job moves to the dead letter queue with full attempt history.
Dead letter — store original payload, merchant ID, event type, and a timestamped log of every attempt (status code/error) on each DLQ entry. GET /v1/webhooks/dead-letter and POST /v1/webhooks/dead-letter/:id/retry (re-enqueue with attempt counter reset) and DELETE /v1/webhooks/dead-letter/:id (dismiss), all JWT-scoped to the requesting merchant.
Idempotency — generate X-OrbitStream-Delivery-Id (UUIDv4) and X-OrbitStream-Timestamp per delivery attempt, sign over delivery_id + timestamp + payload, and document in the integration guide that merchants must dedupe on delivery ID (since retries reuse the same logical event but get fresh IDs per attempt — I'll confirm with you whether delivery ID should be stable across retries of the same job or unique per attempt, since the spec's wording could support either and it changes what merchants need to dedupe on).
Ordering — the part I'd be most careful with. A naive global concurrency: 1 on the worker would serialize all sessions, not just same-session jobs, killing throughput. Instead I'd add a sequence column on webhook_deliveries and use a short-lived Redis lock keyed by session ID at dequeue time: if a job for session X is mid-flight, a second job for session X gets re-enqueued with a small delay instead of processing; jobs for other sessions proceed concurrently. This keeps per-session order without bottlenecking the whole queue.
Network handling — 10s timeout; 2xx marks delivered; 4xx skips retry and goes straight to DLQ (broken endpoint, no point retrying); 5xx and timeouts/network errors retry per the backoff schedule. Full attempt logging either way.
API — the four endpoints as specified, all JWT-authenticated and merchant-scoped.
Testing — unit tests on backoff math, priority ordering, and DLQ placement logic; integration tests against a mocked HTTP endpoint covering the 4xx-no-retry / 5xx-retry split; an ordering test asserting same-session jobs land in sequence even under concurrent enqueue; idempotency header uniqueness; and a load test at 100 concurrent deliveries.

[oomokaro1]

Hi, I'd like to apply for this issue. This is a Redis-backed webhook delivery system with priority-aware retries, a dead letter queue, strict idempotency, and per-session ordering — replacing whatever currently calls dispatchWebhook() directly with a durable queue (webhook-delivery) consumed by a worker, with failures eventually routed to webhook-dead-letter. Plan, by requirement:

Queue — @nestjs/bull queue named webhook-delivery, with a @processor consuming jobs. dispatchWebhook() becomes an enqueue call rather than a direct HTTP send. Priority — map event type to Bull job priority at enqueue time (payment.confirmed = 1 down to session.created/session.cancelled = 3), so the queue naturally services payment-critical events first under load. Backoff — custom backoffDelay(attempt) implementing the 1m/5m/30m/2h/12h schedule with ±20% jitter applied per attempt (not just at enqueue), so retries don't cluster. After attempt 5 fails, the job moves to the dead letter queue with full attempt history. Dead letter — store original payload, merchant ID, event type, and a timestamped log of every attempt (status code/error) on each DLQ entry. GET /v1/webhooks/dead-letter and POST /v1/webhooks/dead-letter/:id/retry (re-enqueue with attempt counter reset) and DELETE /v1/webhooks/dead-letter/:id (dismiss), all JWT-scoped to the requesting merchant. Idempotency — generate X-OrbitStream-Delivery-Id (UUIDv4) and X-OrbitStream-Timestamp per delivery attempt, sign over delivery_id + timestamp + payload, and document in the integration guide that merchants must dedupe on delivery ID (since retries reuse the same logical event but get fresh IDs per attempt — I'll confirm with you whether delivery ID should be stable across retries of the same job or unique per attempt, since the spec's wording could support either and it changes what merchants need to dedupe on). Ordering — the part I'd be most careful with. A naive global concurrency: 1 on the worker would serialize all sessions, not just same-session jobs, killing throughput. Instead I'd add a sequence column on webhook_deliveries and use a short-lived Redis lock keyed by session ID at dequeue time: if a job for session X is mid-flight, a second job for session X gets re-enqueued with a small delay instead of processing; jobs for other sessions proceed concurrently. This keeps per-session order without bottlenecking the whole queue. Network handling — 10s timeout; 2xx marks delivered; 4xx skips retry and goes straight to DLQ (broken endpoint, no point retrying); 5xx and timeouts/network errors retry per the backoff schedule. Full attempt logging either way. API — the four endpoints as specified, all JWT-authenticated and merchant-scoped. Testing — unit tests on backoff math, priority ordering, and DLQ placement logic; integration tests against a mocked HTTP endpoint covering the 4xx-no-retry / 5xx-retry split; an ordering test asserting same-session jobs land in sequence even under concurrent enqueue; idempotency header uniqueness; and a load test at 100 concurrent deliveries.

I love your thorough approach. Please make sure all my ci pass in your pr.

[grantfox-oss]

🦊 GrantFox — @Joycejay17 has been assigned to this issue as part of the Official Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #8)
2. Your PR will be reviewed by the OrbitStream maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign campaign!

@Joycejay17's PR #23 was approved and merged by @oomokaro1.

🏆 @Joycejay17: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (48 total points). Track your full progress on GrantFox.

👏 Great work, @Joycejay17! Keep contributing to OrbitStream.