[Initiative] Controller Watch API with resourceVersion Resume

[juliuskrah]

Summary

Introduce a controller-facing Watch API for catalog state changes so internal controllers can subscribe to ordered deltas instead of polling. The stream uses a monotonic resourceVersion cursor to support disconnect recovery and deterministic replay.

This initiative captures the control-plane event model from the Kubernetes-style brainstorming session and complements existing Git + Scylla work by defining how reconcilers consume change events safely.

In Scope

• Define a Watch API contract for internal controllers to consume catalog change events.
• Add monotonic resourceVersion semantics to stream payloads and checkpoints.
• Support reconnect/resume from last-seen resourceVersion with gap-safe replay behavior.
• Define ordering and delivery guarantees (at-least-once with deterministic replay window).
• Add server-side filtering primitives (namespace/kind) to reduce controller fan-out load.
• Add integration tests covering initial watch, resume after disconnect, and replay from stale cursor.
• Document controller usage patterns and failure semantics.

Out of Scope

• Public storefront real-time subscriptions.
• Full event-sourcing replacement for existing read paths.
• Non-catalog event families.
• UI for stream inspection.

Acceptance Criteria

• A documented Watch API contract exists with explicit resourceVersion semantics.
• Controllers can resume from a previously persisted resourceVersion after disconnect.
• Replay behavior is deterministic and documented for stale or invalid cursors.
• Integration tests verify ordered delta delivery and reconnect behavior.
• Server-side filtering by namespace and/or resource kind is available.
• Operational docs include runbook guidance for lagging watchers and replay recovery.

Area

infra

Priority

p2 – high

Target Milestone / Release

TBD

[juliuskrah]

Dependencies

• Depends on: [Initiative] Datastore Backend: ScyllaDB with Migrations #102 (ScyllaDB datastore — Watch events need a durable, ordered store for resourceVersion bookkeeping)
• Depends on: [Initiative] Decouple API from Git Storage via gRPC Git Service #65 (gRPC Git Service — deltas originate from git-service push/tag events)
• Enables: [Initiative] Tag-Gated Sync Controller for Atomic Release Promotion #132 (Sync Controller reuses the Watch API to reconcile tagged snapshots)
• Related to: [Initiative] Optional KV-Optimized Catalogue Read Layer #112 (KV read-layer — Watch events trigger cache invalidation)