[Initiative] Controller Manager Reconciliation Loop for Core Resources and CRDs

[juliuskrah]

Summary

Define a Kubernetes-style controller manager in gitstore-api that reconciles both core resources and CRD-defined kinds from unified API watch events while preserving strict ownership boundaries: .spec is Git-owned and .status is controller-owned.

This initiative establishes the runtime loop that consumes watch events, executes reconciliation handlers, and writes observed state back through the API contract.

Scope

In Scope

• Define the controller manager runtime model (worker lifecycle, queueing, retry/backoff, idempotency).
• Define reconcile contracts for core resources and CRD kinds with a shared handler interface.
• Define list-then-watch startup flow for controllers (resourceVersion checkpointing and resume).
• Define ownership boundaries for writes: .spec via Git pipeline, .status via API status mutation.
• Define optimistic concurrency requirements for status updates using resourceVersion/preconditions.
• Add integration tests for controller startup, disconnect/reconnect resume, and status write conflict handling.
• Document operational runbooks (lagging controller, replay window exceeded, poisoned work item retries).

Out of Scope

• Buyer-facing AI UX and storefront personalization.
• Mutating admission controllers.
• Replacing GraphQL with REST endpoints.
• Public client-facing realtime subscriptions.

Acceptance Criteria

• Controller manager lifecycle and reconcile interface are documented and implemented.
• Core resources and CRD kinds can be reconciled through the same manager pipeline.
• Controllers resume safely from persisted resourceVersion checkpoints after disconnect.
• Status write path enforces optimistic concurrency and returns deterministic conflict errors.
• Integration tests cover successful reconcile, retry, replay/resume, and conflict scenarios.
• Runbooks exist for replay recovery and controller lag remediation.

Dependencies

• Depends on: [Initiative] Controller Watch API with resourceVersion Resume #131, [Initiative] Dynamic GraphQL Schema Synthesis and Stitching for Custom Kinds #149, [Initiative] Hub-and-Spoke CRD Versioning with WASI Conversion Hooks #164
• Related: [Initiative] gRPC GitEvent Notification Stream from gitstore-git-service to gitstore-api #139, [Initiative] Universal ScyllaDB Resource Table for CRD-Style Resources #140, [Initiative] Agent-Oriented Orchestration Workers in the Go Control Plane #150, [Initiative] GraphQL Status Subresource Contract for Controllers #166

Implementation Plan

1. #180 — Controller manager runtime foundations
2. #181 — Shared reconcile handler contract for core and CRD kinds (blocked by #180)
3. #182 — Startup resume and resourceVersion checkpointing contract (blocked by #180)
4. #183 — Integration tests and operations runbook (blocked by #181 and #182)

Tracking

• Area: infra
• Priority: p2 - high
• Target Milestone / Release: TBD

[juliuskrah]

Dependency note: Blocked by #131 (watch API semantics), #149 (dynamic schema synthesis for CRDs), and #164 (hub-and-spoke conversion model). This initiative should land before #166 so status updates align with controller runtime contracts.

[juliuskrah]

Related follow-up: #169 (Git Event Publication) defines which Git events trigger reconciliation and clarifies draft-vs-published state transitions. Controllers should subscribe to specific event types to drive their loops.