Skip to content

Redis cluster support Phase 4: two-phase XREADGROUP split for multi agent_type workers #13

Description

@zhchxiao123

What to build

Using the RedisStreamOps interface introduced in the Phase 4 prefactor issue (#12), implement the same two-phase read loop as Python/TS for the agent_type ctrl-stream side of WorkerRunner's now-split read paths:

  1. Phase one: concurrently issue one non-blocking single-stream read (via RedisStreamOps, blockMs=null) per active agent_type stream. If any stream returned messages, return them immediately.
  2. Phase two: only if every stream came back empty, issue one real blocking read (blockMs=total_block_ms) against a single "primary" stream, chosen by round-robin across the declared agent_types so no agent_type is permanently starved of the blocking slot.

This eliminates CROSSSLOT (every stream read now touches exactly one stream/slot) without changing the current worst-case delay characteristics: a message that arrives on a non-primary stream while phase two is mid-block still waits up to the full block timeout; a message arriving at any other time is picked up on the next loop iteration.

count/PEL-size semantics are the same as Python/TS: count is applied per-stream, so a worker declaring N agent_types could already pull up to N × count messages per round before this change — this doesn't change that ceiling.

Add multi-agent_type integration test coverage (can run against a local Cluster docker-compose environment marked as a slow/optional CI job) verifying: no CROSSSLOT errors; phase-one non-blocking scan picks up messages within one loop iteration when not blocked in phase two; round-robin primary rotation actually rotates; explicit worst-case latency assertion (bounded by the block timeout, not the loop interval); the split-out workerCtrlStream read path (from #12) continues to function independently and isn't affected by this loop's blocking state.

Acceptance criteria

  • agent_type stream reads in WorkerRunner replaced by the two-phase (concurrent non-blocking scan, then single-primary blocking read) pattern, using RedisStreamOps
  • Round-robin primary rotation implemented and covered by a test asserting no agent_type is starved
  • Worst-case latency test: message arriving during phase-two block on a non-primary stream is picked up only after that block ends
  • No CROSSSLOT errors in a local Cluster integration test with 3+ declared agent_types
  • workerCtrlStream read path from Redis cluster support Phase 4 prefactor: split WorkerRunner.runLoop() + minimal RedisStreamOps interface #12 verified to run independently, unaffected by the agent_type loop's blocking state

Blocked by

#11, #12

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions