Skip to content

feat: parallel-safe playlist ordering + HLS-muxer catch-up encoder #23

Description

@korutx

Goal

Unblock intra-device parallelism for draining a backlog concentrated in few devices (current bottleneck: per-device-serial encoding; ~8 msg/s, CPU 1.6/8).

Compatibility check (publisher)

Verified quarkus-http-pulsar-publisher / RedisPlaylistService:

  • Stamps a uniform #EXTINF:{segment-duration}.0 + TARGETDURATION per segment (does not read real durations) → segments must stay ~segment_duration.
  • Live playlist = last N segments ordered by Redis ZSET score, media-sequence from the first → assumes segments land in number order (i.e. serial today).

Part 1 — parallel-safe ordering (PR #24)

Use the segment's first-frame capture timestamp as the ZSET score instead of time.time(), so ordering is by content time and survives out-of-order (parallel) generation. Cleanup's time-based retention still works (content age). No quarkus change.

Part 2 — HLS-muxer catch-up encoder (prototyped, proven)

One FFmpeg with -f hls -hls_time <dur> -start_number <base> turns a device's whole backlog into many segments in one process. Local prototype vs the quarkus contract:

  • 5 segments, contiguous seg_%06d.ts from base ✅
  • every segment exactly 6.000s → uniform EXTINF valid ✅
  • quarkus-style playlist references exactly the produced files ✅
  • 4x fewer process-spawns for the same output (scales ~1000x on a real backlog)
  • Caveat: PTS continuity across segments needs -output_ts_offset/copyts handling (archive/seek only; live unaffected).

Acceptance

  • Playlist ordering by content time (Part 1).
  • Catch-up encoder behind a flag, used when a device's queue depth is large; live edge stays serial.
  • Segments stay exactly segment_duration; PTS handled for archive.
  • Lint/type/tests pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceIssues affecting system speed or efficiency

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions