6cy — experimental streaming-first archive format (seeking design feedback)

[byte271]

Select Topic Area

Show & Tell

Body

Sharing 6cy — an experimental, Rust reference implementation of a streaming-first archive/container format (repo on GitHub: https://github.com/byte271/6cy). It supports per-block codec polymorphism, a plugin ABI for binary codecs, and robust partial-recovery semantics; this is v0.x and not production-ready. I’m looking for design feedback on the spec (endianness, index layout), the plugin ABI (UUID vs short id, memory model, thread-safety), and ideas for capability negotiation — happy to post plugin ABI details or raw benchmark logs on request.

[midiakiasat]

Streaming-first is correct. The success hinges on strict boundaries and recoverability.

Key points:

• Fix endianness (little-endian). Never negotiate it.
• Make every block self-describing + checksummed (magic, version, codec UUID, sizes, hash).
• Write index at end; make it fully reconstructible by scanning blocks.
• Use UUID for codec identity; optional short IDs as fast-path only.
• Freeze a minimal C ABI for plugins. No shared allocators. Explicit buffers. Declared thread-safety.
• No dynamic negotiation mid-stream container declares required codec, decoder either supports it or fails.
• Checksums mandatory per block for true partial recovery.

If block independence + ABI stability are solid, the format survives. If not, it won’t.