Is your feature request related to a problem? Please describe.
diffusers' quantizer backends are exclusively thin adapters over third-party libraries (autoround, torchao, quanto, bitsandbytes, gguf, modelopt) that only load pre-quantized checkpoints — none implements a quantization algorithm natively, and there is no custom-kernel surface. HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models is a new paper proposing a post-training weight + KV-cache pipeline (per-tile Randomized Hadamard Transform → optimal-lattice quantization onto E8/D4/A2/Z → lossless bit-stripping + near-entropy-optimal Rice coding of lattice indices → KV-cache bias correction) that reports ~3.9x linear-weight compression and ~3.79x KV compression at near-lossless quality at 4 bps on an H100, with no per-frame artifacts on a 19B LTX-2 DiT. But HyperQuant ships no library, no reference kernels, and no pre-quantized checkpoints, so there is nothing to wrap or load today — a diffusers backend would have to reimplement the algorithm from scratch, breaking the established integration model.
Describe the solution you'd like.
A team decision and a scoped first slice. Prototype Randomized Hadamard Transform + lattice (E8/D4) weight quantization for a single DiT (e.g. Ideogram 4) in a standalone notebook and measure reconstruction error and generation fidelity (e.g. CLIP-score / FID on a small prompt set) at 4 bps versus the existing AutoRound and torchao backends — explicitly excluding the inference-speed and memory-compression claims, which require the paper's custom Tensor-Core MMA kernels that diffusers does not host. The result tells the team whether a native quality-only slice is worth merging before kernels exist.
Describe alternatives you've considered.
- Wait and adapt. Wait for a third-party HyperQuant library to mature, then add an adapter backend — the diffusers-consistent path, mirroring AutoRound/torchao.
- Own native + kernels. Commit to a native RHT+lattice implementation plus custom 8/4-bit Tensor-Core MMA kernels (fp8-e4m3, int8, nvfp4, mxfp4; the paper notes int8 beats fp8 on the post-RHT lattice output). This would be the first native quantization algorithm and the first to require bespoke CUDA kernels in diffusers.
- Quality-only proof first. Land a weight-only RHT+lattice prototype that measures reconstruction/generation quality only (no speed/memory), deferring the kernel and packed Rice-coded checkpoint format. Note diffusers currently loads safetensors to standard tensors and has no path to load a packed lattice-index format.
Additional context.
What blocks a clean implementation
- No adapter target. Every diffusers quantizer backend is a thin wrapper over an external library and only loads pre-quantized checkpoints (AutoRound's quantizer explicitly states on-the-fly calibration is unsupported). HyperQuant has nothing to wrap.
- Value needs infrastructure the repo lacks. The measured gains (4-bit inference speedup, ~3.9x memory compression) require (a) the paper's custom Tensor-Core MMA dequantization kernels and (b) a packed, Rice-coded lattice-index checkpoint format. diffusers has neither custom CUDA kernels nor a loader for such a packed format; a PyTorch-only RHT+lattice prototype can measure quality but cannot deliver the speed/memory the headline comparison implies.
- Freestanding-native scope decision. A from-scratch HyperQuant backend would be a meaningful scope/architecture expansion (first native quantizer, first bespoke kernels) the maintainers should sign off on rather than land via a drive-by PR.
Questions to unblock
- Integration model: (a) wait for a third-party HyperQuant library to mature and add an adapter, or (b) own a native RHT+lattice implementation plus custom MMA kernels?
- Scope the measurable slice: is a quality-only proof a worthwhile first PR, or is the speed/memory story a prerequisite for merging?
- Kernel ownership: who maintains the custom int8/fp8 lattice-MMA paths and on which hardware (the suggested AWS Neuron path vs. the paper's H100 CUDA)?
- Checkpoint format: appetite to add a packed Rice-coded loader path, or store HyperQuant outputs as dequantized standard tensors (forfeiting the compression benefit)?
Why this is an Issue instead of a PR
Pre-flight routed this to an Issue before the coding agent ran. The scoped "wire it into the quantizers" slice has no clean drop-in: HyperQuant is a brand-new paper with no released library or checkpoints to wrap, and its headline value (4-bit inference speedup, ~3.9x compression) depends on custom Tensor-Core MMA kernels plus a packed Rice-coded checkpoint format that diffusers has no path for. Routed to an Issue for the team to decide whether to wait for a third-party library to adapt, or to take on native algorithm + kernel ownership.
License & code availability
🟡 No code repository surfaced — no LICENSE could be fetched to evaluate. Worth confirming the paper has an open release before investing in adoption.
- Code / model: no repository or model URL surfaced in the paper, recommendation envelope, or arxiv abstract page.
- License:
(none detected) (class: no-code-link, compat: 0.30).
Drafted by Outrider — paper: arXiv:2606.23406.
Discovery context
Recommended paper: HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models
Confidence: high (Remyx relevance 0.98)
Research interest: diffusers
TL;DR: HyperQuant compresses DiT weights ~3.9x@4-bit via Hadamard+lattice+Rice with custom MMA kernels. No clean PR: diffusers backends wrap 3rd-party libs; no HyperQuant lib/checkpoints exist yet. Decide: wait, or own a native impl+kernels?
Reopen this Issue if you want Outrider to revisit this paper later. While it stays closed, the orchestrator will not re-recommend the same paper.
Is your feature request related to a problem? Please describe.
diffusers' quantizer backends are exclusively thin adapters over third-party libraries (
autoround,torchao,quanto,bitsandbytes,gguf,modelopt) that only load pre-quantized checkpoints — none implements a quantization algorithm natively, and there is no custom-kernel surface. HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models is a new paper proposing a post-training weight + KV-cache pipeline (per-tile Randomized Hadamard Transform → optimal-lattice quantization onto E8/D4/A2/Z → lossless bit-stripping + near-entropy-optimal Rice coding of lattice indices → KV-cache bias correction) that reports ~3.9x linear-weight compression and ~3.79x KV compression at near-lossless quality at 4 bps on an H100, with no per-frame artifacts on a 19B LTX-2 DiT. But HyperQuant ships no library, no reference kernels, and no pre-quantized checkpoints, so there is nothing to wrap or load today — a diffusers backend would have to reimplement the algorithm from scratch, breaking the established integration model.Describe the solution you'd like.
A team decision and a scoped first slice. Prototype Randomized Hadamard Transform + lattice (E8/D4) weight quantization for a single DiT (e.g. Ideogram 4) in a standalone notebook and measure reconstruction error and generation fidelity (e.g. CLIP-score / FID on a small prompt set) at 4 bps versus the existing AutoRound and torchao backends — explicitly excluding the inference-speed and memory-compression claims, which require the paper's custom Tensor-Core MMA kernels that diffusers does not host. The result tells the team whether a native quality-only slice is worth merging before kernels exist.
Describe alternatives you've considered.
Additional context.
What blocks a clean implementation
Questions to unblock
Why this is an Issue instead of a PR
Pre-flight routed this to an Issue before the coding agent ran. The scoped "wire it into the quantizers" slice has no clean drop-in: HyperQuant is a brand-new paper with no released library or checkpoints to wrap, and its headline value (4-bit inference speedup, ~3.9x compression) depends on custom Tensor-Core MMA kernels plus a packed Rice-coded checkpoint format that diffusers has no path for. Routed to an Issue for the team to decide whether to wait for a third-party library to adapt, or to take on native algorithm + kernel ownership.
License & code availability
🟡 No code repository surfaced — no LICENSE could be fetched to evaluate. Worth confirming the paper has an open release before investing in adoption.
(none detected)(class:no-code-link, compat: 0.30).Drafted by Outrider — paper: arXiv:2606.23406.
Discovery context
Recommended paper: HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models
Confidence: high (Remyx relevance 0.98)
Research interest: diffusers
TL;DR: HyperQuant compresses DiT weights ~3.9x@4-bit via Hadamard+lattice+Rice with custom MMA kernels. No clean PR: diffusers backends wrap 3rd-party libs; no HyperQuant lib/checkpoints exist yet. Decide: wait, or own a native impl+kernels?
Reopen this Issue if you want Outrider to revisit this paper later. While it stays closed, the orchestrator will not re-recommend the same paper.