Clockwork Diffusion: Efficient Generation With Model-Step Distillation

[remyx-ai]

Recommended paper: Clockwork Diffusion: Efficient Generation With Model-Step Distillation
Confidence: moderate (Remyx relevance 0.65)
Research interest: (pin-method)

---

TL;DR

Clockwork Diffusion reuses cached low-res UNet features across denoise steps to cut ~32% FLOPs. No natural slot in diffusers core (reuse point is inside UNet forward, beyond callbacks) — decide: modify UNet forward, community pipeline, or wrapper? (2312.08128)

Suggested experiment

Implement Clockwork low-resolution feature caching as an examples/community/ custom pipeline (or UNet wrapper) for StableDiffusionPipeline on SD v1.5: cache the low-res block outputs during one denoising step and replay them on a per-baseline reuse schedule across an 8-step DPM++ run. Acceptance: reproduce >=30% FLOP reduction with negligible FID/CLIP change vs. the uncached baseline, reported alongside wall-clock latency on a fixed GPU.

Why the orchestrator opened an Issue instead of a PR

Pre-flight routed to Issue before implementation

Why this didn't ship as a PR

Clockwork Diffusion is an inference-only feature-caching optimization, so it needs no trainer, new data format, or checkpoint loader — missing-infra doesn't block it. But its reuse point lives inside the UNet forward (caching/replaying low-resolution block outputs), beyond what diffusers' callback system can reach, and there's no natural core slot for per-block caching (comparable methods like DeepCache are maintained externally, not in diffusers core). With no selection rationale and no suggested experiment to pin a slice, the realistic implementation is either an opinionated change to a core model or a freestanding community pipeline that no core code calls, so the team should decide where it belongs first.

Engineering analysis

Clockwork Diffusion is an inference-time efficiency method for diffusion UNets. The core observation: within a UNet, blocks operating on high-resolution feature maps are sensitive to perturbation, while low-resolution blocks (deep down/mid/up blocks) mostly steer the semantic layout and tolerate being reused across denoising steps. Clockwork periodically caches the output of these low-res blocks during one step and replays the cache on subsequent step(s) instead of recomputing them, governed by a per-block reuse schedule. The paper reports ~32% FLOP savings on SD v1.5 (8 DPM++ steps) with negligible FID/CLIP change, and similar gains for image editing.

This consumes a stock checkpoint, requires no training, no new data format, and no new checkpoint loader — so the missing-infra concern does not apply. The open question is purely architectural fit inside diffusers.

What blocks a clean implementation

There is no natural host for the mechanism in diffusers core, and the recommendation spec ships no selection rationale and no suggested experiment to pin a slice:

• The reuse point is inside the UNet forward, not in the scheduler or the denoising loop. diffusers' PipelineCallback / callback_on_step_end exposes step-level state (latents, timestep) but not intermediate block hidden states, so the optimization cannot be expressed as a callback — it must reach into the UNet internals.
• A faithful implementation therefore either (a) adds a caching/replay path to UNet2DConditionModel.forward (and ideally the other conditionable UNets) — an opinionated change to a core model for a single research optimization — or (b) ships as a wrapper UNet or an examples/community/ custom pipeline that no core code imports or calls, i.e. effectively freestanding.
• Comparable methods (DeepCache, block caching, T-GATE) are maintained in separate external repos, not in diffusers core, which suggests the project's convention is to keep per-block feature caching outside the library. There's no existing "inference-cache" slot to drop into.
• The spec names no call sites and proposes no experiment, so there is nothing concrete to validate a slice against.

How to unblock this

Decide where a feature-caching inference optimization should live before implementing:

1. Core vs. extension: Is the team open to a caching/replay path in UNet2DConditionModel.forward (and which other UNets), or should this stay as an examples/community/ custom pipeline / wrapper out of core? Given DeepCache et al. are external, what's the bar for promoting this one in?
2. Scope: Which UNet architectures to target — SD/SDXL UNet only? Flux and other transformer backbones are structured differently and may lack an obvious "low-res block" analog. Confirm SD v1.5 DPM++ as the first benchmark.
3. Mechanism: Cache the output of which blocks specifically, and on what reuse schedule (the paper fixes this per-baseline)? Reproduce the paper's ~32% FLOP / negligible FID+CLIP claim on SD v1.5 8-step DPM++ as the acceptance bar.
4. Evaluation path: Confirm a FLOP-counting + FID/CLIP harness is acceptable for the PR, or whether a lighter perceptual/latency proxy is preferred.

---

Opened by the Remyx Recommendation orchestrator. Pre-flight routed this paper to Issue before the coding agent ran — see the reasoning above for what would need to change to scaffold it as a PR.

Reopen this Issue if you want Outrider to revisit this paper later. While it stays closed, the orchestrator will not re-recommend the same paper.