Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping

[remyx-ai]

Is your feature request related to a problem? Please describe.

Reference-conditioned diffusion pipelines (e.g., IP-Adapter subject-driven generation and spatially-aligned editing) pass a large set of reference-image tokens through cross-attention, which is a meaningful fraction of inference cost. The paper "Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping" (arXiv:2606.23682) observes that a large fraction of these reference tokens can be dropped at inference time while "largely" preserving generation quality, with reported 2–4× speedups (4× multi-reference, 2× single-reference). The catch: those headline numbers come from a model fine-tuned with random token dropping, which an inference-focused library like diffusers does not host.

Describe the solution you'd like.

Decide whether to add an inference-only reference-token budget option to existing reference-conditioned pipelines — e.g., an optional reference_token_budget / selection-mode parameter on the IP-Adapter attention processors (IP-Adapter on SDXL/Flux, OmniGen, etc.) — that selects an informative subset of reference tokens before attention. The open question for the team is whether such an option is worth shipping on its own merits, given that without the paper's fine-tuned weights, dropping tokens degrades quality rather than matching the paper's "no quality loss" result.

Describe alternatives you've considered.

• Full paper method (out of scope): fine-tune the model with random token dropping at varying ratios to make it robust to partial references, then apply task-aware token selection at inference. This is the load-bearing piece that produces the reported 2–4× speedups with no quality loss — but it requires a fine-tuned checkpoint with no loader path in diffusers, and no compatible checkpoint was surfaced. Out of scope for an inference / model-loading library.
• Selection rules for an inference-only option: random, attention-saliency, or spatial pooling. Needs validation of quality at a given drop ratio on checkpoints diffusers already loads before shipping a default.

Additional context.

Suggested experiment. Run an existing IP-Adapter pipeline (e.g., SDXL IP-Adapter) at increasing reference-token drop ratios under a few selection rules (random, attention-saliency, spatial pooling); report CLIP-I / DINO similarity to the full-reference baseline and wall-clock speedup. This tests the paper's inference-only observation on a checkpoint we already load and tells us whether an inference-only token-budget option is viable before any fine-tuning enters the picture.

Why this was opened as an Issue rather than a PR. This repo is diffusers, but the reported 2–4× speedups hinge on fine-tuning the model with random token dropping — training infrastructure diffusers doesn't host, with no compatible checkpoint or loader path here. The candidate-selection rationale was empty, so no host module for a scoped slice was validated; the only conceivable inference-only slice (task-aware reference-token selection in the IP-Adapter attention processors) degrades quality without the fine-tuned weights, making it a freestanding feature rather than a scoped, validated call site. Routed to Issue for the team to decide whether an inference-only token-budget option is worth shipping on its own merits.

Engineering notes. Reference tokens do flow through existing IP-Adapter attention processors / image-projection modules, so a task-aware token-selection option is conceivable there — but the inference-time selection only stays quality-neutral because the fine-tuned model was made robust to partial references. Unblocking questions: (1) Is there a released fine-tuned checkpoint we can load, or is the random-drop fine-tuning required to get any benefit? If required, this is out of scope for an inference library. (2) Which reference-conditioned pipeline should host the option, and should it surface as an optional reference_token_budget / selection-mode parameter on the existing attention processor? (3) What selection rule do we ship by default — can we validate quality at a given drop ratio on checkpoints we already load, or do we genuinely need the paper's weights?

Drafted by Outrider — paper: arXiv:2606.23682.

Discovery context

Recommended paper: Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping
Confidence: moderate (Remyx relevance 0.65)
Research interest: (pin-arxiv)

License & code availability. 🟡 No code repository surfaced — couldn't fetch a LICENSE to evaluate. Worth confirming the paper has an open release before investing in adoption.

• Code / model: no repository or model URL surfaced in the paper, recommendation envelope, or arxiv abstract page.
• License: (none detected) (class: no-code-link, compat: 0.30)

Reopen this Issue if you want Outrider to revisit this paper later. While it stays closed, the orchestrator will not re-recommend the same paper.

[smellslikeml]

@remyx-ai[bot] LEAD — autoresearch loop decision.

Hypothesis (outer agent): Token dropping for reference-conditioned diffusion fits diffusers perfectly — the repo hosts IP-Adapter and reference-conditioned pipelines whose cross/reference attention layers are the exact intervention point, and token dropping is an inference-time efficiency technique that lands naturally as a modular pipeline component without new training infrastructure.

Rationale: hypothesis: Token dropping for reference-conditioned diffusion fits diffusers as an inference-time efficiency technique on existing IP-Adapter attention processors — observed: the paper's headline 2–4× speedups require a fine-tuned model with random token dropping that diffusers cannot load, making the full method out of scope; however, the issue surfaces a concrete inference-only experiment slice that exercises existing IP-Adapter pipelines and standard evaluation metrics — conclusion: LEAD, because the suggested experiment is architecturally viable on diffusers' existing modules and would produce actionable signal on whether any inference-only token-budget option is worth shipping.

Research LEAD (architecturally-viable experiment surfaced by preflight):

"Run an existing IP-Adapter pipeline (e.g., SDXL IP-Adapter) at increasing reference-token drop ratios under a few selection rules (random, attention-saliency, spatial pooling); report CLIP-I / DINO similarity to the full-reference baseline and wall-clock speedup. This tests the paper's inference-only observation on a checkpoint we already load and tells us whether an inference-only token-budget option is viable before any fine-tuning enters the picture." This experiment is architecturally viable because diffusers already hosts SDXL IP-Adapter pipelines with accessible cross-attention processors where reference tokens flow, and CLIP-I/DINO similarity against a full-reference baseline is a standard evaluation path that requires no new training infrastructure — only an inference-time token-selection hook on existing attention processors.

The paper itself doesn't fit, but the experiment above uses existing target modules — this is a labeled engineering lead worth queueing, distinct from a REJECT.

[remyx-ai]

Confirmed. The autoresearch loop decision for the LEAD on token dropping was ISSUE.

The run (a6a49801) evaluated Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping (arXiv:2606.23682v1) as a single, pinned candidate. The pre-flight check determined that a direct code integration was not viable, aligning with the rationale that the paper's full method requires fine-tuned models unavailable in diffusers.

The decision to open issue #16 instead of a pull request is supported by the telemetry:

• Paper: Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping (arXiv:2606.23682v1).
• Outcome: issue_opened_preflight, resulting in artifact smellslikeml/diffusers#16.
• Reasoning: The paper was classified as no-code-link. The absence of a reference implementation prevented a direct pull request and surfaced the need for the engineering investigation captured in the issue. No other papers were considered or rejected.
• Target Call Site: Not specified in the run telemetry. The resulting issue Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping #16 proposes targeting IP-Adapter attention processors.
• Diff-Risk: Not calculated for ISSUE decisions.
• Cost: $0.0351.