Is your feature request related to a problem? Please describe.
My understanding is that diffusers does not currently support original checkpoint-compatible PEFT adapters aside from LoRAs.
Describe the solution you'd like.
I think diffusers should support loading general peft-compatible checkpoints which are compatible with the original checkpoint structure.
In particular, my proposal is that it could be supported by the following strategy:
- Make the
diffusers checkpoint and PEFT checkpoint weights coincide.
- Inject the checkpoint normally using
peft.
See #13861 (comment) for a slightly more fleshed out version of the above.
Describe alternatives you've considered.
diffusers supports LoRA checkpoints by converting the checkpoints to the diffusers format. However, differences between diffusers checkpoints and potential LoRA checkpoints, most notably split Q,K,V projections in diffusers vs fused QKV projections in LoRA checkpoints, means that the adapter checkpoint weights need to be modified to make this possible. In general, if weight shape differences are present, each new PEFT technique (e.g. DoRA, IA3, etc.) would need their own technique-specific conversion logic. So, in particular, for (1) in the proposal above we should make the diffusers checkpoint match the original checkpoint in terms of weight shapes (for example, by fusing the QKV projections) to avoid the need for conversion logic for each PEFT technique.
Additional context.
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
My understanding is that
diffusersdoes not currently support original checkpoint-compatible PEFT adapters aside from LoRAs.Describe the solution you'd like.
I think
diffusersshould support loading generalpeft-compatible checkpoints which are compatible with the original checkpoint structure.In particular, my proposal is that it could be supported by the following strategy:
diffuserscheckpoint and PEFT checkpoint weights coincide.peft.See #13861 (comment) for a slightly more fleshed out version of the above.
Describe alternatives you've considered.
diffuserssupports LoRA checkpoints by converting the checkpoints to thediffusersformat. However, differences betweendiffuserscheckpoints and potential LoRA checkpoints, most notably split Q,K,V projections indiffusersvs fused QKV projections in LoRA checkpoints, means that the adapter checkpoint weights need to be modified to make this possible. In general, if weight shape differences are present, each new PEFT technique (e.g. DoRA, IA3, etc.) would need their own technique-specific conversion logic. So, in particular, for (1) in the proposal above we should make thediffuserscheckpoint match the original checkpoint in terms of weight shapes (for example, by fusing the QKV projections) to avoid the need for conversion logic for each PEFT technique.Additional context.
Add any other context or screenshots about the feature request here.