Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

🌈 Abstract

The paper proposes a novel sampling guidance method called Perturbed-Attention Guidance (PAG) that improves the quality of diffusion model samples in both conditional and unconditional settings, without requiring additional training or external modules. PAG leverages the self-attention mechanism in diffusion U-Net to generate undesirable samples by substituting the self-attention map with an identity matrix, and then guides the denoising process away from these degraded samples. The authors demonstrate that PAG significantly enhances sample quality in ADM and Stable Diffusion models, and also improves performance in various downstream tasks like inverse problems and ControlNet where existing guidance methods are not fully applicable.

🙋 Q&A

[01] Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

1. What is the key motivation behind the proposed Perturbed-Attention Guidance (PAG) method? The key motivation behind PAG is to improve the quality of diffusion model samples in both conditional and unconditional settings, without requiring additional training or external modules. Existing guidance techniques like classifier guidance (CG) and classifier-free guidance (CFG) have limitations, such as the need for additional training or reduced sample diversity.

2. How does PAG work? PAG leverages the self-attention mechanism in the diffusion U-Net to generate undesirable samples by substituting the self-attention map with an identity matrix. These undesirable samples are then used to guide the denoising process away from structural collapse that is commonly observed in unguided diffusion generation.

3. What are the key advantages of PAG compared to other guidance methods? The key advantages of PAG are:

• It improves sample quality in both conditional and unconditional settings without requiring additional training or external modules.
• It can be applied to both pixel-level diffusion models like ADM and latent diffusion models like Stable Diffusion.
• It significantly enhances performance in various downstream tasks like inverse problems and ControlNet where existing guidance methods are not fully applicable.

4. How does PAG relate to and differ from Classifier-Free Guidance (CFG)? PAG can be seen as a generalization of CFG, where the perturbation strategy is not limited to dropping the class label. PAG perturbs the self-attention map of the diffusion U-Net, which is different from the input perturbation used in CFG. This allows PAG to be applicable even in unconditional settings, unlike CFG which requires a class label.

[02] Related Work

1. What are the key advancements in diffusion models discussed in the paper? The paper discusses several advancements in diffusion models, including:

• Improvements in sampling speed through techniques like DDIM.
• The development of latent diffusion models like Stable Diffusion that operate in a compressed latent space.
• The integration of self-attention mechanisms into diffusion U-Net architectures.

2. What are the key sampling guidance techniques for diffusion models mentioned in the paper? The paper discusses the following key sampling guidance techniques:

• Classifier Guidance (CG) which uses a trained classifier to guide the sampling process.
• Classifier-Free Guidance (CFG) which models an implicit classifier without requiring an external module.
• Self-Attention Guidance (SAG) which uses adversarial blurring to guide the sampling process.

3. What is the role of self-attention mechanisms in diffusion models? The paper notes that self-attention mechanisms have been effectively integrated into diffusion U-Net architectures, and their rich representational capabilities have been leveraged across a variety of applications like personalization, image editing, and video generation.

[03] Perturbed-Attention Guidance (PAG)

1. How does PAG leverage the self-attention mechanism in diffusion U-Net? PAG leverages the ability of self-attention maps in diffusion U-Net to capture structural information. It generates undesirable samples by substituting the self-attention map with an identity matrix, and then guides the denoising process away from these degraded samples.

2. What is the key intuition behind perturbing the self-attention map? The key intuition is that perturbing the self-attention map, which is responsible for capturing structural information, can generate samples with collapsed structures. These undesirable samples then serve to steer the denoising trajectory away from structural collapse, which is commonly observed in unguided diffusion generation.

3. How does PAG relate to the concept of an implicit discriminator? PAG defines an implicit discriminator that distinguishes between desirable and undesirable samples during the diffusion process. The gradient of this implicit discriminator's loss function is then used to guide the denoising process towards the desirable distribution and away from the undesirable distribution.

4. How does PAG compare to Classifier-Free Guidance (CFG) in terms of its formulation? PAG's formulation can be seen as a generalization of CFG, where the perturbation strategy is not limited to dropping the class label. PAG perturbs the self-attention map, which is different from the input perturbation used in CFG. This allows PAG to be applicable even in unconditional settings, unlike CFG which requires a class label.

[04] Experiments

1. How does PAG perform compared to other guidance methods on pixel-level diffusion models like ADM? Experiments on the ADM model show that PAG significantly outperforms the baseline without guidance, as well as the Self-Attention Guidance (SAG) method, in terms of FID, IS, Precision, and Recall metrics for both conditional and unconditional generation.

2. How does PAG perform on latent diffusion models like Stable Diffusion? Experiments on Stable Diffusion show that PAG improves sample quality in both unconditional and text-to-image synthesis settings. Combining PAG with Classifier-Free Guidance (CFG) leads to further improvements in text-to-image synthesis.

3. How does PAG perform on downstream tasks like inverse problems and ControlNet? PAG significantly improves the performance of diffusion models on inverse problems like inpainting, super-resolution, and deblurring, where existing guidance methods like CFG cannot be fully utilized. PAG also enhances the quality of ControlNet samples in unconditional generation scenarios with sparse spatial control signals.

4. What are the key findings from the ablation studies on PAG? The ablation studies show that:

• Perturbing the self-attention map by replacing it with an identity matrix is more effective than other perturbation strategies like random masking or additive noise.
• Applying perturbations to deeper layers of the diffusion U-Net generally yields better results than shallower layers.