Diffusion Models for Video Generation

🌈 Abstract

The article discusses the recent progress in using diffusion models for video generation, which is a more challenging task compared to image generation. It covers various approaches, including designing and training diffusion video models from scratch, as well as adapting pre-trained image-to-text diffusion models to generate videos. The article also discusses techniques for maintaining temporal consistency and improving video quality, such as using 3D U-Net architectures, incorporating spatiotemporal attention, and leveraging pre-trained image models.

🙋 Q&A

[01] Parameterization & Sampling Basics

1. What are the key differences in the variable definitions and parameterization compared to the previous post on image generation?

• The article uses a slightly different variable definition, but the underlying math stays the same.
• It introduces the concept of a sequence of noisy variations of the data point $\mathbf{x}$, denoted as ${\mathbf{z}_t \mid t =1 \dots, T}$, with increasing amount of noise as $t$ increases.
• It discusses the $\mathbf{v}$-parameterization, which has been shown to be helpful for avoiding color shift in video generation compared to $\boldsymbol{\epsilon}$-parameterization.

2. How does the DDIM update rule need to be adjusted for video generation?

• For video generation, the diffusion model needs to be able to sample a second video $\mathbf{x}^b$ conditioned on the first $\mathbf{x}^a$, i.e., $\mathbf{x}^b \sim p_\theta(\mathbf{x}^b \vert \mathbf{x}^a)$.
• The article proposes the reconstruction guidance method, which uses an adjusted denoising model to properly condition the sampling of $\mathbf{x}^b$ on $\mathbf{x}^a$.

[02] Model Architecture: 3D U-Net & DiT

1. How do the 3D U-Net and DiT architectures differ from the standard 2D U-Net used for image generation?

• The 3D U-Net extends the 2D U-Net to work for 3D data, where each feature map represents a 4D tensor of frames x height x width x channels.
• The 3D U-Net is factorized over space and time, with separate processing for the spatial and temporal dimensions.
• The DiT (Diffusion Transformer) architecture operates on spacetime patches of video and image latent codes, representing the visual input as a sequence of spacetime patches.

2. What are the key components of the Imagen Video architecture?

• Imagen Video consists of a base video diffusion model and a cascade of interleaved spatial and temporal super-resolution diffusion models.
• The base denoising model performs spatial operations over all the frames with shared parameters, and then a temporal layer mixes activations across frames.
• The super-resolution models condition on the upsampled inputs concatenated with noisy data $\mathbf{z}_t$ channel-wise.

[03] Adapting Image Models to Generate Videos

1. How do the "inflation" approaches, such as Make-A-Video and Tune-A-Video, extend pre-trained image-to-text diffusion models to generate videos?

• They add spatiotemporal convolution and attention layers to extend the network to cover the temporal dimension.
• The new temporal layers are either fine-tuned on unlabeled video data (Make-A-Video) or remain frozen during training (Tune-A-Video).
• Tune-A-Video also incorporates a spatiotemporal attention (ST-Attention) block to capture temporal consistency.

2. What are the key challenges and solutions in the Video LDM and Stable Video Diffusion approaches?

• The main challenge is that the pre-trained autoencoder in LDM only sees images, not videos, which can cause flickering artifacts without good temporal coherence.
• Video LDM and Stable Video Diffusion address this by adding additional temporal layers in the decoder and fine-tuning them on video data, while keeping the encoder frozen to reuse the pre-trained LDM.
• Stable Video Diffusion also emphasizes the importance of dataset curation to improve model performance.

[04] Training-Free Adaptation

1. How does Text2Video-Zero enable zero-shot, training-free video generation?

• Text2Video-Zero enhances a pre-trained image diffusion model with two key mechanisms: 1) sampling the sequence of latent codes with motion dynamics to keep the global scene and background time consistent, and 2) reprogramming frame-level self-attention using a new cross-frame attention to preserve the context, appearance, and identity of the foreground object.

2. What are the key features of the ControlVideo approach?

• ControlVideo aims to generate videos conditioned on text prompt and a motion sequence (e.g., depth or edge maps).
• It introduces three new mechanisms: cross-frame attention, interleaved-frame smoother, and hierarchical sampler to improve temporal consistency and enable long video generation.