Weights & Biases

🌈 Abstract

The article discusses the process of fine-tuning an open-source Text2Video model, specifically the Open-Sora 1.1 Stage3 model, to create stop motion animations. It covers the technical steps involved, the hardware and software used, the dataset creation, and the fine-tuning process. The article also highlights the challenges and areas for improvement in the current model, such as temporal consistency in longer sequences, noise in condition-free generation, and the need for higher resolution and longer sequences.

🙋 Q&A

[01] Fine-tuning an Open-Source Text2Video Model

1. What are the key steps involved in fine-tuning the Open-Sora 1.1 Stage3 model to create stop motion animations?

• The article outlines the following key steps:
  • Obtaining the open-source code (the authors' fork of Open-Sora), dataset, and pre-trained models (32f and 64f)
  • Setting up the training infrastructure, which includes a 32-GPU Lambda 1-Click Cluster with NVIDIA HGX H100 servers and NVIDIA Quantum-2 400 Gb/s InfiniBand networking
  • Preparing the software environment, including NVIDIA CUDA, NCCL, PyTorch, Transformers, Diffusers, Flash-Attention, and NVIDIA Apex
  • Curating the dataset, which consists of high-quality stop motion videos from various YouTube channels, and annotating the video clips using GPT-4
  • Fine-tuning the pre-trained Open-Sora-STDiT-v2-stage3 model on the BrickFilm dataset

2. What are the hardware and software specifications of the training infrastructure used in this project?

• The training infrastructure is a 32-GPU Lambda 1-Click Cluster with the following key specifications:
  • 4 NVIDIA HGX H100 servers, each with 8 NVIDIA H100 SXM Tensor Core GPUs
  • NVIDIA Quantum-2 400 Gb/s InfiniBand networking, providing a node-to-node bandwidth of 3200 Gb/s
  • NVIDIA Driver pre-installed, and a custom Conda environment for managing dependencies (CUDA, NCCL, PyTorch, Transformers, Diffusers, Flash-Attention, NVIDIA Apex)
• This setup enables a training throughput of 97,200 video clips per hour (360p and 32 frames per video).

3. How was the dataset for fine-tuning the model created?

• The dataset was curated from various YouTube channels, including MICHAELHICKOXFilms, LEGO Land, FK Films, and LEGOSTOP Films, which feature high-quality stop motion animations created with LEGO® bricks.
• The videos were processed into 15-200 frame clips and annotated using a vision language model (GPT-4) with a specific prompt.
• The dataset also includes static images, which were extracted as the middle frames of the video clips, to help the model learn object appearance in finer details.

4. What were the key challenges and areas for improvement identified in the current model?

• Temporal consistency in longer sequences: The model's attention mechanism, which processes spatial and temporal dimensions separately, may limit the context window and lead to drifting in generated videos.
• Noise in condition-free generation: The model's learning of brick animation representations can still be improved, potentially by expanding the dataset and finding more efficient ways for the model to learn the representations.
• Resolution and frame count: Pushing the output beyond 360p and 64 frames would enhance the utility and applicability of the model.
• Dataset quality and quantity: Both the quality and the quantity of the dataset can be improved, and the authors mention plans for future dataset releases.

[02] Fine-Tuning Process and Results

1. What were the key results of the fine-tuning process?

• The authors released two fine-tuned models: text2bricks-360p-64f (1017.6 H100 hours of training) and text2bricks-360p-32f (169.6 H100 hours of training).
• The fine-tuning process did not result in a decrease in the loss, but the validation results showed that the quality of the generated images gradually improved, indicating that the model was enhancing its performance in ways not directly reflected by the loss value.
• The authors observed very low CPU usage during the fine-tuning, with the GPUs consistently running at full capacity, highlighting the importance of efficient scaling in training foundational models.

2. What were the key observations made during the fine-tuning process?

• The authors fixed the random seed to ensure apple-to-apple comparisons during the fine-tuning stages, and they provided examples of the model's outputs at different stages of the fine-tuning process.
• They observed that while the loss did not decrease, the quality of the generated images gradually improved, suggesting that the model was enhancing its performance in ways not directly reflected by the loss value.
• The authors also noted the benefits of the Lambda 1-Click Cluster's efficient scaling, with the GPUs running at full capacity during the fine-tuning process.

</output_format>