DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos

🌈 Abstract

The paper presents DepthCrafter, an innovative method for generating temporally consistent long depth sequences with intricate details for open-world videos, without requiring any supplementary information such as camera poses or optical flow. The key contributions are:

• DepthCrafter leverages video diffusion models to generate high-quality video depth, while maintaining generalization ability to diverse open-world videos.
• A three-stage training strategy is proposed to enable generating depth sequences with long and variable temporal context, up to 110 frames, and harvest both precise depth details and rich content diversity.
• An inference strategy is designed to segment-wisely process videos beyond 110 frames and seamlessly stitch them together, enabling depth estimation for extremely long videos.

🙋 Q&A

[01] Introduction

1. What are the key challenges in open-world video depth estimation that the paper aims to address?

• The inherent ambiguity in depth estimation from a single view
• Temporal inconsistency or flickering when directly applying static image depth estimation methods to videos
• The diversity of open-world videos in content, motion, camera movement, and length, making existing video depth estimation methods hard to perform well in practice
• The requirement of additional information like camera poses or optical flow, which are often non-trivial to obtain in open-world videos

2. How does the paper propose to tackle these challenges?

• Formulating video depth estimation as a conditional diffusion generation problem to leverage the strong capability of diffusion models in generating various types of videos
• Designing a three-stage training strategy to enable generating depth sequences with long and variable temporal context, while harvesting both precise depth details and rich content diversity
• Crafting an inference strategy to segment-wisely process extremely long videos and seamlessly stitch the depth sequences together

[02] Method

1. What is the key idea behind DepthCrafter? The key idea is to leverage video diffusion models for video depth estimation, while maintaining the generalization ability to open-world videos. This is achieved through:

• Formulating video depth estimation as a conditional diffusion generation problem
• Training a video-to-depth model from a pre-trained image-to-video diffusion model
• Employing a three-stage training strategy with compiled paired video-depth datasets
• Designing an inference strategy to process extremely long videos

2. How does the three-stage training strategy work? The three-stage training strategy is designed to:

1. Adapt the pre-trained image-to-video diffusion model to the video-to-depth generation task using a large realistic dataset with variable sequence lengths.
2. Fine-tune the temporal layers of the model on the large realistic dataset with longer sequence lengths up to 110 frames, to enable accurate arrangement of the entire depth distributions.
3. Fine-tune the spatial layers of the model on a small synthetic dataset with precise depth annotations, to learn more detailed depth information.

3. What is the purpose of the inference strategy? The inference strategy is designed to enable DepthCrafter to estimate depth sequences for extremely long videos in the open world, beyond the 110-frame limit of the training. It does this by:

• Dividing the input video into overlapped segments
• Estimating depth sequences for each segment independently
• Stitching the segments together seamlessly using a mortise-and-tenon style latent interpolation

[03] Experiments

1. What are the key findings from the quantitative evaluation?

• DepthCrafter achieves state-of-the-art performance in video depth estimation on multiple datasets, including Sintel, Scannet, Bonn, and KITTI.
• Compared to previous methods, DepthCrafter shows significant improvements, e.g., 10% improvement in the metric on the Sintel dataset.
• DepthCrafter also performs competitively in single-image depth estimation on the NYU-v2 dataset.

2. How does the qualitative evaluation demonstrate the effectiveness of DepthCrafter? The qualitative results show that DepthCrafter can produce temporally consistent depth sequences with fine-grained details across various open-world videos, including human actions, animals, architectures, cartoons, and games. In contrast, existing methods exhibit flickering artifacts in the temporal profiles of the depth sequences.

3. What are the key findings from the ablation studies?

• The three-stage training strategy is effective, with the performance improving as the training progresses.
• The inference strategy, including the initialization of overlapped latents and the latent stitching, is crucial for maintaining temporal consistency in the estimated depth sequences.

[04] Applications

1. What are some of the downstream applications facilitated by DepthCrafter? DepthCrafter can facilitate various downstream applications, such as:

• Foreground matting
• Depth slicing
• Fog effects
• Depth-conditioned video generation

These applications rely heavily on the accuracy and consistency of the video depth, which DepthCrafter is able to provide.

2. How do the example results demonstrate the usefulness of DepthCrafter in these applications? The example results show that DepthCrafter can generate temporally consistent depth sequences with fine-grained details, enabling realistic depth-based visual effects like fog simulation. Additionally, the depth maps can be used as effective structural conditions for depth-conditioned video generation, demonstrating the wide applicability of DepthCrafter.