Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models

🌈 Abstract

The paper presents a latent diffusion model for generating 3D scenes from 2D image data. The key ideas are:

• Learning an autoencoder that maps multi-view images to a compressed 3D representation using Gaussian splats
• Training a diffusion model on the latent space to learn an efficient generative model of 3D scenes
• This approach enables fast generation of diverse and high-quality 3D scenes, either from scratch, a single input view, or sparse input views.

🙋 Q&A

[01] Autoencoder

1. Questions related to the content of the section?

• How does the autoencoder map multi-view images to a compressed 3D representation using Gaussian splats?
• What are the key components of the autoencoder architecture?

The autoencoder first encodes the multi-view input images into a low-dimensional latent representation. This is done by passing the images through a multi-view U-Net that allows information exchange between the different views. The final layer of the encoder outputs the mean and log-variance of a Gaussian posterior distribution over the latent space.

The latent features are then decoded to a 3D scene represented as Gaussian splats. Specifically, the latent features are passed through upsampling residual blocks to predict the depth, opacity, color, rotation and scale of splats for each pixel. The 3D position of each splat is then calculated by unprojecting it along the corresponding camera ray using the predicted depth.

The autoencoder is trained as a variational autoencoder, minimizing a reconstruction loss between the rendered splats and the input images, as well as a KL divergence loss to encourage the latent space to match a standard Gaussian prior.

2. Why is conditioning the autoencoder on camera poses important? Conditioning the autoencoder on the relative poses of the input views is important to allow the model to learn the arbitrary scale of the 3D scenes. Without this conditioning, the autoencoder would not be able to resolve the perspective depth/scale ambiguity.

[02] Denoiser

1. Questions related to the content of the section?

• How is the denoising diffusion model trained and used for generation?
• What conditioning signals does the denoiser support?

The denoiser is a multi-view U-Net architecture that is trained using a denoising diffusion probabilistic model (DDPM) on the low-dimensional latent features produced by the autoencoder.

The denoiser is trained jointly for class-conditional and image-conditional generation. For class conditioning, a learned class embedding is added to the timestep embedding. For image conditioning, the conditioning images are encoded by the pretrained autoencoder encoder and concatenated with the noisy latents.

During sampling, the denoiser takes in Gaussian noise in the latent space and uses DDIM sampling to progressively denoise it, either unconditionally, or conditioned on a class label or input image. The final denoised latents are then decoded back to 3D splats using the autoencoder decoder.

2. Why is operating in the latent space beneficial compared to directly denoising the 3D representation? Operating in the latent space is beneficial because it allows the denoiser to work in a much lower-dimensional space compared to directly denoising the full 3D splat representation. This makes the denoising process much more efficient, enabling fast sampling of diverse 3D scenes.

[03] Experiments

1. Questions related to the content of the section?

• How do the authors evaluate their model's performance on generation and reconstruction tasks?
• How does the proposed model compare to the baselines in terms of quantitative and qualitative results?

The authors evaluate their model on both unconditional generation and single-view/sparse-view 3D reconstruction tasks, using the MVImgNet and RealEstate10K datasets.

For unconditional generation, they compare to GIBR, ViewSet Diffusion and RenderDiffusion on the chair, table and sofa classes of MVImgNet. Their model significantly outperforms the baselines in terms of FID score, while also being much faster at sampling.

For single-view 3D reconstruction, they compare to SplatterImage and other baselines. Their model produces more accurate and perceptually better reconstructions according to PSNR and LPIPS metrics, while being an order of magnitude faster than the baselines.

The authors also analyze how the quality of generated scenes varies between the 'denoised' views (used to train the autoencoder) and 'held-out' views. They find only minimal differences, indicating their approach does not overly favor the denoised viewpoints.

2. What are the key benefits of the proposed latent diffusion approach compared to prior work? The key benefits are:

1. Efficiency - the latent diffusion model can sample 3D scenes in just 0.2s, over an order of magnitude faster than prior 3D-aware diffusion models.
2. Quality - the model produces diverse and high-quality 3D scenes, outperforming baselines on both generation and reconstruction metrics.
3. Generality - the model can handle arbitrary scenes, not just pre-segmented or object-centric ones, and supports various tasks like unconditional generation, single-image reconstruction, and sparse-view reconstruction.
4. Simplicity - the model only requires posed multi-view images for training, without any additional 2D/3D supervision.