SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement

🌈 Abstract

The article presents SF3D, a novel method for rapid and high-quality textured object mesh reconstruction from a single image in just 0.5 seconds. SF3D addresses several issues in existing fast 3D reconstruction models, including:

• Light bake-in: SF3D decomposes the illumination and reflective properties using a differentiable shading model to remove baked-in lighting effects.
• Vertex Coloring: SF3D uses a fast box projection-based UV unwrapping technique to generate low-poly meshes with high-resolution textures, instead of relying on vertex coloring.
• Marching Cubes Artifacts: SF3D uses an efficient architecture and DMTet mesh extraction with learned vertex displacements and normal maps to produce smoother mesh surfaces.
• Lack of Material Properties: SF3D predicts non-spatially varying material properties like metallic and roughness to enhance the visual quality of the reconstructed 3D meshes.

The article demonstrates that SF3D outperforms existing and concurrent baselines in both speed and quality of the reconstructed 3D assets.

🙋 Q&A

[01] Enhanced Transformer

1. How does SF3D's enhanced transformer architecture address the issue of aliasing artifacts? SF3D's enhanced transformer architecture addresses the aliasing artifacts by:

• Producing triplanes at a higher resolution of 256x256 with 1024 channels, compared to the lower 128x128 resolution used in prior works.
• Leveraging a two-stream transformer design inspired by PointInfinity, which has linear complexity with respect to the number of tokens. This allows the model to process the higher resolution triplanes without prohibitive computational cost.
• Integrating a pixel shuffling operation to further increase the triplane resolution to 512x512 with 40 feature channels, further reducing aliasing artifacts.

[02] Material Estimation

1. How does SF3D's material estimation module work? SF3D's material estimation module, called the "Material Net", predicts the metallic and roughness parameters of the object's material. To address the challenge of estimating spatially varying materials, which is an inherently ambiguous task, SF3D simplifies the problem by predicting a single set of metallic and roughness values for the entire object.

The Material Net first passes the input image through a frozen CLIP image encoder to extract semantic latents. These latents are then passed through two separate MLPs to predict the parameters of Beta distributions for the metallic and roughness values. During training, the network is optimized to maximize the log-likelihood of the predicted distributions, which helps stabilize the training and prevent collapse to default material values.

During inference, the mode of the predicted distributions is used as the final metallic and roughness values for the object.

[03] Illumination Modeling

1. How does SF3D's illumination modeling component work? SF3D's illumination modeling component, called the "Light Net", aims to explicitly estimate the illumination in the input image to avoid baking lighting effects into the object's texture.

The Light Net takes the high-resolution triplanes from the enhanced transformer as input and passes them through a series of CNN layers and an MLP to predict the grayscale amplitude values for 24 spherical Gaussian (SG) lights. These SG lights are used to implement a deferred physically-based rendering approach, similar to NeRD, to remove the low-frequency illumination effects from the object's appearance.

Additionally, SF3D incorporates a lighting demodulation loss during training, which enforces consistency between the learned illumination and the luminance observed in the training data. This helps resolve the ambiguity between the object's appearance and the lighting conditions.

[04] Mesh Extraction and Refinement

1. How does SF3D's mesh extraction and refinement process work? SF3D uses a differentiable Marching Tetrahedron (DMTet) technique to convert the estimated triplanes into a mesh. To address the staircase artifacts often seen with Marching Cubes, SF3D introduces two additional MLP heads:

1. Vertex Offset Prediction: This MLP predicts vertex offsets that can reduce artifacts from the tetrahedral grid.
2. World Space Normal Prediction: This MLP predicts world space vertex normals, which can add details to the flat mesh triangles.

To stabilize the training of the normal prediction, SF3D uses a spherical linear interpolation (slerp) between the geometry normals and the predicted normals during the initial training steps.

SF3D also employs several loss functions to regularize the mesh estimation, including normal consistency, Laplacian smoothness, and vertex offset regularization losses.

[05] Fast UV-Unwrapping and Export

1. How does SF3D's fast UV-unwrapping process work? To address the computational inefficiency of traditional UV-unwrapping methods, which can take several seconds, SF3D proposes a fast, parallelizable cube projection-based unwrapping approach:

1. The output mesh is first aligned with the cube projection coordinate system based on the most dominant axes.
2. Each mesh face independently selects the appropriate cube face to project onto, based on its surface normal.
3. To address potential occlusions where different faces share the same UV coordinates, SF3D performs 2D triangle-triangle intersection tests and reorganizes the overlapping triangles in the UV atlas.
4. The world positions and occupancy data are then baked into the final UV atlas, allowing for efficient querying of the albedo and normal maps.
5. Margin regions are added to the UV atlas to prevent visible seams at the UV island borders.

This fast UV-unwrapping process, combined with the baking of world positions and occupancy, enables SF3D to generate the final textured 3D mesh in under 0.5 seconds.

[06] Overall Training and Loss Functions

1. How does SF3D's training process work? SF3D's training process consists of three main stages:

1. Pre-training on the NeRF task: SF3D is first pre-trained on the NeRF task, using image-based metrics like MSE and LPIPS to compare the rendered and shaded reconstructions with the ground truth.
2. Transition to mesh training: After the pre-training, SF3D transitions to mesh training, replacing the NeRF rendering with differentiable mesh rendering and SG-based shading.
3. Final stage training: In the final stage, SF3D trains for 80K steps at a higher resolution of 512x512, with a batch size of 96.

The overall loss function combines the image-based metrics (MSE and LPIPS), a mask loss, and separate loss terms for the rendering, mesh regularization, and shading components.

Additionally, SF3D uses larger batch sizes during the light estimation training to aid convergence, as the introduction of the light estimation component was found to benefit from larger batch sizes.

</output_format>