A Unified Framework for Surface Reconstruction

Zehao Yu1     Anpei Chen1,2     Bozidar Antic1     Songyou Peng2,3     Apratim Bhattacharyya1     
Michael Niemeyer1,3      Siyu Tang2      Torsten Sattler4      Andreas Geiger1,3
1University of Tübingen     2ETH Zurich     3MPI for Intelligent Systems, Tübingen 4Czech Technical University in Prague

TL;DR: We provide a unified framework and benchmark for neural implicit surface reconstruction.

About

SDFStudio is a unified and modular framework for neural implicit surface reconstruction, built on top of the awesome nerfstudio project. We provide a unified implementation of three major implicit surface reconstruction methods: UniSurf, VolSDF, and NeuS. SDFStudio also supports various scene representions, such as MLPs, Tri-plane, and Multi-res. feature grids, and multiple point sampling strategies such as surface-guided sampling as in UniSurf, and Voxel-surface guided sampling from NeuralReconW. It further integrates recent advances in the area such as the utillization of monocular cues (MonoSDF), geometry regularization (UniSurf) and multi-view consistency (Geo-NeuS). Thanks to the unified and modular implementation, SDFStudio makes it easy to transfer ideas from one method to another. For example, Mono-NeuS applies the idea from MonoSDF to NeuS, and Geo-VolSDF applies the idea from Geo-NeuS to VolSDF.

Reconstruction Results

NeuS-facto

In this example, we use a proposal network for sampling (inspired by nerfacto and mip-NeRF 360), a big MLP (8 layers with 512 hidden dimension) as scene representations, and train the model on 8 A100 GPUs for ~10 hours on the Heritage dataset.

UniSurf, VolSDF, and NeuS

Reconstruction results of UniSurf, VolSDF and NeuS using a Multi-Res. Feature Grid representation on the DTU datasets.

Geo-UniSurf, Geo-VolSDF, and Geo-NeuS

Geo-NeuS improves reconstruction quality by enforcing multi-view photometric consistency during optimization. Thanks to our unified framework, we can apply the idea from Geo-NeuS to UniSurf and VolSDF.

MonoSDF

Reconstruction result of MonoSDF on the Tanks and Temples dataset Courtroom scene.

Mono-NeuS

We can also apply the idea of MonoSDF to NeuS, as shown here on the Tanks and Temples dataset Auditorium scene:

NeuS-Acc

NeuS with empty space skipping based on nerfacc with monocular prior from MonoSDF, trained for ~17 minutes.

NeuS-RGBD

SDFStudio also supports RGB-D data to obtain high-quality 3D reconstruction. Here for example, we train NeuS on the synthetic rgbd dataset from neural rgbd surface reconstruction:

Viewer

Nerfstudio's viewer can be used for interactive visualization of rendered RGB images, depths and normals.

BibTeX

@misc{Yu2022SDFStudio,
    author    = {Zehao Yu, Anpei Chen, Bozidar Antic, Songyou Peng, Apratim Bhattacharyya,
                 Michael Niemeyer, Siyu Tang, Torsten Sattler, and Andreas Geiger},
    title     = {SDFStudio: A Unified Framework for Surface Reconstruction},
    year      = {2022},
    url       = {https://github.com/autonomousvision/sdfstudio},
}