RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction
A CUDA implementation (100x faster) is now available:
Abstract
This paper introduces a versatile paradigm for integrating multi-view reflectance and normal maps acquired through photometric stereo. Our approach employs a pixel-wise joint re-parameterization of reflectance and normal, considering them as a vector of radiances rendered under simulated, varying illumination. This re-parameterization enables the seamless integration of reflectance and normal maps as input data in neural volume rendering-based 3D reconstruction while preserving a single optimization objective. In contrast, recent multi-view photometric stereo (MVPS) methods depend on multiple, potentially conflicting objectives. Despite its apparent simplicity, our proposed approach outperforms state-of-the-art approaches in MVPS benchmarks across F-score, Chamfer distance, and mean angular error metrics. Notably, it significantly improves the detailed 3D reconstruction of areas with high curvature or low visibility.
Overview
The reflectance and normal maps provided for each view by photometric stereo (PS) are fused, by combining volume rendering with a pixel-wise re-parameterization of the inputs using physically-based rendering.
Results
Comparisons on DiLiGenT-MV
Qualitative comparisons with the MVPS methods Kaya22, PS-NeRF, MVPSNet and Kaya23.
NeuS (mono-illumination) vs Ours (multi-illumination) from sparse viewpoints
Mono-illumination NeuS requires a high number of viewpoints and exhibits a drastic decline in reconstruction quality with a limited number of viewpoints. Conversely, our method demonstrates stable results, benefiting from the rich reflectance and normal maps information obtained through a multi-illumination acquisition setup.
Relighting with reflectance maps
We provide additional results demonstrating relighting capabilities. The videos show results under varying lighting conditions, with the reflectance, roughness and metalness from SDM-UniPS used to create UV maps aligned with our 3D reconstruction results.
Citation
title={RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction},
author={Baptiste Brument and Robin Bruneau and Yvain Quéau and Jean Mélou and François Lauze and Jean-Denis Durou and Lilian Calvet},
booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024}
}
Acknowledgements
Baptiste Brument's doctoral student fellowship is funded by the French Ministry of Higher Education and Research. Robin Bruneau's doctoral student fellowship is funded by the Danish project PHYLORAMA. This work was partly funded by the French National Research Agency through the LabCom project ALICIA-Vision. Lilian Calvet's posdoctoral fellowship is supported by the OR-X - a swiss national research infrastructure for translational surgery - and associated funding by the University of Zürich and University Hospital Balgrist.