The AI Cousin of Michelangelo: Neuralangelo is an AI Model That can Achieve High-Fidelity 3D Surface Reconstruction [Code Included]

Neural networks have advanced quite significantly in recent years, and they have found themselves a use case in almost all applications. One of the most interesting use cases is the 3D modeling of the real world. We have seen neural radiance fields (NeRFs) that can accurately capture the 3D geometry of a scene by using normal, daily cameras. These advancements opened a whole new page in 3D surface reconstruction.

The goal of 3D surface reconstruction is to recover detailed geometric structures of a scene by analyzing multiple images captured from various viewpoints. These reconstructed surfaces contain valuable structural information that can be applied to various applications, including generating 3D assets for augmented/virtual/mixed reality and mapping environments for autonomous robotic navigation. A particularly intriguing approach is a photogrammetric surface reconstruction using a single RGB camera, as it enables users to easily create digital replicas of the real world using common mobile devices.

3D surface reconstruction plays a crucial role in generating dense geometric structures from multiple images, enabling a wide range of applications such as augmented/virtual/mixed reality and robotics. While classical methods, like multi-view stereo algorithms, have been popular for sparse 3D reconstruction, they often struggle with ambiguous observations and produce inaccurate or incomplete results. Neural surface reconstruction methods have emerged as a promising solution by leveraging coordinate-based multi-layer perceptrons (MLPs) to represent scenes as implicit functions. However, the fidelity of current methods does not scale well with MLP capacity.

What if we could have a method that solved the scaling problem? What if we could really accurately generate 3D surface models by just using RGB inputs? Time to meet Neuralangelo. 

Neuralangelo is a framework that combines the power of Instant NGP (Neural Graphics Primitives) and neural SDF representation to achieve high-fidelity surface reconstruction.

Neuralangelo adopts Instant NGP as a neural Signed Distance Function (SDF) representation of the underlying 3D scene. Instant NGP introduces a hybrid 3D grid structure with a multi-resolution hash encoding, along with a lightweight MLP that enhances expressiveness while maintaining a log-linear memory footprint. This hybrid representation significantly improves the representation power of neural fields and excels in capturing fine-grained details.

To further enhance the quality of hash-encoded surface reconstruction, Neuralangelo introduces two key techniques. Firstly, numerical gradients are employed to compute higher-order derivatives, such as surface normals, which contribute to stabilizing the optimization process. Secondly, a progressive optimization schedule is implemented to recover structures at different levels of detail, enabling a comprehensive reconstruction approach. These techniques work in synergy, leading to substantial improvements in both reconstruction accuracy and view synthesis quality.

Neuralangelo naturally incorporates the power of multi-resolution hash encoding into neural SDF representations, resulting in enhanced reconstruction capabilities. Secondly, the use of numerical gradients and eikonal regularization helps improve the quality of hash-encoded surface reconstruction by stabilizing the optimization process. Finally, extensive experiments on standard benchmarks and real-world scenes demonstrate the effectiveness of Neuralangelo, showcasing significant improvements over previous image-based neural surface reconstruction methods in terms of reconstruction accuracy and view synthesis quality.

Check Out The Paper, Code, and Project. Don’t forget to join our 23k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at [email protected]

🚀 Check Out 100’s AI Tools in AI Tools Club