Researchers from Tsinghua University and Harvard University introduces LangSplat: A 3D Gaussian Splatting-based AI Method for 3D Language Fields

In human-computer interaction, the need to create ways for users to communicate with 3D environments has become increasingly important. This field of open-ended language queries in 3D has attracted researchers due to its various applications in robotic navigation and manipulation, 3D semantic understanding, and editing. However, current approaches have limitations of slow processing speeds and limited accuracy.

Consequently, a team of researchers from Tsinghua University and Harvard University has developed a method called LangSplat. The researchers used traditional 3D Gaussian Splatting techniques instead of Neural Radiance Fields (NeRF). It first constructs a 3D language field to produce precise and efficient open-vocabulary queries within three-dimensional spaces. Also, each of these is assigned a unique language embedding. This technique uses a tile-based splatting technique for feature rendering. The exceptional part of LangSplat is that it can generate accurate language features without undergoing computationally expensive processes. To ensure consistent representation across different viewpoints, the researchers used supervision via CLIP embeddings derived from image patches captured from assorted training perspectives.

The researchers further tried reducing memory usage and rendering efficiency using a scene-wise language autoencoder. It compresses high-dimensional CLIP embeddings into a lower-dimensional latent space before generating final language embeddings during decoding. Therefore, memory needs are decreased by LangSplat by avoiding the direct learning of CLIP embeddings. Then, the displayed features are decoded to get the final language embeddings. 

Also, the researchers tried to solve the problem of point ambiguities, which are often encountered in complex scenes. To do this, the researchers used the semantic hierarchy of the Segment Anything Model (SAM) outline. They emphasized that they used SAM as it enabled LangSplat to assign precise CLIP embeddings to individual points in the environment, and, therefore, it helps increase model accuracy. Moreover, SAM-based masks allowed the researchers to query directly at specific semantic levels. This helped tackle the need for extensive searches across numerous absolute scales and additional DINO features.

The researchers performed experiments to evaluate the efficiency of LangSplat. The evaluation showed that LangSplat is superior to other state-of-the-art solutions like LERF. They also noticed that LangSplat has a 199x boost in processing speed and has enhanced performance in open-ended 3D language query tasks. Further, LangSplat has faster rendering speeds and has improved precision compared to previous models.

In conclusion, LangSplat is a significant step in developing 3D language fields. It addresses the limitations of previous models through the innovative use of 3D Gaussian Splatting, a scene-wise language autoencoder, and SAM-based masks. Also, as the researchers focus on further the accuracy and speed of this framework, LangSplat can reshape how to interact with and query information in three-dimensional spaces.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our Telegram Channel