Nvidia AI Research Unveils ‘Align Your Gaussians’ Approach for Expressive Text-to-4D Synthesis

Creating dynamic 3D scenes through generative modeling holds significant promise for transforming how we develop games, movies, simulations, animations, and virtual environments. Although score distillation techniques are proficient at generating diverse 3D objects, they often focus on static scenes, overlooking the dynamic nature of real-world experiences. Unlike image diffusion models, which have successfully been adapted for video generation, more research needs to extend 3D synthesis to encompass 4D generation, incorporating an additional temporal dimension to capture the essence of motion and change in surroundings.

A team of researchers from NVIDIA, Vector Institute, University of Toronto, and MIT have proposed Align Your Gaussians (AYG), which utilizes dynamic 3D Gaussian Splatting with deformation fields as a 4D representation. AYG introduces an approach to regulate the distribution of moving 3D Gaussians, enhancing optimization stability and inducing realistic motion. The method includes a motion amplification mechanism and an innovative autoregressive synthesis scheme for generating and combining multiple 4D sequences, enabling longer and more realistic scene generation. These techniques facilitate the synthesis of vibrant, dynamic scenes, achieving cutting-edge text-to-4D performance. The Gaussian 4D representation allows seamless blending of different 4D animations.

3D Gaussian Splatting represents 3D scenes with N 3D Gaussians, including positions, covariances, opacities, and colors.  Diffusion-based generative models (DMs) are used for score distillation-based generation of 3D objects, such as neural radiance fields (NeRF) or 3D Gaussians. A text-guided multiview diffusion model and a regular text-to-image model are used for synthesizing a static 3D scene. The researchers conducted human evaluations and user studies to assess the quality of their generated 4D scenes, comparing them with MAV3D and performing ablation studies.

AYG is a method for text-to-4D synthesis using dynamic 3D Gaussians and composed diffusion models. The researchers utilize a diligent 4D scene representation, where multiple dynamic 4D objects are composed within a large dynamic scene. AYG incorporates a main 4D stage that involves updating the deformation field using a gradient-based approach. Prompts generate specific 4D scenes, such as “A bulldog is running fast” and “A panda is boxing and punching.” The researchers also mention using a newly trained latent video diffusion model for generating 2D video samples with different fps conditionings.

The study showcases additional dynamic 4D scene samples generated from AYG, demonstrating the effectiveness of their approach. The researchers refer readers to their supplementary video, which showcases almost all their active 4D scene samples. AYG’s newly trained latent video diffusion model is used to generate videos for this work, further highlighting the capabilities of their method. AYG’s dynamic scene generation capabilities can be utilized in synthetic data generation, enabling the creation of realistic and diverse training datasets for various applications.

In conclusion, AYG, an advanced technology for expressive text-to-4D synthesis, leverages dynamic 3D Gaussian Splatting with deformation fields and incorporates score distillation through multiple composed diffusion models. Its innovative regularization and guidance techniques have enabled cutting-edge results in dynamic scene generation. AYG stands out for its capability to demonstrate temporally extended 4D synthesis and compose multiple dynamic objects within a larger scene. The technology has diverse applications in creative content creation and synthetic data generation. For instance, AYG facilitates the synthesis of videos and 4D sequences with precise tracking labels, which is beneficial for training discriminative models.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..