Meta AI Introduces Relightable Gaussian Codec Avatars: An Artificial Intelligence Method to Build High-Fidelity Relightable Head Avatars that can be Animated to Generate Novel Expressions

In a groundbreaking move, researchers at Meta AI have tackled the longstanding challenge of achieving high-fidelity relighting for dynamic 3D head avatars. Traditional methods have often needed to catch up when capturing the intricate details of facial expressions, especially in real-time applications where efficiency is paramount. Meta AI’s research team has responded to this challenge by unveiling Relightable Gaussian Codec Avatars, a method poised to redefine the landscape of avatar realism.

The core problem addressed by the research team is the need for more clarity in capturing sub-millimeter details, such as hair strands and pores, in dynamic face sequences. The inherent complexity lies in efficiently modeling diverse materials in human heads, including eyes, skin, and hair, all while accommodating all-frequency reflections. The limitations of existing methods have spurred the need for an innovative solution that seamlessly blends realism with real-time performance.

Existing approaches to relightable avatars have grappled with a trade-off between real-time performance and fidelity. A persistent challenge has been the need for a method that can capture dynamic facial details in real-time applications. Meta AI’s research team recognized this gap and introduced “Relightable Gaussian Codec Avatars” as a transformative solution.

Meta AI’s method introduces a geometry model based on 3D Gaussians, providing precision that extends to sub-millimeter accuracy. This is a notable leap forward in capturing dynamic face sequences, ensuring that avatars exhibit lifelike details, including the nuances of hair and pores. The relightable appearance model, a key component of this innovative approach, is founded on learnable radiance transfer.

The brilliance of these Avatars lies in their comprehensive approach to avatar construction. The geometry model, parameterized by 3D Gaussians, forms the backbone of the avatars, allowing for efficient rendering using the Gaussian Splatting technique. The appearance model, driven by learnable radiance transfer, combines diffuse spherical harmonics and specular spherical Gaussians. This combination empowers the avatars to undergo real-time relighting with point light and continuous illumination.

Beyond these technical aspects, the method introduces disentangled controls for expression, gaze, view, and lighting. The avatars can be dynamically animated by leveraging a latent expression code, gaze information, and a target view direction. This level of control marks a significant stride forward in avatar animation, offering a nuanced and interactive user experience.

These Avatars aren’t just a theoretical advancement; they deliver tangible results. The method allows for the disentangled control of various aspects, as demonstrated through live video-driven animation from head-mounted cameras. This capability creates dynamic, interactive content where real-time video inputs can seamlessly drive avatars.

In conclusion, Meta AI’s “Relightable Gaussian Codec Avatars” stands as a testament to the power of innovation in addressing complex challenges. By combining a geometry model based on 3D Gaussians with a revolutionary learnable radiance transfer appearance model, the research team has surpassed the limitations of existing methods and set a new standard for avatar realism.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 33k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..