Meet HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions Using Diffusion Models

In response to the challenging task of generating realistic 3D human-object interactions (HOIs) guided by textual prompts, researchers from Northeastern University, Hangzhou Dianzi University, Stability AI, and Google Research have introduced an innovative solution called HOI-Diff. The intricacies of human-object interactions in computer vision and artificial intelligence have posed a significant hurdle for synthesis tasks. HOI-Diff stands out by adopting a modular design that effectively decomposes the synthesis task into three core modules: a dual-branch diffusion model (HOI-DM) for coarse 3D HOI generation, an affordance prediction diffusion model (APDM) for estimating contacting points, and an affordance-guided interaction correction mechanism for precise human-object interactions.

Traditional approaches to text-driven motion synthesis often fell short by concentrating solely on generating isolated human motions, neglecting the crucial interactions with objects. HOI-Diff addresses this limitation by introducing a dual-branch diffusion model (HOI-DM) capable of simultaneously generating human and object motions based on textual prompts. This innovative design enhances the coherence and realism of generated motions through a cross-attention communication module between the human and object motion generation branches. Additionally, the research team introduces an affordance prediction diffusion model (APDM) to predict the contacting areas between humans and objects during interactions guided by textual prompts.

The affordance prediction diffusion model (APDM) plays a crucial role in the overall effectiveness of HOI-Diff. Operating independently of the HOI-DM results, the APDM acts as a corrective mechanism, addressing potential errors in the generated motions. Notably, the stochastic generation of contacting points by the APDM introduces diversity in the synthesized motions. The researchers further integrate the estimated contacting points into a classifier-guidance system, ensuring accurate and close contact between humans and objects, thereby forming coherent HOIs.

To experimentally validate the capabilities of HOI-Diff, the researchers annotated the BEHAVE dataset with text descriptions, providing a comprehensive training and evaluation framework. The results demonstrate the model’s ability to produce realistic HOIs encompassing various interactions and different types of objects. The modular design and affordance-guided interaction correction showcase significant improvements in generating dynamic and static interactions.

Comparative evaluations against conventional methods, which primarily focus on generating human motions in isolation, reveal the superior performance of HOI-Diff. For this purpose, the researchers adapt two baseline models, MDM and PriorMDM. Visual and quantitative results underscore the model’s effectiveness in generating realistic and accurate human-object interactions.

However, the research team acknowledges certain limitations. Existing datasets for 3D HOIs pose constraints on action and motion diversity, presenting challenges for synthesizing long-term interactions. The precision of affordance estimation remains a critical factor influencing the model’s overall performance.

In conclusion, HOI-Diff represents a novel and effective solution to the intricate problem of 3D human-object interaction synthesis. The modular design and innovative correction mechanisms position it as a promising approach for applications such as animation and virtual environment development. Addressing challenges related to dataset limitations and affordance estimation precision as the field progresses could further enhance the model’s realism and applicability across diverse domains. HOI-Diff is a testament to the continual advancements in text-driven synthesis and human-object interaction modeling.

Check out the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 34k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..