Researchers from Cambridge have Developed a Virtual Reality Application Using Machine Learning to Give Users the ‘Superhuman’ Ability to Open and Control Tools in Virtual Reality

Hotkeys are keyboard shortcuts typically found in traditional desktop applications. A team of researchers from the University of Cambridge explores what makes for a suitable alternative to hotkeys in a 3D interaction space where keyboard input is no longer the only option. Scientists have created a VR program that allows users to access and use multiple 3D modeling tools with the wave of a hand. The team at the University of Cambridge employed machine learning to create a system called “HotGestures” that works similarly to the shortcut keys on a computer’s desktop.

Humans are primarily visual creatures. Thus, hand gestures are a natural way to transmit information and create relationships between words. HotGestures allows users to access and use virtual tools quickly through simple hand gestures. This method works well in tandem with standard menu navigation. Through two user tests, researchers can gauge HotGestures’ potential and find that the gesture-based technique offers quick and efficient tool selection and shortcuts. HotGestures was well-received by participants because of its uniqueness, speed, and ease of use and because it complemented the more traditional menu-based interaction. HotGestures allows users to construct virtual reality figures and forms without interacting with a menu, allowing them to maintain attention without distracting themselves.

Despite years of hype, the potential of virtual reality (VR) and associated applications has yet to be completely realized outside of the gaming industry. Hotkeys, or command shortcuts like ctrl+c and ctrl+v, are standard fare for anyone who has used desktop applications. These shortcuts save time by eliminating the need to search through menus for the desired function, but they are only useful if the user already knows the appropriate command. Researchers’ teamwork developed the concept of ‘HotGestures’ to replace hotkeys in 3D virtual reality environments.

For instance, the scissors tool is activated by a cutting motion, and the spray can by a spraying motion tool. The user can access the desired function directly without digging through menus or memorizing keyboard shortcuts. Users may quickly and easily switch between tools without stopping what they’re doing to open a menu or push a button on a controller or keyboard.

The study developed a neural network gesture recognition system that can identify gestures by making predictions on a stream of data containing the positions of the user’s hand joints. The software was developed to remember ten distinct actions related to creating 3D models, including using a pen, cube, cylinder, sphere, palette, spray, cut, scale, duplicate, and delete.

The group conducted two preliminary trials with 30 participants who utilized HotGestures, menu instructions, or both. The gesture-based method allowed for quick and easy access to frequently used tools. HotGestures was well-received by participants because of its uniqueness, speed, and ease of use and because it complemented the more traditional menu-based interaction. The researchers ensured no accidental activations by designing a system that could distinguish between commands and natural hand movements. Overall, the gesture-based system outpaced the menu-based one in terms of speed. The researchers have released the dataset and accompanying source code so VR application developers can include it in their businesses.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 32k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..

We are also on Telegram and WhatsApp.