AI has come a long way in visual perception and language processing. However, these abilities are not enough for building systems that can interact with the physical world. Humans handle objects or make controlled movements using the sense of touch. We feel texture, sense temperature, and gauge weight to guide each action with accuracy. This tactile feedback allows us to manipulate fragile items, use tools with control, and perform intricate tasks smoothly.
Meta, well-known for its work in virtual and augmented reality, is now taking on the challenge of creating AI that can interact with the physical world much like a human. Through its FAIR Robotics initiative, Meta is developing open-source tools and frameworks to enhance robots’ sense of touch and physical agility. These efforts could lead to the development of embodied AI — systems that don’t just see but can also feel and manipulate objects just like humans do.
What Is Embodied AI?
Embodied AI combines physical interaction with artificial intelligence, enabling machines to sense, respond, and engage naturally with their surroundings. Instead of just “seeing” or “hearing” inputs, it allows AI systems to feel and act in the world. Think of a robot that can sense the pressure it applies to an object, adjust its grip, and move with agility. Embodied AI moves AI from screens and speakers into the physical world, making it capable of manipulating objects, performing tasks, and interacting more meaningfully with people.
For example, a robot built on Embodied AI could help an elderly person pick up fragile items without damaging them. In healthcare, it could assist doctors by holding instruments precisely during surgery. This potential extends far beyond robotic arms in labs or automated arms in factories; it’s about creating machines that understand and respond to their physical environment in real time.
Meta’s Approach Towards Embodied AI
Meta is focusing on three key areas to bring embodied AI closer to human-like touch. First, the company is developing advanced tactile sensing technologies that enable machines to detect things like pressure, texture, and temperature. Second, Meta is creating touch perception models that allow AI to understand and react to these signals. Lastly, Meta is building a tactile development platform that integrates multiple sensors with these perception models, offering a complete system for building touch-enabled AI. Here’s how Meta is driving progress in embodied AI across each of these areas.
Meta Digit 360: Human-Level Tactile Sensing
Meta has introduced Digit 360 fingertip, a tactile sensing technology designed to give embodied AI a human-like sense of touch. With over 18 sensing features, it can detect vibrations, heat, and even chemicals on surfaces. Equipped with an AI chip, fingertip processes touch data instantly, allowing for quick responses to inputs like the heat of a stove or the sharp poke of a needle. This technology acts as a “peripheral nervous system” within embodied AI, simulating reflexive responses similar to human reactions. Meta has developed this fingertip with a unique optical system containing over 8 million taxels that can capture touch from every angle. It senses tiny details, down to forces as small as one millinewton, giving embodied AI a finely tuned sensitivity to their environment.
Meta Sparsh: The Foundation for Tactile Perception
Meta is enhancing touch perception capabilities to help AI understand and respond to physical sensations. Named after the Sanskrit word for “touch,” Sparsh acts as a “touch brain” for embodied AI. The model allows machines to interpret complex tactile signals like pressure and grip.
One of Sparsh’s standout features is its versatility. Traditional tactile systems employ separate models for each task, relying heavily on labelled data and specific sensors. Sparsh changes this approach entirely. As a general-purpose model, it adapts to various sensors and tasks. It learns touch patterns using self-supervised learning (SSL) on a massive database of over 460,000 tactile images—without needing labelled data.
Meta has also introduced TacBench, a new benchmark with six touch-based tasks to evaluate Sparsh’s abilities. Meta claims that Sparsh outperformed traditional models by 95.1%, especially in low-data scenarios. Versions of Sparsh built on Meta’s I-JEPA and DINO architectures have demonstrated remarkable abilities in tasks such as force estimation, slip detection, and complex manipulation.
Meta Digit Plexus: A Platform for Tactile System Development
Meta has introduced Digit Plexus to integrate sensing technologies and tactile perception models for creating an embodied AI system. The platform combines fingertip and palm sensors within a single robotic hand to enable more coordinated touch responses. This setup allows embodied AI to process sensory feedback and adjust its actions in real time, like how a human hand moves and reacts.
By standardizing touch feedback across the hand, Digit Plexus enhances the precision and control of embodied AI. This development is especially vital in fields like manufacturing and healthcare, where careful handling is essential. The platform links sensors like the fingertip and ReSkin to a control system, streamlining data collection, control, and analysis—all through a single cable.
Meta is releasing the software and hardware designs for Digit Plexus to the open-source community. The goal is to foster collaboration and accelerate research in embodied AI, driving innovation and progress in these fields.
Promoting Embodied AI Research and Development
Meta is advancing not only technology but also resources to promote embodied AI research and development. A key initiative is the development of benchmarks to assess AI models. One such benchmark, PARTNR (Planning And Reasoning Tasks in humaN-Robot collaboration), evaluates how AI models interact with humans during household tasks. Using the Habitat 3.0 simulator, PARTNR provides a realistic environment where robots assist with tasks like cleaning and cooking. With over 100,000 language-based tasks, it aims to accelerate progress in embodied AI.
Besides internal initiatives, Meta is collaborating with organizations like GelSight Inc. and Wonik Robotics to accelerate the adoption of tactile sensing technologies. GelSight will distribute Digit 360 sensors, while Wonik Robotics will manufacture the Allegro Hand, which integrates Digit Plexus technology. By making these technologies available through open-source platforms and partnerships, Meta is helping create an ecosystem that could lead to innovations in healthcare, manufacturing, and domestic assistance.
The Bottom Line
Meta is advancing embodied AI, taking it beyond just sight and sound to include the sense of touch. With innovations like Digit 360 and Sparsh, AI systems are gaining the ability to feel and respond to their surroundings with precision. By sharing these technologies with the open-source community and partnering with key organizations, Meta is helping accelerate the development of tactile sensing. This progress could lead to breakthroughs in fields like healthcare, manufacturing, and home assistance, making AI more capable and responsive in real-world tasks.
Credit: Source link
