MIT's Ultrasound Bracelet Recognizes Sign Language and Plays Piano with a Robotic Hand

A group of engineers at MIT has developed an ultrasound bracelet that tracks in real-time all 22 degrees of freedom of the human hand, with a latency of about 120 milliseconds, without the need for cameras, gloves, or sensors on the fingers. The research was published in Nature Electronics in March 2026.

The device is worn on the wrist: a wrist-sized adhesive ultrasound label acquires images of the tendons and muscles of the forearm, while the onboard electronics, compact as a mobile phone, process the signal on site. An AI algorithm translates the ultrasonic images into the corresponding positions of the five fingers and palm in real-time.

From Signs to Robotic Hand

The team led by Xuanhe Zhao, a mechanical engineering professor at MIT, along with Gengxi Lu and other co-authors from MIT and the University of Southern California, conducted tests on eight volunteers with varying hand and wrist sizes. The bracelet accurately recognized all 26 characters of the American Sign Language alphabet.

In practical demonstrations, participants controlled a commercial robotic hand to play a piano through gestures and played basketball with a small tabletop hoop. The device also allows manipulation of virtual objects, with the ability to pinch, grasp, and perform zooms, without any physical controller.

VR/AR and Humanoid Robot Training

Zhao is confident about the short-term impact: "We believe this work will have an immediate impact, as it can replace current hand tracking techniques with wearable ultrasound bracelets in virtual and augmented reality." The system could also provide vast amounts of training data for humanoid robots specialized in fine manipulations, including surgical procedures.

The team considers this the most advanced method available for tracking hand movement through wrist-worn imaging. Next steps include further miniaturization of the hardware and training the AI model on a broader repertoire of gestures, with volunteers having more varied hand shapes.

The research was funded by MIT, the National Institutes of Health, the National Science Foundation, the U.S. Department of Defense, and the Singapore National Research Foundation through the Singapore-MIT Alliance for Research and Technology.