Microsoft and Newcastle University develop wrist-mounted motion control sensor

Television remotes and games controllers could soon be things of the past with new technology that allows users to control electronic devices with just a wave of the hand.

Researchers at Newcastle University and Microsoft Research (MSR) Cambridge have developed a sensor the size of a wrist-watch which tracks the 3D movement of the hand and allows the user to remotely control any device.

Known as “Digits,” the technology maps finger movement and orientation using an infrared camera, IR laser line generator, IR diffuse illuminator, and an inertial-measurement unit (IMU) track.

“The Digits sensor doesn’t rely on any external infrastructure so it is completely mobile,” said David Kim, a PhD student at Newcastle University.

“This means users are not bound to a fixed space. They can interact while moving from room to room or even running down the street. What Digits does is finally take 3D interaction outside the living room.”

Kim developed the technology along with Otmar Hilliges, Shahram Izadi, Alex Butler, and Jiawen Chen of MSR Cambridge; Iason Oikonomidis of Greece’s Foundation for Research & Technology; and Professor Patrick Olivier of Newcastle University’s Culture Lab.

Digits can be used to interact spontaneously with electronic devices using simple gestures. For example, users could extend two fingers to shoot an adversary in a video game, or answer their mobile phone without even taking it out of their pocket.

The researchers said that one of the biggest challenges was delivering superior gesture sensing, so that the device can “understand” the human hand, from wrist orientation to the angle of each finger joint.

They developed a real-time signal-processing pipeline that samples key parts of the hand, such as the tips and lower regions of each finger, and also built two kinematic models that enable full reconstruction of hand poses from just five key points.

“We had to understand our own body parts first before we could formulate their workings mathematically,” Izadi explained.

“We spent hours just staring at our fingers. We read dozens of scientific papers about the biomechanical properties of the human hand. We tried to correlate these five points with the highly complex motion of the hand. In fact, we completely rewrote each kinematic model about three or four times until we got it just right.”

To enable ubiquitous 3D spatial interaction anywhere, Digits also had to be lightweight, consume little power, and have the potential to be as small and comfortable as a watch. By instrumenting only the wrist, the user’s entire hand is left to interact freely without wearing data gloves.

The Digits technical paper will be presented at the annual ACM Symposium on User Interface Software and Technology (UIST) 2012. The researchers will put particular emphasis on mobile scenarios, where it can interact with mobile phones and tablets.

A video of the technology in action can be viewed here: