WristWhirl: A Smartwatch That Works Like a Joystick

©WSJ

It’s time to free your hand! Researchers from Dartmouth College and Human-Computer Interaction (HCI) Lab at University of Manitoba have been developing a smart wristwatch which allows users perform touchscreen gestures without even touching the screen. Usually, interacting with smart devices requires both hands such as for flicking or swiping on the screen. Researchers saw this “opposite-side hand” operation very inconvenient especially when user’s hands are full. To obviate such inconvenience, they have looked into the possibility of a one-hand operational smart device and have created a prototype of a new type of smartwatch called WristWhirl. With this smartwatch, users can check emails, listen to music, track fitness activities, and do many other functions just by performing simple wrist movements.

Xing-Dong Yang, assistant professor of computer science at Dartmouth College, explained that research on one-handed gestures for smartwatches has been done in the past, however, the usability of the wrist’s joystick motion for input on smartwatches has not been widely explored as of yet. He said, “WristWhirl is the first to explore gestural input. Technology like ours shows what smartwatches may be able to do in the future, by allowing users to interact with the device using one hand (the one that the watch is worn on) while freeing up the other hand for other tasks” (Dartmouth Press).

So what does it mean by “wrist as joystick”? The wrist is one of the most flexible joints in the human. It can rotate in both directions. It can also bend forward and backward. This flexibility allows a wide range of motions that could be used to turn the wrist into a “joystick” for same-side hand operations on smartwatches.

In developing the prototype, researchers looked closely into the biomechanical properties of the wrist. A user study was done with 15 participants (right-handed) aged 15 to 20 years. To better understand the usability of the wrist’s joystick motions, they tested eight different types of gestures to control the smartwatch. These are called free-form shape gestures and they are shown below:

Figure 1. The eight free-form shape gestures tested for wrist motion usability. The black dot indicates a gesture’s starting point. / ©Gong et al., 2016

A number of factors were considered for the usability test of the prototype. Researchers included eyes-free gestures for input in case users are not able to directly look at the screen while performing the gestures. The eyes-free input can come in very handy during intense user activity. They also featured a position-control mode which enables the physical motion of the wrist to be mapped to actions on the screen. The position of the on-screen visual cues represent the direction and amount of the wrist bend. Gesture delimiter such as finger pinch was also considered.

During the user study, the participants were asked to reproduce the gestures as accurately and as fast as possible in two different situations. First, performing gestures while looking at the display and second, performing the gestures without looking at the display. A difference in response time was recorded, and interestingly, the response time for eyes-free input (877 ms) was less than the former (1043 ms). The participants were able to perform the gestures faster when they were not looking at the display. This can be explained due to participants trying to ensure that they draw the gestures more precisely with the help of visual feedback on the screen.

Figure 2. WisrtWhirl prototype / ©Gong et al., 2016

The WristWhirl includes a 2-inch TFT display and a plastic strap with built-in infrared sensor gap and piezoelectric vibration sensor. All these elements are located inside the strap. The data obtained by the sensor are processed by an Arduino DUE board.

WristWhirl is composed of a 2 in. TFT display and a plastic strap with 12 infrared proximity sensors, each including a pair of IR emitters and detectors (LITON LTE-301 & 302) and operating at 940nm with a maximum sensing distance of approx. 12 cm and a piezoelectric vibration sensor for finger pinch detection (Minisense 100) placed inside the wrist strap. The sensors were connected to Arduino

DUE first and then to a Lenovo ThinkPad x1 Carbon laptop (2.1GHz Intel Core i7 CPU 8GB RAM) to record sensor data (readings from 0 to 1023 with 1023 being the closest proximity) at a speed of 9600 Hz. To start a gesture, a user can pinch, which then turns on the proximity sensors to capture the wrist motion. When finished with the gesture, the user can pinch again which signals the end of the gesture and turns off the proximity sensors to save battery.

The data recorded from each proximity sensor was treated as a vector. The direction of the vector was determined by the location of the sensor along the watch band. The length of the vector was determined by the value of the sensor. The higher value indicates a longer length of the vector. The data was obtained by taking a summary of the three corresponding vectors of the consecutive sensors.

Researchers tested the prototype using Google Maps and off-the-shelf game. WristWhirl has the following four applications:

1. Gesture shortcuts: Similar to popular gestures search app, this shortcut app allows the user to open applications by performing gestures. For example, the user can open a calendar app by gesturing (or drawing) a triangle. The user can also gesture to speed dial a number (e.g. drawing “L” to call Lisa).

2. Music player: A long wrist-swipe allows the user to skip through multiple tracks whereas a short wirst-swipe allows the user to skip to the next song.

3. 2D Navigation: The user can use left or right wrist motions (ulnar or radial) to pan up or down. To pan left or right, the user can use up and down wrist motions (extension or flexion). The users can also zoom in and out by rotating (whirling) the wrist in the clockwise or counterclockwise directions.

4. Game: The user can play Tetris by swiping the wrist left, right, up and down. By swiping up, the user can change the orientation and by swiping down the user can drop the piece.

Preliminary prototype evaluation results showed that the free-form shape gestures were able to be recognized by a $1 gesture recognizer with an accuracy of 93.8%. The future research involves further improving the pinch detection, the proximity sensor to avoid interference from ambient light, exploring multi-touch gestures and supporting alternative designs for different wrist sizes.