The Sound of One Hand:A Wrist-mounted Bio-acoustic

Fingertip Gesture Interface

Brian Amento, Will HillAT&T Labs – Research

Loren TerveenUniversity of Minnesota

Outline

• Motivation

• Gesture Interfaces

• Signal Classifiers

• Prototype Applications

• Future Work

Motivation• Small wearable digital devices increasingly

popular (Cellphones, PDAs, Rios, etc..)

• Nonlinear access to linear media will increase– Voicemail, Music, Video, Radio, Text– Controls: Device Select, Play, Stop, Scan forward,

Scan backward, Faster, Slower, Item Select, Exit

Current Interfaces to Mobile Devices

• Two-handed control mechanisms• Pressing device buttons• Writing/selecting with stylus

– Un-holstering a wearable is a pain (i.e., wristwatches beat pocket watches)

• Speech recognition– Noise or social setting may rule out voice control

• Our Goal: Invisible, weightless, un-tethered and cost-free

How about a gesture interface?

Body tracking

Teresa Martin 1997

Polhemus 2000

Datagloves

Image hand tracking

Cullen Jennings, 1999

Our Approach

“Natural” fingertip gestures

What’s “natural”• Small - max displacement of 5 cm

• Gentle, < 10% of pressing strength (e.g. no finger snap)

• Few gestures, little memory work

• Avoid ring and pinky finger

• Examples:– Thumb as anvil - index, middle as hammer– Thumbpad to fingerpad– Thumbpad to fingernail edge

Fingertip Gestures• Tap, double tap

• Finger and thumb pads rub

• Money gesture and reverse

• Finger and thumb pads press

• Soft Flick

Fingertip Gesture Interface• Wristband-mounted piezo-electric contact

microphones positioned on the styloid bones

• Sense bone conducted sounds produced by gentle fingertip gestures

Simple Classifier

• Allows real-time analysis and control

• 800 samples every 10th of a second

• Take max absolute, quantize to 10 levels

• Finite state machine outputs Taps and Rubs– Intermediate states filter background noise– Buffer states allow continuous gestures

• Surprisingly accurate: ~90%

Example Signals

More Sophisticated Classifier

• Noticeable differences in audio signals

• Hidden Markov Models

• Gesture and noise models trained with sampled data

• Confidence levels for each trained gesture

HMM Classifier Accuracy

• Using 3 subjects, collected 100 instances of gestures rub, tap and flick

• 80 used for training, 20 for testing

Accuracy

Tap 55/60 (92%)

Rub 59/60 (98%)

Flick 56/60 (96%)

Wrist Display Prototype

• Timex Internet Messenger watch

• Tap to cycle through messages

• Double-tap to rewind

Other Prototypes

• Cellphone dialing application– Rub scrolls list in one direction– Tap dials phone number

• Powerpoint slide control– Tap moves forward one slide– Double tap moves back

Future Work

• Miniaturization of device– Hitachi SH5 controller

• Improved gesture classifiers• Finger Identification

– Analyze signals from multiple microphone locations

• User Studies– Usefulness: Compare performance to current cellphone,

PDA and desktop control interfaces.– Social impact: Study how users exploit private control

techniques to mobile devices

Conclusion

• Fingertip gestures– sensed acoustically at the wrist – can be communicated wirelessly to nearby

devices– show promise as a control method.