Enabling Always-Available Input with Muscle-Computer Interfaces

T. Scott SaponasUniversity of Washington

Desney S. Tan Microsoft Research

Dan Morris Microsoft Research

Ravin Balakrishnan University of Toronto

Jim Turner Microsoft Corporation

James A. Landay University of Washington

Mobile Computing Enables…

“How the computer sees us.”

Igoe & O'Sullivan

Hands Busy

Physically Active

Muscle-Computer Interfaces

Muscles Activate via Electrical Signal

Muscles Activate via Electrical Signal

Electrical Signal can be sensed by Electromyography (EMG)

EMG for Diagnostics, Prosthetics & HCI

Jacobsen, et al. “Utah Arm”

Jacobsen, et al. “Utah Arm”

Costanza, et al. “Intimate interfaces in action”

EMG for Diagnostics, Prosthetics & HCI

Jacobsen, et al. “Utah Arm”

Costanza, et al. “Intimate interfaces in action”

Naik, et al. “Hand gestures”

EMG for Diagnostics, Prosthetics & HCI

Jacobsen, et al. “Utah Arm”

Costanza, et al. “Intimate interfaces in action” Wheeler & Jorgensen “Neuroelectric joysticks”

Naik, et al. “Hand gestures”

EMG for Diagnostics, Prosthetics & HCI

Finger Gestures Detected from Upper Forearm

Detecting Finger Gestures Challenging

Offline Classification of Finger Gestures on a Surface

Saponas, et al. CHI 2008

Real-Time Classification ofFree Space & Hands Busy Gestures

Pinch

Mug Bag

Bimanual Gesture

+

dominant handgesture

non-dominant hand squeeze

Sensor Placed on Upper Forearm

Stimulus / Response Training

Gesture Classification Technique

30 millisecond sample

X 6 Sensors

Support VectorMachine

labeledtraining data

user specific model

machine learning

Gesture Classification Technique

30 millisecond sample

Root Mean Square (RMS) ratios between channels

Frequency Energy10 Hz bands

Phase Coherence ratios between channels

X 6 Sensors

Features Support VectorMachine

labeledtraining data

user specific model

machine learning

Gesture Classification Technique

30 millisecond sample

Root Mean Square (RMS) ratios between channels

Frequency Energy10 Hz bands

Phase Coherence ratios between channels

X 6 Sensors

Features

Support VectorMachine

user specific model

machine learning

gesture classification

12 Person Experiment

Pinch

Mug Bag

Training vs Testing in Several Postures

TrainTest

Left Center Right

Left 78% 72% 57%

Center 70% 79% 74%

Right 68% 73% 74%

4 Finger 3 Finger0%

10%20%30%40%50%60%70%80%90%

100%

Hands-Free Gesture Accuracy

Posture Independent Pinching

Bag in Hand Better Recognized

Mug Bags0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Hands-Busy Gesture Accuracy

Four FingersThree Fingers

Worked Well for Those Who “got it”

all bottom 50% top 50%0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Bags in Hands Gestures

Four Fingers

Three Fingers

80% Accurate with 70 Seconds Training

0 25 50 75 100 125 150 1750%

10%20%30%40%50%60%70%80%90%

100%

Quantity of Training Data vs. Classification Ac-curacy

Seconds of Training Data

Clas

sific

ation

Acc

urac

y

Portable Music Player Menus

• Some participants navigated menus easily• Other participants found interaction difficult

Limitations of Current Technique

• Works best for SINGLE user SINGLE session• Wired Sensors with Gel and Adhesive• Sitting or Standing at a Desk in the Lab

Ongoing & Future WorkWireless Armband, Dry Electrodes, Cross-Session Models

Ongoing & Future Work

Walking & Jogging

Wireless Armband, Dry Electrodes, Cross-Session Models

Ongoing & Future Work

Walking & Jogging

Interactive Tabletops

Wireless Armband, Dry Electrodes, Cross-Session Models

Enabling Always-Available Input with Muscle-Computer Interfaces

T. Scott SaponasUniversity of Washington

Desney S. Tan Microsoft Research

Dan Morris Microsoft Research

Ravin Balakrishnan University of Toronto

Jim Turner Microsoft Corporation

James A. Landay University of Washington

Thanks for Listening