Demonstrating the Feasibility of Using Forearm Electromyography for Muscle-Computer Interfaces

T. Scott SaponasUniversity of Washington

Desney S. Tan Microsoft Research

Dan Morris Microsoft Research

Ravin Balakrishnan University of Toronto

2

Physical Transducers Leverage Human Expertise

3

Need for Hands Free Input

4

Advances in Muscle Sensing Enable Muscle-Computer Interfaces

…

5

Muscles Activate via Electrical Signal

6

Muscles Activate via Electrical Signal

Electrical Signal can be sensed by Electromyography (EMG)

7

EMG for Diagnostics, Prosthetics & HCI

Jacobsen, et al. “Utah Arm”

8

EMG for Diagnostics, Prosthetics & HCI

Jacobsen, et al. “Utah Arm”

Costanza, et al. “Intimate interfaces in action”

9

EMG for Diagnostics, Prosthetics & HCI

Jacobsen, et al. “Utah Arm”

Costanza, et al. “Intimate interfaces in action”

Naik, et al. “Hand gestures”

10

EMG for Diagnostics, Prosthetics & HCI

Jacobsen, et al. “Utah Arm”

Costanza, et al. “Intimate interfaces in action” Wheeler & Jorgensen “Neuroelectric joysticks”

Naik, et al. “Hand gestures”

11

Detecting Finger Gestures Challenging

12

Sensors Placed on Upper Forearm

13

Sensors Placed on Upper Forearm

14

Gesture Sets

15

Gesture Sets

16

Lift

17

Tap

18

Position

19

Pressure

20

Pressure

Position

Tap

Lift

21

250 millisecond sample

X 8 Sensors

Features Support VectorMachine

training data

user model

test dataevaluation

machine learning

Gesture Classification Technique

22

Gesture Classification Technique

250 millisecond sample

Root Mean Square (RMS) 28 ratios between channels

Frequency Energy10 Hz bins

Phase Coherence 28 ratios between channels

X 8 Sensors

Features Support VectorMachine

training data

user model

test dataevaluation

machine learning

23

Randomized Block Design

1 2 3 4

random delay

24

Randomized Block Design

1 2 3 4 12 34 X 50random delay

random order random order

25

12 participants

• aged 20 – 63 years (mean 46)• 8 female; 4 male• daily computer users• right-handed• 90 minutes

26

Ten-Fold Cross-ValidationLift

10.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

95%

Gesture

Mea

n A

ccur

acy

ChanceChance

Lift Tap Position Pressure

ChanceChance

Lift

ChanceChance

Lift Tap Position Pressure

ChanceChance

Lift

27

Ten-Fold Cross-ValidationTap

10.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

95%

78%

Gesture

Mea

n A

ccur

acy

ChanceChance

Lift Tap Position Pressure

ChanceChance

Lift Tap

28

Ten-Fold Cross-ValidationPosition

10.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

95%

78% 78%

Gesture

Mea

n A

ccur

acy

ChanceChance

Lift Tap Position Pressure

ChanceChance

Lift Tap Position

29

Ten-Fold Cross-ValidationPressure

10.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

95%

78% 78% 84%

Gesture

Mea

n A

ccur

acy

ChanceChance

Lift Tap Position Pressure

ChanceChance

Lift Tap Position Pressure

30

How much training data?

0 10 20 30 40 50 60 70 800%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%LiftTapPositionPressure

Blocks of Training Data

Mea

n Ac

cura

cy

Chance

31

What are we really measuring?

• Skin moving over muscle creates noise• Distant muscle contractions• Gestures are complex movements

32

Limitations of Current Evaluation

• Works best for SINGLE user SINGLE session• Offline Analysis• Approximation of sensor armband

33

Forearm Electromyography for Muscle-Computer Interfaces

Demonstrated possibility of gesture sets using pressure, position, & all five fingers

Future:• Wireless & dry sensors• Dense auto-configurable band• Cross-user models• Quick compound gestures

34

Interaction Possibilities

• Virtual keyboards• Hands busy controls• 3D gestural interaction• Eye-free mobile interaction

35

thanks!acknowledgements:Sumit Basu, James Fogarty, Jon Froehlich, Kayur Patel, Meredith Skeels and our study participants

T. Scott Saponas University of Washington

Desney S. Tan Microsoft Research

Dan Morris Microsoft Research

Ravin Balakrishnan University of Toronto

http://research.microsoft.com/users/dan/muci/

…

36

37

Pressure

Position

Tap

Lift

38

Labeling Training Data With Best Data

stimulus

stimulus labelrest label

39

Single Sample Classification

stimulus

? ? ? ??

stimulus

thumb 1index 5 middle 1

winner

Whole Trial Classification

40

Collect Pilot Data, Develop Classification Techniques, Evaluation

collect pilot data develop classificationtechniques

define gesture sets

collect test data offline analysisexperiment