mole: motion leaks through smartwatch sensors

MoLe: Motion Leaks through Smartwatch Sensors

Master’s course 29th, Park Joon Young

Contents• Attack concept / Contributions • Previous works • First look • System overview / Assumption • Design details / Evaluation • Discussion • Related works • Future works

Attack concept

Accelerometer

Gyroscope

NO PERMISSIONS

Attack concept

Identifying the leakage - key-press detection - handmotion tracking- cross-user data matching- Bayesian inference

Developing the system - Samsung Gear Live smart watch- experimenting with real users- revealing accuracy

Contributions

Previous Works

Key logging based on side-channels

Attacks using sensors on smartphone

Previous Works

Keyboard Acoustic Emanations (2004) - neural network

TouchLogger (2011) - accelerometer

Timing Analysis SSH (2001) - Hidden Markov Model

KeySweeper (2015) - RF signal

Compromising Electromagnetic Emanations (2009) - electromagnetic

ACCessory (2012) - accelerometer

On the Practicality - (2012) - gyroscope, accelerometer

TapPrints (2012) - gyroscope, accelerometer

(sp)iPhone (2011) - accelerometer

Key logging based on side-channels

Attacks using sensors on smartphone

Previous Works

Keyboard Acoustic Emanations (2004) - neural network

TouchLogger (2011) - accelerometer

Timing Analysis SSH (2001) - Hidden Markov Model

KeySweeper (2015) - RF signal

Compromising Electromagnetic Emanations (2009) - electromagnetic

ACCessory (2012) - accelerometer

On the Practicality - (2012) - gyroscope, accelerometer

TapPrints (2012) - gyroscope, accelerometer

(sp)iPhone (2011) - accelerometer

MoLe (2015)- gyroscope, accelerometer

First look

First look

• Tested with computer vision techniques. (NOT accel / gyro data)

• Left hand only

• “F” is home position

X axis displacements

watch’s X axis

time(sec)

First look

First look

System overview

• MoLe app installed on smartwatch

• Sensor data receiving at the server

System overview

System overview

!

"

#$

%&

Assumptions

• One word at a time

• Only on English

• Only on Samsung smart watch (can compute CPC for other model)

• Appropriate typing fingers

Design details

Design details

Keystroke detector

Point cloud fitting

Bayesian inference

Design details

• Z axis of the watch

• FP / FN occurs

• Bagged decision tree

- Keystroke detector -

Bagging decision tree

• Decision tree

• Bootstrap aggregating-> Bagging

• Attempt again and again, average each samples

Design details

• Z axis of the watch

• FP / FN occurs

• Bagged decision tree

- Keystroke detector -

Design details- Keystroke detector -

Pressed-or-not accuracy

Design details- Keystroke detector -

MoLe against Android API

• Find / Remove gravity

• Calculate displacement

• Kalman smoothing

Design details- Keystroke detector -

Design details

Keystroke detector

Point cloud fitting

Bayesian inference

Design details- Point cloud fitting -

Generate convex hulls of CPC / UPC

Calculate centroids

Rotate & Scale

Design details

Keystroke detector

Point cloud fitting

Bayesian inference

Design details- Bayesian inference -

- Bayesian inference -

• : candidate word(dictionary)

• : observation motion data

• : posterior probability

• : probability word W based on the observed motion data

• : prior probability, captures the word’s occurrence frequency

• : probability of the observation

Design details- Bayesian inference -

same for all possible words

assume, equal among words

Key Goal : obtaining high values

Bayesian inference

Design details- Bayesian inference : Step 1 -

Design details- Bayesian inference : Step 1 -

* example * “apple” -> ap, ap, al, ae, pp, pl, pe, pl, pe, le

t(O) h(O) e(X)

t(O) h(X) e(O)

t(X) h(O) e(O)

Design details- Bayesian inference : Step 2 -

• Consecutive characters

• “er”, “re”, “ea”, “fa”

• Treat as one key

🤔

Design details- Bayesian inference : Step 3 -

• 2D displacements

• Point cloud fitting makes better predict

• Gaussian distribution

Design details- Bayesian inference : Step 3 -

• 2D displacements

• Point cloud fitting makes better predict

• Gaussian distribution

Design details- Bayesian inference : Step 3 -

• 2D displacements

• Point cloud fitting makes better predict

• Gaussian distribution

Design details- Bayesian inference : Step 3 -

• 2D displacements

• Point cloud fitting makes better predict

• Gaussian distribution

Design details- Bayesian inference : Step 3 -

• 2D displacements

• Point cloud fitting makes better predict

• Gaussian distribution Probability density of given character

• Detect sequential movements

• Considers previous character

Design details- Bayesian inference : Step 4 -

• Detect sequential movements

• Considers previous character

Design details- Bayesian inference : Step 4 -

• Missing keys from right hand

• Check time interval every possible character-sequence

• Compensates speed bias between attacker and attackee with a factor

Design details- Bayesian inference : Step 5 -

• Missing keys from right hand

• Check time interval every possible character-sequence

• Compensates speed bias between attacker and attackee with a factor

Design details- Bayesian inference : Step 5 -

🤔

Evaluation

Evaluation• Gyroscope readings at 200Hz with timestamps

• 8 subjects, 5 native English speakers, 3 females

• 300 words randomly selected from 5000 most frequently used words

• word-length ranged from 1 to 14

• re-enter if incorrectly typed

• Between each word, hand position initialized on “F” and “J”

• Two attackers, trained Top-500 longest words in the dictionary on same keyboard

Evaluation

30% for 5 possible words

50% for 24 possible words

🤔(1) How well can MoLe guess each word?

Evaluation

Better results

(1) How well can MoLe guess each word?

Evaluation

(2) What factors affect the rank?

Evaluation

(3) Impact of each Bayesian opportunity

Evaluation

(4) Impact of sampling rate

Evaluation

(5) Keyboard variant

Evaluation

(6) Recovery via human observation

Evaluation

(6) Recovery via human observation

are

Discussion

Discussion

Confined to separate words

Applying nature language processing

Typing activity classifier

Conclusion

Identifying the leakage - key-press detection - handmotion tracking- cross-user data matching- Bayesian inference

Developing the system - Samsung Gear Live smart watch- experimenting with real users- revealing accuracy

Conclusion

Sensor data can leak informations

Diminishing the sampling rate of the sensors can alleviate the attack

Wearable devices could be “double edged sword”

Conclusion

Related works

Related works- (Smart)Watch Your Taps -

NHHT HHT

Related works

• Classification algorithms- Simple linear regression - Random forest - K-nearest neighbors

- (Smart)Watch Your Taps -

Related works

• Classification algorithms- Simple linear regression - Random forest - K-nearest neighbors

- (Smart)Watch Your Taps -

Related works- We can track you .. Metro -

• Tracking metro riders using accelerometers on smartphones

• boosted Naive Bayesian (AdaBoost)

• Decision trees (Random forest)

• Naive Bayesian - family of algorithms based on a common principle - a particular feature is independent of any other feature

• AdaBoost(Adaptive Boosting) - machine learning meta-algorithm - can be used in conjunction with many other types of algorithms- ‘weak learners’ can boost classify

• Decision trees (Random forest)

Related works- We can track you .. Metro -

Related works- We can track you .. Metro -

Question & Answer