a fast eavesdropping attack against touchscreens
DESCRIPTION
The pervasiveness of mobile devices increases the risk of exposing sensitive information on the go. In this paper, we arise this concern by presenting an automatic attack against modern touchscreen keyboards. We demonstrate the attack against the Apple iPhone—2010's most popular touchscreen device—although it can be adapted to other devices (e.g., Android) that employ similar key-magnifying keyboards. Our attack processes the stream of frames from a video camera (e.g., surveillance or portable camera) and recognizes keystrokes online, in a fraction of the time needed to perform the same task by direct observation or offline analysis of a recorded video, which can be unfeasible for large amount of data. Our attack detects, tracks, and rectifies the target touchscreen, thus following the device or camera's movements and eliminating possible perspective distortions and rotations In real-world settings, our attack can automatically recognize up to 97.07 percent of the keystrokes (91.03 on average), with 1.15 percent of errors (3.16 on average) at a speed ranging from 37 to 51 keystrokes per minute.TRANSCRIPT
![Page 1: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/1.jpg)
Federico Maggi, Alberto Volpatto, Simone Gasparini, Giacomo Boracchi, Stefano Zanero
![Page 2: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/2.jpg)
Direct attacks Well-‐known in both literature and industry Very active research community
Other types of attacks Social engineering attacks Side-‐channel attacks Difficult to mitigate (if not through awareness)
![Page 3: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/3.jpg)
Less known yet very effective Digital side-‐channels
Example: decrypting SSL through wifi LAN sniffing Physical-‐world observation
Direct observation ▪ Shoulder surfing
Indirect observation ▪ Sound emanations ▪ Reflections ▪ Magnetic radiations ▪ Desk surface vibrations
![Page 4: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/4.jpg)
![Page 5: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/5.jpg)
First attempt of automatic shoulder surfing Recovery of long texts
![Page 6: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/6.jpg)
2010 survey on 2,252 US citizens 72% use a mobile phone for texting 30% use a mobile phone for instant messaging 38% use a mobile phone for Web browsing
(1970) touchscreen technology was invented 2010: 5 billion US dollars market 159% market grow rate Q3 2010: 417 million of touchscreen devices sold
![Page 7: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/7.jpg)
Non-‐automated not interesting time consuming
Automated Is it feasible? Mobile context poses several constraints
![Page 8: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/8.jpg)
![Page 9: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/9.jpg)
Moving target Fixed observation point not always feasible Very small keyboards No visibility of pressed keys No visible key occlusions
![Page 10: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/10.jpg)
Lack of tactile feedback Early soft keyboards were hard to use UI engineers came up with usable keyboards
![Page 11: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/11.jpg)
![Page 12: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/12.jpg)
![Page 13: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/13.jpg)
![Page 14: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/14.jpg)
Old dilemma More secure, less easy to use Example: Google's 2-‐step authentication
Very secure Very unusable
▪ Wait for the verification code every time you do email Apply also in this context
Feedback-‐less touchscreen keyboards ▪ hard to type on
Feedback-‐rich keyboard keyboards ▪ easy to type on ▪ eyes follow the feedback naturally during typing
![Page 15: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/15.jpg)
![Page 16: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/16.jpg)
![Page 17: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/17.jpg)
Our approach
![Page 18: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/18.jpg)
Requirement 1 iPhone-‐like visual feedback mechanism
Requirement 2 Template of the target screen known in advance
![Page 19: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/19.jpg)
![Page 20: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/20.jpg)
SCREEN TEMPLATE KEY TEMPLATES
QW
E R
T Y
(synthetic, hi-‐res)
MAGNIFIED LAYOUT
(x,y-‐coordinates) (screenshot)
![Page 21: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/21.jpg)
Phase 1 Screen detection and rectification
Phase 2 Magnified key detection
Phase 3 Keystroke sequence reconstruction
![Page 22: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/22.jpg)
Input Image depicting the current scene (current frame)
Output Synthetic image of the rectified, cropped screen
Procedure Screen detection Screen rectification
![Page 23: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/23.jpg)
The current frame is searched for the screen template (Requirement 1)
? +
SCREEN TEMPLATE CURRENT FRAME MATCHING PATCH
![Page 24: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/24.jpg)
SURF features Edges Corners
Invariant to: Rotation Scale Skew Occlusions
Homography estimation
TEMPLATE
CURRENT FRAME
![Page 25: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/25.jpg)
Estimate during screen detection
Successfull matches improve matches in subsequent frames
CURRENT FRAME RECTIFIED FRAME
![Page 26: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/26.jpg)
Input Image of the rectified screen
Output Areas where magnified keys appeared
Procedure Background subtraction
![Page 27: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/27.jpg)
- =
CURRENT FRAME SCREEN TEMPLATE FOREGROUND
![Page 28: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/28.jpg)
FOREGROUND
HIGHLIGHTED KEY (MAGNIFIED-KEY CANDIDATE)
OTHER FOREGROUND ELEMENTS (NOISE)
![Page 29: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/29.jpg)
Input Magnified-‐key candidates
Output Sequence of typed symbols
Procedure Approximate neighbors lookup Best matching key identification Fast pruning Key sequence analysis
![Page 30: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/30.jpg)
Known keyboard layout (Requirement 2) Centroid identification Match centroids with keyboard layout
![Page 31: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/31.jpg)
Q W E R T Y U I O P
A S D F G H J K L
Z X C V B N M
![Page 32: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/32.jpg)
![Page 33: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/33.jpg)
CENTROID 1
CENTROID 2
CENTROID 3
E R T
N M
G H J
![Page 34: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/34.jpg)
Region of interest Key template (Req. 2)
E R T G H J N M LOW HIGH LOW LOW LOW LOW MED MED
![Page 35: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/35.jpg)
Computing the key similarity is expensive Black-‐white distribution of the ROI %B/W-‐heuristic is way faster
B W
B W
B W
NOT A LETTER
NOT A LETTER
MAYBE A LETTER
B W (we don’t know which one, yet)
B W
B W ≠ ≠ =
CANDIDATE FOUND
BASELINE
![Page 36: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/36.jpg)
Find maxima of the key similarity function
![Page 37: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/37.jpg)
Phase 1 C++ OpenCV
Phase 2-‐3 Matlab Compiled into C
Threshold estimation Confidence interval (mean, variance) Video samples collected in “no typing” conditions
![Page 38: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/38.jpg)
DEMO
http://www.youtube.com/watch?v=aPuS8kNI30U
http://www.youtube.com/watch?v=t9BxB3dO0KQ
![Page 39: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/39.jpg)
Types of text Context-‐free Context-‐sensitive
3 attackers, 3 victims Goals
Precision and speed Resilience to disturbances
![Page 40: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/40.jpg)
Typing 3 victims are given the input text Victims type text on their iPhones
Recording A recording camera was used for repeatability
Attack 3 attackers are provided with the videos Attackers have “infinite” time to analyze videos
Comparison Automatic attack vs. human attackers
![Page 41: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/41.jpg)
spent chapter foundation identified because first which material notation summarized time spent volume much technical little system reference figured number measurement lorem referring abstract text introductory shown in the we observing request second objective books relationship astute formidable quantile convenient remainder between utilizable tool law resident minutes exemplified the product then temporarily number will per systematic average accumulated south specialty terminal numerous introduce
![Page 42: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/42.jpg)
close your eyes and begin to relax take a deep breath and let it out slowly concentrate on your breathing with each breath you become more relaxed imagine a brilliant white light above you focusing on this light as it flows through your body allow yourself to drift off as you fall deeper and deeper into a more relaxed state of mind now as i
![Page 43: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/43.jpg)
![Page 44: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/44.jpg)
![Page 45: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/45.jpg)
![Page 46: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/46.jpg)
Non-‐magnifying keys Space (on iPhone only) Layout-‐switching keys Mitigation
▪ Device-‐specific heuristics ▪ E.g., on iPhone, exploit color-‐changing spacebar
Alternative layouts (minor limitation) Mitigation
▪ Detect switch ▪ Loop through different templates during detection
![Page 47: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/47.jpg)
![Page 48: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/48.jpg)
[Raguram, CCS 2011] Appeared at the same conference Completely different approach
Classification-‐based They require training
Really, the very same accuracy 97~98%
![Page 49: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/49.jpg)
Touchscreen mobile devices are widespread Shoulder surfing is automatable Automatic shoulder surfing is precise too Counteract these attacks with privacy screens But…
![Page 50: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/50.jpg)
Challenge How to detect tapping?
![Page 51: A Fast Eavesdropping Attack Against Touchscreens](https://reader034.vdocument.in/reader034/viewer/2022050815/54657884af795907578b690f/html5/thumbnails/51.jpg)
Federico Maggi [email protected]
@vp_lab Dipartimento di Elettronica e Informazione
Politecnico di Milano