gesture recognitionhomepage.cs.uiowa.edu/~ochipara/classes/sensing_the... · motion-based gesture...
TRANSCRIPT
Gesture Recognition
Presented By: Baoluo Meng
Chutian Gao George F Judisch Renjitha Nair
Project Proposal Group
Gestures
• User Interaction in apps today are moving away from mouse and pen
• Smartphones or Video game controllers using gesture recognition are easy to use and provide intuitive human computer interaction
• Pervasive , more physical and tangible
Our Project and Gestures
• What is a Gesture ? Gesture is a promising mobile user-interface modality that ensures eyes-free interaction without stopping or impeding movements
• And our Project : an educational fun game would definitely need to incorporate a variety of these gesture recognition techniques to be intuitive and appealing to the user.
Gesture Recognition on Mobile Devices
Latest smartphones and similar mobile devices are capable of basically identifying three types of Gestures
• Motion Based Gestures • Touch Based Gestures
• Vision Based Gestures
Touch-Based Gesture Recognition
Gesture Coder : A Tool for Programming Multi-Touch Gestures by Demonstration
Hao Lu(University of Washington)
Yang Li ( Google Research )
Multi-Touch Gestures
• Increasing number of modern devices are equipped with multi-touch capability
• They are intuitive, efficient and often have physical metaphors
• However they are difficult to implement
• Challenge : Complicated touch state transitions from multiple fingers and their simultaneous movements
Existing System • Framework supports predefined gestures with one
recognizer for each gesture
• Higher Level Specification Languages-Burden on Developers
• Recognizing gestures from trajectories generated with a single finger or stylus
• Mostly treats gestures as a shortcut for triggering a discrete , one-shot action
Contributions
• interaction techniques and architectural support for programming multi-touch by demonstration
• conceptualization of multi-touch gestures and enables effective machine learning from small no:of gestures
• methods for training the recognizers from gestures and generating user-modifiable code for invoking app specific actions
Gesture Coder
> Automatically generates code that recognizes multi-touch gestures,tracks their state changes and invokes corresponding application actions
Available as an Eclipse Plug-In for Android SDK !!!
Using a Gesture Coder
• Demonstrating gestures on a multi-touch device
• Testing the generated recognizer anytime
• Integrating generated code into developer's project
Demonstrating Gestures
Testing the Generated Recognizer
Integrating Generated Code Into Project
Life-Cycle of a Gesture in Gesture-Coder
• Possible
• Failed
• Began
• Changed
• Cancelled
• Ended
Essentially Implementing a machine Learning Algorithm with decision trees (using C4.5 algorithm from weka-toolkit for training the decision tree)
Demonstration
Defining Multi-touch
Lü and Li define multi-touch as:
"a sequence of finger configura/on changes, and each each finger configura/on might produce a series of repe//ve mo/ons. A mo/on is repe//ve when any of its segments triggers the same type of ac/on as its whole."
Multi-Touch Gestures as State Machines
How an Algorithm May Generate State Machines for the Gestures it is Given
Callbacks
Each gesture has 6 callbacks: • Possible • Fail • Begin • Changed • Cancel • End
These callbacks are invoked when a transition takes place.
Evaluation
Discussion and Scope For Future Work
• Tuning Parameters such as motion segment length and the threshold for low-pass filtering
• Add more features such as the contour of the finger touches to the model, as it is extensible
• Fine-tune recognition further by adding constraints like absolute location of a gesture on the screen
• Of Course training with higher quality and more training data
Motion-Based Gesture Recognition
Based on the Paper:
Kazuya Murao, Tsutomu Terada, Ai Yano, Ryuichi Matsukura, Evaluating Gesture Recognition by Multiple-Sensor-Containing Mobile Devices
Jussi Tohka, Tampere University of Technology Department of Signal Processing, Introduc>on to Pa?ern Recogni>on
• What is Pa?ern Recogni>on?
Introduc*on to Pa-ern Recogni*on
Jussi Tohka, Tampere University of Technology Department of Signal Processing, Introduc>on to Pa?ern Recogni>on
Placing objects to certain classes
• That thing is a dog. • That thing is a car. • That thing is a dog. • That thing is a chicken.
• That thing is a dog. • That thing is a cat.
h?p://www.improvedoutcomes.com/docs/WebSiteDocs/Classifica>on_and_Predic>on/SLAM/An_Introduc>on_to_Classifica>on.htm
Introduction to Pattern Recognition
Introduc*on to Pa-ern Recogni*on
Jussi Tohka, Tampere University of Technology Department of Signal Processing, Introduc>on to Pa?ern Recogni>on
• Why Recogni>on?
• Examples Op>cal Character Recogni>on (OCR) Ac>vity Recogni>on Gesture Recogni>on
Because is not clever enough.
1. Sensing 2. Pre-‐processing and segmenta>on 3. Feature extrac>on 4. Classifica>on 5. Post-‐processing
Introduc*on to Pa-ern Recogni*on
Five stage of Pa?ern recogni>on:
Jussi Tohka, Tampere University of Technology Department of Signal Processing, Introduc>on to Pa?ern Recogni>on
1. Sensing 2. Preprocessing and segmenta>on 3. Feature extrac*on 4. Classifica*on 5. Post-‐processing
Introduc*on to Pa-ern Recogni*on
Five stage of Pa?ern recogni>on:
Jussi Tohka, Tampere University of Technology Department of Signal Processing, Introduc>on to Pa?ern Recogni>on
• That thing is a dog. • That thing is a car. • That thing is a dog. • That thing is a chicken.
• That thing is a dog. • That thing is a cat.
h?p://www.improvedoutcomes.com/docs/WebSiteDocs/Classifica>on_and_Predic>on/SLAM/An_Introduc>on_to_Classifica>on.htm
Introduc*on to Pa-ern Recogni*on
Ac*vity Recogni*on from Accelerometer Data
Eight ac>vi>es: • Standing • Walking • Running • Climbing up stairs • Climbing down stairs • Sit-‐ups
• Vacuuming • Brushing teeth
Nishkam Ravi and Nikhil Dandekar and Preetham Mysore and Michael L. Li?man Department of Computer Science, Rutgers University
X-‐axis readings for different ac>vi>es
Ac*vity Recogni*on from Accelerometer Data
Data collec*on
Four features: • Mean • Standard Devia>on • Energy • Correla>on
Nishkam Ravi and Nikhil Dandekar and Preetham Mysore and Michael L. Li?man Department of Computer Science, Rutgers University
Results
Classifier
Accuracy(%)
Setting1
Setting2
Setting3
Setting4
Boosted 98.86 98.71 89.96 64
kNN
98.15
99.26
72.93
49.67
Plurality Voting
99.57
99.82
90.61
65.33
Nishkam Ravi and Nikhil Dandekar and Preetham Mysore and Michael L. Li?man Department of Computer Science, Rutgers University
Another experiment on Gesture recognition
Overview
• Experiment - Capturing data for 27 kinds of gestures on 8 subjects - Investigated the effects on recognition accuracy of changing the number and positions of sensors, the number and kinds of gestures.
• Results
- The use of multiple sensors and of sensors positioned at specific positions affects accuracy.
- Gestures are interdependent and selecting specific gestures improves recognition accuracy
Equipment
Experimental board with 9 accelerometers and 9 gyroscopes
Board: W117×H155×D16 (mm) weighed 200 g
Sensors: WAA-006, made by Wireless-T Inc a wireless 3-axis accelerometer
a 3-axis gyroscope
Gestures
Experiment
1. How? All the subjects were right-‐handed and
performed the gestures while holding the lower right of the board with their right hands. 2. Data?
Mean and variance over the sensor value of the gesture were extracted over the whole gesture and over the first 1 second(the la?er is be?er).
Experiment cont.
3. Analysis? • Intra-subject analysis -- measures the
averaged accuracy over eight test subjects after 4-fold cross-validation is conducted for each subject
• Inter-subject analysis -- conducts 4-fold cross validation after the data for the eight subjects are merged
Algorithm Support Vector Machine (SVM)
• Basic idea of SVM -‐ Op>mal hyperplane for linearly separable
pa?erns -‐ Extend to pa?erns that are not linearly separable by transforma>ons of original data to map into new space
Support Vectors
• Support vectors are the data points that lie closest to the decision surface
• They are the most difficult to classify
• They have direct bearing on the optimum location of the decision surface
Support Vectors(cont.)
Support Vector Machine (SVM)
• Assume linear separability for now: – in 2 dimensions, can separate by a line
– in higher dimensions, need hyperplanes
• Can find separating hyperplane by linear programming
– separator can be expressed as ax + by = c
Support Vector Machine (SVM)
SVMs maximize the margin around the separa>ng hyperplane.
Results and consideration
Number of Sensors
-‐ Using mul>ple homogeneous sensors improves accuracy
-‐ Mul>ple homogeneous sensors can capture the diversity among the subjects
Results and consideration cont.
Posi>on of Sensors
-‐ Accelerometer : further from the holding posi>on is be?er
-‐ Gyro : on the cross-‐ point of the axes of gyra>on for each gesture works well
Results and consideration cont.
Results and consideration cont.
Kind of gestures
- Gesture that involves parallel displacement and is not consistent are rarely recognized -‐ A gyro is inferior to an accelerometer
Results and consideration cont.
Number of gestures
-‐ Higher accuracy can be achieved without excluding a lot of gestures by selec>ng gestures that do not conflict with each other
Conclusion
This paper investigates the effects on gesture recognition accuracy by changing the number and positions of sensors and the kinds and number of gestures
Sources
• http://yangl.org/pdf/gcoder.pdf • http://www.cs.ucf.edu/courses/cap6412/
fall2009/papers/Berwick2003.pdf • http://ieeexplore.ieee.org/
ielx5/5958741/5959573/05959595.pdf?tp=&arnumber=5959595&isnumber=5959573
• http://www.improvedoutcomes.com/docs/WebSiteDocs/Classification_and_Prediction/SLAM/An_Introduction_to_Classification.htm