![Page 1: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/1.jpg)
1
Mobile Phones based Continuous Sensing Systems
Kiran [email protected]
Computer LaboratoryUniversity of Cambridge
![Page 2: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/2.jpg)
2
• Microphone
• Magnetometer
• GPS
• Bluetooth
• Accelerometer
• Camera
• Ambient light
• Proximity
Sensors in a Smart Phone
![Page 3: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/3.jpg)
3
Microphone – Speaker Recognition
What Can be DoneCamera – Face Recognition
Accelerometer – Activity Recognition
![Page 4: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/4.jpg)
4
Health monitoring
Application ScenariosSocial sensing
Location based services
![Page 5: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/5.jpg)
5
Sense continuously
Energy-Accuracy trade-offs
Continuous Sensing Mobile Systems
![Page 6: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/6.jpg)
6
Mobile Phone Limitations
Energy, processing, and memory constraints
Google Nexus One, and HDC HD2 are equipped with1GHz processor and 512MB RAM
Energy is still a scarce resource
![Page 7: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/7.jpg)
7
EmotionSense
![Page 8: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/8.jpg)
8
Emotions of users and how they are influenced
Social Psychology - Emotions
![Page 9: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/9.jpg)
9
Speech patterns of users
Social Psychology - Speech Patterns
![Page 10: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/10.jpg)
10
Oh yes, I was very happy today
Self reports ◦ But they are biased towards positive emotions
One-time behavioural study in laboratory◦ Users hide their natural behaviour
Social Psychology - Existing Methods
![Page 11: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/11.jpg)
11
• Use Mobile phones• Powerful sensors• Ubiquitous• Unobtrusive
Our Solution
• Challenges• Sensors not built for this purpose• Battery powered• Processing• Main memory limitations• Privacy concerns
![Page 12: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/12.jpg)
12
EmotionSense - Architecture
EmotionSense Manager
Inference Engine
SpeakerMonitor
HTTP Module
Declarative Database
Remote ServerColocationMonior
MovementMonitor
LocationMonitor
HTK
HTK: Hidden markov ToolKit
Implemented in PyS60 and Symbian C++
![Page 13: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/13.jpg)
13
EmotionSense – Flow of Data
Classifiers Facts Base Inference Engine Actions Base
EmotionSense Manager
Sensor Monitors
Raw data
E.g. X Y Z of Accelerometer
E.g. User is MovingE.g. fact(Moving, True)E.g. fact(action, LocationSamplingInterval, 2)
![Page 14: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/14.jpg)
14
Emotion & Speech Recognition
Collect voice data User/Emotion specific modelsTrain background GMM
[2] M. Liberman, K. Davis, M. Grossman, N. Martey, and J. Bell. Emotional prosody speech and transcripts, 2002.
Training Procedure
At Runtime
Record voice on phone Extract PLPs using HCopy Compare using HERest
[1] http://htk.eng.cam.ac.uk
GMM: Gaussian Mixture ModelPLP: Perceptual Liner PredictionHcopy, HERest: Tools of Hidden markov ToolKit (HTK)
Speaker Recognition: Participants voice dataEmotion Recognition: from libraryLoad speaker and emotion models on phone
![Page 15: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/15.jpg)
15
Clustering of emotions
Emotion Categories
Broad Emotion Narrow Emotion
Happy Elation, Interest, Happy
Sad Sadness
Fear Panic
Anger Disgust, Dominant, Hot anger
Neutral Neutral normal, Neutral conversation, Neutral distant, Neutral tete, Boredom, Passive
(a) Used by psychologists (b) Improves accuracy
Why?
![Page 16: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/16.jpg)
16
Optimizations
Adaptive framework
What About Energy?
![Page 17: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/17.jpg)
17
Silence detection◦ train an additional GMM using silence audio
Comparisons driven by co-location information◦ A recorded audio sequence is compared only
with the models co-located users◦ This improves accuracy and saves energy
Speaker Recognition - Optimizations
![Page 18: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/18.jpg)
18
Implemented using Pyke, a knowledge based inference engine [http://pyke.sourceforge.net/]
Activate GPS only when user is moving
Adaptation Framework
set_location_sampling_intervalforeach facts.fact($factName, $value) check $factName == 'Activity' facts.fact($actionName, $currentInterval) check $actionName == 'LocationInterval' $interval = update($value, $currentInterval)assert facts.fact('action', 'LocationInterval', $interval)
Facts Base Inference Engine Actions Base
E.g. fact(Moving, True)E.g. fact(action, GPS, ON)
![Page 19: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/19.jpg)
19
Sensor Sampling
Time
Sleep Sense Sleep Sense Sleep
Events
Time
Sleep Sense
Events
Sense SleepSleep
![Page 20: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/20.jpg)
20
Sensor Sampling Issues Continuous sampling
degrades battery life
Long sleep durations result in loss of sensor data
Not all sensors are similar
Accuracy varies with sensors and classifiers
![Page 21: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/21.jpg)
21
Sampling Interval
Sleep Sense Once
Sleep 0 ∞Minimum Sampling Interval
Maximum Sampling Interval
Constant
![Page 22: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/22.jpg)
22
Design Methodology
Missable Event
Sleep
Not all events are important
E.g.: Microphone recording when there is no audible sound
•Classify events as Unmissable and Missable•Use functions to control the sleep interval
Back-off Function
E.g.: f(x) = 2x, where x is sleep interval
UnMissable Event
Advance Function
E.g.: f(x) = x/2, where x is sleep interval
Sense
![Page 23: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/23.jpg)
23
Back-off and Advance FunctionsType Back-off function Advance function
Linear
Quadratic
Exponential
Minimum N/A Minimum interval
Maximum Maximum interval N/A
€
kxk
x
2x
xe
€
ln(x)
x
x: sleep interval
![Page 24: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/24.jpg)
24
Dynamic Adaptation
Dynamically switch functions from least to most aggressive
Missable Event Sleep
Sequence Count
Sense
Linear back-off function
Quadratic back-off function
Exponential back-off function
Update Sleep
Interval
< Linear Threshold
< Quadratic Threshold
> Quadratic Threshold
![Page 25: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/25.jpg)
25
Micro-benchmarks
Social psychology experiment
Meeting experiment
Evaluation
![Page 26: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/26.jpg)
26
Speaker Recognition - Benchmarks
Accuracy Effect of noise on accuracy
![Page 27: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/27.jpg)
27
Speaker Recognition – Benchmarks
Energy consumption Latency
Why ?
a) Computationally intensive processingb) High latency
Then why compute locally ?
a) Privacy concernsb) Users can use their own sim cards
![Page 28: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/28.jpg)
28
Emotion Recognition - Benchmarks
Accuracy Energy consumption
Clustering helps
![Page 29: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/29.jpg)
29
Nokia 6210 Navigator mobile phones
18 participants, 10 days
Users filled in daily diary questionnaire
Voice data is discarded immediately
All computation performed locally on phone
Social Psychology Experiment
![Page 30: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/30.jpg)
30
Emotion distribution similarity EmotionSense Self reports
Social Psychology Experiment - Results
Users indicated ``happy'' emotion to represent their mental state, and not necessarily verbal expression
![Page 31: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/31.jpg)
31
Social Psychology Experiment - Results
Correlation with time of day and co-location
![Page 32: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/32.jpg)
32
EmotionSense can also be used to analyze the speech patterns
Considerable amount of consistency in verbal behaviour
Social Psychology Experiment - Results
![Page 33: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/33.jpg)
33
11 participants , 30 minutes discussion
We identified conversation leaders in each time slot of length 5 minutes
The result shows the top five most active speakers
Meeting Experiment
![Page 34: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/34.jpg)
34
Continuous sensing mobile systems
EmotionSense◦ Speaker and Emotion Recognition◦ Sensor Monitors◦ Adaptive and Programmable Framework
Evaluation◦ Micro-benchmarks◦ Social Psychology Experiment◦ Meeting Experiment
Summary
![Page 35: Kiran Rachuri kkr27@cam.ac.uk Computer Laboratory University of Cambridge 1](https://reader035.vdocument.in/reader035/viewer/2022081512/5519db0e5503468b0c8b4b34/html5/thumbnails/35.jpg)
35
Thank YouKiran Rachuri
Computer LaboratoryUniversity of Cambridge
EmotionSense http://www.cl.cam.ac.uk/research/srg/netos/emotionsense/