![Page 1: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/1.jpg)
MoodScope:Sensing mood from smartphone usage patterns
Robert LikamwaLin Zhong
Yunxin LiuNicholas D. Lane
Asia
![Page 2: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/2.jpg)
2
Earlier today…
![Page 3: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/3.jpg)
Mood-Enhanced Apps
Socialecosystems
Media recommendation
Personalanalytics
3 of 30
Some time in the future…
![Page 4: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/4.jpg)
Affective Computing(Mood and Emotion)
Audio/Video-based(AffectAura, EmotionSense)
Biometric-based(Skin conductivity,
Temperature, Pulse rate)Highly temporalHigh cost of deploymentHassle
Captures expressionsPower hungrySlightly invasive
4 of 30
![Page 5: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/5.jpg)
Can your mobile phone infer your
mood?
![Page 6: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/6.jpg)
Can your mobile phone infer your
mood?From already-
available, low-power information?*
* No audio/video sensing, no body-instrumentation
![Page 7: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/7.jpg)
MoodScope ∈ Affective Computing
Audio/Video-based
Usage Trace-based(MoodScope)
Biometric-based
Very direct, Fine-grainedHigh cost of deployment
Captures expressionsPower hungrySlightly invasive
Passive, ContinuousHow to model mood?
7 of 30
![Page 8: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/8.jpg)
Mood is…• … a persistent long-lasting state
o Lasts hours or dayso Emotion lasts seconds or minutes
• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns
8 of 30
![Page 9: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/9.jpg)
happysadnervousdepressed excited
relaxed
calm
stressed bored
Circumplex model (Russell 1980)
attentive
9 of 30
![Page 10: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/10.jpg)
Mood is…• … a persistent long-lasting state
o Lasts hours or dayso Emotion lasts seconds or minutes
• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns
10 of 32
• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns
![Page 11: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/11.jpg)
How is the user communicating?
![Page 12: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/12.jpg)
What apps is the user using?
![Page 13: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/13.jpg)
f ( ) = moodusage
13 of 30
![Page 14: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/14.jpg)
iPhone Livelab Logger• Web history• Phone call history• Sms history• Email history• Location history• App usage
Adapted From C. Shepard, A. Rahmati, C. Tossel, L. Zhong, And P. Kortum, "Livelab: Measuring Wireless Networks And Smartphone Users In The Field," In Hotmetrics, 2010.
14 of 30
![Page 15: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/15.jpg)
iPhone Livelab Logger• Web history• Phone call history• Sms history• Email history• Location history• App usage
Runs as shellHash private dataUploads logs to our server nightly
How can we generate mood labels?15 of 30
![Page 16: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/16.jpg)
Mood Journaling App
User-base32 users aged between 18 and 29
11 females 16 of 30
![Page 17: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/17.jpg)
Inference
• Detect a mood pattern
• Validate with only 60 days of data
• Wide range of candidate usage data
• Low computational resources
17 of 30
![Page 18: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/18.jpg)
Daily Mood Averages
• Separate pleasure, activeness dimension
• Take the average over a day _______________
4
Σ( )
18 of 30
![Page 19: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/19.jpg)
Exploring Features• Communication
o SMSo Emailo Phone Calls
• To whom?o# messageso Length/Duration
Consider “Top 10” Histograms
How many phone calls were made to #1? #2? … #10?
How much time was spent on calls to #1? #2? … #10?
19 of 30
?
?
![Page 20: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/20.jpg)
Exploring Features• Communication
o SMSo Emailo Phone Calls
• To whom?o# messageso Length/Duration
• Usage ActivityoApplicationsoWebsites visitedo Location History
• Which (app/site/location)?o# instances
20 of 30
![Page 21: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/21.jpg)
Previous Mood• Use previous 2 pairs of mood labels
21 of 30
![Page 22: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/22.jpg)
Data Type Histogram by: Dimensions
Email contacts# Messages 10# Characters 10
SMS contacts# Messages 10# Characters 10
Phone call contacts# Calls 10Call Duration 10
Website domains # Visits 10Location Clusters # Visits 10
Apps# App launches 10App Duration 10
Categories of Apps# App launches 12
App Duration 12Previous Pleasure and Activeness Averages
N/A 4
22 of 30
![Page 23: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/23.jpg)
??
• Multi-Linear Regressiono Minimize Mean Squared Error
• Leave-One-Out Cross-Validation
• Sequential ForwardFeature Selection during training
23 of 30
Model Design
![Page 24: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/24.jpg)
Sequential Feature Selection
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8Improvement of model as SFS adds more
features
Number of Features Used
Mean
Sq
uare
d E
rror (Each line is
a different user)
24 of 30
1 3 5 7 9 11 13 15 17 19 21 23 25 27 290
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8Improvement of model as SFS adds more
features
Number of Features Used
Mean
Sq
uare
d E
rror
![Page 25: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/25.jpg)
Sample Prediction
0 10 20 30 40 50 602
3
4
Mood (Pleasure)Estimated Mood
Days
Dail
y M
ood
Avera
ge
25 of 30
![Page 26: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/26.jpg)
Error distributions• Error2 of > 0.25 will
misclassify a mood label
93% < 0.25 error2
0.0001
0.001
0.01
0.1
1
25%ile
10%ile
Users
Sq
uare
d E
rror
26 of 30
![Page 27: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/27.jpg)
vs. Strawman ModelsModels using full-knowledge of a user’s data with LOOCV
Model A: Assume User’s Average Mood73% Accuracy
Model B: Assume User’s Previous Mood61% Accuracy
MoodScope Training: 93% Accuracy.
27 of 30
![Page 28: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/28.jpg)
Personalized Training
10 20 30 40 50 590%
20%
40%
60%
80%
100%
Incremental personalized model
Training Days
Mod
el
Accu
racy
28 of 30
All-user modelaccuracy
![Page 29: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/29.jpg)
Personalized/All-userHybrid Training
10 20 30 40 50 590%
20%
40%
60%
80%
100%
Incremental personalized model
Hybrid mood model
Training Days
Mod
el
Accu
racy
29 of 30
![Page 30: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/30.jpg)
Phone
Mood Inputs/Usage Logs
Mood and Usage History
Cloud
Mood ModelMood Model
Current Usage Model
Training
Inferred Mood
Resource-friendly Implementation
30 of 30
![Page 31: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/31.jpg)
Inferred Mood
API
![Page 32: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/32.jpg)
TEXAS MEDICAL CENTER
RICE UNIVERSITY
![Page 33: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/33.jpg)
Inferred Mood
API
![Page 34: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/34.jpg)
MoodScope:Sensing mood from smartphone usage
patterns
• Robustly (93%) detect each dimension of daily moodo On personalized modelso Starts out with 66% on generalized
models
• Validate with 32 users x 2 months worth of data
• Simple resource-friendly implementation
![Page 35: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/35.jpg)
35
Discriminative Features
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8Improvement of model as SFS adds more
features
Number of Features Used
Mean
Sq
uare
d E
rror
Relevant features
![Page 36: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/36.jpg)
36
Discriminative Features
Calls
Emai
lSM
SW
ebApp
s
Loca
...
Prev
....
0
20
40
60
80
100
120 Pleasure
Activeness
Nu
mb
er
of
Featu
res
![Page 37: M oodScope : S ensing mood from smartphone usage patterns](https://reader035.vdocument.in/reader035/viewer/2022062422/5681364b550346895d9dc958/html5/thumbnails/37.jpg)
37
TODO• Wider, longer-term evaluation
o How does the model change over time?
• In-use accuracy metricso Not cross-validation
• Social Factors/Impacto Study Mood-sharingo Provide assistance to psychologists