recognizing daily routines through activity spotting ulf blanke and bernt schiele computer science...
TRANSCRIPT
Recognizing Daily Routines Through Activity SpottingUlf Blanke and Bernt Schiele Computer Science Department, TU Darmstadt
Activity Recognition
Human Activity Recognition support awareness of applications (human to application)
support analysis of human activity (human to human)
Application Areas Healthcare: long term monitoring of patients (months!)
Elderly care: personal diary (weeks to months)
Industrial: workshop activities (minutes to a hours)
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 2
Towards High Level Activities
Low Level Activities E.g., walking, standing, biking… Lasting from seconds to minutes Detected by pose or characteristic motion.
High Level Activities E.g., daily routines: morning routine, dinner, working… More important for many domains (e.g., healthcare) Lasting from minutes to hours Consist of multiple low level activities:
approaching car(walking)
driving leaving car(walking)
commuting
preparing food eating doing dishes
dinner
… … … …
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 3
Related WorkMultilayer Approaches
commuting, working, lunch, dinner…
High-levelactivities
Low levelactivities
Sensor data
Zhang, D., Gatica-Perez, D., Bengio, S., McCowan, I., Lathoud, G. (CVPR 2004)Clarkson, B., Pentland, A. (ICASSP 1999)Oliver, N., Horvitz, E., Garg, A.: (Multimodal Interfaces 2002)
Mid-levelactivities
Morning Routine
wash
undressing drying
Cleaning teeth
scrub
e.g. walking, running, standing, sitting
shower Putting toothpaste
drying
. . . . . .
- Many parameters- Computationally intensive
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 4
Related WorkDirect Approach
commuting, working, lunch, dinner…
High-levelactivities
Low levelactivities
Sensor data
Mid-levelactivities
e.g. walking, running, standing, sitting
Huynh, T., Blanke, U., Schiele, B. (LoCA 2007)
Morning Routine
- High level activities exhibit high inner-class variability- All Data has to be considered
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 5
Can we learn distinctive parts of high level activities?
Can we reduce the amount of data important for recognition?
Research QuestionsActivity Spotting
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 6
Research QuestionsActivity Spotting
LunchHigh-level activities
Low level activities
Sensor data
Which low level parts are important for high level activities?
Automatic selection
Dinner
walking picking up food
eatingPrepfood
eating Doing dishes
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 7
Research QuestionsActivity Spotting
LunchHigh-level activities
Low level activities
Sensor data
Recognizing high level activities by activity spotting feasible?
Dinner
walking picking up food
eatingPrepfood
eating Doing dishes
Activity Spotting
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 8
Method
High-level activities
Low level activities
Sensor data
Doing dishes
K-means clusters
Joint boosting
Feature-Calculation
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 9
Low Level Activity Selection (Joint)Boosting
(1) Combination of low level activities to infer high-level activities
(2) Automatic Selection of most discriminative low level activities
(3) Sharing features (i.e. low level activities)
across high level activities
+
others
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 10
Experiment
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 11
Experimental SetupEvaluation Metrics
Quantitative Analysis
Tradeoff between precision & recall and
number of low-level selected activities?
how much data is needed (occurrence of activities used)?
Qualitative Analysis
Which activities are used – do they make sense?
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 12
7 days of a life from a single person[Huynh, T. - Ubicomp ‘08]
Two layers of annotation 4 high level routines, more than 20 low level routines
Experimental SetupDataset
Wrist
Commuting Commuting
Working Working Dinner
Lunch
2 acceleration sensors
walking
standingin line
having a coffee
Lunch
walkingeating
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 13
High-level activities
Low level activities
Sensor data
Doing dishes
Feature-Calculation
Experimental SetupFixed Parameters
K-means clusters
Joint boosting
Mean and Variance - over 0.4s window- on (x,y,z)-acceleration- of pocket and wrist
Histograms- over 30min
window
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 14
High-level activities
Low level activities
Sensor data
Doing dishes
Feature-Calculation
Experimental SetupVaried Parameters
K-means clusters
Joint boosting
Jointboosting- rounds- Routines’ annotation
Kmean centers- soft and hard- K = 60
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 15
Quantitative Results
Soft Assignments
Rounds 4 10 80 Huynh 08
Precision in % 82.85 82.98 87.81 76.90
Recall in % 83.67 88.12 90.17 65.80
lowlevel activities in%
5.20 11.39 57.93 -
how much data? In % 12.78 17.74 74.30 -
4 10 80
72.71 77.34 86.40
82.67 82.39 90.32
5.19 12.14 50.82
2.11 4.94 45.42
Hard Assignments
Number of lowlevelactivities (clusters)
How much data
Rounds Rounds
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 16
Tradeoff
ResultsClassification scores for one day
Reducing number Low level activities
Precision lossat borders
Sco
res
80 rounds
10 rounds
4 rounds
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 17
Dinner Commute Lunch
sitting/desk activities (47.24%)
driving car (32.90%),
driving car (21.71%)
Time
walking (99.23%)
sitting / desk activities (97.86%)
walking (96.09%)
driving bike (47.86%) walking (22.51%) picking up food (16.81%)
queuing in line (43.86%) picking up food (14.59%)
driving bike (16.76%)
sitting/desk activities (31.20%)
36
6
42
48
29
13
53Lunch WorkCommuteDinner
Time
Time
Distribution of low level label for each selectedlow level cluster
Qualitative AnalysisWhich activities are used?
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 18
Dinner Commute Lunch
sitting/desk activities
driving car
driving car
Time
walking
sitting / desk activities
walking
driving bike walking picking up food
queuing in line picking up food
driving bike
sitting/desk activities
36
6
42
48
29
13
53Lunch WorkCommuteDinner
Time
Time
Distribution of low level label for each selectedlow level cluster
Qualitative AnalysisWhich activities are used?
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 19
Summary & Conclusion
Can we learn distinctive parts of high level activities? Yes.
Automatic Selection of important data
Top down perspective Find discriminative parts of a high level routines
Can we reduce the amount of data used for recognition? Yes.
Fraction (~5-8%) of data sufficient to recognize daily routines (~80%) Filter insignificant data reduce memory usage and computational costs suited for embedded long term activity recognition
Activity Spotting feasible for routine recognition.
Outperforms previous generative approaches on this dataset [Huynh - Ubicomp 08]
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 20
Thank you for your attention.Questions?
ご清聴、ありがとうございました。ご質問はありますか。
Dataset available at www.mis.tu-darmstadt.de
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 21
Date 08.05.2009 | Department of Computer Science | Ulf Blanke | 22
End of Presentation
(Joint)BoostingWeak Classifier
b: confidence, that sample is not part of classa: confidence, that sample is part of class
weak classifier
a b
Total confidence of separation
Strong classifier
Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 23