slice&dice: recognizing food preparation activities using embedded accelerometers cuong pham...
TRANSCRIPT
Slice&Dice: recognizing food preparation activities using embedded accelerometers
Cuong Pham & Patrick OlivierCulture Lab
School of Computing ScienceNewcastle University
Overview
Introduction Instrumented utensils Activity Recognition framework Experiment
Data collection & annotation Evaluation
Reflections
Introduction: Ambient Kitchen project
Goal: help people with dementia live more independent by providing situated services and prompting based on context recognition
Kitchen context: what people are doing objects people are interacting (i.e. food
ingredients) user locations etc.
Introduction: Ambient Kitchen project
Ambient kitchen: a lab-based ambient intelligence environment, designed using high fidelity prototype.
Introduction: prior work
Sensors worn on different parts of users body [Bao2004, Tapia2007, Ravi2005].
Detected outdoor activities such as running, walking, climbing, cycling etc. or high level activities[Wu2007]
Data collected under laboratory [Ravi2005] or semi-realistic conditions [Bao2004]
People with dementia needed fine-grained prompts to complete low-level activities [Wherton2008]
Introduction: system requirements
Detect low-level activities Sensors hidden from users No wires The cost & ease of deployment Comfortable-to-use Reasonable accuracy
Instrumented utensils: Wii ADXL330
A thin, low power, 3-axis accelerometer
Signal conditioned voltage outputs Dynamic acceleration can be
measured motion, shock and vibration
Acceleration can be measured in a range of ±3g
Instrumented utensils
Modified Wii Remotes were embedded in the kitchen utensils
Activity Recognition Framework
Data Communication & Processing acceleration data X, Y, Z sent to the computer
through a bluetooth device pitch and roll were computed for each triple
X,Y,Z Data Segmentation
data stream were segmented into 32, 64, 128, 256, and 512 sample windows
50% overlap between two consecutive windows.
Activity Recognition Framework
Feature Computation Mean Standard deviation Energy Entropy
Classification algorithms (from Weka Lib) Decision Tree C4.5 Bayesian Networks Naïve Bayes
Experiment: data collection
20 subjects 5 IP cameras 4 utensils: 3 knives and one serving spoon Given ingredients: potatoes, tomatoes,
lettuce, carrots, onions, kiwi fruit, grapefruit, peppers, bread, and butter
No instruction and no time-constrained to the subjects
Task: prepare a mixed salad and sandwich
Experiment: data annotation
Collected videos were annotated using Anvil Multimodal Tool [Kipp2001]
Experiment: example
Experiment: example
Experiment: data annotation
Dataset B annotated by one coder
Dataset A independently annotated by three coders only regions where all there coders agreed
were extracted Dataset B is larger than dataset A, but
dataset A is more consistent than dataset B
Experiment: subject independent evaluation
Trained 19 subjects Tested the remaining one Repeated the process for 20 subjects Finally, aggregated the results Subject to test was not included in the
training dataset
Experiment: evaluation results
Algorithm Dataset A Dataset B
Decision Tree 82.9 77.2
Bayesian nets 78.9 71.3
Naïve Bayes 52.4 73.5
Best accuracies were achieved on window size of 256-sample
Experiment: evaluation analysis
Peeling and stirring were highly distinctive (more than 90%)
Chopping, slicing, coring, scooping performed really good (around 80-90%)
Eating, spreading, shaving, scraping and dicing were below 80%: eating sometimes misclassified as scooping spreading sometimes misclassified as shaving
and coring dicing often misclassified as chopping
Reflection
Low-level food preparation activities can be reliably recognized using sensors embedded in kitchen utensils
Our work will continue with finding features most impact on algorithm
performance detecting objects developing Models
Thank you for your attention
Q&A