1

Activity Recognition

Taiwoo ParkMay 7, 2013

Bao, Ling, and Stephen S. Intille. "Activity recognition from user-annotated acceleration data." Pervasive Computing. Springer Berlin Heidelberg, 2004. 1-17.Park, Taiwoo, et al. "E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices."Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems. ACM, 2011.

Some slides are from CSCI 546 course materials by James Reinebold, USC

2

Activity?• Higher level activities

– Giving a lecture, having a breakfast, playing soccer…

• Lower level activities– Lying on a bed, standing still, running, walking, …

3

An easy example• Assumptions:

– Sensors on smartphones are only available• Accelerometer, compass, gyroscope, light, …• You can attach smartphones on your body

– Only three target activities to recognize• Running• Standing still• Lying on a bed

• How can we recognize activities?

4

An easy example (cont’d)

Is the phone being shaken?

Phone orien-tation Activity

No Upright

Yes Upright

No Lying down

Yes Lying down

Variance of accelerometer sensor signalfor the last 3 seconds

Average value ofaccelerometer y-axis sensor signals

for the last 3 seconds

y

xz

Standing still

Running

Lying on a bed

Nothing

5

Activity recognition pipeline

Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

6

An easy example (revisited)

Is the phone being shaken?

Phone orien-tation Activity

No Upright Standing still

Yes Upright Running

No Lying down Lying on a bed

Yes Lying down …?

Variance of accelerometer sensor signalfor the last 3 seconds

Average value ofaccelerometer y-axis sensor signals

for the last 3 secondsWindowing

Feature extraction

Classification

Data collection• Semi-Naturalistic, User-Driven

Data Collection– Obstacle course / worksheet– No researcher supervision while

subjects performed the tasks

• Timer synchronization• Discard data within 10 seconds of

start and finish time for activities

12:30 – 12:50

Walking

…

Bao, Ling, and Stephen S. Intille. "Activity recognition from user-annotated acceleration data." Pervasive Computing. Springer Berlin Heidelberg, 2004. 1-17.

Activities• Walking• Sitting and Relaxing• Standing Still• Watching TV• Running• Stretching• Scrubbing• Folding Laundry• Brushing Teeth• Riding Elevator

• Walking Carrying Items• Working on Computer• Eating or Drinking• Reading• Bicycling• Strength-training• Vacuuming• Lying down & relaxing• Climbing stairs• Riding escalator

Data collection

Source: Bao 2004

Sensors Used• Five ADXL210E accelerometers (manufactured by

Analog Devices)– Range of +/- 10g– 5mm x 5mm x 2mm– Low Power, Low Cost– Measures both static and dynamic acceleration

• Sensor data was stored in a memory card by using “Hoarder Board”

Source: http://vadim.oversigma.com/Hoarder/LayoutFront.htm

Example Signals

Source: Bao 2004

12

Activity recognition pipeline

Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

13

Classification

‘Running’ ‘Standing still’

‘Lying on a bed’ ‘Walking’

Data sample

Collected data samples (in advance)

Question: What is the most similar samples to the current one?

Methods: Naïve bayes, nearest neighbor, decision table/tree,

HMM (Hidden Markov Models), …

?

14

Decision Table

Is the phone being shaken?

Phone orien-tation Activity

No Upright

Yes Upright

No Lying down

Yes Lying down

y

xz

Standing still

Running

Lying on a bed

Nothing

Decision Trees• Make a tree where the non-leaf nodes are the

features, and each leaf node is a classification. Each edge of the tree represents a value range of the feature.

• Move through the tree until you arrive at a leaf node• Generally, the smaller the tree the better.

– Finding the smallest is NP-Hard

Source: http://pages.cs.wisc.edu/~dyer/cs540/notes/learning.html

Decision Tree Example

Phone Orien-tation?

Not lying down: Is the phone being

shaken?

Running Standing still

Lying on a bed

Lying downUpright

NoYes

Nearest Neighbor• Split up the domain into various dimensions, with

each dimension corresponding to a feature.• Classify an unknown point by having its K nearest

neighbors “vote” on who it belongs to.• Simple, easy to implement algorithm. Does not work

well when there are no clusters.

Source: http://pages.cs.wisc.edu/~dyer/cs540/notes/learning.html

Nearest Neighbor Example

Average value of accel Y

Varia

nce

of a

ccel

Lying on a bed Standing still

Running

Naïve Bayes Classifier• Multiplies the probability of an observed datapoint

by looking at the priority probabilities that encompass the training set.– P(B|A) = P(A|B) * P(B) / P(A)

• Assumes that each of the features are independent.• Relatively fast.

Source: cis.poly.edu/~mleung/FRE7851/f07/naiveBayesianClassifier.pdf

20

Activity recognition pipeline

Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

Feature Extraction• Time-domain features [Maurer 2006]

– Mean (average), Root Mean Square, Variance, …

• FFT-based feature computation [Bao 2004]– Sample at 76.25 Hz– 512 sample windows (about 6.71 sec)– Extract mean energy, entropy, and correlation features

Maurer, Uwe, et al. "Activity recognition and monitoring using multiple sensors on different body positions." Wearable and Implantable Body Sensor Networks, 2006. BSN 2006. International Workshop on. IEEE, 2006.

Source: Bao 2004

Source: Bao 2004

Results

• Decision tree was the best performer, but…

Classifier Classification Accuracy (%, Leave-one-subject-out Training)

Decision Table 46.75 +/- 9.296

Nearest Neighbor 82.70 +/- 6.416

Decision Tree 84.26 +/- 5.178

Naïve Bayes 52.35 +/- 1.690

Per-activity accuracy breakdown

Trying With Less Sensors

Accelerometer (s) Left In Difference in Recognition Activity

Hip -34.12 +/- 7.115

Wrist -51.99 +/- 12.194

Arm -63.65 +/- 13.143

Ankle -37.08 +/- 7.601

Thigh -29.47 +/- 4.855

Thigh and Wrist -3.27 +/- 1.062

Hip and Wrist -4.78 +/- 1.331

With only two accelerometers we can get good performance

Lessons• Accelerometers can be used to affectively distinguish

between everyday activities.• Decision trees and nearest neighbor algorithms are

good choices for activity recognition.• Some sensor locations are more important than

others.• Selecting a feature set is important to increase

recognition accuracy.

28

E-Gesture: A Collaborative Architecture

for Energy-efficient Gesture Recognitionwith Hand-worn Sensor and Mobile Devices

29

Motivation

Smartphone

Mobile Gesture Interaction Framework

Hand-worn Sensors

…

Mobile applications using hand gestures

Wristwatch-typemotion sensor

(Accelerometer,Gyroscope)

Mobility!!!!!

30

Challenge: Energy and Accuracy

• Conventional gesture processing pipeline (for gesture recognition in stationary setting)

Data Sensing GestureSegmentation

(Button,Algorithms)

Classification(HMM, DTW)

GestureSamples(Candidate)

Result

Sensor Mobile Device

Accel

Gyro

ContinuousRaw Data

Gesture ‘A’ or ‘B’or non-gesture

31

Challenge: Energy and Accuracy

Data Sensing GestureSegmentation

(Button,Algorithms)

Classification(HMM, DTW)

GestureSamples(Candidate)

Result

Sensor Mobile Device

Accel

Gyro

ContinuousRaw Data

Sensor: 20hrs (250mAh)

Smartphone: 24hrs 17hrs

Mobility Noises

Continuous

data transmissionEnergyAccuracy

Mobility noisesEnergy

Energy-hungry

Gyroscope

(56%)

Accuracy

Mobility noises

Over 90% False segmentation

Only 70% Classification

32

E-Gesture Architecture

Collaborative Gesture Sensing and Segmentation Classification

(Adaptive and Multi-Situation

HMM)

Gesture Samples(Candidate)

Result(e.g. lay-down)Accel Gyro

Trigger

Adaptation

Wristwatch Sensor Device Mobile Device

1. Device-wise collaborationDetection on wristwatch, classification on smartphone

2. Sensor-wise collaborationAccel turns on gyro for energy efficiency

Gyro adapts accel’s sensitivity for mobility changes

Accelerometer: (+)Energy-efficient, (-)Mobility-vulnerableGyroscope: (-)Energy-hungry, (+)Mobility-robust

Sensor-side Energy Savings

33

46mW

39mW (↓15%)

19mW (↓59%)

59% less energy consumption, 2.4x longer lifetime

250mAhLi-ion

Battery

Continuous sensing + transmission:

20 hrs

Device-wise collaboration(reduced transmission)

23.7 hrs (1.2x)

Device-wise, Sensor-wisecollaboration

(gyroscope power control,reduced transmission)

48.7 hrs (2.4x)Energy Consumption

34

Mobile-side Energy Savings

All processing on mobile:42.1hrs

Device-wise collaboration(reduced transmission)

74hrs (1.8x)

122mW

70mW(↓43%)

Nexus One1400mAh

Li-ionBattery

3G/WiFi on

Energy Consumption

35

Implementation

• Sensor node– Atmega128L MCU– Bluetooth, ZigBee– Sensors

• 3-Axis Accelerometer (ADXL335)• 3-Axis Gyroscope (3 XV-3500CB)• 40Hz Sensing

– Vib motor

• Smartphones– Nokia N96, Google Nexus One– Bluetooth Radio

Google Nexus One Sensor node

Nokia N96 Bluetooth Headset

36

Sample Applications

• Swan Boat [Ubicomp09][MM09][ACE09]– Collaborative boat-racing exertion game– Utilizes hand gestures as additional game input

• Punching together, flapping together

• Mobile Music Player, Phone Call Manager– Featuring eye-free, touch-free controls– User can control the application by hand gestures

Hand-worn Sensors

37

Conclusion• Mobile gestural interaction platform

– Collaborative gesture processing• 1.8x longer battery lifetime for Smartphone• 2.4x longer battery lifetime for Hand-worn Sensor

+ preserving gyroscope’s detection performance

– Mobility-robust gesture classification using HMM• Up to 94.6% of classification accuracy on mobile usage

by mobility-considered classification architecture design

– It will greatly facilitate gesture-based mobile applications

• Provided a novel sensor fusion scheme• Serial fusion + feedback control

– Saves energy + preserves detection accuracy

38