overview of life logging
DESCRIPTION
Overview of Life Logging. September 4, 2008 Sung-Bae Cho. Agenda. Life logging Context-aware computing Sensory data for activity recognition Life logging with mobile devices Summary. Advances of Digital Devices. Right now, it is affordable to buy 40 GB (in 2003) - PowerPoint PPT PresentationTRANSCRIPT
Overview of Life Logging
September 4, 2008
Sung-Bae Cho
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
0
0.2
0.4
0.6
0.8
1
2002 2003 2004 2005 2006 2007
Dis
k ca
pac
ity
(TB
)
Source: Microsoft 2003
Advances of Digital Devices
Right now, it is affordable to buy 40 GB (in 2003)
In 3 years 1TB/year is affordable!
It is hard to fill a terabyte/year, but you can:Look at 9,800 pictures a day (300 KB JPEGs)
Read 2,900 documents a day (1MB files)
Listening to audio or view compressed video 24 hours/day (it takes more than 256 kb/s to fill a TB in a year)
Watch 1.5 Mb/s video 4 hours each day.
3/48
4/48
Digital Convergence
Properties of Mobile Devices
Multi-function Mobility
Large Memory Personal Device
Collect Various Log Data of
User’s Everyday Life
Advances of Mobile Devices
Everyday Life with Mobile Devices
I’m always I’m always with you!with you!
5/48
Available Information from Mobile Devices
Photo
Where is the place?
Who is in the media?
Video
How many people in the media?
User Created Contents
What is the activity?
Sensor Log
Bluetooth
Who is nearby?
Bio-sensor
Sleep? Activity Level?
GPS
Location
Vision Sensor
Contexts of environments
Audio
Noisy level?
User Data
Scheduler Personal Profile
Details on daily events Preference
Human Relationships
Demographical information
Address Book
Usage Log
Call Log
DurationReceiver/Caller
Time
SMS Log
ContentsReceiver/Sender
Time
Application Usage
Mp3 Player
6/48
Life(Experience)
DigitalInformation
Life Record
Context DataAlways Bringing Contents Creation
Human Memorize Experience
Life Logging with Personal Digital Devices
Personal Digital Device
7/48
Necessity of Context-aware Computing
Integration of Various Typeof Information
User Modeling fromPersonal Information
Data Mining fromPersonal Information
Context-aware Computing
Personal Information Retrievalbased on Semantic Structure
Episodic Management ofPersonal Information
Summary ofEveryday Life
8/48
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
Definition of Context-aware Computing
• Context : Dey & Abowd (1999)
– Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, physical or computational object
• Context-aware computing
– The use of context to provide task-relevant information and/or services to a user, wherever they may be
• Three important context-aware behaviors
– The presentation of information and services to a user
– The automatic execution of a service
– The tagging of context to information for later retrieval
• Importance of context
– The context of user is changed frequently and drastically in ubiquitous and mobile environment (Pascoe, et al., 1998)
10/48
Trends on Context Awareness
• TEA: EU project for enabling context awareness
– 1998 ~ 2000, with TeCo, Starlab, Omega and Nokia
– scenarios, technologies, market research and demonstrators
• DrWhatsOn concept project
– concept and user scenarios for context aware office PDA device
– main focus in usability and user interface design
• Earlier the focus was in sensor-based context recognition
– recognize and utilize information from user's activity and environment properties
• Now more directions and possibilities exist
– Context-aware computing
– Context-aware communication
– Context-aware services, connectivity, information retrieval, affective computing, etc.
TEA
TEA
DrWhatsOn
11/48
Why Context Awareness is Difficult?
• Context recognition is never 100% reliable
– Contexts are vague, dynamic, overlapping, and ill-defined
• Adaptive user interfaces are scary!
– May need an adaptive, learning SW
• Sensors & algorithms may need constant recalibration
– They may also be too CPU intensive
• Application development frameworks & support are missing
– Security, privacy concerns a lot
• Connectivity to environment
– Limited I/O through wireless links
– What short range connectivity technology should be used?
• Productization
– Where’s the business?
– What everyday problems it really solves?
– Where do we get sensors for sensor-based context recognition?
12/48
Some Technical Challenges
• Understanding structure and behavior of context information
– context are fuzzy, overlapping, and changing in time
• Lack of sensing methods for context information from various sources
• Lack of methods for fusing context information
• Lack of format of context information
• Sensor-based context recognition
– hard to obtain reliable data
– signal processing and recognition consume memory, energy and processing time
– results may be ambiguous
• Connectivity to environment and other devices
– Are profiles available; how about location information?
– Any privacy & security risks?
13/48
Why do we Need Context Awareness?
Maybe avoid disturbing people
at wrong times
Right information, right time, right place
Receding to background
(Calm computing)
New services
(Location-based services are the obvious ones)
Increased user
satisfaction
Better social acceptance
of technology
14/48
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
MIT Media LAB (1)
• Area: Visual Contextual Awareness in Wearable Computing (1998)
• Sensor: Vision
• Probabilistic object recognition
– Probabilistic dependence analysisbased on neighbor vector & object recognition
• O: object, M: measurement
– Task recognition with HMM
16/48
MIT Media LAB (2)
• Activity recognition based on accelerometer (2004)
– 20 activities
• Sensor: Accelerometer on diverse points of body
• Geometric analysis
– Mean, energy, frequency-domain entropy, and correlation
• Classification method
– Decision Tree C4.5, IBL, Naive Bayes
17/48
MIT Media LAB (3)the number of subjects collected under laboratory (L) or naturalistic (N) settings
18/48
MIT Media Lab (4)
• Aggregate confusion matrix for C4.5 classifier
– leave-one-subject-out validation for 20 subjects
19/48
eWatch Sensor Platform
• CMU Computer Science Lab in 2005– Defense Advanced Research Projects Agency (DARPA)
• Activity recognition, improving power consumption, location recognition
• Hardware
– LCD, LED, vibration motor, speaker,Bluetooth for wireless communication
– Li-Ion battery with a capacity of 700mAh
• Sensors
– a two-axis accelerometer (ADXL202; +/- 2g)
– Microphone, light & temperature sensors
• Method
– multi-class SVMs
– HMM based selective sampling
20/48
Classifier
• multi-class SVMs with Gaussian Radial Basis Function kernels
– frequency spectrum-based classification
– time-domain-based classification with SVM
• means, variances, square root of the uncentered second moment, the median absolute differences
21/48
University of Alberta
• National ICT Australia Project: University of Alberta, Canada
• Human activity & gesture recognition
• Sensor
– active, magnetic field, acoustic, laser, camera sensor
• Method
– Coupled hidden Markov model (CHMM)
• Extended HMM model for combining and utilizing concurrent information stream more effectively (By M. Brand et al., 1997)
– M. Brand, N. Oliver, and A. Pentland, “Coupled hidden Markov models for complex action recognition,” in IEEE Intl. Conf. Comp. Vis. Pat. Rec., pp. 994-999, 1997.
22/48
University of Bologna
• Micrel Lab: University of Bologna, Italy (2004)
• Research
– Construction of Ubiquitous environments
– Sensor-based gesture recognition
• Sensor
– Wireless MOCA (Motion capture with integrated accelerometers) sensor
• Accelerometer, gyroscope
• Small size, small power cosume
• Wireless operation
• Sticking on diverse points of human body
• Method
– Hidden Markov Model
23/48
Activity Recognition Summary
• Mostly focused on static classification for pose recognition (relatively easy)
USAMIT
FinlandNokia
JapanATR
Aus. Curtin Univ.
SwissETH
Zurich
BelgiumStarlab.
ChinaSJT Univ.
CanadaCMU
Static classification ○ ○ ○ ○ ○ ○ ○
Temporal classification
○ △
Accelerometer ○ ○ ○ ○ ○ ○ ○ ○
Vision sensor ○ ○
Instant behavior ○ △
Repeated behavior (Pose)
○ ○ ○ ○ ○ ○ ○
Geometric calculation
○ △ ○ △ △ △ ○ △
HMM △ ○ △
SOM ○
24/48
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
Context-aware Mobile Device Trends
• Increasing demands for the multi-functional and high-performance phone
• Prospects
– Increasing demands for Smart Phone
– Decreasing basic functional phone
• Enlarging role as a computing equipment
– Internet surfing
– External storage
• Increasing investment for adding value of phone
• Digital convergence with other functionality
– Mp3 player
– Personal Media Player
– Camera & Camcoder
26/48
Context-aware Phone on Media
The official newsletter of the institute for Complex Engineered Systems (CMU) Sep/Oct 2003
New Scientist, Nov 2004 MIT Sloan Management Review Fall 2004
27/48
SenSay (Sensing and Say)
• Carnegie Mellon University (2003)
• A context-aware mobile phone
– Adapting to dynamically changing environmental and physiological states
– Manipulating ringer volume, vibration, and phone alerts
– Providing remote callers with the ability to communicate the urgency of their calls
– making call suggestions to users when they are idle
– Providing the caller with feedback on the current status of the Sensay user
• Sensors
– accelerometers, light, and microphones
– mounted at various points on the body
28/48
SenSay (2)
• Context recognition by thresholding
29/48
SenSay (3)
• Context classification based on self-organizing map
30/48
MIT Reality Mining Group
• Utilizing Context application from the University of Helsinki (Raento et al., 2005)
• Capturing mobile phone usage patterns
– from one hundred people (MIT students)
– for an extended period of time
• Providing
– insight into both the users
– the ease of use of the device itself
• Method: Bayesian inference
31/48
Nokia
• Nokia 7650 (2002)
– Backlight time adjustment based on proximity & light sensor
– Speaker-phone mode change by proximity sensor
• Cancellation of speaker-phone mode near ear
• Nokia 6230, 6820, 7200
– Presence-enhanced chat service
• Presenting and sharing the user’s status
– SMS messaging based on other’s status
Light SensorProximity Sensor
32/48
Microsoft Research (1)
33/48
Microsoft Research (2)
• Alarm method for incoming call
– Quiet ringing
• Volume down with hand touching
– Acknowledging and ignoring calls
• Calling response: Tilting to body of phone on the pocket
• Notification stop: Holding phone with hand without movement on pocket
– Target device for notifications
• Device selection for alarm: Selecting the recent device if there are several devices of user
– Vibration notification
• Vibration mode change: Holding phone for a long time
34/48
Tilt Sensor & User Behavior
• Upper: forward/back tilt
• Below: left-right tilt
35/48
VTT Electronics (Finland)
• Supported by Nokia
• VTT Electronics – Advanced Interaction Systems – Context Awareness
• Fuzzy/Bayesian Approach
Summary of Context Recognition Procedure(From VTT Publications 511)
Backlight level & font size adjustment
36/48
VTT Technical Research Center of Finland
• Context representation Fuzzy logic
• Context reasoning Naive Bayes, Markov Chain
• Service Fuzzy Control
Context OntologyFuzzification
37/48
VTT Technical Research Center of Finland (2)
• Sensor based contexts
– Bottom: high-level contexts
38/48
VTT Technical Research Center of Finland (3)
• Audio based context
– Bottom: high-level context
39/48
VTT Technical Research Center of Finland (4)
• Naive Bayes based classification
40/48
TEA Project
• High-level context recognition
– Method: Rule & SOM http://www.teco.edu/tea
41/48
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
Summary
• Key components for context-aware applications (life logging)
– Sensor technology to acquire various contextual information
– Intelligence technology to model complex contexts
– Agent technology to provide seamless services
• Future requirements
– Context modeling and cognitive technologies to provide users with more advanced services
43/48
DrWhatsOn (Laerhoven, 2003)
• Concept and user scenarios for context aware office PDA device
• Main focus in usability and user interface design
– Information fusion: Sensor data, device status, personal preference, schedule
– Peripheral attention: supporting appropriate information at appropriate time under prepared conditions
• Scenario for a context sensitive phone of Nokia
– A day of a Finland student Dude
44/48
DrWhatsOn Scenario (1)
• [Nokia’s DrWhatsOn Concept Video, Urpo Tuomela, Nokia Research Laboratories]
– Red mark: Abstracted information from sensor
45/48
DrWhatsOn Scenario (2)
46/48
DrWhatsOn Scenario (3)
47/48
DrWhatsOn Scenario (4)
48/48
Homework #1: Due 9/9
• Survey the state-of-the-art research on the life logging at KIST, Nokia, Microsoft and MIT (10 min presentation per each institute)
– KIST
– Nokia
– Microsoft
– MIT
49/48