![Page 1: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/1.jpg)
Building anAware Home
Irfan Essa
Aware Home Research InitiativeGVU Center / College of Computing Georgia Institute of [email protected]
![Page 2: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/2.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Research Goal
How can your house help, if it is aware (of your whereabouts, activities, needs, intentions, etc.)?
![Page 3: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/3.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Important Goals: Ubiquity
Sensing and output technology that is transparent to everyday activities. Passive Anywhere, anytime input/output. Provide an ability to sense, interact, display information, communicate, without increasing burden/load on users. Aware of residents, sense them!
who, what, where, why? (W4) noninvasive, unobtrusive, perceptual, ubiquitous,
natural interface
![Page 4: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/4.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Sense, Measure, Monitor?
Issues of location: Where are people?Identity: Where are which people?
What about new people?
Local action “Sitting/Getting up”, “Climbing stairs”, “Washing
dishes”, “Reading book”, etc.Extended action
“Eating a meal”,”preparing a meal”, Really extended action
“Change of mobility”, “eating well”
![Page 5: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/5.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Good hard perceptional problems From a perception standpoint, sensing in the Aware Home demands the solution to several classes of fundamental problems:
Sensing user state Understanding user activity Noticing variation over longer time scales
“trending”, “routines”
… a really good set for Computer Vision researchers.… but vision may not be (is not) enough.… a sensor fusion, sensor interpretation problem.
![Page 6: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/6.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
So what form of sensing?
Typical, do-able but: “Grandma fell down and didn’t get up” Why not: Because if that is all you want to do, there
are better, cheaper, more reliable ways (though the failure modes need to be designed well).
Tracking is STILL HARD!Many other sensors can be pervasive but …
If you have a vision infrastructure, and basic primitive capabilities, every new task is not a re-engineering job.It can help focusing on some important event (context)
![Page 7: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/7.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Practical Indoor Sensing
RF ID instrumentation
Floor mats
Below-knee tags
Room-level positioning
Can other sensing build on top of this?
![Page 8: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/8.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Vision infrastructure
20+ Fixed Cameras (Analog & Digital *IEEE 1394*)
16+ PIII PCs (2 cameras / PC)
8 Pan-Tilt-Zoom Cameras
Stereo and other special purpose cameras
![Page 9: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/9.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Vision-based tracking methods
Background Segmentation / Modeling. Color Histograms / Segmentation. Template / Appearance Modeling & Matching. Motion Integration. Calibration, Perspective Modeling. Sensor Fusion (between cameras and other sensors). Learning Methods Client-server Architecture for distributed Processing.
![Page 10: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/10.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Tracking from ceiling sensors
A person is tracked and his activities are reported on the map.
![Page 11: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/11.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Tracking from Above
![Page 12: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/12.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Room mapping2D descriptionsOverlapping cameras
![Page 13: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/13.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
The Gesture Pendant
Wear sensors looking outwards. (1st vs. 2nd vs. 3rd person perspective).
Simplified home control
Biometrics, biomedical, etc.
Starner et al.
![Page 14: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/14.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Eye/Pupil Tracking
![Page 15: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/15.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Audio Sensors
Speech recognition. Augment interaction. Tracking / identification. Affect Determination (anger, stress, sadness, happiness). Noise cancellation. Acoustic Modeling.
![Page 16: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/16.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Auditory Localization I
Phased Array MicrophonesLocalize a speaker and move a pan-tilt-zoom camera to their face4 - 8 microphone systemVision can help with face tracking Sensor-fusion
![Page 17: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/17.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Auditory Localization II
Adaptive Array ProcessingDetermine Time Delay of Arrival (TDoA) to determine source.59 microphone arrayInteraction with NIST (Vince Stanford).
![Page 18: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/18.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Video-based Tracking
Cameras
![Page 19: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/19.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
System Architecture
Video
Locations
Camera 1(Fixed)
Camera 2(Fixed)
ColorTracking
ColorTracking
MotionTracking
MotionTracking
Calibrated
Video
Camera 3(PTZ)
Camera 4(PTZ)
ColorTracking
BeamFormer
FaceTracking
AuditoryLocalization
FaceTracking
Video
Video
More Sensors More Sensors
RoomManager
FaceRecog.
![Page 20: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/20.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Occupancy Grid
![Page 21: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/21.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Combining Sensors
Map of the Room, showing sensors and 2 residents in the room
Visual tracking of a resident
Visual identification of a residentPaper appears in PUI 2001
![Page 22: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/22.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Multi-modal tracking
![Page 23: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/23.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
What Was I Cooking?
Mynatt, Abowd, et al.
![Page 24: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/24.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Video
![Page 25: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/25.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Behavior AnalysisDetection Detection
Behavior AccuracyBehavior AccuracyLow-Risk 92%High-Risk 76%Novice 100%Expert 90%
After ~10 trials per person
![Page 26: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/26.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Routine Activities
share a set of component tasks identify a subset of tasks and measure the demand for the performance of such tasks model and predict successful and independent performance of an activity (Clark, Czaja, & Weber, 1990; Connell & Sanford, 1997; Sanford, Story, &Ringholz, 1998).
![Page 27: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/27.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Routine Household Activities
Activities of Daily Living (ADLs) [dressing, bathing, etc.] Instrumental Activities of Daily Living (IADLs) [house cleaning, laundry, cooking]. Enhanced Activities of Daily Living (EADLs). ADLs, IADLs, and EADLs can potentially be aided by Aware Environments.
![Page 28: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/28.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Face of the House!
PS. Did some facial expression recognition earlier, ask Sandy.
![Page 29: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/29.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Finally, the Context. We need it.
How do we represent what: really the heart of the question What is context? Helps define the target vocabulary of sensing and perception, and input information for decision making.
NEED Experts. Software Engineering: inflow, synchronization, storage, access, delivery (e.g. Context Toolkit, Abowd et al.)
![Page 30: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/30.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Ethical Issues
These visions concern some people (as they should!).
For example, with automated capture:who controls and distributes capture?what about silent and intimidated minority?
Educate & confront Policy
![Page 31: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/31.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
More!
We are interested in building useful (important) “Living Laboratories” (and learning how to build them too). We will build, test, evaluate, and rebuild. “This Aware House.” See:
www.cc.gatech.edu/fce/ahri Email:
[email protected], [email protected] Others
Gregory Abowd, Beth Mynatt, Wendy Rogers, Aaron Bobick, Thad Starner, many UG, MS, PhD students and Research Scientists.
![Page 32: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/32.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
“blob” management b/w clients
![Page 33: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/33.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
All the same person?
![Page 34: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/34.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Natural tasks for vision
Location refinementNon-location determined activity
“Couch potatoes” Basic activity
Contextual-triggers ”Preparing to leave the house” Lots of potential features
Statistical characterization “Slower going up the stairs this week than
last”
![Page 35: Building an Aware Home Irfan Essa Aware Home Research Initiative GVU Center / College of Computing Georgia Institute of Technology irfan@cc.gatech.edu](https://reader035.vdocument.in/reader035/viewer/2022062500/56649e845503460f94b85600/html5/thumbnails/35.jpg)
© Irfan Essa and Georgia Institute of Technology, 2001.
Keeping track of blobs
Overhead cameras These are not plan view cameras, but require
mapping (calibrate if desired). A messy house is not a lab – much less control.Integrate according to 2.1D location – really foot location in plan view by simple learning.The (dreaded) N-to-M problem:
Temporal integration on appearance Probabilistic assignment Finite look ahead and look behind