Building an Aware Home: Understanding the symbiosis

between computing and everyday activities

Irfan Essa, Gregory AbowdFuture Computing Environments

College of Computing, Georgia Institute of Technology

www.cc.gatech.edu/fce

© Irfan Essa and Georgia Institute of Technology, 1999

Who are we?

Faculty (Future Computing Env.):– Gregory Abowd, Chris Atkeson, Aaron

Bobick, Irfan Essa, Blair MacIntyre, Beth Mynatt, Thad Starner

~20 PhD students Affiliations: CoC, GVU, BTC, ECE Collaborations

– ECE (Wireless, DSP), Architecture, Rehab Technologies, Psychology, etc.

© Irfan Essa and Georgia Institute of Technology, 1999

Outline

Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-Aware Applications “Aging in place” Conclusion / Discussion

© Irfan Essa and Georgia Institute of Technology, 1999

Research Objectives

Build a living laboratory in an everyday setting that is aware of its occupants’ activities

.. supports the continuous interactions and activities of a small community.

.. understand the usage of such a laboratory as it applies to symbiosis of computing and everyday living.

© Irfan Essa and Georgia Institute of Technology, 1999

Living Laboratory for Human-Home Symbiosis How can we obtain ubiquitous and

continuous connection?– testbed for technologies

How does this change life / living?– information at fingertips– support human communication– build community– unite over distance

tech-centric

user-centric

© Irfan Essa and Georgia Institute of Technology, 1999

An aware home

A home that is aware of its inhabitants and their activities

… can provide support for day-to-day activities

… do so without increasing the load on the inhabitants

… can augment daily functions … provide connectivity In and around the home

© Irfan Essa and Georgia Institute of Technology, 1999

Specific Applications

awareness and connectedness w/ others

augmentation (cognitive, memory) education monitoring security / surveillance Care Facility (Elder, Child, Health, …)

© Irfan Essa and Georgia Institute of Technology, 1999

Where are we?

Georgia Research Alliance (GRA)– ~$600,000

Broadband Telecommunications Center (BTC)

Ground breaking May 1999. Occupancy by Jan. 2000

© Irfan Essa and Georgia Institute of Technology, 1999

South East

Outside

© Irfan Essa and Georgia Institute of Technology, 1999

Basement

© Irfan Essa and Georgia Institute of Technology, 1999

Living floors (2 floors)

© Irfan Essa and Georgia Institute of Technology, 1999

Other “smart homes”

Home automation– X10, – hobby

Many others

www.cc.gatech.edu/fce/seminar/fa98-info/smart_homes.html (Brad Stenger)

MSFT Research (Barry Brummit)

© Irfan Essa and Georgia Institute of Technology, 1999

Outline

Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-Aware Applications “Aging in Place” Conclusions / Discussion

© Irfan Essa and Georgia Institute of Technology, 1999

Research themes

Human-Home Symbiosis

Human-centered Technology-centered

AwareHome

© Irfan Essa and Georgia Institute of Technology, 1999

Technological Challenges - I

Perception Technologies– make the environment aware of the

users and their activities– what is happening?– ubiquitous sensing– interpret (rich) multi-modal streams– long-term vs. short term

© Irfan Essa and Georgia Institute of Technology, 1999

Technological Challenges - II

Ubiquitous Interfaces / Displays– “Off-the-desktop”– Context-aware applications– capture / integrate / access– anytime, anywhere, ease of use, …– diverse resources/media– software infrastructure– multimedia-based

collaboration/interaction

© Irfan Essa and Georgia Institute of Technology, 1999

Technological Challenges - III

Systems & Networking– fast– distributed– secure– adaptive– storage– easy to deploy / configure

• wireless/wireline

– inside, around, and to the home

© Irfan Essa and Georgia Institute of Technology, 1999

User-centric Challenges - I

Understand the needs of the domain– physical house vs. home (familial

connections)• awareness/connectedness with others

– privacy / security– decrease cognitive load– What home activities are

• desirable• can be improved through technology

– Care facility (elderly, young, health)

© Irfan Essa and Georgia Institute of Technology, 1999

User-centric Challenges - II

Elderly home care / assistive healthcare– prolong independence in familiar

surroundings– understand rhythms, patterns,

deviations– provide contact– memory augmentation

© Irfan Essa and Georgia Institute of Technology, 1999

Outline

Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-Aware Applications “Aging in Place” Conclusions / Discussion

© Irfan Essa and Georgia Institute of Technology, 1999

Aware Spaces

Aware environments that know their inhabitants, their preferences, their activities– Who is there? – Where?– What is happening?– How it can best be supported?

© Irfan Essa and Georgia Institute of Technology, 1999

Computational Perception

Signal Interpretation/Coding– Computer Vision,– Audio/Speech,– Tactile / Contact,– RF/IR emitters,– Sonar,– Usage Sensor, ……

Instrument a Space with Sensors

© Irfan Essa and Georgia Institute of Technology, 1999

Perceptual Analysis

Signal Interpretation to determine– geometry, calibration, context– is anyone there?, who?– locate users/people– recognize their actions, activity, gestures,

expressions– speech, non-verbal, communicative

streams Dynamic / Long-term / Interactive

© Irfan Essa and Georgia Institute of Technology, 1999

Sensors (Optical / Cameras)

High-end vs. low-end Task / Resource specific

NEX V25 microprocessor, powerline modem, ...

Analog / Digital Cameras to commercial PCs

Specific hardware solutions

© Irfan Essa and Georgia Institute of Technology, 1999

Experiences

Reconstruction of a Scene Pose Estimation Multiple Camera-Multiple Person

Tracking Context-based Activity / Object

Recognition

© Irfan Essa and Georgia Institute of Technology, 1999

3D models of rooms(Brostow & Essa, ICCV 1999)

Use motion information to model 3D scenes (models from movement).

© Irfan Essa and Georgia Institute of Technology, 1999

Video

© Irfan Essa and Georgia Institute of Technology, 1999

Multiple Cameras(Stillman, Essa, et al., AVBPA 1999)

Track multiple people with multiple cameras

Develop an architecture to support communication between multiple processors/cameras

Combine fixed and PTZ cameras to track and identify people

© Irfan Essa and Georgia Institute of Technology, 1999

Video

© Irfan Essa and Georgia Institute of Technology, 1999

System Architecture

Video

Locations

Camera 1(Fixed)

Camera 2(Fixed)

ColorTracking

ColorTracking

MotionTracking

MotionTracking

Calibrated

Video

Camera 3(PTZ)

Camera 4(PTZ)

ColorTracking

FaceRecog.

FaceTracking

ExpressionGesture

ColorTracking

FaceTracking

PTZ locations

PTZ locations

Video

Video

More Cameras More Cameras

Server

© Irfan Essa and Georgia Institute of Technology, 1999

Pose tracking(Schödl, Essa, PUI 1998, PDPTA 1999)

Use a 3D model of head Extract texture Match texture on model to moving

head (with non-linear optimization) Repeat for every frame Develop distributed/parallelized

implementation

© Irfan Essa and Georgia Institute of Technology, 1999

Video

© Irfan Essa and Georgia Institute of Technology, 1999

Distributed Tracking

GenerateTest

Parameters

EstimateNew

Minimum

Render Head Model

CalculateMatching Error

& Gradient

Camera

ComputeImage Pyramid

ComputeImage Pyramid

Render Head Model

CalculateMatching Error

& Gradient

xt xt+1

Console

Node 1

Node n

4X

Time t

Console + Parallel Nodes, n=7 in our tests

© Irfan Essa and Georgia Institute of Technology, 1999

Speed-up Curves

0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7Nodes

Frames/Second

1x

2x

3x

10x

40x

RelativeFrame Size

Frame rate as a function of Cluster nodes and the size of image (1x = real frame size, 14.4KB).

© Irfan Essa and Georgia Institute of Technology, 1999

Example: Recognizing Activity(Moore, Essa, Hayes, AVBPA 1999 and ICCV 1999)

Develop an framework (architecture) for relating actions and objects

Track the relations between actions and objects for recognition

Use HMMs for temporal recognition

© Irfan Essa and Georgia Institute of Technology, 1999

Video

© Irfan Essa and Georgia Institute of Technology, 1999

Recognition ResultsDomain Actions Objects RRate

Kitchen Stirring, cutting, scrapping,open, close, cleaning,adjusting controls, shaking,washing, drying, etc.

Bowl, cabinet,cutting board, canopener, appliances,pots/ pans, sink, etc.

72-100%

Office Picking up/ down, returning,fl ip f orward/ backward, open,close, drink, type, point, etc.

Bookcase, book,notepad, keyboard,mouse, phone,printer, etc.

80-100%

Car Change gears, hand-brake,adjustments, turn lef t/ right,roll up/ down, drink, etc.

Gearbox, parkingbreak, radio,steering wheel,window, controls,etc.

80-100%

Overall Total actions: 597 ~92%

© Irfan Essa and Georgia Institute of Technology, 1999

Other Projects (not there yet!) Smart Carpet

– Recognizes people based on their footsteps

Audio-visual tracking– analysis of audio & visual-kinetic data– audio-visual tracking

Auto calibration (inside/outside) Other sensors (wearable etc.)

© Irfan Essa and Georgia Institute of Technology, 1999

Outline

Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-Aware Applications “Aging in Place” Discussion / Future

© Irfan Essa and Georgia Institute of Technology, 1999

What is context?

Characterizing a situation Sensed information

Identity, location, activityof people, places, things

Who? Where? When? What? Why?

© Irfan Essa and Georgia Institute of Technology, 1999

Context-aware applications

Present context information to users– Example: fridge informs the user of

what is running out Tailor the interaction according to

context changes– Examples: Activity in kitchen and near

dinner time provide recipe help based on available food and preferences

© Irfan Essa and Georgia Institute of Technology, 1999

Easier said than done!

Designing and implementing such context-aware applications is difficult!

Goal: Provide software infrastructure to support rapid development

© Irfan Essa and Georgia Institute of Technology, 1999

The Context Toolkit (Salber, Dey, & Abowd CHI 1999)

Separation of concerns– context sensing from reaction– insulate sensors and applications from

each other An analogy to GUI development

– separation of presentation and functionality

We want glue between perception and interaction

© Irfan Essa and Georgia Institute of Technology, 1999

Beyond the GUI analogy

Context widgets are distributed– They can be shared by applications

Context widgets are persistent– They store a history of context

information Context widgets may be unreliable

– They must provide confidence factors

© Irfan Essa and Georgia Institute of Technology, 1999

Components

Context widgets– abstraction of a sensor– taxonomy of context types

Interpreters– translation between context values

Entity servers– persistence and aggregation of

context

© Irfan Essa and Georgia Institute of Technology, 1999

Experience

Electronic In/Out Board

Informal capturing whiteboard

Mobile Conference Assistance

Home Monitoring system

More empirical experience needed

© Irfan Essa and Georgia Institute of Technology, 1999

Outline

Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-aware Applications “Aging in Place” Conclusions & Discussion

© Irfan Essa and Georgia Institute of Technology, 1999

“Aging in Place”

Design aware homes that support elderly– allow them to be independent, yet

connected– supported, cared for– stay home (as opposed to move to an

elder care facility)– health monitoring

© Irfan Essa and Georgia Institute of Technology, 1999

Connected Family

Is Mom doing well? Eating well? (peace of mind)

interface that leads to connectivity see snapshots of “activities”, “day’s

events” active connection (in the periphery) continuously updating “portrait” of

Mom displaying how she is doing.

© Irfan Essa and Georgia Institute of Technology, 1999

Cognitive Support

Assist in daily routines to offset cognitive impairments

Aid memory– take medication– locate lost items– out of site / out of mind (connected)

Avoid institutionalization effects

© Irfan Essa and Georgia Institute of Technology, 1999

Requirements Analysis

What “matters” in free choice envs. What is productivity? Quantify ??!! Why do people move to assistive care

facilities? Why don’t they want to leave their

homes

Ethnographic Interviews ……

© Irfan Essa and Georgia Institute of Technology, 1999

Outline

Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context & Domains “Aging in Place” Conclusions & Discussions

© Irfan Essa and Georgia Institute of Technology, 1999

Test-beds

Future Computing Lab (5/1998) Computational Perception Lab

(1/98) New Labs for “off-the-desktop”

computing (7/1999) “Aware Home” (1/2000)

– Kitchen, Living Room, Entertainment Room, Home-office.

© Irfan Essa and Georgia Institute of Technology, 1999

Future

Pursue both technology-centric and human-centric approaches, understand the domain and build it– better sensing and perceptual

analysis mechanisms– software, systems, networking

infrastructure– evaluate the human-home symbiosis

© Irfan Essa and Georgia Institute of Technology, 1999

Summary

Described the Aware Home Project Making it aware

– sensing– context-enabled

Challenging Application– Care Facility.

© Irfan Essa and Georgia Institute of Technology, 1999

The end

© Irfan Essa and Georgia Institute of Technology, 1999

Why a living laboratory?

It is not sufficient to achieve technological breakthroughs.

Greater contribution lies in the understanding of impact on everyday life.

Domain specific.

© Irfan Essa and Georgia Institute of Technology, 1999

Living laboratory experience

Classroom 2000 (education)– a classroom used daily for 3 years– captured experiences in a classroom

Smart Spaces– Rooms, Offices, …

A large-scale test-bed for research in …