ReachMedia

on-the-move interaction with everyday objects

Assaf Feldman, Emmanuel Muguia Tapia, Sajid Sadi, Pattie Maesand Chris Schmandt

The Media LaboratoryMassachusetts Institute of Technology

Question

Can we merge the virtual world with the physical

world?

Not enough

pixels

Too many clicks

An Interface Problem

Requirements

• Natural and seamless interaction

• Hands and eyes free

• Socially acceptable

IO

Input – RFID,Gestures,Keypad.

Output –audio, text

The smart phone as a general interface

Touch based interaction with objects

Gestures

Slight flick for menu navigation

Select

Next Previous

- Socially acceptable.

Demo

RFID

Multi modal Multi modal interface via interface via accessories accessories

MITe receiver

• Objects ID• Gestural input:

-Next-Previous-Select

RFID RFID Augmented Augmented

physical physical objectsobjects

Audio output:•Earcons •Voice menus

Interaction Interaction mediation via mediation via Mobile deviceMobile device

Object ID, user input

Audio & XML Content

Context Context ManagerManager

Hardware - Sensors

Accelerometers (House_n MITEs

–Cheap

–Small

–Low power

• RFID– Cheap– Great Context

• 25 features - Time domain feature extraction

– Cross correlation (3)

– Power (3)

– Turning points (18)

∑−

=

=1

0

21)(n

ii

an

aP

ji AA

jiji

AAAA

σσρ

),cov(),( =

Signal processing

Classification

• Online (Real Time) classification

• Good accuracy using Naïve Bayes and HMMs

• Preliminary results ~98% accuracy: Previous Next Select Noise

Previous 0.97101 0.014493 0 0.014493Next 0 1 0 0

Select 0 0 1 0Noise 0.017241 0 0 0.98276

Trut

h

The big picture

• Integrate digital world (information & services) and physical world (physical objects/environment)

• Make interfaces more responsive and proactive (objects & environment monitor user and (proactively) present information & services relevant to user’s current needs/interests)

Ambient Intelligence

3rd floor