![Page 1: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/1.jpg)
Prof. Joe Paradiso Responsive Environments Group, MIT Media Lab
http://www.media.mit.edu/resenv Seoul
August-09
Living with Ubiquitous Sensing and Dynamic Responsive Media
![Page 2: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/2.jpg)
2
JAP 4/08
A Historical Perspective
![Page 3: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/3.jpg)
3
JAP 4/08
…to experimental high-energy physics
PhD with Ulrich Becker and Sam Ting (MIT LNS) 1981
![Page 4: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/4.jpg)
4
JAP 4/08
It’s great to do a thesis in Switzerland
CERN is a city of physicists Geneva is between Alps and Jura Any European culture is a short train ride
![Page 5: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/5.jpg)
5
JAP 4/08
PreHistory - 1960
![Page 6: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/6.jpg)
Me - 1976 1974
1976
![Page 7: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/7.jpg)
At Ars Electronica Festival – Sept. 2-9, 2004
![Page 8: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/8.jpg)
8
JAP 4/08
World’s largest home-designed-built synth (1975-1985)
Newton living room - 1986
![Page 9: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/9.jpg)
Musical Interfaces
![Page 10: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/10.jpg)
Sensor Networks as Extension of the Nervous System
Cast our awareness across space, time, scale, modality…
![Page 11: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/11.jpg)
11
JAP 4/08
Bootstrapping a Ubiquitous Sensor Infrastructure
• Sensor networks are the foot soldiers at the front lines of ubiquitous computing
• At this point, few if any customers will buy an ensemble of “UbiComp” sensors
• They will aggregate from established devices – Home security, appliances, utility devices, entertainment…
Just as the web sprouted from a networked ensemble of personal computers, true “ubicomp” will arise from an
armada of networked devices installed for other purposes.
![Page 12: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/12.jpg)
12
JAP 4/08
Power Strips are Everywhere
• Needed in Homes, offices, especially the Media Lab! • Sensors are becoming commodity items
– Cost of adding sensors to a design is becoming incremental • Power strips are ideal base platforms for hosting a
sensor network – Ready access to power – Power line can be a network port – Can monitor the status of devices that are plugged in
![Page 13: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/13.jpg)
13
JAP 4/08 PlugPoint – Power Strips as the backbone of a UbiComp Sensor Infrastructure J. Lifton, M. Feldmeier, Y. Ono (Ricoh)
Power Line provides energy & comm
Monitor current profiles, Switch individual sockets Hosts basic sensors (mic,
light, motion) Expansion Port for others Hub for wireless sensor
network
Collaboration with Ricoh Research
![Page 14: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/14.jpg)
Army of Plugs
35 ON MEDIA LAB THIRD FLOOR
![Page 15: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/15.jpg)
MIT Media Lab
Joe Paradiso / ResEnv
![Page 16: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/16.jpg)
Distributed Acoustic Conversation Shielding
Yasuhiro Ono
Ricoh ML affiliate
Paper in ACM SANET 2007
![Page 17: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/17.jpg)
17
JAP 4/08
Cross Reality
![Page 18: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/18.jpg)
IEEE Pervasive Computing July 09 MIT ML, MIT CMS, Stanford, ETH, University of South Australia, Georgia Tech, Sun, etc.
![Page 19: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/19.jpg)
Shadow Lab - Binding real sensor data to virtual worlds
Simple sensor apparitions to explore basic ideas - Energy use ➔ Amount of smoke
- Sound Levels ➔ Ripples - PIR Motion ➔ Waving Fronds - Temperature ➔ Frond Color
- Light Levels ➔ Frond Height
Third floor of ML built in Second Life
ResEnv Lab rendered in detail - other areas currently
derived from map
Sensor data piped in and interpreted as real-time
graphic phenomena
Lifton 07
![Page 20: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/20.jpg)
Portal Details – Camera System Deployed to cover entire
building (~50 nodes) 3MP Camera Motorized Panning and Focus Dedicated Video DSP/ARM - (TI DaVinci chip) Real-time Linux OS LCD display (Touchscreen) Contains Spinner Gateway/
Sensor board (detailed on next slide)
45 distributed across Media Lab since October 08
Mat Laibowitz
![Page 21: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/21.jpg)
Device Details – Spinner Gateway Works with or without camera
board Communicates with wearable/
mobile devices in mesh network Serves as reference beacon for
location system Ethernet or slave to linux board Audio system with DSP
AVR32 Environmental Sensing
Motion Humidity and Temperature Light Infrared Communication and
Detection/Proximity Talks to badge systems
![Page 22: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/22.jpg)
45 all over the Media Lab
Power Switch
![Page 23: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/23.jpg)
Invitation to Interaction
![Page 24: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/24.jpg)
Live Portal-Portal Video Feed
![Page 25: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/25.jpg)
Portals and Cross Reality
Real World
Virtual World
![Page 26: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/26.jpg)
Will the market demand privacy?
People are bringing these sensors into their homes and lives on the backs of dedicated appliances and devices
If they don’t feel in control of them, they won’t buy them!
How will they manage their dynamic privacy in a world full of potentially invasive sensors?
- You can’t turn them all off because there are too many, and they are often attached to things that need to stay on or can’t be turned off…
![Page 27: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/27.jpg)
world
group
user Accessible for user’s social network
Accessible for public
Accessible for data owner & close colleagues only
Video, Images, Clean Audio, fine localization
Scrubbed images, garbled audio, coarse localization
Image & audio features only, building-level localization
No ID, building-level features
User-Configurable Dynamic Privacy Settings
Privacy can depend on physical location, time, and inquiry identity
![Page 28: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/28.jpg)
Nan-Wei Gong
NoNo Badge for Dynamic Privacy
Dynamic Privacy…
![Page 29: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/29.jpg)
NoNo Badge for Dynamic Privacy
Dynamic Privacy…
Nan-Wei Gong
![Page 30: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/30.jpg)
MIT Media Lab
Joe Paradiso / ResEnv
Privacy requests (button pushes) per user vs time of day
![Page 31: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/31.jpg)
31
JAP 4/08
![Page 32: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/32.jpg)
32
JAP 4/08
Public Misinformation…
“Human” Energy harvesting will do little for sustainability It will only be valuable in extending/eliminating batteries in
portable devices, wearable sensors, etc.
![Page 33: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/33.jpg)
33
JAP 4/08
Sensor networks for energy conservation
• Leveraging dense sensor networks for optimal energy management – 40% of US energy is spent in buildings – Pervasive sensor/actuator network can reduce this
• Optimize heating, AC, lighting for Person not room • Anticipating behavior & build usage models over time
![Page 34: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/34.jpg)
Wireless Monitor
Wearable Integrated vibration, T & H, Light @ µW Mark Feldmeier
Ceiling PIR 802.15.4 bridge
![Page 35: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/35.jpg)
Ceiling HVAC Damper Control
Measure temperature, humidity, airflow, PIR motion Continuous Control of Damper
![Page 36: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/36.jpg)
Open/Close Windows
![Page 37: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/37.jpg)
37
JAP 4/08
Learning Comfort
![Page 38: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/38.jpg)
38
JAP 4/08
Very Early Data – Learned Setbacks
![Page 39: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/39.jpg)
39
JAP 4/08
SportSemble 3x HiG Accl, 3x LowG Accl, 3x HiRateGyro, 3x magnetometer
![Page 40: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/40.jpg)
40
JAP 4/08 Network In Action
![Page 41: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/41.jpg)
To be presented at ISWC 09
![Page 42: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/42.jpg)
Return to Spring Training, 3/09
New device has onboard removable flash for continual data storage (RF synch), Bi-range full IMU
Soon measuring entire team and streaming data to MIT/MGH
![Page 43: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/43.jpg)
Red Sox Spring Training - March 09
Video by Alex Reben
![Page 44: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/44.jpg)
44
JAP 4/08
Exchange Business Card, Bookmark Demos
![Page 45: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/45.jpg)
45
JAP 4/08
Bookmark data posted on website
![Page 46: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/46.jpg)
46
JAP 4/08
Bookmark data posted on website
![Page 47: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/47.jpg)
47
JAP 4/08
Badge Accelerometer & Audio Data
![Page 48: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/48.jpg)
48
JAP 4/08
Interest Meter & Group Dynamics
Collaboration with Human Dynamics Group
Badge-Badge Badge-Demo
Uninterested
Interested
Uninterested
Interested
Classification Value = f (voice, motion, face-face time…)
80% Accuracy
![Page 49: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/49.jpg)
49
JAP 4/08
Affiliated Wearers
Affiliated people tend to spend more time face-face and move together!
Accuracy of inferred affiliation: 93%
Face-Face Time (IR) Correlated Motion (accels)
![Page 50: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/50.jpg)
50
JAP 4/08
Affiliated Wearers from Energy Only
![Page 51: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/51.jpg)
Timekeeping for Talks
T – 3 mins T – 2 mins T – 1.5 mins T – 1 mins T – 30 sec
Out of time!! • 24 talks in the morning (research updates) • 5 Minute time limit on each! • Audience badges flashed time queues • We didn’t run over (first time ever…)!!
![Page 52: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/52.jpg)
SPINNER • Sensate Pervasive Imaging Network for Narrative
Extraction from Reality
• Unites wearable human sensing with video capture • Use of wearable sensing allows access to subject/data
channels far beyond what can be achieved with standard image pixel processing
Mat Laibowitz - PhD finished fall 09
Query multimodal sensor networks in human terms (through narrative/stories)
![Page 53: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/53.jpg)
Overall System Diagram
Behavior Models
Sensor /Action Models
Captured Events
(video clips w / sensor data )
Assembling of Narrative(video )
Meaningful Narrative Discourse
(video)
StoryModel
Design /Input
UserInterface
Sensor NetworkInternal Mappings
Sensor /Imaging Physical Network
Sensor Data
TrainingDataFE
EDB
ACK
FEED
BAC
K
Camera System
![Page 54: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/54.jpg)
The Age of Ubiquitous Media
• Is here, whether you like or not
• Can record your entire life, too much video to ever watch
• Need a mechanism to create/extract meaningful content
![Page 55: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/55.jpg)
Device Details – Spinner Social Sensor • Wearable on collar or as pin/
badge • Audio system with DSP for
analytics and CD quality recording
• Mini-SD Slot for copious memory • Compass for orientation • 3-axis Accelerometer & 2-Axis
Gyro • IR communication and line of
sight detection/proximity • 802.15.4 Radio with RF Location
engine • Captures social signal and group
dynamics • New Badge has OLED Display
![Page 56: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/56.jpg)
![Page 57: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/57.jpg)
System talks to Portals • Cameras will record whenever Spinner badges
are nearby • May also use wearable cameras (e.g., N95) for
1’st person perspective
![Page 58: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/58.jpg)
First ever video edited automatically using sensor data and filtered
from a >40 node distributed camera network
Bright and calm - Active - Dark and calm
Story Arc…
![Page 59: Living with Ubiquitous Sensing and Dynamic Responsive Media](https://reader030.vdocument.in/reader030/viewer/2022012613/6198421ca257bd4c0e52ec81/html5/thumbnails/59.jpg)
59
JAP 4/08
Conclusions • Sensing, computation, and communication become tightly integrated and
commonly embedded • Low power and energy scavenging enable active nodes to be embedded and
“forgotten” • The Ubicomp infrastructure permiates our cities, our dwellings, our objects, our
clothing, and eventually our bodies – Pervasive-Wearable-Implantable
• The 5 human senses locked into our body are augmented by interfaces into ubiquitous sensor network data – Marshall McLuhan for real – Interface devices now - implantables some day – Omniscience…
• This infrastructure mediates everything – Collaboration, business, social interaction, resource use… – Context engines filter, represent, and manifest information
• Google for reality • Brave New World
– Privacy, security, promises vs. perils…