on-body wireless sensing for human- computer interfacesjoep/outgoing/bsn-2008.pdfmarshall mcluhan,...
TRANSCRIPT
-
Prof. Joe ParadisoResponsive Environments Group, MIT Media Lab
http://www.media.mit.edu/resenv BSN - 6/08
On-Body Wireless Sensing for Human-Computer Interfaces
-
At a fascinating intersection
Sensor Networks
Wearable ComputingImplantables
Ubiquitous Computing
-
3
JAP7/05
Topics
• Wearable Sensors– Sports Medicine & Entertainment
• Low Power Techniques– Quasi-Passive & Groggy Wakeup
• Applications in Health and Supply Chain Monitoring– Passive RFID Tags for Precise Localization
• X Reality– Sensor networks connect real and virtual spaces– Virtual worlds for browsing reality
• Sensor-Enhanced Media Creation SensorNetworks for Social Computing
-
4
JAP7/05
Sensing as Commodity
• Sensors are now becoming a commodity, and soon caneasily be designed into most any device.
– Rather than omitting them from a cost/complexity viewpoint, itbegins to make more sense to just include them if there’s anysuspicion that they could be needed.
– This causes a shift in how sensors are used – rather than rely ononly 1 or 2 sensors made a priori to measure particular quantities,many sensors will be used that don’t necessarily exactly measurethe quantity of interest (especially as applications will becomemore general and evolve over time).
-
1997 - Expressive Footwear17 Data Channels
Tilt, shock, rotation, height, bend,location, multipoint pressure
-
The Wearable Gait Laboratory
Stacy Morris
-
7
JAP7/05
Measures 18 Parameters per foot
2
Left Foot Right Foot
PVDF Toe
FSR Metatarsal, Lateral
FSR Metatarsal, MedialFSR Heel, Medial
FSR Heel, Lateral
Bend - Ankle(out of the plane)
Layout of Insole Sensors
Bend - Insole
Capacitive(height)
PVDF Heel
Axes for Gyroscopesand Accelerometers
x
y
z
-
8
JAP7/05
“Shuffle Index” vs. “Stride Excitement” via CART
0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 0.03-40
-35
-30
-25
-20
-15
Variation in Insole Force [normalized by bodyweight]
Minimum pitch [degrees]
NL F Y
NL F
PD F
Min
imum
Foo
t Inc
linat
ion
(deg
)
-
9
JAP7/05
Real-Time Auditory Feedback
• Parametric Personal Music System– Always play ambient music
• Provide meaningful (and correcting) musical feedbackwhen problematic gait changes arise.
The ParkinsonianAntiShuffle
The PronationPreventer
The PressureAvoider
Erik Asmussen
-
Scaling to several dancers…
High Speed Sensor FusionVocabulary of features
Capacitive proximity to 50 cm6-axis IMU - 1 Mbps TDMA radio
100 Hz Full State Updates from 25 nodes
-
Capacitive Sensing System
• Capacitive system measuresrelative spacing between nodes
• Employs a “transmit mode”configuration
• One node transmits a sinusoidalpulse at 90kHz, others measureamplitude of received pulse
• Nodes trade roles as transmitters and receivers as arbitrated by thewireless basestation
• Sensitive up to 0.5 meter with bracelet-sized electrode and sharedground through the body
Typical Capacitive Sensor Response With Strap Electrode and Body Grounded
0
500
1000
1500
2000
2500
3000
3500
4000
4500
0 10 20 30 40 50 60
Spacing (cm)
Raw
Sen
so
r V
alu
es
-
New Developments
• Full deployment at a rehearsal with 20 sensor nodes running onfive dancers
• Feature extraction and mapping algorithms which are fedlogged data at the sample rate to simulate real time operation
-
Rough Demo Mapping
DirectFeatures
IntuitiveFeatures
MusicalParameters
• Audio rendered in real-time asfeatures are calculated
• Runs on a 1.6 GHz G5 w. 1GB RAM
-
Network In Action
-
SportSemble
-
Red Sox Spring Training
Modified sensors for high range and high sample rates
- 3-axis 10,000 °/sec gyro- 3-axis 120 G accelerometer- 3-axis 10 G accelerometer- 3-axis magnetometer- 1 kHz synchronized sampling per node
- 12 seconds recorded on flash at each node
Biomotion measurement of a Red Sox Pitcher
-
Preliminary Results - Red Sox Spring Training
• Acceleration at the wristpeaks well above 100g
• Most of this accelerationoccurs in a 30ms window
• Equates to 30 samples forthe modified inertial system,but only 5 frames on a180Hz video capture system
Acceleration of the Pitch Above Captured at Three Critical Locations - Hand, Wrist, & Biceps
>100 g’s!!
New collaboration, Red Sox Grant - Spring Training in February
-
Players Cluster
-
19
JAP7/05
Miniaturizating & Distribution
• Future of athletic broadcasts– Get content directly from point of expression
• Sensor packs in figure skates, boxing glove, snowboard…– Map dynamic content (music, graphics, specs…)
• Future of exercise– Monitor lower, upper limbs, heart rate, etc.– Map interactive content
• Synchs up and nudges participant to stay on track• Therapy with interactive feedback
-
Interactive Music on N800
• Zigbee interface for N800– Easy implementation of wearable sensors (inertial, etc.)
• Pd (PureData) compiler for N800– Allows artists to graphically compose music interaction
& synthesis– Produces C code (not interpreted)– Fast, efficient execution
Robert Jacobs’ M.Eng. Thesis - demos coming
-
Groggy Wakeup for Efficient Smart Sensor Systems
Processed
DataS
tate
Enable
Sensor 1
Sensor 2
Sensor N
Microcontroller
State Analysis
Data
Sensor Controller
Algorithm Selector
Data Compression Storage
Transceiver
Ari Y. Benbasat
Power-efficient detection of gait phaseGeneral on-node implementation
Accel1
Gyro1Gyro2
Accel2Accel3
Gyro3
TiltAccel1Tilt
Accel1Tilt
Accel2Accel1Tilt
Accel3Accel2Accel1Tilt
Accel1Tilt
DETECT
Gyro1 Gyro1Gyro2
Accel3Accel2
Power
Time
Log and/orProcess
Automated Power-Efficient Sensor HierarchyKeep higher-power sensors OFF unless needed for detection decision
-
Groggy Wakeup for Efficient Smart Sensor Systems Ari Y. Benbasat
Power-efficient detection of walking up stairs
Automated Power-Efficient Sensor HierarchyTrade Power Consumption with Detection Efficiency
“A Framework for the Automated Generation of Power-Efficient Classifiers for Embedded Sensor Nodes,” Ari Y. Benbasat and Joseph A. Paradiso, Proc. of SENSYS 2007
-
Groggy Wakeup for Efficient Smart Sensor Systems Ari Y. Benbasat
Power-efficient detection of gait phase
Automated Power-Efficient Sensor HierarchyTrade Power Consumption with Detection Efficiency
-
24
JAP7/05
Tree Execution Running on the Stack in Real Time
2-level gait classifier tree during actual walking
-
The Disposable Wireless Sensors
• Very simple “featherweight” motion sensor– Cantilevered PVDF piezo strip with proof mass– Activates CMOS dual monostable when jerked– Sends brief (50 µs) pulse of 300 MHz RF– 100 ms dead timer prevents multipulsing– Can zone to within ~10 meters via amplitude– Ultra low power – battery lasts up to shelf life– Extremely cheap – e.g., under $1.00 in large
quantity
-
CargoNet: A Low-Cost MicroPower Sensor Node Exploiting Quasi-Passive
Wakeup for Adaptive Asynchronous Monitoring of Exceptional EventsMalinowski, Moskwa, Feldmeier, Laibowitz, and Paradiso - Presented at ACM Sensys 2007
• Dynamically adaptable thresholds- Adapts to environments with persistent stimuli
• Small and inexpensive• Microampere current draw
- 5 years on a single coin cell battery
-
Power Harvesting ShoesPVDF Stave
Molded into soleEnergy from bend
Ppeak ≅ 10 mW
≅ 1 mW
Flex PZT UnimorphUnder insole
Pressed by heelPpeak ≅ 50 mW
≅ 10 mW
Responsive Environments GroupMIT Media Lab
1998 IEEE Wearable Computing Conference
Raw Powercirca 1% efficient
unnoticable
-
Mobility in Sensor Networks• Forefront research where sensor nets meet robotics
and control• Sensor clusters move to places to optimally:
– Measure dynamic phenomena– Position relays to repair or patch broken network– Dump information at access points (portals)– Get recovered or recharged
What does power harvestingmean in a mobile system?
Energy cost of moving atoms is muchhigher than moving bits…
-
AES Roadmap“
-
Parasitic Mobility in Sensor NetworksImplications- Sensor clusters hitch rides to placeswhere they need to be to optimally:
- Measure relevant phenomena- Relay information peer-peer- Dump information into portals- Get recovered or recharged
Paradiso & Laibowitz
Innovations and Architecture- Interpretation of Energy Harvesting in mobile networks- Two flavors:
- The Tick (e.g., jumps onto a car, attaches magnetically, then disengages)- The Bur (e.g., sticks to passing object, then shakes off)
- Contains GPS, RF, basic sensor suite
- Rapid diffusion of sensors across an environment
- System self-organizes to auto-dispatch nodes to desired regions
Phoresis
-
31
JAP7/05
Parasitic Mobility Research (ParaMoR)
• Paramor Hardware – smallnodes with sensor suite (light,microphone, inertial,proximity, temperature, heat),GPS, RF communication,rechargeable power source,and minimal actuation forattachment/detachment
• Active nodes (ticks)• Passive nodes (burs)• Value-added nodes (pens)• ParaSim – Software
simulator to study behaviorand evaluate controlalgorithms for parasiticallymobile sensor nodes
Mat Laibowitz
Active Node
Passive Node
-
Utags : Passive Real-Time 3-d Localization & Remote Sensing
• Utilizing passive RFID surface acoustic wave (SAW) and low-cost radar technology.
• Target short-range (3-100m) applications– Expect single-measurement resolution of under 10 cm
• Multipath from reader dies out before tag responds• 900 MHZ devices coming out of MIT MTL and nano labs
– Now being characterized and tested
Precise, ultra low cost wide-area tracking of small passive tags for indoor localizationWill revolutionize asset tracking and supply chain operation, search & rescue, etc.
Jason LaPenta
-
Google for Reality
-
34
JAP7/05
Public Misinformation…
“Human” Energy harvesting will do little for sustainabilityIt will only be valuable in extending/eliminating batteries in
portable devices, wearable sensors, etc.
-
35
JAP7/05
Sensor networks for energy conservation
Humidity
Temp.
Comfort
• Leveraging dense sensor networks for optimal urban energymanagement– 40% of US energy is spent in buildings– Pervasive sensor/actuator network can work to minimize this
• Optimize heating, AC, lighting for Person not room• Anticipating behavior & build usage models over time
– My RA, Mark Feldmeier is now a MIT Martin Energy Fellow• Exploring ways to sense “comfort” to optimize distributed utility control
-
Marshall McLuhan, 1911-1980 "After three thousand years of explosion, by means of fragmentaryand mechanical technologies, the Western world is imploding. Duringthe mechanical ages we had extended our bodies in space. Today,after more than a century of electric technology, we have extended ourcentral nervous system itself in a global embrace, abolishing bothspace and time as far as our planet is concerned. Rapidly, weapproach the final phase of the extensions of man - the technologicalsimulation of consciousness, when the creative process of knowing willbe collectively and corporately extended to the whole of humansociety, much as we have already extended our senses and ournerves by the various media."
Marshall McLuhan - Understanding Media (1964)
Electronic media (a.k.a.television) as an extension ofthe central nervous system
-
Sensor Networks as Extension of theNervous System
Cast our awareness across space, time, scale, modality…
-
38
JAP7/05
Bootstrapping a Ubiquitous Sensor Infrastructure
• Sensor networks are the foot soldiers at the frontlines of ubiquitous computing
• At this point, few if any customers will buy anensemble of “UbiComp” sensors
• They will aggregate from established devices– Home security, appliances, utility devices, entertainment…
Just as the web sprouted from a networked ensemble ofpersonal computers, true “ubicomp” will arise from an
armada of networked devices installed for other purposes.
-
39
JAP7/05
Power Strips are Everywhere
• Needed in Homes, offices, especially the Media Lab!• Sensors are becoming commodity items
– Cost of adding sensors to a design is becoming incremental• Power strips are ideal base platforms for hosting a sensor
network– Ready access to power– Power line can be a network port– Can monitor the status of devices that are plugged in
-
40
JAP7/05PlugPoint – Power Strips as the backbone of a UbiComp Sensor InfrastructureJ. Lifton, M. Feldmeier, Y. Ono (Ricoh)
Power Line providesenergy & comm
Monitor current profiles,Switch individual socketsHosts basic sensors (mic,
light, motion)Expansion Port for others
Hub for wireless sensornetwork
Collaboration with Ricoh Research
-
Army of Plugs
35 ON MEDIA LAB THIRD FLOOR
-
Darkerimpliesmore
sound &movement
-
MSR - 4/08
Sensor Networks for Linking Virtual and Real Worlds
X-Reality
-
45
JAP7/05
-
46
JAP7/05
-
47
JAP7/05
Real World can be “browsed” in virtual space
-
Second Life as a the Virtual End of Dual Reality
-
Shadow Lab - Binding real sensor data to virtual worlds
Josh Lifton - Building a virtual Media Lab in SecondLife
Simple sensor apparitions to explore basic ideas- Energy use ➔ height of fire
- Activity (sound, motion) ➔ whirlwinds- Active regions ➔ higher walls
- “Ghost” face ➔ individual presence
Third floor of ML built inSecond Life
ResEnv Lab rendered indetail - other areas currently
derived from map
Sensor data piped in andinterpreted as real-time
graphic phenomena
Lifton 07
-
50
JAP7/05
Mobile User Interfaces for Sensor Networks
Browsing, querying,navigating through
sensor net dataWhat are the interfaceaffordances, displays,
interactionmodalities?
Privacy & Security?Mobile platform
inside of network vsfixed platform
outside?McLuhan Extension
of human senses
-
51
JAP7/05
Nokia 770 Sensor Net Tricorder - Mk. I
-
SPINNER
Sensate Pervasive Imaging Network for NarrativeExtraction from Reality
Unites wearable human sensing with video capture
Maps sensor data to high-level concepts for creation ofmeaningful videoInvestigates how humans perform this mapping (i.e., how they
create stories and narrative)
Use of wearable sensing allows access to subject/datachannels far beyond what can be achieved withstandard image pixel processing
Mat Laibowitz - PhD in Progress
-
Overall System Diagram
Behavior
Models
Sensor /Action
Models
Captured
Events
(video clips w /
sensor data )
Assembling of
Narrative
(video )
Meaningful
Narrative
Discourse
(video)
Story
Model
Design /Input
User
Interface
Sensor Network
Internal Mappings
Sensor /Imaging
Physical Network
Sensor Data
Training
DataFE
EDB
ACK
FEED
BAC
K
Camera
System
-
Sensor Network Devices
Back-endServer
Video/SensorDatabase
WiFi or Ethernet
Imaging Sensor Nodes
Participant
WristSensor
AudioSensor
Participant
WristSensor
AudioSensor
Participant
WristSensor
AudioSensor
WearbleNarrative
Sensor Nodes
Zigbee
-
Device Overview
Spinner devices includeWrist mounted sensor gesture- and bio- sensingCollar mounted sensor social signaling and audio analysis/recordingCamera system
Wearable devices functions and capabilitiesCamera system controlSensor data capture for video footage cataloguingMultimedia browsing
All devicessupport mesh networkingAre equipped with a location/orientation systemHave dedicated DSP processing for real-time classification of event data
-
Device Details – Camera System
Deployed to cover entirebuilding (>100 nodes)
3MP CameraMotorized Panning and FocusDedicated Video DSP/ARM - (TI Davinci chip)Real-time Linux OSLCD displays what is being
captured at all timesContains Spinner
Gateway/Sensor board(detailed on next slide)
-
Device Details – Spinner GatewayWorks with or without camera
boardCommunicates with
wearable/mobile devices inmesh network
Serves as reference beacon forlocation system
EthernetAudio system with DSP
AVR32Environmental Sensing
MotionHumidity and TemperatureLightInfrared Communication and
Detection/Proximity
-
Device Details – Spinner Social SensorWearable on collar or as
pin/badgeAudio system with DSP for
analytics and CD qualityrecording
Compass for orientation3-axis AccelerometerIR communication and line of
sight detection/proximityLocation engineCaptures social signal and group
dynamics
-
Device Details – Spinner Wrist SensorWrist worn device3-axis AccelerometerGalvanic Skin Response (GSR) SensorLocation engineUI for interacting with networkStores and plays videos, providing
ownership of video to end userCaptures gesture and indications of affective
state
-
Expanded Reality
Not just a pipeTechnology removes boundaries from Real World to
allow new content
-
Contributions
A novel methodology for humans and machines tofilter, organize, and understand large streams ofvideo data
A new form of entertainment and communication thatallows you to create media with your social andpersonal behavior
Toolkit for documentation of daily life that may lead tonew and unexpected insights about random events
These capabilities could lead to a new form of onlinecommunity
-
The UberBadge• 16-bit MCU w/ 64kb flash, 2k
RAM, and GCC support• 45-LED Display intended to be read
at distance• IR communications• RF communications with second
processor to handle MAC• Up to 256MB of data memory• Audio input and output with
onboard microphone• Onboard accelerometer• Pager motor for vibratory feedback• Multiple ports for expansion
– Accepts Sensor Stack Modules• Optional LCD display
Mediate Group Interaction & Behavior Modeling
Mat LaibowitzResponsive EnvironmentsGroup
-
63
JAP7/05
Broadcast MessagesInformation on your badge is mainly for other people, not you!
-
64
JAP7/05
Audience Voting in Auditorium
• Red = No, Green = Yes
Yes
63%
No
37%
-
65
JAP7/05
Exchange Business Card, Bookmark Demos
-
66
JAP7/05
Bookmark data posted on website
-
67
JAP7/05
Bookmark data posted on website
-
68
JAP7/05
Badge Accelerometer & Audio Data
Accelerometers
Microphone
-
69
JAP7/05
Auditorium Fidgeting
• Medium-good correlation with length of talk• “Resets” with every presentation
-
70
JAP7/05
Interest Meter & Group Dynamics
Collaboration with Human Dynamics Group
Badge-Badge Badge-Demo
Uninterested
Interested
Uninterested
Interested
Classification Value = f (voice, motion, face-face time…)
80%Accuracy
-
71
JAP7/05
Affiliated Wearers
Affiliated people tend to spend more time face-faceand move together!
Accuracy of inferred affiliation: 93%
Face-Face Time (IR)Correlated Motion (accels)
-
72
JAP7/05
Affiliated Wearers from Energy Only
-
74
JAP7/05
Timekeeping for Talks
T – 3 mins T – 2 mins T – 1.5 mins T – 1 mins T – 30 sec
Out of time!!• 24 talks in the morning (research updates)• 5 Minute time limit on each!• Audience badges flashed time queues• We didn’t run over (first time ever…)!!
-
Smart pendants to Amulets
-
76
JAP7/05
Conclusions• Sensing, computation, and communication become tightly integrated and
commonly embedded• Low power and energy scavenging enable active nodes to be embedded and
“forgotten”• The Ubicomp infrastructure permiates our cities, our dwellings, our objects, our
clothing, and eventually our bodies– Pervasive-Wearable-Implantable
• The 5 human senses locked into our body are augmented by interfaces intoubiquitous sensor network data– Marshall McLuhan for real– Interface devices now - implantables some day– Omniscience…
• This infrastructure mediates everything– Collaboration, business, social interaction…– Context engines filter, represent, and manifest information
• Google for reality• Brave New World
– Privacy, security, promises vs. perils…
-
77
JAP7/05