wearable computing : present and future

Wearable Computing : Present and Future

Swadhin Pradhan

Reading Group Presentation

Business

• In 2013, investors poured $458 million into 49 wearable company deal ( CB Insights )

• $50 Billion Industry by 2017 !! ( Credit Susse )

• Major tech companies like Apple, Google, Samsung and Intel investing heavily in wearables, with non-tech giants like Nike, Under Armour, Adidas, lululemon etc.

Popularity Example

• Pebble

• Kickstarter Campaign

• Seeks : $100K• Raises : $10+ Million

Factors in Wearable Tech Today

• Faster and Cheaper Hardware• Cloud Storage• Location Data• Quantified Self Activity• Gaming Industry• Visual & Voice Technology• User Experience

State-of-the-art

• Devices

• Inputs

• Applications

• Algorithms

Devices

Audio-Video Capturing Devices

Logging Life Events

Google Glass (Smart Glasses)

Narrative Clip (First Person Camera)

Autographer (First Person Camera with Sensors and GPS)

Kapture (Autonomic Audio Capture)

Body Data Gathering Devices

Monitoring Health Status

Jawbone (Activity Monitor)

Fitbit (Fitness Tracker)

Pebble (Smart Wrist Watch)

Emotiv (Brain Activity Tracker –EEG)

Muse (Brain Monitor)

OmSignal (Smart Shirt)

Sensoria (Smart Socks)

FootLogger (Shoe Sole for Fitness Tracking)

Mimo (Baby Status Monitoring)

MC10 (Flexible wearable sensors for health monitoring)

Gesture Recognizing Devices

Myo (Muscle Activity Tracker)

SmartyRing (Smart Ring)

Fin (Another Smart Ring)

Ring GINA (MobiCase 2013) for 3D gestures

Scanadu (Health – heart rate – ECG – Blood Pressure etc. Checker)

SkinPut (MSR)

Wearables discussed in Research Papers

New Wearables – New Applications

• BodyScope: A Wearable Acoustic Sensorfor Activity Recognition (UbiComp 2012)


Detecting Cocaine Usage through Wearable ECG Sensor ( UbiComp 2013)


• An Amulet for Trustworthy Wearable mHealth (HotMobile 2012)


• Wearable Smartphone: Wearable Hybrid Framework for Hand and Foot Gesture Interaction on Smartphone (CVF, ICCV 2013)


• An Interactive Belt Worn Badge (CHI 2012)


• Wearable Device for Visualizing Knee Rehabilitation Exercises (CHI 2013)

And Many More ……

Classification of Wearable Devices (Function Wise)

• Life Logger• Gesture Recognizers• Entertainer– Video– Gaming

• Assistant– For Chore Jobs– For Creative Jobs– For Emergency Jobs

Classification of Wearable Devices (Creation Wise)

• Replacing Daily Wearables with Smarter Alternatives– Watches, Shirts, Shoes, Socks etc.

• Creating New Wearables– Armband, Headband, Shirt Clippers etc.

Inputs

Ambient Sensors

• Accelerometer, Gyroscope, Magnetometer, Barometer, Thermometer, Relative Humidity Sensor, Light Sensor etc.

Accelerometer Gyroscope

Physical Activity Sensor

• MC10 Sensor, EEG sensor etc.

MC10 SensorInteraxon Muse EEG reader

Traditional Inputs/Capture

• Audio(microphone), Video, Photos (camera –head mounted/ body-attached/ smartphone) ...

• Wi-Fi, Cellular, GPS …

Applications

Life Logging (Augmented Memory)

• Google Glass• Narrative Clip Syncing with cloud• Getting summary from the video or audio or

images – an important problem !!• Automatic diary creation for the users or

taking notes for the forgetful users is important.

Activity Tracking/Monitoring

• Calorie Used (Wireless Health ‘13 paper used to guess correctly with accelerometer)

• Sleep Pattern• Steps walked• Detecting Eye Contact using Wearable Eye-

Tracking Glasses (UbiComp 2012)• Wearable Activity Recognition for Dogs !!

(UbiComp 2013)

Healthcare

• Monitors different vitals of users and help them to take informed decisions.– Calory Count using multi-modal Wearable Sensors

(Wireless Health ’12, Hail Kalnatarian et. al.)– SpiroSmart: Using a Microphone to Measure Lung

Function on a Mobile Phone (UbiComp 2012)• Emergency Patient observation and immediate

healthcare notification.– Cognitive Assistance through Wearables (Offloading

through Cloudlets) [Kiryong Ha et. al.]

Gesture Recognizing

• Free from Gesture used for authentication (MobiSys 2014 – MPI,Rutgers)

• Identifying Emotions Expressed by Mobile Users through 2D Surface and 3D Motion Gestures (UbiComp 2012)

• Unobtrusively Wearable Sensor Suite for Inferring the Onset, Causality, and Consequences of Stress in the Field (Sensys 2013)

Assistant

• Anywhere, Anytime Information and Communication

• Sensor-Assisted Facial Recognition: An Enhanced Bio-metric Authentication System for Smartphones (MobiSys 2014) [Trick : Relative Position estimation using sensors]

• Navigation using multimodal sensors (NaviComf, PerCom 2012)

• A Smartphone-Based Obstacle Detection and Classification System for Assisting Visually Impaired People (CVF, ICCV 2013)

Assistant to the special people• Geometric Layout Analysis in a Wearable Reading Device for

the Blind and Visually Impaired (MobiCase 13)

• Parent-Driven Wearable Cameras for Autism Support, CMU, UbiComp poster

Augmented Reality

• Google Glass like view which adds layer of virtual view to normal view.

• After integrating gesture recognition and voice commands, augmented reality can impact retail industry, social networks, and gaming industry.

Algorithms

General Flow

• Feature Selection and Extraction (PCA etc.)• Noise removal and Smoothing (Local

Averaging, DTW etc.)• Peak Detection and Filtering (Butterfly

Filtering etc.)• Unsupervised (K-Means, EM Maximization

etc.) and Supervised (SVM, LDA etc.)

Needs Better Sampling Algorithms

• Wireless Health 2012 paper on better sampling for Body Area Networks (Goudar et. al.)

• How much to sample ? When ? What is the accuracy needed ?

• Should it be application based or activity based ?

Needs Better Robustness for Context Awareness/ Activity Recog• Wireless Health 2012 paper tries to find

bound of Dynamic Time Wrapping Technique to perform moderately (Nimish Kale et. Al.)

• Combination of Wearable Sensor Data and Physiological data to estimate calorie count (Wireless Health 2013) [Marco Altini et. al.]

Needs Customized Machine Learning Algorithms

• Unsupervised Activity Clustering using Single body Sensor and estimating energy consumption (Wireless Health 2013) – Similar Activity Clusters have similar regression models.

• Energy Efficient HMM and KNN for embedded classifiers (Dawud Gordon et. al.)

• Google’s Nine Level Neural Network to read Road Signs, which actually breaks 99.99% Captcha !!

Needs Customized Machine Learning Algorithms

• PhotoOCR: Reading Text in Uncontrolled Conditions (ICCV 2013)

• Motion Primitive-Based Human Activity Recognition Using a Bag-of-Features Approach (IHI 2012)

Interesting Marraiges

Wearable Computing and Smartphone

• Most Common and Obvious• Hub, Connector, and Storage• Bluetooth, Wi-Fi Direct, NFC ( Audio NFC (Dhawni,

SIGCOMM ‘13) & Visible Light (Hotnets 2013) !! )

Wearable Computing and IoT

Wearable Computing and Smart Home

Wearable Computing and Gaming

• Facebook buys Oculus Rift to give user a virtual social gaming experience …

• Microsoft buys Osterhout Design Group in San Francisco, which creates virtual gaming environments.

• Wearable computing can make a daily skype or phone call a direct virtual interaction experience !!

Wearable Computing and Search

• Google Acquires Nest• Physical Graph - Web Graph – Knowledge

Graph• Better and personalized results• Closer to the actual Information Need

Wearable Computing and Social

• Physical Graph – Facebook Graph• Fusing Mobile, Sensor, and Social Data To

Fully Enable Context-Aware Computing (HotMobile 2010)

• Socio-Technical Network Analysis from Wearable Interactions (Katayoun Farrahi et. al.)

Multiple Wearables in sync

MIThrill (MIT Media Lab)

Sixth Sense ( An Old One ?)

Oculus Rift and Myo

Google Glass and Muse

• Muse rejected Google’s buy-out offer • Muse will read your mind and Google Glass

will show you content accordingly.– Show videos to appease query or release tension

( much better than HotMobile 2014 paper QuiltView which only shows videos for a given query )

Concerns

Privacy• Expectation and Purpose: Understanding

Users’ Mental Models of Mobile App Privacy through Crowdsourcing (Privacy depends on Context, UbiComp 2012)

• Give whole level characteristics to the service provider not each user level specific information.

Security

• Wearable Device can be hacked and attacked wirelessly. Patients may die.

• Spoofing and altering are dangerous phenomenon which can actually derail the whole purpose. May create panic.

• Side channel attack through power trace analysis is possible.

Energy

• Less is More: Energy-Efficient Mobile Sensing with SenseLess (MobiHeld 2012)

• Power Constrained Sensor Sample Selection for Improved Form Factor and Lifetime in Localized BANs (Wireless Health 2012)

• Power will come from human body energy !! (Thad Starner, IBM Systems Journal)

• Energy-Efficient Continuous Activity Recognition on Mobile Phones ( Your Activity will define Model (Energy <->Classification), ISWC 2012)

Intrusion

• Too much personalization or assistance will repel users

• Users will be overwhelmed by the huge amount of data and can easily be panicked by misinterpreting any vital health data

• May curb creativity and reduce recall rate

New Ideas

Automatic Text Tagging with Emotions (Google Glass + Muse +

Jawbone)• Each story can be automatically tagged with

emotions by tracking the eye movement, sounds, activity etc.

• User can easily search according to his mood or can be automatically given reading suggestions of a particular position of a book depending upon his mood !!

AutoRemember : A Google for Daily Things (Tile + Google Glass +

Smartphone)• We forget. We can’t find important docs when

we need • Can we use our mobile, our sensors, google

glass, rfids, tiles or qr codes to automatically keeping track of our things?

• This system will also automatically categorize the things for us; sometimes also opportunistically scan some docs to store these in cloud for ubiquitous access.

MindDoctor : Body Language Detection - Mood Inference - Mental

health Suggestion• By intelligently and energy efficiently sensing our

activities and context, a system can easily infer our mood and can set the color & background music of my smart home accordingly.

• The system can suggest some exercises like deep breathing when we are really tensed.

• The system can also detect our body language or postures, and make suggestions according to context – like be confident when in meeting.

Random Thoughts

• Give wearable like benefits using feature phones for developing countries …

• Communication through visible light (hotnets 2013) or audio (Dhawni, SigComm 2012) or omnipresent signals in environment (aereo catches TV signals and SigComm 2013 best paper BackScatter uses it for powerless communication) can be leveraged …

What is Next …

Edible Computers

• "I expect to see edible computers pills, which will act like little medical monitors, downloading information about your state of health to a computer you wear.“– Nicholas Negroponte, MIT Media Lab, 1999

• Motorola Password Pills & Tatoos ..

Planting Computers (Transhumanism?)

wearable computing : present and future

Documents