Telehealth:Opportunities for  

(Communications) Research?Urbashi Mitra

Daphney Zois, Gautam Thatte & Marco LevoratoUniversity of Southern California

Telehealth:Opportunities for  

(Communications) Research?Urbashi Mitra

Daphney Zois, Gautam Thatte & Marco LevoratoUniversity of Southern California

where is the communications?

Telehealth:Opportunities for  

(Communications) Research?Urbashi Mitra

Daphney Zois, Gautam Thatte & Marco LevoratoUniversity of Southern California

where is the communications?it’s at the end…

• The KNOWME Network– Highly interdisciplinary adventure– Computer engineering, wireless communications, machine learning, robotics/human motion modeling, preventive health

• Deploy a wireless body area sensing network– Employ off‐the‐shelf sensors– Original application: pediatric obesity– Optimize all layers of the system

• System architecture• Multimodal physical activity recognition• Energy efficient sensor selection• (human) Energy expenditure estimation• KNOWME on real people/free‐living

Objectives

U.S

. 200

3-20

06, A

ges

12-1

9

• What I learned…

KNOWME

Activity Detection for Health

OXI

ACC

ACC

ECG

GPS

biometric waveform

feature

decision

Q: who to listen to and for how long?

• Nokia N95 cellphone fusion center– Alive Technologies ACC and ECG (chest 

strap) and mobile‐internal Accelerometer | Pulse oximeter not used

– Cost of Bluetooth measurements >> Cost of Nokia ACC

– Bluetooth implementation imposes TDMA!

Energy constraints are different!

our body sensing network

traditionalsensor network

optimize energyresources at N95

optimize energyresources at sensor

sensors are heterogeneous

• Cell phones are not biometric signal processing devices (yet)

Fusion center capabilities

• Electrocardiograph (ECG)

• Features extracted from data– Used to develop hypotheses 

in training– Compared to models in 

testing for activity‐detection

From Sensors to Features

• Accelerometer (ACC)

sample = average variance of tri‐axial accelerometer for windowed data

sample sample sample

ECG period(= sample)

• Certain sensors can better discriminate between specific sets of activities

Heterogeneous Sensor Performance

both ACC and ECG

Only AC

COnly ECG

• Certain sensors can better discriminate between specific sets of activities

Heterogeneous Sensor Performance

both ACC and ECG

Only AC

COnly ECG

• Measurements collected at fusion center– Several features extracted from each sensor– Model temporal correlation via AR(1) process

• Features only weakly correlated; assume independence– Some sensors discriminate better in certain cases– Use of feature selection results in low feature‐feature correlation

• A few challenges inherent to optimizing the system…

Signal Model

measurement noise

mean of Gaussian‐modeled featuremeasurement

# of measurements from kth sensorAR(1) parameter

Multihypothesis Testing

• Discriminate between multiple hypotheses– e.g. Lie down, sit, sit&fidget, stand, stand&fidget, walk, run

• “feature” of Bluetooth implementation: TDMA access• Canonical general Gaussian problem

– Each activity ~ multivariate Gaussian

block‐diagonal

N1 x 1N1 x N1

sensors independent

Non‐linear Decision Regions

• Distinct means and covariance matrices for each subject | personalized training

Decision regions for bivariate Gaussians for six activities

Effect of Integer Allocations

• As allocation changes, structure of covariance matrix changes, e.g. for N = 6

• Linear algebraic gymnastics:  combinatorial integer optimization  Real‐valued vector optimization with continuous N– Real‐time implementation on Nokia N95

Thatte, Li,  Lee, Emken, Annavaram, Narayanan, Spruijt‐Metz, and M, IEEE Trans.  On Signal Processing, April 2011

• System State: activity performed at time k,

– possible activities supported by the system– probability transition matrix

System State

1: Sit2: Stand3: Run4: Walk

example state spaceand Markov chain

POMDP System Execution

state switches

sensors sensing

state detector

belief state updatesensor selection

decision process

Different from typical POMDPs: dependence on previous state, also control does not affect state, but your ability 

to accurately observe the state

• Worst‐case error probability: sensors’ discrimination capabilities

• Energy Cost: transmission power consumption

: communication cost from sensor to cell phone (different for each sensor)

• Total Cost:

Performance Measures

approximated using previous methods

• Imperfect state information requires use of beliefstate

– Sufficient statistics, new update rule needed– ML state estimate

• A general partially observable, stochastic controlproblem

• Solve using DP methods/computationally expensive

State estimation & Optimization

• Can show that cost‐to‐go function                is– Positively homogeneous, concave, piece‐wise linear, superlinear

• Suggests an approximation scheme– Minimum Integrated Cost Time Sharing Sensor Selection (MIC–T3S) algorithm

– Use approximate DP to determine solutions,        , at corner points of simplex

– Final solution via time‐sharing using belief estimate

Approximation Schemes

Complexity:

• 3 sensors: ACC Mean, ACC Variance, ECG Period• 4 activities: Sit, Stand, Run, Walk•

Simulations

• Statistics from realdata

• Costs from real devices• Energy efficiency

– # ACC Mean

– $ ECG Period

• Detection Accuracy– # ACC mean & ACC var– $ ECG period

• Expect limited useof ECG period

ACC mean

ACC var

ECG period

G. Thatte et al., IEEE Trans. on Signal Proc., April 2011

sitstandrunwalk

• EA: every sensor samenumber of samples

Near‐Optimality of MIC‐T3S

• DP and MIC‐T3S attempt to optimize average total cost• Approximate algorithm (MIC‐T3S) performance coincides with

optimal (DP)

Weight factor λ

Average Total Cost

N = 12 samplesL = 5 horizon length

Trade‐off CurvesAverage De

tection Error

Average Energy Cost

Energy gains achieved by 

proposed schemes for detection 

accuracy equal to EA 

Contributions

• Derived implementable performance approximations for MLactivity detection in WBANs

• Proposed a modified POMDP model to optimize detection inheterogeneousWBANs- Different discrimination capabilities & communication costs of sensors- Limited energy budget of fusion center- Imperfect state information

• Derived:• optimal DP algorithm + three approximate schemes

• Energy gains as high as 68% (versus equal allocation) with 99%accuracy and very few samples!!!

Zois, Levorato, & M, in submission  January 2012

Future challenge

States:• Physical activity states

• Contextual states• Emotional states• ACTUATION states

• How to handle state explosion?

• Implications for other large scale (PO)MDPs

the WHOLE system

KNOWME Architecture

• combine health sensors with mobile phone– metabolic: ACC, OXI, ECG– emotional: GSR

three‐tier KNOWME architecture

Internet

Database Server

Web ServerEnd-to-end encryption of sensitive data

Check right to use systemFilter noisy updates

Web enable data access

3GGSMWi-Fi

Doctor

– location: GPS– user initiated: SMS, Tweets, 

video/image/voice tags

Mobile Client Design

• KNOWME execution priority less than other mobile applications (incoming/outgoing calls)

• Device manager– maintains identities of different sensors– sources between different sensors

• Data collector– Receives/synchronizes sensor data from device manager– Health records collected, buffered, sent to local storage, transmitter and analyzer

• KMCore elements determined by implemented Bluetooth/TDMA multiple access

Challenges of System Architecture

• Energy consumption issues are significant– N95 performs all coordination,  

processing, computation– Employ data buffering, adaptive 

sensor throttling, dynamic sensor selection, data transmission choices

• State detection is the most expensive activity– State detection at phone and 

transmit?

Lessons Learned

•– Compress data and do state detection at back‐end server?

• Application stability an issue– Designs done on emulator/debugging

• Floating point:   need smart implementation of signal processing algorithms

WiFI: compression more expensive

• Machine learning techniques– Support vector 

machines– Feature fusion 

methods

• Novel feature design– Cepstral– Time‐domain

• Non‐linear methods

Multimodal signal processing

Accelerometer Temporal Feature Extraction

ECG TemporalFeature Extraction

SVM Modelingwith GLDS Kernel

SVM Modelingwith GLDS kernel

ECG Cepstral Feature Extraction

Accelerometer Cepstral Feature Extraction

GMM Modeling

GMM Modeling with HLDA

DetectedActivities

Weighted Sum Fusion

Feature Extraction Activity Modeling Score Level Fusion

-2 0 2-1

-0.5

0

0.5

1

phase [-:]

Mea

n A

mpl

itude

Lying

-2 0 20

0.1

0.2

0.3

0.4

phase [-:]

Sta

ndar

d D

evia

tion Lying

-2 0 2-1

-0.5

0

0.5

1

phase [-:]

Mea

n A

mpl

itude

Walking

-2 0 20

0.1

0.2

0.3

0.4

phase [-:]

Sta

ndar

d D

evia

tion Walking

Collected ECG Signal

Peak Detection

Phase Alignment

Normalized ECG beats

Amplitude Scaling

DC bias Removal

(a) System overview

(b) ECG pre-processing( c ) Normalized ECG heartbeats

Li, Rozgic, Thatte, Lee, Emken, Annavaram, M, Spruijt‐Metz, Narayanan , IEEE Trans on Neural Systems and Rehab. Eng., August 2010.

• High within individual variability– Sensor placement– Location, user emotion, fitness– ECG signals appear to be more sensitive to robustness issues

• Within a single activity, significant signal variation– How to model?

• Robustness increased via multi‐session training

Lessons learned

What goes around, comes around

• CDMA?

What goes around, comes around

• CDMA?– Significant effort devoted to working around 

Bluetooth/TDMA (implemented on phone/on sensors)

– System architecture issues: device management, thread distinction, maintaining buffers for different sensors

What goes around, comes around

• Pros:– Few sensors  short spreading sequences– Multi‐user detection complexity comparable 

to machine learning methods– No device management, thread distinction, 

buffer maintenance  all done in signal processing

• E.g. no need to keep track of non‐functioning sensors, etc.

• The Scientific Method– Short term research to drive long term implementation

the BIG Conclusion

• There are some things that are unknowable unless you…– Build a system– Work with domain experts– Deploy the system

• Significant room for theory, algorithm design,

• (comm theory, not yet?) http://www.nle2ndgrade.com

• The Scientific Method– Short term research to drive long term implementation

the BIG Conclusion

• There are some things that are unknowable unless you…– Build a system– Work with domain experts– Deploy the system

• Significant room for theory, algorithm design

• (comm theory, not yet?) http://www.nle2ndgrade.com

• “it’s like having a doctor in your pocket”