![Page 1: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/1.jpg)
Internet of Medical Things�Predicting Clinical Outcomes and Digital Phenotyping with Wearables and Machine Learning
Chenyang LuCyber-Physical Systems Laboratory
Department of Computer Science & Engineering
https://www.cse.wustl.edu/~lu/
![Page 2: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/2.jpg)
Internet of Medical Things
Ø Wearables: wristbands, smartwatches... q Continuous monitoring: activity, heart rate, sleep, location…
Ø Connectivity: Bluetooth, WiFi, cellular…q Real-time monitoring and just-in-time intervention
Ø Cloud: computing and storage.q Scalable to large cohorts
Ø Analytics: machine learning and signal processingq Interpret data and predict outcomes
A powerful tool for clinical studiesinside and outside hospitals!
12/19/2019 Chenyang Lu 2
![Page 3: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/3.jpg)
Smartwatch as a Healthcare Tool
12/19/2019
Two-way communication
ecological momentary assessments
Open, programmable platform
Wear OS, Research Kit, onboard analytics
Continuous, passive
measurements activity, heart
rate, sleep, location…
Chenyang Lu 3
![Page 4: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/4.jpg)
Machine Learning for Healthcare
Ø Machine learning: data-driven tools for healthcare.q Predicting clinical outcomes
q Discover risk factors associated with outcome
q Support clinical decisions to improve outcome
Ø Discover the insights behind diverse dataq Mobile Health (mHealth)
• Continuous monitoring outside hospitals• Small data: mHealth studies usually have moderate population
q Electronic Health Record (EHR)• In-patient data• Big data: large patient population
12/19/2019 Chenyang Lu 4
![Page 5: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/5.jpg)
IoMT Use Cases
Ø Predicting readmissionsq Feasibility of monitoring with Fitbit – data yield and compliance
q Predictive models – machine learning with “small data”
Ø Measuring mobilityq Timed Up and Go with smartwatches
Ø Measuring stressq Ecological momentary assessment (EMA)
q Machine learning model based on physiological signals
12/19/2019 Chenyang Lu 5
![Page 6: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/6.jpg)
Predict Readmissions with Fitbit Ø Hospital readmission rate is high for heart failure patients.
q ~25% patients readmitted within 30 days
Ø Predict deterioration (readmission+death) after dischargeq Fitbit provides continuous monitoring of outpatientsq Just-in-time intervention à better outcome and lower cost
12/19/2019
Joint work with Thomas Bailey (Infectious Diseases), Marin Kollef (Critical Care), Dingwen Li (CSE) �D. Li, J. Vaidya, M. Wang, B. Bush, C. Lu, M. Kollef and T. Bailey, Feasibility Study of Monitoring Deterioration of Outpatients Using Multi-modal Data Collected by Wearables, ACM Transactions on Computing for Healthcare, accepted.
Chenyang Lu 6
![Page 7: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/7.jpg)
Objectives Ø Assess the feasibility of collecting multi-modal data from wearablesØ Develop and test predictive models based on machine learning
Ø Provide guidelines for future studies
12/19/2019 Chenyang Lu 7
Data
Model
Day i-2 Day i-1 Day i
Input Predict
Data Label
Deterioration Early Warning
Data Data
Day 1 Day N
Time of discharge from hospital
InputModel
Predict
N=20
Will deteriorate?
Deterioration Risk Prediction
![Page 8: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/8.jpg)
Challenges
Ø Outpatient complianceq Wear devices continuously and properly outside hospitals
q Potential problems• Forget to charge the device• Reluctant to wear the device• Improperly wear the device
Ø Learning from small dataq Overfitting can lead to models that are not generalizable
q Imbalanced dataset can affect predictive performance
12/19/2019 Chenyang Lu 8
![Page 9: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/9.jpg)
Study Protocol
Ø 25 heart failure patients were recruited.q Each patient is given Fitbit ChargeHR wristband.
q Continuously monitor patients after discharge.
q Outcomes: 60-day deterioration events.
Ø Heart rate (HR), step count and sleep stages were collected.q Sampling period: 1 min (step, heart rate); 1 day (sleep)
12/19/2019 Chenyang Lu 9
![Page 10: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/10.jpg)
Yield Ø Yield: fraction of samples successfully collected and stored in databaseØ Participants with data yield > 80%
q Step: 88% of participantsq Heart rate: 60% of participantsq Sleep: 68% of participants
12/19/2019 Chenyang Lu 10
![Page 11: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/11.jpg)
Compliance Ø Participants with data yield > 80%
q Step: 88% of participantsq Heart rate: 60% of participants à wore Fitbit properlyq Sleep: 68% of participants à wore Fitbit at night
12/19/2019
Feasibility of long-term, continuous monitoring
Chenyang Lu 11
![Page 12: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/12.jpg)
Latency Ø Median: 8.6 min; 99% percentile: 22.5 hour
Ø Did not cause data loss, as Fitbit can locally store data for 7 days.
Ø Feasibility for daily interventionØ Can be reduced if analysis performed natively in Fitbit cloud
12/19/2019
Latency
Chenyang Lu 12
![Page 13: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/13.jpg)
Rich Features of Wearable Data Ø Heart rate (HR), step count and sleep quality were collected.
q Sampling period: 1 min (step, heart rate); 1 day (sleep)
Ø Statistical features:q First- and second- order features extracted from sliding windowq 1st order: mean, max, min, skewness, kurtosisq 2nd order: energy, entropy, correlation, inertia and local homogeneity
Ø Detrended Fluctuation Analysisq Determine statistical self-affinity of time seriesq The fluctuation is then used as feature
Ø Sedentary behavior
12/19/2019 Chenyang Lu 13
Important Features for Deterioration Early Warning
![Page 14: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/14.jpg)
Predict Deterioration Risk Ø Input: Fitbit data collected in the first N days since dischargeØ Deterioration: composite outcome of either readmission or deathØ Small data: 25 patients (18 with no deterioration vs. 7 deteriorated)
12/19/2019
Xdaysinthefuture
Data Data
Day1 DayN
Timeofdischargefromhospital
InputModel
Predict
N=20
Willdeteriorate?
Chenyang Lu 14
![Page 15: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/15.jpg)
Assess Predictability Ø Analysis of Variance (ANOVA) à predictability of outcomes
q Test significant difference in the features between patients of different outcomes.• deteriorated patients vs. non-deteriorated patients
q Assess feasibility of learning a stable predictor with the set of features.
Ø Significant differences in features à predictability
12/19/2019
Features F pHR skewness 7.9125 0.0099
HR correlation 5.4789 0.0283
HR DFA 10 5.3353 0.0302
Restless duration 5.2912 0.0308
Time in bed 5.2663 0.0312
Features with the largest F values. High F-statistic and low p-value à significant difference between group means.
Chenyang Lu 15
![Page 16: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/16.jpg)
Feature Selection Ø Select features using sequential forward feature selection
q Avoid overfittingq Improve performance
Ø Features selected by the models have significant differences in ANOVA test
12/19/2019
Features F psleep DFA 60 0.2254 0.6395
min asleep 4.3128 0.0492daily step 4.3625 0.0480restless count 4.2324 0.0512awake count 4.0429 0.0562min awake 2.2073 0.1509
HR LH 4.0282 0.0566HR DFA 10 5.3353 0.0302
Chenyang Lu 16
![Page 17: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/17.jpg)
Test Overfitting Ø Impact of the number of nearest neighbors (K) on KNN performance
12/19/2019
Smaller difference between training and testing errors à less overfitting
Chenyang Lu 17
![Page 18: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/18.jpg)
Evaluation with Small Data Ø Leave-one-out cross validation
q Leave a sample (or a group of samples) out for testing and train the model with rest of sample (or groups of samples).
q Iterate through all the samples.
Ø For imbalanced dataset, do not just look at accuracy!q Example: for 1:9 positive/negative ratio, the predictor can achieve 0.9 accuracy
if predicting everything as negative.
q A good predictor should perform well for sensitivity, specificity and precision.
12/19/2019 Chenyang Lu 18
![Page 19: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/19.jpg)
Deterioration Risk Prediction Ø Small data: 25 patients (18 with no deterioration vs. 7 with deterioration)Ø KNN achieves higher specificity and precision than LACE index used in
clinical practice.Ø KNN performs better than neural network (NN)
q Simple models help avoid overfitting on small data set.
12/19/2019 Chenyang Lu 19
Model Sensitivity Specificity Precision Accuracy
NN 0.0770 0.9459 0.3333 0.720
KNN 0.5385 0.9820 0.9130 0.8667LACE 0.7647 0.6250 0.5556 0.7826
![Page 20: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/20.jpg)
Machine Learning with Small Data Ø Apply statistical analysis to assess predictability with the features.
Ø Mitigate overfitting in modelsq Adopt simple models q Reduce the number of variables through feature selectionq Use an ensemble of models (Gradient Boosted Trees, AdaBoost)
Ø Handle imbalanced data setsq Random oversampling of the minority class(es)q Random undersampling of the majority class(es)q Synthetic minority over-sampling technique (SMOTE)
Ø Evaluationq Don’t just look at accuracy, especially for imbalanced datasetq Assess overfitting by comparing accuracy on training/testing data
12/19/2019 Chenyang Lu 20
![Page 21: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/21.jpg)
Predict Surgical Complications • Patients undergoing pancreatectomy are monitored with Fitbit before
surgery, during hospital stay, and 30 days post-discharge.
• Machine learning models predict complications and readmissions. • 54 patients enrolled (goal: 130).
PreoperativeClinic Surgery OutpatientRecovery
Joint work with Chet Hammill (Surgery), Dingwen Li, Ruixuan Dai (CSE)
12/19/2019 Chenyang Lu 21
![Page 22: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/22.jpg)
IoMT Use Cases
Ø Predicting readmissionsq Feasibility of monitoring with Fitbit – data yield and compliance
q Predictive models – machine learning with “small data”
Ø Measuring mobilityq Timed Up and Go with smartwatches
Ø Measuring stressq Ecological momentary assessment (EMA)
q Machine learning model based on physiological signals
12/19/2019 Chenyang Lu 22
![Page 23: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/23.jpg)
Timed Up and Go with Smartwatch
12/19/2019
Ø Watch appq Remind participants to take the assessmentq Automatically upload the data to the cloud for analysisq Analyze gait and motion featuresq Feedback to physicians and participants
Joint work with Matthew Spraker (Radiation Oncology), Ruixuan Dai (CSE)
Chenyang Lu 23
https://www.cse.wustl.edu/~lu/TUG.mp4
Ø Assess physical health and fall risk during prehabilitation.q 20 participants undergoing neoadjuvant radiotherapy followed by surgeryq Patients will complete TUG at home with the smartwatch for 90 days.
![Page 24: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/24.jpg)
IoMT Use Cases
Ø Predicting readmissionsq Feasibility of monitoring with Fitbit – data yield and compliance
q Predictive models – machine learning with “small data”
Ø Measuring mobilityq Timed Up and Go with smartwatches
Ø Measuring stressq Ecological momentary assessment (EMA)
q Machine learning model based on physiological signals
12/19/2019 Chenyang Lu 24
![Page 25: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/25.jpg)
Measure Stress with EMA and PPG Sensor
12/19/2019
Joint work with Thomas Kannampallil (Anesthesiology, Informatics), Michael Avidan (Anesthesiology), Eric Lenze (Psychiatry), Ruixuan Dai (CSE)
Chenyang Lu 25
![Page 26: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/26.jpg)
EMA in Daily Life Ø EMAs are auto fired 12 times/day with snooze option
q Buzz again when no response in 5 minutes (max: 3 buzzes)q 32 healthy participants
Ø 95.7% response rate (86.8% responded to the first buzz)Ø 85th percentile response time is less than 30s
12/19/2019 Chenyang Lu 26
![Page 27: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/27.jpg)
Do Stressors Work?
Ø Use EMA to validate stressors
12/19/2019
Efficacy of stressors vary à Test with watch-based EMA!
Chenyang Lu 27
![Page 28: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/28.jpg)
Predict Stress with PPG Signals
12/19/2019
ML model based on PPG signals is more consistent with EMA
Chenyang Lu 28
Stress
Sensitivity is fixed at around 0.9
Model Precision Sensitivity Accuracy
SVM(rbf) 0.28 0.86 0.73
Random Forest 0.25 0.86 0.69
Gradient Boosting 0.23 0.87 0.65
AdaBoost 0.21 0.86 0.61
Logistic Regression 0.31 0.90 0.74
SVM (Linear) 0.32 0.88 0.75
Label based on Stressors Label based on EMAsModel Precision Sensitivity Accuracy
SVM(rbf) 0.32 0.86 0.83
Random Forest 0.39 0.87 0.87
Gradient Boosting 0.28 0.87 0.81
AdaBoost 0.24 0.87 0.77
Logistic Regression 0.53 0.91 0.92
SVM (Linear) 0.55 0.87 0.92
![Page 29: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/29.jpg)
IoMT Use Cases
Ø Predicting readmissionsq Feasibility of monitoring with wearables – data yield and compliance
q Predictive models – machine learning with “small data”
Ø Measuring mobilityq Timed Up and Go (TUG) with smartwatches
Ø Measuring stressq Model based on physiological signals
q Ecological momentary assessment (EMA)
12/19/2019 Chenyang Lu 29
![Page 30: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/30.jpg)
Internet of Medical Things
A powerful tool for clinical studiesinside and outside hospitals!
12/19/2019 Chenyang Lu 30
Ø Wearables: wristbands, smartwatches... q Continuous monitoring: activity, heart rate, sleep, location…
Ø Connectivity: Bluetooth, WiFi, cellular…q Real-time monitoring and just-in-time intervention
Ø Cloud: computing and storage.q Scalable to large cohorts
Ø Analytics: machine learning and signal processingq Interpret data and predict outcomes
![Page 31: Internet of Medical Thingslu/talks/IoMT.pdf · Ø KNN performs better than neural network (NN) q Simple models help avoid overfitting on small data set. 12/19/2019 Chenyang Lu 19](https://reader034.vdocument.in/reader034/viewer/2022050223/5f68c2478c0f5069ec1cb00b/html5/thumbnails/31.jpg)
Internet of Medical Things
We can do a lot more when combiningwearables and electronic health records
12/19/2019 Chenyang Lu 31
Ø Wearables: wristbands, smartwatches... q Continuous monitoring: activity, heart rate, sleep, location…
Ø Connectivity: Bluetooth, WiFi, cellular…q Real-time monitoring and just-in-time intervention
Ø Cloud: computing and storage.q Scalable to large cohorts
Ø Analytics: machine learning and signal processingq Interpret data and predict outcomes