Going Beyond the Numbers The Midas+ Journey in Predictive Analytics

Vicky Mahn-DiNicola RN, MS, CPHQ,VP Research & Market Insights

Jim Kirkendall, MBA, VP Analytics

Learning Objectives

• Define “predictive analytics” and “predictive modeling”

• Explain “machine learning” and how it is used in predictive modeling

• Describe strategies for strengthening the power of predictive modeling

• Discuss the Midas+ Roadmap for Advanced Analytics

National Security and Law Enforcement

Numbers Geek Nate Silver

• Statistics guru

• Accurately predicted the

presidential winner in all

50 states in 2012

• Sports Statistics

• ESBN 548 Blog

Obama Campaign CTO Harper Reed

Predictive Analytics Sometimes Used in Spam Filtering

Systems to Identify the Probability that a Message is Spam

Meteorology Predictions

Self-driving Cars

Predictive Analytics: A Definition

•

•

How Will Advanced Analytics

Change Healthcare?

The “Big Question” about “Big Data”

Where can “big data” methods be best

applied to accelerate progress in

transforming healthcare delivery?

Can health care data be integrated in real

time and context aware?

Technical Limitations

Cultural Barriers to Access

Patient Engagement

Midas+ Advanced Analytics

Strategic Objective

11

• Create a scaling platform that combines clinical,

financial, claims, and public data from many sources

across many points of service for the purpose of

research and predictive capabilities.

• Deliver valuable ad hoc or subscription-based analytic

solutions across the healthcare industry.

5/14/2014

Advanced Analytics Foundational

Strategies at Midas+

• Build an analytics data warehouse

• Implement data modeling techniques

• Develop a data validation strategy

• Get the right people trained and focused

Analytics Data Warehouse Challenges

1) Data Volume and Marrying Many Data Sources

Effects of Data Volume on Machine Learning

Xerox Internal Use Only 14

• The best ML models perform worse with smaller data sets than more basic models with a vast amount of data.

Analytics Data Warehouse Challenges

1) Data Volume and Marrying Many Data Sources

2) Data Quality & Speed of Implementing Data

•

•

Analytics Data Warehouse Challenges

1) Data Volume and Marrying Many Data Sources

2) Data Quality & Speed of Implementing Data

3) HIPPA Laws

Analytics Data Warehouse Challenges

1) Data Volume and Marrying Many Data Sources

2) Data Quality & Speed of Implementing Data

3) HIPPA Laws

4) Codified Data from NLP to Achieve Universal Meaning

Analytics Data Warehouse Challenges

1) Data Volume and Marrying Many Data Sources

2) Data Quality & Speed of Implementing Data

3) HIPPA Laws

4) Codified Data from NLP

5) Decouple Analytics from Application

Analytics Data Warehouse Challenges

1) Data Volume and Marrying Many Data Sources

2) Data Quality & Speed of Implementing Data

3) HIPPA Laws

4) Codified Data from NLP

5) Decouple Analytics from Application

6) Include Descriptive Information with Predictive Alerts

Midas+ Advanced Analytics

Designed to be Solution Agnostic

Analytics Data Warehouse

Machine Learning Definition

• A branch of artificial intelligence

• The study of systems that can learn from data – Email filters to classify messages into spam and non-spam

– Amazon and Netflicks learn your preferences and preference of

others like you to suggest more movies and books you’ll like

– Cars that “learn” how to drive

• Some machine learning systems attempt to minimize the

need for human intuition in data analysis, but the intuition

of the data scientist is critical to specify how data will be

represented and which models will be used to search for

meaning

Null Hypothesis:

“Low Blood pressure does not increase a patient’s chances of

going to the ICU”

Null Hypothesis:

“Low Blood pressure does not increase a patient’s chances of

going to the ICU”

Null Hypothesis:

“Low Blood pressure does not increase a patient’s chances of

going to the ICU”

Statistical Learning can Help Us Compute

Correlations on LOTS of variables for a Prediction

Variable #1:

“Is Low Blood pressure a predictor for ICU admission? ”

Pr(>|t|)

Pr(>|t|) = .002 There is a strong correlation between low blood pressure

and ICU transfer

Modeling Techniques

Supervised Learning • Linear or Logistical Regression

• Lasso

• Neural Networks

• SVM (support vector machine)

• Random Forests/Boosted Trees

Unsupervised Learning • Hierarchical Clustering

Supervised Learning

= Socks

= Socks

= Socks = Shirts

= Shirts

= Shirts

Training Set

Supervised Learning

Shirt or Sock? ?

Result: 97% probability it’s a shirt.

Unsupervised Learning

Training Set

Unsupervised Learning

Group 1

Group 2 Group 2 Group 2

Group 1 Group 1

Training Set

Unsupervised Learning

Group 1 or Group 2? ?

Result: 97% probability it’s in Group 2.

How should we label Group 1 and Group 2?

Supervised Learning: Linear Regression

• Calculating the price of a home given age of the home,

square feet, school district, number of bedrooms

Supervised Learning: Logistical Regression

• Computes the probability of a

patient having cancer given

an examination of 800 cells.

• Designer of model can

determine the probability

threshold for when a

prediction of “yes” is made.

Supervised Learning: Lasso • Always better to use the fewest number of features in a model provided

removing features doesn’t lower your predictive abilities.

• When you are building a model and examining the use of a large

number of features, there are diminishing returns on using each

additional feature.

• Lasso helps you understand the performance impact of removing or

keeping each feature.

Supervised Learning: Neural Network

• The algorithm of choice for machine

driven cars.

• Works very well with audio, video,

and data.

• Can speed time manipulating data

since the benefit of the model is

that it learns important features of

the data through its hidden units.

• Training is computationally high.

Supervised Learning: Support Vector Machine

• High accuracy and with an

appropriate kernel they can work

well even if you’re data isn’t linearly

separable.

• Especially popular in text

classification problems where very

high-dimensional spaces are the

norm.

• Memory-intensive, hard to interpret,

and kind of annoying to run and

tune.

Supervised Learning: Random Forests &

Boosted Trees

• Random forests are starting to steal the crown.

• Used by Netflix to classify movies and predict user

preferences.

• They easily handle feature interactions and you

don’t have to worry about outliers or whether the

data is linearly separable.

Unsupervised Learning

• Hierarchical Clustering – Designer must choose number of clusters in advance

– Top-down or bottom-up approach. Top-down…all observations start in one

cluster, and splits are performed recursively

– Model measures the distance between pairs of observations and a

measures the dissimilarity of sets as a function

Two Considerations for Data Validation

• Warehouse – Data used to train the predictive

models must be accurate and

complete if we expect great

results

– Validate interfaced data and

structured data derived from

NLP engines

• Predictive Model – Need to validate the strength of

the prediction

– Need to demonstrate value to

clinicians

Warehouse Data Validation • Validity of data interfaced from the EMR or other sources

• Check the confidence of data sent across interface

• If outside of 2 Standard Deviations we convert it or don’t use it

• Performed on drugs, vitals, and other data elements

Hospital What Client Sends What Midas+ Stores

Hospital 1 98.6 F 98.6 F

Hospital 2 37 C 98.6 F

Hospital 3 377 C NULL

Hospital 4 37 F 98.6 F

Hospital 5 98.6 C 98.6 F

Warehouse Data Validation Check completeness of data by facility and encounter type

Encounter Type Hospital 1 Hospital 2 Hospital 3 Hospital 4

Inpatient 98% 99% 45% 99%

Emergency 22% 98% 88% 90%

Observation 78% 95% 90% 95%

1. What percentage of patients had a History & Physical?

2. What percentage of patients had a pain score?

Encounter Type Hospital 1 Hospital 2 Hospital 3 Hospital 4

Inpatient 95% 99% 95% 99%

Emergency 92% 95% 98% 0%

Observation 98% 95% 92% 94%

Data

completeness

monitoring

serves as a

“Force

Multiplier”.

Data Integrity

Oversight

for all Midas+

Applications

and Future

Predictive

Analytics!

Data Validation on

Predictive Model

• Sensitivity (True vs. False Positives)

• Specificity (True vs. False Negatives)

• How well are we handling missing data (Sparcity)

• How to validate alerts against clinical record

• How to demonstrate clinical value in the alert

Interpreting Predictive Classification Results

True Positive (Sensitivity)

False Positive

False Negative

True Negative (Specificity)

Four possible outcomes in Predictive Analytics

Statistical Primer for Predictive Analytics

True Positive (Sensitivity)

False Positive

False Negative

True Negative (Specificity)

Four possible outcomes in Predictive Analytics

Patient tests positive for the

disease but doesn’t really have it!

Statistical Primer for Predictive Analytics

True Positive (Sensitivity)

False Positive

False Negative

True Negative (Specificity)

Four possible outcomes in Predictive Analytics

Patient tests negative suggesting they are

healthy but they actually have the

disease

Statistical Primer for Predictive Analytics

True Positive (Sensitivity)

False Positive

False Negative

True Negative

(Specificity)

100%

0%

True Positive

Rate

False Positive Rate

ROC Curve is the Area Under the Line Also known as the C-Statistic

True Positive (Sensitivity)

False Positive

(Specificity)

False Negative

True Negative

100%

0%

True Positive

Rate

1.0 = Perfect

Line of no-discrimination

Points above the

Diagonal line Represent better

Than random

Points below the Diagonal line

Represent Worse Than random

0.5 = Random Coin Toss

False Positive Rate

CMS Readmission Calculator

C-Statistic = .60 to .63

100%

0%

True Positive

Rate

CMS Prediction Model Using Logistic Regression and Hierarchical logistic regression models on 2008 to 2010 data

performed better than random guessing

False Positive Rate

Acute MI = .63 Heart Failure = .60 Pneumonia = .63

Sparcity (missing data)

0700 am vitals:

BP 120/80

P 88

R 20

T (not documented)

We could use the previous temperature, use the next temperature, ignore it or

compute it based on a rule……

Nearest Neighbor Algorithm

• Modeling technique to

calculate the missing

data point.

• Take the mean of the

values with patients with

similar vital sign

“constellations”.

Developing Talent and Skills • Foundational Competencies

– Data warehousing

– Statistical Modeling

• Specialized Skills & New Learning – Learn new statistical software packages e.g. R, Matlab, Julia

– Learn PMML (Predictive Model Mark Up Language)

– Sytrue (NLP engine) integration

– Intersystems Solutions • DeepSee (cubing application for complex indicators)

• iKnow (NLP engine)

– Data Integration across multiple EMR systems in the US

• Insight Validation by Clinical Stakeholders

2014 Advanced Analytic Projects

50

Predictive ICU Admission Service

Predictive Readmission Service

Predictive Denied Days Service

Risk of Mortality and Severity of Illness

5/14/2014

Clinical Decline and ICU Transfer

Why the Urgency for a

Predictive ICU Admission Service?

• ICU admissions in U.S. have increased nearly 50% over six years

• 2.79 million in 2002-2003 to 4.14 million in 2008-2009

• ICU admissions contribute nearly 30% of all hospital costs

• 1 out of every 3 dollars spent on healthcare is “critical care”

• Traditional cost containment approaches to decrease critical care resource

utilization have been challenged as cost shifting instead of cost reducing: – Aggressive triage and discharge criteria

– Earlier discussion of end of life decision making and palliative care

– Earlier use of alternative care settings for longer term intensive care needs.

• A predictive algorithm is needed to improve patient outcomes, lower patient

comorbidities, and reduce hospital costs

• Clinicians need real-time actionable intelligence to impact decision making

52

Developing Our Hypothesis Using the

Modified Early Warning Score (MEWS)

1. Patients show clinical signs of

deterioration before going into

cardiac or respiratory arrest

2. Vital signs and lab

measurements can be used to

determine these signals of

clinical deterioration within 4

hours of the event

9 Month Pilot Using MEWS to Identify

Patients at risk for “Code Blue”

• Peninsula Regional Medical Center in Salisbury, Maryland (317 Bed

Tertiary Care)

• Over initial 9 month pilot there was a 67% decrease in code blues and

76% increase in rapid response team calls on initial medical surgical pilot

unit

• No code clues or mortalities in 3 months on a cardiac telemetry unit

• Pilot expanded to six medical surgical units resulting in 64% decrease in

code blues and 55% increase in rapid response calls

• PRMC estimates a potential annual savings of $3.2 million

54

http://www.mckesson.com/about-mckesson/newsroom/press-releases/2013/peninsula-

regional-medical-center-wins--mckesson-award-for-clinical-excellence

Modified Early Warning Score

• A simple guide used by hospital nursing and medical staff as well as emergency medical

services to quickly determine the degree of illness of a patient. It is based on data

derived from four physiological readings.

5/14/2014 55

• Alert-Voice-Pain-Unresponsive (will use in our 2nd iteration)

• Individual component scores are summed. Patients with a MEWS score of 5 or

higher are statistically linked to increased likelihood of death or admission to an ICU

(intensive care unit).

LTHT MEWS Leeds Teaching Hospitals Trust, England 2011

Can Machine Learning do better than MEWS?

5/14/2014 57

If a patient has a heart rate of 110, respiratory rate of 20, and temperature of

38.4 degrees they get 2 points. However, if that patient instead has a heart

rate of 111, respiratory rate of 21, and temperature of 38.5, they get 6 points.

Are these range cutoffs the best we can do?

Should the scoring model be identical regardless of the patient’s condition?

Can we incorporate additional pieces of data?

Next Iteration: Midas+ ICU Predictives Version 2

• Laboratory Results

– Hemoglobin

– Hematocrit

– Platelet Count

– Creatinine

– Sodium

– More

• Medications (all)

• Pain (normalized)

• Methodology Changes

– Temporal optimization

– Excluding patients that expired (used to be considered positive but we’re

concerned that the patients who died may

be skewing the learning….TBD)

– ICD-9 codes generated from NLP

– SNOMED codes

– MS-DRG assignment and relative

weights to active patients “in-

house” derived from ICD-9 codes

derived from NLP (used for real time

risk adjustment)

Initial Predictive ICU Admission Progress

• Developed temporal models for: – Blood Pressure -Temperature

– Heart Rate -Respiratory Rate

– 02 -Urine Output

• Only MEWS scoring on test set – identifies 217 of 862 ICU patients (25.17% sensitivity)

– 1215 false positives

Predictive ICU Admission Progress

with Machine Learning…

5/14/2014 60

• Using additional variables on test set

including attending physician specialty,

age, sex, time of admission.

– Identifies 433 of 862 ICU patients

(50.23% sensitivity)

– 190 false positives (99.1 % specificity)

– 7/10 alerts are true positives

Clinical Validation

• Three Groups – True Positives

• We predicted the patient needed ICU and went.

• If the alert had actually been “live in production”, would earlier

intervention likely have made a difference?

– False Negatives

• We did NOT predict the patients needed ICU but they went.

• Was there something special about this group that we missed?

– False Positives

• We predicated the patient needed ICU and they did NOT go

• Where there any interventions that we could see which would help us

train our learners?)

Training the Learners About Interventions that

May Reduce ICU Admissions

• Nasopharyngeal/oropharyngeal

suctioning

• Additional oxygen

• IV fluid bolus

• IV furosemide (Lasix) bolus

• Non-invasive positive pressure

ventilation

• Respiratory Treatments (albuterol)

• Narcan

• D50

• Vitamin K

Lessons Learned • Time of vital signs,

assessments or interventions

by nursing do not always match

other clinical data e.g. progress

notes, medical orders

• Times may reflect data

validation rather than actual

time of occurrence

• Data entry practices vary

across nursing units

• Workflow has to change so that

vitals and clinical observations

are entered into computer at

point of care immediately after

assessments

2014 Advanced Analytic Projects

64

Predictive ICU Admission Service

Predictive Readmission Service

Predictive Denied Days Service

Risk of Mortality and Severity of Illness

5/14/2014

Integrated Risk Adjustment

• Hierarchical clusters to create homogenous

clinical risk groups

• Risk of mortality and severity of illness scores

are computed for each individual encounter.

• Expected LOS and Expected Charges will be

computer for each encounter.

• Methodology can be run while the patient is in

the hospital and/or immediately after final

coding.

• Model is easily deployable within a Cache

infrastructure via a series of cache object script

routines for ease of maintenance.

We have the

data…..now what?

Time for Questions