Is it really a setback, in general, or not?March 1, 2017

Dale Sanders Executive Vice President, Software

Forbes Magazine: “MD Anderson Benches IBM Watson In Setback For Artificial Intelligence In Medicine”

Let’s Make Things Very Clear• IBM and Watson are a frequent competitor to Health Catalyst• I do NOT celebrate the difficulties of our competitors, especially IBM• “There but by the grace of God, go I”• Watson’s success begets Health Catalyst’s success• Were it not for IBM, I wouldn’t have a career in information

technology• IBM was the backbone of the Air Force information systems that taught me so

very much

Opening Salvo to Stir Things Up • Tying Watson to a “cancer moonshot” created the peak of already inflated

expectations about Watson• Every executive and politician wants to be John F. Kennedy

• We have a generation of political and corporate executives who don’t understand technology and software, even though it’s running their world

• Executives are selling technology they don’t understand and executives are buying technology they don’t understand

• Information asymmetry always leads to an exploited consumer

• Technology professionals have a moral and ethical obligation to speak up when they see this happening

Agenda• My background as it relates to this topic• The fundamental data challenges of applying Watson to healthcare• Health Catalyst’s approach to machine learning and AI in healthcare

• I’m not selling here… I’m just informing you about a different approach• History will be the judge about whether the Catalyst approach works or not

• These slides are purposely bland… this webinar is not about selling Health Catalyst

Data, data, data… for decision supportMy Background

1983 2016

B.S. Chemistry, biology minor

US Air Force Command, Control, Communication, Computers & Intelligence (C4l) Officer

Reagan/Gorbachev Summits

TRW/National Security Agency• START Treaty• Nuclear Non-proliferation• Nuclear command & control

system threat protection• Knowledge Based Systems

Commercialization

Nuclear Warfare Planning and Execution-- NEACP & Looking Glass

Intel Corp, Enterprise Data Warehouse

• Chief Data Guy• Regional Director of

Medical Informatics, Intermountain Healthcare

• CIO, Northwestern

• Chief Data Warehousing Guy

CIO, Cayman Islands National Health System

Product Development, Health Catalyst

The Over-Hype of AI in the 1990s• I lived it. I hyped it.• Military and credit reporting systems managed the largest databases

in the world at the time• They pale in comparison to Silicon Valley data content today

• My team at TRW, the Knowledge Based Systems Group, was tasked with commercializing our military and intelligence technology in expert systems, fuzzy logic systems, neural nets, and genetic algorithms

• Our first target was healthcare. Sound familiar?

I presented the following six slides at a conference in Feb 2012, exactly one year after Watson’s victory on Jeopardy, when hopes for Watson were very high in medicine. I was a fairly lonely contrarian.

8

What About Watson?

Watson

First, a little background on Dale Sanders Natural Language Processing and Text Mining

Watson is revolutionary. It’s the first thing in my IT career that really excited me… everything

else has been incremental or variations of the same flavor

9

10Watson’s Technology

Apache Unstructured Information Management Architecture (UIMA) Hadoop Java, C++

Lexicals and ontologies  DBPedia, WordNet, and Yago 

IBM Content Analytics with Enterprise Search 90 IBM Power 750 servers enclosed in 10 racks 16 Terabytes of memory A 2,880 processor core Linux based

11What is Watson?

Near-word associations coupled with semantic mapping and zillions of sources of knowledge… digitized books, encyclopedias, news feeds, magazines, blogs, Wikipedia, etc.

Equivalent to approximately 240 million pages, in memory

Jeopardy answer “A famous red quaffed clown or just any incompetent fool”

Watson’s correct answer “Who is Bozo?”

Watson searched its indexes for near-word associations, recognized that Bozo was the most common word in the indexes that was missing from the question

12Watson’s Problem With Healthcare

Watson’s training set for Jeopardy was a HUGE collection of human wisdom, academic and otherwise, stretching back thousands of years

What’s the training set for healthcare wisdom? A few decades of clinical trials journals? Claims processing data from a dysfunctional healthcare system that doesn’t include patient

outcomes? Progress notes? Radiology reports? Pathology reports?

Watson is not going to impact healthcare in the near term like many hope it will

13Factoids

More than 50% of all medicines are prescribed, dispensed or sold inappropriately

Less than 40% of patients in the public sector and 30% in the private sector are treated according to clinical guidelines

World Health Organization, May 2010

Key Points• Watson is a text-centric, Natural Language Processing (NLP) engine

• Millions of “near word associations” are processed in seconds

• Although related at some level, that’s different than a generic pattern recognition approach to machine learning used for discrete data and images

• NLP: ”Find things for me faster in all this text.”

• Machine Learning: “Make decisions and suggestions for me, and learn from each decision and suggestion.”

Key Points• 80% of healthcare data is text-based clinical notes and diagnostic

reports, if you don’t count digital images, but that’s still not very much data in terms of sheer volumes, and the quality and consistency of that data varies considerably across clinicians

• The source of Watson’s primary knowledge base in healthcare-- peer-reviewed journals and clinical trials data-- is relatively small in terms of volume and has questionable value in day-to-day healthcare

• Watson’s training set for Jeopardy was at least 100x larger than what’s available to train Watson for healthcare

IBM Watson “Learning” Acquisitions• Phytel• Explorys• Truven• Merge• If the fundamental design of Watson is NLP and text-centric, will these

acquisitions help Watson learn?

Is training Watson on chemotherapy and radiation therapy protocols the right strategy for treating and

preventing cancer?I would argue that it’s not. Current cancer treatment strategies will go down in history alongside bloodletting and trepanation. We need to

apply Watson and similar technology breakthroughs on something other than optimizing the status quo, which is anything but great.

The Cancer Data EcosystemThis is the data you need to prevent and treat cancer. Do we have this data in high volume, across many patients, with reasonable quality and consistency?

No.

• Genomics

• Lifestyle

• Epigenetics

• Microbiome

• Environmental

• Traditional healthcare delivery data

• Quality and length of life outcomes data for long-term survivors

• All the above on healthy patients so we understand the target condition

Health Catalyst’s Approach to AI and Machine Learning

Semantics• Machine learning is one thing. Machine doing is another.• In my definition, it’s not Artificial Intelligence until the machine acts

on your behalf.• We’ll get there in healthcare, but it will take a long time.• In the meantime, I prefer “Suggestive Analytics” based on machine

learning.

Our Simple MissionOur mission is to organize the data in healthcare and make it accessible,

useful, and valuable to the clients, patients, and families we serve.

With data, all things are possible. Without it, not much.

Our fundamental strategy for Machine Learning:

Integrate text and discrete data to inform the vectors and clusters in our models

Your machine learning aspirations must be tempered by the data that’s available, both in breadth and depth.Ironically, it’s easier for us to model and predict bad things in healthcare right now, than good things. We have more data about bad outcomes than good outcomes.

No Data, No Machine Learning• Moore’s Law: Chips double in capacity every 18 months• Sanders’ Law: Machine learning models double in capability every 6

months• But without data content, the models are of no use

25

For the most part, this is the simple three-part pattern recognition model that we are building and that, I would argue, healthcare should broadly pursue

Patients like this [pattern]

Who were treated like this

[pattern]

Had these outcomes and costs [pattern]

The Human Health Data Ecosystem

And, by the way, we don’t have much of any data on healthy patients

We Are Not “Big Data” in Healthcare, Yet

27

Data Volume vs. Machine Learning Model

“But invariably, simple models and a lot of data trump more elaborate models based on less data.”

•“The Unreasonable Effectiveness of Data”, March 2009, IEEE Computer Society; Alon Halevy, Peter Norvig, and Fernando Pereira, of Google

28

29

Google’s Self Driving Car drove 80 million miles before it ever touched a road

Think of a computer sitting in the seat of this computerized driving simulator, not a human

30

Retina: The data collection system for

Feature extraction

Cerebral cortex: The data base and algorithms for Classification

& Clustering

The more times you go through this loop with different ”data”, the faster

and better you become at feature extraction and classifying “people”

31

Pattern recognition process

Data acquisition

Data reduction

Feature extraction

Classification & Clustering

Confidence evaluation

EHRs, billing, outcomes data, lab, meds, vitals, supply chain, et al

Cleaning out the noisy or bad data, identifying general patterns

These are properties of the object. Finding new and specific ways to identify new categories and representations of patient types, outcomes, events, encounters, episodes

Using the features to assign patterns to the categories and representations

Evaluating and correcting the confidence in the model’s output

32

The challenges in healthcareData acquisition

Data reduction

Feature extraction

Classification & Clustering

Confidence evaluation

Very limited data. We think we are big data, but we’re not and generally, what limited data we have, is about sick patients, not healthy patients.

How, then, do we extract Features that Classify a healthy patient so we know how to achieve that “Healthy Patient” pattern?

If we don’t collect outcomes data, how then do we identify the Features to Classify a healthy or sick patient with good or bad outcomes?

33

ess of Predictive AnalyticsThe Machine Learning loop

33

• In healthcare, we have, essentially, no outcomes data, so this is an open loop• If you don’t have a strategy for intervention, predicting something for the sake of

predicting has no value

Troubling factoid

• Of the 1,958 quality metrics in the National Quality Measures Clearinghouse, only 7% of those

measure clinical outcomes and less than 2% of those are based on patient reported outcomes

34N Engl J Med 2016; 374:504-506, February 11, 2016

Thank you for the graphs, PreSonus

Healthcare and patients are continuous flow, analog process and beings

But, if we sample that analog process enough, we can approximately recreate it with digital data

35

We are treating physicians and nurses as if they were digital sampling devices.

“Every new click of the mouse you guys ask me to do, all in the name of data, sucks another piece of my soul away.” --Beleaguered primary care physician

37

Predictive and suggestive analytics in the same

user interface

The efficacy and costs of antibiotic protocols

for inpatients

The Antibiotic Assistant at Intermountain Healthcare: The First Triple Aim

Antibiotic Protocol Dosage Route Interval Predicted

EfficacyAverageCost/Patient

Option 1 500mg IV Q12 98% $7,256

Option 2 300mg IV Q24 96% $1,236

Option 3 40mg IV Q6 90% $1,759

• Antiinfective drugs• Average Savings per Patient = $280

• Cost of Hospitalization• Average Savings per Patient = $13,759

• Annual Savings (12-bed ICU)• Est. Total Savings per Year = $7,925,184

New England Journal of Medicine January 22, 1998

Economic Impact

• 30% reduction in Adverse Drug Events• 27.4% reduction in Mortality• 99.1% “on-time” delivery of pre-operative antibiotics• 84.5% reduction in post-operative antibiotic use• Stabilized antibiotic resistance

Annals of Internal Medicine May 15, 1996

Quality of Care Impact

The Shark Tank Story

• Chicago-based healthcare IT startups• Three hours of 15 minute presentations• Incredibly creative ideas at the

application layer of technology• Absolutely no answer for, or conceptual

understanding of, the challenges at the healthcare data layer

41

This is not an HIE, Clinical Data Repository, or Enterprise Data Warehouse. It’s a little bit of all three but better.

Health Catalyst Data Operating System

Kernel

Metadata

Data Ingest

Real-time Streaming

Machine Learning

NLP

Source Connectors

Catalyst Analytics Engine Core Services

Data Processing

Secure Messaging

Security, Identity& Compliance

Health Catalyst Fabric

RegistriesTerminology & Groupers

EHR Integration ISVsPRBLeading Wisely

Catalyst Apps

Care Management

Apps

Alerting FHIR

Big Data

SAMD & SMD

Measures Patient & Provider Matching

Atlas

Risk Classifications

Patient Attribution

Data QualityData Governance

Data Pattern Recognition

Data Export

43

New Generation Product Briefing

Health Catalyst Data Operating SystemMachine Learning Foundation1

catalyst.ai

• Our machine learning models• Our strategy for embedding machine

learning into all of our products

2 healthcare.ai

• Our tools to automate machine learning tasks

• Democratizing machine learning by releasing as open-source

3

44

healthcare.ai

Our open-source machine learning software product

Automates key tasks in developing models, or customizing existing models using local

data

Makes deployment in an analytics

environment easy and ‘production quality’

45

New Generation Product BriefingScaling People

Data Architects

Great domain knowledge Often looking for opportunities to advance career/skills

With the right tools…

Data architects make great feature engineers Data architects can easily get started in predictive

analytics.

With healthcare.ai, you have the people to do data science right now.

46

The healthcare.ai project listCentral Line-Associated Bloodstream Infection (CLABSI) Risk – Clinical Decision SupportCongestive Heart Failure, Readmissions Risk – Clinical Decision SupportCOPD, Readmissions Risk – Clinical Decision SupportRespiratory (COPD, Asthma, Pneumonia, & Resp. Failure), Readmission Risk – Clinical Decision SupportPredictive Appointment No-shows – Operations and Performance ManagementPre-surgical Risk (Bowel) – Clinical Decision Support and client requestPropensity to Pay – Financial Decision SupportPatient Flight Path, Diabetes Future Risk – Clinical Decision SupportPatient Flight Path, Diabetes Future Cost– Clinical Decision SupportPatient Flight Path, Diabetes Top Treatments – Clinical Decision SupportPatient Flight Path, Diabetes Next Likely Complications (Glaucoma) – Clinical Decision SupportPatient Flight Path, Diabetes Next Likely Complications (Retinopathy) – Clinical Decision SupportPatient Flight Path, Diabetes Next Likely Complications (ESRD) – Clinical Decision Support

In Development

Built

Planned

Sepsis Risk – Clinical Decision SupportPost-surgical Risk (Hips and Knees) – Clinical Decision SupportCharge-denial Risk – Financial Decision SupportCharge-grouping Guidance – Financial Decision SupportPredictive ETL Batch Load Times – PlatformHospital Length of Stay – Operations and Performance ManagementHospital Census – Operations and Performance Management

CAUTI and VTE – Clinical Decision Support Risk-adjusted Comparisons Across Health Systems – CAFÉ1-yr Admission Risk – Population Health and Accountable CareBronchiolitis Admissions Risk – Clinical Decision SupportEmergency C-section Risk – Clinical Decision SupportPalliative Care vs Invasive Procedure Guidance – Clinical Decision SupportMortality Risk in Pre-term Births – Clinical Decision SupportRegistry Automation via Unsupervised Learning – Clinical Decision SupportMortality Risk in PICU – Clinical Decision Support

47

Predictive SeedlingsBronchiolitis Admissions Risk

Emergency C-section Risk

Palliative Care vs Invasive Procedure Guidance

Mortality Risk in Pre-term Births

Mortality Risk in PICU

Deep Learning for Large Tabular Data (1M+ rows)

Patients Like This – Modifiable Risk-factor Recommendation for Patient Attributes

Patients Like This – Optimal Treatment Recommendation

Registry Automation via Unsupervised Learning

Radiology Image Classification via Deep Learning

Pathology Image Classification via Deep Learning

Currently possible with healthcare.ai and the right data

Roadmap for healthcare.ai

In Summary• Watson was overhyped, overbought, oversold… Not maliciously, but

rather, probably naively• But it will have a big impact on society• Healthcare data ecosystem is just not quite ready for Watson,

especially the text content that Watson thrives on• We have a bright future ahead for machine learning in healthcare, if

you adjust your strategy and expectations according to the data content that’s available