1

Using NLP to Identify Physician Documentation Opportunities

Session #241, February 23, 2017

Anupam Goel, VP, Clinical Information, Advocate Health Care

2

Speaker Introduction

Anupam Goel, MD

Vice President, Clinical Information

Advocate Health Care

3

Conflict of Interest

Anupam Goel, MD has no real or apparent conflicts of interest to report.

4

Agenda• Learning objectives, STEPS

• Introduction

• The promise and reality of Natural Language Processing (NLP)

• Using NLP to maximize value of human input for appropriate case detection

• Identifying patients with stroke

• Progress to date

• Other opportunities

• Next steps

• Your questions

5

Learning Objectives

• Describe risks and benefits of using natural language processing for identifying gaps in physician documentation

• Recognize how variations in local practice that may impair the ability of natural language processing to address specific use cases at your facility or health system

• Identify methods to display quantitative information about physician documentation quality to highlight patterns and encourage physician behavior change

6

STEPS argument

7

STEPS – Treatment/Clinical• Identify patients sooner to initiate appropriate diagnostic or treatment

algorithms (stroke, CHF)

• NLP must be embedded within a clinical notification workflow that activates a team member to make a change in clinical care

8

STEPS - Savings• In its current state, the technology is probably best at highlighting

patterns for a human to verify.

• For high-risk or publicly reported conditions, many health systems employ multiple FTEs to sift through physician documentation. NLP could help redeploy those resources to complete other tasks.

http://www.integrity-data.com/improve-productivity-employee-satisfaction-deploy-better-technology/

9

Introduction

http://simsa.dsu.dal.ca/tag/orientation

10

What is Natural Language Processing?• Field of computer science and linguistics concerned with the interactions

between computers and human (natural) languages

• Extracting concepts from free-text

• In its most basic form, it is not a tool

• To identify improper diagnosis or treatment

• To judge documentation content quality

• To determine if health care resources are being wasted

11

Common error modalities with NLP• Misinterpreting errors in syntax and grammar

• Misinterpreting errors in content (voice-to-text errors)

• Misinterpreting context around specific terms

http://computerdocnc.com/computer-errors-indian-trail-nc/

12

The promise and reality of NLP

13

Positive impressions of NLP

https://developer.amazon.com/blogs/post/TxC2VHKFEIZ9SG/First-Alexa-Third-Party-Skills-Now-Available-for-Amazon-Echo

14

NLP sophistication

• Bag of words

• Concepts

• Negation, association

• Identifying themes

• Suggesting alternative pathways

15

What’s your gold standard?

https://www.macobserver.com/tmo/article/head-to-head-comparison-of-13-streaming-music-services

16

How do we measure how good NLP might be?

http://gru.stanford.edu/doku.php/tutorials/sdt

17

Vendor promises

• More complex diagnoses (clinical documentation improvement)

• Real-time feedback for documentation quality

• Improved billing accuracy

18

How good do you need NLP to be?• Immediate life-and-death decisions?

• Retrospective reviews

• Billing and documentation improvement

19

“Secret sauce” for NLP• How do you train an NLP algorithm?

• Variations in documentation style

– Pertinent negatives

– Pertinent positives

• “Black box” algorithm

20

Risks for NLP from a clinician’s perspective• Minimal documentation limits the software’s ability to meaningfully

discriminate between “wanted” and “unwanted” cases

• Voice recognition technology may enable more comprehensive documentation, improving NLP’s effectiveness

• Cases that are difficult for humans to distinguish will also be difficult for NLP algorithms to distinguish

21

Using NLP to maximize value for human input for appropriate case detection• Copy-and-paste

• Missing operative report elements

22

Example 1: Copy-and-paste

• Some physicians abuse computer technology to meet documentation requirements

• There is no easy way to manually review every note that is entered in an electronic medical record for “identical-ness”

• Technology could highlight high levels of identical text without determining if the similarity was appropriate or not

23

What’s the driver?

FRAUD

24

How much similarity is too much similarity?• Specialty

• Patients with a prolonged hospital stay

• Some sections of the note hardly change without adverse effects on patient care

• Absolute cutoff of >95% to trigger a manual review

25

Who should be targeted?

• High absolute numbers of similar notes

• High proportion of similar notes

26

So What?

• Is the copy-and-paste appropriate?

• Is there an escalation path?

• What about continued “bad behavior?”

27

Copy and paste

28

Copy and paste

29

Next steps

• Target sections of a note

• Identify “value-adding” team members

30

Example 2: Operative report completeness• Challenge: capture missing elements within an operative report before the

patient is discharged

• Data elements often scattered throughout the surgeon’s operative report or across multiple documents

31

Defining the note “population”

• What constitutes an eligible note?

• What euphemisms are used by different surgeons to imply different required elements?

• EBL

• Blood loss

• Approximate bleeding

32

Data display

• Number of missing elements

• Number of incomplete reports

• Surgeons with the highest percent of incomplete notes

33

Next steps

• Use the operative report extractions to identify quality metrics to drive quality improvement metrics

34

Identifying patients with stroke

35

How do you identify patients with a stroke?• Current state: manual review of patients admitted to specific floors

• About four FTEs

• Limited bandwidth to review patients across the hospital

• Public reporting implications

36

How do you identify a patient with stroke?

37

Order matters

• When reviewing 200 charts, wouldn’t it be great if the first 40 were the ones most likely to include a stroke?

38

Assessing the probability for stroke

• What terms are most consistent with a stroke?

39

Patient presentation changing over time• Day 1: not a stroke

• Day 2: not a stroke

• Day 3: a stroke

40

Progress to date

41

“Sharpening the tool”

• Ongoing improvements to algorithm by using reviewers to provide new “gold standard” information as additional cases are reviewed

42

Other opportunities

43

Next steps• Apply to other clinical scenarios

– Care pathways not followed

– Identify physicians who are outliers based on specific documentation elements

– Flag patients for aggressive follow-up after discharge

44

Summary

• NLP can provide technological support to help augment manual workflows instead of replacing existing workflows

• NLP may be most effective when targeting case-finding

• Gaining proficiency through low-risk scenarios may improve confidence in the technology without large-scale adverse events

45

Summary

• NLP is only as good as the information that is available for review

• Training an NLP algorithm can take time, but early investments can pay off in FTE savings

• Think about simplifying the data display of information from NLP analysis to facilitate action

• Rather than thinking of NLP as a tool with static operating characteristics, consider using new information to continually tweak the algorithm to improve the software’s yield

46

A Summary of How Benefits Were Realized for the Value of Health ITTreatment/Clinical – begin diagnostic and treatment pathways faster

Savings – FTEs allocated to more productive uses

47

Questions?

• Email: anupam.goel@advocatehealth.com

• Twitter: @anupam1623

• Linkedin: goelanupam

• Please complete the online session evaluation