patient matching ehr ailments: going from placebo to cure ... · •patient records are scattered...
Post on 07-Oct-2020
0 Views
Preview:
TRANSCRIPT
Patient Matching EHR Ailments: Going from Placebo to Cure
Tuesday, March 1st 2016 Adam W. Culbertson, Innovator-in-Residence HHS, HIMSS
Keith J. Miller, Chief Scientist for Identity Intelligence, MITRE
Approved for Public Release; Distribution Unlimited. Case Number 15-4026 ©2016 The MITRE Corporation. ALL RIGHTS RESERVED.
Conflict of Interest
Adam W. Culbertson, MS and
Keith J. Miller, PhD
Have no real or apparent conflicts of interest to report.
Agenda
• Background
– History of Matching
– What is Patient Matching
• Challenges in Matching
– Data Availability
– Data Quality
• Test Evaluation Framework
• Metrics for Algorithm Performance
• Creating Test Data Sets
• Explain why patient matching is a multi-step process requiring a
strategy and not a “one size fits all” solution, the main steps in
developing this strategy, and why determining quality of the data is
key for effective patient matching
• Demonstrate how the framework helps address the multiple steps
needed for an effective patient matching strategy, such as an
understanding of the data and the tradeoffs involved in a good
matching strategy, and why different matching strategies may be
needed for different populations
• Demonstrate how an organization can gain a better understanding
of their data through use of the “Data Variant Taxonomy” and data
characterization tool suite without requiring ongoing hands-on access
• Describe why a gold standard data set is required for a good test
framework, allowing for an “apples-to-apples” comparison of patient
matchers and the issues involved in producing this data set using the
data variant taxonomy
Learning Objectives
• Electronic: Patient matching has been identified as a
key barrier to Interoperability in ONC’s nationwide
Health IT Roadmap
• Prevention & Patient Education: Reduction in patient
safety events caused by missing or incorrectly
matched records
• Patient Engagement/Population Management: More
complete records gathered across disparate health
systems
• Savings: Missing information and reordered tests cost
over $8 Billion annually. Improvement in patient
matching can reduce this cost.
• Improvements in patient matching can reduce deaths
healthcare costs and fraud caused by incorrectly
matched data
How Patient Matching Benefits Health IT
Background
Significant Dates in (Patient) Matching
A Framework for Cross-Organizational
Patient Identity Management
2015
Kho, Abel N., et al Design and
Implementation of a Privacy Preserving
Electronic Health Record Linkage Tool
HIMSS Patient Identity
Integrity
Grannis, et al Privacy and Security
Solutions for Interoperable Health Information
Exchange
2009
Joffe et al A Benchmark Comparison
of Deterministic and Probabilistic Methods for Defining Manual Review
Datasets in Duplicate Records Reconciliation
Dusetzina, Stacie B., et al Linking Data for Health
Services Research: A Framework and Instructional Guide
HIMSS hires Innovator In Residence (IIR) focused
on Patient Matching
Audacious Inquiry and ONC
Patient Identification and Matching Final Report
2014
HIMSS Patient Identify Integrity Toolkit,
Patient Key Performance
Indicators
Winkler Matching and
Record Linkage
2011
Newcombe, Kennedy, & Axford
Automatic Linkage
of Vital Records
1959
Dunn Record Linkage
1946
Soundex US Patent
1261167
1918
Fellegi & Sunter A Theory of
Record Linkage
1969
Grannis, et al Analysis of Identifier Performance Using a Deterministic Linkage
Algorithm
2002
Campbell, K et al A Comparison of Link Plus, The Link King, and a “Basic”
Deterministic Algorithm
RAND Health Report
Identity Crisis: An Examination of the Costs and Benefits of a Unique Patient Identifier for the US Health Care System
2008
Patient Matching Definition
Patient matching: Comparing data from multiple
sources to identify records that represent the
same patient.
• In Healthcare involves matching varied
demographic fields from different health data
stores to create a unified view of a patient.
Identity Matching / Identity Resolution
Identity analysis:
link analysis, data mining
Identity resolution:
Merge/dedupe records
Identity matching
Measure record similarity. Search/retrieval
Attribute matching Compare name, DOB, COB, address, etc.
Identity data
repository
Structured and unstructured
data sources
“Patient had an onset of diabetes, which is
accompanied by an odd change in race, and
the medication worked extremely well, and
in subsequent visits no longer occurred.”
Wes Richel: ONC HIT Privacy and Security Tiger Team Hearing, December 2010
Challenges in Matching
Challenges
• Lack of adoption of metrics
• Data availability
• Patient records are scattered across the health care
system in various data silos including; laboratory
systems, hospitals and primary care provider
EMRs.
• Differences in electronic health record vendors
– Data attributes collected
– Variation in output formats
– 12/01/1985, 12-01-1985, 01-12-1985
Availability of Data Attributes
% Availability of Attributes Over Region
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Firs
t N
ame
Mid
dle
Nam
e
Last
Nam
e
Dat
e o
f B
irth
Bir
th Y
ear
Gen
der
Soci
al S
ecu
rity
Nu
mb
er
Ad
dre
ss (
full)
Stre
et A
dd
ress
Lin
e 1
Cit
y
Stat
e
Po
stal
Co
de
Co
un
try
Ab
bre
viat
ion
Co
un
try
Full
Nam
e
Ph
on
e N
um
ber
(an
y)
Ho
me
Ph
on
e N
um
ber
Ce
ll P
ho
ne
Nu
mb
er
Wo
rk P
ho
ne
Nu
mb
er
Emai
l Ad
dre
ss
Nic
knam
e
Insu
ran
ce N
um
be
r (f
ree
text
)
Dri
vers
Lic
en
se N
um
be
r
Rac
e (
OM
B)
Rac
e (
fre
e te
xt)
Eth
nic
itiy
Lan
guag
e
Occ
up
atio
n
Inco
me
Mar
ital
Sta
tus
Hei
ght
(cm
)
Hei
ght
(m)
Hei
ght
(in
)
Hei
ght
(ft)
We
igh
t (l
bs)
We
igh
t (k
g)
Blo
od
Typ
e
Site B
Site A
Site C
• Data Quality is a Key
– Garbage in and Garbage out
• Data entry errors are compound data matching complexity
– Various algorithmic solutions to address these, not perfect
• Types of errors:
– Missing or Incomplete Values
– Inaccurate data
– Fat finger errors
– Information is out of date
– Transposed names
– Misspelled names
Data Quality
• Transposition errors
• Mary Sue vs Sue Marie
• Smitty, John vs John, Smitty
• Names change over time
• Marriage, Divorce
• More than one way to spell name
• Jon, John
• Data entry
– Fat-finger = typo, transposition, etc.
• Phonetic variation
– Double names may not be given in full
Data Quality
Variant Taxonomy
• Element Variation
– Data Errors • OCR
• Typos
• Truncations
– Short forms • Abbreviations
• Initials
– Spelling variations • Alternate Spellings
• Transliterations
– Particles • Particle Segmentation
• Particle Omission
– Nicknames & Diminutives
– Translation variants
– Non-word characters
– Presence/Absence of TAQ
– Case variation
• Structural Variation – Additions/deletions
– Fielding variations
– Permutations
– Placeholders
– Element segmentation
Names
© 2014 The MITRE Corporation. All rights reserved.
Variant Taxonomy
• Element Variation
– Data errors • OCR
• Typos
• Truncations
• Removals
– Short forms • Abbreviations
• Initials
• Numerals
• Symbols
– Spelling variations • Alternate Spellings
• Transliterations
• Segmentation
– Translation variants
– Aliases
– Substitutions
– Element length
– Case variation
• Structural variation – Additions/deletions
– Fielding variations
– Permutations
– Placeholders
Addresses
Variant Taxonomy
• Element Variation
– Data Errors • OCR
• Typos
• Truncations
– Particles • Particle substitutions
• Particle omission
– Short forms • Abbreviations
• Month numbers
• Dropping leading zeros
• Dropping leading year digits
• Structural Variation
– Additions/deletions
– Fielding variations
– Placeholders
– Element segmentation
Dates (of birth) IDs (SSN/other)
• Element Variation
– Data Errors
• OCR
• Typos
• Dropping leading zeros
– Particles
• Particle substitutions
• Particle omission
– Short forms
• Structural Variation
– Missing data/deletions
– Fielding variations
– Placeholders
– Element segmentation
Variant Taxonomy
Paper / Poster presented at
AMIA 2013 Summit on Clinical Research Informatics
Test Evaluation Framework
• What is the question you are trying to answer?
• What data attributes do you have?
• What is the quality of the attributes?
• What is matching method do you want doing use?
• How effective is your matching method?
Framework Applied to Patient Matching
Metrics for Algorithm Performance
• Ideal outcome of any matching exercise is correctly answering this one question hundreds or thousands of times, Are these two things the same thing?
– Correctly identifying all the true positives and true negatives while minimizing the number of errors, false positives and false negatives
Patient Matching Goal
• True Positive- The two records represent the same patient
• True Negative- The two records don't represent the same patient
Patient Matching Terminology
• False Negative: The algorithm misses a record that should be matched
• False Positive: The algorithm creates a link to two records that don’t actually match
Patient Matching Terminology
EHR A EHR B Truth
(Gold Standard)
Algorithm Match
Type
Jonathan Jonathan Match Match True Positive
Jonathan Sally Non-Match Non-Match True Negative
Jonathan Sally Non-Match Match False Positive
Jonathan Jonathan Match Non-Match False
Negative
Evaluation
Good
Bad
EHR A EHR B Truth
(Gold Standard)
Algorithm Match
Type
Jonathan Jonathan Match Match True Positive
Jonathan Sally Non-Match Non-Match True Negative
Jonathan Sally Non-Match Match False Positive
Jonathan Jonathan Match Non-Match False
Negative
Evaluation
Bad
Truth
Algorithm
Positive Negative
Positive True Positive False Positive
Negative False Negative True Negative
Evaluation
Recall
Precision
Precision = True Positives / (True
Positives + False Positives)
Recall = True Positives /
(True Positives + False
Negatives)
• Calculation
– Precision = True Positives / (True Positives +
False Positives)
– Recall = True Positives / (True Positives +
False Negatives)
• Tradeoffs between Precision and Recall
– F Measure
Evaluation
Creating Test Data Sets
Development of Test Data Set
Patient Database
Select Potential Matches
(aka Adjudication Pool)
Compare Algorithm and
Test Data Set
Human-Reviewed Match Decisions
(Answer Key == Ground Truth Data Set)
Manual
Reviewer 1
Manual
Reviewer 2
Manual
Reviewer 3
Development of Ground Truth Sets • Identify data set that reflects real word use case
• Develop potential duplicates
• Human adjudication review and classification
– Match or Non-Match
• Estimate truth
– Pooled methods using multiple matching methods
Issues In Establishing Ground Truth
Examples
B Smith Bill Smythe William Smythe W Smith ??
DOB: 10/12/1972 October 11, 1972
December 10, 1972 12/10/72 October 12, 1927
Adjudication Exercise
Identity Matching Adjudication Collector (IMAC) User Interface
One screen of the Adjudication Collector continually provides
questions to the adjudicator which need to be answered. These
screens first ask the question with no dates provided and then again
asks the question with dates shown.
Issues In Establishing Ground Truth
• Different truth for different applications
– Credit check
– Security applications
– Customer support
– De-duplication of mailing lists
• What is the cost of missing a match?
– New record entered into database
– Irritated customer
– Lives are lost
• Criteria for truth must be carefully established and well-
understood by annotators
– Question posed to annotators must be carefully phrased
Issues In Establishing Ground Truth
• How much time / expertise is available to judge (/discount) false positives?
• Needs to reflect real word test use case
• Evaluation results are only as good as the truth on which they are based
– And only as appropriate as the evaluation is to the task that will be performed with the operational system
• Absolute recall impossible to measure without completely known test set (i.e. “You don’t know what you’re missing.”)
– Estimate with pooled results
Issues In Establishing Ground Truth • First step in evaluation is to determine why the
evaluation is being conducted
• Different truth for different applications
– Security Applications vs Patient Health Record
• What is the cost of missing a match?
– Security: Lives are lost
– Health: Patient safety event, missed medications, allergies, etc… death But…this is situation today.
• What is the cost of wrongly identifying a match?
– Security : Passenger is inconvenienced / delayed
– Health: Patient safety event, wrong medication, treatment, liability, death
• Criteria for truth must be carefully established and well-understood
– E.g. Question posed to annotators must be carefully phrased
Summary for
Healthcare
Use Case
Next Steps
• Build Ground Truth dataset to enable evaluation of complementary approaches to patient matching
– “Exam” for patient matching systems
• Encourage adoption and understanding of metrics-based decisions with respect to implementation of patient matching systems.
• ONC Patient Matching Projects
– Patient Matching Aggregation and Linking (PMAL) Project
– PCOR Patient Matcher Test Harness
– Patient Matching Challenge
Future Work
• Electronic: Patient matching has been identified as a
key barrier to Interoperability in ONC’s nationwide
Health IT Roadmap
• Prevention & Patient Education: Reduction in patient
safety events caused by missing or incorrectly
matched records
• Patient Engagement/Population Management: More
complete records gathered across disparate health
systems
• Savings: Missing information and reordered tests cost
over $8 Billion annually. Improvement in patient
matching can reduce this cost.
• Improvements in patient matching can reduce deaths
healthcare costs and fraud caused by incorrectly
matched data
How Patient Matching Benefits Health IT
Keith J. Miller: keith@mitre.org
Adam W. Culbertson : aculbertson@himss.org
Contact Information
Questions?
Questions
BackUp
What is the first step in an effective patient matching strategy?
A. Understanding your data.
B. Understanding the question you are trying to answer for patient matching in your organization.
C. Implementing a patient matcher software solution.
D. Improving data entry processes.
Question 1
Correct Answer:
B. Understand the question you are trying to answer. The approach you take will be dependent upon the question, as this will determine how you address tradeoffs that will be needed, for instance in timeliness of a response vs. accuracy.
Incorrect Answers:
A: Understanding your data is the next step. The first step is understanding what exactly you want to do.
C, D: Implementing a patient matching solution should happen only after understanding your use cases and your data, and at the same time as improving data entry processes.
Answer 1
What is the data variant taxonomy?
A. A taxonomy used to describe the way errors can happen in the demographic data.
B. A taxonomy for describing how patient health data varies between patients.
C. A taxonomy for describing the cultural variation in patient populations.
D. A taxonomy for describing errors in the collection of patient health data.
Question 2
Correct Answer:
A. A. A taxonomy used to describe the way errors can happen in the
demographic data. This variant taxonomy provides a unified way to
describe errors that can happen in patient demographic data, for
instance, truncation of dates.
Incorrect Answers:
B. Incorrect because this is not related to health data.
C. Incorrect because this describes the types of errors commonly seen in
data, not the cultural make-up of the population that is being matched.
D. This is related to errors in demographic data, not health data.
Answer 2
True or False: You need to understand your data because the
approach you take varies depending upon the mix of cultures and
naming conventions, some matchers are better than others at dealing
with different types of errors in the data, demographics such as
predominance of age groups can change your matching approach.
Answer: TRUE - all of the above are true for reasons in understanding
the data before undertaking patient matching
Question 3
BACKUP
The Trade-off Between False Positive and False Negative Matches
• As the match score threshold is increased, the number of false positives decreases, but false negatives increase. (increasing precision)
• As the match score threshold is lowered, the number of false negatives decreases, but false positives increase (increasing recall)
Source: Grannis, S. Introduction to Record Linkage. September 27, 2012
Basic IR Metrics: Precision and Recall
“Subject”:
MAHMOUD ABDUL HAMEED
12/10/1945
False
positives
False
negatives
“Target
List”:
‘True’ Answers
System returns
Precision (P) = X/Y
Recall (R) = X/Z
X
Y
Z
MOREY APPLEBAUM
MOHAMMED ABDUL HAMID
MAHMOUD ABD EL HAMEED
MAKMUD ABDUL HAMID
MAHMOUD ABD ALHAMID
(2/4)
(2/3)
True
Positives
Precision and Recall Inversely Related
(1) Database
‘True’ Hits
System returns
Recall Increased, but
Precision Fell
The ‘Low Hanging Fruit’
phenomenon – more false
hits will come in for every
true one
Precision and Recall Inversely Related (2)
Database
‘True’ Hits
System returns
Precision Increased, but
Recall Fell
More selective matching
What Makes a Good Evaluation?
• Objective – gives unbiased results
• Replicable – gives same results for same inputs
• Diagnostic – can give information about system improvement
• Cost-efficient – does not require extensive resources to repeat
• Understandable – results are meaningful in some way to appropriate people
• Well-documented – also contextualizes results in terms of purpose of the evaluation and task
• Lack of Transparency in How Patient Matching Algorithms Perform
• Varied Claims in Algorithm Performance
• Need greater transparency in system performance
• Better education around patient matching understanding the science.
• Little work done to quantify match rates on data sets on real work clinical data sets
• Need Reporting on Match rates in terms of precision and recall
Problem Statement
IMAC – Admin Interface
An administrative screen allows the ability to manage IMAC users as
well as manage the questions asked of users. This includes the
ability to set the priority of questions and the number of judges to be
used for each question.
IMAC – Admin Interface (2) Viewing and resolution of conflicting adjudications can also be
performed from the administrative screen.
Evaluation: Like IR Tasks • Metrics
– F-measure - harmonic mean of precision and recall • F = (β2 + 1) P R / ( (β2 P) + R) where
P = precision = correct system responses / all system responses
R = recall = correct system responses / all correct reference responses
β = beta factor– provides a mean to control the importance of recall over precision
– Additional Measures
• False positives – items that are identified as correct responses that are not correct responses
(= 1 – Precision)
• False negatives – correct responses not identified
(= 1 – Recall)
• Fallout = non-relevant responses / all non-relevant reference responses (related to, but not directly calculable from precision / recall)
Issue:
• Annotation Standard for Development of Ground Truth
• Large Affects on performance due to algorithm tuning
• Tuning is need specific
• Setting Cut-offs
– Upper Thresholds
– Feature Selection
– Feature Weighing
• Blocking
Algorithm Tuning
Algorithm Performance
Algorithm
Algorithm Tuning
Data Quality
Framework for Evaluation: EAGLES 7-Step Recipe/ISLE FEMTI*
1. Define purpose of evaluation – why doing the evaluation
2. Elaborate a task model – what tasks are to be performed with the data
3. Define top-level quality characteristics
4. Produce detailed system requirements
5. Define metrics to measure requirements
6. Define technique to measure metrics
7. Carry out and interpret evaluation Originally developed as an evaluation framework for Machine Translation, but authors note that it
should be able to be used as a generic evaluation framework. *Acronyms: EAGLES – European Advisory Group on Language Engineering Standards
ISLE – International Standards for Language Engineering
FEMTI – Framework for the Evaluation of Machine Translation in ISLE
(http://www.issco.unige.ch/femti)
top related