informing the design and deployment of health information
TRANSCRIPT
University of South FloridaScholar Commons
Graduate Theses and Dissertations Graduate School
10-26-2015
Informing the Design and Deployment of HealthInformation Technology to Improve CareCoordinationDiego A. MartinezUniversity of South Florida, [email protected]
Follow this and additional works at: http://scholarcommons.usf.edu/etd
Part of the Medicine and Health Sciences Commons, and the Other Earth Sciences Commons
This Dissertation is brought to you for free and open access by the Graduate School at Scholar Commons. It has been accepted for inclusion inGraduate Theses and Dissertations by an authorized administrator of Scholar Commons. For more information, please [email protected].
Scholar Commons CitationMartinez, Diego A., "Informing the Design and Deployment of Health Information Technology to Improve Care Coordination"(2015). Graduate Theses and Dissertations.http://scholarcommons.usf.edu/etd/5987
Informing the Design and Deployment of Health Information Technology to Improve Care
Coordination
by
Diego A. Martinez
A dissertation submitted in partial fulfillmentof the requirements for the degree of
Doctor of PhilosophyDepartment of Industrial and Management Systems Engineering
College of EngineeringUniversity of South Florida
Major Professor: Jose L. Zayas-Castro, Ph.D.Peter Fabri, M.D.Ali Yalcin, Ph.D.
Shuai Huang, Ph.D.Alex Savachkin, Ph.D.
Adriana Iamnitchi, Ph.D.
Date of Approval:October 20, 2015
Keywords: Hospital Readmission, Health Information Exchange, Healthcare SystemsEngineering
Copyright c© 2015, Diego A. Martinez
Table of Contents
Abstract ii
Chapter 1 Introduction 11.1 Research Contributions 2
Chapter 2 A Literature Review of Preventable Hospital Readmissions 5
Chapter 3 Preventable Readmission Risk Factors for Patients with Chronic Conditions 6
Chapter 4 A User Needs Assessment to Inform Health Information Exchange Designand Implementation 7
Chapter 5 Uncovering Hospitalists’ Information Needs From Outside Healthcare Fa-cilities in the Context of Health Information Exchange Using AssociationRule Learning 9
Chapter 6 A Strategic Gaming Model for Health Information Exchange Markets 11
Chapter 7 Conclusion 12
Appendices 17Appendix A Copyright Permissions for Manuscripts Presented in Appendices B,
C, D, E and F 18Appendix B A Literature Review of Preventable Hospital Readmissions 25Appendix C Preventable Readmission Risk Factors for Patients with Chronic
Conditions 82Appendix D A User Needs Assessment to Inform Health Information Exchange
Design and Implementation 99Appendix E Uncovering Hospitalists’ Information Needs From Outside Health-
care Facilities in the Context of Health Information Exchange Us-ing Association Rule Learning 111
Appendix F A Strategic Gaming Model for Health Information Exchange Mar-kets 140
i
Abstract
In the United States, the health care sector is 20 years behind in the use of information tech-
nology to improve the process of health care delivery as compared to other sectors. Patients have
to deliver their data over and over again to every health professional they see. Most health care
facilities act as data repositories with limited capabilities of data analysis or data exchange. A
remaining challenge is, how do we encourage the use of IT in the health care sector that will
improve care coordination, save lives, make patients more involved in decision-making, and save
money for the American people? According to Healthy People 2020, several challenges such as
making health IT more usable, helping users to adapt to the new uses of health IT, and monitoring
the impact of health IT on health care quality, safety, and efficiency, will require multidisciplinary
models, new data systems, and abundant research. In this dissertation, I developed and used sys-
tems engineering methods to understand the role of new health IT in improving the coordination,
safety, and efficiency of health care delivery.
It is well known that care coordination issues may result in preventable hospital readmissions.
In this dissertation, I identified the status of the care coordination and hospital readmission issues
in the United States, and the potential areas where systems engineering would make significant
contributions (see Appendix B). This literature review introduced me to a second study (see
Appendix C), in which I identified specific patient cohorts, within chronically ill patients, that
are at a higher risk of being readmitted within 30 days. Important to note is that the largest
volume of preventable hospital readmissions occurs among chronically ill patients. This study
was a retrospective data analysis of a representative patient cohort from Tampa, Florida, based on
multivariate logistic regression and Cox proportional hazards models. After finishing these two
ii
studies, I directed my research efforts to understand and generate evidence on the role of new
health IT (i.e., health information exchange, HIE) in improving care coordination, and thereby
reducing the chances of a patient to be unnecessarily readmitted to the hospital.
HIE is the electronic exchange of patient data among different stakeholders in the health care
industry. The exchange of patient data is achieved, for example, by connecting electronic medical
records systems between unaffiliated health care providers. It is expected that HIE will allow
physicians, nurses, pharmacists, other health care providers and patients to appropriately access
and securely share a patient’s vital medical information electronically, and thereby improving the
speed, quality, safety and cost of patient care. The federal government, through the 2009 Health
Information Technology for Economic and Clinical Health (HITECH) Act, is actively stimulating
health care providers to engage in HIE, so that they can freely exchange patient information.
Although these networks of information exchange are the promise of a less fragmented and more
efficient health care system, there are only a few functional and financially sustainable HIEs across
the United States. Current evidence suggests four barriers for HIE:
• Usability and interface issues of HIE systems
• Privacy and security concerns of patient data
• Lack of sustainable business models for HIE organizations
• Loss of strategic advantage of "owning" patient information by joining HIE to freely share
data
To contribute in reducing usability and interface issues of HIE systems, I performed a user
needs assessment for the internal medicine department of Tampa General Hospital in Tampa,
Florida. I used qualitative research tools (see Appendix D) and machine learning techniques (see
Appendix E) to answer the following fundamental questions: How do clinicians integrate patient
information allocated in outside health care facilities? What are the types of information needed the
iii
most for efficient and effective medical decision-making? Additionally, I built a strategic gaming
model (see Appendix F) to analyze the strategic role of "owning" patient information that health
care providers lose by joining an HIE. Using bilevel mathematical programs, I mimic the hospital
decision of joining HIE and the patient decision of switching from one hospital to another one.
The fundamental questions I tried to answer were: What is the role of competition in the decision
of whether or not hospitals will engage in HIE? Our mathematical framework can also be used by
policy makers to answer the following question: What are the optimal levels of monetary incentives
that will spur HIE engagement in a specific region? Answering these fundamental questions will
support both the development of user-friendly HIE systems and the creation of more effective
health IT policy to promote and generate HIE engagement.
Through the development of these five studies, I demonstrated how systems engineering tools
can be used by policy makers and health care providers to make health IT more useful, and to
monitor and support the impact of health IT on health care quality, safety, and efficiency.
iv
Chapter 1: Introduction
The elderly constitute 13.7% of the population in the United States, and they consume 42%
of the hospital expenditures. In addition, during FY 2006, it was found that 72% of Medicare
hospitalizations were treated in teaching hospitals, many of whom are critically ill patients in need
of advanced care. Unfortunately, care coordination among health care providers during patient
treatment is not optimal. Gaps in communication during health care delivery can cause unnecessary
hospital readmissions and serious breakdowns in care. These gaps in communication have been
recognized as the leading root cause of sentinel events by The Joint Commission between 1995 and
2006. To put this into context, patient hand off during hospital transfers represent a critical situation
where inaccessible clinical information delays understanding of patient’s health condition, and
consequently hinders his/her timely treatment. Having timely access to a patient’s medical history
should improve the delivery of care during a patient hand off. Health information exchange
(HIE) has emerged as a mechanism to foster care coordination and reduce communication gaps.
Although the 2009 HITECH Act has directed substantial funding to promote HIE, recent studies
have reported low engagement across hospitals and other health care providers in the United States.
This engagement is particularly low for large academic tertiary care institutions in competitive
markets. Several authors claim that better designed HIE systems would stimulate HIE engagement.
The objective of this dissertation is to inform the design and deployment of health IT aiming
at improving care coordination and reducing hospital readmissions. The rationale underlying this
investigation is that, once the health professionals information needs during treatment of hospital-
1
ized patients are understood, better HIE systems will be designed, representing an opportunity to
improve the adoption and utilization of HIE across the United States.
1.1 Research Contributions
The research contributions of the studies presented in Appendices B, C, D, E, and F are
described next.
1. In the first study (see Appendix B), I synthesized published evidence on the status of the
hospital readmission problem in the United States, as well as identifying research gaps where
systems engineering can make a significant impact.
2. In a following study (see Appendix C), I identified risk factors associated with 30-day
preventable hospital readmission for congestive heart failure, acute myocardial infarction,
pneumonia, and diabetes patients. Important to note is that the largest proportion of hospital
readmissions is among chronically ill patients.
3. Since improving care coordination is key to reducing hospital readmissions, I directed my
efforts towards analyzing the role of new health IT (i.e., health information exchange, HIE)
in improving care coordination. The study introduced in Appendix D revealed physicians’
preferences, habits, and barriers to collect and use patient information allocated in electronic
medical records of other health care facilities. This study is the first user needs assessment
previous HIE implementation in a teaching hospital.
4. In the study introduced in Appendix E, I measured physicians’ actual information-gathering
habits in electronic medical records of other health care facilities. This study innovates by
explicitly incorporating the health care providers’ needs and voice in what data/information
an HIE must deliver.
2
5. Although HIE has the potential of supporting care coordination efforts, there are still few
functional HIE networks in the United States. One of the barriers for hospitals to engage
in HIE is the potential loss of competitive advantage by freely sharing patient data with
other competing hospitals. In the work presented in Appendix F, I generated a deeper
understanding of the role of competition in the decision of whether or not a hospital will
join an HIE network.
6. Finally, I designed and built a mathematical framework to find the optimal levels of federal
monetary incentives that will spur HIE adoption in a given region (see Appendix F). Many
modeling studies about HIE adoption have already been undertaken. A crucial difference
among these studies is the type of interaction that is assumed among competing hospitals.
In more competitive models, the type of interaction can often be summarized in terms of
the hospital’s conjectural variation, in which each hospital has about the way its competitors
may react if it varies its decision to join HIE. The models presented in Appendix F make
the following contribution. Unlike previous approaches, they calculate an oligopolistic
equilibrium of HIE adoption in a given region using the hospital utility function conjectural
variations, while considering the discrete range patient’s decision of where to receive (or
purchase) health care. I argue this is a more realistic representation of the HIE market. The
resulting optimization problem for each hospital is a bi-level mathematical program.
In summary, the work presented in this dissertation provide guidelines, anchored in systems
engineering methods, to developers to better design HIE systems, to health IT policy makers to
find optimal levels of monetary incentives that will spur HIE engagement, and to researchers as
to where significant contributions can be made to contribute in the care coordination and hospital
readmission problems. These contributions will be significant because design guidelines based on
providers’ needs should result in HIE systems with a higher degree of personalization, facilitating
use and adoption, and therefore improved care coordination and health care delivery. It is expected
3
to have an impact in the creation of better HIE systems, as well as the development of further
longitudinal studies that will provide stronger evidence-based guidelines.
4
Chapter 2: A Literature Review of Preventable Hospital Readmissions
Preventable readmissions are a large and growing concern throughout healthcare in the United
States, representing as many as 20% of all hospitalizations (30-day post-discharge) and an esti-
mated $17 to $26 billion in unnecessary costs annually. National quality initiatives and Medicare
reimbursement financial incentives have stimulated significant efforts by healthcare organizations
to reduce readmissions via a number of approaches and interventions. Given the severity and
complexity of this problem, this paper summarizes the recent literature describing descriptive and
predictive readmission studies as well as proposed interventions. A total of 112 publications were
identified and grouped into three general categories: descriptive analyses, intervention studies, and
predictive analyses. While a significant amount of work has been conducted in each of these areas,
very few industrial engineering or operation research studies focused directly on readmissions have
been reported in the literature. This paper, therefore, concludes with a discussion of potential areas
in which industrial engineers might make meaningful contributions to this important problem. The
complete manuscript A Literature Review of Preventable Hospital Readmissions, under review in
IIE Transactions on Healthcare Systems Engineering, can be found in the Appendix B.
5
Chapter 3: Preventable Readmission Risk Factors for Patients with Chronic Conditions
Evidence indicates that the largest volume of hospital readmissions occurs among patients
with preexisting chronic conditions. Identifying these patients can improve the way hospital
care is delivered and prioritize the allocation of interventions. In this retrospective study, we
identify factors associated with readmission within 30 days based on claims and administrative
data of nine hospitals from 2005 to 2012. We present a data inclusion and exclusion criteria to
identify potentially preventable readmissions. Multivariate logistic regression models and a Cox
proportional hazards extension are used to estimate the readmission risk for 4 chronic conditions
(congestive heart failure [CHF], chronic obstructive pulmonary disease [COPD], acute myocardial
infarction, and type 2 diabetes) and pneumonia, known to be related to high readmission rates.
Accumulated number of admissions and discharge disposition were identified to be significant
factors across most disease groups. Larger odds of readmission were associated with higher
severity index for CHF and COPD patients. Different chronic conditions are associated with
different patient and case severity factors, suggesting that further studies in readmission should
consider studying conditions separately. The article Preventable Readmission Risk Factors for
Patients with Chronic Conditions, published in the Journal for Healthcare Quality, can be found in
the Appendix C.
6
Chapter 4: A User Needs Assessment to Inform Health Information Exchange Design and
Implementation
Important barriers for widespread use of health information exchange (HIE) are usability and
interface issues. However, most HIEs are implemented without performing a needs assessment
with the end users, healthcare providers. We performed a user needs assessment for the process
of obtaining clinical information from other health care organizations about a hospitalized patient
and identified the types of information most valued for medical decision-making. Quantitative and
qualitative analysis were used to evaluate the process to obtain and use outside clinical information
(OI) using semi-structured interviews (16 internists), direct observation (750 h), and operational
data from the electronic medical records (30,461 hospitalizations) of an internal medicine depart-
ment in a public, teaching hospital in Tampa, Florida. 13.7% of hospitalizations generate at least
one request for OI. On average, the process comprised 13 steps, 6 decisions points, and 4 different
participants. Physicians estimate that the average time to receive OI is 18 h. Physicians perceived
that OI received is not useful 33âAS66% of the time because information received is irrelevant
or not timely. Technical barriers to OI use included poor accessibility and ineffective information
visualization. Common problems with the process were receiving extraneous notes and the need
to re-request the information. Drivers for OI use were to trend lab or imaging abnormalities,
understand medical history of critically ill or hospital-to-hospital transferred patients, and assess
previous echocardiograms and bacterial cultures. About 85% of the physicians believe HIE would
have a positive effect on improving healthcare delivery. Although hospitalists are challenged by
a complex process to obtain OI, they recognize the value of specific information for enhancing
7
medical decision-making. HIE systems are likely to have increased utilization and effectiveness if
specific patient-level clinical information is delivered at the right time to the right users. The article
A User Needs Assessment to Inform Health Information Exchange Design and Implementation,
published in BMC Medical Informatics and Decision Making, can be found in the Appendix D.
8
Chapter 5: Uncovering Hospitalists’ Information Needs From Outside Healthcare Facilities in the
Context of Health Information Exchange Using Association Rule Learning
Important barriers to health information exchange (HIE) adoption are clinical workflow disrup-
tions and troubles with the system interface. Prior research suggests that HIE interfaces providing
faster access to useful information may stimulate use and reduce barriers for adoption; however,
little is known about informational needs of hospitalists. Our objective was to study the association
between patient health problems and the type of information requested from outside healthcare
providers by hospitalists of a tertiary care hospital. We searched operational data associated with
fax-based exchange of patient information (previous HIE implementation) between hospitalists of
an internal medicine department in a large urban tertiary care hospital in Florida, and any other
affiliated and unaffiliated healthcare provider. All hospitalizations from October 2011 to March
2014 were included in the search. Strong association rules between health problems and types
of information requested during each hospitalization were discovered using Apriori algorithm,
which were then validated by a team of hospitalists of the same department. Our results indicate
that only 13.7% (2,089 out of 15,230) of the hospitalizations generated at least one request of
patient information to other providers. The transactional data showed 20 strong association rules
between specific health problems and types of information exist. Among the 20 rules, for example,
abdominal pain, chest pain, and anaemia patients are highly likely to have medical records and
outside imaging results requested. Other health conditions, prone to have records requested, were
lower urinary tract infection and back pain patients. The presented list of strong co-occurrence
of health problems and types of information requested by hospitalists from outside healthcare
9
providers not only informs the implementation and design of HIE, but also helps to target future
research on the impact of having access to outside information for specific patient cohorts. Our
data-driven approach helps to reduce the typical biases of qualitative research. The complete
manuscript Uncovering Hospitalists’ Information Needs From Outside Healthcare Facilities in
the Context of Health Information Exchange Using Association Rule Learning, under review in
Applied Clinical Informatics, can be found in the Appendix E.
10
Chapter 6: A Strategic Gaming Model for Health Information Exchange Markets
Here we describe a strategic gaming model for estimating willingness of healthcare organiza-
tions to adopt HIE, and to demonstrate its use in HIE policy design. We formulated the model
as a bi-level integer mathematical program. Multi-hospital mixed strategy Nash equilibrium is
searched using a quasi-Newton method, and are interpreted as the hospitals’ willingness to adopt
HIE based on its competitors decisions. We applied our model to 1,093,177 encounters over a 7.5-
year period in 9 hospitals located within three adjacent counties in Florida. For this community
and under a particular set of assumptions, proposed federal penalties of up to $2,000,000 have
a higher impact on increasing HIE adoption than current federal monetary incentives. Medium-
sized hospitals are more reticent to HIE than large-sized hospitals. In the presence of a 4-hospital
collusion to not adopt HIE, neither federal incentives nor proposed penalties increase hospitals’
willingness to adopt HIE. Hospitals may set HIE adoption decisions to threaten the value of
interconnectivity even with federal incentives in place. Competition among hospitals, coupled
with volume-based payment systems, creates no incentives for smaller hospitals to exchange data
with competitors. Medium-sized hospitals need targeted actions to mitigate market incentives to
not adopt HIE. Strategic gaming modeling clarified HIE adoption decisions and market conditions
at play in an extremely complex technology implementation, which may inform other communities
trying to achieve EMR interconnectivity and the development of new and stronger HIE policy. The
complete manuscript A Strategic Gaming Model for Health Information Exchange Markets, under
review in the Journal of the American Medical Informatics Association, which is under review in
the Journal of the American Medical Informatics Association can be found in the Appendix F.
11
Chapter 7: Conclusion
This dissertation has answered, to some extent, the five questions we began with:
• Question 1: What is the current status of the hospital readmission problem in the United
States?
Answer: Hospital readmissions are a large and growing concern representing as many as
20% of all hospitalizations, with an estimated annual cost of $17 billion. During the last 10
years, most of the published evidence has concentrated on data analysis to identify those at
a higher risk of readmission and assessment of interventions aiming at reducing such risk.
Only a few large-scale unified studies have been conducted. Moreover, the scope of most
studies is either disease specific (limited to one disease), fairly localized (limited to a single
hospital) or too broad (limited to nationwide hospitalizations with no clinical information).
• Question 2: What are the conditions that make a patient more likely to be readmitted?
Answer: For chronically ill patients, the more days the patient stays in the hospital, the
higher the likelihood of being readmitted within 30 days. Particularly for a patient with
heart failure, having behavioral health issues is associated to a higher likelihood of being
readmitted. In terms of payer class, it was found that patients with Medicaid and Medicare
have a higher risk of being readmitted as compared to commercial insurance. Finally, those
admitted though the emergency department are at a higher risk of being readmitted.
• Question 3: How do clinicians integrate patient information allocated in outside health care
facilities to improve medical-decision making and care coordination?
12
Answer: In an urban tertiary care hospital, although hospitalists are challenged by a com-
plex process to integrate patient information allocated in outside health care facilities, they
recognize the value of specific data types. It was found that, on average, the process to
obtain patient records comprises 13 steps, 6 decision points, 4 different participants, and
lasts 18 hours. Most of the time, physicians find that the patient information received is
irrelevant or late. Common problems with the process are receiving extraneous notes and
the need to re-request information. Common situations where obtain patient records is key
are trending lab results abnormalities, understanding medical history of critically ill patients
or hospital-to-hospital transferred patients, and assessing previous electrocardiograms and
bacterial cultures. About 85% of the hospitalists believe HIE will have a positive effect on
improving health care delivery.
• Question 4: What are the types of information needed the most for efficient and effective
medical decision-making?
Answer: In the internal medicine department of a urban tertiary care hospital, outside med-
ical records are commonly request for abdominal pain and anemia patients. For abdominal
pain patients, for example, medical records are usually requested to find previous MRIs, CTs
and endoscopies.
• Question 5: What is the role of competition in the decision of whether or not hospitals will
engage in HIE?
Answer: Our simulation experiments indicate that the higher the competition among hospi-
tals in a given region, the higher incentives/penalties are needed for HIE engagement. It was
also found that penalties, instead of incentives, would have a stronger impact on generating
collaboration via HIE engagement.
This dissertation has advanced the current understanding of the hospital readmission problem.
Through a literature review, it has discussed definitions, measurements, and descriptive analyses
13
reported in the literature, as well as the many interventions utilized by health care providers to
reduce patient readmission risk. It has also identified and discussed the current research gaps
that could be addressed by systems engineers. For instance, several opportunities exist to conduct
predictive analytics to identify those patients at a high risk of readmission. Also, the development
of new health information technology to support care coordination efforts, such as HIE, may have
a key role in reducing hospital readmission. Through statistical modeling, this dissertation has
identified risk factors for preventable hospital readmission. The list of risk factors may be useful
to other investigators who are trying to predict whether or not a patient will be readmitted. Also,
recognizing those patients cohorts at high risk of readmission, may help health care providers to
target their interventions.
This dissertation has also advanced the current understanding of HIE, and its role in support-
ing care coordination and medical decision-making. Through qualitative methods, it has more
deeply described the clinicians’ expectations and values regarding HIE, as reflected in individual
internists’ usage of a fax-based HIE system. The simple framework of drivers and barriers may be
useful to other investigators who are trying to understand users needs in the context of HIE design
and implementation.
Trough quantitative methods, it has documented internists information requests patterns in
the context of HIE. Outcomes of this investigation will help HIE developers and implementers
recognize commonly requested clinical information by the patient chief complaint, and thereby
prioritize information display. This knowledge could be used to inform the design of new technical
functionalities beyond simple data exchange. For instance, electronic decision support systems
that identify, retrieve and present, at the point of care, patient clinical data allocated in information
systems from other health care providers.
Through mathematical models, it has generated a deeper understanding of the role of compe-
tition in the HIE participation decision, which may help modify current policies and incentives
structures, which seek to foster HIE participation and thereby collaboration among competitors.
14
With the increasing evidence supporting the effect of HIE use on reduced utilization and costs in
emergency departments, there is the need of stronger policies and incentives to convince competing
organizations to share patient data electronically.
Further research is needed to predict hospital readmission. Data accumulating from wide-
spread use of electronic medical records (EMR) and HIE networks provide an underexploited
opportunity to perform individualized patient care using data-driven approaches. A hospital read-
mission may be influenced by numerous factors including physiologic indices of case severity,
treatment strategies, and socioeconomic factors. Accordingly, developing predictive models for
readmissions requires hypothesis-driven selection of predictors, robust sample sizes, and the use
of computational methods that may exploit these large datasets. Supervised machine learning
methods may be used to leverage heterogeneous (structured and unstructured) demographic, phys-
iologic, laboratory and imaging data to improve early identification of patients at high risk for HF
readmission.
Future research is also needed to determine the effect of clinician access to information from
HIE networks. Linking HIE to patient outcomes is important to demonstrate its value and to
promote HIE engagement. To develop clinical decision support systems that are fed by HIE data,
more research needs to be done to understand clinician-user and the system in which the users and
the technology interact. Improved knowledge of different kinds of care transitions (e.g., hospital
transfers) would be essential to understand the value of HIE. Such knowledge could also be used to
inform the design of new technical functionalities beyond simple data exchange. HIE will evolve
to support richer forms of collaboration among health care stakeholders including health care
providers, patients, health IT vendor companies, public health specialists, federal policy experts,
and the HIE organizations that supply data exchange services.
Health information technology, in the form of HIE, presents enormous opportunities for im-
proving care coordination and for other secondary uses, especially related to quality analysis and
population/personalized health care analytics, which may be essential to achieve sustainability in
15
HIE organizations and improvements in health care delivery. After many years of failed attempts
to have an interconnected health care system, HIE may be on a path toward success, now that the
federal government and other important stakeholders are engaged and have invested considerable
resources. However, it may still take many years and experiments before HIE realizes its potential.
It will be important to learn from the successes and failures, and to continue employing systems
engineering tools to understand and improve HIE.
16
Appendices
17
Appendix A: Copyright Permissions for Manuscripts Presented in Appendices B, C, D, E and F
Appendix A includes the copyright approvals for the material presented in this dissertation.
18
9/1/15, 2:24 PMUniversity of South Florida Mail - Re: Fw: Re: Requesting copyright permission
Page 1 of 3https://mail.google.com/mail/u/0/?ui=2&ik=7ad69269f1&view=pt&search=inbox&th=14f63a6f4b19b2d5&siml=14f63a6f4b19b2d5
Diego Martinez <[email protected]>
Re: Fw: Re: Requesting copyright permission1 message
Birgit Lang <[email protected]> Tue, Aug 25, 2015 at 3:02 AMTo: [email protected]
Dear Dr. Martinez, thanks for your request. You may use the manuscript as part of your dissertation. Kind regardsBirgit
Schattauer GmbH – Publisher for Medicine and Natural Sciences
i.A. Birgit Lang, Mrs.Editorial Office Phone: +49 711 22987-34
Fax: +49 711 22987-65E-mail: [email protected]: www.schattauer.com
Hoelderlinstrasse 370174 StuttgartGermany Here you will find our social media profiles.
Schattauer GmbHDistrict Court Stuttgart Register Court HRB 3357Chief Executive Officers: Dieter Bergemann / Dr. Wulf Bertram / Jan Haaf
VAT No. DE147831168
Original Message processed by david®
Re: Requesting copyright permission (24-Aug-2015 23:51)
From: Diego Martinez
To: Jess HolzerCc: Peter Henning
Thank you, Jess.
Appendix A (continued)
19
9/1/15, 2:24 PMUniversity of South Florida Mail - Re: Fw: Re: Requesting copyright permission
Page 2 of 3https://mail.google.com/mail/u/0/?ui=2&ik=7ad69269f1&view=pt&search=inbox&th=14f63a6f4b19b2d5&siml=14f63a6f4b19b2d5
Hi Peter -- please, let me know if further information is required.
Regards,Diego
On Mon, Aug 24, 2015 at 5:50 PM, Jess Holzer <[email protected]> wrote:Diego,
You will need to contact Schattauer for that permission. I have CC'ed Peter Henning, who should be able to help.
Best,Jess
Managing Editor, ACI Journal
On Mon, Aug 24, 2015 at 5:23 PM, Diego Martinez <[email protected]> wrote:Dear Editor,
Hope this message finds you well.
I am writing to request copyright authorization to use the following manuscript as part of my dissertationmaterial.
Title: Uncovering hospitalists’ information needs from outside healthcare facilities in the context of healthinformation exchange using association rule learningShort Title: Hospitalist information needs and HIEAuthors: Diego A. Martinez, Elia Mora, Martino Gemmani, José Zayas-CastroTopic: eHealth SystemsSubmission type: Research ArticleManuscript ID: ACI-2015-06-RA-0068.R1
Thank you in advance.
Best regards,Diego
--
Appendix A (continued)
20
9/1/15, 2:24 PMUniversity of South Florida Mail - Re: Fw: Re: Requesting copyright permission
Page 3 of 3https://mail.google.com/mail/u/0/?ui=2&ik=7ad69269f1&view=pt&search=inbox&th=14f63a6f4b19b2d5&siml=14f63a6f4b19b2d5
Diego A. Martinez, M.I.E.
Ph.D. CandidateDepartment of Industrial and Management Systems EngineeringEGN 129
University of South Florida4202 East Fowler Avenue, Tampa, FL 33620(813) [email protected]
-- Diego A. Martinez, M.I.E.
Ph.D. CandidateDepartment of Industrial and Management Systems EngineeringEGN 129
University of South Florida4202 East Fowler Avenue, Tampa, FL 33620(813) [email protected]
Appendix A (continued)
21
9/1/15, 2:41 PMUniversity of South Florida Mail - 00583381 re:Requesting copyright permission
Page 1 of 2https://mail.google.com/mail/u/0/?ui=2&ik=7ad69269f1&view=pt&search=inbox&th=14f6301ce1cc2a00&siml=14f6301ce1cc2a00
Diego Martinez <[email protected]>
00583381 re:Requesting copyright permission1 message
"Jorge Menil" <[email protected]> <[email protected]> Mon, Aug 24, 2015 at 11:58 PMTo: "[email protected]" <[email protected]>
Dear Dr Martinez
Thank you for contacting BioMed Central.
The article you refer to is an open access publication. Therefore you are free to use the article for the purposerequired, as long as its integrity is maintained and its original authors, citation details and publisher are identified.
For detailed information about the terms please refer to the open access license:
http://www.biomedcentral.com/about/license.
If you have any questions please do not hesitate to contact me.
Best wishes
Jorge MenilCustomer Servicesinfo@biomedcentral.comwww.biomedcentral.com--------------Your Question/Comment -----------------
Dear Editor,
Hope this message finds you well.
I am writing to request copyright authorization to use the following manuscript as part of my dissertation material.
Title: A User Needs Assessment to Inform Health Information Exchange Design and Implementation Authors: Alexandra T Strauss, Diego A Martinez, Andres Garcia-Arce, Stephanie Taylor, Candice Mateja, Peter JFabri and Jose L Zayas-Castro
Journal: BMC Medical Informatics and Decision Making
Manuscript ID: 4130412391654976
Thank you in advance.
Best regards,Diego
Appendix A (continued)
22
Institute of Industrial Engineers
3577 Parkway Lane, Suite 200 · Norcross, GA 30092 · (770) 349-1110 August 25, 2015 Diego A. Martinez, M.I.E. Ph.D. Candidate Department of Industrial and Management Systems Engineering EGN 129 University of South Florida 4202 East Fowler Avenue, Tampa, FL 33620 (813) 974-5553 [email protected] www.dmartinezcea.com RE: COPYRIGHT PERMISSION Dear Diego Martinez: The Institute of Industrial Engineers hereby grants permission to use material from its publication in your dissertation, and warrants that it is the sole owner of the rights granted. We ask that you note the following reprint lines respectively:
Copyright©2015. Reprinted with permission of the Institute of Industrial Engineers from IIE Transactions on Healthcare Systems Engineering All rights reserved.
For: “A Literature Review of Preventable Hospital Readmissions”
Authors: Wan, Hong; Zhang, Lingsong; Witz, Steve; Musselman, Kenneth; Yi, Fang; Mullen, Cody; Benneyan, James; Zayas-Castro, José; Martinez, Diego; Rico, Florentino; Cure, Laila Please fax this signed agreement to my attention at (770) 263-8532. Regards, Donna Calvert
Appendix A (continued)
23
9/1/15, 2:49 PMRightslink® by Copyright Clearance Center
Page 1 of 1https://s100.copyright.com/AppDispatchServlet#formTop
Title: Preventable Readmission RiskFactors for Patients With ChronicConditions.
Author: Florentino Rico, Yazhuo Liu,Diego Martinez, et al
Publication: Journal for Healthcare QualityPublisher: Wolters Kluwer Health, Inc.Date: Jan 1, 9000Copyright © 9000, (C) 2015 National Association forHealthcare Quality
If you're a copyright.comuser, you can login toRightsLink using yourcopyright.com credentials.Already a RightsLink user orwant to learn more?
This reuse is free of charge. No permission letter is needed from Wolters Kluwer Health, LippincottWilliams & Wilkins. We require that all authors always include a full acknowledgement. Example:AIDS: 13 November 2013 - Volume 27 - Issue 17 - p 2679-2689. Wolters Kluwer Health LippincottWilliams & Wilkins© No modifications will be permitted.
Copyright © 2015 Copyright Clearance Center, Inc. All Rights Reserved. Privacy statement. Terms and Conditions. Comments? We would like to hear from you. E-mail us at [email protected]
Appendix A (continued)
24
Appendix B: A Literature Review of Preventable Hospital Readmissions
Appendix B shows the manuscript titled, "A Literature Review of Preventable Hospital Read-
missions", which is under review in IIE Transactions on Healthcare Systems Engineering.
25
A Literature Review of Preventable Hospital Readmissions
Hong Wan, Lingsong Zhang, Steven Witz, Kenneth J. Musselman, Fang Yi, Cody J.
Mullen, James Benneyan, Jose L. Zayas-Castro, Diego A. Martinez, Florentino Rico,
Laila N. Cure
Preprint Submitted to IIE Transactions on Healthcare Systems Engineering
Copyright©2015. Reprinted with permission of the Institute of Industrial Engineers from
IIE Transactions on Healthcare Systems Engineering. All rights reserved
Abstract
Preventable readmissions are a large and growing concern throughout healthcare in the
United States, representing as many as 20% of all hospitalizations (30-day post-
discharge) and an estimated $17 to $26 billion in unnecessary costs annually. National
quality initiatives and Medicare reimbursement financial incentives have stimulated
significant efforts by healthcare organizations to reduce readmissions via a number of
approaches and interventions. Given the severity and complexity of this problem, this
paper summarizes the recent literature describing descriptive and predictive readmission
studies as well as proposed interventions. A total of 112 publications were identified and
grouped into three general categories: descriptive analyses, intervention studies, and
predictive analyses. While a significant amount of work has been conducted in each of
these areas, very few industrial engineering or operation research studies focused
directly on readmissions have been reported in the literature. This paper, therefore,
concludes with a discussion of potential areas in which industrial engineers might make
meaningful contributions to this important problem.
Appendix B (continued)
26
Keywords: Readmissions, re-hospitalizations, bounce backs, discharge process
Appendix B (continued)
27
1. Background
Hospital readmissions and their associated costs have become an increasing concern over
the last several years (Boutwell, 2011), with provisions of the 2010 Patient Protection and
Affordable Care Act establishing penalties for hospitals with higher than average avoidable
readmission rates (Santamour, 2011). These penalties are an attempt to curb the rising number
of readmissions and their associated costs, which are significant. The Agency for Healthcare
Research and Quality reported that in 2011 there were approximately 3.3 million adults, all-
cause, 30-day readmissions in the United States at an estimated cost of $41.3 billion (Hines,
Barrett, Jiang, & Steiner, 2014). The cost of readmissions for Medicare patients alone stands at
an estimated $26 billion annually, out of which $17 billion are potentially preventable (Goodman,
Fisher, Chang, Raymond, & Bronner, 2013; Robert Wood Johnson Foundation, 2013).
While the problem is compelling, its underlying causes are difficult to analyze. Readmission
studies are often hampered by a lack of information on follow-up data among different care sites
and the cohort of hospitals used in the studies (public vs. private hospitals, Medicare vs. Non-
Medicare patients). For example, Chen et al. (2010) estimated a hospital cost model per
medical condition, and used the observed mean cost of care per case for Medicare patients and
a predicted mean cost of care to compare hospitals in a certain location and with specific
characteristics. This study is limited by the current inability of tracking patients going to different
hospitals.
Examples of common initial (“index”) diagnoses for hospitalizations and subsequent
readmissions include congestive heart failure (CHF), renal failure, urinary tract infection (UTI),
pneumonia, and chronic obstructive pulmonary disease (COPD) (Ouslander, Diaz, Hain, &
Tappen, 2011; Press et al., 2010), with common causes including incomplete care during a
hospital stay (Benbassat & Taragin, 2000; Ornstein, Smith, Foer, Lopez-Cantor, & Soriano,
2011), exacerbation of the initial condition or complication of the initial treatment (Marcantonio et
al., 1999), substandard care during the transition out of the hospital (Boutwell, 2011), adverse
Appendix B (continued)
28
drug events post discharge (Allaudeen, Vidyarthi, Maselli, & Auerbach, 2010), and poor
compliance to medication, exercise, and diet instructions after patients are discharged
(Krumholz et al., 2002).
Estimates of the percent of discharged adult patients readmitted within a month of their
original hospitalization range from 5% to 29% (Thomas & Holloway, 1991), with 90% of these
readmissions estimated as unplanned (Jencks, Williams, & Coleman, 2009). For Medicare fee-
for-service beneficiaries discharged between July 2005 and June 2008, the median 30-day
readmission rates were 19.9% for acute myocardial infarction (AMI) and 24.4% for heart failure
(HF) (Krumholz, Merrill, & Schone, 2009), with the overall annual cost of unplanned re-
hospitalizations estimated at $17.4 billion in 2004 (Jencks et al., 2009). According to hospital
discharge data for residents of New York, Pennsylvania, Tennessee, and Wisconsin, from
January to July in 1999, hospital costs for preventable readmissions were roughly $730 million
(Friedman & Basu, 2004). Readmitted patients also tend to have significantly poorer outcomes
and longer lengths of stay. More broadly, readmissions often are proposed as a general marker
for the quality of care received during an index admission (Weissman et al., 1999). For example,
early unplanned readmissions of patients with heart failure, diabetes, and obstructive lung
disease have been linked to the quality of care during their previous hospital stay (Ashton,
Kuykendall, Johnson, Wray, & Wu, 1995).
Despite this evidence and ensuing efforts to reduce readmissions, Karen E. Joynt and Jha
(2012) found that risk-adjusted 30-day readmission rates for congestive heart failure,
pneumonia and acute myocardial infarction between 2002 and 2009 showed little improvement,
arguing that overall 30-day readmission rates for these conditions may not appropriately reflect
the quality of care because causes for most of those readmissions may not be under the
hospital’s control. The Dartmouth Atlas Project in collaboration with the Robert Wood Johnson
Foundation (2013) reported that overall improvement in 30-day readmission rates between 2008
and 2010 has been “slow and inconsistent” throughout academic hospitals in the United States.
Appendix B (continued)
29
The report points out that focusing on 30-day readmission rates may not improve the health of
patients because it may lead to neglecting other important aspects of care, such as the
prevention of longer term readmissions for patients with chronic diseases and the increase in
hospital mortality (Goodman et al., 2013; Robert Wood Johnson Foundation, 2013). Still, 30-day
readmission rates continue to be the metric used to evaluate the performance of hospitals.
The Centers for Medicare and Medicaid Services (CMS) began reporting 30-day risk-
standardized readmission rates as a measure of hospital quality in 2009. In 2012, they
introduced a reimbursement system that penalizes hospitals with a high rate of readmissions for
pneumonia, congestive heart failure, or acute myocardial infarction (AMI) patients. The penalty
is assessed across all Medicare reimbursements for services rendered in a given hospital.
Given the magnitude of the readmission problem, financial pressures, and considerable
national focus within healthcare, this manuscript summarizes recent literature describing the
general problem, analytical studies, and intervention approaches. The intent is to provide
sufficient background to enable systems engineers and related researchers to contribute
meaningfully applied and theoretical work to this important area. Where useful, representative
studies are cited to provide context and additional insight, although the intent is not to
exhaustively review all papers.
A total of 112 papers from 1987 through 2011 were generated by a keyword search within
PubMED and reviewed for their key contributions. As summarized in Figure 1, the number of
papers in each category increased significantly in the past few years, somewhat coinciding with
the 2009 introduction of Medicare’s new reimbursement policy. Partly driven by these reporting
and financial motivations, institutions and researchers have developed a variety of strategies to
identify and reduce preventable readmissions. Some studies have focused on describing the
readmissions landscape at the national level while others have focused on the local and hospital
levels. There have been a number of predictive studies exploring risk factors for different patient
groups to better understand the dynamics of readmissions. These studies have shown a
Appendix B (continued)
30
pervasive lack of standard systems or processes to ensure post-discharge compliance to
exercise treatment instructions (e.g. medication, diet, and follow-up care) (Krumholz et al.,
2002), so a number of the studies have focused on developing interventions to improve
information transfer and other aspects of the discharge process. We grouped the papers into
three categories: descriptive analysis (43), intervention studies (34), and statistical or predictive
models (35).
Figure 1. Publications categorized as descriptive, intervention, or predictive.
The remainder of this paper is organized as follows: Section 2 discusses definitions,
measurements, and descriptive analyses reported in the literature; Section 3 summarizes
common preventive approaches proposed, evaluated or practiced by healthcare institutions;
and Section 4 reviews statistical and predictive models discussed in the literature. A discussion
0
5
10
15
20
25
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Fre
qu
en
cy
Year
Number of Publications per Year
Descriptive Intervention Predictive
Appendix B (continued)
31
of research gaps and opportunities for future work is presented in Section 5, the last section of
this paper.
2. Definitions, Measurements, and Descriptive Analyses
Depending on the study or context, hospital readmissions are typically defined using a time
window from the time of discharge, i.e. “n-day readmission” (common windows being 14, 30, 90,
and 180-day readmission rates). A study by Heggestad and Lilleeng (2003) found 28% of all
readmissions occur within 10 days, 49% within 30 days, and 79% within 90 days. Estimating
exact readmission rates, however, is problematic due to a variety of data accuracy and patient
tracking issues. For example, the primary and secondary diagnoses of readmitted patients often
are not the same as their index admissions, even when the cases are linked. Moreover, same-
hospital readmissions capture only 80.9% of all-hospital readmissions, with a significant number
of patients being readmitted to a different hospital (Nasir et al., 2010).
Figure 2 illustrates the general context within which readmissions occur. After the initial
(index) admission and treatment, a patient is usually released home following a discharge
process in which home care, diet, medication, exercise, and other instructions are reviewed with
the patient and his or her family. Depending on the patient’s condition and the particular
healthcare organization, in the time between this initial discharge and subsequent readmission,
the patient may be contacted by phone to review discharge instructions and address any
questions, be visited by a home health nurse or other provider, or be monitored by some form of
home monitoring technology. Later, the patient may be readmitted to a hospital under the same
or different diagnostic coding. For example, a patient could be readmitted for a broken leg when
his or her index admission was the result of heart failure. Adding to the complexity, the patient
may return for care, but not to the same hospital. For example, in examining Medicare patients
readmitted within 30 days after undergoing one of three common surgical procedures, Gonzalez,
Shih, Dimick, and Ghaferi (2013) found that only 64% were readmitted to the same hospital.
Finally, reasons for a readmission can vary. They include, but are not limited to, non-compliance
Appendix B (continued)
32
to discharge instructions, the quality or completeness of care received during the initial hospital
stay, and an iatrogenic injury. This care cycle for the patient may occur several times between a
discharge location, such as the patient’s home, and a hospital or set of hospitals.
Many factors can come into play when investigating readmissions. For instance, if in the
above example the patient’s admission due to a broken leg to the same hospital is counted as a
readmission, it may cause misleading conclusions about the quality of care that patient received
during his or her index admission for heart failure. Moreover, readmissions analyses often do
not consider readmissions to another hospital due to lack of data, whereas these readmissions
may be an indicator of unsatisfactory patient care at the index admission hospital. Also, the time
between readmissions may be reflective of the quality of hospital care or post-discharge care.
For example, short cycling may be due to the patient’s poor adherence to discharge instructions
and have nothing to do with the quality of care provided by the hospital.
Figure 2. General readmissions context
Readmissions can also be classified as planned or unplanned, where planned refers to an
intentional admission that is a scheduled part of a patient’s care plan, such as chemotherapy or
rehabilitation. One study estimated 47.1% of patients readmitted within 30 days were unplanned
(Maurer & Ballmer, 2004). Unplanned readmissions can be either (potentially) preventable (e.g.,
Index admission for reason 1
Discharge
Readmission due to reason i
Timet 1 t 7t 6t 5t 4t 3t 2 t m +1t mt 8
Hospital n
Hospital 2
Hospital 1
LOS1 ICF
Index admission
LOS2
Readmission1
Home LOS3 Home LOS4 Home LOS5
Readmission2
to different facility
Unrelated
Readmission3
Readmission4
to different facility
LOS: Length of stay
ICF: Intermediate care facilityi
...
...
//1 2
1
1
Appendix B (continued)
33
congestive heart failure, bacterial pneumonia, urinary tract infection, surgical wound infection) or
non-preventable (e.g., trauma, unexpected finding of malignancy). While estimates vary for the
percent of unplanned admissions that are preventable, Jiang, Russo, and Barrett (2009) reports,
in a study of nearly 4.4 million admissions in 2006, that 18% of the adult admissions were
potentially preventable. Ascertaining whether a patient’s condition is preventable or not can
exacerbate the accurate identification of a readmission. In practice, making this determination is
often assessed by various types of clinical experts (e.g. surgeons, general physicians) whose
background may influence their analyses and conclusions.
Preventable readmission rates range widely in the literature from 5.5% to 49.3% (see Table
1 in Appendix), due to practice-to-practice variations, different diagnoses, and a lack of
consistent definition and measurement criteria (Clarke, 1990). Some authors agree that the use
of readmission rates as an indicator of the quality of care in a previous admission may not
always be reasonable (Benbassat & Taragin, 2000; Chen et al., 2010; Weissman et al., 1999).
Therefore, factors beyond those solely related to quality of care during a hospital stay should be
considered as potential causes of readmissions.
3. Prevention Interventions
Most of the intervention articles reviewed culled recommendations from the literature or
experimental studies. Summaries of many of these interventions can be found in Greenwald,
Denham, and Jack (2007); Kanaan (2009); Olson et al. (2011); Simmons (2010) and Taylor
(2010). Osei-Anto, Joshi, Audet, Berman, and Jencks (2010) and Jweinat (2010) summarize
successful interventions and provide a framework for the development of readmission
prevention programs in hospitals. Two of the papers Trisolini, Aggarwal, Leung, Pope, and
Kautter (2008) and Healthleaders Media (2010) focus on healthcare quality.
Table 2 in the appendix summarizes common interventions discussed in the literature. A
large majority of these publications tend to focus on a few diagnoses or a specific population of
patients. Table 3 shows the patient diagnoses most commonly cited, including congestive heart
Appendix B (continued)
34
failure (CHF) and acute myocardial infarction (AMI). High-risk patients were often determined
using some form of assessment (Bisognano & Boutwell, 2009; Rayner, Temple, Marshall, &
Clarke, 2002).
Most of the interventions can be grouped into general improvements for transitions of care,
redesigning the discharge process, or enhanced follow-up care strategies. Interventions to
improve transitions of care included: (1) enhanced assessment of patient needs (such as quality
of inpatient care, accurate medication reconciliation, effective education and communication at
discharge, post-discharge support, follow-up referrals, effective communication of clinical
prognosis, and proactive end-of-life care planning) (Bisognano & Boutwell, 2009; Institute for
Healthcare Improvement, 2009b); (2) general guidelines for readmission prevention efforts
(such as assessing, prioritizing, implementation and monitoring) (Osei-Anto et al., 2010), and (3)
models for improved care coordination/transition between settings (Bodenheimer, 2008; Institute
for Healthcare Improvement, 2010b).
The main components in interventions focusing on the discharge process consisted of: (1)
the careful design of the discharge process and all related activities (Clancy, 2009; Institute for
Healthcare Improvement, 2009a); (2) the use of patient-centered approaches (Jack et al., 2008;
Jweinat, 2010); (3) the simplification of the discharge process for patients and caregivers
(Balaban, Weissman, Samuel, & Woolhandler, 2008); (4) providing patients with clear
instruction on risks, symptoms, complications, and their adequate management (Grafft et al.,
2010; Patient Safety Authority, 2005); and (5) the use or development of information technology
for the communication of key discharge information (Motamedi et al., 2011). Better education of
patients and medical staff was also found to decrease readmission rates (Bisognano & Boutwell,
2009).
Common interventions directed towards post-discharge, follow-up care included the
following: (1) increased frequency or intensity of follow-up activities (Rayner et al., 2002; Rich et
al., 1995); (2) increased primary care access (Cline, Israelsson, Willenheimer, Broms, & Erhardt,
Appendix B (continued)
35
1998; Strunin, Stone, & Jack, 2007; Weinberger, Oddone, & Henderson, 1996); (3) high-risk
screening tools to determine the need for intervention (Manning, 2011); (4) home health
monitoring technology (Institute for Healthcare Improvement, 2010a); (5) improved
communication between primary care and inpatient providers to facilitate timely and accurate
transfer of key patient information (Ornstein et al., 2011); (6) healthcare worker (e.g., physician,
nurse, physiotherapists) visits after discharge (Andersen et al., 2000; Ornstein et al., 2011); and
(7) phone-based follow-up after discharge (Harrison, Hara, Pope, Young, & Rula, 2011; Kasper
et al., 2002) or a combination of visits and phone calls after discharge (Naylor et al., 1999).
Performance metrics used to evaluate the effectiveness of interventions include compliance
rates, readmission rates, days until readmission, readmission lengths of stay, readmission costs,
emergency department visit costs, overall cost of care, mortality rates, inpatient/outpatient
resource utilizations, patient satisfaction, and quality of life. Compliance rates attempt to
measure the extent to which an intervention is being carried out (e.g., rates of follow-up and
counts of incomplete outpatient workups (Balaban et al., 2008). Two articles proposed
measures to better evaluate readmissions (Bhalla & Kalkut, 2010; Institute for Healthcare
Improvement, 2003). However, there is still a need to define and implement standardized
performance metrics that can assist in assessing or validating the level of success of an
intervention. Studies should incorporate a measure of the fidelity of the actual intervention
implementation as a predictor variable for the performance metrics being evaluated. The
development of these metrics should reflect the priorities of patients and healthcare providers,
and should facilitate the identification of specific areas in need for reengineering.
Even though most studies developed their proposed interventions based on widely accepted
good clinical practices and patient-centered care, three studies did not find significant
differences between intervention and control groups (Grafft et al., 2010; Rayner et al., 2002;
Weinberger et al., 1996). One study found that the efficacy of their intervention was relatively
smaller in congestive heart failure patients as compared to other patients (Naylor et al., 1999),
Appendix B (continued)
36
which may suggest the need to tailor interventions according to the needs of different patient
groups. A recent report from the Agency for Healthcare Research and Quality on the
effectiveness of interventions to improve transitions for acute stroke and myocardial infarction
patients found that while some outcomes, such as hospital length of stay and mortality, are
often improved by intervention, most studies have not been able to clearly demonstrate a
positive or negative effect on metrics of systems’ or patient’s outcomes (Olson et al., 2011).
Five studies included a cost analysis based on costs per patient, annual healthcare cost per
patient, total Medicare reimbursements for health services at 24 weeks after discharge,
discharge costs, and possible implications of readmission cost policies on care quality (Balaban
et al., 2008; Cline et al., 1998; Naylor et al., 1999; Rich et al., 1995; Simmons, 2010). Cost
benefit analysis of interventions are especially important in the light of the Medicare
reimbursement penalty for those hospitals with consistently increased readmission rates.
The actual adoption of intervention strategies to reduce readmission rates in hospitals is
questionable (Butler & Kalogeropoulos, 2012). Bradley et al. (2012) found that although most
hospitals in the hospital-to-home (H-2-H) quality improvement initiative had a written objective
related to reducing preventable readmissions for patients with heart failure or AMI, actual
interventions and levels of implementation varied widely. The survey study found that less than
50% of the hospitals surveyed had fully implemented any single key practice and less than 3%
were currently using all of the 10 practices investigated in the study. The practices with the
highest adoption level included: partnering with community hospitals (49.3%), partnering with
local hospitals to manage high risk patients (23.5%), linking inpatient and outpatient prescription
records (28.9%), and consistently sending the discharge summary to the patient’s primary
medical doctor (25.5%). Regardless of the intervention strategies selected, the implementation
of such strategies needs to be carefully planned and executed to maximize their potential for
success.
Appendix B (continued)
37
Measuring the success of an intervention is still a challenge because of the difficulty of
defining variables that capture the quality of healthcare delivery, patient satisfaction, health
status, and healthcare provider satisfaction. Consequently, some interesting challenges may
exist when conducting statistical and predictive analysis of both intervening factors and outcome
variables, which is discussed in the next section.
4. Statistical and Predictive Analysis
The most common statistical approaches used in analyzing readmission data are logistic
regression and survival analysis (Almagro et al., 2006; Beck, Khambalia, Parkin, Raina, &
Macarthur, 2006; Epstein, Tsaras, Amoateng-Adjepong, Greiner, & Manthous, 2009; French,
Bass, Bradham, Campbell, & Rubenstein, 2008; Greenblatt et al., 2010; Hannan et al., 2003;
Hasan et al., 2010; Hendryx et al., 2003; Holloway & Thomas, 1989; Jasti, Mortensen, Obrosky,
Kapoor, & Fine, 2008; Luthi, Burnand, McClellan, Pitts, & Flanders, 2004; Mudge et al., 2010;
Neupane, Walter, Krueger, Marrie, & Loeb, 2010; Philbin & DiSalvo, 1999; Tsuchihashi et al.,
2001; van Walraven et al., 2010; Weiss, Yakusheva, & Bobay, 2010). Other more sophisticated
statistical models have also been applied in specific situations. For example, Medress and
Fleshner (2007) used Wilcoxon nonparametric and Fisher’s exact tests to compare continuous
and categorical variables, respectively. Allaudeen et al. (2010) employed multivariable
generalized estimating equations for clustering of patients within physician assignments and
calculating the adjusted odds ratios to identify factors significantly associated with readmissions.
Generally speaking, standard statistical tests and criteria are typically used to identify
associated factors (e.g. t-test, chi-square test, Pearson correlations); and more sophisticated
techniques are used for prediction models. For example, Glasgow, Vaughn-Sarrazin, and Kaboli
(2010) used t-tests to analyze continuous variables and chi-square tests to analyze categorical
variables to compare patient baseline characteristics between two groups (those discharged
against medical advice and those with a standard discharge), multivariable Cox proportional
hazard models to predict the time to readmission, and stepwise model selection to “determine
Appendix B (continued)
38
which of the remaining covariates also represented significant risk factors in each separate
model.”
The work to identify factors associated with readmissions is summarized in Table 4 in the
Appendix. We can see that a fair amount of work has been published studying factors
associated with readmissions in specific patient populations. Heart failure and pneumonia are
by far the most commonly studied diseases. The factors considered include patients’ biological,
social, and economical characteristics and hospital discharge and post-discharge processes. It
should be noted that several research articles have demonstrated that education (Koelling,
Johnson, Cody, & Aaronson, 2005; Krumholz et al., 2002), intervention (Hernandez et al., 2010;
Riegel et al., 2002), and hospital discharge programs (Jack et al., 2009; Lappe et al., 2004)
have had positive effects on readmissions.
Another important body of literature has to do with constructing statistical models to predict
readmission rates. Table 7 in the appendix summarizes papers from 1989 through 2010 related
to readmission prediction; and Table 8 summarizes the focus of each paper and the frequency
of the common predictive factors. Age and gender were the two most common predictive factors
analyzed, and have appeared in roughly two-thirds of all examined papers. Comorbidity, length
of stay, prior admissions, and ethnicity were also commonly identified predictors. Other studies
focused on very specific predictive factors, especially those that considered a subset of patients,
with specific diagnoses or diseases sometimes tested as independent or causal variables. In a
study of psychiatric patients, for instance, Hendryx et al. (2003) examined the association
between a primary diagnosis of schizophrenia and subsequent readmission.
While some authors examined all types of admissions and readmissions, it is more common
to limit the patient sample to a diagnosis or demographic subset. For instance, Lagoe,
Noetscher, and Murphy (2001) and Luthi et al. (2004) both focused on patients diagnosed with
heart failure, since this is the leading diagnosis associated with readmission.
Appendix B (continued)
39
Generally speaking, the data sources used in these predictive studies can be classified into
one of two levels:
(1) Hospital, in which data are typically collected and analyzed within one to three specific
healthcare facilities. An example is the study reported by Hendryx et al. (2003) at the
Harborview Medical Center in Seattle, WA.
(2) Database, in which data are typically collected and analyzed at the state or national level.
Examples include studies reported by Hannan et al. (2003); Holloway and Thomas
(1989); Philbin and DiSalvo (1999), and Lagoe et al. (2001) conducted in New York
State hospitals.
Tables 5 and 6 in the appendix summarize these hospital and database studies, respectively.
The latter type of study generally had larger sample sizes because of their wider service regions.
A focus on heart failure patients is even more common in database studies, as seen in Hofer
and Hayward (1995); Keenan, Normand, and Lin (2008); Krumholz et al. (2000); Luthi et al.
(2003); and Philbin and DiSalvo (1999). In addition, two studies used hospitals rather than
patients as the unit of analyses. In one, Boulding, Glickman, Manary, Schulman, and Staelin
(2011) investigated the relationship between patient satisfaction survey results aggregated at
the hospital level and 30-day hospital readmission rates. In the other, Hansen, Williams, and
Singer (2011) explored the relationship between 30-day risk-adjusted readmission rates and
patient safety climates, assessed through employee surveys.
5. Challenges and Opportunities for Industrial Engineers
As shown in the literature review, we have witnessed a growing analysis of various aspects
related to hospital readmissions. During the last decade much of the work has concentrated on
data analysis and the design and assessment of interventions. A fair amount of consulting and
proprietary methods are also increasingly appearing in hospitals and conferences. The
IE/STAT/OR community has become more and more involved in the area, and we are
presented with several promising opportunities.
Appendix B (continued)
40
While the analysis methods used tend to be fairly rigorous, few large-scale unified studies
have been conducted. The scope of most studies are either disease specific, fairly localized (i.e.
limited to a single hospital) or very broad (i.e. statewide admissions). Opportunities exist in the
IE/STAT/OR research domain to develop models that better capture the necessary granularity
that can be integrated in a more generalizable manner. This will require proposing and
validating new readmission metrics, especially as they relate to all-cause, comorbid and
longitudinal (i.e., over 30 days) conditions. Research into readmission patterns that extend
beyond the ubiquitous frequency measures may also prove to be helpful. Additionally, the need
for care coordination and population health studies abound. Out of this should come new
methodologies that better incorporate the human experiences.
Several opportunities exist to contribute to the analysis and improvement of readmissions.
One of the most common limitations throughout the various studies was the availability of data
to identify, manage and prevent readmissions. In the case of intervention implementation and
evaluation, the most common barriers included a lack of uniform data about factors that may be
related to readmissions (Harrison et al., 2011), difficulty in sharing information across
organizations, assessing and ensuring patient and provider compliance (Grafft et al., 2010;
Patient Safety Authority, 2005), and a lack of validated processes for determining if the
readmissions were related or not to an index admission (Andersen et al., 2000; Institute for
Healthcare Improvement, 2010b).
Evaluating the risk of (preventable) readmissions is a challenge due to the lack of clinical
data in the identification of significant factors. Clinical data is available; however, physician
notes, test results, and images are not structured and are not easily extracted for statistical
analyses. Moreover, the existence of confounding factors can limit data analysis, a problem not
easily overcome for observational studies (Hernandez et al., 2010). For example, Moore,
Wisnivesky, Williams, and McGinn (2003) retrospectively analyzed medical errors related to
care discontinuity between inpatient and outpatient settings, although patients with work-up
Appendix B (continued)
41
errors may be subsequently managed differently than others. Weissman et al. (1999) studied
care quality during initial admissions, but did not consider post-discharge care, while van
Walraven, Seth, Austin, and Laupacis (2002) analyzed the effect of discharge summary
availability, but did not control for care during the initial hospitalization.
The classification of readmissions (e.g., planned versus unplanned, avoidable versus
unavoidable) can also limit analyses, especially those mainly focused on a specific type of
readmission. For example, Jencks et al. (2009) focused on related adverse readmissions
(RAR) and non-RARs, classifying readmissions as planned or unplanned and avoidable or
unavoidable. Classification errors can also occur due to the lack of a second independent
examiner to confirm (Maurer & Ballmer, 2004), potentially introducing noise into subsequent
statistical analyses. Some studies do not distinguish between planned and unplanned
readmissions (Dormann et al., 2004; Nasir et al., 2010). Again, this is often due to a lack of
data. A standardized system for classifying readmission types, therefore, would make results
more generalizable and cross-comparable, especially to facilitate selection of appropriate
intervention strategies or predictive models.
As in most health services research, clinical information systems or administrative data are
used predominantly in retrospective studies, which can limit the types of available data and
reduce the ability to conduct meaningful analyses. The effects of potentially important factors,
consequently, are likely to be underestimated (Elixhauser, Steiner, Harris, & Coffey, 1998;
Harrison et al., 2011; Marcantonio et al., 1999) and incomplete data can restrict the
generalization of results. There are opportunities for improvement at all levels of data
procurement, including data collection, data selection, population selection, definition of
guidelines to classify events and patients, and identification of confounding factors. The current
effort, however, to develop data exchange standards and information systems for tracking
patients across institutions should enable better implementation and research opportunities.
Some of this research might include geospatial and socio-demographic analysis of healthcare
Appendix B (continued)
42
seeking behaviors to better understand where, how often, and why patients seek the care they
do. This understanding could lead to adopting strategies for better coordinated, patient-
centered care.
Other limitations in many of the published studies also include the short time spans of
sampled data (Miles & Lowe, 1999) and the use of nonrandomized or observational
comparisons (Lappe et al., 2004) or narrow sample groups (Ashton et al., 1995; Koelling et al.,
2005; Krumholz et al., 2009). For example, Ashton et al. (1995) studied the association
between the quality of inpatient care and early readmission only among males using Veterans
Affairs hospitals, potentially limiting the generalizability of the results.
In terms of study populations, many papers focused on particular disease types, age
groups, or social statuses. In the case of studies related to interventions, addressing specific
patient populations has shown significant benefits since these efforts can focus more effectively
on the particular needs of these patient groups (Grafft et al., 2010).
Regarding the use of interventions, implementation-specific factors and intervention
characteristics were not explicitly addressed in a majority of the studies. For example, most
interventions are formed by a set of activities or strategies that may or may not work as a whole
(e.g., assessment methodology and follow-up procedure variables, such as time to follow-up or
type of follow-up). The majority of studies focused on validating the overall effectiveness of the
proposed intervention, but few attempted to find the specific characteristics of the population or
the particular activities and strategies that made the intervention successful (Naylor et al.,
1999). For example, an intervention to reduce readmissions of patients with heart failure
discharged to skilled nursing facilities found that enhanced communication among caregivers
was key to reducing the corresponding preventable readmissions (Jacobs, 2011). It is important
to distinguish between strategies that are effective for the general population and strategies that
are effective for specific patient groups, so that risk assessment can be used to determine the
“optimal intervention plan” needed, if any.
Appendix B (continued)
43
Although many studies identified factors associated with readmissions, most did not draw
conclusions about causality nor offer guidelines on how to optimize any particular intervention to
reduce readmission rates (Balaban et al., 2008; Bell et al., 2009; Chen et al., 2010; Krumholz et
al., 2002; van Walraven & Bell, 2002). For example, van Walraven and Bell (2002) found that
readmission risk may decrease with better discharge summary availability during post-discharge
visits, but was unable to determine how dissemination of discharge summaries to follow-up
physicians might avoid readmissions.
From an industrial engineering perspective, several opportunities exist to contribute to the
above efforts and issues. Perhaps most obvious are opportunities to conduct various types of
statistical modeling, potentially including data mining of large unstructured data sets and novel
predictive modeling methods beyond those already being used. Additionally, data reduction
methods such as feature recognition and principal components analysis, pseudo experimental
design methods to test causality, and modern visual exploration data analysis methods could
have particular value. Research more aligned with operations research might include
deterministic and probabilistic intervention optimization, stochastic patient flow and transition
models, comparative and cost effectiveness models for interventions, and agent-based or game
theoretic models.
Despite the heightened focus on preventing readmissions, it is not always clear if, where,
and why readmission rates are improving. Ross et al. (2010), for example, found no reduction in
readmission rates nor significant differences in rates among hospitals from 2004 through 2006
for Medicare beneficiaries discharged after hospitalization for heart failure. Thus, development
and use of methods to better estimate readmission rates and causality would seem useful as
well. Similarly, performance measures to evaluate intervention strategies (e.g., compliance,
frequency, coverage) are needed to monitor their effectiveness. Given the complexities, human
interactions, and interdependencies of multiple factors, exploring various socio-technical
analyses that better address the human factor seem especially appropriate. System dynamics
Appendix B (continued)
44
models also might be useful here, possibly including analysis of various financial and public
reporting incentives and of the introduction and optimal design of accountable care
organizations and other new integrated delivery system concepts.
In summary, numerous opportunities exist for industrial engineering and operations research
methods to complement, support, and extend the hospital readmissions work done to date,
which is now mostly being conducted within other disciplines. Given the importance of this
problem across the entire United States healthcare system, it is appropriate for industrial
engineers to begin to apply their expertise to this challenging area.
Appendix B (continued)
45
Appendix
Table 1: Proportion of preventable readmissions among unplanned readmissions
Study Group Design1
Number patients
Time interval, day
Number of Readmissions / Rates
Preventable readmissions, % of all readmissions
Clarke (1990)
General medical and geriatric Surgical
R
207
166
60
48
0-6
21-27
0-6
21-27
(in total 100 random case notes ) (74 were available) 25 case notes (18 available) 25 case notes (19 available) 25 case notes (19 available) 25 case notes (18 available)
31.5 6.3 (Total: 16.5) 49.3 19.0 (Total: 34.6)
Miles and Lowe (1999)
All RA data from JHH2 in Oct. 1998 by ACHS3 indicator
R 3,081 admissio
ns
28 437 readmissions with adequate data involving 371 patients
5.5 (out of the 437 readmissions)
Maurer and Ballmer (2004)
DIM4 of KSW5
P 884 IA6 30 90
12.3% 19.5% (planned & unplanned)
9.4 18.5 (out of unplanned)
Friedman and Basu (2004)
Persons with initial PQI7 admission
R 345,651 3 mo 6 mo
- 35.3%
13.3% 19.4% (out of the PQI admissions)
1 R: retrospective, P: prospective, 2 JHH: John Hunter Hospital, 3ACHS: Australian Council on
Healthcare Standards, 4 DIM: Department of Internal Medicine, 5 KSW: Kantonsspital Winterthur,
6 IA: index admissions, 7 PQI: Prevention Quality Indicator
Appendix B (continued)
46
Table 2: Summary of common interventions discussed in the literature
Intervention type
Intervention References
Discharge planning
Disease and treatment education
(Balaban et al., 2008; Bickmore, Pfeifer, & Jack, 2009; Bisognano & Boutwell, 2009; Cline et al., 1998; Institute for Healthcare Improvement, 2009a, 2009b, 2010a; Jack et al., 2008; Manning, 2011; Naylor et al., 1999; Ornstein et al., 2011; Patient Safety Authority, 2005; Rich et al., 1995; Weinberger et al., 1996)
Review of medication (Bisognano & Boutwell, 2009; Cline et al., 1998; Fleming & Haney, 2013; Institute for Healthcare Improvement, 2009a, 2010a; Kasper et al., 2002; Osei-Anto et al., 2010; Rich et al., 1995; Weinberger et al., 1996)
Prescribed diet (Rich et al., 1995)
Assignment of PCP (Osei-Anto et al., 2010; Weinberger et al., 1996)
Self-management education
(Cline et al., 1998; Coleman, Parry, Chalmers, & Min, 2006; Fleming & Haney, 2013; Institute for Healthcare Improvement, 2009a, 2010a; Jack et al., 2008; Manning, 2011; Osei-Anto et al., 2010; Patient Safety Authority, 2005)
Identify sources of error/risk at discharge
(Anthony et al., 2005; Institute for Healthcare Improvement, 2009b)
Risk screen patients (Institute for Healthcare Improvement, 2010b; Manning, 2011; Osei-Anto et al., 2010)
Interdisciplinary/multi-disciplinary clinical team
(Osei-Anto et al., 2010)
Transitions of care
Computer-enabled discharge communication
(Motamedi et al., 2011)
Effective patient and family engagement
(Institute for Healthcare Improvement, 2010a, 2010b)
Coordination among care sites
(Bisognano & Boutwell, 2009; Bodenheimer, 2008; Coleman et al., 2006; Fleming & Haney, 2013; Institute for Healthcare Improvement, 2009a, 2009b, 2010a, 2010b; Jacobs, 2011; Manning, 2011; Motamedi et al., 2011; Ornstein et al., 2011; Osei-Anto et al., 2010; Press et al., 2010)
Assignment of a care transitions coordinator / transitions coach
(Coleman et al., 2006; Fleming & Haney, 2013)
Follow-up Home visits (Andersen et al., 2000; Naylor et al., 1999; Osei-Anto et al., 2010; Rich et al., 1995)
Appendix B (continued)
47
Telephone contact (Balaban et al., 2008; Bisognano & Boutwell, 2009; Cline et al., 1998; Harrison et al., 2011; Institute for Healthcare Improvement, 2009a; Jacobs, 2011; Kasper et al., 2002; Naylor et al., 1999; Osei-Anto et al., 2010; Rich et al., 1995; Weinberger et al., 1996)
Compliance with instructions given at hospital
(Harrison et al., 2011; Jacobs, 2011; Motamedi et al., 2011; Rich et al., 1995; Weinberger et al., 1996)
Primary care clinic follow-up appointment
(Coleman et al., 2006; Grafft et al., 2010; Institute for Healthcare Improvement, 2009b, 2010a; Jordan et al., 2012; Kasper et al., 2002; Osei-Anto et al., 2010; Rayner et al., 2002; Weinberger et al., 1996)
Access to nurse consultation (short notice)
(Cline et al., 1998; Naylor et al., 1999)
Medical rehabilitation/therapy after discharge
(Jordan et al., 2012; Mudrick et al., 2013)
Appendix B (continued)
48
Table 3: Common diagnoses mentioned in the intervention literature.
Patient Group References
CHF (Bisognano & Boutwell, 2009; Cline et al., 1998; Coleman et al., 2006; Institute for Healthcare Improvement, 2010a, 2010b; Kasper et al., 2002; Manning, 2011; Rich et al., 1995; Weinberger et al., 1996)
Diabetes (Coleman et al., 2006; Weinberger et al., 1996)
COPD (Coleman et al., 2006; Weinberger et al., 1996)
AMI (Andersen et al., 2000; Coleman et al., 2006; Institute for Healthcare Improvement, 2010a; Mudrick et al., 2013)
Ambulatory surgery (Patient Safety Authority, 2005)
General (Balaban et al., 2008; Bickmore et al., 2009; Bodenheimer, 2008; Grafft et al., 2010; Harrison et al., 2011; Institute for Healthcare Improvement, 2009a, 2009b, 2010b; Jack et al., 2008; Jacobs, 2011; Jweinat, 2010; Motamedi et al., 2011; Ornstein et al., 2011; Osei-Anto et al., 2010; Press et al., 2010; Rayner et al., 2002)
Other (Coleman et al., 2006; Jordan et al., 2012)
Appendix B (continued)
49
Table 4: Factors associated with readmissions
Study Factor Sample Group, N=sample
size Results
Elixhauser et al. (1998)
Comorbidity
Non-maternal inpatients from in 438 acute care hospitals California N=1,779,167
Comorbidities were associated with longer length of stay, higher hospital charges, and mortality and had different effects among different patient groups
van Walraven et al. (2002)
Discharge summary availability
Patients discharged for acute medical illness from Ottawa Civic Hospital with OHIP1 number N=888
A decreased trend in readmissions was found when the factor was added (relative risk, 0.74)
Krumholz et al. (2002)
Education and support
Patients in YNHH2with heart failure from Oct. 1997 to
Sep.1998, age≥=50 N=88
Intervention group had a significantly lower risk of readmission (hazard ratio, 0.56)
Riegel et al. (2002)
nurse case-management telephone intervention
Patients with heart failure from 2 southern California hospitals N=358
The heart failure hospitalization rate was 45.7% and 47.8% lower in the intervention group at 3 and 6 months
Moore et al. (2003)
Medical errors related to discontinuity care from inpatient to outpatient setting
General patients who had been hospitalized at a large academic medical center N=86
49% of patients experienced at least 1 medical error and patients with work-up error were 6.2 times more likely to be re-hospitalized within 3 months
Dormann et al. (2004)
Adverse drug reactions
General patients from internal medicine of UHEN3; N=1000 admissions
ADRs were not significant with readmissions but with LOS
Lappe et al. (2004)
Hospital-based discharge medication program (DMP)
Cardiovascular disease from the 10 largest hospitals in UIHS4: Pre-DMP(1996-1998): N=26000; DMP (1999-2002): N=31465
Reduced relative risk for death and readmissions (hazard ratios, 0.81, 0.92)
Ather, Chung, Gregory, and Demissie (2004)
Insurance provider
Adults with asthma from NJDHHS5; N=15864
Significant increased risk of 7-day readmission for managed care patients compared to indemnity (OR, 1.67) and LOS is also significant for readmissions
(Koelling et al., 2005)
One-hour discharge education
Patients with chronic heart failure from University of Michigan Hospital; N=223; Control group=116
Patients receiving the education intervention had lower risk of re-hospitalization (relative risk, 0.65)
(Vira, Colquhoun, &
Medication reconciliation
Generally from a Canadian community hospital; N=60
18% of patients were detected having clinical important unintended variance after
Appendix B (continued)
50
Etchells, 2006)
reconciliation
(Kartha et al., 2007)
Depression Adults inpatient with at least 1 hospital admissions in the past 6 month; N=144
Depression tripled the odds of re-hospitalization (odds ratio, 3.3)
(Bailey et al., 2009)
Risks of severity
Indigenous and non-indigenous children of bronchiolitis from Royal Darwin Hospital, age≤2; N=101
No significant difference for readmission rates among the 2 groups, but indigenous children had more Severe illness
(Jha, Orav, & Epstein, 2009)
Public reporting of discharge planning
Congestive Heart Failure, using HQA6 database
NO large reduction in unnecessary readmissions
(Jack et al., 2009)
A reengineered hospital discharge program
Adults patients admitted to medical teaching service of Boston Medical Center; N=749
The intervention group(N=370) had a lower rate of hospital utilization (0.314 vs 0.451 visit per person per month )
(Hernandez et al., 2010)
Early physician follow-up
Patients ≥65 with heart failure from 225 hospitals; N=30316
Patients who are discharged from hospitals that have higher early follow-up rates have a lower risk of 30-day readmission
(Boulding et al., 2011)
Patient satisfaction 430,982 patients with acute myocardial infarction (AMI) 1,02 9,578 patients with heart failure 912,522 patients with pneumonia
Higher overall satisfaction and satisfaction with discharge planning are associated with lower 30-day risk-standardized readmission rates
(Hansen et al., 2011)
Hospital patients safety climate
36,375 employees in 67 hospitals
There is positive association between lower safety climate and higher readmission rates for AMI and HF
(K. E. Joynt, Orav, & Jha, 2011)
Race and site of care (non-minority and minority)
Medicare beneficiaries
(3.1 million in 2006 - 2008)
Black patients were more likely to be readmitted after hospitalization for AMI, congestive HF and pneumonia
(Onukwugha et al., 2011)
Discharges against medical advice(AMA)
348,572 patients from nonfederal acute care hospitals in Maryland with CVD (Cardiovascular disease)
The percentage of patients who were readmitted was higher among AMA group versus non-AMA group
1OHIP: Ontario Health Insurance Plan, 2YNHH: Yale New Haven Hospital, 3UHEN: University
Hospital Erlangen-Nuremberg, 4UIHS: Utah-based Intermountain Health Care System,
5NJDHHS: New Jersey Department of Health and Senior Services, 6HQA: Hospital Quality
Alliance Program
Appendix B (continued)
51
Table 5: Summary of hospital-level studies
Paper Location/Type Sample Size Notes
(Allaudeen et al., 2010)
550-bed tertiary care academic medical center in San Francisco, CA
6805 patients 10,359 admissions
General medicine
(Almagro et al., 2006)
Acute-care teaching referral center in Barcelona, Spain.
129 patients COPD
(Capelastegui et al., 2009)
400-bed teaching hospital in the Basque country (northern Spain)
1117 patients Pneumonia
(Halfon et al., 2002)
Centre Hospitalier Universitaire Vaudois, Lausanne, Switzerland (CHUV) - 800-bed university hospital
3474 patients
(Hendryx et al., 2003)
Harborview Medical Center in Seattle, WA
1384 patients Psychiatric
(Jasti et al., 2008) 7 hospitals in Pittsburg 577 patients CAP
(Lagoe et al., 2001)
3 hospitals in Syracuse, New York: Community-General Hospital-306 beds; Crouse Hospital-566 beds; St. Joseph's Hospital Health Center-431 beds
1500+ discharges CHF
(Luthi et al., 2004) 3 Swiss academic medical centers (all urban public university hospitals)
934 patients HF
(Medress & Fleshner, 2007)
Cedars-Sinai Medical Center in Los Angeles, CA
202 patients Colitis
(Mudge et al., 2010)
Internal Medicine Department of a tertiary teaching hospital in Brisbane, Australia.
142 patients
(Weiss et al., 2010)
4 Midwestern hospitals 162 patients Medical-surgical
Appendix B (continued)
52
Table 6:Summary of database-level studies
Paper Location/Type Sample Size Notes
(Beck et al., 2006) Canadian Institute for Health Information database
334,959 Pediatric patients
(Boult et al., 1993) Longitudinal Study of Aging (LSOA)
5,876 Elderly people 70 years old and older
(French et al., 2008)
National Medicare and Veterans Health Administration (VHA) facilities.
41,331 Medicare Elderly veterans
(Glasgow et al., 2010)
129 acute care Veterans Administration hospitals
32,819 patients 1,930,947 admissions
Left against medical advice veterans
(Greenblatt et al., 2010)
Centers for Medicaid and Medicare Services
42,348 patients Colectomy
(Goldfield et al., 2008)
249 Florida inpatient hospitals 4,311,653 admissions
(Hannan et al., 2003)
New York State hospitals 16,325 patients CABG surgery
(Hasan et al., 2010)
Multi Center Hospitalist Study data (designed in six academic medical centers in the US)
10,946 patients General medicine
(Hofer & Hayward, 1995)
190 hospitals in the statewide Michigan Inpatient Database
603,959 patients
HF, gastrointestinal, neuologic, pulmonary disease
(Holloway & Thomas, 1989)
1980 National Medical Care Utilization and Expenditure Survey data
2206 patients
(Keenan et al., 2008)
2002-2005 Medicare claims data frfom the Medicare Enrollment Database
>1 million admissions
HF
(Krumholz et al., 2000)
18 Connecticut Hospitals 2176 patients HF 65+
(Luthi et al., 2003)) 50 community hospitals in Colorado, Connecticut, Georgia, Oklahoma, and Virginia
2943 patients HF
(Onukwugha et al., 2011)
Maryland Health Services Cost Review Commission Database
348,572 patients CVD
(Philbin & DiSalvo, 1999)
New York State Department of Health
42,731 patients CHF
Appendix B (continued)
53
(van Walraven & Bell, 2002)
11 hospitals (6 university-affiliated, 5 community) in Ontario
4812 patients Medical or surgical
(van Walraven et al., 2010)
Discharge Abstract Database (DAD), which records all discharges from Ontario hospitals
2.4 million patients
Non-elective admissions adult
Table 7: Summary of papers from 1989 through 2010 related to readmission prediction
Author Dates R/P
Readmission Definition
Diagnosis
Sample group
Readmission Rate
Method Significant Factors
(Allaudeen et al., 2010)
Jun 2006 May 2008
R
30-days unplanned
General medicine patients
Sample size: 6805; The University of California , San Francisco Medical Center
17.0%
Multivariable generalized estimating equations
Black race, Medicaid as payer, High risk medications, Comorbidities (CHF, renal disease, cancer, weight loss, iron deficiency anemia)
(Allaudeen, Schnipper, Orav, Wachter, & Vidyarthi, 2011)
Mar 2008 Apr 2008
R 30-days
general medicine patients
Sample size: 164; University of California , San Francisco Medical Center
32.7%
Receiver-operating characteristic (ROC) curves
Older age, male sex, poor self-rated general health, availability of an informal caregiver, coronary artery disease, diabetes, hospital admission within last year, more than six doctor visits during the previous year
(Almagro et al., 2006)
Oct 1996 May 1997
P 1-year COPD
Sample size: 129; Acute care teaching referral center, Barcelona, Spain
58.1% Multivariable logistic regression
Previous hospitalization for, COPD, Hypercapnia at discharge, Poorer quality of life
Appendix B (continued)
54
(Beck et al., 2006)
Jan 1996 Dec 2000
R 30-days
Pediatric
Sample size: 334,959; Pediatric population (Age≤18) Canadian Institute for Health Information Discharge Abstract Database
3.4% 3.6%
(discharged on Friday) 3.3%
(discharged on Wednesday)
Multivariable logistic regression
Number of diagnoses; In-hospital complications; Hospital admission within prior 6 months
(Berman et al., 2011)
2008 R 30-days
Advanced liver disease
Sample size: 447; Hepatology service at Indiana University Hospital and University of Colorado Hospital
20% Multivariate analyses
End-stage liver disease scores; presence of diabetes; male gender
(Boulding et al., 2011)
July 2005 June 2008
R
30-day risk standardized
AMI, HF, Pneumonia
Unit of analysis was hospital; AMI: 1798 hospitals, HF: 2561 hospitals, Pneumonia: 2562 hospitals. Hospital Compare database by the US Department of Health and Human Services; HCAHPS patient satisfaction survey data
20% (for all clinical areas)
Logistic regression
Overall patient satisfaction for AMI, HF, pneumonia (negatively); Patient satisfaction with discharge planning for HF (negatively)
Appendix B (continued)
55
(Boult et al., 1993)
1984 R 4-year
Elderly people
Sample size: 5876; 70 years old and older; Longitudinal Study of Aging (LSOA) data
28.4% Multivariate logistic regression
Age, Gender, Self-rated general health, Availability of an informal caregiver, Coronary artery disease, Previous hospital admission, More than six doctors visit, Diabetes
(Capelastegui et al., 2009)
Jul 2003 Jun 2007
P
30-day admission-related & admission-unrelated
CAP
Sample size: 1,117; Galdako Hospital, Spain
7.3%
Cox proportional Hazard regression models
Pneumonia related: Treatment failure, Instability factors upon discharge Pneumonia unrelated: Age >65, Charlson index>2, Decompensated comorbidities
(Demir, Chaussalet, Xie, & Millard, 2008)
1997-2004
R All types
COPD, Stroke, CHF
Sample size: COPD: 696,911; Stroke: 546,406; CHF: 533,439; The Department of Health in England's Hospital Episode Statistics
COPD: 39%
Stroke: 21% CHF: 36%
Coxian phase-type distribution fitting via maximum likelihood Bayesian classification
Optimal time windows: COPD: 45 days Stroke: 16 days CHF: 39 days
(Fleming & Haney, 2013)
1999-2002
R 30-days
Hip fractures
Sample size: 41331; Medicare patients (≥65 years old); National Medicare and VA
18.3% Logistic regression
Men, Long inpatient stay, Elixhauser comorbidities
Appendix B (continued)
56
(Glasgow et al., 2010)
Oct 2003 Sep 2008
R
30-days all-cause Readmission to any VA hospital
General medicine patients
Sample size: 1,930,947; 32,819 AMA patients; Specified in patients left AMA; Veteran Administration Hospital
11% (patients who
discharged
home) 17.7% (AMA patient
s)
Multivariable Cox proportional hazards model
Discharge AMA, Age, Income Comorbidities (Arrythmia, dementia, fluid disorder, MI, psychosis, Non-white race
(Goldfield et al., 2008)
2005-2006
R
15 days index admission related Readmission to same&any hospital
All types
Sample size: 4,311,653; 249 Florida inpatient hospitals
6.% (15 days, same
hospital)
7.9% (15
days, any
hospital)
-
Reason for admission, Severity of illness, Extremes of age, Presence of mental health diagnoses, Substance abuse problems
(Greenblatt et al., 2010)
1992-2002
R
30-days Readmission to any hospital
Patients who had colectomy
Sample size: 42,348; Surveillence, Epidemiology, and End Results (SEER)-Medicare database (Age≥66)
11% Multivariate logistic regression
Male, Asian/Pacific race, Region, Prior hospitalization, Comorbidity, Emergent admission, Prolonged hospital stay, Blood transfusion, Ostomy, Postoperative complication, Discharge to SNF, Hospital procedure volume (negatively)
(Halfon et al., 2002)
Jan 1997 Dec 1997
P 31-day All types
Sample size: 3,474; Centre Hospitalier Universitaire Vaudois, Switzerland
23%
Stepwise selection beased on Wald statistic
Previous hospitalization, Long LOS, High Charlson comorbidity index, Surgical stay and low
Appendix B (continued)
57
Charlson score (negative)
(Hannan et al., 2003)
Jan 1999 Dec 1999
R
30-days CABG related statewide readmission
CABG
Sample size: 16325; New York State's Cardiac Surgery Reporting System
15.3% Stepwise logistic regression
Older age, Women, Having larger body surface area, Having a myocardial infarction, Comorbidities (hepatic failure, dialysis), Hospital annual surgery volume < 100, Hospitals with high risk-adjusted mortality rates, Discharge to SNF, Longer LOS
(Hansen et al., 2011)
2006-2007 (survey data); 2008 (readmission rates)
R
30-day risk-standardized
AMI, HF, Pneumonia
Unit of analysis: Hospitals, Sample size: 67 hospitals. Patient Safety Climate in Healthcare Organizations survey data responses
- Multiple regression
Hospital safety climate for AMI and HF(negatively).
(Hasan et al., 2010)
Jul 2001 Jun 2003
R
30-days all-cause, to index or another hospital
General medicine patients
Sample size: 7287 (derivation), 3659 (validation); Multicenter Hospitalist Study data
17.5% Multivariable logistic regression
Insurance type, Marital status, Having a regular physician, Charlson index, Physical Medical Outcomes, Admissions in last year, LOS longer than 2 days
Appendix B (continued)
58
(Hendryx et al., 2003)
1997 R
1-year statewide readmission
Psychiatric patients
Sample size: 1384; Harborview Medical Center, Seattle, Washington State Department of Social and Health Services , Mental Health Division database
8.2% (Depression: 1.5%; Bipolar disorder: 7.1%; schizophrenia: 16%; other: 8.8%)
Continuous variables: Least-squares linear; Categorical variables: Maximum-likelihood logistic multiple regression
Substance abuse, Global assessment of functioning score, Prior hospitalization or outpatient service use , Age, Social support unreliability, Activity of daily living dysfunction
(Holloway & Thomas, 1989)
1980 R
31-days all-cause
All types
Sample size: 2946; National Medical Care Utilization and Expenditure Survey data
9.5% (all-
cause) 3.1%
(linked) 6.1%
(same-conditio
n)
Multiple logistic regression
Very high risk or high risk condition group for the index stay, Poor or fair health status, Surgery during the index stay to a patient with health-related activity limitations
(Jasti et al., 2008)
Feb 1998 Mar 1999
R
30-days CAP-related Comorbidity-related
CAP
Sample size: 577; 7 hospitals in Pittsburg, Pennsilvania
12.00% Multiple logistic regression
Low education level; Unemployment; Coronary artery disease; COPD
(Keenan et al., 2008)
2002-2005
R
30-days all-cause
HF
Sample size: 567,447; Medicare Standard Analytic Files, Medicare Enrolment Database (Age≥65)
23.6% Hierarchical logistic regression
Age, Gender, 9 cardiovascular variables, 26 comorbidities
Appendix B (continued)
59
(Krumholz et al., 2000)
1994-1995
R
6-months all-cause statewide readmissions
HF
Sample size: 1129(derivation), 1047(validation); Medicare patients (≥65 years old); 18 Connecticut Hospital
49% (all
cause) 23% (HF-
related)
Cox proportional Hazard models
Prior readmission within 1 year, Prior heart failure, Diabetes, Creatinine level>2.5 mg/dL
(Lagoe et al., 2001)
1998-1999
R
30-days unplanned same category diagnosis
CHF
Sample sizes: 465 (Crouse Hospital); 575 (St. Joseph's Hospital); 366(Community General Hospital)New York Statewide Planning and Research Cooperative System
9%( Crouse
Hospital)
10.8% (St.
Joseph's
Hospital)
11% (Comm
unity Genera
l Hospita
l)
Manual stepwise regression
Crouse Hospital: Secondary diagnosis of cardiomyopathy or renal failure, 60 to 69 years old, inpatient stays of 6 days or more. St. Joseph's Hospital: Secondary diagnosis of renal failure and diabetes, 60 to 69 years old. Community General Hospital: Secondary diagnosis of renal failure and diabetes
(Lin, Chang, & Tseng, 2011)
Aug 2006 Dec 2008
P
30, 90, 180, and 360-days
acute stroke
Sample size: 2,657; community hospital in southern Taiwan
30-day – 10% 90-day – 17% 180-day – 24% 360-day – 36%
Kaplan-Meier method; Cox proportional hazard models
age, previous stroke, atrial fibrillation, coronary artery disease, complications at the index hospitalization, longer length of stay, dependency at discharge
Appendix B (continued)
60
(Luthi et al., 2004)
Jun 1995 Sep 1996
R 21-months
HF (LVSD)
Sample size: 611; Medicare database (Age≥65),
70.0% Bivariate analysis
Receiving no or low dose ACEI, prior MI, History of heart failure, Diabetes, Elevated creatinine level
(Luthi et al., 2003)
Jan 1999 Dec 1999
R
30-days all-cause
HF
Sample size: 1055; Three Swiss academic medical centers
13.2% Multivariate logistic regression
None of the quality of care factors were significant
(Medress & Fleshner, 2007)
Aug 2001 Aug 2006
R
30-days unplanned, to index or another hospital
Patients who had colectomy
Sample size: 202; Cedars-Sinai Medical Center, Los Angeles
19.0%
Median comparison with Wilcoxon nonparametric test; Categorical variables' comparison: Fisher's exact test
No preoperative or surgical factor was associated with readmissions
(Mudge et al., 2010)
Feb 2006 Feb 2007
P
6-months unplanned
All types
Sample size: 142; Age≥50; Had prior two or more hospitalizations; Tertiary teaching hospital, Brisbane, Australia
39.0% Multiple logistic regression
Chronic conditions, Body Mass Index, Depressive symptoms
(Neupane et al., 2010)
Jul 2003 Apr 2005
P
90-days all-cause
CAP
Sample size: 717; 2 Canadian cities; Age ≥65;
11.2% Logistic regression
Male, Vitamin E supplement given
Appendix B (continued)
61
(Onukwugha et al., 2011)
2000-2005
R
CVD-related, 7-day, 31-day, 180-day after discharge AMA, to the same hospital
CVD
Sample size: 348, 572; Maryland Health services Cost Review Commission
7-day: 2%;
31-day: 6%; 180-day: 14%
Generalized estimating equations regression
Discharge AMA, Age, Gender, Insurance type, Weekend discharge, HF, Drug abuse, PTCA, Race, Residence, Stroke, Alcohol abuse, CABG
(Philbin & DiSalvo, 1999)
1995 R 1-year CHF
Sample size: 42731; Black and White race; New York State Department of Health Statewide Planning and Research Cooperative System database
21.3% Logistic regression
Black race, Medicaid/Medicare insurance, Home helthcare services, Comorbidities, Use of telemetry monitoring Negative factors: Rural hospital, Discharge to SNF, Echocardiogram, Cardiac catheterization
(Tsuchihashi et al., 2001)
Jan 1997 Dec 1997
R 1-year CHF-related
CHF
Sample size: 230; 5 institutions in Fukuoka, Japan
35.0% Multivariate logistic regression
Prior CHF admission, LOS, Hypertension, No occupation, Professional support, Poor follow-up visits
(van Walraven & Bell, 2002)
Mar 1999 Mar 2000
R
30-days unplanned
All types
Sample size: 2,403,181; Ontario Discharge Abstract Database
5.4% Proportional Hazards Modeling
Discharge on Friday
(van Walraven et al., 2010)
Oct 2002 Jul 2006
P 30-day unplanned
All types
Sample size: 4,812; 11 Hospitals in Ontario
8% (Read
mission&
mortality rate)
Multivariable logistic regression
Length of stay (L), Acuity of the admission (A), Comorbidity of the patient (C), Emergency
Appendix B (continued)
62
department use (E)
(Weiss et al., 2010)
- R
30-days unplanned
Medical-surgical patients
Sample size: 162 nurse-patient pairs; 4 Midwestern hospitals, Age>18
- Logistic regression
Readiness for Hospital Discharge Scale-Nurse version-(inverse effect), Age, Medical type admission
R: Retrospective, P: Prospective, SNF: Skilled Nursing Facility, VA: Veterans Administration,
LOS: Length of Stay, AMA: Against Medical Advice, LVSD: Left Ventricular Systolic Dysfunction,
CHF: Congestive Heart Failure, HF: Heart Failure, COPD: Chronic Obstructive Pulmonary
Disease, CAP: Community Acquired Pneumonia, CABG: Coronary Artery Bypass Graft, MI:
Myocardial Infarction, ACEI: Angiotensin-Converting Enzyme Inhibitor, CVD: Cardiovascular
Diseases, PTCA: Percutaneous Transluminal Coronary Angioplasty, HCAHPS: Hospital Care
Quality Information from the Consumer Perspective
Appendix B (continued)
63
Table 8: Focus of readmission prediction papers and common predictive factors, 1989
through 2010 P
aper
Age
Gender
Com
orb
idity
Length
of
sta
y
Specific
dia
gnosis
/
dis
ease
Prior
adm
issio
ns
Race
Clin
ical
In-h
ospital
pro
cess
Dis
charg
e
pro
cess
Fam
ily/
support
S
ocio
-
econom
ic
Genera
l health
Tre
atm
ent
Adm
issio
n
pro
cess
Insura
nce
Qualit
y o
f lif
e
(Allaudeen et al., 2010)
x x x x x x x x x x
(Almagro et al., 2006)
x x x x x x x x
(Beck et al., 2006)
x x x x x
(Boult et al., 1993)
x x x x x x x
(Capelastegui et al., 2009)
x x x x x x x x x
(French et al., 2008)
x x x x
(Glasgow et al., 2010)
x x x x x
(Greenblatt et al., 2010)
x x x x x x x x x x x x x x
(Halfon et al., 2002)
x x x x x x x
(Hannan et al., 2003)
x x x x x x x x x x
(Hasan et al., 2010)
x x x x x
(Heggestad & Lilleeng, 2003)
x
(Hendryx et al., 2003)
x x x x x x x x x x x
(Hofer & Hayward, 1995)
(Holloway & Thomas, 1989)
x x x x x x x x
(Jasti et al., 2008))
x x x x x x x x x x x x x
(Keenan et al., 2008)
x x x x x
Appendix B (continued)
64
(Krumholz et al., 2000)
x x x x x x x x x
(Lagoe et al., 2001)
x x x x x x x x x x
(Luthi et al., 2003))
x x x x x x x
(Luthi et al., 2004)
x x x x x x
(Medress & Fleshner, 2007)
x x x x x x
(Mudge et al., 2010)
x x x x x x x x x
(Neupane et al., 2010)
x x x x x x x x x
(Nasir et al., 2010)
x
(Onukwugha et al., 2011)
x x x x x x x x
(Philbin & DiSalvo, 1999)
x x x x x x x x
(Tsuchihashi et al., 2001)
x x x x x x x x x x x
(van Walraven & Bell, 2002)
x
(van Walraven et al., 2010)
x x x x x x x x x
Total 24 24 18 16 16 15 14 13 10 11 12 11 10 9 7 7 4
Appendix B (continued)
65
References
Allaudeen, N., Schnipper, J. L., Orav, E. J., Wachter, R. M., & Vidyarthi, A. R. (2011). Inability of
Providers to Predict Unplanned Readmissions. J Gen Intern Med. doi: 10.1007/s11606-
011-1663-3
Allaudeen, N., Vidyarthi, A., Maselli, J., & Auerbach, A. (2010). Redefining Readmission Risk
Factors for General Medicine Patients. Journal of Hospital Medicine, 000(000), 1-7.
Almagro, P., Barreiro, B., Ochoa de Echagüen, A., Quintana, S., Rodriguez Carballeira, M.,
Heredia, J., & Garau, J. (2006). Risk Factors for Hospital Readmission in Patients with
Chronic Obstructive Pulmonary Disease. Respiration, 73(3), 311-317.
Andersen, H. E., Schultz-Larsen, K., Kreiner, S., Forchhammer, B. H., Eriksen, K., & Brown, A.
(2000). Can Readmission After Stroke Be Prevented? : Results of a Randomized
Clinical Study: A Postdischarge Follow-Up Service for Stroke Survivors. Stroke, 31(5),
1038-1045.
Anthony, D., Chetty, V. K., Kartha, A., McKenna, K., Rizzo DePaoli, M., & Jack, B. (2005). Re-
engineering the Hospital Discharge: An Example of a Multifaceted Process Evaluation.
In K. Henriksen, J. B. Battles, E. S. Marks, & D. I. Lewin (Eds.), Advances in Patient
Safety: From Research to Implementation (Vol. 2, pp. 379-394). Rockville, MD: Agency
for Healthcare Research and Quality.
Ashton, C. M., Kuykendall, D. H., Johnson, M. L., Wray, N. P., & Wu, L. (1995). The Association
Between the Quality of Inpatient Care and Early Readmission. Ann Intern Med, 122(6),
415-421.
Ather, S., Chung, K. D., Gregory, P., & Demissie, K. (2004). The Association Between Hospital
Readmission and Insurance Provider Among Adults With Asthma. J Asthma, 41(7), 703-
707.
Appendix B (continued)
66
Bailey, E. J., Maclennan, C., Morris, P. S., Kruske, S. G., Brown, N., & Chang, A. B. (2009).
Risks of Severity and Readmission of Indigenous and Non-Indigenous Children
Hospitalised for Bronchiolitis. J Paediatr Child Health, 45(10), 593-597. doi: JPC1571
[pii]10.1111/j.1440-1754.2009.01571.x
Balaban, R. B., Weissman, J. S., Samuel, P. A., & Woolhandler, S. (2008). Redefining and
Redesigning Hospital Discharge to Enhance Patient Care: A Randomized Controlled
Study. J Gen Intern Med, 23(8), 1228-1233. doi: 10.1007/s11606-008-0618-9
Beck, C. E., Khambalia, A., Parkin, P. C., Raina, P., & Macarthur, C. (2006). Day of Discharge
and Hospital Readmission Rates Within 30 Days in Children: A Population-Based Study.
Paediatr Child Health, 11(7), 409-412.
Bell, C. M., Schnipper, J. L., Auerbach, A. D., Kaboli, P. J., Wetterneck, T. B., Gonzales, D.
V., . . . Meltzer, D. O. (2009). Association of Communication Between Hospital-based
Physicians and Primary Care Providers with Patient Outcomes. J Gen Intern Med, 24(3),
381-386. doi: 10.1007/s11606-008-0882-8
Benbassat, J., & Taragin, M. (2000). Hospital Readmissions as a Measure of Quality of Health
Care: Advantages and Limitations. Arch Intern Med, 160(8), 1074-1081.
Berman, K., Tandra, S., Forssell, K., Vuppalanch, R., Burton, J. R., Jr., Nguyen, J., . . .
Chalasani, N. (2011). Incidence and predictors of 30-day readmission among patients
hospitalized for advanced liver disease. Clinical Gastroenterology and Hepatology, 9(3),
254-259. doi: S1542-3565(10)01107-9 [pii] 10.1016/j.cgh.2010.10.035
Bhalla, R., & Kalkut, G. (2010). Could Medicare Readmission Policy Exacerbate Health Care
System Inequity? Ann Intern Med, 152(2), 114-117. doi: 0003-4819-152-2-201001190-
00185 [pii]10.1059/0003-4819-152-2-201001190-00185
Bickmore, T. W., Pfeifer, L. M., & Jack, B. W. (2009). Taking the Time to Care: Empowering
Low Health Literacy Hospital Patients with Virtual Nurse Agents. Paper presented at the
Conference on Human Factors in Computing Systems, Proceedings of the 27th
Appendix B (continued)
67
international conference on Human factors in computing systems, Boston, MA, USA.
https://dl-web.dropbox.com/get/Readmissions/papers/2009_Reengineering Hospital
Discharge A Protocol_Clancy.pdf?w=315d1abd
Bisognano, M., & Boutwell, A. (2009). Improving Transitions to Reduce Readmissions. Frontiers
of Health Services Management, 25(3), 3-10.
Bodenheimer, T. (2008). Coordinating Care -- A Perilous Journey through the Health Care
System. N Engl J Med, 358(10), 1064-1071. doi: 10.1056/NEJMhpr0706165
Boulding, W., Glickman, S. W., Manary, M. P., Schulman, K. A., & Staelin, R. (2011).
Relationship between patient satisfaction with inpatient care and hospital readmission
within 30 days. Am J Manag Care, 17(1), 41-48. doi: 12805 [pii]
Boult, C., Dowd, B., McCaffrey, D., Boult, L., Hernandez, R., & Krulewitch, H. (1993). Screening
elders for the risk of hospital admission. Journal of American Geriatric Society, 41(8),
811-817.
Boutwell, A. (2011). Reduced Readmissions: Reform's Low-Hanging Fruit. Healthcare Executive,
86-88.
Bradley, E. H., Curry, L., Horwitz, L. I., Sipsma, H., Thompson, J. W., Elma, M., . . . Krumholz, H.
M. (2012). Contemporary Evidence About Hospital Strategies for Reducing 30-Day
ReadmissionsA National Study. Journal of the American College of Cardiology, 60(7),
607-614. doi: 10.1016/j.jacc.2012.03.067
Butler, J., & Kalogeropoulos, A. (2012). Hospital Strategies to Reduce Heart Failure
ReadmissionsWhere Is the Evidence?. Journal of the American College of Cardiology,
60(7), 615-617. doi: 10.1016/j.jacc.2012.03.066
Capelastegui, A., Espana Yandiola, P. P., Quintana, J. M., Bilbao, A., Diez, R., Pascual, S., . . .
Egurrola, M. (2009). Predictors of Short-term Rehospitalization Following Discharge of
Patients Hospitalized with Community-Acquired Pneumonia. Chest, 136(4), 1079-1085.
doi: chest.08-2950 [pii]10.1378/chest.08-2950
Appendix B (continued)
68
Chen, L. M., Jha, A. K., Guterman, S., Ridgway, A. B., Orav, E. J., & Epstein, A. M. (2010).
Hospital Cost of Care Quality of Care, and Readmission Rates: Penny Wise and Pound
Foolish? Arch Intern Med, 170(4), 340-346. doi: 170/4/340
[pii]10.1001/archinternmed.2009.511
Clancy, C. M. (2009). Reengineering Hospital Discharge: A Protocol to Improve Patient Safety,
Reduce Costs, and Boost Patient Satisfaction. Am J Med Qual, 24(4), 344-346. doi:
1062860609338131 [pii]10.1177/1062860609338131
Clarke, A. (1990). Are Readmissions Avoidable? BMJ, 301(6761), 1136-1138.
Cline, C. M., Israelsson, B. Y., Willenheimer, R. B., Broms, K., & Erhardt, L. R. (1998). Cost
Effective Management Programme for Heart Failure Reduces Hospitalisation. Heart,
80(5), 442-446.
Coleman, E. A., Parry, C., Chalmers, S., & Min, S. J. (2006). The Care Transitions Intervention:
Results of a randomized control trial. Arch Intern Med, 166, 1822-1828.
Demir, E., Chaussalet, T. J., Xie, H., & Millard, P. H. (2008). Emergency Readmission Criterion:
A Technique for Determining the Emergency Readmission Time Window. IEEE Trans Inf
Technol Biomed, 12(5), 644-649. doi: 10.1109/TITB.2007.911311
Dormann, H., Neubert, A., Criegee-Rieck, M., Egger, T., Radespiel-Troger, M., Azaz-Livshits,
T., . . . Hahn, E. G. (2004). Readmissions and Adverse Drug Reactions in Internal
Medicine: the Economic Impact. J Intern Med, 255(6), 653-663. doi: 10.1111/j.1365-
2796.2004.01326.xJIM1326 [pii]
Elixhauser, A., Steiner, C., Harris, D. R., & Coffey, R. N. (1998). Comorbidity Measures for Use
with Administrative Data. Medical Care, 36(1), 8-27.
Epstein, C. D., Tsaras, G., Amoateng-Adjepong, Y., Greiner, P. A., & Manthous, C. (2009).
Does Race Affect Readmission to Hospital After Critical Illness? Heart & Lung, 38(1),
66-76. doi: S0147-9563(08)00015-0 [pii]10.1016/j.hrtlng.2008.01.001
Appendix B (continued)
69
Fleming, M. O., & Haney, T. T. (2013). Improving patient outcomes with better care transitions:
The role for home health. Cleveland Clinic Journal of Medicine, 80(e-Suppl 1), e-S2-e-
S6. doi: 10.3949/ccjm.80.e-s1.02
French, D. D., Bass, E., Bradham, D. D., Campbell, R. R., & Rubenstein, L. Z. (2008).
Rehospitalization After Hip Fracture: Predictors and Prognosis from a National Veterans
Study. J Am Geriatr Soc, 56(4), 705-710. doi: JGS1479 [pii]10.1111/j.1532-
5415.2007.01479.x
Friedman, B., & Basu, J. (2004). The Rate and Cost of Hospital Readmissions for Preventable
Conditions. Med Care Res Rev, 61(2), 225-240. doi: 10.1177/1077558704263799
Glasgow, J. M., Vaughn-Sarrazin, M., & Kaboli, P. J. (2010). Leaving Against Medical Advice
(AMA): Risk of 30-Day Mortality and Hospital Readmission. J Gen Intern Med, 25(9),
926-929. doi: 10.1007/s11606-010-1371-4
Goldfield, N. I., McCullough, E. C., Hughes, J. S., Tang, A. M., Eastman, B., Rawlins, L. K., &
Averill, R. F. (2008). Identifying Potentially Preventable Readmissions. Health Care
Financ Rev, 30(1), 75-91.
Gonzalez, A., Shih, T., Dimick, J. B., & Ghaferi, A. A. (2013). Does same-hospital readmission
rate correlate with all-hospital readmission rate? Journal of the American College of
Surgeons, 217(3), S105. doi: 10.1016/j.jamcollsurg.2013.07.240
Goodman, D. G., Fisher, E. S., Chang, C. H., Raymond, S. R., & Bronner, K. K. (2013). After
Hospitalization:A Dartmouth Atlas Report on Readmissions Among Medicare
Beneficiaries. In Robert Wood Johnson Foundation (Ed.), The Revolving Door: A Report
on U.S. Hospital Readmissions.
Grafft, C. A., McDonald, F. S., Ruud, K. L., Liesinger, J. T., Johnson, M. G., & Naessens, J. M.
(2010). Effect of Hospital Follow-up Appointment on Clinical Event Outcomes and
Mortality. Arch Intern Med, 170(11), 955-960. doi: 170/11/955
[pii]10.1001/archinternmed.2010.105
Appendix B (continued)
70
Greenblatt, D. Y., Weber, S. M., O'Connor, E. S., LoConte, N. K., Liou, J. I., & Smith, M. A.
(2010). Readmission After Colectomy for Cancer Predicts One-Year Mortality. Ann Surg,
251(4), 659-669. doi: 10.1097/SLA.0b013e3181d3d27c
Greenwald, J. L., Denham, C. R., & Jack, B. W. (2007). The Hospital Discharge: a Review of a
High Risk Care Transition With Highlights of a Reengineered Discharge Process.
Journal of Patient Safety, 3(2), 97-106.
Halfon, P., Eggli, Y., van Melle, G., Chevalier, J., Wasserfallen, J. B., & Burnand, B. (2002).
Measuring Potentially Avoidable Hospital Readmissions. J Clin Epidemiol, 55(6), 573-
587. doi: S0895435601005212 [pii]
Hannan, E. L., Racz, M. J., Walford, G., Ryan, T. J., Isom, O. W., Bennett, E., & Jones, R. H.
(2003). Predictors of Readmission for Complications of Coronary Artery Bypass Graft
Surgery. JAMA, 290(6), 773-780. doi: 10.1001/jama.290.6.773290/6/773 [pii]
Hansen, L. O., Williams, M. V., & Singer, S. J. (2011). Perceptions of hospital safety climate and
incidence of readmission. Health Serv Res, 46(2), 596-616. doi: 10.1111/j.1475-
6773.2010.01204.x
Harrison, P. L., Hara, P. A., Pope, J. E., Young, M. C., & Rula, E. Y. (2011). The impact of
postdischarge telephonic follow-up on hospital readmissions. Popul Health Manag, 14(1),
27-32. doi: 10.1089/pop.2009.0076
Hasan, O., Meltzer, D. O., Shaykevich, S. A., Bell, C. M., Kaboli, P. J., Auerbach, A. D., . . .
Schnipper, J. L. (2010). Hospital Readmission in General Medicine Patients: A
Prediction Model. J Gen Intern Med, 25(3), 211-219. doi: 10.1007/s11606-009-1196-1
HealthleadersMedia. (2010). 20 People Who Make Healthcare Better. from
http://www.healthleadersmedia.com/20people/
Heggestad, T., & Lilleeng, S. E. (2003). Measuring Readmissions: Focus on the Time Factor. Int
J Qual Health Care, 15(2), 147-154.
Appendix B (continued)
71
Hendryx, M. S., Russo, J. E., Stegner, B., Dyck, D. G., Ries, R. K., & Roy-Byrne, P. (2003).
Predicting Rehospitalization and Outpatient Services from Administration and Clinical
Databases. J Behav Health Serv Res, 30(3), 342-351.
Hernandez, A. F., Greiner, M. A., Fonarow, G. C., Hammill, B. G., Heidenreich, P. A., Yancy, C.
W., . . . Curtis, L. H. (2010). Relationship Between Early Physician Follow-up and 30-day
Readmission Among Medicare Beneficiaries Hospitalized for Heart Failure. JAMA,
303(17), 1716-1722. doi: 303/17/1716 [pii]10.1001/jama.2010.533
Hines, A. L., Barrett, M. L., Jiang, H. J., & Steiner, C. A. (2014). Conditions with the Largest
Number of Adult Hospital Readmissions by Payer, 2011;HCUP Statistical Brief #172. In
Agency for Healthcare Research and Quality (Ed.). Rockville, MD.
Hofer, T. P., & Hayward, R. A. (1995). Can Early Re-Admission Rates Accurately Detect Poor-
Quality Hospitals? Med Care, 33(3), 234-245.
Holloway, J. J., & Thomas, J. W. (1989). Factors Influencing Readmission Risk: Implications for
Quality Monitoring. Health Care Financ Rev, 11(2), 19-32.
Institute for Healthcare Improvement. (2003). Move Your Dot™: Measuring, Evaluating, and
Reducing Hospital Mortality Rates (Part 1). White paper. Institute for Healthcare
Improvement. Boston, MA. Retrieved from
http://www.ihi.org/IHI/Results/WhitePapers/MoveYourDotMeasuringEvaluatingandReduc
ingHospitalMortalityRates.htm
Institute for Healthcare Improvement. (2009a). Avoiding Readmissions: Optimizing the
Transition from Hospital to Home. From
http://www.ihi.org/IHI/Programs/ConferencesAndSeminars/HospitaltoHomeOptimizing
theTransitionJune09.htm
Institute for Healthcare Improvement. (2009b). Reducing Readmissions by Improving
Transitions in Care Collaborative. Collaboratives. from
Appendix B (continued)
72
http://www.ihi.org/IHI/Programs/Collaboratives/IHICollaborative
IReducingReadmissionsbyImprovingTransitionsinCare.htm?TabId=0
Institute for Healthcare Improvement. (2010a). Good Heart Failure Care Follows Patients Home.
5.
http://www.ihi.org/IHI/Topics/ChronicConditions/AllConditions/ImprovementStories/Good
HeartFailureCareFollowsPatientsHome.htm
Institute for Healthcare Improvement. (2010b). State Action on Avoidable Rehospitalizations
(STAAR). from
http://www.ihi.org/IHI/Programs/StrategicInitiatives/STateActiononAvoidableRehospitaliz
ationsSTAAR.htm
Jack, B. W., Chetty, V. K., Anthony, D., Greenwald, J. L., Sanchez, G. M., Johnson, A. E., . . .
Culpepper, L. (2009). A Reengineered Hospital Discharge Program to Decrease
Rehospitalization: A Randomized Trial. Ann Intern Med, 150(3), 178-187. doi: 150/3/178
[pii]
Jack, B. W., Greenwald, J. L., Forsythe, S. R., O'Donnell, J. K., Johnson, A., Schipelliti, L., . . .
Chetty, V. K. (2008). Developing the Tools to Administer a Comprehensive Hospital
Discharge Program: The ReEngineered Discharge (RED) Program. In Agency for
Healthcare Research and Quality (Ed.), (AHRQ Publication No. 08-0034-3 ed., Vol. 3, pp.
1-15).
Jacobs, B. (2011). Reducing heart failure hospital readmissions from skilled nursing facilities.
Prof Case Manag, 16(1), 18-24. doi: 10.1097/NCM.0b013e3181f3f68401269241-
201101000-00005 [pii]
Jasti, H., Mortensen, E. M., Obrosky, D. S., Kapoor, W. N., & Fine, M. J. (2008). Causes and
Risk Factors for Rehospitalization of Patients Hospitalized with Community-acquired
Pneumonia. Clin Infect Dis, 46(4), 550-556. doi: 10.1086/526526
Appendix B (continued)
73
Jencks, S. F., Williams, M. V., & Coleman, E. A. (2009). Rehospitalizations Among Patients in
the Medicare Fee-for-Service Program. N Engl J Med, 360(14), 1418-1428. doi:
360/14/1418 [pii]10.1056/NEJMsa0803563
Jha, A. K., Orav, E. J., & Epstein, A. M. (2009). Public Reporting of Discharge Planning and
Rates of Readmissions. N Engl J Med, 361(27), 2637-2645. doi: 361/27/2637
[pii]10.1056/NEJMsa0904859
Jiang, H. J., Russo, C. A., & Barrett, M. L. (2009). Nationwide frequence and costs of potentially
preventable hospitalizations, 2006. In Agency for Healthcare Research and Quality (Ed.).
Rockville, MD.
Jordan, C. J., Goldstein, R. Y., Michels, R. F., Hutzler, L., Slover, J. D., & Bosco, J. A. (2012).
Comprehensive program reduces hospital readmission rates after total joint arthroplasty.
Am J Orthop, 41(11), E147-151.
Joynt, K. E., & Jha, A. K. (2012). Thirty-Day Readmissions — Truth and Consequences. New
England Journal of Medicine, 366(15), 1366-1369. doi: doi:10.1056/NEJMp1201598
Joynt, K. E., Orav, E. J., & Jha, A. K. (2011). Thirty-day readmission rates for Medicare
beneficiaries by race and site of care. JAMA, 305(7), 675-681. doi: 305/7/675
[pii]10.1001/jama.2011.123
Jweinat, J. J. (2010). Hospital Readmissions Under the Spotlight. J Healthc Manag, 55(4), 252-
264.
Kanaan, S. B. (2009). Homeward Bound: Nine Patient-Centered Programs Cut Readmissions
(pp. 35): California HealthCare Foundation,.
Kartha, A., Anthony, D., Manasseh, C. S., Greenwald, J. L., Chetty, V. K., Burgess, J. F., . . .
Jack, B. W. (2007). Depression is a Risk Factor for Rehospitalization in Medical
Inpatients. Prim Care Companion J Clin Psychiatry, 9(4), 1-7.
Kasper, E. K., Gerstenblith, G., Hefter, G., Van Anden, E., Brinker, J. A., Thiemann, D. R., . . .
Gottlieb, S. H. (2002). A Randomized Trial of the Efficacy of Multidisciplinary Care in
Appendix B (continued)
74
Heart Failure Outpatients at High Risk of Hospital Readmission. J Am Coll Cardiol, 39(3),
471-480. doi: S0735109701017612 [pii]
Keenan, P., Normand, S., & Lin, Z. (2008). An Administrative Claims Measure Suitable for
Profiling Hospital Performance on the Basis of 30-Day All-Cause Readmission Rates
Among Patients With Heart Failure. Circ Cardiovasc Qual Outcomes, 2008(1), 29-37. doi:
10.1161/CIRCOUTCOMES.108.802686
Koelling, T. M., Johnson, M. L., Cody, R. J., & Aaronson, K. D. (2005). Discharge Education
Improves Clinical Outcomes in Patients With Chronic Heart Failure. Circulation, 111(2),
179-185. doi: 01.CIR.0000151811.53450.B8 [pii]10.1161/01.CIR.0000151811.53450.B8
Krumholz, H. M., Amatruda, J., Smith, G. L., Mattera, J. A., Roumanis, S. A., Radford, M. J., . . .
Vaccarino, V. (2002). Randomized Trial of an Education and Support Intervention to
Prevent Readmission of Patients With Heart Failure. J Am Coll Cardiol, 39(1), 83-89. doi:
S0735109701016990 [pii]
Krumholz, H. M., Chen, Y.-T., Wang, Y., Vaccarino, V., Radford, M. J., & Horwitz, R. I. (2000).
Predictors of Readmission Among Elderly Survivors of Admission with Heart Failure.
American Heart Journal, 139(1), 72-77.
Krumholz, H. M., Merrill, A., & Schone, E. (2009). Patterns of Hospital Performance in Acute
Myocardial Infarction and Heart Failure 30-Day Mortality and Readmission. Circ
Cardiovasc Qual Outcomes, 2009(2), 407-413. doi: DOI:
10.1161/CIRCOUTCOMES.109.883256
Lagoe, R. J., Noetscher, C. M., & Murphy, M. P. (2001). Hospital Readmission: Predicting the
Risk. J Nurs Care Qual, 15(4), 69-83.
Lappe, J. M., Muhlestein, J. B., Lappe, D. L., Badger, R. S., Bair, T. L., Brockman, R., . . .
Anderson, J. L. (2004). Improvements in 1-year Cardiovascular Clinical Outcomes
Associated with a Hospital-Based Discharge Medication Program. Ann Intern Med,
141(6), 446-453. doi: 141/6/446 [pii]
Appendix B (continued)
75
Lin, H. J., Chang, W. L., & Tseng, M. C. (2011). Readmission after stroke in a hospital-based
registry: risk, etiologies, and risk factors. Neurology, 76(5), 438-443. doi:
WNL.0b013e31820a0cd8 [pii]10.1212/WNL.0b013e31820a0cd8
Luthi, J. C., Burnand, B., McClellan, W. M., Pitts, S. R., & Flanders, W. D. (2004). Is
Readmission to Hospital an Indicator of Poor Process of Care for Patients With Heart
Failure? Qual Saf Health Care, 13(1), 46-51.
Luthi, J. C., Lund, M. J., Sampietro-Colom, L., Kleinbaum, D. G., Ballard, D. J., & McClellan, W.
M. (2003). Readmissions and the Quality of Care in Patients Hospitalized with Heart
Failure. Int J Qual Health Care, 15(5), 413-421.
Manning, S. (2011). Bridging the Gap Between Hospital and Home: A New Model of Care for
Reducing Readmission Rates in Chronic Heart Failure. J Cardiovasc Nurs, 00(0), 0. doi:
10.1097/JCN.0b013e318202b15c
Marcantonio, E. R., McKean, S., Goldfinger, M., Kleefield, S., Yurkofsky, M., & Brennan, T. A.
(1999). Factors Associated with Unplanned Hospital Readmission Among Patients 65
Years of Age and Older in a Medicare Managed Care Plan. The American Journal of
Medicine, 107(1), 13-17.
Maurer, P. P., & Ballmer, P. E. (2004). Hospital Readmissions--Are They Predictable and
Avoidable? Swiss Med Wkly, 134(41-42), 606-611. doi: smw-10706 [pii]2004/41/smw-
10706
Medress, Z., & Fleshner, P. R. (2007). Can We Predict Unplanned Hospital Readmission After
Colectomy for Ulcerative Colitis and Indeterminate Colitis? Am Surg, 73(10), 998-1001.
Miles, T. A., & Lowe, J. (1999). Are Unplanned Readmissions to Hospital Really Preventable? J
Qual Clin Pract, 19(4), 211-214.
Moore, C., Wisnivesky, J., Williams, S., & McGinn, T. (2003). Medical Errors Related to
Discontinuity of Care from an Inpatient to an Outpatient Setting. JGIM: Journal of
General Internal Medicine, 18(8), 646-651. doi: 10.1046/j.1525-1497.2003.20722.x
Appendix B (continued)
76
Motamedi, S. M., Posadas-Calleja, J., Straus, S., Bates, D. W., Lorenzetti, D. L., Baylis, B., . . .
Ghali, W. A. (2011). The efficacy of computer-enabled discharge communication
interventions: a systematic review. Qual Saf Health Care, 20, 403-415. doi:
bmjqs.2009.034587 [pii]10.1136/bmjqs.2009.034587
Mudge, A. M., Kasper, K., Clair, A., Redfern, H., Bell, J. J., Barras, M. A., . . . Pachana, N. A.
(2010). Recurrent Readmissions in Medical Patients: a Prospective Study. J Hosp Med,
2(000), 61-67. doi:10.1002/jhm.811
Mudrick, D. W., Shaffer, L., Lalonde, M., Ruhil, A., Lam, G., Hickerson, J., . . . Snow, R. (2013).
Cardiac Rehabilitation Participation Reduces 90-Day Hospital Readmissions after Acute
Myocardial Infarction or Percutaneous Coronary Intervention. Journal of the American
College of Cardiology, 61(10), E1418. doi: 10.1016/s0735-1097(13)61418-7
Nasir, K., Lin, Z., Bueno, H., Normand, S. T., Drye, E., Keenan, P. S., & Krumholz, H. M. (2010).
Is Same-Hospital Readmission Rate a Good Surrogate for All-Hospital Readmission
Rate? Medical Care, 48(5), 477-481.
Naylor, M. D., Brooten, D., Campbell, R., Jacobsen, B. S., Mezey, M. D., Pauly, M. V., &
Schwartz, J. S. (1999). Comprehensive Discharge Planning and Home Follow-up of
Hospitalized Elders: a Randomized Clinical Trial. JAMA, 281(7), 613-620. doi: joc80991
[pii]
Neupane, B., Walter, S. D., Krueger, P., Marrie, T., & Loeb, M. (2010). Predictors of Inhospital
Mortality and Re-hospitalization in Older Adults with Community-Acquired Pneumonia: A
Prospective Cohort Study. BMC Geriatr, 10(1), 22. doi: 1471-2318-10-22
[pii]10.1186/1471-2318-10-22
Olson, D. M., Bettger, J. P., Alexander, K. P., Kendrick, A. S., Irvine, J. R., Wing, L., . . .
Graffagnino, C. (2011). Transition of Care for Acute Stroke and Myocardial Infraction
Patients: From Hospitalization to Rehabilitation, Recovery, and Secondary Prevention.
(11(12)-E011). Rockville, MD: AHRQ.
Appendix B (continued)
77
Onukwugha, E., Mullins, C. D., Loh, F. E., Saunders, E., Shaya, F. T., & Weir, M. R. (2011).
Readmissions after unauthorized discharges in the cardiovascular setting. Med Care,
49(2), 215-224. doi: 10.1097/MLR.0b013e31820192a5
Ornstein, K., Smith, K. L., Foer, D. H., Lopez-Cantor, M. T., & Soriano, T. (2011). To the
hospital and back home again: a nurse practitioner-based transitional care program for
hospitalized homebound people. J Am Geriatr Soc, 59(3), 544-551. doi: 10.1111/j.1532-
5415.2010.03308.x
Osei-Anto, A., Joshi, M., Audet, A. M., Berman, A., & Jencks, S. F. (2010). Health Care Leader
Action Guide to Reduce Avoidable Readmissions. In Health Research & Educational
Trust, The Commonwealth Fund, & John A. Hartford Foundation (Eds.). Chicago, IL.
Ouslander, J. G., Diaz, S., Hain, D., & Tappen, R. (2011). Frequency and Diagnoses Associated
With 7- and 30-Day Readmission of Skilled Nursing Facility Patients to a Nonteaching
Community Hospital. J Am Med Dir Assoc, 12(3), 195-203. doi: S1525-8610(10)00078-2
[pii]10.1016/j.jamda.2010.02.015
Patient Safety Authority. (2005). Unanticipated Care After Discharge from Ambulatory Surgical
Facilities PA-PSRS Patient Safety Advisory (Vol. 2, pp. 1-32).
Philbin, E. F., & DiSalvo, T. G. (1999). Prediction of Hospital Readmission for Heart Failure:
Development of a Simple Risk Score Based on Administrative Data. J Am Coll Cardiol,
33(6), 1560-1566. doi: S0735-1097(99)00059-5 [pii]
Press, M. J., Silber, J. H., Rosen, A. K., Romano, P. S., Itani, K. M. F., Zhu, J., . . . Volpp, K. G.
(2010). The impact of resident duty hour reform on hospital readmission rates among
medicare beneficiaries. J Gen Intern Med, 24(6), 405-411. doi: 10.1007/s11606-010-
1539-y
Rayner, H. C., Temple, R. M., Marshall, T., & Clarke, D. (2002). A Comparison of Hospital
Readmission Rates Between Two General Physicians With Different Outpatient Review
Practices. BMC Health Serv Res, 2(1), 12.
Appendix B (continued)
78
Rich, M. W., Beckham, V., Wittenberg, C., Leven, C. L., Freedland, K. E., & Carney, R. M.
(1995). A Multidisciplinary Intervention to Prevent the Readmission of Elderly Patients
with Congestive Heart Failure. N Engl J Med, 333(18), 1190-1195. doi:
10.1056/NEJM199511023331806
Riegel, B., Carlson, B., Kopp, Z., LePetri, B., Glaser, D., & Unger, A. (2002). Effect of a
Standardized Nurse Case-Management Telephone Intervention on Resource Use in
Patients with Chronic Heart Failure. Arch Intern Med, 162(6), 705-712. doi: ioi10218 [pii]
Robert Wood Johnson Foundation. (2013). The Revolving Door: A Report on U.S. Hospital
Readmissions.
Ross, J. S., Chen, J., Lin, Z., Bueno, H., Curtis, J. P., Keenan, P. S., . . . Krumholz, H. M. (2010).
Recent National Trends in Readmission Rates After Heart Failure Hospitalization. Circ
Heart Fail, 3(1), 97-103. doi: 10.1161/circheartfailure.109.885210
Santamour, B. (2011). Reining in Avoidable Readmissions American Hospital Association (pp.
8).
Simmons, J. (2010). Reducing Readmissions: Are Quality Payments a Carrot or Stick?
HealthLeaders Media,, 2. https://dl-
web.dropbox.com/get/Readmissions/papers/2010_Reducing Readmissions Are Quality
Payments_Simmons.pdf?w=2d4e5880
Strunin, L., Stone, M., & Jack, B. (2007). Understanding Rehospitalization Risk: Can Hospital
Discharge be Modified to Reduce Recurrent Hospitalization? J Hosp Med, 2(5), 1-22. doi:
10.1002/jhm.206
Taylor, M. (2010). Shutting the Door on Readmissions. Hosp Health Netw, 84(1), 33-34, 36.
Thomas, J. W., & Holloway, J. J. (1991). Investigating Early Readmission as an Indicator for
Quality of Care Studies. Medical Care, 29(4), 377-394.
Trisolini, M., Aggarwal, J., Leung, M., Pope, G., & Kautter, J. (2008). Lessons Learned on
Improving Quality and Efficiency in Health Care. In The Commonwealth Fund, Centers
Appendix B (continued)
79
for Medicare and Medicaid Services, & Agency for Healthcare Research and Quality
(Eds.), The Medicare Physician Group Practice Demonstration (pp. 58): The
Commonwealth Fund, Centers for Medicare and Medicaid Services,.
Tsuchihashi, M., Tsutsui, H., Kodama, K., Kasagi, F., Setoguchi, S., Mohr, M., . . . Takeshita, A.
(2001). Medical and Socioenvironmental Predictors of Hospital Readmission in Patients
with Congestive Heart Failure. American Heart Journal, 142(4), E7-E7.
van Walraven, C., & Bell, C. M. (2002). Risk of Death or Readmission Among People
Discharged from Hospital on Fridays. CMAJ, 166(13), 1672-1673.
van Walraven, C., Dhalla, I. A., Bell, C., Etchells, E., Stiell, I. G., Zarnke, K., . . . Forster, A. J.
(2010). Derivation and Validation of an Index to Predict Early Death or Unplanned
Readmission after Discharge from Hospital to the Community. CMAJ. doi: cmaj.091117
[pii]10.1503/cmaj.091117
van Walraven, C., Seth, R., Austin, P. C., & Laupacis, A. (2002). Effect of Discharge Summary
Availability During Post-discharge Visits on Hospital Readmission. J Gen Intern Med,
17(3), 186-192. doi: jgi10741 [pii]
Vira, T., Colquhoun, M., & Etchells, E. (2006). Reconcilable Differences: Correcting Medication
Errors at Hospital Admission and Discharge. Qual Saf Health Care, 15(2), 122-126. doi:
15/2/122 [pii]10.1136/qshc.2005.015347
Weinberger, M., Oddone, E. Z., & Henderson, W. G. (1996). Does Increased Access to Primary
Care Reduce Hospital Readmissions? The New England Journal of Medicine, 334(22),
1441.
Weiss, M., Yakusheva, O., & Bobay, K. (2010). Nurse and Patient Perceptions of Discharge
Readiness in Relation to Postdischarge Utilization. Med Care, 48(5), 482-486. doi:
10.1097/MLR.0b013e3181d5feae
Weissman, J. S., Ayanian, J. Z., Chasan-Taber, S., Sherwood, M. J., Roth, C., & Epstein, A. M.
(1999). Hospital Readmissions and Quality of Care. Med Care, 37(5), 490-501.
Appendix B (continued)
80
Appendix B (continued)
81
Appendix C: Preventable Readmission Risk Factors for Patients with Chronic Conditions
Appendix C includes the article titled, "Preventable Readmission Risk Factors for Patients with
Chronic Conditions", published in the Journal for Healthcare Quality.
82
Preventable Readmission Risk Factors forPatients With Chronic ConditionsFlorentino Rico, Yazhuo Liu, Diego A. Martinez, Shuai Huang, José L. Zayas-Castro, Peter J. Fabri
IntroductionThe U.S. Federal Government is seeking toeliminate unnecessary care and to controlgrowing spending by Medicare thatreached $556 billion in 2012 (Rau, 2012).Readmission rates have been established ashospital performance measures with theobjective of promoting quality, patient-centeredness, and accountability (CMS,2013). Readmissions are a costly element ofMedicare spending. Almost one fifth of the11,855,702Medicare beneficiaries who hadbeen discharged from a hospital were re-admitted within 30 days, and 34% werehospitalized within 90 days of which only10% were likely to have been planned(Jencks et al., 2009). Moreover, the cost ofreadmissions is estimated at $26 billionannually for Medicare only, and $17 billionof it are potentially preventable (RobertWood Johnson Foundation, 2013).
A hospital readmission can be definedas an admission to a hospital within a finitetime frame after an original admission anddischarge. A readmission can occur ateither the same hospital or a differenthospital, and it can involve planned orunplanned surgical or medical treatments(Stone and Hoffman, 2010). In general,preventable readmissions can be dividedinto three broad categories: complicationsor infections arising directly from the initialhospital stay, poorly managed transitionsduring discharge, and readmissions due toa chronic condition (Center forHealthcareQuality and Payment Reform, 2011).
The largest volume of readmissions oc-curs among patients with chronic con-ditions (Stone and Hoffman, 2010).According to Stone and Hoffman (2010),a number of factors might be contributingto this relatively high readmission rate: poordischarge planning and follow-up, low care
instructions compliance, inadequate familysupport, disease complications, and medi-cal errors. Thus, this study assesses read-mission risk by chronic condition group toidentify and compare significant factorsassociated with readmission.
There is still much that is unknownabout which patient and hospital factorsresult in a higher probability of a hospitalreadmission. Hospital-based studies pro-vide opportunities to identify these pa-tients and improve the way hospital care isdelivered (Center for Healthcare Qualityand Payment Reform, 2011). Identifyingthe significant factors can help in thecreation and implementation of inter-ventions to target these specific conditionsand high-risk patient groups.
Keywordsrehospitalizationmachine learningrisk factorslogistic regressionproportional hazardmodel
Journal for Healthcare QualityVol. 00, No. 0, pp. 1–16© 2015 National Association forHealthcare Quality
Abstract: Evidence indicates that the largest volume of hos-pital readmissions occurs among patients with preexistingchronic conditions. Identifying these patients can improve theway hospital care is delivered and prioritize the allocation ofinterventions. In this retrospective study, we identify factorsassociated with readmission within 30 days based on claimsand administrative data of nine hospitals from 2005 to 2012.We present a data inclusion and exclusion criteria to identifypotentially preventable readmissions. Multivariate logisticregression models and a Cox proportional hazards extensionare used to estimate the readmission risk for 4 chronic con-ditions (congestive heart failure [CHF], chronic obstructivepulmonary disease [COPD], acute myocardial infarction, andtype 2 diabetes) and pneumonia, known to be related to highreadmission rates. Accumulated number of admissions anddischarge disposition were identified to be significant factorsacross most disease groups. Larger odds of readmission wereassociated with higher severity index for CHF and COPD pa-tients. Different chronic conditions are associated with differ-ent patient and case severity factors, suggesting that furtherstudies in readmission should consider studying conditionsseparately.
1Vol. 00 No. 0 Month 2015
Appendix C (continued)
83
Literature ReviewThere is no standard definition of read-mission in the literature. Kansagara andcolleagues (2011) conducted a systematicliterature review on risk prediction modelsfor hospital readmissions. From this review,differences in the definition of read-missions are identified: the readmissiontime window (from 15 days to 12 months),type of hospital visit (all-included, poten-tially preventable, planned, or unplanned),source of data collection (administrativedata, prospective clinical data collection, orreal-time data collection), population andsetting (age range, Medicare, Medicaid, 1or multiple hospital networks, and depart-ments within the hospital), and themedicalcondition under study. Although the defi-nition of readmission varies across studiesin the literature, most study analyses aredriven by policy and decisions at the gov-ernment level. The Centers for Medicareand Medicaid Services (CMS) annuallydefines and calculates 30-day readmissionrates based on claims and administrativedata for public reporting for acute myo-cardial infarction (AMI), heart failure(HF), and for pneumonia (CMS, 2013).
A number of studies measure read-mission rates for specific medical con-ditions. Congestive heart failure (CHF)(Hamner and Ellison, 2005; Keenan et al.,2008; Kosiborod et al., 2003; Rosati et al.,1991), AMI, chronic obstructive pul-monary disease (COPD), pneumonia(Lindenauer et al., 2010), and type 2 dia-betes are the most common diseasesstudied in readmissions models. However,other disease-specific readmission analy-ses include cancer (Greenblatt et al., 2010;Reddy et al., 2009) and sickle cell disease(Sobota et al., 2010; Frei-jones and Field,2009). Studying readmissions and patientsby disease group allows studies to usea more homogeneous cohort and im-plementation of interventions to reducereadmissions.
Logistic regression (LR) is the mostcommonly used classification techniquein readmission research (Allaudeenet al., 2011; Bahadori et al., 2009; Bermanet al., 2011; Callaly et al., 2010; Feudtner
et al., 2009; Lindenauer et al., 2011;Nantsupawat et al., 2012; Neupane et al.,2010; Whitlock et al., 2010). A major rea-son for the widespread use of LR is its easeto adjust for different sampling schemes.Cox proportional regression models havealso been implemented to assess the riskover time with the proportional hazardsassumption. Thismethod is able to identifystatistically significant factors related toreadmission and high-risk populationgroups (Capelastegui et al., 2009; Lauet al., 2001; Lipska et al., 2010), althoughthey are limited in their ability to establisheither cause and effect or the actualimportance of these factors. Studies useboth LR and Cox proportional regressionmodels to find significant factors affectingreadmission (Belfort et al., 2010; Khawajaet al., 2012; Strouse et al., 2008).Moreover,other studies (Alkalay et al., 2010; Bisgaardet al., 2011; Courtney et al., 2009) usedunivariate statistical analysis and hypothe-sis testing to identify significant differencesbetween patients that were readmittedversus those that were not readmitted. Theresults in these models differ in deter-mining which factors are significant. Thevariability and lack of consistency in thepublished relationships could be due toa large number of factors, many of whichrelate to statistical inference and cause–effect inference.
Readmission risk prediction continuesto be difficult and current readmissionpredicting models perform poorly.Among published articles, the highestpredicting ability, in terms of the areaunder the receiver operating characteris-tic, is 0.80 (Shulan et al., 2013). Limi-tations identified include the lack ofgeneralizability of the results since moststudies are done for a specific cohort ofpatients (Cline et al., 1998; Fontanella,2008; Koelling et al., 2005; Rich et al.,1995), and the limitations of administra-tive data that may reduce the ability toidentify predictors due to absence ofimportant clinical information (Curtiset al., 2009; Frei-jones and Field, 2009;Reddy et al., 2009; Tsuchihashi et al.,2001). To provide more generalizable re-sults, a representative sample size, and
2 Journal for Healthcare Quality
Appendix C (continued)
84
relevant data, both clinical and adminis-trative data are suggested (Kaben et al.,2008). However, it has been noted thatadding additional risk factors has addedcomplexity without improving the pre-dictive power ofmodels (Spiva et al., 2014).
There are still significant opportunitiesto advance the understanding of the cau-ses and important risk factors associatedwith readmissions. The identification ofhigh-risk patient groups could foster pre-ventive interventions (Lin et al., 2011;Reddy et al., 2009), an area where pre-dictive modeling could have a majorimpact. Although much work has beendone to determine the most appropriatedefinition of readmission, our reviewshows that there is still no consensus onwhich readmission definition is best. Ourdefinition of readmission is mostly basedon the CMS definition of readmission, andthe predictive models built presented inthis study are used to identify risk factors,but not as a risk adjustment model. Thus,we believe that it makes sense to identifyand predict in advance potentially pre-ventable readmissions.
PurposeThe aims of this study are to identifypotentially preventable readmissions based
on claims and administrative data, todetermine significant factors associatedwith the risk of being readmitted througha multivariate 30-day LR model and anextension of the Cox proportional hazardmodel with recurrent events, and to com-pare the effects of patient factors, caseseverity, and hospital factors associatedwith readmission across disease groups thatare related to readmissions and their costs.
Study Design and MethodsThe data used in this retrospective studyare extracted from the administrativeclaims data of nine hospitals geo-graphically localized within three adjacentcounties in Florida. The types of hospitalsin the study include general, teaching, andspecialized hospitals. The initial datasetincludes 594,751 patients accounting for1,093,177 patient discharges from January2005 through July 2012. The data wereprocessed in three phases:
Phase I: Exclusion CriteriaThe data were filtered based on the exclu-sion criteria in Table 1. This study excludedsingle events (admissions) or the entirepatient record in the database to classifythose readmissions that are avoidable and
Table 1. Excluded Single Admissions or Patient RecordsAdmissions Patients
The record of the admission (single event) was excluded if itwas due to:
The entire patient record was excluded ifhe/she was:
Continued care in the same hospital due to same-dayinternal hospital transfer (This was represented asa readmission in the same day in the database)
Discharged to hospice care
Newborn delivery Diagnosed with cancer: ICD-9 code“malignant neoplasm” and ongoingcancer treatment
Trauma Diagnosed with renal disease and ongoingtreatment
RehabilitationOutside transfer and discharge planning is performedElopement: leaving without medical advice and/ortreatment
Death and subsequent to death (i.e., organ donation)
3Vol. 00 No. 0 Month 2015
Appendix C (continued)
85
potentially unavoidable. The recordsexcluded are considered to be routine,planned, or unavoidable. After this pro-cess, the final dataset has 470,147 patientsand 763,289 hospitalizations with a 30.2%elimination rate.
Phase II: Study Cohort by Disease TypeThis study focuses on admissions forspecific chronic conditions or diseasesthat are known for high readmissionsrates. Using the International Classificationof Diseases, 9th Revision, Clinical Modifica-tion (ICD-9-CM), primary diagnosis codewas used to identify admissions for CHF(codes 428.*,402.01, 402.91, 404.01,404.03, 404.11, 404.13, 404.91, 404.93),COPD (codes 491.0, 491.1, 491.2, 491.20,491.21, 490, 492, 496), AMI (codes 410.*),type 2 diabetes (codes 250.*2), andpneumonia (codes 480–483, 485–486,510, 511.0, 511.1, 511.9 and a primarydiagnosis of a pneumonia-related symp-tom [codes 780.6, 780.6, 786.00, 786.05,786.06, 786.07, 786.2, 786.3, 786.4, 786.5,786.51, 786.52, 786.7] and a secondarydiagnosis of pneumonia, emphysema, orpleurisy) as index admissions for these 5illnesses.
Phase III: Planned/Unplanned ReadmissionsWe used the definition of planned/unplanned readmissions stated in theHospital-Wide All-Cause Unplanned Re-admission Measure final report for CMS(Horwitz et al., 2008). Planned read-missions were defined as those in whichone of a prespecified list of procedurestook place. This analysis considered onlyunplanned admissions within 30 days asthe outcome of interest in the predictivemodels. This time frame was used tofollow the CMS readmission definitionstandards to estimate high readmissionpenalties.
Study VariablesThe descriptive statistics for the dataand variables’ categories are shown inTable 2. After discussions with hospital
experts, we classified the variables for thisstudy in three categories: (1) “patientfactors”: age range, gender, marital sta-tus, race/ethnicity, and language; (2)“case severity factors”: severity of illness(from 1 =minor to 4 = extreme as defined by3M APR DRG; 3M Health InformationSystems, 2008), behavioral health co-morbidities (1 if present as a secondarydiagnosis, 0 otherwise), Charlson co-morbidity index (Charlson Co; calcu-lated based on the comorbid conditionsand severity; Charlson et al., 1994), andlength of stay (LOS) (days); (3) “hospitalfactors”: hospitalist (1 if present, 0 oth-erwise), payer class, discharge disposi-tion, admission type, and year (overseven years).
Analytical MethodsA LR model and a proportional hazardmodel were used to identify statisticallysignificant variables and assess their 30-dayunplanned readmission relative risk andthe readmission risk over time (hazardratio [HR] for recurrent events).
Logistic Regression and 30-Day ReadmissionRisk. We built a LR model to predictan unplanned readmission within 30days of discharge as a binary outputvariable (Y = 1, if readmitted within 30days, or 0 otherwise). The results areinterpreted using the quantity log p
12p(the “log odds”) to compare the relativerisks among the different class levels ofthe independent variables. Goodness-of-fit is evaluated using theHosmer–Lemshowstatistic and cross-validation. A Wald testis used to test the statistical significanceof each coefficient (b) in the modeland to create the 0.95 confidence inter-vals (CIs).
Proportional Hazard Model WithRecurrent Events. We applied a Coxproportional hazards extension to esti-mate effects of covariates which arereported as HRs. The motivation forusing proportional hazard model withrecurrent events is that 1 patient might
4 Journal for Healthcare Quality
Appendix C (continued)
86
Table 2. Descriptive Statistics for Study VariablesCHF COPD AMI Pneumonia Type 2 Diabetes
No. of patients 7,287 5,946 9,688 10,897 4,879No. of admissions 9,590 7,921 11,210 12,130 6,158*
Patient factorsAge
18–45 4.83 4.61 6.07 16.62 24.9045–55 9.76 14.97 16.88 14.64 22.7355–65 13.54 24.07 23.07 14.95 19.3165–75 17.02 25.08 19.86 15.34 15.4375–85 27.82 21.78 21.08 21.73 12.11851 14.93 6.19 7.79 9.32 3.73Null 12.10 3.31 5.25 7.40 1.78
GenderFemale 51.41 56.93 41.28 55.90 49.97Male 48.59 43.07 58.72 44.10 50.03
Marital statusDivorced/Separated 11.29 19.88 10.34 11.83 16.29Married 39.74 35.89 51.27 41.28 35.85Single 21.30 23.65 22.75 27.13 35.62Widowed 27.67 20.59 15.64 19.77 12.24
RaceBlack 15.21 8.98 6.17 11.78 28.28Hispanic 8.08 4.94 8.26 8.68 12.85White 75.31 84.86 82.40 77.71 56.94Other 1.40 1.21 3.17 1.83 1.93
LanguageEnglish 70.22 79.52 78.55 75.19 78.73Other 29.78 20.48 21.45 24.81 21.27
Case severity factorsSeverity of illness
1 = Minor 9.35 20.26 25.22 10.84 21.602 = Moderate 45.29 43.23 40.95 48.41 33.873 = Major 35.33 24.25 22.74 31.55 23.224 = Extreme 5.52 3.04 9.05 6.10 3.00Null 4.52 9.22 2.03 3.10 18.30
Behavioral health comorbidityNo 76.53 65.24 80.09 70.26 74.76Yes 23.47 34.76 19.91 29.74 25.24
Charlson comorbidity0 15.90 0.00 34.87 28.12 10.021 24.59 47.54 31.01 37.00 32.972 22.90 26.70 16.76 18.10 18.273 15.45 12.08 8.18 7.64 15.614 9.69 6.77 4.30 4.43 10.5651 11.47 6.91 4.88 4.71 12.59
Length of stay (days)Mean (min, max) 4.6 (0, 19) 3.8 (0, 56) 4.1 (0, 78) 5.2 (0, 15) 3.8 (0, 90)
(Continued)
5Vol. 00 No. 0 Month 2015
Appendix C (continued)
87
have multiple records of admission duringthe seven years of data. Also, data might beheterogeneous across individuals and eventdependent. Several survival models ofrecurrent events have been extended basedon semiparametric Cox proportional haz-ardmodels (Gjessing et al., 2010). Based onthe special features of the readmissionproblem, a conditional frailty model thatcombines a randomeffect with stratificationof events is recommended (Box-Steffen-smeier and De Boef, 2006). The model as-sumes that the contributions to the kth
admission are restricted to only those pa-tients who have previously experienced the
k 2 1th admission. The hazard of kth eventoccurring for the ith subject is
likðt ; ZikÞ5 l0kðt 2 tk2 1Þeb9ZikðxikÞ1vi ;
(1)
where Xik and Zik, respectively, denotethe observation time and covariate vectorfor the ith subject with respect to the kthevent, and b is the unknown regressionparameter vector. l0k is the baseline haz-ard rate and (t2 tk 2 1) represents the gaptime between kth and k 2 1th events. vi isthe vector of random effects (frailties)across events.
Table 2. (Continued )CHF COPD AMI Pneumonia Type 2 Diabetes
Hospital factorsHospitalist
Yes 25.85 29.10 27.27 28.62 32.64No 74.15 70.90 72.73 71.38 67.36
Payer classCommercial 9.49 10.96 26.52 18.39 19.96Medicaid 10.32 14.47 8.26 12.56 21.14Medicare 75.89 67.44 55.98 60.00 44.71Other 4.30 7.13 9.24 9.05 14.19
Discharge dispositionNonacute facility 43.02 29.57 26.43 33.79 32.49Routine/home 52.74 67.10 57.22 63.45 64.08Specialty hospital 2.89 1.00 14.99 0.88 0.99Other 1.35 2.34 1.36 1.88 2.44
Admission typeEmergency 83.67 82.07 77.25 87.36 69.29Routine 4.53 9.22 2.08 3.10 18.32Urgent 6.61 3.64 9.22 4.23 5.31Other 5.19 5.08 11.45 5.31 7.08
No of previous admissionsMean (min, max) 2.8 (1, 36) 3.3 (1, 45) 1.9 (1, 49) 2.4 (1, 59) 3.1 (1, 52)
YearH 19.26 13.26 14.89 16.07 14.31I 16.03 12.11 13.31 14.55 13.41J 13.23 12.02 15.58 13.72 13.30K 13.69 14.76 16.33 14.06 14.70L 12.40 17.04 14.99 15.00 15.54M 14.58 17.28 14.59 15.42 15.85N–O 10.81 13.53 10.31 11.19 12.89
*Includes 55 patients who are younger than 18 years.AMI, acute myocardial infarction; CHF, congestive heart failure; COPD, chronic obstructive pulmonary
disease.
6 Journal for Healthcare Quality
Appendix C (continued)
88
Institutional Review Board ApprovalThis project was formally exempted by theUniversity of South Florida InstitutionalReview Board because it does not meet thedefinition of human subjects research.
ResultsThe LR model and the conditional frailtyproportional hazard model were built inSAS (version 9.3) and R (version 3.0.2),respectively. In the LR modeling predict-ing the 30-day risk of readmission, statisti-cally significant variables are selectedusing a stepwise selection (entry = 0.10,stay = 0.10) removing insignificant variablefrom themodel before adding a significantvariable to the model in every step. For theproportional hazard model, variables areselected based on the level of statisticalsignificance (P # .10) as well.
The statistically significant factors(P # .05) in the prediction of readmissionvaried across disease groups and predictionmodels, especially for patient and caseseverity factors. A large amount of hospitalfactors were found to be statistically signif-icant (P# .05) inbothmodels andacross alldiseases: accumulated number of admis-sions, year, and discharge disposition. Thepresence of a hospitalist and the dischargeday of week were not found statisticallysignificant in any of the models. The list ofstatistically significant factors found in eachmodel across disease groups and the per-formance for the LR model, in terms of itsdiscriminatory power (c-statistic), is pre-sented in Table 3. The relative risks for thepredictors’ class levels are analyzed usingtheodds ratio (OR) from theLRmodel andthe HR from the proportional hazardmodel. The OR and HR estimates are ex-pressed as a ratio point estimate and the0.95 CI upper and lower limits in Table 4.
Hospital FactorsThehigher the accumulated times a patienthas been readmitted to the hospital (ORfrom 1.06 to 1.15), the more likely it is thatthis person will be readmitted within 30days. The OR and HR showed a consistent
decreasing trend in readmission risk overthe years in the data analyzed. Dischargedisposition to another acute hospital orspecialty hospital has the higher odds ofbeing readmitted among other dispositions(routine home, nonacute facility, or other).Payer class was identified as significant forCHF, COPD, pneumonia, and type 2 dia-betes. In most of the cases, patients withMedicaid and Medicare had the higherratio (OR) of readmission among the payerclassifications (commercial insurance). Thetype admission for the patient is consideredfor CHF, AMI, and type 2 diabetes; more-over, patients admitted as emergency havehigher odds of readmission.
Case Severity FactorsLength of stay was statistically significant inacross all disease groups, except for AMI.Themore days the patient has stayed in thehospitals, the higher the likelihood of beingreadmitted with 30 days and risk of read-mission over time. The proportional hazardmodel identified the Charlson comorbidityindex as a significant factor in patients withCHF, AMI, Pneumonia and Type 2 Diabe-tes; moreover, patients with an index of 3 orhigher have the highest odds of read-mission HR over time (OR are also higherin this range for pneumonia and type 2diabetes). Severity of illness index wasincluded in one or both models for CHF,COPD, and pneumonia, and the odds ofreadmission increases as severity index ishigher. Having a comorbidity related toa behavioral health condition was found forCHF patients, and the probability of read-mission for having this comorbidity is 1.18times higher than not having it.
Patient FactorsThe differences of significant factors dif-fered drastically across disease groups.The LR model found age to be significantonly in the type 2 diabetes cohort. How-ever, the proportional HR found it signif-icant in four of the five disease groups.Gender was only included in the pro-portional hazard model, with higher HRfor female patients.
7Vol. 00 No. 0 Month 2015
Appendix C (continued)
89
DiscussionThe objective of this study was to furtherunderstand the risk factors associated withunplanned readmissions within 30 days inprespecified disease cohorts. Using twopredictive modeling techniques, we wereable to identify and compare factors asso-ciated with the patient, hospital stay, anddisease case severity.
Both the LR model and the proportionalhazards model for 30-day readmission gen-
erate a different mix of significant risk fac-tors in all five diseases. Thus, we performedanalyses for readmission for specific diseasesto better understand specific factors ofa given disease. In most cases, factors wereconsistent across the specified diseases. Forexample, patients with commercial insur-ance always have lower risk of being read-mitted, and longer LOS is associated witha higher probability of readmission. Wefound common significant factors across
Table 3. Significant Factors in Prediction Models
CHF COPD AMI Pneumonia Type 2 Diabetes
30-DayRisk
HazardRatio
30-DayRisk
HazardRatio
30-DayRisk
HazardRatio
30-DayRisk
HazardRatio
30-DayRisk
HazardRatio
c = 0.63 c = 0.68 c = 0.74 c = 0.67 c = 0.73
Patient factorsAge x x x x xLanguage x x x x xMarital status x x x xRace x x xGender x
Case severityfactorsBehavioralhealth
x
Severity ofillness
x x x x x
Length of stay x x x x x x xCharlsoncomorbidity
x x x x x x
Hospital factorsHospitalist*
Discharge dayof week*
Admissiontype
x x x
Payer class x x x x x xNo. ofpreviousadmissions
x x x x x x x x x x
Year x x x x x x x x x xDischargedisposition
x x x x x x x x x x
*Variable was not found significant by eithermodel for the disease groups studied. It will not be included in theanalysis of results.
AMI, acute myocardial infarction; CHF, congestive heart failure; COPD, chronic obstructive pulmonarydisease.
8 Journal for Healthcare Quality
Appendix C (continued)
90
Table4.
Mod
elPa
rameter
RelativeRisks
CHF
COPD
Odd
sRatio
HazardRatio
Odd
sRatio
HazardRatio
Patie
ntfactors
Age 18
–45
11
45–55
0.94
(0.77–
1.15
)1.52
(1.18–
1.97
)55
–65
0.78
(0.64–
0.96
)1.6(1.25–
2.06
)65
–75
0.73
(0.59–
0.9)
1.46
(1.12–
1.91
)75
–85
0.78
(0.63–
0.96
)1.27
(0.97–
1.68
)85
10.81
(0.65–
1.01
)1.35
(0.98–
1.85
)Gen
der
Female
Male
Marita
lstatus
Divorced
1Married
0.84
(0.75–
0.95
)Sing
le0.93
(0.82–
1.05
)Widow
ed0.98
(0.86–
1.13
)Race Black
1Hispa
nic
0.86
(0.73–
1.02
)White
0.81
(0.73–
0.91
)Other
0.57
(0.38–
0.85
)Lan
guage
Eng
lish
11
1Other
1.17
(0.99–
1.38
)1.13
(1–1.27
)1.27
(1.01–
1.6)
Caseseverity
factors
Disease
severity
11
11
21.23
(0.99–
1.52
)1.17
(0.97–
1.41
)0.99
(0.88–
1.1)
31.32
(1.06–
1.66
)1.39
(1.13–
1.72
)1(0.88–
1.14
)4
1.33
(0.97–
1.85
)1.62
(1.09–
2.41
)0.94
(0.71–
1.24
)Beh
avioralh
ealth
comorbidity
01
11.18
(1.04–
1.34
)
(Continued)
9Vol. 00 No. 0 Month 2015
Appendix C (continued)
91
Table4.
(Contin
ued)
CHF
COPD
Odd
sRatio
HazardRatio
Odd
sRatio
HazardRatio
Cha
rlsonco
morbidity
01
11.14
(0.99–
1.3)
21.22
(1.06–
1.39
)3
1.3(1.12–
1.51
)4
1.34
(1.14–
1.59
)51
1.26
(1.06–
1.49
)Len
gthof
stay
(days)
1.02
(1–1.03
)1.04
(1.02–
1.06
)1.03
(1.02–
1.04
)Hospitalfactors
Payerclass
Com
mercial
11
1Med
icaid
1.36
(1.14–
1.62
)1.94
(1.45–
2.6)
1.56
(1.3–1.87
)Med
icare
1.23
(1.04–
1.46
)1.44
(1.11–
1.88
)1.38
(1.16–
1.64
)Other
0.87
(0.68–
1.11
)1.55
(1.09–
2.22
)1.48
(1.19–
1.84
)Accum
ulated
numbe
rof
admission
s1.15
(1.12–
1.17
)1.08
(1.07–
1.1)
1.15
(1.13–
1.17
)1.09
(1.08–
1.1)
Disch
arge
disposition
Non
acutefacility
11
11
Rou
tine/
home
0.83
(0.73–
0.93
)1.05
(0.96–
1.15
)0.9(0.77–
1.05
)1.04
(0.94–
1.16
)Sp
ecialty
hospita
l2.43
(1.85–
3.2)
1.74
(1.4–2.17
)2.13
(1.27–
3.58
)1.45
(0.98–
2.15
)Other
1.59
(1.04–
2.44
)1.27
(0.93–
1.72
)1.78
(1.21–
2.62
)1.58
(1.21–
2.06
)Adm
ission
type
Emerge
ncy
1Other
0.8(0.65–
0.99
)Rou
tine
0.83
(0.7–0.98
)Urgen
t0.87
(0.73–
1.04
)Ye
ar 11
11
12
0.88
(0.73–
1.06
)0.86
(0.76–
0.97
)0.96
(0.75–
1.23
)0.91
(0.78–
1.06
)3
0.77
(0.61–
0.97
)0.83
(0.71–
0.97
)0.88
(0.65–
1.2)
0.84
(0.72–
0.98
)4
0.84
(0.66–
1.05
)0.84
(0.71–
0.98
)0.85
(0.63–
1.15
)0.79
(0.68–
0.91
)5
0.72
(0.57–
0.92
)0.7(0.6–0.83
)0.78
(0.58–
1.04
)0.69
(0.6–0.81
)6
0.76
(0.6–0.96
)0.74
(0.63–
0.87
)0.72
(0.53–
0.97
)0.62
(0.53–
0.72
)7–
80.57
(0.44–
0.74
)0.43
(0.35–
0.52
)0.54
(0.39–
0.75
)0.3(0.25–
0.37
)
(Continued)
10 Journal for Healthcare Quality
Appendix C (continued)
92
Table4.
(Contin
ued)
AMI
Pne
umon
iaTyp
eII
Diabe
tes
Odd
sRatio
HazardRatio
Odd
sRatio
HazardRatio
Odd
sRatio
HazardRatio
Patie
ntfactors
Age 18
–45
11
145
–55
1.07
(0.89–
1.27
)1.8(0.55–
5.87
)1.01
(0.84–
1.21
)55
–65
1.03
(0.86–
1.23
)1.03
(0.31–
3.4)
0.68
(0.55–
0.84
)65
–75
0.84
(0.68–
1.03
)1.52
(0.46–
5.05
)0.67
(0.51–
0.88
)75
–85
0.79
(0.65–
0.97
)1.8(0.54–
6)0.73
(0.55–
0.97
)85
10.83
(0.66–
1.05
)2.11
(0.61–
7.37
)0.65
(0.43–
0.98
)Gen
der
Female
1Male
0.89
(0.79–
1.01
)Marita
lstatus
Divorced
11
1Married
1.13
(0.95–
1.36
)0.77
(0.64–
0.92
)0.82
(0.65–
1.03
)Sing
le0.92
(0.75–
1.12
)0.85
(0.7–1.03
)0.91
(0.72–
1.14
)Widow
ed1.12
(0.91–
1.39
)0.72
(0.59–
0.89
)0.62
(0.44–
0.87
)Race Black
11
Hispa
nic
0.79
(0.6–1.04
)0.8(0.64–
1.01
)White
1.03
(0.85–
1.24
)0.61
(0.34–
1.08
)Other
0.85
(0.51–
1.39
)0.95
(0.81–
1.1)
Lan
guage
Eng
lish
11
Other
1.19
(1.02–
1.4)
1.13
(0.95–
1.34
)Caseseverity
factors
Disease
severity
11
12
1.09
(0.86–
1.39
)1.2(0.99–
1.45
)3
1.32
(1.03–
1.7)
1.36
(1.11–
1.65
)4
1.55
(1.12–
2.16
)1.35
(1.04–
1.77
)Beh
avioralh
ealth
comorbidity
0 1
(Continued)
11Vol. 00 No. 0 Month 2015
Appendix C (continued)
93
Table4.
(Contin
ued)
AMI
Pneu
mon
iaTyp
eII
Diabe
tes
Odd
sRatio
HazardRatio
Odd
sRatio
HazardRatio
Odd
sRatio
HazardRatio
Cha
rlsonco
morbidity
01
11
11
11.03
(0.9–1.19
)1.16
(0.98–
1.36
)1.26
(1.1–1.44
)0.9(0.62–
1.3)
0.95
(0.73–
1.24
)2
1.01
(0.85–
1.19
)1.27
(1.05–
1.53
)1.37
(1.18–
1.6)
1.73
(1.19–
2.5)
1.58
(1.2–2.07
)3
1.13
(0.9–1.42
)1.4(1.11–
1.77
)1.47
(1.22–
1.78
)2.01
(1.38–
2.91
)1.74
(1.32–
2.29
)4
1.35
(1.03–
1.78
)1.57
(1.2–2.06
)1.5(1.2–1.89
)1.9(1.28–
2.83
)1.96
(1.46–
2.63
)51
1.03
(0.77–
1.39
)1.55
(1.19–
2.02
)1.56
(1.25–
1.94
)1.87
(1.25–
2.78
)1.67
(1.23–
2.26
)Len
gthof
stay
(days)
1.02
(1–1.03
)1.01
(1.01–
1.02
)1.03
(1.01–
1.04
)1.03
(1.02–
1.04
)Hospitalfactors
Payerclass
Com
mercial
11
1Med
icaid
1.6(1.26–
2.02
)1.73
(1.44–
2.08
)1.51
(1.23–
1.85
)Med
icare
1.47
(1.21–
1.78
)1.79
(1.5–2.14
)1.31
(1.06–
1.63
)Other
1.02
(0.76–
1.37
)1.02
(0.81–
1.28
)1.07
(0.85–
1.35
)Accum
ulated
numbe
rof
admission
s1.12
(1.09–
1.15
)1.14
(1.09–
1.18
)1.09
(1.07 –
1.11
)1.06
(1.05–
1.08
)1.11
(1.09–
1.12
)
Disch
arge
disposition
Non
acutefacility
11
11
11
Rou
tine/
home
0.6(0.52–
0.69
)0.74
(0.65–
0.85
)0.72
(0.62–
0.82
)0.83
(0.74–
0.93
)0.88
(0.72–
1.07
)1.52
(1.09–
2.1)
Specialty
hospita
l6.74
(5.82–
7.81
)41
.1(33.99
–49
.7)
3.26
(2.14–
4.97
)2.9(1.98–
4.26
)3.95
(2.21–
7.04
)0.92
(0.8–1.07
)Other
1.1(0.71–
1.72
)1.36
(0.86–
2.16
)1.62
(1.12–
2.35
)1.55
(1.13–
2.11
)2.15
(1.41–
3.29
)3.35
(1.92–
5.85
)Adm
ission
type
Emerge
ncy
11
Other
1.1(0.78–
1.55
)0.8(0.62–
1.04
)Rou
tine
0.73
(0.58–
0.9)
0.9(0.63–
1.27
)Urgen
t0.84
(0.69–
1.01
)0.73
(0.52–
1.03
)Ye
ar 11
11
11
12
0.85
(0.7–1.04
)0.86
(0.7–1.06
)0.96
(0.78–
1.17
)1.01
(0.87–
1.18
)0.73
(0.55–
0.98
)0.68
(0.55–
0.84
)3
1(0.81–
1.24
)0.9(0.71–
1.13
)0.89
(0.72–
1.1)
0.89
(0.76–
1.04
)0.65
(0.48–
0.87
)0.83
(0.67–
1.02
)4
0.85
(0.69–
1.06
)0.8(0.64–
1.01
)0.91
(0.74–
1.12
)0.95
(0.81–
1.11
)0.63
(0.47–
0.84
)0.71
(0.57 –
0.87
)5
0.91
(0.73–
1.14
)0.76
(0.6–0.96
)0.76
(0.61–
0.93
)0.77
(0.65–
0.91
)0.59
(0.44–
0.79
)0.59
(0.48–
0.73
)6
0.74
(0.59–
0.93
)0.66
(0.51–
0.84
)0.76
(0.62–
0.94
)0.74
(0.62–
0.87
)0.51
(0.38–
0.69
)0.52
(0.41–
0.65
)7–
80.73
(0.57–
0.94
)0.56
(0.43–
0.75
)0.7(0.55–
0.88
)0.56
(0.45–
0.68
)0.48
(0.35–
0.66
)0.39
(0.3–0.51
)
Ratio
values
areex
pressedas
pointe
stim
ate(0.95co
nfide
nceinterval).
AMI,acutemyocardialinfarction;
CHF,
cong
estivehe
artfailure;C
OPD
,chron
icob
structivepu
lmon
arydisease.
12 Journal for Healthcare Quality
Appendix C (continued)
94
diseases: discharge disposition, Charlson co-morbidity index, and number of previousadmissions.
Interesting patterns are found forsome factors. For instance, as LOS in-creases, risk of readmission increases. Fora large number of potential factors (i.e.,case severity), LOS can be a surrogatemeasure. In the scope of this study, wecannot explain this behavior, and moreclinical information is needed to under-stand potential causation. People speak-ing languages other than English havehigher risk of readmission. In the litera-ture, it has been found that dischargeinstructions are important in the reduc-tion of readmissions, and one canhypothesize that patients who do notspeak English need better means ofcommunication for their discharge in-structions. In the case of the patients’ age,different risk patterns are observed acrossdiseases. For type 2 diabetes patients,younger to middle-aged patients havehigher readmission risk than elderly pa-tients. However, COPD patients betweenthe ages of 45 and 65 years have higherrisk than others.
Most of the significant variables foundare reasonable. However, some resultsneed further investigation. For example,for hospital factors, is payer class differ-ence due to the socioeconomic status orthe hospital systems? Commercial insur-ance holders have a lower chance ofbeing readmitted compared with allother payer classes. Moreover, anotherstudy also found that commercial insur-ance holders to have lower odds of read-missions compared with Medicare andMedicaid (Kruse et al., 2013), and thismight be due to common characteristicsthat a patient in this group share (e.g.,age, healthy enough to be employed, andincome). Payer class can be an estimatorof the socioeconomic situation of thepatients admitted. We also find that olderpatients have a lower chance to be read-mitted in the case of CHF. One study(Kosiborod et al., 2003) shows that the useof transfusions or other treatments forpatients with anemia aged 65 years orolder with HF could be the reason for
lower readmission rate. However, ourstudy lacks information of treatmentduring the stay.
LimitationsOur study provides important insights intothe hospital readmission problem basedon a network of hospitals located in Flor-ida over 7 years of data and patients olderthan 18 years. However, there are severallimitations in our study. First, our datasetcomes from the administrative data col-lected that does not contain completeclinical information for the admission.These hospitals are located in the sameextendedmetropolitan area, whichmeansthat the study population cannot be gen-eralized to other areas in the country. Theunavailability of clinical records andmedical tests limits our ability to evaluateother variables that may be more closelyrelated to how the patient was treatedduring a hospital stay. We believe that lackof patient transfer and discharge infor-mation also hinders tracking patients’ vis-its to other facilities outside the network.Finally, model performance was modest interms of the c-statistic achieved by themodels (c-statistics between0.63 and 0.74),but this performance is comparable withcurrent predictive models in the literature(Kansagara et al., 2011; Kruse et al., 2013).
Directions for Future ResearchIn future studies, predictive modelsshould explore the addition of otherclinical factors associated with the patientvisit to the hospital. This might enhancethe identification of risk factors beyondthe administrative claims data. To improveaccuracy and discriminatory power ofpredictive models, other machine learn-ing tools can be used to exploit more datacomplexity (i.e., decision trees, randomforest, and support vector machine). Inthe practice, this study suggests that hos-pital further evaluates potential inter-ventions for specific patient population athigher risk of readmission. However, in-terventions are already being designed toaddress specific needs such as patient
13Vol. 00 No. 0 Month 2015
Appendix C (continued)
95
education and discharge protocols(Koelling et al., 2005; Manning, 2011;Younis et al., 2012), analysis of racial dis-parities to reduce readmission rates fora specific population (Joynt et al., 2011),and the impact of specific medical inter-vention pertinent to a given disease toreduce mortality and readmission rates(Curtis et al., 2009). Finally, to capturepatient characteristics more precisely,competing risk models for the inter-actions, one, two, ormore diseases can alsobe studied, since a large number of pa-tients with disease combination could be atrisk for all potential diseases.
AcknowledgmentsThe Authors would like to thank theanonymous reviewers for their valuablefeedback and comments. There are nofinancial relationships with any organ-izations that might have an interest in thesubmitted work in the previous 3 years,and no other relationships or activitiesthat could appear to have influenced thesubmitted work.
References3M Health Information Systems. The 3M all
patient refined DRG (APR DRG) classifica-tion system. 2008. Available at: www.3Mhis.com. Accessed March 3, 2014.
Alkalay, A.L., Bresee, C.J., & Simmons, C.F.Decreased neonatal jaundice readmissionrate after implementing hyperbilirubinemiaguidelines and universal screening for bili-rubin. Clin Pediatr (Phila) 2010;49:830–833.
Allaudeen, N., Vidyarthi, A., Maselli, J., &Auerbach, A. Redefining readmission riskfactors for general medicine patients. J HospMed 2011;6:54–60.
Bahadori, K., FitzGerald, J.M., & Levy, R.D.,et al. Risk factors and outcomes associatedwith chronic obstructive pulmonary diseaseexacerbations requiring hospitalization.Can Respir J 2009;16:e43–e49.
Belfort, M., Clark, S.L., & Saade, G.R., et al.Hospital readmission after delivery: evi-dence for an increased incidence of non-urogenital infection in the immediatepostpartum period. Am J Obstet Gynecol 2010;202:35.e1–35.e7.
Berman, K., Sweta, T., & Forsell, K., et al.Incidence and predictors of 30-day read-mission among patients hospitalized for
advanced liver disease. Clin GastroenterolHepatol 2011;9:254–259. Available at:www.sciencedirect.com/science/article/pii/S1542356510011079. Accessed February 7,2014.
Bisgaard,T.,Kehlet,H.,&Bay-Nielsen,M., et al.Anationwide study on readmission, morbidity,and mortality after umbilical and epigastrichernia repair. Hernia 2011;15:541–546.
Box-Steffensmeier, J.M., & De Boef, S.Repeated events survival models: the con-ditional frailty model. Stat Med 2006;25:3518–3533.
Callaly, T., Hyland, M., & Trauer, T., et al. Re-admission to an acute psychiatric unit within28days of discharge: identifying those at risk.Aust Health Rev 2010;34:282–285.
Capelastegui, A., España Yandiola, P.P., &Quintana, J.M., et al. Predictors of short-termrehospitalization following discharge of pa-tients hospitalized with community-acquiredpneumonia. Chest 2009;136:1079–1085.
Center for Healthcare Quality and PaymentReform. Reducing hospital readmissions.Healthc Exec 2011. Available at: www.chqpr.org/readmissions.html. Accessed February10, 2013.
Charlson,M., Szatrowski, T.P., &Peterson JaneyGold, J. Validation of a combined co-morbidity index. J Clin Epidemiol 1994;47:1245–1251.
Cline,C.M., Israelsson,B.Y.,&Willenheimer,R.B.,et al. Cost effective management pro-gramme for heart failure reduces hospital-isation. Heart 1998;80:442–446. Available at:www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1728835&tool=pmcentrez&rendertype=abstract. Accessed January 25, 2014.
CMS. Hospital quality Initiative: outcome meas-ures. A federal government website managedby the centers for Medicare & Medicaid serv-ices. 2013. Available at: www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/OutcomeMeasures.html. Accessed January3, 2014.
Courtney, M., Edwards, H., & Chang, A., et al.Fewer emergency readmissions and betterquality of life for older adults at risk of hos-pital readmission: a randomized controlledtrial to determine the effectiveness of a 24-week exercise and telephone follow-upprogram. J Am Geriatr Soc 2009;57:395–402.
Curtis, J.P., Schreiner, G., & Wang, Y., et al.All-cause readmission and repeat revas-cularization after percutaneous coronaryintervention in a cohort of Medicare pa-tients. J Am Coll Cardiol 2009;54:903–907.
Feudtner, C., Levin, J.E., & Srivastava, R., et al.How well can hospital readmission be pre-dicted in a cohort of hospitalized children?A retrospective, multicenter study. Pediatrics2009;123:286–293.
14 Journal for Healthcare Quality
Appendix C (continued)
96
Fontanella, C. The Influence of clinical, treat-ment, and healthcare system characteristicson psychiatric readmission of adolescents.Am J Orthopsychiatry 2008;78:187–198.
Frei-jones, M.J., & Field, J.J. Risk factors forhospital readmission within 30 days: a newquality measure for children with sickle celldisease. Pediatr Blood Cancer 2009;2008:481–485.
Gjessing, H.K., Røysland, K., Pena, E., &Aalen, O.O. Recurrent events and the ex-ploding cox model. Lifetime Data Anal 2010;16:525–546.
Greenblatt,D.Y.,Weber, S.M.,&O’Connor, E.S.,et al. Readmission after colectomy for cancerpredicts one-year mortality. Ann Surg 2010;251:659–669.
Hamner, J.B., & Ellison, K.J. Predictors of hos-pital readmission after discharge in patientswith congestive heart failure. Heart Lung2005;34:231–239.
Horwitz,L.,Lin,Z.,&Herrin, J., et al.ReadmissionMeasure Final Technical Report Submitted byYaleNewHavenHealthServicesCorporation/Center forPrepared forOutcomesResearch&Evaluation (YNHHSC/CORE). Available atwww.qualitynet.org/dcs/ContentServer?c=Page&pagename=QnetPublic%2FPage%2FQnetTier4&cid=1219069855841. 2008: 11–33 Accessed November 12, 2013.
Jencks, S.F., Williams, M.V., & Coleman, E.Rehospitalizations among patients in theMedicare fee-for-service program. N EnglJ Med 2009;360:1418–1428.
Joynt, K.E., Orav, E.J., & Jha, A.K. Thirty-dayreadmission rates forMedicare beneficiariesby race and site of care. JAMA 2011;305:675–681.
Kaben, A., Corrêa, F., & Reinhart, K., et al.Readmission to a surgical intensive careunit:incidence, outcome and risk factors. CritCare 2008;12:R123.
Kansagara, D., Englander, H., & Salanitro, A.,et al. Risk prediction models for hospitalreadmission a systematic review. JAMA 2011;306:1688–1698.
Keenan, P.S., Normand, S-.L.T., & Lin, Z., et al.An administrative claims measure suitablefor profiling hospital performance on thebasis of 30-day all-cause readmission ratesamong patients with heart failure. Circ Car-diovasc Qual Outcomes 2008;1:29–37.
Khawaja, F.J., Shah, N.D., & Lennon, R.J., et al.Factors associated with 30-day readmissionrates after percutaneous coronary inter-vention. Arch Intern Med 2012;172:112–117.
Koelling, T.M., Johnson, M.L., Cody, R.J., &Aaronson, K.D. Discharge education im-proves clinical outcomes in patients withchronic heart failure. Circulation 2005;111:179–185.
Kosiborod, M., Smith, G.L., & Radford, M.J.,et al. The prognostic importance of anemia
in patients with heart failure. Am JMed 2003;114:112–119.
Kruse, R.L., Hays, H.D., & Madsen, R.W., et al.Risk factors for all-cause hospital read-mission within 30 days of hospital discharge.J Clin Outcomes Manage. 2013;20:203–214.
Lau, A.C., Yam, L.Y., & Poon, E. Hospital re-admission inpatientswith acuteexacerbationof chronic obstructive pulmonary disease.Respir Med 2001;95:876–884.
Lin, H., Chang, W., & Tseng, M. Readmissionafter stroke in a hospital- based registry.Neurology 2011;76(5):438–43.
Lindenauer, P.K., Bernheim, S.M.,&Grady, J.N.,et al. The performance of US hospitals asreflected in risk-standardized 30-day mor-tality and readmission rates for Medicarebeneficiaries with pneumonia. J Hosp Med2010;5:E12–E18.
Lindenauer, P.K.,Normand, S-.L.T.,&Drye, E.E.,et al. Development, validation, and results ofa measure of 30-day readmission followinghospitalization for pneumonia. J Hosp Med2011;6:142–150.
Lipska, K.J., Wang, Y., & Kosiborod, M., et al.Discontinuation of antihyperglycemic ther-apy and clinical outcomes after acute myo-cardial infarction in older patients withdiabetes. Circ Cardiovasc Qual Outcomes 2010;3:236–242.
Manning, S. Bridging the gap between hospitaland home: a newmodel of care for reducingreadmission rates in chronic heart failure.J Cardiovasc Nurs 2011;26:1–9.
Nantsupawat, T., Limsuwat, C., & Nugent, K.Factors affecting chronic obstructive pul-monary disease early rehospitalization.Chron Respir Dis 2012;9:93–98.
Neupane, B.,Walter, S.D., &Krueger, P., et al.Predictors of inhospital mortality andre-hospitalization in older adults withcommunity-acquired pneumonia: a pro-spective cohort study. BMC Geriatr 2010;10:22.
Rau, J. Hospitals face pressure to avert read-missions.NewYorkTimes. 2012;pD1. Availableat: www.nytimes.com/2012/11/27/health/hospitals-face-pressure-from-medicare-to-avert-readmissions.html?_r=1&. AccessedFebruary 10, 2014.
Reddy, D.M., Townsend, C.M., & Kuo, Y-.F.,et al. Readmission after pancreatectomy forpancreatic cancer in Medicare patients.J Gastrointest Surg 2009;13:1963–1974; dis-cussion 1974–1975.
Rich, M.W., Beckham, V., & Wittenberg, C.,et al. A multidisciplinary intervention toprevent the readmission of elderly patientswith congestive heart failure. N Engl J Med1995;333:1190–1195.
Robert Wood Johnson Foundation. TheRevolving Door: A Report on U. S. Hospital Re-admissions. Lebanon, NH: The Dartmouth
15Vol. 00 No. 0 Month 2015
Appendix C (continued)
97
Institute for Health Policy & Clinical Prac-tice; 2013.
Rosati, R.J., Huang, L., Navaie-Waliser, M., &Feldman, P.H. Risk factors for repeated hos-pitalizations among home healthcare recipi-ents. J Healthc Qual 1991;25:4–10; quiz 10–1.Availableat:www.ncbi.nlm.nih.gov/pubmed/12659074. Accessed February 12, 2014.
Shulan, M., Gao, K., & Moore, C.D. Predicting30-day all-cause hospital readmissions.Health Care Manag Sci 2013;16:167–175.
Sobota, A., Graham, D., Heeney, M.M., &Neufeld, E.J. Corticosteroids for acute chestsyndrome in childrenwith sickle cell disease:variation in use and association with lengthof stay and readmission. Am J Hematol 2010;85:24–28.
Spiva, L., Hand,M., VanBrackle, L., &McVay, F.(2014), Validation of a Predictive Model toIdentify Patients at High Risk for HospitalReadmission. J for Healthcare Qual. doi: 10.1111/jhq.12070. Accessed March 05, 2013.
Stone, J., & Hoffman, G.J. Medicare HospitalReadmissions: Issues, Policy Options and PPACA.2010. Available at: www.ncsl.org/documents/health/Medicare_Hospital_Readmissions_and_PPACA.pdf. Accessed December 19,2013.
Strouse, J.J., Takemoto, C.M., & Keefer, J.R.,et al. Corticosteroids and increased risk ofreadmission after acute chest syndrome inchildren with sickle cell disease. Pediatr BloodCancer 2008;50:1006–1012.
Tsuchihashi, M., Tsutsui, H., & Kodama, K.,et al. Medical and socioenvironmental pre-dictors of hospital readmission in patientswith congestive heart failure. Am Heart J2001;142:E7.
Whitlock, T.L., Repas, K., & Tignor, A., et al.Early readmission in acute pancreatitis:incidence and risk factors. Am J Gastroenterol2010;105:2492–2497.
Younis, J., Salerno, G., & Chaudhary, A., et al.Reduction in hospital reattendance due toimproved preoperative patient educationfollowing hemorrhoidectomy. J Healthc Qual2012;35:24–29.
Author’s BiographyFlorentino Rico, MSEM, MSIE, is a doctoralcandidate in the Department of Industrial andManagement Systems Engineering at the Uni-versity of South Florida (USF), Tampa, FL. Hisprimary role at USF is data analytics. Other areasof research interests include quality improvement,biostatistics, and decision support systems.
Yazhuo Liu, MIE, is a doctoral candidate in theDepartment of Industrial and ManagementSystems Engineering at the University of SouthFlorida (USF), Tampa, FL. Her role at USF isconducting healthcare related research and as-sisting courses.
Diego A. Martinez, MIE, is a doctoral candidatein the Department of Industrial and Manage-ment Systems Engineering at the University ofSouth Florida (USF), Tampa, FL. He conductsresearch in healthcare systems and the role of newhealth information technologies in improvingcare coordination.
Shuai Huang, PhD, is an Assistant Professor inthe Department of Industrial and SystemsEngineering at the University of Washington.His research interests are statistical learning anddata mining with applications in healthcareand manufacturing.
José L. Zayas-Castro, PhD, is Professor ofIndustrial and Management Systems Engi-neering at the University of South Florida (USF),Tampa, FL. As part of his responsibilities at theUSF, he leads an interdisciplinary research teamthat conducts research, and develops curricula,in health systems engineering and improving thedelivery of care to patients.
Peter J. Fabri, MD, PhD, FACS, is Professor ofSurgery and Professor of Industrial Engineering atthe University of South Florida. He has heldnumerous positions of academic leadership, buthas developed the past 10 years toward developinga “hybrid” field of health systems engineering,teaching systems engineering and statistics tomedical students and residents as well as healthdelivery to engineering students. He continues toteach basic medical school classes in the college ofmedicine and basic engineering classes in the col-lege of engineering.
For more information on this article, contactFlorentino Rico at [email protected].
Supported by the Regenstrief Foundationthrough the Regenstrief Center for HealthcareEngineering at Purdue University.
The authors declare no conflict of interest.
16 Journal for Healthcare Quality
Appendix C (continued)
98
Appendix D: A User Needs Assessment to Inform Health Information Exchange Design and
Implementation
Appendix D presents the article titled, "A User Needs Assessment to Inform Health Informa-
tion Exchange Design and Implementation", published in BMC Medical Informatics and Decision
Making.
99
RESEARCH ARTICLE Open Access
A user needs assessment to inform healthinformation exchange design andimplementationAlexandra T. Strauss1*, Diego A. Martinez2, Andres Garcia-Arce3, Stephanie Taylor4, Candice Mateja1,Peter J. Fabri5 and Jose L. Zayas-Castro3
Abstract
Background: Important barriers for widespread use of health information exchange (HIE) are usability and interfaceissues. However, most HIEs are implemented without performing a needs assessment with the end users, healthcareproviders. We performed a user needs assessment for the process of obtaining clinical information from other healthcare organizations about a hospitalized patient and identified the types of information most valued for medicaldecision-making.
Methods: Quantitative and qualitative analysis were used to evaluate the process to obtain and use outside clinicalinformation (OI) using semi-structured interviews (16 internists), direct observation (750 h), and operational data fromthe electronic medical records (30,461 hospitalizations) of an internal medicine department in a public, teachinghospital in Tampa, Florida.
Results: 13.7 % of hospitalizations generate at least one request for OI. On average, the process comprised 13 steps, 6decisions points, and 4 different participants. Physicians estimate that the average time to receive OI is 18 h. Physiciansperceived that OI received is not useful 33–66 % of the time because information received is irrelevant or not timely.Technical barriers to OI use included poor accessibility and ineffective information visualization. Common problemswith the process were receiving extraneous notes and the need to re-request the information. Drivers for OI use wereto trend lab or imaging abnormalities, understand medical history of critically ill or hospital-to-hospital transferredpatients, and assess previous echocardiograms and bacterial cultures. About 85 % of the physicians believe HIE wouldhave a positive effect on improving healthcare delivery.
Conclusions: Although hospitalists are challenged by a complex process to obtain OI, they recognize the value ofspecific information for enhancing medical decision-making. HIE systems are likely to have increased utilization andeffectiveness if specific patient-level clinical information is delivered at the right time to the right users.
Keywords: Health information technology, Health information exchange, Medical decision making, Hospital medicine,Medical record linkage, Computer communication networks, Continuity of patient care, Care coordination
* Correspondence: [email protected] of Internal Medicine, College of Medicine, University of SouthFlorida, Tampa, FL, USAFull list of author information is available at the end of the article
© 2015 Strauss et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 DOI 10.1186/s12911-015-0207-x
Appendix D (continued)
100
BackgroundIn the United States, 125 million people live with chronicconditions [1], and most of them receive care from mul-tiple health care providers [2]. For these patients, carecoordination is a necessity. Without care coordination, pa-tients may undergo avoidable procedures, receive contra-indicated treatments and incur unnecessary costs [3, 4].To foster care coordination, federal incentives have beenin place since 2009 to promote health information ex-change (HIE). HIE refers to the electronic movement ofhealth-related information among health care organiza-tions intended to facilitate a safer and more timely, effi-cient, effective and equitable delivery of care [5].Mixed evidence supports the ability of HIE to add
value to healthcare systems [6, 7], to detect patient safetyissues [8, 9] and to reduce healthcare delivery time andredundant testing [10–16]. For instance, Bailey and col-leagues found HIE reduces repeated imaging testing forback pain and headache admissions in emergency de-partments, but has a negligible effect on reducing costs[11, 12]. Frisse and colleagues found a negative associ-ation between HIE usage and hospital admissions, com-puterized tomography (CT) scans and laboratory tests[17]. Vest and Miller found better patient satisfactionlevels in those hospitals with HIE versus those withoutHIE [18]. Nguyen and colleagues reported a perceivedneed by healthcare providers and social service providersfor improved health information sharing [19]. In contrast,Overhage and colleagues found no significant effect ofHIE on reducing testing and number of admissions [13].Lang and colleagues found HIE use associated with dupli-cation of specialty consultations, as well as no significanteffect of HIE on reducing number of hospital admissions,length of stay and number of tests [20]. Finally, Hansagiand colleagues found HIE use improved physician satisfac-tion, but no significant effects were observed on the num-ber of emergency department, primary care and specialtyvisits [21]. A potential reason for the mixed evidence, assuggested by recently published systematic reviews [6, 7],is that widespread adoption of HIE across the UnitedStates is still limited. To date, only 14 % of solo practicesand non-primary care specialties, 30 % of hospitals, and10 % of ambulatory clinics are engaged in an HIE, withtypical rates of access from 2 to 10 % of patient visits[22–24]. Despite substantial progress in electronic med-ical record (EMR) adoption, physician engagement inHIE remains low in office settings [24].Research revealing how health professionals use HIE
systems to obtain information from other institutionscan help improve HIE functionality and subsequentlyimprove HIE utilization. Some have explored the user’sinteraction in ambulatory care situations [25]. Althoughearly studies concentrated on identifying drivers andbarriers for HIE adoption [18, 25–28], recent studies
have shed light on HIE use patterns. For example, it hasbeen found that physicians are more likely to access radi-ology reports than any other health professional [29, 30],and that all users engage with HIE systems in a minimalfashion by accessing only the select patient screen and therecent encounters summary screen [31]. Additionally, ithas been shown that time constraints are an importantbarrier to HIE usage [27, 28, 32–34], which might resultin health professionals being reluctant to engage in HIE.Based on these results, we suggest that tailoring the typeof information displayed on the first screens of HIEsystems by type of user (e.g., physician, nurse) and discip-line (e.g., emergency medicine, pediatrics) might improveHIE utilization by providers. Furthermore, most priorstudies were performed in emergency departments withproviders already using HIE. New products often benefitfrom a user needs assessment before, during, and after thedevelopment cycle. We believe HIE systems will be moresuccessful if they are developed with a priori input fromits future users. Our work is unique as it provides a clin-ician needs assessment prior to HIE implementation, sothe providers have not developed biases of using an HIE.Furthermore, our research expands the current evidenceby focusing on an unexplored clinical setting in regards toHIE: an Internal Medicine (IM) Hospitalist Department.In this study, we investigated an IM Department in a
teaching hospital in Tampa, Florida before HIE imple-mentation. Our objectives were to understand theprocess of obtaining medical information from otherfacilities prior to HIE, explore provider perceptions ofthe usage of outside information for medical decision-making, and to analyze their views on the potentialimpact of HIE. Improving HIE developers’, policymakers’, and administrators’ understandings about howdocuments from outside institutions, referred to asoutside information (OI), are collected and utilized byclinicians can inform HIE design and implementationwhich could improve HIE usability.
MethodsWe used a convergent mixed-methods study design togather insights about the performance of the current fax-based process to request OI, the use of OI for medical-decision making, and the physicians’ perceptions of HIEimplementation. We conducted semi-structured inter-views with both IM third-year residents and attendingphysicians and performed direct observation of the work-flows in the IM Department. In addition, we collecteddemographic and clinical data of hospitalizations that gen-erated at least one request for OI. Institutional reviewboard approval was granted for this study by the hospital’sOffice of Clinical Research and the University of SouthFlorida (IRB Number: Pro00014574).
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 2 of 11
Appendix D (continued)
101
Study setting and datasetsThis research was performed in the IM Department ofa public, teaching hospital in Tampa, Florida. The hos-pital is a 1018-bed hospital serving 23 counties inTampa using the electronic medical record system(EMR) Epic (EpiCare; Verona, WI) with no HIE func-tionality enabled. We considered three sources of data:direct observation, interviews, and the EMR. First, weobserved approximately 750 h of the workflows andmedical decision-process related to the request of OI.Second, we interviewed resident and attending physiciansfrom the IM Department from January to February 2014.Finally, from the hospital’s EMR, we extracted demo-graphic and clinical factors for each hospitalization fromOctober 2011 to March 2014 that generated at least onerequest for OI. We also extracted operational data relatedto the request for OI: timestamps for the request and re-ceipt of OI, type of health professional requesting OI, andtype of information received.
Process mappingWe followed a two-step method of observation and val-idation to document the process to request and collectOI. We created a process chart that represents the activ-ities performed, resources used, and people involved inorder to obtain OI. To construct these diagrams, ourteam of industrial engineers and physicians observed theprocess and created preliminary flow process charts.During observation, the team shadowed and interviewedmedical teams, nurses and personnel from the medicalrecords department. Three people each performed 30observation periods. During each period, between 6 and10 h were observed. Observations were performed everyday of the week and during working hours. During theseobservations, between 3 and 5 providers were observedon both attending and resident physicians. Observers re-corded their observations when necessary. The initialflow process charts were then validated by subject mat-ter experts, which included physicians and the medicalrecords department. We validated the process map dur-ing semi-structured interviews with the third year resi-dents and attending physicians until saturation. Duringthis validation process, we discussed perceived processtimes and any additional comments about each step inthe process.
InterviewsA semi-structured interview (see Additional file 1) includ-ing 8 questions was performed with 16 physicians fromthe IM Department. All attending physicians in the IMDepartment and all third year resident physicians were e-mailed to be invited to participate in the study. We used anon-probabilistic convenience sampling approach. In aneffort to reduce interviewer bias, a team member with
expertise in interviewing methods prepared a 1-day train-ing for the other members of the team. Additionally, thequestions included in the interviews were discussed withsubject experts to avoid potential bias imposed by theteam. The duration of the interview was 30 min. Aninformed consent was reviewed and signed by each phys-ician. Each interview was audio recorded and transcribedfor posterior analysis. Afterwards, the de-identified tran-scripts were analyzed to code the main themes reportedby the subjects using Atlas.ti version 6.0 [35]. The codingprocess was performed concurrently by three study mem-bers with experience in medicine, systems engineering,and qualitative analysis. In case of disagreement, the studymembers discussed the alternatives and a majority votedetermined the final result.
ResultsInterview respondentsSixteen out of thirty-eight physicians participated(42.1 % response rate). The 16 study subjects included11 third-year resident physicians and 5 attending physi-cians. There were an equal number of male and femalesubjects. On average, interviewees had been using thesame EMR system for 2.5 years prior to the study. The30-min interviews were transcribed and generated afree text document containing 37,579 words that wasanalyzed using Atlas.ti.
EMR dataTable 1 describes the hospitalizations for which OI wasrequested. The study population was 50.7 % female and98.2 % English speaking followed by 4.5 % Spanishspeaking preference. The mean age was 53.5 years old.
Pre-HIE process map of obtaining OIUsing the information collected from shadowing medicalteams, interviewing physicians and meeting with medicalrecords personnel, a final flow process chart was created(see Fig. 1). The boxes with curved bottoms representsteps in the process involving paper. Each step was sepa-rated depending on the person or location in which ittook place. The current process to obtain outside re-cords comprises eight steps, five paper generation steps,six decision points and at least four different personnel.The pre-HIE process flow chart demonstrates where HIEcan improve the sharing of information. The process mapshows that various individuals with different levels ofmedical expertise and in different locations are requiredto complete myriad steps at different times. Many stepsinvolve paper documents to be generated and moved. Forexample, documents housed in one hospital need to befaxed page by page by an individual which generate an-other set of documents at the receiving hospital. Then, theduplicated paper documents are scanned into a computer,
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 3 of 11
Appendix D (continued)
102
stored and later shredded. These actions require humanand physical resources, as well as time. These types ofwaste could be largely replaced by a few clicks in an effect-ively designed HIE system.Figure 2 represents a simplified flow process chart.
Physicians believed that the time between identifying theneed for OI and placing the request ranges between1 min and 5 days, with a mode of 45 min. Our evalu-ation on the time actual orders to obtain HIE were en-tered into the EMR indicated that the median delaybetween admission and electronic order of OI requestwas 10-h. This demonstrates potential time that couldbe saved by effective HIE implementation if informationwas available immediately on admission to the hospital.Physicians estimated that the time between the requestand when the information was viewed ranged between 1and 72 h, with a mode of 18 h.The interviews revealed that providers want alerts
upon the arrival of OI. We found OI is sometimes faxeddirectly to the nurse’s station or the hospital’s Health In-formation Management Department depending on what
information is sent with the request. When OI arrives,physicians must wait for the OI to be scanned into thehospital EMR to have access to the information, andmust repeatedly check to see if the information is avail-able. This suggests that effective HIE designs should in-clude a feature to alert providers once OI is available forviewing. Another insight elicited through the interviewswas that physician satisfaction with the OI received washigher among those who made follow-up phone calls tooutside facilities to inquire about the record request.Also, physicians specifying exactly which data items theyneed in the OI request improved the value of the OIreceived.
Perceptions on use of OI compared to EMR dataTo explore physicians’ perceptions we asked, “What per-centage of your patients do you request for OI?” Mostphysicians believe they request outside records for 5 to10 % of their patients. We were able to compare the pro-vider perceptions to the quantitative data and found thatout of 15,230 admissions to the IM Department duringthe study timeframe, 2091 generated at least one requestfor OI (13.7 %). In addition, we were able to explorewhat factors influenced when the physician did not needOI. Responses to the question, “In which situations doyou know OI exists but you do not request for records?”are presented in Table 2. Most physicians answered thatif the current admission is unrelated to OI (i.e., “…itmay be unrelated to the acute [issue] they are coming infor.”), then they do not need that data. About 25 % ofphysicians reported that the process would take toolong, so they did not feel it was useful to request the in-formation (i.e., “I rarely request them because it’s so dif-ficult to get them. But I find it is usually not worth thetime.”). Most of the physicians (75 %) estimated that theinformation was not received or incorrect more than33 % of the time. Our analysis of EMR data showed thatin 814 out of 2091 (38.9 %) admissions, OI was re-quested but no documents were received.The majority of physicians stated that the information
received is often a large amount of data that is not orga-nized for quick clinical use. The majority of physiciansbelieved that between 33 and 66 % of all OI received isnot useful. They elaborated that they might only be look-ing for specific data items, but an abundance of dailymonitoring notes make it difficult to find relevant infor-mation. They also reported OI was not useful because itwas not the information they had requested. See Table 2for physician responses to the prompt: “Give examples inwhich outside information was requested and you en-countered problems. What percentage?”. This perceptionwas compared to our findings from the data from theEMR. OI received from outside facilities are indexed as“medical record”, “imaging”, “history and physical”, “note”,
Table 1 Demographic and clinical factors of hospitalizationswith at least one request for outside information
No. (%) N = 2091
Female 1061 (50.7)
Language preference
English 1949 (93.2)
Spanish 95 (4.5)
Unknown/Other 47 (2.3)
Marital status
Single 1361 (65.1)
Married 652 (31.2)
Unknown/Other 78 (3.7)
Primary care provider 1235 (59.1)
Payer class
Commercial 627 (30)
Medicare 817 (39.1)
Medicaid 465 (22.2)
HCHCP 137 (6.6)
Other 45 (2.1)
Admission source
Emergency room 1921 (91.9)
Physician-referral 84 (4)
Outside hospital 84 (4)
Other 2 (0.1)
Mean (SD)
Age 53.5 (17.3)
Length of stay 6.7 (10)
HCHCP Hillsborough Country Health Care Plan
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 4 of 11
Appendix D (continued)
103
Fig. 1 Flow process chart of obtaining outside information. Abbreviations: OI, outside information
Fig. 2 Simplified flow process chart of obtaining outside information from physician perspective
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 5 of 11
Appendix D (continued)
104
“discharge summary”, “electrocardiogram”, or “consult-ation”. As shown in Table 3, most of the documents re-ceived were medical records (n = 2343) followed byimaging (n = 567) and history and physical (n = 395).Therefore, most received documents are labeled am-biguously as “medical records”, consistent with phys-ician perceptions that the OI is usually not useful.Mitigating an overabundance of data with efficientcategorization of records is key for the successful futureof HIE.
Physician-identified clinical drivers for future HIE useThrough our user needs assessment, we were able to iden-tify common themes of clinical drivers for physiciansrequesting OI and medical decision-making using OI. Byfocusing on the drivers of OI requests, HIE designers andadministration can work with clinicians to give physicians
information they need at a time that it is clinically rele-vant. Physicians were asked, “In which specific clinicalsituations would timely OI influence your medical deci-sions?”. The research team classified the clinical driversfor OI described by physicians into three groups: general,test-related, and health condition. As shown in Fig. 3, 10out of 16 interviewed physicians reported “knowing previ-ous workup or treatment”, “medication reconciliation”and “comparing lab abnormalities” as clinical driverswhere having OI may influence medical decisions. In gen-eral, physicians found OI most beneficial if the patient wasunable to communicate and information was not availablefrom family members.Specific test-related clinical drivers for OI requests are
presented in Fig. 4. Responses included imaging and la-boratory tests. Imaging was the most frequently requestedtest, indicated by 11 of the 16 interviewed physicians.Specifically CT scan was identified by 6 physicians andmagnetic resonance imaging (MRI) was identified by 6physicians. Echocardiograms, cardiac catheterizations,electrocardiograms and troponin levels were mentionedby 10, 7, 4 and 1 of the 16 interviewed physicians, respect-ively. Bacterial cultures from urine, blood, or other sourceswere recognized as important to clinical decision-makingby 7 physicians. Physicians also wanted specific informa-tion about blood cultures including speciation, antibioticsusceptibility and amount of bacteria present. Withoutthis information, tests may need to be repeated and effect-ive treatment is delayed or unnecessary treatment isprovided.
Table 2 Summary of physician perceptions of current, pre-HIE use of outside information requested from outside hospitals
Reasons for not requesting Problems encountered
1. Time 1. Process
● Outside information is too old ● Need to re-request
● Physician assumes the OI request process takes too long ● Delay in sending or scanning outside information after work hours
● Emergent situations ● Transitions-of-care communication problems
● Brief Hospital stay ● Problems with outside information transfer patients
● Do not receive any outside information
● OI comes too late
● Delay waiting for imaging to be loaded from CD
● Unaware of where outside information is in the process or if it has arrived
2. Relevance 2. Information
● Current admission unrelated to outside information ● Unhelpful physician or nursing notes
● Unnecessary to request outside information based onclinical expertise
● Difficulty finding useful information in unorganized and abundant amount ofoutside information
● Skepticism of imaging or culture reads from outside facility
3. Patient
● Patient or family is good historian and record keeper
● Patient does not know where to request outsideinformation from
OI outside information
Table 3 Document types received from outside health carefacilities
Document type Number of documents received (%) N = 2091
Medical record 1637 (78)
Imaging 383 (18)
History and physical 255 (12)
Note 206 (10)
Discharge summary 164 (8)
Electrocardiogram 153 (7)
Consultation 151 (7)
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 6 of 11
Appendix D (continued)
105
Figure 5 shows the diverse health conditions that wereidentified as influential on medical decisions. The most fre-quently identified conditions were chest pain, acute cardiacconditions and infection, followed by kidney injury andcancer. 19 % of physicians discussed pneumonia and sepsis.Anemia was mentioned by 13 % of the interviewees. Theremaining diagnoses were: thrombocytopenia, pulmonaryhypertension, pulmonary embolism, malingering, lymph-adenopathy, falls, Crohn’s disease, acute respiratory dis-tress, urinary tract infection, liver disease, identifying drug-seekers, altered mental status and chronic obstructivepulmonary disease.
Other critical clinical drivers for OI were admissionsto the intensive care unit (ICU) and transfers from otherhospitals. 19 % of physicians identified critically ill pa-tients as key examples of when OI would be valuable.The physicians elaborated that knowing the prior work-up of a critically ill patient can expedite life-saving pa-tient care decisions. Studies have shown that patientsunable or unwilling to communicate their health status,which is common in the ICU, are targets for using HIE[26]. Additionally, patients transferred from other hospi-tals are an important population because they are oftensicker patients with complex medical conditions. Infor-mation about the workup done at the originating hospitalis critical to the receiving providers to provide effectivecare to the patient. Unfortunately, transitions of care aredifficult in these situations because of the emergent natureand abundance of information. In our interviews, 50 % ofthe physicians recognized “hospital transfers” as an oppor-tunity for using HIE, which is consistent with other re-ports [36]. Six interviewees identified that they frequentlyget incomplete OI in these cases, and five intervieweessaid there was poor communication with transfers.
Perceptions on pre-HIE electronic viewing of OI andpotential for HIEAfter discussion about situations where OI was influentialin medical decisions, we wanted to explore how physiciansphysically interact with the outside records received. Atthe study hospital, outside documents are scanned intothe EMR when they are received by fax, where they canthen be viewed electronically. The original paper docu-ments are stored in the patient’s bedside chart for tempor-ary access. Physicians were asked, “Do you view themajority of the outside records in paper or electronic
Fig. 3 Response distribution to the question “In which specific (general) clinical situations would timely OI influence your medical decisions?”Abbreviations: ICU, intensive care unit
Fig. 4 Response distribution to the question “In which specific(test type) clinical situations would timely OI influence yourmedical decisions?” Abbreviations: MRI, magnetic resonanceimaging; EKG, electrocardiogram; CT, computed tomography
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 7 of 11
Appendix D (continued)
106
format? What percentage?”. Then, a discussion was gener-ated about the positives and negatives of viewing each for-mat. Physicians responded that they view OI electronicallyless than 40 % of the time. The negative aspects identifiedfor electronic viewing were “excessive clicking” and “itdoes not facilitate parallel tasking”. Because there islimited screen space, it is difficult to view the outsidedocuments while viewing current clinical information.Therefore, it is cumbersome to compare lab values orincorporate data into current documentation. Also, be-cause of excessive amounts of records received andneeding to adjust the zoom frequently to view contentproperly, the process requires extensive clicking. Oneof the benefits of electronic viewing was “remote accessto records”.
At the end of the interviews, we explored physicians’perceptions about HIE implementation in the future.Most physicians regarded HIE implementation positively;of the total number of responses to their perceptionsabout HIE, 85 % of the answers were coded as “positive”.Most providers recognize the need for universal access topatient records and anticipate streamlined patient care.The most frequent positive responses were that HIE will“facilitate better patient care”, lead to “less test redun-dancy” and “reduce costs”. Some other perceptions werethat HIE will “reduce patient harm”, “decrease delays” and“improve transitions of care.” One physician mentionedthat it would only be “beneficial if done the right way.”The negative feelings towards HIE were “concerns withHIPAA”, “access to meaningless data” and “slow down
Fig. 5 Response distribution to the question “In which specific (health condition) clinical situations would timely OI influence your medicaldecisions?” Abbreviations: ICU, intensive care unit; COPD, chronic obstructive pulmonary disease; CHF, congestive heart failure
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 8 of 11
Appendix D (continued)
107
patient care”. This largely positive perception of the po-tential for HIE is an interesting contrast to providersthat have experienced the problems of HIEs afterimplementation.
DiscussionOur study suggests that the drivers for HIE utilizationare the treatment of complex patients with a high num-ber of comorbidities or with frequent previous health-care visits, consistent with previous research [27]. Ourstudy identifies the difficulties faced by physicians in anIM Department in a large hospital in order to obtainoutside information prior to HIE implementation andprovides a user needs assessment to inform HIE designand implementation. Our research begins to address thegap identified by O’Malley and colleagues between thepolicy makers’ expectations and the clinicians’ experi-ences with HIE [37]. We identified information that isimportant to physicians in specific clinical situations.Finally, we provided physicians’ insight into their percep-tions of future implementation of HIE.
User needs assessment to inform HIE designOur results suggest that efficient organization of datashared by HIE is paramount to effective use. Prior datashowing low usage by providers may be partly due tothe user-unfriendly nature of current HIE, which weredesigned without empiric a priori end-user input. Table 4presents a design for the implementation of HIE in-formed by the results of our study. By identifying pat-terns in responses by the physicians, we were able tostart creating networks of clinical drivers and importantinformation needs to inform medical decision-making.An example clinical domain is congestive heart failure.
Many physicians identified congestive heart failure as acondition in which specific OI, such as echocardiograms,electrocardiograms and weight measurements, likely in-fluence clinical decisions and patient outcomes. Thisfinding from the interviews is particularly important be-cause the Centers for Medicare and Medicaid Services(CMS) require all congestive heart failure patients tohave an up-to-date echocardiogram documented [38].One of our recommendations is having visual indicatorsthat alert the user when OI in the HIE is relevant to spe-cific diagnoses within the local system. For example, if aprovider were treating a patient with heart failure, theHIE would indicate that an echocardiogram is availablefrom an outside hospital. These clinically relevant fea-tures of an HIE would promote provider satisfaction byfacilitating their HIE interface experience and potentiallyimprove compliance with quality measures.
Problems amenable to HIE and factors that will remainproblematicOur analysis of physician interviews identified problemsamenable to HIE and factors that will remain problem-atic despite HIE implementation. Some factors that willbe alleviated by HIE are the physician not requesting OIbecause they assume the process will take too long oryield incorrect information. The current fax based sys-tem is inefficient, so often providers proceed with lessinformation. However, a well designed HIE could pro-vide some information faster and more reliably. This willbe helpful especially in critical situations, such as the ICUor hospital transfers. Another factor amenable to HIE iswhen the patient does not know from where to requestOI. In some HIEs, the provider will be able to see the loca-tion of all OI. Also, the difficult process to find moreinformation after initial review of OI will be mitigatedbecause the provider will not need to fill out requestforms, fax them again, and wait for their return (See Figs. 1and 2). They will only require re-accessing HIE to findmore information. The problem of not being able to getOI after office hours will be eliminated as the HIE will beautomated without relying on personnel to manually faxinformation.Some problematic factors that will remain despite
HIE implementation are if the OI is old informationand needs to be repeated despite having easy access toit. HIE will also be challenged by an abundance ofunorganized information received if it is not designedproperly. Viewing original radiology imaging may beslow using HIE, so the need for imaging disks may notbe alleviated by HIE completely. There may still beskepticism of the results from outside facilities, whichwill lead to repetitive testing. Similarly, the HIE willonly have final reports for bacterial cultures and there
Table 4 Design recommendations for health informationexchange in an Internal Medicine Department in a publichospital
Design recommendations
1. Allow keyword search functionality in OI
2. Provide the telephone number of the OI source for follow upquestions
3. Provide the list of previous medications for medication reconciliation
4. Facilitate remote access to patients’ medical records
5. Provide computer screens that facilitate parallel tasking whilereviewing documents electronically
6. Visual indicators for when OI is potentially relevant to specificdiagnoses
7. Provide 1-click access to imaging, echocardiograms, bacterial cultures,cardiac catheterizations and CTs results (not only reports)
8. Prioritize OI access to patients with acute cardiac issues, chest pain,infection, cancer, and kidney injury
9. Prioritize OI access for hospital transfers and ICU patients
OI outside information, CT computerized tomography, ICU intensive care unit
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 9 of 11
Appendix D (continued)
108
may still be doubt as to the laboratory techniques forcertain results (i.e., which location cultures were drawnfrom).
Limitations & future workOur study has limitations. First, the semi-structured inter-views were a very powerful approach to obtain even subtleperceptions from the people who are involved in theprocess of requesting OI. However, by directly interview-ing physicians, we are disturbing the environment andtherefore the responses may be influenced by the presenceof the research team. Second, because of the sample sizeand the specific setting (a teaching hospital using Epic),the conclusions obtained in this study may not begeneralizable. However, this study represents an advancein the community of HIE knowledge since this researchhas not been carried out before in IM Departments withina hospital. Additionally, as of March 2015, Epic Systems isone of the top three EMR vendors comprising nearly 60 %of the market share of primary certified EMRs [39]. Futureresearch should be done using a longitudinal approach,and ideally a larger number of settings. Finally, we alsohad attrition bias due to non-responses and we did notaddress any potential confounding due to user characteris-tics. For example, the level of computer skills may havebiased physicians’ responses. Nonetheless, all the inter-viewees had at least 2.5 years of experience in the sameIM Department and with Epic.There are various aspects that can be addressed in fu-
ture work. First, the effect of provider access to clinicallyrelevant OI on length of stay and resource utilizationshould be assessed. Linking OI to patient outcomes iskey to demonstrating HIE value. Second, patients withabdominal pain and cardiac problems should be specific-ally explored since these patients represent a large amountof OI requests. Third, HIE research should focus on ICUpatients or hospital transfer admissions, as others have ex-plored the challenges of communication between hospital-ists and primary care physicians [40].
ConclusionBy using mixed-methods we were able to map the currentprocess of requesting OI, define provider perceptions, andcompare those perceptions to quantitative data. Thisknowledge provides a user needs assessment for informingfuture HIE design and implementation. Further, our studycombined with other research can direct future financialincentives to specifically promote evidence-based func-tionality that improves important outcomes. As meaning-ful use has improved EMR adoption, incentives for HIEpaired with physician-guided implementation can likelyimprove the utilization of HIE.
Additional file
Additional file 1: Semi-structured interview: list of close- andopen-ended questions used during the semi-structured interviews.(DOCX 23 kb)
AbbreviationsCHF: Congestive heart failure; CMS: Centers for Medicare and MedicaidServices; COPD: Chronic obstructive pulmonary disease; CT: Computerizedtomography; EKG: Electrocardiogram; EMR: Electronic medical record;HIE: Health information exchange; ICU: Intensive care unit; IM: Internalmedicine; MRI: Magnetic resonance imaging; OI: Outside clinical information.
Competing interestsNone.
Authors’ contributionsAS and DM contributed to the idea conception, study design, acquisitionand analysis of qualitative and quantitative data. AG contributed to thestudy design and acquisition and analysis of qualitative data. CM and STcontributed to the study design and acquisition of qualitative andquantitative data. PF contributed to the analysis of quantitative andqualitative data. JZ is guarantor and contributed to the idea conceptionand study design. All authors contributed equally in preparing andreviewing multiple versions of the manuscript and provided importantintellectual content. All authors read and approved the final version ofthis manuscript.
AcknowledgmentsWe would like to thank Peter Chang, Scott Arnold, Athena Muse and thephysicians and nurses from hospital evaluated in this study for theircontributions in this study. No funding was provided for the completionof this study.
Author details1Department of Internal Medicine, College of Medicine, University of SouthFlorida, Tampa, FL, USA. 2Johns Hopkins Department of Emergency Medicine,Baltimore, MD, USA. 3Department of Industrial and Management SystemsEngineering, College of Engineering, University of South Florida, Tampa, FL,USA. 4Department of Internal Medicine, Carolinas Medical Center, Charlotte,NC, USA. 5Department of Surgery, College of Medicine, University of SouthFlorida, Tampa, FL, USA.
Received: 26 March 2015 Accepted: 5 October 2015
References1. Anderson G, Knickman JR. Changing the chronic care system to meet
people’s needs. Health Aff. 2001;20:146–60.2. Pham HH, Schrag D, O’Malley AS, Wu B, Bach PB. Care patterns in Medicare
and their implications for pay for performance. N Engl J Med.2007;356:1130–9.
3. Gandhi TK, Sittig DF, Franklin M, Sussman AJ, Fairchild DG, Bates DW.Communication breakdown in the outpatient referral process. J Gen InternMed. 2000;15:626–31.
4. Jha AK, Chan DC, Ridgway AB, Franz C, Bates DW. Improving safety andeliminating redundant tests: cutting costs in U.S. hospitals. Health Aff(Millwood). 2009;28:1475–84.
5. Blumenthal D. Stimulating the adoption of health information technology.N Engl J Med. 2009;360:1477–9.
6. Rudin RS, Motala A, Goldzwelg CL, Shekelle PG. Usage and effect of healthinformation exchange: a systematic review. Ann Intern Med. 2014;161:803–12.
7. Rahurkar S, Vest JR, Menachemi N. Despite the spread of health informationexchange, there is little evidence of its impact on cost, use, and quality ofcare. Health Aff (Millwood). 2015;34:477–83.
8. Ballard J, Rosenman M, Weiner M. Harnessing a health informationexchange to identify surgical device adverse events for urogynecologicmesh. In: AMIA Annual Symposium proceedings, vol. 2012. 2012. p. 1109–18.
9. Kaelber DC, Bates DW. Health information exchange and patient safety. JBiomed Inform. 2007;40(6 Suppl):S40–5.
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 10 of 11
Appendix D (continued)
109
10. Kouroubali A, Starren JB, Clayton PD. Costs and benefits of connectingcommunity physicians to a hospital WAN. In: AMIA Annual Symposiumproceedings. 1998. p. 205–9.
11. Bailey JE, Wan JY, Mabry LM, Landy SH, Pope RA, Waters TM, et al. Doeshealth information exchange reduce unnecessary neuroimaging andimprove quality of headache care in the emergency department? J GenIntern Med. 2012;28(2):176–83.
12. Bailey JE, Pope RA, Elliott EC, Wan JY, Waters TM, Frisse ME. Healthinformation exchange reduces repeated diagnostic imaging for back pain.Ann Emerg Med. 2013;62:16–24.
13. Overhage JM, Dexter PR, Perkins SM, Cordell WH, McGoff J, McGrath R, et al.A randomized, controlled trial of clinical information shared from anotherinstitution. Ann Emerg Med. 2002;39:14–23.
14. Callen J, Paoloni R, Li J, Stewart M, Gibson K, Georgiou A, et al. Perceptionsof the effect of information and communication technology on the qualityof care delivered in emergency departments: a cross-site qualitative study.Ann Emerg Med. 2013;61:131–44.
15. Ozkaynak M, Brennan PF. Revisiting sociotechnical systems in a case ofunreported use of health information exchange system in three hospitalemergency departments. J Eval Clin Pract. 2013;19:370–3.
16. Thorn SA, Carter MA. The potential of health information exchange to assistemergency nurses. J Emerg Nurs. 2013;39:e91–6.
17. Frisse ME, Johnson KB, Nian H, Davison CL, Gadd CS, Unertl KM, et al. Thefinancial impact of health information exchange on emergency departmentcare. J Am Med Inform Assoc. 2012;19(3):328–33.
18. Vest JR, Miller TR. The association between health information exchangeand measures of patient satisfaction. Appl Clin Inform. 2011;2:447–59.
19. Nguyen OK, Chan CV, Makam A, Stieglitz H, Amarasingham R. Envisioning asocial-health information exchange as a platform to support a patient-centered medical neighborhood: a feasibility study. J Gen Intern Med.2014;30(1):60–7.
20. Lang E, Afilalo M, Vandal AC, Boivin J-F, Xue X, Colacone A, et al. Impact ofan electronic link between the emergency department and familyphysicians: a randomized controlled trial. CMAJ. 2006;174:313–8.
21. Hansagi H, Olsson M, Hussain A, Ohlén G. Is information sharing betweenthe emergency department and primary care useful to the care of frequentemergency department users? Eur J Emerg Med. 2008;15:34–9.
22. Adler-Milstein J, Jha AK. Health information exchange among U.S. hospitals:who’s in, who’s out, and why? Healthcare. 2014;2:26–32.
23. Rudin R, Volk L, Simon S, Bates D. What affects clinicians’ usage of healthinformation exchange? Appl Clin Inform. 2011;2:250–62.
24. Furukawa MF, King J, Patel V, Hsiao C-J, Adler-Milstein J, Jha AK. Despitesubstantial progress in EHR adoption, health information exchange andpatient engagement remain low in office settings. Health Aff (Millwood).2014;33(9):1672–9. hlthaff.2014.0445.
25. Leu MG, Cheung M, Webster TR, Curry L, Bradley EH, Fifield J, et al. Centersspeak up: the clinical context for health information technology in theambulatory care setting. J Gen Intern Med. 2008;23:372–8.
26. Finnell JT, Overhage JM. Emergency medical services: the frontier in healthinformation exchange. AMIA Annu Symp Proc. 2010;2010:222–6.
27. Vest JR, Zhao H, Jasperson J, Jaspserson J, Gamm LD, Ohsfeldt RL. Factorsmotivating and affecting health information exchange usage. J Am MedInform Assoc. 2011;18:143–9.
28. Vest JR, Jasperson ‘S, Zhao H, Gamm LD, Ohsfeldt RL. Use of a healthinformation exchange system in the emergency care of children. BMC MedInform Decis Mak. 2011;11:78.
29. Vest JR, Grinspan ZM, Kern LM, Campion TR, Kaushal R. Using a healthinformation exchange system for imaging information: patterns andpredictors. AMIA Annu Symp Proc. 2013;2013:1402–11.
30. Campion TR, Ancker JS, Edwards AM, Patel VN, Kaushal R. Push and pull:physician usage of and satisfaction with health information exchange. AMIAAnnu Symp Proc. 2012;2012:77–84.
31. Vest JR, Jasperson ‘S. How are health professionals using health informationexchange systems? Measuring usage for evaluation and systemimprovement. J Med Syst. 2012;36:3195–204.
32. Ross SE, Schilling LM, Fernald DH, Davidson AJ, West DR. Health informationexchange in small-to-medium sized family medicine practices: motivators,barriers, and potential facilitators of adoption. Int J Med Inform.2010;79:123–9.
33. Wright A, Soran C, Jenter CA, Volk LA, Bates DW, Simon SR. Physicianattitudes toward health information exchange: results of a statewide survey.J Am Med Inform Assoc. 2010;17:66–70.
34. Sicotte C, Paré G. Success in health information exchange projects: solvingthe implementation puzzle. Soc Sci Med. 2010;70:1159–65.
35. GmbH. ATLAS.ti version 6.2.23. [Computer Software]. Berlin, Germany. 2009.Licensed through University of South Florida.
36. Rudin RS, Salzberg CA, Szolovits P, Volk LA, Simon SR, Bates DW. Caretransitions as opportunities for clinicians to use data exchange services:how often do they occur? J Am Med Inform Assoc. 2011;18:853–8.
37. O’Malley AS, Grossman JM, Cohen GR, Kemper NM, Pham HH. Are electronicmedical records helpful for care coordination? Experiences of physicianpractices. J Gen Intern Med. 2010;25:177–85.
38. Physician Quality Reporting System. [http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/PQRS/index.html?redirect=/PQRS/].Access date Oct 8 2015.
39. Electronic Health Record Vendors Reported by Hospitals Participating in theCMS EHR Incentive Programs,’ Health IT Quick-Stat #29 [dashboard.healthit.gov/quickstats/pages/FIG-Vendors-of-EHRs-to-Participating-Hospitals.php]. Accessdate Oct 8 2015.
40. Hennrikus E. Communication between the primary care physician and thehospital at the time of patient admission. J Clin Outcomes Manag.2012;19:453–9.
Submit your next manuscript to BioMed Centraland take full advantage of:
• Convenient online submission
• Thorough peer review
• No space constraints or color figure charges
• Immediate publication on acceptance
• Inclusion in PubMed, CAS, Scopus and Google Scholar
• Research which is freely available for redistribution
Submit your manuscript at www.biomedcentral.com/submit
Strauss et al. BMC Medical Informatics and Decision Making (2015) 15:81 Page 11 of 11
Appendix D (continued)
110
Appendix E: Uncovering Hospitalists’ Information Needs From Outside Healthcare Facilities in the
Context of Health Information Exchange Using Association Rule Learning
Appendix E exhibits the manuscript titled, "Uncovering Hospitalists’ Information Needs From
Outside Healthcare Facilities in the Context of Health Information Exchange Using Association
Rule spotLearning", which is under review in Applied Clinical Informatics.
111
Uncovering Hospitalists’ Information Needs from Outside Healthcare Facilities in the
Context of Health Information Exchange Using Association Rule Learning
Diego A. Martinez, Elia Mora, Martino Gemmani, Jose L. Zayas-Castro
Preprint Submitted to Applied Clinical Informatics
ABSTRACT
Background: Important barriers to health information exchange (HIE) adoption are clinical
workflow disruptions and troubles with the HIE system interface. Prior research suggests that
interfaces of HIE systems providing faster access to useful information may stimulate use and
reduce barriers for adoption; however, little is known about informational needs of hospitalists.
Objective: Study the association between health problems and the type of information
requested from outside healthcare providers by hospitalists of a tertiary care hospital.
Methods: We searched operational data associated with the fax-based exchange of patient
information (previous HIE implementation) between hospitalists of an internal medicine
department in a large urban tertiary care hospital in Florida, and any other affiliated and
unaffiliated healthcare provider outside the hospital. All hospitalizations from October 2011 to
March 2014 were included in the search. Strong association rules between health problems and
the types of information requested during each hospitalization were discovered using Apriori
algorithm, which were then validated by a team of hospitalists of the same department.
Results: Only 13.7% (2,089 out of 15,230) of the hospitalizations generated at least one
request of patient information to other providers. The transactional data showed 20 strong
association rules between specific health problems and types of information exist. Among the
20 rules, for example, abdominal pain, chest pain, and anemia patients are highly likely to have
Appendix E (continued)
112
medical records and outside imaging results requested. Other health conditions, prone to have
records requested, were lower urinary tract infection and back pain patients.
Conclusions: The presented list of strong co-occurrence of health problems and types of
information requested by hospitalists from outside healthcare providers not only informs the
implementation and design of HIE, but also helps to target future research on the impact of
having access to outside information for specific patient cohorts. Our data-driven approach
helps to reduce the typical biases of qualitative research.
Keywords: health information exchange; medical record linkage; medical decision making;
hospital medicine; patient handoff; medical informatics applications.
Appendix E (continued)
113
1. INTRODUCTION
In the United States, people suffering from chronic health conditions constitute 49.8% of the
adult population [1], and they consume 84% of the health care expenditures [2]. For people to
achieve a safe, effective, and efficient health care, a coordinated effort is often required among
unaffiliated providers. Lack of care coordination may lead to medication errors, avoidable
hospital readmissions, duplicated testing, and delays in understanding the patient condition [3–
10]. Since 2009, to support improvements in care coordination, the federal government has
been stimulating the adoption and use of health information exchange (HIE). However, recent
studies report HIE adoption across hospitals is still low [11,12]. As noted in the systematic
review by Rudin and colleagues, one of the important barriers to HIE adoption are clinical
workflow disruptions and troubles with the system interface [13]. Several authors claim better-
designed interfaces for HIE systems would stimulate its usage since clinicians will have quicker
access to useful patient information [14–16].
To improve HIE systems, it is imperative to understand physician information needs from
outside health care facilities. Healthcare providers are increasingly constrained by the time they
have to diagnose and treat patients, while trying to both follow evidence-based
recommendations and consider the unique needs, characteristics, and preferences of the
patients [17–22]. Given that the voluntary usage of additional information sources, such as HIE,
can be discouraged by time constraints [23], there is a need to make the information displayed
on HIE systems more valuable than the opportunity costs. For instance, screen redesign, single
sign-on, enhanced record searches, or eliciting user needs could all be means to address the
need. Additionally, the expected benefits of HIE might be fruitless if clinicians do not have
access to a system that takes into account users’ unique needs, cognitive tasks, and workflow
processes [24]. However, there is no clear understanding and agreement of what data elements
are needed from outside health care facilities to assist physicians in their decision-making [25].
Appendix E (continued)
114
Therefore, the information needs of the physicians are needed to inform the design and
deployment of the HIE and health IT policy. Most of the published studies on physicians’
information needs have focused on the communication between hospital-based (i.e.,
hospitalists) and primary care physicians [26]. However, in the context of HIE, the information
sharing will include a bigger spectrum of healthcare providers. The communication between
hospitalists and primary care providers has particular perspectives that may influence
information needs and resource preferences.
Additionally, the collection of meaningful data on information needs may be problematic.
Beyond the usual drawbacks of surveys and interviews, physician self-assessments of
information-seeking behavior can be unreliable. For example, physicians may be unaware of
their needs at the time of applying the self-assessment instrument. The information channels
they use and their methods of using them, which are influenced by study habits adopted as
early as medical school or college, may not provide the most efficient, accurate, and
comprehensive information necessary for medical decision-making [27]. Many physicians are
unaware of, or uncomfortable with, ever-evolving sources of information. In previous years,
investigations have used questionnaires (e.g., [28–32]) and interviews (e.g., [33–36]) to shed
light on physician’s sources of information and how these influence workflow. Unfortunately,
limited conclusions can be drawn from these data due to limitations in the internal validity and
generalizability. In many of the investigations, for example, less than 50% of the sample
population participated in the study.
This article reports the results of a study to document hospitalists’ information needs in a
large urban tertiary care hospital in Florida with no HIE functionality, and in planning stages for
implementation. Our objective was to uncover associations between the health problems of the
patient and the type of clinical information requested from outside health care facilities. An
attempt was made to reduce selection and recall biases by mining a large number of data
Appendix E (continued)
115
transactions from October 2011 to March 2014 of all hospitalists and residents working in the
internal medicine department. Since other researchers have successfully used association rule
learning (ARL) algorithms to analyze healthcare data (e.g., [37–39]), we implemented the Apriori
algorithm to discover strong associations between the patients’ health problems and the clinical
information requested. The outcome of our investigation will help HIE developers and
implementers recognize commonly requested clinical information from outside health care
facilities by specific health problems, and thereby prioritize information display.
2. METHODS
The transactional data used in this study were collected from the Internal Medicine
Department of Tampa General Hospital (TGH) in Tampa, Florida. TGH is a 1,018-bed tertiary
care hospital serving over four million people from 23 counties in West Central Florida with no
HIE functionality, and in planning stages for implementation. During the study timeframe, thre
was no functional HIE in the region where TGH is located, and thereby most of the health
information transactions between healthcare providers were performed via fax and telephone. A
list of disease-information association rules was mined from transactional data using the Apriori
algorithm, and validated by senior internists working at the same department. Transactional
data includes all types of clinical information requested from outside healthcare providers during
a patient hospitalization (denoted as outside information, OI) via fax and telephone, which was
then scanned into the patient’s electronic medical record. Our approach comprised four major
phases: data collection and pre-processing, association rule building, post-processing and
association rule selection, and clinical expert validation.
2.1. Data collection and pre-processing
Appendix E (continued)
116
Our dataset included all hospitalizations from October 2011 to March 2014 with at least one
request for OI. The dataset was constructed with the list of health problems, and the list of OI
requested in each hospitalization. The list of health problems corresponds to the discharge
problem list, which are directly recorded by physicians during the patient hospitalization. We
also collected demographic and clinical factors associated with each hospitalization.
Independently, to assure consistency, three co-authors detected and corrected inaccurate
health problem terms in the dataset. Any discrepancies between the co-authors were discussed
and resolved by consensus, and uncertainty was referred to the fourth co-author.
2.2. Association rule building
We used ARL to discover strong associations between the health problems (antecedent)
and OI requested (consequent). Since previous investigations found HIE useful only in particular
cases [40], we hypothesize that a strong association between a health problem and an OI type
indicates an important information need. Association rules are antecedents implying
consequences of the form 𝑋 → 𝑌, in our study, health problems implying OI requests. The
association 𝑋 → 𝑌 measures how likely the event 𝑌 is, given 𝑋. We measured the quality of an
association rule in terms of support and confidence, and the quality of an association rule in
terms of lift. Support corresponds to the statistical significance of a rule given by the proportion
of transactions in the dataset containing a given set of health problems and OI types. A high
support denotes a high popularity for the given set of health problems and OI types. Confidence
is a measure of a rule’s strength and is calculated as the conditional probability of the
consequent given the antecedent, which is understood as the probability that a health problem
occurs if it is known that a particular OI type was requested. Lift denotes the strength of the rule
over the random co-occurrence of the antecedents and the consequent. Particularly, a lift
greater than 1 implies the association between the set of health problems and the set of OI
Appendix E (continued)
117
types is more significant than if the two sets were independent. In our context, an association
rule with a lift value of 2 means that a physician who serves a patient with disease 𝑋 is twice
more likely to request outside information type 𝑌 than the general physician, and similarly, the
physician who request 𝑌 is twice as likely to being serving a patient type 𝑋, since lift is a
symmetric measure. The stronger the association is–the larger the lift. In epidemiological terms,
support and confidence are related to the terms of prevalence and positive predictive value,
respectively.
The association rules were mined using Apriori algorithm[41], which was executed in R
using the Arules package[42]. Apriori calculates a set of strong rules given an arbitrarily
selected minimum value for support and confidence. The strategy behind Apriori is to
decompose the task of finding strong rules into two major subtasks; the frequent itemset
generation and the rule generation. Frequent itemset generation finds those itemsets satisfying
an arbitrarily selected minimum support value. On the other hand, rule generation extracts all
the high-confidence rules from the previously generated frequent itemsets. These extracted
rules are denoted as strong rules. Apriori algorithm assumes items within an itemset to be
independent, and thereby it may disregard hidden interrelationships among items. This is
important when dealing with many real-world applications, since the data under study are
usually far from being perfect. For example, a distributed information environment with data
being collected from different sources with imprecise and vague documentation methods. In our
study, we assume that the dataset under study is precise and contain no ambiguity. We support
this assumption in the fact that all data collected for this study were documented by highly
trained individuals in a single EMR system. More precisely, hospitalists document the health
problems during a hospitalization and coders from the hospital electronic medical records
department document the OI types received from outside healthcare providers.
Appendix E (continued)
118
2.3. Post-processing and association rule selection
Once the set of strong rules was generated, we selected those in which both of the following
conditions were satisfied: at least one health problem was present in the antecedent, and at
least one OI type was present in the consequent. We denote these extracted rules as strong
and potentially meaningful rules. Additionally, a chi-square test was utilized to determine the
statistical significance of each association rule, where the rule-corresponding two-by-two table is
given by the cells 𝑋 ∩ 𝑌, 𝑋𝑐 ∩ 𝑌, 𝑋 ∩ 𝑌𝑐, and 𝑋𝑐 ∩ 𝑌𝑐, where 𝑐 refers to the complement of a
given itemset. To facilitate calculations, we used the results of [43] to derive the chi-square
value of each rule in terms of its support, confidence, and lift, and of the total number of data
instances n. A p-value providing an upper bound on the type I error (i.e., the risk of discovering
a rule that is actually false) of each rule is then computed from the chi square value by
consulting the chi square distribution with one degree of freedom. Due to the high risk of type I
error inherent to ARL algorithms, we adjusted the p-values to control for false discovery using
an improved Bonferroni-type procedure: the Benjamini-Hochberg correction method[44]. This
method allows us to control type I error during the identification of statistically significant rules in
our exploratory study. Another approach to evaluate statistical significance of association rules
is to test tentative rules on a validation dataset. However, this approach is problematic to use in
exploratory studies, as in our context, due to the limited data availability. In our study, we
consider those rules for which the chi square values lead to a corrected statistical significance
level or type I error of 0.10 or lower to be statistically significant.
2.4. Clinical expert validation
We validated the set of strong and potentially meaningful rules with three internists from the
TGH Internal Medicine Department. To assure consistency, the three internists independently
Appendix E (continued)
119
assessed the set of rules generated by our research team. By consensus, any discrepancies
between the internists were discussed and resolved. These validated rules are denoted as our
final set of association rules.
3. RESULTS
3.1. Population and Dataset
Only 13.7% (2,089 out of 15,230) of the hospitalizations in the internal medicine department
generated at least one request for OI. As shown in Table 1, 50.7% of the patients were female,
with 93.2% English speakers followed by 4.5% Spanish speakers. Although 91.9% of the
patients were admitted through the emergency department, most of them (59.1%) had a primary
care provider at the time of their admission. The mean age was 53.5 years old, and the mean
length of stay was 6.7 days.
Table 1. Demographic and clinical factors of hospitalizations, with at least one request for
clinical information from outside healthcare providers, in the Internal Medicine Department of the
Tampa General Hospital. Abbreviations: HCHCP, Hillsborough Country Health Care Plan.
N=2,089 No. (%)
Female 1,059 (50.7) Language preference English 1,948 (93.2) Spanish 94 (4.5) Unknown/Other 47 (2.3) Marital status Single 1,361 (65.1) Married 650 (31.2) Unknown/Other 78 (3.7) Have a primary care provider 1,235 (59.1) Payer class Commercial 627 (30) Medicare 817 (39.1) Medicaid 465 (22.2)
Appendix E (continued)
120
HCHCP 137 (6.6) Other 45 (2.1) Admission source Emergency room 1,919 (91.9) Physician-referral 84 (4) Outside hospital 84 (4) Other 2 (0.1) Mean (SD)
Age 53.5 (17.3) Length of stay 6.7 (10.0)
Hospitalists from the internal medicine department under study do no routinely collect OI,
and if they do, the patient or their relatives have to authorize the released of patient information
from outside healthcare facilities. As noted in Table 2, 75% of the requests for OI are made
within 22 hours from patient admission and only 10% of the requests are made within 1 hour.
Based on this data, the OI requests were not part of a routine during patient admission, and
they seem to play an important role, perhaps, when the clinical picture of the patient becomes
less clear than initially appeared. The most common health problems and OI requested in the
2,089 hospitalizations under study are presented in Table 3. The majority of the requests for OI
were from rather non-specific health problems such as chest pain, 18.5%, abdominal pain,
15.1%, and dyspnea, 9.9%. This pattern is aligned with the patient population and clinical
setting under study. On the other hand, the most frequent OI requested were outside medical
records with 77.9%, followed by laboratory test results with 18.5% and imaging results with
18.2%. Important to note is that the frequency analysis presented in Table 3 may result in
overlap between the different classes of health problems and outside information types.
Table 2. Analysis of duration from patient admission to when the request for OI was made by a
hospitalist in the Internal Medicine Department of Tampa General Hospital.
Quantile Duration in minutes Duration in hours
Appendix E (continued)
121
100% Max 51,894 865
99% 18,456 308
95% 6,072 101
90% 3,534 59
75% Q3 1,309 22
50% Median 575 10
25% Q1 224 4
10% 49 1
5% 23 0
1% 0 0
0% Min 0 0
Table 3. Common health problems seen and outside information types requested during
hospitalizations in the Internal Medicine Department of the Tampa General Hospital.
Abbreviations: COPD, congestive obstructive pulmonary disease; CHF, congestive heart failure;
EKG, electrocardiogram; GI, Gastrointestinal.
Health Problems Number of hospitalizations (%)
Chest pain 387 (18.5) Abdominal pain 315 (15.1) Anemia 261 (12.5) Dyspnea 206 (9.9) Hypertension 199 (9.5) Diabetes mellitus 195 (9.3) Leukocytosis 182 (8.7) Renal Failure 177 (8.5) Vomiting 152 (7.3) Nausea 150 (7.2) Altered mental status 133 (6.4) Fever 122 (5.8) Cancer 109 (5.2) Tachycardia 107 (5.1) Hypotension 100 (4.8) Lower urinary tract infection 97 (4.6) Hypokalemia 96 (4.6) Hyponatremia 92 (4.4) Back pain 88 (4.2) Syncope 88 (4.2) Coronary artery disease 84 (4.0) Pneumonia 81 (3.9) COPD 78 (3.7) CHF 76 (3.6)
Appendix E (continued)
122
GI bleed 75 (3.6) Cellulitis 73 (3.5) Headache 69 (3.3) Alcohol abuse 69 (3.3) Weakness 66 (3.2) Others Diagnosis 325 (15.6) Outside Information Types
Outside medical records 1635 (77.9) Outside laboratory results 389 (18.5) Outside imaging results 382 (18.2) Outside history and physical test results 255 (12.2) Outside notes 206 (9.8) Outside consultation 173 (8.2) Outside discharge summary 164 (7.8) Outside EKG results 153 (7.3) Outside surgery or procedure notes 151 (7.2)
3.2. Association Rules
The final set of association rules is presented in Table 4. We fixed the minimum support at
2%, minimum confidence at 75%, lift values greater than 1, and the association rules had to
have at least one health problem in the antecedent and one OI type in the consequent. Clinically
relevant rules are presented in Table 4. A total of 20 association rules were found to be clinically
relevant, of which the two with the lowest p-values (rules 3 and 16 in Table 4) do not satisfy 𝑝 <
0.01. By the Benjamini-Hochberg correction method, we concluded that since 0.01 = (2/20)0.1,
these two results are not statistically significant at the corrected level 𝑃 < 0.1. All of the rules
were determined by chi square analysis and Benjamini-Hochberg correction not to be
significant. Although our conservative approach resulted in no statistically sound association
rules, there seems to be a trend between health problems and OI types for specific patient
cohorts. For example, in terms of support, the stronger association rules found are {abdominal
pain → outside medical records} and {anemia → outside medical records}. That is, outside
medical records are frequently requested for abdominal pain and anemia patients with a support
of 12% and 10%, respectively. When requesting OI for abdominal pain patients, there is an 83%
confidence of requesting outside medical records. Similarly for anemia patients, there is an 80%
Appendix E (continued)
123
confidence of requesting outside medical records. The Internal Medicine Department usually
serves people carrying several chronic conditions as comorbidities of an acute condition.
Hence, most of the requests for outside medical records were for chronically ill patients. Despite
this fact, the collected data show acute cases such as lower urinary tract infections typically
trigger requests for outside medical records as well. For this particular patient cohort, there is an
86% chance of requesting outside medical records. Other acute conditions found among the 20
strong association rules were patients with abdominal pain, chest pain, nausea, and vomiting.
Table 4. The strong association rules between health problems and types of information
requested during hospitalizations in the Internal Medicine Department of the Tampa General
Hospital. Abbreviations: OMR, outside medical record; CHF, congestive heart failure; EKG,
electrocardiogram; BH-FDR, Benjamini-Hochberg false discovery rate.
ID Association Rules
Support Confidence Lift N χ2 Uncorrected P-values
BH-FDR corrected p-values
1 Abdominal pain → OMR
12% 83% 1.06 261 0.57 0.55 0.10
2 Anemia → OMR 10% 80% 1.03 210 0.09 0.24 0.04 3 Dyspnea → OMR 8% 79% 1.01 163 0.01 0.06 0.01 4 Hypertension →
OMR 8% 81% 1.04 162 0.10 0.25 0.05
5 Diabetes mellitus → OMR
8% 82% 1.04 159 0.10 0.25 0.05
6 Renal failure →
OMR 7% 83% 1.06 147 0.18 0.33 0.07
7 Cancer → OMR 5% 88% 1.13 96 0.34 0.44 0.10 8 Lower urinary
tract infection →
OMR
4% 86% 1.09 83 0.12 0.27 0.03
9 Hypotension → OMR
4% 83% 1.06 83 0.05 0.18 0.06
10 Back pain →
OMR 4% 85% 1.09 75 0.11 0.26 0.06
11 Pneumonia →
OMR 3% 89% 1.14 72 0.18 0.32 0.01
12 Chest pain, Outside imaging
3% 93% 1.19 71 0.31 0.42 0.02
Appendix E (continued)
124
→ OMR
13 Anemia, Outside laboratory results → OMR
3% 93% 1.19 68 0.29 0.41 0.02
14 Abdominal pain, Nausea, OMR
3% 83% 1.06 63 0.03 0.14 0.03
15 Abdominal pain, Vomiting → OMR
3% 85% 1.09 63 0.07 0.20 0.04
16 CHF → OMR 3% 82% 1.04 62 0.01 0.09 0.07 17 Anemia, Outside
imaging → OMR 3% 94% 1.20 58 0.28 0.40 0.08
18 Hypertension, Diabetes mellitus → OMR
3% 85% 1.09 57 0.06 0.19 0.09
19 Abdominal pain, Vomiting, Nausea → OMR
3% 85% 1.08 55 0.05 0.17 0.09
20 Chest pain, Outside EKG →
OMR
2% 98% 1.25 48 0.23 0.37 0.08
4. DISCUSSION
We sought to uncover the relationship between the patients’ health problems and the
information needed from outside health care facilities in a large academic medical center. ARL
was used to mine two and a half years of transactional data from the hospital EMR previous HIE
implementation. Although previous investigations have made valuable contributions to the
knowledge base on informational needs of physicians and patterns of use of HIE systems (e.g.,
[45,46]), most of them focus solely on hospital and primary care provider communication. We
construct on these investigations considering the entire spectrum from which a hospital
physician (i.e., hospitalist) may request patient records. With an increased number of handoffs
between providers [47], due to the shift towards hospital medicine, studying informational needs
of hospitalists becomes essential for improving HIE functionality, and thereby reducing barriers
to adoption. We have also identified an important gap in the literature – most of the HIEs are
built and implemented without first performing a user needs assessment. We believe HIE will be
more successful if it is evaluated before, during and after implementation. To the best of our
Appendix E (continued)
125
knowledge, there is no previous study serving as both needs assessment and baseline of
informational needs prior to HIE implementation. Important to note is that hospitalists working in
the department under study identified specific situations where they know outside information
exists, but they do not request for records. For example, physician assumes the OI request
process takes too long or the patient does not know where to request outside information from.
These situations are amenable to HIE, therefore, physician OI request behavior may change
after HIE implementation. We plan on capturing these variations in a future study.
Previous investigations suggest users have determined HIE is useful in some, but not all
cases [40]. Our results indicate those patients hospitalized with chest pain were the target of
outside information requests to obtain EKG results and other imaging test results. Other patient
cohorts that were a common target of outside information requests were urinary tract infection
patients and back pain patients. Indeed, Bailey and colleagues found HIE usage was associated
with 64% lower odds of repeated imaging testing for back pain patients [6]. These findings can
be translated into HIE design recommendations; for example, HIE systems should provide 1-
click access to imaging, echocardiograms, bacterial cultures, cardiac catheterizations and CT
scans allocated in other healthcare facilities for those patients with acute cardiac issues, urinary
tract infection and back pain. Not only did our results indicate which patient populations are
more prone to have outside records requested, they also indicated where future HIE research
should focus to elucidate the value of information exchange among providers. Still, work lies
ahead in elucidating whether or not streamlined access to outside information improves medical
decision-making for other patient populations, and hence lower health care costs and improve
patient outcomes. Future research should focus on determining the effects of having quick
access to outside information in those patient cohorts previously unexplored; for example,
urinary tract infection patients. Additionally, we would like to point out that few hospital transfers
and physician referrals were included in our study. Since previous research found that
incomplete patient records during transfers may lead to costly duplicated testing (e.g., [48]),
Appendix E (continued)
126
future investigations should focus on the role of HIE during the admission of transferred
patients.
A crucial step in improving information exchange between inpatient and other settings of
care is the discharge summary [49–51]. Although The Joint Commission on Accreditation of
Healthcare Organizations requires a discharge summary for every patient, usually, they do not
provide timely and sufficient information for appropriate care transitions [52–54]. Kripalani and
colleagues, in their 2007 systematic review of deficits in communication and information transfer
between hospitalists and primary care physicians, infer that new health information technology
and standardized methods of information exchange bears particular promise to improve care
coordination [26,45]. Computer-generated summaries offer a quick way to present and highlight
key elements of the hospitalization, and they are ready for delivery sooner than traditional
summaries [55]. However, information needs and collection habits are not generic but instead
vary among different types of physicians. Previous investigations found information needs and
expectations of computers are influenced by specialty and practice setting [28,33,56,57]. Future
research must determine differences between informational needs due to a variety of factors
that include the young physician’s lack of experience with fundamental clinical principles and the
senior physician’s lack of experience with information technology.
We found few other studies analyzing informational needs in the context of information
exchange among healthcare organizations. Two studies, focused on the emergency department
(ED) and outpatient care settings, found most OI users accessed patient summary data
displayed by default in the HIE system followed by detailed laboratory and radiology information,
which is consistent to what we found [58,59]. We contribute to this body of research by focusing
on the inpatient care setting and hospitalists, who are key actors in coordinating the care of the
patient within and outside the hospital. Ozkaynak and Brennan, during direct observation of ED
workflows, found clinicians were more likely to request OI for admissions of chronic pain
Appendix E (continued)
127
patients [60], which is consistent with our findings as well. However, during follow-up interviews,
they found ED clinicians requested OI to identify drug seekers, which may not be the same
motivation of hospitalists. Further research should explore hospitalists’ perceptions on the value
of OI to support medical decision-making.
There are important limitations to our work. First, we do not know if information-seeking
efforts of hospitalists were successful. The collected transactional data have no information on
whether or not the user located the desired information. Second, our study was restricted to a
single hospital and thus a single EMR. However, most of the features of the in-use EMR were
the same as the majority of hospitals across the nation. Third, the results of this work have
limited generalizability in terms of the setting of care. Information users from other settings of
care, even within the same hospital, may have different information needs. Yet, in the presence
of data, our methodological approach can be reproduce to elucidate information needs in other
clinical settings. Fourth, the usage of direct communication to verbally request OI (i.e.,
telephone call to the outside healthcare provider), which is then directly documented by the
clinician in the patient’s medical record were not included in this study. Finally, we did not
address potential confounding due to region characteristics (e.g., the number of unaffiliated
outside healthcare providers and their electronic medical record adoption rates).
5. CONCLUSION
We proposed a new approach to study informational needs of clinicians in the context of
HIE. In particular, we uncovered the relationship between health problems and the most critical
information requested, from outside health care facilities, in an internal medicine department of
a tertiary care hospital. After data preparation, a set of disease-information association rules
was built and then validated by clinical experts. This knowledge should inform the design and
implementation of HIE in similar clinical settings, and in the presence of data, our approach can
Appendix E (continued)
128
be used in other clinical settings as well. Our study contributes to fill the existing gap in knowing
and understanding the clinical information needs in the context of new health information
technology. With better knowledge of clinical information needs, it will become possible to
conduct prospective studies of the clinical benefit of providing doctors with decision support
tools that meet their outside information needs. Evidence can then be collected on whether
improved access to outside information will result in more efficient or effective clinical decision-
making or improved patient health outcomes. The effectiveness of health information exchange
can thereby obtain its most eloquent validation.
6. CLINICAL RELEVANCE STATEMENT
Health information exchange is expected to facilitate a better delivery of care to patients.
This study assists that goal by uncovering the most commonly requested clinical information
from outside health care facilities by specific health problems. In the hands of HIE developers
and implementers, our framework may facilitate screen redesign and enhanced record
searching, and thereby reduce clinical workflow disruptions and troubles with the system
interface.
7. CONFLICT OF INTEREST
The authors declare that they have no conflicts of interest in this study.
8. HUMAN SUBJECTS PROTECTIONS
This study was performed in compliance with the World Medical Association Declaration of
Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was
Appendix E (continued)
129
reviewed by the Tampa General Hospital Office of Clinical Research and the University of South
Florida (IRB # Pro00014574).
9. ACKNOWLEDGEMENTS
We thank Drs. Alexandra Strauss, Candice Mateja, and Stephanie Taylor for their valuable
contributions in our study. We also thank Andres Garcia-Arce for his contributions during early
stages of this project. Finally, we would also like to show our gratitude to the four anonymous
reviewers for their comments that greatly improved our manuscript.
10. REFERENCES
1. Ward BW, Schiller JS, Goodman RA (2014) Multiple chronic conditions among US adults:
a 2012 update. Prev Chronic Dis 11: E62. doi:10.5888/pcd11.130389.
2. Anderson G (2010) Chronic Conditions: Making the Case for Ongoing Care. Princeton,
NJ.
3. Lucas DJ, Ejaz A, Haut ER, Spolverato G, Haider AH, Pawlik TM (2014) Interhospital
transfer and adverse outcomes after general surgery: implications for pay for
performance. J Am Coll Surg 218: 393–400. Available:
http://www.ncbi.nlm.nih.gov/pubmed/24468232. Accessed 2 September 2014.
4. Kho AN, Lemmon L, Commiskey M, Wilson SJ, McDonald CJ (2008) Use of a Regional
Health Information Exchange to Detect Crossover of Patients with MRSA between Urban
Hospitals. J Am Med Informatics Assoc 15: 212–216. doi:10.1197/jamia.M2577.
Appendix E (continued)
130
5. Bailey JE, Wan JY, Mabry LM, Landy SH, Pope RA, Waters TM, Frisse ME (2013) Does
health information exchange reduce unnecessary neuroimaging and improve quality of
headache care in the emergency department? J Gen Intern Med 28: 176–183. Available:
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3614141&tool=pmcentrez&ren
dertype=abstract. Accessed 16 January 2015.
6. Bailey JE, Pope R a, Elliott EC, Wan JY, Waters TM, Frisse ME (2013) Health
Information Exchange Reduces Repeated Diagnostic Imaging for Back Pain. Ann Emerg
Med. Available: http://www.ncbi.nlm.nih.gov/pubmed/23465552. Accessed 13 March
2013.
7. Unertl KM, Johnson KB, Lorenzi NM (2012) Health information exchange technology on
the front lines of healthcare: workflow factors and patterns of use. J Am Med Inform
Assoc 19: 392–400. Available:
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3341790&tool=pmcentrez&ren
dertype=abstract. Accessed 10 September 2012.
8. Carr CM, Krywko DM, Moore HE, Saef SH (2012) The Impact of a Health Information
Exchange on the Management of Patients in an Urban Academic Emergency
Department: An Observational Study and Cost Analysis. Ann Emerg Med 60: S15.
Available: http://dx.doi.org/10.1016/j.annemergmed.2012.06.062.
9. Vest JR, Miller TR (2011) The association between health information exchange and
measures of patient satisfaction. Appl Clin Inform 2: 447–459. Available:
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3612996&tool=pmcentrez&ren
dertype=abstract. Accessed 6 June 2013.
Appendix E (continued)
131
10. Solberg D, Roberts J (2009) “Pipe dream” HIE proves challenging. A community hospital
network, concerned that each clinic’s needs could not be entirely met, decided on a
standard EHR platform and a shared community network. Health Manag Technol 30: 22–
23, 30.
11. Furukawa MF, King J, Patel V, Hsiao C-J, Adler-Milstein J, Jha AK (2014) Despite
Substantial Progress In EHR Adoption, Health Information Exchange And Patient
Engagement Remain Low In Office Settings. Health Aff (Millwood): hlthaff.2014.0445 – .
Available: http://content.healthaffairs.org/content/early/2014/08/05/hlthaff.2014.0445.
12. Adler-Milstein J, Jha AK (2014) Health information exchange among U.S. hospitals:
Who’s in, who's out, and why? Healthcare 2: 26–32. Available:
http://www.sciencedirect.com/science/article/pii/S2213076414000025. Accessed 8
January 2015.
13. Rudin RS, Motala A, Goldzwelg CL, Shekelle PG (2014) Usage and Effect of Health
Information Exchange: A Systematic Review. Ann Intern Med 161: 803–812.
14. Richardson JE, Abramson EL, Kaushal R (2012) The value of health information
exchange. J Healthc Leadersh 4: 17–23.
15. Vest JR, Jasperson ’Jon Sean, Zhao H, Gamm LD, Ohsfeldt RL (2011) Use of a health
information exchange system in the emergency care of children. BMC Med Inform Decis
Mak 11: 78. Available:
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3295672&tool=pmcentrez&ren
dertype=abstract. Accessed 10 September 2012.
Appendix E (continued)
132
16. Hincapie AL, Warholak TL, Murcko AC, Slack M, Malone DC (2011) Physicians’ opinions
of a health information exchange. J Am Med Inform Assoc 18: 60–65.
doi:10.1136/jamia.2010.006502.
17. Kassirer JP (1998) Doctor discontent. N Engl J Med 339: 1543–1545.
doi:10.1056/NEJM199811193392109.
18. Fischman J (2005) Whi will take care of you? Issues: 31.
19. Mechanic D (2003) Physician discontent: challenges and opportunities. JAMA 290: 941–
946. doi:10.1001/jama.290.7.941.
20. Morrison I, Smith R (2000) Hamster health care. BMJ Br Med J 321: 1541–1542.
doi:10.1136/bmj.321.7276.1541.
21. Morrison I (2000) The Future of Physicians’ Time. Ann Intern Med 132: 80. Available:
http://annals.org/article.aspx?doi=10.7326/0003-4819-132-1-200001040-00013.
22. Trude S (2003) So much to do, so little time: physician capacity constraints, 1997-2001.
Track Rep: 1–4.
23. Sicotte C, Paré G (2010) Success in health information exchange projects: Solving the
implementation puzzle. Soc Sci Med 70: 1159–1165. Available:
https://www.healthypeople.gov/2020/topics-objectives/topic/health-communication-and-
health-information-technology.
24. Karsh B-T (2009) Clinical Practice Improvement and Redesign: How Change in Workflow
Can Be Supported by Clinical Decision Support: 1–34. Available:
Appendix E (continued)
133
http://www.nachc.com/client/Clinical Practice Improvement and Redesign_How Workflow
can Support CDS.pdf.
25. Stead WW, Lin HS (2009) Computational Technology for Effective Health Care :
Immediate Steps and Strategic Directions. Program 22: 121. Available:
http://books.google.com/books?hl=en&lr=&id=HUAXQzIPxV8C&oi=fnd&a
mp;pg=PT1&dq=Computational+teChnology+for+effeCtive+health+Care+immediate
+StepS+a+n+d+StrategiC+direCtionS&ots=Sh4Pzoy3Fh&sig=P1WQnADNNP
GkSeFLgy92VF4Ssv0.
26. Kripalani S, LeFevre F, Phillips CO, Williams M V, Basaviah P, Baker DW (2007) Deficits
in communication and information transfer between hospital-based and primary care
physicians: implications for patient safety and continuity of care. JAMA 297: 831–841.
doi:10.1001/jama.297.8.831.
27. Stross JK, Harlan WR (1979) The dissemination of new medical information. JAMA 241:
2622–2624. doi:10.1001/jama.1979.03290500030017.
28. Strasser TC (2012) The information needs of practicing physicians in Northeastern New
York State. J Med Libr Assoc 100: G. Available:
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3571668&tool=pmcentrez&ren
dertype=abstract.
29. Curry L, Putnam RW (1981) Continuing medical education in Maritime Canada: The
methods physicians use, would prefer and find most effective. Can Med Assoc J 124:
563–566.
Appendix E (continued)
134
30. Cohen SJ, Weinberger M, Mazzuca S a, McDonald CJ (1982) Perceived influence of
different information sources on the decision-making of internal medicine house staff and
faculty. Soc Sci Med 16: 1361–1364. doi:10.1016/0277-9536(82)90032-6.
31. Northup DE, Moore-West M, Skipper B, Teaf SR (1983) Characteristics of Clinical
Information-Searching: Investigation Using Critical Incident Technique. J Med Educ 58:
873–881. Available:
http://journals.lww.com/academicmedicine/Abstract/1983/11000/Characteristics_of_clinic
al_information_searching_.6.aspx.
32. Kochen M, Cohen L, Wulff Y (1985) Information systems and clinical research by
residents in internal medicine. Methods Inf Med 24: 85–90.
33. Stinson ER, Mueller DA (1980) Survey of health professionals’ information habits and
needs. Conducted through personal interviews. JAMA 243: 140–143.
doi:10.1001/jama.243.2.140.
34. Christensen DB, Wertheimer AI (1979) Sources of information and influence on new drug
prescribing among physicians in an HMO. Soc Sci Med 13A: 313–322.
35. Covell DG, Uman GC, Manning PR (1985) Information needs in office practice: are they
being met? Ann Intern Med 103: 596–599. doi:10.1059/0003-4819-103-4-596.
36. Rudin R, Volk L, Simon S, Bates D (2011) What Affects Clinicians’ Usage of Health
Information Exchange? Appl Clin Inform 2: 250–262. Available:
/pmc/articles/mid/NIHMS335831/?report=abstract.
37. Abdullah U, Ahmad J, Ahmed A (2008) Analysis of effectiveness of apriori algorithm in
medical billing data mining. Proceedings - 4th IEEE International Conference on
Appendix E (continued)
135
Emerging Technologies 2008, ICET 2008. pp. 327–331.
doi:10.1109/ICET.2008.4777523.
38. Sharma N, Om H (2014) Extracting Significant Patterns for Oral Cancer Detection Using
Apriori Algorithm. Intell Inf Manag 06: 30–37. Available:
http://www.scirp.org/journal/PaperDownload.aspx?DOI=10.4236/iim.2014.62005.
39. Ilayaraja M, Meyyappan T (2013) Mining medical data to identify frequent diseases using
Apriori algorithm. Proceedings of the 2013 International Conference on Pattern
Recognition, Informatics and Mobile Engineering, PRIME 2013. pp. 194–199.
doi:10.1109/ICPRIME.2013.6496471.
40. Vest JR, Zhao H, Jasperson J, Jaspserson J, Gamm LD, Ohsfeldt RL (2011) Factors
motivating and affecting health information exchange usage. J Am Med Inform Assoc 18:
143–149. Available:
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3116259&tool=pmcentrez&ren
dertype=abstract. Accessed 1 August 2012.
41. Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items
in large databases. ACM SIGMOD Rec 22: 207–216. doi:10.1145/170036.170072.
42. Hahsler M, Buchta BG, Hornik K (2015) Arules: Mining Association Rules and Frequent
itemsts. Available: https://cran.r-project.org/web/packages/arules/index.html.
43. Alvarez SA (2003) Chi-squared computation for association rules: preliminary results.
Chestnut Hill, MA.
44. Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and
powerful approach to multiple testing. J R Stat Soc Ser B 57: 289–300.
Appendix E (continued)
136
45. Smith K (2014) Effective communication with primary care providers. Pediatr Clin North
Am 61: 671–679. doi:10.1016/j.pcl.2014.04.004.
46. Koopman RJ, Steege LMB, Moore JL, Clarke MA, Canfield SM, Kim MS, Belden JL
(2015) Physician Information Needs and Electronic Health Records (EHRs): Time to
Reengineer the Clinic Note. J Am Board Fam Med 28: 316–323. Available:
http://www.jabfm.org/cgi/doi/10.3122/jabfm.2015.03.140244.
47. Landrigan CP, Conway PH, Edwards S, Srivastava R (2006) Pediatric hospitalists: a
systematic review of the literature. Pediatrics 117: 1736–1744. doi:10.1542/peds.2005-
0609.
48. Stewart BA, Fernandes S, Rodriguez-Huertas E, Landzberg M (2010) A preliminary look
at duplicate testing associated with lack of electronic health record interoperability for
transferred patients. J Am Med Inform Assoc 17: 341–344.
doi:10.1136/jamia.2009.001750.
49. Pantilat SZ, Lindenauer PK, Katz PP, Wachter RM (2001) Primary care physician
attitudes regarding communication with hospitalists. Am J Med 111: 15–20.
doi:10.1016/S0002-9343(01)00964-0.
50. Schabetsberger T, Ammenwerth E, Andreatta S, Gratl G, Haux R, Lechleitner G,
Schindelwig K, Stark C, Vogl R, Wilhelmy I, Wozak F (2006) From a paper-based
transmission of discharge summaries to electronic communication in health care regions.
Int J Med Inform 75: 209–215. doi:10.1016/j.ijmedinf.2005.07.018.
51. Laye PG (1997) Tying up loose ends. Thermochim Acta 300: 237–245.
doi:10.1016/S0040-6031(97)00076-2.
Appendix E (continued)
137
52. El-Kareh R, Roy C, Brodsky G, Perencevich M, Poon EG (2011) Incidence and predictors
of microbiology results returning postdischarge and requiring follow-up. J Hosp Med 6:
291–296. doi:10.1002/jhm.895.
53. Roy CL, Poon EC, Karson AS, Ladak-Merchant Z, Johnson RE, Maviglia SM, Gandhi TK
(2005) Patient safety concerns arising from test results that return after hospital
discharge. Ann Intern Med 143: 121–128. doi:10.1016/j.athoracsur.2010.09.032.
54. Walz SE, Smith M, Cox E, Sattin J, Kind AJH (2011) Pending laboratory tests and the
hospital discharge summary in patients discharged to sub-acute care. J Gen Intern Med
26: 393–398. doi:10.1007/s11606-010-1583-7.
55. Van Walraven C, Laupacis A, Seth R, Wells G (1999) Dictated versus database-
generated discharge summaries: A randomized clinical trial. Cmaj 160: 319–326.
56. Teach RL, Shortliffe EH (1981) An analysis of physician attitudes regarding computer-
based clinical consultation systems. Comput Biomed Res 14: 542–558.
doi:10.1016/0010-4809(81)90012-4.
57. Singer J, Sacks HS, Lucente F, Chalmers TC (1983) Physician attitudes toward
applications of computer data base systems. JAMA 249: 1610–1614.
doi:10.1001/jama.1983.03330360050035.
58. Campion TR, Edwards AM, Johnson SB, Kaushal R (2013) Health information exchange
system usage patterns in three communities: practice sites, users, patients, and data. Int
J Med Inform 82: 810–820. Available:
http://www.sciencedirect.com/science/article/pii/S1386505613001056. Accessed 28
November 2014.
Appendix E (continued)
138
59. Johnson KB, Unertl KM, Chen Q, Lorenzi NM, Nian H, Bailey J, Frisse M (2011) Health
information exchange usage in emergency departments and clinics: the who, what, and
why. J Am Med Inform Assoc 18: 690–697. doi:10.1136/amiajnl-2011-000308.
60. Ozkaynak M, Brennan PF (2012) Revisiting sociotechnical systems in a case of
unreported use of health information exchange system in three hospital emergency
departments. J Eval Clin Pract: 1–4. Available:
http://www.ncbi.nlm.nih.gov/pubmed/22420774. Accessed 27 September 2012.
Appendix E (continued)
139
Appendix F: A Strategic Gaming Model for Health Information Exchange Markets
Appendix F presents the manuscript titled, "A Strategic Gaming Model for Health Information
Exchange Markets", which is under review in the Journal of the American Medical Informatics
Association.
140
A STRATEGIC GAMING MODEL FOR HEALTH INFORMATION EXCHANGE MARKETS
Diego A. Martinez, Felipe Feijoo, Jose L. Zayas Castro, Tapas K. Das
Preprint submitted to the Journal of the American Medical Informatics Association
ABSTRACT
Objective: To describe a mathematical model for estimating the willingness of health care
organizations to adopt HIE under different scenarios of federal incentives and health information
blocking, and to demonstrate its use in HIE policy design.
Methods: We built a bi-level integer program (BiIP), in which the upper-level emulates the
hospital decision of adopting HIE, and the lower-level emulates the patient decision of switching
hospital. Multi-hospital Nash equilibria, in which each hospital solves the BiIP, are calculated
and interpreted as the willingness of a hospital to adopt HIE based on its competitors decision.
We applied our model to 1,093,177 patient encounters over a 7.5-year period in nine hospitals
geographically located within three adjacent counties in Tampa, Florida.
Results: For this community and under a particular set of assumptions, hospitals may set HIE
adoption decisions to threaten the value of HIE even with federal monetary incentives in place.
Medium-sized hospitals are more reticent to adopt HIE compared to large-sized institutions.
Collusions to not join HIE significantly reduce the effectiveness of current and proposed federal
incentive structures.
Discussion: Although health information blocking is commonly attributed to health IT
developers, health care providers may also become a significant barrier for nationwide HIE.
Smaller hospitals are more reticent to HIE, which may be attributed to market share loses and
limited HIE adoption budgets and health IT infrastructure. Competition between hospitals
coupled with volume-based payment systems create no incentives for smaller hospitals to
exchange their data with competitors.
Appendix F (continued)
141
Conclusion: Our model can be used by policy makers to find incentive structures that will spur
HIE participation in a given community. Although the recent shift from volume- to value-based
medicine may amplify the benefits of HIE for providers, medium-sized hospitals need targeted
actions to mitigate market incentives to not adopt HIE.
1. BACKGROUND AND SIGNIFICANCE
Over the next 10 years, it is expected that all health care organizations in the United States be
able to exchange electronic patient data through health information exchange (HIE) with
affiliated and unaffiliated organizations. From the late 1990s, relevant stakeholders and the
research community have recommended that all electronic medical record systems (EMR) be
interoperable to facilitate care coordination and cost savings.[1,2] The federal government has
taken an active role to stimulate such interconnectivity. Enacted in 2009, the Health Information
Technology for Economic and Clinical Health (HITECH) Act has been providing a base incentive
of $2,000,000 for those hospitals electronically exchanging patient information with unaffiliated
providers. Although recent evidence shows mixed results about the positive impact of HIE, two
recent systematic reviews suggest it may be due to a lack of widespread HIE adoption.[3,4]
There has been an uptick in HIE adoption since the enactment of the HITECH Act, however
only 30% of hospitals and 14% of solo practices are conducting HIE activities with significant
state-to-state variations.[5,6] Common barriers to HIE adoption include interface and workflow
issues, privacy and security concerns of patient data, and the financial sustainability of
organizations facilitating information exchange.[7–11] A less studied but equally important
barrier is the strategic role of “owning” patient information.
A recent report from the Office of the National Coordinator for Health Information
Technology (ONC) establishes that current market conditions create incentives for some entities
to exercise control over patient data in ways that unreasonably limit its availability and use.[12]
Appendix F (continued)
142
This issue, named health information blocking, is used as a mean of locking-in patients to
enhance market share and reinforce market dominance of established entities. Empirical and
modeling studies on HIE capabilities and trends provide the necessary context for
understanding the nature and extent of health information blocking. Recent evidence shows that
large for-profit hospitals are less likely to adopt HIE compared to non-profit hospitals and
hospitals with no significant market share or with operations in less concentrated markets.[5]
Another study found large health systems more likely to exchange electronic patient data
internally but are less likely to exchange with competitors and unaffiliated providers.[13]
Although providers are legally required to share patients’ records, there is also anecdotal
evidence that providers are hesitant to release records to patients transferring to other
providers.[12,14–17] Hospital administration have outlined concerns about losing competitive
advantages by ceding full control of “their” data.[18] While the evidence is limited, there is little
doubt that health information blocking is occurring and is interfering with nationwide HIE.
Various modeling studies on HIE have been undertaken to study HIE network structure and
financial sustainability.[36–42] However, only a few have focused on issues related to health
information blocking and the strategic decision of adopting HIE. Zhu and colleagues proposed a
game theoretic approach to studying the strategic behavior of data owners and HIE
adoption.[43] Desai developed a game theoretical model to analyze the potential loss of
competitive advantage due to HIE adoption.[20] A crucial difference among these studies on
health information blocking is the type of interaction assumed between hospitals and patients,
and among competing hospitals. In hospital competition focused models, hospital interactions
can be summarized in terms of conjectural variation (i.e., each hospital’s decision to adopt HIE
is predicated on the way it perceives its competitors may react). The proposed model, unlike
previous approaches, calculate oligopolistic equilibriums of HIE adoption using the hospital
utility function conjectural variations while considering the discrete range patients’ options of
Appendix F (continued)
143
where to purchase health care services. The resulting bi-level mathematical program can be
used to deepen our understanding of health information blocking under different market
structures. More importantly, policy makers can use our model to answer the fundamental
question of, what should be the optimal levels of federal incentives that will spur HIE adoption
across United States?
2. OBJECTIVE
There is a need of stronger and targeted policy that stimulates competing health care
organizations to adopt HIE. Our objective is to describe a mathematical model for estimating the
willingness of health care organizations to adopt HIE, which considers different levels of federal
incentive structures and health information blocking.
3. MATHERIAL AND METHODS
3.1. Market assumptions
In our model, we establish a finite number of hospitals serving a finite number of patients.
Hospitals decide whether or not to adopt HIE. The patient then decides whether or not to switch
the hospital where they consume health care services based upon an extension of the utility
function used in [20]. By not adopting HIE, hospitals may be able to increase their patient
volume and profit by reducing patient migration to other hospitals. Alternatively, by adopting
HIE, hospitals may increase volume and profit by treating patients migrating from other hospitals
and by receiving marginal benefits of joining an HIE network. In a community served by a multi-
hospital system, a Nash equilibrium will occur when no hospital has any incentive to unilaterally
change its HIE adoption decision. The model presented in [20] is similar to ours, except for two
differences. The first difference is that in [20] a duopoly market is assumed—the multi-hospital
equilibriums are not calculated neither discussed. We instead consider reactions of more than
two competing hospitals in a given community, which we argue is a more realistic
Appendix F (continued)
144
representation of HIE markets. Second, our model are constrained by hospital HIE adoption
budgets and by patient allocation needs, i.e., patients in our model have specific care needs
that cannot be served by every hospital (see Section 3.3 for further details).
In this model, we assume all hospitals are for-profit institutions maximizing expected
payoffs. The hospitals have a designated budget for HIE implementation, and must not run a
budget deficit. We assume only the hospitals manipulate the decision to adopt HIE. On the other
hand, patients are considered to maximize their utilities, which are measured in terms of the
quality of care offered by each hospital, the personal preference each patient has for each
hospital, and the switching costs generated at the time of moving health information from one
health care provider to another one. We assume all patients purchase medical insurance, and
thereby they are insensitive to price changes on health care services.[21] The timing of the
model timing is as follows. First, patients are randomly assigned to a hospital (index hospital)
with imperfect information about their personal hospital preference. Second, patients learn their
hospital preference perfectly, and we assume the prospect of the hospital adopting HIE causes
no impact on the patient’s utility function. Third, hospitals decide whether or not to adopt HIE.
Finally, patients decide whether or not to switch the index hospital. If the index hospital decides
to adopt HIE, then the switching costs for the patient are reduced to zero. We also assume that
patient switching costs are reduced to zero even if only the index hospital decides to adopt HIE.
We have developed two utility-based models representing the interactions of hospitals
and patients in a health care delivery market in the context of HIE. The bi-level model can be
phrased as follows. There are some dominant hospitals in the market, each deciding whether or
not to adopt HIE. The model tries to determine the optimal HIE adoption decision for each
hospital. Hospitals can be thought of as a leader of a Stackelberg game, and the leader
calculates its decision based on anticipating what the patients in a given community would do.
Appendix F (continued)
145
The patients’ assumed reactions are based on their utility functions and are considered by
solving one integer program representing the patient’s purchase decision.
3.2. Mathematical formulation
Mathematically, the HIE market can be formulated as an oligopolistic market equilibrium model
on a network consisting of the node sets 𝐼 and 𝐽, where the set 𝐼 corresponds to the hospitals in
a given community and the set 𝐽 corresponds to the patients served by the multi-hospital
network. There are several hospitals in the market, each serving specific members of the
population. In this section, we give the precise formulation of the single-hospital problem, and
the solution strategy for a multi-hospital problem.
3.3. The single-hospital problem
In essence, the single-hospital problem is a two-level constrained optimization problem in which
a hospital takes as inputs its perceived market conditions (including any competitors’ service
and demand functions) and maximizes profit under a set of equilibrium constraints. In the
terminology of a bi-level optimization problem, the upper-level variables consist of the hospital’s
decision to adopt HIE and the lower-level is the patient’s decision as to switch hospital. The
upper-level problem is parameterized by the patient’s willingness to switch which is restricted by
given bounds; such bounds constitute the upper-level constraints. The upper-level objective is
the hospital’s profit, equal to its revenues less its costs.
The single-hospital problem focuses on a hospital denoted by 𝑖∗ ∈ 𝐼. The following is the
notation used in the formulation of this problem.
Sets:
𝐼 Set of all hospitals
Appendix F (continued)
146
𝐽 Set of all patients
𝑇𝑗 Set of all hospitals where patient 𝑗 cannot purchase health care services
Indices:
𝑖 Hospital in the network
𝑗 Patient in the network
Parameters:
𝛼 A scalar
𝑣𝑖 Vertical quality component for hospital 𝑖
𝑟𝑖𝑗 Personal preference for hospital 𝑖 by patient 𝑗
𝑠 Switching cost
𝑝 Price of service
𝑞𝑖 Number of patients served by hospital 𝑖
𝑓𝑖 Quantity of federal monetary incentive for adopting HIE
𝛽𝑖 Marginal benefit per patient a hospital 𝑖 receives from HIE
𝐶𝐻𝐼𝐸 Fixed HIE adoption cost
𝐵𝑖 Budget allocated by hospital 𝑖 for HIE adoption
Lower-level patient decision variables:
𝑡𝑖𝑗 1 if patient 𝑗 consumes from hospital 𝑖 and 0 otherwise
𝑦𝑖𝑗 1 if patient 𝑗 migrates from hospital 𝑖 and 0 otherwise
Upper-level hospital decision variables:
𝑒𝑖 1 if hospital 𝑖 adopts HIE and 0 otherwise
Appendix F (continued)
147
The lower-level patient switching problem is formally stated as the following
mathematical program in variable 𝑡𝑖𝑗 and 𝑦𝑖𝑗, parametrized by decision 𝑒𝑖 for 𝑖 ∈ 𝐼.
Maximization of patient’s payoff
max𝑡𝑖𝑗,𝑦𝑖𝑗∈{0,1}
2∑∑𝑡𝑖𝑗[𝛼(𝑣𝑖 + 𝑟𝑖𝑗) − (1 − 𝑒𝑖)𝑠]
𝑗𝑖
(1)
constrained by the set of hospitals to which a patient cannot migrate due to special
health care needs: for all patients 𝑗 ∈ 𝐽,
∑ 𝑡𝑖𝑗 = 0
𝑖∈𝑇𝑗
(2)
by the migration of a patient to a unique hospital: for all patients 𝑗 ∈ 𝐽,
∑𝑡𝑖𝑗 = 𝑦𝑖∗𝑗𝑖≠𝑖∗
(3)
and, by the binary decision variables
𝑡𝑖𝑗 , 𝑦𝑖𝑗 ∈ {0,1}2 (4)
With the lower-level problem defined, we may now complete the upper-level problem
that hospital 𝑖∗ ∈ 𝐼 solves to determine its decision of adopting HIE. Specifically, taking 𝑡𝑖𝑗 and
𝑦𝑖𝑗 for all 𝑗 ∈ 𝐽 as given, hospital 𝑖∗ ∈ 𝐼 maximizes its payoff.
Maximization of hospital’s profit
max𝑒𝑖∈{0,1}
𝑝 [𝑞𝑖∗ +∑𝑡𝑖∗𝑗𝑗
−∑𝑦𝑖∗𝑗𝑗
] + 𝑒𝑖 [𝛽𝑖 (𝑞𝑖∗ +∑𝑡𝑖∗𝑗𝑗
−∑𝑦𝑖∗𝑗𝑗
) − 𝐶𝐻𝐼𝐸 + 𝑓𝑖] (5)
constrained by the budget that each hospital allocates for HIE adoption: for all hospitals
𝑖 ∈ 𝐼,
Appendix F (continued)
148
𝑒𝑖[𝐶𝐻𝐼𝐸 − 𝑓𝑖] ≤ 𝐵𝑖 (6)
and by the binary decision variables
𝑒𝑖 ∈ {0,1}. (7)
Rewriting the resulting formulation (1) – (7), we obtain the following bi-level integer
program, to which we refer as BiIP. The upper-level of problem (8) represents the interest of
hospital 𝑖 ∈ 𝐼, while the lower-level represents the interest of patient 𝑗 ∈ 𝐽. The hospital is
classified as leader of the bi-level program, and the patients are classified as followers.
BiIP:
max𝑒𝑖∈{0,1}
𝑝 [𝑞𝑖∗ +∑𝑡𝑖∗𝑗𝑗
−∑𝑦𝑖∗𝑗𝑗
]
+𝑒𝑖 [𝛽𝑖 (𝑞𝑖∗ +∑𝑡𝑖∗𝑗𝑗
−∑𝑦𝑖∗𝑗𝑗
) − 𝐶𝐻𝐼𝐸 + 𝑓𝑖]
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑒𝑖[𝐶𝐻𝐼𝐸 − 𝑓𝑖] ≤ 𝐵𝑖, ∀𝑖,
𝑒𝑖 ∈ {0,1},
(𝑡𝑖∗𝑗, 𝑦𝑖∗𝑗) ∈ max𝑡𝑖𝑗,𝑦𝑖𝑗∈{0,1}
2
{
∑∑𝑡𝑖𝑗[𝛼(𝑣𝑖 + 𝑟𝑖𝑗) − (1 − 𝑒𝑖)𝑠]
𝑗𝑖
:
∑ 𝑡𝑖𝑗 = 0, ∀𝑗
𝑖∈𝑇𝑗
,∑ 𝑡𝑖𝑗 = 𝑦𝑖∗𝑗, ∀𝑗
𝑖≠𝑖∗
,
𝑡𝑖𝑗 , 𝑦𝑖𝑗 ∈ {0,1}2
}
.
(8)
3.4. Solution strategy for the single and multi-hospital problem
Bi-level optimization models have been widely used to study strategic behavior of market
participants in different markets.[22–24] Bi-level models include two mathematical programs,
Appendix F (continued)
149
where one serves as a constraint on the other. For a lower level model, with convex and
feasible space and objective function, the first order necessary conditions represent a solution
for the model.[25] The model presented in Section 3.3 does not comply with these assumptions
since the lower-level model is a non-convex model due to the presence of integer decision
variables. A number of solution approaches have been discussed to tackle problems of this
type. However, most of these approaches do not necessarily guarantee a solution to be
optimal,[26] and if they do, computational requirements are cost prohibitive for large problems
as the one under study.[27]
To guarantee that an optimal solution is obtained for the bi-level formulation presented in
Section 3.3, the bi-level model is solved in two steps. First, we fixed the hospital’s decision of
whether to adopt (𝑒𝑖 = 1) or not (𝑒𝑖 = 0) HIE; after that, given the hospital’s decision, the lower
level model becomes a single level mixed integer problem, which can be solved independently.
Once the lower level model is solved for both each possible value of 𝑒𝑖, the optimal solution for
hospital 𝑖 ∈ 𝐼 can be obtained by choosing the maximum between 𝐹(𝑒𝑖 = 1) and 𝐹(𝑒𝑖 = 0),
where 𝐹(𝑒𝑖) represents the profit of hospital 𝑖 ∈ 𝐼.
When multiple hospitals participate in the HIE market, the equilibrium strategies among
those hospitals need to be obtained. In this context, each hospital faces and needs to solve the
bi-level model. Since the bi-level solution approach considers testing each possible hospital
strategy, the game and the corresponding market equilibrium can be formulated as a matrix
game. Each position in the matrix game represents the profit of each hospital for a unique
combination of strategies 𝐸(𝑒1, 𝑒2, 𝑒3, … , 𝑒𝑖). The representation of the matrix game and solution
approach for obtaining the market equilibrium is presented in Figure 1.
Appendix F (continued)
150
Figure 1. Diagram of the solution approach for obtaining market equilibrium in a multi-hospital
problem. Abbreviations: HIE, health information exchange.
As stated earlier, each position in the matrix game represents a combination of strategies
𝐸(𝑒1, 𝑒2, 𝑒3, … , 𝑒𝑖) of the hospitals. In order to obtain an equilibrium, we evaluate each
combination of these strategies in the lower-level problem and calculate the profit for each
hospital according to the hospital’s objective function described in section 3.3. Once each
possible strategy combination in the matrix is populated with the corresponding hospitals’
profits, the equilibrium can be obtained. A strategy profile 𝐸∗(𝑒1∗, 𝑒2
∗, 𝑒3∗, . . . , 𝑒𝑖
∗) is a Nash
equilibrium (NE) if no unilateral deviation in strategy by any single player is profitable for that
player. That is, the strategy 𝐸∗(𝑒1∗, 𝑒2
∗, 𝑒3∗, . . 𝑒𝑖
∗) is said to be a NE if:
∀𝑖 ∈ 𝐼, 𝐹𝑖(𝑒𝑖∗, 𝑒−𝑖
∗ ) ≥ 𝐹𝑖(𝑒𝑖, 𝑒−𝑖∗ )
If a pure NE cannot be found, a mixed strategy NE can be always found based as proven by
[28]. A mixed strategy NE assigns a probability distribution to the set of strategies that hospitals
can take. The probability distribution is understood in our context as the willingness of hospitals
to join and HIE network.
Appendix F (continued)
151
4. RESULTS
We now illustrate how the proposed model can assist the analysis of HIE markets and
development of HIE policy. Using a sample hospital network, the model can be used to assess
HIE adoption levels in a given region under various scenarios of federal monetary incentives, as
well as different levels of health information blocking (i.e., collusions to avoid HIE adoption). We
conduct three numerical studies to answer the following questions: 1) How will HIE adoption
rates be affected by federal incentives? 2) How will HIE adoption be affected by market power?
3) What degree of market power results in significant market inefficiencies that should be
mitigated? To answer the first question, we evaluated a set of existing federal incentive
structures and a set of proposed penalties. To answer the second and third questions, we
simulated collusions by randomly assigning a subset of hospitals to not adopt HIE. In our model,
the number of hospitals in the fictitious collusions varies as in the following levels: none, no
hospitals colluded; minor, two hospitals colluded; moderate, four hospitals colluded; severe, six
hospitals colluded; and extreme, eight hospitals colluded. We then evaluated the impact of the
collusion level on the other hospitals’ willingness to engage in HIE. We also use the moderate
collusion scenario for evaluating a number of ad-hoc incentive structures that vary within current
incentives and proposed penalties. These experiments allow a deeper understanding of the
effectiveness of existing and proposed actions to promote HIE adoption.
4.1. Sample hospital network and model validation
For the numerical studies proposed above, patient flow data were collected from administrative
claims of nine hospitals geographically located within three adjacent counties in Tampa, Florida.
Hospitals with 88-218 beds were classified as medium-sized and those with more than 218
beds as large-sized. The dataset includes 1,093,177 patient encounters (594,751 unique
patients) from January 2005 to July 2012. The vertical quality component of each hospital, 𝑣𝑖,
Appendix F (continued)
152
and the patients’ personal preferences, 𝑟𝑖𝑗, are randomly generated in the interval [0,1]. The
switching cost is assumed to be $50, and the average price of service is set to $9,700 as
presented in [29]. To be conservative, the marginal benefit per patient a hospital 𝑖 receives from
HIE are set to 60-70% of the values presented in [30] of $26 per admission, so at least we
account for HIE benefits in those encounters initialized through the emergency departments.
The federal monetary incentives given to each hospital for HIE adoption are up to $2,000,000.
[31] Since evidence on the costs of HIE adoption are scarce, we set HIE adoption cost at
$900,000 based on anecdotal evidence. [32] Finally, the HIE adoption budget of each hospital 𝑖
is randomly generated in the interval [800000, 1000000]. Hospital network characteristics and
model parameters are summarized in Table 1.
Table 1. Hospital network characteristics and model parameters. Medium-sized hospital, 88-218
beds; large-sized hospital, >218 beds.
Hospital
1 2 3 4 5 6 7 8 9
Size Large Medium
Large Medium
Large Medium
Medium
Large Medium
Average patient volume per year [patients]
4,013 2,162 7,830 1,205 3,425 1,759 2,358 7,813 1,106
𝛼 [$] 150
𝑣𝑖 𝑢𝑛𝑖𝑓(0,1) 𝑟𝑖𝑗 𝑢𝑛𝑖𝑓(0,1)
𝑠 [$] 50
𝑝 [$] 9,700
𝑓𝑖 [$] 2,000,000
𝛽𝑖 [$] 15.36 16.5 13.86 16.47 19.72 13.2 16.12 15.18 16.47
𝐶𝐻𝐼𝐸 [$] 900,000
𝐵𝑖 [$] 882,107
846,300
796,943
731,111
796,010
856,995
852,583
840,047
863,011
Appendix F (continued)
153
To validate the model, we compared the actual versus simulated average patient
volume. The vertical quality components for each hospital, 𝑣𝑖, were manipulated within the [0, 1]
interval until divergences from the actual patient volume were lower than 5% (see Table 2).
Table 2. Model calibration results.
Hospital
1 2 3 4 5 6 7 8 9
Actual average patient volume per year [patients]
4,013 2,162 7,830 1,205 3,425 1,759 2,358 7,813 1,106
Simulated average patient volume, 𝑞𝑖 [patients]
4,072 2,221 8,203 1,230 3,528 1,782 2,465 8,063 1,107
Estimated vertical
quality component, 𝑣𝑖 0.810 0.735 0.925 0.680 0.790 0.715 0.75 0.92 0.675
Error [%] 1.5 2.7 4.8 2.1 3.0 1.3 4.5 3.2 0.1
4.2. Market and policy analysis
The BiIP model was implemented in GAMS and solved using CPLEX.[33] The multi-hospital
Nash equilibrium search was performed using the algorithm presented in [34] and implemented
in MATLAB.[35] Numerical studies are presented next to illustrate the usefulness of the
proposed model.
4.3. How will HIE adoption rates be affected by federal incentives?
To investigate the impact of federal incentives on HIE adoption in the community under study,
we calculated multi-hospital Nash equilibrium under scenarios of penalties of up to $2,000,000
for those hospitals not joining HIE and incentives of up to $2,000,000 for those hospitals joining
HIE. As presented in Figure 2, we found higher sensitivity to penalties than incentives. We also
found that not always a greater incentive (or penalty) is the most effective strategy to promote
HIE adoption. For example, our results suggest that a penalty of $500,000 is more effective than
a penalty of $1,000,000 to generate significant engagement of the hospitals in the community
Appendix F (continued)
154
under study. To investigate these patterns further, we compared the behavior of medium-sized
versus large-sized hospitals. In Figure 3, we can see medium-sized hospitals reticent to adopt
HIE. Possible explanations of such behavior are that medium-sized hospitals are more afraid of
losing significant market share due to patient migration or that they are limited by HIE adoption
budgets and health IT infrastructure. These results are aligned with empirical evidence
suggesting that large hospital systems are more likely to have greater HIE capabilities than
small and single practice providers. [13] In summary, under a particular set of assumptions,
hospitals set HIE adoption decisions to threaten the value of HIE even with federal monetary
incentive structures in place.
Figure 2. Influence of federal monetary incentive structures on promoting HIE engagement in a
community served by nine hospitals. Abbreviations: HIE, health information exchange
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Lik
elih
ood o
f H
IE p
art
icip
ation
Federal monetary incentive structure
Hospital 1
Hospital 2
Hospital 3
Hospital 4
Hospital 5
Hospital 6
Hospital 7
Hospital 8
Hospital 9
Overall
Appendix F (continued)
155
Figure 3. Influence of federal monetary incentive structure on promoting HIE engagement in a
community served by five medium-sized hospitals and four large-sized hospitals. Abbreviations:
HIE, health information exchange.
4.4. Influence of federal monetary incentives on promoting HIE adoption in a
community suffering health information blocking
We now address the following fundamental questions, how will HIE adoption be affected by
health information blocking? What degree of health information blocking results in significant
market inefficiencies that should be mitigated? To investigate further the issue of health
information blocking, we use our model to simulate collusions among a subset of hospitals to
not join HIE, and then evaluate the impact of these collusions on HIE adoption. Collusions are
an agreement between two or more market participants to limit open competition and thereby
gaining an unfair market advantage. In the context of HIE, most stakeholders are committed to
achieve nationwide interconnectivity, but current economic and market conditions create
business incentives for some market participants to exercise unreasonable control over patient
data. Practices of health information blocking include, among others, providers implementing
health IT in non-standard ways that are likely to increase the costs and complexity of electronic
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%A
vera
ge lik
elih
ood o
f H
IE p
art
icip
ation
Federal monetary incentive structure
Medium sized hospitals
Large sized hospitals
Appendix F (continued)
156
exchange of health information. Providers may collude to not join HIE as a means to control
referrals and enhance their market dominance. As presented in Figure 4, we found that
moderate collusions to not join HIE reduce the effectiveness of current (and proposed) federal
incentive structures. Although health information blocking complaints are frequently attributed to
health IT developers, we found health care providers may also become a significant barrier for
nationwide interconnectivity.
Figure 4. Influence of federal monetary incentive structures on promoting HIE engagement in a
community with health information blocking. To simulate the moderate collusion scenario, two
medium-sized (2 & 4) and two large-sized (3 & 8) hospitals were randomly selected and forced
not to adopt HIE. Abbreviations: HIE, health information exchange.
DISCUSSION
The intent of the HITECH Act was to drive the rapid adoption of interoperable EMRs to support
care and efficiency improvements in the United States health care system. While the intent was
and is clear to the majority of stakeholders, some entities are knowingly interfering with
electronic information exchange across disparate and unaffiliated providers to gain market
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Avera
ge lik
elih
ood o
f H
IE e
ngagem
ent
Federal monetary incentive structure
Moderate
None
Appendix F (continued)
157
advantage. We propose a strategic gaming model for assessing health care provider decision to
adopt HIE, which simulates an oligopolistic health care delivery market consisting of several
dominant hospitals. In our model, the interactions between hospitals and patients are modeled
as a Stackelberg game, in which the hospital is the leader, and the patient is the follower. Each
patient decides whether or not to switch the hospital where they consume health care services
based upon an extension of the utility function presented in [20], which includes personal
preferences, perceived hospital quality, and switching costs. As reported in [12], switching costs
may arise when there exists: 1) contract terms, policies or other business practices that restrict
individuals’ access to their electronic health information, 2) fees for data exchange among
providers, and 3) non-standard health IT technologies that increase the costs and complexity
electronic exchange of patient information. We assume that patient switching costs are reduced
to zero when a hospital adopts HIE. Therefore, hospitals not adopting HIE may exercise health
information blocking to increase their profit by reducing patient migration.
A deeper understanding of the role of health information blocking and federal incentives
to promote HIE adoption can help modify and improve current HIE policy. With the increasing
evidence supporting the effect of HIE use on reduced utilization and costs in emergency
departments,[3] there is the need for policies and incentives to stimulate competing
organizations to freely share patient data electronically and minimize health information
blocking. There are several ways to explore, understand, and anticipate the effects of new HIE
policy. First, ex-post analysis of current markets to empirically determine whether or not
hospitals are engaged in HIE (e.g., [6]) Second, ex-ante analysis of market concentration using
the Herfindahl-Hirschman Index (e.g., [19]), which focuses on hospital market share and ignores
HIE adoption costs and health information blocking. Third, ex-ante experimental analysis
investigating interactions of HIE market structures and participant behavior. However, they often
involve naïve subjects and their associated cost makes replication, sensitivity analysis, and
Appendix F (continued)
158
generalization to other circumstances limited. Last, ex-ante modeling analysis using artificial
subjects is capable of integrating HIE adoption incentives, blocking behaviors, and market share
- all factors that affect HIE adoption. These types of models allow us to calculate HIE adoption
levels in a given region, and are more easily generalized and analyzed for sensitivity.
When evaluating the behavior of hospitals under no incentive structures, our model
suggest that in the community under study six out of nine hospitals had market incentives to
adopt HIE–the three hospitals not willing to adopt HIE were medium-sized hospitals. Market
incentives to adopt HIE were driven by direct benefits of adopting HIE, such as reductions of
repeated testing and reduction of hospital readmissions, as well as market share gains
facilitated by HIE. In a meta-analysis published in 2012, Fareed found that large hospitals have
lower mortality rates than smaller hospitals, and therefore patients may have incentives to
switch from medium- to large-sized hospitals. Such market incentives, combined with HIE’s
potential on lowering patient switching costs,[44] may be perceived by smaller hospitals as a
threat for market share and thereby a barrier to adopting HIE. Competition between hospitals
coupled with volume-based payment systems create no incentives for smaller hospitals to
exchange their data with competitors because they want to keep lucrative services within their
hospital.[45–48] Although we believe the recent shift from volume- to value-based medicine will
only amplify the benefits of HIE adoption across all providers, medium-sized hospitals may need
targeted actions to mitigate market incentives to not adopt HIE.
In a recent report to the Congress,[12] the ONC recognizes health information blocking
as an important and unexplored barrier for HIE adoption. In order to deepen our understanding
about health information blocking, we used our proposed model to analyze the effect of a
collusion between two or more hospitals to not join HIE. Our model suggest that health care
provider health information blocking is a significant barrier for nationwide interconnectivity.
Appendix F (continued)
159
Moreover, current monetary incentives, as well as proposed penalties, had little or no effect on
stimulating HIE adoption in the community under study. Our results highlight the need for a new
and comprehensive strategy to remedy health information blocking. Current federal monetary
incentives are not enough to reach nationwide HIE. Although a common practice of providers is
to justify not adopting HIE due to privacy and data security concerns, there are reports of
privacy laws being cited in situations in which they do not in fact impose restrictions. The Health
Insurance Portability and Accountability Act (HIPAA), enacted in 1996, does not restrict patient
data from being shared between providers. The HIPAA Privacy Rule only establishes national
standards of privacy protections and rights, which applies to health plans, health care
clearinghouses, and providers. The Rule requires appropriate safeguards to protect the privacy
of personal health information, as well as setting limits and conditions on the uses and
disclosures that may be made of such information without patient authorization. In other words,
as long as patient consent is obtained, no further restrictions are imposed by HIPAA in a patient
information transaction between providers.
In the same report, ONC proposes to strengthen the regulatory environment that is
conducive to the exchange of electronic health information. More precisely, ONC seeks to work
with CMS to coordinate payment incentives and leverage other market drivers to reward
interoperability and exchange, and to discourage health information blocking. Among several
policy layers that are under discussion, new incentives to adopt HIE and penalties that raise the
costs of not moving to interoperable health IT systems were proposed by ONC. In light of these
debates, under particular market assumptions, our results suggest that penalties may be more
effective than incentives to promote HIE adoption in the particular community under study. Still
abundant research is needed to estimate the optimal design of proposed penalties.
Appendix F (continued)
160
Study limitations and future research are discussed next. First, our research does not considers
the physician opinion or willingness to use electronical medical records (EMR). Rather, the
model decides from a net economic perspective. Therefore, we cannot assess the influence of
individual, organizational, and contextual factors on hospital adoption of HIE. Second, our NE
search method does not provides the one and unique equilibrium of a game. Instead, the
method finds the one equilibrium out of many a game may have that is best in the sense that all
players have optimized their payoffs/utilities rather than adjusted to their beliefs about other
players in the game. Third, although out of the scope of this investigation, health information
blocking behavior can also be generated by health IT developers (i.e., EMR vendor
competition), or by coordinated actions between developers and their health care provider
customers. For instance, developers charge fees that make it cost-prohibitive for providers to
engage in HIE with other providers using a competitor EMR system. Future work will study the
role of competition in the health IT developers market, and how their actors behave under
different market structures.
CONCLUSION
A practical and efficient bi-level model for calculating oligopolistic HIE participation equilibrium in
health care provider markets has been developed and illustrated. The equilibrium is a mixed
strategy Nash equilibria interpreted as the willingness of each health care provider to share
freely data with other providers. An important barrier for reaching interoperability of EMR
systems is the strategic role of “owning” patient information that providers may lose by joining
HIE. The existing evidence, containing both empirical and modeling studies, helps to support
the design of HIE networks and to assess the potential impact of HIE policies. Our research
extends the existing evidence by incorporating the strategic behavior providers have at the time
of deciding whether or not to adopt HIE. This type of behavior and interaction can be illustrated
in terms of a health care provider’s conjectural variation–what does each hospital assume about
Appendix F (continued)
161
its competitors’ responses to its actions? The proposed model allows for deeper understanding
of why hospitals do not engage in HIE and the circumstances in which they do. Using sample
data from hospitals in Florida, we studied the potential impact of current and proposed HIE
policy, as well as the impact of health information blocking in the level of participation in HIE.
The proposed model can be used by policy makers to find incentive structures that will spur HIE
participation in a given community. HIE organizations can also benefit from the proposed model
by using it to inform their capacity expansion planning. For instance, HIE organization leaders
would be able to prioritize their efforts to seek new customers by identifying those providers at
the higher likelihood of joining HIE. Future work will investigate the hospitals’ HIE participation
decision over time, and extend the application of the model in evaluating other HIE networks
and other markets where inter-organizational cooperation for the common good is necessary.
COMPETING INTERESTS
The authors declare no competing interests.
FUNDING
No funding was provided for the completion of this study.
AUTHORS’ CONTRIBUTION
DM contributed to the idea conception, study design, model development, and acquisition and
analysis of results. FF contributed to the study design, model development and analysis of
results. TD and JZ are guarantors and contributed to the study design and analysis of results.
All authors contributed equally in preparing and reviewing multiple versions of the manuscript
and provided significant intellectual content. All authors read and approved the final version of
this manuscript.
Appendix F (continued)
162
REFERENCES
1 Kohn LT, Corrigan JM, Molla S. To Err Is Human: Building a Safer Health System. 2000.
2 Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st
Century. Washington, DC: 2001.
3 Rudin RS, Motala A, Goldzweig CL, et al. Usage and Effect of Health Information
Exchange. Ann Intern Med 2014;161:803. doi:10.7326/M14-0877
4 Rahurkar S, Vest JR, Menachemi N. Despite the spread of health information exchange,
there is little evidence of its impact on cost, use, and quality of care. Health Aff (Millwood)
2015;34:477–83. doi:10.1377/hlthaff.2014.0729
5 Adler-Milstein J, Jha AK. Health information exchange among U.S. hospitals: who’s in,
who's out, and why? Healthcare 2014;2:26–32. doi:10.1016/j.hjdsi.2013.12.005
6 Furukawa MF, King J, Patel V, et al. Despite Substantial Progress In EHR Adoption,
Health Information Exchange And Patient Engagement Remain Low In Office Settings. Health
Aff (Millwood) 2014;:hlthaff.2014.0445 – . doi:10.1377/hlthaff.2014.0445
7 Leu MG, Cheung M, Webster TR, et al. Centers speak up: The clinical context for health
information technology in the ambulatory care setting. J Gen Intern Med 2008;23:372–8.
doi:10.1007/s11606-007-0488-6
Appendix F (continued)
163
8 Vest JR, Zhao H, Jasperson J, et al. Factors motivating and affecting health information
exchange usage. J Am Med Inform Assoc 2011;18:143–9. doi:10.1136/jamia.2010.004812
9 Finnell JT, Overhage JM. Emergency medical services: the frontier in health information
exchange. AMIA Annu Symp Proc 2010;2010:222–
6.http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3041358&tool=pmcentrez&renderty
pe=abstract
10 Vest JR, Jasperson ’Jon Sean, Zhao H, et al. Use of a health information exchange
system in the emergency care of children. BMC Med Inform Decis Mak 2011;11:78.
doi:10.1186/1472-6947-11-78
11 Vest JR, Miller TR. The association between health information exchange and measures
of patient satisfaction. Appl Clin Inform 2011;2:447–59. doi:10.4338/ACI-2011-06-RA-0040
12 Office of the National Coordinator for Health Information Technology. Report on Health
Information Blocking.
https://www.healthit.gov/sites/default/files/reports/info_blocking_040915.pdf
13 Miller AR, Tucker C. Health information exchange, system size and information silos. J
Health Econ 2014;33:28–42.
14 Cohen E. Patients demand: ‘Give us our damned data’ CNN. 2010.
15 CNN. Patients want records, hospitals turn deaf ear. CNN.com. 2010.
Appendix F (continued)
164
16 Reps. D. Black and M. Honda. Letter to M. Tavenner and F. Mostashari.
2013.http://op.bna.com/hl.nsf/id/kcpk-99mnkx/$File/711EHRletter.pdf
17 White J. Halth IT Now Coalition. 2015.http://www.healthitnow.org/what-is-information-
blocking/
18 Grossman JM, Kushner KL, November EA, et al. Creating sustainable local health
information exchanges: can barriers to stakeholder participation be overcome? 2008.
19 Adler-Milstein J, DesRoches CM, Jha AK. Health information exchange among US
hospitals. Am J Manag Care 2011;17:761–8.http://www.ncbi.nlm.nih.gov/pubmed/22832592
20 Desai S. Electronic Health Information Exchange, Switching Costs, and Network Effects.
Switch Costs, Netw Eff (November 1, 2014) NET Inst Work Pap 2014.
21 Ringel JS, Hosken SD, Vollaard BA, et al. The Elasticity of Demand for Health Care: A
review of the literature and its application to the military health system. Santa Monica, CA: 2002.
http://www.rand.org/content/dam/rand/pubs/monograph_reports/2005/MR1355.pdf
22 Sun H, Gao Z, Wu J. A bi-level programming model and solution algorithm for the
location of logistics distribution centers. Appl Math Model 2008;32:610–6.
doi:10.1016/j.apm.2007.02.007
23 Feijoo F, Das TK. Design of Pareto optimal cap-and-trade policies for deregulated
electricity networks. Appl Energy 2014;119:371–83. doi:10.1016/j.apenergy.2014.01.019
Appendix F (continued)
165
24 Fampa M, Barroso LA, Candal D, et al. Bilevel optimization applied to strategic pricing in
competitive electricity markets. Comput Optim Appl 2007;39:121–42. doi:10.1007/s10589-007-
9066-4
25 Hu X, Ralph D. Using EPECs to Model Bilevel Games in Restructured Electricity
Markets with Locational Prices. Oper Res 2007;55:809–27. doi:10.1287/opre.1070.0431
26 Li H, Zhang L, Jiao Y-C. An interactive approach based on a discrete differential
evolution algorithm for a class of integer bilevel programming problems. Int J Syst Sci 2015;:1–
12. doi:10.1080/00207721.2014.993348
27 Saharidis GK, Ierapetritou MG. Resolution method for mixed integer bi-level linear
problems based on decomposition technique. J Glob Optim 2009;44:29–51.
doi:10.1007/s10898-008-9291-0
28 Nash JF. Equilibrium points in n-person games. Proc Natl Acad Sci 1950;36:48–9.
doi:10.1073/pnas.36.1.48
29 Pfuntner A, Wier LM, Steiner C. Costs for hospital stays in the united States. Stat Br
2010;146.
30 Overhage JM, Dexter PR, Perkins SM, et al. A randomized, controlled trial of clinical
information shared from another institution. Ann Emerg Med 2002;39:14–23.
doi:10.1067/mem.2002.120794
Appendix F (continued)
166
31 Blumenthal D. Stimulating the Adoption of Health Information Technology. N Engl J Med
2009;:1477–9.http://www.nejm.org/doi/full/10.1056/nejmp0901592 (accessed 30 May2013).
32 eHealth Initiative. Health Information Exchange: From Start-up to Sustainability.
Washington, DC: 2007. http://www.hci3.org/sites/default/files/files/HRSA CCBH Final Report
Revised.pdf
33 GAMS Development Corporation. General Algebraic Modeling System (GAMS). 2013.
34 Chatterjee B. An optimization formulation to compute Nash equilibrium in finite games.
In: Methods and Models in Computer Science, 2009. ICM2CS 2009. Proceeding of International
Conference on. 2009. 1–5.
35 The MathWorks Inc. MATLAB. 2012.
36 Berman O, Zahedi F, Pemble KR. A decision model and support system for the optimal
design of health information networks. IEEE Trans Syst Man Cybern Part C (Applications Rev
2001;31:146–58. doi:10.1109/5326.941839
37 Brennan PF, Ferris M, Robinson S, et al. Modeling participation in the NHII: operations
research approach. AMIA Annu Symp Proc AMIA Symp AMIA Symp 2005;2005:76–
80.http://www.ncbi.nlm.nih.gov/pubmed/16779005
38 Ferris M, Brennan PF, Tang L, et al. Creating operations research models to guide RHIO
decision making. AMIA Annu Symp Proc AMIA Symp AMIA Symp 2007;2007:240–
4.http://www.ncbi.nlm.nih.gov/pubmed/18693834
Appendix F (continued)
167
39 Sridhar S, Brennan PF, Wright SJ, et al. Optimizing financial effects of HIE: a multi-party
linear programming approach. J Am Med Inform Assoc 2012;19:1082–8. doi:10.1136/amiajnl-
2011-000606
40 Merrill JA, Deegan M, Wilson R V, et al. A system dynamics evaluation model:
implementation of health information exchange for public health reporting. J Am Med Inform
Assoc 2013;20:e131–8. doi:10.1136/amiajnl-2012-001289
41 Mustafee N, Katsaliaki K, Gunasekaran A, et al. Electronic health records: A simulation
model to measure the adoption rate from policy interventions. J Enterp Inf Manag 2013;26:165–
82.
42 Yaraghi N, Du AY, Sharman R, et al. Network effects in health information exchange
growth. ACM Trans Manag Inf Syst 2013;4:1.
43 Zhu Q, Gunter C, Basar T. Tragedy of anticommons in digital right management of
medical records. In: Proceedings of the 3rd USENIX conference on Health Security and Privacy.
2012. 10.
44 Strauss AT, Martinez DA, Garcia-Arce A, et al. A user needs assessment to inform
health information exchange design and implementation. BMC Med Inform Decis Mak
2015;15:81. doi:10.1186/s12911-015-0207-x
Appendix F (continued)
168
45 Kruse CS, Regier V, Rheinboldt KT, et al. Barriers Over Time to Full Implementation of
Health Information Exchange in the United States. JMIR Med Inf 2014;2:e26.
doi:10.2196/medinform.3625
46 Sheikh A, Sood HS, Bates DW. Leveraging Health Information Technology to Achieve
the ‘Triple Aim’ of Healthcare Reform. J Am Med Inform Assoc 2015;44:1–9.
doi:10.1093/jamia/ocv022
47 O’Malley AS, Bond AM, Berenson RA. Rising hospital employment of physicians: better
quality, higher costs? Issue Br Cent Stud Heal Syst Chang 2011;:1–
4.http://www.ncbi.nlm.nih.gov/pubmed/21853632
48 O’Malley AS. Testimony to the interoperability task force of the health IT policy
committee, ONC. 2015.
Appendix F (continued)
169