the rural health safety net under pressure: rural hospital ... · year, the chartis center for...

The Rural Health Safety Net Under Pressure:

Rural Hospital Vulnerability

Research Methodology | February 2020

1 Rural Hospital Vulnerability | The Chartis Center for Rural Health

ince 2010, The Chartis Center for Rural Health (CCRH) and iVantage Health Analytics have

been tracking the loss of rural hospitals across America. As the closure crisis begins its tenth

year, The Chartis Center for Rural Health has been a leading participant in this national

conversation through research into the stability of the rural health safety net.

Last year the number of rural hospital closures since 2010 surpassed 100 and now stands at 120.1

In an effort to determine if there is a “tipping point” toward closure, CCRH examined financial and

operational data for the three years prior to closure for closed hospitals . The analysis shows that

between three years and one year prior to closure – operating margin and revenue decline steadily

(Figure 1).

Figure 1. Financial Trends Prior to Closure

Informed by this initial analysis, The Chartis Center for Rural Health has developed a rigorous

statistical model that: (A) identifies key indicators most likely to impact a hospital’s ability to

sustain operations during the two years prior to closure, and (B) identifies the number of open

rural hospitals that are quantitatively similar across selected covariates to those rural hospitals

1 Cecil G. Sheps Center for Health Services Research, University of North Carolina, January 1, 2020.

S


that have closed since January, 2010, and (C) explores the performance level of open hospitals.

From this analysis, we identify the portion of currently open hospitals that may be vulnerable to

closure.

1. Analytic Methods

The Chartis Center for Rural Health’s rural hospital vulnerability analysis is based on Healthcare

Provider Cost Report Information System (HCRIS) data pertaining to 76 rural hospitals closed

since 2010 compared to 1,844 rural hospitals currently ‘open’ as of September 2019. In

September 2019, the number of closed rural hospitals since 2010 was 113. The list of closed

hospitals is sourced from the Cecil G. Sheps Center for Health Services Research at the University

of North Carolina. The analysis excluded 37 closed hospitals due to missing or invalid data. To

adjust for geographical variation in socioeconomic disadvantage, the model includes data from

the Area Deprivation Index developed by researchers at the University of Wisconsin-Madison.

CCRH research identified statistically significant indicators for hospital closure and the main

effects attributed to those indicators through multilevel logistic regression.2 A varying intercept

(multilevel) model at the state level was implemented to account for the nested structure of the

data with each hospital’s Medicaid expansion status being nested within a state. The dependent

variable is 1 if the hospital is closed and 0 otherwise. A total of 16 independent variables were

selected for inclusion by CCRH and were theorized to be potentially important indicators of

hospital closure.

The 16 indicators3 are:

• Case Mix Index: This is the total sum of DRG weights for each hospital divided by

the number of discharges from HCRIS. This adjusts for the severity of cases.

• Government Control Status: Using type of control variable from HCRIS, CCRH

created a 0/1 variable for government-controlled facilities (type of control 7-13)

where 1 is government controlled (Table 1).

2 Used the glm function in RStudio Version 1.2.1335 3 Included value for most recent year for each indicator unless otherwise specified


Table 1. Type of Control Descriptions

Type of Control Type of Control Description

1 Voluntary Nonprofit - Church

2 Voluntary Nonprofit - Other

3 Proprietary - Individual

4 Proprietary - Corporation

5 Proprietary - Partnership

6 Proprietary - Other

7 Governmental - Federal

8 Governmental - City-County

9 Governmental - County

10 Governmental - State

11 Governmental - Hospital District

12 Governmental - City

13 Governmental - Other

• Critical Access Hospital: This is a 0/1 variable to indicate a (1) Critical Access

Hospital or (0) other. Critical Access status is determined by CMS.

• Number of Beds: The total number of beds from HCRIS.

• Percent Operating Revenue: Percentage of total revenue that is outpatient from

HCRIS.

• Capital Efficiency: Net Patient Revenue/Total Revenue from HCRIS.

• Occupancy: Percent occupancy as defined by HCRIS.

• Average Age of Plant: Depreciation/Annual Depreciation Expense from HCRIS.

• Average Length of Stay: Hospital average length of stay from HCRIS.

• State Status for Medicaid Expansion: A variable indicating whether the hospital

resides in a Medicaid expansion state with a value of (1) for expanded and (0) for no

expansion. For the purpose of this analysis, Idaho, Maine, Nebraska, Utah and

Virginia – states that have adopted but not yet implemented expansion – are

considered expansion states.

• Medicare/Medicaid Discharges: Percentage of discharges that are Medicare or

Medicaid from HCRIS.


• Operating Margin (Positive/Negative): Hospitals with a positive operating margin

have a value of 1 and 0 otherwise from HCRIS.

• Change in Total Revenue: Percentage change in total revenue from most recent two

years of data from HCRIS.

• Change in Net Days in Account Receivables: Percentage change in days in Net AR

from most recent two years of data from HCRIS.

• Area Deprivation Index: National percentile ranking of area deprivation index by

census block group.4

• System Affiliation: A variable derived from HCRIS indicating whether the hospital is

part of a system with a value of (1) for system hospitals and a value of (0) otherwise.

2. Transformations

• Removed 31 records where the total revenue is 0 or less.

• Removed 45 records where the days in Net AR is less than 0.

• Removed 161 records where the average age of plant is less than 0.

• Capped occupancy at 100%.

• Removed 66 records with outliers for total revenue, days in net accounts receivable,

average age of plant, or average length of stay.

• Imputed the median for missing values of Average Age of Plant, Case Mix Index, and

Number of Beds (CAH median and Rural PPS median separately).

• Took the logarithm of the Number of Beds and Average Age Plant in order to correct

for skewness.

4 University of Wisconsin School of Medicine and Public Health. 2015 Area Deprivation Index v2.0. Downloaded from https://www.neighborhoodatlas.medicine.wisc.edu/ November 12, 2019


• Given the prior knowledge that the number of beds depends on whether the hospital

is a Critical Access Hospital (CAH) or not, the model includes an interaction term

between the Number of Beds and CAH variables.

3. Results

The relative effects (standardized odds ratios) produced by CCRH’s logistic regression model for

each of the nine statistically significant independent variables are provided in Figure 2 and

sorted from largest relative effect size to smallest relative effect size and including 90%

confidence intervals (CI).

Figure 2. Logistic Model Relative Effects

The average scaled effect sizes reveal some variation across these covariates, however, the 90%

CIs overlap across most of the variables showing no statistical difference in effect size among

them. The percent occupancy variable demonstrates the largest protective effect against

hospital closure.

The main effects (not standardized odds ratios) are detailed below. These odds ratios are

simpler to interpret even though their magnitude cannot be directly compared due to the

difference in scale across the covariates.


The main effects (and 90% CIs) can be described in the following way:

• A one percentage increase in Occupancy is associated with a 4.7% (3.2%, 6.2%) decrease

in the odds of closure on average.

• A one percentage increase in Case Mix Index is associated with a 3.3% (1.4%, 5.2%)

decrease in the odds of closure on average.

• Having Government Control Status is associated with a 70.4% (43.9%, 84.4%) decrease in

the odds of closure on average.

• A one percentage increase in the proportion of Outpatient Revenue is associated with a

4.5% (2.6%, 6.3%) decrease in the odds of closure on average.

• A one percentage increase in the Percent Change in Total Revenue is associated with a

2.9% (1.5%, 4.3%) decrease in the odds of closure on average.

• Being in a Medicaid Expanded State is associated with a 62.3% (24.5%, 81.2%) decrease

in the odds of closure on average.

• A one unit change in Log Average Age of Plant is associated with a 35.5% (17.8%, 49.4%)


• A one percentage increase in Capital Efficiency is associated with a 2.0% (1.0%, 2.9%)


• Health System Affiliation is associated with a 46.6% (5.8%, 69.7%) decrease in the odds of

closure on average.

This resulting model was used to assign a predicted probability (p) to each hospital within the

sample and out of the sample. Initially, CCRH selected a threshold for p of 0.05 that optimizes

accuracy (specificity and sensitivity). Therefore, closed hospitals are identified as hospitals with a

predicted probability of closure greater than 0.05. Using a 4-fold cross validation to assess out

of sample accuracy, the validation model produces 71% sensitivity and 88.4% specificity.

Open hospitals that also fall above the 0.05 threshold probability are determined to be

financially similar to 86.8% (66) of closed hospitals based on the selected covariates. These 216

open hospitals are considered the ‘most vulnerable’ to closure out of the total 1,844 open

hospitals included in the study.


Because the 0.05 threshold does not capture any vulnerable hospitals for several states, CCRH

evaluated a second tier of vulnerability with a threshold of p of 0.02 in order to recognize

vulnerability risk nationwide. When the threshold is relaxed to p > 0.02, the model correctly

identifies 6 additional closed hospitals. Applied to open hospitals, this second threshold

indicates an additional 237 open hospitals may also be ‘at risk’ to closure.

The 216 ‘most vulnerable’ hospitals identified exhibit the following characteristics:

• 162 (or 75%) located in Non-Medicaid Expansion states

• 119 (or 55%) are Rural PPS while 97 (or 44%) are Critical Access Hospitals

• 165 (or 76%) do not have government control status

• 124 (or 57%) are affiliated with a system

• Median percentage change in total revenue was -1.4%

• Median occupancy is 20.7%

• Median capital efficiency is -6.3%

• Median percentage of outpatient revenue is 75.9%

• Median operating margin is -8.6%

The 237 ‘at risk’ hospitals identified exhibit the following characteristics:

• 146 (or 62%) located in Non-Medicaid Expansion states

• 92 (or 39%) are Rural PPS while 145 (or 61%) are Critical Access Hospitals

• 160 (or 68%) do not have government control status

• 114 (or 48%) are affiliated with a system

• Median percentage change in total revenue was 1.7%

• Median occupancy is 26.9%

• Median capital efficiency is -1.1%

• Median percentage of outpatient revenue is 77.6%

• Median operating margin is -2.6%

Rural hospital vulnerability is not evenly distributed across the nation. The maps below show the

geographical variation of vulnerable hospital counts across the United States.


Distribution of 453

Vulnerable Hospitals

Distribution of Cohort 1: 216

‘Most Vulnerable’ Hospitals

Distribution of Cohort 2: 237

‘At Risk’ Hospitals


4. Limitations

• There are several factors that are relevant determinants of hospital closure that could not

be quantified and therefore were not included in this study. These include, but are not

limited to:

o Community sentiment toward a hospital

o Local political decisions or pressures that could lead to hospital closure

o Opening of new hospitals in close proximity

o Provider retention

o Service line retention

• Between January 2010 and September 2019, 113 rural hospitals had closed. Once outliers

and invalid records were removed from the analysis, the total count of closed rural

hospitals becomes 76. Results saw negligible improvements in statistical significance and

accuracy when experimenting with up-sampling methods. Therefore, no adjustments

were made to account for small sample size.

• iVantage Health Analytics’ Hospital Strength INDEX is not included in this analysis, but

the financial indicator (capital efficiency) is included as a variable. The INDEX

components for Quality and Patient Perspectives are too variable in meaning year-over-

year to include in this analysis. The outpatient and inpatient market share could be

included in future analysis.

• Wage Index was not included in this analysis because a data source for wage index has

not been identified for closed facilities. Given that Medicare bases payments on wage

index, and that officials in some states have argued a connection between ‘unfair’ wage

index and hospital closures, a historical review of wage index could provide an added

lens to this research.

• This study does not match on fiscal year as hospital closures were spread out over the

span of nearly 10 years. Instead, the most current two years of indicators were used to

determine whether an open hospital is vulnerable based on our understanding of how

hospitals progress toward closure in the final two years of operation.

https://www.modernhealthcare.com/government/cms-throws-rural-hospitals-lifeline-wage-index-changes


Research Team

Michael Topchik, MA, National Leader, The Chartis Center for Rural Health

Michael has led the development and program operations of more than 20 rural health

network initiatives around the country including the development and management of

the OH CAH Network since 2009. He is a frequent presenter at state, regional and

national rural health events and brings a wealth of experience utilizing “big data” for

hospital benchmarking and performance improvement. Michael offers his expertise and

knowledge as a key resource on matters impacting rural healthcare for media outlets

such as CNN, The Washington Post, Forbes, Reuters, The Boston Globe and FiveThirtyEight.

Ken Gross, PhD, Chief Data Scientist, The Chartis Group

Ken Gross is the Chief Data Scientist of The Chartis Group. He has over 15 years of

experience as a thought leader for advanced analytic techniques and solution

development across the healthcare provider industry. At Chartis, he serves as a senior

advisor and industry expert to healthcare providers, aiming to advance their analytic

capabilities and methods, and leads the development of new analytic methodologies

and algorithms that support the firm’s consulting practices.

Prior to joining The Chartis Group, Dr. Gross was founder and Principal of Quantitative

Innovations, a data strategy consulting practice, where he advised hospital systems and ACOs on

implementation of population health data analytic strategies. He also served as the Director of Research

and Evaluation for the Camden Coalition of Healthcare Providers, where he developed innovative

quantitative and spatial analytic methods for understanding and addressing the needs of high utilization

patients. Prior to his work with the Camden Coalition, Dr. Gross held positions as a Senior Associate at The

Reinvestment Fund, and an Epidemiologist for the City of Philadelphia, Division of Maternal and Child

Health.

Melanie Pinette, MEM, Data Analysis, The Chartis Center for Rural Health

Melanie possesses extensive experience working with analyzing healthcare data. She

works closely with CCRH’s state networks, providing insight into performance

improvement opportunities. Prior to joining CCRH, Melanie served as a manager of

Business Development for GNS Healthcare’s managed care team and led several

program analyses and research efforts related to population health at Onpoint Health

Data. She also worked with state health entities to implement and evaluate ACO

networks designed to improve patient outcomes and lower total cost of care.


Troy Brown, Client Services Manager, The Chartis Center for Rural Health

Troy spent 10 years at Charles A. Dean Memorial Hospital, a critical access hospital in

Maine, serving in a variety of roles including: Director of Business Services, Registration,

Patient Accounts, HIM, IT, Materials Management, Performance Improvement and

Community Relations and Development. Troy has built upon this foundation at The

Chartis Center for Rural Health, facilitating a variety of strategic performance

improvement-related projects with independent rural hospitals, system affiliated

facilities and state-wide networks of rural and Critical Access Hospitals nationally.

Billy Balfour, Director, Communications, The Chartis Center for Rural Health

Billy leads The Chartis Center for Rural Health’s marketing initiatives. In his role, he

works closely with state networks to promote and coordinate various network

initiatives designed to educate and help participating hospitals optimize the use of

INDEX-related benchmarks. Billy oversees development and marketing of CCRH’s

thought leadership activities, including executive-level presentations at national rural

health conferences and ongoing research into the rural health safety net.

Hayleigh Kein, Analyst, The Chartis Center for Rural Health

In her role, Hayleigh works closely with The Chartis Center for Rural Health’s clients to

better understand and assess performance metrics, including the Hospital Strength

INDEX. She’s actively involved in the development of market and population health

assessments for The Chartis Center for Rural Health’s state network clients that provide

hospital leadership teams with a new lens into dynamic factors impacting market

share, patient volumes, as well as the quality and the delivery of care.

the rural health safety net under pressure: rural hospital ... · year, the chartis center for...

Documents