1

Methodological Considerations in Using PROs, PROMs and PRO-PMs in ESRD

John D. Peipert and Ron D. Hays (February 10, 2017)

White Paper Commissioned by the Kidney Care Quality Alliance

2

Table of Contents

I. Executive Summary ............................................................................................................. 3

II. Intro and Definitions ............................................................................................................ 4

III. Why, When, and How PRMs are Used in Medical Research and Clinical Settings ........ 8

IV. Types of PRMs ............................................................................................................... 11

V. Administering PRMs to Dialysis Patients ......................................................................... 21

VI. Challenges ...................................................................................................................... 26

VII. Recommendations .......................................................................................................... 29

VIII. Conclusions .................................................................................................................... 31

References ................................................................................................................................. 31

3

I. Executive Summary

Patient reported measures (PRMs), including patient-reported outcomes (PROs), play a critical

role in dialysis care, as one of two primary sources of information about clinical outcomes of

patients. The usage of PRMs and PROs is extensive in dialysis clinics. While there are excellent

PRMs to choose from, and their implementation as part of quality improvement and performance

monitoring is extensive, there are still challenges to be addressed. In this paper, we summarize

the literature on PRMs and their use in dialysis and in medicine generally, and we offer

recommendations for improving their use in dialysis related to Selection of PRMs, Mode of

Administration, and Support for PRM Use:

v Continue the use of KDQOL-36 for dialysis centers’ internal quality improvement

activities and the ICH-CAHPS for public dialysis center performance monitoring, but

promote efforts to modify these instruments by incorporating PROMIS general health

items (KDQOL-36) and reducing the length of the ICH-CAHPS.

v Adopt a PRM of whether dialysis patients have been informed about their option for

transplant.

v Evaluate equivalence between electronic and paper versions of PRMs prior to

widespread use of electronic administration.

v Explore reimbursement of costs of PRM administration by the Centers for Medicare and

Medicaid Services.

v Continue development of provider trainings in PRM administration and interpretation.

These recommendations will help dialysis care decision-makers, clinicians, and applied

researchers take the next steps toward enhancing PRM use in dialysis.

4

II. Intro and Definitions Patient reported measures (PRMs), including patient-reported outcomes (PROs), play a critical

role in medical care, as one of two primary sources of information about clinical outcomes of

patients.1 The other source of information is “non-patient-reported” biologically-based patient

data (e.g., laboratory values such as hemoglobin level, calcium, phosphorus, parathyroid

hormone albumin, and hemoglobin). PRM assessment may be included at several stages of the

patient-provider clinical encounter such as during the time when the health issue is elicited, when

the course of treatment or action is discussed, when the treatment plan is created, during the

course of treatment itself, and after the treatment is concluded.1 PRMs are rarely included in

clinical registries or incorporated into clinical decision-making,1 but they are widely-used in

clinical trials of new medications2 and other medical treatments or health interventions.3 For

instance, PROs have become instrumental in the field of oncology, as many chemotherapy drugs

can successfully destroy cancer cells, but the subsequent impact on the patient’s health-related

quality of life (HRQOL) may differ substantially.4 Therefore, in addition to measures of clinical

performance, such as patient survival, HRQOL is often an important secondary outcome in

comparisons of drugs’ effectiveness.5,6

The National Quality Forum (NQF) has adapted the FDA definition of a PRO, which is:

‘‘any report coming from patients about a health condition and its treatment, without

interpretation of the patient’s response by a clinician or anyone else’’ (p. 2).2 The NQF makes

the further distinctions of patient-reported outcome measures (PROMs), defined as an,

“Instrument, scale, or single-item measure used to assess the PRO concept as perceived by the

patient, obtained by directly asking the patient to self-report,” and PRO-based performance

measures (PRO-PMs), defined as, “a performance measure that is based on PROM data

5

aggregated for an accountable healthcare entity.” The broad scope of the FDA’s definition of

PROs is consistent with what others have referred to as PRMs.1 In theoretical frameworks such

as Andersen’s Behavioral Model of Access to Health Care,7 outcomes have a more limited

scope, including only the patient’s health status (perceived or evaluated) and satisfaction with

care. On the other hand, important factors that may be patient-reported, such as health behaviors,

attitudes toward health, and patients’ experiences with health care are not themselves outcomes,

and therefore do not fall under PROs.

Fung and Hays offered a framework that articulates the relationships between types of

PRMs, including PROs.1 (Figure 1) At its most basic level, this framework distinguishes between

the underlying concepts to be measured

(circles) and direct indicators that do not

represent an underlying concept (boxes).

Both direct indicators and underlying

concepts can be measured with PRMs.

For example, a health behavior such as

adherence to medications can be

measured by patient reports on whether

they have or have not adhered to a

prescribed medication regimen in

response to questions asking directly about medication taking (e.g., “Did you take your

medications yesterday at the prescribed times?”); such questions assess whether the behavior

occurred or not, and they do not intend to represent a larger, underlying concept. On the other

6

hand, HRQOL is a concept that cannot be indicated directly and usually requires a series of

questions that each represent related aspects of the larger concept.

Major concepts in this framework include preferences for care, HRQOL, quality of care,

and satisfaction with care. The direct indicators include patient characteristics, technical quality

of care, needs assessment, patient reports about care, and health behaviors. Of the many

relationships between these constructs and indicators, it is important to note that HRQOL is

exclusively an outcome and itself does not influence the other depicted constructs or indicators.

On the other hand, HRQOL is influenced by other concepts and indicators, including health

behaviors and quality of care, which do not count as outcomes, and therefore are not PROs.

Though satisfaction with care is generally considered a PRO, it also impacts health behaviors. It

is important to distinguish between patients’ satisfaction with care, and patients’ experiences

with care. Satisfaction with care regards discrepancies between patients’ expectations for care

and the care they actually receive.8 Experience with care refers to objective dimensions of the

care patients receive and interactions with different elements of the health care system.8

Indicators like patient reports about care may be used to measure their experiences with care,

along with their preferences for care and ratings of quality of care. Of these, the needs

assessment and patient reports about care would be distinguished as PRMs, while technical

quality of care would not. Additionally, other patient characteristics, like demographics, are also

considered PRMs.

Table 1. provides each of the underlying concepts and direct indicators that Fung and

Hays count as PRMs, and gives examples of PRMs used to capture each. For HRQOL, the

PROMIS-29 instrument is a generic multi-dimensional measure that can elicit patient reports of

HRQOL. (Detailed in Section III below). Similarly, satisfaction with care can be assessed using

7

the Functional Assessment of Chronic Illness Therapy Treatment Satisfaction (FACIT TS)

instrument. The Clinician & Group CAHPS represents an example of a patient report about care.

Questions about whether and how often patients use tobacco are examples of health behaviors,

which are typically measured directly.

Table 1. Underlying Concepts and Direct Indicators with Example Patient-Reported Measures Concept/Indicator Example of PRM Health-Related Quality of Life PROMISa-299 Satisfaction with Care FACITb Treatment Satisfaction10 Patient Reports about Care Clinician & Group CAHPSc,11 Needs Assessment Control Preferences Scale12 Patient Demographic Characteristics

Age, gender, race/ethnicity, education attainment

Health Behaviors Tobacco Use (e.g., yes vs. no, frequency of use) aPatient Reported Outcomes Measurement Information System; bFunctional Assessment of Chronic Illness Therapy; cConsumer Assessment of Health Care Providers and Systems

In the field of dialysis, PRMs are used as performance measures, or PRO-PMs. Since the

Centers for Medicare and Medicaid Services (CMS) covers the cost of patient care for most renal

replacement therapy, an extensive effort has been made to track patient experiences with care

and HRQOL, with large data collection projects funded by government agencies. If dialysis

centers have not demonstrated that the mandated PRM assessment has occurred, their

reimbursement from CMS is in jeopardy.

Using the Fung and Hays framework as a starting point, this report will identify key

PRMs relevant to dialysis patients, and then review key methodological issues around the use of

these measures. We focus on how PRMs are used in research and clinical settings, which

measures are available for use with dialysis patients, and provide recommendations for

overcoming challenges in administering PRMs. Within this discussion, we will also comment on

how current PRMs may be used for performance measurements (PRO-PMs). The anticipated

8

audiences for this report are dialysis medical providers and applied researchers who seek to use

PRMs with dialysis patients, as well as dialysis care regulators (e.g., CMS), payors (insurance

companies), and policy makers.

III. Why, When, and How PRMs are Used in Research and Clinical Settings PRMs offer many potential benefits. Perhaps most importantly, they provide information about

aspects of health that are important to patients and that are best, or that can only be, obtained

from patients directly.13 A patient’s health status can be measured by providers without patient

input. However, provider reports in this case are more unreliable and less valid than patient

reports.14 PRMs operate on the philosophy that patients are the best source of information about

their own health experiences and perspectives. Therefore, in addition to key non-patient reported

indictors of health, PRMs play an important role in evaluating a patient’s health. PRMs also

provide an opportunity for patients to report on their experiences with health care providers and

health services, which is associated with how likely patients are to use those services again and

contributes to comparative information about the quality of care.13,15

Given the importance of PRMs to understanding patients’ health and experiences with health

care, it is often advantageous to include them in health care interventions. The International

Society for Quality of Life Research (ISOQOL) put forth guidance for incorporation of PROs in

clinical care.16,17 In doing so, they use Greenhalgh and colleagues’ taxonomy, which makes the

following recommendations for implementing PRMs in clinic.18

v First, PRMs are used to screen for health problems.

v Once health problems are identified, PRMs are used to monitor those problems over

time.

9

v Finally, clinicians need PRM information to facilitate shared decision-making about

treatment.

This framework helps identify which patients might benefit most from interventions to

improve health, or to signal which interventions are most appropriate or effective. For example,

reports of physical functioning will help determine whether interventions that include increased

physical activity are likely to be effective.1,13 Additionally, PRMs can be used to elucidate

problems with the care experience and environment, which can enhance understanding of the

most appropriate interventions for specific subgroups of patients.13 Regarding dialysis

specifically, patients with renal failure must obtain renal replacement therapy, but have several

options about how to treat this condition, including several different types of dialysis, as well a

pursuit of transplant from a deceased or living donor, or no treatment at all.

The Consolidated Standards of Reporting Trials (CONSORT) statement addresses inclusion

of PROs.19 For example:

v Scoggins and Patrick examined the use of PROs in ClinicalTrials.gov randomized

controlled trials (RCTs) between 2004-200720. From 17,704 registered RCTs, 14%

(n=2,481) reported using a PRO, but only 41% of these identified the instrument used.

However, the literature on the impact of PROs incorporated into interventions has been mixed.

v A recent report from a RCT with 766 cancer patients on chemotherapy who were

assigned either usual care or to a symptom-reporting and management intervention

wherein nurses were alerted of patient-reported worsening symptoms electronically found

that those receiving the intervention had significantly fewer visits to the emergency room

and better survival at 1 and 5 years after enrollment in the study.21,22

10

v Valderas and colleagues conducted a review of 28 studies that tested the impact of

administering PRMs in clinical settings on improvements in processes of care (e.g.,

facilitating patient education, increasing referrals), outcomes of care (e.g., improving

functional status), and physician satisfaction with care (e.g., physician-rated usefulness of

new information provided).13 The proportion of interventions with a significant, positive

impact varied by the type of outcome, but occurred in 40-50% of the studies, showing

promise for PROs to improve care.

Despite the potential benefit of administering PRMs in clinic, their uptake and incorporation

in clinical care has been slower than desired. Literature on barriers to implementing PRMs,1,13 in

dialysis23 and in other clinical settings,24,25 have identified the following types of barriers:

insufficient provider time, training and education to administer PRMs; skepticism about the

reliability, validity, or responsiveness of PRMs; burden on patients to complete the measures;

high financial costs associated with routine PRM administration; and insufficient motivation to

use the measures among providers. Relatedly, Fung and Hays point to multiple studies showing

that physicians do not change their treatment plans when provided with data from HRQOL

instruments,1 calling into question whether providers judge the data resulting from PRMs to be

useful.

The field of dialysis represents one area where PRMs have been well-integrated into clinical

interventions with patients, and they are used as PRO-PMs. In their Conditions for Coverage (42

CFR §494.90), CMS mandated that each dialysis patient’s physical and mental health must be

monitored, and this often occurs with the use of a standardized HRQOL measure.26 The patient

reports of HRQOL are then used to create individually-tailored interventions that focus on the

areas where the patient’s HRQOL needs most improvement. In addition, reports of patient

11

experience with care using the CAHPS® In-Center Hemodialysis survey is included in the

Quality Incentive Program (QIP) evaluation metrics for dialysis centers.27

IV. Types of PRMs PROs

PROMIS®

The most frequently assessed PRO is HRQOL. Over the past decade, the NIH PROMIS® project

produced the state-of-the-science of HRQOL measures. PROMIS took an innovative approach to

the development and evaluation of PROs by use of item response theory (IRT) and computer

adaptive testing (CAT), drawing from large banks of items to generate efficient, reliable, and

parsimonious individually-tailored measures of HRQOL.9 Figure 2 shows the PROMIS

conceptual model that features

Global Health, Physical

Health, Mental Health, Social

Health, and several domains

within these areas.

The PROMIS project

brought together a national

brain trust of multiple content

area experts, methodological experts, clinicians from academia, and NIH project officers. Large

scale field testing was conducted with internet survey panels and patients from 6 primary study

sites to represent the general population and those with chronic conditions such as heart disease,

cancer, rheumatoid arthritis, osteoarthritis, psychiatric illnesses, chronic obstructive pulmonary

disease, spinal cord injury, and kidney disease.9,28

12

The PROMIS measures can be assessed as static “short forms” or through CAT using a

range of data capture and scoring platforms, including Assessment Center

(http://www.assessmentcenter.net/). Both forms of administration have demonstrated exceptional

psychometric properties.9,29,30 The PROMIS measures use a T-score metric, which has a mean of

50 and standard deviation of 10, with the mean referenced to the U.S. general population. Using

this metric, different subgroups and individual patients can be compared to the general

population or other referent groups.

There are many PROMIS measures, for both adult and pediatric populations. The

PROMIS-29 is analogous to the SF-36, with physical and mental health summary scores

components arising out of 8 domains: 1) physical function; 2) anxiety; 3) depression; 4) fatigue;

5) sleep disturbance; 6) ability to participate in social roles and activities; 7) pain interference; 8)

pain intensity. In addition, a range of domain-specific measures are available in physical health

(e.g., physical function, sexual function), mental health (e.g., cognitive function, anger), and

social health (e.g., ability to participate in social roles, companionship, social isolation).

The PROMIS measures are generic. The measures are designed to be equally applicable

to a variety of chronic conditions, patients of all races, and patients of all age groups (separate

instruments for adult and pediatrics). For this reason, PROMIS measures may be well-suited to

capture the global aspects of HRQOL for dialysis, but may not fully reflect the specific issues

facing dialysis patients. Though there is increasing attention paid to the potential for PROMIS

measures for use in kidney disease,31,32 the PROMIS measures have not been systematically used

with adult dialysis patients. However, the PROMIS pediatric measures have begun to be

implemented with pediatric ESRD patients.33,34

Kidney Disease Quality of Life (KDQOL) Measures

13

Targeted measures, in contrast, are designed to be relevant to a particular subgroup. The

relative advantages and disadvantages of generic versus targeted HRQOL measures have been

discussed.35 Generic measures in theory can be administered to anyone and compared across

different subgroups. Targeted measures are putatively more applicable and sensitive to the group

being targeted. When possible, it is recommended that a combination of generic HRQOL

measures be used. For example, the original KDQOL short-form (KDQOL-SF) includes the SF-

36 as its generic core, supplemented by 11 kidney disease targeted domains:

Symptoms/Problems of Kidney disease (34 items), Effects of Kidney Disease (20 items), Burden

of Kidney Disease (4 items), Work Status (4 items), Cognitive Function (6 items), Quality of

Social Interactions (4 items), Sexual Function (4 items), Sleep (9 items), Social Support (4

items), Dialysis Staff Encouragement (6 items), and Patient Satisfaction (2 items).

The original psychometric analyses of the KDQOL-SF showed that these targeted scales had

acceptable to excellent internal consistency reliabilities,36 ranging from 0.76 (Sleep) to 0.94

(Effects of Kidney Disease). A factor analysis revealed that these 11 scales and the SF-36 scales

loaded substantially on 4 factors representing physical health, mental health, kidney disease-

targeted health, and patient satisfaction. Construct validity was evidenced through significant

associations with patient-reported good days, patient-reported bad days, self-rating of quality of

life compared to those without kidney disease, ability to do everything the patient wants to do,

number of disability days in the last 30 days, and overall rating of health.

The KDQOL-36 includes the SF-12 as the generic core and 24 additional items targeted at

kidney disease (Symptoms/Problems, Effects of Kidney Disease, and Burden of Kidney

Disease). The Symptoms/Problems scale includes 12 potential symptoms kidney patients may

face (e.g., chest pain, dry and itchy skin, nausea). Patients are asked to rate how bothered they

14

are by each symptom from “Not at all bothered” to “Extremely bothered”, then a Symptoms/

Problems scale score is created by calculating the mean of these responses (ranging from 1-5,

with higher scores indicating lower symptom burden). The Effects of Kidney Disease scale

contains 8 items and examines the practical impacts of kidney disease on the patient’s activities

(e.g., diet and travel), independence (e.g., dependence upon doctors), and self-concept (e.g.,

personal appearance). The response options and scoring for this scale are the same as that of the

“Symptoms/Problems” scale, and higher scores are interpreted as lower negative effect from

kidney disease. Finally, the Burden of Kidney Disease scale contains 4 items representing

potential ways kidney disease may have an overall burden on their life (e.g., “My kidney disease

interferes too much with my life”) and asks patients to rate each from “Definitely true” to

“Definitely false”. The scoring is the same as the other two KDQOL scales, and they are

interpreted as higher scores indicate lower overall burden from kidney disease.

Table 2. provides details of the KDQOL scales and several other HRQOL instruments used

in dialysis. One generic HRQOL measure, the SF-36, is featured due to its frequent application

among dialysis patients, as well as its use as the generic HRQOL measures included in the

KDQOL-SF. The remaining scales are kidney or dialysis-targeted. For each scale, advantages

and disadvantages to use in dialysis are specified. Important advantages of the measures include

patient involvement in measure development, comprehensive coverage of domains and

symptoms relevant to kidney patients, good psychometric properties, and sufficient brevity to

reduce administrator (provider) and respondent (patient) burden. Disadvantages largely comprise

the absence of these characteristics.

15

Table 2. Advantages and Disadvantages of HRQOL Measures for Dialysis Patients Measure Advantages Disadvantages SF-3637,38 • Developed with patient input

• Multi-dimensional HRQOL scale with physical or mental health component scores and 8 domain subscales

• Evidence of reliability and validity

• No kidney-targeted items or scales • May be too long for all applications

Kidney Disease Quality of Life-Short Form (KDQOL-SF)36,39

• Developed with patient input • Contains generic and targeted HRQOL

scales • Multi-dimensional set of scales with 11

kidney disease subscales representing many HRQOL domains

• Evidence of reliability and validity

• Full instrument may be too long for all applications

KDQOL-3640

• Brief • Evidence of reliability and validity

• Less comprehensive coverage of HRQOL domains than KDQOL-SF

100-Category Checklist41

• Comprehensive coverage of symptoms and problems associated with kidney disease

• Evidence of reliability and validity

• More focus on functioning and symptoms over wellbeing

• Not developed with patient input

Short Version Checklist42

• Brief • Evidence of reliability and validity • More patient input to item selection

compared to 100-Category Checklist

• More focus on functioning and symptoms over wellbeing

CHOICE Health Experience Questionnaire (CHEQ)43

• Developed with patient input • Contains 21 kidney disease targeted

domains • Evidence of reliability and validity

• May be too long for all applications • Rigorous approach to assessment of

dimensional structure not taken

Dialysis Symptom Index (DSI)44

• Patient-generated, comprehensive list of symptoms associated with dialysis

• Evidence of reliability and validity

• Focused on symptoms only

ESRD-SCLa,45 • Multi-dimensional symptom scale • Evidence of reliability and validity

• Not developed with patient input • Focused on symptoms only • May be too long for all applications

Ferrans & Powers Quality of Life Index46

• Multi-dimensional HRQOL scale representing 4 domains with dialysis-targeted items

• Evidence of reliability and validity

• Not all items and domains health-related • May be too long for all applications

Hemodialysis Quality of Life Questionnaire47

• Multi-dimensional HRQOL scale with dialysis-targeted items

• Items generated with patient input • Evidence of reliability and validity

• May be too long for all applications

Kidney Disease Questionnaire48

• Developed with patient input • Multidimensional profile measure (5

total scales) covering physical, mental, and social health

• Evidence of reliability and validity

• No generic HRQOL items included, potentially limiting comparability among patient populations

Physical Symptom Distress Scale49

• Evidence of reliability and validity • Brief

• Not developed with patient input • Focused on physical symptoms only

aEnd-Stage Renal Disease Symptom Checklist – Transplant Module

16

Considering the advantages and disadvantages of the HRQOL measures considered in

Table 2., we recommend the continued use of the KDQOL-36 instrument with dialysis

patients for the purposes of dialysis centers’ internal quality improvement. The KDQOL-36

has attractive psychometric properties, and it has been successfully applied to date with many

thousands of dialysis patients, providing an unrivaled opportunity to compare individual patient

scores to norms from the general dialysis population or from subgroups within this population.

Despite these properties, we do not recommend, at this time, that HRQOL be used by CMS for

dialysis quality assessment, including KDQOL-36 scores. This issue has been debated in the

literature50,51, but further research on the potential ramifications of using any HRQOL measure to

rate dialysis center performance should be conducted before this strategy is pursued.

There are opportunities to improve the KDQOL-36. The incorporation of the SF-12 as the

KDQOL-36’s generic HRQOL companion is no longer ideal. As described in the PROMIS

subsection above, there have been major advancements in HRQOL measurement science, and

the PROMIS measures now represent the state-of-the-science in generic HRQOL measures.

Additionally, while the KDQOL-36 subscales have represented important dimensions of

HRQOL for dialysis patients, they were developed over 20 years ago, and a changing dialysis

population could signal the need for an update, or at least re-assessment, of these kidney-targeted

scales. Therefore, we recommend that a new version of the KDQOL-36 be developed with

PROMIS measures as a generic core, and exploration of potential for fine-tuning among

the kidney disease-targeted scales. An important caveat to this recommendation is that an

approach should be taken such that new scores yielded from an updated version of the KDQOL-

36 should be statistically linked to the original version. Over and above its incorporation into a

newer version of the KDQOL-36, we do not recommend additional use of the PROMIS

17

measures for mandated quality improvement or outcomes monitoring in dialysis centers.

However, specific uses of any of the PROMIS measures, like research projects in which

PROMIS-relevant domains are involved, are strongly recommended.

Other Types of PRMs

In addition to the PROs described in the previous section, there are many other types of

PRMs that play a critical role in understanding patients’ health and health care experiences in

dialysis. Considering Fung and Hays’s framework (Figure 1), other types of PRMs that need to

be considered include patients’ health behaviors, preferences for care, patients’ experiences with

care, and even patients’ decision-making characteristics about how they treat their kidney

disease. There are multiple PRMs that fit these categories in use in research and in clinical

practice, though there is significant opportunity to expand their use.

CAHPS In-Center Hemodialysis Survey

The CAHPS In-Center Hemodialysis Survey was supported by the Agency for Health

Research and Quality and CMS.52 CAHPS surveys are based on a definition of patient

experience as “the range of interactions that patients have with the health care system, including

their care from health plans, and from doctors, nurses, and staff in hospitals, physician practices,

and other health care facilities”.52 CMS has adopted several CAHPS measures for quality

improvement in addition to ICH-CAHPS, including the CAHPS Hospital, Home and

Community-Based Services, Hospice, Surgery, and Medicare ambulatory surveys. 53 The ICH-

CAHPS survey items are targeted at care provided to hemodialysis patients, and these items

would not be appropriate for consumers of other types of health services.

The ICH-CAHPS includes 3 composites, including Nephrologists Communication and

Caring (6 items; e.g., “In the last 3 months, how often did your kidney doctors explain things in a

18

way that was easy for you to understand?”), Providing Information to Patients (9 items; “Did

dialysis center staff at this center ever review your rights as a patient with you?”), and Quality of

Dialysis Center Care and Operations (17 items; “In the last 3 months, how often did the dialysis

center staff show respect for what you had to say?"). Additionally, 3 other items provide global

ratings of patients’ experience with their kidney doctors, dialysis center staff, and dialysis center.

Most ICH-CAHPs items (but not all) have a range of four response options from “Never” to

“Always” or a range of two options, including “Yes” and “No”. For each composite, scores are

created by determining the proportion of answers to each response option for all questions in the

composite, then averaging the proportion of those responding to each answer choice in all

questions; e.g., “Top Box” scores refer to the average proportion of the most positive responses.

Item-scale correlations and internal consistency reliability estimates have provided supported for

the reliability of these scales: Nephrologists Communication and Caring α = 0.89, Quality of

Dialysis Center Care and Operations α = 0.93, and Providing Information to Patients α = 0.75.54

Due to CMS’s use of ICH-CAHPS as a clinical measure in the payment year (PY) 2019 QIP,

it is assessed twice yearly in all dialysis centers throughout the United States. Figure 3 shows

examples of national and state averages of ICH-CAHPS surveys in 2015. The results of patients’

19

reports about different dialysis clinics with the ICH-CAHPS are available to view on CMS’s

Dialysis Facility Compare website: https://www.medicare.gov/dialysisfacilitycompare/. These

comparisons show differences between centers of interest, and to state and national averages. An

example of comparison of two centers is given in Figure 4. Recent reports from CMS indicate

that the ICH-CAHPS will continue to play a large role in dialysis service evaluation and figure

into CMS’s ratings of dialysis center performance.55

Given its attractive measurement properties and its ability to be used for

comparisons among clinics and to state and national norms, we recommend the continued

use of the ICH-CAHPS for CMS’s dialysis center performance monitoring. However, there

are also opportunities to optimize this measure, especially to make it more parsimonious,

reducing burden among patients and providers. A recent report detailed efforts to shorten the

CAHPS Clinician and Group adult survey without significant reduction in reliability or

clinically-important content.56 These analyses found that Provider Communication and the

Access scales could be reduced from 6 and 5 to 2 items, respectively. Noting the length of the

ICH-CAHPS composites at 6 items (Nephrologists Communication and Caring), 9 items

(Providing Information to Patients), and 17 items (Quality of Dialysis Center Care and

Operations), the ICH-CAHPS is ripe for similar analyses.

20

Additional PRMs

There are several other PRMs relevant to dialysis patients. Commonly assessed health

behaviors include diet and exercise regimen;57 use of alcohol, tobacco, or other substances;57 and

adherence to medications,58 dialysis appointments,59 or other prescribed treatments. Similarly,

preferences for care encompasses many patient preferences about use of health care like the

degree of agency in their relationship and communication with providers60 to preferences for

end-of-life care.61 Finally, and somewhat related to preferences for care, understanding the

patient characteristics that influence their decision-making about their care represent important

PRMs and may include readiness for particular types of treatment (e.g,, peritoneal dialysis),

perceived benefits and costs of different treatments, self-efficacy to pursue different treatment

options, and knowledge of treatment options.62

Patients’ decision-making about their treatment is a particularly important domain where the

use of PRMs should be expanded in clinical dialysis care. As noted above, dialysis is not the

only treatment open to kidney patients, and CMS has required that patients have the opportunity

to learn about their other options. According to CMS’s 2008 Conditions for Coverage, dialysis

centers must provide information about the option for kidney transplant to each dialysis patient,

and indicate that they have done so on CMS Form-2728 at the time of initiation of chronic

dialysis. In part, this requirement reflects the need to ensure that patients are able to make an

informed decision, and therefore give informed consent, to their dialysis treatment. To date, this

report on Form-2728 is made by the dialysis provider, but there is evidence that patients actually

report being educated about transplant less-often than providers report educating them when data

from Form-2728 are compared to those from patient surveys.63 Studies like this indicate that

incorporation of patient reports about whether they have received adequate education for their

21

treatment options, along with reports about their preferences, may be better indicators of whether

informed decision-making and consent around treatment choices actually occur among dialysis

patients. Therefore, we recommend that a PRM of whether patients have been informed

about their option for transplant be adopted.

V. Administering PRMs to Dialysis Patients

There are several options for administering PRMs. As the use of PRMs in dialysis clinics has

expanded, clinicians and researchers have attempted to identify the best ways to administer these

measures. In 2015, ISOQOL conducted a comprehensive assessment of the resources needed and

tradeoffs associated with different modes of administration of PRMs.17 (Table 3) Each mode of

administration must be considered within the context of its location, including in the clinic or in

the patient’s home or another personal location, where surveys can be administered by mail,

phone, or on the internet (web-based). Within the clinic, surveys can be self-administered,

interview-administered, or computer administered. On the phone, the concentration is on

interview administration or administration through an automated, voice-activation survey

system. By mail, surveys are strictly self-administered by the patient and returned by mail.

Similarly, on the web, surveys are self-administered by patients. Each of these modes requires

specific resources that have implications for their feasibility. For example, when surveys are

administered in the clinic, a personal space is required for the survey to be completed to ensure

the patient’s privacy and confidentiality. Mail, telephone, and web-based surveys can overcome

this need since they can be administered in the patient’s home. However, successfully

conducting surveys with any of these methods requires consideration of needs for staff,

technology, and potentially informatics infrastructure that can be expensive.

22

Table 3. International Society for Quality of Life Research Summary of Mode of Administration for PRMs

Resources Needed Advantages Disadvantages

In-Clinic Self

Admin. • Personnel to supervise

and assist • Space • Personnel for data

entry

• Low-technology requirements

• Implemented in any clinical setting

• Relatively low cost

• Problem with low literacy patients & visual handicap

• Difficult with other special populations (e.g., very young, very old)

• Higher rate of missing data Interview Admin.

• Skilled interviewer • Space • Personnel for data

entry

• More personal • In-depth questioning • No issues with literacy

and/or visual handicap

• Relatively expensive • Social desirability bias • Staff time

Computer Admin.

• Personnel to supervise and assist

• Software to collect & report data

• Efficient data capture and entry

• Problems finding space/providing privacy

• Costs to obtain & maintain PRO system

• Potential software problems Mail

Self Admin.

• Personnel to manage mailing

• Personnel for data entry

• Low-technology requirements

• Potentially simpler logistics than in-clinic administration

• Relatively low cost

• High non-response rate • Cannot ensure patient completes

questionnaire alone • Hard to respond immediately to patient

needs • Challenges scheduling assessment near

clinical visit • Other limitations similar to Self-

Administered In-Clinic Telephone Interview Admin.

• Skilled interviewer • Personnel for data

entry

• More personal • More convenient for

patient • Largely circumvents

literacy problem and/or visual handicap

• Lack of visual cues as compared to face-to-face

• Relatively expensive • Potential problem with social

desirability • Some topics may be more difficult to

address Voice Activated

• Personnel to oversee data collection

• Validated interactive voice response (IVR) system

• Low cost due to automation

• May not be accepted by patients • Costs to obtain & maintain IVR

system • Requires process to track and respond

to any urgent problem reported by patients

• Other disadvantages similar to Live Telephone Interview, plus impersonal nature

Telephone Web-Based • Systems management

personnel • Software to collect and

report the PRO data • Training patients

• Efficient data capture with simultaneous data entry

• Convenient for patient • Flexible timing for data

collection

• Difficult to ensure privacy • Upfront costs for the PRO system and

maintenance • Potential software problems

Reprinted from Aaronson N, Choucair A, Elliott T, et al. User’s Guide to Implementing Patient-Reported Outcomes Assessment in Clinical Practice. International Society for Quality of Life Research; 2015.

23

The potential efficiencies of electronic survey administration methods have seen this

approach grow in recent years. One particular benefit accrued in web-based surveys is the direct

input of data into a database that can be immediately sourced for analysis. Through this

approach, the need for data entry into a database is eliminated, which is attractive not only

because it reduces the amount of personnel and time dedicated to administering PRMs, but also

because it prevents data entry error. Once data is entered through a web interface, the ease of

integrating these data within the electronic medical record is increased as well. Given that the

ultimate objective of collecting PRMs is to combine them with other important clinical data, the

ability to do so with ease and in real time is a considerable benefit.

One example of electronic data entry platforms for PRMs in dialysis clinics is found in the

Medical Education Institute’s (MEIs) administration of the KDQOL-36.64 The MEI uses the

KDQOL-36 in dialysis centers throughout the United States as part of their KDQOL-Complete

program. The KDQOL-Complete programs helps meet the CMS requirement to create

individualized care plans for each dialysis patient.64 The KDQOL-Complete program assesses

dialysis patients using the KDQOL-36 and tailors care plans for each patient in order to increase

areas of HRQOL that are below expectations. The ability to enter data electronically into the

KDQOL-Complete computerized platform allows for automated scoring of the KDQOL-36 so

that individual patients’ scores can be viewed immediately. Additionally, this computerized

system can compare an individual patient’s scores to national norms and generate illustrative

graphics to help the dialysis provider and patient understand the scores.

Despite these benefits, some considerations for the integrity of electronic PRM

administration should be made before pursuing this strategy. Many instruments were

designed for paper/pencil, but there is a growing interest in whether or not they can be

24

administered electronically.65 PRMs often do not need to be completely redeveloped for

electronic administration, but additional testing for equivalence should be conducted, leading to

some instrument modification.65 The types of changes needed range from relatively minor to

extensive. Examples of small changes include things like updates to instructions and formatting.

Examples of moderate changes include things like updates to item wording. Examples of

significant changes include serious changes to item wording or response options. When tested,

there often turns out to be little difference between electronic and paper and pencil versions of

instruments. Gwaltney and colleagues performed a meta-analysis of RCTs testing the differences

between these modes of administration among PRMs.66 Among 46 studies, they found that mean

difference in scores between electronically and paper-and-pencil administered instruments was,

on average, 0.2%, and the average weighted correlation between scores, including multiple types

of correlation coefficient and weighted kappa, was 0.90 (95% CI: 0.87-0.94). Additionally,

regarding PROMIS measures specifically, a cross-over design RCT was used to compare mode

of administration on item responses for physical function, fatigue, and depression scales, and

found no differences in responses when administered on an interactive voice response system,

paper-and-pencil response, and electronic survey approaches, including on computers.67 These

results are encouraging, and similar efforts should be made for key PRMs used in dialysis

to determine if any modifications are needed before electronic administration is advised.

In addition to the mode of administration, there are differences in the mode of survey

dissemination that are important to consider, and may have an impact on the success of the

survey. Anastario and colleagues compared protocols for the distribution of the CAHPS clinician

and group surveys.68 In comparison to a mail-based dissemination protocol, a hand-out protocol

had a sub-optimal distribution rate (74%) and a lower response rate (40% vs 58%).

25

The appropriate timing of PRM administration in clinic depends on the purpose of the

assessments. In the context of clinical trials, the FDA’s Center for Drug Evaluation and Research

identified the following possible assessment approaches: at the beginning and end of a particular

treatment, at or after important medical events occur (e.g., starting a new type of dialysis,

development of a clinically significant comorbidity), or at regular, repeated intervals to examine

progress of a chronic condition.69,70 Though all of these approaches may be relevant to PRM

assessment as part of standard clinical care in dialysis, long term monitoring over repeated

intervals may be most relevant. For instance, a HRQOL measure may be assessed when a patient

begins dialysis then administered at regular intervals thereafter to track potential improving or

worsening health.

Indeed, the most benefit may be realized from administering PRMs at multiple occasions

with patients.71 Multiple administrations allow clinicians to track changes over time to monitor

disease progression or to examine responses to changes in treatments. Despite this benefit, most

often, PRMs are only administered once to patients.71 This concurs within the field of dialysis as

well. Although CMS has mandated the assessment of HRQOL and patient experience, these

PRMs are most often assessed only once or twice for each patient. While single timepoint

assessments are certainly superior to no PRM assessment at all, repeated assessments drastically

increase the capacity to understand patients’ health and experiences with dialysis so that

adjustments in care can be made. The value of longitudinal assessment of PRMs has been raised

specifically in regards to the CAHPS measures.72 CAHPS items ask about experiences with care

over the previous 6 months; if repeated every 6 months, experiences with care occurring over

specific durations of time can be isolated, and the reports generated can be easily used for quality

improvement if necessary. Other approaches, wherein all care received in the past is asked about,

26

do not offer the opportunity for longitudinal assessment, since repeated assessments of such

measures would yield uninterpretable information due to the inability to pinpoint which specific

durations of care were being described by the reports.73

There are a few challenges to longitudinal assessment of PRMs. First, repeated measures on

the same patient over time may entail a lack of timely reports of results to providers. Second, due

to frequent patient turnover in some clinics, longitudinal measurement may not be possible for

all patients, creating a potential challenge to standardized longitudinal measurement for all

patients. Finally, many of the barriers to cross-sectional PRM measurement may be compounded

with repeated measurement, including increased costs, provider burden, and patient burden.

Considering both the benefits and burdens of repeated PRM measurement, we recommend

that key PRMs be assessed twice annually for each patient so that some change can be

observed in response to changes in treatment plans. The CMS requirements for the ICH-

CAHPS already adhere to this recommendation, though we suggest that the KDQOL-36

assessment recommendations be altered so that it is also assessed twice annually.

VI. Challenges Though there are many clear benefits to administering PRMs in dialysis centers, there are also

challenges. First, though we noted in the previous section the significant benefits to assessing

PRMs over multiple timepoints, doing so can be burdensome for dialysis providers and dialysis

patients. There is evidence that dialysis providers often have an extensive workload, and adding

of PRMs adds to this workload.74 A related practical challenge regards the additional cost

associated with administering PRMs in the dialysis clinic. The staff time and resources, as well

as material costs, required to administer a PRM, along with entering the data, then interpreting

the results and incorporating the learning into clinical practice is not free, and may be difficult to

27

justify in clinics without significant discretionary spending.1 We recommend that new

explorations be launched to identify mechanisms for CMS to reimburse these costs.

Another important barrier regards staff training to administer key PRMs1. When PRMs are

administered in an interview style from the dialysis staff, they require understanding of

standardized survey administration techniques, including ways to elicit unbiased, accurate

responses and trouble shoot when patients have questions, understand potentially complex skip-

patterns, and screen for patient responses that may be untruthful or not genuine (e.g., a patient

gives several of the same responses consecutively quickly in order to complete the assessment).

Even when PRMs are administered through self-administered surveys (e.g., mailed to the

patients), data entry protocols to reduce error are recommended, and these require training.

Therefore, the continued development of effective, low-cost training programs to help

providers administer PRMs, including e-learning programs, should be a top priority.

Additionally, patients’ willingness or ability to participate in surveys of PRMs represents a

potential barrier. Response rates to such surveys vary considerably and by the mode of

administration. For example, in a randomized study of response rates to PRM-based surveys

among 2,400 hip-replacement patients in Sweden, Rolfson and colleagues found a 92% response

rate for paper surveys disseminated by mail vs. a 49% response rate for internet-based survey

(p<0.001) .75 In a larger study of 131,447 knee, hip, hernia, or varicose vein surgery patients in

England, response rates to a post-surgical mail survey ranged between 65% (varicose vein

surgery) to 85% (hip surgery) .76

Though the most significant contribution of PRMs is that they represent the patient’s

perspective, patient reports may generate biased and inaccurate data. Patients may give

inaccurate responses in interview-based surveys in which they are speaking directly to a care

28

provider, either in the clinic or over the phone.17 This may be because of social desirability or

wanting to please the provider by reporting that their health is good or that they have had good

experiences with care. Relatedly, patients may not trust providers enough to divulge sensitive

information. A particularly apt example involves reports of patients’ satisfaction with care when

reported directly to the providers of that care. In this circumstance, patients who feel unsatisfied

with their care may not feel comfortable to report this to providers. An additional source of bias

for PRMs is recall bias, wherein patients may not have sufficient memory of the events or health

states they are asked about to provide accurate responses.77 One review found that recall bias was

particularly high for reporting on HRQOL (e.g., health transition) and pain intensity.78 Another

important issue with administering PRMs in clinic regards the potential for response burden.

Some patients, especially ones who are very sick, may feel burdened by completing extensive

surveys or interviews. These issues and other can have a significant impact on the quality of the

data from PRMs administered in dialysis centers.

Finally, the PRMs themselves can create barriers to clinical implementation. PRMs without

established guidelines for clinically or minimally important differences (e.g., for change in health

status) may be difficult to interpret. Similarly, measures without established population and case-

mix adjusted norms for comparison may be less meaningful to discuss with patients since it

cannot be easily determined how any one patient compares to average dialysis patients, or to

other dialysis patients with similar demographic and clinical characteristics. PRMs that are used

with individuals in clinic must have high reliability, 0.90 or higher.79 Achieving such high

reliability requires including many items in PRMs, which itself can increase response burden for

patients. Therefore, it is important to select measures that balance brevity and high reliability,

29

though trade-offs in one, if not both, of these areas may be required in choosing a measure that is

feasible to implement in dialysis centers.

VII. Recommendations

The philosophy underlying PRMs is that patients have critical information about their own

health that should be included in creating treatment plans. In choosing PRMs to administer in

dialysis clinics, the top priority should be on collecting information that elicits patients’

perspectives and preferences for their health and use of healthcare, and that can lead to

improvements in these domains through clinical interventions. In selecting measures, the

psychometric properties should be considered, and measures with extensive support for their

reliability and validity should be given priority. When the goal is to use the measure for

individual assessment and intervention, it is critical that measures with high reliability be

selected.

As performance monitoring and incentives for quality improvement expand within the

field of dialysis, and PRMs play a potentially larger role in this enterprise, it is important to

recognize the benefits that may result and how PRMs can improve dialysis care. CMS is already

incentivizing the use of PRMs in dialysis, which rewards centers that invest more in patient-

centered care in their QIP program. Another way CMS leverages the results of PRMs to impact

care is by publishing the results of patients’ reports of their health and ratings of experiences of

care and allowing patients to choose providers based on these results.1 In this way, the patient’s

voice can define high- versus low-quality care, and chart a path toward truly patient-oriented

dialysis. Dialysis providers should embrace this trend. Increasing the patient’s voice in the

provision of dialysis will not supplant or replace the provider’s, but will provide critical feedback

on ways that providers can monitor the impact of dialysis and make changes where necessary.80

30

To date, little to no data on the impacts of assessing HRQOL and patient experience in the QIP

process is available. This is an area of the highest importance in ongoing research.

Although some research on the provision of PRMs in medical care in general, and in

dialysis specifically, has emerged, there are many areas requiring further exploration. First, more

research should be conducted to determine the most efficient and successful modes of

administering PRMs among dialysis patients. This should take the form of a randomized trial

comparing response rates and completeness of responses between different modes of

administration and delivery like paper in-clinic versus electronic surveys in patients’ homes.

Additionally, though large-scale investigations of the impact of administration of PRMs in clinic

have been generated for other types of treatment, such evaluations, also randomized, are needed

within dialysis.

In addition to these general recommendations, we summarize the major, specific

recommendations generated throughout the sections above are summarized here in Table 4.

Table 4. Specific Recommendations for Continued Use of PRMs in Dialysis Centers Category Recommendations Selection of PRMs v Continue the use of KDQOL-36 for dialysis centers’

internal quality improvement activities and the ICH-CAHPS for public dialysis center performance monitoring, but promote efforts to modify these instruments by incorporating PROMIS general health items (KDQOL-36) and reducing the length of the ICH-CAHPS.

v Adopt a PRM of whether dialysis patients have been informed about their option for transplant.

Mode of Administration v Evaluate equivalence between electronic and paper versions of PRMs prior to widespread use of electronic administration.

Support for PRM Use v Explore reimbursement of costs of PRM administration by the Centers for Medicare and Medicaid Services.

v Continue development of provider trainings in PRM administration and interpretation.

31

VIII. Conclusions In conclusion, there is a lot to celebrate in the field of PRMs in dialysis. Many strong measures

have been developed and validated, and their use in dialysis centers is extensive. Despite these

successes, there is significant room for improvement. We have identified several pointed

recommendations for improving the use of PRMs in dialysis. These recommendations are

intended to help dialysis care decision-makers, clinicians, and applied researchers continue to

improve the excellent track record of PRM use in dialysis.

References

1. Fung CH, Hays RD. Prospects and challenges in using patient-reported outcomes in clinical practice. Qual Life Res. 2008;17(10):1297-1302.

2. Food and Drug Administration. Guidance for Industry Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. Rockville, MD: U.S. Department of Health and Human Services,;2009.

3. Waterman AD, Robbins ML, Paiva AL, et al. Your Path to Transplant: a randomized controlled trial of a tailored computer education intervention to increase living donor kidney transplant. BMC nephrology. 2014;15:166.

4. Ganz PA. Quality of life and the patient with cancer. Individual and policy implications. Cancer. 1994;74(4 Suppl):1445-1452.

5. Cella D, Grunwald V, Nathan P, et al. Quality of life in patients with advanced renal cell carcinoma given nivolumab versus everolimus in CheckMate 025: a randomised, open-label, phase 3 trial. The Lancet. Oncology. 2016;17(7):994-1003.

6. Beaumont JL, Salsman JM, Diaz J, et al. Quality-adjusted time without symptoms or toxicity analysis of pazopanib versus sunitinib in patients with renal cell carcinoma. Cancer. 2016;122(7):1108-1115.

7. Andersen RM. Revisiting the behavioral model and access to medical care: does it matter? Journal of health and social behavior. 1995;36(1):1-10.

8. AHRQ. What is Patient Experience? 2016; https://www.ahrq.gov/cahps/about-cahps/patient-experience/index.html. Accessed February 11, 2017.

9. Cella D, Riley W, Stone A, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. Journal of clinical epidemiology. 2010;63(11):1179-1194.

10. Peipert JD, Beaumont JL, Bode R, Cella D, Garcia SF, Hahn EA. Development and validation of the functional assessment of chronic illness therapy treatment satisfaction (FACIT TS) measures. Qual Life Res. 2014;23(3):815-824.

32

11. Dyer N, Sorra JS, Smith SA, Cleary PD, Hays RD. Psychometric properties of the Consumer Assessment of Healthcare Providers and Systems (CAHPS(R)) Clinician and Group Adult Visit Survey. Medical care. 2012;50 Suppl:S28-34.

12. Degner LF, Sloan JA, Venkatesh P. The Control Preferences Scale. The Canadian journal of nursing research = Revue canadienne de recherche en sciences infirmieres. 1997;29(3):21-43.

13. Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Quality of Life Research. 2008;17(2):179-193.

14. Justice AC, Rabeneck L, Hays RD, Wu AW, Bozzette SA, Group ftOCotACT. Sensitivity, Specificity, Reliability, and Clinical Validity of Provider-Reported Symptoms: A Comparison With Self-Reported Symptoms. JAIDS Journal of Acquired Immune Deficiency Syndromes. 1999;21(2):126-133.

15. Palmer SC, de Berardis G, Craig JC, et al. Patient satisfaction with in-centre haemodialysis care: an international survey. BMJ open. 2014;4(5):e005020.

16. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Quality of Life Research. 2012;21(8):1305-1314.

17. Aaronson N, Choucair A, Elliott T, et al. User’s Guide to Implementing Patient-Reported Outcomes Assessment in Clinical Practice. International Society for Quality of Life Research;2015.

18. Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Quality of Life Research. 2009;18(1):115-123.

19. Calvert M, Blazeby J, Altman DG, et al. Reporting of patient-reported outcomes in randomized trials: The consort pro extension. Jama. 2013;309(8):814-822.

20. Scoggins JF, Patrick DL. The Use of Patient-Reported Outcomes Instruments in Registered Clinical Trials: Evidence from ClinicalTrials.gov. Contemporary clinical trials. 2009;30(4):289-292.

21. Basch E, Deal AM, Kris MG, et al. Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. Journal of Clinical Oncology. 2016;34(6):557-565.

22. Basch E. Patient-Reported Outcomes - Harnessing Patients' Voices to Improve Clinical Care. N Engl J Med. 2017;376(2):105-108.

23. Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples. Qual Life Res. 2009;18(1):125-136.

24. Antunes B, Harding R, Higginson IJ. Implementing patient-reported outcome measures in palliative care clinical practice: A systematic review of facilitators and barriers. Palliative Medicine. 2014;28(2):158-175.

25. Price RA, Elliott MN, Cleary PD, Zaslavsky AM, Hays RD. Should Health Care Providers be Accountable for Patients' Care Experiences? Journal of General Internal Medicine. 2015;30(2):253-256.

26. Sledge R. KDQOL-36 and the interdisciplinary team. Nephrology news & issues. 2010;24(7):36-38.

27. Gabbay E, Meyer KB. Incentives for caution: the in-center hemodialysis consumer assessment of healthcare providers and systems survey and experience of care. Clin J Am Soc Nephrol. 2014;9(6):1005-1006.

33

28. Rothrock NE, Hays RD, Spritzer K, Yount SE, Riley W, Cella D. Relative to the general US population, chronic diseases are associated with poorer health-related quality of life as measured by the Patient-Reported Outcomes Measurement Information System (PROMIS). Journal of clinical epidemiology. 2010;63(11):1195-1204.

29. Cella D, Yount S, Rothrock N, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Medical care. 2007;45(5 Suppl 1):S3-S11.

30. Reeve BB, Hays RD, Bjorner JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical care. 2007;45(5 Suppl 1):S22-31.

31. Jhamb M, Argyropoulos C, Steel JL, et al. Correlates and outcomes of fatigue among incident dialysis patients. Clin J Am Soc Nephrol. 2009;4(11):1779-1786.

32. Finkelstein FO, Arsenault KL, Taveras A, Awuah K, Finkelstein SH. Assessing and improving the health-related quality of life of patients with ESRD. Nat Rev Nephrol. 2012;8(12):718-724.

33. DeWalt DA, Gross HE, Gipson DS, et al. PROMIS((R)) pediatric self-report scales distinguish subgroups of children within and across six common pediatric chronic health conditions. Qual Life Res. 2015;24(9):2195-2208.

34. Selewski DT, Massengill SF, Troost JP, et al. Gaining the Patient Reported Outcomes Measurement Information System (PROMIS) perspective in chronic kidney disease: a Midwest Pediatric Nephrology Consortium study. Pediatr Nephrol. 2014;29(12):2347-2356.

35. Guyatt GH, Feeny DH, Patrick DL. Measuring health-related quality of life. Annals of internal medicine. 1993;118(8):622-629.

36. Hays RD, Kallich JD, Mapes DL, Coons SJ, Carter WB. Development of the kidney disease quality of life (KDQOL) instrument. Qual Life Res. 1994;3(5):329-338.

37. Ware JE, Jr., Gandek B. Overview of the SF-36 Health Survey and the International Quality of Life Assessment (IQOLA) Project. Journal of clinical epidemiology. 1998;51(11):903-912.

38. Hays RD, Sherbourne CD, Manzel RM. User's Manual for the Medical Outcomes Study (MOS) Core Measures of Health-Related Quality of Life. Santa Monica, CA: RAND;1995.

39. Hays RD, Kallich JD, Mapes DL, et al. Kidney disease quality of life short form (KDQOL-SF): A manual for use and scoring. Third ed. Santa Monica, CA: RAND; 1997.

40. RAND Health. Kidney Disease Quality of Life Instrument (KDQOL). 2017; http://www.rand.org/health/surveys_tools/kdqol.html. Accessed January 21, 2017.

41. Tsutsui H, Koike T, Yamazaki C, et al. Identification of Hemodialysis Patients' Common Problems Using the International Classification of Functioning, Disability and Health. Therapeutic Apheresis and Dialysis. 2009;13(3):186-192.

42. Tsutsui H, Ohkubo T, Tsuruta Y, Kato S, Yasuda Y, Oshida Y. Development and validation of a short-version checklist for patients undergoing hemodialysis based on the International Classification of Functioning, Disability and Health. Clinical and Experimental Nephrology. 2015;19(5):953-960.

43. Wu AW, Fink NE, Cagney KA, et al. Developing a Health-Related Quality-of-Life Measure for End-Stage Renal Disease: The CHOICE Health Experience Questionnaire. American Journal of Kidney Diseases. 2001;37(1):11-21.

34

44. Weisbord SD, Fried LF, Arnold RM, et al. Development of a symptom assessment instrument for chronic hemodialysis patients: the dialysis symptom index. Journal of pain and symptom management. 2004;27(3):226-240.

45. Franke GH, Reimer J, Kohnle M, Luetkes P, Maehner N, Heemann U. Quality of Life in End-Stage Renal Disease Patients after Successful Kidney Transplantation: Development of the ESRD Symptom Checklist – Transplantation Module. Nephron. 1999;83(1):31-39.

46. Ferrans CE, Powers MJ. Psychometric assessment of the Quality of Life Index. Research in nursing & health. 1992;15(1):29-38.

47. Churchill DN, Wallace JE, Ludwin D, Beecroft ML, Taylor DW. A comparison of evaluative indices of quality of life and cognitive function in hemodialysis patients. Control Clin Trials. 1991;12(4 Suppl):159S-167S.

48. Laupacis A, Muirhead N, Keown P, Wong C. A disease-specific questionnaire for assessing quality of life in patients on hemodialysis. Nephron. 1992;60(3):302-306.

49. Chiou C-P. Development and Psychometric Assessment of the Physical Symptom Distress Scale. Journal of pain and symptom management. 1998;16(2):87-95.

50. Moss AH, Davison SN. How the ESRD quality incentive program could potentially improve quality of life for patients on dialysis. Clin J Am Soc Nephrol. 2015;10(5):888-893.

51. Finkelstein FO. Performance measures in dialysis facilities: what is the goal? Clin J Am Soc Nephrol. 2015;10(1):156-158.

52. Agency for Healthcare and Research Quality. CAHPS: Surveys and Tools to Advance Patient-Centered Care. 2016; https://www.ahrq.gov/cahps/index.html. Accessed January 22, 2017.

53. In-Center Hemodialysis CAHPS Survey. 2016; https://ichcahps.org/Home.aspx. Accessed January 21, 2017.

54. Weidmer BA, Cleary PD, Keller S, et al. Development and evaluation of the CAHPS (Consumer Assessment of Healthcare Providers and Systems) survey for in-center hemodialysis patients. Am J Kidney Dis. 2014;64(5):753-760.

55. Richardson MM, Grobert ME. ICH-CAHPS: what signal on the chadburn? Am J Kidney Dis. 2014;64(5):670-672.

56. Stucky BD, Hays RD, Edelen MO, Gurvey J, Brown JA. Possibilities for Shortening the CAHPS Clinician and Group Survey. Medical care. 2016;54(1):32-37.

57. Lantz PM, House JS, Lepkowski JM, Williams DR, Mero RP, Chen J. Socioeconomic factors, health behaviors, and mortality: Results from a nationally representative prospective study of us adults. Jama. 1998;279(21):1703-1708.

58. Osterberg L, Blaschke T. Adherence to Medication. New England Journal of Medicine. 2005;353(5):487-497.

59. Denhaerynck K, Manhaeve D, Dobbels F, Garzoni D, Nolte C, De Geest S. Prevalence and consequences of nonadherence to hemodialysis regimens. American journal of critical care : an official publication, American Association of Critical-Care Nurses. 2007;16(3):222-235; quiz 236.

60. Vick S, Scott A. Agency in health care. Examining patients' preferences for attributes of the doctor–patient relationship. Journal of Health Economics. 1998;17(5):587-605.

61. Holley JL, Hines SC, Glover JJ, Babrow AS, Badzek LA, Moss AH. Failure of advance care planning to elicit patients' preferences for withdrawal from dialysis. American Journal of Kidney Diseases. 1999;33(4):688-693.

35

62. Glanz K, Rimer B, Viswanath K. Health Behavior and Health Education: Theory, Research, and Practice. 4th ed. San Francisco, CA: Jossey-Bass; 2008.

63. Salter ML, Orandi B, McAdams-DeMarco MA, et al. Patient- and provider-reported information about transplantation and subsequent waitlisting. J Am Soc Nephrol. 2014;25(12):2871-2877.

64. Medical Education Institute. KDQOL Complete. 2016; https://www.kdqol-complete.org/. Accessed December 28th, 2016.

65. Coons SJ, Gwaltney CJ, Hays RD, et al. Recommendations on Evidence Needed to Support Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR ePRO Good Research Practices Task Force Report. Value in Health. 2009;12(4):419-429.

66. Gwaltney CJ, Shields AL, Shiffman S. Equivalence of Electronic and Paper-and-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review. Value in Health. 2008;11(2):322-333.

67. Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, Ware JE. Difference in method of administration did not significantly impact item response: an IRT-based analysis from the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative. Quality of Life Research. 2014;23(1):217-227.

68. Anastario MP, Rodriguez HP, Gallagher PM, et al. A randomized trial comparing mail versus in-office distribution of the CAHPS Clinician and Group Survey. Health Serv Res. 2010;45(5 Pt 1):1345-1359.

69. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: Draft version. Health and Quality of Life Outcomes. 2006.

70. Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ. The routine use of patient reported outcome measures in healthcare settings. Brit Med J. 2010;340.

71. Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: a structured review. Journal of Evaluation in Clinical Practice. 2006;12(5):559-568.

72. Cleary PD, Lubalin J, Hays RD, Short PF, Edgman-Levitan S, Sheridan S. Debating survey approaches. Health affairs. 1998;17(1):265-268.

73. Allen Jr HM, Rogers WH. Consumer Surveys of Health Plan Performance: A Comparison of Content and Approach and a Look to the Future. The Joint Commission Journal on Quality Improvement. 1996;22(12):775-794.

74. Flynn L, Thomas-Hawkins C, Clarke SP. Organizational traits, care processes, and burnout among chronic hemodialysis nurses. Western journal of nursing research. 2009;31(5):569-582.

75. Rolfson O, Salomonsson R, Dahlberg LE, Garellick G. Internet-Based Follow-Up Questionnaire for Measuring Patient-Reported Outcome after Total Hip Replacement Surgery—Reliability and Response Rate. Value in Health. 2011;14(2):316-321.

76. Hutchings A, Neuburger J, Grosse Frie K, Black N, van der Meulen J. Factors associated with non-response in routine use of patient reported outcome measures after elective surgery in England. Health Qual Life Outcomes. 2012;10:34.

77. Schwarz N, Sudman S. Autobiographical Memory and the Validity of Retrospective Reports. New York: Springer; 1994.

36

78. Schmier JK, Halpern MT. Patient recall and recall bias of health state and health status. Expert review of pharmacoeconomics & outcomes research. 2004;4(2):159-163.

79. Nunnally JC. Psychometric theory. 2nd ed. New York: McGraw-Hill; 1978. 80. Agency for Healthcare and Research Quality. Improving the Quality of In-Center

Hemodialysis Care. 2013; https://www.ahrq.gov/cahps/quality-improvement/reports-and-case-studies/Report_ICH-QI-Project.html. Accessed February 9, 2017.