Improving the Evaluation of Measures of Patient Reported Outcomes using Content

Validity Analysis:A Bayesian Randomized Equivalency Experiment

Byron J. GajewskiAssociate Professor of Biostatistics

Associate Professor of Nursing

w/ Coffland, Boyle, Bott, Leopold, Oberhelman, & Dunton

Valmi D. Sousa, PhD, APRN, BC

Associate Professor of Nursing

1958-2010

Background: Measures of Patient Reported Outcomes (PRO)

As evolving healthcare therapies, treatments, and policies are introduced in the U.S., it is more critical than ever to quickly develop new valid and reliable instruments for measuring patient reported outcomes

Examples:o disease management (e.g. diabetes)

o fine motor function

o activities of daily living of older adults (e.g. bathing, dressing…)

o cognitive impairment

Background: Other Behavioral Measures (Scales and Instruments)

Nursing home staff perception of Culture Change

Nursing home staff perception of End-of-Life practices

Registered Nurse (RN) Job Enjoyment in hospitals

Background: Validating measures of PRO & other behavioral measures

Core: Psychometrics “Instruments” (“reliability”)o Also known as “Scales,” or “paper & pencil questionnaires”

Background worko Develop theory

o Draft potential questions (or items)

Validity o Criterion Validity (Gold Standard)

• Often Gold Standard not directly measureable (Job Enjoyment)

o Alternative to Criterion Validity

• Content Validity: Elicit content experts’ opinion (“essential” or “relevance”)

• Construct Validity: Test on participants

– How many “constructs” are we measuring?

– Factor Analysis

Example Instrument: RN Job Enjoyment

strongly agree (6) to strongly disagree (1)

Nurses with whom I work with would say that they:

# Item

1 Are fairly well satisfied with their jobs.

2 Would not consider taking another job.

3 Have to force themselves to work much of the time.

4 Are enthusiastic about their work almost every day.

5 Like their jobs better than the average worker does.

6 Feel that each day on their job will never end.

7 Find real enjoyment in their work.

6

One-Factor analysis measurement model

7

f

ρ1

Z2

Z1

Z4

Z3

ρ2

ρ3

ρ4

Z6

Z5

ρ5

ρ6

Z7

ρ7ρj = corr (f, zj)

f = standardized latent domain scoreo (mean = 0 variance = 1)

zj = standardized response for item j

(1) Validate Factor Structure

(2) Good estimates of ρj

One-Factor analysis measurement model

8

(1) Validate Factor Structure

(2) Good estimates of ρj

(3) Sample sizes? 10/parameter estimate

7 means + 7 variances + 7 correlations = 21 “=“ 210 subjects

Note also: se{g(ρj)}=1/ √n

Outline

Introduction

Content Validity

Related Literature

Study Designo Exact Model

o Approximate model

o Hypotheses

o Sample size calculations and equivalency trial

Results

Discussion & Limitations

Introduction

Instrument validation methodso Content validity

o Construct validity

Traditionally analyzed separately

Integrated analysis of content and construct validity (IACCV)o Both datasets on the same metric

Accomplished using Bayesian methodologyo Expert data – prior distribution

o Participants’ data utilized to update prior information via a posterior distribution

Introduction

Previous use of combination of IACCV and Bayesian methodology

o Instrument measuring nursing home culture change

o Stable estimates of psychometrics parameters via posterior distribution

o Useful results with small sample size

Introduction: Potential Impact of IACCV -> Efficiency!

~76,000 participants will be in instrument development studies in next five years

using IACCV reduces this participant number by 37%, big decrease in manpower and time

IACCV transfers some response burden from vulnerable participants to expert panelso Disabilities/Cancer/low populations

Content Validity

Typical content validity procedureso Content experts

o Instrument available to experts

o Content review tool

o Definition of the construct

o Assessment of item relevance using a four-point Likert scale

• 1 = content not relevant

• 2 = content somewhat relevant

• 3 = content quite relevant

• 4 = content highly relevant

Content Validity

Content validity index for each itemo Calculating proportion

o Focus on responses of “quite” and “highly” relevant

o Minimum item content validity index of 0.80

• At least 80% experts agree that an item is quite or highly relevant

Justification of the content validity index cut-pointo IACCV/Bayesian methodology valid????

Purpose of Study

We hypothesize that experts equate

“relevance” and “correlation” scale

ρj = corr (f, zj)

f = standardized latent domain scoreo (mean = 0 variance = 1)

zj = standardized response for item j

Purpose of Study

Relevancy Responses Corr. Correlation Scale

1 = Not Relevant ↔No Correlation

[0.0 – 0.10)

2 = Somewhat Relevant ↔Small Correlation

[0.10 – 0.30)

3 = Quite Relevant ↔Medium Correlation

[0.30 – 0.50)

4 = Highly Relevant ↔Large Correlation

[0.50 – 1.00)

16

Overall Design

Bayesian design with two group, randomized equivalency study

Registered Nurse Job Enjoyment Scale (Taunton et al. 2004)o National Database of Nursing Quality Indicators TM (NDNQI®)

Role of the site coordinator

Subjects are voluntary from the NDNQI® site coordinator pool

Overall Design

Site Coordinators

(Randomized)

Relevance Group

Relevance Scale

Not Relevant-Highly

Relevant

Correlation Group

Correlation Categories

Categories 0.00 to 1.00

18

Study Participants

Content expertso Total 1,226 site coordinators emailed

o 397 eligible participants volunteered

o All are registered nurses

o 120 participants randomly chosen

o Participants randomized into two groups

• n₁ = n₂ = 60

o Over sampled by 22 to obtain at minimum of 98 completed surveys (see slides later)

Study Design

Tools created using Survey Monkey (http://www.surveymonkey.com/)

Participants received the respective link by email

Survey with 11 itemso Eight items based on Job Enjoyment (7 actual, 1 “sabotage”)

o Three items for basic demographics

Exact Model

Combining all expert responses for all items

g (ρjkm) = g (ρjm) + ejkm

m = 1 ‘relevance’ group and m= 2 ‘correlation’ group

r = 8 Job Enjoyment items

j = item 1,…,8

k = expert opinion 1,…,nm

ρjm = correlation between item and domain

(pooling within the mth group)

ejkm = normally distributed, mean 0, and variance σ²

Exact Modelg (ρjkm) = g (ρjm) + ejkm

ρjkm = kth expert opinion for jth itemo Transformation g(ρ)=1/2log{(1+ρ)/(1-ρ)}

o Allows for correlations of -1 to 1

o Related to xjkm (observed ordinal values)

Measurement model for the correlation of the first item to its

domain from six experts.

ρ1

1

1

1

1

1

1

x12

x11

x14

x13

x16

x15

ρ11

ρ12

ρ13

ρ14

ρ15

ρ16

Approximate Model

xjkm = µjm + exjkm

Modeling x’s on the ordinal scale 1-4

µjm = mean response for jth item from kth expert in the mth

group

exjkm = normally distributed, mean 0, and variance σ2

j

Hypotheses

H1j :| ρj1 - ρj2 | < 0.25

• Exact Model

• g (ρjkm) = g (ρjm) + ejkm

H*1j :| µj1 - µj1 | < 0.5

• Approximate Model

• xjkm = µjm + exjkm

27

Posterior Calculations

Exact Model o Complicated by cutpoints and untransformed scale inferences

o Posterior distribution calculations of ρjkm using Markov chain Monte Carlo (MCMC)

o WinBUGS

o Burned in 1,000 draws and used the next 10,000 iterations

Approximate Modelo Calculations are in closed form

o Can be done in Excel

o Easy sample size calculations

PriorsModel Parameter Distribution Median

(95% Crl)Pr

Approximate µjm N (2.5, σj / √4)n0m = 4 & σ = 1

2.5 (1.52, 3.48) - -

µj1 - µj2 N (0, σj / √2) 0.0 (-1.39, 1.39) 0.52

Exact σ 1/σ2~U(0.01,100)n0m = 8 & σ = 1/n

0.50 (0.03, 0.98) - -

ρjm g(ρjm)~ N(0.5493, 1/ √8) 0.50 (-0.15, 0.85) - -

ρj1 - ρj2 Simulation 0.00 (-0.74, 0.74) 0.52

29

Equivalency Analysis

Posterior calculations

o Approximate and exact models

o Posterior median (50th percentile)

o 95% Credible Intervals (Crl)

• 2.5th percentile

• 97.5 percentile

o Pr(H1j)

Sample Size Calculations Equivalency Trial Sample size calculations

o Each item observed as:

• Approx. normal random variable

•

– Mean µj1 - µj2

– Variance σ2j (1/n1 + 1/n2)

o Prior distribution for µj1 - µj2

• Normal

– Mean 0

– Variance σ2j (1/4 + 1/4)

• Equivalent to four in each group

1 2j jx x

Sample Size Calculations Equivalency Trial Sample size for n1 + n2

(|µj1 - µj2| < 0.5) > λ

OR

P µj1 - µj2({| µj1 - µj2| < 0.5} |xj1, xj2, n1, n2, σ2

j ,4, 4) > λ

(parameters to the right)

A = (xj1, xj2, n1, n2, σ2j ,4, 4)’

Sample Size Calculations Equivalency Trial λ = 90%

= -0.25 and σ2j = 1

Suspect mean differences are “0”

Allow for deviation between 0 and boundary (0.5)

Find the minimal integer n=n1=n2:Min{n | P µj1 - µj2

({| µj1 - µj2| < 0.5} | = -0.25, n, A) > 0.90

Noting the posterior distribution

[ µj1 - µj2 | = -0.25, n, A ] ~ N (-0.25, √2/(4+n)

1 2j jx x

1 2j jx x

1 2j jx x

Sample Size Calculations Equivalency Trial

Plot indicates that n = 49 is sufficient

n=n1=n2=60 to account for possible dropout

Response Rates

Relevance group (m=1) with 59 subjects

Correlation group (m=2) with 51 subjects

Response rates greater in the relevance groupo Beta-Binomial distribution with uniform priors

o Posterior probability = 0.9984

o 95% Credible Interval in difference is (0.04, 0.23)

o Demonstrates correlation group has a significant smaller response rate

RN Demographics

Length of time in Position %

1-5 years in current position 40.4%

6-19 years in current position 34.8%

> 20 years in current position 24.8%

Experience

Total RN experience in US > 20 years 70%

Highest Academic Degree %

Diploma/Associate 9%

Baccalaureate 37%

Masters 47%

Doctorate 6%

Not Applicable 1%

36

Summary Statistics

# Item m=1 ‘Relevance’ (n=59)Response %

m=2 ‘Correlation’ (n=51)Response %

1 2 3 4 1 2 3 4

1 Satisfied with job 0 10 25 64 0 6 26 68

2 Consider another job 3 27 25 44 2 14 44 40

3 Force themselves to work 5 15 29 51 8 10 20 62

4 Enthusiastic to work 0 12 27 61 0 2 34 64

5 Like job better than averageworker does

5 24 37 34 4 10 36 50

6* Are clinically competent 17 31 24 29 10 30 40 20

7 Feel job will never end 7 25 25 42 6 8 34 52

8 Real enjoyment in their work 0 2 22 76 0 6 16 78

Relevance responses (1=“not relevant” to 4=“highly relevant”)Correlation responses (1=“0.00-0.10” to 4=“0.50-1.00”)

37

Summary Statistics

#m=1 (Relevancy) m=2 (Correlation)

n1 s1 n2 s2

1 3.54 59 0.68 3.62 50 0.60

2 3.10 59 0.92 3.22 50 0.76

3 3.25 59 0.90 3.36 50 0.96

4 3.49 59 0.70 3.62 50 0.53

5 3.00 59 0.89 3.32 50 0.82

6* 2.64 59 1.08 2.70 50 0.91

7 3.03 59 0.98 3.32 50 0.87

8 3.75 59 0.48 3.72 50 0.57

1x 2x

38

Approximate Model(Relevancy-Correlation)

Differences

# σ E(µ1-µ2) SE 2.5%-tile 97.5%-tile Prob(H1*)

1 0.64 -0.08 0.12 -0.32 0.17 1.00

2 0.85 -0.12 0.16 -0.44 0.20 0.99

3 0.93 -0.11 0.18 -0.46 0.24 0.99

4 0.63 -0.13 0.12 -0.37 0.11 1.00

5 0.86 -0.32 0.17 -0.64 0.00 0.86*

6* 1.00 -0.06 0.19 -0.43 0.32 0.99

7 0.93 -0.29 0.18 -0.64 0.06 0.88*

8 0.52 0.03 0.10 -0.17 0.22 1.00

*Correlation groups had higher ratings39

Exact Model Results(Relevancy-Correlation)

# m=1 m=2 Differences

E(ρ1) sd(ρ1) E(ρ2) sd(ρ2) E(ρ1-ρ2) sd(ρ1-ρ2) 2.50% 97.5% Pr(H1)

1 0.56 0.03 0.58 0.03 0.02 0.04 -0.05 0.09 1.00

2 0.45 0.03 0.47 0.03 0.02 0.04 -0.06 0.10 1.00

3 0.49 0.03 0.52 0.03 0.03 0.04 -0.04 0.11 1.00

4 0.54 0.03 0.57 0.03 0.03 0.04 -0.04 0.10 1.00

5 0.42 0.03 0.50 0.03 0.07 0.04 0.00 0.15 1.00

6* 0.35 0.03 0.36 0.03 0.00 0.04 -0.08 0.09 1.00

7 0.44 0.03 0.50 0.03 0.06 0.04 -0.01 0.14 1.00

8 0.62 0.03 0.61 0.03 0.00 0.04 -0.08 0.07 1.00

Median (95% Crl for σ was 0.24 (0.22-0.26)

40

Discussion

The assumption that relevance and correlation corresponds substantiated

Comparing modelso Exact model

• Precise estimates on the correlation scale

• Can infer the portion of experts scoring items with medium to large correlation

o Approximate model

• Conservative approach for calculating sample sizes

• If sample size is appropriate for the approximate model, the size is large enough for the exact model

DiscussionCombining m=1 (relevance) and m=2 (correlation)

# Item 1(%) 2(%) 3(%) 4(%) CVI CVIa CVIe

1 Satisfied with job 0 8 25 66 0.91 0.81 0.92

2 Consider another job 3 21 34 42 0.76 0.56 0.78

3 Force themselves to work 6 13 25 56 0.81 0.61 0.85

4 Enthusiastic to work 0 7 30 62 0.93 0.80 0.91

5Like job better than average worker

does5 18 37 41 0.78 0.56 0.78

6 Are clinically competent 14 31 31 25 0.56 0.37 0.60

7 Feel job will never end 7 17 29 47 0.76 0.57 0.79

8 Real enjoyment in their work 0 4 19 77 0.96 0.92 0.96

42

Limitations (1) Negatively worded items and impact on the interpretation

of correlations.

(2) responses “correlation” group < responses “relevance” group.

(3) The two tools may agree for spurious reason?

(4) Lack of training of experts regarding correlation.

Conclusions

The relevance tool and the correlation tool found to be equivalent

Content validity justified with correlation argument

Implications for replicating this method with other psychometric instrumentso Recall that se{g(ρj)}=1/ √n

o From exact model (Relevance group) se{g(ρj)}=0.037

• 12.4 participants per expert (about 10)

o Typically 6 experts used, experts are 6*10=60 participants

• Originally we needed 210, content validity reduced to 150!

Acknowledgments

Thanks to site coordinators and NDNQI staff member Kim Boyle

This research is supported by:o Contract from the American Nurses Association, NDNQI (PI: Nancy Dunton)

o University of Kansas Research Institute Bridging Grant (Chair Peter Smith)

o Department of Biostatistics (Chair Matt Mayo)

o School of Nursing Office of Grants and Research (Associate Dean Marge Bott)

o Lauren Aaronson & Carol Smith (grant writing mentors)