Ensuring Consistency in Assessment of Continuing Care Needs:

An Application of Differential Item Functioning Analysis

R. Prosser, M. Gelin, D. Papineau, B. Zumbo,

Presentation at CES ConferenceJune 3, 2003

Outline

• Background– The MDS-HC Assessment Tool & Its Uses– The Challenge

• Methodology– The IADL Scale– Differential Item Functioning Analysis

• Results of Our Look at 5 Raters• Implications for Practice

What is MDS?• Standardized assessment of functional,

psychological, social and environmental needs, administered by clinician raters

• Target groups include older adults and people with disabilities or chronic diseases

• Minimum Data Set: a tool for collecting the minimum amount of data needed for producing a comprehensive and accurate profile

• Our focus is on MDS for Home Care (HC) clients• Mandated for use in B.C.

A Word About InterRAI• International group of 40+ researchers and

clinicians– Registered as not-for-profit corporation and owns

international copyright on RAI instruments– Conducts multinational collaborative research to

develop, implement and evaluate the instruments and their related applications

– Tools available in 12 languages

– Lead Canadian researcher: Dr. John Hirdes (University of Waterloo)

Uses of MDS

Assessment

Care Planning

Outcome Measures Quality Indicators

Resource Allocation

Components of the MDS System

– Minimum Data Set assessment form– Assessment protocols (trigger issues to

review in developing client’s care plan)– Quality indicators (standards and targets)– Health outcome scales (effectiveness)– Resource utilization groups: case mix

system of groups with homogeneous resource requirements

Key Outcome Scales

• Cognitive Performance Scale (CPS)• Depression Rating Scale (DRS)• Activities of Daily Living (ADL) Performance

Scales• Independent Activities of Daily Living (IADL)

Scales

Uses of Outcome Scales

• Provide additional information to determine service needs and to plan care

• Evaluate the effects of treatments• Compare results in different settings for

client groups with similar health issues and needs

Uses of MDS in Planning & Implementing a New Service Model

• Establish / refine service inclusion and exclusion criteria using RAI-HC

• Permits more reliable and equitable service allocation

• Determine number of clients on current caseloads who could benefit from this service

• Estimate future demand for this service

The Challenge• Need to have an objective basis for evaluating

and demonstrating assessor comparability and improving the MDS rating process – To ensure that service allocation decisions based on

assessments are made fairly—and transparently so (i.e., can be explained and justified)

– To provide reliable information for care planning and program evaluation

• Time pressure (long instrument)• Small samples

The IADL Scale• 7 items

– Meal Preparation– Ordinary Housework– Managing Finance– Managing Medications– Phone Use– Shopping– Transportation

• Four point scale for each item– Independent, Some Help, Full Help, By Others

Differential Item Functioning (DIF)• Analytic Foundation: Item Response Theory of

measurement• Answers the question “Do item response curves

differ appreciably for subgroups of a population, e.g., males & females or blacks & whites?”

• Often used to provide evidence in relation to item bias in achievement testing

• In our case, subgroups are seniors rated by DIFFERENT RATERS

• Our Question: Do different raters use an item the same way?

Item Characteristic Curves & DIF

Ability

43210-1-2-3-4

Pro

babi

lity

of a

"Ye

s"1.0

.9

.8

.7

.6

.5

.4

.3

.2

.10.0

Our Method

• Began by looking at DIF via logistic regression

• Decided that graphical approach is much more informative J. Ramsay’s Testgraf software

Our Sample

• Seniors from one of the four subregions within North & West Vancouver

• Selected the five raters for that area who had the largest caseloads

• Included only cases with complete data (i.e., no ratings omitted due to the target activity not being performed within the time frame) N = 335

Raters’ Subgroups

5489847038N =

rater id

54321

clie

nt a

ge a

t ass

essm

ent

100

90

80

70

60

50

40

30

20

10

0

283

157155314329

330334289

282

125114123

3204

310

97

134177

129

34321

256

281

71

57

73

67

79

0

10

20

30

40

50

60

70

80

90

1 2 3 4 5

Rater ID

Perc

ent F

emal

e

The IADL Scale

0

0.5

1

1.5

2

2.5

IADL Item Number

mean 1.54 2.07 1.27 0.82 0.4 1.79 1.57

SD 1.16 1.06 1.26 1.12 0.84 1.15 1.24

1 2 3 4 5 6 7

Scree Plot

Factor Number

7654321

Eig

enva

lue

5

4

3

2

1

0

Curve for Meal Preparation Item

Curve for Meal Preparation Item

Curves for Housework Item (5 raters)

Housework item (Option curves)Option 1 Option 2

Option 3 Option 4

Curves for Transportation Item(Item curve)

Transportation Item (Option curves)Option 1 Option 2

Option 3 Option 4

Summary of Key Findings

• The five raters behaved similarly for most items

• Rater 5 appears to be using items differently than the other raters, particularly item 7

Implications for Practice• Consider elaboration / clarification of scoring

instructions provided to raters• Conduct periodic reviews / monitoring with

raters to ensure that consistency is maintained• Provide additional training to particular raters

on particular items as needed

• May want to have raters rate random samples of clients or stratify the analyses

Benefits of this Approach to DIF Analysis

• Can be used with relatively small samples of the population being rated

• Raters can rate different seniors provided the samples are equivalent random assignment

• Short scales (small numbers of items) are OK• Graphical approach is easy to interpret yet

reveals details needed for making changes to instructions / training of raters

Areas for Future Work

• Testing with raters– How well does the feedback work in practice?– Is transparency increased?

• Determining rules of thumb for use– Minimum numbers of cases to rate– Scale features, e.g., minimum length needed

For Further Information

• InterRAI Web Site: http://nt8380.hrca.harvard.edu/

• Differential Item Functioning: http://www.educ.ubc.ca/faculty/zumbo/DIF/

• TestGraf Web Site: ftp://ego.psych.mcgill.ca/pub/ramsay/testgraf/

• Author e-mail: bprosser@vrhb.bc.ca