Evaluating research in developmentaldisabilities: a conceptualframework forreviewing treatmentoutcomes

Charlene Butler* EdD;

Henry Chambers MD;

Murray Goldstein DO MPH;

Susan Harris PhD;

Judy Leach RPT;

Suzann Campbell PhD;

Richard Adams MD;

Johanna Darrah PhD; Treatment Outcomes Committee,

American Academy for Cerebral Palsy and Developmental

Medicine, Rosemont, IL, USA.

*Correspondence to first author at 1818 Westlake Avenue

North, Suite 106, Seattle, Washington, 98109-2707, USA.

E-mail: argonauta@foxinternet.net

The study of developmental disabilities, not being confined to

one medical field, poses a challenge in evaluating outcomes

research. It is a multidisciplinary area of study which encom-

passes health-care, rehabilitation, psychosocial, educational,

and biotechnology specialties and involves biological, social,

and behavioral effects of intervention. Consequently, there is a

lack of consistency in what has been studied, how the out-

comes have been measured, and where these results have been

recorded. Naturally, these disparate outcomes data need to be

consolidated in such a way that comparison of treatments can

be made, within and across professional disciplines.

In an attempt to facilitate this, the American Academy for

Cerebral Palsy and Developmental Medicine (AACPDM) is

committed to providing critical and useful appraisal of the sci-

entific literature to help clinicians keep abreast in their own as

well as other relevant disciplines1. The Academy has explored

several classification systems to consolidate and interpret data,

and has established a two-part conceptual framework (1) to

aggregate treatment outcomes and construct evidence tables

based on a model of disablement which classifies treatment

outcomes by the dimension in which they have an effect; and

(2) to determine the degree of confidence that can be placed in

the scientific evidence available in support of an intervention.

Classification systems and models of disablement Before selecting a classification system, the AACPDM exam-

ined and carefully considered the merits of the most promi-

nent classification in disablement models in rehabilitation as

well as a system currently used to classify effects of orthope-

dic surgery. The rehabilitation models were found to be

more applicable to the multidisciplinary nature of develop-

mental medicine.

In 1980, the World Health Organization (WHO) developed

a model of disablement; this model has had the most univer-

sal influence on rehabilitation. The model is described in two

companion publications: the International Classification of

Disease (ICD-10)2 which classifies diseases, disorders or

injuries; and the International Classification of Impairment,

Disability and Handicap (ICIDH)3 which describes the con-

sequences of health conditions. The WHO model describes a

key concept which is the basis of its classification system, i.e.

that the consequences of disease occur at different levels:

disease at the level of molecules and cells; impairment at the

level of organs; disability at the level of individuals; and

handicap at the level of society. The WHO’s use of the term

handicap for one of these levels became the subject of vigor-

ous international debate. Consequently, the US Institute of

Medicine (IOM)4 and the US National Center for Medical

Rehabilitation Research (NCMRR)5 chose not to use the lan-

guage of the ICIDH. Instead, they adopted the classification

language of a conceptually similar model which was pro-

posed by Nagi in 19696.

Table I shows that this concept of reference levels is central

to all the rehabilitation models, but that the number and nam-

ing of levels is not consistent7. For example, disability refers to

the person level in the 1980 WHO model, but in the IOM

model, disability refers to the societal level. The NCMRR

model added a fifth level which addresses how society may

impose disablement on a person with a disability. Although

disability and other related terms have different meanings

within these existing classification systems, a common con-

ceptual framework of levels of reference underlies all of them.

In addition to the controversy over the term handicap and

the confusion of terminology which resulted from efforts to

improve on the WHO classification system, conceptual think-

ing about disablement has occurred over the years. Current

thinking by the WHO8, the IOM9, and the NCMRR (personal

communication, Marcus Fuhrer, Director of NCMRR, 1998)

has: (1) refocused the classification terminology toward neu-

tral rather than negative attributes (i.e. function instead of

functional limitation) to reflect a more positive approach or

enablement model; (2) acknowledged that the responsibility

for the disadvantages of disablement may be assigned to the

individual, to the environment, or both; and (3) recognized

that the consequences of disease are not necessarily uni-

directional and linear.

A common and non-inflammatory classification system has

been needed to facilitate the measurement, management, and

research of rehabilitation outcomes and to help remove the

barriers between medical and social models of rehabilitation

service delivery10. In 1993, the WHO initiated a worldwide

effort to accomplish this. Involving multiple partners in a con-

sensus-building exercise guided both by current scientific

thinking and the practical needs of use in the field, the WHO

revised the ICIDH. The draft revision, known as the ICIDH-2

and published in 1997, reflects changes proposed by many

users, experts, WHO-collaborating centers, and international

task forces8. This draft version is being subjected to systematic

Developmental Medicine & Child Neurology 1999, 41: 55–59 55

field trials and further consultation until 1999 and will be final-

ized after the field-trial results. Ongoing revision will extend

the main document to address specific use of the model in

clinical applications and medical research.

Compatibility with the WHO revision in progressIn 1995, when the Academy needed to select a classification

system, the WHO revision was in its pre-draft stage. The

Academy elected to use the NCMRR classification and contin-

ue to monitor the WHO revision process. The US (represented

by the Public Health Service, the IOM, and the NCMRR as well

as other groups) has subsequently joined Canada and Europe

in supporting the current and ongoing WHO revision effort11.

Once the WHO revision is fully accomplished with con-

sensus from disability activists, medical rehabilitation service

delivery systems, and research communities, the AACPDM

will consider changing to the WHO classification schema.

Similarities between the conceptual levels of reference used

in the WHO draft and the NCRMM model have been investi-

gated to assure that only the names given to the levels would

change (Table II). Thus future comparisons between inter-

ventions using evidence tables developed under the former

and latter classifications would be possible.

It will take time for any new terms of disablement to take

hold in a widespread and uniform manner. However, a com-

mon language is crucial to the clarification of rehabilitation

concepts, outcome definitions, and development of mea-

surement tools.

The NCMRR modelThe NCMRR, established in 1990 by the US National Institute

of Health, required a model to guide their agenda for mea-

surement and research of rehabilitation outcomes. An advi-

sory board of scientists, clinicians, and individuals with

disabilities had to determine the best way to translate med-

ical findings into clinical benefits for individuals with disabil-

ities. It examined the usefulness of current disability models

and how well they reflected the multidisciplinary nature of

rehabilitation and the importance of environmental factors

in mediating disabilities. The NCMRR constructed a model in

1993 based on, but also elaborating, the earlier Nagi, WHO,

and IOM models. Given the model’s intended purpose and

that its classification included levels from both medical and

social models, the NCMRR classification system seemed well-

suited to the needs of the AACPDM.

The NCMRR model describes five dimensions in which

congenital abnormalities, developmental disorders, genetic

conditions, injury or disease and their consequences may

occur (see Table III). The term ‘dimensions’ replaced that of

‘levels’ to suggest the multidirectional nature of the interac-

tions proposed; this change in terminology has also been

adopted in the WHO revision. Table IV shows how the effects

of cerebral palsy can be described using this model.

Table V gives one example of how evidence tables based

on this model can consolidate data from existing research for

analysis.

Evidence tables: how much and what kind of research isavailable? Evidence tables are a convenient way to summarize research

findings to determine what is known, what is not yet known,

and what it would be helpful to know about an intervention.

Campbell et al.12 demonstrated the usefulness of the

NCMRR model for analyzing data from their review of the

effects of intrathecally-administered baclofen on adults and

children with spasticity. They determined which dimension

of the disabling process was measured or observed for each

outcome published and then constructed a detailed evi-

dence table organized by these dimensions. Table V is a sum-

mary of that evidence table; it demonstrates one of the

simplest ways in which disparate data can be presented to

show the kind of evidence, as well as how much, exists for an

intervention. There are studies on how baclofen works in

the pathophysiological dimension, but the authors did not

attempt to review these or any work on societal limitations,

choosing, instead, to concentrate on those dimensions of

primary interest to rehabilitation specialists, i.e. impair-

ment, functional limitation, and disability. Table V shows

56 Developmental Medicine & Child Neurology 1999, 41: 55–59

Table I: Comparison of classification terminology in models of disablement

Terms used by Levels of reference for impact of disabilityCells/tissue Organ Person Social External barriers

Nagi 1969 Pathophysiology Impairment Functional limitation Disability –

WHO 1980 – Disease Impairment Disability Handicap

IOM 1991 – Pathology Impairment Functional limitation Disability

NCMRR 1993 Pathophysiology Impairment Functional limitation Disability Societal limitation

Table II: Comparison of NCMRR and revised WHO classification

Terms used by Levels of reference for impact of disabilityCells/tissue Organ Person Social External barriers

NCMRR 1993 Pathophysiology Impairment Functional limitation Disability Societal limitation

WHO 1997 Health condition Impairment Activity Participation Contextual– environmental

(revised draft) factors

that our scientific knowledge, based on this review of

human studies, is restricted to effects in the dimension of

impairment and, to a lesser extent, in the dimension of func-

tional limitation. All 20 studies assessed one or more effects

on impairment, specifically: effects on abnormal postural

tone, reflexes, spasms, pain, skin integrity, bladder capacity,

and so on. Effects on functional limitations (e.g. sleep dis-

turbance, dressing, hygiene care, sitting, and mobility) were

reported in 15 of the studies, but in three of them only anec-

dotal findings are reported. Only eight of the studies provid-

ed any information on effects in the dimension of disability

(e.g. ability to drive own van, full-time employment, sexuali-

ty) and this was exclusively anecdotal.

Linkage of levelsAnother type of analysis to which such an evidence table

lends itself is an examination of the interactions across

dimensions, which may occur as a result of an intervention.

One is tempted to think of these as five levels of a hierarchical

causal pathway; for example, in cerebral palsy, periventricu-

lar leukomalacia causes spasticity which is responsible for

awkward gait, and so on. While this may be true, the corollary

that successful treatment at one ‘level’ (i.e. the pathophysio-

logical or impairment dimension) will automatically and

positively affect another ‘level’ (i.e. functional limitation or

disability) has been an untested assumption underlying

intervention and research.

As a result of this assumption, studies have rarely incorpo-

rated measures in multiple dimensions to test specifically for

associated effects of an intervention. However, frequently

reported clinical observations did lead to a 1990 study of

selective posterior rhizotomy for cerebral palsy in which out-

come measures were included for both impairment and

functional limitation dimensions13. The results showed that,

in spite of the reduction in spasticity (i.e. a measure of

impairment) achieved by the intervention, children in the

study did not begin to use normal movement patterns (i.e. a

measure of functional limitation). In other words, positive

effects of treatment in one dimension did not automatically

produce effects in another dimension.

Almeida et al.14 published a case study in 1997 which sup-

ports a multidimensional assessment of outcomes. These

authors specifically designed a study to investigate the link-

age of effects of a treatment across dimensions, i.e. the use of

intrathecal baclofen in a child with cerebral palsy. Outcomes

were assessed longitudinally and included measures of

impairment (spasticity, speed of movement, range of

motion, and strength), functional limitations (gross motor

activities and activities of daily living), and disability (client

and family reports of changes at home and at school).

Although use of abnormal movement patterns did not

decrease in the child, reduced spasticity and increased speed

of movement were associated with improved independence

in transfers, dressing, and toileting; and the child won an

award for improved physical fitness in school as well as hav-

ing improved ball control during wheelchair basketball.

The NCMRR model emphasizes the possibility that its

dimensions may prove to be multidirectional rather than

hierarchical. The following examples demonstrate the com-

plex interactions suggested by the NCMRR model, i.e. that

effects may be top-down, bottom-up, or lateral. Antispasticity

drugs may intervene in the pathophysiological dimension by

affecting neurotransmitters; orthopedic surgery may act in

Evaluating Research in Developmental Disabilities Charlene Butler et al. 57

Table III: NCMRR model of disablement: five dimensions ofhuman functioning

Dimension Description

Pathophysiology Interruption or interference of normal

physiology and developmental processes

or structures

Impairment Loss or abnormality of body structure or

body function

Functional limitation Restricted participation in typical

societal roles

Disability Inability to participate in typical societal

role functions

Societal limitation Barriers to full participation in society that

result from attitudes, architectural barriers

and social policies

Table IV: Examples of selected effects of cerebral palsy ondimensions of human functioning

Level Examples

Pathophysiology Cystic lesions and white matter loss as a

result of periventricular leukomalacia of the

premature infant’s brain

Impairment Spasticity, contractures, low endurance,

perceptual dysfunction

Functional limitation Awkward walking with fatigue; difficulty

dressing; poor concentration and sustained

listening; reading problems

Disability Learning delays; education in restricted

environment; limited sports activity;

interference with dating and sexuality; not

able to take communion at church; cannot

participate in family activity by doing chores

at home; unable to achieve independent

living

Societal limitation Exclusion from school/city team sports;

denial of medical treatment or equipment

by insurer; government action that blocks

the building of independent living units for

people with disabilities; failure of voters to

support funding of wheelchair lifts for

public buses

Table V: Summary of a review of 20 studies of effects ofintrathecally administered baclofen on adults and childrenwith spasticity of differing etiology (1986 to 1993)13

Impairment Functional Disabilitylimitation

Nr of studies reporting 20 12 0

measured outcomes

Nr of studies reporting 1 3 8

anecdotal outcomes

the impairment dimension by changing the musculoskeletal

system. There may be positive, negative, or no effects of these

drug or surgical interventions in the dimensions of functional

limitation and disability. Assistive technology, such as pow-

ered mobility, may compensate for functional limitations by

providing an alternative means of efficient locomotion. It may

decrease disability by allowing a student to be independent

and to move about school faster with less effort. Its use may or

may not have positive or negative effects in the impairment

dimension, such as improved head control(positive), skin

breakdown (negative), or increased knee contractures (nega-

tive). Use of assistive technology may have effects in the func-

tional-limitation dimension through loss of ability for

independent transfers, or conversely, increase of transfer abil-

ity due to high motivation and effort to access public trans-

portation. Training for the Wheelchair Olympics may increase

ability. It may or may not also decrease emotional impairment

for a person with depression or low self-esteem. The relations

among the five dimensions of functioning have not been sys-

tematically investigated and need attention.

Grading studies: how good is the evidence?The ideal method for determining efficacy of a treatment is

through randomized clinical trials, but such trials are often

difficult to pursue for a number of reasons. As a result,

many studies employ less well-controlled research

designs. The variety of research designs in studies man-

dates use of a method to help evaluate diverse studies and

give weight to their findings. To determine the degree of

confidence that can be placed in the evidence available

about an intervention, a grading system commonly known

as Sackett’s rules of evidence can be used. Sackett15 has

described a method for grading research based on five lev-

els of evidence (Table VI).

Grade A, B, and C recommendations in support of an

intervention can be generated based on these five levels of

evidence. Outcomes supported by at least one and prefer-

ably more Level-I studies yield grade A recommendations.

Outcomes supported by at least one Level-II study receive a B

grade. Grade C recommendations are given to outcomes

supported by Level-III, -IV, or -V studies. The grade indicates

the degree of certainty of a conclusion, generated by the

strength of the supporting research evidence. Grade A con-

clusions are generated from the strongest research evidence;

they are the most definitive. Grade B conclusions are based

on weaker evidence and are only tentative. Grade C conclu-

sions are based on the weakest, suggestive evidence only;

thus, they are the least reliable.

While this system has much to offer in summarizing the

quality of evidence, there are some limitations. The major

limitation is that this system depends on large group studies

because it grew out of clinical work in epidemiology and

internal medicine where large group studies, even random-

ized controlled trials, are not only possible but increasingly

common. Unlike reviewers of research in developmental dis-

abilities, where small group studies and single subject

designs prevail, Sackett simply excluded less rigorous stud-

ies and relied on ones with stronger designs. In addition, this

system can be misleading because it focuses on the type of

design rather than on the actual design (i.e. the extent to

which threats to validity have actually been controlled).

Finally, there is potential for confusion. For example, two

randomized controlled trials may show opposite results. In

this classification, these Level I studies produce grade A evi-

dence in support of an intervention.

Despite these limitations, the AACPDM believes that ‘lev-

els of evidence’ is a useful concept and has therefore consid-

ered where to incorporate the single subject designs found

more commonly in the developmental disabilities research

into the Sackett schema. In fact, Sackett has more recently

promoted the use of the N-of-1 randomized trial, with an

individual subject to establish the efficacy of a treatment trial.

Furthermore, Sackett points out that using a levels-of-evi-

dence classification does not mean that we discard the body

of uncontrolled evidence that is available to us. Instead,

when more rigorous studies are not available, we must ‘fol-

low the trail to the next best external evidence and work

from there’16 (p 4).

ConclusionThis two-part conceptual framework which provides the orga-

nizational structure for evaluation of interventions in develop-

mental disabilities will be used by the AACPDM. Evidence

tables and analyses of current interventions will appear in sub-

sequent issues of this journal. Those wishing to learn more

about developing reviews of research using this framework

should visit the AACPDM website at http://www.aacpdm.org

for a step-by-step process, including an elaboration of the

NCMRR classification and the Sackett grading systems with

operational definitions of terms and case reports. Here you

will find a guideline entitled ‘Methodology for Developing

Evidence Tables and Reviewing Treatment Outcomes

Research’ written by The Treatment Outcomes Committee of

the AACPDM.

58 Developmental Medicine & Child Neurology 1999, 41: 55–59

Table VI: Sackett’s16 method for grading research

Level Description

Level I Large randomized trials, producing results with

high probability of certainty. These include studies

with positive effects that show statistical

significance and studies demonstrating no effect

that are large enough to avoid missing a clinically

significant effect

Level II Small randomized trials, producing uncertain

results. These are studies which have a positive

trend that is not statistically significant to

demonstrate efficacy or studies showing a negative

effect that are not sufficiently large to rule out the

possibility of a clinically significant effect

Level III Non-randomized prospective studies of concurrent

treatment and control groups, i.e. cohort

comparisons between contemporaneous subjects

who did and did not receive the intervention

Level IV Non-randomized historical cohort comparisons

between subjects who did receive the intervention

and earlier subjects who did not

Level V Case series without controls. The clinical course of

a group of clients is described, but no control of

confounding variables is undertaken. This is a

descriptive study which can generate hypotheses

for future research but does not demonstrate

efficacy

The Academy expects that using this format for organizing

the information on research outcomes will have many bene-

fits for the field of developmental disabilities. With such a

tool, consensus can be reached about the levels for which

there is evidence of efficacy. Meaningful comparisons

between interventions can be made. The levels for which

adequate information is lacking can be visualized and future

research encouraged to address the gaps in our knowledge.

Use of the model prompts the research community to

include multiple outcome measures in study protocols, so

that effects of an intervention in one level can be traced

across other levels. It will help professionals and clients alike

to see that recommendations may not be seen as conflicting

but rather as complementary by showing how different inter-

ventions can have effects in different levels. This can help

clients make informed decisions about treatment options

based on client values and on what the treatment offers. It

can lead us to think about intervention differently, i.e.

whether an individual child is best served at a particular

point by altering the environment rather than attempting to

change the child’s impairment or by considering whether

the disability of social isolation may be caused by depression

rather than limited mobility. Finally, it will serve to remind

each of us that as individuals in our communities, we have

the opportunity to reduce further the burden of disability in

the dimension of societal limitation by efforts to initiate and

support actions to remove all barriers to full participation in

society of people with disabilities.

Accepted for publication 9th October 1998.

References1. Butler C. (1995) Outcomes that matter. Developmental Medicine

and Child Neurology 37: 753–4.2. World Health Organization. (1992–1994) International

Statistical Classification of Diseases and Related HealthProblems, 10th Revision familiarly abbreviated as ICD-10.Geneva, Switzerland: WHO

3. World Health Organization. (1980) International Classificationof Impairments, Disabilities and Handicaps.Geneva,Switzerland: WHO.

4. Pope AM ,Tarlov AR, editors. (1991) Disability in America:Toward a National Agenda for Prevention. Washington, DC:National Academy Press.

5. National Institute of Health. (1993) Research Plan for theNational Center for Medical Rehabilitation Research. NIHPublication no. 93-3509. Washington, DC: US Department ofHealth and Human Services.

6. Nagi SZ. (1965)Disability and Rehabilitation. Columbus, OH:Ohio State University Press.

7. Dittmar SS ,Gresham GE. (1997) Functional Assessment andOutcome Measures for the Rehabilitation HealthProfessional.Gaithersburg, MD: Aspen Publishers.

8. World Health Organization. (1997) ICIDH-2: InternationalClassification of Impairments, Activities and Participation.Geneva, Switzerland: WHO.

9. Brandt EN, Pope AM. editors (1997) Enabling America: Assessingthe Role of Rehabilitation Science and Engineering. Washington,DC: National Academy Press.

10. Wilkerson D. (1997) On the language and classification ofdisablement and a new ICIDH. Rehabilitation Outlook.Winter: 5–7.

11. Seltser R. (1992) PHS Task Force on Improving Medical Criteriafor Disability Determination: Summary Report. Washington,DC: Public Health Service.

12. Campbell SK, Almeida GL, Penn RD ,Corcos DM. (1995) Theeffects of intrathecally administered baclofen on function inpatients with spasticity. Physical Therapy 75: 352–62.

13. Cahan LD, Adams JM, Perry J, Beeler LM. (1990) Instrumentedgait analysis after selective dorsal rhizotomy. DevelopmentalMedicine and Child Neurology 32: 1037–43.

14. Almeida GL, Campbell SK, Girolami GL, Penn RD ,Corcos DM.(1997) Multidimensional assessment of motor function in achild with cerebral palsy following intrathecal administration ofbaclofen. Physical Therapy 77: 751–64.

15. Sackett DL. (1989) Rules of evidence and clinicalrecommendations on the use of antithrombotic agents. Chest95: (Suppl.) 2–4.

16. Sackett DL, Richardson WS, Rosenberg W ,Haynes RB. (1997)Evidence-based Medicine: How to Practice and Teach EBM.New York: Churchill Livingstone.

Evaluating Research in Developmental Disabilities Charlene Butler et al. 59