reading 3601 smith

What Is Evidence-Based Behavior Analysis?

Tristram SmithUniversity of Rochester Medical Center

Although applied behavior analysts often say they engage in evidence-based practice, theyexpress differing views on what constitutes evidence and practice. This article describes apractice as a service offered by a provider to help solve a problem presented by a consumer.Solving most problems (e.g., increasing or decreasing a behavior and maintaining this change)requires multiple intervention procedures (i.e., a package). Single-subject studies are invaluablein investigating individual procedures, but researchers still need to integrate the procedures intoa package. The package must be standardized enough for independent providers to replicate yetflexible enough to allow individualization; intervention manuals are the primary technology forachieving this balance. To test whether the package is effective in solving consumers problems,researchers must evaluate outcomes of the package as a whole, usually in group studies such asrandomized controlled trials. From this perspective, establishing an evidence-based practiceinvolves more than analyzing the effects of discrete intervention procedures on behavior; itrequires synthesizing information so as to offer thorough solutions to problems. Recognizing theneed for synthesis offers behavior analysts many promising opportunities to build on theirexisting research to increase the quality and quantity of evidence-based practices.Key words: evidence-based practice, clinical trials, treatment effectiveness evaluation,

behavior analysis, behavior modification

A Google search for evidence-basedbehavior analysis yields 16,500,000hits. Most of the top hits containaffirmations that behavior analysis isindeed evidence based. Similar state-ments appear regularly in traditionalmedia (e.g., Smith, 2012). Despitethis refrain, many behavior analystsexpress misgivings about publishedoperational definitions of the termevidence based (Green, 2008; ODon-ohue & Ferguson, 2006). Such de-finitions usually include guidelinesfor rating the quality of individualstudies, aggregating findings acrossstudies, and classifying the overalllevel of empirical support for anintervention. For example, guidelinesin education and psychology listspecific indicators of high-qualitystudies, set a threshold for thenumber of such studies needed toestablish an intervention as evidence

based, and delineate categories suchas probably or possibly effica-cious (Chambless & Hollon, 1998;Kratochwill & Stoiber, 2002; Odomet al., 2005; West et al., 2002). Manyguidelines in medicine specify that thegold standard for identifying evi-dence-based interventions is a sys-tematic, statistical analysis of datafrom multiple randomized controlledtrials (RCTs), in which investigatorsassign large numbers of participantsby chance to treatment or controlgroups (Guyatt, Oxman, et al., 2008).By examining how closely the avail-able research approaches this stan-dard and how favorable the resultsare, reviewers can rank or grade theintervention (United States Preven-tive Services Task Force, 2007).Behavior analysts have exposed

many limitations of RCTs (Johnston,1988; Johnston & Pennypacker, 1993)and statistical methods for combiningdata from different studies (Baron& Derenne, 2000). One practicalproblem is that individuals who con-sent to enroll in RCTs and risk beingassigned to the control group maydiffer from individuals who declineto enroll (Christenson, Carlson, &

The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:20 7 Cust # MS TBA12-15

Correspondence concerning this articleshould be addressed to Tristram Smith,Division of Neurodevelopmental and Behav-ioral Pediatrics, Department of Pediatrics,University of Rochester Medical Center, 601Elmwood Ave., Box 671, Rochester, NewYork 14642 (e-mail: [email protected]).

The Behavior Analyst 2013, 36, 733 No. 1 (Spring)

7

Valdez, 2002). Quasiexperimentalstudies, in which treatment and con-trol groups are matched on partici-pant characteristics but are assignednonrandomly (e.g., according to theavailability of providers to deliver theexperimental treatment), may mini-mize this problem. Nevertheless, amore fundamental difficulty remains:Both RCTs and quasiexperimentalstudies test the average change withinan intervention group instead of themore clinically relevant analysis ofchange made by each individualparticipant as a function of a specificintervention (Johnston, 1988).Statistical metrics that merge data

across studies compound the prob-lems in RCTs (Salzberg, Strain, &Baer, 1987; Strain, Kohler, & Gresh-am, 1998). Basically, these metricsrepresent an average of the averagechange reported in different studies.The studies almost always vary fromone another in terms of their inter-vention protocols, outcome mea-sures, and criteria for enrolling par-ticipants. Thus, the metrics arederived from an amalgamation ofdata obtained by an assortment ofmethods. False precision is apt to bethe result.Because of these limitations of

group studies (RCTs and quasiex-perimental research), behavior ana-lysts favor studies with single-subjectexperimental designs (SSEDs). SSEDstudies involve conducting repeatedobservations to compare an individ-uals behavior during a baselineperiod when the individual receivesno intervention to the behavior inone or more intervention phases.They are intended to detect changesthat occur as soon as an interventionbegins. Thus, SSED studies are muchbetter suited than group studiesfor establishing a functional relationbetween a particular intervention andchange of a specific behavior (John-ston, 1988; Johnston & Pennypacker,1993). If the functional relation isrobust, the change should be evidentfrom visual inspection of a graphical

display of the data, obviating theneed for statistical analyses. Replica-tions of intervention procedures with-in and across participants will eluci-date how and when the relationoccurs (Johnston, 1996).In relation to evidence-based prac-

tice, behavior analysts advocate forthe recognition of SSED studies asimportant sources of data (Detrich,2008; Green, 2008) and have pro-posed criteria for classifying an inter-vention as evidence based on thestrength of findings from such studies(Horner et al., 2005; Kratochwill etal., 2010; National Autism Center[NAC], 2009). These guidelines in-clude indicators of high-quality re-ports, notably clearly described inter-vention procedures and multiplereplications of intervention effects,assessed by direct, reliable observa-tions of the target behavior (Horneret al., 2005; NAC, 2009). Reviews ofSSED studies find numerous appliedbehavior-analytic (ABA) proceduresthat meet criteria for classification asevidence based (e.g., Kurtz, Boelter,Jarmolowicz, Chin, & Hagopian,2011; Lee, 2005; NAC, 2009; Sulzer-Azaroff & Austin, 2000; Vegas,Jenson, & Kircher, 2007).From such work, some behavior

analysts conclude that we have per-suasively rebutted standard defini-tions of evidence-based practice(Keenan & Dillenburger, 2011), re-placed them with definitions of ourown (e.g., Horner et al., 2005), anddocumented that many of our inter-ventions merit this designation (e.g.,NAC, 2009). However, such conclu-sions are too hasty, in this writersjudgment. Inspection of the ABAinterventions that are described asevidence based and the nature ofempirical support adduced for themreveal that some basic questionsremain unanswered.First of all, what is a practice? Lists

of evidence-based ABA practices arean incongruous lot. For example,after a meticulous literature searchand coding process, the National


8 TRISTRAM SMITH

Standards Project (NAC, 2009) oninterventions for individuals withautism spectrum disorders (ASD)classified 11 practices as evidencebased:

1. Antecedent package2. Behavioral package3. Comprehensive behavioral treat-

ment for young children4. Joint attention intervention5. Modeling6. Naturalistic teaching strategies7. Peer training package8. Pivotal response treatment9. Schedules10. Self-management11. Story-based intervention pack-

age (NAC, p. 43)

Two items in this list, modeling andschedules, are specific cuing proce-dures that have a wide range ofapplications for individuals with andwithout ASD. Most other items arecombinations of procedures that varyfrom study to study. However, threeitems refer to more standardizedpackages. Comprehensive behavioraltreatment originated in the work ofLovaas (1987) on early intensivebehavioral intervention (EIBI) fortoddlers and preschoolers with ASD.It has evolved into many differentmodels (Handleman & Harris, 2001),all of which combine a multiplicity ofintervention procedures that are de-livered 20 to 40 hr per week for 2 to3 years (Smith, 2011). Pivotal re-sponse treatment (Koegel & Koegel,2006) is one approach to usingnaturalistic teaching strategies, withpriority given to behaviors identifiedby the developers as especially impor-tant for promoting childrens devel-opment. Story-based intervention isderived from a trademarked pro-gram, Carol Grays Social Stories(Gray, 2004). As acknowledged inan incisive commentary, the onlycommon element of all 11 of theseitems is that each is comprised ofintervention procedures or combina-tions that seem to share core charac-teristics (NAC, 2009).

What is evidence based? Goingbeyond our longstanding contentionthat SSED studies have advantagesover between-groups studies (Sidman,1960), behavior analysts assert thatsuch studies can stand alone as the solesource of empirical support for apractice (Horner et al., 2005). Is thismore expansive assertion tenable?Going still further, some writers arguethat many ABA intervention ap-proaches (e.g., differential reinforce-ment and shaping) are so well estab-lished and are monitored so closely foreach individual who receives them thatthey can be considered evidence basedeven without reference to a particularset of SSED studies (Keenan &Dillenburger, 2011). Does this line ofreasoning withstand scrutiny? Also,many systems have been proposed forclassifying practices as evidence basedon account of findings from eitherSSED or between-groups studies(Chambless & Hollon, 1998; NAC,2009; National Secondary TransitionTechnical Assistance Center, 2010;Odom et al., 2005; Reichow, Volkmar,& Cicchetti, 2008). How can two suchdisparate methodologies, often char-acterized as separate research tradi-tions by behavior analysts (e.g., John-ston, 1988), both be used to identifyevidence-based practices?These questions are not merely

about semantics. The purpose ofidentifying evidence-based practicesis to help consumers and providerschoose among intervention approach-es (Detrich, 2008). Essentially, apractice is a service offered by provid-ers to consumers, and representing aservice as evidence based is a formof endorsement or recommendation,backed by an appeal to science.Insurers and other third-party payersincreasingly use information aboutevidence-based practices to determinewhich services to cover (Green, 2008).The pliable definitions and criteria

for evidence-based ABA practicesnow in use are troublesome becausethey put us at risk for overestimatingour accomplishments and helpfulness


EVIDENCE-BASED BEHAVIOR ANALYSIS 9

to consumers and providers while weoverlook topics in need of morestudy. The goal of the present articleis to help behavior analysts provideuseful guidance to consumers, pro-viders, and third-party payers byclarifying what is or is not a practiceand what is or is not suitableevidence. Accordingly, the articleexamines what constitutes a practice(a procedure or package of proce-dures, standardized or not) andpersuasive evidence for the practice(findings from SSED or group stud-ies). In so doing, prior discussions ofthese issues by behavior analysts arereconsidered in light of the muchlarger literature currently available,and ABA interventions that behavioranalysts have identified as evidence-based practices are critiqued. Theseconsiderations are relevant for ABAproviders, who need to recognizewhich services they offer have ade-quate scientific support and whichdo not. Moreover, they raise criticalquestions to be addressed by ABAresearchers, who must determinewhat is known and unknown aboutinterventions and set priorities forfuture programs of inquiry.

PROCEDURES OR PACKAGES?

In their formative description ofABA as a discipline, Baer, Wolf, andRisley (1968) emphasized that ABAis technological, meaning that itsintervention techniques or proce-dures are precise, detailed, and repli-cable. A key aspect of ABA researchever since has been to discover suchprocedures and understand how andwhy they work. However, Baer (1975,2004) pointed out that, as researchprogresses, it may become appropri-ate to combine procedures into apackage or program and test thecombination. A single interventionprocedure (i.e., a format for teachinga new skill or tactic to decrease abehavior) is seldom enough to fix aproblem by itself; as Baer (2004)remarked, Peoples problems seem

to require a program (p. 310). Azrin(1977) added that a package orprogram should contain strategiesfor deploying techniques effectively,efficiently, and credibly, even whenconfronted with real-world con-straints such as resource limitations,variations in skill level and enthusi-asm of interventionists, and competi-tion from alternative programs. Inother words, a package should helpproviders make decisions on how touse procedures in their practice.The need for intervention packages

led Azrin (1977) to recommend mov-ing quickly to conducting researchon packages. However, Birnbrauer(1979) countered that it is prematureto evaluate packages unless theircomponents have been thoroughlyinvestigated. Johnston (1996) addedthat evaluations of componentsshould go through a sequence begin-ning with investigations in controlledsettings that allow rigorous experi-mental analyses and proceeding tofield settings that might require com-promises (e.g., reliance on inexperi-enced interventionists or introductionof an intervention after a baselinethat was too short to establish astable rate of behavior).Although they warned against

forming packages comprised of in-completely understood procedures,both Birnbrauer (1979) and Johnston(1996) agreed that movement fromprocedures to packages is essential.Birnbrauer averred that no onecan argue with (p. 19) the valueof packages that improve outcome,minimize costs, and are acceptable toconsumers. He added that SSEDstudies on an individual procedureare at best, slower and moredifficult to sell to consumers (Birn-brauer, p. 19) than research onpackages. Likewise, Johnston (1993,1996) pointed out that the goal ofABA research is fully identifying allof the individuals and behaviors ofinterest and understanding the webof social contingencies that influ-ence these individuals and behaviors


10 TRISTRAM SMITH

(Johnston, 1993, p. 329). He furtherobserved,

The challenge is to design the bits and piecesof a procedure into an integrated and oftencomplex whole. At some point, however,everything that has been learned shouldsuggest a reasonable combination of proce-dural elements as a focus for further analyticattention. (p. 330)

In short, analysis of procedures yieldsthe information needed to form acoherent package that can itself bestudied.Nevertheless, controversies over

efforts to create intervention packag-es show that behavior analysts con-tinue to struggle with how and whento do so. For example, Carr et al.(2002) reproached behavior analystsfor overrating studies of isolatedprocedures:

For decades, applied behavior analysts haveprided themselves on the publication of manysuccessful research demonstrations that in-volve the application of single interventions.These demonstrations have made for greatscience but ineffective practice. A comprehen-sive approach involving multicomponent in-tervention is necessary to change the manyfacets of an individuals living context that areproblematic. (p. 9)

In a rejoinder, Johnston, Foxx, Ja-cobson, Green, and Mulick (2006)disputed the charge that ABA proce-dures look impressive in a researchcontext yet fail in practice. They alsocriticized many aspects of the frame-work advocated by Carr et al. forcreating packages to address problembehavior (positive behavior interven-tion and support; PBIS). It is impor-tant to note, however, that they toodescribed effective intervention forproblem behavior as involving acombination of procedures and in-deed worried that advocates of PBIShave downplayed some key proce-dures, especially consequence-basedstrategies such as differential rein-forcement. Moreover, they admittedthat behavior analysts have much tolearn about how to package inter-ventions that nonspecialists would be

willing and able to adopt. Even morestrikingly, they acknowledged a dis-connect between what appears in anoutlet such as the Journal of AppliedBehavior Analysis (JABA) and whathappens in practice: Although someof the features of JABA research maycertainly be found in routine practice,the pages of that journal do notrepresent the full range of practicesin ABA (Johnston et al., 2006,p. 61).The exchanges between Azrin

(1977) and Birnbrauer (1979) andbetween Carr et al. (2002) andJohnston et al. (2006), along withthe comments by Baer (1975, 2004),suggest that a broad consensus hasemerged on what constitutes a prac-tice. Formulating and evaluatingpractices, however, remain conten-tious. Thus, it is worth exploringthe consensus on what constitutes apractice and then to consider researchapproaches that would increase suc-cess in developing practices.There is general accord among

behavior analysts that a practice isa service aimed at solving peoplesproblems. Further, although no onehas ruled out the possibility that thesolution might consist of a singleprocedure in some cases, behavioranalysts concur that the solution ismore likely to involve a combinationof procedures, along with strategiesfor deploying the practice in typicalservice settings (i.e., a package). Forexample, a practice aimed at reduc-ing a problem behavior is likely toinclude a functional analysis or as-sessment, followed by the use offunction-based interventions that in-volve altering both antecedents andconsequences of the behavior whilestrengthening replacement behaviors(Johnston et al., 2006). A practiceaimed at overcoming skill deficits islikely to include a careful evaluationof the individuals current skill leveland the skills that would be mosthelpful for the individual to learn, aswell as a task analysis of those skills,followed by the implementation of a



range of teaching strategies to helpthe individual acquire, generalize,and maintain the skills (Greer,2002). Either kind of practice proba-bly also involves considering contex-tual factors such as the individualsfamily environment, peer networks,and expectations in school or at theworkplace, In addition, the practicerequires determining what is feasiblegiven constraints such as resourcelimitations and preferences of thepeople involved.These views are consistent with

standard definitions of a practiceacross professions. Particularly whenused in the phrase evidence-basedpractice, a practice refers to a fusionof research findings, consumer pref-erences, and clinical assessment(American Psychological Association[APA], 2005; Sackett, Rosenberg,Muir Gray, Haynes, & Richardson,1996). This meaning is encapsulatedin one helpful synopsis:

There is an abundance of definitions ofevidence-based practice (EBP). Fortunately,most of them say essentially the same thing. The goal of EBP is the integration of (a)clinical expertise/expert opinion, (b) externalscientific evidence, and (c) client/patient/care-giver values to provide high-quality services.(American Speech-Language Hearing Associ-ation, n.d.)

Thus, evidence-based practice in psy-chology is defined as the integrationof the best available research withclinical expertise in the context ofpatient characteristics, culture, andpreferences (APA, 2005, p. 5). Sim-ilarly, evidence-based medicine isdefined as an integration of clinicalexpertise and research evidence forthe conscientious, explicit and judi-cious use of current best evidence(Sackett et al., 1996, p. 71) to guidedecisions about service delivery. Inthese definitions a practice is not anisolated procedure. Rather, it is aprocedure or (more commonly) a setof procedures that a skilled providercan adapt to meet the needs of anindividual case in the context of a

service setting and the individualslife circumstances (i.e., a package).Definitions of evidence-based prac-

tice by behavior analysts use differentwords to make virtually the samepoint. Detrich (2008) described thefunction of evidence-based practiceas consumer protection. Detrich,Keyworth, and States (2007) stressedthat the identification of evidence-based practices involves the review ofresearch findings to guide providersand consumers in their choice ofinterventions and the developmentof strategies to help them implementthe interventions even in the presenceof social, political, fiscal, and hu-man resource (p. 4) barriers thatmight exist. Similarly, Horner andKratochwill (2011) referred to apractice as any operationally de-fined set of procedures that are usedby a specified target audience, underdefined conditions/contexts, to achievevalued outcomes for one or moredefined populations (p. 2). In keepingwith definitions in other professions,these definitions characterize a practiceas a package for implementing aprocedure or procedures. Horner andKratochwills definition is especiallyclear that an operationally definedprocedure is only one part of a practice.Practices can vary in the range of

behaviors they target. For example,as emphasized by Carr et al. (2002),practices developed within PBIS of-ten aim to reduce a problem behaviorby improving the individuals overallquality of life. In contrast, functionalcommunication training is a practicethat focuses more specifically onreplacing a problem behavior with acommunication skill that serves thesame function as the problem behav-ior (Carr & Durand, 1985). Asanother example, EIBI is intendedto accelerate all aspects of develop-ment in young children with ASD(Lovaas, 1987), whereas the pictureexchange communication system isdirected mainly toward helping chil-dren with ASD begin to communi-cate (Bondy & Frost, 2002).


12 TRISTRAM SMITH

However, it is a misnomer todescribe an individual procedure(e.g., modeling) as a practice, as hasbeen done in some ABA reports (e.g.,NAC, 2009), unless the procedure byitself offers a solution to a problemand providers have guidance on howto use it in practice. If research showsthat a procedure reliably changesbehavior, it may become a compo-nent of a practice. It may also beworthy of dissemination in its ownright. Embry and Biglan (2008) de-scribed some procedures as ker-nelsfundamental units that canbe incorporated into a wide range ofintervention or prevention packages.Examples include time-out, writtenpraise notes, self-monitoring, framingrelations among stimuli to affect thevalue of a given stimulus, and phys-iological strategies such as nasalbreathing when upset (p. 77). Theynoted that kernels tend to be simplerthan complete packages and thus maybe more readily disseminated in somecircumstances. However, they ac-knowledged that users will need tocombine kernels with other approach-es and that kernels are not a substi-tute or replacement for packages.Even under the best of circum-

stances, community providers andconsumers would have great difficul-ty determining how to use proceduresin practice. For example, expandingon the NAC (2009) report, Odom,Hume, Boyd, and Stabel (2012)identified 24 evidence-based teachingprocedures for children with ASDsuch as discrete-trial training, promptdelay, and visual schedules. They alsolaid out step-by-step instructions forsetting up and implementing eachprocedure, along with written tutori-als and instructional videos to showhow to use the procedures correctly.To supplement this information,Odom et al. offered general guide-lines on the kinds of skills that can betaught with each procedure (e.g.,noting that visual schedules havebeen found to be especially usefulfor promoting independent comple-

tion of self-help activities). However,they did not identify a way to selectprocedures that match the needs ofan individual child with ASD, thescope and sequence of skills to teach,the frequency and length of time todevote to each procedure, or methodsto deploy the procedures expedientlyin community settings such as schools.Therefore, although Odom et al.s setof procedures is a valuable resource, itmainly consists of pieces (i.e., kernels)that need to be assembled by provid-ers and consumers. Thus, it consti-tutes an important step toward for-mulating a practice but remains awork in progress.Going beyond procedures, packages

(e.g., intervention programs developedwithin PBIS) do include specific man-uals for assessment of the needs ofindividual consumers (Sugai, Lewis-Palmer, Horner, & Todd, 2005), de-velopment of an intervention plan foran individual (Dunlap et al., 2010) or agroup (Sugai et al., 2010), and evalu-ation of the fit between the interven-tion and the context in which it takesplace (Horner, Salentine, & Albin,2003). As discussed above, this kindof integration of procedures into pack-ages is crucial for creating practices.Moreover, given the voluminous liter-ature of SSED studies on the domaintargeted in PBIS (problem behavior),one might hope that investigators havemet Birnbrauers (1979) and John-stons (1996) criteria for studyingprocedures thoroughly before packag-ing them. However, it is possible thatthe literature may still contain gaps. AsJohnston (1996) commented, manySSED studies are aimed at respondingto an immediate need that arises in aservice setting, rather than advancing asystematic program of research. Con-sequently, the considerable size of theapplied literature may suggest morethan it can actually deliver (p. 39).Accordingly, a careful review of theliterature may be warranted to deter-mine whether the research is sufficient-ly far along to guide the creation of apackage. Another issue to consider is



that, because packages encompass amultiplicity of procedures and strate-gies for their deployment, they cantake an almost infinite variety of forms(Baer, 2004). Thus, if existing packages(e.g., PBIS approaches) raise concerns(Johnston et al., 2006), investigatorscan propose alternatives.Investigators may debate whether

to conduct SSED studies to fill ingaps in the literature on problembehavior, focus on PBIS, or developnew intervention packages. Such de-bates also may arise in connectionwith interventions for other prob-lems. What is clear, however, is thatthe key to resolving these debates isto identify ways to promote progressfrom isolated procedures to integrat-ed packages. The establishment ofevidence-based practices in ABA de-pends on achieving this integration.To summarize, a practice is a

solution to a problem presented bya consumer. Most problems requirea combination of procedures, alongwith strategies for implementingthem in a service setting (i.e., apackage). The process of implemen-tation necessitates the integration ofevidence about the practice with thepreferences of the individual who isbeing served and the skills of theservice provider. Although SSEDstudies are invaluable in the identifi-cation of individual procedures toinclude in a practice, researchers stillmust find ways to put procedurestogether into a package for use in apractice setting. The next sectiondescribes methodologies for examin-ing whether a procedure is likely tobe useful in a practice setting and forpackaging procedures.

Research Implications

The potential utility of proceduresin practice. Behavior analysts oftenemphasize the need to study theeffects of ABA procedures in thecontext of typical practice settings(e.g., Johnston, 1996). However, re-views indicate that the large majority

of our research focuses on interven-tions delivered by study personnel,usually in tightly controlled environ-ments such as laboratories, special-ized ABA classrooms, or distraction-free areas set up to provide one-to-one instruction (e.g., Kasari & Smith,in press; Rehfeldt, 2011; Trahan,Kahng, Fisher, & Hausman, 2011).This discrepancy may reflect a dilem-ma that behavior analysts have hadtrouble resolving: We recognize thatconducting studies in practice settingsmay require sacrificing some scientif-ic rigor because the primary missionof such settings is to deliver servicesrather than conduct research (John-ston, 1996), yet we regard the qualityof many studies in these settings asunacceptable (Johnston et al., 2006).The root of this dilemma is that

findings from poorly designed studiesare difficult or impossible to inter-pret, no matter where they take place.Thus, the best way to move forwardis to seek opportunities to carry outrigorous studies in practice settings.Of course, this is easier said thandone, but several areas of researchoffer guidance on how to proceed.One such area is implementationscience (Fixsen, Naoom, Blase, Fried-man, & Wallace, 2005), which grewout of studies on ABA programs suchas Achievement Place. Related areasof investigation, notably diffusion ofinnovations (Rogers, 2003), socio-technical technology transfer (Backer,David, & Soucy, 1995), and improve-ment models (Langley et al., 2009),arose from efforts to deploy othertechnologies in community settings.Research in these areas has highlight-ed many factors that contribute tosuccess in working with communityagencies. Particularly important ele-ments include the establishment of anongoing partnership with the agency,identification of a champion withinthe agency for introducing a newtechnology, assembly of a team towork together to implement the tech-nology, securing support from admin-istrators to ensure that adequate


14 TRISTRAM SMITH

resources are available, obtainingapprobation from other influentialindividuals in the agency, and startingwith small pilot projects (Damschro-der et al., 2009). A commendableexample in ABA research is theBehavior-Analytic Consultation toSchools program, which has complet-ed a large number of high-qualityfunctional analyses in public schools(Mueller, Nkosi, & Hine, 2011). Otherexamples are the studies of EIBI byHoward, Sparkman, Cohen, Green,and Stanislaw (2005) and Cohen,Amerine-Dickens, and Smith (2006),which were made possible when oneof the authors (Cohen) negotiated afour-way agreement among stateagencies, local school districts, privateservice providers, and parents on asystem for publicly funding EIBI.Along with the need to test the

generality of intervention effectsacross service settings and providers,it is also important to test thegenerality of effects across individu-als who receive interventions and toidentify factors that predict success orfailure of an intervention for a givenindividual. Despite the caveats notedin the introduction, meta-analysescan help fulfill this objective bysystematically organizing data ob-tained across many SSED studies.(Because meta-analyses are also usedin group studies, they may have theadded advantage of presenting resultsfrom SSED studies in a way that isunderstandable and convincing toresearchers outside ABA.) However,reliable information on generality ofeffects is difficult to obtain from theABA literature because failures arehardly ever reported. For example,data from all treatment articles inVolumes 40 through 44 of JABAindicate that every study participantresponded favorably to intervention,ordinarily within a few sessions. Tobe sure, the results were tempered insome cases by individual variations inresponse (e.g., Dunn, Sigmon, Thom-as, Heil, & Higgins, 2008), incom-plete generalization (e.g., Shillings-

burg, Kelley, Roane, Kisamore, &Brown, 2009), or a need to augmentthe original intervention (e.g., Moore& Fisher, 2007). Still, a 100% rateof improvement that usually occursright away belies the practical realitythat many individuals make slow orminimal progress, even in exemplary,intensive ABA intervention programs(cf. Hagopian & Wachtel, n.d.; Leaf,Taubman, McEachin, Leaf, & Tsuji,2011).The tendency to publish only

positive results may be due to thepriority given in ABA research to theestablishment of experimental controlthat shows a functional relationbetween an intervention and a behav-ior. Beginning with Skinner (1956),behavior analysts have held that it ismore informative to present data thatepitomize experimental control thanto present data on a large number ofparticipants or on an average re-sponse to the intervention. As acorollary, null findings, which reflecta lack of experimental control, maybe uninformative and hence unwor-thy of publication.However, behavior analysts are

interested not only in establishingexperimental control but also inperforming replication studies todetermine the generality of findings(Johnston & Pennypacker (2009).Given that ABA interventions arenot universally effective, this processmust include documentation of bothsuccesses and failures. Although, tothis writers knowledge, behavioranalysts have not described ways toprovide such documentation, we canconsider approaches developed inother disciplines as a starting point.In group studies on intervention, thestandard approach is to use anintention-to-treat analysis (Schultz,Altman, Moher, & CONSORTGroup, 2010). In this kind of analysis,researchers report outcome data fromall participants who entered thestudy, not just those who completedit. To put these data in context, theyalso describe the flow of participants



through the study. This descriptionincludes the recruitment procedures(e.g., in person or by telephone ormail, from one agency or more), theinclusion and exclusion criteria forentry into the study, the number ofpotential participants who werescreened for eligibility but did notenroll, the reasons why they did notenroll (e.g., declining to give consent),and the number of dropouts, alongwith the timing and reasons fordropping out (Schultz et al., 2010).The goal is to disclose the full breadthof outcomes of intervention, includ-ing the possibility of not receiving theintervention at all, rather than pre-senting only successes.In other social sciences, researchers

who conduct studies with small sam-ples incorporate purposeful sam-pling strategies (Creswell, 2006;Miles & Huberman, 1994). For exam-ple, researchers sometimes seek max-imum variation in the target problem(e.g., problem behavior that rangesfrom mild to severe) or actively lookfor disconfirming cases. Alternatively,they may seek participants who areconsidered typical or representative insome way of a larger population. Suchapproaches could be valuable inSSED studies. As an illustration,although experimental functionalanalysis has been highly productivein ABA research, it is seldom used incommunity practice (Dixon, Vogel, &Tarbox, 2012). Researchers have at-tempted to bridge the gap betweenresearch and practice by streamliningfunctional analysis procedures (Iwata& Dozier, 2008), showing that theycan be completed in real-world envi-ronments (Mueller et al., 2011), andtesting their social validity (Lang-thorne & McGill, 2011). However,another barrier may be that, becausefunctional analysis emerged fromwork in inpatient or day-treatmentprograms with individuals who exhib-it severe problem behavior (e.g.,aggression or self-injury), it is notclear whether such analysis is neces-sary or helpful with individuals who

display routine problem behaviorsuch as talking out of turn, refusingto complete tasks, or breaking rules.Deliberately sampling such individu-als may be an important step towardascertaining the utility of functionalanalysis across a wide spectrum ofproblem behavior.Overall, although many ABA in-

tervention procedures show consider-able promise, the tendency of SSEDstudies on these procedures to takeplace in controlled settings, rely onstudy personnel to deliver interven-tions, and focus on successes mayexaggerate the potential benefits ofthe procedures in practice. As acorrective, researchers would do wellto take advantage of findings fromimplementation science and relatedareas of research on how to carry outhigh-quality research in practice set-tings. In addition, it may be advisableto use intention-to-treat analyses orpurposeful sampling strategies (orboth).Packaging intervention procedures.

Baer, Wolf, and Risley (1987) drewattention to a formidable technolog-ical challenge in developing interven-tion packages: the need to determinewhich procedures in the packageshould be followed faithfully, nomatter where or when the program isused, and which procedures usersshould be allowed, and even encour-aged, to modify to fit their localsituations and contingencies (p. 321).In the years since Baer et al. high-lighted this challenge, investigatorshave made little progress in elucidat-ing principles that govern how tostrike an appropriate balance be-tween standardization and individu-alization. The differing levels ofstandardization in ABA interventionpackages for individuals with ASD(NAC, 2009) or other problems mayreflect this uncertainty.Nevertheless, investigators have

honed a technology for proposing abalance that can be tested: an inter-vention manual. The advent of man-uals has transformed research on


16 TRISTRAM SMITH

behavioral, psychological, and edu-cational treatments (McHugh & Bar-low, 2012). Manuals standardize in-tervention packages by giving step-by-step instructions for each stage ofimplementation, along with problemsthat the user is likely to encounterand possible solutions to these prob-lems (Hibbs et al., 1997). They alsodefine the problem or population forwhom the intervention package isintended and the qualifications thatproviders who implement the manualshould possess (Hibbs et al., 1997).At the same time, they allow con-strained flexibility (MacMahon,2004) by delineating a limited set ofacceptable variations. For example,the package may be divided intomodules with decision rules or as-sessment procedures for selectingwhich modules to implement andunder what circumstances (Weisz etal., 2012). The manual may describedifferent ways for providers to deliverthe package (e.g., procedures forimplementing the intervention one-to-one or in groups), incorporate theconsumers own interests into activi-ties, and collaborate with the con-sumer to set goals (cf. Kendall &Chu, 2000; Kendall, Chu, Gifford,Hayes, & Nauta, 1998; Tee & Ka-zantsis, 2011).Manuals may take a variety of

forms. For example, a manual foroutpatient therapy might lay out asequence of 1-hr sessions, whereas amanual for an educational programmight consist of a curriculum inwhich the learner has to master eachstep before going to the next, alongwith techniques for the teacher to usein delivering instruction. For an ABAintervention package, it might beorganized around functional rela-tions (e.g., a set of procedures forproblem behavior that serves toescape or avoid situations, anotherset of procedures for behavior thatserves to gain access to tangibleitems). The manual might present acomplete script for a session ormerely outline key points to cover

(Frankel, 2004). It may include oneor more types of supporting materialssuch as handouts, worksheets, videosfor training interventionists or forillustrating concepts for consumers,data sheets for monitoring progress,and so on. A manual may undergoseveral revisions to take into accountfindings from initial testing of itsacceptability (social validity) andusability (extent to which providersimplement procedures with fidelityand recipients of the interventionadhere to the protocol). As describedpreviously, the manual may encom-pass one intervention procedure ormany, provided that it offers asolution to a problem presented inpractice and strategies that commu-nity providers can use to implementthe procedures effectively and effi-ciently. If the manual centers on onlya few procedures, it may be possibleto embed it within a research article;otherwise, it is likely to require aseparate publication (Chambless &Hollon, 1998).The availability of a manual is

widely viewed as a requirement forclassifying an intervention as evi-dence based (Chambless & Hollon,1998; Society for Prevention Re-search, 2004). As stated by Cham-bless and Hollon, research projectsfor which a treatment manual wasnot written and followed are oflimited utility in terms of assessmentof treatment efficacy (p. 11). Thisview is well founded. It would bevirtually impossible to generate rep-licable findings about unstandardizedpackages such as an antecedentpackage or a behavioral package(NAC, 2009) to reduce problembehavior. There are no criteria fordetermining when to use one or theother package, how many elements toinclude, which elements should al-ways be included in the package andwhich can vary across individuals, inwhat sequence to introduce them,what recourse is available if the initialpackage is insufficient, or how tomonitor fidelity of implementation.



Without such criteria, researchersand providers cannot determinewhether or not they have accuratelyreplicated an unstructured package.In contrast, as previously noted,interventions with manuals, such asthose developed within PBIS (e.g.,Dunlap et al., 2010), do includespecific methods to address theseissues. Thus, manuals create the op-portunity to test a well-defined inter-vention package across consumersand providers, as must occur to amassa body of research that would justifyclassifying an intervention as evidencebased. Although future researchersmay develop technologies other thanmanuals for devising replicable inter-vention packages, manuals are cur-rently the only such technology(McHugh & Barlow, 2012).In this writers judgment, the role

of manuals in establishing evidence-based practices may explain why theefficacy of schoolwide PBIS interven-tions has been evaluated much morebroadly (i.e., in much larger studies,with independent researchers) thanunstructured ABA packages for prob-lem behavior (see Horner, Sugai, &Anderson, 2010) or even ABA pack-ages that have been described in moredepth but not presented in manuals(e.g., functional communication train-ing; Tiger, Hanley, & Bruzek, 2008).The same pattern holds for otherABA interventions with manualscompared to unstructured packagesfor the same problems. The efficacy ofthe UCLALovaas model of EIBI hasbeen replicated much more broadlythan any other comprehensive ABAintervention for children with ASD(Smith, 2011), pivotal response treat-ment more broadly than other ap-proaches to incidental teaching (Koe-gel, Koegel, Vernon, & Brookman-Frazee, 2010), the picture exchangecommunication system more broadlythan other forms of augmentative oralternative communication (Flippin,Reszka, & Watson, 2010), integrativebehavioral marital therapy morebroadly than unstandardized forms

of acceptance and commitment ther-apy (Christensen et al., 2004), thecommunity reinforcement approachplus vouchers more often than othercontingency management systems forsubstance abuse (Higgins et al., 2003),and the comprehensive behavioralintervention for tics more broadlythan other packages that involve habitreversal (Piacentini et al., 2011).The requirement for replicability

by independent providers (often ne-cessitating the availability of a man-ual) sharply reduces the number ofABA approaches that can be regard-ed as evidence based. Of the 11interventions for ASD identified inthe NAC (2009) report, only the threethat have been standardized in man-uals might be considered to bepractices, and even these may beincomplete. For example, a manualis available for writing stories toinclude in a story-based interventionpackage (Gray, 2004), but not forhow to use the stories to change thebehavior of an individual with ASD.Thus, there are many ABA proce-dures and unstructured packages, butrelatively few practices.Nevertheless, some possible draw-

backs of manuals deserve mention. Apotential weakness of practices thatinvolve intervention packages withmanuals is that the step-by-stepinstructions are liable to give theimpression that unskilled providerscan implement the package (John-ston et al., 2006). In so doing, theymay tempt agencies to employ suchproviders, thus diminishing the effec-tiveness of the package. It is certainlytrue that, when demand has grownfor ABA interventions such as EIBI,unskilled providers have proliferated(Green, 1999). Thus, it may benecessary to anticipate this unintend-ed consequence by specifying mini-mum qualifications for providers anddisseminating information to con-sumers about these qualifications(e.g., Autism Special Interest Groupof the Association for BehaviorAnalysis, 2007; Shook, Johnston, &


18 TRISTRAM SMITH

Mellichamp, 2004). A manual is anaid to clinical decision making, not asubstitute. With its balance of stan-dardization and flexibility, it struc-tures the package yet gives providerslatitude to tailor it to individualcases.Another concern is that manuals

may stifle expert providers (Johnsonet al., 2006). Analogous concernshave been voiced by professionals inother disciplines (see Addis & Kras-now, 2000). Research in these disci-plines provides scant evidence thatexpert providers achieve better out-comes if left to their own devices thanif given a manual (e.g., Durlak &DuPre, 2008; Woolf & Johnson,2005). However, the large and com-plex literature of SSED studies onmany ABA procedures raises thepossibility that there really could bean advantage to letting expert pro-viders rely on the literature and theirexperience rather than a manual.Indeed some writers have portrayedbehavioral artists (Foxx, 1996)who have an uncanny knack forcreating and administering ABA in-terventions. It may be worth evalu-ating whether these providers aremore effective without a manual. Ifso, consistent with the technologicalfocus of ABA, the next step would beto conduct research that seeks toencapsulate what these providers doso that others can replicate it. Thus,this line of inquiry would lead towardimproving manuals rather than dis-carding them.An additional concern is that,

because standardized packages aremore detailed than unstructuredpackages, they may be more difficultto disseminate in community settings.Unstructured packages can be (andfrequently are) promulgated widely inthe form of practice guidelines orrecommendations. For example, re-lying largely on ABA research, theAmerican Academy of Pediatrics(Committee on Psychosocial Aspectsof Child and Family Health, 1998)developed guidelines for pediatricians

to counsel parents on how to disci-pline their typically developing chil-dren. Parents are encouraged to usepositive reinforcement frequently forappropriate behavior and to discour-age acting-out behavior with proce-dures such as time-out rather thanverbal reprimands or corporal pun-ishment. However, in keeping withthe previous discussion about thedifficulty of replicating unstandard-ized packages, little is known abouthow best to promote accurate imple-mentation of such guidelines (Grim-shaw et al., 2004).In sum, interventions that are

bundled into a package and detailedin a manual are not the be-all-and-end-all of treatment research. Indi-vidual procedures in the form ofbehavioral kernels and unstructuredpackages in the form of guidelinesalso have utility. Nevertheless, inter-ventions with manuals are central toevidence-based practice because theyprovide the integration necessary toaddress a problem thoroughly andthe standardization needed to repli-cate findings across studies.

SSEDS OR GROUP DESIGNS

Behavior analysts concede thatgroup designs have some valid uses,particularly for addressing actuarialquestions such as whether one inter-vention tends to be more effectivethan another (Johnston, 1988; Skin-ner, 1968). Nevertheless, we haverepeatedly asserted that SSED studiesare sufficient to establish an inter-vention as evidence based. To high-light the idiographic tradition inABA (i.e., the focus on evaluationof interventions for each individualrather than for a group), Keenan andDillenburger (2011) described ABAas evidence based because interven-tions are evaluated carefully andcontinuously for each individualwho receives them, and the standardintervention procedures are part of ageneral-purpose technology distilledfrom more than a century of research



in laboratory and field settings witha variety of human and nonhumanpopulations. Examples of such pro-cedures include differential reinforce-ment, prompting and prompt fading,shaping, and chaining.To support their position, Keenan

and Dillenburger (2011) cited Reese,Howard, and Reeses (1978) generalprocedure for applied behavior anal-ysis, which includes strategies foridentifying target behaviors, conduct-ing a task analysis, selecting amonggeneral-purpose ABA interventions,and testing outcomes. Comparablesystems have been proposed by oth-ers. For example, Kameenui andSimmons (1990, p. 87) proposed fivegeneric design-of-instruction fea-tures for academic tasks. Thesefeatures involve the use of ABAprinciples to assess what childrenneed to learn, to select curriculummaterials, to plan lessons, to gener-alize skills to new contexts, and topinpoint successful and unsuccessfulaspects of the lessons. Reese et al.sand Kameenui and Simmonss sys-tems present their frameworks forimplementation of ABA technologywithout reference to any specificstudies to support their deploymentfor a particular problem or with aparticular population of individuals.Keenan and Dillenburger (2011)

used the example of shaping toillustrate why a procedure can beconsidered evidence based withoutthe need for prior RCTs or otherstudies to document its effectiveness.They note that, when implementedfor an individual learner, shaping isnot based on fixed treatment reci-pes (p. 8) copied from previousresearch. Rather, it is a dynamicprocess that is individualized for thelearner and that evolves continuouslybased on data obtained on theindividuals progress. These dataconstitute the evidence on the efficacyof shaping for that individual.Certainly, it is necessary to develop

an individualized intervention planfor each recipient of services, to

monitor progress regularly, and toadjust the plan as new informationbecomes available. In the process, itmakes sense to start with tried-and-true procedures such as shaping.But the dangers of drifting too fartoward the idiographic are that weend up reinventing the wheel everytime a new client comes into theoffice or clinic (Barlow, 1997,p. 448). As shown in the precedingsection, reinventing the wheel is aformidable undertaking. Thus, inves-tigators have recommended ways tostandardize even dynamic processessuch as shaping (Galbicka, 1994).Moreover, documenting that thewheel is working requires more thanjust showing an uptick on a graph ofan individuals behavior. A behaviormay change but remain a problem forthe individual, or may be only a smallcomponent of a much larger clusterof problems such as addiction ordelinquency (Baer et al., 1987). Also,a procedure may be appropriate foran individual yet can be administeredin the context of a program that maynot be inappropriate.Such complications abound in

ABA intervention, as illustrated bythe following examples:

1. Many ABA interventions havebeen directed toward increasing therate at which preschoolers answerquestions appropriately (Ingvarsson,Tiger, Hanley, & Stephenson, 2007)or the rate at which children withdisabilities initiate or respond toovertures from peers (Odom et al.,1999). However, such gains may notbe enough to overcome the childrensinitial problems in these areas. Fur-thermore, to be liked and makefriends, children are likely to needmany other skills (e.g., skills forsustaining interactions around sharedinterests and for resolving conflicts).2. Most ABA interventions for

teaching children with disabilities toread focus on sight reading, butliteracy also entails other critical read-ing skills such as decoding, phonemic


20 TRISTRAM SMITH

awareness, and comprehension (Brow-der, Wakeman, Spooner, Ahlgrim-Delzell, & Algozzine, 2006).3. Some ABA studies have shown

success in teaching children to handleguns safely, but, as investigators haveemphasized, it is at least as importantto help families store firearms secure-ly (Miltenberger et al., 2005).4. Most studies of ABA interven-

tions for feeding difficulties docu-ment an increase in acceptance ofnovel foods. However, some popula-tions that are prone to feedingdifficulties, such as children withASD, are seldom malnourished orunderweight (Hyman et al., 2012).Further, investigators seldom showthat a particular child with ASD in astudy on ABA feeding interventionshas health problems (but see McCart-ney, Anderson, & English, 2005, for anotable exception), Thus, for manyof these children, outcomes otherthan acceptance of novel foods areprobably more important. Such out-comes might include reduced stress atmealtimes for children and caregiversor a more varied overall diet thatpromotes long-term health.5. The goal of EIBI is to normalize

the functioning of children with ASDby helping them catch up to peers(Lovaas, 1987). However, despite theacquisition of many new skills, somechildren do not reach this goal(Smith, 2011). For these children,providers and families need to gobeyond the data on acquisition ofindividual skills to determine whetheradjusting the intensity, content ormethod of instruction, or goalswould make the program a better fit(Delmolino & Harris, 2011).

In principle, providers could at-tempt to address these issues byconducting exceptionally rigorous,thorough evaluations of an individu-als progress, with SSED studies ofall behaviors of concern and data onsocial validity (the extent to whichothers see the intervention as accept-able and useful). In reality, however,

the exigencies of field settings seldomallow providers the luxury of spend-ing so much time on data collectionand analysis (Johnston, 1993). Thus,some behavior analysts emphasizethe need for a sequence of studieson an intervention, in addition tocase-by-case evaluation. Johnston(1993, 1996) recommended beginningthe sequence with studies in relativelycontrolled settings and then movingto more typical field settings. Thesequence consists entirely of SSEDstudies instead of actuarial groupstudies.To illustrate this sequence, John-

ston (1996) cited a series of studiesthat led to the creation of an inter-vention for teaching women to per-form breast self-examinations. Thefirst studies elucidated the character-istics of lumps in breast tissue. Laterstudies refined procedures for teach-ing women to identify these lumpsin laboratory settings. Finally, studiestested the procedures under condi-tions that increasingly resembledreal-world situations (Pennypacker& Iwata, 1990). Johnston describedthe teaching procedure as the stateof the art in breast self-examination(p. 41). Pennypacker (2010) lamentedthat the Cochrane Collaboration,which conducts influential reviewsof the medical literature to identifyevidence-based practices, concludedthat breast self-examination failed toqualify as evidence based. Indeed,SSED studies on ABA procedures forbreast self-examination provide im-pressive analyses of what to teachand how to teach it (see Pennypacker& Iwata, 1990).However, the status of breast self-

examination as an evidence-basedintervention hinges on a quintessen-tial actuarial question: Does it pre-vent deaths from breast cancer? TheCochrane review examined RCTsthat addressed this question andfound that the answer was no,perhaps because learning a discreteprocedure for self-examination failsto address barriers to performing the



examinations consistently or accu-rately (Koster & Gtzsche, 2008).It is possible that outcomes would

improve with greater reliance on theABA teaching approach cited byPennypacker and Iwata (1990). Inany case, further SSED studies couldhelp find ways to increase adherenceto or accuracy of self-examinationprocedures. However, the only wayto determine whether doing so pre-vents deaths from breast cancer is tostudy large groups of women. Corre-lational studies can examine whether,all other things being equal, womenwho perform self-examinations havelower mortality rates than otherwomen. RCTs can compare groupsthat are taught to conduct self-examinations against groups that donot receive this instruction. In con-trast, single-case studies, which involvea small number of participants, all ofwhom receive the intervention (instruc-tion on how to conduct self-examina-tions), cannot show whether or not theintervention reduces mortality.As another example that may be

more pertinent to the customaryscope of practice for behavior ana-lysts, Baer (1975) drew on the historyof a program for juvenile delinquents(Achievement Place). He suggestedthat research might begin with SSEDstudies that test the effects of a singleprocedure. For instance, the studiesmight use a multiple baseline designacross participants. In this approach,all participants start in a baseline (nointervention) condition, and then theintervention procedure is introducedfor one participant at a time. Thenext stage of research might involvetesting a package of procedures (asopposed to an individual procedure)in a multiple baseline design acrossparticipants, and a subsequent stagemight be to evaluate the package ina multiple baseline design acrossgroups of participants, with thepackage being introduced to onegroup at a time. Although actualresearch programs do not follow sucha neat progression, SSED studies that

involve large groups of participantsdo exist (Biglan, Ary, & Wagenaar,2000). However, such studies areequivalent to commonly used groupdesigns. As recognized by Bailey andBurch (2002), Baers example corre-sponds to a group design, referred toas a wait-list control, wherein onegroup starts intervention at the outsetof the study and is compared toanother group that is placed on awaiting list to start intervention whenthe first group finishes. Biglan et al.(2000) demonstrated that their exam-ple could be described as a groupdesign called an interrupted timeseries study (Shadish, Cook, &Campbell, 2002).In these examples, SSED studies

are pivotal in the development andrefinement of interventions, butgroup studies are required to estab-lish the intervention package asevidence based (i.e., to show thatthe intervention helps to solve thetarget problem). Thus, SSED studiesmay serve as demonstration pro-jects (Kazdin, 2005) or proof ofconcept (Smith et al., 2007), provid-ing a foundation for larger studiesbut not standing alone as the sourcefor the establishment of an interven-tion as evidence based. Stated differ-ently, the process of development andvalidation of an intervention mayrequire multiple stages, as in Baers(1975) example, with SSED studieshaving a central role in earlier stagesand group studies becoming increas-ingly important in later stages (cf.Forness, 2005; Robey, 2004; Schlos-ser, 2009).The reason that this progression is

likely to be necessary is that mostproblems pose important actuarialquestions (e.g., determining whetheran intervention package is moreeffective than community treatmentas usual; deciding whether to investin one intervention package or an-other, both, or neither; and deter-mining whether the long-term bene-fits justify the resources devotedto the intervention). Hence, group


22 TRISTRAM SMITH

studies, which are designed to addresssuch questions (Johnston, 1988), areneeded. A particularly importantactuarial issue centers on the identi-fication of the conditions underwhich the intervention is most likelyto be effective. Although SSEDstudies can explore the effects ofsystematically altering experimentalconditions (Sidman, 1960), groupstudies can better test the generalityof these effects through formal anal-yses of moderators and mediators(variables that influence the effects ofthe intervention; Kasari & Smith, inpress).This perspective may help to re-

solve an apparent contradiction thathas emerged from efforts to identifyevidence-based practices in educationand psychotherapy. Even thoughmany published criteria in psycholo-gy (Chambless & Hollon, 1998),education (Odom et al., 2005), andother professions (e.g., RomeiserLogan, Hickman, Harris, & Heriza,2008) count both single-case andgroup studies as acceptable sourcesof evidence, group studies are muchmore commonly cited in support ofclassifying an intervention as evidencebased (e.g., Nathan & Gorman,2002). For example, the What WorksClearinghouse, which evaluates re-search on educational practices, hasnot identified any practices as evi-dence based from SSED studies as ofthis writing, although it may do so inthe future, in that it has recentlybroadened its review criteria to incor-porate such studies (Kratochwill etal., 2010). Reviews of rehabilitationinterventions for neurological disor-ders mention SSED studies as a usefulresearch design but do not specifyevidence-based practices that haveemerged from such studies (e.g.,Beeson & Robey, 2006; Slifer &Amari, 2009) The initial report onevidence-based practices by theClinical Psychology Division of theAmerican Psychological Association(Chambless et al., 1996) listed threeevidence-based interventions with

support from SSED studies: behaviormodification for encopresis, tokeneconomies for individuals with schizo-phrenia in inpatient settings, andbehavior modification for develop-mental disabilities. However, the lasttwo items were deleted from the nextreport because they were judged to beinsufficiently specific (Chambless &Hollon, 1998), leaving only behaviormodification for encopresis. Althoughencopresis can be a source of substan-tial distress and functional impair-ment, it is a circumscribed difficultythat can usually be overcome in a fewweeks (OBrien, Ross, & Christopher-sen, 1986), and no known alternativestoABA interventions have been testedother than medical treatments. As thisexample suggests, SSED studies canbe the sole source of evidence for abrief intervention to address a narrowproblem, but such interventions arethe exception, not the rule.In contrast to criteria in psycholo-

gy and education, criteria for evi-dence-based medicine usually givegreater weight to RCTs than to anyother design (e.g., Guyatt, Oxman,et al., 2008). Interestingly, however,medical researchers regard someSSED approaches as types of RCTs,which they call N-of-1 randomizedtrials (Guyatt et al., 1986). Forexample, a reversal design involvesobservation of a participants behav-ior in baseline, then the introductionof an intervention, then withdrawalof the intervention to see whether thebehavior reverts to its preinterventionlevel, and then reintroduction of theintervention. In this design, partici-pants behaviors, rather than theparticipants themselves, are assignedto the baseline and interventionconditions (Guyatt et al., 1986; seealso Kratochwill & Levin, 2010).Medical researchers explicitly bor-rowed the reversal design from ABAresearch (Guyatt et al., 1986) anddescribe it as being at least asmethodologically sound as a groupRCT (Guyatt, Rennie, Meade, &Cook, 2008). Nevertheless, important



limitations have also been recognized(Evans & Ildstad, 2001). For exam-ple, it may be possible to obtain aclear baseline and reversal for chronicconditions such as diabetes or hyper-tension, but not for acute or rapidlydeteriorating conditions such as inju-ries or infections. Moreover, it maybe possible to observe clear changeswith fast-acting treatments, but notwith treatments that are slower totake effect (e.g., drugs that takeseveral weeks to build up in the bodyto a level needed to ameliorate thecondition being treated) or that areintended to improve long-term out-comes (e.g., reduction of an individ-uals risk of developing coronaryheart disease later in life). Thus, inprinciple, SSED studies (N-of-1 ran-domized trials) could be the primaryapproach used to establish a medicalintervention as evidence based, butthey seldom are.Consistent with the perspective

presented here, medical investigatorsconduct a variety of types of studiesbefore RCTs are undertaken. Forexample, new drugs are initiallydeveloped in the laboratory and thentested in humans in three or fourphases. Phase 1 studies evaluatesafety and appropriate dosing insmall groups of participants. Phase2 studies provide initial tests of safetyand efficacy. Phase 3 studies are largeRCTs, and Phase 4 studies are RCTsthat test different possible uses of thedrug. Investigators may go on toconduct additional RCTs to deter-mine how best to use the drug incommunity settings (Szilagyi, 2009)and how to help providers prescribe itappropriately (Thornicroft, Lempp, &Tansella, 2011). Thus, although dif-ferent designs are used in earlier stagesof developing interventions, RCTsbecome indispensable in later phases.Medical researchers also concur

with behavior analysts that groupRCTs often include heterogeneousgroups of participants and regularlyshow wide variability in response toa medical treatment. They further

agree that this variability may limitthe generalizability of study findingsto a given individual (Larson, 1990).Therefore, following group RCTs,SSED studies may be recommendedfor testing the efficacy of the inter-vention in different individuals (Lar-son, 1990). Barlow and Nock (2009)likewise advocated such studies aftergroup RCTs of behavioral interven-tions. Thus, SSED studies may beimportant not only for innovation (asnoted above) but also for later stagesof research on individualizing inter-ventions. Still, group RCTs usuallyneed to occur in between.RCTs are not the only design

needed to establish an interventionas evidence based, but neither areSSEDs. The process of developmentand validation involves multiple stag-es and is likely to require multiplekinds of studies. RCTs tend to bebetter suited to the later stages of thisprocess than are SSEDs.

DISCUSSION

Identification of evidence-basedpractices entails more than the anal-ysis of behavior. It requires synthesisof findings into a package thatindependent providers can adoptand that offers a thorough solutionto problems presented by consumers.Although SSED studies are vital foranalysis, other approaches are neces-sary for synthesis, especially thecreation of a manual that combinesprocedures into a standardized yetflexible format and carrying outgroup studies such as RCTs toevaluate effects that are broader thana change in a single target behavior.Behavior analysts have achieved sucha synthesis for diverse problems thatrange from ASD (Mayville & Mu-lick, 2011), substance abuse (Higginset al., 2003), Tourette syndrome(Piacentini et al., 2011), and maritaldiscord (Christensen et al., 2004).However, at times, we have becomeso mired in analysis that we endorseinterventions that are too fragmen-


24 TRISTRAM SMITH

tary, protean, or incompletely testedto be of much use to providers orconsumers.Fortunately, recognition of the

need for synthesis immediately opensup a plethora of exciting opportuni-ties for behavior analysts (Lutzker &Whitaker, 2005). Many problems ofinterest to us have attracted a largenumber of SSED studies on proce-dures that appear to be ready tocompile into packages for larger scaletesting. The following are some ex-amples, and readers are likely toidentify others:Intellectual and developmental dis-

abilities (IDD).ABA intervention forIDD is a prominent area of SSEDresearch. However, parent trainingfor the promotion of adaptive behav-ior and reduction of problem behav-ior (Baker, Ambrose, & Anderson,1989) is the only ABA interventionpackage that has been developedspecifically for such individuals, stan-dardized in a manual, and tested ingroup studies (cf. Handen & Gilchr-ist, 2006). Thus, it is unknown wheth-er ABA interventions might helpindividuals with IDD catch up topeers or improve their quality of life.Parenting. Because of the obvious

importance of parenting, Skinner(1945) and subsequent behavior ana-lysts have eagerly sought to use ABAprinciples to help parents raise theirchildren in ways that are mutuallyfulfilling and stress free. Group stud-ies show that parent training derivedfrom ABA principles can help par-ents to reduce their childrens severe-ly oppositional, aggressive, or delin-quent behaviors (Dretzke et al., 2009)and can lower the risk that someparents will abuse or neglect theirchildren (Edwards & Lutzker, 2009).However, ABA interventions withmanuals have not been developedand tested for counseling parents inprimary care pediatric settings aboutmilder problem behaviors displayedby typically developing children. Thisis a significant gap, because suchbehaviors are among the most com-

mon concerns that parents express(Friman, 2010). Also, although manySSED studies have centered on ABAinterventions for other common con-cerns such as difficulties with feeding(Sharp, Jacquess, Morton, & Herzin-ger, 2010), this research has notprogressed to group studies to eval-uate the impact of intervention pack-ages on parents stress or on chil-drens success in learning to feedindependently and appropriately.Health-related behaviors. Obesity is

among the most urgent public healthconcerns in developed countries inthe 21st century, and ABA proce-dures to promote physical exerciseand healthy eating are widely viewedas especially promising interventionstrategies (Freedman, 2011). SSEDstudies are available on such proce-dures (e.g., De Luca & Holborn,1992), and more such studies areunderway (Freedman, 2011). Therehave also been at least 40 SSEDstudies on ABA procedures to im-prove performance in various sports(Martin, Thompson, & Regehr,2004). However, group studies onthe extent to which such proceduresactually reduce obesity, promote fit-ness, and enhance health are only justbeginning (e.g., Stark et al., 2011).Education. Low academic achieve-

ment has been a perennial concernamong researchers, including behav-ior analysts beginning with Skinner(1954). Many SSED studies have beenconducted on ABA interventions foreducation, and some interventionshave been packaged into curriculasuch as Direct Instruction (Engel-mann, 1980), Headsprout (Twyman,Layng, Stikeleather, & Hobbins,2004), and EdMark (EdMark Read-ing Program, 2004). However, DirectInstruction is the only one to havebeen evaluated in controlled groupstudies. In addition, a federally spon-sored review indicated that the qualityof all but one of these studies was lowand that the overall level of evidencewas small (What Works Clearing-house, 2007). Thus, the impact that



ABA curricula might have on aca-demic achievement remains uncertain.Besides the myriad possibilities for

scaling up ABA interventions, behav-ior analysts are well positioned toaddress some critical overarchingissues. As previously noted, one suchissue is the relative merit of dissem-inating intervention packages withmanuals versus unstructured packages(guidelines) or individual procedures(kernels). Another is whether expertproviders are helped or hindered bymanuals. A third is whether thelimitations of SSED studies in medi-cine (i.e., difficulty using such studiesin research on unstable conditions orslow-acting treatments) are also rele-vant to research on behavioral oreducational problems.Other issues pertain to the manuals

themselves. Most manuals that havebeen used in research are comprisedof approximately 10 sessions in whicha provider works one-to-one withan individual or his or her parent(Kazdin, 1997). Despite the successesof these manuals, they have impor-tant limitations. Interventions thatadhere to the manuals often reduceconsumers target problems but areseldom a cure (Kendall, 2012). Thus,the search for more effective manualsmust continue. Also, the manualshave greatly facilitated disseminationof evidence-based practices to pro-viders, but even now only a smallpercentage of individuals who mightbenefit from such practices haveaccess to them (Kazdin 2011). Forthis reason, ways to disseminateevidence-based practices more widelyare still needed (Embry & Biglan,2008; Kazdin, 2011). As experts inoperationalizing procedures, behav-ior analysts may be uniquely quali-fied to develop innovative ways topackage intervention procedures. In-deed, although the format of somemanuals for ABA interventions fol-lows the customary 10-session model(e.g., Piacentini et al., 2011), others arestrikingly different. For example, be-havior analysts have developed manu-

als for intervention programs such asEIBI that involve many more than 10sessions (Lovaas, 1987) and for parenttraining programs that adjust theamount of service depending on need(Edwards & Lutzker, 2009).An additional topic of interest is

whether the progression from SSEDstudies to manuals to group studies(as outlined in this article) offers trueadvantages. Although medical re-searchers have outlined sequences ofsteps to develop and validate treat-ments such as new medications,nonmedical researchers have hadlittle to say about this process (Mace,1991). For example, some writersdescribe how to create a manual butdo not discuss what kind of evidence,if any, is needed as a prerequisiteto this activity (Hibbs et al., 1997;Kendall et al., 1998). Some indirectevidence suggests that starting withSSED studies can improve outcomes.For example, Seligman (1993) ob-served that psychotherapies for anx-iety disorders appear to be moresuccessful than psychotherapies fordepression, and he hypothesized thatanxiety may be intrinsically moremalleable than depression. An alter-native possibility, however, is thatinterventions for anxiety are moreeffective because they are groundedin SSED studies (see Barlow, 2002),whereas interventions for depressionare not (Beck, 1987). Similarly, Gur-alnick (1998) reviewed studies indi-cating that early intervention appearsto produce larger gains in intellectualfunctioning for children with ASDthan for children with other disabil-ities. Notably, early intervention pro-grams for ASD emerged from SSEDstudies (Mayville & Mulick, 2011),whereas programs for children withother disabilities did not (Guralnick,1997). More direct comparisons ofinterventions derived from SSEDstudies and those derived from othersources would be informative.Despite the potential benefits of

moving from SSED studies to man-uals and group studies, some cau-


26 TRISTRAM SMITH

tions must be acknowledged. First,there is no guarantee that this processwill come to fruition. For instance,investigators reported substantialsuccess in SSED studies on tokeneconomies to improve the behaviorof individuals with schizophrenia(Paul & Lentz, 1977) and created amanual for this approach (Paul &Menditto, 1992). However, the inter-vention was seldom tested in largerscale studies (McMonagle & Sultana,2009) or adopted in practice (Paul &Menditto, 1992). Commentators de-bate whether the intervention lan-guished because investigators andproviders hold distorted views aboutit (Wong, 2006) or because the dataare less favorable than proponentsclaim (Wakefield, 2006). Regardless,the implication is that research anddissemination may stall even for inter-ventions that appear to be promising.An additional caution is that group

studies demand far more resourcesthan most SSED studies. BecauseSSED studies involve few partici-pants, they can be performed byindependent providers and smallteams (Hayes, Barlow, & Nelson-Gray, 1999). In contrast, group stud-ies may be feasible only in largeresearch centers. As an illustration,an RCT of the breast self-examinationprocedure discussed earlier wouldprobably need to involve experts inthe procedure, oncologists, statisti-cians, and data managers acrossmultiple sites with access to a largerecruitment base. Collaboration, then,will be essential (Johnston, 2000).In short, an evidence-based prac-

tice is a service that helps to solve aconsumers problem. Thus, it is likelyto be an integrated package of proce-dures, operationalized in a manual,and validated in studies of sociallymeaningful outcomes, usually withgroup designs. This definition is morerestrictive than what many behavioranalysts have proposed, but it is inkeeping with behavior analysts viewthat ABA is a technology. AlthoughBaer et al. (1968) and subsequent

behavior analysts have rightly em-phasized that, as a technology, ABAfocuses on procedures that are welldefined and thoroughly analyzed,they have also pointed out that thegoal of technology is to make usefulproducts (Baer, 1991; Johnston,2000). Achieving this goal requires acomplex, often challenging process ofbreaking down the product into itscomponent parts and then assemblingthe parts into a consumer-friendlypackageanalysis and synthesis. Be-havior analysts can increase oursuccess in making one type of prod-uct, evidence-based practices, by at-tending to the second stage of thisprocess as we have to the first.

REFERENCES

Addis, M. E., & Krasnow, A. D. (2000). Anational survey of practicing psychologistsattitudes towards psychotherapy treatmentmanuals. Journal of Consulting and ClinicalPsychology, 68, 331339.

American Psychological Association. (2005).Policy statement on evidence-based practicein psychology. Retrieved from http://www.apa.org/practice/resources/evidence/evidence-based-statement.pdf

American Speech-Language Hearing Asso-ciation. (n.d.). Introduction to evidence-basedpractice: What it is (and what it isnt). Re-trieved from http://www.asha.org/Members/ebp/intro/

Autism Special Interest Group of the Associ-ation for Behavior Analysis. (2007). Con-sumer guidelines for identifying, selecting,and evaluating behavior analysts workingwith individuals with autism spectrum disor-ders (2nd rev.). Retrieved from http://www.abainternational.org/Special_Interests/AutGuidelines.pdf

Azrin, N. H. (1977). A strategy for appliedresearch: Learning based but outcome ori-ented. American Psychologist, 32, 140149.

Backer, T. E., David, S. L., & Soucy, G.(Eds.). (1995). Reviewing the behavioralscience knowledge base on technology trans-fer (NIDA Research Monograph 155, NIHPublication No. 95-4035). Rockville, MD:National Institute on Drug Abuse.

Baer, D. M. (1975). In the beginning, therewas the response. In E. Ramp & G. Semb(Eds.), Behavior analysis: Areas of researchapplication (pp. 1630). Englewood Cliffs,NJ: Prentice Hall.

Baer, D. M. (1991). Tacting to a fault.Journal of Applied Behavior Analysis, 24,429431.



Baer, D. M. (2004). Program evaluation:Arduous, impossible, and political. In H.E. Briggs & T. L. Rzepnicki (Eds.), Usingevidence in social work practice: Behavioralperspectives (pp. 310322). Chicago, IL:Lyceum Books.

Baer, D. M., Wolf, M. M., & Risley, T. R.(1968). Some current dimensions of appliedbehavior analysis. Journal of Applied Behav-ior Analysis, 1, 9197.

Baer, D. M., Wolf, M. M., & Risley, T. R.(1987). Some still-current dimensions ofapplied behavior analysis. Journal of Ap-plied Behavior Analysis, 20, 313327.

Bailey, J. S., & Burch, M. R. (2002). Researchmethods in applied behavior analysis. Thou-sand Oaks, CA: Sage.

Baker, B. L., Ambrose, S. A., & Anderson,S. R. (1989). Parent training and developmen-tal disabilities. Monographs of the AmericanAssociation on Mental Retardation, 13.

Barlow, D. H. (1997). Its yet anotherempirical question: Commentary on CanContextualism Help? Behavior Therapy,28, 445448.

Barlow, D. H. (2002). True alarms, falsealarms, and learned (conditioned) anxiety:The origins of panic and phobia. In D. H.Barlow (Ed.), Anxiety and its disorders: Thenature and treatment of anxiety and panic(2nd ed, pp. 219251). New York, NY:Guilford.

Barlow, D. H., & Nock, M. K. (2009). Whycant we be more idiographic in ourresearch? Perspectives on Psychological Sci-ence, 4, 1921.

Baron, A., & Derenne, A. (2000). Quantitativesummaries of single-subject studies: Whatdo group comparisons tell us about individ-ual performances? The Behavior Analyst, 23,101106.

Beck, A. T. (1987). Cognitive therapy. In J. K.Zeig (Ed.), The evolution of psychotherapy(pp. 149178). New York, NY: Brunner/Mazel.

Beeson, P. M., & Robey, R. R. (2006).Evaluating single-subject research: Lessonslearned from the aphasia literature. Neuro-psychological Review, 16, 161169.

Biglan, A., Ary, D. V., & Wagenaar, A. C.(2000). The value of interrupted time-seriesexperiments for community interventionresearch. Prevention Science, 1, 3149.

Birnbrauer, J. S. (1979). Applied behavioranalysis, service, and the acquisition ofknowledge. The Behavior Analyst, 2, 1521.

Bondy, A., & Frost, L. (2002). A picturesworth: PECS and other visual communica-tion strategies in autism. Bethesda, MD:Woodbine House.

Browder, D. M., Wakeman, S. Y., Spooner,F., Ahlgrim-Delzell, L., & Algozzine, B.(2006). Research on reading instruction forindividuals with significant cognitive dis-abilities. Exceptional Children, 72, 392408.

Carr, E. G., Dunlap, G., Horner, R. H.,Koegel, R. L., Turnbull, A. P., & Sailor, W.,Fox, L. (2002). Positive behavior support:Evolution of an applied science. Journal ofPositive Behavior Interventions, 4, 416.

Carr, E. G., & Durand, V. M. (1985).Reducing behavior problems through func-tional communication training. Journal ofApplied Behavior Analysis, 18, 111126.

Chambless, D. L., & Hollon, S. D. (1998).Defining empirically supported therapies.Journal of Consulting and Clinical Psychol-ogy, 66, 718.

Chambless, D. L., Sanderson, W. C., Shoham,V., Bennett Johnson, S., Pope, K. S., Crits-Christoph, P., Woody, S. R. (1996). Anupdate on empirically validated therapies, I.The Clinical Psychologist, 49(2), 518.

Christensen, A., Atkins, D. C., Berns, S.,Wheeler, J., Baucom, D. H., & Simpson,L. E. (2004). Traditional versus integrativebehavioral couple therapy for significantlyand chronically distressed married couples.Journal of Consulting and Clinical Psychol-ogy, 72, 176191.

Christenson, S. L., Carlson, C., & Valdez, C. R.(2002). Evidence-based interventions inschool psychology: Opportunities, challenges,and cautions. School Psychology Quarterly,17, 466474.

Cohen, H., Amerine-Dickens, M. T., & Smith,T. (2006). Early intensive behavioral treat-ment: Replication of the UCLA model in acommunity setting. Journal of Developmen-tal Behavioral Pediatrics, 27, 143155.

Committee on Psychosocial Aspects of Childand Family Health. (1998). American Acad-emy of Pediatrics guidance for effectivediscipline. Pediatrics, 101, 723728.

Creswell, J. W. (2006). Qualitative inquiry andresearch design: Choosing among five ap-proaches (2nd ed.). Thousand Oaks, CA:Sage.

Damschroder, L., Aron, D., Keith, R., Kirsh,S., Alexander, J., & Lowery, J. (2009).Fostering implementation of health servicesresearch findings into practice: A consoli-dated framework for advancing implemen-tation science. Implementation Science, 4, 50.

De Luca, R. V., & Holborn, S. W. (1992).Effects of a variable-ratio reinforcementschedule with changing criteria on exercisein obese and nonobese boys. Journal ofApplied Behavior Analysis, 25, 671679.

Delmolino, L., & Harris, S. L. (2011).Matching children on the autism spectrumto classrooms: A guide for parents andprofessionals. Journal of Autism and Devel-opmental Disorders, 42, 11971204.

Detrich, R. (2008). Evidence-based education:Can we get there from here? ABAI News-letter, 31(3). Retrieved from http://www.abainternational.org/ABA/newsletter/vol313/Detrich.asp

Detrich, R., Keyworth, R., & States, J. (2007).A roadmap to evidence-based education:


28 TRISTRAM SMITH

Building an evidence-based culture. SavageControversies, 1(1), 35.

Dixon, D. R., Vogel, T., & Tarbox, T. (2012).A brief history of functional analysis andapplied behavior analysis. In J. L. Matson(Ed.), Functional assessment of challengingbehavior (pp. 323). New York, NY:Springer.

Dretzke, J., Davenport, C., Frew, E., Barlow,J., Stewart-Brown, S., Bayliss, S., Hyde,C. (2009). The clinical effectiveness ofdifferent parenting programmes for childrenwith conduct problems: A systematic re-view. Child and Adolescent Psychiatry andMental Health. Retrieved from http://www.capmh.com/content/3/1/7

Dunlap, G., Iovannone, R., English, C., Kin-caid, D., Wilson, K., & Christiansen, K., English, C. (2010). Prevent-teach-reinforce:A school-based model of positive behaviorsupport. Baltimore, MD: Brookes.

Dunn, K. E., Sigmon, S. C., Thomas, C. S.,Heil, S. H., & Higgins, S. T. (2008).Voucher-based contingent reinforcement ofsmoking abstinence among methadone-maintained patients: A pilot study. Journalof Applied Behavior Analysis, 41, 527538.

Durlak, J. A., & DuPre, E. P. (2008).Implementation matters: A review of re-search on the influence of implementationon program outcomes and the factorsaffecting implementation. American Journalof Psychology, 41, 327350.

EdMark Reading Program Level 1 and 2Combo Kit [Computer software]. (2004).Austin, TX: Pro-Ed.

Edwards, A., & Lutzker, J. R. (2009).Iterations of the SafeCare model. Anevidence-based child maltreatment preven-tion program. Behavior Modification, 32,736756.

Embry, D. D., & Biglan, A. (2008). Evidence-based kernels: Fundamental units of behav-ioral influence. Clinical Child and FamilyPsychology, 11, 75113.

Engelmann, S. (1980). Direct instruction. Eng-lewood Cliffs, NJ: Educational Technology.

Evans, C. H., & Ildstad, S. T. (Eds.). (2001).Small clinical trials: Issues and challenges.Washington, DC: National Academy ofSciences.

Fixsen, D. L., Naoom, S. F., Blase, K. A.,Friedman, R. M., & Wallace, F. (2005).Implementation research: A synthesis of theliterature. Tampa, FL: University of SouthFlorida, Louis de la Parte Florida MentalHealth Institute, The National Implemen-tation Research Network (FMHI Publica-tion 231).

Flippin, M., Reszka, S., & Watson, L. R.(2010). Effectiveness of the picture exchangecommunication system (PECS) on commu-nication and speech for children with autismspectrum disorders: A meta-analysis. Amer-ican Journal of Speech-Language Pathology,19, 179185.

Forness, S. R. (2005). The pursuit of evidence-based practices in special education forchildren with emotional or b

reading 3601 smith

Documents