practical functional assessment | understanding problem

17
1 23 Education and Treatment of Children ISSN 0748-8491 Educ. Treat. Child. DOI 10.1007/s43494-020-00015-1 Practical Functional Assessment: A Case Study Replication and Extension with a Child Diagnosed with Autism Spectrum Disorder Julia L. Ferguson, Jeremy A. Leaf, Joseph H. Cihon, Christine M. Milne, Justin B. Leaf, John McEachin & Ronald Leaf

Upload: others

Post on 02-Mar-2022

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Practical Functional Assessment | Understanding Problem

1 23

Education and Treatment of Children ISSN 0748-8491 Educ. Treat. Child.DOI 10.1007/s43494-020-00015-1

Practical Functional Assessment: A CaseStudy Replication and Extension with aChild Diagnosed with Autism SpectrumDisorder

Julia L. Ferguson, Jeremy A. Leaf, JosephH. Cihon, Christine M. Milne, JustinB. Leaf, John McEachin & Ronald Leaf

Page 2: Practical Functional Assessment | Understanding Problem

1 23

Your article is protected by copyright and all

rights are held exclusively by Association for

Behavior Analysis International. This e-offprint

is for personal use only and shall not be self-

archived in electronic repositories. If you wish

to self-archive your article, please use the

accepted manuscript version for posting on

your own website. You may further deposit

the accepted manuscript version in any

repository, provided it is only made publicly

available 12 months after official publication

or later and provided acknowledgement is

given to the original source of publication

and a link is inserted to the published article

on Springer's website. The link must be

accompanied by the following text: "The final

publication is available at link.springer.com”.

Page 3: Practical Functional Assessment | Understanding Problem

BRIEF REPORT

Practical Functional Assessment: A Case Study Replicationand Extension with a Child Diagnosed with AutismSpectrum Disorder

Julia L. Ferguson & Jeremy A. Leaf & JosephH. Cihon &

Christine M. Milne & Justin B. Leaf & John McEachin &

Ronald Leaf

# Association for Behavior Analysis International 2020

Abstract Recent literature has described the analysisand treatment of problem behavior that involves anopen-ended interview to inform the conditions underwhich problem behavior is likely to occur, conductinga synthesized contingency analysis, and implementingtreatment that includes functional communication train-ing, delay and denial tolerance training, and increasingthe amount of demands presented after a denial (e.g.,Hanley et al. Journal of Applied Behavior Analysis,47(1), 16–36, 2014). This process has been describedas an Interview-Informed Synthesized ContingencyAnalysis (IISCA) or practical functional assessment.Much of the literature evaluating the practical functionalassessment has originated from authors within the sameresearch group. The purpose of the present study was toreplicate and extend previous research on the practicalfunctional assessment with a different group of re-searchers and in a different setting (i.e., an early inten-sive behavioral intervention clinic). This study sought toextend previous literature by including additional mea-sures of social validity on the open-ended interview,contingency analysis, treatment, and pre-post measureson parental stress. The results were similar to previousresearch with an overall reduction in problem behavior

and increases in functional communicative responsesand compliance with demands.

Keywords Precursor and serious problem behaviors .

Autism spectrum disorder . Practical functionalassessment . Functional communication training

Introduction

Individuals diagnosed with autism spectrum disorder(ASD) often exhibit higher rates of problem behavior,including, but not limited to, aggression, self-injuriousbehavior, stereotypy, property destruction, yelling, andother disruptive behaviors (Matson and Nebel-Schwalm2007). Within clinical practice and behavior analyticresearch, the common approach to treating problembehaviors consists of first conducting a functional as-sessment (Ala’i-Rosales et al. 2018). A functional as-sessment can take many forms, including indirect as-sessments such as questionnaires or interviews (Durandand Crimmins 1988), descriptive assessments based onobservations (Bijou et al. 1968), or experimental func-tional analyses such as a trial based functional analysisor a standard experimental functional analysis (Iwataet al., 1982/Iwata et al. 1994). The results of the func-tional assessment then inform treatment, which oftenconsists of teaching a functionally equivalent responseto replace the problem behavior. Commonly, the re-sponse that replaces problem behavior is a functionalcommunicative response that is taught through function-al communication training (FCT). FCT includes

Educ. Treat. Child.https://doi.org/10.1007/s43494-020-00015-1

J. L. Ferguson (*) : J. A. Leaf : J. H. Cihon :C. M. Milne : J. B. Leaf : J. McEachin :R. LeafAutism Partnership Foundation, 200 Marina, Dr. Seal Beach, CA90740, USAe-mail: [email protected]

J. H. Cihon : C. M. Milne : J. B. LeafInstitute of Behavioral Studies, Endicott College, Beverly, MA,USA

Author's personal copy

Page 4: Practical Functional Assessment | Understanding Problem

teaching a functional communicative response that willbe reinforced while all other problem behavior isextinguished. This process has been shown to be effec-tive in decreasing problem behavior and increasing thecommunicative response (e.g., Carr and Durand 1985).

Recent research has described a contingency analysisinformed by an interview (e.g., Hanley et al. 2014; Jesselet al. 2016; Jessel et al. 2018; Santiago et al. 2016; Slatonet al. 2017; Strand and Eldevik 2018). Test and controlconditions are based on information obtained during theinterview. Unlike a standard experimental functionalanalysis, test conditions can involve a combination ofcontingencies or synthesized contingencies (Hanleyet al. 2014; e.g., escape from demands to attention andtangibles) instead of isolating one contingency (e.g., es-cape from demands). The results of the functional analy-sis then informs treatment. With this approach, treatmentinvolves several components, including teaching an ap-propriate functional communicative response, delay anddenial tolerance training, compliance chaining, and ex-tension of the treatment to relevant people, contexts, andtime periods (Hanley et al. 2014).

The results of the synthesized contingency analysisand the corresponding treatment within this body ofresearch has demonstrated socially valid improvements(i.e., overall decreases in problem behavior, increases inappropriate behavior, and positive ratings from stake-holders). Although this is a promising line of research,much of the literature evaluating the interview informedsynthesized contingency analysis (IISCA) or practicalfunctional assessment has originated from researcherswithin the same group and occurred within a universityoutpatient clinical setting, which families would visit 3 to4 days per week for 1 h of treatment (e.g., Hanley et al.2014). To date, only one other research group has pub-lished a successful replication of an IISCA and treatmentin a home-based setting within an early intensive behav-ior intervention program (Strand and Eldevik 2018).However, given the perceived drastic change in the ap-proach to problem behavior, replication across multiplesites and researchers is still warranted. Therefore, thepurpose of the present study was to replicate and extendprevious research on the practical functional assessmentwith a different group of researchers in a different setting(i.e., an early intensive behavioral intervention clinic).This study also sought to extend previous literature byassessing the social validity of the open-ended interview,contingency analysis and treatment, as well as assessmentof parental stress prior to and following intervention.

Methods

Participant

Heather was an 8-year-old female with an independentdiagnosis of ASD. She had received ABA-based ser-vices since she was 3 years old, but had only beenreceiving ABA-based services from the current clinicfor 1 year at the time of this study. Heather had aVineland-3 (Sparrow et al. 2016) overall adaptive be-havior composite score of 81. Her Vineland-3 commu-nication standard score was 102, scoring in the age-typical range for an 8-year-old. Her standard scores forthe socialization and daily living skills were below thenormative range, 72 and 78, respectively. Heather’sscores on the Aberrant Behavior Checklist (ABC;Aman and Singh 1986) irritability subscale was 21 andhyperactivity was 27; both of which were elevated.Heather was above grade level academically but had along history of problem behavior in the school setting.Heather’s problem behavior included hitting, kicking,biting, and punching staff members and parents, scream-ing, yelling threats (e.g., “I’m going to kill you”), elop-ing from rooms and buildings, spitting, throwing items,property destruction (e.g., drawing on walls, tearingwallpaper off of walls, knocking over furniture, etc.),and sometimes self-injurious behavior (e.g., biting self,hitting head on wall or floor). Relevant functional rela-tionships that had been determined by her current clin-ical supervisor prior to the study were attention, escape,and access to tangibles, but addressing these functionsseparately through behavior analytic interventions (e.g.,differential reinforcement of alterative behaviors, func-tional communication training, offering choices) had notbeen successful at decreasing rates of problem behavior.In 2nd grade, 3 months into the school year, Heather hadbeen removed from her general education classroomdue to problem behavior and started to receive 30 hper week of behavioral intervention at a private clinicas an alternative setting to school. Due to the intensityand frequency of problem behavior Heather displayed,she was only able to be in one specific room within theclinic for safety reasons in which all the furnishings hadbeen removed. The remainder of her 2nd grade schoolyear (i.e., 10 months) was spent at the private clinic. Shewas in 3rd grade at the time of this study, 2 months intothe school year, having resumed attending public schoolfor 4.5 h one day a week with paraprofessional support(~25 h was still at the clinic). However, she was only

Educ. Treat. Child.

Author's personal copy

Page 5: Practical Functional Assessment | Understanding Problem

placed in general education for 30 min due to high ratesof problem behavior.

Setting and Materials

All sessions were conducted at a private clinic in SouthernCalifornia. Sessions were conducted 1 to 5 days a weekand between 2 to 28 times per day. The schedule ofsessions per day and week was determined by the inter-ventionists’ schedule and availability. The contingencyanalysis (described below)was conducted in a small roomat the clinic equipped with a one-way mirror. All inter-vention sessions were conducted as part of Heather’sregular programming and occurred in a regular therapyroom in the clinic (i.e., a room inwhich individual therapysessions were regularly scheduled and designed for be-havioral intervention). During the parent training phase ofintervention, sessions occurred at Heather’s home or inthe community once a day up to 6 days a week.

During all sessions, materials included various aca-demic instructional materials (e.g., textbooks, pens, pen-cils, workbooks), preferred items and activities (identi-fied through the interview conducted) such as Pokémonfigurines, Pokémon cards, Nintendo DS, NintendoSwitch, dinosaur figurines, and an iPad with variouspreferred games and apps.

Interventionists

Heather’s contingency analysis and intervention was con-ducted by two Board Certified Behavior Analysts®(BCBAs®). The first author and second author servedas the interventionists. The first author had 6 years ofexperience providing ABA services to children diag-nosed with ASD and had worked with Heather for overa year at the time of the study as a direct interventionist atthe clinic. The second author had 9 years of experienceproviding ABA services to children diagnosed withASD, was Heather’s case supervisor, and had workedwith her for 2 years at the clinic. Sessions were conductedby just one of the interventionists at a time. The determi-nation of who conducted the session was based on thescheduling availability of each interventionist.

Dependent Variables

There were nine dependent variables evaluated withinthe research study: (1) problem behavior, (2) functionalcommunication responses (FCRs), (3) functional delay

responses (FDRs), (4) tolerating responses, (5) compli-ance with easy demands, (6) compliance with harddemands, (7) social validity, (8) parental stress, and (9)clinical session behavior.

Problem Behavior The first measure was the frequencyof problem behavior. Two types of problem behaviorwere identified, precursor behavior and severe problembehavior. Precursor behaviors that typically precededmore dangerous problem behaviors were defined priorto conducting the contingency analysis so that the inter-ventionist could reinforce the precursor behaviors dur-ing the analysis in order to keep Heather and the inter-ventionist safe. Heather’s precursor behaviors were de-fined as threats (e.g., “I’m going to kill you”), yelling,and tensing (e.g., clinching/tensing her whole body).Heather displayed four topographies of serious problembehavior, which were defined as aggression (e.g.,kicking, hitting, pinching, biting) and property destruc-tion (e.g., breaking items, throwing items). Taken col-lectively, the precursor behaviors plus the serious prob-lem behaviors constituted the set of problem behaviorsto be addressed in this analysis.

Functional Communication Responses (FCRs) FCRswere separated into four topographies with varying levelsof complexity. Simple FCRs were defined as Heathersaying, “my way.” Intermediate FCRs were defined asHeather asking, “Can I have my way please?” ComplexFCRs were defined as Heather saying, “Excuse me[name], can I have my way please?” Ideal FCRs wereverbally identical to the complex FCR but requiredHeather to be seated in a chair, not have preferred tangibleitems in her hands, and to look at the interventionist whilestating, “Excuse me [name], can I have my way please?”

Functional Delay Responses The FDR was defined asHeather asking, “Can I wrap it up?”

Tolerating Response The tolerating response was de-fined as Heather saying, “Okay, no problem” whilegiving a thumbs up to the interventionist.

Compliance with Easy Demands Easy demandsconsisted of any non-academic tasks already withinHeather’s behavioral repertoire (e.g., clapping hands,touching head, or naming three colors). Compliancewith easy demands was defined as Heather completingthe given task without engaging in problem behavior.

Educ. Treat. Child.

Author's personal copy

Page 6: Practical Functional Assessment | Understanding Problem

Compliance with Hard Demands Hard demandsconsisted of any academic tasks (e.g., math facts orwriting). Compliance with hard demands was definedas Heather completing a given task without engaging inproblem behavior.

Social Validity A social validity questionnaire was pro-vided to Heather’s parents at two times during the study.The first was immediately following the interview andcontingency analysis, and the second was after the in-tervention. The questionnaires were provided toHeather’s parents in unmarked folders that could beprovided to an individual not associated with the studyafter their completion (e.g., receptionist). On each in-stance that the questionnaires were provided, Heather’sparents took the questionnaires home and brought themback 1 to 2 days later. The researchers were not presentwhile Heather’s parents filled out the questionnairessince they were completed at their home. The socialvalidity questionnaire for the interview and contingencyanalysis consisted of five questions using a 7-pointLikert scale (1 representing “not acceptable/comfort-able” and a 7 representing “highly acceptable/comfort-able”). The social validity questionnaire following in-tervention consisted of 10 questions using the same 7-point Likert scale and one open-ended question to pro-vide any additional comments about treatment andoutcomes.

Parental Stress To evaluate parental stress, we providedHeather’s mother with the Parenting Stress Index-4Short Form (PSI-4 SF; Abidin 2012) prior to and fol-lowing intervention. The PSI-4 SF is a standardizedassessment consisting of 36 questions using a 5-pointLikert scale, and consists of three subscales: (a) parentaldistress, (b) parental child-dysfunctional interaction, and(c) difficult child subscale. Scores on each subscale andthe total stress score are considered elevated if they fallabove the 86th percentile.

Clinical Session Behavior Data were also collected onHeather’s behavior during her regularly scheduled clin-ical sessions prior to and following the study. Data werecollected on whether Heather was in programming, outof programming, or in play/reinforcement. In program-ming was defined as Heather engaging in behaviorsrelated to her current programming or teaching (e.g.,following instructions or attending to the intervention-ist) in the absence of problem behavior. Out of

programmingwas defined as Heather engaging in prob-lem behavior (defined above). Play/reinforcement wasdefined as Heather engaging in play or on a break frominterventionist led activities. A 20 min momentary timesampling procedure was used to score each of thesebehaviors.

Measurement and Reliability

All contingency analysis and intervention sessions werevideo recorded. A stopwatch or smartphone was used totime the duration of the sessions. All sessions at theclinic were scored in vivo via paper and pencil. Parenttraining sessions were video recorded and scored thenext day using the same scoring procedures. Sessions inthe clinic lasted 5, 10, or 15min based upon the phase ofintervention (described below). Parent training sessionsaveraged 23 min (range, 10 to 36 min).

Interobserver agreement (IOA) was collected for33% of sessions. IOA was calculated by dividing thenumber of agreements by the number of agreementsplus disagreements and multiplying by 100. IOA forproblem behavior averaged 99.0% across conditionsand sessions (range, 93.3–100%). IOA for simple FCRsaveraged 98.6% (range, 93.3–100%), intermediate FCRIOA was 100%, complex FCR IOA averaged 98.4%(range, 93.3–100%), and IOA for ideal FCRs averaged98.9% (range, 90–100%). IOA for the tolerating re-sponse averaged 99.2% (range, 93.3–100%) and forthe FDR IOA was 99.8% (range, 96.7–100%). TheIOA for compliance with easy demands was 99.5%(range, 93.3–100%) and IOA for compliance with harddemands was 99.5% (range, 90–100%). IOA for clinicalsession behavior was 100% for in programming, out ofprogramming, and play/reinforcement. Treatment integ-rity was not measured throughout the analysis andintervention.

Design

A multielement design (Kazdin 2011) was used duringthe contingency analysis to compare test and controlconditions. A changing criterion design (Hartmann andHall 1976) was used to evaluate the effectiveness of theintervention. The criteria for moving up a phase duringtreatment was contingent upon (1) the absence of prob-lem behavior for two consecutive sessions, (2) indepen-dent communicative and/or tolerating responses, and (3)compliance with task demands (if applicable).

Educ. Treat. Child.

Author's personal copy

Page 7: Practical Functional Assessment | Understanding Problem

Functional Assessment Procedure

Open-Ended Interview An open-ended interview wasconducted with Heather’s parents prior to conductingthe contingency analysis. The interview was conductedby the first author and consisted of 20 questions from theinterview developed by Hanley (2012). The interviewwas intended to help define problem behavior, deter-mine various contextual variables to be included incontingency analysis (e.g., evocative stimuli), and hy-potheses of behavioral function. Heather’s parents wereinterviewed together but were each given an opportunityto provide a response to each question.

Contingency Analysis The control and test conditionspresent during the functional analysis were developedusing the answers from the open-ended interview. Thepurpose of the control condition was to establish acontext in which problem behavior was least likely tooccur. The control condition consisted of providingHeather non-contingent access to preferred items andactivities, no explicit demands, non-contingent access toattention from the interventionist, and interventionistcompliance with any reasonable requests from Heather.Unreasonable requests were any request that could notbe fulfilled by the interventionist (e.g., going to the icecream store, purchasing apps on the iPad, or going toDisneyland). If any unreasonable request was made, theinterventionist responded by telling Heather it was un-reasonable and that s/he could not fulfill the request.

The test condition included evocative events inwhich the interventionist restricted access to all pre-ferred items and activities, placed demands, and deniedany of Heather’s requests. The evocative events stoppedcontingent upon any instance of problem behaviordisplayed by Heather. The contingency analysisconsisted of five sessions: a control condition, a testsession, a control session, and two consecutive testsessions. The analysis was concluded following differ-entiation between conditions.

Intervention

Functional Communication Training Functional com-munication training consisted of verbal instructions,modeling, practice, and feedback. Functional communi-cation training occurred with Heather prior to interven-tion sessions. To teach Heather the targeted FCR theinterventionist instructed her that if she was told to do

something she did not want to do she could say the FCR(e.g., “my way”), and then she could go back to engag-ing in the activity of her choice. After instructing Heath-er of what she could say, Heather and the interventionistthen role-played the interaction. The role-playsconsisted of the interventionist providing a general in-struction (e.g., “Okay Heather, it is time to work”) to setthe occasion for Heather to engage in the target FCR. IfHeather engaged in the targeted FCR then the interven-tionist said, “Okay you can have your way” whichsignaled to Heather that she could go back to engagingwith the activity of her choice, with attention from theinterventionist, and the interventionist complying withall reasonable requests Heather made. After role-playingwith the interventionist, the interventionist providedfeedback (e.g., praise or corrective) based on how sheperformed in the role-play. Once Heather was consis-tently stating the target FCR independently during role-play scenarios, intervention sessions began. Interventionsessions lasted 5 min. The criterion to move from onephase to the next was Heather independently engagingin the targeted responses 100% of opportunities in theabsence of any problem behavior for two consecutivesessions.

Simple Functional Communicative Response Duringthis condition, simple FCRs were targeted (previouslydescribed) following an evocative event (e.g., demandfrom the interventionist, restricting access to a preferreditem/activity). Evocative events were presented by theinterventionist on an average of every one minute (i.e., 5evocative events presented per session). Once Heatherengaged in the simple FCR the evocative event wasterminated, and she had free access to reinforcing itemsand activities. If Heather did not engage in the simpleFCR independently, then the interventionist verballyprompted her to state the simple FCR. Once she en-gaged in the prompted response, the evocative eventwas terminated, and she had free access to reinforcingitems and activities.

I n t e rmed i a t e Fun c t i o na l Commun i c a t i v eResponse Intermediate FCRs were targeted during thiscondition following an evocative event. Evocativeevents were presented by the interventionist on an aver-age of once per minute (i.e., five evocative events pre-sented per session). Once Heather engaged in the inter-mediate FCR she was allowed access to any preferreditems or activities and the evocative event was

Educ. Treat. Child.

Author's personal copy

Page 8: Practical Functional Assessment | Understanding Problem

terminated. If Heather engaged in the previouslytargeted FCR (i.e., simple FCR) the interventionist ei-ther waited for Heather to self-correct and emit theintermediate FCR independently or prompted her tostate the intermediate FCR before terminating the evoc-ative event.

Complex Functional Communicative Response Oncethe complex FCR was taught, the prior FCRs (i.e.,simple and intermediate) were placed on extinctionand no longer reinforced. The complex FCR consistedof Heather saying “Excuse me [interventionist name],can I have my way please?”, waiting for the interven-tionist to respond with “yes”, then Heather could goback to “her way” consisting of free access to all pre-ferred items and activities, no demands placed by theinterventionist, access to attention from the intervention-ist, and the interventionist would comply with all rea-sonable demands placed by Heather. Evocative eventswere presented by the interventionist on an average ofonce per minute (i.e., 5 evocative events presented persession).

Ideal Functional Communicative Response Similar tothe other FCRs, once the ideal FCR was taught the otherFCRs were placed on extinction. Once Heather engagedin the ideal FCR she had access to “her way”which wasset up to be the same as the control condition in thefunctional analysis. The ideal FCR consisted of theinterventionist presenting an evocative event about ev-ery minute (e.g., “Okay, it is time to get to work”),Heather moving to sit in a chair at a table, orienting tothe interventionist, and stating, “Excuse me [interven-tionist name]”, waiting for the interventionist to ac-knowledge her, and then asking, “Can I have my wayplease”? Once Heather engaged in this response theinterventionist reinforced this response by allowingHeather access to “her way”. If Heather did not engagein the ideal FCR, then the interventionist would prompther through the interaction with verbal prompts (e.g.,“Remember to look at me when you ask for your way”)and reinforced the prompted ideal FCR by allowingHeather access to “her way”.

Denial and Delay Tolerance Training A denial from theinterventionist consisted of stating “no” or “not rightnow” following an ideal FCR emitted by Heather.Heather was taught the tolerating response of saying,“Okay, no problem,” while giving a thumbs-up sign,

and orienting toward the interventionist. Teaching thetolerating response was similar to functional communi-cation training. Teaching the response occurred outsideof intervention sessions and consisted of instructions,modeling, role-play, and feedback based on Heather’sperformance during the role-play. During denial anddelay tolerance intervention sessions, Heather was told“no” or “not right now” after an ideal FCR on 40–60%of opportunities (i.e., following an evocative event, twoor three of every five ideal FCRs were denied by theinterventionist and required Heather to engage in thetolerating response, the remaining two or three producedreinforcement following the ideal FCR). Responses toreinforce (i.e., ideal FCR or tolerating response) wererandomly determined prior to sessions. If the trial wasplanned as a tolerating response trial, she would bereinforced for producing the tolerating response (i.e.,saying “OK no problem” and giving thumbs up gesture)by allowing Heather to go back to “her way”. If Heatherdid not engage in the tolerating response following adenial, then the response was verbally prompted by theinterventionist (e.g., “Say, okay no problem”) and onceHeather engaged in the tolerating response, she could goback to “her way”. Sessions during this phase of inter-vention lasted 5 min and an evocative event occurredfive times throughout the session.

Functional Delay Response During the denial and delaytolerance phase of intervention, Heather continued toengage in consistent, but low levels of problem behav-ior. Problem behavior typically occurred while Heatherwas playing a game on the iPad that could not be pausedand was asked to stop and engage in instructional activ-ities. As a result, the interventionists taught Heather afunctional delay response (i.e., saying, “Can I wrap itup?”) that allowed her an extra 30 s to produce the idealFCR. Heather was taught the FDR through verbal in-structions, modeling, role-play, and feedback. The func-tional delay response consisted of a series of interactionsbetween Heather and the interventionist. The interactionbegan with the interventionist providing an evocativeevent (e.g., “Time to get back to work”) which set theoccasion for Heather to engage in the FDR. If Heatherengaged in the FDR, the interventionist alwaysresponded, “Yes”. Heather was then given 30 s to finda point at which the activity or game could be paused orended. Once she paused or ended the game/activityHeather emitted the ideal FCR. If Heather did not endthe activity or game within 30 s the interventionist

Educ. Treat. Child.

Author's personal copy

Page 9: Practical Functional Assessment | Understanding Problem

would begin to provide instructions (e.g., “Okay, it’stime to work”) until Heather ended the game and pro-duced the ideal FCR.

Compliance Chaining Compliance chaining consistedof nine phases (See Table 1). Each phase increased thenumber and difficulty of demands. Five planned evoc-ative events occurred in each session during compliancechaining. Table 1 displays the responses that Heatherwould need to engage in in order to receive reinforce-ment in the form of access to “her way”. Phases 1 and 2had one sequence of responses that were candidates forreinforcement. The order of the responses that would bereinforced was randomized and determined prior to eachsession. Phases 3 through 9 had two sequences of re-sponses that were candidates for reinforcement in whichthe sequences were alternated between per session. Theorder of responses to reinforce within each session wasrandomized and determined before each session began.Sessions during phases 1 through 3 were 5 min long.Sessions during phases 4 through 8 lasted 10 min, andsessions during phase 9 lasted 15 min. The criterion formoving up through the phases was fulfillment of all thefollowing for two consecutive sessions: absence ofproblem behavior, producing all independent communi-cative and tolerating responses, and independently com-plying with all task demands presented.

Treatment Extension-Parents Once Heather had com-pleted all phases of compliance chaining at the clinic,treatment extension with Heather’s parents began. Ses-sions were conducted by either Heather’s mother (65% ofsessions) or father (35% of sessions) and only one sessionwas conducted per day. Parent training with Heather’sparents occurred prior to the treatment extension sessionsand consisted of written and verbal instructions, model-ing, role-play, and feedback on how to conduct interven-tion sessions with Heather. Heather’s parents were firsttrained to reinforce the ideal FCR, then training focusedon reinforcing the tolerating response, and then reinforc-ing compliance with task demands after a denial. Trainingprogressed as Heather’s parents were accuratelyimplementing the treatment and reinforcing the targetresponse in role-play scenarios with Heather. Parent train-ing lasted 1.5 h in duration for Heather’s mother andlasted 2 h in duration for Heather’s father. When treat-ment extension sessions began at Heather’s home, herparents started implementing phase 6 of compliancechaining. Criteria to move from one phase of compliance

chaining to the next was contingent on Heather indepen-dently engaging in the targeted responses 100% of op-portunities in the absence of any problem behavior fortwo consecutive sessions.

Table 1 Responses to reinforce during compliance chaining*

Phase Responses reinforced per trial

1 1. Ideal FCR2. Ideal FCR3. Tolerating response4. 1 easy task5. 1 easy task

2 1. Ideal FCR2. Tolerating response3. 1 easy task4. 1 easy task5. 2 easy tasks

3 1. Ideal FCR2. Tolerating response3. 2 easy tasks4. 3 easy tasks5. 1 hard task

1. Ideal FCR2. Tolerating response3. 2 easy tasks4. 1 hard tasks5. 3 hard tasks

4 1. Ideal FCR2. Tolerating response3. 3 easy tasks4. 5 easy tasks5. 1 hard task

1. Ideal FCR2. Tolerating response3. 1 easy task4. 5 easy tasks5. 3 hard tasks

5 1. Ideal FCR2. Tolerating response3. 4 easy tasks4. 6 easy tasks5. 2 hard tasks

1. Ideal FCR2. Tolerating response3. 2 easy tasks4. 4 hard tasks5. 6 hard tasks

6 1. Ideal FCR2. Tolerating response3. 5 easy tasks4. 7 easy tasks5. 3 hard tasks

1. Ideal FCR2. Tolerating response3. 7 easy tasks4. 3 hard tasks5. 5 hard tasks

7 1. Ideal FCR2. Tolerating response3. 2 easy tasks4. 10 easy tasks5. 7 hard tasks

1. Ideal FCR2. Tolerating Response3. 7 easy tasks4. 2 hard tasks5. 10 hard tasks

8 1. Ideal FCR2. Tolerating response3. 2 easy tasks4. 13 easy tasks5. 10 hard tasks

1. Ideal FCR2. Tolerating response3. 13 easy tasks4. 2 hard tasks5. 10 hard tasks

9 1. Ideal FCR2. Tolerating response3. 3 easy tasks4. 10 easy tasks5. 20 hard tasks

1. Ideal FCR2. Tolerating response3. 10 easy tasks4. 3 hard tasks5. 20 hard tasks

*Order of responses to reinforce was randomized prior to eachsession

Educ. Treat. Child.

Author's personal copy

Page 10: Practical Functional Assessment | Understanding Problem

Results

Functional Analysis

The answers provided by Heather’s parents during theopen-ended interview suggested Heather’s problem be-havior occurred during adult-directed activities in whichHeather’s access to preferred items and activities wasrestricted and when an adult was not complying withHeather’s requests. Figure 1 provides the results of thecontingency analysis. Heather did not engage in anyinstances of problem behavior during the control condi-tion. Heather only engaged in problem behavior duringthe test condition, during which problem behavior oc-curred upon every instance of an evocative event occur-ring at an average rate of 0.93 instances per minute.

Intervention

Figure 2 displays Heather’s behavior during functionalcommunication training and delay and denial tolerancetraining. The first panel of Fig. 2 represents Heather’scompliance with instructions placed; however, no in-structions were placed during this phase, and this panelis provided to remain consistent with the subsequentfigures. The remaining panels of Fig. 2 consist of prob-lem behavior (second panel), FCRs (third panel), andtolerating responses (fourth panel). Once the simpleFCR was taught, there was an immediate decrease inproblem behavior and an increase in the frequency of

simple FCRs. When the intermediate FCR was taught,problem behavior remained at low rates with the excep-tion of only two instances of problem behavior duringthis phase of functional communication training. Itshould be noted, the problem behavior that occurredduring this phase was tensing, a behavior that wasconsidered a precursor to other dangerous problem be-havior. During the intermediate FCR condition, the fre-quency of simple FCRs decreased with an increase inthe frequency of intermediate FCRs. No instances ofproblem behavior occurred during the complex FCRcondition, and the previously reinforced intermediateFCRs decreased in frequency, and the frequency ofcomplex FCRs increased. During teaching of the idealFCR, the frequency of complex FCRs decreased whilethe frequency of ideal FCRs increased. During thisphase, there were four sessions during which problembehavior occurred. Similar to the intermediate FCRphase, these instances of problem behavior were in-stances of body tensing.

During denial and delay tolerance sessions, Heatherrequired more sessions to emit the tolerating responseindependently (see, panel 4 of Fig. 2). Complex FCRsand ideal FCRs continued to occur at similar frequencies(see, panel 3 of Fig. 2). During this phase, there was alsoan increase in the frequency of problem behavior. Sim-ilar to prior phases of treatment, this problem behaviorconsisted of non-dangerous precursor behavior with theexception of session 50, in which Heather engaged intwo instances of aggression. Due to the consistent ratesof problem behavior during the denial and delay toler-ance training, an FDR was taught. Following teachingan FDR, the frequency of problem behavior decreased.

Following denial and delay tolerance training, thecompliance chaining condition began (see Fig. 3). Dur-ing phase 1, Heather complied with instructions but alsoengaged in variable rates of problem behavior. Follow-ing a spike in session 100, there was a gradual decline tozero for the remainder of phase 1. All instances ofproblem behavior consisted of precursor problem be-havior (e.g., verbal threats or body tensing). Duringphase 2, Heather did not engage in any problem behav-ior, complied with all instructions, and continued toengage in ideal FCRs and tolerating responses indepen-dently. Throughout the remaining phases, Heather en-gaged in variable rates of problem behavior but consis-tently complied with instructions and engaged in idealFCRs and tolerating responses independently. Similar toother phases of treatment, Heather’s problem behavior

1 2 3 4 5

Prob

lem

Beh

avio

r Pe

r M

inut

e

0

1

2CONTROL TEST

SessionsFig. 1 Contingency analysis

Educ. Treat. Child.

Author's personal copy

Page 11: Practical Functional Assessment | Understanding Problem

Sessions

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80

Tole

rati

ng R

espo

nse

Per

Min

ute

0 .0

.5

1.0

Inst

ruct

ions

/Com

plia

nce

0

10

20

30

40

Prob

lem

beh

avio

r pe

r m

in

0

1

2

FCR

Res

pons

es p

er m

in

0

1

2

3 sFCRinFCRcFCRidFCRFDR

BL Funct ional Communicat ion TrainingDenial and Delay

Tolerance TrainingsFCR inFCR cFCR idFCR dFCR

Fig. 2 Functional communication training and denial and delay tolerance training

Educ. Treat. Child.

Author's personal copy

Page 12: Practical Functional Assessment | Understanding Problem

during each phase primarily consisted of precursor prob-lem behaviors (e.g., body tensing or verbal threats).

Only nine sessions out of the 105 sessions during thecompliance chaining condition included problem

Sessions

85 90 95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170 175 180 185 190

Tole

rati

ng R

espo

nse

Per

Min

ute

0 .0

.5

1.0

Inst

ruct

ions

/Com

plia

nce

wit

h Th

erap

ist

0

10

20

30

40

Easy tasksEasyComplianceHard tasksHardCompliance

Prob

lem

beh

avio

r pe

r m

in

0

1

2

FCR

Res

pons

es p

er m

in

0

1

2

3

cFCRidFCRFDR

Compliance ChainingP1 P2 P3 P4 P5 P6 P7 P8 P9

Fig. 3 Compliance chaining

Educ. Treat. Child.

Author's personal copy

Page 13: Practical Functional Assessment | Understanding Problem

behavior that was considered dangerous (e.g., aggres-sion or property destruction).

Following the compliance chaining condition at theclinic, parent training began in which intervention wasextended to Heather’s parents at home (see Fig. 4). OnceHeather’s parents implemented the intervention athome, Heather complied with instructions, engaged inideal FCRs and tolerating responses independently, anddisplayed minimal problem behavior during the firstphase of the treatment extension. The next phase result-ed in an increase in problem behavior with the highestfrequency during session 202, which eventually de-creased to a frequency of zero across two consecutivesessions. Following the introduction of the next phase,there was an immediate increase in the frequency ofproblem behavior (i.e., sessions 213–215). This wasfollowed by a decrease in the frequency of problembehavior from session 216 to 221. During sessions222–228 Heather engaged in variable frequencies ofproblem behavior, which were followed by a decreasein frequency in session 229. Throughout the treatmentextension with Heather’s parents, Heather consistentlycomplied with instructions, continued to engage in idealFCRs, and tolerating responses independently.

Social Validity

We first evaluated the social validity of the interviewand contingency analysis using a 7-point Likert re-sponse scale. Heather’s parents responded to five ques-tions: (1) found the interview process to be acceptable;(2) comfortable during the interview process; (3) foundthe functional analysis of my child’s problem behaviorto be acceptable; (4) I consider the functional analysis tobe safe for my child and the therapist; and (5) I wascomfortable watching the functional analysis of mychild’s problem behavior. Heather’s mother and fatherscored a 7 (e.g., very comfortable or very acceptable) onevery single question.

Table 2 includes Heather’s parents’ responses to theassessment of the social validity of the intervention. Thefirst four questions pertained to how satisfied Heather’sparents were with her improvement, her ongoing prob-lem behavior at home, how helpful they thought thetreatment was at home, and how confident they feltimplementing the treatment strategies. Overall, theirresponses were positive; however, both parents indicat-ed they were still concerned about her problem behaviorat home. Two questions on the social validity

questionnaire asked how comfortable they felt takingaway preferred activities or asking Heather to do some-thing before and after the intervention. Heather’s parentsreported low comfortability ratings prior to interventionand much more comfortability taking away Heather’stoys or asking her to do something following interven-tion. Two other questions asked Heather’s parents howthey felt taking her out in public before and after theintervention. Her parents reported low comfortabilityratings prior to intervention and more comfortabilityfollowing intervention. The final two questions on thesocial validity questionnaire asked how comfortableHeather’s parents felt asking Heather to go to sleepbefore and after the intervention. Heather’s parents re-ported that they felt very comfortable asking Heather togo to sleep before and after the intervention. In additionto filling out the social validity questionnaire, Heather’smother and father made additional comments such as,“This process has been fairly painless. Thank you somuch. We are so happy that Heather buys in, which hasnever been the case before” and “Thrilled to have hadthe opportunity to learn this information!”

Parental Stress

Prior to treatment, Heather’s mother had a parental dis-tress score of 35 (78 percentile), a parent-child dysfunc-tional interaction score of 37 (92 percentile), difficult childscore of 52 (>99 percentile), and a total stress score of 124(98 percentile). All of these scores with the exception ofthe parental distress domain would be considered elevated(i.e., fall above the 86th percentile). Following interven-tion, Heather’s mother had a parental distress score of 33(72 percentile), a parent-child dysfunctional interactionscore of 35 (86 percentile), difficult child score of 49(>99 percentile), and a total stress score of 117 (92 per-centile). Heather’s mother still reported elevated stress onthe difficult child and total stress domains; however, theresults showed that parental stress decreased overall andacross all domains following intervention.

Clinical Sessions

Prior to intervention, Heather was engaged in program-ming an average of 40% of the time, was out of pro-gramming an average of 50% of the time, and was inplay/reinforcement an average of 10% of the time. Fol-lowing intervention, Heather was engaged in program-ming an average of 60% of the time, out of

Educ. Treat. Child.

Author's personal copy

Page 14: Practical Functional Assessment | Understanding Problem

Prob

lem

beh

avio

r pe

r m

in

0

1

2

Inst

ruct

ions

/Com

plia

nce

wit

h Pa

rent

0

10

20

30

40 Easy tasksEasyComplianceHard tasksHardCompliance

Sessions

195 200 205 210 215 220 225 230 235

Tole

rati

ng R

espo

nse

Per

Min

ute

0 .0

.5

1.0

FCR

Res

pons

es p

er m

in

0

1

2

3

idFCRFDR

Compliance Chaining

P6 P7 P8

Fig. 4 Treatment extension with Heather’s parents

Educ. Treat. Child.

Author's personal copy

Page 15: Practical Functional Assessment | Understanding Problem

programming an average of 10% of the time, and inplay/reinforcement and average of 40% of the time.

Discussion

This study demonstrated the successful replication ofthe practical functional assessment and treatment de-scribed previously in several studies by a limited num-ber of researchers within an early intensive behavior

intervention (EIBI) program in a clinical setting. Fol-lowing the contingency analysis and treatment, de-creases in the frequency of problem behavior and in-creases in the frequency of desirable behavior wereobserved. Anecdotally, it should also be noted that afterintervention Heather was able to work in a variety ofrooms in the clinic, with regular furnishings, unlike priorto intervention. Results of the assessment of social va-lidity also indicated that Heather’s parents were com-fortable with the interview and functional analysis andfound the treatment to be acceptable and easy to imple-ment at home. Additionally, Heather’s mother’s parentalstress index scores decreased with the exception of thedifficult child subscale.

Similar to Strand and Eldevik (2018), this studyreplicated the IISCA and subsequent treatment withinan EIBI program and did not require any additionalresources or cost of the program. Differences in theimplementation of treatment included several sessionsper day (i.e., up to 28), sessions conducted as many daysof the week as possible (i.e., up to 5 days a week), andmultiple measures of social validity.

Although this study replicated the results of previousstudies (e.g., Strand and Eldevik 2018) limitations canbe noted. Perhaps the main limitation is the inclusion ofonly one participant. Future studies should include rep-lications across multiple participants in an EIBI clinic.Additionally, all sessions in the clinic were conductedby a BCBA®. Although this did not increase the cost ofservices for Heather’s family at this clinic, it may becostly and time-consuming for other BCBAs® withlarger caseloads at other behavior analytic clinics. Fu-ture researchmaywish to look at a model of treatment inwhich a BCBA® is supervising and overseeing treat-ment but not implementing treatment similar to Santiagoet al. (2016).

Another limitation was that although we did see anoverall decrease in the frequency of problem behavior,variable frequencies of problem behavior continued tooccur throughout treatment. Some of the problem be-havior included aggression, property destruction, andthrowing, but the majority was non-dangerous, precur-sor behavior (e.g., yelling or body tensing). Specifically,body tensing occurred most frequently throughout treat-ment sessions. Although Heather’s parents reported inthe open-ended interview that tensing was part of theresponse class comprising problem behavior and was asignal that more dangerous problem behavior was likelyto occur, it may be the case that tensing was not actually

Table 2 Answers to questionnaire on assessment for the treat-ment process and outcomes

Question Mother’sresponses

Father’sresponses

Rate the extent to which you aresatisfied with the amount ofimprovement seen in Heather’sproblem behavior during the practicesessions

6 6

Rate the extent to which you areconcerned about Heather’s ongoingproblem behavior at home

6 6

Rate the extent to which you havefound the assessment and treatmentprovided by our team helpful to yourhome situation up to this point

7 7

Rate the extent to which you feelconfident applying the samestrategies you have seen in thepractice sessions, when addressingHeather’s problem behavior at home

7 6

How comfortable were you takingaway Heather’s preferred activitiesand asking her to do something elseBEFORE you were trained on theintervention?

3 2

How comfortable are you taking awayHeather’s preferred activities andasking her to do something else nowAFTERyou have been trained on theintervention?

6 5

How comfortable were you takingHeather to public places BEFOREyou were trained on the intervention?

1 3

How comfortable are you takingHeather to public places nowAFTERyou have been trained on theintervention?

4 6

How comfortable were you askingHeather to go to sleep BEFORE youwere trained on the intervention?

7 7

How comfortable were you askingHeather to go to sleep AFTER youwere trained on the intervention?

7 7

Educ. Treat. Child.

Author's personal copy

Page 16: Practical Functional Assessment | Understanding Problem

a precursor to problem behavior and outside of theproblem behavior response class. That is, tensing oc-curred throughout treatment in isolation without leadingto more dangerous problem behavior and other problembehavior occurred in the absence of tensing; thus, it isunlikely that tensing was a part of the problem behaviorresponse class.

Another limitation of this study is that the variablefrequencies in problem behavior also contributed to thelength of the intervention. It took over 200 sessions tocomplete the function-based treatment in the clinic andextend the treatment to Heather’s parents.When lookingat how long treatment took in the clinic in terms of days,it took 21 days, and parent training took 61 days due tothe constraints Heather’s parents had of just running onesession per day. Future researchers maywish to comparethis treatment to other function-based treatments in or-der to look at the efficiency of the procedures. It couldbe that other treatments might be more efficient than theone implemented in this study.

Additionally, measures of treatment integrity werenot collected across therapists or Heather’s parents.Taking measures of treatment integrity should be donein future replications of the IISCA function-based treat-ment. If interventionists and parents’ implementation oftreatment varies from person to person this could lead totreatment taking longer to be successful and effective.Since measures of treatment integrity were not collectedin this study, it is unclear how this could have affectedHeather’s progress through treatment.

Within this study we presented data on Heather’streatment extending to her parents at home, but treat-ment was also extended to all of Heather’s staff withinthe clinic and Heather’s classroom at school. Unfortu-nately, data were not collected on the treatment exten-sion to school or to additional staff within the clinic.Although data were not collected onHeather’s treatmentextension at school, anecdotally, the extension was re-ported as successful and resulted in an increase in thenumber of days spent at school (i.e., 5 days a week asopposed to 1 day a week prior to the intervention) andhours spent in a general education classroom with sameaged-peers. Future researchers may wish to collect dataon treatment extension or generalization to all relevantpersons and environments in the client’s life.

While this study was not without its limitations, itscontributions to clinical practice and research literatureshould not be discounted. To our knowledge, this studyrepresents only the second successful replication of the

practical functional assessment and IISCA-informedtreatment within a different research lab. The practicalfunctional assessment offers several advantages over theconventional approach to problem behavior such ashighlighting the importance of synthesized contingen-cies, ability to move quickly to treatment, and arrangingthe environment to resemble the criterion environmentin order to teach the functional skills to replace problembehavior. However, to be considered evidenced basedby several entities (e.g., Horner et al. 2005; NationalAutism Center (NAC) 2015; What Works Clearing-house 2014), it requires replication across several dif-ferent variables, including research labs. It is our hopethat this study helps to contribute to this body of litera-ture and inspires other research labs to replicate andextend the findings of this and other studies.

References

Abidin, R. R. (2012). Parenting stress index (4th ed.). Lutz, FL:PAR.

Ala’i-Rosales, S., Cihon, J. H., Currier, T. D. R., Ferguson, J. L.,Leaf, J. B., Leaf, R., McEachin, J., & Weinkauf, S. M.(2018). The big four: Functional assessment research informspreventative behavior analysis. Behavior Analysis inPractice, 12(1), 222–234. https://doi.org/10.10007/s40617-018-00291-9.

Aman,M. G., & Singh, N. N. (1986). Aberrant behavior checklist:Manual. East Aurora, NY: Slosson Educational Publications.

Bijou, S. W., Peterson, R. F., & Ault, M. H. (1968). A method tointegrate descriptive and experimental field students at thelevel of data and empirical concepts. Journal of AppliedBehavior Analysis, 1(2), 175–191. https://doi.org/10.1901/jaba.1968.1-175.

Carr, E. G., & Durand, V. M. (1985). Reducing behavior problemsthrough functional communication training. Journal ofApplied Behavior Analysis, 18, 111–126. https://doi.org/10.1901/jaba.1985.18-111.

Durand, V. M., & Crimmins, D. B. (1988). Identifying the vari-ables maintaining self-injurious behavior. Journal of Autismand Developmental Disorders, 18, 99–117.

Hanley, G. P. (2012). Functional assessment of problem behavior:Dispelling myths, overcoming implementation obstacles, anddeveloping new lore. Behavior Analysis in Practice, 5(1),54–72.

Hanley, G. P., Jin, C. S., Vanselow, N. R., & Hanratty, L. A.(2014). Producing meaningful improvements in problem be-havior of children with autism via synthesized analyses andtreatments. Journal of Applied Behavior Analysis, 47(1), 16–36. https://doi.org/10.1002/jaba.106.

Hartmann, D. P., & Hall, R. V. (1976). The changing criteriondesign. Journal of Applied Behavior Analysis, 9(4), 527–532.https://doi.org/10.1901/jaba1976.9-527.

Educ. Treat. Child.

Author's personal copy

Page 17: Practical Functional Assessment | Understanding Problem

Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., &Wolery, M. (2005). The use of single-subject research toidentify evidence-based practice in special education.Exceptional Children, 71, 165–179. https://doi.org/10.1177/001440290507100203.

Iwata, B. A., Dorsey, M. F., Slifer, K. J., Bauman, K. E., &Richman, G. S. (1994). Toward a functional analysis ofself-injury. Journal of Applied Behavior Analysis, 27, 197–209. https://doi.org/10.1901/jaba.1994.27-197 (reprintedfrom Analysis and Intervention in DevelopmentalDisabilities, 2, 3-20, 1982).

Jessel, J., Hanley, G. P., & Ghaemmaghami, M. (2016). Interview-informed synthesized contingency analyses: Thirty replica-tions and reanalysis. Journal of Applied Behavior Analysis,49(3), 576–595. https://doi.org/10.1002/jaba.316.

Jessel, J., Ingvarsson, E. T., Metras, R., Kirk, H., & Whipple, R.(2018). Achieving socially significant reductions in problembehavior following the interview-informed synthesized con-tingency analysis: A summary of 25 outpatient applications.Journal of Applied Behavior Analysis, 51(1), 130–157.https://doi.org/10.1002/jaba.436.

Kazdin, A. E. (2011). Single-case research designs: Methods forclinical and applied settings. New York, NY: OxfordUniversity Press.

Matson, J. L., & Nebel-Schwalm, M. (2007). Assessing challeng-ing behaviors in children with autism spectrum disorders: A

review. Research in Developmental Disabilities, 28(6), 567–579. https://doi.org/10.1016/j.ridd.2006.08.001.

National Autism Center (NAC). (2015). Finding and conclusions:National standards project phase 2. Randolph, MA: NationalAutism Center.

Santiago, J. L., Hanley, G. P., Moore, K., & Jin, C. S. (2016). Thegenerality of interview-informed functional analyses:Systematic replications in school and home. Journal ofAutism and Developmental Disorders, 46(3), 797–811.https://doi.org/10.1007/s10803=015-2617-0.

Slaton, J. D., Hanley, G. P., & Raftery, K. J. (2017). Interview-informed functional analyses: A comparison of synthesizedand isolated components. Journal of Applied BehaviorAnalysis, 50(2), 252–277. https://doi.org/10.1002/jaba.384.

Sparrow, S. S., Cicchetti, D. V., & Saulnier, C. A. (2016).Vineland-3: Vineland adaptive behavior scales (3rd ed.).Bloomington, MN: NCS Pearson, Inc..

Strand, R. C., & Eldevik, S. (2018). Improvements in problembehavior in a child with autism spectrum diagnosis throughsynthesized analysis and treatment: A replication in an EIBIhome program. Behavioral Interventions, 33(1), 102–111.https://doi.org/10.1002/bin.1505.

What Works Clearinghouse (2014). Procedures and standardshandbook: Version 3.0. Retrieved from:https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v3_0_standards_handbook.pdf.

Educ. Treat. Child.

Author's personal copy