single-case research designs: training protocols in visual analysis wendy machalicek university of...
TRANSCRIPT
![Page 1: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/1.jpg)
Single-Case Research Designs:Training Protocols in Visual Analysis
Wendy MachalicekUniversity of Oregon
www.uoecs.org
Acknowledgement:
Rob HornerTom Kratochwill
Joel LevinChris SwobodaGeorge Sugai
Hariharan SwaminathanDan Maggin
![Page 2: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/2.jpg)
Specific Goals
Review training protocols for competency and fluency in visual analysis
www.singlecase.org
![Page 3: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/3.jpg)
Single-Case Research Designs:Training Protocols in Visual Analysis
Wendy MachalicekUniversity of Oregon
www.uoecs.org
Acknowledgement:
Rob HornerTom Kratochwill
Joel LevinChris SwobodaGeorge Sugai
Hariharan SwaminathanDan Maggin
![Page 4: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/4.jpg)
Specific Goals
Review training protocols for competency and fluency in visual analysis
www.singlecase.org
![Page 5: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/5.jpg)
Context: Is visual analysis reliable across analysts?• Some have found visual analysis to be unreliable, but these studies
have been conducted with novice analysts• e.g. Liberman, Yoder, Reichow, & Wolery (2010); Ottenbacher (1993)
• When analysts are trained and experienced, we see high agreement • Kahng et al. (2010)
![Page 6: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/6.jpg)
WWC White Paper
• Single-Case Intervention Research Design Standards• • Thomas R. Kratochwill
• University of Wisconsin-Madison
• • John H. Hitchcock
• Ohio University
• • Robert H. Horner
• University of Oregon
• • Joel R. Levin
• University of Arizona
• • Samuel M. Odom
• University of North Carolina at Chapel Hill
• • David M. Rindskopf
• City University of New York
• • William R. Shadish
• University of California Merced
Available atwww.uoecs.org“Materials”
![Page 7: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/7.jpg)
What is needed
• Better standards for training in single-case methods
• More precision in RFA stipulations• Establishes expectations for reviewers
• Better standards for reviewing single-case• Journal editors• Grant reviewers
• Development of meta-analysis technology
![Page 8: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/8.jpg)
Evaluate the Design
Meets Design Standards Meets with Reservations Does Not Meet Design Standards
Evaluate the Evidence
Strong Evidence Moderate Evidence No Evidence
Effect-Size Estimation
Social Validity Assessment
Does the Design allow
Documentation of Experimental
Control ?
Do the data document
Experimental Control ?
Is the effect something we should care
about?
Stop
Stop
![Page 9: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/9.jpg)
Visual analysis of single case designs• Determine viability of design first① Assess Baseline② Assess the within phase pattern of each phase③ Assess the basic effects with each phase comparison④ Assess the extent to which the overall design documents experimental
control• E.g. Three demonstrations of basic effect, each at a different point in time
• Assess effect size• Assess social validity
![Page 10: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/10.jpg)
Assessing within phase “pattern” and Between phase “basic effect”• Level• Trend• Variability +• Overlap• Immediacy of Effect• Consistency across similar phases_________________________________________• Other: vertical analysis; intercept gap
Within Phase
Between
Phases
![Page 11: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/11.jpg)
Effective training protocols
Fisher, Kelley, & Lomas (2003) • Train practitioners to use dual-criterion method with AB graphs• Baseline split-middle and mean level lines projected onto treatment
phase• Specified proportion of data must fall above/below both lines
• Visual analysis improved when graphs included criterion lines, have written rules, and brief verbal instruction/modeling training provided
![Page 12: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/12.jpg)
Effective training protocols
Stewart, Carr, Brandt, & McHenry (2007)• Train practitioners to use (1) 8-min recorded lecture, demos,
conservative dual-criterion method with AB graphs & (2) 12-min recorded lecture• Lecture alone didn’t improve accuracy
• Visual analysis improved to 80% accuracy when graphs included criterion lines, have written rules, and brief verbal instruction/modeling training provided
![Page 13: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/13.jpg)
Effective training protocolsWolfe & Slocum (2015)• Random assignment of
participants to (a) computer-based instruction, (b) 36 min recorded lecture or (c) control condition• AB graphs varied on trend
and level
![Page 14: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/14.jpg)
Training Protocol for Visual Analysis of Single-case data • Development of visual analysis graphs
• Rob Horner, Bruce Wampold, Richard Freund, Rick Albin, Tom Kratochwill, Chris Swoboda• Level, trend, variability, overlap• Purpose is to emphasize interaction
• Examine and score each graph (ABAB, MBD, ATD graph sets)• Use ALL data in the graph• Combine assessment of:
• Level, trend, variability, overlap, immediacy of effect, consistency of data patterns in similar phases.• Use scoring metric:
• 1 = no functional relation• 2,3,4 = considerable weaknesses • 5 = moderate functional relation (publishable)• 7 = strong functional relation
• Compare scores with Excel file• Median scores from five “expert” Single-case Researchers
http://www.singlecase.org
![Page 15: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/15.jpg)
Using www.singlecase.org• Anyone, any time may use the website to assess visual analysis
competency• For Instructors• Contact Rob with course code, and dates needed
• Set up “coordinator”• Students enter code and name• You can download their scores, number of sessions, time
• For Research• Inter-rater agreement with visual analysis• Training protocols for teaching visual analysis.
![Page 16: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/16.jpg)
Visual analysis of single case designs
• Determine viability of design first• Assess Baseline• Assess the within phase pattern of each phase• Assess the basic effects with each phase comparison• Assess the extent to which the overall design documents experimental
control• E.g. Three demonstrations of basic effect, each at a different point in time
• Assess effect size• Assess social validity
![Page 17: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/17.jpg)
Screen shot of results
![Page 18: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/18.jpg)
Screen shot of results
![Page 19: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/19.jpg)
Summary• Visual Analysis of Single Case Designs• Determine viability of design first• Assess Baseline• Assess the within phase pattern of each phase• Assess the basic effects with each phase comparison• Assess the extent to which the overall design documents experimental
control• E.g. Three demonstrations of basic effect, each at a different point in time
• Assess effect size• Assess social validity
YOUR TURN: The full set of
140 graphs
![Page 21: Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom](https://reader030.vdocument.in/reader030/viewer/2022032709/56649e9f5503460f94ba11d1/html5/thumbnails/21.jpg)
Design activity
• Design a single-case research or Group experimental design evaluating use of www.singlecase.org or another training protocol to improve participant acquisition, fluency of visual analysis in single-case research