Topics in Special Education Research

Session 3-Experimental & Quasi-experimental Research

This Evening’s Agenda

• Review Course Requirements & Upcoming Assignments

• Ethics in Research• Review for, Take, & Correct Quiz• Discussion & Lecture on Experimental

Research• Activity• Dismissal

Discussion Guides- Please submit to dropbox folder and label◦ 3 main points for all of the readings for that session.◦ No need to double space◦ Address all of the headings (even the difficulties,

concerns, questions section).◦ Use complete sentences◦ These are designed to guide your discussions.◦ Please put your questions for ME on the feedback

guide, so if they aren’t answered by your discussion group, type them on the feedback guide!

Updates/Questions

Today NIH/CITI training modules August 5th : Annotated Bibliography (individual assignment)

◦ Each individual reviews 3 research articles regarding their topic ◦ See Example:

http://rxsped596.pbworks.com/w/file/fetch/54804527/Example%20Annotated%20Bibliography.pdf

August 5th : Conceptual Framework (group assignment)◦ Group submits short summary of literature and presents a conceptual

framework for theories that drive their proposal.

August 7th : Article Review Assignment (group assignment)

August 12th : Written Research Proposal (group assignment)

August 14th : Presentation of Research Proposal (group assignment)

Upcoming Assignments

Briefly and clearly state how each research question will be addressed.

For example, “This research question will be answered by comparing the end-of-year state wide test scores of students who received the intervention and those who did not receive the intervention.”

Briefly present the proposed theoretical and practical implications of the findings.

e.g., “The results of this study may have implications for the use of the evidence-based Super-Duper Reading Intervention by elementary special education teachers…”

Research Question(s) Guidelines

Framed based to operationalize (clearly define so that it can be replicated) the objectives of the proposed research project.

Mention the IV and DV and how they will affect each other

Framed based on methodology:◦ Experimental/Quasi-experimental/Single-subject: “Is there a

causal of functional relationships between IV and DV”◦ Correlational: “Is there a correlational relationship between IV

and DV” ◦ Descriptive/Qualitative: Describe a phenomenon or issue better,

“What is the prevalence of intellectual disabilities in African-American middle school students?”

Research Question Tips

What did you learn from doing the CITI course?

Regarding…◦ Research misconduct◦ Data Management◦ Responsible authorship◦ Collaboration in Research

◦ http://www.youtube.com/watch?v=W7sfIA1dIGQ◦ http://www.youtube.com/watch?v=yr5cjyokVUs◦ http://www.youtube.com/watch?v=b8McGyYAwcU

Ethics in Research

1. Respect for Persons- ◦ Required to: obtain informed consent & respect privacy of

participants

2. Beneficence-◦ Required to: use best possible design to maximize benefits &

minimize harms, show they can perform the procedures and handle the risks, prohibit research that that is without a favorable risk-benefit relationship

3. Justice◦ Required to: select participants equitably & avoid exploitation

of vulnerable populations or populations of convenience.

The Belmont Report (1979), National Commission—3 Principles

PSU Human Subjects Research Review Committee (HSRC)

http://www.rsp.pdx.edu/compliance_human.php

Portland State University (PSU) is responsible for the rights and welfare of human subjects involved in research sponsored or conducted by the university.   In order to meet this responsibility, the University established the Human Subjects Research Review Committee. 

Members are charged with reviewing all research

conducted under the auspices of PSU that involves human subjects to ensure adequate protections are in place.

Review for Quiz

In-text Citations: Formatting Quotations

Caruth (1996) states that a traumatic response frequently entails a “delayed, uncontrolled repetitive appearance of hallucinations and other intrusive phenomena” (p.11).

A traumatic response frequently entails a “delayed, uncontrolled repetitive appearance of hallucinations and other intrusive phenomena” (Caruth, 1996, p.11).

When quoting, introduce the quotation with a signal phrase. Make sure to include the author’s name, the year of publication, the page number, but keep the citation brief—do not repeat the information.

In-text Citations: A Work with Two Authors

When citing a work with two authors, use “and”in between authors’ name in the signal phrase yet “&” between their names in parenthesis.

According to feminist researchers Raitt and Tate (1997), “It is no longer true to claim that women's responses to the war have been ignored” (p. 2).

Some feminists researchers question that “women's responses to the war have been ignored” (Raitt & Tate, 1997, p. 2).

In-text Citations: A Work with Three to Five authors

When citing a work with three to five authors, identify all authors in the signal phrase or in parenthesis.

(Harklau, Siegal, & Losey, 1999)

In subsequent citations, only use the first author's last name followed by "et al." in the signal phrase or in parentheses. (Harklau et al., 1999)

In-text Citations: A work with 6 or more authors

• When citing a work with more than 6 authors, you do NOT have to identify all authors in the signal phrase or parenthesis.

• You identify the first author and use et al. after his/her name with the date of the publication

(Carr et al., 1999) ORsignal phrase in text e.g.:

“Carr et al. (1999) conducted a meta-analysis of behavior support practices…”

Steps in the Research/Scientific Process

1. Identify socially important issue

2. Review current literature

3. Define conceptual model

4. Define specific hypothesis(es) and research question(s)

5. Define dependent variable(s)/measure

6. Identify independent variable(s)/measures

7. Select appropriate research design

8. Obtain consents 9. Collect data 10. Analyze data 11. Communicate

results Written presentation Oral presentation

Theory that drives the research. Guides our thinking and provides “rules”,

“principles” that guides the research and practice.

Structure of assumptions, principles, and rules that holds together the ideas of a broad concept.

Outlines your research

Conceptual Models (or theoretical models)

“Logical Flaws” of FBA use in public schools (Scott et al., 2005)

FBA is used mainly as a reactionary approach.opportunity is lost to utilize FBA technology to develop interventions that address minor behaviors that usually precede more serious problems.

FBA is restricted to set of procedures used by “experts”The rich supply of information from people with whom the student interacts with the most is lost.

FBA is restricted to rigorous procedures that are unrealistic for public school settings.

Disincentive for using FBA technology. Cynicism as to the practicality of FBA .

Proactive…Parsimonious…Practical FBA in schools

FBA conceptualized by Scott et al. as a proactive pre-referral routine that uses the most parsimonious procedures required to create an effective behavior support plan.

Given the time & resource constraints in schools, we must encourage schools to “work smarter” to develop capacity to implement technology to effectively support more students.

Use Practical FBA procedures to develop capacity within a school to utilize FBA technology.

Practical FBA Logic Model

School-wide Positive Behavioral Supports

80% of Students

Secondary Group

Supports10-15% of Students

Individualized Supports

5% of Students

Behavior Specialist responsible for 25 FBAs in school of 500

Personnel with “flexible” roles conduct proactive Practical FBA to expand the scope of FBA, prevent intensive problem behaviors, & decrease reliance on specialist.

A) You should educate readers about the topic and provide a clear rationale as to why the study is important and necessary based on the previous research and writing on the topic. B) Within your literature review you will present the logic or conceptual framework as to why and how your current study (topics, methods, designs) is organized the way it is.

C) Make this section compelling. Concisely explain the social importance of what you are studying.e.g., Start with a powerful statement or statistic

Literature Review Guidelines:

Quiz

Correct Quiz

Get together with 2-3 other people and use your discussion guides to “guide” your discussion.

Spend more time on the chapter reading and the Quality Indicator Article as we will be doing an activity on the other two articles later!

Discussion

Experimental & Quasi Experimental Research

Research Designs & Threats to Validity

Attempt tonight to apply the principle of teaching “less more thoroughly”….

https://www.youtube.com/watch?v=qtLnBz6lbRQ

Lecture

“Gold Standard” in Research (Moore & McCabe, 1993; Feuer, Towne, & Shavelson, 2002; Slavin, 2002).

Rooted in postpositivist paradigm.

Seeks to make causal conclusions.

Difference between experimental design and quasi-experimental design is the use of random selection of participants.

Experimental Design

Emphasis is on operationally defining the variables (dependent & independent) and the context of the research.

Dependent variable(s)- outcome variables (e.g., reading scores)

Independent variable(s)- variable that is manipulated (the intervention or practice; e.g., reading curriculum)

Context- defined clearly so replication can occur (e.g., K-3 school with 200 students, etc.)

In both experimental & quasi-experimental designs….

Experimental (or treatment group)- receives the intervention

Control group- business as usual.

For true “experimental” research, participants are randomly assigned to each group. ◦ In order for to be considered random, every

person must have an equal chance of being in either group

Experimental Group vs Control Group

Direct manipulation of an independent variable (intervention)

Experimental and quasi-experimental research involves…

Refers to whether a study is able to scientifically answer the questions it is intended to answer.

Extent to which your test (or study) measures what it intends to measure.

Validity

Changes observed in the dependent variable (outcome) are due to the effect of the independent variable (intervention)…..& not to some other unintended variables (extraneous, alternative explanations)

12 threats to internal validity (noted by Mertens, 2010)

https://www.youtube.com/watch?v=_UPUtlHDM0A

Internal Validity

1. History- events other than IV affected results?

2. Maturation- changes in participants?3. Testing- participants became “test-wise”?4. Instrumentation- difference between pre- and

posttests? 5. Statistical regression- extreme groups used?6. Differential selection- groups differed in ways

other than exposure to IV?

12 Threats to Internal Validity (think when something other than the IV affect the results of a study)

7. Experimental mortality- drop outs of study?8. Selection-maturation- was selection a

problem based on the characteristics of the sample? (e.g., participants in one group may have been older)

9. Experimental treatment diffusion- treatment & control groups shared information?

10. Compensatory rivalry by the control group

11. Compensatory equalization of treatments –extra resources given to control group?

12. Resentful demoralization of the control group

Threats to Internal Validity continued

External Validity= extent to which findings in one study can be applied to another situation.

AKA: ecological validity, generalizability

10 threats posed as questions (noted by Mertens, 2010)

External Validity (think generalizability)

1. Were the variables, context, and treatment described in sufficient detail?

2. Were multiple treatments used? Did they interfere with each other?

3. Was the Hawthorne effect (special attention as part of study affecting results) operating?

4. Was the treatment influenced by being novel or disruptive?

5. What was the influence of the individual experimenter?

10 Threats (questions) to External Validity

6. Were the participants sensitized by taking a pretest?

7. Were the participants sensitized by taking a posttest?

8. What was the influence of the type of measurement used for the dependent variable?

9. Was there an interaction of history and treatment effects?

10. What was the influence of the time period that elapsed between the treatment and administration of the dependent variable?

Questions to external validity continued

Treatment fidelity- implementer of the independent variable follows the exact procedures specified for administering the treatment(s).

Treatment Fidelity…another threat

May not be reasonable to expect participants to improve given the “strength” of the intervention.

Intervention may not have been tried long enough or delivered with adequate intensity.

Strength of treatment…another threat

R= Random assignment of subjects to conditions

X=Experimental treatment (e.g., reading curriculum)

O= Observation of the dependent variable (e.g., test or observation measure)

Coding system used for research designs

One-shot case studyX O

Threats:History, maturation, & mortality (drop out)

Other concerns using this design?No control group & No pretest to know if it

was the intervention that affected outcome. Very weak design

Single Group Designs

One-Group Pretest-Posttest DesignO X O

Threats: History, maturation, What would help control for these threats?

◦ Control group- both groups taking the tests at same time, but one not receiving the intervention

◦ But sometimes it is difficult to find a control group

Involves measurement of the dependent variable at periodic intervals.

O O O O O X O O O O O If behavior is stable in baseline (before

intervention), then change can be attributed to intervention.

Controls for several threats◦ Maturation, testing, differential selection (same persons

involved) https://www.youtube.com/watch?v=GUq_tO2BjaU

Time Series Design

Pre-test-Post-test Control Group DesignR O X OR O O

Controls for what threats? history, differential selection, mortality (pre-test can show differences in drop-outs)

Posttest-only Control Group DesignR X OR O

Controls for what threats?◦ Same as above, except for mortality (no pretest)

Experimental Designs

Single-Factor Multiple-Treatment DesignsX1= intervention 1X2= intervention 2

…….R O X1 OR O X2 OR O O

Controls for threats because participants randomly assigned to comparison groups and pre-post-tests conducted.

If worried about pretesting affecting validity

R O X OR O O R X OR O

https://www.youtube.com/watch?v=7RRVW4iO7gA

Solomon 4-group Design

Similar designs to experimental designs, except for……

Lacking…. Random assignment Can’t make a strong causal statement

https://www.youtube.com/watch?v=_WpBxERGNVw

Quasi-experimental designs

How do we know if a research study involves rigorous, systematic and objective procedures? CEC-Division for Research

Sponsored prominent researchers to author papers to propose Parameters for establishing that reported

research has been conducted with high quality (quality indicators)

Criteria for determining whether a practice has been studied sufficiently (enough high-quality research studies conducted on its effectiveness) and shown to improve student outcomes (effects are strong enough)

Graham, S. (2005). Criteria for evidence-based practice in special education [special issue]. Exceptional Children, 71.

Exceptional Children (2005) volume 71(2) Group Experimental and Quasi-

Experimental Research (Gersten, Fuchs, Comptom, Coyne, Greenwood, & Innocenti)

Single-Subject Research (Horner, Carr, Halle, McGee, Odom, & Wolery)

Correlational Research (Thompson, Diamond, McWilliam, Snyder, Snyder)

Qualitative Studies (Brantlinger, Jimenez, Klingner, Pugach, & Richardson)

Quality Indicators (QIs) for Experimental (and Quasi-Experimental) Research Describing Participants

Sufficient information about participants and interventionists, selection procedures as well as comparability across conditions

Implementation of Intervention and Description of Comparison Conditions Clear description of intervention (and comparison

conditions) with implementation fidelity assessed

Outcome Measures Use of multiple measures at appropriate times

Data Analysis Analysis techniques appropriate to questions

and unit of analysis with effect size calculated

Prepared by M. Hara (mhara@pdx.edu)

Statistics, statistics

Descriptive StatisticsWho is in your data?

sample population

Inferential StatisticsWhat your sample says about the population

sample

population

Mean, Median, Mode, Standard Deviation, Variance Tests of significance

(t-, F-Tests)

Tests of Significance Statistical analyses to determine whether a

difference is statistically significant (probability for result to occur by chance).

Yes or No answer

Alpha level (p=)◦ An established probability level which serves as the

criterion to determine whether to accept or reject the null hypothesis

◦ Common levels in education .01 .05 .10

Objectives 4.1 & 6.1

Prepared by M. Hara (mhara@pdx.edu)

Statistics, statistics

Descriptive StatisticsWho is in your data?

sample population

Inferential StatisticsWhat your sample says about the population

sample

population

Mean, Median, Mode, Standard Deviation, Variance Tests of significance

(t-, F-Tests)

T tests- used when have two groups to compare. ◦ Independent samples t- if groups are independent

Different people in each group◦ Dependent samples t-: if two sets of scores are available for the same

people (e.g., pre and post-tests of same group) Matched groups

ANOVA (analysis of variance)- when you have more than 2 groups to compare OR more than one independent variable (reports an F-statistic, which is basically a t-value squared)

ANCOVA (analysis of covariance)- ANOVA that allows for control of the influence of an IV (e.g., characteristics of people) that may vary between your groups before treatment is introduced. ◦ Post-hoc method for matching groups on variables such as age, prior

education, SES, or a measure of performance

Inferential Statistics

• Way of quantifying the difference between two groups.

• Not just was there an effect, but the magnitude of the effect.

• Many ways to calculate

• ES= [Mean of experimental group] – [Mean of control group]/Standard Deviation

• R-squared, Cohens-D

• Standard deviation is how well the mean summarizes the data

Effect Size

• Over 800 Meta-analyses

• https://www.youtube.com/watch?v=sng4p3Vsu7Y

Visible Learningby John Hattie

Implications for your practice- why you selected the article

Introduction- what was the aim of the article, what were the specific research questions

Methods- Describe:◦ Participants◦ Setting of study◦ Target behaviors- dependent variables◦ How they measured the dependent variables◦ Materials used◦ Intervention procedures used- specify◦ Were the procedures replicable?

Article Review: Use questions as your guide

Methods used for treatment integrity◦ This asks how well does the author measure the

implementation of the intervention What was the design?

◦ Experimental, quasi-experimental, single-subject (specify what type of design…multiple baseline, etc.)

Results- give quantifiable information of how the intervention worked

Discussion- issues discussed, limitations? Conclusion- did they answer the research

question? Reflection- commentary, questions

Article Review: more questions

Get together with a partner.

Practice completing an article review for one of the articles you read this week.

Note the design of the study.

In-Class Activity

A detailed explanation of the assignment is posted on the wiki

What should you be doing in your groups?◦ At this point you should have a topic and start

coming up with your framework for your research project (based on literature).

◦ Start to draft your conceptual framework, research questions & identify your dependent and independent variables

◦ You should walk away from your group time with a list of tasks to complete.

Proposal Assignment & Group Work

1. Socially Important Issue:

2. Conceptual Model/Hypothesis:

3. Research Question(s):

4. Dependent Variable:

5. Dependent Variable Measure:

6. Independent Variable:

7. Independent Variable Measure:

8. Research Design:

Self-evaluation & feedback