rti professional learning webinar series: using research to select and design effective...

RTI Professional Learning Webinar Series: Using Research to Select and Design Effective Interventions – Session 4October 15, 2009

Our session will start momentarily. While you are waiting, please do the following:

Enter/edit your profile information by going to:•Tools - Preferences - My Profile…•Fill out the info on the “identity” tab and click “OK”•To view the profile of another use, hover your mouse over his or her name in the participants window Configure your microphone and speakers by going to:•Tools – audio – audio setup wizard

Confirm your connection speed by going to:•Tools – preferences – connection speed

RTI Professional Learning Webinar Series:

Using Research to Select and Design

Effective Interventions

.

2

Session #4 10/15/09 3-4:30 pmCritical Review of Research Studies

on Interventions

Session #4 10/15/09 3-4:30 pmCritical Review of Research Studies

on Interventions

Counting ALL Participants After you enter your profile information to

join the webinar as usual:

1. Go to the Chat Window

2. Type your district name and school name or organization name or “parent”

3. Type the names of every person in attendance with you

4. Send it to “This Room”

3

Who are we? Teachers Administrators Counselors or Social Workers County Office or DOE Parents PK-5, 6-8, 9-12, post-secondary Statewide Organizations Other?

4

Welcome and Introductions

John Wight, GaDOE

Dr. Kimberly Anderson, REL-SE at SERVE Center at UNCG

5

6

REGIONAL EDUCATION LABORATORY- SOUTHEAST (REL-SE)

Executive Director Dr. Ludwig van Broekhuizen

Toll Free: 800-755-3277 www.serve.org

Operated by the SERVE Center at UNCG

7

Serving Alabama, Florida, Georgia, Mississippi, North Carolina, and South Carolina

Georgia liaison: Dr. Kim [email protected]

404-657-6174

8

The Regional Educational Laboratory (REL) System

REL-SE MISSION

To serve the educational needs of the

Southeast, using applied research,

development, dissemination, and

training and technical assistance, to

bring the latest and best research and

proven practices into school

improvement efforts.

9

REL-SE Services

1. Outreach and dissemination of research, evaluation, and policy information

2. Technical assistance to SEAs and LEAs

3. “Issues & Answers” publications

4. Randomized Controlled Trial experimental studies on interventions of relevance to our region

• AMSTI/Alabama and K-PAVE/Mississippi

10

Series Framework: EBDM Cycle

11

12

WEBINAR SERIES OVERVIEW

SESSION 1 SESSION 2 SESSION 3 SESSION 4

Webinar Series Overviewsessions 5 & 6 session 7 session 8 session 9

13

TODAY’S OBJECTIVES 1. Reach back to Session 3

2. Understand how to assess the rigor and findings of studies on interventions

3. Practice using the Critical Reading Protocol for Studies About Interventions

4. Debrief the Protocol process and discuss the relevance of this type of process for RTI decision making

5. Preview session #4

14

Types of Research Methods handout posted on GaDOE website

Carnegie Learning 2001 & Witzel Multisensory Algebra studies

Critical Reading Protocol For Studies About Interventions handout

Here we go!

Objective #1

Reach back to Session #3:

Types of Research Methods

15

We discussed 5 types of research methods. What do you remember?

What thoughts/questions remain?

Descriptive Correlational Quasi-Experimental Experimental Meta-Analysis

2 min. ChatFacilitators lead their groups

Individuals utilize chat box

16

Quick Poll 1: RecallStudies using which method can make

claims that the intervention studied caused changes in student achievement?

A. Descriptive B. Correlational C. Quasi-Experimental D. Experimental E. Meta-Analysis

17

Quick Poll 2: RecallStudies using which method can discover

relationships between variables (e.g., a particular instructional strategy and academic achievement)?


18

Quick Poll 3: ID This Research Design Citation: Bennett, J., Lubben, F., & Hogarth, S. (2006).

Bringing science to life: A synthesis of the research evidence on the effect of context-based and STS approaches to science teaching. Science Education, 10, 347-370.

Research Question: What is the average effect across studies examining the impact of context-based and STS approaches to science teaching?

Findings: The 17 studies suggested that context-based and STS approaches to science teaching improved attitudes towards science as compared to control groups.

19

What type of research design do you think the researchers used? Why?


20

Quick Poll 4: ID This Research Design Citation: Rainey, L., Harman, P., & McColskey, W. (1996). A

report of the integrated science student impact study at the University of Alabama. Greensboro, NC: SERVE Center

Research Question: Do students in the Integrated Science Program (“ISP”) perform significantly better on performance assessments as compared to students not enrolled in the ISP?

Methods: Students in the ISP and not in the ISP were matched based on race, gender, and academic ability.

Findings: In a test of science process skills, students enrolled in the ISP performed significantly better than students not in the ISP.

21

What type of research design do you think the researchers used? Why?


22

Objective #2

Put our knowledge of research methods together with other key information to assess the rigor and findings of studies on interventions

23

Critical Reading Protocol for Studies About Interventions

Looks at key elements of a research study’s design and results in order to determine its quality and rigor.

This protocol can be used in the process of compiling information about interventions and strategies, and determining “what works.”

This protocol is in the GaDOE RTI manual.

24


See Protocol handout

1. Authorship: Who is the author? Is it the vendor or a third party? Who funded the study? Is there an obvious bias?

2. Sample: What was the sample? How closely do the participants in the study mirror your population?

25


3. Research design: What kind of design did the researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed?

*use your Types of Research Methods handout

26


4. Results: What kinds of outcomes were reported on? Are the results statistically significant? Practically significant?

5. If you were to implement this intervention what outcomes could you reasonably expect to achieve?

27

Objective #3

Practice using the Critical Reading Protocol for Studies About Interventions

Participants read sample program evaluation publications and utilize Protocol to determine the quality and rigor of the study

Then, whole group discussion

28

Sample Publication #1: Carnegie Learning 2001 Cognitive Tutor

1 min. introduction to this publication

10 min. read and answer Protocol questions (in small facilitated groups and/or in chat box)

10 min. whole group discussion: What did we learn about this research study and its application for our RTI decision making?

29

POLL: Protocol Question #1 Authorship: Who is the author? Is it the vendor or

a third party? Who funded the study? Is there an obvious bias?

A.The author is/has a relationship to the vendor, and as such may be biased

B.The author is a third party, and as such should be neutral

C.Cannot determine

D.We have a question about this…?

30

POLL: Protocol Question #2

Sample: What was the sample? How closely do the participants in the study mirror your population?

A.The participants closely mirror our population

B.The participants are not similar to our population

C.Cannot determine


31

POLL: Protocol Question #3Research design: What kind of design did the

researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed?

A. The design was quasi-experimental

B. The design was correlational

C. The design was experimental

D. Cannot determine or we have a question about this…?

32


Results part 1: What kinds of outcomes were reported on?

A. Results were Texas End of Course Exam

B. Results were local assessments

C. Cannot determine

D. We have a question about this…?

33

POLL: Protocol Question #4Results part 2: Are the results statistically

significant? Practically significant?

A. Results were statistically significant

B. Results were statistically AND practically significant (in our view, the difference between scores of C.T. students and comparison group students would justify C.T.’s cost, for us)

C. Cannot determine


34


If you were to implement this intervention what outcomes could you reasonably expect to achieve? Why?

A. We would expect to achieve similar results

B. We would not expect to achieve similar results

C. Cannot determine


35

Sample Publication #2: Witzel 2005 Multisensory Algebra

1 min. introduction to this publication

10 min. read and answer Protocol questions (in small facilitated groups and/or in chat box)

10 min. whole group discussion: What did we learn about this research study and its application for our RTI decision making?

36

POLL: Protocol Question #1 Authorship: Who is the author? Is it the vendor or

a third party? Who funded the study? Is there an obvious bias?

A.The author is/has a relationship to the vendor, and as such may be biased

B.The author is a third party, and as such should be neutral

C.Cannot determine


37


Sample: What was the sample? How closely do the participants in the study mirror your population?

A.The participants closely mirror our population

B.The participants are not similar to our population

C.Cannot determine


38

POLL: Protocol Question #3Research design: What kind of design did the

researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed?

A. The design was quasi-experimental

B. The design was experimental

C. The design was descriptive

D. Cannot determine or we have a question about this…?

39


Results part 1: What kinds of outcomes were reported on?

A. Results were SC’s “PACT” test scores

B. Results were scores on a local assessment designed specifically for use in the study

C. Cannot determine


40

POLL: Protocol Question #4Results part 2: Are the results statistically

significant? Practically significant?

A. Results were statistically significant

B. Results were statistically AND practically significant (in our view, the amount of difference between scores of M.A. students and comparison would warrant M.A.’s cost, for us)

C. Cannot determine


41


If you were to implement this intervention what outcomes could you reasonably expect to achieve?

A. We would expect to achieve similar results

B. We would not expect to achieve similar results

C. Cannot determine


42

OBJECTIVE #4

Debrief the Critical Reading Protocol process, and

Discuss the relevance of this type of process as we consider purchasing/creating interventions

43

Debrief & Discuss

1. What do we think about the Protocol? 2. How does this Protocol process compare to how we

normally think about the research behind interventions we are considering?

3. What is the value (or not) of this type of process as we consider purchasing/creating interventions?

3 min. ChatFacilitators lead their groups

Individuals utilize chat box

44

Objective #5Preview session #4

October 28, 2009 3-4:30pm

State of Research on RTI Math: What do we know? What do we not know?

Dr. Sybilla Beckmann, UGA

45

Webinar Series Overviewsessions 5 & 6 session 7 session 8 session 9

46

http://ies.ed.gov/ncee/wwc/publications/practiceguides/

47

Accessing session #5 materials and webinar

John Wight

48

Thank you for being here

today!

Dr. Kim Anderson – [email protected] – 404-657-6174

The SERVE Center/REL-SE– 800-755-3277– www.serve.org

John Wight, GaDOE [email protected] 404-656-0478

49

rti professional learning webinar series: using research to select and design effective...

Documents