Washington State LASER2008 Evaluation Report

Dave WeaverKelly Jarvis

RMC Research Corporation111 SW Columbia, Suite 1200

Portland, Oregon 97201dweaver@rmccorp.com

(800) 788–1887

2

Presentation Contents

• Analysis of 2007 Science WASL Results– School-Level Analysis– Student-Level Analysis– Longitudinal Analysis

• Module-Level Assessment Results• Sentinel Site Visits Preliminary Results• Recommendation• Upcoming Activities

3

Analysis of 2007 ScienceWASL Results

• Evaluation Question– To what extent did teacher professional

development on inquiry-based science instruction contribute to improved student achievement on the Grade 5 and 8 Washington Assessment of Student Learning of science (science WASL)?

4

School-Level Analysis• Data Sources

– School-Level Science WASL Data for 2007– School-Level Mathematics WASL Data for 2006– School-Level Demographic Data– School-Level Professional Development (PD) Index

• Total PD Hours of Teaching Staff for 2-Years Per FTE

• Methods: Linear Regression Analysis – Dependent Variable:

• Percent of Students Who Met Science Standard on 2007 WASL– Independent Variable:

• School-Level 2-Year PD Index– Control Variables:

• Percent of Students Eligible for Free or Reduced Price Lunch (FRL)• Students Per Classroom Teacher

5

Student-Level Analysis• Data Sources

– 2007 Student-Level Science WASL Results for Grade 5 & 8 with Student Demographics

– 2006 Student-Level Mathematics WASL Results for Grades 4 & 7– Teacher-Level 3-Year PD Index

• Total PD Hours Teacher for 3-Years Prior to WASL

• Methods: Linear Regression Analysis– Dependent Variable

• 2007 Science WASL Scale Score

– Independent Variable• Teacher-Level 3-Year PD Index

– Control Variables• FRL• Previous Grade Mathematics WASL Scale Score

6

Student-Level Data Map

7

Longitudinal Analysis• Data Sources

– 2007 Student-Level Science WASL Results for Grade 8 with Student Demographics

– 2004 Student-Level Science WASL Results for Grades 5– School-Level 3-Year PD Index

• Total PD Hours Teacher for 3-Years Prior to WASL

• Methods: Linear Regression Analysis– Dependent Variable

• 2007 Science WASL Scale Score– Independent Variable

• School-Level 3-Year PD Index– Control Variables

• FRL• Previous Grade Mathematics WASL Scale Score

8

Longitudinal Data Map

9

Finding 1

• The number of professional development hours in which a student’s science teacher participated was a small but significant predictor of student performance on the science WASL above and beyond what could be explained by socioeconomics (FRL) and the student’s skill level (previous math WASL).

• This finding is consistent with earlier studies.

10

School-Level Analysis Results

30.1%

33.8%34.6%

37.0%

20%

22%24%

26%

28%

30%32%

34%

36%38%

40%

Adjusted Means

2-Year PD Per FTE

Per

cent

of

Gra

de 5

Stu

dent

s W

ho

Met

/Exc

eede

d Sc

ienc

e St

anda

rd

Less Than or Equal to 3Hours

Greater Than 3 andLess Than or Equal to7.5 hours

Greater Than 7.5 andLess Than or Equal to13.4 hours

Greater Than 13.4hours

Means adjusted by previous year math scores,

class size, and FRL.

11

Longitudinal Analysis Results

394.8394.3

395.5

397.1

390391392393394395396397398399400

Adjusted Means

School-Level PD Index Category (Secondary)

Gra

de 8

Sci

ence

WA

SL S

cale

Sc

ore

Greater Than 0 andLess Than or Equal To1.25

Greater Than 1.25 andLess Than 3

Greater Than or Equalto 3 and Less than 6.3

Greater than or Equalto 6.3

Means adjusted by Grade 5 Science Scale Score, FRL,

Non-Minority, Special Education, and Gifted.

12

Finding 2

• Grade 5 teachers must have participated in at least 18 hours of professional development before an impact on student achievement was measurable, and the impact of the professional development on student achievement increased as the number of professional development hours increased.

Grade 5 Multiple Regression Analysis Across Teacher-Level PD Index Categories

PD Index Interval N Beta t p Adj. R2 SE

More than 0 but less than 9 hours

5,633 -.013 -1.410 .158 .542 14.682

At least 9 hours but less than 18 hours

5863 .011 1.279 .201 .584 14.105

At least 18 hours but less than 36 hours

7589 .025 3.176 .001* .538 15.318

At least 36 hours but less than 54 hours

2740 .034 2.658 .008* .547 14.710

54 hours or more 1366 .093 5.009 <.001* .534 15.010

NOTE: Dependent Variable is the Grade 5 science WASL scale score. Control variables are the previous year math scale score and FRL Results are weighted by the number of students assessed.* denotes significance at the .05 level.

14

Finding 3• Grade 5 teachers who had participated in

science professional development served:– A larger proportion of students eligible for free

or reduced-price lunch;– A greater proportion of Hispanic and Latino

students; and– A smaller proportion of White students.

• These differences may account for the differences in student achievement.

Variable No PD Any PD

Total Number of Students 41,400 30,006

Student Achievement

Mean 2007 Science WASL Score 391.17 389.53

Mean 2006 Math WASL Score 409.26 406.71

Ethnicity Hispanic/Latino 12.3% 17.6% White (Not Hispanic Origin) 69.0% 63.5%

Special Program

Free or Reduced Price Lunch 37.1% 43.1%

Differences in Demographics and Achievement of PD Participants vs. Non-Participants

16

Module-Level Assessment of Student Content Knowledge • Evaluation Questions

– What science content knowledge gains do students make as a result of their use of the inquiry-based instructional modules?

– What is the relationship between the science PD of the teacher and the gains in student content knowledge?

17

Module-Level Assessment of Student Content Knowledge

• Assessment– Obtained Permission to use Horizon Research

Assessment– Selected Horizon Research Instruments

• Human Body from FOSS• Levers and Pulleys from FOSS• Rocks and Minerals from STC• Electric Circuits from STC

18

Module-Level Assessment of Student Content Knowledge

• Administration Plan– Recruit teachers with a range of PD levels– Data collection for each student

• Pre and post student scores• Student demographics

– Gift certificate compensation to teachers– Data collected spring 2008

• 53 classes completed both pre and post assessments• 1087 students

56 Teachers RecruitedKit PD Level Teachers

Electric Circuits High (>54 hours) 12

Medium (>18 but <54 hours) 5

Low (<18 hours) 5

None 4

Total: 26

Human Body High (>54 hours) 3

Medium (>18 but <54 hours) 4

Low (<18 hours) 3

None 7

Total: 17

Rocks and Minerals High (>54 hours) 2

Medium (>18 but <54 hours) 5

Low (<18 hours) 5

None 1

Total: 13

20

Finding 4

• Students demonstrated significant improvement between the preassessment and the postassessment on all 3 modules: on average, student scale scores increased more than 19 points.

21

Module-Level Student Assessment Results

42.7 42.537.4

62.1 62.456.5

0102030405060708090

100

Electric Circuits Human Body Rocks and Minerals

Mea

n Sc

ale

Scor

e

Pre Post

22

Finding 5• The relationship between student performance and teacher

professional development was inconclusive.– The performance of the students whose teachers participated in

fewer than 18 hours of professional development was significantly lower than the performance of students whose teachers participated in 18 or more hours professional development.

– Students whose teachers did not participate in professional development outperformed students whose teachers participated in some professional development.

• Additional data will be collected during the 2008–2009 school year.

Student Assessment Gains as a Function of Professional Development

24

Sentinel Site Selection• Schools With Significant LASER Participation• Identified 36 Schools

– 13 Schools Refused Participation– Added 11 New Schools

– Visited 34 Schools

• Defined 2 Groups of Schools Based on Science WASL Change Between 2006 and 2007 – Demonstrated Significant Positive Gains– Demonstrated Little, No, or Negative Gains

25

Sentinel Site Visits• 10 Site Visitors• Each site visit: 1 ½ to 2 days

– Principal Interview– At Least 5 Teacher Survey & Interviews– At Least 3 Classroom Observations

• Data Collection Instruments– http://www.rmccorp.com/LASERSiteVisits/

• Conducted 3 web-based training session• Round 1: Spring 2008• Results Reported As Rubric Scores

26

Finding 6• The site visitors gave the vast majority of the

schools visited very high ratings in several areas indicating that the schools served by the LASER Alliances had:– Successfully established a core sequence of inquiry-

based instructional modules across all of the grade levels, and

– Had materials support system that delivered modules to teachers on time and ready to use.

27

Finding 7

• There were several significant differences between schools whose students demonstrated increased performance on the science WASL compared to schools whose students did not.

28

Schools with Grade 5 Students• Schools whose Grade 5 students’ science

WASL scores increased had significantly greater:– Participation in professional learning communities– Time during the workday for school-based

professional development– Evidence of positive parent and community support– Evidence of teachers integrating literacy with science

29

Schools with Grade 8 Students• Schools whose Grade 8 students’ science WASL

scores increased had:– Time during the work day for school-based professional

development– Greater participation in professional development that

involved examining student work– Lessons that more often engaged students in

discussion based on scientific evidence– Lessons whose content was more significant, accurate

and worthwhile

30

Schools with Grade 8 Students (cont.)• Schools whose Grade 8 students’ science WASL scores

increased had:– Lessons that more often built on prior ideas and

experiences– Lessons whose content was more closely connected to

the lesson activities– Lessons that were effective at fostering student

understanding– Lessons that included mechanisms to monitor students’

understanding of science content– Lessons that helped students make sense of scientific

concepts– Lessons that routinely provided opportunities for students to

reflect on their new understanding

31

Finding 8• The schools that demonstrated an increase in the

percentage of students who met the science standards had teachers with significantly fewer years of education experience than the schools with no gains.

• This difference was more pronounced in the schools that served Grade 8 students.

– Testing for demographic differences between the schools that improved their science WASL scores and schools that did not could not account for the findings. Factors examined include:

• School size• Ethnic/racial composition of the student population• Program membership of the students• Teacher education and experience

32

Conclusion• The infrastructure to support the use of a core

curriculum of inquiry-based science instructional modules is in place and is functioning adequately in the schools visited.

• Although these conditions are necessary for the implementation of inquiry-based science instruction, they are not sufficient to raise student achievement as measured by the science WASL.

33

Recommendation 1• Ensure that the professional development on

research-based instructional practices is consistent and explicit across all of the LASER Alliances– Help teachers understand the elements of effective

science instruction and use the modules as a means of carrying out the element with their students.

34

Elements of Effective Science Instruction• Intrinsic Motivation

– Relevant, interest to student, resolution of dissonance• Eliciting Student’s Prior Knowledge

– Articulating what they already believe about a concept• Intellectual Engagement

– Students do the thinking• Use of Evidence

– Discourse: Justifications and explanations based on evidence• Sense-Making

– Closure, meta-cognition, reflection

Banilower, E., Cohen, K., Pasley, J., & Weiss, I. (2008). Effective science instruction: What does research tell us? Portsmouth, NH: RMC Research Corporation, Center on Instruction.

35

Recommendation 2

• Increase support for school-based professional development that helps teachers:– Assume accountability for student learning

that results from the use of the modules, and

– Collaboratively implement the elements of effective science instruction.

– Ample structure and leadership for success

36

Evaluation Activities for 2008-09• Sentinel Site Visits (30 to 35)• Student Content Assessments

– Fall, Winter, & Spring– 7 Instructional Modules

• Analysis of 2008 WASL Data• Online Surveys

– School Principal Survey– Alliance Directors

• PD Provider Online Surveys (3)

Rocks & Minerals

Human Body

Electric Circuits

Motion & Design

Levers & Pulleys

Environments

Variables

37

ASK-IT Schools Show What Science Engaged Schools Can Do

ASK IT schools had teachers who: Used formative assessment to

determine student understanding of key concepts

Met in professional learning communities to inform their teaching based on student data

Increased their science teaching time so most teachers completed each unit.

Environments Proficiencies 5th Grade Matched Pair 1

-2

-1.5

-1

-0.5

0

Pre Post

Treatment

Control

Human Body Overall Proficiencies

-2.5

-2

-1.5

-1

-0.5

0

Pre Post

Treatment

Control

Magnetism & Electricity Overall Proficiencies

-2

-1.5

-1

-0.5

0

0.5

Pre Post

Treatment

Control

38

Questions ???Dave Weaver

RMC Research Corporation111 SW Columbia, Suite 1200

Portland, Oregon 97201dweaver@rmccorp.com

(800) 788–1887