Policy Definitions, Achievement Level Descriptors, Science Achievement Standards (Cut Scores)

for General Education and for Alternate Assessment

1

Overview Kathleen Vanderwall, OAIS

External Evaluation Mary Seburn, Educational Policy Improvement Center

Implementing 2009 Content Standards for Instructional Improvement

Monument School District, Rachel Aazzerah North Clackamas School District, Meagan Sternberg

2

3

Policy Definitions are general statements that describe rigor across grade levels and content areas and provide an overarching definition for each achievement level.  They are used for all subjects.

Achievement Level Descriptors (ALDs) explain a student’s knowledge and skills at a specific Achievement Level by grade group and subject and are a key element in the establishment of cut scores.

ODE focused the Achievement Standards Verification panelists’ attention by using outside references with data on national and international benchmarks.

4

Specify the knowledge, skills and abilities a student needs to know in order to be classified as:

• Does Not Yet Meet• Nearly Meets• Meets• Exceeds

5

Scores on Oregon’s scale. Describe what students should be able to do in each achievement level.

Participants actively discuss their expectations of the “target student” in each achievement level using a booklet of 70 Oregon test items ordered from easiest to most difficult.

6

May-September 2010 Moving Science Forward Professional Development

2009, 2010 October Statewide In-Service- new standards and assessments presentations September 2009-present Science Monthly Updates,

AA Updates, Superintendents Pipeline February 2011 Draft 2009 Test Specifications and

Blueprint Documents posted to web January-May 2011 Announcements regarding

reporting, assessments, scoring guides and Science Standard Verification

August-September 2011 Survey October 2011 – May 2012 Communication Strategy

7

General Policy Definitions (adopted March 2010)

Science Policy Definitions(provide consistency across grades-approved by Science Panel January 2011)

Science Achievement Level Descriptors for grade 5, 8, HS

(adopted May 2011)

Final Achievement Level Descriptors and Achievement Standards linked to outside references will be available to inform partners.

8

Dr. Mary Seburn Director Research, Design and Analytics DivisionEducational Policy Improvement Center

9

• Modified Bookmarking method • Comparative data from NAEP, PISA, OUS, other states• Nearly Meets, Meets, and Exceeds • Grades 5, 8, and high school Science• 24 stakeholders • Three days

• Recommended increased achievement standards for all grade levels• Except for “Nearly Meets”; current standards confirmed

10

11

Note: No changes were recommended to the Nearly Meets Achievement Standard.

12

The Educational Policy Improvement Center (EPIC) •Documented evidence of technical adequacy

• Procedural validity, as observed• Standards for Educational and Psychological Testing• NCLB Peer Review• Best practice

•Materials review•Observations•Interviews •Evaluations

Twenty-one evaluation elements total, all met:

13

Panelist Selection Panelist and Facilitator

Training Methodology Process Documentation

“The training materials were helpful.”

“I am confident I understand my role in the standards verification process.”

“The training clearly identified the goals for the standards verification procedure.”

14

“I understood how to place my bookmarks.”

“Overall, I am satisfied with my group's final bookmarks.”

“I am confident that the bookmark procedure used produced valid Achievement Standards.”

“The bookmark procedure was well described.”

15

16

• “Steps in the process were very good, group and individual work time allowed for flexibility and allowed us to come to consensus.”

• “I would love to do this again. I can’t think of anything they could do better. Very successful.”

• “I really believe in the process, it is difficult, but it’s why democracy works.”

Rachel Aazzerah7-12 Science/Math TeacherProfessional Learning Community Leader/Data Driven Decision Making Coordinator

Monument School District

17

Interaction and Change Life Science (6.2L.2)

Explain how individual organisms and populations in an ecosystem interact and how changes in populations are related to organisms.  

18

Interaction and Change Earth and Space Science (7.2E.3)

Evaluate natural processes and human activities that affect global environmental change and suggest and evaluate possible solutions to problems. 

19

Structure and Function Life Science (H.1L.3)

Explain and apply laws of heredity and their relationship to the structure and function of DNA. 

20

Meagan SternbergK-12 Science Instructional Coach North Clackamas School District

21

North Clackamas School District is in the middle of reviewing and realigning science courses with a focus on making sure students are exposed to Engineering Design content, practices and work samples at each grade level.

This is challenging in some science courses. However, we feel it is critical work with the

new focus on Engineering and Technology at the national level.

22

Happy Valley and Rock Creek Middle Schools have an elective called “Engineer It!”.

At HVMS, almost all students take this class.

23

Raising the Science Achievement Standards increases the rigor and importance of teaching and learning science in Oregon schools.

24

State Board adoption of recommended achievement standards and achievement level descriptors

Peer Review Submission- December 2011

25

Thank you!

Questions?

26

Dianna Carrizales-Engelmann

Brad Lenhardt

Office of Student Learning and Partnerships

Daniel Farley

University of Oregon

27

28

Population District Uses Diploma Options

Post-Secondary Implications

Students with the most significant cognitive disabilities

AYP reporting (up to 1%) and district achievement reporting

•Extended Diploma

•Modified Diploma

Supported living facilities to post-secondary education and vocational careers

29

General Assessment Alternate Assessment

Population All students Students with IEPs

Participation ‘10

~296,090 ~5,776

Decision Default decision IEP team decision

Development Based on grade level content

Based on grade level content reduced in depth, breadth, and complexity

Administration

Primarily Online Paper and pencil, individually administered

Assessment areas

Reading, Writing*, Mathematics, Science

Reading, Writing*, Mathematics, Science

Stakes Used to demonstrate Essential skills towards a regular diploma

Can be used to demonstrate Essential Skills in Reading toward the Modified Diploma or can lead to the Extended Diploma 30

General Assessment

Alternate Assessment

Participation (2010)

123,036 1736

Achievement Standards

RIT scores Scaled Scores

ALDs Developed by staff approved by expert panelists

Developed by staff approved by expert panelists

Standard Setting process

Verification of staff- recommended cutscores

Panel-recommended cutscores based on knowledge of population and assessment

31

University of Oregon Behavioral Research and Teaching Dr. Gerald Tindal

Bookmarking process 13 educators representing 9 counties SPED and content expertise 3 rounds Review of impact data Recommendations: Both increases and decreases

related to refinements to the new content standards

34

Achievement Level

CurrentScience Cut Scores

ProposedScience Cut Scores

Change to Cut Score(+/- Resulting from

Recommended Minus Current Cut Score)

Grade 5 Grade 8 HS Grade 5 Grade 8 HS Grade 5 Grade 8 HSNearly Meets 100 89 91 100 95 98 0 +6 +7

Meets 108 96 107 107 101 103 -1 +5 -4Exceed

s 116 114 114 114 113 109 -2 -1 -5

35

Cut Scores Impact Data

GradeNearly Meets Meets Exceeds

Does Not Yet Meet

Nearly Meets Meets Exceeds

Meets /Exceeds 2011

Meets /Exceeds 2010

5 100 107 114 28% 11% 37% 23% 60% 58.10%8 95 101 113 20% 12% 53% 15% 68% 80.20%

HS 98 103 109 36% 21% 31% 12% 43% 33.60%

36

Does Not Yet Meet Nearly Meets Meets Exceeds

Science Policy Definitions(Apply to all grade groups)

Performance indicates that the student has a limited understanding of specific academic concepts linked to grade-level content that has been reduced in depth, breadth, and complexity.Demonstration of Science Knowledge and Skill in familiar situations with significant limitations.

Performance indicates an inconsistent understanding of specific academic concepts linked to grade-level content that has been reduced in depth, breadth and complexity.Demonstration of Science Knowledge and Skill in familiar situations with some success.

Performance indicates an understanding of specific academic concepts linked to grade-level content that has been reduced in depth, breadth, and complexity.Demonstration of Science Knowledge and Skill in familiar situations with success.

Performance indicates a consistent understanding of the academic concepts linked to grade-level content that has been reduced in depth, breadth, and complexity.Demonstration of Science Knowledge and Skill in familiar situations with consistent success and in unfamiliar situations with some success.

37

Rigor of process Appropriateness of panelists Quality of personnel/oversight Thoroughness of training Validity of assessment Panelist Confidence Assessment data

38

Thank you!

Questions?

39

Does Not yet Meet: Students do not demonstrate mastery of grade-level knowledge and skills required for proficiency.

Nearly Meets: Students demonstrate partial mastery of grade-level knowledge and skills required for proficiency.

Meets: Students demonstrate mastery of grade-level knowledge and skills required for proficiency.

Exceeds: Students demonstrate mastery of grade-level knowledge and skills exceeding the requirement for proficiency.

40

Does Not Yet Meet: Students demonstrate limited mastery of science, scientific inquiry and engineering design knowledge and skills, through the application of basic concepts with occasional success.

Nearly Meets: Students demonstrate partial mastery of science, scientific inquiry and engineering design knowledge and skills, through the application of basic concepts with regular success.

Meets: Students demonstrate mastery of science, scientific inquiry and engineering design knowledge and skills, through the interpretation and application of grade-level concepts with consistent success.

Exceeds: Students demonstrate mastery of science, scientific inquiry and engineering design knowledge and skills, through the interpretation and application of advanced concepts with consistent success.

41

Proficient: Proficient: A student who is proficient has solid academic performance.

Mastery: Mastery: A student who has mastered a subject is able to clearly explain aspects of the subject to fellow students.

42