From Z to A: Setting the STAAR Cut Points from End to Beginning

Gloria Zyskowski, TEA

Kimberly O’Malley, Pearson

Session Overview

• STAAR Assessment Program

• Alignment of Content Standards

• Standard Setting 101

• Alignment of Performance Standards

• STAAR Standard-Setting Process

STAAR Assessment Program

• Emphasis on college and career readiness

• Aligned system of assessments

• Increased rigor Focus—on the curriculum (TEKS) that are most

critical to assess Clarity—regarding what will be assessed and

how the assessed content standards are preparing students for their next step

Depth—is emphasized over breadth in assessing student expectations

College Readiness Goals

• To become one of the top 10 states in the nation in terms of college readiness by the 2019–2020 school year

• High school graduates prepared for postsecondary opportunities

Texas Definition of College Readiness

Texas Education Code (TEC) §39.024:

• “…the level of preparation a student must attain in English language arts and mathematics courses to enroll and succeed, without remediation, in an entry-level general education course for credit in that same content area.” at four-year college and universities at institutions that offer associate degrees and

certificates

Alignment of Content Standards

• Start with college readiness

• Align to high school standards

• Vertically align down to elementary school

Texas College and Career Readiness Standards (CCRS)

• Legislation required TEA and THECB to establish vertical teams to develop standards in English language arts Mathematics Science Social studies

• Approved in January 2008

• Have been compared with other national standards

Aligned to High School Standards

• Gap analysis and alignment study

• TEKS refinements and revisions

• Identification of critical skills in English III Algebra II

Vertically Aligned to Elementary

• Map critical skills back to third grade

• Focus on essential skills for current grade and important to be prepared for the next grade

• Readiness vs. supporting student expectations Readiness student expectation emphasized in

the STAAR assessments

College and Career Readiness Standards (CCRS):

• First integrated into the EOC assessment at the end of the sequence (English III and Algebra II)

• Then mapped backwards across courses and grades to form a content-aligned system

Standard Setting 101

• Different types of standards Content Standards Performance Standards Accountability Standards

• Process of determining the level of knowledge and skill students need to demonstrate to be classified into the various performance levels.

• Involves obtaining recommendations from those who are knowledgeable of the content/skills to be assessed the test population the ways in which test scores are used

Standard Setting 101

• Combines content, data, and policy expertise

Alignment of Performance Standards

• Start with college readiness

• Align to high school standards

• Vertically align down to elementary school

College Readiness Standards

TEC § 39.024 mandates:

• The setting of college readiness performance standards for English III and Algebra II Informed by research studies Research studies conducted prior to initial standard

setting and at least every three years thereafter

• The comparison of standards to national and international “college readiness” assessments and success in military service or the workforce

• The conducting feasibility studies for science and social studies EOC assessments

Aligned to High School Standards

• Performance standards in lower-level EOC assessments informed by studies that relate student performance in one course with student performance in the next course Algebra I standard aligned with Algebra II English II standard aligned with English III English I standard aligned with English II

Vertically Aligned to Elementary

• Performance standards in middle and elementary school also informed by studies, relating student performance in one grade/course with student performance in the next grade/course Grade 8 math standard aligned with Algebra I Grade 8 reading standard aligned with English I Standards in grades 3–7 reading and

mathematics aligned with next grade/course

Performance standards based on empirical evidence from student performance

• across courses and grades

• on external assessments

STAAR Standard-Setting ProcessEight steps for setting STAAR performance standards:

1. Conduct validity and linking studies

2. Develop performance labels and policy definitions

3. Develop grade/course specific performance level descriptors

4. Standard-setting committee

5. Policy review committee

6. Approval of performance standards

7. Implementation of performance standards

8. Review of performance standards

1. Conduct Validity and Linking Studies

Studies to inform the college readiness standard:

• Follow students from high school to college

• Compare EOC performance and other test performance SAT ACT ACCUPLACER THEA

• College students take STAAR EOC assessments

1. Conduct Validity and Linking Studies

Studies to inform standards for STAAR EOC assessments:

• Relationship between test performance in the same content area

• Relationship between test and course performance

• Comparison to TAKS

• Comparison with NAEP

1. Conduct Validity and Linking Studies

Studies to inform standards for STAAR 3–8 assessments:

• Relationship between test performance in different grades in the same content area

• Relationship between grade 8 and high school

• Comparison with TAKS

• Comparison with NAEP

• Vertical Scale

Why Studies for Setting Performance Standards?

• Texas goal is to be in the Top 10 in terms of college readiness by 2019–2020.

• Comparisons of Texas standards with national and international standards are important in meeting this goal.

• Studies will be conducted at least every three years to update standards and monitor progress.

Example: Algebra II and SAT

SAT Mathematics (not SAT Subject Test)

• Arithmetic operations

• Algebra

• Geometry

• Statistics

• Probability

• Etc…

STAAR Algebra II

• Quadratic Functions and Relations

• Square Root Functions

• Rational Functions

• Exponential and Logarithmic Functions

• Etc…

Content Overlap

• SAT (general) is broader, but STAAR is deeper

Example: Algebra II and SAT

Timing

• STAAR Algebra II Spring of junior year Spring of senior year

• SAT (General Test) Spring of junior year Fall of senior year

• Students take the assessments around the same time of year. However…

Example: Algebra II and SATData are not available on the same timeline

• SAT data are typically available to the state after students graduate

• Initial standards will be set using data from tests not used for students’ graduation requirements.

• Standards review will incorporate data from tests used for students’ graduation requirements.

STAAR SAT

Initial Standard Setting

Seniors 2011

Juniors 2010 (Algebra II FT)

Graduating seniors 2011

Standards Review

Juniors 2014 Graduating seniors 2015

Additional Considerations

• SAT—no college ready benchmark for comparison ACT does have a college ready benchmark and it will

be used in comparisons

• Student motivation now and then

• Algebra II is a newly-tested content area Required course as part of 4x4 Required assessment for graduation (recommended

and distinguished achievement programs) Instruction will improve Curriculum may be changed/updated

• SAT is typically taken by college-bound students

2. Performance Labels and Policy Definitions

• TEA and THECB convened a committee September 30–October 1, 2010

• 26 committee members represented by diverse stakeholders in: Public education Higher education Business community Legislature

Committee Charges

• Assume that the state assessment system will be implemented under current federal and state statute, both of which require a minimum of three performance levels.

• Reach consensus on recommendations for the names of the performance labels (categories of performance) for student achievement on the assessments (general, modified and alternate).

• Make recommendations for key words/phrases to be used in drafting the policy definitions that will define student performance within each category.

Performance Label & Policy Definition

TAKS Example

• Performance Label Met the Standard

• Policy Definition Satisfactory academic achievement, students

performed at a level that was at or somewhat above the state passing standard, students demonstrated a sufficient understanding of the knowledge and skills measured at this grade.

Committee Process

Step 1: Brainstorm key words/phrases to be used in developing the policy definitions

Step 2: Share recommendations for key words/phrases

Step 3: Reach consensus on recommendations for key words/phrases to be used in developing the policy definitions

Step 4: Brainstorm performance labels for each of the performance categories

Step 5: Share recommendations for performance labels

Step 6: Reach consensus on recommendations for performance labels

Final labels and definitions to be approved by the

commissioner of education by December 31, 2010.

3. Specific Performance Level Descriptors

• Committees to meet in 2011

• Committees to primarily include educators from both public education and higher education

• Translate the policy definitions into grade/course and content specific descriptions

3. Specific Performance Level Descriptors

TAKS Exit-level ELA Example

Met the StandardSatisfactory performance; at or above state

passing standard; a sufficient understanding of the ELA TEKS

curriculum

Students Who Met the Standard1. Use appropriate strategies to

comprehend both fiction and nonfiction most of the time.

2. Understand, for the most part, how literary techniques, such as symbolism, foreshadowing, and flashback, contribute to the development of a story.

3. Have some awareness of personal strengths and weaknesses as a writer.

4. Etc…

Performance Label

Policy Definition

Specific Performance Level Descriptors (PLDs)

4. Standard-Setting Committee• STAAR EOC: February 2012

• STAAR 3–8: October 2012

• Committee members to represent broad perspectives including: Educators (from both public and higher

education) Policy experts (including business

representatives)Dual expertise in education and policy

4. Standard-Setting Committee

• Committee will follow a research-based standard-setting process

• Cut scores will be informed by Test content (item difficulty, required skills) Alignment within content area External study results (e.g. SAT, ACT) Linking studies (e.g. Algebra I to Algebra II) Student performance (estimated % passing) Expert judgment

• Cut score will be set starting with highest grade/course, with lower grade/course vertically aligned to higher grade/course

5. Policy Review Committee

• STAAR EOC: March 2012

• STAAR 3–8: November 2012

• Committee of higher and public education policy experts from prior committee and additional policy experts

• Review recommended cut scores across the entire STAAR program – all grades/courses and content areas

6. Approve Performance Standards

• STAAR EOC: March 2012

• STAAR 3–8: December 2012

• College readiness performance standards approved by commissioner of education and commissioner of higher education

• All other performance standards approved by commissioner of education

7. Implement Performance Standards

• STAAR EOC: May 2012 (first high stakes administration)

• STAAR 3–8: Late fall 2012 or early 2013

• New standards are expected to be used in state and federal accountability systems starting in 2013

8. Review Performance Standards

• Legislative requirement to review performance standards at least once every three years

• First review in 2013

• Consider additional data from research studies Longitudinal data (follow cohorts of TX students

from EOC to college and careers) Substitute tests (AP, IB, SAT subject) Military service Workforce College readiness for science and social studies

(depending on findings of feasibility studies)

Summary

The STAAR Assessment Program will feature:

• Content and performance standards that are an aligned system from grade 3 to college and career readiness to prepare all students for postsecondary opportunities

• Performance standards that are not set in isolation but informed by data from research studies that include comparisons with national and international assessments

• A multi-step research-based standard-setting process that includes diverse stakeholders from higher and public education and involves frequent review of standards

Additional information can be found in the transition report submitted to the legislature on December 1.

Questions?

Session Code

<Insert Session Code here>