Symposium: Assessment, Accountability, Instruction, and Learning in Urban Districts

 • Research funded by the Joyce Foundation and Helen Bader Foundation• Center for Systemic Reform in the Milwaukee Public Schools (SSR-MPS)• Wisconsin Center for Education Research (WCER)• University of Wisconsin-Madison• 3 papers:

- William Clune, et al. Milwaukee Middle School Proficiencies- Robert Meyer Value-added & School Performance- Norman Webb Assessment Literacy

• All focus on Milwaukee district• 3 Commentators:

- Warren Chapman Joyce Foundation- Deborah Lindsey Milwaukee Public Schools- Andrew Porter WCER

• Papers on website: http://www.wcer.wisc.edu/mps

The Milwaukee Middle School Proficiencies: Systemic school reform through high stakes

assessments and a network of schools • William H. Clune, with Sarah Mason, Cecilia Pohs, Chris Thiel, and Paula A. White

• Paper prepared for the annual meeting of the AERA, New Orleans, April 2, 2002

• Paper on website: http://www.wcer.wisc.edu/mps

Overview of Points

 

• What are the Proficiencies?

• Research Methods

• Impact on instruction and achievement

• Centralized/ decentralized policy & implementation

• Value of standardized and performance assessments

• Importance of evaluation

The Milwaukee Middle School Proficiencies: What are they?

•  Promotion requirements from 8th grade, year 1999-2000 through 2001-02

• 4 Areas: Communications, Mathematics, Science, Research Paper

• Multiple assessments, multiple opportunities to pass, grades 6-8

• Purpose: assure readiness for high school (not just pass or fail)

Assessment Types

• Traditional standardized tests (e.g., State of Wisconsin)

• On-demand district performance assessments• Performance assessments embedded in instruction (tasks, scoring rubrics)

• Most weight put on embedded assessments

• Can pass w/o "proficient" on State test• Alternative completion project (all 4 proficiencies) (Adopt-a-City)

Research Methods

• 17 interviews in Spring 2000

• 11 school sites (range of reputed success)

• 9 interviews with learning coordinators, 3 principals

• 5 with respondents from school network ("Middle School Collaborative")

• Taped, transcribed, coded with NUD*IST 4

• Names & schools not disclosed in paper

Findings

• Strong impact on instruction, learning, school organization

• Unclear impact on student achievement, weak positive evidence

• Centralized/ decentralized implementation/ policy formation

 Instruction• Provided a focus for teaching (9 respondents)• Students re-do work w teachers (4)• Increased hands-on work (4)• Aligned with curriculum (9) Learning• Students:

- take greater responsibility (6)- improved in writing (5)- improved in reading (3)- improved in math/ science (6)- improved on standardized tests (2)

 School organization• Schools did major re-organization (4)• Special proficiency classes helpful (4)• Summer school programs helpful (3)

Finding 1: Strong Impact on Instruction, Learning,School Organization

Finding 2: Unclear Impact on Student Achievement, weak positive evidence

 

• Intended to improve achievement (proficiency beyond assessments)

• Performance assessments not statistically reliable

• Annual 8th grade testing poor method of evaluation

• Wisconsin 8th grade scores rose for two years then fell

• Technical problems: change of test date, test forms

• District study: increase in high school grades and 9th grade promotion

• Also more transitional ("8-T") students in grades 8-9

• Much better: value-added from annual standardized tests (Meyer paper)

Finding 3: Centralized/ decentralized implementation/ policy formation 

In general

• Unfolding requirements and implementation (incremental "roll out")• Much formative activity by District staff, Middle School Collaborative, Learning Coordinators, Lead Principals The Middle School Collaborative (network of Middle School Principals) • Had independent funding (Danforth)• Summer retreat for guiding vision ("all children can succeed")• Prevented repeal of Proficiencies• Successfully advocated alternative completion mechanism (Adopt-a-City)• Advocated fewer midstream policy changes

Discussion/ Significance of Findings 

• Policy strength from the top and bottom

• Tradeoff of measurement reliability and instructional validity 

Discussion point 1: Policy strength from thetop and bottom

 • Good fit with Porter et al (1988) framework (authority, power, consistency, specificity)• Importance of infrastructure in systemic reform (e.g., Clune, 2001) • At the top

- Authority (School Board, broad support for performance assessments)- Power (promotion for students, high resources)- Specificity (clear expectations for students)- Consistency (coherent design across subjects, grades)

 • From the bottom

- Authority (Principals in Collaborative)- Power (huge voluntary resources for implementation)- Specificity (many details worked out between and w/i schools)- Consistency (Vision and details managed decentrally)

 

Discussion point 2: Tradeoff Of Measurement Reliability And Instructional Validity

• Traditional standardized tests: high reliability, low instructional validity• District performance assessments: low reliability, high instructional validity• Standardized performance assessments rejected as too

expensive• Two good options for district:

- Combination of both, but move to annual standardized testing for evaluation

- Revisit standardized performance assessments

 

Conclusion: Importance of Evaluation

• Important for both successful and unsuccessful programs

• To discontinue, continue, refine

• Proficiencies used enormous resources (mostly labor)

• Intensive resources = political vulnerability = need for good evaluation