Using Curricular Measuresfor Description &

AnalysisSEC Collaborative Meeting

Tampa, FLFebruary 21, 2005

John L. Smithson, Director, Measures of Enacted CurriculumWisconsin Center for Education Research, University of Wisconsin-Madison

johns@education.wisc.edu

Surveys of Enacted Curriculum

The SEC Data - Sets

DistinctionsOn-Line Off-line

•Data collection/processing/reporting•Descriptive Data•Limited Reporting Options•Easy Access / Indiv. Results

•Analytic analyses•Unlimited reporting options•Requires data manipulation

Conducting Inquiry Using SEC Data

Forms of Inquiry Collaborative or Evaluative

Teacher EnrichmentSchool ImprovementProfessional Lrng. Comm.

Program EvaluationIndicator ReportingProgram ManagementPerformance Modeling

• MSP-PD Study (In progress: Blank, Smithson, Porter, Garet, Birman)

Use of SEC data for program evaluation

A short history of SEC research

• Reform-Up-Close (Porter, Kirst, Osthoff, Smithson, Schneider, 1993)

Validation of teacher self-report survey data.

• Upgrading Mathematics (Gamoran, Porter, Smithson, White, 1997)

First content analysis of assessment using content language.

Predictive validity of alignment index comparing instruction & assessments

• Data on Enacted Curriculum (Blank, Porter, Smithson, 2004)

Use of SEC data to facilitate school improvement efforts

First content analysis of state standards

Alignment Relationships in Standards-based Reform

Instruction

Assessment

Standards

Intersection of what is taught with what is tested.

Intersection of what is taught with what is in standards.

Intersection of whatis taught with what is in the standards

Taught,tested, and inthe standards

Alignment Relationships in Standards-based Reform

A Quantitative Approach to Alignment

Porter-Smithson/SEC Alignment Process

Content analyses of curriculum documents and reports of practice by content experts using two-dimensional content language.

Multiple raters (w/ content & assessment expertise) using independent ratings in combination with team discussions.

Content Description [Topic(s) by Cognitive Demand(s)]

Yields Alignment Index based on:

Alignment Analyses

The analytic power of quantitative alignment measures

• Prediction

• Program Evaluation*

• Planning

• Performance Modeling

* Not teacher evaluation

Alignment Analyses

Students perform best when tested on material for which they have been provided the opportunity to learn.

Class Gains Student Gains

Alignment Index 0.451 0.259

Correlation of Alignment Index to Achievement Gains

From: Gamoran, et.al. (1997). Uprgading High School Mathematics Instruction. EEPA v19n4pp325-338.

Time 0 Time 1 Time 2

Time 0 Time 1 Time 2

Learning Gains by Course Type

Learning Gains Controlling for Content

RegentsAlgebraStretch RegentsMath A/B/UCSMPGen. Mth. / Pre-alg.

From:Upgrading High School Mathematics Instruction,(Gamoran, Porter, Smithson, & White, 1997),EEPAv19n4

12

11

10.5

12

9.5

10

11.5

12

11

10.5

12

9.5

10

11.5

Explaining variation in student learning gains

Alignment Index:Instruction to Standards

MathematicsAcross 4 Districts

Alignment Analyses for School Improvement

Using alignment as an outcome measure

(Measuring change in alignment over time)

Treatment 99

Control 124

Leaders 16

Counts

Alignment Analyses for Planning Purposes

Mapping Curriculum Materials

(www.seconline.org) SEC - Online

Administrative Functions:Administration Set-upReview Registrants, Completion RatesAdministrator Report Generator

SEC Reports

Floating Bars

Dynamic SampleSelection

Dynamic DataDisaggregation

Sample andDisaggregateCounts

ComparisonCharts

IndividualResults

SEC Reports: Instructional Content

Coarse Grain Maps

Content Areas

by

Cognitive Demand

SEC Reports: Instructional Content

Fine Grain Maps

Topics

by

Cognitive Demand

Alignment as a Systemic Tool

Fine Grain

X 1 2 3 Y 1 2 3

z 1 2 3

Alignment= 0.6

C0.1 0 0

B0 0.1 0

0.2 0 0

A0.3 0.1 0

C

0.3 0.2 0

B 0.1 0.1 0.1

AA

B

C

0.5 0.1 0.1

0 0.2 0

0.1 0 0

Calculating Alignment1-((|x-y|(1-n))/2)