Designing an Assessment Designing an Assessment System for the Race to System for the Race to

the Topthe Top

Edward HaertelSchool of EducationStanford University

RTTT Public Meeting, General AssessmentDenver, ColoradoDecember 1, 2009

1

What’s Different This What’s Different This Time?Time?In the absence of well-defined and demanding standards, education in the United States has gravitated toward de facto national minimum expectations, with curricula focusing on low-level reading and arithmetic skills and on small amounts of factual material in other content areas. Most current assessment methods reinforce the emphasis on these low-level skills and processing bits of information rather than on problem solving and critical thinking. The adoption of world-class standards would force the Nation to confront today's educational performance expectations that are simply too low. 

NCEST Report, 1992

2

New IdeasNew IdeasCommon Standards

◦fewer, clearer, higher?Aligned assessments of individual

growthExplicit attention to range of

assessment types / formats, content sampling, item release plans, etc.

21st century technology◦computer-based testing, longitudinal growth

models, Evidence-Centered Design, …

3

Old IdeasOld IdeasIf assessments are aligned to

standards, then growth in test scores will track intended schooling outcomes

Essentially all valued schooling outcomes can be captured by test performances of◦students working alone◦without “smart workplace” resources◦answering clear questions with clear

answers

4

Old IdeasOld IdeasOne assessment system will “do it all”

◦track progress of students / schools / …

◦inform decisions about graduation / college readiness /

career readiness / college course placement◦support improvement

in teaching / learning / instructional programs◦determine effectiveness

of schools / principals / teachers

5

6

Theories of Action are, if anything, less clear than in 1992

A Theory of Action is a chain of logical propositions that “connects the dots,” explaining step by step, in commonsense terms, how the assessment system, accountability system, and related reforms will lead to improved learning outcomes

Rethinking Old Ideas To Rethinking Old Ideas To Make New Ideas WorkMake New Ideas WorkAligning tests and curricula to

common standards is not sufficient to create a coherent system

7

Content Standards

Curriculum &

Instruction

Assessments

?

Rethinking Old Ideas To Rethinking Old Ideas To Make New Ideas WorkMake New Ideas WorkAligning tests and curricula to

common standards is not sufficient to create a coherent system

8

Content Standards

Curriculum &

Instruction

Assessments

?

11th Grade History /11th Grade History /Social StudiesSocial Studies

US11.11.2.Discuss the significant domestic policy speeches of Truman, Eisenhower, Kennedy, Johnson, Nixon, Carter, Reagan, Bush, and Clinton (e.g., education, civil rights, economic policy, environmental policy).

9

Item Testing US11.11.2Item Testing US11.11.2

10

High School BiologyHigh School BiologyBI6.fStudents know at each link in a food web some energy is stored in newly made structures but much energy is dissipated into the environment as heat. This dissipation may be represented in an energy pyramid.

11

Item Testing BI6.fItem Testing BI6.f

12

Rethinking AlignmentRethinking AlignmentIt is not enough for each item to be

somehow connected to one or more standards

Assessments, standards, and curriculum and instruction must all be considered together ◦Curriculum-embedded formative

assessments◦Portfolios incorporating well-specified,

externally scorable components13

Rethinking Old Ideas To Rethinking Old Ideas To Make New Ideas WorkMake New Ideas WorkTesting need not imply that

students work alone, without references or resources, on small tasks with one right answer◦Well-structured problems

(Frederiksen, 1984)◦Content-lean, process-constrained tasks

(Baxter & Glaser, 1998)◦Sequestered Problem Solving

(Bransford & Schwartz, 1999)

14

Rethinking What It Means Rethinking What It Means to Take A Testto Take A TestA comprehensive assessment system

must also include◦ill-structured problems with multiple solutions

that are better or worse, not right or wrong◦content-rich, process-open tasks requiring

flexible, adaptive application of coherent knowledge structures to solve novel problems

◦tests of preparation for future learning – measuring skill in approaching new problems, “knowing with” as well as “knowing how” or “knowing that”

15

Rethinking Old Ideas To Rethinking Old Ideas To Make New Ideas WorkMake New Ideas WorkA single form of testing cannot

satisfy the full set of RTTT design requirements◦Sole reliance on external, on-demand

tests will fall short, even if constructed-response items are included

◦Sole reliance on external, on-demand tests will fall short, even if interim / benchmark assessments are included

16

Rethinking Multiple Rethinking Multiple IndicatorsIndicatorsA comprehensive assessment system

must use multiple forms of evidence◦curriculum-embedded assessments◦extended projects◦a range of student performances including

finding, organizing, and evaluating information analyzing and synthesizing data planning and carrying out investigations solving unfamiliar, open-ended problems written reports and oral presentations

17

Rethinking Multiple Rethinking Multiple IndicatorsIndicatorsDifferent forms of assessment,

organized in different ways, may be required for◦improving instructional decision

making in the classroom◦tracking student growth and

predicting college or career readiness

◦accountability

18

Rethinking Multiple Rethinking Multiple IndicatorsIndicatorsStandards, assessments, curriculum,

and instruction must be aligned and coherent◦horizontal links across content areas◦vertical links across grade levels

Assessment at the classroom or school level may employ matrix sampling◦different students answer different

questions to provide better group-level information

◦some assessment system components may not yield individual scores

19

SummarySummaryWe must RETHINK Alignment

◦not just test-to-standards, but instead a coherent, simultaneous consideration of standards, assessments, curriculum and instruction, and I might add, teacher preparation

20

SummarySummaryWe must RETHINK what it means to

take a test◦not just sequestered problem solving,

with tidy questions and routine problems, but also complex tasks with multiple solutions judged by multiple

criteria sometimes worked out over an extended time sometimes involving collaboration sometimes requiring students to figure out

what information they need and where to find it

21

SummarySummaryWe must RETHINK multiple

indicators◦not just a broader mix of item formats◦not just interim plus end-of-year tests◦some portion of classroom assessment

must be sufficiently documented and structured so as to be incorporated into comprehensive assessment systems to meet multiple needs

22

23

Thank you