“some” implications of large scale assessment for msps marge petit center for assessment may...

“Some” Implications of Large Scale Assessment

for MSPs

Marge PetitMarge Petit

Center for AssessmentCenter for Assessment

May 2004May 2004

Accountability that only Accountability that only focuses on identifying focuses on identifying schools is not going to schools is not going to improve schools. improve schools. Brian GongBrian Gong

May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 2

Some Definitions

Large-scale assessment;Large-scale assessment; Standards; Standards; Standards-based large-scale assessment;Standards-based large-scale assessment; Grade level expectations; andGrade level expectations; and Grade level assessments.Grade level assessments.


NCLB and Challenges (Jim Pelligrino, NRC Conference 2004)

§ Designing and implementing assessment systems to respond effectively to the multiple provisions of No Child Left Behind;

§ Defining the appropriate targets for assessment relative to content standards;

§ Determining how targets might be apportioned across different elements of a comprehensive assessment system; and

§ Developing and supporting assessment systems that provide information to support the enhancement of learning and instruction as well as accountability.

The multiple sets of goals and needs is a major conceptual and operational design challenge.


Primary Goal of MSPs

To improve student learning in mathematics and science!


Givens All schools associated with MSPs are responsible for improving All schools associated with MSPs are responsible for improving

student learning in mathematics and science as they relate to state student learning in mathematics and science as they relate to state standards and grade level expectations;standards and grade level expectations;

All schools associated with MSPs are responsible for improving All schools associated with MSPs are responsible for improving student performance on state assessments;student performance on state assessments;

All schools are responsible for making Adequate Yearly Progress All schools are responsible for making Adequate Yearly Progress in mathematics as defined by each state consistent with in mathematics as defined by each state consistent with requirements of NCLB;requirements of NCLB;

The state level assessment is currently the basis for determining The state level assessment is currently the basis for determining AYP;AYP;

One measure of the effectiveness of your MSP will probably be One measure of the effectiveness of your MSP will probably be an increase in the percentage of student at proficient or above on an increase in the percentage of student at proficient or above on state grade level assessments.state grade level assessments.


Implications for MSPs The quality of grade level assessments will impact your work and The quality of grade level assessments will impact your work and

mathematics and science instruction and learningmathematics and science instruction and learning . . Learn about the Learn about the issues facing your state, and get involved in the ongoing issues facing your state, and get involved in the ongoing development of the grade level assessments in your state.development of the grade level assessments in your state.

The quality of grade level expectations will impact your work and The quality of grade level expectations will impact your work and the quality of mathematics and science instruction and learning. the quality of mathematics and science instruction and learning. All All that is valued in mathematics and science may or may not be articulated that is valued in mathematics and science may or may not be articulated in state standards and grade level expectations. in state standards and grade level expectations. Find the gaps and fill.Find the gaps and fill.

Large scale assessment CANNOT provide ALL the information that Large scale assessment CANNOT provide ALL the information that is needed to improve mathematics/science instruction and learning.is needed to improve mathematics/science instruction and learning. Build a “coherent and balanced” assessment system to support Build a “coherent and balanced” assessment system to support MSP schoolsMSP schools..



mathematics and science instruction and learningmathematics and science instruction and learning . . Learn about the Learn about the issues facing your state, and get involved in the development of the issues facing your state, and get involved in the development of the grade level assessments in your state.grade level assessments in your state.

The quality of grade level expectations will impact your work and The quality of grade level expectations will impact your work and the quality of mathematics and science instruction and learning. the quality of mathematics and science instruction and learning. All All that is valued in mathematics and science may or may not be articulated that is valued in mathematics and science may or may not be articulated in state standards and grade level expectations. in state standards and grade level expectations. Find the gaps and fillFind the gaps and fill..



“Some” Real time - Practical and Political issues interacting with technical and educational issues

Alignment;Alignment; Testing time – time of year;Testing time – time of year; Turn-around time and the impact on item types;Turn-around time and the impact on item types;

Item types and Depth of Knowledge;Item types and Depth of Knowledge; Integrity of the content;Integrity of the content; Level of information derived;Level of information derived;

Access for the greatest number of students;Access for the greatest number of students; Science LiteScience Lite Standard setting; and Standard setting; and Implementation cost.Implementation cost.


Dimensions of Alignment (e.g., Webb, 1997;

Achieve, Inc., Porter, 2002; WestEd)

ContentContent Cognitive demand (e.g., Depth of Cognitive demand (e.g., Depth of

Knowledge (Webb); Level of Challenge Knowledge (Webb); Level of Challenge (Achieve)(Achieve)

Balance;Balance; RangeRange


Porter Alignment on 2 dimensions

Grade 8Grade 8

MemorizeMemorize Perform Perform

ProceduresProcedures

Communicate Communicate UnderstandingUnderstanding

Solve non-Solve non-routing routing

problemsproblems

Conjecture, Conjecture, Analyze, ProveAnalyze, Prove

SimilaritySimilarity

Pythagorean Pythagorean TheoremTheorem

Porter alignment dimensions


12 ft

8 ft

A

B C

What is the length of ?AC

Range of items

Wyoming BOE

An Alignment Quiz


How much alignment is enough?

A: The state standards and state assessment should be perfectly matched/aligned. (True or False – Why or why not?)

State Standards/grade level expectations



B: The state standards should be a subset of the objectives assessed on the state assessment.

State Assessment




The objectives assessed on the state assessment should be a subset of the state standards/grade level expectations.



State Assessment

C


State Assessment

D


State Standards/Grade Level Expectations

Mentally add and subtract whole numbers Mentally add and subtract whole numbers to ….to ….

Design and conduct an experiment to Design and conduct an experiment to answer a student or teacher generated answer a student or teacher generated experiment…experiment…


Alignment

Grade 8Grade 8

MemorizeMemorize Perform Perform

ProceduresProcedures

Communicate Communicate UnderstandingUnderstanding

Solve non-Solve non-routing routing

problemsproblems

Conjecture, Conjecture, Analyze, ProveAnalyze, Prove

Lines, slopes Lines, slopes and and interceptsintercepts

xx xx xx

Local Curriculum: Students will demonstrate conceptual understanding of linear relationships as a constant rate of change by determining the slope of a line, explaining the meaning of slope and intercept in context from a table,graph or situation, and by solving routine and non-routine problems involving the relationship between the rate of change, slope, and intercept.


Two Types of Grade Level Expectations

Test Specification for Test Specification for the large scale the large scale assessmentassessment

Specification for local Specification for local curriculum and curriculum and assessment.assessment.

Test Specification

for large-scale


Some Characteristics of Two Types

Test SpecificationTest Specification Must be assessable in an Must be assessable in an

on-demand large-scale on-demand large-scale setting;setting;

Should be a prioritized set.Should be a prioritized set.

Local Curriculum and Local Curriculum and AssessmentAssessment

Can include concepts and Can include concepts and skills not easily assessable skills not easily assessable in an on-demand setting;in an on-demand setting;

Can include foundational Can include foundational skills as they develop skills as they develop across grades.across grades.


Examples

Test SpecificationTest Specification M–DSP–3–1M–DSP–3–1 Interprets a Interprets a

given representationgiven representation ((line line plots, bar graphs, tally charts, plots, bar graphs, tally charts, or tables) to answer questions or tables) to answer questions related to the data, to analyze related to the data, to analyze the data to formulate the data to formulate conclusions, or to make conclusions, or to make predictionspredictions.. (New England (New England Common Assessment Program)Common Assessment Program)

Local Curriculum and Local Curriculum and AssessmentAssessment

GLE #M26: In response to a GLE #M26: In response to a teacher or student generated teacher or student generated question or hypothesis, question or hypothesis, collects appropriate data, collects appropriate data, organizes the data, organizes the data, displays/represents the data and displays/represents the data and makes observations about the makes observations about the data to draw conclusions about data to draw conclusions about the questions or hypothesis the questions or hypothesis being tested. (Local Vermont being tested. (Local Vermont GLE)GLE)


Why a balance of item types?

•To adequately assess GLEs within a given time period;•To appropriately assess the content and cognitive demand in GLEs; and •To appropriately reflect instructional and assessment practices consistent with GLEs.

“Large scale assessments should be substantially consistent with high quality classroom assessments though procedurally separate.” Shepard 2000

Link to item type instruction


Item Types“(1) you get what you assess... [and] (2) you do not get what you do not assess... (Resnick & Resnick,

1992, p. 59)

Write a rule using symbols for the nth term in the following arithmetic sequence.

Modified - Texas 2002 8th grade

Generalize a variety of patterns, and represent the

patterns symbolically.


Generalize a variety of patterns, and represent the

patterns symbolically.

Item Types“(1) you get what you assess... [and] (2) you do not get what you do not assess... (Resnick & Resnick, 1992, p. 59

Texas 2002 8th grade


Science “Lite” Ed Week Jan. 2004 Science “Lite” Science “Lite”

Items that don’t require scienceItems that don’t require science

Item with technical design flawsItem with technical design flaws

39 state study - Carnine, Miller Metzenberg


Science “Lite” Ed Week Jan. 2004

Doesn’t require Doesn’t require sciencescience

A lever is a bar that turns around a fixed point called a fulcrum. A pair of scissors is made of two levers that move in opposition [an illustration of a pair of scissors are provided with A – D]. Which of the following points is the fulcrum of the two levers?

A) Point AB) Point BC) Point C

D) Point D


Science “Lite” Ed Week Jan. 2004

Science “Lite” Science “Lite”


Access for the greatest number of students … … provide the greatest number of students the provide the greatest number of students the

opportunity to demonstrate their knowledge and opportunity to demonstrate their knowledge and skills in relationship to the skills in relationship to the mathematical or mathematical or science constructscience construct being assessed. being assessed.

Universal Design


Conserving the Mathematical Construct (Petit and Lager 2003)

Explicitly aligning items with mathematical Explicitly aligning items with mathematical construct being assessed– content and process construct being assessed– content and process demands in GLE;demands in GLE;

Make intentional decisions of when and how to Make intentional decisions of when and how to use context to assess the mathematics constructuse context to assess the mathematics construct;;

Streamlining language (Lager, 2003) to provide Streamlining language (Lager, 2003) to provide access without compromising mathematical access without compromising mathematical construct being assessed;construct being assessed;

Appropriately using graphics, pictures, graphs, Appropriately using graphics, pictures, graphs, tables, diagrams, and models.tables, diagrams, and models.




The quality of grade level expectations will impact your work and The quality of grade level expectations will impact your work and the quality of mathematics and science instruction and learning. the quality of mathematics and science instruction and learning. All All that is valued in mathematics and science may or may not be articulated that is valued in mathematics and science may or may not be articulated in state standards and grade level expectations. in state standards and grade level expectations. Find the gaps and fill.Find the gaps and fill.

Large scale assessment CANNOT provide ALL the information that Large scale assessment CANNOT provide ALL the information that is needed to improve mathematics/science instruction and learning.is needed to improve mathematics/science instruction and learning. Build a “coherent and balanced” assessment system to support Build a “coherent and balanced” assessment system to support MSP schoolsMSP schools. .


At the foundation of any assessment or accountability system based on student performance is the framework of concepts and skills upon which they are to be built.


The ultimate validity test…

“If a school is identified under NCLB for low performance in mathematics, the response by the school should be to strengthen the mathematics instruction based upon a solid

set of grade level expectations.” Tim Kurtz, NH Assessment Director


Primary Goal of MSPs

To improve student learning in mathematics and science!

MSP Responsibility: Go beyond the bullets on the GLEs to support curriculum, instruction, and assessment with deep and lasting understanding of concepts and skills.


New Jersey Grade 2---By the end of Grade 2, students will:A. Number Sense1. Use real-life experiences, physical materials, and technology to construct meanings for numbers (unless otherwise noted, all indicators for grade 4 pertain to these sets of numbers as well).

•Whole numbers through millions• Commonly used fractions (denominators of 2, 3, 4,5, 6, 8, 10, 12, 16)

2. Demonstrate a sense of the relative magnitude of numbers. 3. Understand the various uses of numbers; 4. Use concrete and pictorial models to relate whole numbers, commonly used fractions, and decimals to each other, and to represent equivalent to forms of the same number;5. Compare and order whole numbers.6. Compare and order numbers.7. Explore settings that give rise to negative numbers.

What does it mean to develop deep and lasting

understanding of…. ?


By the end of Grade 4, students will:A. Number Sense1. Use real-life experiences, physical materials, and technology to construct meanings for numbers (unless otherwise noted, all indicators for grade 4 pertain to these sets of numbers as well).

•Whole numbers through millions• Commonly used fractions (denominators of 2, 3, 4,5, 6, 8, 10, 12, 16)

2. Demonstrate a sense of the relative magnitude of numbers. 3. Understand the various uses of numbers; 4. Use concrete and pictorial models to relate whole numbers, commonly used fractions, and decimals to each other, and to represent equivalent to forms of the same number;5. Compare and order whole numbers.6. Compare and order numbers.7. Explore settings that give rise to negative numbers.

What does it mean to develop deep and lasting

understanding of…. ?


An Example – Does this sequence promote deep and lasting understanding

Shade 1/3 of the figure.

What is 1/3 of 27?


Students progress through four levels of Students progress through four levels of partitioning: sharing; algorithmic halving; partitioning: sharing; algorithmic halving; evenness; oddness; and composition. (Pothier and evenness; oddness; and composition. (Pothier and Sawada, 1990, cited in Bezuk and Bieck, 1993, Sawada, 1990, cited in Bezuk and Bieck, 1993, pp.124 – 125)pp.124 – 125)

““Fair share” means the same number of pieces or Fair share” means the same number of pieces or parts regardless of their size. Partitions are drawn parts regardless of their size. Partitions are drawn in a shape for the correct number of parts, in a shape for the correct number of parts, disregarding the size of the parts (e.g., Cutting a disregarding the size of the parts (e.g., Cutting a circle with four vertical lines). (Payne, 1975) circle with four vertical lines). (Payne, 1975)

•Proper fractions (denominators of 2, 3, 4, 8, 10)


Shared with permission of the Vermont Mathematics Partnership


Questions about GLEs and local curriculum

Do the GLEs promote and support quality Do the GLEs promote and support quality instruction and assessment in mathematics and instruction and assessment in mathematics and science?science?

Have aspects of science and mathematics not Have aspects of science and mathematics not easily assessed in large scale, but important, been easily assessed in large scale, but important, been included for local curriculum and assessment?included for local curriculum and assessment?

Is local curriculum and assessment aligned with Is local curriculum and assessment aligned with state GLEs?state GLEs?

Are the GLEs and local curriculum prioritized?Are the GLEs and local curriculum prioritized? Other?Other?




The quality of grade level expectations will impact your work and The quality of grade level expectations will impact your work and the quality of mathematics and science. the quality of mathematics and science. All that is valued in All that is valued in mathematics and science may or may not be articulated in state standards mathematics and science may or may not be articulated in state standards and grade level expectations. and grade level expectations. Find the gaps and fill the, but doFind the gaps and fill the, but do not not support “mile wide and inch deep” curriculum or assessment. Help support “mile wide and inch deep” curriculum or assessment. Help Prioritize!Prioritize!



Large Scale Assessments are Not valuable for… Lorrie Shepard, Assessment in Support of Learning

Providing detailed understanding of Providing detailed understanding of individual student learning on an ongoing individual student learning on an ongoing basis!basis!

Capturing synthesis of knowledge or some Capturing synthesis of knowledge or some concepts that are not easily assessed in concepts that are not easily assessed in large-scale assessment.large-scale assessment.


Local Assessment: Webb, DoK Level 4 – Curriculum Embedded Assessment


UsesUses Level of Level of InfluenceInfluence

Large Scale State Level Large Scale State Level AssessmentAssessment

Local – Large Scale Local – Large Scale AssessmentAssessment

Classroom summative Classroom summative assessmentassessmentTests, quizzes, projects, etc.Tests, quizzes, projects, etc.

Formative AssessmentFormative Assessment

OtherOther

How balanced is the assessment system in MSP schools?


Characteristics of a Coherent and Balanced

Assessment System

• Comprehensive; (Adapted from Knowing What Student Know(NRC, 2001)

• Coherent; (Adapted from Knowing What Student Know(NRC, 2001)

• Continuous; (Adapted from Knowing What Student Know(NRC, 2001)

• Integrated; (Distilled from other reports listed on last page)

• Includes quality assessments. (Distilled from other reports

listed on last page)


What do I know? What do I need to find out? What actions should I take? Assessment design issues facing my state;Assessment design issues facing my state; The design of grade level assessments;The design of grade level assessments; The nature of school and student level information The nature of school and student level information

derived from large scale assessment;derived from large scale assessment; The status of the grade level expectation The status of the grade level expectation

development and implementation;development and implementation; The degree to which educators have the range of The degree to which educators have the range of

information needed to make program adjustments information needed to make program adjustments and to make instructional decisions – “on time”;and to make instructional decisions – “on time”;


References

Bezuk, N. S., and Bieck, M. (1993). Current Research on Rational Numbers and Common Fractions: Summary and Implications for Teachers. In D.T. Owens (Ed.), Research Ideas for the Classroom: Middle Grades Mathematics (pp. 118-136). New York: Macmillan.

Lager, C., Petit, M., (2003) Conserving the Mathematical Construct, TSNE Test Specifications v. 12,,

National Research Council (2001a), Adding it Up: How Children Learn Mathematics. Mathematics Learning Study Committee, Jeremy Kilpatrick, Jane Swan, and Bradford Findell (Eds.)., Washington D.C., National Academy Press.

National Research Council (2003), Assessment in Support of Student Learning: Bridging the Gap Between Large-scale and Classroom Assessment. Mathematical Science Education Board, Board on Testing and Measurement, and Committee on Science Education Washington D.C., National Academy Press.

National Research Council (2001b), How People Learn: Brain, Mind, Experience, and School. Committee in the Development of the Science of Learning. John Bransford, Ann Brown, and Rodney Cocking (Eds.) Washington D.C., National Academy Press.

National Research Council (2000), Knowing What Students Know: The Science and Design of Educational Assessment. Committee on the Foundations of Assessment. James Pelligrino, Naomi Chudowsky, and Robert Glaser (Eds.) Washington D.C., National Academy Press.

National Research Council (2000), Learning about Assessment, Learning Through Assessment,. Mathematical Sciences Education Board. Mark Driscol and Deborah Bryant (Eds.) Washington D.C., National Academy Press.

National Research Council (2000), Classroom assessment and the national science education standards, . J. Myron Adkins, Paul Black, and Janet Coffey (Eds.) Washington D.C., National Academy Press.

Payne, J. N.. (1975) Review of research on fractions. In R. Lesh and Bradbard (Eds.) Number and measurement: Papers from a research workshop. (Page 145 – 187), Columbus, Ohio.

Petit, M., (2003) Aligning to What?, RILS Annual Meeting. www.nciea.orgPopham, J. [In press} (2003) Crafting Curricula Aims for Instructionally Supportive AssessmentWebb, N., (2002)Resnick, L. B., & Resnick, D. P. (1992). Assessing the thinking curriculum: New tools for educational reform. In B. R. Gifford & M. C. O'Conner (Eds.), Changing assessments: Alternative views of aptitude, achievement and instruction. Boston: Kluwer Academic