using progress variables to map intellectual development cathleen a. kennedy and mark wilson
DESCRIPTION
Using Progress Variables to Map Intellectual Development Cathleen A. Kennedy and Mark Wilson University of California at Berkeley [email protected]. Outline. Outline. Purpose of the study Method BEAR Assessment System Calibration with Common Item Equating Setting Standards - PowerPoint PPT PresentationTRANSCRIPT
1
Using Progress Variables to Using Progress Variables to
Map Intellectual DevelopmentMap Intellectual Development
Cathleen A. Kennedy and Mark WilsonUniversity of California at Berkeley
2
OutlineOutline Purpose of the study Method
• BEAR Assessment System• Calibration with Common Item
Equating• Setting Standards
Findings • Progress Variables• Items• Progress Guides• Calibration• Performance Standards• Mapping Intellectual Development
Next Steps
Outline
3
Purpose of the StudyPurpose of the Study
Develop a framework for modeling intellectual development within a curricular unit:
1. Determine relevant constructs and instructionally useful performance levels
2. Construct formative assessments aligned with those levels
3. Establish performance expectations and associated cut-points
4. Demonstrate the use of graphical progress charts for formative feedback
Purpose
4
Context of the Study
Established FAST curriculum; 10-wk unit on Buoyancy Established m/c pre/post test Established instructional activities ≈ 300 students across 8 CA schools
Tasks: Gather more useful assessment data
1. Formalize data gathering in selected performance activities2. Extend some pre/post test items
Assess at least two learning dimensions: 1. A curriculum-specific content knowledge domain
2. A universal inquiry skill
Purpose
5
MethodMethod
Develop Assessments using BEAR* Assessment System• Determine Progress Variables• Design Items • Develop Progress Guides• Establish Measurement Model
Calibrate with Common Item Equating Set Performance Level Standards Data
Method
* Berkeley Evaluation & Assessment Research Center
6
BEAR Assessment System
Principle ToolA developmental perspective of learning Progress Variables
Alignment of assessment to Instruction Items Design
Useful to teachers Progress Guides
Evidence of quality and fairness Measurement Model
Method: BAS
Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates.
7
Calibrate with Common Item Equating
Perf. Act. 1
Perf. Act. 2
Perf. Act. 3
Pretest Post Test
Method: Calibration
8
Standard Setting
Respondents with limited knowledge.
Define performance levels as logit
ranges.
Respondents with extensive knowledge.
Respondents with some knowledge.
Respondents with more knowledge.
Method: Standard Setting
9
?
?
?
Respondents with limited knowledge.
Respondents with extensive knowledge.
Respondents with some knowledge.
Respondents with more knowledge.
Define performance levels as logit
ranges.
Standard Setting
Method: Standard Setting
10
DataData
8 CA middle schools
1 teacher per school (most new to curriculum)
14 classrooms
220 students completed all assessments
• Calibration
• Student growth
75 additional students completed post tests
• Calibration
Method: Data
11
FindingsFindings1. Developed Two Progress Variables
• Buoyancy: WTSF (specific content)
• Reasoning (generic inquiry)
2. Modified existing assessment activities to elicit evidence of the two variables
3. Developed progress guides for the two variables
4. Calibrated Progress Variables
5. Set Performance Levels
6. Mapped Student Growth
Findings
12
BEAR Assessment System Principle 1BEAR Assessment System Principle 1 Classroom assessment system should be based
on a developmental perspective of student learning.
Building Block: Progress Variables*• Defines low, high and intermediate levels• Visual metaphor for
− how students develop and− how their responses change
Development of Progress Variables
Findings: Progress Variables
* Progress Variables are also referred to as Construct Maps in the BEAR Assessment System.
13
Development of Progress Variables
1. How many variables?
2. Which are most appropriate for the purpose?
3. How many levels are needed on each variable?
Findings: Progress Variables
14
Candidate Progress Variables
Mass Volume Density
Graphing
Process InquiryBuoyancy
Why things sink & float
Building Explanations
Findings: Progress Variables
15
Mass Volume Density
Graphing
Process InquiryBuoyancy:
WTSF
Why things sink & float
ReasoningBuilding
Explanations
Final Two Progress Variables
Findings: Progress Variables
16
Useful Levels for Formative Feedback: WTSF
Instruct ional
Focus
Relative Density
Density: Objects
Density: Medium
Mass and Volume
Mass Volume
Introductionto the curriculum
1 2 3 4 5 6 7 8 9 10 11 12
Investigations
Findings: Progress Variables
17
Levels of WTSF from the Curriculum
Understands mass or volume alone
Understands mass & volume
Understands density of matter
Understands relative density
Findings: Progress Variables
18
More WTSF Levels after Reading Student Work
Misconceptions
Unresponsive
Understands mass or volume alone
Understands mass & volume
Understands density of matter
Understands relative density
Findings: Progress Variables
19
BEAR Assessment System Principle 2BEAR Assessment System Principle 2
What is taught and what is assessed must be clearly aligned.
Building Block: Items Design • A framework for designing tasks to elicit specific
kinds of evidence about student knowledge, as described in one or more progress variables, seamlessly integrated into the instructional activities of a course.
Modification of Assessments
Findings: Items Design
20
Modification of Assessments
1. What is the purpose of assessing?
2. When should formal assessment take place?
3. What types of assessment tasks are appropriate?
Findings: Items Design
21
Critical Junctures in Instruction: WTSF
Instruct ional
Focus
Relative Density
Density: Objects
Density: Medium
Mass and Volume
Mass Volume
Introductionto the curriculum
1 2 3 4 5 6 7 8 9 10 11 12
Investigations
Purpose is to determine readiness for next part of instruction.
RL4 RL7 RL10
Findings: Items Design
22
Graphing Activity
Below is a data table and graph for the Sinking-Straws activity.
Depth of Sinking 4 cm 5 cm 6 cm 7 cm 8 cm
Mass of Straw and BBs
0.8 g 1.0g 1.2 g 1.4 g 1.6 g
Explain in detail what the data and graph tell you about mass and depth of sinking. Use information and evidence from the graph as well as information from what you know. You may write and draw a picture to show your explanation.
Findings: Items Design
23
Predict-Observe-Explain Activity
We have three bottles. We know the mass of each bottle. We already measured their displaced volumes in water. Record the mass and displaced volume of each bottle in Table 1.
Table 1
Bottle # Mass (g) Displaced volume (ml)
Bottle 1
Bottle 2
Bottle 3
Based on what you know about mass and volume, in Table 2 PREDICT whether each bottle will float, sub-surface float, or sink when your teacher places the bottles into water. Explain why.
Table 2
Prediction Why do you think that the bottle will float, subsurface float, or sink?
Bottle 1
Float Sub-surface float Sink
Bottle 2
Float Sub-surface float Sink
Bottle 3
Float Sub-surface float Sink
Findings: Items Design
24
Essay Activity
Explain below why things sink and float. Write as much information as you need to explain your answer. Use evidence and examples to support your explanation.
Findings: Items Design
25
Multiple Choice with Justification Items
7. Amy put a wooden block into an overflow can filled with water. The block floats. She finds that the displaced water is 30 ml. What is the mass of the wooden block?
a. 10g
b. 30g
c. 60g
d. 90g
Please explain why you chose that answer:
___________________________________
___________________________________
___________________________________
Findings: Items Design
26
Development of Progress Guides
Teachers are the primary managers and users of assessment data.
Building block: Progress Guides*• Categories of student responses must make sense to
teachers in the context of relevant progress variables.
BEAR Assessment System Principle 3BEAR Assessment System Principle 3
Findings: Progress Guides
* Progress Guides are also referred to as the Outcome Space in the BEAR Assessment System.
27
Level “MV” on WTSF Progress Variable
Code Student knows Student needs Sample response
MV Mass and Volume
Student knows that floating depends on having a small mass and a large volume.
To progress to the next level, student needs to understand the concept of density as a way of combining mass and volume into a
single property.
“An object floats when its mass is small and
its volume is large.”
For each performance level, define:
Findings: Progress Guides
28
Levels “RD” – “M,V” on WTSF
Level What the Student Already Knows What the Student Needs to Learn
RD
Relative DensityStudent knows that floating depends on having less density than the medium.“An object floats when its density is less than the density of the medium.”
DDensityStudent knows that floating depends on having a small density.“An object floats when its density is small.”
To progress to the next level, student needs to recognize that the medium plays an equally important role in determining if an object will sink or float.
MV
Mass and VolumeStudent knows that floating depends on having a small mass and a large volume.“An object floats when its mass is small and its volume is large.”
To progress to the next level, student needs to understand the concept of density as a way of combining mass and volume into a single property.
M V
MassStudent knows that floating depends on having a small mass.“An object floats when its mass is small.”
Volume Student knows that floating depends on having a large volume.“An object floats when its volume is large.”
To progress to the next level, student needs to recognize that changing EITHER mass OR volume will affect whether an object sinks or floats.
Findings: Progress Guides
29
Level What the Student Already Knows What the Student Needs to Learn
PM
Productive MisconceptionStudent thinks that floating depends on having a small size, heft, or amount, or that it depends on being made out of a particular material.“An object floats when it is small.”
To progress to the next level, student needs to refine their ideas into equivalent statements about mass, volume, or density. For example, a small object has a small mass.
UF
Unconventional FeatureStudent thinks that floating depends on being flat, hollow, filled with air, or having holes.“An object floats when it has air inside it.”
To progress to the next level, student needs to refine their ideas into equivalent statements about size or heft. For example, a hollow object has a small heft.
OT
Off TargetStudent does not attend to any property or feature to explain floating.“I have no idea.”
To progress to the next level, student needs to focus on some property or feature of the object in order to explain why it sinks or floats.
NR No ResponseStudent left the response blank.
To progress to the next level, student needs to respond to the question.
Similar coding guide developed for Reasoning variable.
Findings: Progress Guides
Levels “PM” – “NR” on WTSF
30
You have six blocks. Blocks 1, 2 and 3 are made of one material, while blocks 4, 5 and 6 are made of another material.
The density of blocks 1, 2 and 3 is 0.91 g/cm3, while the density of blocks 4, 5, 6 is 1.2 g/cm3.
For each block, predict whether it will sink, float or subsurface float in water. Give your reasons for your predictions.
3
Sink
Float
Subsurface Float
Block 3 will float because objects float when their density is less than 1.
WTSF – D
Reas. – P
Student Response
Reflective Lesson 10B
Findings: Progress Guides
Two Codes
31
CalibrationCalibration
Classroom assessment requires reliability and validity evidence and evidence for fairness.
Building Block: Measurement model• Seeking interpretive quality• Multidimensional partial credit model in which
− Order of item difficulties is the same for all respondents− Order of respondents is the same for all item subsets− One parameter model for interpretive needs
BEAR Assessment System Principle 4BEAR Assessment System Principle 4
Findings: Calibration
32
Respondent Location
Goal of Calibration: Interpretive Quality (1)
1. Person locations are interpreted in the context of item content (criterion referenced).
Buoyancy depends on the mass of the object.
Buoyancy depends on the object being flat, hollow,
filled with air, etc.
XRL4_A.Mass
RL4_A.ProductiveMisconceptions
Findings: Calibration
33
Goal of Calibration: Interpretive Quality (2)
2. Student change derives meaning from items.
Buoyancy depends on the density of the object.
Buoyancy depends on the mass and volume of the
object.
Buoyancy depends on the mass of the object.
Buoyancy depends on the object being flat, hollow,
filled with air, etc.
RL4_A.Mass
RL4_A.ProductiveMisconceptions
XRL4_A.Mass&Volume
X
Findings: Calibration
34
Calibrated All Items onto Same Scale
Five forms of post test contained all items from all instruments.
Anchored Pretest, RL4, RL7 and RL10 item difficulties from post test calibration.
Findings: Calibration
35
Set Performance LevelsSet Performance Levels
Based on Thurstonian thresholds (50-50)
Find mean of thresholds for each step
Cut-point is midpoint between means
X
13.MV
6.MV
4.M
16.M
3.M
Findings: Set Perf. Levels
36
WTSF Performance Levels
Findings: Set Perf. Levels
37
WTSF Performance Levels
Findings: Set Perf. Levels
38
Buoyancy depends on the density of the object.
Buoyancy depends on the object being flat, hollow, filled with air, etc.
Buoyancy depends on the density of the object relative to the density of the medium.
Buoyancy depends on the mass and volume of the object.
Buoyancy depends on the mass or volume of the object.
1.4
0.4
-0. 6
-1.6
Relative Density
Density
Mass & Volume
Mass or Volume
Misconception
Set Criterion-Referenced Cut Points
Does not attend to any property or feature to explain floating
Unresponsive-2.3
Findings: Set Perf. Levels
39
Map Student PerformanceMap Student Performance
Class performance status
Individual student change
Comparisons to curricular expectations
Item diagnostics
Findings: Map Performance
40
Class Performance Status
Amy Tom Brian
After Lesson
7
Findings: Map Performance
41
Performance for Amy through Lesson 7
Pretest
Lesson 4
Lesson 7
Findings: Map Performance
42
Compared to Curricular Expectations
Findings: Map Performance
43
Diagnostic MapStudent - Brian
Variable - WTSFItem Set - RL7
Response Level Unachieved Levels
XXX
Item Diagnostics
Findings: Map Performance
44
Item DiagnosticsDiagnostic MapStudent - Brian
Variable - WTSFItem Set - RL7
Response Level Unachieved Levels
RL7b.MV RL7b.MorV XXX
RL7b.Mis
Performed as expected on this P-O-E item.
Findings: Map Performance
45
Item DiagnosticsDiagnostic MapStudent - Brian
Variable - WTSFItem Set - RL7
Response Level Unachieved Levels
RL7d.MV XXX RL7d.MorV RL7d.Mis
RL7d.UR
Performed more poorly than expected on this P-O challenge item.
Findings: Map Performance
46
Next StepsNext Steps1. Procedure tested in FOSS curriculum Fall 2006
2. Gathering teacher reactions to
a. Using standards-based graphs for formative feedback
b. Using software to record item codes
c. Consistency with intuition about students
Next Steps