andrew c. porter vanderbilt university measuring the content of instruction: uses in research and...
DESCRIPTION
Content MatrixTRANSCRIPT
![Page 1: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/1.jpg)
Andrew C. PorterVanderbilt University
Measuring the Content of Instruction:
Uses in Research and Practice
![Page 2: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/2.jpg)
Tools
Teacher surveys of instructionDaily logsEnd-of-semester or end-of-year surveys
Content analyses ofStandardsTestsCurriculum materials
Alignment indices—e.g, alignment between assessment and standards
![Page 3: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/3.jpg)
Categories of Cognitive Demand
TopicsMemorize Perform
ProceduresCommunicateUnderstanding
Solve Non-Routine
Problems
ConjectureGeneralize
ProveMultiple
StepEquations
Inequalities
LiteralEquations
Lines /Slope andIntercept
Operationson
Polynomials
QuadraticEquations
Content Matrix
![Page 4: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/4.jpg)
Instruction
Vertical and Horizontal Alignment
Achievement
Assessment
Assessment Standards
StandardsDistrict
State
![Page 5: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/5.jpg)
Example Matrices to Measure Alignment
.3 0 .10 .1 00 .2 .10 .1 .1
.2 0 .10 .2 0.1 .2 .10 0 .1
Assessment Standards
Topics
Cognitive Demand
Alignment Index =1 -X=Assessment Cell ProportionsY=Standards Cell Proportions
∑ |X-Y|
2
![Page 6: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/6.jpg)
Alignment of Assessments with Standards7th-Grade Math:Goals Study
Standards Assessment States B D E F
B .37 .39 .37 .45 D .35 .37 .36 .40 E .36 .33 .43 .31 F .32 .35 .30 .41
NCTM .34 .40 .33 .47
Average Within-State Alignment = .40Average Between-State Alignment = .39Average State-Test-to-NCTM Alignment = .39
![Page 7: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/7.jpg)
Alignment of Instruction with Assessment8th-Grade Math:SCASS Study
Instruction AssessmentState H J K L E O NAEP
H 0.35 0.22 0.19 0.28 0.21 0.04 0.38J 0.34 0.21 0.18 0.25 0.20 0.05 0.38K 0.42 0.28 0.21 0.29 0.25 0.05 0.39L 0.36 0.24 0.19 0.29 0.22 0.05 0.40E 0.39 0.26 0.17 0.26 0.24 0.04 0.38O 0.35 0.21 0.16 0.26 0.20 0.05 0.38
Average Within-State Alignment = .22Average Between-State Alignment = .23Average State-to-NAPE Alignment = .39
![Page 8: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/8.jpg)
States H J K L E O G I M N
HJ .73
K .59 .66L .56 .64 .67E .65 .71 .78 .70
O .71 .80 .63 .65 .70G .71 .81 .66 .67 .71 .84I .73 .82 .63 .66 .68 .79 .80M .68 .77 .61 .62 .66 .73 .76 .79N .62 .69 .58 .61 .62 .71 .70 .67 .65
Average Alignment = .69
Alignment of Instruction with Instruction 8th-Grade Math: SCASS Study
![Page 9: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/9.jpg)
7th Grade StandardsState E State F NCTM
![Page 10: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/10.jpg)
Quality of Data Response rates Interrater agreement for content
analyses Validity of teacher self-report
Explaining between-teacher variance in alignment to NAEPPredicting student achievement gains
[Note: The need for a reform-neutral language]
![Page 11: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/11.jpg)
Uses of Tools Describing Instructional Practices
ResearchServe as dependent variable in teacher decision-making researchDescribe the implemented curriculum Measure implementation of new curriculaAssess the validity of transcript studies
PracticeInform teacher reflections on their own instructional practices
[Note: Should not be used for teacher accountability.]
![Page 12: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/12.jpg)
Uses of Tools Describing Instructional Materials
ResearchResearch effects of textbooks on instructionAssess the breadth and depth of content in instructional materials
PracticeBuild testsWrite content standardsDevelop national, state, or district indicator systems
![Page 13: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/13.jpg)
Uses of Indices of Alignment Research
Serve as a control variableServe as a dependent variable Serve as a descriptive variable
PracticeAlign state tests to state standardsAlign instructional materials to standards or course frameworks
![Page 14: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/14.jpg)
Increasing Validity and Value Getting the content languagecontent language right Using time samples to describe instruction for an entire
school year Replicating the finding that alignment predicts student
achievement gains Identifying contexts in which teacher self-reportteacher self-report on the
content of instruction is more or less accurate Improving the level and consistency of interrater interrater
agreementagreement in content analyses Understanding the distributional propertiesdistributional properties of the
alignment statistics Building powerful professional developmentpowerful professional development programs
for data-based decision making Developing a content language for reading
![Page 15: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/15.jpg)
Conclusions Much progress has been made in recognizing the
importance of instructional content as a variable in education research.
Some progress has been made in building tools for including content in education research.
There have been several innovative uses of these new tools in both research and practice, and more are on the horizon.
But there is much more work to be done.
![Page 16: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/16.jpg)
![Page 17: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/17.jpg)
7th-Grade Standards--Close ViewState E State F NCTM
Number Sense and Numeration
![Page 18: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/18.jpg)
7th-Grade Standards--Close ViewState E State F NCTM
Data Analysis and Probability
![Page 19: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/19.jpg)
Response Rates for Survey
Eisenhower LongitudinalWave 1 75%Wave 2 74%Wave 3 75%
Eisenhower Cross-Sectional 72%Reform Up Close 75%
![Page 20: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/20.jpg)
Interrater Agreement
Assessment Mean Range Goals Study .51 .77 to .34 CCSSO Study .47 .60 to .37
Standards Goals Study .48 .59 to .33
[Note: In each study, there was one outlier rater.]
![Page 21: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/21.jpg)
Eisenhower Longitudinal Study
42% of variance explained by level (elementary, middle, high school) and subject
27% of variance explained by teachers in the same school
0% of variance explained by between school or between years
Longitudinal data on instruction alignment to NAEP yielded:
![Page 22: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/22.jpg)
Alignment to Predict Achievement Gains
Index Cross-Sectional Gain
Longitudinal Gain
Level .336 (p=.02) .238 (p=.11) Level and Configuration .425 (p=.01) .341 (p=.07)
Level x Configuration .438 (p=.00) .304 (p=.04)
Cell Correlation .335 (p=.02) .240 (p=.11)
Total Configuration .312 (p=.04) .335 (p=.02)
![Page 23: Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice](https://reader035.vdocument.in/reader035/viewer/2022062504/5a4d1b7f7f8b9ab0599ba690/html5/thumbnails/23.jpg)
Index Intercorrelations
L LXC CELL TOTAL
Level 1.0
Level x Configuration .69 1.0
Cell Correlation .62 .72 1.0
Total Configuration .77 .59 .86 1.0