1 the thinking behind · pact· performance assessment for california teachers raymond pecheone...

11

The Thinking BehindThe Thinking Behind

··PACT·PACT·PPerformanceerformance

AAssessment forssessment for CCaliforniaalifornia TTeacherseachers

The Thinking BehindThe Thinking Behind

··PACT·PACT·PPerformanceerformance

AAssessment forssessment for CCaliforniaalifornia TTeacherseachers

Raymond PecheoneRaymond Pecheone Stanford UniversityStanford University

April 16, 2008April 16, 2008

22

The PACT Assessment The PACT Assessment SystemSystem


• A performance assessment for teacher A performance assessment for teacher candidates created in response to SB 2042, candidates created in response to SB 2042, with new subject matter standards, new with new subject matter standards, new program standards, and new assessment program standards, and new assessment standardsstandards

• Alternate assessments permitted must meet Alternate assessments permitted must meet California Quality Standards for California Quality Standards for reliability/validity reliability/validity (i.e., AERA/APA test standards).(i.e., AERA/APA test standards).

• Aligned with the California Teaching Aligned with the California Teaching Performance Expectations (standards) and Performance Expectations (standards) and California Content StandardsCalifornia Content Standards

• High stakes assessment designed to initially High stakes assessment designed to initially license beginning teacherslicense beginning teachers

22

33

PACT InstitutionsPACT InstitutionsPACT InstitutionsPACT Institutions

‣ UC BerkeleyUC Berkeley

‣ UC DavisUC Davis

‣ UC IrvineUC Irvine

‣ UCLAUCLA

‣ UC RiversideUC Riverside

‣ UC San DiegoUC San Diego

‣ UC Santa BarbaraUC Santa Barbara

‣ UC Santa CruzUC Santa Cruz

33

‣ Cal Poly — SLOCal Poly — SLO

‣ CSU Channel IslandsCSU Channel Islands

‣ CSU ChicoCSU Chico

‣ CSU Dominguez HillsCSU Dominguez Hills

‣ CSU Monterey BayCSU Monterey Bay

‣ CSU NorthridgeCSU Northridge

‣ Humboldt StateHumboldt State

‣ Sacramento State Sacramento State

‣ San Diego StateSan Diego State

‣ San Francisco StateSan Francisco State

‣ San Jose StateSan Jose State

‣ Sonoma StateSonoma State

‣ StanfordStanford

‣ Holy Names UniversityHoly Names University

‣ Mills CollegeMills College

‣ Notre Dame de Namur Notre Dame de Namur UniversityUniversity

‣ Pepperdine UniversityPepperdine University

‣ St. Mary’s College of St. Mary’s College of CaliforniaCalifornia

‣ University of the University of the PacificPacific

‣ University of San University of San DiegoDiego

‣ Antioch UniversityAntioch University

‣ USCUSC

‣ San Diego InternSan Diego Intern

444444



Assessments Embedded in Local Assessments Embedded in Local ProgramsPrograms

— — examples —examples —

Assessments Embedded in Local Assessments Embedded in Local ProgramsPrograms

— — examples —examples —

Observation/Supervisory Evaluation & Observation/Supervisory Evaluation & FeedbackFeedback

Observation/Supervisory Evaluation & Observation/Supervisory Evaluation & FeedbackFeedback

Child Child Case Case StudiesStudies

Child Child Case Case StudiesStudies

Analyses Analyses of Student of Student LearningLearning

Analyses Analyses of Student of Student LearningLearning

Curriculum Curriculum /Teaching /Teaching AnalysesAnalyses

Curriculum Curriculum /Teaching /Teaching AnalysesAnalyses

The Capstone The Capstone Teaching EventTeaching EventThe Capstone The Capstone

Teaching EventTeaching Event

Teaching EventTeaching EventDemonstrates :Demonstrates :

‣PlanningPlanning

‣InstructionInstruction

‣AssessingAssessing

‣ReflectingReflecting‣Academic Academic LanguageLanguage

Teaching EventTeaching EventDemonstrates :Demonstrates :

‣PlanningPlanning

‣InstructionInstruction

‣AssessingAssessing

‣ReflectingReflecting‣Academic Academic LanguageLanguage

55

Teaching Event Teaching Event Records of Practice*Records of Practice*Teaching Event Teaching Event

Records of Practice*Records of Practice*Instructional and Social ContextInstructional and Social Context

3 to 5 Days3 to 5 Days

PlanningPlanning•Lesson PlansLesson Plans•Handouts, Handouts, overheads, overheads, student workstudent work•Lesson Lesson CommentaryCommentary

InstructiInstructionon•Video Video clip(s)clip(s)•Teaching Teaching CommentaryCommentary

AssessmeAssessmentnt•Analysis of Analysis of Whole Class Whole Class Assessment Assessment •Analysis of Analysis of learning of 2 learning of 2 studentsstudents

ReflectioReflectionn•Daily Daily ReflectionsReflections•Reflective Reflective CommentaryCommentary

Evidence of Academic LanguageEvidence of Academic Language

55

* 24 Teaching Events in 13 credential areas

66

Teaching Event Teaching Event Subject AreasSubject Areas

Teaching Event Teaching Event Subject AreasSubject Areas

• Multiple Multiple SubjectsSubjects

‣ LiteracyLiteracy

‣ MathematicsMathematics

66

• Single Subject Single Subject

‣ AgricultureAgriculture

‣ English language English language artsarts

‣ History social History social sciencescience

‣ MathematicsMathematics

‣ ScienceScience

‣ ArtArt

‣ MusicMusic

‣ Physical EducationPhysical Education

‣ World languagesWorld languages

77

Guiding Questions and Guiding Questions and Analytic RubricsAnalytic Rubrics

Guiding Questions and Guiding Questions and Analytic RubricsAnalytic Rubrics

• PLANNINGPLANNING‣ Establishing a Balanced Establishing a Balanced

Instructional FocusInstructional Focus

‣ Making Content Making Content AccessibleAccessible

‣ Designing AssessmentsDesigning Assessments

• INSTRUCTIONINSTRUCTION‣ Engaging Students in Engaging Students in

LearningLearning

‣ Monitoring Student Monitoring Student Learning During Learning During InstructionInstruction

77

• ASSESSMENTASSESSMENT‣ Analyzing Student Work Analyzing Student Work

From an AssessmentFrom an Assessment

‣ Using Assessment to Using Assessment to Inform TeachingInform Teaching

• REFLECTIONREFLECTION‣ Monitoring Student Monitoring Student

ProgressProgress

‣ Reflecting on TeachingReflecting on Teaching

• ACADEMIC ACADEMIC LANGUAGELANGUAGE‣ Understanding Language Understanding Language

DemandsDemands

‣ Supporting Academic Supporting Academic Language DevelopmentLanguage Development

88

PACT Rubrics PACT Rubrics (one example)(one example)PACT Rubrics PACT Rubrics (one example)(one example)

88

ELEMENTARY LITERACY TEACHING EVENT ( 2004-05 PILOT)ELEMENTARY LITERACY TEACHING EVENT ( 2004-05 PILOT)GUIDING QUESTION:How does the candidate use analysis GUIDING QUESTION:How does the candidate use analysis of student learning to propose next steps in instruction?of student learning to propose next steps in instruction?

Level 1Level 1 Level 2Level 2 Level 3Level 3 Level 4Level 4

• Next steps are Next steps are vaguely related to vaguely related to or not aligned with or not aligned with the analysis of the analysis of student student misunderstandings misunderstandings and needs.and needs.

— OR — — OR —

• Next steps are not Next steps are not described in described in sufficient detail to sufficient detail to understand them.understand them.

— OR — — OR —

• Next steps are Next steps are based on inaccurate based on inaccurate conclusions about conclusions about student student development from development from the assessment the assessment analysis.analysis.

• Next steps focus on Next steps focus on improving student improving student performance performance through support through support that addresses that addresses student student misunderstandings misunderstandings or needs.or needs.

• Next steps are Next steps are based on broad based on broad patterns of patterns of performance on the performance on the assessment.assessment.

• Next steps focus on Next steps focus on improving student improving student performance performance through targeted through targeted support to support to individuals and individuals and groups to address groups to address specific specific misunderstandings misunderstandings or needs.or needs.

• Next steps are Next steps are based on analysis of based on analysis of whole class patterns whole class patterns of performance, of performance, some patterns for some patterns for individuals and/or individuals and/or subgroups and subgroups and general knowledge general knowledge of indvidiual of indvidiual students and/or students and/or subgroups.subgroups.

All components of All components of Level 3 plus:Level 3 plus:•Next steps Next steps demonstrate a strong demonstrate a strong understanding of both understanding of both the identfied content the identfied content and language and language standards and of standards and of individual students individual students and/or subgroups.and/or subgroups.

Level 2Level 2

•Next steps focus on improving Next steps focus on improving student performance through student performance through support that addresses student support that addresses student misunderstandings or needs.misunderstandings or needs.

•Next steps are based on broad Next steps are based on broad patterns of performance on the patterns of performance on the assessment.assessment.

99

2-Day Subject Specific 2-Day Subject Specific Scorer TrainingScorer Training

2-Day Subject Specific 2-Day Subject Specific Scorer TrainingScorer Training

• DAY 1DAY 1

‣ Overview of PACT Teaching Event and scoring Overview of PACT Teaching Event and scoring processprocess

‣ Discussion on biasDiscussion on bias

‣ Note taking and DocumentationNote taking and Documentation

‣ Understanding Level “2”Understanding Level “2”

•• DAY 2DAY 2



‣ Independently score a Calibration Independently score a Calibration Teaching Event & DebriefTeaching Event & Debrief

1100

PACT ScoresPACT ScoresInter-rater Reliability Inter-rater Reliability

PACT ScoresPACT ScoresInter-rater Reliability Inter-rater Reliability

Level of AgreementLevel of Agreement PercentPercent

Exact MatchExact Match 46%46%

± 1 point± 1 point 34%34%

± 2 points or greater± 2 points or greater 10%10%

Sample SizeSample Size · 2,580 · 2,580Spearman-Brown Reliability EstimateSpearman-Brown Reliability Estimate · · 0.880.88

1111

• Content validityContent validity

‣ Development teams, Development teams, Program directors, Program directors, Program faculty, & Program faculty, & Leadership teamLeadership team

‣ TPE alignment studyTPE alignment study

• Concurrent validityConcurrent validity

‣ Evaluation of score Evaluation of score validityvalidity

‣ Decision Consistency · Decision Consistency · Holistic vs. Analytic Holistic vs. Analytic ratingsratings

• Bias and fairness Bias and fairness reviewreview

PACT Validity StudiesPACT Validity StudiesPACT Validity StudiesPACT Validity Studies

1111

• Construct validityConstruct validity

‣ Factor AnalysisFactor Analysis(2002-03 Pilot Year):(2002-03 Pilot Year):

• Reflection Reflection & Assessment& Assessment

• InstructionInstruction

• PlanningPlanning

• Predictive Validity Predictive Validity (Carnegie/CT Study)(Carnegie/CT Study)

1122

What We Learn from What We Learn from the PACT Analysesthe PACT Analyses

What We Learn from What We Learn from the PACT Analysesthe PACT Analyses

•How our candidates do:How our candidates do:‣ On different aspects of teachingOn different aspects of teaching

‣ In different subject areasIn different subject areas

‣ In comparison to other institutionsIn comparison to other institutions

‣ Over timeOver time

‣ With different kinds of supportsWith different kinds of supports

1122

1133

Task Mean Item Scores by Campus

1.75

2.00

2.25

2.50

2.75

3.00

3.25

Total MISPlanningInstructionAssessment

Reflection

Acad Language

Category Title

Mean Item Score

A

B

C

D

E

F

G

H

I

J

K

Data Charts · 2003-04Data Charts · 2003-04Campus/Task ScoresCampus/Task Scores

Data Charts · 2003-04Data Charts · 2003-04Campus/Task ScoresCampus/Task Scores

1144

Task Mean Item Scores by Content Area

1.75

2.00

2.25

2.50

2.75

3.00

Total MIS

Planning MISInstruction MISAssessment MIS

Reflection MIS

Academic Language

Category Title

Mean Item Score

EL

EM

ELA

MTH

HSS

SCI

Data Charts · 2003-04Data Charts · 2003-04Content Area/Task ScoresContent Area/Task ScoresData Charts · 2003-04Data Charts · 2003-04Content Area/Task ScoresContent Area/Task Scores

1155

PACT Scores - PACT Scores - Assessment of Student Assessment of Student Learning (2003-2005)Learning (2003-2005)

PACT Scores - PACT Scores - Assessment of Student Assessment of Student Learning (2003-2005)Learning (2003-2005)

Student Learning Score Frequency 2003

0

50

100

150

200

250

1 2 3 4

Score

Frequency


0

50

100

150

200

250

1 2 3 4

Score

Frequency


0

100

200

300

400

1 2 3 4

Rubric Scores

Frequencies

1166

Faculty Learning & Faculty Learning & Program ImprovementProgram Improvement

Faculty Learning & Faculty Learning & Program ImprovementProgram Improvement

• Increased articulation across Increased articulation across courses, structures and rolescourses, structures and roles

• Changes in content of some Changes in content of some coursescourses

• Structural changes in Teacher Structural changes in Teacher Education Program Education Program

1177

PACT Teaching Event · PACT Teaching Event · DNADNA

PACT Teaching Event · PACT Teaching Event · DNADNA

• Documents teaching of learning Documents teaching of learning segment segment (3-5 lessons or hours of instruction)(3-5 lessons or hours of instruction)

• Subject specificSubject specific

• Standardized tasks & core questions Standardized tasks & core questions across programsacross programs

• Scored with common rubrics, passing Scored with common rubrics, passing standardstandard

• During student teachingDuring student teaching1177

1188

For More Information...For More Information...For More Information...For More Information...

• See Teaching Event See Teaching Event Handbooks and Rubrics Handbooks and Rubrics at www.pacttpa.org.at www.pacttpa.org.

1199

ActionsActionsActionsActions

ScorinScoring g

theTEtheTE

ScorinScoring g

theTEtheTE

CollaborativCollaborativeplanning eplanning

acrossUniveacrossUniversity &K-12 rsity &K-12

schoolsschools

CollaborativCollaborativeplanning eplanning

acrossUniveacrossUniversity &K-12 rsity &K-12

schoolsschools

ProfessionProfessional al

DevelopmeDevelopmentnt

ProfessionProfessional al

DevelopmeDevelopmentnt

PACT PACT AdvisoAdviso

rr

PACT PACT AdvisoAdviso

rr

Analysis Analysis of of

CandidatCandidate Worke Work

Analysis Analysis of of

CandidatCandidate Worke Work

Program Program MeetingMeeting

ss

Program Program MeetingMeeting

ss

2200

Total Mean Item Scores and Task Mean Item Scores by Campus (2003-04 Pilot Year)

1.50

1.70

1.90

2.10

2.30

2.50

2.70

2.90

3.10

3.30

Total MIS Planning Instruction AssessmentReflection

Acad Language

Mean Item Score

A

B

C

D

E

F

G

H

I

J

K

2211

The Research Base for The Research Base for Teacher Licensing TestsTeacher Licensing TestsThe Research Base for The Research Base for Teacher Licensing TestsTeacher Licensing Tests

• Weak relationship between traditional Weak relationship between traditional licensing tests and teacher effectiveness licensing tests and teacher effectiveness (NRC, 2001)(NRC, 2001)

‣ Strauss & Sawyer (1986)Strauss & Sawyer (1986)

‣ Ferguson (1991, 1998)Ferguson (1991, 1998)

‣ Ferguson & Ladd (1996) Ferguson & Ladd (1996)

‣ Clotfelter, Ladd, & Vigdor (forthcoming) Clotfelter, Ladd, & Vigdor (forthcoming)

‣ Goldhaber (2005, 2006) Goldhaber (2005, 2006)

• Effect sizes quite small in recent value Effect sizes quite small in recent value added research (.01 .06)added research (.01 .06)

2211

2222

Educative AssessmentEducative AssessmentEducative AssessmentEducative Assessment

• Teachers MatterTeachers Matter

• Subject Matter MattersSubject Matter Matters

• Preparation (support) Preparation (support) MattersMatters

• Authenticity MattersAuthenticity Matters

• Integration of Practice Integration of Practice MattersMatters

2222

2233

Total Mean Item Scores and Task Mean Item Scores by Campus (2003-04 Pilot Year)

1.50

1.70

1.90

2.10

2.30

2.50

2.70

2.90

3.10

3.30

Total MIS Planning Instruction AssessmentReflection

Acad Language

Mean Item Score

A

B

C

D

E

F

G

H

I

J

K

2244

Total Mean Item Score and Task Mean Item Score by Content Area (2003-04 Pilot Year)

1.5

1.7

1.9

2.1

2.3

2.5

2.7

2.9

3.1

Total MISPlanning MIS

Instruction MISAssessment MIS

Reflection MIS

Academic Language

Mean Item Score

EL

EM

ELA

MTH

HSS

SCI

2255

California Teaching California Teaching Performance Performance ExpectationsExpectations

California Teaching California Teaching Performance Performance ExpectationsExpectations

‣ TPE 1 · TPE 1 · Specific Pedagog-Specific Pedagog-ical Skills for Subject ical Skills for Subject Matter InstructionMatter Instruction

‣ TPE 2 · TPE 2 · Monitoring Student Monitoring Student Learning During InstructionLearning During Instruction

‣ TPE 3 · TPE 3 · Interpretation and Interpretation and Use of AssessmentsUse of Assessments

‣ TPE 4 · TPE 4 · Making Content Making Content AccessibleAccessible

‣ TPE 5 · TPE 5 · Student Student EngagementEngagement

‣ TPE 6 · TPE 6 · Developmentally Developmentally Appropriate Teaching Appropriate Teaching PracticesPractices

2255

‣TPE 7 · TPE 7 · Teaching English Teaching English LearnersLearners

‣TPE 8 · TPE 8 · Learning about Learning about StudentsStudents

‣TPE 9 · TPE 9 · Instructional Instructional PlanningPlanning

‣TPE 10 · TPE 10 · Instructional Instructional TimeTime

‣TPE 11 · TPE 11 · Social Social EnvironmentEnvironment

‣TPE 12 · TPE 12 · Professional, Professional, Legal, and Ethical Legal, and Ethical ObligationsObligations

‣TPE 13 · TPE 13 · Professional Professional GrowthGrowth

2266

What is Subject Specific What is Subject Specific about the Teaching about the Teaching

Event?Event?

What is Subject Specific What is Subject Specific about the Teaching about the Teaching

Event?Event?• Focus of learning segment & aligned to Ca. Focus of learning segment & aligned to Ca.

content standardscontent standards

• Teaching/learning tasks on video clip(s)Teaching/learning tasks on video clip(s)

• Additional prompts in some content areas Additional prompts in some content areas (e.g., misconceptions in science, (e.g., misconceptions in science, dispositions in mathematics, description of dispositions in mathematics, description of text in text in English/language arts)English/language arts)

• Common and subject specific rubricsCommon and subject specific rubrics

• BenchmarksBenchmarkswithin subject areaswithin subject areas 22

66

2277

ScoringScoringScoringScoring

• Trained and calibrated subject Trained and calibrated subject specific assessorsspecific assessors

• Campus based with central audits Campus based with central audits & regional scoring& regional scoring

• Rubric based scoring in real time Rubric based scoring in real time (web based platforms)(web based platforms)

• Organized around dimensions of Organized around dimensions of teaching (PIARA) and guiding teaching (PIARA) and guiding questionsquestions

• Sequentially Scored By PIARA TasksSequentially Scored By PIARA Tasks 2277

2288

PACT ScoresPACT ScoresAssessment of Student Learning (2003 - Assessment of Student Learning (2003 -

2005)2005)

PACT ScoresPACT ScoresAssessment of Student Learning (2003 - Assessment of Student Learning (2003 -

2005)2005)

Student Learning Score Frequency

0

100

200

300

400

1 2 3 4

— Score —

— Frequency —

2003

2004

2005