formative assessment system for teachers - confex · formative assessment system for teachers:...

Formative Assessment System for Teachers: Close the Gap with Something That Works

Dr. Sarah BrownIowa Department of EducationDr. Theodore ChristUniversity of Minnesota & Fast Bridge LearningDr. Megan E. CoxMinnesota Department of Education

Caveats & Disclosures• Potential Conflicts of Interest

• Theodore J. Christ, PhD has equity and royalty interests in, and will serve on the Board of Directors for, FastBridge Learning (FBL) LLC, a company involved in the commercialization of the Formative Assessment System for Teachers (FAST). The University of Minnesota also has equity and royalty interests in FBL LLC. These interests have been reviewed and managed by the University of Minnesota in accordance with its Conflict of Interest policies.

• Federal Funding• The research reported here was supported by the Institute of Education

Sciences (R324B060014, R324A090038, R305A120086, R324A130161) and Office of Special Education Programs (H327A060014-07, H327A090064, H327A100012, H327A110024, H327S150004), U.S. Department of Education, through grants to The University of Minnesota. The opinions expressed are those of the authors and do not represent views of the Office, Institute, or Department.

Summary (What was Promised)Instructionally relevant assessment systems can enhance opportunity and equity; especially if they are efficient and provide timely information. The characteristics of such systems has eluded developers and users of formative assessment and evaluation. This may be related to misplaced paradigmatic loyalties (computer adaptive versus nonadaptive, multiple choice versus performance-based, psychometric versus behavioral/observational), which confuse and distract educators who just want something that works.

This symposium will present findings and perspectives from state-level implementations of the Formative Assessment System for Teachers (FAST; FastBridge Learning) in Iowa and Minnesota along with district-level implementations in other states.

Summary (What was Promised)

• Introduction (10 min, Dr. Christ)

• Minnesota Kindergarten Entry Program (15 min, Dr. Cox)

• Iowa State-Wide Early Warning System (15 min, Dr. Brown)

• Using Data & Closing (10 min, Christ)

• Questions from audience (10 min, Drs. Brown, Cox, & Christ)

METHODS, PARADIGMS, EPISTEMOLOGY, AND PURPOSE

Data Collection, Interpretation, and Use

Paradigms as Method

Paradigms of Assessment• Psychometric Assessment

– Classical Test Theory– Generalizability Theory– Item Response Theory

• Behavioral Assessment– Curriculum-based measurement– Curriculum-based assessment– Direct behavior ratings– Systematic direct observation

• Diagnostic C & I Assessment– Teacher made– Curriculum embedded

Methods of Assessment• Modality

– Standardized, unstandardized– Individual, group– Adaptive, Non-adaptive– Selection, production– Timed, untimed– Computer, paper, in vivo

• Scoring– Standardized, unstandardized– Qualitative, quantitative– Dichotomous, polytomous– Automated, person scored

• Interpretation– Norm-referenced– Criterion-referenced (benchmark)– Self-referenced (intra-individual)

Paradigm as EpistemologyParadigms of Assessment(Serafini, 2000)

• Assessment as Measurement– Positivistic (knowledge is factual)– Large-scale, standardized,

psychometric, selection, automated scoring, understand learned in sterile conditions

• Assessment as Inquiry– Constructivist (knowledge is subjective)– Small-scale, unstandardized,

individualized, diagnostic C&I, production, person scored, understand learner in context

• Assessment as Procedure– Positivistic (knowledge is factual)– Elements of Measurement & Inquiry– Small-to-large scale, semi-standardized,

psychometric w/ and performance-based responses, production, person scored, emphasis on learner in context

Roles of Assessment Paradigms (Nagy, 2000)

• Gatekeeping• Accountability

• Instructional Diagnosis

• Instructional Diagnosis• Maybe

– Gatekeeping & Accountability

Paradigms as Epistemology

Epistemology – study of “what we know?”Realist –objective fact counts as knowledgeRelative – subjective experience counts as knowledge

Ontology – study of “how we know?”Realist – knowledge derives from discovery of factsRelativism – knowledge derives from subjective experience

Assessment as InquiryAssessment as Procedure

Assessment as Measurement

58%33%

8% 0%

Paradigms as Purpose

Purpose and Use of Data• Formative (Interim)

– Identify Problems– Analyze Problems– Plan Solutions– Monitoring Solutions

• Summative (Interim)– Evaluate Problems– Evaluate Solutions

Targets of Assessment• Systems

– Individual, group

• Programs– One, group

• Educators– Individual, group

• Students– Individual, group

Systems are at the CoreProblems are rarely specific to a student.

10

Systematic Problem Solving• Problem Solving

– Problem Identification– Problem Analysis– Plan Development– Plan Implementation & Monitoring– Plan Evaluation

• Problem (P)– difference between what is expected (E) and what occurs (O)

• (Professional) Problem Solver→ (Professional) Educator– someone who acts to reduced or eliminate a problem– data are collected to inform the change of professional behavior

Systematic Processes Solve Well-Identified ProblemsBeginning with Core Systems.

12

Systemsare at the core ofimprovement

Systematic Problem Solvingof Common Problems

Think about the Layers/Domains

Pillar of Systems Thinking

Learn to improve with aSystematic Process

that iterates across Systems, Domains and Problems.

18

&DBR

Paradigms

Paradigms

ConvergentPurposes

ConvergentPurposes

ConvergentPurposes

MINNESOTA

Minnesota’s Early Education Assessment Landscape2002-2015

• Statutory Requirements for Assessment– Birth to Five

• School Based Preschool Programs

– Kindergarten Entry• Kindergarten Readiness Assessment

– Reading Well By Third Grade• Yearly Reporting

Minnesota Department of Education - DO NOT COPY

National Perspective• Early childhood historically

focused on environmental quality and process quality until system reform efforts redefined how we view children’s learning– 50 states have early learning

standards– 26 states have a formal

definition of school readiness– 37 states measure children’s

readiness at kindergarten entry

• Federal initiatives provided funding for assessment design and revision:

• Race to the Top – Early Learning Challenge

• USDOE, Enhanced Assessment Grants

• USDOE, Preschool Development/ Enhancement Grants

Center for Early Education Learning Outcomes, 2014National Center for Education Statistics, 2013

Expectations for Young Children1995• NAEYC Position

Statement• Few states adopt school

readiness definition

Today• States continue ecological

perspective and include:– School Readiness

definitions– Early Childhood

Standards• 5 domains

– Lack of operational definitions

Community Ready

School Ready

Family Ready

Child Ready

The Kindergarten Entry Assessment pilot

Two goals– Refocus energy to classroom practice– Focus on standards

• Empirical equivalence to standards– Using both early learning and K

standards

• Two broad phases– Phase 1 – alignment– Phase 2 – assessment equivalency

Nominate

Phase 1

Phase 2


Developmental Milestones• Constructed using the Early Childhood

Indicators of Progress

Measures academic and non-academic skills

Criterion referenced, observational tool

Other tools tested

– Beginning Kindergarten Assessment– Brigance Inventory of Early Development– Desired Results Developmental Profile*– Early Learning Scale– Teaching Strategies Gold*– Work Sampling System*– Formative Assessment System for

Teachers*Minnesota Department of Education - DO

NOT COPY

Method

Mixed method approach• Tool-to-standard crosswalks• Content validation process• Claim and evidence

summaries• Technical adequacy• Empirical alignment to

standards• Relative difficulty• Concurrent calibration• Standard setting

Sample• Phase 1: 1,694 • Phase 2: 1,504• Mean age = 5.78 years• 78% white• 94% English speakers


Pilot PhasesPreliminary

criteria

Crosswalk coverage

Evidentiary reports

Statistical alignment Teacher use

Cadre scores

Equivalency analysis

Performance across groups

Scalability

Phase 1

Phase 2

Tools can be piloted and re-piloted


Conceptual Alignment

Phase 1 - Relative Difficulty

Example Item Maps- FAST DevMilestones Cognitive


Example Item Maps- FAST Language Literacy

Phase 1 - Relative Difficulty


Phase 2 – Concurrent Calibration

.5-1.5 -.5 1.5 2.5 3.5-2.5-3.5 4.5 5.5

1 12 23 3

4 4 5 56 6

DRDP

Language, Literacy, and Communication

0 0 1 1

2 23 34 4

C CO ON NS S

FAST

0

GOLD

01 12 23 3

4 45 56 6

7 78 8

WSS1 12 2 3 3

Phase 2 – Concurrent Calibration2-2 0 4 6 8-4-6 10

1 1 2 2 3 3 4 4

5 56 6

DRDP

FAST

GOLD

WSS

0 01 1 3 32 2 4 4

0 01 12 2

3 34 45 5

6 67 78 8

1 1 2 2 3 3

Social Emotional

Significance

• Links measurement and learning standards • Fits within EAG, PDG and RTT-ELC federal

initiatives• Allows districts and states choice of tool based

on quality criteria• May provide legislative support for statewide

expansion of KEAs


Iowa Context

• State provided earlyReading, R-CBM, and aReading as part of MTSS implementation in 2013-2014.

• Legislation changes led to offering the tools statewide.

Iowa Context

• Early Literacy Implementation Law implemented in 15-16

• Requires• K-3 Screening 3 times/year• K-3 Progress monitoring weekly for all

learners below benchmark targets• Screening and PM using DE Approved

Measures

Iowa Context

'13-'14

'14-'15

'15-'16

10% Implementation

94% Implementation

Over 95% implementation

2015-2016 FAST Implementation

• Screening: • 2,767,775 administrations

• Progress Monitoring: • 2,043,829 administrations

• CBM-R• 2,954,884 administrations

• aReading: • 345,323 administrations

CBMreading & aReading

712,823279,532

956,939

1,949,294

922,592

345,323

2,043,829

3,311,744

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

CBMreading Screen aReading Screen CBMreading PM Total

2014-15 2015-16

Note. Each CBMreading screening included three administrations per occasion (administrations = 2,138,470; 2,767,775). Screening occurred three times per year.Once CBMreading PM was used weekly for students below the screening benchmarks.

EarlyReading Measures371,480

262,852229,709 222,491

99,41460,174 41,621 37,171

11,164 1,7500

50,000100,000150,000200,000250,000300,000350,000400,000

Administrations

FASTBridge Spanish4,421

656 655 586302 200 101 98 95

0500

1,0001,5002,0002,5003,0003,5004,0004,5005,000

Administrations

IGDIs Measures36,130.00 35,891.00 35,765.00 35,749.00

22,909.00

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

Picture Naming Rhyming Sound ID WODNB First Sounds

Administrations

State Connections: Special Education

Iowa’s State Personnel Development Grant• OSEP Grant: 5 years, 2015-2020• Purpose: to improve practitioner ability to

diagnose, design, and deliver high-quality Specially Designed Instruction (SDI) for diverse learners so that learners with disabilities are successful.

• Focus on Early Literacy

Using growth on FASTBridge assessments as a grant evaluation measure.

State Connections: Accountability

Assessment and Data-Based

Decision-Making

Intervention SystemLeadership

UniversalInstruction

Infrastructure

Healthy Indicators

Assessment SystemHealthy Indicator Ideal Cut Scores Data Source

Percent of learners screened with a valid and reliable universal screening tool.

Intensive: 0-79%Targeted: 80-94%Universal: 95-100%

Spring 2016 Screening

Percent of learners not at benchmark assessed with a valid and reliable progress monitoring tool at least 90% of the weeks between screening periods.


Winter – Spring 2016 Progress Monitoring

• Purpose of administration• Testing vs. Teaching• Data Use

Assessment System

Universal InstructionHealthy Indicator Ideal Cut Scores Data Source

Percent of learners at benchmark in a screening period.


Spring 2016 Screening

Percent of learners at or above benchmark in the fall remaining at or above benchmark in a subsequent screening period.


Fall 2015 – Spring 2016 Screening

Universal Instruction

• Priority area of focus• Classwide Intervention

Intervention SystemHealthy Indicator Ideal Cut Scores Data Source

Percent of learners below benchmark two consecutive screening periods receiving intervention.


Fall 2016 Substantially Deficient designation and intervention scheduling

Percent of learners below benchmark in the fall who then score at or above benchmark in a subsequent screening period.


Fall 2015 - Spring 2016 Screening

Challenges and Next StepsChallenges• Practitioner understanding of the purpose of screening

and progress monitoring.• Early childhood integration.• Relationship to proficiency measure currently used.Next Steps• Continued training.• Support for utilizing progress monitoring data to make

instructional decisions.

PURPOSE AS PARADIGMINTERPRETATION AND USE OF DATA

Methods, paradigms, epistemology, and purpose

54

&DBR

Phase 2 – Concurrent Calibration

.5-1.5 -.5 1.5 2.5 3.5-2.5-3.5 4.5 5.5

1 12 23 3

4 4 5 56 6

DRDP

Language, Literacy, and Communication

0 0 1 1

2 23 34 4

C CO ON NS S

FAST

0

GOLD

01 12 23 3

4 45 56 6

7 78 8

WSS1 12 2 3 3

Phase 2 – Concurrent Calibration2-2 0 4 6 8-4-6 10

1 1 2 2 3 3 4 4

5 56 6

DRDP

FAST

GOLD

WSS

0 01 1 3 32 2 4 4

0 01 12 2

3 34 45 5

6 67 78 8

1 1 2 2 3 3

Social Emotional

& DBR

CBMreading & aReading

712,823279,532

956,939

1,949,294

922,592

345,323

2,043,829

3,311,744

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

CBMreading Screen aReading Screen CBMreading PM Total

2014-15 2015-16

Note. Each CBMreading screening included three administrations per occasion (administrations = 2,138,470; 2,767,775). Screening occurred three times per year.Once CBMreading PM was used weekly for students below the screening benchmarks.

0%9%

36%34%32%

29%16%

6%6%

2%2%2%4%

17%21%

32%34%

34%35%

43%56%

65%89%

85%74%74%

16%20%

32%33%33%

36%41%

38%29%

9%13%

23%23%

44%32%

0%0%0%0%0%0%0%0%0%0%0%

23%19%

0%0%

0%0%0%0%0%0%0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0123456789

101112

0 1 2 3 4 5 6 7 8 9 10 11 12CBM-R 0% 9% 36% 34% 32% 29% 16% 6% 6% 2% 2% 2% 4%aReading 17% 21% 32% 34% 34% 35% 43% 56% 65% 89% 85% 74% 74%aMath 16% 20% 32% 33% 33% 36% 41% 38% 29% 9% 13% 23% 23%eReading 44% 32% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%eMATH 23% 19% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

MANY NEEDS, MANY PURPOSES, MANY PARADIGMS…

PARADIGM AS PURPOSE.NOTHING LESS WILL DO.

Multi-PurposeMulti-MethodMulti-SourceMulti-Paradigm

77

formative assessment system for teachers - confex · formative assessment system for teachers:...

Documents