2010 c washington (future of assessments) [no notes] rev 1 1
DESCRIPTION
The future of educational assessmentTRANSCRIPT
11N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
The future of AssessmentsLessons learned internationally
Washington, 9 March 2010
Andreas SchleicherHead, Indicators and Analysis Division
OECD Directorate for Education
22N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rThe future of assessments
The Holy Grail? Or the Alchemists’ Stone? Know why you are looking
You cannot improve what you cannot measure The yardstick for success is no longer
just improvement by national standards but the best performing education systems globally
Know what you are looking for A new assessment culture Responsive to changing skill requirements Capitalising on methodological advances Not sacrificing validity gains for efficiency gains
Know how you will recognise it when you find it Gauging predictive validity Impact on improving learning and teaching
Implications and lessons learned .
33N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rThe old bureaucratic system The modern enabling system
Hit and miss Universal high standards
Uniformity Embracing diversity
Provision Outcomes
Bureaucratic look-up Devolved – look outwards
Talk equity Deliver equity
Prescription Informed profession
Conformity Ingenious
Curriculum-centred Learner-centred
Interactive Participative
Individualised Community-centred
Delivered wisdom User-generated wisdom
Management Leadership
Public vs private Public with private
Culture as obstacle Culture as capital
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
1995Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
Graduate supply
Cost
per
stu
den
t
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
1995Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
United States
Finland
Graduate supply
Cost
per
stu
den
t
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
2000Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
Australia
FinlandUnited Kingdom
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
2001Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
2002Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
2003Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
2004Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
2005Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States
0 10 20 30 40 50 60 700
5000
10000
15000
20000
25000
30000
2006Ex
pend
iture
per
stu
dent
at t
ertia
ry le
vel (
USD
)
Tertiary-type A graduation rate
A world of change – higher education
United States
Australia
Finland
1313 E
duca
tion Indic
ato
rs
Pro
gra
mm
e20
09 e
dit
ion o
f Ed
uca
tion a
t a G
lance
Latin America then…
Hanushek 2009
GDP/pop 1960
Years schooling
Asia 1891 4
Sub-Saharan Africa 2304 3.3
MENA 2599 2.7
Latin America 4152 4.7
Europe 7469 7.4
Orig. OECD 11252 9.5
1414 E
duca
tion Indic
ato
rs
Pro
gra
mm
e20
09 e
dit
ion o
f Ed
uca
tion a
t a G
lance GDP/pop
1960Years
schooling
Asia 1891 4
Sub-Saharan Africa 2304 3.3
MENA 2599 2.7
Latin America 4152 4.7
Europe 7469 7.4
Orig. OECD 11252 9.5
Latin America then and now…
GDP/pop 1960
Years schooling
Growth 1960-2000
GDP/pop 2000
Asia 1891 4 4.5 13571
Sub-Saharan Africa 2304 3.3 1.4 3792
MENA 2599 2.7 2.7 8415
Latin America 4152 4.7 1.8 8063
Europe 7469 7.4 2.9 21752
Orig. OECD 11252 9.5 2.1 26147
Hanushek 2009
1515 E
duca
tion Indic
ato
rs
Pro
gra
mm
e20
09 e
dit
ion o
f Ed
uca
tion a
t a G
lance
Latin America then and now…Why quality is the key
Hanushek 2009
GDP/pop 1960
Years schooling
Growth 1960-2000
GDP/pop 2000
PISAscore
Asia 1891 4 4.5 13571 480
Sub-Saharan Africa 2304 3.3 1.4 3792 360
MENA 2599 2.7 2.7 8415 412
Latin America 4152 4.7 1.8 8063 388
Europe 7469 7.4 2.9 21752 492
Orig. OECD 11252 9.5 2.1 26147 500
1616N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Know what you are looking for
The Holy Grail was a well-described object, and there was only one true grail…
1717N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Schooling in the medieval age:
The school of the church
1818N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Schooling in the industrial age:
Uniform learning
1919N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Schooling in the industrial age:
Uniform learning
The challenges today:
Universal quality
Motivated and self-reliant citizens
Risk-taking entrepreneurs, converging and continuously emerging professions tied to globalising contexts and technological advance
2020N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rHow the demand for skills has changed
Economy-wide measures of routine and non-routine task input (US)
1960 1970 1980 1990 200240
45
50
55
60
65 Routine manual
Nonroutine manual
Routine cognitive
Nonroutine analytic
Nonroutine inter-active
(Levy and Murnane)
Mean t
ask
inp
ut
as
perc
en
tile
s of
the 1
960
task
dis
trib
uti
on
The dilemma of assessments:The skills that are easiest to teach and test are also the ones that are easiest to digitise, automate and outsource
2323N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r Education today needs to prepare students…
… to deal with more rapid change than ever before…
… for jobs that have not yet been created…… using technologies that have not yet been
invented…… to solve problems that we don’t yet know will
arise It’s about new…
Ways of thinking– involving creativity, critical thinking, problem-solving and
decision-making Ways of working
– including communication and collaboration Tools for working
– including the capacity to recognise and exploit the potential of new technologies
The capacity to live in a multi-faceted world as active and responsible citizens.
2424N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rMathematics in PISA
The real world The mathematical World
A real situation
A model of reality A mathematical model
Mathematical results
Real results
Understanding, structuring and simplifying the situation
Making the problem amenable to mathematical
treatment
Interpreting the mathematical results
Using relevant mathematical tools to solve the problemValidating
the results
2525N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rNational assessment systems differ…
Who is assessed
System assessment
School evaluation
Teacher appraisal
Student assessment
For what? E.g.• Accountabilit
y• Improvement
By whom? • Evaluators• Users of
feedback• Evaluation
agencies
How? Methods and procedures, mix of criteria and instruments• Mapping of
feedback to different units
System
School
Classroom
What?• Inputs• Processes• Outcomes
With whom? • Agents
involved
2626N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rAssessment cultures
Putting the pressure on top of the education system is the easy part, building capacity is harder
Administrative external
Participative/internal
Summative Formative
Interactive, reflective, critical
friend
Survey
InspectorateStandardised assessment
2828N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r…but there are global trends
Multi-layered, coherent assessment systems from classrooms to schools to regional to national to international levels that…
Support improvement of learning at all levels of the education system
Are largely performance-based Make students’ thinking visible and allow for
divergent thinking Are adaptable and responsive to new
developments Add value for teaching and learning by
providing information that can be acted on by students, teachers, and administrators
Are part of a comprehensive and well-aligned continuum, communicate what is expected and hold relevant stakeholders accountable .
• Integrate, synthesize and creatively apply content knowledge in novel situations
• Activate students as owners of their own learning and activate students as learning resources for one another
• Understanding what the assessment reveals about students’ thinking to shape better opportunities for student learning
• Responding to assessments can enhance student learning if tasks are well crafted to incorporate principles of learning
• Capitalise on improved data handling tools and technology connectivity to combine formative and summative assessment interpretations for a more complete picture of student learning
2929N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Know how you will recognise it when you find it
The Alchemists’ stone was to be recognised by transforming ordinary metal into gold…
3030N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Age 19
Age 21
Age 21
048
121620
Level 2Level 3
Level 4Level 5
Increased likelihood of postsec. particip. at age 19/21 associated with PISA reading proficiency at age 15
(Canada)after accounting for school engagement, gender, mother
tongue, place of residence, parental, education and family income (reference group PISA Level 1)
Odds ratioCollege entry
School marks at age 15
PISA performance at age
15
3131N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
20102011201220132014201520162017201820192020202120222023202420252026202720282029203020312032203320342035203620372038203920402041204220432044204520462047204820492050205120522053205420552056205720582059206020612062206320642065206620672068206920702071207220732074207520762077207820792080208120822083208420852086208720882089209020912092209320942095209620972098209921002101210221032104210521062107210821092110-5%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Relationship between test performance and economic outcomes
Annual improved GDP from raising performance by 25 PISA pointsPe
rcent
add
itio
n t
o G
DP
3333N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Implications and lessons learned
The medieval Alchemists’ followed the dictates of a well-established science but that was built on wrong
foundations
The search for the Holy Grail was overburdened by false clues and cryptic symbols
3434N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rFrom assessment-inhibited practice
towards outcome driven reform
Weak focus on processes
Strong focus on processes
Weak outcome-based
management
Strong outcome-based management
Integrated quality management
Good willand trust
External control, uninformed prescription
Deprivation
3535N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rSome criteria used in the world
Coherence Built on a well-structured conceptual base—an
expected learning progression—as the foundation both for large scale and classroom assessments
Consistency and complementarity across administrative levels of the system and across grades
Comprehensiveness Using a range of assessment methods to ensure
adequate measurement of intended constructs and measures of different grain size to serve different decision-making needs
Provide productive feedback, at appropriate levels of detail, to fuel accountability and improvement decisions at multiple levels
Continuity A continuous stream of evidence that tracks the
progress of both individual students .
3737N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rUnderstanding learning
progressions Learning targets
Defining what mastery means for a given skill level Progress variables
Delineate a pathway that characterise the steps that learners typically follow as they become more proficient
Evaluation of students reasoning in terms of the correctness of their solutions as well as in terms of their complexity, validity and precision
Levels of achievement Describing the breadth and depth of the learner’s
understanding of the domain at a particular level of advancement
Learning performances The operational definitions of what student’s
understanding would look like at each of the stages of progress . Wilson, ATC21S
3939N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Context- Personal
- Social/public- Global
Competencies- Identify scientific issues- Explain phenomena scientifically- Use scientific evidence
Knowledge- Knowledge of science- Knowledge about science
Attitudes-Interest in science-Support for scientific enquiry-Responsibility
IdentifyingRecognising issues that can be investigated scientificallyIdentifying keywords in a scientific investigationRecognising the key features of a scientific investigation
ExplainingApplying knowledge of science in a situationDescribing or interpreting phenomena scientifically or predicting change
Using evidenceInterpreting scientific evidence and drawing conclusionsIdentifying the assumptions, evidence and reasoning behind conclusions
OECD Level 6
OECD Level 2 Students can demonstrate
ability to understand and articulate the complex modelling inherent in the design of an investigation.
Students can determine ifscientific measurement can be applied to a given variable in an investigation. Students can appreciate the relationship between a simple model and the phenomenon it is modelling.
Students can draw ona range of abstract scientific knowledge and concepts andthe relationships between these in developing explanations ofprocesses
Students can recall anappropriate, tangible, scientific fact applicable in a simple and straightforward context and can use it to explain or predict an outcome.
Students demonstrateability to compare and differentiate among competing explanations byexamining supporting evidence. They can formulate arguments by synthesising evidence from multiplesources.
Students can point to an obvious feature in a simple table in support of a given statement. They are able to recognise if a set of given characteristics apply to the function of everydayartifacts.
4040N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
rSome methodological challenges
Can we sufficiently distinguish the role of context from that of the underlying cognitive construct ?
Do new types of items that are enabled by computers and networks change the constructs that are being measured ?
Can we drink from the firehose of increasing data streams that arise from new assessment modes ?
Can we utilise new technologies and new ways of thinking of assessments to gain more information from the classroom without overwhelming the classroom with more assessments ?
What is the right mix of crowd wisdom and traditional validity information ?
How can we create assessments that are activators of students’ own learning ?Wilson, ATC21S
4141N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Low policy value
High policy value
Low feasibility High feasibility
Money pits
Must haves
Low-hanging fruits
Quick wins
Examine individual, institutional and systemic
factors associated with performance
Establish the relative standing of students and
schools
Extending the range of competencies through which
quality is assessed
Measuring growth in learning
A real-time assessment environment that bridges the gap between formative and
summative assessment .
Monitor educational progress
Assuming that every new skill domain is orthogonal
to all others
4242P
ISA
OE
CD
Pro
gram
me
for
Inte
rnat
iona
l Stu
dent
Ass
essm
ent
See
ing
Japa
nese
sch
ools
thro
ugh
the
pris
m o
f PIS
A
•Principled•Strategic partnership
•Negotiated•Pragmatic .
•Top-down•Antagonistic .
•Leading•Evidence-driven•Achieving high reliability and innovation .
• Enabling• Incentivising
.
•World class performance.
•Continuous learning and innovation .
Good Great
•Accommodating•Evidence-based•Adopting best . practice
•Regulating .•Capacity-building
•Transparency .•Spreading best practice
• Implementing•Accepting evidence•Adopting minimum standards
•Prescribing .• Justifying
• Tackling underperformance
Adequate GoodPoor Adequate
Main focus of assessment
Role of government
Role of professions
Nature of relationship between government
and professions
Phases of development
Main outcomes
• Improvement in outcomes
•Reduction of public anxiety.
•Steady improvement
•Growing public satisfaction .
•Consistent quality•Public engagement and co-production .
Getting the sequencing right
4343N
atio
nal C
onf
ere
nce
on
Nex
t G
ener
atio
n A
sses
smen
tsW
ashi
ngto
n, 8
-9 M
arc
h 2
010
Fu
ture
of
As
sess
men
tsA
ndre
as S
chle
iche
r
Thank you !
www.oecd.org; www.pisa.oecd.org– All national and international publications– The complete micro-level database
email: [email protected] Twitter: @SchleicherEDU
…and remember:
Without data, you are just another person with an opinion