problematising assessment
DESCRIPTION
A discussion of some issues inherent in the practice of assessment in education; assessment drift and Type 1/Type 2 errorsTRANSCRIPT
Problematising Assessment (as if it needed it)
James Atherton11 March 13
Problematising Assessment (as if it needed it)
James Atherton11 March 13
Balloons in this colour are
additional notes for the online version
3.3: Understand theories, principles and applications of formal and informal assessment
This is the outcome to which the
session relates
3.3: Understand theories, principles and applications of formal and informal assessment
And if I were teaching Ofsted style I should
now recite the objectives...
Confused
And for once I will. At the end of this
session you should be–
Confused
...but at a higher level than before
Confused
...but at a higher level than before
Probably from Kelley, 1951, but attributed to various sources
It is frowned upon for you to
confuse your students.
Confused
...but at a higher level than before
Probably from Kelley, 1951, but attributed to various sources
It is frowned upon for you to
confuse your students.
...which may well be the biggest
limitation on your teaching.
Confusion can be constructive in teaching—
like ploughing before planting
The Problem of Proxies
1: The Problem of Proxies
...or surrogates, or substitutes, or
stand-ins for the real thing
1: The Problem of Proxies
...or surrogates, or substitutes, or
stand-ins for the real thing
Assessment is rife with them, and diluted by their use—but we are
stuck with them
This is the essence of intuitive heuristics:
when faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution
Kahneman 2011: 12
Thinking Fast and Slow, Penguin
This is the essence of intuitive heuristics:
when faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution
Kahneman 2011: 12
And this is exactly what we do in assessment
Content Assess-ment
Content Assess-ment
In principle our teaching is governed by content, and the
assessment is just to check that it has been learned
Content Assess-ment
Content Assess-ment
In practice, the demands of the assessment can all
to easily take over
Content Assess-ment
In practice, the demands of the assessment can all
too easily take over
“Will we be tested on this?”
AspectsForms
Purposes
AspectsForms
Purposes
Here are some traditional
perspectives on assessment...
• Diagnosis
• Feedback
• Standards
AspectsForms
Purposes
• Diagnosis
• Feedback
• Standards
AspectsForms
Purposes
Pre-teaching
• Diagnosis
• Feedback
• Standards
AspectsForms
Purposes
During teaching
• Diagnosis
• Feedback
• Standards
AspectsForms
Purposes
After teaching
• Validity
• Reliability
• Fairness
• Security
AspectsForms
Purposes
• Validity
• Reliability
• Fairness
• Security
AspectsForms
Purposes
Traditional criteria for evaluating assessment
• Criterion-referenced
• Norm-referenced
• Ipsative
AspectsForms
Purposes
• Criterion-referenced
• Norm-referenced
• Ipsative
AspectsForms
Purposes
Judging against fixed pre-specified criteria
• Criterion-referenced
• Norm-referenced
• Ipsative
AspectsForms
Purposes
Judging against other people’s performance
• Criterion-referenced
• Norm-referenced
• Ipsative
AspectsForms
Purposes
Judging against your own prior
performance: personal best
• Formative
• Summative
AspectsForms
Purposes
...etc. I could now test you on your
knowledge of assessment, but
• 12 items of jargon
See what I’ve done? I’ve reduced the whole topic to
Validity
• Does it do what it says on the tin?
• Is it really assessing the outcome?
What the area of practice
actually requires
Let’s look at the whole process of assessment drift.
What the area of practice
actually requires
Let’s look at the whole process of assessment drift.
Based on the work of Howard Becker and
Etienne Wenger, among others
What the course sets out to teach
What the course sets out to teach
There’s about 80% overlap—never a
perfect fit
What the course actually does
teach
What the course sets out to assess
What the course actually does
assess
What the course actually does
assess
What the area of practice
actually requires
That’s all the overlap left
What the course actually does
assess
What the area of practice
actually requires
And if you don’t pass very well...
2: False positives and false negatives:
the inherent limitations of testing
2: False positives and false negatives:
the inherent limitations of testing
I got into some trouble in this section!The maths are correct, but the problem comes with the labelling of the False Positives (or Type 1 errors) and what happens if you try to
eliminate them simply by making the assessment stricter (rather than by targeting it more precisely), so to avoid unnecessary extra
confusion, I’ve taken that out of this version.
Take a hundred people and train them for something....
In the real world, 80% are competent at it, and 20% aren’t
Competent (80%)Not
com
pete
nt
(20%
)
In the real world, 80% are competent at it, and 20% aren’t
But we’re not in the real world—we’re in a college—and we have to devise a test to determine who can be let loose on the public
Accurate (80%)
Inaccurate (20%)
... but tests aren’t always good predictors. You devise the best you can, but it may be only, say, 80% accurate.
Competent (80%)
Accurate (80%)
Inaccurate (20%)N
ot c
ompe
tent
(2
0%)
So the 80% the test passes are not the same as the 80% who are genuinely competent
True + (64%)True –16%
False – (16%)False +(4%)
True + (64%)True –16%
False – (16%)False +(4%)
These are the “true positives”—they passed
the test, and so they should have
True + (64%)True –16%
False – (16%)False +(4%)
These are the true negatives: they
failed and so they should have done.
True + (64%)True –16%
False – (16%)False +(4%)
These are the unfortunates: the test failed them, but it was wrong. That is technically a ‘Type 2’ error.
True + (64%)True –16%
False – (16%)False +(4%)
These are the ‘Type 1’ errors: they should have failed, but the test passed them.
This test will always be 20% wrong. So you can only reduce the False
Positives at the cost of increasing the False Negatives.
See the notes for more on this.
So I hope you are now confused at a higher level
than before...
• Becker H (1963) “Why school is a lousy place to learn anything in” reprinted in R J Burgess (ed.) Howard Becker on Education Buckingham; Open University Press, 1998
• Kahneman D (2011) Thinking, fast and slow London; Penguin
• Kay J (2011) Obliquity; why our goals are best achieved indirectly London; Profile Books
• Wenger E (1998) Communities of Practice; learning, meaning and identity Cambridge; C.U.P.
