the future of educational accountability: notes for a political economy of measurement michael j....
TRANSCRIPT
The Future of Educational Accountability:Notes for a Political Economy of Measurement
Michael J. Feuer
National Research Council
The National Academies
CRESST
January 2007
Celebrating Bob Linn
2
How do we sound...?
A bit depressed? Is anything all right?
A bit complicated?
Famous Speeches - Video Gallery
3
Robert’s rules
1- do rigorous science
2- convey findings simply, without simplifying
3- build public and policy maker confidence
4- keep smiling, even when the Denver airport is closed
4
Pretest
For this section, solve each problem and decide which is the best of the choices given. Fill in the corresponding circle on the answer sheet. You may use any available space for scratchwork.
(The use of a calculator is permitted.)
5
1. Scholarship
Bob Linn’s contributions to measurement theory and practice cover which of the following topics:
a) Validity, reliability, fairnessb) Standard settingc) Military, K-12, employment, personality testingd) Admissions and placemente) Intelligencef) Classroom based assessmentg) Performance measurementh) ELLs and SWDi) Test linking and equatingj) Accountabilityk) All of the above
6
2. Mentorship
Choose the adjective that best characterizes Bob Linn in his work with students, colleagues, and freeloaders (like the NRC):
a) Accessibleb) Honestc) Friendlyd) Supportivee) Patientf) Modestg) All of the above
7
3. Public service
During his career Bob Linn has been involved in which of the following activities:
a) District level school reform consultingb) Statewide assessment design and evaluationc) National policy review and strategy developmentd) International comparative assessments of school and teacher qualitye) Chairmanship or participation in national, state, local commissionsf) All of the above
8
3. Statesmanship
In answering the question, “Bob, would you be willing to help us on a project ...,” what phrases are missing from Bob’s lexicon:
a) Nob) How much are you paying?c) Do I get first authorship?d) Can I fly first class?e) What, that question AGAIN?f) All of the above
9
Sneak-peek at the answers...
Sorry, that would be cheating...
10
Oh, what the heck. A little test coaching won’t kill anyone...*
If an item looks like it might have more than one correct answer, and if you have the option of choosing “all,” then you should. (Keep this advice handy for later...)
Congratulations on scoring above basic!
*with apologies to Dan Koretz
11
And now for a quick summary of my paper...
Defining terms Reasoning by analogy An audacious theorem From optimization to optimism
12
Political Economy
Markets Externalities and market failure
Government, visible hands, collective action Rowhouses and outdoor lighting
Accountability
cause and remedy Some examples
13
An idea from the department of clever symmetries:
Political economy in general is about the measurement of externalities.
In the context of educational accountability it’s about
the externalities of measurement
14
Test based accountability: externality considerations
Origins: from Horace Mann to Bobby Kennedy
Principles:
• education provided through system of “acceptable coercion,” i.e., taxation• Legitimate public demand for information about quality of schooling• Concern for intended and unintended consequences of instruction
• Good news if you like democracy: Educational accountability is part of the broader family of social arrangements designed to instill discipline in the quality and provision of public goods.
• Caution from Jim March: “the demand for accountability is a sign of pathology in the social system”
15
In other words...
Like most (all?) beneficial technologies, high stakes testing produces its own externalities:
narrowing of the curriculum, score inflation, excessive rote memorization at the expense of deeper learning, reduction of educational goals to low minimum standards, etc.
incentives for opportunism, manipulation, cheating perpetuation of inequality, differential expectations Erosion of support for education due to persistent disappointment Etc.
16
Ideas possibly worth stealing from other policy arenas
Common approaches to management of externalities Regulation, enforcement, accountability
Upstream and downstream remedies Examples
17
Testing in context: recap
Assumptions Multiple purposes Validity and reliability Consequences Freedom of information The power of parsimony Accountability and democracy (trust but verify)
18
Pop quiz
IN A MORE PERFECT WORLD, accountability systems would have which of the following features:
l Domain robustness: authentic criterion-based representations of complex cognitive functioning based on efficient domain sampling
l Recursive stability: immunity of scores to opportunistic behavioral responses and degradation of validity l Causal neutrality: description of population differences without risk of implied attribution of cause and/or
implied acceptance of differential or discriminatory standardsl Policy precision: incentives for improved performance in desired domains without unintended degradation of
performance in other domainsl Instrument selectivity: use of tests solely for the purposes for which they were designed and validatedl Diagnostic generality: measures of aggregate performance (or growth) based on population sampling that
yield valid inferences for high stakes individual-level decisions l Minimal burden: comprehensive information based on multiple measures within constraints on budget and
testing timel Rational exuberance: standards set high enough to motivate public commitment to improvement without risk of
cyclical disappointment and erosion of morale
ALL OF THE ABOVE?
19
Sso much for coaching...
20
Some discouraging news
Theorem:
There is no accountability system that satisfies all those conditions simultaneously.
{Proof: left as an exercise to the reader...}
21
But don’t despair...
Other impossibility theorems can be EVEN MORE depressing:
Like democracy and majority rule? Try this:
From the presentation to Kenneth Arrow of the Nobel Prize in economics, 1972:
“... Let us assume that in a society one has a number of alternative conditions to choose between and that each individual in the society can rank all these alternatives in order of desirability. Is it ... possible to find ethically acceptable, democratic rules, for making a collective (or social) ranking of the different alternatives in order of desirability?”
Arrow’s answer: No.*
The conclusion: “rather discouraging as regards the dream of democracy...”
*Original paper: Arrow, K., "A Difficulty in the Concept of Social Welfare," Journal of Political Economy, 58, 4, August, 1950.
22
From impossibility to reason
Cognitive science applied to policy Procedural rationality* Assessing the counterfactuals
Suppose NCLB really does die... The predicament we face
Measuring (!) externalities: downstream Compensation or redress: what to do with false positives?
Anticipating externalities: upstream “CPA”
*see Feuer, Moderating the Debate: Rationality and the Promise of American Education, Harvard Education Press, 2006
23
Thank you. I have benefited from this opportunity, and hope the externality I produced was at least a little
bit positive...
Comments: write me at