What is evaluation What is evaluation knowledge for practice?knowledge for practice?
Thomas A. SchwandtThomas A. Schwandt
22
Evaluation as EvidenceEvaluation as Evidence--forfor--UseUse
Reinvigorated link between Reinvigorated link between evaluation and scientific thinking evaluation and scientific thinking (e.g., experimentalism, realist (e.g., experimentalism, realist evaluation, theoryevaluation, theory--based evaluation)based evaluation)Collections of evidence modeled on Collections of evidence modeled on Cochran CollaborationCochran CollaborationDefining evaluation in the context of Defining evaluation in the context of managerialismmanagerialism
33
OutlineOutline
The meaning of The meaning of evaluationevaluation and and practicepracticeHow they can be relatedHow they can be relatedSignificance of this analysis for the Significance of this analysis for the way we think about evaluationway we think about evaluation
44
Why approach the topic this Why approach the topic this way?way?
What are we assuming about ‘evaluation’and ‘practice’ when we direct our evaluative lens on the work of practitioners?
Learning from the ‘practice turn’
The importance of language
55
What is What is ““evaluationevaluation””? ? Two ways to talk about evaluation:Two ways to talk about evaluation:•• E E valuationvaluation•• e e valuation valuation Both concerned with making distinctions Both concerned with making distinctions
of worthof worth——that is, decisions about the that is, decisions about the value, merit, significance of some action value, merit, significance of some action taken or about to be takentaken or about to be taken
Each reflects a different rationality & a Each reflects a different rationality & a different idea of evaluation knowledgedifferent idea of evaluation knowledge
66
E E valuationvaluationThe most familiar to usThe most familiar to usTools and approaches for making Tools and approaches for making decisions about value guided by a general decisions about value guided by a general logic:logic:•• Establish criteria of meritEstablish criteria of merit•• Generate evidence of performance on criteriaGenerate evidence of performance on criteria•• Synthesize performance into overall judgmentSynthesize performance into overall judgment
Professional activityProfessional activity
77
Examples of Examples of E E valuationvaluation
Functional outcome assessmentFunctional outcome assessmentQuality of care approachQuality of care approach——performance monitoring and auditingperformance monitoring and auditingEvidenceEvidence--based practicebased practice
88
Functional AssessmentFunctional Assessment
Functional Assessment Taxonomy for Functional Assessment Taxonomy for disability determinationdisability determination•• Operationally defined and measured Operationally defined and measured
constructs for workconstructs for work--related assessments related assessments used preused pre-- and postand post--interventionintervention
99
Quality of Care Approach:Quality of Care Approach:Evaluating the service systemEvaluating the service system
DomainDomainTechnical careTechnical care
Process of careProcess of care
Structure of careStructure of care
DefinitionDefinitionClient outcomesClient outcomes
Organization of service Organization of service team, activities directed at team, activities directed at client motivation, etc.client motivation, etc.
Qualities of space, Qualities of space, facilities, equipment, etc.facilities, equipment, etc.
1010
EvidenceEvidence--based practice based practice approachapproach
What worksWhat works----causal efficacy causal efficacy of the treatment/interventionof the treatment/intervention
1111
EvidenceEvidence--based practice based practice approachapproach
Taxonomy of evidenceTaxonomy of evidenceLevel I Level I –– Evidence from Evidence from RCTsRCTs or systematic reviewsor systematic reviewsLevel II Level II –– Evidence of one RCTEvidence of one RCTLevel III Level III –– Evidence from cohort or caseEvidence from cohort or case--controlled controlled
studiesstudiesLevel IV Level IV –– Evidence from nonEvidence from non--experimental studiesexperimental studiesLevel V Level V –– Opinions of authorities, clinical Opinions of authorities, clinical
experience, descriptive studiesexperience, descriptive studies
1212
What What E E valuation Assumes About valuation Assumes About Practice, Rationality, & KnowledgePractice, Rationality, & Knowledge
1)1) World of actors with intentional World of actors with intentional states/ individual motivations states/ individual motivations distinct from and acting ondistinct from and acting on
2)2) A preA pre--existing world of existing world of ‘‘practicepractice’’. . Thus,Thus,
3)3) Assume relationship of detachmentAssume relationship of detachment
1313
Individual mind of actor/ Individual mind of actor/ External world of actionsExternal world of actions
1414
What What E E valuation Assumes About valuation Assumes About PracticePractice
4)4) Practice is something deliberately Practice is something deliberately chosen and aligned with strategies chosen and aligned with strategies and goals in the minds of actorsand goals in the minds of actors
5)5) This practice is capable of This practice is capable of observation and classification by observation and classification by the evaluator in terms of inputs, the evaluator in terms of inputs, processes, and outputsprocesses, and outputs
1515
What What EEvaluation Assumes About valuation Assumes About PracticePractice
6)6) The authority of evaluation resides The authority of evaluation resides in its ability to supersede everyday in its ability to supersede everyday experience in the way in which it experience in the way in which it studies practicestudies practice
1616
E E valuation authorityvaluation authority
Performance of A Practice
1717
What What E E valuation Assumes About valuation Assumes About PracticePractice
7)7) Practice = an instrumental means Practice = an instrumental means to solve problems; governed by to solve problems; governed by instrumental rationalityinstrumental rationality
8)8) Practice = complex set of Practice = complex set of techniques, attitudes, dispositionstechniques, attitudes, dispositions
9)9) Normative state of affairs for Normative state of affairs for practice = clarity, relative certainty, practice = clarity, relative certainty, orderorder
1818Lack of clarity and confusion and ambiguity in practice….
1919
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
East OfficeWest OfficeNorth Office
BRINGING ORDER TO PRACTICE
2020
Ah! Clarity and the elimination of ambiguity….!
2121
What What E E valuation Assumes About valuation Assumes About PracticePractice
‘Outside’, Expert Knowledge
Problems ofAction inPractice
Applied
to
2222
What What E E valuation Assumes About valuation Assumes About PracticePractice
Because practices are local, Because practices are local, contingent, contextual, contingent, contextual, E E valuation valuation knowledge must be adapted to the knowledge must be adapted to the demands of a given contextdemands of a given contextThe problem, however, lies not in the The problem, however, lies not in the nature of the knowledge but in the nature of the knowledge but in the ‘‘contextcontext’’
2323
SummarySummary
Actors are selfActors are self--conscious, intentionalconscious, intentionalActions are guided by clear goals and Actions are guided by clear goals and strategies; they are purposeful and strategies; they are purposeful and plannedplannedAction can be reconstructed in logic Action can be reconstructed in logic of explanationof explanationAll of this is capable of observation All of this is capable of observation and classification by the evaluatorand classification by the evaluator
2424
SummarySummary
A nonA non--collaborative collaborative ‘‘sciencescience’’ of of evaluationevaluation““SuccessSuccess”” of the evaluation judged of the evaluation judged by independent standards for by independent standards for rational behavior on the part of rational behavior on the part of evaluation expertsevaluation experts
2525
ee valuationvaluation
A kind of evaluation A kind of evaluation knowledge and rationality in knowledge and rationality in agreement with everyday lifeagreement with everyday life
2626
What What e e valuation assumes about valuation assumes about practicepractice
Practice is not simply what people Practice is not simply what people ‘‘dodo’’Practice is a social site where Practice is a social site where identities of practitioners and identities of practitioners and strategies are strategies are relationallyrelationally coco--constructedconstructed
2727
““DwellingDwelling””
A mode of engagement with the A mode of engagement with the world (of people and objects) in world (of people and objects) in which the familiar things around us which the familiar things around us are nonare non--deliberately and effortlessly deliberately and effortlessly coco--opted into our current activity to opted into our current activity to form an extension of our being and form an extension of our being and doing doing
2828
RelationalityRelationalityThe ceaseless actions of daily life constitute a fluctuating network of connections and disconnections with the various objects that extend and reflect our sense of human agency. Instead of an external world of stable objects that support our needs and purposes, the space and time of daily life begins to look like a restless scene of acts in permanent suspension… Acts…connect and disconnect. They express relationships between things rather than the things themselves. (Cooper, 2005, p. 1690-1691)
2929
RelationalityRelationality
I have in mind the hundreds of practical judgments, the everyday taken-for-granted routines and practices, the explicit and tacit knowledge that is brought to bear on concrete situations, the moving about in the legal-moral environment of large administrative bureaucracies, the mastering of difficult human-emotional situations, the negotiating of discretionary space, and the interactive give and take with colleagues that make up everyday public administration. (Wagenaar, 2004, p.644)
3030
““PracticePractice””
Is inherently dialogical & interactiveIs inherently dialogical & interactiveIt is about transactions with others It is about transactions with others that are simultaneously cognitive and that are simultaneously cognitive and emotiveemotiveIs fluid, changeable, dynamicIs fluid, changeable, dynamicIs about the particularIs about the particularIs indeterminate (impossible to know Is indeterminate (impossible to know precisely)precisely)
3131
““PracticePractice””
Requires a kind of knowledge we call Requires a kind of knowledge we call practical judgment, wise judgment, practical judgment, wise judgment, practical wisdom, practical wisdom, phronesisphronesis, that, in , that, in turn, requires perception and turn, requires perception and deliberation or judgmentdeliberation or judgment
3232
Practical perceptionPractical perception[The] capacity to identify with clarity and discernment the features of a complex situation that are significant for determining the most appropriate action. It brings to the fore aspects of a situation that have gone unnoticed and gives one an intuitive sense of what is important for making a moral choice. Practical perception is cultivated through one’s experiences of moral choice and through observation of the choices made by [others]. (Polkinghorne, 2004, p. 117)
3333
What does this view of practice What does this view of practice have to do with the way we think have to do with the way we think
about evaluation?about evaluation?
A different rationality and A different rationality and a different kind of knowledge a different kind of knowledge
3434
Why Why ee valuation?valuation?
It draws our attention to the kind of It draws our attention to the kind of decisions about distinctions of worth decisions about distinctions of worth inherent in our practicesinherent in our practicesIt is the practical art of living well, of It is the practical art of living well, of realizing the realizing the ‘‘goodgood’’ constitutive of a constitutive of a morally worthwhile form of human morally worthwhile form of human activity (e.g., teaching counseling, activity (e.g., teaching counseling, parenting, social work, etc.)parenting, social work, etc.)
3535
ee valuationvaluation
Focused on the moralFocused on the moral--political political activity called praxisactivity called praxisAntidote to the view that the only Antidote to the view that the only rational rational ‘‘sciencescience’’ of evaluation is a of evaluation is a methodical sciencemethodical science
3636
ee valuationvaluation
A science of practical philosophy; a A science of practical philosophy; a science that guides praxisscience that guides praxisA science that reclaims the sphere of A science that reclaims the sphere of praxis from its modern assimilation praxis from its modern assimilation into the sphere of technical and into the sphere of technical and instrumental rationalityinstrumental rationality
3737
What is What is praxispraxis??1.1. Making wise choices as we pursue Making wise choices as we pursue
common goalscommon goals2.2. Acting rationally in the absence of rules Acting rationally in the absence of rules
and proceduresand procedures3.3. Arriving at understandings of what it is Arriving at understandings of what it is
right to do through conversationright to do through conversation4.4. Allowing ourselves to be questionedAllowing ourselves to be questioned5.5. Recognizing that our decisions are always Recognizing that our decisions are always
imperfect, provisional, never imperfect, provisional, never ‘‘finishedfinished’’
3838
A practical science of A practical science of ee valuationvaluation
““Practical knowledge and Practical knowledge and understanding can only be developed understanding can only be developed and advanced by practitioners and advanced by practitioners engaging in the kind of dialogue and engaging in the kind of dialogue and conversation through whichconversation through which…… [the] [the] assumptions implicit in their practice assumptions implicit in their practice can be made explicit and their can be made explicit and their collective understanding of their collective understanding of their praxis transformedpraxis transformed”” (Carr, 2006)(Carr, 2006)
3939
A practical science of A practical science of ee valuationvaluation
Goal is not knowledge to control or Goal is not knowledge to control or repair what practitioners dorepair what practitioners doGoal is to initiate and sustain Goal is to initiate and sustain processes of critical reflectionprocesses of critical reflectionEvaluator as collaboratorEvaluator as collaboratoree valuation is dialogical and socially valuation is dialogical and socially cooperativecooperative
4040
Socially cooperative form of Socially cooperative form of inquiryinquiry……
4141
Disagreement as productive Disagreement as productive feature of feature of praxispraxis……
4242
Critical reflection on Critical reflection on praxispraxis
Mapping a practiceMapping a practice----puzzles, puzzles, dilemmas, challenges; dilemmas, challenges; communicative strategies, communicative strategies, dispositions, normative expectations, dispositions, normative expectations, guiding concepts, rules, proceduresguiding concepts, rules, procedures
4343
Critical reflection on Critical reflection on praxispraxis
Examining practical knowledge:Examining practical knowledge:1.1. Rules of practiceRules of practice2.2. Practical principlesPractical principles3.3. ImagesImages----Embodied, experiential Embodied, experiential
complexes of meaning that guide complexes of meaning that guide decisionsdecisions
4444
Final thoughtsFinal thoughts……
EEvaluation and valuation and eevaluation valuation ≠≠ Macro Macro versus micro view of evaluationversus micro view of evaluationThe important issue is that each is a The important issue is that each is a different form of rationality and different form of rationality and requires a different form of requires a different form of knowledgeknowledge
4545
Different orientations to Different orientations to everyday lifeeveryday life
Instrumental and Instrumental and technical rationalitytechnical rationalityPractice is an Practice is an objectobject--like entity like entity that can be that can be inspectedinspectedEvaluation Evaluation knowledge is knowledge is means to an endmeans to an end
Practical rationalityPractical rationalityPractice is a Practice is a relational dwellingrelational dwellingEvaluation Evaluation knowledge is knowledge is knowledge about knowledge about how to engage in how to engage in critical reflection critical reflection on praxison praxis
4646
What is What is ‘‘evaluationevaluation’’ knowledge knowledge for?for?
KnowledgeKnowledge--forfor--action: the action: the application of application of knowledge and knowledge and technique to the technique to the instrumental instrumental problems of problems of practice (e.g., practice (e.g., effectiveness and effectiveness and efficiency)efficiency)
KnowledgeKnowledge--inin--action: action: Understanding how Understanding how to make wise to make wise decisions acquired decisions acquired from critical from critical reflection on reflection on experience of experience of actual situations; actual situations; means and ends means and ends defined defined interactivelyinteractively
4747
Consequences of Consequences of ‘‘EE valuationvaluation’’ knowledgeknowledge
Dominant belief that everyday life Dominant belief that everyday life should be rationalizedshould be rationalized----reordered reordered according to a technical model of according to a technical model of applied knowledgeapplied knowledgeTendency to surrender responsibility Tendency to surrender responsibility for decisions about the value of for decisions about the value of practice to expertspractice to experts
4848
Consequences of Consequences of ‘‘ee valuationvaluation’’ knowledgeknowledge
Recognize that practitioners are not Recognize that practitioners are not functionariesfunctionaries----simply carrying out specialized simply carrying out specialized tasks on behalf of societytasks on behalf of societyRecognize that practices are not simply Recognize that practices are not simply ‘‘delivery delivery mechanismsmechanisms’’ for providing services to clientsfor providing services to clientsRecognize that it is in practices such as social Recognize that it is in practices such as social work, teaching, etc. that we become aware of work, teaching, etc. that we become aware of what it means to be human, to live together, to what it means to be human, to live together, to flourishflourish
4949
Consequences of Consequences of ‘‘ee valuationvaluation’’ knowledgeknowledge
Recognize that praxis is not simply a matter of Recognize that praxis is not simply a matter of finding suitable means to achieve finding suitable means to achieve pregivenpregivenpurposes (this is technology)purposes (this is technology)Recognize that praxis is matter of collectively Recognize that praxis is matter of collectively making wise choices in pursuit of common goalsmaking wise choices in pursuit of common goalsRecognize that we are morally and politically Recognize that we are morally and politically responsible for our choicesresponsible for our choices
5050
EVALUATION SHAPES OUR EVALUATION SHAPES OUR SENSE OF SELFSENSE OF SELF
5151
At the heart of practices are At the heart of practices are practitioners wrestling with making practitioners wrestling with making countless decisions about what is the countless decisions about what is the right, honest, just, and wise thing to do right, honest, just, and wise thing to do when facing a client, student, patient, or when facing a client, student, patient, or subordinate.subordinate.
If we want better practices, programs, If we want better practices, programs, and policies, then we need to help them and policies, then we need to help them do this kind of do this kind of e e valuation.valuation.
5252
Thank youThank youfor listeningfor listening