evaluation: a process supporting skills and intervention development
DESCRIPTION
EVALUATION: A PROCESS SUPPORTING SKILLS AND INTERVENTION DEVELOPMENT. Réginald Savard, Ph.D . Université de Sherbrooke. Evaluation: The Research. Do you know of many studies aimed at determining… How interventions have been useful to clients? - PowerPoint PPT PresentationTRANSCRIPT
Réginald Savard, Ph.D.Université de Sherbrooke
EVALUATION: A PROCESS SUPPORTING SKILLS AND INTERVENTION
DEVELOPMENT
Do you know of many studies aimed at determining… How interventions have been useful to clients? How interventions have contributed to
compensating for their difficulties? How interventions have helped to meet their
expectations and needs? How interventions have improved their future or
social trajectory?
2
Evaluation: The Research
3
1 •Evaluate for whom?
2 •Evaluate by whom?
3 •Why evaluate?
4 •Evaluate what?
5 •Evaluate when?
6 •Evaluate how?Eval
uatio
n
Evaluate for whom?
4
Intervention
EMPLOYER (manager)
COUNSELLOR
CLIENT
Evaluate for whom?Organizations Evaluate skills and performance Evaluate learnings post trainingClients Evaluate effects of interventions (on the basis of expectations and
needs) Evaluate level of satisfactionCounsellors Evaluate effectiveness of interventions Evaluate factors during the intervention process
5
Evaluate by whom?
6
The actors concerned, their motivations, their interests…
Evaluate by whom?
7
The actors concerned, their motivations, their interests…
Evaluate by whom?
8
The actors concerned, their motivations, their interests…
Evaluate by whom?
9
The actors concerned, their motivations, their interests…
Evaluation for CounsellorsA FEW ISSUES
Satisfaction with service offered Job enrichment Work satisfaction
Professional skills Self-esteem, humiliation, shame Feeling of personal efficacy Etc.
10
Who evaluates? Who evaluates whom?
Intervention
EMPLOYER (manager)
COUNSELLOR
CLIENT
11
External evaluator
3 perspectives Internal (self-evaluation)• Counsellors and clients Organizational External
12
Why evaluate? A phrase from the Viennese philosopher Ludwig
Wittgenstein. What he meant, so far as I understand him, was that
any thinking person sees the world like a fly looking out of his bottle.
That's to say that we see the world clearly, but we're limited by confines beyond our control: in the fly's case, the bottle.
In the human's case, our singular experience.
Why evaluate?
13
Why evaluate?
14
15
Why evaluate?To grasp the reality of individuals in constant
interaction with their environment and to understand the relationships within that environment.
To grasp the reality of individuals seeking to meet their needs and the ability of their environment to meet those needs. To evaluate factors related to the individual and those related
to the environment. To evaluate the individual’s ability to meet his/her needs in
his/her environment.
Why evaluate? Through evaluation, counsellors (or the organization)
learn about themselves (or the client), their functioning, the meaning of their results, their processes (Bouvier, 1998).
Evaluation practices encourage counsellors to seek to gain better self-knowledge, particularly through results analysis, by engaging in reflection in order to better understand the processes implemented.
Evaluation makes it possible to understand what they have done, what they are in the process of doing, what they plan to do, and what they will do.
16
Why evaluate?
To situate the person in his/her developmental trajectory in relation to the different spheres that help to explain his/her problem and adaptation, as well to identify his/her needs (Pronovost and Moreau, 2004).
Evaluation is necessary to gain a better knowledge and understanding of the problem situation and to make better decisions about the best intervention (Alfoldi, 1999).
17
Why evaluate? Evaluation reveals the person’s potential for change
and ability to adapt (Pronovost and Moreau, 2004). Evaluation may help to activate a change process with
clients Become aware of their abilities Become aware of their difficulties Formulate new ideas Propose methods never before contemplated Identify worthwhile projects
Why evaluate? Evaluation makes it possible to step back, which affords some
protection for counsellors who are very involved in the problem situation (Alfoldi, 1999).
Evaluation makes it possible to clarify the roles and responsibilities of each person, as well as the intervention targets (Alfoldi, 1999).
According to Alfoldi (1999), evaluation makes it possible to Have a different understanding of the person’s functioning (needs, abilities,
etc.) and sources of obstacles or dysfunction Plan the intervention Develop an action plan.
According to Pronovost and Moreau (2004), Determine the service to be offered.
Why evaluate? Evaluation enables the actors to clarify and
adjust objectives, verify practices, improve action, and help with decision making
Regulation of intervention practices Development of human resources Management of training plan Support for, or piloting of, quality initiatives Definition of an organization’s strategic directions
20
Why evaluate? Evaluation makes it possible to identify the client’s
initial functioning and determine whether that functioning improved following the intervention.
Evaluation is a way of taking a step back from a client’s problem that sometimes makes counsellors emotionally vulnerable.
21
Evaluation: a process
OBSERVER
MEASURER
ANALYZER
Intervention
22
Why evaluate?
This process should make it possible to plan, carry out, and re-evaluate the intervention (Hayes, Nelson and Jarret, 1987).
Evaluate what? Evaluation is a (dynamic) process through
which we prepare (identify), observe, and collect useful information (obtain) and synthesize it in the form of advice or judgements (reviews) for the purpose of disseminating it to others (counsellors, employers, decision makers, etc.). Oriented towards a specific goal (or goals), making it possible to judge possible decisions.
23Stufflebeam et al. 80
Evaluate what? Process (1) through which we define (2), obtain (3) and
provide (4) information (5) that is useful (6) and enables us to judge possible decisions (7)1. process = ongoing activity2. define = identify pertinent information3. obtain = collect, analyze, measure data4. provide = communicate those data5. information = facts to be interpreted6. useful information = that satisfies relevance criteria7. possible decisions = orientation actions, etc.
24Stufflebeam et al. 80
Evaluate what? Evaluating the quality of the product
delivered means asking questions about the criterion that defines that quality.
Quality is meeting client expectations and needs.
Evaluating the quality of the product delivered is based on the idea of querying resources about their ability to meet client expectations and needs.
25
Evaluate what? Relevance of interventions in relation to the client’s
expectations and needs and the organization’s priorities. Effectiveness, i.e., the degree to which interventions were
(or should be) carried out and their impacts Impact evaluation is a systematic identification of the positive or
negatives effects, intentional or not, on different elements generated by an activity, in this case a program or project.
Efficiency, i.e, the degree to which interventions were (or should be) carried out given the resources available.
26World Development Bank
Evaluate what? Evaluation requires selecting the subject of the evaluation
and the most appropriate tool. There are a wealth of tools available, and that should make it
possible to conduct an analysis that is as realistic and dynamic as possible of the person in terms of his/her career path (Birk and Cassereau, 2003).
Wealth of kaleidoscopic pictures (changing and moving).
Evaluation must be able to be argued, justified, and consequently based on a certain number of parameters or variables that can be identified (Steffatre and Pachoud, 2003).
Evaluate what? Combine the subjective and the objective to avoid a
distancing from the actual experience of the clients and the counsellor (Lecomte and Savard, 2003).
According to Patterson (2002), Evaluation forces us to specify the topic to be discussed to
avoid confusion. Evaluation requires systematic logic in the face of reality,
corroboration of intuition through observation. To intuition and logical discovery, add experience and
knowledge through empirical demonstration.
28
Evaluate what? Combine the subjective and the objective to avoid a
distancing from the actual experience of the clients and the counsellor (Lecomte and Savard, 2003).
Evaluation forces us to specify the topic to be discussed to avoid confusion.
Evaluation requires systematic logic in the face of reality, corroboration of intuition through observation.
What are the factors that help to explain the results obtained?
Contribution of counsellor’s characteristics Contribution of client’s characteristics Contribution of techniques
29
Heppner and Heppner, 2000; Swanson, 2002
30
Specific elements: 8%(Techniques)Specific interventions
General factors: 92%(Common factors) Counsellor variables Client variables Working alliance
Evaluate what?
Specific Genéral0
10
20
30
40
50
60
70
80
90
100
Effectiveness
Wampold, 2001
General
31
Evaluate what?
Client variables
Counsellor variables
RelationshipTechniques
Evaluation
of intervention
Evaluate when?
32
Time 1
Before
Time 2
During
Time 3
After
Time 4
3 months6 months12 monthsEtc.
INTERVENTION
Evaluate how?Qualitative approach: evaluation through a
combination of data collection and information processing techniques to determine significance
Quantitative approach: evaluation using standardized toolsSystematic observation gridsQuestionnairesStandardized tests
33
Evaluate how? Standarized tools would be required to guide the
intervention effectively. The criteria for selecting an evaluation instrument are
as follows:It must be flexible.It must not affect the relationship between the
counsellor and the client.It must take into account several factors in order to
reflect the complexity of the intervention.
Beutler and Martin, 2002
Evaluate for whom, by whom, why, what, when, and how?
Evaluation of intervention
InputWho?
ProcessWhat?
ResultsFor whom?
35
Intermediate results
Perhaps the essence of research can be summed up in this quotation from Schopenhauer:
“The task is not so much to see what no one has yet seen but to think what nobody has yet thought about that which everybody sees.”
36