when the results of evaluation can make a difference… dr. liljana rihter, faculty of social work,...

WHEN THE RESULTS OF WHEN THE RESULTS OF EVALUATION CAN MAKE A EVALUATION CAN MAKE A

DIFFERENCE…DIFFERENCE…

Dr. Liljana Rihter, Faculty of Social Work, University of Ljubljana, Slovenia

EFTC, June 6th, 2007

STRUCTURE OF PRESENTATIONSTRUCTURE OF PRESENTATION

What is the understanding of term evaluation?

Do we all have the same needs and interests as regard evaluation?

Establishing the evaluation system for social protection programmes in Slovenia

1. Typology of the social protection programmes2. Evaluation model3. Evaluation criteria4. Measuring instruments

Proposed scenario of evaluation

Conclusions

WHAT IS THE UNDERSTANDING WHAT IS THE UNDERSTANDING OF TERM EVALUATION?OF TERM EVALUATION?

Many definitions of evaluation In narrow sense (evaluation = estimation):

examinations and judgements about accomplishments and effectiveness

In broader sense (evaluation = evaluation research): systematic data collection on potentially broad question sets about programme/project activities, characteristics and effects to be able to judge about programme, to plan future activities.

Source: Patton, 1997

WHAT IS THE UNDERSTANDING WHAT IS THE UNDERSTANDING OF TERM EVALUATION?OF TERM EVALUATION?

Evaluation is:

evaluation OF SOME OBJECT

for some (more or less defined) PURPOSE

with before known and defined CRITERIA.

DO WE ALL HAVE THE SAME NEEDS AND DO WE ALL HAVE THE SAME NEEDS AND INTEREST AS REGARD EVALUATION?INTEREST AS REGARD EVALUATION?

The need to evaluate:Outside tensions:

EU, UN, WHO,…Financers of programme (state)

Inside tensions:Professionals: ethically obliged to do the best

and professionally correct work.

Money, control of resources, costs, results

Control of procedures and processes


The goals of financers:To know which programmes are ‘better/worse’,

to allocate the money to the best ones ‘better’: efficient (comparison of inputs and results),

effective (comparison of goals of programme with real outcomes)

Evaluation should enable comparison of outcomes of (rather) similar programmes

SUMMATIVE EVALUATION (to obtain the judgement on value, purpose, efficiency and effectiveness of programme)


The goals of professionals carrying out the programme:To be informed about procedures and

processes in programme Is concrete procedure implemented in the planned

manner; do we perform the work ‘well’?Evaluation should enable researching

procedures and processes, to obtain the information to plan future activities

FORMATIVE EVALUATION (change the programme)


The goals of users of programme:To receive appropriate services in comparison

with their needs Satisfaction with the services, benefits

Evaluation should enable to monitor the changes

EVALUATION OF GOALS, OUTCOMES (are the goals of users in concordance with programme goals; are they achieved)

DO WE ALL HAVE THE SAME NEEDS AND DO WE ALL HAVE THE SAME NEEDS AND

INTEREST AS REGARD EVALUATION?INTEREST AS REGARD EVALUATION?

Problems if evaluation responded only to the needs of one stakeholder:

Financers would not obtain the data on comparison of outcomes of programmes; which render impossible to efficiently plan the politics

For professionals the evaluation would be merely control

The users could think that they are solely ‘means of production’


Evaluation should be a response to the needs of as many stakeholders as possible; therefore following is needed: To create a model, encompassing all main

questions Planning and implementing evaluation in

accordance with the scientific criteria of objectivity, reliability and validity.

COMPLEX EVALUATION

ESTABLISHING THE EVALUATION SYSTEM

THE IMPETUS FOR ESTABLISHING THE SYSTEM TO EVALUATE THE IMPLEMENTATION OF SOCIAL PROTECTION PROGRAMMES IN SLOVENIA:- necessary adjustments of systems of social protection in Europe;- Slovenian legislation (enabling the function of an integrated and transparent social protection system)- evaluations have been carried out according to different models, diverse criteria,…THE AIMS OF THE SYSTEM:- to enable a comparison of effects between similar programmes (requested by the Ministry)- to provide the professionals with feedback information to enable them to plan and direct their future work

Typology of the social protection programmes

REVIEW OF THE SOCIAL PROTECTION PROGRAMMES AND TYPOLOGY OF THE PROGRAMMES (Rode, 2001)

momentary (transitory) programme

prevention correction(preventive (therapeutic programme) programme)

permanent programme

TC

Typology of the social protection programmes

DURATION dimension:- momentary, one-off services- transitional programmes- permanent programmes

PREVENTION – CORRECTION dimension- purely informative programmes- combinations of preventive and curative programmes- therapeutic programmes

Evaluation model

(Yates, 1999)

SOURCES: means and personnel needed to carry out programme

PROCEDURES: activities and services which contribute to realising the programme’s aims

PROCESSES: all changes in the users, reactions, eventsOUTCOMES: changes in psychological and social

statuses caused by the programmeThe evaluation of LINKAGES between sources,

procedures, processes, outcomes

sources outcomesprocedures processes

Evaluation criteria

ESTABLISHING THE CRITERIA: necessity of co-operation of all parties involved in the implementation of social protection programmes (users, providers, financier)

- The general criteria (to measure the aims from the National Programme of Social Protection):- The quality of life- Strength perspective- Normalisation

- Special criteria (encountered by various kinds of social protection programmes)- Functional ability- Therapeutic effects

- Specific criteria (formed by individual programmes for self-evaluation)

EVALUATION CRITERIA – analysis of criteria in the programmes, co-financed by the Ministry

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

programi za urejanje socialnih stisk, povezanih z uživanjem akoholika

kratkotrajno dnevno in celodnevno obravnavo in oskrbo otrok in mladoletnikov…

sprejemališča in zavetišča za brezdomce

materinski domovi…

stanovanjske skupine ipd. za osebe s težavami v duševnem zdravju ...

bivalne skupnosti za neodvisno življenje invalidov…

specializirani preventivni programi za otroke s težavami v odraščanju …

programi ipd. za uživalce drog,

psihosocialna pomoč žrtvam nasilja

zmanjševanje socialne izključenosti starih

svetovanje preko telefona...

kakovosti življenja normalizacija funkcionalna sposobnost terapevtski učinki perspektiva moči več konceptov skupaj ostalo - nerazvrščeno

Evaluation criteriaEvaluation criteria

evaluation criteria

0%

20%

40%

60%

80%

100%

120%

all target group -drug users

other

strength perspective

therapeutic effects

functional ability

normalisation

quality of life

Evaluation criteriaEvaluation criteria

DEFINTION OF SPECIAL AND SPECIFIC CRITERIA

Focus group discussion with:- providers of programmes and services’ users;- direct involvement into establishing the system to evaluate social protection programmes.

Purpose: - to obtain insight into concrete aims; to form

criteria for measuring quality of work and achievements of aims.

Evaluation criteriaEvaluation criteriaDEFINTION OF SPECIAL AND SPECIFIC CRITERIA in

programmes: therapeutic communities and other programmes which offer accommodation for drug users; counselling and social rehabilitation centres of illegal drug users who need everyday treatment

Participants: 4 representatives of 3 different programmes

Structure and results of discussion:- services and goals of activities: various, depend on

interference into users life (→ separated discussions)- existing methods of self-evaluation; identification of

common criteria (→ select one ‘good’ already existing methodology): no minimum agreement on how to proceed



Positive effects of suggestions: saving the time and money (no need to develop new methodology, measuring instruments etc.)

Negative effects of suggestions: - professionals from other organizations and programmes might not adopt the ‘forced’ methodology and criteria- users and financiers would not have the right to present their criteria



Necessary steps:- to follow principles of dialogic evaluation - the minimum evaluation criteria should be agreed (each

programme can have also its own, additional criteria)

Positive experience form other groups of programmes:

- agreement on possible criteria was feasible in other programmes, where also next steps have been already done

Measuring instruments

Prepared on the basis of:- general criteria- criteria set by the focus group discussions.

Three questionnaires for measuring general criteria:- Questionnaire for measuring the quality of life

(Lancashire Quality of Life Profile)- Questionnaire for measuring the use of normalisation

principles- Guidelines for the interview for the use of the strength

perspective.

THE PROPOSED SCENARIO OF PROGRAMME EVALUATION

1. GENERAL PRINCIPLES- To ensure the highest possible level of

objectivity (several evaluators for one programme, comparison of their estimations).

- To ensure dialogue between evaluators and the representatives of organisations (to reveal possible reasons for good/bad results).

- Two similar programmes are evaluated simultaneously to ensure the comparability of evaluation.

THE PROPOSED SCENARIO OF PROGRAMME EVALUATION

2. THE SCENARIO OF EVALUATION- Two similar programmes are evaluated by two

trained evaluators.- Every evaluator has a collaborator (a service

provider). - Each of four participants (two evaluators and

the two co-workers) estimates both programmes.

- The evaluators discuss the data and estimations, write an evaluation report and present it to both organisations and to the Ministry.

PILOT EVALUATION

PURPOSE:

- to test feasibility of the uniform procedure of the proposed evaluation system

- to test measurement instruments

PILOT EVALUATIONEXAMPLE OF PILOT EVALUATION FROM THE PROGRAMME Institutional

Care for the AgedPilot evaluation proceeded as follows:Introductory meeting (October 2004):

- purpose of evaluation, evaluation procedure and method- review of proposed measurement instruments, critical assessment of the feasibility of questionnaire- arrangements about data collection

Data collection for users upon admission (October 2004 – January 2005)Second joint meeting (January 2005):

- report on data collection procedures, problems- the way of collecting the data with the questionnaire for users upon the evaluation- presentation of the collection of data on resources, procedures, processes and other criteria

Two visits in each home (March – May 2005)- review of the documents- interview with the representative of the home- survey of the users- data collection and fill in customised form of Yates (Yates, 1999)

Data processing (January – May 2005) and preparation of reportReport sent to all four evaluators (beginning of June 2005)Harmonisation meeting (end of June 2005) and final report

CONCLUSION

- Evaluation scenario is feasible in practice- Problems: measurement instruments, data

collection

- Constant review of criteria is necessary- More involvement of users

- Importance of giving value not only to efficiency but also to effectiveness

LITERATURE AND SOURCESLITERATURE AND SOURCESD.E. Chambers, K.R. Wedel,M.K. Rodwell (1992), Evaluating Social

Programs. Massachusetts: Allyn and BaconJ. Cheetam, R. Fuller, G. McIvor, A. Petch (1992), Evaluating Social

Work Effeciveness. Buckingham: Open University Press.L.L. Martin, P.M. Kettner (1996), Measuring the Performance of

Human Service Program. London: Sage Publications.M.Q. Patton (1997), Utilization-Focused Evaluation. London: Sage

Publications. L. Rihter (2004), Evalvacije na področju socialnega varstva in njihov

pomen za prilagajanje sodobnih držav blaginje na izzive globalizacije. Univerza v Ljubljani: Fakulteta za družbene vede (doktorska disertacija).

N. Rode, L. Rihter, B. Kobal (2006), Evalvacija programov v socialnem varstvu: model in postopek izvedbe. Ljubljana: Fakulteta za socialno delo, Inštitut RS za socialno varstvo.

M. J. Smith (1990), Program Evaluation in the Human Services. New York: Springer Publishing Company.

B.T. Yates (1996), Analysing Costs, Procedures, Processes and Outcomes in Human Services. London, New Delhi, Thousand Oaks: Sage Publications.

when the results of evaluation can make a difference… dr. liljana rihter, faculty of social work,...

Documents