facilitator™s workshop guide - who | world health...

1Facilitator�s workshop guide

WHO/MSD/MSB 00.2j

Facilitator�sworkshop guide

Evaluation of costs and effects ofpsychoactive substance use treatment

2 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2j

WHOWorld Health Organization

UNDCPUnited Nations International Drug Control Programme

EMCDDA European Monitoring Center on Drugs and Drug Addiction

World Health Organization, 2000c

This document is not a formal publication of the World Health Organization (WHO) and all rights are reserved bythe Organization. The document may, however, be freely reviewed, abstracted, reproduced and translated, inpart or in whole but not for sale nor for use in conjunction with commercial purposes. The views expressed indocuments by named authors are solely the responsibility of those authors.


WHO/MSD/MSB 00.2j

Acknowledgements

The World Health Organizationgratefully acknowledges the contri-butions of the numerous individualsinvolved in the preparation of thisworkshop and workbook series, in-cluding the experts who provideduseful comments throughout itspreparation for the Substance AbuseDepartment, directed by Dr. MaryJansen. JoAnne Epping-Jordan(Switzerland) wrote this Facilitator'sWorkshop Guide and designed the5-day workshop from which theguide is based. Maristela Monteiro(WHO’s Substance Abuse Depart-ment) also contributed to its devel-opment. Brian Rush of Virgo Plan-ning and Evaluation Consultants(Canada) contributed materials forcreating Programme Logic Models,and provided general input on theworkshop agenda.

Financial support for this projectwas provided by the United NationsDrug Control Programme (UNDCP),the European Monitoring Centre forDrugs and Drug Abuse (EMCDDA),and the Swiss Federal Office ofPublic Health.


WHO/MSD/MSB 00.2j

Table of contents

Background and goals of overall project 7

Introduction to the workshop 7

Structure of this guide 7

Structure of the overall project 8Summary of the workbook component 9Summary of the workshop component 10Summary of the workshop follow-up component 10Summary of the project evaluation component 11

Pre-workshop planning 12

1. Preparing participants 12Selecting participants 12Working productively with participants 12Giving participants advance information 13

2. Preparing for facilitation 13Qualities of a good facilitator 14Working with your team to maximise effectiveness 14Working cross-culturally 15During the workshop 15A final word... About participant ambivalence 16

3. Room structure, equipment, and materials 18Room structure 17Equipment and materials 17Overhead transparencies 17

4. Suggested workshop schedule 18

Day 1 20

Opening activities 20

Presentation: introduction of workbooks 21

Exercise: fears and expectations 21

Site presentations of local treatment 22

Presentation: why is treatment evaluation important? 23

Exercise: the pros and cons of evaluation for me 23


WHO/MSD/MSB 00.2j

Day 2 24

Presentation: the 8 steps of planning an evaluation 24

Presentation and exercise: identify andprioritise evaluation needs 25

Presentation: describe your programme for evaluation 25

Exercise: describe your programme for evaluation 28

Day 3 29

Presentation and exercise: define your evaluation questions 29

Presentation and exercise: determine your evaluation measures 30

Day 4 32

Presentation and exercise: prepare a data collection plan 32

Presentation and exercise: ensure thatyour resources are sufficient 33

Presentation: the 6 steps of implementing an evaluation 34

Exercise: prepare your site presentation 35

Day 5 36

Presentation: summary 36

Site presentations of evaluation plans 36

Presentation: writing an evaluation plan 37

Closing activities 38

Problems and solutions 39

AnnexesAnnex 1: How to describe your local treatment 42

Annex 2: Questionnaires 44

Annex 3: Written evaluation plan criteria and checklists 74

Annex 4: Overhead prototypes 82


WHO/MSD/MSB 00.2j


WHO/MSD/MSB 00.2j

Background and goalsof the overall project

Introduction to the workshopThis facilitator’s workshop guide provides theessential information to organise and imple-ment a training workshop on the Costs andEffects of Treatment for Psychoactive Sub-stance Use Disorders. During the workshop,participants will be guided by facilitators toplan an evaluation of their substance usetreatment programme or network. Facilitatorsassist participants to design an evaluation thatsuits their unique resources and needs. Theproposed training workshop is fundamentallyintegrated with a complementary nine-volumeworkbook series that explains how to plan andimplement treatment evaluations. Many presen-tations and exercises are taken directly from theworkbook series. The workbooks are describedin greater detail on page 9 of this guide.

The workshop schedule is planned for fivedays. Experience suggests that five days aresufficient, but not excessive, to plan evalua-tions. Participants are introduced to the gen-eral framework for evaluation on the first day.The bulk of their evaluation planning happensduring the second through fourth days. Onthe fifth and final day, participants present theirevaluation plans and discuss next steps.

The immediate goals for the workshop are:

• To introduce participants to the workbookseries

• To guide participants to plan their owntreatment evaluation

Secondary or longer-term goals of the work-shop are:

• To enhance participants’ attitudes towardstreatment evaluation

• To enhance participants’ general abilitiesto plan and implement treatment evalua-tions

As a facilitator, it is important to familiariseyourself with this guide, and the workbookseries from which it is derived, well in ad-vance of the workshop. Begin your prepa-ration for the workshop by reading thisguide. After you have finished reading, youcan begin specific preparations with yourfacilitation team and review the workbookseries.

Structure of this guideThis guide is divided into several sections.“Pre-Workshop Planning” covers the essen-tial information you need to prepare for theworkshop. Detailed explanations of the proposedpresentations and exercises, divided into fivedays, are presented next. This is followed by asection entitled “Problems and Solutions,” which

presents suggestions for managing commonproblems that arise during workshops. Theannexes provide supplementary forms foryour use before, during, and after the work-shop. Each of the annexes is explained withinthe main body of this guide. Refer to the Tableof Contents for more specific information.


WHO/MSD/MSB 00.2j

The programme logic model illustratesthe overall project structure and goals.Project components include the work-books, the training workshop, workshop

Structure of theoverall project

follow-up, and project evaluation.Implementation objectives, short-termgoals, and long-term goals are listed un-der each component.

• To enhance sites’ capacities to conduct high quality treatment evaluations

EvaluationWorkshopfollow -upWorkshopWorkbook

Projectcomponents

• To assess sitesthat receivethe workbooksand/or attendthe workshop

• To gatherinformationabout theworkshop’sandworkbooks’effectiveness

• To introduceworkbooks tosites

• To enhancesites’ abilities touse workbookseffectively

• To guide sites toplan a localtreatmentevaluation

• To transferneededknowledge andplanning skills

• To provide aclear anddetailed guide

Implementationobjectives

Short-termgoals

• To support sitesto effectivelyimplement theirtreatmentevaluations

Long-termgoals

• To integrate evaluation systematically with treatment decision making

• To improve quality, coverage, and cost-effectiveness of treatment

• To improveworkbooks andworkshop

• To prepare local evaluation plans incollaboration with relevant stakeholders


WHO/MSD/MSB 00.2j

Summary of the workbook componentThe first component of the project is the workbook series. The first threemanuals are introductory and foundation workbooks, whereas the nextsix manuals are specialised workbooks.

Introductory WorkbookFramework Workbook

Foundation WorkbooksWorkbook 1: Planning EvaluationsWorkbook 2: Implementing Evaluations

Specialised WorkbooksWorkbook 3: Needs Assessment EvaluationsWorkbook 4: Process EvaluationsWorkbook 5: Cost EvaluationsWorkbook 6: Client Satisfaction EvaluationsWorkbook 7: Outcome EvaluationsWorkbook 8: Economic Evaluations

The workbooks are designed to be used inconjunction with the training workshop de-scribed in this facilitator guide. The workbooksalso can be used without workshop training,but better outcomes are expected whenparticipants attend a training workshop in ad-dition to reading the workbooks. This is be-cause training workshops provide participantswith individualised attention through the evalu-ation planning process. Data are being col-lected currently regarding whether this hy-pothesis is supported.

Facilitators should keep these points in mind:

• The foundation workbooks are the mainfocus of the workshop, and should be usedby all participants.

• The specialised workbooks provide moredetailed information about specific types ofevaluation. Most participants will use onlyone or two specialised workbooks during theweek. The choice of specialised workbookwill depend on the evaluation question.

All workbooks contain “It’s your turn” par-ticipatory exercises. These exercises are de-signed to assist participants to apply work-book information to their own evaluations.“It’s your turn” exercises are incorporated intothe suggested workshop curriculum.

Most workbooks contain case examples,which are real life accounts of treatmentevaluations conducted around the world.The cases can be useful learning tools.They show how evaluators have conducteddifferent kinds of evaluations, logisticalproblems they overcame, methods theyused, and how results were applied to im-prove treatment services. Editorial com-ments about each of the cases also are pro-vided. An “It’s your turn” exercise aftereach case example will help you to reflectthem critically.

You should review all workbooks, including“It’s your turn” exercises and case examples,in advance of the workshop.

1 0 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2j

Summary of the workshop componentSecondary or longer-term goals of the work-shop are:

• To enhance participants’ attitudes towardstreatment evaluation

• To enhance participants’ general abilitiesto plan and implement treatment evalua-tions

The workshop is fundamentally integratedwith the workbook series. Many presenta-tions and exercises are taken directly fromthe workbooks. Participants should have theworkbook series available throughout theworkshop, and should be instructed to re-fer to them as appropriate during the pre-sentations.

The second component of the overall projectis the training workshop. The workshop is de-signed to be conducted over the course offive days by a facilitation team (for a sug-gested schedule, see page 23 of this guide).Detailed information about workshop prepa-ration and curriculum is presented later in thisfacilitator’s guide.

The immediate goals for the workshopweek are:

• To introduce participants to the workbookseries

• To guide participants to plan their owntreatment evaluation

Summary of workshopfollow-up componentThe third component of the overall project isworkshop follow-up. The main goal of theworkshop follow-up is to support participat-ing sites in implementing their treatment evalu-ations.

Frequently, participating sites are not finishedcompletely with evaluation planning by the endof the workshop. They may need additionaltime and/or to consult with others beforefinalising their plans. Follow-up and assistanceduring this planning phase are often useful.

After planning is complete, sites also mightneed assistance to implement their evaluations.Questions or problems could arise with datacollection strategies, data analysis, or usingresults effectively. Facilitators and/or otherconsultants can be quite helpful in many ofthese situations.

Follow-up can occur via telephone, email, fax,or post. Site visits also are possible, depend-ing on available funding.

1 1Facilitator�s workshop guide

WHO/MSD/MSB 00.2j

This project is currently being evaluated byWHO/ UNDCP/ EMCDDA. Results fromthe project evaluation will be used to improvethe workbooks and workshop curriculum, withthe goal of creating written materials for gen-eral dissemination with established utility andeffectiveness. Results also will be used toclarify whether the workshop produces en-hanced outcomes, relative to those who re-ceive only the workbook series.

Local or regional evaluation of the workshopand workbook series also is important.Through your own evaluation of the work-shop and workbook series, specific problemsfor your region, language, or culture can beidentified. Local evaluation also can assesswhether your participants are satisfied withvarious components of the project, as wellas whether their participation helps themto plan and implement meaningful treatmentevaluations.

Follow the basic steps for planning and imple-menting treatment evaluations (as outlined inthe foundation workbooks) when designingyour local evaluation of the workshop andworkbook series. Make sure that you under-stand your evaluation questions, have a spe-cific plan for data collection and data man-agement, and know how you will use yourresults to improve the workshop and/or work-book series.

The participant questionnaires being used byWHO/ UNDCP/ EMCDDA to evaluate theoverall project are reproduced in this guide(pages 79 to 121) for your convenience.

There are pre- and post-workshop question-naires, designed to be completed at the be-ginning and end of the workshop. The pre-workshop evaluation questionnaire can be sentto participants in advance of the workshop,or completed during opening activities. Theadvantage of sending questionnaires in ad-vance (and asking for their return in advance)is that they can provide valuable informationabout the participants that can be useful forworkshop planning. If participants have for-gotten to complete the questionnaire, ask themto finish it during opening activities. For par-ticipants who receive the workbook series,but do not attend a training workshop, thereis a “Pre-Workbook Questionnaire” on page91 of this guide. This questionnaire is designedto be completed and returned by participantsbefore they receive the workbooks. In addi-tion, there are follow up questionnaires aboutthe workbook series, designed to be com-pleted by all participants six months and 12months following participation in the work-shop and/or receipt of the workbook series.These follow-up questionnaires are locatedstarting on page 96 of this guide.

Comments from organisers and facilitators tothe World Health Organization are most wel-come, and should be addressed to:

Dr. Maristela MonteiroSubstance Abuse DepartmentWorld Health Organization20 Avenue Appia1211 Geneva 27 · SwitzerlandFax: ++41 22 791 48 51E-mail: [email protected]

Summary of the projectevaluation component


WHO/MSD/MSB 00.2j

Pre-workshop planning

Preparing participantsSelecting participantsWorkshop participants should be chosen bysite with the assistance of selected criteria.A site could be a country, a defined regionwithin a country, or a specific treatment ser-vice. An important criterion for selection isthat each site should have three individualsprepared to commit to the evaluation projectand work together — one person involved inplanning and funding treatment programmes,another involved in direct service delivery, andthe third with technical and research skills.Each site should be asked to bring three par-ticipants, representing these different sectors,with the purpose of discussing and reachingagreement on the evaluation to be undertakenand going through the planning process dur-ing the workshop. Regardless of background,

participants should hold positions in which theyhave decision making authority.

Group definitions for this facilitator guide:

Individual level: each participant worksalone (1 person)

Site level: participants work with the otherparticipants from their treatment site (usually3 people)

Triplet level: three sites work together (usu-ally 9 people)

Group level: all participants meet together(everyone)

Working productively with participantsPrior experience indicates that grouping sitesinto larger groups of three sites each, or trip-lets, is beneficial for exchange between sitesand monitoring by facilitators. Throughout theworkshop, triplets meet with assigned facili-tators to exchange ideas and receive feed-back about their evaluation plans. Triplets cre-ate the opportunity for cross-site peereducation, and allow participants to draw fromeach other’s knowledge and experience to

create high quality evaluation plans. Facilita-tors should foster cross-site discussion andsupport within the triplets. By doing so, theyremove themselves from the role of expertand rely instead on participants to help eachother. Over the course of the workshop,cross-site collaboration usually grows stron-ger, and lasting collaborations are frequentlyestablished.

1.


WHO/MSD/MSB 00.2j

Ideally, one facilitator should be assigned toeach triplet (three sites) for the duration ofthe workshop. Triplet formations and facilita-tor assignments should be made by facilita-tors in advance of the workshop.

As a general guideline, minimal time shouldbe spent on lectures. The presentationsshould be short and focused on orienting par-ticipants to the exercises to be completed.Assigned facilitators should be responsible for

guiding their sites through the exercises andproviding ongoing, individualised feedbackduring evaluation planning. Facilitators shouldsee their role less as expert and more asassistant.

Most participants will become fatigued occa-sionally during the workshop week. Be alertfor signs of tiredness and take breaks asneeded.

Giving participants advance informationIt is usually helpful to provide participants withbasic information about the workshop beforethe actual training. Accordingly, in advanceof the workshop, it is suggested that you sendparticipants copies of the Framework Work-book, which presents the general model fortreatment evaluation and some introductoryinformation about the workbook series. Alsosend the workshop agenda. Participants canread this information in a relaxed manner, andin doing so, begin to orient themselves towardsthe task of evaluation planning.

Participants also should be encouraged tobegin thinking about their local treatmentservice or network in a systematic way. Oneway to accomplish this goal is to ask eachsite to create a written summary of their

local treatment in preparation for the work-shop. A copy of detailed written instruc-tions for completing this task, entitled “Howto describe your local treatment”, is locatedon page 74 of this guide. Once sites finishthe written summary, they should return itto the organising facilitator. Upon receiptof the summary, the organiser should re-quest that the site also prepare a 15 minuteoral presentation of the summary informa-tion, to be presented during Day 1 of theworkshop. The thought of an oral presen-tation may create a considerable amount ofanxiety for some participants, so it is sug-gested that the organiser wait to requestthis task until the written summary is com-plete. The organiser also should emphasisethat the oral presentation can be informal.

Preparing for facilitationThis workshop is designed to be facilitatedby a team of qualified individuals. Why ateam? Team facilitation increases the over-all knowledge and skill base available to par-ticipants. It also allows facilitators to giveindividualised attention to participants asthey plan their evaluations. Ideally, thereshould be at least one facilitator for everythree participating sites (or, stated differ-ently, one facilitator per triplet), or a mini-mum of two facilitators if there are less thansix participating sites.

Ideally, the facilitation team should representa diversity of professional and cultural back-grounds. Diversity increases the overallknowledge and skill base of the team, bringsmultiple perspectives to the process, and re-flects the varying backgrounds of partici-pants. Also, many participants will feelmore comfortable with facilitators fromtheir own cultural or professional back-grounds. For similar reasons, inclusion ofwomen (both facilitators and participants)also should be encouraged.

2.


WHO/MSD/MSB 00.2j

Qualities of a good facilitator• Familiarity with the culture, needs,

strengths, and limitations of participants,and respect for individual differences

• Organisation: equipment, supplies, timeschedule

• Ability to strike the delicate balance be-tween being directive and non-directive,and to know when each stance is neededat different points during the workshop

In addition to these qualities, facilitators alsoshould have relevant knowledge and experi-ence in evaluation planning and implementa-tion, and/or a good understanding of typicalbarriers and benefits of treatment evaluation.Formal scientific training is advantageous forcertain types of evaluation, but is no guaran-tee of good facilitation.

• Trust in others and their capacities (bothco-facilitators and participants)

• Good listening skills

• Good presenting skills

• Confidence without arrogance

• Ability to create an atmosphere of confi-dence among participants

• Respect for the opinion of others (not im-posing ideas)

• Flexibility in changing methods and se-quences as needed

• Knowledge of group process, including theability to sense the group’s mood at anygiven time and adjust the programme ac-cordingly

Working with your team tomaximise effectivenessCo-operation is essential among co-facilita-tors: mutual trust; recognition and acceptanceof individual strengths and weaknesses; andan attitude of reciprocal learning and assis-tance. The success of the workshop does notdepend on individual performance, but rather,on how well the co-facilitators work togetheras a team to enhance participants’ evaluationplanning. If problems arise with participantsduring the workshop, or if you feel like youneed help, tell your co-facilitators. The teamshould be able to support one another throughall difficulties.

Well ahead of the workshop, facilitatorsshould familiarise themselves with the sub-ject matter and meet with their team to planthe workshop in detail. Facilitators should dis-cuss their backgrounds and professionalstrengths at this time. Based on this informa-tion, specific facilitators can be assigned tospecific presentations and exercises. If it be-

comes apparent that there is a gap in theknowledge base of the co-facilitators, effortsshould be made to bring a complementaryfacilitator to the workshop (even for only partof the workshop). In the absence of this op-tion, facilitators can make note of questionsthat arise during the workshop and have themanswered by an expert consultant after theworkshop ends.

Throughout the workshop, facilitators shouldmeet for about one hour at the end of eachday. During this time, they should discuss anyproblems or issues that have arisen, as wellas their general impressions about how wellparticipants are progressing with their plan-ning. All problems or issues (among partici-pants or facilitators) should be resolved asquickly as possible. A general discussion anddebriefing among facilitators after the work-shop ends also are recommended.


WHO/MSD/MSB 00.2j

Working cross-culturallyDuring many workshops, participants and fa-cilitators will represent a range of culturalbackgrounds, which are not restricted to na-tionality. Cultural differences also can includedifferences in work environments (university,government, or treatment settings), statuswithin organisations (director or employee),roles within organisations (physicians or ad-ministrators), and perspectives on gatheringinformation (quantitative or qualitative ap-proaches).

As a facilitator, it is important to respect andwork with the values of participants from dif-ferent cultures. The key for overcoming cul-tural barriers during the workshop is to seekbackground information on the cultural prac-tices of participants, to anticipate potentialproblems, and to be willing to adjust your ap-proach as needed. While it is impossible tochange some of your personal characteris-tics, such as your age or nationality, modifi-cations to clothing, presentation style, and fa-cilitation techniques can be made asappropriate.

Facilitators also should be aware of the im-pact of their own cultural preferences. It isnatural to be interested in people who shareyour opinions, native language, or ways ofdoing things. However, when you are a fa-cilitator, it is important to be even-handed witheveryone, and to avoid any appearance ofpreferential regard. Throughout the workshop,you should consider whether you are beingfair to all concerned.

Finally, language barriers might present sig-nificant problems for facilitators and partici-pants from diverse backgrounds. In these situ-ations, remember to speak slowly, and avoidcomplicated sentence structure and/or vo-cabulary. For foreign languages, most peopleread better than they understand oral com-munication. Use the written word to reinforcekey concepts via overhead transparencies,chalk boards, flip charts, and the workbooksthemselves. Pictures or diagrams also can beuseful.

During the workshopBefore the workshop begins each day, facili-tators should review the suggested curricu-lum and make modifications if needed (e.g.,in the case that some sites are workingthrough the material more quickly or slowlythan anticipated).

Facilitators should consult informally with eachother throughout the day, to ensure that ev-erything is progressing smoothly and/or totroubleshoot any problems that might arise.

Each facilitator should take notes throughoutthe workshop about their sites’ progress inevaluation planning. Notes provide useful ref-erences for discussing progress or problemswith other facilitators, and also serve as a goodmemory tool for oneself.

At the end of the workshop, it is useful forfacilitators to compile their notes into a writ-ten summary about each of their sites. These

summaries should document each site’sprogress, and can be shared with workshoporganisers, future site consultants, and projectfunders. The summary should be about 20lines (more or less) for each site. Componentsshould include:

• site background information

• prior evaluation experience of participants

• other skills participants bring to evaluationplanning and implementation

• barriers to evaluation

• type of programme logic model (national,regional, local)

• main evaluation question(s) and how thatrelated to their programme logic model


WHO/MSD/MSB 00.2j

• major issues that sites struggled withduring the workshop, and how (or if) theseissues were resolved

The site summaries should focus on objec-tive information rather than facilitators? sub-jective opinions. Limit your comments to factsabout what happened during the workshopweek. Avoid judgmental statements, compari-sons between groups, and your personal im-pressions.

Finally, remember that facilitators have a spe-cial role during the workshop: to help every-one feel comfortable and valued. There areseveral ways to accomplish this goal:

• take time to converse informally with par-ticipants

• attempt to solve any problems that par-ticipants have, even if they are not relateddirectly to the workshop itself

• organise after hours social events for ev-eryone to attend. If participants have trav-elled a long distance to attend the work-shop, or are not familiar with the region,they may be particularly grateful to attendorganised events.

A final word...About participant ambivalenceParticipants may have mixed feelings aboutaspects of this project. Although possible con-cerns are unlimited in nature, participants mayworry about the viability of doing treatmentevaluations locally, have scepticism about fa-cilitators understanding or supporting them,and/or feel too under-educated to be success-ful in planning and implementing evaluations.Others may feel they are over-qualified andthere is no point in participating.

It is important for facilitators to be aware thatambivalence of one sort or another is com-mon among participants, but that many timesthis ambivalence remains unexpressed. Itis the role of the facilitator to be alert for pos-sible ambivalence, and to have strategies fordealing with it effectively.

One useful method for dealing with ambiva-lence is to increase participants’ motivationfor doing treatment evaluations by helpingthem to see how evaluations help meet theirgoals. One way to do this is to ask partici-pants to discuss their treatment services andobjectives. By listening carefully to their re-sponses, you can gain insight into possiblemotivating factors and help to draw new con-

nections for participants. For example, a sitemay report limited funding for services as amajor problem in its setting, and securing newfunding as a major goal. In this situation, youcan point out how evaluation results can beused to make programmes more efficient andsecure new funding in some cases. Othercommon benefits to evaluation are: finding outwhether programmes are working as hoped,obtaining information that can improveprogrammes, and improving local knowledgeand experience about how to conduct evalu-ations. The key is to personalise motivationalmessages to particular sites as much as pos-sible.

It also is important to understand possible in-hibiting factors (or barriers) to doing evalua-tions. Barriers could include systemic prob-lems, or involve negative attitudes abouttreatment evaluations. For example, partici-pants might think that they are too busy ortoo inexperienced to conduct evaluations.Others might not really want to know if theprogramme is working as hoped, because theyfear change or discontinuation of funding.These factors can be discussed and, if pos-sible, problem-solved during the workshop.


WHO/MSD/MSB 00.2j

Room structure,equipment, and materials

Room structureThe workshop site should have plenty ofspace for sites to be able to spread out andwork separately without disturbing eachother. Auditorium settings are not advisable,however, because of the participatory na-ture of the workshop. If you have more than

three participating sites, consider adjoiningrooms for each site triplet. If you use mul-tiple rooms, one room should be largeenough to hold everyone for group presen-tations. Also, rooms should not be far apartfrom one another.

Equipment and materialsThe suggested equipment (listed below) shouldbe prepared in advance of the workshop.While the recommended equipment is idealfor the various activities, modifications canbe made if needed. For example, chalk boardscould be used in place of an overhead projec-tor or flip charts.

Facilitator equipment

• overhead projector

• acetate sheets for overhead projector

• multicoloured pens for overhead projec-tor

• flip chart (make sure you have plenty offlip chart paper available)

• multicoloured pens for flip charts

• note pads

• pens or pencils

• name tags

Participant equipment

• complete copy of workbook series - 1 setper participant

• note pads - 1 pad per participant

• pens or pencils

• flip chart and paper - 1 chart per site

• coloured construction paper - at least 2different colours (for writing hopes andfears; louder and slower signs: see pages27 and 29) - about 8 sheets per participant

• multicoloured marking pens - 1 set per site

• post-it notes or small pieces of paperbacked with tape (for creating programmelogic model) - 1 package per site

• name tags

Overhead transparenciesfor your use throughout the workshop. Pleasenote that you should review the prototypes inadvance of the workshop, and choose thosethat best suit your needs.

A set of overhead transparency prototypes islocated at the back of this facilitator guide. Ifyou plan to use an overhead projector, theseprototypes can be made into transparencies

3.


WHO/MSD/MSB 00.2j

Suggestedworkshop scheduleThis workshop schedule is based on thepremise that each participating site will sendthree individuals to participate. During theworkshop, participants will be guided, step bystep, through planning a local evaluation oftheir substance use treatment services or net-work. In the process of planning, they will beintroduced to the workbook series, and learnto use it as a resource during and after theworkshop.

The workshop is designed to tailor evaluationplans to participating sites’ different levels ofresources and/or expertise. Evaluation planscould be as simple as a needs assessmentsurvey, or as complex as a full economicevaluation. The key for facilitators is to assistin tailoring evaluation planning to each site’sunique needs and capacity for evaluation.

The workshop format combines brief presen-tations with participatory exercises. Partici-pants listen to formal presentations based onworkbook material, then meet with other par-ticipants from their site to put the informationto work immediately in planning their ownevaluations. Facilitators work closely witheach site throughout the planning exercises.Participants also provide and receive feed-back with other sites via triplet exercises.

The schedule is planned for five days. Priorexperience indicates that this amount of timeis sufficient, but not excessive, to accomplishevaluation planning. Suggested time durationsare listed for each activity. Lunch and otherbreaks can be inserted between activities asneeded. The schedule can be modified forthose who have more or less time available,although it is recommended that you do notalter the sequence of activities.

4.


WHO/MSD/MSB 00.2j

Day 2

Day 3

Day 4

• presentation: summary (15 minutes)

• site presentations of evaluation plans (15 minutes per site)

• presentation: writing an evaluation plan (15 minutes)

• presentation and exercise: overcoming barriers to evaluation (1hour)

• closing activities (2 hours)

Day 1 • opening activities (2 hours)

• presentation: introduction of workbooks (15 minutes)

• exercise: fears and expectations (30 minutes)

• site presentations of local treatment (15 minutes per site)

• presentation: why is treatment evaluation important? (15 minutes)

• exercise: The pros and cons of evaluation for me (1 hour)

• presentation and exercise: the 8 steps of planning an evaluation(1 hour)

• presentation and exercise: identify and prioritise evaluation needs(1 hour)

• presentation: describe your programme for evaluation (1 hour)

• exercise: describe your programme for evaluation (4 hours)

• presentation and exercise: define your evaluation questions (3-4hours)

• presentation and exercise: determine your evaluation measures(3-4 hours)

• presentation and exercise: prepare a data collection plan (2 hours)

• presentation and exercise: ensure that your resources are suffi-cient (1 hour)

• presentation: the 6 steps of implementing an evaluation (1 hour)

• exercise: prepare your site presentation (3 hours)

Day 5


WHO/MSD/MSB 00.2j

Day 1one exercise that moves from the site to thetriplet level. The major components of theday’s suggested agenda are:

• opening activities

• presentation: introduction of workbooks

• exercise: fears and expectations

• site presentations of local psychoactivesubstance use treatment

• presentation: why is treatment evaluationimportant?

• exercise: The pros and cons of evaluationfor me

Each of these components is explained ingreater detail below.

The first day of the workshop has severalgoals, including:

• making introductions and establishing rap-port

• orienting participants to the workshopagenda and goals

• introducing participants to the workbookseries

• learning about each site’s background andcurrent situation

• managing practical details of the workshopweek as needed (e.g., lodging, meals,transportation, site funding)

During the first day, most activities happen atthe group level. At the end of the day, there is

Opening activities(group level)It is a good idea to arrive at the workshop siteearly, to set up the room(s) and arrange yourequipment. Wear a name tag to identify your-self as a facilitator, and give participants theirown name tags as they arrive.

The specific content of opening activities willvary by workshop. Suggested components in-clude:

• opening statements by organisers and hostsite representative

• introduction of facilitators

• introduction of participants (ask partici-pants to describe their professional role andlist the skills that they bring to their evalu-ation team)

• proposed time structure and agenda forthe week

• assignment of sites to specific triplets andfacilitators

• orientation to workshop site (e.g., locationof toilets, telephones, refreshments, etc.)

• any outstanding practical issues (e.g., sitefunding, transportation, etc.)

• pre-workshop evaluation by participants(questionnaire on page 80 of this guide)*

* The pre-workshopevaluationquestionnaire can besent to participants inadvance of theworkshop, or completedduring openingactivities. Theadvantage of sendingquestionnaires inadvance (and asking fortheir return in advance)is that they can providevaluable informationabout the participantsthat can be useful forworkshop planning. Ifparticipants have notreturned thequestionnaire, ask themto finish it at this point.


WHO/MSD/MSB 00.2j

If participants have different primary lan-guages, instruct participants to write thewords slower and louder in their nativelanguages on coloured paper (tell partici-pants which colour to use for each word).Then, at anytime during the workshop, par-ticipants should show the appropriate pa-per if they would like the current speakerto talk slower or louder. This is a funmethod to acknowledge cultural differ-

ences, learn two words from eachparticipant’s language, and facilitate usefulcommunication.

If participants and facilitators share a com-mon primary language, the words louder andslower can be depicted graphically by eachperson. The same rules could apply: show theappropriate paper when a speaker is talkingtoo quickly or quietly.

Presentation: introductionof workbooks(group level)Each participant should be provided with aset of workbooks. One facilitator should ex-plain the workbook series structure, and tellparticipants that the workbooks will be thebasis for the upcoming workshop. Partici-pants should be told that Workbooks 1 and2 provide a solid foundation of general in-

formation about conducting evaluations,whereas the specialised workbooks (Work-books 3 through 8) present detailed infor-mation for different types of evaluations.Facilitators should give participants aminute or two to look through the workbooksbefore proceeding.

Exercise: fears and expectations(group level)

This exercise is a modification of question3 of the Step 1A exercise located in Work-book 1 (page 11). Participants should be giventwo different colours of paper for writing theirfears and expectations, and directed whichcolour to use for each category.

Give the following instructions (available asoverhead prototype in the back of this guide):

Working individually, list on the provided cards1) your expectations about the workshop, andthen 2) your fears or concerns. Use differentcard colours for expectations vs. fears. Writelegibly and use large print, so your cards can

be read from a distance. To protect your pri-vacy, do not put your name on the cards.

When finished, the cards should be collectedby a facilitator and grouped by colour.

Facilitators next should review briefly thecards, and use these as a means to demon-strate similarities and uniqueness of ideas. Theidentity of card writers should not be askeddirectly by facilitators. Instead, cards shouldbe clustered and discussed at the group level.Following the presentation, cards should betaped to the wall and displayed for the dura-tion of the workshop.


WHO/MSD/MSB 00.2j

In past workshops, the following themes havebeen expressed:

Fears

• The workshop will not be relevant

• Home treatment programme/network willnot be motivated to implement an evalua-tion

• There are insufficient funds to implementan evaluation

• There is insufficient time to cover all theworkbooks during the workshop

• Communication and language barriers

• Personal and/or facilitator inexperience

Expectations

• Transmitting knowledge to home region orcountry

• Solving problem of addiction in home re-gion or country

• Increase skills

• To get a simple method of evaluation

• To come back with practical projects

• Learn something concrete

• Enhance future work

• Establish good contacts

• Break barriers of communication

Site presentations of psychoactivesubstance use treatment(group level)

Each site presents briefly (15 minutes) to theentire group an overview of its treatment ser-vice or system. Presentations should be ar-ranged with sites in advance, and should bebased on the written summary created by eachsite (see Preparing Participants on page 12of this guide).

The purpose of the presentations is to obtaininformation about each site’s programmes and

capacity for evaluation. Facilitators alsoshould use these presentations as an oppor-tunity to listen for participant ambivalence andother potential barriers to evaluation. Feed-back from facilitators should be limited at thisstage, however.

Following these presentations, facilitatorsshould be able to begin formulating possibleevaluation questions for each site.


WHO/MSD/MSB 00.2j

Presentation: why is treatmentevaluation important?(group level)

The presentation from this section can bebased on material from the Framework Work-book. Consider using the sections “Whyis treatment evaluation Important”,“Treatment evaluation as feedback”, and“Levels of evaluation” for content. In-clude these ideas:

• historically, policy decisions regardingtreatment have been made without con-sidering evaluation results;

• evaluation is not a technology, nor a col-lection of information or statistics, but anattitude towards change;

• the underpinning of the project is the de-velopment of a healthy culture of evalua-tion, as part of ongoing collection of infor-mation for decision making at theprogramme, service, and system level;

• participants should be ready to accept andimplement changes to their programmesaccording to the results of the evaluations,regardless of their own views on specifictreatments or approaches.

Exercise: The pros and consof evaluation for me(site to triplet level)

Following the presentation, participants shouldbe directed to complete the exercise describedbelow. This exercise should be started at thesite level, then move to the triplet level fordiscussion. The purpose of this exercise is todiscover information about each site’s moti-vators and barriers to successful evaluation,and secondly, to increase their motivation todo evaluation in their setting. Those facilita-tors familiar with motivational interviewingtechniques* may find them useful with par-ticipants at the latter point of this exercise.

* Motivationalinterviewing is anapproach based onprinciples of experimentalsocial psychology, andhas the goal of increasingintrinsic motivation tochange by helping todevelop a discrepancybetween goals andcurrent behavior.Although MotivationalInterviewing wasdeveloped originally forclinical use with peoplewho have alcoholproblems, it is a highlyeffective tool for anyonewho is attempting tonegotiate behaviorchange in any setting(such as facilitators ofthis workshop).For more information,see MotivationalInterviewing: PreparingPeople to ChangeAddictive Behavior byW.R. Miller and S.Rollnick, Guilford Press,1991.

The instructions are as follows (available asoverhead prototype in back of this guide):

1 Each site should work separately and writeon flip chart paper the pros and cons ofdoing treatment evaluation at its local site.Allow 15 minutes to complete this step.

2 Each site should meet with at the tripletlevel to discuss the pros and cons that theywrote.


WHO/MSD/MSB 00.2j

Day 2activities also are included. The suggestedagenda includes:

• presentation and exercise: the 8 steps ofplanning an evaluation

• presentation and exercise: identify andprioritise evaluation needs

• presentation: describe your programme forevaluation

• exercise: describe your programme forevaluation

Each of the activities is described in detail onthe next pages.

The goals for the second day include:

• introducing the basic steps of evaluationplanning

• assisting sites to begin to identify andprioritise their evaluation needs

• teaching a structured method for describ-ing programmes for evaluation(programme logic model)

• assisting sites to begin constructing theirown programme logic models

Today, participants begin working intensivelyat the site level. Some triplet and group level

Presentation and exercise: the 8steps of planning an evaluation(group, site, and triplet levels)This presentation should begin by over view-ing the 8 steps of evaluation planning (as out-lined on page 5 of Workbook 1). Follow-ing this, a more detailed presentation can bemade for Steps 1 and 2.

Step 1, Decide Who Will Be Involved In TheEvaluation, was covered to a large extentduring Day 1. Highlight the information pre-sented on pages 10 to 11 of Workbook 1for participants’ future reference.

Step 2, Assess Your Resources, is an impor-tant but potentially intimidating task for par-ticipants. In your presentation, be sure toemphasise that high quality evaluation CANbe conducted with very few resources. Thekey is to evaluate resources accurately, andthen choose a project that is practical withthe resources available.

If organisers plan to give resources to sitesfor their evaluations, the sites should be madeaware of the type and extent of resourcesavailable. This will help them to evaluate theirsituations correctly (see below), and mayeliminate some concerns about being under-resourced for evaluation efforts.

After you have finished your presentation,direct participants to complete the “It’s yourturn” Step 2 exercise, located on page12 of Workbook 1. The exercise questionsare designed to assist sites to evaluate accu-rately their financial/material, expertise, andtime resources for evaluation. Participantsshould work at the site level, and then moveto the triplet level for feedback and exchangeif time permits. Facilitators should workclosely with sites and triplets during the exer-cise.


WHO/MSD/MSB 00.2j

Presentation and exercise: identifyand prioritise evaluation needs(group, site, and triplet levels)

see the range of possibilities for evaluations,and perhaps to see some common trends inideas. The next step (Step 3: Describe YourProgramme for Evaluation) will help to fur-ther clarify needs and priorities.

After the presentation, participants shouldbe directed to complete the “It’s yourturn” Step 4 exercise, questions 1 and2, located on page 19 of Workbook 1.They should complete this exercise at thesite level. If time permits, triplet discussionof the exercise should follow. As with allexercises, facilitators should work with theirsites and triplets to provide assistance andfeedback as needed.

The following points might be useful to in-clude in this presentation. It is natural forsite participants to have a diversity of ideasabout what to evaluate. For example, ad-ministrators may be most interested in costefficiency, while clinicians may be inter-ested in the benefits of a new treatment. Itis important for all participants from eachsite to understand one another’s perspec-tives, then begin to come to a decision abouttheir overall needs for evaluation. This stepof evaluation planning (Step 3: Evaluate andPrioritize Evaluation Needs) is a first steptowards accomplishing this goal. Partici-pants should be encouraged to explorefreely different options for evaluation. Thegoal at this stage is to assist participants to

Presentation: describe yourprogramme for evaluation(group level)This is your opportunity to explain the pro-cess and benefits of creating a programmelogic model (PLM). This step of the work-shop is particularly important, because thePLMs will form the basis for the evaluationplans created later in the week. The PLMshelps sites to achieve consensus on the evalu-ation questions that should be asked.

Before site participants create PLMs, theyneed to consider the treatment level at whichthey want to conduct their evaluation. Referparticipants to pages 14 and 15 of theFramework workbook for a reminder aboutdifferent treatment levels. In some cases, siteswill need to decide if they want to conducttheir evaluations at the activity, service,agency, or system level. Whatever they de-cide, their PLM level should match their evalu-ation level. For example, if a site wants to

conduct an evaluation at the service level, thenthe PLM should portray the service level, too.In some cases, site participants will not know,or disagree about, their desired level of evalu-ation. Discussion among participants to de-termine treatment evaluation level is the firstpriority in these cases.

Refer to the information contained in Step 3of Workbook 1 (starting on page 14) tostructure the rest of your presentation.Overheads, which may help orient partici-pants, are contained at the end of this guide.

In addition to the information provided in theworkbook, consider including the followingquestions as part of your explanation of howto create programme logic models. (Thesequestions are provided courtesy of Dr. BrianRush, of Virgo Planning and Evaluation Con-


WHO/MSD/MSB 00.2j

sultants, London, Ontario, Canada.) Ask par-ticipants to answer the following questions inreference to their own treatment programme:

1 In the long run, what should be differentin the community, or the “target popula-tion”, as a result of your programme be-ing delivered? What are the changes youhope for, even recognising yourprogramme may only be playing a smallpart in achieving these changes? Thesechanges are your long-term objectives.Some of them may be quite general andreflect broad goals or aspects of your mis-sion statement.

2 In the shorter term, what changes do youhope will occur in the community or the“target population”, as a result of yourprogramme being delivered? What short-term changes are needed in order toachieve your longer term objectives(goals) identified above? How are theseobjectives linked over time? In otherwords, what changes lead to what otherchanges to eventually achieve your longerterm objectives? Write down all thechanges you think might happen as a re-sult of your programme and then ask your-self how they are inter-connected. Whatis the logic or rationale connecting onechange to another?

3 Next, rather than focusing on the changesyou hope to make, think about how youare trying to do this. What services or ac-tivities do you currently deliver or whichyou plan on delivering in the near future?These are your implementation (processobjectives). They reflect what you do,rather than the outcomes (changes) youexpect to achieve. This is an importantdistinction since delivering the service isonly a means to an end; the services youprovide are not themselves the out-comes. Your activities or services prob-ably cluster into different program com-ponents. There is not hard and fast ruleas to how you cluster them — some-times it's because a set of activities orservices all relate to a particular groupof people (e.g., youth vs elderly) or theyall relate to a particular programme func-tion or role (e.g., smoking cessation vs.alcohol education). The examples pro-vided give a flavour of how to organise

your implementation (process) objectivesinto components. Be careful not to gettoo detailed at this stage, but try to cap-ture, in as specific terms as possible, themain elements of your program activitiesand services.

4 The last stage is to show the connectionbetween your various programme compo-nents and your short-term objectives (andeventually your long-term objectives). Thisis done in a diagram connecting the vari-ous activities and outcomes. Again, theexamples provided will illustrate how thisis done. The most important point, how-ever, is that there should be some logic orrationale underlying these connections. Inother words, why should delivering theseactivities or services produce thesechanges?

5 Once you have a draft of a diagram put-ting all of the above together, you shouldexpect to revise it several times. This is agreat tool for communicating with eachother about your programme. You can useit to discuss how your program hasevolved over time and should adjust in thefuture. In other words, it is a planning andprogramme management tool as muchas an evaluation tool.

Once you have a good draft you can startusing it to plan your evaluation activities. Ba-sically, your task is to use the logic model toprioritise those aspects of the programme tobe evaluated and to develop and select thespecific evaluation questions to be addressed.The chart that accompanies the logic modelwill help you do this by giving you a frame-work to translate each of your programmeobjectives (from the logic model) to an evalu-ation strategy for getting feedback on theachievement of that objective. It helps in thisprocess to have numbered each objective inyour logic model (or group of objectives) andtransfer them one at a time to this table usingthe number to cross-reference.

What will emerge as you complete this tableis a very large “shopping list” for your evalu-ation. Now, as a group, your task is to decidewhat is feasible for an evaluation plan givingdue consideration to the resources you have,measurement and design issues, ethical andconfidentiality considerations, etc.


WHO/MSD/MSB 00.2j

Experience indicates that specific ex-amples of programme logic models helpparticipants better understand the basicprinciples. The programme logic modelfrom the fictional heroin detoxification

ReferralMaincomponents

Implementationobjectives

Short-termoutcomeobjectives

Long-termoutcomeobjectives

Diagnosis& Treatmentplanning

Withdrawalmanagement& Treatment

• To confirm clients’eligibility for theprogram

• To determineclients’ motivationto engage intreatment

• To determineclients individualneeds

• To obtain standardsomatic, mentaland psychologicalstatus of the client

• To form therapeuticalliance with theclient

• To formulatetreatment plan

• To signtherapeuticcontract

• To monitorwithdrawalsymptoms

• To prescribestandardmedication

• To provide a safeand supportiveenvironment

• To conduct labor-atory/other tests

• To motivate clientto finish theprogram

• To provide safewithdrawalmanagement

• To provideinformationabout otherprograms forfurther drug-freetreatment

• To motivateclients tocontinuetreatment in oneof theseprograms

• To collect thenecessaryinformation fordevelopment ofadequatetreatment

• To enhanceclients’motivation tostay in thedetoxificationprogram

• To cope with withdrawal symptoms

• To stabilise mental and physical status

• To maximise client satisfaction with theprogram

• To reduce involvement in criminalactivities

• To increase clientsknowledge aboutfurther treatmentpossibilities

• To increase clientsknowledge aboutconsequences ofPSU and AIDS

• To maximise thenumber of clientswho are referredto long-termtreatment

• To reduce drug use behaviour

• To prevent consequences of drug use

• To improve mental and physical well-being of clients

• To collect adatabase ofstandardisedclient information

• To improvequality of care

Assessment &Intake

programme in Workbook 1 and 2 is pre-sented below and in the overhead proto-type section in the back of this guide.Consider referring to this example as youmake your presentation.


WHO/MSD/MSB 00.2j

Exercise: describe yourprogramme for evaluation(site level)

understanding the basic PLM structure, whileothers might disagree among themselvesabout their activities or goals. Regardless, itis especially important for facilitators to pro-vide intensive assistance and feedback dur-ing this exercise. Listen to sites’ discussions,provide advice or examples if needed, andmake yourself available for questions as theyarise. Expect that most sites will need sev-eral hours and multiple versions before beingsatisfied with their PLMs.

It is a good idea to move sites to the tripletlevel at least once during this exercise. In-put from other sites can spur new thinkingor resolve dilemmas. Do not wait until theend of the exercise to get feedback at thetriplet level; move sites to their triplets wellin advance of completion of the PLMs.After receiving triplet level feedback, par-ticipants can continue working on theirmodels at the site level.

After you have finished describing programmelogic models, direct participants to move tothe site level and complete the “It’s your turn”Step 3 (A and B), located on pages 15and 17 of Workbook 1. The majority of timeshould be spent on exercise 4B. Post-it notes(small pieces of note paper with adhesive onone side) are useful for creating programmelogic models: if each component of the model(e.g. a single activity or objective) is writtenon one post-it note, transformation and edit-ing of the model is easy to perform. If post-itnotes are not available, small pieces of paperbacked with tape can be used instead. Sitesshould be given adequate space to create theirmodels (e.g. a portion of an empty wall, or alarge flip chart).

Prior experience indicates that for most sites,creating a programme logic model is the mostchallenging and thought-provoking task of theworkshop. Some sites might have difficulty


WHO/MSD/MSB 00.2j

Day 3Today, participants will continue to spend themajority of time at the site level, although sometime will be spent at group level presentations,and with triplet level feedback. Specific ac-tivities for the day include:

• Presentation and exercise: define yourevaluation questions

• Presentation and exercise: determine yourevaluation measures

Each of the activities is described in detailbelow.

The goals for the third day include:

• using the programme logic model that wasdeveloped during Day 2, sites will definespecific evaluation questions

• participants will be instructed how to usethe foundation and specialised workbookstogether

• participants will learn guidelines for choos-ing evaluation measures

• sites will begin to choose their evaluationmeasures

Presentation and exercise:define your evaluation questions(group, site, and triplet levels)The presentation and exercise should be basedon Step 5 of Workbook 1 (pages 20 to21). The presentation should be given at thegroup level. It can be brief (15 minutes), butshould include some general remarks abouthow to narrow the scope of possible evalua-tion questions (see below for specific contentsuggestions). Following the presentation, sitesshould be given several hours to formulateand discuss their evaluation questions.

A general structure for moving from theprogramme logic model to evaluation ques-tions and data collection methods is presentedin the overhead prototype entitled “Basic For-mat for Moving from Logic Model to MoreDetailed Evaluation Planning” located in theback of this guide. Consider showing this formto participants at the beginning of the presen-tation and making copies for their use duringthis phase of evaluation planning.

Experience shows that many sites will wantto “answer it all” with their first evaluation.This approach usually ends unfavourably. Itis essential that the facilitators caution against

this approach, and then assist the sites to nar-row the questions to a feasible number andscope given their time, experience, and otherresources.

In addition to feasibility, sites should be en-couraged to create evaluation questions thatare meaningful to the treatment programmeor service. That is, questions should be linkedclearly to a specific component or goal of theprogramme logic model, and have clear im-plications for improving treatment services.Sites might want to ask themselves: How willthe results from the proposed evaluation ques-tion be used to improve our treatment ser-vices? If there is no clear answer, then evalu-ation questions should be reformulated.

At this point in the workshop, participants donot need to choose specific evaluation mea-sures. (They will be chosen later in the day.)Participants should be instructed to focus onformulating general evaluation questions, en-suring that they are feasible and meaningful.Following the presentation, participants shouldbe directed to move to the site level and com-


WHO/MSD/MSB 00.2j

plete questions 1 through 3 of the “It’s yourturn” Step 5 exercise, located on page 21of Workbook 1. Participants will not be able todo question 4 at this point, as it involves consul-tation with expected users of the results.

At a middle point of the exercise, sites shouldbe moved to the triplet level, where they canpresent possible evaluation questions to com-ment and feedback from other participantsand the assigned facilitator. Triplet feedbackserves a dual purpose: to give participants andfacilitators an opportunity to provide commen-tary to sites on the feasibility and appropri-ateness of evaluation questions, and for fa-cilitators to teach participants about differentevaluation types. As sites are discussing theirevaluation questions, the facilitator should takenotes. Once every site has finished, the pos-sible questions should be used to explain dif-ferent evaluation types (see Frameworkworkbook, pages 18 to 27):

• Needs Assessment• Process• Costs• Client satisfaction• Outcome• Economic

Use the site evaluation questions to highlightkey differences in each type of evaluation.

During the discussion, provide constructivecommentary and guidance to sites on theirevaluation questions. Ideally, these two dis-cussion points should unfold seamlessly andsimultaneously.

Experience indicates that some participantswill confuse process and outcome evaluationquestions. In other words, they might attemptto answer an evaluation question about out-come by measuring process, or vice versa.As you listen to possible evaluation questions,ask yourself whether your sites are makingthis common error. In preparation for thiseventuality, consult Workbooks 4 and 7 formore information about the differences be-tween process and outcome evaluations.Briefly, process evaluations seek to understandthe extent to which a treatment programmeis operating as planned. The evaluation ques-tions do not ask about changes in clients thatresult from services being provided. Rather,the questions concern coverage and process.Outcome evaluations measure how clients andtheir circumstances have changed, and second,attempt to show that treatment has been a fac-tor in causing this change. Typically, processevaluations should be conducted in advance ofoutcome evaluations. Why? Evaluators mustfirst establish that treatment is happening in aconsistent and desirable manner before evalu-ating whether it helps clients.

Presentation and exercise:determine your evaluation measures(group, site and triplet levels)Before beginning the main presentation, thepresenting facilitator should make a few com-ments about how to use the specialised work-books in conjunction with the foundation work-books. Participants should be reminded thatWorkbooks 1 and 2 provide a solid founda-tion of general information about conductingevaluations, whereas the specialised work-books (Workbooks 3 through 8) present de-tailed information for different types of evalu-ation. Sites should begin consulting thespecialised workbook that is applicable fortheir evaluation question, while still using the

foundation workbooks for general informa-tion. An overhead prototype listing thespecialised workbooks content is located inthe back of this guide.

The rest of the presentation should cover keymaterial from Step 6 of Workbook 1 (pages22 to 41). You should tailor this presentationto match participants’ prior knowledge of mea-surement issues and evaluation questions. Ifparticipants are relative novices concerningmeasurement issues, a discussion of basicconcepts may be needed. In other cases,


WHO/MSD/MSB 00.2j

higher-level discussions may be appropriate.Regardless, limit your comments to 45 - 60minutes maximum.

Possible areas for presentation include:

• The language (meaning) of measurement

• Why bother with indicators?

• Quantitative and qualitative measurement

• Reliability and validity

• Types of measures

• Tips for selecting or preparing a data col-lection instrument

Overhead prototypes for each of these top-ics are located in the back of this guide.

Regardless of participants’ prior experience,remind them during the presentation that ifthey don’t take measurement issues seriouslyduring programme evaluation, any claims theymake about the programme will always beopen to criticism. Someone could always say:“This sounds nice, but how do you actuallyknow your programme accomplishes that?”By using good measures and data collectiontechniques, participants can provide a betterresponse to this question. For this reason, sys-tematic measurement in programme evalua-tion is the best tool for convincing people aboutwhat a programme does, how it functions,what outcomes are achieved and what hasbeen done to improve it.

Caution participants to limit their evaluationmeasures to those that are necessary to an-swer the evaluation questions. Some partici-pants might want to collect as much data aspossible, for example measuring client depres-sion using three different scales. This ap-proach leads to unnecessary burdens uponevaluation participants, higher drop-out rates,and more cumbersome data management.Limiting measures to those that are absolutelyneeded to answer the evaluation questions canavoid these kinds of problems.

Also advise participants that there are manyadvantages to using existing instruments thathave been proven to be valid and reliable.They need to be sure, however, that the mea-sures collect the right kind of data, and thatthey are applicable for their specific cultureand/or setting. In addition, participants should

be made aware that any revisions such asrewording, eliminating, adding, or reorderingitems might diminish the validity and reliabil-ity of an established instrument.

Finally, tell participants that sometimes theycannot rely on existing instruments, scales,forms, or interview schedules to create theirdata collection instruments. If they must de-sign a new indicator or question, they shoulduse the following checklist for constructing it(reproduced on page 60 of Workbook 1 and asan overhead prototype in the back of this guide):

1 Are the words simple, direct and familiarto all?

2 Is the question as clear and specific aspossible?

3 Is it a double question?

4 Does the question have a double negative?

5 Is the question too demanding?

6 Are the questions leading or biased?

7 Is the question applicable to all respon-dents?

8 Is the question objectionable?

9 Will the answers be influenced by responsestyles?

10 Have you exhausted the response alter-natives?

The exercise following the presentation (It’sYour Turn Step 6F from Workbook 1:page 41) is designed to assist sites to selecttheir own data collection methods. Note thatsites will not be able to answer question 3 atthis time, because it requires consultation withexpected users of the results. If sites usedthe form, “Basic Format for Moving fromLogic Model to More Detailed EvaluationPlanning,” during the morning session, theyshould continue to use it during this exercise.Facilitators should provide guidance asneeded. In many cases, sites will need spe-cific suggestions about measures to use.Many measures are presented in the work-books and case examples, so be sure tofamiliarise yourself with these materials inadvance of the workshop. In particular, seethe tables on page 11 of Workbook 4 fora comprehensive listing of relevant mea-sures. In addition to the workbooks, it is agood idea to have copies of other commondata collection measures available.*

*A review of theAddiction ResearchFoundation OutcomeMeasures Directory(undated) is highlyrecommended. ThisDirectory containsmany potentiallyuseful instrumentsand discussesreliability, validity,and practical issues inadministration.Information about thispublication can beobtained from:Addiction ResearchFoundation, 100Collip Circle, Suite200, London, Ontario,Canada, N6G 4X8.


WHO/MSD/MSB 00.2j

Day 4This day covers a large amount of workbookmaterial: the remainder of Workbook 1 andWorkbook 2. Presentations based on Work-book 1 are fairly thorough, while Workbook 2presentations just introduce key concepts andrecommend further reading. The final part ofthe day is spent preparing for Day 5's sitepresentations.

• presentation and exercise: prepare a datacollection plan

• presentation and exercise: ensure that yourresources are sufficient

• presentation: the 6 steps of implementingan evaluation

• exercise: prepare your site presentation

Main goals for Day 4 include:

• Review how to prepare for data collec-tion, including choosing a time frame andsampling strategy. Assist sites to developtheir own data collection strategies.

• Ensure that sites’ evaluation plans are re-alistic given the time and resources theyhave available.

• Introduce the 6 steps of implementingevaluations,

• Give sites a clear structure for presentingtheir evaluation plans during Day 5.

Presentation and exercise:prepare a data collection plan(group, site, and triplet levels)

This group level presentation should corre-spond to the information presented in Step 7of Workbook 1 (pages 42 to 47). Key con-cepts include a) setting a time frame for datacollection, and b) determining the sample tobe used. In other words, sites must decideabout the “when?” and “who?” of data col-lection. Their decision-making should dependon their evaluation questions, limitations onresources, and other practical constraints.

When setting a time frame, sites should con-sider type of evaluation that they are doingand their resources for collecting and man-aging data. They need to decide two things:the number of times to collect data, and theamount of time between the data collection

points. In terms of the number of times tocollect data, more is not always better. Why?Frequently repeated data collection (e.g.,weekly during treatment) results in increaseddata collection resources and more compli-cated data analysis. And in many cases, fre-quently repeated data collection is not neces-sary to answer evaluation questions. Manyevaluations require one data collection pointonly. Even for more complicated outcomeevaluations, one pre-treatment and one post-treatment data collection point (and perhapsone additional follow-up data collection) isusually sufficient. If multiple data points willbe collected, decisions about time intervalsbetween data points also should be driven byevaluation questions and practical constraints


WHO/MSD/MSB 00.2j

on resources. In any case, sites should be di-rected to consult the appropriate specialisedworkbook for specific suggestions.

Sampling is an important issue, yet one thatnovice researchers often downplay or over-look while conducting evaluations. Beginningevaluators may see no harm in sampling themost “convenient” group of people withoutconsidering the group’s representativeness.Consequently, results can be biased and non-generalisable. Participants should befamiliarised with the important concept ofrepresentativeness, and with different meth-ods for sampling populations (see page 45of Workbook 1).

Following the presentation, direct participantsto move to the site level and complete thetwo “It’s your turn” exercises for Step7, located on pages 44 and 47. Please notethat participants will not be able to answereach exercise’s question 3 at this point, be-cause each question requires input from ex-pected users of results. Facilitators shouldassist sites in their decision making as needed.After sites have discussed their plans, movethem to the triplet level for feedback. By thispoint, triplets should be working well together,allowing for meaningful cross-site educationand support.

Presentation and exercise: ensurethat your resources are sufficient(group and site levels)

This group level presentation is based on Step8 of Workbook 1 (pages 48 to 49). Thegoal of this presentation and exercise is toensure that each site’s resources (financial/material, expertise, and time) are sufficientfor the evaluation plan that its participants havecreated. The presentation itself should be briefand focused upon orienting participants to thesite level exercise.

Ask participants to move to the site level andcomplete the “It’s your turn” exercise lo-cated on page 49 of Workbook 1. Becausesome participants might be sensitive aboutdiscussing their resources in the presence ofother sites, it is suggested that participantsremain at the site level for the duration of thisexercise. Facilitators, however, should assistsites to examine their resources realistically,and redirect them as needed.

The goal of this exercise is to have partici-pants review their evaluation planning anddecide whether they have created a reason-

able, well-organised plan that also is feasibleto complete successfully. If facilitators havebeen alert for potential problems throughoutthe planning process, then this exercise shouldnot create the need for major revisions.

In some cases, workshop organisers might beproviding funding and/or other support to sites.If you have not already done so, this is thetime to negotiate any outstanding issues,which might include:

• the project time frame

• seed funding

• availability of follow up consultants

• means of communication with theorganisers following the workshop

• deadlines for evaluation plans and progressreports


WHO/MSD/MSB 00.2j

Presentation: the 6 steps ofimplementing an evaluation(group level)

Step 2...

This presentation should be a brief overviewof salient issues for data collection. Becauseparticipants will not yet be at this stage oftheir evaluations, you can simply point outimportant issues to consider, and direct themto pages 22 to 23 of Workbook 2 for fu-ture reference.

Step 3...

Step 3, data analysis, can be a highly techni-cal area. Your comments should be tailoredto the participants’ levels of expertise andevaluation questions.

Depending on your audience, key conceptsinclude:

• What can data analysis tell us?

• The importance of planning for data analy-sis in advance

• General factors to consider when choos-ing a data analysis strategy

• The importance of seeking technical helpas needed

Steps 4 to 6...

Remember to convey the important conceptthat evaluation does not end with data analy-sis. Reporting results, making use of what waslearned, and starting again are equally vitalaspects of the evaluation process. The “Cre-ating a Healthy Culture for Evaluation”graphic located on page 8 of the Frame-work Workbook (and available as an over-head prototype in this guide) can be usedto structure this part of the presentation.

This group level presentation should overviewthe 6 steps of implementing an evaluation (asoutlined in Workbook 2). Because of timeconstraints, it is impossible to cover each ofthese steps in detail during the workshop.However, facilitators should mention key con-cepts from each of the six steps (outlinedbelow).

Step 1...

Preparing for data collection is an importantstep in the process of conducting evaluations.Key components include:

Managing ethical issues

• Informed consent

• Manage risks of participation

• Confidentiality

Developing a data management plan

• Design a record keeping system

• Acquire and process data

• Train people who will collect data

• Standardisation

Conducting a pilot test

• Identify flaws

• Revise data collection plan as needed

Writing an evaluation plan

(this task will be covered in detail during Day5)

Participants should be directed to Step 1 ofWorkbook 2 (pages 10 to 21) for more in-formation about these topics.


WHO/MSD/MSB 00.2j

Exercise: prepare yoursite presentation(site level)

Briefly show your programme logic modelcomponents. Indicate the part of the modelthat is the focus of the evaluation.

Section II. Evaluation question, measures,data collection

Show the one or two most important evalua-tion questions and how you will answer them(evaluation plan)

Section III. Using the results for decision-making

Describe who will receive the results fromthe evaluation and how it will be used for de-cision-making.

If an overhead projector is available, each siteshould be instructed to create one overheadtransparency per section to structure theirevaluations. Photocopies of these transpar-encies can then be made for facilitators andother sites to keep as references. If you donot have an overhead projector, flip charts orchalk boards also can be used.

The final part of the day should be devoted topreparing for Day 5’s site presentations, dur-ing which time each site will be asked topresent briefly (15 minutes) an overview ofits evaluation plan. The primary purpose ofthe presentations is to cross-inform partici-pating sites about their evaluation plans. Thesecondary purposes of the presentations area) to provide a “positive reinforcement” op-portunity for each site’s evaluation planningefforts, b) to build a sense of community be-tween the sites, and c) to increase each site’scommitment to complete its evaluationthrough public proclamation of its plan.

Most sites will need to wait until they returnhome and consult with others before finalisingtheir evaluation plans. In these cases, sitesshould do this exercise with the planning theyhave completed at this point.

Participants should be given the followingstructure for preparing their site presentations(available as an overhead prototype in theback of this guide):

Section I. Programme logic model andprogramme description


WHO/MSD/MSB 00.2j

Day 5Like Day 1, today’s activities occur mostly atthe group level. Participants are provided theopportunity to reconvene and learn about oneanother’s progress during the workshop, andto give feedback to facilitators. The specificactivities include:

• presentation: summary

• site presentations of evaluation plans

• presentation: writing an evaluation plan

• exercise: develop a specific action plan

• closing activities

Goals for the fifth and final workshop day in-clude:

• Each site will give a 15 minute presenta-tion of its evaluation plan

• Explain how to write an evaluation plan

• Assist each site develop a specific actionplan

• Address remaining questions or problems

• Obtain constructive feedback from par-ticipants about ways to improve the work-shop

Presentation: summary(group level)One facilitator should provide a brief sum-mary presentation of the work that has beenaccomplished so far, and the remaining itemsfor the day. The facilitator also should elicitfeedback from the group regarding any otheroutstanding issues. Relatively simple questionsor outstanding issues probably can be ad-dressed during closing activities. More com-

plicated issues may require individual atten-tion and/or modification of the day’s agenda.

Other facilitators should listen carefully to thesummary and participant questions, identifygaps in the information provided by the pre-senting facilitator, and interject as needed.

Site presentations of evaluation plans(group level)During this interval, each site should presenta brief overview (15 minutes) of its evalu-ation plan. Presentations should follow thestructure presented on page 35 of thisguide. As stated earlier, the primary pur-pose of the presentations is to cross-informparticipating sites about their evaluationplans. The secondary purposes of the pre-

sentations are a) to provide a “positive re-inforcement” opportunity for each site’sevaluation planning efforts, b) to build asense of community between the sites, andc) to increase each site’s commitment tocomplete its evaluation through public proc-lamation of its plan. In light of these pur-poses, it is important for facilitators to keep


WHO/MSD/MSB 00.2j

the majority of their comments congratula-tory and positive.

Experience indicates that most sites will needassistance to keep their presentations withinthe 15 minute time frame. Consider appoint-ing a time keeper, who can watch the timeand tell speakers when their time is close tofinished.

Depending on the time available, consider al-lowing a few minutes at the end of each pre-sentation for comments from other sites. Thisis the only structured opportunity for partici-pants to learn about all other sites, plans, andcross-site feedback at this level can be veryuseful.

Presentation: writing anevaluation plan (group level)After the site presentations, a facilitator shouldpresent the rationale and general guidelinesfor writing an evaluation plan. This informa-tion is presented on pages 20 to 21 of Work-book 2, and summarised as an overheadprototype in the back of this guide. Insome cases, workshop organisers might re-quire that sites submit a written plan follow-ing the workshop in order to receive feed-back and/or financial support. In other cases,sites will be on their own to implement theirevaluations. Regardless, writing a structuredplan helps evaluators organise their thoughtsclearly. It also serves as a reminder to every-one involved about the purpose of the evalu-ation and the questions and decisions the re-sults are intended to address.

The basic elements of an evaluation plan (ex-plained in greater detail on page 18 of Work-book 2) are:

1) Background and general purpose

2) Programme logic model

3) Evaluation team

4) Evaluation questions to be addressed

5) Data collection strategy

6) Data management plan

7) Staff training

8) Pilot test

9) Strategy for using results

Sites should be encouraged to complete theirown written evaluation plans after they re-turn home and consult with expected usersof the results. In some cases, these plans alsowill be forwarded to workshop organisers forreview.

Sites involved with the initial WHO/ UNDCP/EMCDDA project were asked to submit writ-ten evaluation plans for review. A set of guide-lines, along with checklists for sites and re-viewers to complete, were created toencourage consistency in the writing and re-view process. These guidelines and check-lists are reproduced starting on page 74 ofthis guide for your convenience.

Exercise:

After participants thought about their evalua-tion plan, they should be encouraged to thinkabout a concrete action plan which will helpthem transfer their plan into action.

The facilitator should direct participants towrite a response to the following questions:

Why is a concrete action plan helpful to ini-tiate your evaluation?

Think about 2-3 concrete task that should becarried out at the beginning of your imple-mentation?

When should these task be completed? Howmuch time will they occupy? Who will be in-volved in completing these tasks?


WHO/MSD/MSB 00.2j

Tell the participants to be as specific as pos-sible when thinking about these concretesteps. They also should think about feasibledeadlines for these steps. Instruct participantsto keep their responses anonymous. Afterparticipants are finished writing, papers shouldbe collected and discussed.

Reminder: do not ask the group who wrotespecific responses. If a person wants tomake himself/herself known, that is his/herchoice, but the information should not be“forced” by facilitators.

Closing activities

This section should begin with closing com-ments for the workshop. Consider using thephrase “Creating a Healthy Culture for Evalu-ation” (see page 8 of the FrameworkWorkbook ; figure available as overheadprototype in the back of this guide) as anorganisational theme for the presentation.

This is your final opportunity to “tie up looseends” with participating sites. Any outstand-ing issues regarding continuing collaborationwith sites, expectations for site evaluation re-ports, financial support, etc., should be handlednow.

This also is an opportunity for participants topose remaining questions to facilitators. Pos-sible questions could include general aspectsof evaluation planning or implementation, orsomething specific to their site or evaluationplan.

Following the question and answer period,participants should be directed to completethe post-workshop evaluation questionnaire(located on page 48 of this guide). Afterthe questionnaires are completed, facilitatorsshould revisit the fears and expectations (writ-ten on Day 1 of the workshop). As a group,review the fears, and determine to what ex-tent expectations were met. The outcome ofthis discussion can provide a qualitative evalu-ation of the workshop’s success.

After the workshop is finished, remember toconduct a final debriefing session with all fa-cilitators (see “Preparing for Facilitation” onPage 13). Also take the time to collect andorganise facilitator notes for each site (seepage 15).


WHO/MSD/MSB 00.2j

Problems and solutions

Experience indicates that various kinds ofproblems might arise during the workshop. A list ofcommon problems and suggested solutions areoutlined in the section below.

Problem: There are significant language barriers

Solution: In situations with significant lan-guage barriers, remember to speak slowly, andavoid complicated sentence structure and/orvocabulary. Write important ideas using over-head transparencies, chalk boards, and flipcharts. Pictures or diagrams also can be use-

ful. If participants are still having trouble fol-lowing key ideas, refer them to relevant sec-tions of the workbooks, where they can readat their own pace. Finally, encourage partici-pants to use their “louder” and “slower” cards(see page 27) whenever needed.

Problem: Participants are not present for the entire workshop

Solution: Absences from the workshophinder evaluation planning and should beavoided if possible. In situations where ab-sences are unavoidable, it is better if at leasttwo people from a site are present, so thatthey can discuss during the workshop andmake presentations to others upon their re-turn. Sites that arrive late to the workshopshould be moved forward into the mainstream

schedule and triplet work as rapidly as pos-sible, to foster cross-site collaboration andcollective learning. A designated facilitatorshould work intensively to cover material thatthey missed in order to achieve this goal.“Mini-presentations” can be held while othersites are working on planning activities. As-signments can be given for evening work tohelp them catch up quickly to other sites.

Problem: Participants become too tired to work effectively.

Solution: The workshop schedule is demand-ing, so it is no surprise that some participantsbecome tired after a few days. Remember toschedule breaks throughout each day. Evena few minutes to stand and/or walk aroundthe room can make a big difference. If tired-ness is a big problem, consider reschedulingyour time so participants will have a half-day

free to rest and relax. While in session, try tokeep the tone of the workshop cheerful. Forvariety, move around the room while mak-ing presentations, and make use of multipleteaching tools (overhead projector, flipchart, chalk board). Remember to minimise“lectures”, and focus on involving partici-pants in discussions.


WHO/MSD/MSB 00.2j

Problem: Facilitators are too tired to meet at the end of the day.

together with fellow site participants. Thisfacilitator could raise some hypotheticalproblems (like those mentioned above) asexamples, then discuss how they interferewith planning a meaningful evaluation. Oncethe site exercises are underway, facilitatorsshould remain alert for possible problems,discuss them with their co-facilitators, andintervene as appropriate. If you need to in-tervene, consider using a straightforwardapproach, such as “In my experience, sitesdevelop better evaluation plans when allparticipants discuss their views openly. Per-haps we can try that now. X, what do youthink about this question?” In other cases,a more subtle approach will be more suit-able, such as “X, I noticed that you haven’tsaid anything about this question yet. Whatare your views?”

Solution: Like participants, facilitators’ feel-ings of tiredness are often related to mentalexertion. Yet, it is very important for facilita-tors to meet each day to discuss how theirsites are progressing, and to get suggestionsfrom their team about the day ahead. Forthese reasons, daily facilitator meetings makea large difference in how well the workshopproceeds. If you are too tired in the evenings,morning meetings are an option. There are

some drawbacks, though: morning meetingsare frequently rushed, and by the next morning,facilitators will not remember as clearly whathappened the day before. For these reasons,immediate daily debriefing is preferable. Try tomake the meeting as enjoyable as possible. Meetin a picturesque location, or consider going for awalk as you talk together. In addition to work-shop practicalities, discuss your frustrations,fears, or anything else that is concerning you.

Problem: Participants are not working well together at the site level.

Solution: “Not working well together” canbe defined in multiple ways. In some cases,participants will prefer to work separatelyfrom one another, thus eliminating meaning-ful site level discussions. In other cases, oneparticipant will dominate with his/her ideas oropinions, while others remain silent. In yetother situations, site participants will disagreestrongly with one another about the bestcourse of action. A certain amount of vari-ability in how people relate to each other is tobe expected, especially cross-culturally. Fa-cilitators should be respectful of cultural andpersonal differences, yet be aware of situa-tions that interfere with the goal of high qual-ity evaluation planning. Before site level ex-ercises begin, it might be helpful for onefacilitator to make a few comments at thegroup level about the importance of working

Problem: Participants react negatively to a facilitator.

Solution: Your age, gender, nationality, ap-pearance, experience, and/or behaviour cansometimes be the focus of negative reactionsfrom participants. As a facilitator, it is impor-tant to understand these potential pitfalls, andto adjust your approach accordingly. Regard-

less of the specific issue, the key is to antici-pate these potential problems, to seek back-ground information on the cultural practicesof participants’ countries in advance of theworkshop, and to be willing to adjust your ap-proach as needed.

Problem: Sites are not keeping up with the workshop schedule.

Solution: If most or all sites are behindschedule, you should re-examine youragenda and consider whether you are mov-ing through the material too quickly. Moretypically, however, you will find that somesites simply work through the material morequickly than others. For sites that are be-hind schedule, assess whether participantsare having difficulties working well together

(see above). Address these problems asneeded. In any event, encourage sites to staywith the agenda as much as possible. Even ifthey are not finished completely with an ex-ercise, they should move forward to the nextsteps as the agenda suggests. Final decisionswill need to be taken after the workshop, andunfinished exercises also can be completedat that time.


WHO/MSD/MSB 00.2j

Problem: Sites are moving through the material too quickly.

ing fully each topic. It is possible that thesesites are being dominated by a single mem-ber, thus curtailing meaningful discussion. Al-ternatively, all participants from a certain sitemight be treating the discussions too superficially.As a facilitator, it is your role to assess theseissues and intervene appropriately.

Solution: The potential issues and solutions tothis problem are similar to those listed above. Ifnearly all sites are working ahead of schedule, itis possible that your agenda allows too muchtime. If, on the other hand, only certain sitesare working ahead of schedule, you shouldassess whether these participants are discuss-

Problem: Sites� evaluation questions are not �meaningful�, or linked directly toprogramme objectives listed in the programme logic model.

Solution: The determination of what is“meaningful” is highly subjective. But for thepurposes of this project, a meaningful evalu-ation question is defined as one that is linkeddirectly to assessing whether programme ob-jectives are being met. As a facilitator, it isyour role to ensure that sites have chosenquestions that are meaningful according to thisdefinition. One way to help yourself is to meetdaily with your co-facilitators to discuss sites’progress. In this context, you can present your

sites’ plans to your team, and get feedbackas to whether their evaluation questions aresufficiently meaningful. If a question is notthought to be meaningful, you should raise thisissue with the site the next day. Tell the siteparticipants that you are confused about howtheir question(s) will improve their programme,and ask them to explain this to you. Their re-sponse will help you determine whether you havemissed an important justification, or whether thesite needs to adjust its plans.

Problem: Sites� evaluation questions are not feasible.

Solution: In some cases, sites will createevaluation plans that are not reasonable, giventheir resources (time, expertise, funding). Theymight feel pressured to “answer it all” withone evaluation, or might have an unrealisticperception of their limitations. As a facilita-tor, you should assist sites to choose evalua-tions that are feasible. The first step is to geta good understanding of each site’s availableresources (accomplished during Day 2). Later,

you should listen to their plans with theirunique situation in mind. If you think that thereis a significant problem, you should raise thiswith the site participants. It is often useful totake a non-threatening approach by tellingthem that you are confused, and asking themto explain to you how they will be able to ac-complish their plan with their available re-sources. Once they start explaining, they willusually see the problem for themselves.


WHO/MSD/MSB 00.2j

Annex 1

How to Describe your Local Treatment is astructured format for workshop participants to usein advance of the workshop. Please see page 15 ofthis guide for more information about its use.

How to describe yourlocal treatment

In preparation for the upcoming workshop,we would like you to prepare a two to threepage summary of your current substanceuse treatment activity, service, agency, ornetwork. The main purpose of the summaryis to help us gain a better sense of yourclinical services, so that we can help youdesign an evaluation plan that is well-suitedfor your situation.

A treatment activity involves a particulartreatment modality (e.g., cognit ive-behavioural therapy).

A treatment service (e.g., outpatientcare) can include more than one treatmentactivity or modality (e.g., cognitive-behavioural therapy, relapse prevention,social skills training).

A treatment agency may offer more thanone type of treatment service (e.g., detoxi-fication, inpatient, outpatient and continu-ing care).

A community treatment system is com-prised of many different services, agen-cies and treatment settings. It involvespeople with substance use disorders and

includes both specialised and generalistservices and agencies.

Depending on the size of your clinical ser-vices, you may or may not have a service,agency, or system within your setting. Youshould focus your summary on the level ofcare that is available, and also at the levelyou want to evaluate.

To help guide the preparation of your sum-mary, please follow the guidelines below:

• Make sure that you keep your summaryto three pages or less. If you have a largetreatment service or system, you willneed to choose the most important in-formation to present. Remember that thepurpose of the presentation is tofamiliarise us with your overall situation.Undue details are not necessary.

• Your accuracy is important to us.Please answer our questions as cor-rectly as possible.

• Use the following questions to guide thedevelopment of your summary.


WHO/MSD/MSB 00.2j

General substance use issues

1 What are the major substance use prob-lems requiring treatment in your region?

2 What is the current drug policy of yourregion? (Include pertinent informationabout criminal laws and treatment policiesfor those who use illegal substances.)

3 What are the government’s and/orcommunity’s attitudes towards conductingtreatment evaluation and improving sub-stance use treatment services?

Overview of clinical services

1 Briefly overview your local clinical ser-vices.

2 Are the clinical services private or public?

3 How are your clinical services related tothe national health system and/or generalmedical services in your region?

4 How are your clinical services related tothe general community?

Resources

1 Who are your employees?

- educational background

- roles

- how many

2 What kind of facilities are available (of-fice space, hospital beds, etc.)?

3 What is the annual operating budget?

4 Are waiting lists common? How long isthe waiting list on average?

Activities

1 Describe the type(s) of clinical servicesthat you provide.

- Do you have structured substanceuse activities, or is clinical care un-

structured and provided on an as-needed basis?

- If you have structured activities, pleasedescribe them in terms of the targetpatient population, length ofprogramme, and the typical course ofone patient through the programme.

2 Who are your typical patients?

- Age

- Gender

- How many served

- Common substance(s) used

- Common types of substance-relateddisorders/problems

Goals of treatment

What are the overall goals of your clinicalservices? (For example: reduce patients? sub-stance use, assist patients to obtain employ-ment, prevent children from initiating sub-stance use, etc.)

Current evaluation efforts

Is evaluation happening now? If yes, pleasedescribe current evaluation efforts.

Costs and benefits ofevaluation

1 What are the main barriers to doing treat-ment evaluations in your setting?

2 What are potential positive benefits fordoing treatment evaluations in yoursetting?

Optional questions

1 If you could change one thing about yourclinical services, what would it be?

2 What are the largest challenges facing yourclinical services?


WHO/MSD/MSB 00.2j

Annex 2

by people who plan to use the workbooks,but who are unable to attend a work-shop. This questionnaire should be com-pleted and returned before the workbooksare released.

The six month evaluation questionnaire(page 96) is designed to be completed sixmonths after the workshop and/or re-ceipt of the workbooks.

The twelve month evaluation question-naire (page 109) is designed to be com-pleted 12 months after the workshop and/or receipt of the workbooks.

Questionnaires

The following pages contain five participantquestionnaires. The purpose of each is describedbelow. For more information about the evaluationcomponent of this project, please see page 11.

The pre-workshop evaluation question-naire (page 80) can be sent to participantsin advance of the workshop, or completedon Day 1. The advantage of sending question-naires in advance (and asking for their returnin advance) is that they can provide valuableinformation about the participants that can beuseful for workshop planning.

The post-workshop evaluation question-naire (page 86) is designed for completion byworkshop participants on Day 5 of the workshop.

The pre-workbook evaluation question-naire (page 91) is designed for completion

Pre-workshop evaluation questionnaireThis workshop and workbook series are be-ing evaluated for their effectiveness in help-ing people like you to conduct your evalua-tions of substance use treatment. Our goal isto make future versions as useful as possible,and you are an important part of this process.

All workshop participants are requested tocomplete evaluations at the beginning and endof the training programme. In addition, we arerequesting that users of our workbook seriescomplete questionnaires about the workbooks

after using them in their own settings. Yourresponses will help us to understand yourneeds and whether our workshop and work-book series is helpful to you.

In addition to this questionnaire, you will beasked to complete a follow up questionnaireabout the workshop and its materials atthe end of the week. Also, follow up ques-tionnaires about the workbook series willbe mailed to you in about six months and 12months.


WHO/MSD/MSB 00.2j

Background Information

Name

Title

Name of yourinstitution

Address

Telephone number

Fax number

E-mail address

Your professionalbackground

Please check ALL boxes that describe your employment activities:

researcher

clinician (direct patient care)

treatment service or network administrator

government planner or policy maker

other

Please check ALL boxes that describe your anticipated role(s) in local evaluationactivities:

evaluation planning

oversight of evaluation implementation

data collection

data analysis

preparation of evaluation report

implementation of changes based on evaluation results

other


WHO/MSD/MSB 00.2j

Questions about workshop

Date of workshop: Location of workshop:Please answer thefollowing questionsas honestly andaccurately as youcan. Respond withwhat you really thinkand what you reallyknow, not with whatyou �should� thinkor �should� know.

How helpful do you think this workshop will be in assisting you to plan your evaluationof substance use treatment?

none very little somewhat very much extremely

How helpful do you think this workshop will be in assisting you to implement yourevaluation of substance use treatment?


What are your concerns or worries about the workshop?

Questions about workbook series

How helpful do you think this workbook series will be in assisting you to plan yourevaluation of substance use treatment?


How helpful do you think this workbook series will be in assisting you to implementyour evaluation of substance use treatment?


What are your concerns or worries about the workbook series?

How would you describe your knowledge of substance use disorders and theirtreatment?

none very little some a lot extensive

How would you describe your knowledge of how to plan treatment evaluations?


How would you describe your knowledge of how to implement treatment evaluations?



WHO/MSD/MSB 00.2j

Rate the extent to which you agree with the following statements:

Neutral AgreeDisagreeStronglydisagree

Stronglyagree

• Treatment evaluations are not needed to knowif substance use treatments are effective.

• Most patients with substance use problemswill not change their behaviour, regardlessof the quality of the treatment programme.

• Treatment evaluations take away limited re-sources from direct clinical care.

• Treatment evaluations frequently result incuts to programme funds.

• We don’t have the experience to do treat-ment evaluations in our setting.

• We don’t have the money to do treatmentevaluations in our setting.

• We don’t have the time to do treatmentevaluations in our setting.

To what extent is your current work related to delivering substance use disordertreatments?

none very little somewhat very much all the time

To what extent is your current work related to conducting treatment evaluations?


N o Yes

Indicate whether you have ever personally completed the following:

• Planned any kind of research study

• Planned a treatment evaluation

• Collected data from any source

• Collected data from patients

• Entered data into a data book

• Entered data into a computer file

• Analysed data by hand

• Analysed data using a computer

• Written an evaluation report

• Published results in a scientific journal

• Presented results orallyThank you!


WHO/MSD/MSB 00.2j

Post-workshop evaluationquestionnaireThis workshop and its materials are beingevaluated for their effectiveness in helpingpeople like you to conduct your evaluationsof substance use treatment. Our goal is tomake future workshops as useful as possible,and you are an important part of this process.

We are requesting that all workshop partici-pants complete questionnaires at the end of

the training programme. Your responses willhelp us to understand whether our workshopwas helpful to you, and how we can improvein the future.

Additional follow up questionnaires about theworkbook series will be mailed to you inabout six months and 12 months.

Background information

Date of workshop: Location of workshop:

Questions about workshop

Name of facilitator working most closely with you during workshop:

Name

Title


Address

Telephone number

Fax number

E-mail address



WHO/MSD/MSB 00.2j

Please answer thequestions on thefollowing pages ashonestly andaccurately as youcan. Respond withwhat you reallythink and feel, notwith what you�should� think or�should� feel.

Overall, the workshop training programme:Neutral AgreeDisagreeStrongly

disagreeStronglyagree

What overall rating would you give to the workshop training programme?

1 2 3 4 5

Lowest Highest

• Allocated time appropriately, given the to-tal time available

• Covered the subject matter in adequatedetail

• Used effective teaching methods

• Presentations were helpful

• Exercises were helpful

• Helped us to plan a better evaluation


Stronglyagree

Would you recommend this workshop training programme to a colleague?

no yes

Why or why not?

Complete the following ratings about the facilitators.

Overall, how helpful were the facilitators?


The facilitators:

• Had sufficient knowledge

• Had sufficient teaching skills

• Created a positive learning atmosphere

• Helped us to plan a better evaluation

Additional comments about the facilitators:


WHO/MSD/MSB 00.2j

Please finish the following sentences.

• The most useful aspect of the workshop training programme was:

• The least useful aspect of the workshop training programme was:

• Unnecessary information was given about:

• I wish more information were provided about:

• Additional comments or suggestions:

Thank you!

Pre-workbook questionnaire

Thank you for your interest in the WHO/UNDCP/ EMCDDA workbook series projecton the costs and effects of treatment for psy-choactive substance use disorders. Theseworkbooks are being evaluated for their ef-fectiveness in helping people like you to con-duct your evaluations of substance use treat-ment. Our goal is to make future versions ofthese workbooks as useful as possible, andyou are an important part of this process.

We are requesting that all users of our work-book series complete questionnaires before

and after using them in their own settings.Your responses will help us to understand yourneeds and to evaluate whether our workbookseries is helpful to you.

Follow up questionnaires will be mailed to yousix months and 12 months following your re-ceipt of the workbook series.

Please complete the following questions andreturn this form to the address listed below.We also welcome any other comments, ques-tions, or suggestions that you might have.


WHO/MSD/MSB 00.2j

Background Information

Please check ALL boxes that describe your employment activities:

researcher

clinician (direct patient care)

treatment service or network administrator

government planner or policy maker

other


evaluation planning


data collection

data analysis



other

Name

Title


Address

Telephone number

Fax number

E-mail address



WHO/MSD/MSB 00.2j

How helpful do you think this workbook series will be in assisting you to plan yourevaluation of substance use treatment?


How helpful do you think this workbook series will be in assisting you to implementyour evaluation of substance use treatment?


What are your concerns or worries about the workbook series?





How would you describe your knowledge of how to implement treatmentevaluations?


Please answer thequestions on thefollowing pages ashonestly andaccurately as youcan. Respond withwhat you really thinkand what you reallyknow, not with whatyou �should� thinkor �should� know.


WHO/MSD/MSB 00.2j



Stronglyagree












N o Yes

Indicate whether you have ever personally completed the following:











• Presented results orallyThank you!

Please return thisquestionnaire to theaddress listed onthe first page.


WHO/MSD/MSB 00.2j

Six month follow-up questionnaireful as possible, and you are an importantpart of this process.

Please complete the following questions assoon as possible and return this form to theaddress listed below. We also welcome anyother comments, questions, or suggestionsthat you might have.

Our records indicate that you received theworkbook series on the costs and effectsof treatment for psychoactive substanceuse disorders six months ago. Accordingly,we are enclosing a questionnaire to help usevaluate whether our workbook series hasbeen helpful to date. Our goal is to makefuture versions of these workbooks as use-

Name

Title


Has your contact information changed? If yes, complete the following:

Address

Telephone number

Fax number

E-mail address


Did you attend a workbook series training workshop?

no yes

If no, how did you receive the workbook materials?


If yes, what was the date and location of the workshop you attended?


WHO/MSD/MSB 00.2j








evaluation planning


data collection

data analysis



other

Please answer thefollowing questionsas honestly andaccurately as youcan. Respond withwhat you reallythink and what youreally know, not withwhat you �should�think or �should�know.



Stronglyagree









WHO/MSD/MSB 00.2j

Has the workbook series contributed to your completion of these activities?

no yes

If yes, in what way?





N o Yes

Indicate whether you have ever (including the past 6 months) personally completedthe following:











• Presented results orally


WHO/MSD/MSB 00.2j

Indicate whether you completed the following tasks within the last six months. Foreach “no” response, please indicate the reason(s) you did not complete the task.

N o Yes• Read the Framework workbook

• Read Workbook 1 (Planning Evalua-tions)

• Read Workbook 2 (ImplementingEvaluations)

• Read any of the Specialised Workbooks(Workbooks 3 - 8)

• Completed “It’s your turn” Exercises

• Formed a local evaluation planningworkgroup

• Used the workbook guidelines to planan evaluation

• Used the workbook guidelines to imple-ment an evaluation

• Finished an evaluation

• Reported the evaluation findings

• Used the evaluation findings to guidechanges in your treatment services

• Planned a second evaluation based onresults from first evaluation

Reason(s) for not completing

Have you engaged in any OTHER form of evaluation training activity (in addition tothis workbook series) over the past 6 months?

no yes

If yes, what type of activity?

Read book(s)

Attended training workshop(s)

Other


WHO/MSD/MSB 00.2j

If you have planned and/or implemented an evaluation using the workbook series, indi-cate the evaluation type:

Needs evaluation

Process evaluation

Cost evaluation

Client satisfaction evaluation

Outcome evaluation

Economic evaluation

Please describe your evaluation here:

Please base your responses to the following questions on your experiences with theworkbook series to date.

Overall, how helpful was this workbook series in assisting you to plan your evaluationof substance use treatment?


Overall, how helpful was this workbook series in assisting you to implement yourevaluation of substance use treatment?


The workbook series:

• was clear to understand

• was easy to use

• was useful for our local needs

• covered all the relevant informationwe needed

• terminology was easy to understand

• writing style was clear


Stronglyagree


WHO/MSD/MSB 00.2j

Would you recommend this workbook series to a colleague?

no yes

Why or why not?

Complete the following ratings about the Introductory/Framework Workbook.

I read the Framework Workbook:

no yes

Overall, how helpful was the Framework Workbook?


Overall, the Framework Workbook:


• was easy to use


• covered all the relevant information weneeded

• contained terminology that was easy tounderstand

• used a clear writing style

• helped us to plan and/or implement abetter evaluation

Additional comments about the Framework Workbook:


Stronglyagree


WHO/MSD/MSB 00.2j

Complete the following ratings about the Foundation Workbooks (Workbooks 1 and 2).

Check the workbook(s) that you read.

• Workbook 1(planning evaluations)

• Workbook 2 (implementing evaluations)

Overall, how helpful were the Foundation Workbooks (Workbooks 1 and 2)?


N o Yes


Stronglyagree

Workbooks 1 and 2:

• were clear to understand

• were easy to use

• were useful for our local needs


• contained terminology that was easy to un-derstand


• helped us to plan and/or implement a bet-ter evaluation

Additional comments about the Foundation Workbooks:

Complete the following ratings about the Specialised Workbooks (Workbooks 3 - 8).


• Workbook 3 (needs evaluations)

• Workbook 4 (process evaluations)

• Workbook 5 (cost evaluations)

• Workbook 6 (client satisfaction evaluations)

• Workbook 7 (outcome evaluations)

• Workbook 8 (economic evaluations)

N o Yes


WHO/MSD/MSB 00.2j

Overall, how helpful were the Specialised Workbooks that you read?


The Specialised workbooks that I read:









Stronglyagree

Additional comments about the Specialised Workbooks:

Complete the following ratings about the “It’s your turn” exercises.

Check the workbook(s) in which you completed “It’s your turn” exercises.

• Framework Workbook

• Workbook 1 (planning evaluations)








N o Yes


WHO/MSD/MSB 00.2j

Overall, how helpful were the “It’s your turn” exercises?


The exercises that I completed:




• enhanced my understanding of theworkbook text

• helped our evaluation group worktogether more effectively

• helped us plan and/or implement a betterevaluation


Stronglyagree

Additional comments about the “It’s your turn” exercises:

Complete the following ratings about the case examples (located in the back of most ofthe workbooks).

Check the workbook(s) in which you read the case example(s).








Overall, how helpful were the case examples?


N o Yes


WHO/MSD/MSB 00.2j

The case examples that I read:









Stronglyagree

Additional comments about the case examples:


The most useful aspect of the workbook series was:

The least useful aspect of the workbook series was:

Unnecessary information was given about:

I wish more information were provided about:

Additional comments or suggestions:

Thank you! Please return this questionnaire to the address listed on the first page.


WHO/MSD/MSB 00.2j

Twelve month follow-up questionnaireOur records indicate that you received theworkbook series on the costs and effectsof treatment for psychoactive substanceuse disorders twelve months ago. Accord-ingly, we are enclosing a questionnaire tohelp us evaluate whether our workbookseries has been helpful to you. Our goal isto make future versions of these workbooks

as useful as possible, and you are an im-portant part of this process.

Please complete the following questions assoon as possible and return this form to theaddress listed below. We also welcome anyother comments, questions, or suggestionsthat you might have.

Name

Title


Has your contact information changed? If yes, complete the following:

Address

Telephone number

Fax number

E-mail address


Did you attend a workbook series training workshop?

no yes

If no, how did you receive the workbook materials?


If yes, what was the date and location of the workshop you attended?


WHO/MSD/MSB 00.2j








evaluation planning


data collection

data analysis



other

Please answer thefollowing questionsas honestly andaccurately as youcan. Respond withwhat you reallythink and what youreally know, notwith what you�should� think or�should� know.



Stronglyagree









WHO/MSD/MSB 00.2j

Has the workbook series contributed to your completion of these activities?

no yes

If yes, in what way?





N o Yes

Indicate whether you have ever (including the past 12 months) personally completedthe following:











• Presented results orally


WHO/MSD/MSB 00.2j

Indicate whether you completed the following tasks within the last 12 months. For each“no” response, please indicate the reason(s) you did not complete the task.

N o Yes• Read the Framework workbook

• Read Workbook 1 (Planning Evalua-tions)

• Read Workbook 2 (ImplementingEvaluations)

• Read any of the Specialised Workbooks(Workbooks 3 - 8)

• Completed “It’s your turn” Exercises

• Formed a local evaluation planningworkgroup

• Used the workbook guidelines to planan evaluation

• Used the workbook guidelines to imple-ment an evaluation

• Finished an evaluation

• Reported the evaluation findings

• Used the evaluation findings to guidechanges in your treatment services

• Planned a second evaluation based onresults from first evaluation

Reason(s) for not completing

Have you engaged in any OTHER form of evaluation training activity (in addition tothis workbook series) over the past 12 months?

no yes

If yes, what type of activity?

Read book(s)

Attended training workshop(s)

Other


WHO/MSD/MSB 00.2j

If you have planned and/or implemented an evaluation using the workbook series, indi-cate the evaluation type:

Needs evaluation

Process evaluation

Cost evaluation

Client satisfaction evaluation

Outcome evaluation

Economic evaluation

Please describe your evaluation here:

Please base your responses to the following questions on your experiences with theworkbook series to date.

Overall, how helpful was this workbook series in assisting you to plan your evaluationof substance use treatment?


Overall, how helpful was this workbook series in assisting you to implement yourevaluation of substance use treatment?


The workbook series:


• was easy to use


• covered all the relevant informationwe needed




Stronglyagree


WHO/MSD/MSB 00.2j

Would you recommend this workbook series to a colleague?

no yes

Why or why not?

Complete the following ratings about the Introductory/Framework Workbook.

I read the Framework Workbook:

no yes

Overall, how helpful was the Framework Workbook?


Overall, the Framework Workbook:


• was easy to use






Additional comments about the Framework Workbook:


Stronglyagree


WHO/MSD/MSB 00.2j

Complete the following ratings about the Foundation Workbooks (Workbooks 1 and 2).


• Workbook 1(planning evaluations)


Overall, how helpful were the Foundation Workbooks (Workbooks 1 and 2)?


N o Yes


Stronglyagree

Workbooks 1 and 2:





• contained terminology that was easy to un-derstand


• helped us to plan and/or implement a bet-ter evaluation

Additional comments about the Foundation Workbooks:

Complete the following ratings about the Specialised Workbooks (Workbooks 3 - 8).








N o Yes


WHO/MSD/MSB 00.2j

Overall, how helpful were the Specialised Workbooks that you read?


The Specialised workbooks that I read:









Stronglyagree

Additional comments about the Specialised Workbooks:

Complete the following ratings about the “It’s your turn” exercises.

Check the workbook(s) in which you completed “It’s your turn” exercises.

• Framework Workbook









N o Yes


WHO/MSD/MSB 00.2j

Overall, how helpful were the “It’s your turn” exercises?


The exercises that I completed:





• helped our evaluation group worktogether more effectively

• helped us plan and/or implement a betterevaluation


Stronglyagree

Additional comments about the “It’s your turn” exercises:

Complete the following ratings about the case examples (located in the back of most ofthe workbooks).

Check the workbook(s) in which you read the case example(s).








Overall, how helpful were the case examples?


N o Yes


WHO/MSD/MSB 00.2j

The case examples that I read:









Stronglyagree

Additional comments about the case examples:


The most useful aspect of the workbook series was:

The least useful aspect of the workbook series was:

Unnecessary information was given about:

I wish more information were provided about:

Additional comments or suggestions:

Thank you! Please return this questionnaire to the address listed on the first page.


WHO/MSD/MSB 00.2j

Annex 3

General criteria for sites

Written evaluation plancriteria and checklists

To help ensure that you are covering all im-portant areas in your written evaluationplan, we have developed a specific struc-ture for what to include. Following thisstructure will help you to organise yourthoughts clearly. It also will demonstrate tous that you have thought through the im-portant aspects of your proposed evalua-tion, and that you are ready and able to com-plete this particular evaluation.

Remember, we want to assist you to de-sign an evaluation plan that is meaningfuland feasible, given your unique situation

and priorities. A meaningful evaluation planis one that is logically related to theprogramme logic model. A feasible evalua-tion plan is one that is simple, reasonable forthe budget and other available resources, andachievable within the one-year time frame.

As you prepare your written evaluationplan, use the workbooks that were given toyou at the workshop. The workbooks aredesigned to assist you to independently de-velop your evaluation. They provide infor-mation and suggestions on all aspects ofplanning.

Structured outline for written evaluation plans

Organise your written evaluation plan according to these sections:

Section 1. Background and general purpose (approximately one-half page)

This is a summary of your evaluation plan. Itshould include a brief description of the pur-pose of the evaluation, how you plan to con-duct the evaluation, who will receive the re-

sults, and how results will be used. Do notpresent specific details in this section; simplyprovide the reader with a brief overview.


WHO/MSD/MSB 00.2j

Section 2. Programme logic model (one page)

This section describes the programme, ser-vice, or system that you plan to evaluate inthe form of a programme logic model. Referto Step 3 of Workbook 1 for more informa-tion about this description.

The model should describe:• The type(s) of services that you provide• Your process or implementation objectives• The short-term and long-term goals of

various parts of your programme.

Section 3. Evaluation team (approximately one page)

This section lists the people who will be in-volved with your evaluation project. For eachperson, give the following information:

• Name• Professional title• Educational background• Type of experience (if any) conducting

evaluations - brief summary only• Anticipated roles in the proposed evaluation

Important: Indicate the person (one only,please) who is responsible for communicat-ing with us throughout your evaluation. Pro-vide that person’s mailing address, telephonenumber, and if available, fax number and emailaddress.

Section 4. Evaluation question(s) (approximately one page)

This section lists your specific evaluation ques-tions. Refer to Step 5 of Workbook 1 for someexamples of well-phrased evaluation questions.

Important:

a) your evaluation questions should followlogically from your programme logicmodel. In other words, each of your ques-tions should be linked to one of yourprogramme’s implementation objectives orgoals in a specific way.

b) your evaluation questions should be writ-ten in the form of a question, not a state-ment.

c) for each evaluation question, you shouldindicate what type of evaluation you areconducting through addressing thisquestion.

Here are two examples of how to do this:

Programme objective or goal Evaluation type

Implementation objective: Toprovide initial assessments topatients in a timely manner.

What percentage of patientsreceive initial assessmentswithin 2 weeks of contactingour programme?

Process evaluation(Workbook 4)

Is problem drinking reducedfollowing participation in ourprogramme?

Long-term goal: To reduceproblem drinking among pa-tients.

Outcome evalua-tion (Workbook 7)

Evaluation question

Write your evaluation questions in this sec-tion using the same format. This will showus how your evaluation questions are linkedto your programme logic model, and whattype of evaluation you are conducting.

Once you have identified your evaluationtype(s), refer to the appropriate specialisedworkbook for information and examples ofhow to complete that kind of evaluation.


WHO/MSD/MSB 00.2j

Do not phrase your question in the form of astatement. For example, “We are going toassess the total number of people with sub-

stance use problems in our community” is notan acceptable evaluation question (becauseit is not a question at all).

Section 5. Data collection strategy (approximately two pages of text, not includingyour data collection instruments)

This section is very important, because it isyour road map for how you are going to getfrom your evaluation questions to meaningfulresults. Many evaluators make the mistakeof writing this section too vaguely, assumingthat details can be decided as the evaluationprogresses. This attitude places the entireevaluation in jeopardy, because details areforgotten frequently and/or procedures are notfollowed in a standardised manner through-out the evaluation.

Follow this guideline: could an outside con-sultant read your data collection strategy sec-tion and then be able to implement your evalu-ation exactly as you are planning? If not, youneed to be more specific here.

Refer to Steps 6 and 7 of Workbook 1 forinformation and suggestions about creatingyour data collection strategy. For each evalu-ation question you listed in section 4 (above),you should indicate:

• Whether you will use a standardised datacollection instrument. If so, indicate whichone(s), and state the specific variables(subscales or other summary scores) youplan to use.

• Whether you will develop your own datacollection instrument. If you plan to useyour own instrument, you need to be veryspecific about exactly what you plan toask. Include the specific questions andscoring criteria you plan to use. Note: youmay refine your data collection instrumentafter pilot testing, but it is still important topresent your plan for specific questionsand scoring criteria here.

• How you will collect the data (self-reportquestionnaire, interview, focus group, re-view of records, etc.)

• Who will administer the data collectioninstrument(s)

• How many participants you will assess

• Your sampling strategy: how you willchoose participants for your evaluation

• Your time line for data collection (howmany weeks/months)

Section 6. Ethical concerns (approximately one-half page of text, not including yourconsent form)

This section covers relevant ethical issues.Most important, you should provide docu-mentation that you have received approvalto conduct your evaluation from your localethics committee. You also should presentyour plan for how you will keep participantinformation confidential. If you will use aconsent form with your participants, in-

clude it here. (Some evaluations, such asthose using record reviews, may not requirea consent form.)

Refer to Step 1-A of Workbook 2 for moreinformation and suggestions for how to man-age ethical issues during your evaluation. Asample consent form also is presented.

Section 7. Data management plan (approximately one page)

You should explain your record keeping sys-tem, including how you will store your dataand maintain central records.

Refer to Step 1-B of Workbook 2 for moreinformation and suggestions for how to de-velop a data management plan.


WHO/MSD/MSB 00.2j

Section 8. Staff training (approximately one-half page)

In this section, explain if any additional staffwill be needed for the evaluation. Statewhether you anticipate needing outside con-sultants, and if so, for how long and in whatcapacity.

Regardless of whether new staff or consult-ants will be needed, describe how you will

train all evaluation staff (data collectors,record keepers) to conduct their jobs in astandardised way.

Refer to Step 1-B of Workbook 2 for moreinformation and suggestions for how to en-sure standardisation of your procedures.

Section 9. Pilot test (approximately one-half page)

This section should describe your specificplans for conducting a pilot test of your evalu-ation. What type of a pilot test will you con-duct, and how many people will participate ineach portion of it? If you are using a newlydeveloped data collection instrument, you

should plan to include it as part of your pilottesting, and then make changes to it as neededafter the pilot testing.

Refer to Step 1-C of Workbook 2 for moreinformation and suggestions.

Section 10. Data analysis (approximately one page)

This section presents an overview for howyou plan to analyse your data. You shouldindicate the specific statistical tests you planto use to answer each of your evaluationquestions (section 4). State the specificanalysis you are planning for each evalua-tion question.

Selected information regarding data analysisis presented in Step 3 of Workbook 2. If youthink that you will need further assistance tochoose specific statistical tests and/or con-duct the data analysis, state this clearly here.Skip descriptions of specific statistical tests ifyou are unsure.

Section 11. Reporting results (approximately one-half page)

Describe how you will present your results,and the format you will use to present them.Will you present the information in oral formand/or written form, and what kind of vi-

sual aids (e.g., bar graphs) will be used tosummarise results? Refer to Step 4 ofWorkbook 2 for more information on report-ing results.

Section 12. Strategy for using results (approximately one page)

This section is very important, because it ex-plains how your evaluation results can be ap-plied to the long-term goal of improving sub-stance use treatment. Explain who will receiveresults, and how this information could be usedto make changes to improve treatment.

Remember, evaluation is an ongoing pro-cess of asking questions and applying re-

sults. We do not expect that this singleevaluation will be the only step needed toimprove treatment in your setting. In real-ity, multiple evaluations may be needed be-fore large changes can be made. However,you should explain clearly how these initialevaluation results can serve as a beginningstep to enhance treatment.


WHO/MSD/MSB 00.2j

Section 13. Budget (approximately one page)

• Have you indicated clearly how your re-sults will be used to improve substance usetreatment?

• Can your team complete the proposedevaluation with available resources?“Available Resources” includes money,necessary equipment and facilities, andavailability of outside consultants ifneeded.

• Can the proposed evaluation be com-pleted within the expected time frame?

• Are ethical concerns adequately ad-dressed?

• Other impressions

It is likely that sites will need to revise theirwritten evaluation plans at least one timeafter reviewers have read them.

Given the amount of money available, de-scribe how you plan to allocate the fundsto complete the evaluation. If your site isreceiving additional funding for this evalu-ation, include the relevant information inyour budget plan.

If you will be receiving payment to completeyour evaluation, the first instalment will beauthorised upon successful completion of thewritten evaluation plan. “Successful comple-tion” depends upon reviewers’ assessment ofthese factors:

• Have you followed the structured outlinefor written evaluation plans? Are all sec-tions present and complete? (If not, yourplan will be returned to you for revision.)

• Is the accompanying checklist completedand included with the plan?

• Are your evaluation questions relatedmeaningfully to your programme logicmodel?

• Is your proposed data collection strategysufficiently detailed?

• Are your proposed data collection and dataanalysis strategies adequate to answer yourevaluation question(s)?

• Considering your plans for pilot testing,staff training, and record keeping, will yourdata collection proceed smoothly and pro-duce reliable information?

Section 14. Time line (approximately one page)

Present a time line for when you will accom-plish each step of your evaluation.

Total written evaluation plan page length:approximately 12.5 pages of text (programmelogic model, data collection instruments, andconsent form extra).


WHO/MSD/MSB 00.2j

Written evaluation plan checklistThis checklist is intended to assist siteparticipants to ensure that pertinent in-formation is included in the written evalu-

ation plan. Please complete this check-list and return it with your written evalu-ation plan.

Evaluators’ names and proposal title

Please check that the following components are included in your plan:

Section 1: Background and general purpose

Section 2: Programme logic model

• programme logic model included

Section 3: Evaluation team

• every evaluation team member included

• person responsible for communication is indicated

Section 4: Evaluation questions

• each evaluation question is linked to a specific component of the programmelogic model

• evaluation questions are written in the form of questions

• evaluation type is indicated for each evaluation question

Section 5: Data collection strategy for each evaluation question, state:

• data collection instrument (indicate whether standardised or newly developed)

• specific variables to be used

• specific questions and scoring criteria for all newly developed data collectioninstruments

• method for data collection (interview, self-report, etc.)

• number of participants

• sampling strategy

• time line for data collection


WHO/MSD/MSB 00.2j

Section 6: Ethical concerns

• documentation of ethics committee approval

• plan for keeping participant information confidential

• consent form for participation (if needed)

Section 7: Data management plan

• explanation of record keeping system

• explanation of data storage plan

Section 8: Staff training

• indication if additional staff will be needed

• indication if outside consults will be needed

• explanation of staff training procedures

Section 9: Pilot test

• type of pilot test

• number of participants for pilot test

Section 10: Data analysis

• specific statistical tests to be conducted, and/or

• indication of whether consultant will be needed to choose/conduct data analysis

Section 11: Reporting results

• plan for reporting results

Section 12: Strategy for using results

• who will receive results

• specific changes that could be made using results

Section 13: Budget

Section 14: Time line


WHO/MSD/MSB 00.2j

Written evaluation plan checklistreviewer version

Evaluators’ names and proposal title

This checklist is intended to assist re-viewers to thoroughly assess sites’ writ-ten evaluation plans. Use this checklist

as a guide while reviewing site plans andformulating recommendations.

Check if “yes”:

• Is the site checklist completed and included with the plan?

• Are all 14 sections included in the plan as requested?

• Are the evaluation questions linked meaningfully to the programme logicmodel?

• Is the proposed data collection strategy sufficiently detailed?

• Is the proposed data collection strategy adequate to answer the evaluationquestions?

• Considering proposed plans for pilot testing, staff training, and record keeping,will data collection proceed smoothly and produce reliable information?

• Is the proposed data analysis strategy adequate to answer the evaluationquestions?

• Has the site indicated clearly how results will be used to improve treat-ment?

• Can the site team complete the proposed evaluation with their availableresources? “Available Resources” includes money, necessary equipmentand facilities, qualifications and experience of evaluation team members,and projected availability of outside consultants.

• Can the proposed evaluation be completed within the expected time frame?

• Are ethical issues adequately addressed?

Other impressions or concerns:

Yes


WHO/MSD/MSB 00.2j

Annex 4

Overhead prototypes

Day 1


WHO/MSD/MSB 00.2j

Summary of the workbook seriesIntroductory WorkbookFramework Workbook

Foundation WorkbooksWorkbook 1: Planning EvaluationsWorkbook 2: Implementing Evaluations

Specialised WorkbooksWorkbook 3: Needs Assessment EvaluationsWorkbook 4: Process EvaluationsWorkbook 5: Cost EvaluationsWorkbook 6: Client Satisfaction EvaluationsWorkbook 7: Outcome EvaluationsWorkbook 8: Economic Evaluations

WHO/UNDCP/EMCDDA

Project on costs andeffects of treatment forpsychoactive substanceuse disorders


WHO/MSD/MSB 00.2j

Fears and expectationsWorking individually, list on theprovided cards:

1) your expectations about the workshop

2) your fears or concerns

Use different card colours for expectations vs. fears.Write legibly and use large print, so your cards canbe read from a distance. Do not put your name on thecards.

Why is treatmentevaluation important?••••• Treatment services are growing

••••• Despite scarcity, resources aregiven to ineffective treatments

••••• Treatment evaluation improvesquality of care and saves money forservices that are effective


WHO/MSD/MSB 00.2j

Pros and cons of evaluation1. Each site should work separately andwrite on flip chart paper the pros andcons of doing treatment evaluation at itslocal site. Allow 15 minutes to completethis step.

2. Each site should meet with at thetriplet level to discuss the pros and consthat they wrote.

Day 2


WHO/MSD/MSB 00.2j

The 8 steps of planningan evaluation1. Decide who will be involved in the evaluation.2. Assess your resources.3. Identify and prioritise the evaluation needs.4. Describe your programme for evaluation.5. Define your evaluation questions.6. Determine your evaluation measures.7. Determine your evaluation design.8. Ensure that your resources are sufficient.

If not, return to Step 3.

Possible evaluation partners••••• patients/clients••••• family members••••• senior managers••••• board members••••• staff (therapists/clinicians)••••• researchers••••• funder(s)••••• representatives of other programmes or

service systems in your community••••• people living in your community


WHO/MSD/MSB 00.2j

Evaluation resources

••••• Financial/Material

••••• Expertise

••••• Time

Developing aprogramme logic modelTypes of objectives

IMPLEMENTATION

(Process) Concerned with the program itself. Level/quality of services. The meanswhereby outcomes are to be achieved and the target group(s).

OUTCOME What you hope to change. The ends you pursue.

• • • • • Immediate

• • • • • Short-term

• • • • • Medium term

• • • • • Long-term

Ü

ÜÜÜ Goals


WHO/MSD/MSB 00.2j

Types of objectives (cont'd)

Implementation • to establish ............... for• to provide .................. to• to prescribe .............. for• to teach ..................... to• to give ........................ to

Outcome: • to improve• to decrease/increase• to change/modify

Basic program logic modelMain components(e.g. assessment, detoxification, follow-up)

Implementation objectives(e.g. to determine correct diagnosis, toprovide substance resistance skills, tomonitor health status)

Planned short-termoutcome objectives(e.g. to increase motivation for furthertreatment, to decrease the likelihood ofrelapse)

Planned long-termoutcome objectives(e.g. to decrease substance use, toimprove quality of life)

ÜÜ

ÜÜ

ÜÜ

ÜÜ Ü


WHO/MSD/MSB 00.2j

• To reduce drug use behaviour

• To prevent consequences of drug use

• To improve mental and physical well-being of clients

• To increase clients knowledgeabout further treatmentpossibilities

• To increase clients knowledgeabout consequences of PSUand AIDS

• To maximise the number ofclients who are referred to long-term treatment

• To cope with withdrawalsymptoms

• To stabilise mental andphysical status

• To maximise clientsatisfaction with theprogram

• To reduce involvement incriminal activities

• To collect a databaseof standardised clientinformation

• To improve quality ofcare

Long-termOutcomeObjectives

Short-termOutcomeObjectives

• To collect thenecessary informationfor development ofadequate treatment

• To enhance clients�motivation to stay inthe detoxificationprogram

• To provideinformationabout otherprograms forfurther drug-freetreatment

• To motivateclients tocontinuetreatment in oneof theseprograms

• To monitor withdrawalsymptoms

• To prescribe standardmedication

• To provide a safe andsupportive environment

• To conductlaboratory/other tests

• To motivate client tofinish the program

• To provide safewithdrawal management

• To formulatetreatment plan

• To signtherapeuticcontract

• To confirm clients�eligibility for theprogram

• To determine clients�motivation to engagein treatment

• To determine clientsindividual needs

• To obtain standardsomatic, mental andpsychological status ofthe client

• To form therapeuticalliance with the client

ImplementationObjectives

ReferralMainComponents

Assessment &Intake

Diagnosis &TreatmentPlanning

WithdrawalManagement &Treatment

Ü Ü Ü ÜÜÜÜÜ

Ü Ü Ü


WHO/MSD/MSB 00.2j

Day 3

Objective 2

EvaluationQuestions

EvaluationMeasures

DataCollectionStrategy

ProgramObjectives(from thelogic model)

Objective 1

Basic format for moving from logic model tomore detailed evaluation planning


WHO/MSD/MSB 00.2j

Evaluation types

••••• Needs assessment••••• Process evaluation••••• Cost evaluation••••• Client satisfaction••••• Outcome evaluation••••• Economic evaluation

The specialised workbooks

If you are conducting a� Then you should review�

Needs Assessment Workbook 3

Process Evaluation Workbook 4

Cost Evaluation Workbook 5

Client Satisfaction Workbook 6

Outcome Evaluation Workbook 7

Economic Evaluation Workbook 8


WHO/MSD/MSB 00.2j

What are indicators?ProgramComponent

Client Assessment

ImplementationObjective

Outputs(indicators of servicedelivery and characteristicsof those served)

Why bother with indicators?

Indicator#2 Indicators#3

(1)Trained

assessmentworker isavailable

(2)Comprehensive

Assessmentprotocol

completed

(3)Treatment planprepared and

client assignedto treatment

Indicator#1Ü Ü Ü

Ü Ü Ü

Claim ConfidenceLevel

Indicator

�Our PSawareness iseffective�

Very low I heard it through the grapevine

Low I talked to John, who participated and he said itworked for him

High Compared to pre-test scores, scores on an alcoholknowledge test went up by an average of two points

Medium 40 out of 50 people in the programme checked off�effective� in a satisfaction questionnaire completedat the end of the programme

Highest Compared to a control group of people who weren�texposed to the programme at all, participants in theprogramme increased scores on seven out of eightmeasures of PS awareness


WHO/MSD/MSB 00.2j

Reliabilityrefers to whether an indicator isconsistent across time and/orobservers

Validityconcerns the extend to which youare actually measuring what youintend to measure

Types of measures

••••• Observation••••• Questionnaires••••• Interviews••••• Focus group discussions••••• Examining routine records


WHO/MSD/MSB 00.2j

Language of measurement

Quantitative approach:measurement of variablethrough numbers

Qualitative approach:measurement of variablethrough words

You Me

Variable

Indicator

A trivial example...

Height

Mark on the wall

[Variable]

Tape measure [Two indicators]

[Data]

[The abstract output or outcome of interest]

[Different types of measures of the output or outcome]

[The specific observations you make]

Indicator

01 02 03 04 01 02 03 04

You Me


WHO/MSD/MSB 00.2j

Prepare for data collection

••••• Set a time frame

••••• Determine a sampling procedure1) Representativeness2) Randomness

Sampling procedures

1.Accidental2.Reputational3.Random4.Stratified5.Cluster6.Quota


WHO/MSD/MSB 00.2j

Writing good questions1. Are the words simple, direct and familiar to all?2. Is the question as clear and specific as possible?3. Is it a double question?4. Does the question have a double negative?5. Is the question too demanding?6. Are the questions leading or biased?7. Is the question applicable to all respondents?8. Is the question objectionable?9. Will the answers be influenced by response styles?10.Have you exhausted the response alternatives?

Day 4


WHO/MSD/MSB 00.2j

The 6 steps ofimplementing an evaluation

1. Prepare for data collection.

2. Collect data.

3. Analyse data.

4. Report results.

5. Make use of what was learned.

6. Start again.

Prepare for data collectionManage ethical issues••••• Informed consent••••• Manage risks of participation••••• Confidentiality

Develop a data management plan••••• Design a record keeping system••••• Acquire and process data••••• Train people who will collect data••••• Standardisation

Conduct a pilot test••••• Identify flaws••••• Revise data collection plan as needed

Write an evaluation plan


WHO/MSD/MSB 00.2j

DefinitionsFrequency Number of responses in each category

of an indicator

Mode Score that occurs most frequently

Median Score that separates the upper half of scoresfrom the lower half of scores

Mean Average score

Range Lowest score to highest score

Variability Extent to which scores deviate from theircentral tendency

Indices • Variability• Standard deviation

Descriptive vs.Explanatory analysisDescriptive analysis

Goal:To summarise the measurementsfor each relationships betweenvariable

Examples:How many clients reported�?How much time did staff spend...?How much variability wasthere in�?

Explanatory analysis

GoalTo explain relationship betweenvariables and groups

Examples:Do male and female clients differ...?Compared to clients assigned to acontrol group, were clients exposedto... more likely to...?


WHO/MSD/MSB 00.2j

Structure for sitePresentationsSection I. Programme logic model and programme descriptionBriefly show your programme logic model components. Indicate thepart of the model that is the focus of the evaluation.

Section II. Evaluation question, measures, data collectionShow the one or two most important evaluation questions and howyou will answer them (evaluation plan)

Section Ill. Using the results for decision-makingDescribe who will receive the results from the evaluation and how itwill be used for decision making.

Day 5


WHO/MSD/MSB 00.2j

Written evaluation plan1. Background and general purpose2. Programme logic model3. Evaluation team4. Evaluation questions to be addressed5. Data collection strategy6. Data management plan7. Staff training8. Pilot test9. Strategy for using results

facilitator™s workshop guide - who | world health...

Documents