economic evaluations - emcdda home page · full economic evaluations require two or ... studies...

1Workbook 8 · Economic Evaluations

WHO/MSD/MSB 00.2i

EconomicEvaluations

Workbook 8

2 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2i

WHOWorld Health Organization

UNDCPUnited Nations International Drug Control Programme

EMCDDA European Monitoring Center on Drugs and Drug Addiction

World Health Organization, 2000c

This document is not a formal publication of the World Health Organization (WHO) and all rights are reserved bythe Organization. The document may, however, be freely reviewed, abstracted, reproduced and translated, inpart or in whole but not for sale nor for use in conjunction with commercial purposes. The views expressed indocuments by named authors are solely the responsibility of those authors.


WHO/MSD/MSB 00.2i

Acknowledgements

The World Health Organizationgratefully acknowledges the con-tributions of the numerous indi-viduals involved in the preparationof this workbook series, includingthe experts who provided usefulcomments throughout its develop-ment for the Substance Abuse De-partment, directed by Dr. MaryJansen. Financial assistance wasprovided by UNDCP/EMCDDA/Swiss Federal Office of PublicHealth. Alan Ogborne (Canada)wrote the original text for thisworkbook and Brian Rush(Canada) edited the workbook se-ries in earlier stages. JoAnne

Epping-Jordan (Switzerland) wrotefurther text modifications and editedthe workbook series in later stages.Munira Lalji (WHO, SubstanceAbuse Department) and JenniferHillebrand (WHO, Substance AbuseDepartment) also edited the work-book series in later stages. Thisworkbook’s case examples werewritten by David Cooper (UnitedKingdom), Ambros Uchtenhagen etal. (Switzerland), and edited byGinette Goulet (Canada). MaristelaMonteiro (WHO, Substance AbuseDepartment) provided editorial inputthroughout the development of thisworkbook.


WHO/MSD/MSB 00.2i


WHO/MSD/MSB 00.2i

Table of contents

Overview of workbook series 7

What is an economic evaluation? 8

Why do an economic evaluation? 9

How to do a cost evaluation: general steps 10

1. Defining the economic question and theperspective of the study 112. Determining the treatments to be evaluated 113. Choosing the study design 134. Identifying, measuring and valuing the costs of the alternativetreatments 195. Identifying, measuring and valuing the benefits of the alternativetreatments 216. Adjusting costs and benefits for differential timing 247. Measuring the incremental costs and benefits 248. Putting the costs and benefits of the alternatives together andanalysing the results 259. Testing the sensitivity of results 26It�s your turn 27

Conclusion and a practical recommendation 29

Sample questionnaires 30

Discounting tables 31

References 31

Comments about case example 32

Case example of economic evaluationCost-effectiveness evaluation of substance misure interventions 33It�s your turn 37References for case example 39


WHO/MSD/MSB 00.2i

Introductory WorkbookFramework Workbook

Foundation WorkbooksWorkbook 1: Planning EvaluationsWorkbook 2: Implementing Evaluations

Specialised WorkbooksWorkbook 3: Needs Assessment EvaluationsWorkbook 4: Process EvaluationsWorkbook 5: Cost EvaluationsWorkbook 6: Client Satisfaction EvaluationsWorkbook 7: Outcome EvaluationsWorkbook 8: Economic Evaluations

Overview ofWorkbook SeriesThis workbook is part of a seriesintended to educate programmeplanners, managers, staff and otherdecision-makers about the evalua-tion of services and systems for thetreatment of psychoactive substanceuse disorders. The objective of thisseries is to enhance their capacityfor carrying out evaluation activities.The broader goal of the workbooksis to enhance treatment efficiencyand cost-effectiveness using the in-

formation that comes from theseevaluation activities.

This workbook (Workbook 8) isabout economic evaluation. Eco-nomic evaluations involve the iden-tification, measurement and valu-ation, and then comparison of thecosts (inputs) and benefits (out-comes) of two or more alternativetreatments or activities.


WHO/MSD/MSB 00.2i

What is aneconomicevaluation?Economic evaluations involve the identifica-tion, measurement, and valuation, and thencomparison of the costs (inputs) and ben-efits (outcomes) of two or more alternativePSU treatments or activities.

In economic evaluations, the costs and con-sequences of alternative interventions or sce-narios are compared to examine the best useof the scarce resources. The specific ques-tion being addressed may be:

• a comparison of the costs and benefits ofa new intervention to some current thera-peutic approach

• a comparison of the costs and benefitsbetween treatment and prevention activi-ties

• a comparison of the costs and benefitsbetween treatment and law enforcementactivities

Economic evaluations differ according totheir scope and intent. They can have avery narrow focus, whereby evaluators areonly concerned about the resource conse-quences for their agency. In these evalua-tions, any new intervention which shiftscosts to another agency may be preferred.Alternatively, economic evaluations canexamine wider social costs. In these evalu-ations, a new intervention that shifts costsbut does not reduce total costs may nothave good “value”. Similarly, for healthagencies, the outcomes of prime impor-tance are likely to be the health of the indi-vidual user. From a societal perspective,however, the costs of crime and other so-cial effects may be of greatest concern.


WHO/MSD/MSB 00.2i

Why doan economicevaluation?

Using any resource for the treatment of PSUdisorders means the opportunity to use thatresource for something else is lost. There-fore, cost-effectiveness (or “value” for moneyspent on treatment services) is of central con-cern in most health care and government sys-tems. Economic evaluation is one of the toolsavailable to help choose wisely from a rangeof alternatives and implement efficient re-sources.

In general, economists prefer the widestpossible societal perspective: general ques-tions about the use of scarce resources andsocietal well-being. However, in certaincases, policy makers may wish to knowthe answers to narrower questions, forexample, restricting the perspective tohealth outcomes and health care expendi-ture, or restricting it to a specific area, forexample, the effects on crime and the crimi-nal justice system.

Economicevaluation isone of thetools availableto help choosewisely from arange ofalternativesand implementefficientresources.

Full economic evaluations are rarely com-pleted. One reason is that economic evalua-tions are resource intensive and typically re-quire a high level of research expertise. It isimportant, prior to undertaking this type ofstudy, to determine whether a full economicevaluation is warranted or required. For someresearch questions, answers can be ad-dressed through a cost evaluation (Work-book 5), which is generally less intensive tocomplete.

Full economic evaluations should only beundertaken after an initial analysis to gaugethe usefulness of the study. Prospective eco-nomic analyses are best undertaken along-side other evaluations, particularly outcomestudies (Workbook 7). In themselves, eco-nomic components of research need not beexcessively expensive. There is, however,great merit in examining the economic de-sign from the beginning of a research plan-ning process as results may affect the overalldesign of the study as well as the detail ofdata collection.


WHO/MSD/MSB 00.2i

How to do acost evaluation:general steps

Before doing an economic evaluation, it

is important to review Workbook 1 of this

series, which outlines general steps to

evaluation planning. The specific steps

for undertaking economic evaluations

are outlined in this section.


WHO/MSD/MSB 00.2i

The main steps are:

1. Defining the economic question and theperspective of the study

2. Determining the treatments to be evaluated

3. Choosing the study design

4. Identifying, measuring and valuing the costs ofthe alternative treatments

5. Identifying, measuring and valuing the benefitsof the alternative treatments

6. Adjusting costs and benefits for differentialtiming

7. Measuring the incremental costs and benefits

8. Putting the costs and benefits together andanalysing the results

9. Testing the sensitivity of the results


WHO/MSD/MSB 00.2i

1. Defining the economic question andthe perspective of the study

Your choice of study questions will dependon the specifics of your situation and yourevaluation priorities. Use Workbook 1 tohelp you define your specific evaluation goals.Some pros and cons of different approachesare discussed below.

You may want to know the cost of your ownprogramme, in your own organisational con-text. While this approach is likely to gener-ate some useful data, results generated fromsuch a study cannot be generalised acrossservices.

Alternatively, you may want to know whethera new therapy should be adopted. The pro-viders of substance use service may be in-terested in the detailed analysis of the costsand consequences for their own organisation.Funders of services may be interested in thewider implications for health service deliveryunder their jurisdiction. National or state au-thorities may be more interested in the soci-etal perspective. It is important that you as-sess the potential consequences of taking anynarrower perspective than that of society.

Questions about adopting a new treatmentcan, depending on your situation, be put in anumber of different ways. For example, youmay have a set budget for substance use treat-ments. In this situation, your question maybe whether a new treatment can deliver morebenefits within the same budget constraint asother existing approaches. In other situations,you may be more concerned with meeting ahealth target and then the question may beposed as to which type of therapy involvesthe least net cost for some target level ofbenefit.

More general questions, for example, aboutthe overall level of funding of substance usetreatments in a local area, or whether thebalance between prevention and treatmentfunding is bringing the highest health gains,require a broader approach and perspective.Answering such broad questions requiresdata at a more aggregate level and assumesthat many of the detailed evaluations havealready been undertaken.

2. Determining the treatments to beevaluated

Full economic evaluations require two ormore PSU treatments for evaluation. Thechoice of treatments is a very important partof the evaluation process. An economicevaluation is not useful if a potential treat-ment of greater benefits and lesser costs hasbeen omitted. On the other hand, it is im-possible, for practical reasons, to evaluateall possible alternative PSU treatments.

A clear evaluation question for all new treat-ments is: what are the costs and outcomes ofthe new treatment compared to current prac-

tice? (Sometimes this is interpreted as cur-rent “best” practice.) For more fundamentalquestions of the value of treatment, the ques-tion may imply a comparison with a no-treat-ment option. This implies, however, a fullevaluation of the no-treatment alternative. Forexample, a certain proportion of the groupmay stop taking PS or reduce the harm as-sociated with their PSU without formal treat-ment. However, obtaining data on a samplereceiving no-treatment is difficult, and re-search including such a no-treatment optionmay be deemed unethical. In practice, there

Given thecurrent scarcityof economicevaluations,it is importantto generate abody of well-conductedresearch.However, suchstudies need tobe resourced atan appropriatelevel.


WHO/MSD/MSB 00.2i

is usually some attempt to evaluate the newtherapy compared to some form of minimalintervention.

The case example located at the end of thisworkbook presents an evaluation of a rela-tively new case approach versus “usual care”for people with severe mental illness and PSdependence. “Usual care” was defined asparticipation in a community-based PSU sup-port group (alcoholics/narcotics anonymous).

Some questions require more complexevaluation designs. For example, you maywant to evaluate a system of treatments orsome form of stepped care. These types ofquestions lend themselves to a decision-treeapproach. Zarkin and colleagues (1994) cre-ated one structure to consider the impact ofevaluating outcomes and costs over morethan one treatment episode. This approachassumes that a PSU client will have repeatedencounters with treatment services through-out his/her lifetime, and helps to identify al-ternative PSU policy interventions that mightaffect outcomes. Considering any one PSUclient’s lifetime history, he/she will be on aparticular “branch” of a decision tree at anygiven time (e.g., A) stopped PSU after initialtreatment vs. B) continued to use PS afterinitial treatment and was admitted for furthertreatment vs. C) continued to use PS afterinitial treatment but refused additional treat-ment. By estimating the proportion of the totalclient population that may follow each ofthese paths, you can examine the effect ofpotential policy changes in terms of numbersof clients affected, costs, and expected out-comes. For example, you can assess the pro-portion of your PSU population that is clas-sified into category C (above), and estimatewhether an intervention directed at motivat-ing this category of patients to return to treat-ment is as cost effective as directing similarresources at initial intervention efforts.

Alternative structures could be composedlooking, for example, at the potential differ-

ent outcomes from compliance to mainte-nance therapies. Glazer and Ereshefsky(1996) present a model with antipsychotictherapy that could be adapted. In their model,the first step is to identify all the treatments incurrent clinical practice, then to identify thepossible outcomes of each of the treatmentalternatives (e.g., client will comply or notcomply, client will remain abstinent from PSUor begin using PSU again, etc.). The next stepis to establish the estimated probability ofeach of the outcome combinations (or “path-ways”) for each treatment alternative. Ide-ally, these probabilities should be derivedfrom previous outcome evaluations (Work-book 7). Costs for each treatment alterna-tive are compiled separately. Finally, costsfor each treatment alternative are comparedby multiplying the costs associated with eachoutcome pathway by the cumulative prob-ability that a client will reach this particularoutcome. After repeating this process foreach of the outcome pathways associatedwith each treatment alternative, costs can beadded together to yield the total cost of thattreatment strategy.

Another approach would be to use a deci-sion analysis to evaluate a stepped careprogramme where “failures” are entered intodifferent or progressively more intensivetherapies. The advantage of the decision- treeapproach is that thought must be given to allthe alternative courses. The disadvantage isthat data are required on the probabilities ofoutcomes at different stages and for evalua-tion to be feasible at each of these stages.

Given the current scarcity of economic evalu-ations, it is important to generate a body ofwell-conducted research. However, studiessuch as these need to be resourced at anappropriate level. The danger is that with sofew studies conducted, the results of a studythat is designed to answer a very specificquestion, with very selected alternatives be-ing considered, may be inappropriatelygeneralised.


WHO/MSD/MSB 00.2i

3. Choosing the study design

Many existing studies fall short of full eco-nomic evaluations. Drummond and col-leagues (1997) outlined different types ofpartial evaluations and emphasised two char-acteristics necessary for full economic evalu-ation:

• a comparison of two or more alternatives

• both costs and benefits of the alternativesare considered

No intervention can be cost-effective if it isnot effective in terms of clinical outcomes (seeWorkbook 7). Therefore, the most robustdesign for a full economic evaluation is arandomised controlled trial. This is the de-sign that was used in the case example evalu-ation located at the end of this workbook.Other evidence of cost-effectiveness is lessrobust. In particular, studies using differencesbetween before and after treatment, with nocontrol group, tend to overestimate benefitsof treatment. These benefits are even moreprone to overestimation if only those whocomplete treatment are included in the study.

Ideally, cost data is collected at the same timeand with the same degree of accuracy asoutcome data. While this is increasingly be-coming the practice, most studies have ei-ther attempted to estimate costs for alterna-tive therapies retrospectively, or model costsand consequences for the alternatives beingconsidered using literature reviews of effec-tiveness data and models of resource costs.For more complex structures, or where thereare longer term benefits, some modelling andmodel predictions will always be required.Any modelling or predictions require someassumptions to be made.

Part of the study design stage involves thechoice of an economic evaluation method:cost-minimisation; cost-effectiveness; cost-utility or cost-benefit (described below).There is a need to match the choice of eco-

nomic analysis to the questions being ad-dressed in the analysis. For example, if thequestion is about the best way to improvethe health of PSU users, a cost-effectivenessdesign may be adopted. However, if therewas a need to make wider comparisons itmay be more appropriate to use a cost-util-ity framework. For studies attempting to lookat the full range of costs and benefits of pro-viding treatment compared to no treatment,the most appropriate design may be cost-benefit analysis.

There are four types of full economic evalu-ation:

• cost minimisation

• cost-effectiveness

• cost-utility

• cost-benefit

The main difference between the four types offull economic evaluation is how the benefits tothe individual are measured and valued.

Cost minimisation analysis

In cost minimisation, the effect of the alter-native interventions on the individuals health-related quantity and quality of life are as-sumed to be equal. In these studies, all otherresource consequences are measured inmonetary terms. Some of these resourceconsequences, such as reduced future levelsof crime or health care costs, can be seen as“benefits” of the intervention, whereas otheraspects such as the direct costs of the inter-ventions can be clearly defined as “costs.”Published studies vary in the name given tosome of the non-individual “benefits” — insome studies these are considered as part ofthe cost calculations but as benefits these sumsare subtracted from other costs to give a netcosts total.

Only wherethere are twoor morealternatives,and both costsand benefitsare examined,is the studyclassified as afull economicevaluation. It isonly from fulleconomicevaluationsthat questionsabout value formoney can beaddressed.


WHO/MSD/MSB 00.2i

If two interventions have the same individualhealth effects, then the one which can be judgedas the most value for money will be the inter-vention which minimises the net costs. It is, how-ever, a strong assumption to assume that indi-vidual health effects are the same between twoor more alternative treatments (or treatmentscenarios if more complex questions are beingposed). It would be an even stronger assump-tion to include all other benefits of treatment asequal. The advantage of the cost minimisationapproach is that the measurement problem isreduced to just examining resource conse-quences. However, the assumptions are diffi-cult to justify prior to any experimental study.This method, therefore, has only limited appli-cation within the PSU field.

An example of a cost minimisation study isan examination of several standard metha-done facilities in which client group andexpected consequences are assumed to bethe same. The figures in the table on thenext page illustrate some potential results.They are taken from a study by Bradleyand colleagues (1994), and it should bestressed that this study was only concernedwith costs and no claims were made aboutthe outcomes of the different sites. Of thethree sites reported in the table below, SiteA was hospital based whereas Sites B andC were free standing facilities. In this case,Site A is the cost minimising option, be-cause of the lower average staff costs.

Cost-effectiveness Analysis

The majority of the published economicevaluations have been cost-effectivenessanalyses. In this type of economic evalua-tion, the effect of treatment is measured ina single natural health unit. Costs and otherconsequences also are measured in mon-etary terms in the same way as for costminimisation analysis. The requirement foran economic study to have a single, prin-cipal outcome measure is needed to con-

The advantageof the costminimisationapproach isthat themeasurementproblem isreduced to justexaminingresourceconsequences.However, theassumptionsare difficult tojustify prior toanyexperimentalstudy.

Site A Site B Site C

Capital cost

Rent and maintenance

Staff costs

Telephone, office supplies, utility costs etc.

Contracted services(laboratory tests, pharmacists, accountants etc)

TOTAL Costs

Number of clients

Average cost per client

Illustrative cost minimisation analysis for 3 standard methadonetreatment programmes (1990)

17,190

44,876

663,257

111,298

131,645

968,266

250

3,873

83,107

102,326

843,323

199,206

237,205

1465,167

333

4,400

105,340

6,508

414,812

222,273

39,339

788,272

210

3,754

struct some cost-effectiveness ratio indi-cating the net costs required for each unitof outcome. For some health care inter-ventions, the natural health unit outcomemeasure may be best reflected by deathsavoided or gains in life years. Most PSUstudies have used some measure of PSUrather than a health measure, for example,the net costs per abstinent day, or per per-centage reduction in PSU.


WHO/MSD/MSB 00.2i

One down side of this technique is that usinga single measure is that the total effects ofPSU treatment may not be reflected in anyone health or PSU variable. Further, manyof the effects of treatment on the individualmay have wider impact on the quality of lifethan just that of health. Hence, a narrow uni-dimensional outcome measure used as acomparison may fail to “measure” the fullimpact of the different therapies and lead toa misleading conclusion on the relative“worth” of the therapies under consideration.

The choice of outcome measure not only af-fects the validity of the study, but also theuse of study results. PSU quantity measuresmay be preferred by therapists as being theonly measures relevant to their client group.However, funders of treatment could not usesuch studies to examine the comparativeworth of expanding PSU treatments vs. ex-panding vaccination programmes because ofthe lack of a common generic health statusmeasure.

To illustrate the use of cost-effectivenessstudies, consider the following example of theeffectiveness of brief interventions comparedto a control intervention for those drinkingalcohol above a low risk level. In this ex-ample, both costs and effects are measuredas the excess over the control intervention.

• Systematic reviews of the effectivenessevidence suggest that alcohol consump-tion is reduced on average by 20 per centfollowing brief interventions. Assume thatin a hypothetical problem- PSU popula-tion, this would translate into a reductionof 6.02 alcohol units per person perweek. (One unit is equal to 8 grams ofalcohol).

• Based on these results, administratorswant to implement a screeningprogramme and delivery opportunisticbrief interventions in a primary care set-ting for 100 men and 100 women. Be-fore doing so, they want to understandthe cost-effectiveness.

In this typeof economicevaluation,the effect oftreatment ismeasured in asingle naturalhealth unit.Costs andotherconsequencesalso aremeasured inmonetary termsin the sameway as for costminimisationanalysis.

• The screening costs of applying an alco-hol use questionnaire, 2 minutes, are cal-culated to be between £0.8 and £2.40per person, total costs £160 to £480.

• The questionnaire is estimated to suggestthat 46 people would need a brief inter-vention (36 true positive and 10 falsenegative)

• It is estimated that 15 minutes is neededto deliver each intervention, with addi-tional costs of leaflets, etc., giving a costof each brief intervention of between £8and £20

• For the 46 people receiving the interven-tion, this yields a service cost of between£368 and £920

• Total programme costs including thescreening are therefore between £528 and£1400

• The cost per at risk drinker is between£14.6 and £38.9

• Expressing this in terms of effectivenessevidence yields an average extra costs of£2.4 to £6.5 per “unit” of reduced alco-hol consumption.

Note this example is conducted from the pri-mary care perspective and only a limitedrange of cost and consequences are exam-ined.

Cost-utility Analysis

All resources have an opportunity cost: op-portunities to do something different with re-sources are lost when resources are com-mitted in a certain direction. Within healthcare, there is a need to make decisions onthe balance of resources, for example, be-tween terminal care and prevention interven-tions. Such comparisons, however, requiresome common outcome measures that canincorporate quantity and quality of lifechanges. Such measures can be seen as mea-sures of utility (or value of health) to indi-viduals. Economic evaluations using suchoutcome measures are hence called cost-utility studies.


WHO/MSD/MSB 00.2i

Drummond et al (1997) suggest this methodshould be used when quality of life is theimportant outcome. PSU cost-utility studiesmight involve the evaluation of social careprogrammes designed to help individualswho have been in long-term residentialprogrammes. Or, it might be used to com-pare interventions that have effects both onthe length and quality of life. Finally, thereare those programmes that have a range ofdifferent outcomes arising from interventionsand some common measure is required tomake comparisons between them.

There are a number of different aspects toconstructing and using health utility measuresin economic evaluations. It is necessary toidentify, measure, and value the health gainsfrom any extension of life and improved qual-ity of life. Some treatments may improve bothaspects, but others may influence only thelength of life or the quality of life. Whereascost-effectiveness studies measure the out-come a particular point in time, for exampleone year after treatment ends, cost-utilitymeasures must estimate how long the treat-ment effects will last.

Most cost utility studies measure qualityadjusted life years (QALYs) among theirparticipants. A QALY is based on the ideathat categorising people merely as “alive” or”dead” (i.e., quantity of life) does not cap-ture adequately multiple states of health, orquality of life, that exist in individuals’ livesfollowing PSU treatment. QALYs assign thescore of 1.000 to a (hypothetical) personwho is in a state of perfect health. Then, de-ductions from 1.000 are taken for differentsymptom reports while answering quality of lifequestions. For example, use of a cane may re-duce a person’s QALY by.060 (1.000 -0.060), while wheezing or shortness or breathmay reduce QALY by.257 (1.000 - 0.257).

Note that not all QALYs are calculated inthe same way. Scoring for different ques-tionnaires may be based on different ideasabout what constitutes quality of life. Forexample, is regular fainting “worse” thanchronic pain? Different quality of life mea-sures will “weight” or value these items dif-ferently. Different QALY measures alsohave different health dimensions. TheEuroQol EQ-5D, for example, has 5 di-mensions: mobility, self-care, usual activ-ity, pain/discomfort and anxiety/depression(Dolan et al., 1995). Another measure, theWHOQOL-BREF, has four dimensions:physical health, psychological, social rela-tionships, and environment. Copies of bothmeasures and scoring instructions are lo-cated in Workbook 1, Appendix 2.

All other costs and resource consequencesare measured and valued in a similar wayas in all the other types of economic evalu-ations.

As an example of this type of study, con-sider the three treatment programmes de-scribed in the previous cost-minimisationtable. If a cost utility study of these threeprogrammes were completed, it would havebeen possible to chart both differences in anyoverdose or other mortality while in theprogramme and the improvement in generalhealth.

Using this example, assume that Site C (withgreater proportionate staff input) resulted ineach client having large health improvementsusing a standard quality of life measures.Combining the health and reductions in mor-tality yielded the following average total healthgains in the three programmes over a yearprogramme. No future health gains werethought to arise from these programmes, aminimum estimate position.

.. this methodshould beused whenquality of lifeis theimportantoutcome.

Site A 160 QALYs (210 original clients)

Site B 291 QALYs (333 original clients)

Site C 345 QALYs (250 original clients)


WHO/MSD/MSB 00.2i

Combining these findings with cost data(assuring the original cost figures were ad-justed for the premature mortality (and

hence reduced annual cost) yielded thefollowing average cost per QALY figuresshown in the table below.

All other costs and consequences are as-sumed to be the same between the sites. Inthis example, Site C has a far lower cost perQALY than the other two sites. Given thedifferent mortality rates, there would be dif-ferential health gains in future years even if allexisting clients reverted back to former lev-els of PSU.

Cost-benefit Analysis

In cost-benefit analysis, all individual benefitsare measured in monetary terms. This meansthat all costs and consequences are measuredin the same units. The method is useful whenthere are a wide range of diverse outcomesassociated with the treatments being evalu-ated. Because the results can be expressedin terms of whether the monetary value ofbenefits outweighs the costs, such studies areoften seen to provide more powerful argu-ments for implementing programmes (or not)than other forms of economic evaluation.However, the relevance of any study to de-cision-making depends on the alternativeoptions being evaluated and the scope of theevaluation.

Measuring health gains in monetary terms issometimes viewed as problematic. For ex-ample, market values of the value of life,based on foregone earnings have beenthought to undervalue some groups in soci-ety, particularly older and poorer people.This method of valuation is now rarely used.Other methods include using market values

Illustrative example of a cost utility study

Total Programme QALYs gained Average costcosts ($) per QALY ($)

Site A 788,272 160 4,927

Site B 1,465,167 291 5,035

Site C 968,266 345 2,807

of risk or asking individuals to put monetaryvalues on different health states using a will-ingness-to-pay approach (see Johannessonet al. (1996) for a review of these methods).

An illustration of the results gained by will-ingness to pay methods can be drawn lossesassociated with fatal road accidents in theUnited Kingdom. The government agencyreviewed the available estimates from all dif-ferent methods, and choose a value in 1987of £500,000 per life. This compared to a fig-ure of £283,000 calculated using a foregoneearnings method. In 1996 terms, this con-verts to a value of £23,000 for each lost(gained) life year.

Willingness-to-pay methods may be seen asan alternative measurement system to thatused for utility measures. The differencesbetween them may be in the weighting sys-tem used for different groups of the popula-tion. Utility measures usually have an equityelement built in, with one quality adjusted lifeyear being deemed of equal value to all indi-viduals. This is not always the case with mon-etary measures as some may be biased togiving greater weight to those with more in-come. This method may be appropriate inhealth care systems were individuals are re-sponsible for paying for health care.

French and colleagues (1996) have proposedrecently a method for estimating the mon-etary value of PSU treatments. The method-ology they proposed is a mixture between

In cost-benefitanalysis, allindividualbenefits aremeasured inmonetaryterms.


WHO/MSD/MSB 00.2i

using health utility measures and monetaryvaluations. Estimates of quality adjusted lifeyears are calculated for different age/gen-der and race cohorts and a dollar value ofa QALY applied. The average value wastaken from assuming a value $5 million forthe statistical life of an average white maleat age 38. In essence, the methodologyproposed could be used within either acost-utility or cost-benefit analysis. Differ-ent disease outcomes were related to ageneral index of values of different healthstates. For example, compared to a per-fect health value of 1 and death as 0, amoderately severe case of Hepatitis B wasthought to generate a value of 0.96 over 2months duration. This is an interesting “lowcost” methodology for estimating individualhealth gains. It may have useful applica-tions for PS injection users who are vul-nerable to a number of different diseaseswith measurable consequences. It may bepossible to provide reasonable estimatesof the “avoided cases” by treatment. How-ever, these data would need to be esti-

mated from available epidemiological dataand it is not clear that “avoided disease” canbe accurately estimated for the range of dif-ferent PSU. Also, by concentrating onavoided disease, the measure may fail to cap-ture the full individual benefit of treatment.This may be particularly important for treat-ments applied to less dependent users.

The fullest study examining both costs andbenefits in monetary terms was conductedby Gerstein and colleagues (1994). Thiswas a before and after study without acontrol group and therefore can becriticised on methodological grounds. Thestudy, however, illustrates the size of po-tential gains to a number of agencies. Con-sequences included criminal justice costs,an estimate of victim losses from crime,health care cost and productivity conse-quences. The summary figures are given inbelow, showing all modalities apart frommethadone discharge resulted in greaterbenefits than costs for an average episodeof treatment.

CALDATA examples of cost benefit analysis

Residential Social Model Outpatient Methadone MethadoneDischarge Continuing

Savings per Day During $22.19 $12.79 $10.60 $14.14 $29.68

Savings per Day After $24.51 $14.43 $7.50 $-3.79 N/A

LOS (average) 69 79 150 60

Cost per Day of Treatment $61.47 $34.41 $7.87 $6.79 $6.37

Total Cost Per Episode $4,405 $2,712 $990 $405 $(2,325)

Total Benefits $10,744 $6,509 $2,853 $-1,206 $(10,833)

Benefits to Cost 2.44 2.40 2.88 -2.98 4.66


WHO/MSD/MSB 00.2i

Cost-offset, cost-outcomeand other types of partialeconomic evaluations

There also are a number of valuable partial,economic evaluations that have been under-taken for PSU treatments. These are givendifferent names in the literature including cost-offset and benefit-cost studies (often referredto somewhat erroneously as cost-benefitstudies).

• An outcome description, a cost de-scription or a cost-outcome descriptionis a study that examines only one treat-ment (or possibly one treatment system).

• A cost analysis is a study that considersone or more alternatives, and only costsare examined (see Workbook 5).

• Cost-offset studies examine the costs ofdifferent treatments or treatment systemsand the consequent impact on future healthcare costs. The idea being tested is that

the costs of substance use treatments canbe fully or partly offset by reductions infuture health care costs. The purpose ofthese studies is to lend support for the in-clusion of coverage of alcohol and othersubstance use treatments in insuranceplans. It is an attempt to partially addressthe question is treatment worthwhile. Ingeneral, the analysis has involved obser-vational data on individuals through timeand comparing health care costs beforeand after treatment.

Undertaking a partial economic cost study isone practical way to collect data with limitedresources. Such data can give a broad pic-ture of a service, although great caution isneeded in making any comparisons eitherwithin the service or across services. Suchstudies provide some evidence on the broadworth of treatments while not being of suffi-cient rigour to answer more detailed ques-tions on how services could be changed toyield more benefits with less resource use.

4. Identifying, measuring and valuingthe costs of the alternative treatments

Undertakinga partialeconomiccost study isone practicalway tocollect datawith limitedresources.

The table on the next page outlines all thepotential costs of a PSU treatment. The fourgeneral areas identified where costs mayoccur are:

• costs to service providers

• costs to the individuals and families intreatment

• costs to other agencies or individuals

• productivity costs


WHO/MSD/MSB 00.2i

Checklist of costs of substance use treatments

1. Costs to service providers

• Capital

- land

- buildings

- equipment

• Running costs

- paid staff

- volunteers

- administrative and managerial costs

- consumables including drugs prescribed and their dispensing costs, toxicologycosts etc.

2. Costs to the individuals and their families in treatment

• Out of pocket expenses

- travelling and other direct expenses

- contribution to treatment costs (if not included in A)

• Leisure time and other costs associated with input to treatment Costs

- pain, distress etc. associated with changing habits, or with process of treatment

3. Costs to other agencies or individuals

••••• Referrals to other health or social agencies linked to the treatment

••••• Increases in potential problems associated with treatment

- leakage of prescribed drugs to illicit markets

4. Productivity costs


WHO/MSD/MSB 00.2i

The areas that may be included in your analysiswill depend on your situation. For example,an evaluation from the health service per-spective may only concentrate on the cost toproviders and any other consequent cost forother health agencies. If a wider perspectiveis taken, more variables may be analyses. Inthese situations, it is important to avoid doublecounting. For example, individuals may loseincome while undertaking treatment. Includ-ing an allowance for the full cost of lost pro-ductivity from the time spent in treatmentwould mean that the lost labour “resource“would be erroneously counted twice. It shouldalso be noted that the list refers only to “re-source“ costs – those actions which meanthere is a loss of scarce raw materials, land,labour or capital. Many PS users may be inreceipt of welfare payments from the state— these are transfers from one group (thetaxpayer) to another group. Changes intransfer payments are not included in eco-nomic evaluations. However, such changesmay be of prime interest to state or na-tional governments and may need someseparate analysis.

It may be relatively easy to measure and valuethe provision of the costs of treatment. Someother potential consequences pose moreproblems. For example, it is often difficult totrace the full impact of different treatmentson other agencies. Alternatively, certain treat-ments may be associated with more distressboth to the individual and their family. While

it may not always be possible to measure andvalue all these effects, some analysis can beundertaken to check whether they are likelyto differ among the alternatives being evalu-ated.

A controversial area is whether costs arisingfrom lost productivity while in treatmentshould be included in different economicevaluations. For long residential treatments,the estimates of these costs can be consid-erable. A sizeable group of substance usersin treatment may have been unemployed forsome time and therefore the valuation of thisitem will depend crucially on whether theestimates are based on some unadjustedvalue of time, or adjusted for labour marketdemand conditions – i.e. adjusted for the riskof unemployment.

A number of questions also arise in applying“values“ to material resources. For example,some resources may be more expensive inrural areas because of transportation costswhereas others, including buildings, may bemore expensive in urban areas because ofscarcity. It is helpful to present results in re-source use terms as well as applying mon-etary values so that individual readers canrelate the results to their own situation.

More detail on how individual treatmentsmay be costed and further discussion ofsome costing issues are contained in Work-book 5.

It may berelatively easyto measureand value theprovision ofthe costs oftreatment.Some otherpotentialconsequencespose moreproblems.

5. Identifying, measuring and valuing thebenefits of the alternative treatments

In the table on the next page, the range of pos-sible benefits that may arise from treatmentis outlined. The five broad areas are:

• direct health benefits to the individual

• non-health improvements in quality of lifefor the individual and family

• reduced use of other health care interven-tions

• benefits to other agencies

• productivity benefits


WHO/MSD/MSB 00.2i

Consequences of PSU Treatments

1. Direct health benefits to the individual

• Quality and quantity of health improvements

- exact measurement depending on economic analysis type

- associated with reduction in drug use

- reduced risk of injection-transmitted disease

- more healthy lifestyle in general

- less any adverse effects of treatment

2. Non-health improvements in quality of life for the individualand family

••••• Reduction in PSU- related violence

••••• Improvements in social functioning

••••• Other benefits to the family

3. Reduced use of other health care interventions

4. Benefits to other agencies

••••• Reduced use of resources from other social care and welfare services

••••• Reduced criminal justice system costs

••••• Benefits net of any adverse consequences to “community and socialenvironment”

5. Productivity benefits

••••• Benefits in individual productivity as a result of the treatment


WHO/MSD/MSB 00.2i

Most studies are likely to include a numberof different individual outcome measures. Forexample, many controlled trials may includesome measure of substance use, some spe-cific substance-related outcome measure anda more generic health status instrument. Foreconomic evaluations, there may be a needto value the health benefits and a need tochoose both the method of valuation and thegroups from which values are sought, forexample, from PS users or the whole popu-lation.

Most existing general health measures arefocused on health-related quantity and qual-ity of life changes. This makes sense becausemaximising health gain is a very importantobjective. PSU, however, affects a numberof other dimensions of life quality of bothusers and their families. Some PSU-relatedoutcome measures have attempted to includesome of these dimensions. There is a needto evaluate whether such “non-health” ben-efits to PS users can be measured and val-ued perhaps through “willingness to pay” orpreference-based measures on total healthand non-health-related quality of life. At thecurrent time, there are no such ready-to-useeconomic measures.

The current body of cost-offset and costoutcome studies suggest that one of the con-sequences of individuals receiving treatmentis the reduction in demand for other healthservices. These potential gains could be size-able. The ease by which health use can bemeasured varies with health care systems.Where there is some charging mechanism,even if individuals do not pay directly be-cause of social insurance, there may berecords of all health care use, including theresource cost of the different treatment epi-sodes. However, in many countries it maybe necessary to ask individuals about theiruse of health care over a period before, dur-ing and after treatment and then use averagevalues of the costs of such use. Cost-offsetstudies using insurance record data can trackindividuals within the plan over considerableperiods. This would be more difficult toachieve with self-report data. It is not clearhow accurate recall would be over long pe-

riods especially of frequently used health ser-vices. This is one of the important challengesto researchers.

As well as the benefits to the health care sys-tem, PSU treatments also are likely to re-duce the use of other social care and welfareagencies. It should be noted again that theseresources relate to services received ratherthan changes to welfare benefits, which aretransfer payments not resource costs. Thedifficulty for evaluators is tracing such changesin use and finding some means of valuing thediverse range of effects that may arise. Itwould of course be far too costly and timeconsuming to individually trace all potentialeffects. Some may be excluded from theanalysis but the effects of such exclusions hasto be considered carefully. It is clear, for ex-ample, that there is a large impact on the coststo the criminal justice system from treatingsome dependent drug users. It would be im-portant to consider whether the alternativetreatments being evaluated had differentialeffects on crime before it was decided toexclude them, even if the main focus of thestudy was the impact on health care services.Some important benefits from successfultreatments may be more and difficult to mea-sure. For example, an adequate system ofservices for PS users may well have afavourable impact on a community and theenvironment over and above some of thereductions in direct problems. For example,reductions in crime rates may also producereductions in the fear of crime among com-munity members. Not all consequences oftreatments may be beneficial. It is importantto consider possible negative consequencesof the alternative interventions being evalu-ated.

Finally, there are the benefits that result fromgains in productivity. As with productivitycosts, there is some debate about the inclu-sion of such effects and if included how theyshould be measured. Treatment is likely toimprove employment prospects and increasethe productivity of those in work. However,the actual changes will depend on the stateof the local labour market.

PSU...affects anumber of otherdimensions oflife quality ofboth users andtheir families.

Not allconsequencesof treatmentsmay bebeneficial.It is importantto considerpossiblenegativeconsequencesof thealternativeinterventionsbeingevaluated.


WHO/MSD/MSB 00.2i

6. Adjusting costs and benefitsfor differential timing

For all costs and consequences, there is aneed to consider the time period over whichany effects will be measured and valued. Inresearch terms, observed follow-ups aregenerally limited in time. In some instances,epidemiological data may be available tomodel plausible outcomes over time includ-ing an allowance for relapse. In other cases,assumptions may have to be made and arange of results presented. It is these typesof issues and how they are resolved whichillustrate some of the assumptions and com-promises that have to be made in practicaleconomic evaluations.

Many of the effects from treatment may lastmore than one year. This is particularly trueif the interventions under study extend the lifeof the participants. However, we tend to puta lower value on events occurring in the fu-ture than those that occur in the current year.One step in an economic evaluation is toconvert all costs and benefits to a “presentvalue” so that they can be compared. Thisprocess is called discounting. While there isgeneral agreement on the need to discountmost resource consequences, there is lessagreement on whether future health benefits

should be discounted. Is a life year saved inthis year worth twice as much as a life-yearsaved ten years from now? Or should all life-years saved be treated as equal even if thesaving does not occur until 20 years in thefuture? In practice, most guidelines suggestboth discounted and undiscounted figures forhealth benefits should be made available toreaders of the study. Discount rates are usu-ally based on current financial interest rates.Applying discount rates to data is relativelyeasy and tables are available for convertingfigures across a number of years to presentvalues. Tables from Drummond et al., 1997,are contained at the end of this workbookfor reference purposes.

To illustrate, consider potential gains fromeliminating alcohol related road traffic deathsin one year. In England and Wales, this wouldresult in a total gain of 148,500 life years in a30 year period. Discounting the future gainsof life years at 5 per cent per annum, how-ever, reduces the total to a figure of 85,800life years. If a higher discount rate of 10 percent was used, the calculated health gain re-duces to 57,000 life years, less than half theundiscounted figure.

7. Measuring the incremental costsand benefitsLists of costs and benefits are createdin terms of totals (or average per indi-vidual) associated with each of the al-ternative treatments. For economicevaluations, it is important to knowwhether the costs or benefits vary withthe level of service provided. Measur-ing the incremental (or extra) costs andbenefits as more treatment is undertakenis important for this task.

Clearly, not all costs rise at the same rate asthe number of treatment admissions increases.There are some fixed costs, such as build-ings and equipment. As numbers rise, thesefixed costs are spread over a larger numberuntil some capacity limit is reached. At thiscritical point, however treating a few extrapeople can involve a large amount of extraresources. One of the largest inputs into treat-ment is therapists’ time. This can be regarded


WHO/MSD/MSB 00.2i

Marginal costs for a year from standard methadone treatments (1992)

as semi-fixed and obviously the extra costsof treating an extra client fall until the thera-pist has a full load.

Other costs and benefits also may changeaccording to the level of activity. It is there-fore important to undertake additionalmeasurement and modelling of costs and ben-efits to fully understand the processes of differ-ent therapies as they expand or contract.

Illustration of incremental costs was given inthe study by Bradley et al (1994): see thetable below. In this example, an extra clientis assumed only to require moreconsumables, such as methadone. However,an extra 25 clients would require more staffand other programme costs, although notextra building space. In this example, Site Chas the lowest marginal costs for 1 and 25clients.

For some research questions, the whole studycould be framed in an incremental way. Manypolicy decisions concern how much eachprogramme should be funded rather than achoice of funding only one. The incremental

approach would be an appropriate way ofevaluating some stepped programme of care.In this case, the additional benefits and costsof providing extra units of care against thealternative of no additional care.

8. Putting the costs and benefits of thealternatives together and analysingthe resultsAfter valuing all the costs and benefits of thealternatives, discounting the sums to presentvalues, and completing some incrementalanalysis, the results still need to be broughttogether. Review Workbook 2 for morethorough information about analysingand reporting research results.

In cost-minimisation studies, the analysisinvolves straight comparisons of the costs (ornet costs if some resource, rather than indi-vidual health benefits, have been included).

For cost-effectiveness and cost-utility stud-ies, there are two ways of presenting results.Cost-effectiveness ratios are generally calcu-lated by comparing the net costs divided by theindividual health outcomes of one programmeto the costs divided by the individual healthoutcomes of the second programme. For ex-ample, one intervention could result in £1000per quality-life-year (QALY) gained and an-other £1200 per QALY. These ratios are gen-erally based on average costs and benefits.

Site A Site B Site C1 Client 25 Clients 1 Client 25 Clients 1 Client 25 Clients

Methadone & otherclient related costs 1,019 49,571 776 41,909 164 28,090

Staff – 296 – 1,335 1,785

Other costs – 5,118 – 11,547 15,822

Marginal costs 1,019 54,986 776 54,791 164 45,697


WHO/MSD/MSB 00.2i

The alternative is to compare costs and out-comes directly:

(Costs1 - Costs

2 )/ (Outcomes

1 - Outcomes

2)

to give a net figure for the difference be-tween programme 1 and 2 (if the design ofthe study is comparing two alternatives inthis way). This is usually calculated usingincremental costs and benefits and canclearly be adapted for some of the morecomplex economic evaluation designs (seeDrummond et al., 1997).

With a simple study comparing two alterna-tive programmes, there are four possible re-sults:

• Programme 1 has more benefits and lowercosts than programme 2 (at all levels ofimplementation). In this case programme1 clearly dominates programme 2.

• Programme 2 has more benefits and lowercosts than programme 1 and 2 will domi-nate 1.

• Programme 1 has more benefits but alsomore costs than programme 2. In this casethe decision is not so simple and may de-pend on whether incremental figures showan advantage of one over another.

• Programme 2 has more benefits and morecosts than programme 1. Again decisionis unclear.

With cost-benefit analysis, the benefits andcosts are measured in the same unit, moneyvalues, and the results may be presented asthe net benefits of the alternatives (benefits -costs) or in terms of cost-benefit ratios (costs/benefits). Ratios are not particularly usefuland can be manipulated because some ben-efits can be redefined as averted costs andaffect the ratio. Ratios do not give any ideasof the size of the scale of benefits or costs.This may be important when comparingprogrammes.

The results will be more complex if a morecomplex design is used. Similarly, there maybe subsidiary analysis to consider if the de-sign of the study allows. For example, it maybe possible to consider whether costs orbenefits vary with severity or other charac-teristics. However, unlike in other areas, thereare no standard measures for case-mix forsubstance use services.

9. Testing the sensitivity of results

Undertaking a full economic evaluation re-quires a large number of assumptions to bemade. It is important to have some ideawhether the overall results of the study wouldvary if different assumptions had been taken.Some assumptions can be tested systemati-cally by using sensitivity analyses. This mayinvolve, for example, using different levels ofeffectiveness varying the main cost variablesor using different discount rates and assess-ing the impact on the results.

As Drummond et al. (1997) suggest, sen-sitivity analysis may be needed when esti-

mates are subject to debate. This may oc-cur if no estimates are available, the esti-mates are subject to imprecision, or thereis methodological controversy such asthose surrounding discount rates. Upperand lower bounds on estimates for the sen-sitivity analysis can be set by using evidencefrom other studies, current practice in theliterature or by soliciting judgements fromthose who will be making decisions. Cal-culations can be made using a combina-tion of best guess, most conservative andleast conservative estimates.


WHO/MSD/MSB 00.2i

Put the information from this workbook touse for your own setting. Complete theseexercises below.

Remember to use the information fromWorkbooks 1 and 2 to help you complete afull evaluation plan. Review that informationnow, if you have not already done so.

1 Decide whether a full economic evalua-tion is needed or warranted, given yourresearch questions and your research re-sources. Is a partial economic evaluation(reviewed in this workbook) or a costevaluation (Workbook 5) more appro-priate?

2 Decide the scope of your study and thetreatment alternatives that you want toevaluate. Will you conduct an economicevaluation within an agency, across sev-eral agencies, or evaluate wider socialcosts?

••••• Within an agency

••••• Across several agencies

••••• Wider social costs

It�s your turn

3 Using the information contained in thisworkbook, choose the study design thatis most appropriate for your researchquestions and resources.

••••• Cost-minimisation

••••• Cost-effectiveness

••••• Cost-utility

••••• Cost-benefit

••••• Partial economic evaluation

4 List programme cost sources that youwant to evaluate. If evaluating servicesacross agencies, decide the commonmeasurement(s) you will use. Meet withplanners from the other agency(ies) toachieve consensus on the evaluationmethods.

We have started the list as an aide foryou. Cross out the sources that do notapply to your situation, and add othersthat are not already listed.

1) Costs to service providers

••••• Capital:- land- buildings- equipment- vehicles

••••• Running costs:- paid staff- volunteers- administrative and managerialcosts


WHO/MSD/MSB 00.2i

- building related expenditure(heating, lighting, maintenance,etc.)- consumables including drugsprescribed and their dispensingcosts, toxicology costs etc.

2) Costs to the individuals and theirfamilies in treatment

••••• Out of pocket expenses:- travelling and other direct ex-penses- contribution to treatment costs(if not included in A)

••••• Leisure time and other costs as-sociated with input to treatmentcosts:- pain, distress etc. associatedwith changing habits, or with pro-cess of treatment

3) Costs to other agencies or individuals

••••• Referrals to other health or so-cial agencies linked to the treat-ment

••••• Increases in potential problemsassociated with treatment- leakage of prescribed drugs toillicit markets

4) Loss of patient productivity costs

5 Decide how you will assess the “benefits”of treatment. This will depend partly on theevaluation design that you choose (Exer-cise 3), and may include a combination ofcost, quality of life, and other outcomedata. Determine what information youhave available, and what other informa-tion you will still need to find out. If youneed to collect additional data, decidewhat method you will use to do this. Re-view Workbook 2 to help you choose anappropriate data collection measure.

Here’s a list to get you started. Cross outitems that you will not measure, and addothers as needed.

1) Direct health benefits to the individual

••••• Quality and quantity of healthimprovements:

- exact measurement dependingon economic analysis type- associated with reduction indrug use- reduced risk of injection-trans-mitted disease- more healthy lifestyle in general- less any adverse effects of treat-ment

2) Non-health improvements in quality oflife for the individual and family

••••• Reduction in PSU - relatedviolence

••••• Improvements in social func-tioning

••••• Other benefits to the family

3) Reduced use of other health care in-terventions

4) Benefits to other agencies

••••• Reduced use of resources fromother social care and welfareservices

••••• Reduced criminal justice systemcosts

••••• Benefits net of any adverse con-sequences to “community andsocial environment”

5) Productivity benefits

••••• Benefits in individual productiv-ity as a result of the treatment

6 Review what you have planned inthese exercises. Will your plans an-swer your research questions? Areyour plans realistic, given your re-search resources? If not, make modi-fications as needed.


WHO/MSD/MSB 00.2i

Conclusionand a practicalrecommendationIn this workbook, we have outlined the ba-sic principles and practices of economicevaluations within PSU services and sys-tems. In undertaking economic evaluations,it is essential that you pay close attentionto the principles and practices of planningand implementation as outlined in Work-books 1 and 2. Trade-offs have to be madeas to the rigour with which you collect andanalyse information to answer your evalu-ation questions, and the resources youhave available. You must strive to achievethe best possible information with the timeand resources available to you. You mustcarefully document the limitations of yourfindings and conclusions. With these prin-ciples in mind, you will be able to under-take practical and useful cost evaluationswithin your treatment service or system.

After completing your evaluation, you wantto ensure that your results are put to prac-tical use. One way is to report your resultsin written form (described in Workbook2, Step 4). It is equally important, how-ever, to explore what the results mean foryour programme. Do changes need to hap-pen? If so, what is the best way to accom-plish this?

Return to the expected user(s) of the evalu-ation with specific recommendations basedon your results. List your recommenda-tions, link them logically to your results, andsuggest a period for implementation ofchanges. The example below illustrates thistechnique.

Based on the finding that programme A,compared to programme B, results in 20%cost savings yet equivalent quality of lifeoutcomes, we recommend thatprogramme A is adopted on a larger-scalebasis.

Remember, economic evaluations are acritical step to better understanding the dayto day functioning of your PSU services.It is important to use the information thateconomic evaluations provide to redirecttreatment services. Through careful exami-nation of your results, you can develophelpful recommendations for yourprogramme. In this way, you can take im-portant steps to create a “healthy culturefor evaluation” within your organisation.


WHO/MSD/MSB 00.2i

References

Bradley, C.J., French, M.T. and Rachal,J.V. Financing and cost of standard andenhanced methadone treatment. Journalof Substance Abuse Treatment, 1994,11: 433-442.

Dolan, P., Gudex, C., Kind, P. and Will-iams, A. A social tariff for the EuroQol:results from a UK general populationsurvey. Centre for Health Economics Dis-cussion Paper 138, University ofYork:York, 1995.

Drummond, M.F., O’Brien, B., Stoddart,G.L. & Torrance, G.W. Methods for theEconomic Evaluation of Health CareProgrammes. 2nd edition, Oxford MedicalPublication: Oxford, 1997.

French, M.T., Mauskopf, J.A., Teague, J.L..& Roland, E.J. Estimating the dollar valuehealth outcomes from drug-abuse interven-tions. Medical Care, 1996, 34(9): 890-910.

Gerstein, D.R., Johnson, R.A., Harwood,H.J., Fountain, D., Suter, N. and Malloy, K.Evaluation recovery services: the Califor-nia drug and alcohol treatment assessment(CALDATA). California Department of Al-cohol and Drug Programs: Sacramento,1994.

Glazer, W.M. and Ereshefsky, L. Apharmaeconomic model of outpatient antip-sychotic therapy in “revolving door” schizo-phrenic patients. Journal of Clinical Psy-chiatry, 1996, 57(8): 337-345.

Johannesson, M., Jönsson, B., & Karlsson,G. Outcome measurement in economicevaluation. Health Economics, 1996, 5(4):279-296.

Zarkin, G.A., French, M.T., Anderson, D.W.& Bradley, C.J. A conceptual frameworkfor the economic evaluation of substanceabuse interventions. Evaluation and Pro-gram Planning, 1994,17(4): 409-414.


WHO/MSD/MSB 00.2i

The following case example presents a modi-fied cost-effectiveness economic evaluation.The evaluation compared a case managementapproach to a “usual care” approach forpeople with severe mental illness and PSUdependence. In this evaluation, case man-agement included intense individualised as-sistance and monitoring by a team of profes-sionals. Usual care was defined asparticipation in a community-based PSU sup-port group (Alcoholics/Narcotics Anony-mous). Costs of providing each programmewere calculated on a per client basis, andcompared against changes in societal costs

that were incurred by participants. Societalcosts included psychiatric, medical, legal, andfamily resources. Results indicated that bothtreatments resulted in cost savings. However,the case management approach also resultedin significantly improved psychiatric symp-toms and role functioning relative to usualcare. Evaluators did not provide a cost-ef-fectiveness ratio, thus departing from stan-dard cost-effectiveness evaluation techniquesas described in this workbook. As a result,their conclusions had to be based on a gen-eral overview of cost and outcome data ratherthan a single measure of effectiveness.

Comments aboutcase example


WHO/MSD/MSB 00.2i

Who was asking thequestion(s) and whydid they want theinformation?

Towards the late 1980’s, a California countymental health administrator was very muchconcerned about the rising costs of mentalhealth care and the prevalence of substancemisuse. The administrator was searching fora services program which might be able tocontain the rising costs of care, and, perhapsat the same time, improve the mental healthstatus and functioning of clients. In response,the county allocated funds for carrying outalternative models for treatment. At the sametime, evaluation funding from the federal gov-ernment was received to evaluate the costsand outcomes of this experiment.

This case study focuses on the treatment ofpeople with severe mental illness and alco-hol and drug abuse in a California county(Jerrell & Hu, 1996; Jerrell, Hu, & Ridgeley,1994; Jerrell & Ridgeley, 1995). A casemanagement program was developed for thisclient population since they are very high us-

ers of public health and other services. Thecase management program involved inten-sive individualised assistance and monitoringby a team of clinicians and paraprofession-als. The clients involved in the case manage-ment program were compared to those in-volved with Alcoholics Anonymous orNarcotics Anonymous (AA/NA), a form ofsupportive counselling to help people workthrough the 12-step recovery process. Staffmet with clients both individually and in groupsand provided an additional three to four hoursper week of services in addition to their men-tal health services. Staff actively engaged inteaching patients the 12-step recovery ap-proach, and linked them to existing AA/NAmeetings in the community. Thus, the 12-stepAA/NA approach served as the controlgroup or as a “usual care” group.

Qualified patients were randomly assignedinto the two intervention programs. In the fi-nal analysis, 39 patients were in the AA/NAprogram and 45 patients were in the casemanagement program. It was anticipated thatthe individualised assistance and monitoringwithin the case management program wouldproduce superior outcomes for this clientpopulation and be more cost-effective.

The author alone isresponsible for theviews expressed inthis case example.

Case example of aeconomic evaluation

Cost-effectiveness evaluation ofsubstance misuse interventionsByTeh-wei Hu, Ph.D.


WHO/MSD/MSB 00.2i

What resourceswere needed tocollect and interpretthe information?

To carry out this evaluation, the director ofthe evaluation department in the county re-cruited a health economist and a clinical psy-chologist to design the study. Two graduatestudents and two staff were hired to collectand process the data. Outcome data werecollected from personal interviews, and costdata were mainly collected from county ser-vices claim records and personal interviewswith clients and their families.

How were thedata collected?

Outcomes - This study examines the out-comes of treatment of dual diagnosis clients.It placed much emphasis on psychiatric andsubstance disorder symptoms, social func-tioning, and life satisfaction. Outcome infor-mation was collected from several surveyinstruments: the Social Adjustment Scale(SAS), (Schooler et al., 1979), the RoleFunctioning Scale (RFS), (Green et al.,1987), and the Satisfaction with Life Scale(SLS), (Stein and Test, 1980), augmentedinformation on client use of drugs and alco-hol, and their mental health and medical con-ditions.

Costs - Resources utilised by clients withsevere mental and substance disorders ineach of these intervention programs involvethe public and private mental health sector,general medical sector, judicial system, so-cial service agencies, and families. Data werecollected on all these sources of support.

Intensive mental health services that wereprovided to each client in the study (i.e., in-patient days, skilled nursing days, residential

treatment days, and emergency visits), wereseparated from the mental health supportiveintervention services (i.e., case managementhours, outpatient visits, medication visits, sup-portive housing days, and day service days).This was done to compare the cost differ-ences in providing supportive services througheach intervention. The cost impact of theseprograms on the use of other mental healthservices, such as acute and subacute, inten-sive services and non-mental-health serviceswas then compared.

Data on general medical services were alsocollected, because many of these clientsalso have co-occurring chronic medicalproblems, or have a propensity to seektreatment in medical emergency or outpa-tient health services. Medical treatment in-cluded inpatient, outpatient, and emer-gency visits, as well as nursing home care.All these clients were eligible for Medic-aid, so the costs of these services wereobtained from the billing system in the lo-cal public hospital or clinics, and fromMedicaid claims data for clients served inthe private sector.

Criminal justice and social services wereprovided to some clients in the study aswell. Criminal justice services includedpolice contacts, arrests, court appear-ances, attorney services, jail, probation,and conservator services. The major chal-lenge in estimating criminal justice costswas the complexity of the process of re-trieval and placing a cost value to each unitof those contacts. Criminal justice systemutilisation data were obtained through thecriminal justice data system. The unit costswere obtained from the county executive’soffice, which had previously undertaken aspecial cost accounting study to determinewhat the public’s direct costs were by typeof charge (misdemeanor or felony, drugrelated and non-drug related) at each stageof the criminal justice contact. Data on useof other social services, including mentalhealth conservatorship or guardianship,were collected from client interviews and


WHO/MSD/MSB 00.2i

clinical public guardian records. Informa-tion regarding average cost per unit ofthese services were obtained from the rel-evant departments.

Costs incurred by the family in providingcare to these clients were included in thecost estimation procedures. This included,for example, the actual family expendituresfor treatment, transportation, legal services,as well as the time family members spentwith the client in treatment and transpor-tation. These data were obtained from fam-ily/ care giver interviews. Market value ofthe transportation costs and wage rate ofservices were then used to estimate thecosts to the family. The issue of whethermaintenance costs, such as food, lodging,and clothing should be included in this typeof analysis is debatable. One approach isto treat all maintenance costs as treatmentcosts (Rice, Kelman, & Miller 1992). An-other approach is to treat only a portionof the total as treatment costs (McGuire etal., 1987). Maintenance costs incurredwhile living with family members or alonein a house are not usually considered treat-ment costs because these maintenance ex-penditures are part of daily living expenses,even of persons who are ill. On the otherhand, in the context of this study, the costsof employing a paid caretaker to assist aclient with basic daily living activities shouldbe considered as treatment costs. The lat-ter reasoning was adopted in these analy-ses. In this study, we performed a sepa-rate accounting of all these maintenancecosts to reflect the magnitude of those costsof daily living. These data were collectedin client and family/ care giver interviews.

Finally, to understand the relevant financialburdens among various sectors of society,transfer payments were also recorded in ourdata set. Transfer payments are from oneparty to another (i.e., parents to children,government welfare payments, etc.) that arenot accompanied by an exchange of servicesor goods of comparable value. These pay-ments are not treatment costs because they

reflect only a shifting of existing resources.However, they are an important indicatorreflecting the government/ taxpayer shareof the cost of illness. The amount of trans-fer payments also serves as a useful out-come indicator demonstrating treatmentproviders’ success in connecting clients toentitlement programs that are likely to en-hance overall income level and, therefore,quality of life. These data are collectedfrom client interviews as well as from pub-lic or private guardian records.

After all the public and private service el-ements related to the evaluation were iden-tified, we determined a standard unit ofmeasurement for each type of services andobtained the unit cost of each service. Thisunit cost was then multiplied by the num-ber of units of services and summed toobtain various types of subtotal and totalcosts.

How were thedata analysed?

Cost and outcome data were compiled forthe 6 months prior to each client’s entryinto the study, and then for each 6-monthperiod that they remained in the study. Costand outcome variable were found usingstatistical computing software. Each costand outcome variable was summarisedusing the mean, variance (standard devia-tion), minimum, and maximum value, theuse of which provides a basic understand-ing of these variables and also helps tocheck for any possible outliers or unrea-sonable values. A number of cross-tabu-lations were constructed with socio-demo-graphic variables (age, gender, ethnicity,etc.) to provide a description of the studypopulations between study groups. Toevaluate the possible differences be-tween programs, multiple regressionanalysis was used for cost and outcomedata analysis.


WHO/MSD/MSB 00.2i

What did they find out?

Psychiatric symptoms include depression,manic episodes, and schizophrenia whileusing drugs and alcohol. Compared to theAA/NA program, patients under case man-agement had an overall lower mean valueof schizophrenia symptoms and depres-sion. There was no difference in drug andalcohol use. These results were obtainedafter controlling for sociodemographic dif-ferences and baseline illness condition us-ing multiple regression analysis. Overall lifesatisfaction measures showed that casemanagement program patients improved interms of their living situation, global satis-faction with life situation, and mental healthcondition, as compared to AA/NA pro-gram patients. The overall role functioningmeasures show that case management pro-gram patients improved in independent liv-ing, but were rated lower in extended so-cial involvement in the community, ascompared to AA/ NA program patients.Again, all these findings are based on mul-tiple regression models, controlling forother sociodemographic factors.

Changes of psychosocial outcome mea-sures scores were measured and com-pared between the baseline and 12-monthperiod through regression analysis. It wasfound that adjustment of family interaction(SAS) was improved by 0.75 for casemanagement clients, as compared to AA/NA clients. Similarly, the score of GlobalSatisfaction of Life (SLS) for higher forcase management clients by 1.74. Further-more, mental health symptoms (Schizo-phrenia, -1.88; Depression, -2.19; Mania,-0.96) were all reduced among case man-agement clients as compared to AA/NAclients.

The findings from these effectiveness mea-surements indicate that the case manage-ment program provides some improvementin psychiatric symptoms, life satisfactionmeasures, and independent living, but nosignificant improvement in drug or alcoholsymptoms.

Tables 1 and 2 provide a summary of men-tal health costs and average societal costsfor the two interventions. The detailed cat-egories of these costs changes over a 12-month period are illustrated. It can be seenthat both programs significantly reducedmental health costs from the baseline pe-riod: AA/NA program reduced costs by50%, while the case management programreduced costs by 41.2%. Similarly, totalaverage societal costs were reduced46.8% for the AA/NA program, the casemanagement program reduced costs by39.7%.

When comparing effectiveness measure-ments, it seems that case management ismore cost effective than the AA/NA pro-gram. The AA/NA reduced costs by 10%,but had less improvement in patient out-comes. On the other hand, the case man-agement program achieved both cost re-duction (4%), and improved some of thepsychosocial conditions of participatingpatients.

Given the nature of multiple outcome mea-sures in numerous scales, it is very difficultto provide a meaningful cost-effectivenessratio. However, it is clear from this analy-sis that the case management program hasnot only reduced (or saved) the costs oftreatment, but also improved the outcomesof participating clients. In other words, itachieved both cost savings and improvedeffectiveness.


WHO/MSD/MSB 00.2i

Table 1: Changes in average mental health service costs per client for two dualdiagnosis treatment programs

AA/NA Case managementService (n=39) (n=45)

Baseline 12 months % Change Baseline 12 months % Change

Intensive mental health services

Inpatient 7,660 2,196 -71% 2,860 1,563 -45%

Skilled nursing 1,158 159 -86% 1,606 707 -55%

Residential 568 384 -32% 701 201 -71%

Emergency 405 184 -55% 426 157 -63%

Subtotal 9,791 2,923 -70% 5,593 2,628 -53%

Supportive mental health services

Medication 724 604 -17% 794 565 -29%

Outpatient 1,852 1,870 +1% 1,611 1,251 -22%

Case management 466 602 +24% 506 539 -7%

Housing 349 485 +39% 460 378 -18%

Day services 196 189 -4% 237 100 -58%

Partial hospitalization 0 0 0 94 0 -100%

Subtotal 3,587 3,750 +5% 3,702 ,833 23%

Total Costs $13,378 $6,673 -50% $9,295 $5,461 -41%

How were theresults used?

These two programs have achieved costsavings primarily because of the major re-duction in the use of intensive mental healthservices. Therefore, the dual diagnosistreatment programs studied succeeded intheir goal to reduce cost.

The findings of this evaluation were re-ported by the county director of evalu-

ation to the Director of the Bureau ofMental Health Services, the CountyMedical Director, and the County Su-pervisor. As a result, the County de-cided that case management programsshould be continued and clients shouldbe encouraged to utilise case manage-ment services so that total care costs arereduced and treatment outcomes areimproved. In fact, the County has alsoexpanded case management services tomental health and juvenile delinquentservices program.


WHO/MSD/MSB 00.2i

Table 2: Comparison of average societal cost per client for 12-month time periods fortwo dual diagnosis treatments

AA/12-step Program Case management(n=39) (n=45)

Baseline 12 months % Change Baseline 12 months % Change

Psychiatric

Psychiatric Intensive 9,791 2,923 -70% 5,593 2,628 -53%

Psychiatric Supportive 3,587 3,750 +5% 3,702 2,833 -23%

Subtotal 13,378 6,673 -50% 9,295 5,461 -41%

Medical

Medical Inpatient 134 371 +176% 193 341 +77%

Medical Emergency 377 117 -69% 235 121 -49%

Medical Outpatient 104 27 -74% 6 77 +1183%

Subtotal 615 515 -16% 434 539 +24%

Legal

Court, jail, etc. 1,151 995 -14% 1,657 977 -41%

Conservatorship 23 54 +134% 34 11 -68%

Subtotal 1,174 1,049 -11% 1,691 988 -42%

Family

Support 687 176 -74% 363 139 -62%

Travel 57 55 -4% 41 9 -78%

Subtotal 739 231 -69% 404 148 -63%

Grand total $15,906 $8,468 -47% $11,824 $7,136 -40%


WHO/MSD/MSB 00.2i

Derogatis, L.R., & Spenser, P.M. BriefSymptom Inventory: Administration andScoring Procedures Manual. Washington,D.C.: Clinical Psychometric Evaluation,1982.

Hu, T., & Jerrell, J. Cost-effectiveness ofalternative approaches in treating severelymentally ill in California. Schizophrenic Bul-letin,1991, 17: 461-468.

Jerrell, J., Hu, T., & Ridgeley, M. Cost-ef-fectiveness of substance disorder interven-tion for people with severe mental illness. TheJournal of Mental Health Administration,1994, Summer:83-297.

Jerrell, J., & Ridgeley, M. (1995). Compara-tive effectiveness of three approaches to serv-ing people with severe mental illness and sub-stance abuse disorders. Journal of Nervousand Mental Disorders, 183: 586-96.

References for case example

McGuire, T., Dickey, G., Shiveley, E., &Strumwasser, I. Differences in resource useand cost among facilities treating alcohol, drugabuse, and mental health disorders: Implica-tions for design of a prospective paymentsystem. American Journal of Psychiatry,1987, 144:616-20.

Rice, D., Kelman, S., & Miller L. The eco-nomic burden of mental illness. Hospital andCommunity Psychiatry, 1992, 43:1227-32.

Schooler, N., Weissman, M., & Hogarty, G.Social Adjustment Scalefor Schizophrenics.In: W. Hargreaves, C. Atkisson, & J.Sorenson (Eds.).Resource Material forCommunity Mental,1979.

Health Program Evaluators, USHEWPublication No (ADM) 799328 Washing-ton DC: U.S. Government Printing Office,pp. 203-303.

economic evaluations - emcdda home page · full economic evaluations require two or ... studies...

Documents