chapter 1 ensuring effective evaluation of tobacco control ... · ensuring effective evaluation of...

IInnttrroodduuccttiioonn

This volume is concerned withmethods for evaluating the evidencefor the effects of policy initiatives. Bypolicies we mean the enacteddecisions of governments and theirconsequences on the environment(legal, social and physical) in whichtobacco use takes place or ontobacco use directly; that is, specificinstances of the policy’s mani-festations (interventions). Thismeans evaluating the effects oflaws, regulations, taxes, admin-istrative decisions, programmes andefforts to publicise or disseminatediscrete interventions such assmoking cessation aids. It includesevaluation of policies that have theexplicit goal of tobacco control, aswell as policies that affect tobaccouse incidentally, although our focusis primarily on the former. TheWorking Group (WG) is primarilyinterested in evaluating inter-ventions that are designed to haveeffects at a population level,especially those enacted at anational level, but the principlesapply to many subnational- andeven local-level policies. While thefocus of the WG is on how to assesspolicy consequences of govern-ments, the evaluation framework wehave developed could equally apply

to the disseminated programmes ofnon-governmental agencies.

This chapter provides anintroduction to the importance ofhaving well-evaluated, population-level tobacco control interventionsand of having a framework forachieving them. It outlines criteriaused to evaluate constructs andmeasures, and how these relate tostrategies for most effectivelygathering information to evaluatethe effectiveness of interventions,the mechanisms by which theywork, and the conditions thatmoderate their effects.

Cigarette smoking is not only themost prevalent form of tobacco use,it is also among the most harmful,as it kills one in two long term usersprematurely. In the 20th century,cigarette smoking caused anestimated 100 million deathsworldwide. Most of these deathswere in developed countries of theworld where cigarette smoking firstbecame popular in the 1920s to1940s. This resulted in an epidemicof smoking-induced cancer, heartdisease, and chronic obstructivepulmonary disease (COPD) deaths.In 2000, tobacco use wasresponsible for approximately 4.83million deaths, evenly dividedbetween the industrialised and non-industrialised worlds (Ezzati &

Lopez, 2003). If current trendscontinue, it will cause some 10million deaths each year by 2030,with around 70% in low-resourcecountries (Peto & Lopez, 2001;Ezzati & Lopez, 2004). Thisprojected shift is due, in part, toincreasing population size andincreased smoking in low-resourcecountries, but it is also partly due togreater success in controllingsmoking in many higher-resourcecountries. In the 21st century, ifcurrent usage patterns persist,smoking will cause approximately1000 million deaths, a tenfoldincrease over the previous century(Gajalakshmi et al., 2000). Asubstantial fraction of theseexpected deaths could be avertedby efforts to discourage tobacco useand to assist those addicted totobacco to quit (IARC, 2007a).

Most countries have ratified theWorld Health Organization’s (WHO)Framework Convention for TobaccoControl (FCTC). It is the first pieceof international law emanating fromthe WHO. Its objective is:

“…to protect present and futuregenerations from the devastatinghealth, social, environmental andeconomic consequences of tobaccoconsumption and exposure totobacco smoke by providing a

1

Chapter 1Ensuring effective evaluation of tobacco control interventions

CHAPITRE1.janvier12:Layout 1 12/01/2009 13:29

IARC Handbooks of Cancer Prevention

2

framework for tobacco controlmeasures to be implemented bythe Parties at the national, regionaland international levels in order toreduce continually and substantiallythe prevalence of tobacco use andexposure to tobacco smoke.”(Article 3) (WHO, 2003).

To achieve this objective, theWHO FCTC calls for acomprehensive range ofmeasures, specifically:• Price and tax measures to

reduce demand (Article 6)• Protection from exposure to

tobacco smoke (Article 8)• Regulation of the contents of

tobacco products (Article 9)• Regulation of tobacco product

disclosures (Article 10)• Controls on packaging and

labelling of tobacco products(Article 11)

• Programmes of education,communication, training andpublic awareness (Article 12)

• Bans on tobacco advertising,promotion and sponsorship(Article 13)

• Programmes to promote andassist tobacco cessation andprevent and treat tobaccodependence (Article 14)

• Elimination of illicit trade intobacco products (Article 15)

• Measures to prevent sale ofand promotion of tobacco toyoung people (Article 16)

• Provision of support foralternative crops to tobacco(Article 17)

In addition, Part VII of theWHO FCTC, on “Scientific and

Technical Cooperation and Com-munication of Information” spellsout a framework for research,surveillance and technical coop-eration to facilitate the achieve-ment of the policy goals.

Article 20, “Research, surveil-lance and exchange of informa-tion”, calls for “The parties [to]undertake to develop and promotenational research and to coordi-nate research programmes at theregional and international levels inthe field of tobacco control.” Thearticle, among other things, callsfor the development and promo-tion of national research efforts,national systems of surveillance oftobacco consumption and relatedsocial, economic and health indi-cators; coordination of activities sothat data can be compared acrosscountries; exchange of publiclyavailable scientific, technical,socio-economic, commercial andlegal information, as well as infor-mation regarding practices of thetobacco industry; and that the fi-nancial and institutional resourcesbe put in place to allow this to hap-pen.

Article 22, “Cooperation in thescientific, technical, and legalfields and provision of relatedexpertise”, expands on Article 20with regard to such things asproviding developing countrieswith technical and materialsupport and training, andidentifying methods for tobaccocontrol, including comprehensivetreatment for nicotine addiction.

The WHO FCTC will likelyresult in the proliferation of policiesand associated programmes

designed to reduce tobacco use.These will include but not berestricted to those mandated orrecommended by the Convention.Ensuring the right mix of policiesrequires an understanding of thedeterminants of tobacco use andof how tobacco harms health.

Tobacco use is determined bymultiple factors, and attempts tocontrol the epidemic requirechanges in societies as well asindividuals (see Figure 1.1).Analysis of the factors thatinfluence tobacco use shouldencompass smokers, thosevulnerable to uptake, tobaccoproducts, those who produce andsell tobacco products, andgovernments who determine theparameters of use. The role ofcultural and economic diversityshould also be considered.Further, we need to understandhow both the determinants of useand actual use and/or exposuresare affected by interventions.

Policies and the disseminatedprogrammes that result from pol-icy decisions are of particular in-terest because of their potential toaffect large numbers of people, insome cases entire populations. Asa result, it is important to be ableto show that they achieve their ob-jectives and do so in a cost-effec-tive way, with any incidentaleffects ideally having net benefits.Evaluation allows the most effec-tive interventions to be maintained(and perhaps improved further)while less effective interventionsare either improved or aban-doned.


Ensuring effective evaluation of tobacco control interventions

3

TToobbaaccccoo aanndd hheeaalltthh

The amount of harm created bytobacco use in a given populationis a function of the toxicity of theproducts, the site(s) of exposure,the toxins taken in, the period overwhich exposures occur, and thedistribution of those exposures inthe population (IARC, 2004,2007b). The harms from tobaccouse are mainly from long-termuse, which is made more likely bythe addictive nature of theproduct. Calculation of thepotential harms that tobacco

products cause should considerthe composition of what isingested and how the products aredesigned to be used. Thus forcombusted tobacco products, thefocus needs to be on the smoke,rather than on the unburnedproduct, although the compositionof the unburned product isrelevant to the extent that itinfluences the composition and/ordensity of the smoke. Mode ofingestion is often ignored;however, some chemicals aremore toxic when absorbedthrough the lungs than through the

mouth lining or stomach becausethe lungs are more sensitive. Theevidence that exclusive cigar orpipe smokers have notably lesshealth risk than cigarette smokers(Doll, 2004) is probably becausethese smokers tend to only takethe smoke into their mouths.

Decades of research on thehealth effects of tobacco haveidentified numerous diseasescausally related to tobacco use,including several sites of cancer(including lung, oral cavity, esoph-agus, larynx, stomach and pan-creas), major vascular diseases

Figure 1.1 Major influences on tobacco use and its consequencesUsed with permission of Ron Borland

Tobaccomarketing

Cues

Active and passive exposures

Health outcomes

Socio-economic effects

Tobaccoproducts

Tobaccoindustrycontrol

HostBiology

TobaccoUse Control

PPeeooppllee:: Awareness, appraisals, experiences, habits, values,expectancies, choices,etc

Tobacco use

Environment: Physical, institutional, communication, policy, legal, scientific,cultural, social & inter-personal



4

(including ischemic heart disease,peripheral vascular disease andcerebrovascular disease), majorrespiratory diseases (includingchronic obstructive pulmonary dis-ease, tuberculosis, and pneumo-nia), reproductive effects andreduced bone health. Epidemio-logical methods have been ap-plied to estimate how much ofthese diseases in different popu-lations with different tobacco usehistories is due to tobacco (Peto etal., 1992).

While prolonged exposures areresponsible for most fatalconsequences of tobacco use,there is increasing evidence ofadverse short-term effects, seenmost clearly in the rapidlyreversible impacts of passivesmoke exposures on non-smokers (Raitakari et al., 1999;Wong et al., 1999; Wakefield etal., 2003a). There is no safe levelof exposure to tobacco smoke.Risks of cardiovascular problemsare largely reversible, and effectsseem to asymptote at lower dosesthan those related to cancers andchronic lung conditions (e.g.emphysema), where the dose-response curve is more linearacross typical exposure patterns(Law & Wald, 2003; Pechacek &Babb, 2004). The addictive natureof tobacco makes it likely thatpeople who begin to use it will notbe able to stop before the negativeeffects associated with long-termharm start to occur.

Nicotine is the main psycho-active ingredient of tobacco and thesource of its addictiveness, but isotherwise a minor contributor to theharm (Murray et al., 1996; Beno-

witz, 1999). Most of the harm is dueto other constituents in tobacco andtobacco smoke (IARC, 2004). Thusnicotine only indirectly contributes tomost of the harms, by leading toprolonged use of dirty deliverysystems, especially cigarettes.

The epidemiology is clear. Thehealth risks of smoking are fargreater than those associated withsmokeless tobacco use. Thehealth risk of each kind ofsmokeless tobacco varies signi-ficantly as a function of theirtoxicity. For smoked products, thelikely variability in toxicity does notseem to translate into cleardifferences in health risks. To date,cigarettes with levels of toxinsreduced by enough to be plausiblyless harmful are not used bysmokers, so are irrelevant totobacco control efforts.

Some harms, particularly minorharms and those related tocardiovascular disease, arereversible on quitting smoking.While quitting can improve health,cutting down on consumption doesnot seem to (Hecht et al., 2004;Tverdal & Bjartveit, 2006). Thismay be in part because, for someillnesses much of the harm occursat relatively small doses, and partlybecause smokers who reduce thenumber of cigarettes they smoke,often smoke the remainingcigarettes harder, ingesting moretoxins per cigarette, thus reducingor eliminating the potential benefitsof smoking less (National CancerInstitute, 2001). There has beensome success in reducing thetoxicity of smokeless tobaccoproducts. Changing from smokedto smokeless products (particularly

the toxin-reduced forms) canreduce harm, but does noteliminate it (Critchley & Unal,2003; Foulds et al., 2003; Roth etal., 2005; Henley et al., 2007).Reducing or eliminating smokedtobacco use is a higher priority forhealth than reducing smokelesstobacco use. Research is neededto determine whether smokelesstobacco might play a role in this orwhether nicotine replacementproducts and other cessation aidsare all that is needed.

PPaatttteerrnnss ooff ttoobbaaccccoo uussee

Tobacco is a plant containing thepsychoactive and addictive drugnicotine. It has a long history ofuse and has been used in a widevariety of forms. The two mainforms of tobacco use are by smok-ing and by chewing or parkingwads of tobacco in the mouth andallowing the active ingredients tobe absorbed (smokeless use). Inthe 20th century, the use of ciga-rettes came to dominate both thesmoked and overall markets innearly all countries. It is also theproduct that has been the focus ofmost of the research. In mostcountries factory-made cigarettesdominate the market; however“roll your own” cigarettes have en-joyed a resurgence in some coun-tries. In other countries, mostnotably India, people consume adiverse range of tobacco prod-ucts, both smoked and smoke-less. Among smoked products,the “bidi” (tobacco hand-rolled in aleaf) is the predominant form usedin the Indian sub-continent. Use ofwater pipes is common, particu-



5

larly in the Middle East. Cigars oc-cupy a position as a ‘luxury’ to-bacco product, but use isgenerally low. All forms ofsmoked tobacco are extremelydangerous to health, and therehas been no major progress to-wards creating less toxic versionsof these products that are suffi-ciently acceptable to consumers tobe successfully marketed.Smokeless tobacco is not used inmany parts of the world, but use issignificant in other parts, with theproducts used ranging widely inplaces like India (e.g. gutka, usewith betel quid, nicotine tooth-paste), but is limited to one mainform in others (e.g. snuff (pow-dered tobacco) either in loose orprepackaged, small tea-bag-likeportions). Use of smokeless to-bacco is increasing in some places(e.g. Sweden) (Foulds et al., 2003).Non-cigarette tobacco use isunder-researched in comparison tocigarette use.

The proportion of the populationwho use tobacco varies greatlyfrom around 20% to around 60%(Shafey et al., 2003). In many coun-tries, few women smoke, often ac-companied by high smoking ratesin males (e.g. in Asia). By contrast,in most developed countries femalesmoking rates are typically only afew percentage points below that ofmales. There has been some pre-dictability in these patterns of use,leading to Lopez, Collishaw &Piha’s (1994) four-stage model ofthe tobacco epidemic, with devel-oped countries first to experience it.In this model, female smoking ini-tially lags behind male smoking,with female rates eventually rising.

The experience of countries likeSingapore and Thailand, whichhave so far successfully preventedfemale uptake, suggest that theLopez et al. model does not de-scribe a necessary progression, butthat the epidemic may be able to belargely averted in some sub-popu-lations, most notably women, wheneffective tobacco control policiesare implemented.

Over the last 20–30 years,smoking prevalence has fallenmarkedly in some countries. Thisis well documented for some in-dustrialised countries (Gilmore,2000; Giovino, 2002; White et al.,2003). One country, Bhutan, hasbanned the sale of tobacco prod-ucts to its citizens. However, insome other countries, rates of to-bacco use may have increased.The great diversity both betweencountries and within countriesover time creates huge challengesand opportunities for scientific un-derstanding. One challenge, forexample, concerns preventingwomen from smoking in societieswhere few currently smoke. Thischallenge needs to be taken up inways that are not contrary to thegreater emancipation of women inthose societies. In developedcountries, e.g. in North Americaand Western Europe, the tobaccoindustry skilfully used femaleemancipation as a strategy forlinking smoking to images of themodern woman. The slogan“You’ve come a long way baby”from the notoriously successfulVirginia Slims advertising cam-paign typifies this strategy (USDepartment of Health and HumanServices, 2001).

The most comprehensivechange in tobacco control hasbeen in attitudes and rules aboutsmoking in enclosed public placesand workplaces. As late as 20years ago, smoking waseffectively ubiquitous in mostcountries, with smoking allowedvirtually everywhere (exceptwhere there was a danger of firesor damage to equipment). In somecountries, this environment hastransformed; several countries(starting with Ireland and Norway)now prohibit smoking in all publicplaces and workplaces, and othercountries are following rapidly.

The social acceptance ofsmoking is declining in most placeswhere it has been studied. Thisdecline seems to be related to thelength of time the society has takento regard the problem as serious,and to progress in the imple-mentation of smoke-free places. InThailand, for example, equivalentlevels of smokers see their habit asnon-normative (i.e., that societydisapproves) as in Westerncountries such as Australia,Canada, the UK and the USA, allof which have decades of strongaction. By contrast, even thoughpersonal disapproval of smoking ishigh in neighbouring Malaysia,which has only recently taken up theissue systematically, smokers arefar less likely to perceive societaldisapproval (ITC South East Asiaproject, unpublished data).

However, it is not just trends intobacco use and tobacco-relatedknowledge that are likely to affectefforts to control tobacco use.Broader societal issues may alsoplay a key role. The rapid



6

emergence of China and othercountries as economic power-houses is likely to affect tobaccouse, at least in those countries, asmore and more people havemoney to spend on consumerproducts like tobacco that aremarketed to appeal to “modern”sensibilities. Worldwide concernsabout the environment, includingthe issue of global warming, andthe rise of religious funda-mentalism in some countries arealso likely to have effects, but it is

beyond the scope of this volume tospeculate as to what these effectsmight be. However, unless effortsare made to understand howtobacco control fits into broadersocial changes that are sweepingthe world, important determinantsof use may be missed, with theresultant reduction in the capacityto identify and implement policiesand programmes that work.

In thinking about the potentialhealth benefits of interventions, itis important to consider both their

potency and their timing (seeFigure 1.2). While the under-standing of their potency is focal tothis volume, it needs to beremembered that the sooneraction is taken, the more lives willbe saved. Every year of delayadds millions to the eventualburden of lives lost. Enough isknown to act in a comprehensivemanner now. The evaluation effortis primarily about helping us refinethose interventions, to ensure theyare delivered in ways thatmaximise their effects, and onlysecondarily, to the development ofnew more effective interventions.

WWhheerree ddooeess tthhiiss vvoolluummee ffiittwwiitthhiinn TToobbaaccccoo CCoonnttrrooll??

This Handbook is not intended tobe a one-stop resource for alltobacco control evaluation needs.It is designed to present aframework for evaluation directedat policy effects and to providestrategies and measures that arespecific to tobacco control, ratherthan try to replicate material that isgeneral to all forms of evaluation.

In analysing the potentialcontribution of research to policyevaluation, it is useful to outlinethe various roles it can play.Applied science proceeds througha series of iterative stages once aproblem has been identified (inthis case tobacco as a cause ofhealth harm): elaboration of atheory or theories as to the causeof the problem and of possiblesolutions, observation and des-cription of the problem informedby the theory, understandingcausal mechanisms, intervention

Figure 1.2 Projected impact of population-level tobacco controlinterventions on estimated cumulative tobacco deathsEstimated cumulative tobacco deaths 1950-2050 showing the effects ofdifferent intervention strategies. In red baseline, in blue if proportion of youngadults taking up smoking halves by 2020 and in green, if adult consumptionhalves by 2020Adapted from Jha & Chaloupka (1999), The World Bank

IImmppaacctt ooff ppoolliicciieess ddeeppeennddss oonn ffaaccttoorrss iinncclluuddiinngg::-- IInntteerrvveennttiioonn ddaattee-- EEffffeecctt ssiizzee



7

development, intervention deploy-ment and evaluation, andre-evaluation of the problem. Fromthis, there might be the need fornew or revised solutions, whichmay require refinement of thetheory or development of a newone. Research can play a numberof important roles in the process ofdeveloping and disseminating themost effective policy interventions.It can be used to:

1. help in the development of newinterventions;

2. help make the case for anintervention being adopted;

3. fine-tune an intervention beforeimplementation to meet localneeds (formative evaluation);

4. document the quality ofimplementation (process evalu-ation);

5. assess the effectiveness ofcomponent parts, or of theintervention under ideal cir-cumstances;

6. evaluate the effects of theintervention as implemented,both intended and incidental;

7. determine the cost-effective-ness of the intervention; and

8. assess the cumulative effectsof changes in outcomes onhealth.Of these, only number 6 is of

focal interest here. All of the oth-ers are important, but to have cov-ered them all would have madethe volume too broad and toolong. We also do not considerevaluation of the efficacy of dis-crete interventions that can readilybe tested in randomised trials; e.g.smoking cessation aids. TheCochrane Collaboration (www.

cochrane.org, for reviews) pro-vides regularly updated reviews ofevidence in these areas. How-ever, we are concerned with theevaluation of effects of these in-terventions when applied to popu-lations.

The focus of this volume is theevaluation of tobacco control poli-cies in the short to medium term.We concluded that for policies di-rected at tobacco use, tobaccouse was the outcome of interest,rather than on the subsequenthealth effects. Clearly, as wemove forward, we will want toevaluate the summative effects ofall the efforts to reduce tobaccouse, and the consequential healthoutcomes. For a few jurisdictionsthat have had active tobacco con-trol programmes for decades, thisprocess is already underway(Thun & Jemal, 2006). However,the reality is that for most coun-tries, we will never know exactlyhow many tobacco-caused deathsare being averted, because thereis insufficient data on how manysuch deaths are currently occur-ring. The global estimates referredto earlier are a result of careful ex-trapolation from those countrieswhere good data is available andfrom studies that have been ableto estimate the fraction of deathsfrom various causes that are dueto tobacco. The methods for doingthis are beyond our remit, as areways to model the potential im-pacts of interventions on smokingprevalence or on the burden ofdisease (e.g. Levy et al., 2006).

The typical evaluation researchstudy can be thought of as havingfive components:

1. A research design2. The choice of constructs and

measures to assess them(predictors and outcomes)

3. A sampling strategy4. Study implementation5. Data analysis

Of these, we only focus on thefirst two, although some attentionis given to issues of sampling,particularly of the value of havingrepresentative samples as a corepart of the research design. We donot consider data analysis as thetools here are largely generic andare covered in the main computeranalysis packages, including theemerging techniques of GEEmodels (Generalized EstimatingEquations) (Hanley et al., 2003).

This Handbook was not writtenwith the needs of those conductingevaluations at a community level inmind. However, much within it islikely to be relevant, at least at aconceptual level. The cumulativeapproach adopted means that forevaluations of interventions thathave been shown to be effective incomparable situations, the needfor intense evaluation will be less,as the evaluation can rely onindicators validated in previouswork. However, for novel inter-ventions, the more powerfulmethods outlined here should stillbe used wherever the resourcesallow. The US Centers for DiseaseControl (CDC) has published auseful guide to the evaluation ofmore local programmes (Mac-Donald et al., 2001). A majordifference between that guide andthe present volume is the capacityto use national surveys and datacollections in ways that are not



8

usually possible for local initiatives.That said, to evaluate localinitiatives country-level data canbe used as a control, withcomplementary data collectedfrom the community to assess theintervention effects.

PPoolliiccyy aarreeaass nnootteemmpphhaassiisseedd iinn tthhiiss vvoolluummee

There are a number of tobaccocontrol policy domains that areeither not included, or notemphasised. This is not becausethe WG believes that they are notimportant, but because it sought tokeep the size of the volumemanageable. Policy domains notfocussed on include some that aredesigned to affect tobacco usedirectly, such as sales to minors,restrictions on sales outlets, andschool-based prevention. Othersare directed more at the tobaccoindustry, or parts of it, and includeprevention of illicit trade, industrysubsidisation, controls on accessof for-profit companies into themarket (and the role ofgovernment monopolies), andagricultural policies that affect leafproduction.

The most significant area wehave not focussed attention on inthe volume is the lack of detailedattention to population-level pre-vention policies. There is a largebody of evidence on the effective-ness of school-based educationprogrammes (Thomas & Perera,2006). The current evidence showsthat, taken in isolation of other soci-etal efforts, the impact of school-based programmes is generally

weak, and there exists the poten-tial for poorly thought-through pro-grammes to actually becounterproductive. Most of the re-search on the effects of preventionprogrammes in schools is from in-dustrialised countries. School pro-grammes are plausibly of moreimportance in non-industrialisedcountries, where school is a con-duit for new knowledge into thecommunity in a way it no longer isin industrialised countries. The dif-ficulty of developing successfulprevention education comes atleast in part from the problem thatraising the issue engenders inter-est and thus curiosity about theproducts. Doing this in a way thatovercomes the potential threat ofcuriosity leading to increased ex-perimentation, and that has a netnegative effect on use, has provendifficult. This may explain the in-terest of some tobacco companiesin promoting such strategies. Tothe extent that educational pro-grammes are translated into themass media, strategies for evalu-ating them are covered in Section5.6 on Measuring the Impact ofAnti-Tobacco Public Communica-tion Campaigns.

Another prevention strategy wedo not address the evaluation of ispolicies to prohibit sales of tobaccoproducts to minors, and to enforcethese laws by using young peopleattempting purchases. Suchprogrammes can result in adecline in the proportion of suchattempts that result in sales, butthe evidence that this actuallyreduces youth smoking is notstrong (Stead & Lancaster, 2000).

In the broad area of tobaccoindustry control, there is someconsideration of illicit trade in thesection on sources of productionand trade (Section 4.2) and in thesection on tax policies (Section5.1). Neglected areas includerestrictions on the number or typeof outlets in which products aresold. There are few examples ofattempts to restrict the number ortype of outlet selling tobacco.However, it seems inevitable thatin the future some jurisdictions willtry to restrict access to allsmokers, not just youth.

We also do not address theevaluation of policies that restrictfor-profit companies from opera-ting in the market. Some countrieshave actual or virtual statemonopolies on the sale orproduction of tobacco products.Several countries have beenforced to abandon these mono-polies by the World TradeOrganisation. It has been arguedthat non-profit control of theindustry should make tobaccocontrol efforts easier (Borland,2003), but there is little workevaluating either the move to freemarkets or the potential ofrestricting the markets. In boththese areas, research is neededto evaluate possible options andto estimate likely effects.

AA ccrriittiiqquuee ooff ccuurrrreenntt aapppprrooaacchheess ttoo eevvaalluuaattiioonn

To achieve maximally effectivetobacco control requires thedevelopment and ongoingrefinement of a viable set of



9

methods for integrating researchand evaluation in the imple-mentation of tobacco controlinterventions. The populationhealth challenge is to use scientificmethods to ensure that systemsare set up to understand theeffects of the policy initiatives insuch a way as to allow theirevolution into the most effectiveways of controlling the epidemic oftobacco use and related harms.Evaluation researchers in tobaccocontrol, like professionals in otherareas of population health, havebeen concerned for some timeabout limitations in the evaluationframework used.

The current dominant model ofintervention evaluation for im-proving population health involvesextrapolation from the use ofrandomised controlled trials(RCTs) of clinical (most typically,pharmaceutical) therapies. It isbased on the desire to identify theactive therapeutic agent or agentswithin any intervention. Thismodel is important and extremelysuccessful for testing the efficacyand often effectiveness of discreteinterventions offered at theindividual (and even small group)levels, particularly where doubleblinding is possible. This is whereneither researcher nor participantknow who is getting the thera-peutic agent under evaluation andwho is getting either a placebo orthe existing best-practice inter-vention. RCTs produce consi-derable certainty about causes.However, reliance on RCTs is notalways possible or appropriate forthe evaluation of policy impact in

the population for a number ofrelated reasons. First, imple-mented policies cannot berandomised and analoguestudies, where randomisation canoccur, may lack critical elementsof policy interventions (e.g.authority of law, or it being appliedto all in the community). Second,over-reliance on RCTs, whichfocus on the detection ofintervention effects, can lead to aneglect of theory, which is criticalfor generalising from results torelated areas, and forunderstanding the mechanismsby which interventions work.Third, RCTs are not able toanswer questions about therelative effectiveness of inter-ventions across differentpopulations. Fourthly, when RCTsare compromised, in terms ofdeviation from the double-blindedideal, they are less powerful, andmay be less strong thanalternative methods with differentvalidity limitations. Finally, focus-sing on RCTs to provide answersto questions can result in aneglect of other evaluationtechniques, which although not asinferentially strong as RCTs, mayhave complementary strengths. Itis important to understand theconditions under which RCTs arelimited and what the implicationsare for inference.

LLiimmiittaattiioonnss ooff RRCCTTss

Determining whether a discreteintervention works involvesanswering three questions, whichsometimes can only be answered

separately: the questions of effi-cacy, effectiveness, and dissemi-nation (Flay et al., 2005). First isthe efficacy question: Can thisintervention work? That is, whenimplemented in a controlled andoptimal way, does it work? Herethe double-blinded randomisedcontrolled trial (RCT) is the goldstandard, where possible. Secondis the question of effectiveness:does it actually work whenimplemented under real-worldconditions, and with what degreeof variation? Third is the questionof dissemination: Is the inter-vention used by enough of thepopulation who would benefit fromit to have an impact? An effectiveintervention that few are preparedto offer or few are prepared to useis of little benefit. One must alsoconsider the extent to which theintervention is similarly attractivefor all with the problem. When onlya subset of the populationbenefits, any barriers to selectiveadoption or influence should beexamined. As we move fromaddressing questions of efficacy,through effectiveness, to dissemi-nation issues, it becomesincreasingly difficult to fit theconditions for RCTs, even forclinical interventions.

RCTs involve a number of(usually implicit) assumptions.First, RCTs assume that themeasurement required for theevaluation does not affect theintegrity of the intervention.Second, it is presumed that theinterventions can be evaluated inisolation of environmental factors,including the society’s response to



10

the intervention and to othercultural trends; i.e., that theeffectiveness of the interventioncan be determined prior to itswidespread implementation. Third,it is assumed that any impact ofpersonal choice over whether tohave the intervention can beseparated from the core thera-peutic effect. Fourth, it is assumedthat the intervention is uniformlyeffective for all who are eligible tobe given it. None of theseassumptions are tenable for policyinterventions and disseminatedprogrammes.

The assumption that a givendose of an intervention isassumed to have an equivalenteffect on all who have thecondition it is intended to treat isproblematic even with manypharmaceuticals. The solution tothis problem has been to treateach identified population as noveland to require separate RCTs.This might work for major distinctdifferences, but when there aremany possible populations toconsider, the strategy becomescumbersome and costly. Moreefficient strategies are required.

RCTs are similarly a cum-bersome method for evaluatinginterventions that vary continu-ously, as they involve creatingdiscrete categories for randomi-sation. This means there is, forexample, poor quality informationon optimal dosage, both amountper dose and duration of use. Thismakes RCTs a particularlycumbersome method for evalu-ating interventions where the doseof an intervention can varyconsiderably.

Finally, there is no capacity toconsider closely related — indeed,functionally equivalent — inter-ventions as a class, and developdifferent criteria for evaluating newversions of essentially the sameintervention. For example, differentexecutions of a cognitive-beha-vioural cessation treatment or eventhe various forms of NicotineReplacement Therapy (NRT) gettreated as independent interven-tions. In the case of NRT, allvariants have had to go through thesame process of testing throughindependent randomised trialsbefore they were able to bemarketed.

Population interventions tendto be different in observable wayswherever they are implemented.Information-based interventionsare dependent on language, andthe language used must vary byculture, not just linguistic group.Language must be kept up-to-date to make it contemporary, andthus attract interest (and some-times increase) comprehension.People-based interventions in-variably differ. Policy-relatedinterventions encompass thosemajor aspects of the system thatallow them to operate, not just thecore requirements. It is notreasonable to assume thatpopulation-based interventionshave their effects independent ofanything the person does orthinks, unlike most pharma-ceutical interventions. Likevirtually all psychological andsocial interventions, as well assome pharmaceutical and otherones, the effectiveness of policyinterventions is critically depen-

dent on how the individualresponds to them. For clinicalinterventions, the frame is quitedifferent. Their questions areframed: If the appropriate systemis put in place to ensure theperson with the illness uses theintervention properly (or is given itproperly), then can wedemonstrate a benefit? Thequestion the WG ask is quitedifferent and much broader: Cana system be put in place that willmake the intervention work, andhow can that system be optimisedunder different conditions?

Where limitations exist on theinternal validity of RCTs formaking the inferences of interest,the strategy of using meta-analyses of similar studies to drawinferences is similarly problematic.Alternative means are required tocontrol for these threats toinference. It is only in the contextof being able to assumegenerality, having few enoughinterventions to assume each isan independent case, and havingthe capacity to test interventions inisolation of their context, that themodel of RCTs as the keystone ofevaluation is possible.

The allure of having a simplemodel based on RCTs to allowdefinitive inferences about theeffects of interventions treated inisolation seems to have distractedus from considering the potentialutility of other approaches. Inparticular, the RCT-focussedframework tends to neglect therole of theory and of the potentialcontribution of combined studieswith different kinds of limitations.



11

The contribution of theory isundervalued in tobacco controland in public health moregenerally. We agree with thenoted psychologist Kurt Lewin:“There is nothing so practical as agood theory.” Some in the socialsciences take theory to refer to theexisting, sometimes demonstrablylimited social science models, andtake the theories from other areas(typically from the biologicalsciences) to be accepted fact,rather than theoretical models; e.g.of how a chemical will affectbehaviour. Theory is thought of inan encompassing sense of theaccumulation of our under-standing of how things work, notmerely the original ideas. Theoryprovides the mechanism tosystematically use existing know-ledge to understand likely futureeffects. The aim should be todevelop consistent sets of ideas(theories) that describe and predictactual outcomes. A hunch or apast empirical finding is anunarticulated theory of what willhappen in the future. Unlessarticulated, these implicit theoriescannot be subject to properscrutiny. If they turn out to predictoutcomes, there is no capacity towork out why without firstarticulating them.

Theories specify mechanismsor mediating pathways of effects,allowing these pathways to betested. They also can specify con-ditions under which interventionswill work (i.e. moderate interven-tion impact). One can test whetherthese factors affect outcomes, andthus be better placed to developthe suite of interventions needed to

provide maximal help to all, or toproduce the desired structural orcultural changes. No single theorycan encompass the complexity ofcontrolling tobacco use; however,more can be done to consider howtheories that deal with different as-pects of the problem interrelate, in-cluding different timescales forchange (e.g. behaviour changeversus change in cultural normsand practices). The set of theoriesused should be compatible witheach other, even if the nature ofthe interrelationships is not fully ar-ticulated.

The most important implicationof considering theory is that itallows explicit linkage of tobaccocontrol to relevant existing know-ledge. A focus on evaluatinginterventions in isolation tends todistract from what is known,specifically:

• Information campaigns canincrease knowledge abouttobacco.

• Knowledge can change beliefsand attitudes.

• Beliefs and attitudes can affecttobacco use.

• Advertising can change beha-viour independent of consciousawareness of the influence.

• There are programmes andaids that can help people quitusing tobacco.

• There are ways that the toxicityof products can be reduced.

• Price rises affect levels ofconsumption of tobacco pro-ducts.

• Poorly designed and/or exe-cuted communications canhave boomerang effects.

This knowledge is part of afoundation that is sometimesforgotten. The question we arereally asking is: Under whatconditions can the desired effectsbe optimised? This includesconcern about the form of theintervention, the ways it isdelivered, and various charac-teristics of the populations towhom it is provided.

A new evaluation framework,one that is less reliant on the RCT,is required. It should have asystems perspective; use the bestpossible methods, including RCTswhere appropriate; allow a morecentral role for theory, to allowmore efficient consideration ofpossible variation in effects acrosspopulations; and provide a moreefficient means of understandingeffects of dosing and otheraspects of implementation.

One approach to evaluationthat is popular among publichealth practitioners, but that hasless credibility with researchers, isthat of programme evaluation (e.g.Patton, 1997). These models havegrown in areas where there are nosimple relationships betweenprogrammes and sought policyoutcomes, yet there is a need todemonstrate progress. Thus thefocus of these models of programevaluation is often on determiningintermediate effects when it isdifficult to demonstrate effects onthe main outcome goals. Webelieve that there is value inextending these models toconsideration of outcomes as well.The essence of these approachesis to test the theory behind theprogramme, sometimes also



12

called the “programme logic”, toassess whether the variousaspects of a programme can beshown to contribute to theachievement of its goals (Mac-Donald et al., 2001). The WG hasadopted the idea of using logicmodels as a core element of theframework we have developed.We found that doing so increasedconceptual clarity and provided auseful organising frame forthinking about the policies and amore coherent way to organise thechapters and sections.

FFrraammeewwoorrkk ffoorr ttoobbaaccccooccoonnttrrooll eevvaalluuaattiioonn

The role of evaluation is todetermine the effects of inter-ventions, determine under whatcircumstances these effectsoccur, and help identify ways tomake the interventions moreeffective. To do this involvesdetermining how the interventionswork, and diagnosing any prob-lems that either prevent them fromworking as desired or diminishtheir impact, in particular anydifferences of effects within thetarget population (equity issues).In doing this one should considerthe totality of effects, bothintended and incidental. To do

effective evaluation we need toconsider what effects might occur(theory), and design studies thatallow detection of effects in thevariables of interest (description)and making of valid causalinferences about the contributionof the intervention to the observedchanges in outcomes.

TThheeoorryy

Evaluation must begin with atheoretical evaluation of how anintervention might work. Oftenthere will be one clear theoreticalmechanism, generally provided aspart of the justification of havingthe policy, but sometimesalternative modes of effect mightbe postulated. This is particularlythe case when the head of power(constitutional source of capacityto legislate/regulate) under whichpolicies are enacted is limited.Thus policies to protect workersfrom exposure to passive smokingcannot explicitly consider thepossible benefits of smoke-freeplaces for reducing cigaretteconsumption or for enhancingquitting. Good evaluation requiresconsideration of all potentiallyimportant outcomes, not just thoseused to justify or provide a legalbasis for the policy.

Evaluation is enhanced byshowing the mechanisms of theeffects, not just restricting itself todetermination of effect size. Thisis critical in population researchbecause most of the outcomes weare interested in are potentiallydetermined by multiple factors;thus it helps demonstrate acontribution from the focalinterventions as distinct from otherinterventions happening at thesame time. Thus, the theoryneeds to spell out the mediationalmodel of how an interventionmight work. Mediational modelsallow us to test each step along aproposed causal chain fromintervention exposure to beha-viour (see Figure 1.3). If somerelationships are not as predicted,the intervention may not beworking, at least in the way it wasintended to work. In cases wherethe intervention is known to bepotent, evaluation of mediatorsmay only need to proceed as faras assessing uptake/exposure.However, where the potency isunproven, testing the inter-vention’s impact through to thedesired outcomes (e.g. smokingcessation) becomes necessary. Inan area like tobacco control wherethe main outcomes of interest(e.g. smoking cessation, pre-

Policy as implemented

Policy- specificmediators

Generalmediators

Policyoutcomes

Figure 1.3 A generalised model of mediation



13

vention of uptake) are determinedby multiple factors, mediationalmodels can also help establish therelative contribution of specificinterventions. Testing mediationalmodels can also enhance under-standing of basic mechanismsand facilitate the development ofnew and improved interventions.

Other theoretically importantfactors are those that maymoderate the relationshipbetween the intervention andoutcomes. That is, what conditionsaffect the efficacy of theintervention, or how does itseffectiveness vary by identifiablesub-groups. Where one finds ortheorizes moderator effects, it isimportant to understand wherethey occur along the proposedmediational pathways, or indeedwhether different mediationalpathways exist for different groupsor situations (see Figure 1.4). Forexample, if an intervention is notseen to be relevant to or targetedat a group, this group may notrespond to it. Here, making theintervention relevant might be allthat is needed to remove themoderating effect. A goodexample of this is advertisements

whose spokespeople are old,which are typically not seen asrelevant to young people (theconverse is less likely to be true).Something as simple as choice ofactor can create moderatoreffects, which under otherconditions would not be present(or be so small as to be ignored).

Incidental effects must also beconsidered. Sometimes it can beuseful to separate these out fromthe intended effects (see Figure1.5). Incidental effects can occurfor a range of reasons; some maybe theoretically expected, whileothers may not. Some can occuras a result of counter-actions ofsections of the tobacco industry toreduce the threats of policies totheir profitability. These effectscan be incorporated within themore general model (Figure 1.4)as all such effects can be eitherdue to reactions to the policy, or toindependent other factors (andthus should be treated asmoderators).

DDeessccrriippttiioonn

The relevant theory tells us whichconstructs to measure. Evaluation

requires a good description of theproblem and its context, and ofhow these are changing. Thisinvolves finding appropriatemeasures of the constructs ofinterest and of collecting datausing the appropriate measures.The goal here is to providepopulation estimates of whatpeople do and think, focusing onkey outcomes. It involvescollecting data in four principaldomains: 1) who uses tobacco,what they use, how much, andwhere and when they use it, aswell as any relevant knowledge,beliefs and attitudes (includingthose of ex-smokers and non-smokers); 2) tobacco industrybehaviour, including charac-teristics of their products; 3)tobacco control activities to whichpeople are exposed; and 4)aspects of the broader environ-ment that might affect tobacco useor tobacco harm outcomes(cultural norms, controls onactivities like alcohol consumptionthat are linked to tobacco use).High-quality data collections, suchas regular cross-sectional sur-veys, are essential to describingthe nature of the problem and the

Policy asimplemented


Generalmediators

Policyoutcomes

Moderators

Figure 1.4 A generalised model of mediation, making allowance for moderator effects



14

quantification of trends in tobaccouse and in key determinants ofuse. In tobacco control, becausethe tobacco industry or sections ofit might be motivated to moderatethe effects of policies, it isimportant to conduct surveillanceof possible counteractions topolicies. More generally, possibleincidental effects of policiesshould always be considered andmeasured where appropriate.

There are five broad types ofoutcomes that relate to indivi-duals: improvements in knowl-edge, changes to attitudes andrelated normative beliefs, changesto behaviour patterns, changes inexposures, and health outcomes(particularly acute ones that canbe detected soon after a policy isimplemented). Interventions typi-cally change the environmentalconditions that affect and thussustain these outcomes. Mecha-nisms for behaviour change can

be through rules and restrictions,making available alternatives orsubstitutes, and/or providing rele-vant resources and/or skills. Themediational pathways vary bothfor outcomes and policies. For ex-ample, mediational pathways toknowledge acquisition are shorterthan ones to smoking cessation.

IInnffeerreennccee

The core of good evaluation isdesigning studies to detectchanges in outcomes that mightbe attributable to a specificintervention, and putting in placemeasures to rule out alternativeexplanations. These alternativeexplanations are of three types:those related to systematic errorsof measurement (bias), thoserelated to alternative mechanismsof effect (confounding), andchance effects. Bias occurs wherethe measures used to assess the

constructs of interest actuallymeasure something different(usually a closely related con-struct) or are contaminated bysome systematic error (e.g. socialdesirability can affect responsesabout beliefs and intentions).Confounding occurs when theassociation with the outcome ofinterest appears stronger orweaker than it truly is as a resultof an uncontrolled associationbetween the intervention andother mechanisms of effect (e.g. adifferent policy intervention). Thecontribution of chance is afunction of naturally occurringvariability in outcomes of interest,and its impact is controlled for byensuring adequate sample sizes.

The quality of evidence fromany single study is a joint functionof the study design and of thequality of the measures used: thatis, their reliability and validity.Where optimal research designs

Figure 1.5 A generalised model of mediation, making allowance for both moderator and unintended orincidental effects

Policy asimplemented

Incidental effects


Generalmediators

Targeted policyoutcomes

Moderators



15

are not available, one must focuson the relative strengths ofdifferent designs. It is not enoughto conduct meta-analyses of theindividually strongest studies. Adiversity of research designs (andassociated measures) withcomplementary strengths, shouldbe combined, and that informationcombined in ways that increasethe validity of inferences. Demon-stration of similar effects withdifferent methods and/or mea-sures increases confidence in thereality of effects and of theplausible causal mechanisms.

EEvvaalluuaattiioonn aass aa ddyynnaammiiccpprroocceessss

The evaluation of policy inter-ventions occurs after they areinstituted, as they first must beimplemented somewhere before itis known how they actually work.Because the authority ofgovernment policy or law mayaffect compliance, it is notpossible to confidently generalisefrom the results of analoguestudies conducted prior to imple-mentation. This means onecannot in principle be certain ofthe effectiveness of interventionsbefore they are implemented;hence, lack of evidence needs tobe used with caution as a reasonfor delaying needed policychange. Scientific methods canbe used to help us minimise ourrisk of error, but they can nevereliminate it completely. Scienceshould not inhibit action whenthere is a need for action, butrather act to maximise thechances of success and minimise

the risks of wasting resources.This involves a model in whichscience plays a role of evaluatinginterventions once they aredisseminated, not just restrictingits activity to evaluating inter-ventions before they aredisseminated. It is a science ofevidence in action, not just ofevidence preceding action. Oneaim of this volume is to provide theconceptual framework and some ofthe tools to allow more effectiveevaluation of implemented policiesand programmes. It is designed tocomplement the often (necessarily)limited evaluation of interventionsthat occurs before they areimplemented.

There is the possibility thatempirical work will show thetheoretical model used to developand or evaluate the intervention tobe flawed: either incorrect in someof its assertions (includinginclusion of factors that have littleor no influence), or incomplete byignoring important factors. It isonly by specifying models that onecan systematically work to makethem better.

A model of evaluation isrequired that is designed for thedynamic, ever-changing world inwhich we live. The potential of theworld’s diversity must be viewedas a tool to aid in understanding,not an obstacle to be overcome.Each action of government is anattempt to influence outcomes inways consistent with policy goals,which, hopefully, aim to improvethe health and well-being of thecommunity. Similarly, the actionsof tobacco companies are alsodesigned to affect smoking, in this

case in ways that enhanceshareholder value, which is whythey are almost invariably directedat increasing or at least main-taining use. Even the bestthought-through interventionssometimes fail to work asexpected, and policies that work inone context sometimes stopworking when the contextchanges. Because neither pastexperience nor theory can berelied upon to always deliver thebest solution to our problems,methods must be established tocheck when and how things areworking. This is what modernevaluation is about. A frameworkfor effectively evaluating policyinterventions is essential.

Such a model places lessstringent tests on demonstratingthat something has equivalenteffects in a new context or whendelivered in a new form (wherethere is no reason to expectchanges in efficacy) than it doesfor evaluation of truly novelinterventions or their implemen-tation under conditions wheredifferences in effects is plausible.However, it still calls for strongerevaluation methods when evi-dence accumulated to question anassumption of equivalence. Thusit provides an explicit link betweenthe roles of ongoing auditing ofprogrammes to ensure continuedeffectiveness and more intensiveevaluation activity when there areconcerns. As these decisions arebased around clearly articulatedtheories, the framework is open toscrutiny and should allow the mostcost-effective possible evaluationby demanding plausible reasons



16

before testing for differences ineffects.

CChhaarraacctteerriissttiiccss ooff iinntteerrvveenn--ttiioonnss

Typically, policy interventions aredesigned to have sustainedeffects, but in some cases thismay require designing ongoingprogrammes to ensure that thishappens. Further, there may beshort-term onset effects. Forexample, there is evidence thatwarning labels on cigarette packshave an onset effect as well as asustained effect (Hammond et al.,2007a). We need evaluationmethods that can differentiateonset effects from sustainedeffects, and which also can helpus understand the conditionsunder which both kinds of effectsare maximised.

It is necessary to understandwhat, if anything, is required tosustain potential enduring effects:that is, what endures withoutfurther intervention and whatrequires regular updating, or asustained presence. For example,anti-smoking mass media cam-paigns have a short-term impacton quitting (Snyder, 2001). Itseems important to maintain cuesin the environment to remindpeople of information for thatinformation to have a maximalimpact. The form of some kinds ofinterventions may also need tochange over time if the effects ofthe intervention are to besustained. This applies particularlyto communication-based inter-ventions. What is seen asup-to-date, and thus of most

relevance for communication,changes quite rapidly in somecommunities. Similarly, acrosscultures, intervention may need tobe framed differently to ensurecultural relevance. Under somecircumstances it can be useful toconceptually separate the coreconcepts in an intervention fromthe mode of communication usedto convey them. Thus evaluationmight focus on the culturalrelevance of the intervention or onits underlying potency, or both.Analogous to the way societiesand/or people change, inter-ventions need to change tomaintain their relevance. This callsfor an equivalent model to that ofhow to create new immunizationsfor emerging strains of influenza.Here, the rate of change in theproblem is too rapid for RCTs tobe practical, and quite differentmethods are used.

Changes to interventions mayalso be required as a societyprogresses through the adoptionof an innovation cycle for adoptingnew sets of values and beha-vioural options for tobacco use.Take, for example, encouragingthe adoption of smoke-freehomes. This happens first in theface of social disapproval, or atleast lack of understanding. Anentity instituting a ban will often beasked to justify it, and some mightsee it as unreasonable. However,as such bans become morecommon, there comes a tippingpoint where smoke-free environ-ments become the norm. Sincejustification is no longer neces-sary, smokers often just do notsmoke when indoors, and those

without such bans feel a need tojustify their positions. Before thetipping point, even quite intenseinterventions may have limitedimpact (as has been the case forimplementing smoke-free homes(Hovell et al., 2000)), while after itpeople may be readily able tochange without help (as evi-denced by rapid adoption of thepractice in some countries (e.g.Borland et al., 1999)). Wherethings change, the rate of changemust be considered as well. Whenit is more rapid than the time forthe institutionalisation of inter-ventions through traditional testingof efficacy and so on, then newmethods must be adopted to allowinterventions to be changed intrain with the changing context.This is one of the reasons why it isimportant to pre-test the mes-sages used for cultural relevance,even for proven interventionswhen applied in new contexts.This is also why it is important toconduct ongoing evaluation ofdisseminated interventions.

HHooww ppoolliiccyy iinntteerrvveennttiioonnsstthhaatt ttaarrggeett bbeehhaavviioouurr wwoorrkk

Evaluations of population-level in-terventions are typically interestedin determining the overall effect ofthe intervention. As a conse-quence, it is not so much aboutasking whether an intervention ofthis kind can work, but of askingunder what circumstances does itwork and how to optimise thoseconditions to get maximal impact.This involves consideration of thereach of the intervention (some-times no more than awareness),



17

the ways people respond to it andits underlying potency or efficacy.

There are three key aspects ofinterventions from the perspectiveof the individual: awareness of, ac-ceptance of, and actions taken in re-sponse to policies. Evaluation mustdeal with all three. The first aspectis determining the extent to whichthe target population is aware of theintervention, which is a function ofits implementation, dissemination,and surrounding publicity about it.Awareness is generally a prerequi-site of policy effects, except in thoserare cases where the policy createsenvironmental conditions that canhave direct conditioned effects; i.e.independent of conscious aware-ness.

The second aspect isdocumenting attitudes towards theintervention by the targetpopulation, as this can affect theirresponses to it. Policies that areunpopular are more likely to beresisted, and forms of assistancethat seen as inappropriate to theperson’s needs are unlikely to beadopted. Thus, a smoker whoobjects to smoke-free rules ismore likely to ignore the rules or toseek convenient alternatives,while a smoker who approves andsees this as an opportunity to gaingreater control over their smoking,may not only comply, but use theopportunity to either quit alto-gether or reduce theirconsumption. A price increase willonly cause smokers to try to quit ifthey see the increased price asmaking smoking no longer worththe cost. Alternatively they couldsmoke more of each cigarette tomaintain the value, or shift to a

cheaper brand, or seek outsources of cheaper cigarettes, oreven re-interpret smoking assomething more exclusive andthus desirable. Like awareness,acceptance can only really beevaluated at a population level,although it is typically theacceptance of each individual thatis critical. In some collectivistcultures, the views of communityleaders are also critical, as theydetermine what it is acceptable tothink and do. These roles are inaddition to the roles of leaders inall cultures as policy makers.

The third aspect is theevaluation of the actions thatresult: that is, the consequencesor outcomes of the intervention interms of both intended andunintended incidental effects. Thisis a function of both the actionstaken by the individual and thepotency of the intervention. Whiletraditional intervention evaluationrestricts its focus on outcomesamong those who are encouragedto use the interventions, for policyinterventions this is not a usefulrestriction; one must consider thetotal impact on the population,including those who areunaffected. Outcomes should beconsidered as a joint function ofthe potency of the interventions,the ways they are used orresponded to (a function ofattitudes to them), and the degreeof exposure to them.

TThhee tthheeoorriieess bbeehhiinnddttoobbaaccccoo ccoonnttrrooll

A critical step in developing anevaluation framework is having a

coherent theory or set of theoriesas to what tobacco control isabout. This should extend beyondthe list of tasks identified in theWHO FCTC to an analysis of howthe various domains of inter-vention are theorised to contributeto the overall goal. The nature ofthe relationship between tobaccouse and harm must be sufficientlyunderstood to know whatbehavioural aims are appropriate.Such an analysis should considerthe broad scope of potentialimpacts, not just those that arepart of the rationale forimplementing any particular policyinitiative. For example, the impactof smoke-free places, introducedto protect non-smokers, also havebeneficial effects on smokers anddo not appear to have some of theadverse effects on economicactivity that some had feared(Scollo et al., 2003). Detailedanalysis of the conceptual foun-dations of specific interventions isprovided in the relevant sectionslater in this volume. Here the WGaddresses a few broader issues.

A broad schematic overview ofkey influences on tobacco use andtobacco-related harm is providedin Figure 1.1. This figure makes itclear that policy and socio-culturalinfluences have indirect effects onuse and that the most proximal de-terminants of use are the product;cues in the environment; charac-teristics of people, including cog-nitions about the products; and theperson’s biology (both conditionedand innate). Further, the behav-iour and the product jointly deter-mine exposures, which, ininteraction with existing biology,



18

determine harm (see Figure 1.6).The role of a systematic science oftobacco control is to analyse andclarify the components of this sys-tem and their interrelationshipsover time, with the aim of introduc-ing interventions that will minimisethe harms. Figure 1.6 is a generic

model for this. It is possible to elab-orate this figure to include other im-pacts of policies (see Figure 1.7).With generic models of this kind,areas that require greater attentioncan be expanded upon and boxeswhere things are more straightfor-ward can be combined.

Tobacco control efforts can befocussed on users and potentialusers of tobacco products (e.g.changing knowledge and beliefs),or they can be designed to directlyreduce use (e.g. price andavailability controls), or to reduceuse indirectly by changing theenvironment to increase cues toinhibit use (e.g. warning labels onpacks), or reduce cues to use (e.g.by constraining tobacco com-panies’ marketing practices), or bychanging the nature of thetobacco products on the market(see Figure 1.8). Efforts can alsobe directed at reducing the toxicityof tobacco products (targeting theindustry), and at reducing theexposures of non-smokers (tar-geting tobacco users). Tointervene in any of these wayswith either people or companiesrequires a good understanding(theory) of how the factorsproducing unwanted effectsoperate and how the interventionwill affect those operations. It isbeyond the scope of this volumeto spell out such a complex theory,although in each section, relevantelements are canvassed.

TToobbaaccccoo iinndduussttrryy ccoonnttrroollss

Tobacco industry controls areabout targeting the 4 Ps of mar-keting: Product, Price, Place (oravailability) and Promotion; towhich a fifth P can be added,Packaging; and, unrelated to mar-keting, the imposition of specificobligations to provide information(for example, warning material) re-gardless of its impact on the mar-ketability of the products. This is

Policy-relatedinterventions

Propositionsabout tobacco

Sensory stimuli

Otherinfluences

Tobaccoindustry

Tobaccoproducts

Tobacco productcontents

Tobacco product yields

Conscious processing

Patterns ofuse

Cumulative exposure

Tobacco harms

Toxin exposureper use

Tobacco use

Figure 1.6 Schematic diagram of main pathways by which policiesaffect tobacco use, tobacco exposures and tobacco harms



19

Policy-relatedinterventions

Propositions abouttobacco

Sensory stimuli

Otherinfluences

Tobaccoindustry

Tobaccoproducts

Tobaccoproductcontents

Tobaccoproduct yields

Passiveexposures

Tobacco use

Patterns ofuse

Toxin exposureper use

Conscious processing

Cumulative exposure

Tobacco harms

Figure 1.7 Model from Figure 1.6 expanded to illustrate where effects other than on tobacco use fit in

achieved through a mix of lawsand agreements, generally tar-geted at manufacturers or distrib-utors, but in other cases, at otherpoints in the supply chain (e.g. re-tailers). Evaluation of tobacco in-dustry controls also requires ananalysis of possible industry ac-

tions to counter the intended ef-fects, or to otherwise minimise ad-verse effects on their business.

Product controls (see Section5.3) include rules about what typesof products can be sold (e.g.smokeless tobacco is banned fromsale in some jurisdiction), levels of

constituents or emissions (e.g.upper limits on tar, nicotine andcarbon monoxide as measured byISO standard testing; restrictionson additives/ ingredients), or onengineering features (e.g. man-dating reduced ignition propensitycigarettes, filters). The aims of



20

Tobaccoproducts

Production regulations- types- constituents- engineering

Price controls and taxes

Constraints onavailability

Controls on promotion

Controls on packaging

Informationrequirements; warning labels

TToobbaaccccoo UUssee CCoonnttrrooll

Education campaigns

Rules on use, e.g.smoke-free policies

Cessation aids

Tobaccoharms

Tobacco use

Cues to use

Cues toinhibit use

Passiveexposures

Tobacco users andpotential users

Figure 1.8 Schematic overview of tobacco control interventions and how they relate to tobacco products,users and potential users

TToobbaaccccoo IInndduussttrryy CCoonnttrrooll



21

product rules vary from preventingnew forms of tobacco (to amarket) becoming established(e.g. bans on smokeless), toreducing their appeal (e.g. banson flavourings), both of which aredesigned to reduce use, and rulesto reduce the harmfulness of theproducts (e.g. constituent limits),which can also have direct effectson the harm caused.

Price controls (see Section 5.1)includes efforts to damperdemand through increasing prices(e.g. taxation of various forms),which can have direct effects onuse, as well as strategies toprevent price-related marketing(e.g. setting minimum and/ormaximum prices to prevent dis-counting and other forms ofprice-related marketing).

Place or availability controlsrefer to efforts to reduce theavailability of the products andinclude restrictions on the numberor types of outlets, and to whomthey can be sold (e.g. age limitsand bans on vending machines).Many of the existing rules havebeen put in place to discourageuse by young people, but res-trictions could also be used toreduce impulsive purchases and/orto discourage use in certain venues(e.g. bans on sales in bars).

Packaging controls includerules about what can be on thepack (e.g. use of terms like “Light”and “Mild”; see Section 5.5). Italso includes rules that prohibitsale of single cigarettes andestablish a minimum pack size tostop use of packs with smallnumbers of cigarettes, which areknown to appeal primarily to

young people (Wilson et al., 1987;Assunta & Chapman, 2004a;Prokhorov et al., 2006). Theeffects of such policies mayoperate through reducing cues touse, or by making the product lessattractive, reduce the value ofusing such products.

Controls on promotion (seeSection 5.4) are the most promi-nent form of control on the indus-try. They are essentially aboutreducing cues to use, but in doingso, might also reduce the appealof the products. Controls includebans on paid advertising, spon-sorships, and product placement,and encompass restrictions onpackaging (including controls onthe use of trademarks, e.g.generic packaging). Because to-bacco is sold in a competitive mar-ket, some signs differentiatingproducts as belonging to a manu-facturer/marketer are necessary.Even in places when brand dis-plays and advertising is banned atpoint of sale, a generic sign say-ing that tobacco is sold is allowed.This promotes availability. To-bacco retailers can also promoteproducts to customers by word ofmouth.

The final type of rules is inde-pendent of attempts to controlmarketing, and is about what formand content are required for warn-ings. The content may includefacts about the adverse effects oftobacco use, benefits of quitting,and information about toxin levels(see Section 5.5). Here the aim isto discourage use or at least en-sure that any continuing or newuse occurs in the context of someinformation about the risks; that is,

it provides cues to inhibit use.Warning and other risk-related in-formation can be required on pack-ets, at the point of sale, on anypermitted advertisements, or inconjunction with any depiction oftrademarks or commercial mentionof products.

Tobacco industry controls areoften about reducing cues to usetobacco, while tobacco use controlefforts and information provisionrequirements directed at industryare about increasing cues todiscourage use. For cues to use,the effect on behaviour is oftenconditioned such that they willstimulate tobacco use unlessactively resisted. By contrast, cuesto inhibit use are more likely tooperate via conscious processing.

Evaluation of tobacco industrycontrol is first about assessingcompliance with the rules. This isunlikely to be an issue where therules are to control obviousactivities of small numbers ofcompanies (e.g. compliance withlabelling requirements), but can bean issue where there is morepotential for avoidance (e.g. manypotential actors or where theaction is not so obvious; e.g.payment/avoidance of taxes).Evaluation is next about deter-mining the effects of the rules.What is involved here varies as afunction of whether the rulesmandate some actions (e.g.warning labels, higher prices) orwhether they mandate removingsomething (e.g. promotional cuesto smoke) that would otherwise bethere. In the former case, issues ofreactions to the change need to beevaluated. In the latter, the extent



22

of previous response to the cues(or other things) removed must beknown before the impact of theirremoval can be effectively evalu-ated. As noted above, it isnecessary to monitor and evaluateany industry actions that mightoccur to reduce the impact of therules on their businesses.

TToobbaaccccoo uussee ccoonnttrrooll

Tobacco use interventions arethose targeted at tobacco users orpotential users directly. They in-clude rules about use, attempts toprovide messages aimed at pro-viding information and changingattitudes and beliefs, and pro-grammes to deliver interventionsthat can facilitate appropriate be-haviour change, or in the case ofprevention interventions, effec-tively inoculate against uptake ofany of addiction-level use.

Rules about use includepolicies to make various placessmoke-free (see Section 5.2).Smoke-free rules are generallydesigned to protect non-smokers,although in doing so they haveeffects on smokers that need to beunderstood. Rules could also beabout which products could beused, and by whom. However,where there are restrictions onuse of products or who can usethem, they are usually alsocodified as rules against sellingsuch products (e.g. smokelesstobacco) or selling to particularindividuals (e.g. minors), so theseare best considered underindustry control even when theparallel restrictions are imposedon individuals as well.

Provision of messages essen-tially relates to mass mediacampaigns, where the intent is toexpose as many people aspossible to the campaign (seeSection 5.6). This may includecampaigns to promote pro-grammes. Campaigns aredesigned to inform people and tomake the issue emotionally salientenough to stimulate appropriateaction. One of the enduringchallenges of tobacco control isthat because the main adverseeffects of smoking are not evidentuntil after a long lag time, smokersdo not experience any significantsense of the harm they are doing,and thus tend to underestimate itsharmfulness (Slovic, 1998). Thereare extra issues to consider in theevaluation of prevention cam-paigns. Focussing on an issueincreases awareness of it and mayincrease interest, which ifunchecked could lead to increasedexperimental use. Designing pre-vention campaigns or programmesin ways that overcome thisincreased interest requiresthought. There is evidence thatsome prevention campaigns,especially those emanating fromtobacco companies, can haveadverse effects (Wakefield et al.,2006), presumably through theincreased interest in the issue theyengender.

Programmes to disseminateinterventions include rules regu-lating cessation medications,provision of services, and sub-sidies to products or services (seeSection 5.7). The kinds ofproducts/services vary, includingself-help resources, stop-smoking

pharmaceuticals and coaching oradvice programmes of varioustypes. As noted earlier, thisvolume is not concerned withevaluating the efficacy of theseproducts or services themselves,but on evaluation of their com-munity-wide dissemination anduse. Beyond this, there is interestin considering the effects of theexistence of cessation services onthe broader community. There issome evidence that awareness ofthe availability of quit-smokingprogrammes can stimulate quittingactivity even among those who donot use the services (Ossip-Kleinet al., 1991).

Evaluation of tobacco useinterventions should consider boththeir intended effects andincidental effects. They need to beinformed by a sophisticatedunderstanding of psychologicalprinciples, and where there arecompeting psychological pro-cesses involved, it is important toput in place measures of allrelevant processes. Where addi-tional effects to those sought areknown (or hypothesised) they canbecome further justifications foraction (or inaction, if they are ormight be undesirable).

UUssee ooff llooggiicc mmooddeellss

Achieving a comprehensiveapproach to tobacco controlrequires adoption of a range ofdifferent strategies, underpinnedby differing constructs andtheories. It is important to spellout the relevant concepts toconsider in each area in which apolicy intervention might be



23

planned. The WG has adopted thestrategy of encouraging the use oflogic models or flow charts to spellout the main constructs that needto be measured for each type ofpolicy. The criterion we adoptedwas to divide an area to the pointwhere the causal pathways weresufficiently different to makedealing with the various possi-bilities difficult within the oneframe. The WG used Figures 1.4and 1.5 as generic models, but aswill be seen, found the need tomodify them considerably for somepolicy areas. We accept that asknowledge about how some ofthese interventions work accu-mulates, new distinctions maybecome necessary, which couldlead to further subdivisions ofintervention type. Further, in somecases, distinctions may be shownto be of lesser importance, allowingsome of the existing boxes to becombined. It is only once acoherent theoretical model of thedomain has been established thatdetermining the constructs tomeasure becomes possible.

MMeeaassuurreemmeenntt iissssuueess

Measurement is critical toevaluation. To measure the con-cepts of interest, these conceptsmust first be defined in ways thatmake them amenable to measure-ment. These definitions constitutethe constructs. Constructs can beoperationalised in many ways.This operationalisation must comefrom a clear consideration of theconcepts and thus of theunderlying theory. Because con-structs are defined in terms of the

theory and not directly inrelationship to what measuresthem, error is localised in theimperfect relationship between theunderlying construct and themeasures used to assess it.Many of the concepts that need tobe measured are not directlyobservable, or, where they are,they sometimes stretch thecapacity of the respondent torecall or otherwise come up with avalid answer (e.g. rememberingquit attempts months or yearsago). As a result, most measuresare subject to a range of possiblebiases as indicators of their targetconstructs. Exceptions arecharacteristics such as sex anddate of birth, which in mostcultures at least can be reportedvery reliably (although not in all).One of the great challenges ofmeasurement is that the mea-sures that are most easilyobtained are often not idealoperationalisations of the con-structs of interest. For self-reported data, most things peoplereport are used as indicators ofbehaviour patterns or of under-lying beliefs, behaviour patternsand/or understanding, not assimple answers to the question.The lack of direct measures alsooccurs for many physical mea-sures. For example, cotininelevels are sometimes used toassess intake of nicotine or extentof smoking. However, becausepeople differ both in size and inrate of nicotine metabolism,cotinine is a biased measure ofintake or exposure at an individuallevel, although it can be a goodestimator at a population level.

The problem of the measure that isavailable not being a directmeasure of the construct of interestmay be greater when existing dataare used, as compromises arecommonly made in the interests ofbeing able to use what is at hand.These data were often collected forquite different purposes to those offocal interest, and thus themeasures used are often of relatedconstructs, not the exact onesbeing studied. Dependent on thestudy, evaluators may be forced touse measures of constructs withdifferent limitations. They need alanguage to help them talk aboutthe quality of measures inrelationship to the constructs theyare using the measures to assess.Unfortunately there is no con-sistent language for talking aboutthese distinctions, and the WGwere unable to develop one for thisvolume. The WG views thedevelopment of such a languageas critical to reducing the potentialfor conceptual confusion that canoccur from failing to consider thelimits of specific measures toactually measure the constructsevaluators are interested inmeasuring.

DDeetteerrmmiinniinngg wwhhaatt ttoo mmeeaassuurree

Choice of potential measuresbegins with an elaboration of thetheory or theories as to how theintervention might work, includingthe range of expected outcomesand potentially mediating (orintermediate) and moderatingvariables (effect modifiers), as wellas incidental effects. It might also



24

consider questions like: “Whatoutcomes will lead to healthgains?” and “What might influencepolicy adoption and/or continu-ation?” Evaluators should alsoconsider whether the sameoutcomes are relevant to allcultures. For example, in Islamiccountries and others wherealcohol use is prohibited or notsocially significant, considerationof smoking policies in bars is oflittle interest. Also the relevance ofsome issues can change as afunction of a society’s status inregards to tobacco control efforts.For example, support for andreports of smoke-free hospitalsare now so high in manycountries, it is no longernecessary to ask. However, incountries where passive smokinghas not become an issue, askingabout smoke-free hospitals maybe critical to assessing emergingcommunity concern. This analysisidentifies the concepts that itwould be desirable to measure.

Next, evaluators need to con-sider how they want to operational-ize the concepts as constructs.This needs to be done in a way thatensures that the constructs arestructurally independent of relatedconstructs they might want to relatethem to in causal pathways. Fur-ther, they need to consider whetherthe construct can always be meas-ured in the same way. Physicalmeasures typically measure thesame thing regardless of context,but answers to questions may not.For example, the direction of socialdesirability biases might switch assmoking becomes less sociallynormative. For any given study,

they must assess how well theconstructs of interest can be meas-ured. Where adequate measuresdo not exist, there will unavoidablybe gaps in the modelling. Some-times these gaps can be covered,at least in part, by using sets ofmeasures of related constructs.

In Chapters 4 and 5 of thisHandbook the WG providesguidance on measures that mightbe used in various evaluationcontexts. For any domain ofinterest we attempt to characteriseconstructs that might bemeasured as one of:1. Core constructs: those that

should be included wheneverthis domain is being studied.These will include key out-comes along with majortheorized mediators andmoderators. Not having mea-sures of any of these is likely tocompromise the study, or atleast limit the range ofinferences that can be drawn.

2. Important complementary con-structs, to use for detailedinvestigation of a domain.

3. Other measures or indicatorsthat may add some limited oruncertain value, but which wecannot recommend (for oragainst), or only recommend inlimited circumstances.

4. Not recommended: these onlyneed to be specified for com-monly used measures that havebeen shown to have no utility.The quality or validity of the

measures used for each constructalso must be considered. Validityof measures refers to the extent towhich they actually assess theconstruct they are designed to.

This can be assessed through therelationship between the measureand a gold-standard measure (cri-terion validity), or by showing thatthe measure related to other theo-retically related constructs as hy-pothesized (convergent validity).One form of convergent validity ispredictive validity, where themeasure is shown to predict out-comes as theorised. A valid meas-ure of one construct is unlikely tobe an equally valid measure ofeven a closely related construct.Also, the validity of a measuremay vary as a function of how it isbeing used. Thus reports ofawareness of environmental cuesare not a valid measure of the ex-tent to which any single individualis exposed (because of differ-ences in sensitivity), but may be avalid measure of overall commu-nity exposure (as the individual er-rors are assumed to cancel outacross the population). Validityalso only relates to the contexts inwhich it is established. As thecontext changes the validity of ameasure may vary. For example,self-reported age is generally avalid measure of how old some-body is. This is so in cultureswhere birthdays (anniversaries)are important occasions, but maybe less so in cultures where peo-ple take no notice of birthdays.Also the validity of measuresvaries directly with the precisionrequired of the measures: meas-ures that may be valid for detect-ing large-scale effects might notbe adequate for detecting smalleffects.

The WG uses the followingbroad categories to provide an



25

indication of the quality ofmeasures:

• Gold standard measure. Estab-lished valid measure of aconstruct of interest that isbetter than alternatives in allways.

• Clearly validated outcome orpredictor. There is evidence thatthis is a good way of measuringthe construct, in at least somespecifiable contexts. Limits tovalidity should be noted.

• Evidence of utility. There existssome validity data, but it is notstrong. It might be one of arange of alternatives with noclear way of differentiatingbetween them. These shouldonly be chosen when no bettermeasure is available.

• Face validity. This involves ananalysis of the extent to whichthe question taps the construct,and may be all that is availablefor single item self-reportmeasures. Where possible, we also

provide an indication of thesensitivity of the construct tomeasurement error. For example,how robust is a question todifferences in wording? Or indeed,might wording or contextualizingstatements need to differ bycontext and/or by characteristicsof the respondent? For example,some questions need to changefor use with current smokers ascompared to ex-smokers; e.g.“How confident are you that youwill be able to stay quit, if/whenyou try (The last qualifying phaseis not needed for ex-smokers)?”Users of this manual should keep

in mind that the quality of ameasure may be dependent onthe type of study in which it iscollected and the use to be madeof it. The assessments made hereassume the measures are madein appropriate circumstances.

TTyyppeess ooff ddaattaa uusseedd iinn eevvaalluuaattiioonn

The type of data needed forevaluation varies, and in somecases it can be found in existingdata collections, although some-times measured in ways that areless than ideal for the newpurposes to which it is going to beput. In some cases, measures ofthe variables of interest areavailable from more than onesource. In these cases, decisionsneed to be made as to whichsources of information are mostuseful. Issues to consider hereare validity, practicality ofcollection, and the extent to whichthe data can be related to specificindividuals. However, in mostcases, the necessary informationwill need to be collected, givingthe researcher greater controlover the ways in which therelevant constructs are measured.Some of the main types of dataand major ways of collecting it areoutlined below.

1. Documentation of policies.Critical to any form of evaluation isdocumenting the nature of theintervention. Documentation ofpolicy can occur at two levels: theespoused intent or formal policy(something that is typicallydocumented), and the actual

program of activity that is put inplace to implement it (which isusually more difficult todocument). Policy documentsshould be collated and coded inways that allow appropriatecomparisons to be made. There isnow an international repository ofinformation about the content ofnational tobacco control policies(See Section 4.1), making thistask easier, at least for national-level policies. Some countriescollect this information for sub-national policies, but in mostcases, the information will need tobe collected from each jurisdiction.Where there are many such setsof rules (e.g. of workplaces, localgovernments), it is usually moreconvenient to either obtainsamples of policies, or to userespondents in population studiesto report on the rules that apply tothem. Clearly, this latter form issubject to the problem thatordinary people often do not knowabout rules, and where they donot, may respond in terms of whatthey remember. For example,when asked if there are bans onsmoking in their workplace, somewill know the formal rules andrespond appropriately, whereasothers may know the rules butrespond in terms of what actuallyhappens (e.g. if there is a rule, butit is ignored, they will report thatthere is no rule, interpreting thequestion to mean, “Can peoplesmoke?”). Others will only be ableto answer in terms of what theyinfer from their recalled obser-vations, e.g. “Nobody smokesthere, so it must be banned.” Thismeans that such reports may not



26

be able to help differentiatebetween policy existence andpolicy implementation. Indeed,generally there are difficulties indirectly determining implemen-tation, especially for complexpolicies independent of theireffects. This is only a problemwhen the research questionsinclude asking whether problemswith a policy occur at the level ofpolicy content, or are a problem ofimplementation.

2. Identifying changes in theenvironment or factors thatmight moderate policy effects.The challenges of doing thisdiffer by the environment underconsideration.

a) Mass media. Monitoring ofnational and regional media,with sampling of communitiesfor audit of local media, is themost objective source of whatis potentially available. Thisdoes not cover some importantsources like the Internet. Anaggregated respondent reportis useful where there aresufficient observations percommunity unit. Individualreports are subject to sen-sitivity bias, such that whenthinking about quitting, or tryingto quit, the person is likely to besensitized to mentions orimages of tobacco or smoking.This means that respondentreports should not be used asindicators of exposure in mostindividual-level analyses.

b) Physical environment. Theseconsist of rules about publictobacco use and cues totobacco use from things like

point-of-sale displays, bill-boards, and posters. They canbe collected through obser-vation in sampled settings.They may also be estimatedfrom reports from relevantorganisations (e.g. of work-places as to the restrictions onsmoking), but are assessedmore often by reports fromordinary citizens as to what theyexperience, or for smokers,what they actually did (e.g.“when last at a restaurant, didyou smoke?”). These reportscan be averaged acrosscommunities to estimate overalllevels of these features. Likeother respondent reports, theseare subject to sensitivity bias,limiting their use for individual-level analyses.

c) Production and sales data.Various forms of sales data, orproxies for sales data, may beavailable, usually related toreporting on taxes and excises.These may be national-level,but in some cases can beseparated by type of outlet orlocality. At a national level,there are some internationalrepositories of this information(see Section 4.2). Self-report ofprice paid is a fairly accurateindicator of prices, but little isknown of possible systematicbiases.

d) Characteristics of tobaccoproducts on the market. Theseinclude composition and engi-neering features of productsand performance characteris-tics. These can either be gath-ered from the manufacturers orthrough independent testing.

3. Effects on and characteristicsof individuals

a) Self-report data. Characteris-tics of individuals (knowledge,attitudes and behaviour) aregenerally only available fromself-reports (some scope forproxy reports, but limited be-yond smoking status). Self-re-port data can be of internalcognitive states that are not in-dependently verifiable (e.g. ofattitudes, knowledge or experi-ences), as well as of things thatcan, at least in theory, be vali-dated, such as behaviours.Sometimes answers to ques-tions can also be used to inferinternal states of which the re-spondent is either not aware ornot thought able to report accu-rately (e.g. personality traits).Many countries have routinebehavioural risk factor surveil-lance studies and/or tobaccospecific surveillance studies,and these can be useful in arange of contexts. Many coun-tries use standardised methodsand questions, and are workingtowards common repositoriesof data (see Section 4.3). Self-reports are affected by ques-tion wording and by otheraspects of the ways in whichthe information is collected (seeSection 2.2 for some exam-ples).

b) Physical measures. This in-cludes biological and chemicalmeasures (e.g. of cotinine lev-els). These are often used tomeasure behaviour indirectly,but this should be done withcaution. Limitations of thesemeasures as well as their



27

strengths are well documented(Benowitz, 1996a; Matt et al.,1999; Al Delaimy, 2002 ).

c) Proxy reports. For observableaspects of behaviour, reportsof others who know the targetindividual may be useful.

SSuurrvveeyy mmeetthhooddss ffoorr eevvaalluuaa--ttiioonn

Survey methods are crucial tomany forms of policy evaluation.These can range from surveys ofindividuals to surveys of informantsabout the activities of organisations(e.g. of governments or work-places). Two key issues areaddressed here: the samplingframe and the way the questionsare asked and answered.

Sampling: To be able to gener-alise to a population, the sampleneeds to be representative of thepopulation. This is a function ofboth the sampling frame and par-ticipation. It is thus desirable tohave broadly representative sam-ples, recognizing that true repre-sentativeness is unattainable.Participation is also crucial. Any bi-ases in participation threaten rep-resentativeness. Because oftennothing is known about all or someof those who do not participate,quantitative estimation of biases iseither impossible, or partial at best,meaning their likely effects need tobe inferred. The higher the re-sponse rate, the less likely majorbiases are, but unless the rates areclose to 100%, biases can occur.

Sample size is anotherimportant consideration. The twomain factors to consider here arethe size of effects that are expected

(or the required power to detect)and the desire to explore potentialmoderator effects. In principle,making a study larger does notimprove its representativeness.However, because size does in-crease power to detect moderatoreffects, larger samples can beused to increase confidence in thegeneralisability of the findings to allgroups who have a sufficientsample size for such possibleinteractions to be tested.

Question asking: The mainissue with surveys is inconsistencyand bias in the ways in which peo-ple respond to questions. This ispart of a general phenomenon ofthe frame of reference or contextfor the question affecting how it isunderstood, and thus how it is re-sponded to. Variation in frame ofreference includes mode of sur-veying (e.g. face to face vs. phoneinterview vs. self-completion).There is emerging evidence thatsome modes of surveying result inbetter response rates for sub-sec-tions of the population. There is anurgent need for research to de-velop optimal methods for calibrat-ing both questions and samplecharacteristics across modes (seeDillman & Christian, 2005, for a dis-cussion of general issues concern-ing mixed-mode surveying). As it isbeyond the scope of this volume todocument the entire range of is-sues corresponding to questions(there are several excellent textson this topic; e.g. Foddy, 1993;Fowler, 2001), we deal only withtwo issues in this chapter. Theseare the time frame over which an-swers apply, and cultural factors ininterpreting question meaning.

The time interval over whichthe response is deemed to bevalid is a crucial issue in testingcausal models. Causes precedeeffects, so one must assume thatpredictor variables when mea-sured at the same time as out-comes, predated the occurrenceof the outcomes. Sometimes ques-tions are given a time frame or tim-ing of events is asked for to assistin determining sequences. Self-re-ports of periods or of dates aresubject to biases in reporting withevents sometimes displaced intime. Self-reports are typically bet-ter for recent events (due to mem-ory effects). Salient events may bereported as experienced more re-cently than in reality, and lesssalient events are prone to be for-gotten.

Aside from issues concerningthe context of survey delivery, theway in which respondentsinterpret questions and responseformats affects their answers. Onekey aspect is the extent to whichthe conceptual framework under-pinning the questions reasonablyapplies across the cultural con-texts under consideration. Asresearch moves from studyingissues like tobacco within WesternEuropean and North Americancultures, to studying tobacco useacross cultural settings wherethere may be different values andassumptions, there is a need toquestion the underlying assum-ptions that frame the research.Within all cultures, there will bevariation that researchers shouldtry to characterise and under-stand. The possibility that culturaldifferences may compromise the



28

utility of some questions needs tobe reviewed on a case-by-casebasis. Some of these issues andmethods for overcoming them arecovered in Section 2.2.

In principle, the response to aquestion can be directly comparedwhen the respondents are an-swering the same question. Peo-ple generally assume this meansthe same wording. However,under some conditions, the samewording can result in quite differ-ent questions being answered,and different wording may be re-quired to achieve equivalence.The most obvious example is ask-ing questions in different lan-guages, but it can occur for thesame language where respon-dents’ assumptions about what isbeing asked can vary systemati-cally, and achieving equivalencerequires different contextualisingwords for different individuals.This can be caused by words hav-ing different nuances in differentcultures, or effects due to the fa-miliarity and or normativeness ofthe issues being asked about.

As surveys become stan-dardised, there is a tendency forsurveys to converge on commonways of asking questions, thusimplicitly operationalising theconstructs they are interested in.To the extent that either theoperationalisation has an arbitraryelement or the measure is flawed,there is a risk of institutionalizingerror. To avoid this, it may beimportant to analyze whetherdifferent ways of asking questionsmay improve the ability tomeasure a construct. There isalways a role for asking questions

in different ways. Where theanswers are relatively invariant tothe form of wording, one can haveconsiderable confidence in gene-ralisability across the inevitablewording differences betweenlanguages. However, where res-ponses are sensitive to wording, itis less likely that different formsare actually measuring the sameconstruct, and extra care will berequired in translation.

SSttuuddyy ddeessiiggnnss ffoorr eevvaalluuaattiinnggppooppuullaattiioonn iinntteerrvveennttiioonnss

To best understand the impli-cations of policy change (includingcommunity-wide dissemination ofinterventions), research designsshould be as strong as possible. InSection 2.1 the relative strengthsof various evaluation designs arecan-vassed. In short, evaluation isstrengthened with more obser-vations (both before and after theintervention) within the populationan intervention occurs in, the morepopulations that are studied inparallel, and the more alternativeexplanations for outcomes thatare assessed within each study. Inaddition, the use of cohorts addsconsiderable power by allowingmediation and moderation effectsto be tested more precisely.Finally, representativeness of thesample to the study populationcan increase the generalisability offindings. The ITC study (Fong etal., 2006a) is a good example ofwhat can be achieved byattempting to implement as manyof these attributes as possible.

Achieving the strongest pos-sible evaluation involves putting in

place measures of key outcomes(at least) as long as possiblebefore the policies are imple-mented. Obviously the best way todo this is if the measures can bepart of the country’s ongoingsurveillance system. Where this isnot possible, the studies should beimplemented as early in theprocess of discussing policychange as possible.

For detection of trends, it isimportant that both samplingframe and participation ratesremain constant. This is tomaximise the likelihood thatbiases are likely to remainconstant so that any changes areunlikely to be due to a samplingeffects. Repeatability is moreimportant than representativenessfor determination of trendsbecause it requires comparabilitybetween estimates over time.

Such a research agenda re-quires monitoring of all relevantvariables in a diverse range ofcommunities or jurisdictions overa period of time in which there aredifferences in policy implementa-tion between those communities.This will include use of repeatedcross-sectional surveying, andwhere possible, more in-depth lon-gitudinal cohort studies of samplesof relevant individuals (e.g. smok-ers, and young people at risk ofuptake), to begin to explore howthe changes come about andwhether some groups are affecteddifferently to others. This survey-ing will need to be complementedby longitudinal monitoring of eco-logical variables. The level (nation,state, local area) of the variablemeasurement will determine the



29

practicality of maintaining ongoingmonitoring of all activity or whethersome sampling is necessary.

Such a program of data collec-tion is needed to provide the infra-structure necessary for under-standing the mechanisms of pop-ulation level change. Among otherthings, it would increase under-standing of which factors are cul-ture-sensitive, and which are not,and how the roles of various fac-tors change as a person’s positiontowards changing and adopting tar-get behaviour changes. Similarly, itwould allow for an understandingof how community readiness tochange affects realized changeand how readiness can be modified,as well as the conditions that facili-tate the institutionalization ofchange. For policy makers, it canprovide information on need for fur-ther action.

DDrraawwiinngg ccoonncclluussiioonnss aabboouuttccaauusseess

The approach the WG has taken toevaluation shares more with themethods used in epidemiology todetermine causes of illness, thanthe reliance on RCTs to assessclinical interventions. As a result,when considering criteria to use indrawing conclusions about theeffectiveness of policy inter-ventions, we have adapted thecriteria used in the epidemiology ofdisease (Hill, 1965). The adaptedcriteria are:

• Magnitude of the observedeffect, particularly in rela-tionship to known naturallyoccurring variations;

• Temporal relationship betweenintervention and change intarget outcome;

• Exposure-response gradient;• Biopsychosocial plausibility;

that is, the effects can be ex-plained as occurring through aplausible mix of biological, psy-chological and/or social pro-cess;

• Coherence across lines of evi-dence with different threats tovalidity, e.g. similar resultsusing aggregate data and self-reported consumption couldrule out response biases;

• Coherence of results fromdemonstrations of effects ondifferent parts of the theorisedcausal pathway, or by demon-strating efficacy of components(e.g. the evidence of efficacy ofmany cessation aids makes itmore likely that they have ef-fects when delivered as part ofprogrammes of help);

• Evidence that this type of inter-vention can have effects onother comparable outcomes(e.g. on other behaviour pat-terns);

• Consistency of observed ef-fects across studies and popu-lations, or clear patterns in thevariability to demonstrate limitsto generalisability;

• To which we would add: Elimi-nation of theoretically possiblealternative mechanisms for ex-plaining the observed effects.Policy evaluation has added

challenges to other forms ofoutcome evaluation, becausepolicies usually occur in a mix andpolicies are only one set of factorsthat are responsible for the

outcomes of interest. Smokingprevalence or rates of quitting aredetermined by multiple factors,and establishing the contributionof each individual intervention isdifficult. The task of differentiatingthe contribution of all possiblecontributors to the observedeffects is difficult.

In providing a summativeevaluation of the effects of anintervention, we need to not onlyconsider the size and nature ofeffects, we also need to considerthe possibility that there is nomeaningful effect. In particular, itis important to make a cleardistinction between evidence ofthe absence of effects, and thesituation where there is a lack ofevidence; that we really do notknow whether an interventionworks or not. We recognize thatscience cannot prove the nullhypothesis, but it can and shouldmake statements about inter-ventions where there is aconsistent failure to find evidenceof any meaningful effect.

We need to qualify effects witha statement about generalisability.Some interventions have similareffects in most contexts, otherscan be quite context-specific. Thisconsideration needs to cover cul-tural adjustments to the interven-tion itself, as well as factors in theenvironment that might affect itspotency (effect moderators). It isalso important to consider the di-rection of effects. Some interven-tions might prove counter-productive. Clearly less evidenceshould be required to stop an in-tervention where the evidencesuggests that it is counter-produc-



30

tive, than if it suggested no effector only a small positive effect.

The levels of evidenceframework used to evaluatediscrete interventions is notappropriate for use in evaluatingpolicy interventions. We see morepromise in adapting the criteriaused by the International Agencyfor Research on Cancer (IARC)for its Cancer PreventionHandbooks. This is essentially afour-level system: Sufficient evi-dence of an effect, Limitedevidence, Insufficient evidence,and Evidence suggesting lack ofeffect. The WG’s concerns withadapting this framework to ourpurposes, is that it does not allowfor gradations in confidence ofconcluding no effects, it does notclearly differentiate adverseeffects, and it does not considerissues of generalisability, all ofwhich are desirable qualifiers inthe policy context. One possibilitywould be to adopt a matrix asshown on this page, withadditional statements on effectsize (for established effects) andon generalisability.

The effect size could be ratedas: Small, Medium, or Large (orundetermined). Considerationneeds to be given to whether thehighest level of certainly could beapplied to interventions wherethere had not been a directdemonstration of effects on thetarget outcome, or whetherinferred effects could ever berated as better than Probable. Forexample, it has been shown thatlarger health warnings lead tomore thought about quitting, andthat more thoughts predict future

quitting. However, nobody hasshown that there is more quittingin the context of stronger healthwarnings being introduced. Howreliably can one conclude thatstronger health warnings stimulatequitting?

Finally, once the effectivenessof an intervention is established,less powerful research designswill be needed to monitorcontinuation of effects and/or toassess whether similar magni-tudes of effect are attained withnew populations. It is only whenthere is reason to believe thatthere are real differences thatstronger research methods mightneed to be reapplied.

HHooww ttoo uussee tthhiiss HHaannddbbooookk

This Handbook is designed as aguide for program and policyevaluators. The WG hopes it willbe used as a tool for training newevaluators and those who need tounderstand evaluation principles.It can act as a reference source forarguments about the role ofevaluation and the way to thinkabout evaluation, and byextension the development ofeffective interventions. In doingso, we hope it provides aframework for increasing thescientific credibility of the field, by

helping to show that policyevaluation has rigorous methodsand can make importantcontributions to knowledge.

We also hope it will act as astimulus for further action toimprove evaluation methods andmeasures. As such, this Hand-book will need to be kept as up-to-date as possible. This mightinvolve periodic revisions once theprinciples have been tested, orsome other mechanism formoving our expected standardsforward. There is a particular needto update the material on specificmeasures and on the status ofdata repositories, as these are ina constant state of change.

We hope this Handbook willprovide a stimulus to work towardsgreater coordination of the ways inwhich policy evaluation operatesand the development and/orexpansion of international reposi-tories to collect the relevant dataand reports, and user-friendlyways to extract this informationand synthesise it.

Some future actions the WGwould like to see:

• Work to coordinate and arriveat a set of core terms that aremost useful for our field.

• Work on what the criteria forvalidation should be for the

The evidence matrix

No evidence is available

Possible effect: Negative Not meaningful Positive

Probable effect: Negative Not meaningful Positive

Established effect: Negative Not meaningful Positive



31

various kinds of measuresused, and how that relates tothe different types of mea-sures.

• Development and agreementon use of prototype formats forreporting on frequently re-peated interventions, such asmass media campaigns. Thiswill facilitate their combinationinto meta-analytic studies, es-pecially important for under-standing where and whenthings work.

In conclusion, this volumeshould be thought of as an impor-tant step in a process, rather thanas a static recipe book for evalu-ating tobacco control interven-tions. The methods described andthe measures provided are thebest available today. The princi-ples outlined in this volume willpersist, but those principles re-quire that methods and measuresbe adapted to the changing world.The WG has built into this Hand-book some guidelines for seeking

out the latest methods and someguidance in assessing the need tomove beyond the measures andmethods described here. We be-lieve that this dynamic but sys-tematic approach is the best wayto approach the future because itprovides a framework that allowsevidence to guide action both be-fore and after programmes or poli-cies are implemented.


chapter 1 ensuring effective evaluation of tobacco control ... · ensuring effective evaluation of...

Documents