who vip webinar 2011 1 evaluation considerations: measures & methods shrikant i. bangdiwala, phd...

WHO VIP Webinar 20111

Evaluation Considerations: Measures & Methods

Shrikant I. Bangdiwala, PhD

Professor of Research in BiostatisticsInjury Prevention Research CenterUniversity of North Carolina at Chapel Hill, USA

presented at


Content

Purpose of evaluation Cycle of program planning & evaluation Indicators Study designs Statistical modeling Challenges

Safety 2010 London3

What are we ‘evaluating’?

Actions, programs, activities Conducted in a community setting, over a

period of time Aimed at reducing deaths, injuries, and/or

events and behaviors that cause injuries



Example: Suwon, South Korea

area of ‘safety promotion’

http://www.phs.ki.se/csp/safecom/suwon2.htm


Why do we ‘evaluate’?

To know ourselves what works and if we are doing some goodIn performing some activityIn the communityIn the country

To convince funders and supporters that their investment is worthwhile

To convince the community about the benefits of the multiple activities and actions carried out

WHO VIP Webinar 20116 NSC Chicago 2010 6

Main purposes of evaluation

Evaluation helps determine:How well a program/policy works relative to its

goals & objectivesWhy a program/policy did or didn’t work, relative

to planned processHow to restructure a program/policy to make it

work, or work betterWhether to change funding for a program


Methodological complications

Multiplicities Multiple components of a programMultiple populations at riskMultiple study designsMultiple types of effects/impacts/outcomes &

severitiesMultiple audiences/objectives of ‘evaluation’Multiple methods for conducting evaluation

WHO VIP Webinar 20118 8

When should evaluation be considered? Evaluation needs to begin in, and be part of,

the planning process… Otherwise, “if you do not know where you are

going, it does not matter which way you go, and you will never know if you got there or not!”

Lewis Carroll (1872)

Alice in Wonderland

Adapted from M. Garrettson


ProgramPlanningPhase

Formative EvaluationHow can the program activities be improved before

implementation?

Process EvaluationHow is/was the program (being) implemented?

PostProgramPhase

Impact / OutcomeDid the program succeed in achieving the intended

impact or outcome?

Program Implementation

Phase

Types of evaluation depending on program phase


Cycle of program planning and evaluation

Adapted from C Runyan


Identify population & problem Surveillance data Other needs assessment strategies

key informant interviewsfocus groupssurveys evaluations of past programs literatureconsultation with peersother info…


Test & refineimplementation

Test, Refine,Implement

Choose strategies

Set goals/objectives

Disseminate

Identifyproblem

& population

Define target audience

Evaluation: •Formative

Implement

Identify resources

Evaluation:•Process•Impact•Outcome



To whom is the program directed?Whose injuries need to be reduced?Who is the target of the program?

• at risk persons• care givers (e.g. parents)• general public• media• decision makers


Understand target audience

What are their characteristics?Special needs (e.g. literacy)Interests, concerns, prioritiesAttitudes & beliefs re: problem & solutions to

problem Cultural issues




Choose strategies


Disseminate

Identifyproblem

& population



Implement

Identify resources



Identify resources

Community partnersinterest in topicworking on similar projects

On-going activities Sources of financial support Interests in community




Choose strategies


Disseminate

Identifyproblem

& population



Implement

Identify resources



Set goals & objectives

Goalbroad statement of what program is trying to

accomplish

ObjectivesSpecificMeasurableTime-framed




Choose strategies


Disseminate

Identifyproblem

& population



Implement

Identify resources



Choose Strategies

Identify existing strategies/programsLiterature: evidence based? promising practice?WHO manualsSuccesses from other communities-regions-

countries Develop new strategies:

Logic model (how would it work)Haddon matrix


Haddon Matrix

Person Vehicle/vector

Physical Environ.

Social Environ.

Pre-event

Event

Post-event

Haddon 1970 Am J Public Health


Pre-Event Other??

Decision CriteriaDecision Criteria

PhasesPhases

3-dimensional Haddon Matrix

P

erso

n

Veh

icle

/V

ecto

r)

P

hysi

cal

Env

iron.

Soc

ial

Env

iron.

FactorsFactors

Event

Post-event

FeasibilityPreferences

StigmatizationEquity

FreedomCost

Effectiveness

Runyan 1998 Injury Prevention




Choose strategies


Disseminate

Identifyproblem

& population



Implement

Identify resources



Formative Evaluation

What is the best way to influence the target population?

Will the activities reach the people intended, be understood and accepted by target population?

How can activities be improved?

Improves (pilot-tests) program activities before full-scale implementation

May increase likelihood program or policy will succeed

May help stretch resources

Why it’s usefulQuestions it answers

* Modified from Thompson & McClintock, 2000




Choose strategies


Disseminate

Identifyproblem

& population



Implement

Identify resources



Implementation

As planned, with attention to detail

Documented clearly so others can replicate if appropriate




Choose strategies


Disseminate

Identifyproblem

& population



Implement

Identify resources



Process evaluation

Purpose is to address:What was done?How was it implemented?How well was it implemented?Was it implemented as planned?


Process evaluation – examples of questions

• Who carried out intervention?• Was this the appropriate person/group?• Who supported and opposed intervention?• What methods/activities were used?


Process evaluation - why is it useful?

• Allows replication of programs that work.• Helps understand why programs fail.

* Modified from Thompson & McClintock, 2000


The intervention cannot be a black box…

OutcomeIdea ?

It must be clearly understood




Choose strategies


Disseminate

Identifyproblem

& population



Implement

Identify resources



Impact evaluation

Purpose is to address changes in: knowledge attitudes beliefs/ values skills behaviors / practices


Using impact measures for

Establishing effectivenessSuppose we have a public safety campaign as our

strategyNeed to show Campaign Behavior Outcome

If we already have demonstrated that Behavior Outcome

We simply need to show Campaign Behavior


Outcome evaluation

Purpose is to address changes in:injury events (e.g. frequency, type, pattern) morbidity (e.g. frequency, severity, type)mortality (e.g. frequency, time to death)cost (e.g. direct and indirect)


OutcomesImpacts Intervention

Physician counseling

parents

Enforcement of helmet law

Media campaign

Parental attitudes

toward child helmet use

Purchase of helmets

Use of helmets by children

Head injury in bike crashes

Deaths from head injury in crashes

Example: Bike helmets


Evaluation examples of questions for local policy of smoke alarms

Did the local policy of smoke alarms in apartments… Get passed Where people aware of it? Did people have access to smoke alarms? Did people get them installed properly? Do people keep them maintained? Lead to a reduction in the number or rates of:

events (e.g. apartment fires) injuries deaths costs (e.g. burn center costs, family burden, property loss)


Evaluation – selection of measures‘Quantitative Indicators’

Process Impact Outcome

Health related Financial


Choice of measure or indicator

We need to choose appropriate impact and outcome measures ‘Soft’ (more difficult to measure) outcomes –

Perceptions constructs: fear, insecurity, wellbeing, quality of life Knowledge, attitudes and behaviors constructs

Hard outcomes – Deaths, hospitalizations, disabilities due to injuries and violence Societal impacts – local development indicators

Economics outcomes – Direct $/€/£/¥, indirect DALYs, QALYs, opportunities lost, burdens


Evidence of effectiveness

Obtain qualitative ‘evidence’ to complement the quantitative ‘evidence’Ex. Are “multisectorial collaborations and

partnerships” friendly and functioning well?Ex. Is “community participation” optimal?

Incorporate process indicators Incorporate narratives & testimonials




Choose strategies


Disseminate

Identifyproblem

& population



Implement

Identify resources



Dissemination

Dissemination not done wellNot attemptedNot based on research about how to disseminate

information to intended audience Dissemination done well

Defining audienceHow to access audienceHow best to communicate change message to themPresentation of clear, straightforward messages


Evaluation measures

Lead to evidence of effectiveness But only if the research and study

methodologies, and the statistical analyses methodologies, are appropriate to convince the funders and supporters, the skeptics, the stakeholders, the communityand understandable


Research methodology approach:Evidence of effectiveness

Obtain quantitative ‘evidence’ that favors the hypothesis that the intervention is effective as opposed to the (null) hypothesis that the intervention is not effective.

How? Experimental study designs - randomized clinical trials,

grouped randomized experiments, community-randomized studies

Quasi-experimental study designs - non-randomized comparative studies, before-after studies

Observational studies - cohort studies, case-control studies and comparative cross-sectional studies


Randomized controlled trial (RCT) / Experiment

‘strongest’ evidence

Randomize

InterventionGroup

Control Group

O X O

O OX’


Quasi-experimental designs

‘qualified’ evidence

Intervention Group

Comparison Group

O X O

O O


One group pre/post

‘weak’ evidence

InterventionGroup

O X O


One group – multiple pre / multiple post

better ‘weak’ evidenceIntervention

GroupO O O X O O O O O


One group, post only

‘basically ignorable’ evidence

Intervention Group

X O

Safety 2010 London505050

Observational designs- cohort study

evidence?Self-chosen Intervention Group

Self-chosen Non-intervention Group

X X X

O O O



Observational designs- case-control study

evidence? Cases

Controls

X

O

X

O



Observational designs- cross-sectional study

evidence?

InjuredX X O XO

O X OO O

Non-injured



Statistical analysis methodologies

Choice - often guided by what has been done previously, or what is feasible to do, or easy to explain

Choice should be tailored to the audience & their ability to understand results; but also on the ability of the presenter to explain the methodologies


Statistical analysis Determined by research question(s) Guided by study design – experimental or observational

Group randomized controlled experiment Non-randomized comparison study Single site pre/post; surveillance study Retrospective or cross-sectional

Guided by whether outcome is studied at a single time point or multiple time points Time series analyses

Guided by audience Visual and descriptive appreciation


Visual and descriptive analysis – longitudinal time series

Example:

Espitia et al (2008) Salud Pública Mexico


Visual and descriptive analysis – comparisons over time

Example:

www.gapminder.org



Statistical analysis - challenge

But what we as a field have not done as well as other fields, is to draw strength from numbers develop collective evidence

Combine results from multiple studiesSystematic reviews (of observational studies)Meta analysis (of experimental & observational studies)Meta regression (of heterogeneous studies)Mixed treatment meta regression (for indirect

comparisons)


Systematic reviews

A protocol driven comprehensive review and synthesis of data focusing on a topic or on related key questions formulate specific key questions developing a protocol refining the questions of interest conducting a literature search for evidence selecting studies that meet the inclusion criteria appraising the studies critically synthesizing and interpreting the results


Example – Systematic review

Shults et al (2001) Amer J Prev Med


Systematic reviews

Of particular value in bringing together a number of separately conducted studies, sometimes with conflicting findings, and synthesizing their results.

To this end, systematic reviews may or may not include a statistical synthesis called meta-analysis, depending on whether the studies are similar enough so that combining their results is meaningful Green (2005) Singapore Medical Journal

Zaza et al (2001) Amer. J Preventive Medicine – motor vehicle


Meta analysis

A method of combining the results of studies quantitatively to obtain a summary estimate of the effect of an interventionOften restricted to randomized controlled trialsRecently, the Cochrane Collaboration is

‘branching out’ to include both experimental and observational studies in meta analyses


Meta analysis

e.g. Liu et al (2008) Cochrane Collaboration


Meta analysis

The combining of results should take into account: the ‘quality’ of the studies

• Assessed by the reciprocal of the variance the ‘heterogeneity’ among the studies

• Assessed by the variance between studies


Meta analysis – estimation of effect

The estimate is a weighted average, where the weight of a study is the reciprocal of its variance

In order to calculate the variance of a study, one can use either a ‘fixed’ effects model or a ‘mixed’/’random’ effects model Fixed effects model:

utilizes no information from other studies Random effects model:

considers variance among and within studies

*2

2

)var(

)var(

iYi

iii

VY

eY

Borenstein et al (2009) Introduction to Meta Analysis

iYii VeY )var()var(


Meta analysis & meta regression

Dealing with ‘heterogeneity’ among the studies - 2 Decompose the total variance into among and within

components using mixed effects models for getting a more precise estimate of the intervention effect

If there is still residual heterogeneityExpand the mixed effects model to include study-level

covariates that may explain some of the residual variability among studies meta regression


Meta regression

e.g.

iiiii eXXY 2211

Overall mean

X1 study variable – EU/USA

X2 study variable – population type

study random effect

random error


Meta analysis

Standard meta-analytical methods are typically restricted to comparisons of 2 interventions using direct, head-to-head evidence alone.

So, for example, if we are interested in the Intervention A vs Intervention B comparison, we would include only studies that compare Intervention A versus Intervention B directly.

Many times we have multiple types of interventions for the same type of problem, and we hardly have head-to-head comparisons

We may also have multiple component interventions


Mixed treatment meta analysis

Let the outcome variable be a binary response 1 = positive response 0 = negative response

We can calculate the binomial counts out of a total number at risk on the kth intervention in the jth study

We can then calculate

the estimated probability of the outcome (risk of response) for the kth intervention in the jth study

kj

kjkj n

rp

:

::

kjr :kjn :

Welton et al 2009 Amer J Epid



Let each study have a reference ‘‘standard’’ intervention arm, sj, with study-specific ‘‘standard’’ log odds of outcome, j .

The log odds ratio, j:k, of outcome for intervention k, relative to standard sj, is assumed to come from a random effects model with mean log odds ratio , and between-study standard deviation

where dk is the mean log odds ratio of outcome for intervention k relative to control (so that d1 = 0).

)(jsk dd




This leads to the following logistic regression model:

where ]),[(~: jskkj ddN

k

s

p

p

kjj

jj

kj

kj

int

int

1ln

::

:



Mixed treatment meta analysis- multiple-methods interventions

If we have multiple methods in the ith intervention

Plus we have multiple times when the outcome is assessed

itiititiit eXMMtY 423121

,...,, 321 iii MMM

Study effect

Time effect

Components 1 & 2 effects

Study covariable

Error term


Statistical analysis

Methodology does exist for developing stronger collective evidence, evaluating the effectiveness of community based interventions, using different types of study designs and interventions

Developing “practice-based evidence”


Dissemination

We should not stop at developing the evidence We must work alongside economists in developing

ways to effectively communicate ‘what works’ methodology and cost models do exist for estimating the “return on investment”

Money talks !!



Challenges – Evaluation requires

Integration of evaluation from the beginning Appropriate measures, possible to be collected objectively,

unbiasedly, easily and with completeness Appropriate qualitative and process information, to complement

the quantitative information Concrete and convincing evidence of what aspects work in

individual communities Formal methodological statistical evaluation of specific elements

of programs Collective evidence of what common elements of programs work Effective dissemination strategies – “return on investment”

who vip webinar 2011 1 evaluation considerations: measures & methods shrikant i. bangdiwala, phd...

Documents

evaluation slide

vip webinar

program slide

process evaluation

resources evaluation

program phase slide

community slide

evaluation needs