design and conduct of evaluations of cvd control programs (part i) gilles paradis, md, msc, frcpc...

Design and conduct of evaluations of CVD control programs (part I)

Gilles Paradis, MD, MSc, FRCPC

Jennifer O’Loughlin, PhD

McGill University Health Center

Department of Epidemiology and Biostatistics

McGill University

Outline Part I

Why evaluate? What’s evaluation Evaluate what? Scope of evaluation Methodological issues

Why evaluate?

1. AccountabilityReport on the attainment of objectives and use of limited resources

2. ImprovementTreatment, program performance.

3. AdvocacyEnhance programs, build consensus, support coalitions

Why evaluate?

Social responsibility beyond "Primum non nocere"

Many (well established) interventions have been subsequently shown to be useless or harmful

M.I.: Prolonged bed restMagnesiumClass I antiarrhythmicsCa++ channel blockers

Prevention: caroteneHRT (?)

What is evaluation?

Process of systematic data collection or information gathering to shed light on some aspects of an action or intervention

Respond to specific questions regarding a program "Who is being reached by…?"

Support decision making "Which of two alternative strategies is more effective?"

What is evaluation?

Enhance community participation "What are key community concerns?"

Improve the understanding of mechanisms of action"How can I reach low SES populations

with this program?"

Support community mobilization "What do key stakeholders expect from a coalition?"

Evaluate what?

Primary prevention programsReduce exposure to risk factorsDecrease incidence

Secondary preventionPrevent progression among affectedasymptomatic individuals (HBP, …)Screening, case-finding

Individual practiceDiagnostic, preventive, therapeutic

Organizational or community changesStructural (inputs, resources mobilized)Process (quality of services)Outcomes (attainment of objectives)

Tertiary preventionDecrease morbidity, mortality amongsymptomatic individualsImprove QOL, functioning

Scope of Evaluation

Broad approaches

1 - Normative

2 - Evaluative research

Scope of Evaluation

1.1 - Quality of preventive care GOAL: Compare practices to standards of

excellence or criterias EXA: Rules for use of resources

Who gets fasting lipoprotein profiles? Who gets 24 hour BP monitoring? Streptokinase or tpa?

Criterias of quality preventive care Management of HBP, type II diabetes Management of pts with IHD

METHODS: Chart auditsSurveys

1 - Normative

Scope of Evaluation

1.2 - Quality of programs GOAL: Structural: Appropriate use of

resources?Process: Target population

attained?Program implemented

asintended?

Impact: Were objectives achieved?

EXA: HBP screening in worksites Methods: Review of reports, existing databases

Key informant interviewsSurveys

1.3 - Evaluation of (public health) organizations Structure, functioning, planning, etc.

1 - Normative

Scope of Evaluation

Efficacy Effectiveness Efficiency (cost-benefit, cost-

effectiveness) Quality of preventive care (decision

analysis)

2 - Evaluative research

1 - Specification of theoretical model

2 - Design

3 - Measures (what and how)

4 - Biases

5 - Analysis

Methodological issues

1 - Theoretical model

Avoid “Black Box” phenomenon Observe connecting processes between

inputs and outputs Key to understand and improve

interventions Describes how program produces the effect Blueprint for selection of variables, guiding

analysis, interpreting results

Methodological issues

Methodological issues

2 - Design

General modelInitial state Subsequent state

t0 Intervention t1

Effect of intervention, time or other?

Initial state Subsequent state

t0 Intervention t1

Initial state Subsequent state

Methodological issues

Methodological issues

2 - Design

Repeat cross-sectional surveys Cohort RCT (Case-control)

Methodological issues

2 - Design

Cohort Individual behavior change Non-anonymous participation Attrition related to behavior

evaluated Repeat testing, co-intervention Maturation, aging More long-term residents 1-

Repeat C / S Community-wide prevalence Anonymous

Less of a problem Cross-contamination 1-

Methodological issues

Methodological issues

2 - Design

RCT Unbiased allocation Similar distribution of R.F. (known or unknown)

to groups Comparability of groups Validity of statistical tests Feasability, costs Other options to minimize biases (matching,

stratification, …)

Methodological issues

Methodological issues

3 - Measures

3.1 - What? Mortality, morbidity Q O L Risk factors Behaviors Physical and social environments

Proximal impact easier to measure than distal

Methodological issues

3 - Measures

3.2 - How?ReliabilityValidity

Self-reported behaviors Social desirability Pre-testing instruments Objective measures / Gold standard Environmental measures(shelf space, no-smoking

signs, …) Surrogate reports from next of kin Bogus measurements

Methodological issues

4 - Biases

“Distortion in the estimate of effect of an exposure”

due to Selection of subjects How information is collected Confounding

Methodological issues

4 - Biases

Community programs particularly prone to biases

Random allocation is rare Limited # of clusters Important differences between groups

(absolute and secular trends) Multiple co-interventions Blinding is impossible

Methodological issues

Solutions:

Matching:

# of pairs

# measurements

Methodological issues

Methodological issues

5 - Analysis

Effects measured at the individual level but allocation and intervention are at the community level

1- High intra-class correlationsBiased standard error at the individual level ( false -positive results)Standard error must be computed at the community level

• requires N• adjustment for sampling procedures• # data collection

Methodological issues