applicability and transferability role of review authors

The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community WellbeingMelbourne School of Population Health

Applicability and Transferability

Role of review authors

Belinda BurfordRebecca ArmstrongEmma Tavender


Some definitions

• applicability and transferability are two very important and related concepts (often used interchangeably)

• Two sides of the one coin:– Applicability: whether or not it would be feasible to implement this

intervention in a particular setting

• OR “Is it possible?”

– Transferability: whether we can expect to see similar results after implementing this intervention in a particular setting

• OR “Even if it’s possible, will it work?”


Some challenges for complex reviews

• Heterogeneity, heterogeneity, heterogeneity!• Population

– Interactions between intervention, population characteristics and individual preferences/attitudes

– Groups rather than individuals• Intervention

– Multiple components• ‘active’ components?; interactions between components?

– Fidelity/integrity difficult to measure• Comparisons

– Poorly described• Outcomes

– Proxy/surrogate outcomes• Setting

– Important! Often poorly described


Imag

e fr

om:

http:

//w

ww

.har

lem

-sch

ool.c

om/1

0TH

/mat

h_p

df/m

ath.

htm

l

http://www.harlem-school.com/10TH/math_pdf/math.html

http://www.harlem-school.com/10TH/math_pdf/math.html


Organising the information

What are some approaches you have taken in your reviews to help the reader consider applicability/transferability?


What do end-users need from our reviews?

• Who are your end-users?– Differing needs

• What do they need to know1:– Could it work in my jurisdiction?– Will it work?– What would it take to make it work?– Is it worth it?

• Many resources for end-users for assessing applicability of systematic reviews:– SUPPORT tools to support policy-making– Applicability/transferability checklist2

– Many, many more…

1 Lavis et al. Use of research to inform public policymaking Lancet 2004; 364: 1615–21 Lancet 20052 Wang et al 2005. Applicability and transferability of interventions in evidence-based public health. Health Promotion International, 21(1):76-83.


Starting from the end-user perspective: Applicability

End-user question1 What can the review provide?2

Does the political environment of the local society allow for this intervention to be implemented? • Study setting/context

Is there any political barrier to implementing this intervention? • Study setting/contextWould the general public and the targeted (sub) population accept this intervention? Does any aspect of the intervention go against local social norms? Is it ethically acceptable?

• Characteristics of study populationStudy setting/context

Can the contents of the intervention be tailored to suit the local culture? • Intervention components

Are the essential resources for implementing this intervention available in the local setting? • Resources required

Does the target population in the local setting have a sufficient educational level to comprehend the contents of the intervention?

• Characteristics of study population

Which organisation will be responsible for the provision of the intervention in the local setting? • Organisational context

Is there any possible barrier to implementing this intervention due to the structure of that organisation?

• Barriers to implementation identified in studies

Does the provider of the intervention in the local setting have the skill to deliver this intervention? If not will training be available?

• Who delivered the intervention? Was training provided?

1 Questions from: Wang et al 2005. Applicability and transferability of interventions in evidence-based public health. Health Promotion International, 21(1):76-83.2 Table from: Burford et al. Forthcoming


End-user question1 What can the review provide?2

What is the baseline prevalence of the health problem of interest in the local setting? What was the difference in prevalence between the study setting and the local setting?

• Baseline prevalence in study populations

Are the characteristics of the target population comparable between the study setting and the local setting? Is it possible that the characteristics of the target population, such as ethnicity, socioeconomic status, educational level etc will have an impact on the effectiveness of the intervention?

• Characteristics of study population• Outcomes analysed by relevant

socio-demographic factors

Is the capacity to implement the intervention comparable between the study setting in such matters as political environment, social acceptability, resources, organisational structure and the skills of the local providers?

• Study setting/context• Resources required• Intervention deliverers

Starting from the end-user perspective: Transferability

1 Questions from: Wang et al 2005. Applicability and transferability of interventions in evidence-based public health. Health Promotion International, 21(1):76-83.2 Table from: Burford et al. Forthcoming


Organising the information

Place Population• Geographical• Setting• Scale• plus…• consider factors important for

your review…

• Age• Gender• Race/Ethnicity• Income-level• Education• plus…

Intervention Characteristics Intervention Components• Types of roles involved• Interaction between roles• Delivery• Flexibility/tailoring• Component within a broader

system/program• plus…

• Number of components• Type of components• Frequency• plus…


Approaches for dealing with this in your review

• Extrinsic Approach– Give context-dependent guidance for applying

the review findings– 1-2 paragraphs in the discussion– Implementation tables

• Intrinsic Approach– Build into the review planned subgroup

analyses• Good to do both


EXAMPLE: Interventions for preventing obesity in children1

• Data extraction– “implementation factors” based on “practice-

relevant tool”• Extrinsic approach

– Described these across studies in results section and summarised in the discussion

• Intrinsic approach– Meta analyses to explore heterogeneity

1 Waters E, de Silva-Sanigorski A, Hall BJ, Brown T, Campbell KJ, Gao Y, Armstrong R, Prosser L, Summerbell CD. Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews 2011, Issue 12. Art. No.: CD001871. DOI: 10.1002/14651858.CD001871.pub3.


Extrinsic Approach - Results

• Effectiveness– Maintenance / Sustainability of effects– Equity

• PROGRESS-Plus– Harm-adverse/unintended effects

• Implementation– Intervention design and theoretical basis– Process evaluation (including: intensity/dose, acceptability,

barriers/facilitators)– Resources needed (including: who delivered, funding, hours of

implementation, other?)– Strategies to address disadvantage/diversity

Waters E, de Silva-Sanigorski A, Hall BJ, Brown T, Campbell KJ, Gao Y, Armstrong R, Prosser L, Summerbell CD. Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews 2011, Issue 12. Art. No.: CD001871. DOI: 10.1002/14651858.CD001871.pub3.


Extrinsic approach: Discussion

Applicability of the evidenceThis review included studies from high income countries as well as lower-middle- and upper-middle-income countries, with five studies conducted in countries within the latter two groupings (Thailand, Brazil, Chile and Mexico). This means that, while predominantly conducted within high-income settings, the findings from this review may be generalisable to a number of settings. A total of nineteen studies specifically reported incorporating strategies to target socio-economic and/or cultural diversity or disadvantage. One such study was conducted outside of the high-income country setting, in Chile, an upper-middle-income country. Of the remaining eighteen studies, seven studies conducted in the USA were of interventions targeting African American children and their communities and another two studies targeted Native American communities. Other studies targeted participants of low socio-economic status, or were implemented in areas of social disadvantage. By far the most common setting for interventions included in this review were schools (43 studies). Other interventions were (or included) home-based (14 studies), community-based (six studies), or were set in a health service (two studies) or care setting (two studies). Eleven studies incorporated interventions across multiple settings. Most interventions took a combined dietary and physical activity approach to obesity prevention (31 studies). As a single strategy, targeting physical activity alone was more popular (17 studies) than targeting diet alone (seven studies). The predominant theoretical basis for interventions in this review was behaviour change theory. Other theories represented include environmental change strategies, the socio-ecological framework, social learning theory, health promotion theory, transtheoretical models, and youth development and resiliency based approaches. The theoretical basis for interventions was explicitly reported in approximately half of the included studies.



Intrinsic approach: Synthesis

• Meta analyses to explore heterogeneity:– Age group of participants– Setting:

• Education only• Education plus other settings• Non-education

– Intervention type:• PA• Diet• PA/diet combined

– Intervention duration:• <=12 months• > 12 months



The next update?

• Intrinsic– meta-regression: investigate multiple factors simultaneously

(continuous and categorical) for their possible influence on the size of the intervention effect• Caution:

– Generally not considered when <10 studies in meta-analysis– Characteristics should be pre-specified (supported by scientific rationale)

[ours will be post-hoc]– Select a small number of characteristics– Careful interpretation (Ch 9 of Cochrane Handbook)

• Extrinsic– Implementation tables


Using qualitative synthesis to explore heterogeneity of complex interventions

2 reviews:• Qualitative review on HIV/AIDS patients perspectives on adherence• Cochrane Review of intervention trials to improve adherence to therapy in HIV/AIDS patients

Tabulated whether the interventions in the CR corresponded with the patients’ perspectives on how to help them

Candy et al. BMC Medical Research Methodology 2011, 11:124


Exploring temporal study design: Handwashing behaviour in LMIC

Vind

igni

et a

l. Sy

stem

atic

revi

ew: h

andw

ashi

ng b

ehav

iour

in lo

w- t

o-m

iddl

e-in

com

e co

untr

ies:

ou

tcom

e m

easu

res

and

beha

viou

r mai

nten

ance

. Tro

pica

l Med

icin

e an

d In

tern

ation

al H

ealth

. 20

11;1

6(4)

:466

-477


Handwashing: Temporal study design

• Community settings (n=16 studies)– Formative research: average = 4.8 months– Intervention: average = 17.2 months– Evaluation: average = 9.6 months

• Schools (n=4 studies)– Formative research: 1 study– Intervention: average = 4.8 months– Evaluation: average = 5 months

• Health care settings (n=7 studies)– Formative research: 1 study– Intervention: average = 2.5 months– Evaluation: average = 3.2 months

Vindigni et al. Systematic review: handwashing behaviour in low-tomiddle-income countries: outcome measures and behaviour maintenance. Tropical Medicine and International Health. 2011;16(4):466-477


Example from the EPOC Group

Population typeStudies

satisfying EPOC Design Criteria

Comparative studies not satisfying EPOC

Design Criteria

Descriptive studies only Total

Urban 7 12 16 35

Urban Disadvantaged 0 1 6 7

Rural 1 4 14 19

Rural Disadvantaged 1 0 11 12

Total 9 17 47 73

Are there different amounts of relevant evidence?

Eg. Specialist outreach clinics in primary care and rural hospital settings


Specialist Outreach review

applicability and transferability role of review authors

Documents

promotion of mental

public health

health promotion international

population characteristics

wang applicability checklist

public policymaking

support tools

wang et