what is actually measured in process evaluations for ... 3.pdf · (3) explore the relationship...
TRANSCRIPT
What is actually measured in process evaluations for
worksite health promotion programs: a systematic review
Wierenga D, Engbers LH, van Empelen P, Duijts S, Hildebrandt VH, van Mechelen W.
BMC Public Health 2013, 13(1):1190
3
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
42
Chapter 3
Abstract
Background: Numerous worksite health promotion program (WHPPs) have been
implemented the past years to improve employees’ health and lifestyle (i.e., physical
activity, nutrition, smoking, alcohol use and relaxation). Research primarily focused on the
effectiveness of these WHPPs. Whereas process evaluations provide essential information
necessary to improve large scale implementation across other settings. Therefore, this
review aims to: (1) further our understanding of the quality of process evaluations alongside
effect evaluations for WHPPs, (2) identify barriers/facilitators affecting implementation, and
(3) explore the relationship between effectiveness and the implementation process.
Methods: Pubmed, EMBASE, PsycINFO, and Cochrane (controlled trials) were searched
from 2000 to July 2012 for peer-reviewed (randomized) controlled trials published in English
reporting on both the effectiveness and the implementation process of a WHPP focusing
on physical activity, smoking cessation, alcohol use, healthy diet and/or relaxation at work,
targeting employees aged 18-65 years.
Results: Of the 307 effect evaluations identified, twenty-two (7.2%) published an additional
process evaluation and were included in this review. The results showed that eight of those
studies based their process evaluation on a theoretical framework. The methodological
quality of nine process evaluations was good. The most frequently reported process
components were dose delivered and dose received. Over 50 different implementation
barriers/facilitators were identified. The most frequently reported facilitator was strong
management support. Lack of resources was the most frequently reported barrier. Seven
studies examined the link between implementation and effectiveness. In general a positive
association was found between fidelity, dose and the primary outcome of the program.
Conclusions: Process evaluations are not systematically performed alongside effectiveness
studies for WHPPs. The quality of the process evaluations is mostly poor to average,
resulting in a lack of systematically measured barriers/facilitators. The narrow focus on
implementation makes it difficult to explore the relationship between effectiveness and
implementation. Furthermore, the operationalization of process components varied
between studies, indicating a need for consensus about defining and operationalizing
process components.
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
43
Systematic review of implementation studies
3
Introduction
Employees with unhealthy lifestyle behaviors and overweight or obese employees are less
productive at work, show a decreased work ability and take more sick days compared to
employees with a healthy lifestyle [1-4]. An unhealthy lifestyle can be characterized by one
or more of the following behaviors; low physical activity levels, an unhealthy diet, smoking,
frequent alcohol use and poor levels of relaxation (i.e. mental health and vitality). Because
employed adults spend about half of their workday waking hours at their workplace, the
worksite may be an effective setting to increase employees health and productivity [5-8].
Therefore, in the past decade, numerous worksite health promotion programs (WHPPs)
have been conducted to improve employees’ health and lifestyle. However, not all of these
programs were successful [5-7].
In order to change employees’ health and lifestyle, programs need to work as intended
(and therefore avoid theory failure). Additionally, they must be effectively transferred
from research to practice and be maintained over time [9]. Programs often fail to reach
potential participants adequately due to a lack of adoption, communication, or program
sustainability which could lead to low participation levels (in other words, program failure)
[10]. Furthermore, research starts to recognize the importance of evaluating different
implementation outcomes (such as recruitment, dose delivered, dose received, fidelity,
satisfaction, and maintenance) and the contextual factors that hinder or facilitate the
implementation of a WHPP [11-14]. These contextual factors are related to characteristics
of the context, organization, implementer, program and participants [11, 12, 15]. Since
WHPPs are often comprehensive interventions with multiple components, it is difficult to
determine the overall level of implementation and investigate which specific intervention
components have been successful [14, 16].
A review by Durlak and Dupre (2008) showed that higher levels of implementation
improve program outcomes, suggesting that there should be an adequate focus on the
implementation process [13]. However, most research focuses primarily on measuring the
effects of a WHPP, with an occasional process evaluation performed after implementation.
However, systematic process evaluations can produce valuable insights into the interpretation
of the (lack of) effects of an intervention by identifying successful and unsuccessful
program components, thereby allowing researchers to optimize their program [16-19].
Furthermore, process evaluations can help to identify barriers and/or facilitators influencing
the implementation process, while taking into account the different actor levels at which
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
44
Chapter 3
these factors play a role [16, 18]. These are valuable outcomes that can be used to improve
program implementation in the future and across other settings. Hence, effect evaluations
should be accompanied by systematic and real-time process evaluations [20, 21].
Murta et al. (2007) looked at the quality of process evaluations accompanying controlled
trials studying individual based stress management interventions. They showed that process
evaluations are often incomplete and not systematically conducted, meaning that they lacked
a theoretical framework and were not planned prior to implementation [22]. However, the
quality of the process evaluations that were included in this review was insufficient to make
sound conclusions in terms of best predictors for successful interventions. Additionally,
Murta’s review only focused on evaluations of occupational stress management
interventions published between 1977 to 2003. So their conclusion cannot directly be
transferred to recent WHPP focusing on other lifestyle behaviors. In their review, Murta also
concluded that the framework of Steckler and Linnan proved to be a useful tool to conduct
process evaluations. But in this framework little attention has been given to the great variety
of contextual factors that can either hinder or facilitate the implementation process. We
therefore recently proposed a framework for a systematic and comprehensive process
evaluation based on several theoretical frameworks to gain insight into the implementation
process (figure 1) [11, 12, 14, 15, 23]. The four main aspects of this framework relate to
determinants of implementation that may influence the implementation process, and the
implementation process itself (i.e. adoption, implementation and continuation). These four
main aspects are operationalized using a combination of the framework of Steckler and
Linnan for process evaluations and the RE-AIM framework [14, 23]. So, in order to gain
insight in the implementation process, eight different process components at three different
actor levels (macro-level: organization and management; meso-level: implementer; micro-
level: participant) need to be evaluated using a mixed methods approach [11]. Of these eight
components, six components focus on implementation (reach, recruitment, dose delivered,
dose received, fidelity, and satisfaction), one component (maintenance) on continuation and
the eighth component (context) refers to the determinants of implementation. Since many
implementation determinants can be identified, context is further defined by categorizing
the barriers and/or facilitators into five main categories: 1) characteristics of the socio-
political context, 2) characteristics of the organization, 3) characteristics of the implementer,
4) characteristics of the intervention program, 5) characteristics of the participant [11, 12,
15].
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
45
Systematic review of implementation studies
3
Some reviews have focused on process evaluations and relevant implementation
barriers and/or facilitators [10, 12, 13, 22, 24, 25]. However, these reviews focused on
domains other than WHPPs [12, 13, 24, 25], limited the identification of barriers and/or
facilitators to those affecting WHPP participation levels [10], or focused only on process
evaluations accompanying stress management interventions [22]. To our knowledge, there
has been no systematic review that has examined process evaluations for worksite health
promotion programs targeting lifestyle change among employees. The aim of this review was
therefore to: (1) further our understanding of the quality of process evaluations alongside
effect evaluations for worksite health promotion programs (WHPPs), (2) identify barriers/
facilitators affecting implementation, and (3) explore the relationship between effectiveness
and the implementation process.
Characteristics of the socio-political context
Characteristics of the organization
Characteristics of the innovation
Characteristics of the adopting person/user
Adoption
Implementation
Continuation
Implementation strategy
Innovation process Innovation determinants
Figure 1. Theoretical framework
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
46
Chapter 3
Methods
Literature search and study selection
For this systematic review, peer-reviewed studies were eligible for inclusion when they
reported an effect evaluation as well as a process evaluation for a worksite health promotion
intervention focusing on stimulating a healthy lifestyle (physical activity, nutrition, smoking,
alcohol use and relaxation) published in English between 2000 and July 2012. The literature
search was conducted in two steps.
Step 1: a literature search was in the online databases Pubmed, EMBASE, PsycINFO, and
the Cochrane Central Register of Controlled trials for peer-reviewed WHPP effect evaluations
published in English from 2000 to July 2012. Searches included the following combination of
keywords: (Randomized controlled trial OR controlled trial) AND (worker* OR employee* OR
worksite OR work environment) AND (worksite health promotion OR lifestyle intervention)
AND (absenteeism OR sickness absence OR body mass index OR lifestyle OR health behavior
OR cholesterol level). To ensure no studies were overlooked this search string was repeated
in each database with the addition of the following keywords for each lifestyle behavior
separately: ‘physical activity’, ‘smoking’, ‘alcohol use’, ‘healthy diet’ and ‘relaxation’. Since
we were only interested in studies focusing on change in actual behavior rather than change
in attitude, social norm or self-efficacy keywords were added on outcome measures related
to actual behavior change. All the keywords were specified for each database using Mesh
or Thesaurus terms. The complete search strategy for each database and lifestyle behavior
are presented in appendices 1. The inclusion criteria for the first stage of the selection
process were: (1) a randomized controlled (RCT) or controlled trial (CT) published in English
between 2000 and July 2012, (2) an evaluation of the effects of worksite health promotion
interventions focusing on physical activity, smoking cessation, alcohol use, healthy diet and/
or relaxation at work, and (3) targeting employees aged 18-65 years. With respect to the
second inclusion criteria it should be noted that studies investigating interventions that
primarily aim to promote a healthy lifestyle as well as interventions that primarily aim to
prevent musculoskeletal disorders of which the intervention contains one of the lifestyle
components mentioned above were included. The first author (DW) performed an initial
selection based on the titles and abstracts of all papers reporting on effect evaluation.
The abstracts of the effect evaluations were presented to the fourth author (SD), who was
blinded for authors, affiliations, journal and year of publication. In a consensus meeting
between both authors (DW and SD), a final selection of effects evaluations was made. When
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
47
Systematic review of implementation studies
3
an abstract contained insufficient information or when the authors disagreed, the full paper
was retrieved and read. When disagreement persisted, the second author (LE) was asked to
decide about eligibility.
Step 2: DW checked whether the selected effect evaluations were accompanied by
a peer-reviewed process evaluation. The inclusion criteria for the second stage of the
selection procedure were: (1) published in English between 2000 and July 2012, and (2)
reporting on implementation/process outcomes. DW first contacted the corresponding
authors by e-mail to ask whether the effect evaluation had been accompanied by a paper
on implementation/process outcomes. If the corresponding author did not respond or
could not provide the requested information, DW first checked the full text paper of the
effect evaluation for references to a related process evaluation (in the reference list and/or
text). If nothing was found, DW searched all four databases using the title of the study (if
known), trial registration number and the names of all authors for an additional formative,
implementation or process evaluation. If based on title and abstract it was unclear whether
the paper included information on the implementation process of the intervention, the full
text paper was retrieved and read.
The full papers were retrieved for all eligible effect evaluation papers paired with a
published process evaluation paper. These complete studies were independently assessed
for eligibility by the first three authors (DW, LE and PE) before being included in the review.
During this final round, LE and PE were blinded for authors, affiliations, journal and year of
publication.
All references used in eligible effect evaluations and process evaluations were checked
by the first author (DW) for other relevant publications that might have been missed in the
electronic search (‘snow ball’ procedure).
Methodological quality assessment
In order to answer the first research question, a methodological quality assessment
was performed. The methodological quality of all papers included in the review was
independently assessed by the first three authors (DW, LE and PE) using a checklist (Table 1).
Disagreements between reviewers were discussed and resolved during consensus meetings.
The criteria for the assessment of the included effect evaluations were based on the
methodological guidelines for systematic reviews developed by the Cochrane Back Review
Group [26]. These guidelines were developed for RCT’s studying low back pain. Some
Cochrane criteria were therefore omitted or adapted to fit the studies included in this
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
48
Chapter 3
review. Other reviews focusing on worksite interventions have used an adapted version
of this guideline [5, 27]. The final criteria list consisted of three main categories: internal
validity (n=8), descriptive criteria (n=5) and analysis (n=3). Items were scored as positive (+),
unsatisfactory (+/-) or negative (-). If an item was not applicable, this was stated. The quality
of studies was considered to be ‘above average’ if the overall validity score (V) was above
50%, and a score above 75% was considered to represent relatively good quality [26, 28].
The overall validity score (V) was based on the eight internal validity criteria (V1-V8). When
items were not applicable (N/A), they were not included in the percentages.
Given the absence of a standardized assessment form, we defined our own criteria for
the methodological quality assessment of process evaluations on the basis of our proposed
conceptual framework, shortly explained in the introduction and published elsewhere [11].
These criteria are described in detail in table 1. The final criteria list included nine items.
Items were scored as positive (+), unsatisfactory (+/-) or negative (-). If an item was not
applicable, this was stated. The quality of studies was considered to be ‘above average’ if the
overall validity score (T) was above 50%, and a score above 75% was considered to represent
relatively good quality. This validity score (T) was based on the nine internal validity criteria
(T1-T9). Non-applicable items (N/A) were not included in the percentages.
Table 1. Criteria list for the methodological quality assessment of the studies and definitions of the criteria
Effect evaluationsInternal validity / study designV1 Randomization procedure Positive if a random (unpredictable) assignment procedure
sequence of subjects to the study groups was used and if there was a clear description of the procedure and adequate performance of the randomization
V2 Similarity of companies Positive if they controlled for variability in included companiesV3 Similarity of study groups Positive if the study groups were similar at the beginning of the
studyV4 Dropout Positive if the percentage of dropouts during the study period did
not exceed 20% for short-term follow-up (≤ 3 months) or 30% for long-term follow-up (> 3 months) and adequately described
V5 Timing of outcome measurement
Positive if timing of outcome assessment was identical for intervention and control groups and for all important outcomes assessments.
V6 Blinding Positive if the person performing the assessments was blinded to the group assignment
V7 Co-interventions Positive if co-interventions were avoided or comparable.V8 Outcome Positive when data on outcome was selected with standardized
methods of acceptable quality
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
49
Systematic review of implementation studies
3
Descriptive criteriaD1 Eligibility criteria (in- and
exclusion criteria)Positive if in- and exclusion criteria of participants were specified
D2 Baseline characteristics Positive if an adequate description of the study groups was given for demographic variables: gender, age, type of work, hours a week working, education level, baseline main outcome measures
D3 Company characteristics Positive if an adequate description of the included companies was given (type of industry, organizational characteristics)
D4 Intervention Positive if an adequate description was given of the interventions(s): number of intervention aspects, type of interventions, frequency of sessions, intensity of intervention(s)
D5 Follow-up Positive if a follow-up of 6 months or longer was described.AnalysisA1 Sample size Positive if an adequate sample size calculation was describedA2 Confounders Positive if the analysis controlled for potential confoundersA3 Intention to treat Positive if the intervention and control subjects were analysed
according to the group belonging to their initial assignment, irrespective of non-compliance and co-interventions.
Process evaluationsT1 Model used for evaluation Positive if a theoretical framework for the evaluation was used and
adequately described.T2 Level of evaluation Positive if implementation was evaluated on 2 or more levels (i.e.
macro, meso, micro)T3 Definition of outcome
measure (process components)
Positive if the definition of the outcome measures (process variables and barriers and/or facilitators) were accurately described
T4 Reported process variables a. Positive if four or more process evaluation variables are evaluated (in process evaluation)
b. Positive if barriers or facilitators on 1 or more levels are presented
T5 Data collection Positive if 2 or more techniques for data collection were used (triangulation).
T6 Timing of data collection Positive if measurements of barriers and/or facilitators were performed pre-, during and after implementation.
T7 Quantitative outcome measures
Positive if data on quantitative outcome was selected with methods of acceptable quality and data on multiple process components was measured.
T8 Qualitative data a. Positive if study design for qualitative data (theoretical framework, participant selection, setting, data collection) were adequately described
b. Positive if qualitative data was analyzed by two researchers.
T9 Outcome related to implementation of intervention
Positive if outcomes (barriers and/or facilitators) are related to the quality of implementation
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
50
Chapter 3
Data extraction
The first author (DW) extracted the data using a predefined format. For the effect
evaluations, information was extracted about study design, company type, study
population, intervention content, and intervention goal. In addition, although not the main
aim of this review, the proportion of affected primary outcomes was extracted to provide
some information about the effectiveness of the program. For the process evaluations
information was extracted about data collection, methods, timing of measurements, level
of evaluation, type of evaluation, linking effect to implementation outcomes, model for
process evaluation, reported process components and reported barriers and/or facilitators.
During data extraction, the process components reported in the studies were classified
under the eight process components (recruitment, reach, dose delivered, dose received,
fidelity, satisfaction and maintenance, and context) on the basis of the definitions proposed
by Wierenga et al. [11, 14]. If a study used another framework/model with additional
components, these were classified separately. The barriers and/or facilitators found in
the studies were assigned to five categories based on the review by Fleuren et al. (2004):
socio-political context, organization, implementer, program and participants [12, 15]. When
something was unclear, advice was asked from the second (LE) and third author (PE). The
complete data extraction form can be obtained from the corresponding author.
Results
Study selection
The initial computerized search identified 9112 articles (see flow chart in figure 2). Initial
screening of the titles and abstracts produced 704 potentially relevant articles looking at
all lifestyle behaviors. 8408 articles were excluded because they were not (R)CT related to a
WHPP with the aim of changing employee lifestyles. We excluded 397 of the 704 potentially
relevant articles on the grounds that articles were found in multiple databases (Pubmed,
EMBASE, PsycINFO, Cochrane Central Register of Controlled Trials), or in the different
searches based on lifestyle behavior. Interventions not related to any of the five health
behaviors or not targeting employees were also excluded.
A total of 307 full-text articles were retrieved and checked for related process evaluation
articles, resulting in the exclusion of another 277 articles. The main reasons for exclusion
were: no additional published process evaluation for the same study, not targeting
employees, or no lifestyle component. Thirty studies reporting on both the effects and the
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
51
Systematic review of implementation studies
3
implementation process of the same WHPP were checked for eligibility. After reading the
papers, eight studies were excluded because, contrary to expectations, they did not perform
a process evaluation. No articles were added after snowballing the references in the
selected studies. Finally, twenty-two studies met the selection criteria and were included in
this review (table 2; Appendices 2).
49
because, contrary to expectations, they did not perform a process evaluation. No articles were
added after snowballing the references in the selected studies. Finally, twenty-two studies met the
selection criteria and were included in this review (table 2; Appendices 2).
Figure 2. Flowchart of study selection process
Intervention and study characteristics
The second column in table 2 presents the general study characteristics in order to give an overview
of the type of studies included. Half of the studies were conducted in the USA [29-39], and the other
half in Europe [40-50], mostly in the Netherlands [40, 41, 45, 47-49]. Nineteen (86%) studies used a
randomized controlled trial design [29, 30, 33-36, 38-50] and three (14%) a controlled trial design
[31, 32, 37]. Fourteen (63%) studies reported on interventions targeting physical activity (PA) [29-34,
40-44, 48-50]. Two of these studies focused mainly on preventing musculoskeletal disorders [40, 44],
but the intervention included a PA component and so these studies were included in the review.
Twelve studies (55%) reported on the effect and process evaluation of interventions targeting
healthy nutrition [29, 32-37, 41, 45, 47-49]. Furthermore, five (23%) studies reported interventions
for smoking and tobacco use [36-39, 41], and two (9%) studies focused on relaxation [46, 48]. None
of the twenty-two studies targeted alcohol use.
Methodological quality assessment
9112 of records identified through database searching
0 of additional records identified through other sources
9112 records screened
307 of records after duplicates removed and full-text articles
assessed for eligibility
8549 records excluded
22 of studies included in the qualitative synthesis
285 of full-text articles excluded:
- no additional process evaluation - intervention not related to any of the lifestyle behaviours - target population not employees
Figure 2. Flowchart of study selection process
Intervention and study characteristics
The second column in table 2 presents the general study characteristics in order to give an
overview of the type of studies included. Half of the studies were conducted in the USA
[29-39], and the other half in Europe [40-50], mostly in the Netherlands [40, 41, 45, 47-
49]. Nineteen (86%) studies used a randomized controlled trial design [29, 30, 33-36, 38-
50] and three (14%) a controlled trial design [31, 32, 37]. Fourteen (63%) studies reported
on interventions targeting physical activity (PA) [29-34, 40-44, 48-50]. Two of these studies
focused mainly on preventing musculoskeletal disorders [40, 44], but the intervention
included a PA component and so these studies were included in the review. Twelve studies
(55%) reported on the effect and process evaluation of interventions targeting healthy
nutrition [29, 32-37, 41, 45, 47-49]. Furthermore, five (23%) studies reported interventions
for smoking and tobacco use [36-39, 41], and two (9%) studies focused on relaxation [46,
48]. None of the twenty-two studies targeted alcohol use.
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
52
Chapter 3
Tabl
e 2.
Cha
ract
eris
tics
of th
e st
udie
s in
clud
ed in
this
revi
ewG
ener
al s
tudy
info
rmati
onPr
oces
s ev
alua
tion
info
rmati
on
Refe
renc
eD
esig
n eff
ect e
valu
ation
: Ran
dom
ized
co
ntro
lled
tria
l (RC
T) o
r Co
ntro
lled
tria
l (CT
)Ty
pe o
f com
pany
St
udy
popu
lati
onIn
terv
enti
on c
onte
ntIn
terv
enti
on g
oal
Prop
orti
on o
f aff
ecte
d pr
imar
y ou
tcom
es
Dat
a co
llecti
on m
etho
d of
pro
cess
ev
alua
tion
(PE)
Tim
ing
of p
roce
ss e
valu
ation
(PE)
Proc
ess
eval
uati
on (P
E) e
valu
ation
leve
ls:
Mac
ro, m
eso,
mic
roTy
pe o
f pro
cess
eva
luati
on (P
E)Eff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Des
crib
ed o
r N
ot d
escr
ibed
Mod
el u
sed
for
eval
uati
onM
easu
red
proc
ess
com
pone
nts:
Defi
nitio
n us
ed in
the
stud
y, -
=not
des
crib
ed
Dri
esse
n et
al.
(201
0,20
11)
[29-
31]
Net
herl
ands
Des
ign
effec
t ev
alua
tion
: RCT
Type
of c
ompa
ny: U
nive
rsity
, rai
lway
tr
ansp
orta
tion,
air
line,
ste
elSt
udy
popu
lati
on: B
lue
and
whi
te c
olla
r In
terv
enti
on c
onte
nt:
Wor
ksite
par
ticip
ator
y er
gono
mic
s pr
ogra
mIn
terv
enti
on g
oal:
Prev
ent l
ow b
ack
pain
and
ne
ck p
ain
Prop
orti
on o
f aff
ecte
d pr
imar
y ou
tcom
es:
2 of
14
Dat
a co
llecti
on m
etho
d PE
:M
ixed
met
hods
Tim
ing
PE: D
urin
g an
d po
st-in
terv
entio
nPE
eva
luati
on le
vels
: Mes
o, M
icro
Type
of P
E: E
valu
ation
of i
nter
venti
ons
and
impl
emen
tatio
n pr
oces
sEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Not
des
crib
ed
Mod
el u
sed
for
eval
uati
on:
Adj
uste
d ve
rsio
n of
Lin
nan
and
Stec
kler
Mea
sure
d pr
oces
s co
mpo
nent
s:-
Recr
uitm
ent:
inte
rven
tion
depa
rtm
ents
, wor
king
gro
ups
form
ed, w
orki
ng
grou
p m
embe
rs fo
r tr
aini
ng, r
espo
nse
rate
bas
elin
e qu
estio
nnai
re-
Reac
h: w
orks
ite v
isits
by
trai
ner,
atten
danc
e ra
tes
impl
emen
ters
, att
enda
nce
rate
s of
impl
emen
ters
in tr
aini
ng-
Dos
e de
liver
ed: p
erce
ived
impl
emen
tatio
n of
the
inte
rven
tion
acco
rdin
g to
im
plem
ente
rs-
Dos
e re
ceiv
ed: p
erce
ived
impl
emen
tatio
n of
the
inte
rven
tion
of th
e em
ploy
ees
- Fi
delit
y: E
xten
t to
whi
ch th
e in
terv
entio
n w
as d
eliv
ered
as
inte
nded
- Sa
tisfa
ction
: of i
mpl
emen
ters
abo
ut im
plem
enta
tion
proc
ess
and
inte
rven
tion,
and
of e
mpl
oyee
s ab
out t
he in
terv
entio
n-
Cont
ext:
Per
ceiv
ed b
arri
ers
and/
or fa
cilit
ator
s to
impl
emen
tatio
n of
in
terv
entio
n-
Mai
nten
ance
: -
Gro
enev
eld
et
al. (
2010
, 201
1)[3
2-34
]
Net
herl
ands
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Con
stru
ction
St
udy
popu
lati
on: B
lue
colla
r at
ris
k fo
r CV
DIn
terv
enti
on c
onte
nt:
Indi
vidu
ally
bas
ed li
fest
yle
inte
rven
tion
by
mea
ns o
f moti
vatio
nal i
nter
view
ing
Inte
rven
tion
goa
l: Ch
ange
phys
ical
acti
vity
, die
t and
sm
okin
g be
havi
orPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 8o
f 15a
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mes
o, M
icro
Type
of P
E: E
valu
ation
of i
nter
venti
ons
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: D
escr
ibed
for
body
wei
ght
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- D
ose
deliv
ered
: Num
ber
of s
essi
ons
perf
orm
ed b
y im
plem
ente
r an
d nu
mbe
r of
item
s di
scus
sed
duri
ng s
essi
on-
Dos
e re
ceiv
ed: N
umbe
r of
ses
sion
s att
ende
d by
par
ticip
ants
- Fi
delit
y: A
dher
ence
to in
terv
entio
n pr
otoc
ol b
y im
plem
ente
rs a
nd th
e qu
ality
of t
he in
terv
entio
n de
liver
y-
Satis
facti
on: P
artic
ipan
ts o
pini
on o
n th
e co
mpe
tenc
e an
d sk
ills
of th
e im
plem
ente
r. Im
plem
ente
rs o
pini
on o
n th
eir
own
com
pete
nce
and
skill
s on
de
liver
y of
inte
rven
tion
and
over
all d
ifficu
lty o
f del
iver
ing
inte
rven
tion
- Re
crui
tmen
t, R
each
, Mai
nten
ance
, Con
text
: -
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
53
Systematic review of implementation studies
3
Fren
ch e
t al.
(201
0) [3
5, 3
6]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Bus
gar
ages
Stud
y po
pula
tion
: Blu
e co
llar
Inte
rven
tion
con
tent
: Env
ironm
enta
l wor
ksite
ob
esity
pre
venti
on in
terv
entio
nIn
terv
enti
on g
oal:
Impr
ove
heal
thfu
l foo
d ch
oice
s an
phy
sica
l acti
vity
leve
lsPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 4
of 2
0
Dat
a co
llecti
on m
etho
d PE
: Qua
ntita
tive
Tim
ing
PE: P
ost-
inte
rven
tion
PE e
valu
ation
leve
ls: M
icro
Ty
pe o
f PE:
Eva
luati
on o
f int
erve
ntion
sEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Not
des
crib
ed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- D
ose
deliv
ered
: Am
ount
of i
nter
venti
ons
mad
e av
aila
ble
duri
ng th
e in
terv
entio
n pe
riod
- D
ose
rece
ived
: Par
ticip
ation
rate
s of
em
ploy
ees
in in
terv
entio
n-
Recr
uitm
ent,
Rea
ch, F
idel
ity, S
atisf
actio
n, M
aint
enan
ce, C
onte
xt: -
Dis
hman
& W
ilson
et
al.
(200
9, 2
010)
[3
7, 3
8]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Ret
ail
Stud
y po
pula
tion
: Whi
te c
olla
rIn
terv
enti
on c
onte
nt:
Soci
al-e
colo
gic
inte
rven
tion
by p
erso
nal a
nd
team
goa
l-setti
ngIn
terv
enti
on g
oal:
Incr
ease
leis
ure-
time
phys
ical
acti
vity
Pr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 3
of 3
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Dur
ing
and
post
-inte
rven
tion
PE e
valu
ation
leve
ls: M
eso,
Mic
roTy
pe o
f PE:
Eva
luati
on o
f int
erve
ntion
s an
d im
plem
enta
tion
proc
ess
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: D
escr
ibed
for p
hysi
cal a
ctivi
ty le
vels
Mod
el u
sed
for
eval
uati
on:
Dur
lak
& D
upre
Mea
sure
d pr
oces
s co
mpo
nent
s:-
Dos
e de
liver
ed: A
mou
nt o
f int
erve
ntion
del
iver
ed-
Dos
e re
ceiv
ed: A
mou
nt o
f int
erve
ntion
rece
ived
by
parti
cipa
nts
- Fi
delit
y: D
egre
e to
whi
ch th
e in
terv
entio
n w
as im
plem
ente
d as
pla
nned
- Co
ntex
t: Im
plem
enta
tion
barr
iers
- Re
crui
tmen
t, R
each
, Sati
sfac
tion,
Mai
nten
ance
: -
Yap
et a
l. (2
009,
20
10) [
39-4
1]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: C
TTy
pe o
f com
pany
: Man
ufac
turi
ng p
lant
Stud
y po
pula
tion
: Blu
e co
llar
Inte
rven
tion
con
tent
:Ta
ilore
d e-
mai
l int
erve
ntion
Inte
rven
tion
goa
l: In
crea
se in
tenti
onal
ph
ysic
al a
ctivi
tyPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 2
of 2
Dat
a co
llecti
on m
etho
d PE
: Qua
litati
ve
Tim
ing
PE: P
ost-
inte
rven
tion
PE e
valu
ation
leve
ls:
Mic
ro
Type
of P
E: E
valu
ation
of i
nter
venti
ons
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: N
ot d
escr
ibed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- D
ose
rece
ived
: Use
of i
nter
venti
on-
Satis
facti
on: S
atisf
actio
n w
ith th
e in
terv
entio
n co
mpo
nent
s-
Recr
uitm
ent,
Rea
ch, D
ose
deliv
ered
, Fid
elity
, Mai
nten
ance
, Con
text
: -
Gils
on e
t al.
(200
7,
2008
) [42
, 43]
Uni
tes
King
dom
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Uni
vers
itySt
udy
popu
lati
on: W
hite
col
lar
Inte
rven
tion
con
tent
:Ro
ute
and
task
-bas
ed w
alki
ng In
terv
enti
on
goal
: Im
prov
e w
ork
day
step
cou
nts
and
heal
th
stat
usPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 1o
f 4
Dat
a co
llecti
on m
etho
d PE
:Q
ualit
ative
Ti
min
g PE
: Pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mic
roTy
pe o
f PE:
Eva
luati
on o
f int
erve
ntion
s an
d im
plem
enta
tion
proc
ess
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: N
ot d
escr
ibed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- Re
crui
tmen
t: S
trat
egie
s to
app
roac
h an
d att
ract
em
ploy
ees
- Sa
tisfa
ction
: Exp
erie
nces
dur
ing
inte
rven
tion
- Co
ntex
t: B
enefi
ts/p
ositi
ves
and
prob
lem
s/ba
rrie
rs a
ssoc
iate
d w
ith
parti
cipa
tion
in in
terv
entio
n-
Reac
h, D
ose
deliv
ered
, Dos
e re
ceiv
ed, F
idel
ity, M
aint
enan
ce: -
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
54
Chapter 3
Goe
tzel
, DeJ
oy,
Wils
on e
t al.
(200
7, 2
009,
201
0,
2011
) [44
-48]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: C
TTy
pe o
f com
pany
: Man
ufac
turi
ng
Stud
y po
pula
tion
: Blu
e an
d w
hite
col
lar
Inte
rven
tion
con
tent
: Env
ironm
enta
l wei
ght
man
agem
ent i
nter
venti
on
Inte
rven
tion
goa
l: Im
prov
e ph
ysic
al a
ctivi
ty
and
heal
thy
eatin
gPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 6
of 8
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Dur
ing
and
post
-inte
rven
tion
PE e
valu
ation
leve
ls: M
acro
, Mes
o, M
icro
Type
of P
E: E
valu
ation
of i
nter
venti
ons
and
impl
emen
tatio
n pr
oces
sEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Not
des
crib
ed
Mod
el u
sed
for
eval
uati
on:
Inte
grati
ve m
odel
Mea
sure
d pr
oces
s co
mpo
nent
s:-
Cont
ext:
Man
agem
ent s
uppo
rt, j
ob/t
ask
fact
ors
(phy
sica
l and
psy
chol
ogic
al
dem
ands
of s
peci
fic jo
bs),
envi
ronm
enta
l fac
tors
(phy
sica
l wor
k en
viro
nmen
t and
the
soci
al-o
rgan
izati
onal
env
ironm
ent
- Re
crui
tmen
t, R
each
, Dos
e de
liver
ed, D
ose
rece
ived
, Fid
elity
, Sati
sfac
tion,
M
aint
enan
ce: -
Lem
on, E
stab
rook
et
al.
(201
0, 2
011)
[4
9, 5
0]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Hos
pita
lSt
udy
popu
lati
on: W
hite
col
lar
Inte
rven
tion
:Ec
olog
ical
and
env
ironm
enta
l mul
tilev
el
inte
rven
tion
Inte
rven
tion
goa
l: Pr
even
t wei
ght g
ain
by
targ
eting
hea
lthy
eatin
g an
d ph
ysic
al a
ctivi
tyPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 0
of 1
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Pre
-, du
ring
, and
pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mac
ro, M
eso,
Mic
roTy
pe o
f PE:
Eva
luati
on o
f int
erve
ntion
s an
d im
plem
enta
tion
proc
ess
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: D
escr
ibed
for
Body
Mas
s In
dex
Mod
el u
sed
for
eval
uati
on:
RE-A
IMM
easu
red
proc
ess
com
pone
nts:
- Re
crui
tmen
t: A
mou
nt o
f rec
ruite
d se
tting
s fo
r pa
rtici
patio
n an
d nu
mbe
r th
at a
gree
d to
take
par
t, re
pres
enta
tives
of p
artic
ipati
ng s
etting
s (A
dopti
on)
- Re
ach:
Par
ticip
ation
rate
s, re
pres
enta
tiven
ess
of p
artic
ipan
ts,
char
acte
risti
cs o
f par
ticip
ants
and
non
-par
ticip
ants
- D
ose
deliv
ered
: Im
plem
ente
d in
terv
entio
n co
mpo
nent
s-
Dos
e re
ceiv
ed: U
se o
f int
erve
ntion
com
pone
nts
- Fi
delit
y: E
xten
t to
whi
ch th
e pr
ogra
m w
as d
eliv
ered
as
inte
nded
at p
rogr
am
and
indi
vidu
al le
vel (
Impl
emen
tatio
n)-
Mai
nten
ance
: Ext
ent t
o w
hich
the
beha
vior
or
polic
y of
inte
rest
mai
ntai
ned
over
the
long
term
at i
ndiv
idua
l and
insti
tutio
nal l
evel
- Co
ntex
t: M
anag
emen
t sup
port
, job
/tas
k fa
ctor
s (p
hysi
cal a
nd p
sych
olog
ical
de
man
ds o
f spe
cific
jobs
), en
viro
nmen
tal f
acto
rs (p
hysi
cal w
ork
envi
ronm
ent a
nd th
e so
cial
-org
aniz
ation
al e
nviro
nmen
t-
Effec
tiven
ess:
Beh
avio
ral o
utco
mes
and
oth
er o
utco
mes
incl
udin
g im
pact
on
qua
lity
of li
fe-
Satis
facti
on: -
And
erse
n et
al.
(201
1) [5
1-53
]
Den
mar
k
Des
ign
effec
t ev
alua
tion
: RCT
Type
of c
ompa
ny: O
ffice
Stud
y po
pula
tion
: Whi
te c
olla
rIn
terv
enti
on c
onte
nt:
Prog
ress
ive
resi
stan
ce tr
aini
ng (e
xerc
ise
prog
ram
)In
terv
enti
on g
oal:
Relie
ve n
eck/
shou
lder
pai
nPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 4
of 6
Dat
a co
llecti
on m
etho
d PE
:Q
uanti
tativ
eTi
min
g PE
: Pre
-inte
rven
tion
PE e
valu
ation
leve
ls: M
icro
Type
of P
E: E
valu
ation
of i
nter
venti
ons
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: N
ot d
escr
ibed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- Co
ntex
t: P
rogn
ostic
fact
ors
for
adhe
renc
e to
inte
rven
tion
- Re
crui
tmen
t, R
each
, Dos
e de
liver
ed, D
ose
rece
ived
, Fid
elity
, Sati
sfac
tion,
M
aint
enan
ce: -
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
55
Systematic review of implementation studies
3
Hau
kka,
Peh
kone
n et
al.
(200
9, 2
010)
[5
4, 5
5]
Finl
and
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Mun
icip
al k
itche
nsSt
udy
popu
lati
on: B
lue
colla
r In
terv
enti
on c
onte
nt:
Parti
cipa
tory
erg
onom
ics
inte
rven
tion
Inte
rven
tion
goa
l: D
ecre
ase
phys
ical
and
m
enta
l wor
kloa
dPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 0
of 8
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Pre
-, du
ring
, and
pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mes
o, M
icro
Type
of P
E: E
valu
ation
of i
nter
venti
ons
and
impl
emen
tatio
n pr
oces
sEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Not
des
crib
ed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- D
ose
deliv
ered
: Tim
e an
d vi
sits
mad
e to
wor
ksite
- D
ose
rece
ived
: Par
ticip
ation
rate
s -
Satis
facti
on: G
ener
al o
pini
ons
rega
rdin
g pr
oces
s, e
xper
ienc
es o
f the
in
terv
entio
n,
- Co
ntex
t: B
enefi
ts o
r di
fficu
lties
exp
erie
nces
of t
he p
roje
ct, f
acili
tatin
g fa
ctor
s fo
r im
plem
enta
tion,
sup
port
and
tim
e ne
eded
for
deve
lopm
ent,
or
gani
zatio
n of
wor
k ta
sks
duri
ng in
terv
entio
n-
Recr
uitm
ent,
Rea
ch, F
idel
ity, M
aint
enan
ce:-
Sore
nsen
, Hun
t (2
005,
200
7) [5
6,
57]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Man
ufac
turi
ngSt
udy
popu
lati
on: B
lue
colla
rIn
terv
enti
on c
onte
nt: W
orks
ite in
terv
entio
n ta
rgeti
ng fr
uit a
nd v
eget
able
con
sum
ption
, re
d m
eat c
onsu
mpti
on, m
ultiv
itam
in u
se a
nd
phys
ical
acti
vity
Inte
rven
tion
goa
l: Ca
ncer
pre
venti
onPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 2
of 4
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mes
o, M
icro
Type
of P
E:Ev
alua
tion
of in
terv
entio
ns a
nd
impl
emen
tatio
n pr
oces
sEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Not
des
crib
ed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- Re
ach:
Pro
gram
aw
aren
ess
- D
ose
deliv
ered
: Del
iver
ed in
terv
entio
ns-
Dos
e re
ceiv
ed: P
rogr
am p
artic
ipati
on
- Co
ntex
t: M
anag
emen
t con
tact
s re
gard
ing
orga
niza
tiona
l cha
nges
, fac
tors
th
at in
fluen
ces
impl
emen
tatio
n-
Recr
uitm
ent,
Fid
elity
, Sati
sfac
tion,
Mai
nten
ance
: -
Bere
sfor
d et
al.
(200
0, 2
001,
201
0)
[58-
60]
Uni
ted
Stat
es
Des
ign
effec
t ev
alua
tion
: RCT
Type
of c
ompa
ny: H
ospi
tals
, edu
catio
nal,
gove
rnm
enta
l, pr
ofes
sion
al a
genc
ies,
co
nstr
uctio
n, m
anuf
actu
ring
, fina
ncia
l, re
tail,
w
hole
sale
, ser
vice
Stud
y po
pula
tion
: Blu
e an
d w
hite
col
lar
Inte
rven
tion
con
tent
: Env
ironm
enta
l and
in
divi
dual
str
ateg
ies
Inte
rven
tion
goa
l: In
crea
se fr
uit a
nd v
eget
able
in
take
Pr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 2
of 2
b
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mes
o, M
icro
Type
of P
E: E
valu
ation
of i
nter
venti
ons
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: D
escr
ibed
for
frui
t and
veg
etab
le
inta
ke
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- Re
ach:
See
ing
or re
adin
g th
e in
terv
entio
n m
ater
ials
(use
of i
nter
venti
on)
- D
ose
deliv
ered
: Doc
umen
tatio
n of
inte
rven
tion
activ
ities
by
impl
emen
ters
- D
ose
rece
ived
: Ave
rage
inte
rven
tion
expo
sure
per
em
ploy
ee-
Cont
ext:
Wor
ksite
cha
ract
eris
tics
- Re
crui
tmen
t, F
idel
ity, S
atisf
actio
n, M
aint
enan
ce: -
Sore
nsen
, Hun
t et
al. (
2007
, 201
0)
[61,
62]
Uni
ted
Stat
es
Des
ign
effec
t ev
alua
tion
: RCT
Type
of c
ompa
ny: C
onst
ructi
on
Stud
y po
pula
tion
: Blu
e co
llar
Inte
rven
tion
con
tent
: Tai
lore
d te
leph
one-
deliv
ered
and
mai
led
inte
rven
tion
Inte
rven
tion
goa
l: Pr
omot
e sm
okin
g ce
ssati
on
and
incr
ease
frui
t and
veg
etab
le c
onsu
mpti
onPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 3
of 3
Dat
a co
llecti
on m
etho
d PE
:Q
ualit
ative
Ti
min
g PE
: Pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mic
roTy
pe o
f PE:
Eval
uatio
n of
inte
rven
tions
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: N
ot d
escr
ibed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- D
ose
deliv
ered
: Num
ber
of in
terv
entio
ns d
eliv
ered
- D
ose
rece
ived
: Rec
eivi
ng in
terv
entio
n an
d m
ater
ials
- Fi
delit
y: E
xten
t to
whi
ch th
e in
terv
entio
n w
as im
plem
ente
d-
Satis
facti
on: P
artic
ipan
t sati
sfac
tion
with
inte
rven
tion
- Re
crui
tmen
t, R
each
, Mai
nten
ance
, Con
text
: -
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
56
Chapter 3
Stee
nhui
s et
al.
(200
4) [6
3, 6
4]
Net
herl
ands
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Gov
ernm
enta
l and
larg
e co
mpa
nies
Stud
y po
pula
tion
: Whi
te c
olla
r In
terv
enti
on:
Envi
ronm
enta
l int
erve
ntion
s in
clud
ing
labe
ling
of h
ealth
y fo
ods,
food
sup
ply
prog
ram
and
ed
ucati
onal
pro
gram
abo
ut h
ealth
y fo
od
Inte
rven
tion
goa
l: In
crea
se a
vaila
bilit
y an
d kn
owle
dge
of h
ealth
y fo
ods
Prop
orti
on o
f aff
ecte
d pr
imar
y ou
tcom
es:
1 of
3
Dat
a co
llecti
on m
etho
d PE
:Q
ualit
ative
Tim
ing
PE: P
ost-
inte
rven
tion
PE e
valu
ation
leve
ls: M
eso
Type
of P
E:Ev
alua
tion
of in
terv
entio
ns a
nd
impl
emen
tatio
n pr
oces
sEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Not
des
crib
ed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- Sa
tisfa
ction
: man
ager
s op
inio
ns o
n in
terv
entio
n -
Cont
ext:
diffi
culti
es w
ith im
plem
enta
tion,
per
ceiv
ed b
enefi
ts o
f int
erve
ntion
- Re
crui
tmen
t, R
each
, Dos
e de
liver
ed, D
ose
rece
ived
, Fid
elity
, Mai
nten
ance
: -
Sore
nsen
, Q
uinti
liani
et a
l. (2
010)
[65,
66]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: C
TTy
pe o
f com
pany
: Tru
ckin
g te
rmin
als
Stud
y po
pula
tion
: Blu
e co
llar
Inte
rven
tion
: Tel
epho
ne a
nd p
rint
-del
iver
ed
inte
rven
tion
Inte
rven
tion
goa
l: Pr
omot
e to
bacc
o us
e ce
ssati
on a
nd im
prov
e w
eigh
t man
agem
ent b
y he
alth
y nu
triti
onPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 1
of 2
Dat
a co
llecti
on m
etho
d PE
:Q
uanti
tativ
eTi
min
g PE
: Pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mes
o, M
icro
Type
of P
E: E
valu
ation
of i
nter
venti
ons
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: D
escr
ibed
for
smok
ing
cess
ation
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- D
ose
deliv
ered
: Num
ber
of d
eliv
ered
inte
rven
tions
- D
ose
rece
ived
: Per
cepti
on o
f rec
eive
d in
terv
entio
n co
mpo
nent
s an
d m
ater
ials
(Eng
agem
ent i
n in
terv
entio
n co
mpo
nent
s)-
Satis
facti
on: H
elpf
ulne
ss o
f int
erve
ntion
- Re
crui
tmen
t, R
each
, Fid
elity
, Mai
nten
ance
, Con
text
: -
Stod
dard
, Hun
t et
al. (
2003
, 200
5)
[67,
68]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Ret
ail
Stud
y po
pula
tion
: Whi
te c
olla
r (t
eens
)In
terv
enti
on c
onte
nt: B
ehav
iora
l tob
acco
co
ntro
l int
erve
ntion
Inte
rven
tion
goa
l: D
ecre
ase
smok
ing
prev
alen
cePr
opor
tion
of a
ffec
ted
mai
n ou
tcom
es: 0
of 2
Dat
a co
llecti
on m
etho
d PE
:Q
uanti
tativ
eTi
min
g PE
: Dur
ing
and
post
-inte
rven
tion
PE e
valu
ation
leve
ls: M
eso,
Mic
roTy
pe o
f PE:
Eval
uatio
n of
inte
rven
tions
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: N
ot d
escr
ibed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- D
ose
deliv
ered
: Am
ount
of i
nter
venti
on d
eliv
ered
- D
ose
rece
ived
: am
ount
of i
nter
venti
on re
ceiv
ed, p
artic
ipati
on ra
tes,
m
otiva
tion
for
parti
cipa
tion
in in
terv
entio
n, p
rogr
am a
war
enes
s-
Recr
uitm
ent,
Rea
ch, F
idel
ity, S
atisf
actio
n, M
aint
enan
ce, C
onte
xt: -
Volp
p, K
im e
t al
. (20
09, 2
011)
[6
9, 7
0]
Uni
ted
Stat
es
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Mul
tinati
onal
St
udy
popu
lati
on: N
ot d
escr
ibed
Inte
rven
tion
con
tent
: Offe
ring
fina
ncia
l in
centi
ves
for
smok
ing
cess
ation
Inte
rven
tion
goa
l: Im
prov
e sm
okin
g ce
ssati
on
rate
sPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 1
of 1
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Pos
t-in
terv
entio
n PE
eva
luati
on le
vels
: Mic
roTy
pe o
f PE:
Eva
luati
on o
f int
erve
ntion
sEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Des
crib
ed fo
r sm
okin
g ce
ssati
on
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- Re
ach:
Pro
gram
aw
aren
ess
- D
ose
deliv
ered
: Am
ount
of i
nter
venti
on d
eliv
ered
- D
ose
rece
ived
: am
ount
of i
nter
venti
on re
ceiv
ed, p
artic
ipati
on ra
tes,
m
otiva
tion
for
parti
cipa
tion
in in
terv
entio
n-
Recr
uitm
ent,
Fid
elity
, Sati
sfac
tion,
Mai
nten
ance
, Con
text
: -
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
57
Systematic review of implementation studies
3
Has
son
et a
l. (2
005,
201
0) [7
1,
72]
Swed
en
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Inf
orm
ation
tech
nolo
gy,
med
iaSt
udy
popu
lati
on: W
hite
col
lar
Inte
rven
tion
con
tent
: Web
-bas
ed h
ealth
pr
omoti
on a
nd s
tres
s m
anag
emen
t tra
inin
gIn
terv
enti
on g
oal:
Dec
reas
e un
wan
ted
stre
ss
and
prom
ote
heal
th a
nd re
cove
ryPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 5
of 5
Dat
a co
llecti
on m
etho
d PE
:Q
ualit
ative
Tim
ing
PE: D
urin
gPE
eva
luati
on le
vels
: Mic
roTy
pe o
f PE:
Eva
luati
on o
f int
erve
ntion
sEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Not
des
crib
ed
Mod
el u
sed
for
eval
uati
on:
Not
des
crib
edM
easu
red
proc
ess
com
pone
nts:
- D
ose
rece
ived
: Fre
quen
cy o
f rep
lyin
g to
the
scre
enin
g to
ol-
Cont
ext:
Fac
tors
that
det
erm
ine
use
of p
rogr
am-
Recr
uitm
ent,
Rea
ch, D
ose
deliv
ered
, Fid
elity
, Sati
sfac
tion,
Mai
nten
ance
: -
Verm
eer
et a
l. (2
011)
[73,
74]
Net
herl
ands
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Wor
ksite
caf
eter
ias
of
hosp
itals
, uni
vers
ities
, pol
ice
depa
rtm
ents
, co
mpa
nies
Stud
y po
pula
tion
: Whi
te c
olla
rIn
terv
enti
on c
onte
nt:
Envi
ronm
enta
l int
erve
ntion
intr
oduc
ing
a sm
all
hot m
eal i
n ad
ditio
n to
the
exis
ting
size
and
a
prop
ortio
nal p
rici
ng s
trat
egy
in c
afet
eria
sIn
terv
enti
on g
oal:
Stim
ulat
e w
orke
rs to
re
plac
e th
eir
larg
er m
eal w
ith a
sm
alle
r m
eal
Prop
orti
on o
f aff
ecte
d pr
imar
y ou
tcom
es:
0 of
2
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Pre
-, du
ring
and
pos
t-in
terv
entio
nPE
eva
luati
on le
vels
: Mes
o, M
icro
Type
of P
E:Ev
alua
tion
of in
terv
entio
nsEff
ect o
f im
plem
enta
tion
on
outc
ome
mea
sure
: Not
des
crib
ed
Mod
el u
sed
for
eval
uati
on:
Bara
now
ski &
Sta
bles
and
Rog
ers
Mea
sure
d pr
oces
s co
mpo
nent
s:-
Recr
uitm
ent:
Att
racti
ng a
genc
ies,
impl
emen
ters
, or
pote
ntial
par
ticip
ants
fo
r co
rres
pond
ing
part
s of
the
prog
ram
- D
ose
rece
ived
: Ext
ent t
o w
hich
par
ticip
ants
vie
w o
r re
ad th
e m
ater
ials
that
re
ach
them
- Fi
delit
y: E
xten
t to
whi
ch th
e pr
ogra
m is
impl
emen
ted
as d
esig
ned
- M
aint
enan
ce: K
eepi
ng p
artic
ipan
ts in
volv
ed in
the
prog
ram
mati
c an
d da
ta c
olle
ction
, and
ext
ent t
o w
hich
par
ticip
ants
con
tinue
to d
o an
y of
the
activ
ities
- Co
ntex
t: A
spec
ts o
f the
env
ironm
ent o
f an
inte
rven
tion
- Co
ntam
inati
on: e
xten
t to
whi
ch p
artic
ipan
ts re
ceiv
ed in
terv
entio
ns fr
om
outs
ide
the
prog
ram
and
the
exte
nt to
whi
ch th
e co
ntro
l gro
up re
ceiv
es th
e tr
eatm
ent
- Re
sour
ces:
Mat
eria
ls o
r ch
arac
teri
stics
of a
genc
ies,
impl
emen
ters
, or
parti
cipa
nts
nece
ssar
y to
att
ain
proj
ect g
oals
- Re
ach,
Dos
e de
liver
ed, S
atisf
actio
n:-
Stri
jk e
t al.
(201
1,
2012
) [75
-77]
Net
herl
ands
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Aca
dem
ic h
ospi
tal
Stud
y po
pula
tion
: Whi
te c
olla
rIn
terv
enti
on c
onte
nt: V
italit
y in
terv
entio
n co
nsis
ting
of w
eekl
y yo
ga a
nd w
orko
ut
sess
ion,
wee
kly
unsu
perv
ised
aer
obic
exe
rcis
e,
free
frui
t dur
ing
sess
ions
and
thre
e vi
sits
of
pers
onal
vita
lity
coac
hIn
terv
enti
on g
oal:
Impr
ove
lifes
tyle
beh
avio
rs
Prop
orti
on o
f aff
ecte
d pr
imar
y ou
tcom
es:
3 of
10
Dat
a co
llecti
on m
etho
d PE
:Q
uanti
tativ
eTi
min
g PE
: Dur
ing
and
post
-inte
rven
tion
PE e
valu
ation
leve
ls: M
eso,
Mic
roTy
pe o
f PE:
Eval
uatio
n of
inte
rven
tions
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: N
ot d
escr
ibed
Mod
el u
sed
for
eval
uati
on:
Stec
kler
and
Lin
nan
Mea
sure
d pr
oces
s co
mpo
nent
s:-
Reac
h: p
ropo
rtion
of w
orke
rs p
artic
ipati
ng in
inte
rven
tion
- D
ose
deliv
ered
: num
ber
of in
terv
entio
n co
mpo
nent
s de
liver
ed b
y im
plem
ente
rs-
Dos
e re
ceiv
ed: E
xten
t to
whi
ch th
e w
orke
rs w
ere
enga
ged
in th
e in
terv
entio
n (a
tten
danc
e ra
tes)
- Fi
delit
y: E
xten
t to
whi
ch th
e in
terv
entio
n w
as im
plem
ente
d as
pla
nned
- Sa
tisfa
ction
: Wor
kers
atti
tude
tow
ards
the
inte
rven
tion
- Co
ntex
t: O
rgan
izati
onal
and
env
ironm
enta
l fac
tors
con
cern
ing
the
inte
rven
tion
- Re
crui
tmen
t, M
aint
enan
ce:-
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
58
Chapter 3
Verw
eij e
t al.
(201
1, 2
012)
[7
8-80
]
Net
herl
ands
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Uni
vers
ity, b
ank,
nur
sing
ho
me,
spi
ce fa
ctor
y, p
acka
ging
com
pany
, m
unic
ipal
ity, c
onsu
mer
goo
ds c
ompa
nySt
udy
popu
lati
on: B
lue
and
whi
te c
olla
r In
terv
enti
on c
onte
nt: O
ccup
ation
al h
ealth
gu
idel
ine
Inte
rven
tion
goa
l: pr
even
ting
wei
ght g
ain
by
incr
easi
ng P
A, d
ecre
asin
g se
dent
ary
beha
vior
, in
crea
sing
frui
t con
sum
ption
or
redu
cing
en
ergy
inta
ke d
eriv
ed fr
om s
nack
sPr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 2
of 6
Dat
a co
llecti
on m
etho
d PE
: Mix
ed m
etho
dsTi
min
g PE
: Dur
ing
and
post
-inte
rven
tion
PE e
valu
ation
leve
ls: M
eso,
Mic
roTy
pe o
f PE:
Eval
uatio
n of
inte
rven
tions
and
im
plem
enta
tion
proc
ess
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: D
escr
ibed
for
body
wei
ght a
nd
wai
st c
ircum
fere
nce
Mod
el u
sed
for
eval
uati
on:
Stec
kler
and
Lin
nan
Mea
sure
d pr
oces
s co
mpo
nent
s:-
Recr
uitm
ent:
Sou
rces
and
pro
cedu
res
used
to a
ppro
ach
and
attra
ct
pote
ntial
par
ticip
ants
, the
num
ber
of ra
ndom
ized
OPs
, and
the
num
ber
of
empl
oyee
s th
at fi
lled
out t
he b
asel
ine
ques
tionn
aire
- Re
ach:
Num
ber
of e
mpl
oyee
s w
ho a
tten
ded
the
coun
selin
g se
ssio
ns,
reas
on fo
r m
isse
d co
unse
ling
sess
ions
and
the
perc
enta
ge o
f dro
p-ou
ts
incl
udin
g re
ason
- D
ose
deliv
ered
: The
num
ber
of in
terv
entio
n m
ater
ials
or
com
pone
nts
actu
ally
del
iver
ed b
y O
Ps, a
nd th
e du
ratio
n an
d fo
rm o
f the
cou
nsel
ing
sess
ions
- D
ose
rece
ived
: The
ext
ent t
o w
hich
par
ticip
ants
use
mat
eria
ls, r
esou
rces
, or
tech
niqu
es re
com
men
ded
by p
rogr
am-
Fide
lity:
The
ext
ent t
o w
hich
the
inte
rven
tion
was
del
iver
ed a
s pl
anne
d: if
O
Ps a
dher
ed to
the
guid
elin
e an
d ad
equa
tely
per
form
ed b
ehav
ior
chan
ge
coun
selin
g-
Satis
facti
on: P
artic
ipan
ts a
ttitu
des
tow
ard
the
cont
ent,
use
and
lim
itatio
ns
of th
e gu
idel
ine
(OP)
, or
tow
ard
the
inte
rven
tion,
mat
eria
ls a
nd O
P (e
mpl
oyee
)-
Cont
ext:
Org
aniz
ation
al c
hara
cter
istic
s th
at a
ffect
inte
rven
tion
impl
emen
tatio
n, in
clud
ing
phys
ical
, soc
ial,
politi
cal,
and
econ
omic
feat
ures
- M
aint
enan
ce: -
Jorg
ense
n et
al
.(20
11, 2
012)
[8
1-83
]
Den
mar
k
Des
ign
effec
t eva
luati
on: R
CTTy
pe o
f com
pany
: Hos
pita
ls, c
lean
ing
com
pani
es, l
arge
bus
ines
s w
ith in
-hou
se
clea
ning
ser
vice
sSt
udy
popu
lati
on: B
lue
colla
r In
terv
enti
on c
onte
nt: P
hysi
cal c
oord
inati
on
trai
ning
Inte
rven
tion
goa
l: Im
prov
e ph
ysic
al
coor
dina
tion
trai
ning
or
cogn
itive
beh
avio
ral
reso
urce
s Pr
opor
tion
of a
ffec
ted
prim
ary
outc
omes
: 3
of 6
Dat
a co
llecti
on m
etho
d PE
:Q
uanti
tativ
eTi
min
g PE
: Dur
ing
inte
rven
tion
PE e
valu
ation
leve
ls: M
eso
Type
of P
E: E
valu
ation
of i
nter
venti
ons
Effec
t of i
mpl
emen
tati
on o
n ou
tcom
e m
easu
re: N
ot d
escr
ibed
Mod
el u
sed
for
eval
uati
on:
Ada
pted
ver
sion
Ste
ckle
r &
Lin
nan
Mea
sure
d pr
oces
s co
mpo
nent
s:-
Recr
uitm
ent:
Pro
cedu
res
used
to a
ppro
ach
and
attra
ct p
artic
ipan
ts-
Dos
e de
liver
ed: i
nter
venti
on d
eliv
ered
- D
ose
rece
ived
: adh
eren
ce to
inte
rven
tion
(att
enda
nce
rate
s)-
Fide
lity:
Qua
lity
of in
terv
entio
n de
liver
y-
Cont
ext:
Una
ntici
pate
d ev
ents
at t
he w
ork
plac
e-
Reac
h, S
atisf
actio
n, M
aint
enan
ce: -
RCT:
Ran
dom
ized
con
trol
led
tria
l; CT
: Con
trol
led
tria
l; PE
: pro
cess
eva
luati
on; M
acro
leve
l: co
mpa
ny/m
anag
emen
t; M
eso
leve
l: Im
plem
ente
r; M
icro
: em
ploy
eea 3
out
com
es w
ere
sign
ifica
nt a
t sho
rt a
nd lo
ng te
rm, 3
out
com
es o
nly
at s
hort
term
, 2 o
utco
mes
onl
y at
long
term
b bo
th o
utco
mes
wer
e si
gnifi
cant
ly im
prov
ed a
t sho
rt te
rm, h
owev
er th
ey w
ere
not s
tatis
tical
ly s
igni
fican
t at l
ong
term
but
stil
l im
prov
ed in
favo
r of i
nter
venti
on
grou
p
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
59
Systematic review of implementation studies
3
Methodological quality assessment
Table 3 shows the methodological quality scores of the included studies. Initial disagreement
between the reviewers about 50 of the 448 effect evaluation criteria (11%) and 46 of the
207 process evaluation criteria (22%) were mainly attributable to the interpretation of the
items. The differences concentrated primarily on drop-out (V4), company characteristics
(D3), intention to treat (A3), data collection for process evaluation (T5), and the timing of
data collection for process evaluation (T6). Full agreement was reached after two discussion
sessions, thereby completing the scoring process.
The methodological quality scores for the effect evaluations of the studies ranged from
37.5% to 100%. Fifteen (68%) effect evaluations were considered to be ‘above average’
(>50%) [29-33, 37, 39-41, 43, 44, 46, 48, 48-50]. Only six of these effect evaluations (27%)
were relatively good (quality score >75%) [31, 40, 41, 43, 48, 49]. The quality scores for
the process evaluations ranged from 12.5% to 100%. Eight process evaluations (36%) were
relatively good (quality score >75%) [51-58].
Only three (14%) studies scored relatively good (quality score >75%) on both the process
as well as the effect evaluation with respect to their methodological quality [51, 57, 58]. In
four (18.2%) studies the methodological quality of both the effect as well as the process
evaluation was poor (quality score 50% or less) [36, 38, 42, 45]. No relation was observed
between the quality of effect evaluations and process evaluations.
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
60
Chapter 3Ta
ble
3. O
vera
ll sc
ores
of t
he m
etho
dolo
gica
l qua
lity
of th
e in
clud
ed s
tudi
es
Firs
t aut
hor
(Yea
r)M
etho
dolo
gica
l qua
lity
asse
ssm
ent c
rite
rion
– e
ffec
t ev
alua
tion
sM
etho
dolo
gica
l qua
lity
asse
ssm
ent c
rite
rion
– p
roce
ss
eval
uati
onV
1V
2V
3V
4V
5V
6V
7V
8Va
lidit
y sc
ore
(V) i
n %
T1T2
T3T4
T5T6
T7T8
T9Va
lidit
y sc
ore
(T) i
n %
Dri
esse
n et
al.
(201
0, 2
011)
**
++
+/-
+/-
++
++
87,5
++
++
+/-
++
++
94,4
Gro
enev
eld
et a
l. (2
010,
201
1) *
+N
/A+
+/-
++/
-N
/A+
83,3
-+
++
-+
+N
/A+
75Fr
ench
et a
l. (2
010)
*+/
-+
+/-
N/A
+N
/A+/
-+
75-
N/A
--
-+/
-+/
-N
/A-
14,3
Dis
hman
& W
ilson
et a
l. (2
009,
201
0)+
+/-
+/-
-+
-+/
-+
56,2
5+
++
++
++
++
100
Yap
et a
l. (2
009,
201
0) *
N/A
N/A
++
+N
/AN
/A+
100
--
--
--
N/A
+-
12,5
Gils
on e
t al.
(200
7, 2
008)
+N
/A+
-+/
--
-+
50-
-+
--
+/-
N/A
+/-
-25
Goe
tzel
, DeJ
oy, W
ilson
et a
l. (2
007,
200
9-20
11) a
***
N/A
+/-
+/-
-+
N/A
+/-
+75
++
++/
-+
++
+-
83,3
Lem
on, E
stab
rook
et a
l. (2
010-
2011
)+/
-+
+/-
++
--
+62
,5+
++
++
++
++
100
And
erse
n et
al.
(201
1)+
N/A
++
++
++
100
--
+/-
--
-+
N/A
+31
,25
Hau
kka,
Peh
kone
n et
al.
(200
9,20
10)
+-
++
+N
/A-
+71
,4-
++
++/
-+
++
+83
,3So
rens
en, H
unt e
t al.
(200
5, 2
007)
-+
++/
-+/
--
-+
50-
++
+-
++
+-
66,6
7Be
resf
ord
et a
l. (2
000,
200
1, 2
010)
*-
++
-+
--
+50
-+
++/
-+/
-+
+N
/A+
75So
rens
en, H
unt e
t al.
(200
7, 2
010)
-N
/A+
++
--
+/-
50-
-+/
--
-+/
-+/
-N
/A+/
-25
Stee
nhui
s et
al.
(200
4)-
+/-
+/-
-+
--
+37
,5-
--
+/-
--
N/A
-+/
-12
,5So
rens
on, Q
uinti
liani
et a
l. (2
010)
_+
++
+/-
--
+56
,25
-+
+-
-+
+N
/A+
62,5
Stod
dard
, Hun
t et a
l. (2
003,
200
5)-
++
N/A
+-
-+/
-50
-+
+/-
+/-
-+
+N
/A-
50Vo
lpp,
Kim
et a
l. (2
009,
201
1)
+-
++
+-
-+
62,5
+/-
-+/
--
+/-
++
++
61,1
Has
son
et a
l. (2
005,
201
0)+
-+
++
--
+62
,5+
-+
-+
++
N/A
+/-
68,7
5Ve
rmee
r et
al.
(201
1)+/
--
-+/
-+/
--
-+
31,2
5+
++
++
++
+-
88,8
9St
rijk
et a
l. (2
011,
201
2)+
++
+/-
++
++
93,7
5+
++
++/
-+
+N
/A-
81,2
5Ve
rwei
j et a
l. (2
011,
201
2)+
-+
++
++
+87
,5+
++
++
++
++
100
Jorg
ense
n et
al.
(201
1, 2
012)
+-
+-
++/
-+
+68
,75
+-
++
+/-
-+
N/A
-56
,25
a Con
trol
led
tria
l*
two
effec
t arti
cles
sco
red
toge
ther
** tw
o pr
oces
s ev
alua
tions
sco
red
toge
ther
***
thre
e eff
ect a
rticl
es s
core
d to
geth
er a
nd 2
impl
emen
tatio
n ar
ticle
s sc
ored
toge
ther
N/A
, not
app
licab
le. +
, pos
itive
. +/-
, not
suffi
cien
t; -,
neg
ative
. All
tria
ls a
re r
ando
miz
ed t
rial
s ex
cept
for
the
tria
ls in
dica
ted
with
a s
uper
scri
pt ‘a
’. Th
e m
axim
um s
core
for
met
hodo
logi
cal q
ualit
y of
effe
ct e
valu
ation
s is
8 (b
ased
on
valid
ity s
ectio
n V1
-V8)
. The
max
imum
sco
re fo
r m
etho
dolo
gica
l qua
lity
of p
roce
ss e
valu
ation
is 9
(bas
ed o
n se
ction
T1-
T9).
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
61
Systematic review of implementation studies
3
Tabl
e 3.
Ove
rall
scor
es o
f the
met
hodo
logi
cal q
ualit
y of
the
incl
uded
stu
dies
Firs
t au
thor
(Yea
r)M
etho
dolo
gica
l qua
lity
asse
ssm
ent c
rite
rion
– e
ffec
t ev
alua
tion
sM
etho
dolo
gica
l qua
lity
asse
ssm
ent c
rite
rion
– p
roce
ss
eval
uati
onV
1V
2V
3V
4V
5V
6V
7V
8Va
lidit
y sc
ore
(V) i
n %
T1T2
T3T4
T5T6
T7T8
T9Va
lidit
y sc
ore
(T) i
n %
Dri
esse
n et
al.
(201
0, 2
011)
**
++
+/-
+/-
++
++
87,5
++
++
+/-
++
++
94,4
Gro
enev
eld
et a
l. (2
010,
201
1) *
+N
/A+
+/-
++/
-N
/A+
83,3
-+
++
-+
+N
/A+
75Fr
ench
et a
l. (2
010)
*+/
-+
+/-
N/A
+N
/A+/
-+
75-
N/A
--
-+/
-+/
-N
/A-
14,3
Dis
hman
& W
ilson
et a
l. (2
009,
201
0)+
+/-
+/-
-+
-+/
-+
56,2
5+
++
++
++
++
100
Yap
et a
l. (2
009,
201
0) *
N/A
N/A
++
+N
/AN
/A+
100
--
--
--
N/A
+-
12,5
Gils
on e
t al.
(200
7, 2
008)
+N
/A+
-+/
--
-+
50-
-+
--
+/-
N/A
+/-
-25
Goe
tzel
, DeJ
oy, W
ilson
et a
l. (2
007,
200
9-20
11) a
***
N/A
+/-
+/-
-+
N/A
+/-
+75
++
++/
-+
++
+-
83,3
Lem
on, E
stab
rook
et a
l. (2
010-
2011
)+/
-+
+/-
++
--
+62
,5+
++
++
++
++
100
And
erse
n et
al.
(201
1)+
N/A
++
++
++
100
--
+/-
--
-+
N/A
+31
,25
Hau
kka,
Peh
kone
n et
al.
(200
9,20
10)
+-
++
+N
/A-
+71
,4-
++
++/
-+
++
+83
,3So
rens
en, H
unt e
t al.
(200
5, 2
007)
-+
++/
-+/
--
-+
50-
++
+-
++
+-
66,6
7Be
resf
ord
et a
l. (2
000,
200
1, 2
010)
*-
++
-+
--
+50
-+
++/
-+/
-+
+N
/A+
75So
rens
en, H
unt e
t al.
(200
7, 2
010)
-N
/A+
++
--
+/-
50-
-+/
--
-+/
-+/
-N
/A+/
-25
Stee
nhui
s et
al.
(200
4)-
+/-
+/-
-+
--
+37
,5-
--
+/-
--
N/A
-+/
-12
,5So
rens
on, Q
uinti
liani
et a
l. (2
010)
_+
++
+/-
--
+56
,25
-+
+-
-+
+N
/A+
62,5
Stod
dard
, Hun
t et a
l. (2
003,
200
5)-
++
N/A
+-
-+/
-50
-+
+/-
+/-
-+
+N
/A-
50Vo
lpp,
Kim
et a
l. (2
009,
201
1)
+-
++
+-
-+
62,5
+/-
-+/
--
+/-
++
++
61,1
Has
son
et a
l. (2
005,
201
0)+
-+
++
--
+62
,5+
-+
-+
++
N/A
+/-
68,7
5Ve
rmee
r et
al.
(201
1)+/
--
-+/
-+/
--
-+
31,2
5+
++
++
++
+-
88,8
9St
rijk
et a
l. (2
011,
201
2)+
++
+/-
++
++
93,7
5+
++
++/
-+
+N
/A-
81,2
5Ve
rwei
j et a
l. (2
011,
201
2)+
-+
++
++
+87
,5+
++
++
++
++
100
Jorg
ense
n et
al.
(201
1, 2
012)
+-
+-
++/
-+
+68
,75
+-
++
+/-
-+
N/A
-56
,25
a Con
trol
led
tria
l*
two
effec
t arti
cles
sco
red
toge
ther
** tw
o pr
oces
s ev
alua
tions
sco
red
toge
ther
***
thre
e eff
ect a
rticl
es s
core
d to
geth
er a
nd 2
impl
emen
tatio
n ar
ticle
s sc
ored
toge
ther
N/A
, not
app
licab
le. +
, pos
itive
. +/-
, not
suffi
cien
t; -,
neg
ative
. All
tria
ls a
re r
ando
miz
ed t
rial
s ex
cept
for
the
tria
ls in
dica
ted
with
a s
uper
scri
pt ‘a
’. Th
e m
axim
um s
core
for
met
hodo
logi
cal q
ualit
y of
effe
ct e
valu
ation
s is
8 (b
ased
on
valid
ity s
ectio
n V1
-V8)
. The
max
imum
sco
re fo
r m
etho
dolo
gica
l qua
lity
of p
roce
ss e
valu
ation
is 9
(bas
ed o
n se
ction
T1-
T9).
Process evaluation design
The third and fourth columns in table 2 list the characteristics of the included process
evaluations. Most studies (50%) used a mixed methods approach (qualitative and
quantitative) to look at the implementation process [51-56, 58-62]. With regard to the
number of measurements, ten (45%) studies conducted a post-process evaluation only [29,
55, 59, 61-67]. Three (14%) studies collected process evaluation information at three points
in time: before, during and after the intervention [54, 56, 60].
Only two (9%) studies evaluated the implementation process at all three actor levels
(the macro-, meso- and micro-levels, see above) [53, 60]. The majority (50%) of the studies
evaluated the implementation process at both the meso- and micro-levels [51, 52, 54-59,
61, 67, 68]. A minority of the studies (n=8; 36%) reported using a theoretical framework to
guide their process evaluation [47, 51-53, 57, 58, 60, 69]. Of these eight studies, four used
the framework of Steckler and Linnan (or an adapted version of that framework) [51, 57,
58, 69], and the remaining four studies applied either the integrative model [53], the RE-
AIM model [60], a model based on Durlak and Dupre[52], or a model based on Baranowski
and Stables, in combination with Rogers’ framework for the diffusion of innovations [56].
It should be noted that the integrative model is originally a model for the development of
workplace environmental interventions and not specifically for process evaluations [53].
Reporting of process evaluation components
The fourth column of table 2 lists the predetermined process components that were
measured. Seven of these components (reach, recruitment, dose delivered, dose received,
fidelity, satisfaction and maintenance) measure the degree of implementation. The eighth
component, context, maps the barriers and/or facilitators that affect implementation. The
average number of process evaluation components found in the studies was 3.9, ranging
from 1 to 8. Ten (45%) studies evaluated fewer than four process components [29, 53, 62-
64, 66-68, 70, 71]. Six (27.5%) studies reported five or more process components [51, 56-58,
60, 69]. The studies focused mainly on dose received (82%) [29, 51, 52, 54-63, 65, 67-69,
71], dose delivered (68%) [29, 51, 52, 54, 55, 57-62, 65, 67, 68], and context (68%) [52, 54-
57, 60, 61, 64, 66, 69-74]. Fewer than half of the studies (41%) looked at fidelity [51, 52,
56-60, 65]. Nine (41%) studies measured satisfaction [51, 54, 57-59, 63-66]. Relatively few
studies reported on reach (n=7; 32%) [51, 55, 57, 58, 60, 61], or recruitment (n=5; 23%) [56,
58, 60, 64, 69]. In addition, two (9%) studies looked at maintenance [56, 60].
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
62
Chapter 3
Implementation barriers and/or facilitators
Next, we identified barriers and/or facilitators which can affect implementation. However,
the barriers and/or facilitators were often not systematically measured by means of
questionnaires or interviews and often only observed and documented on the basis of the
researchers’ experience. Nevertheless, fifty-four different barriers and/or facilitators were
obtained from nineteen (86%) studies (table 4) [52, 54-57, 59-64, 66, 68-74]. On average, the
studies described 6.5 [range 1-22] barriers and/or facilitators. None of the studies reported
a context analysis prior to implementation. The majority of barriers and facilitators in the
five categories as a whole were reported only in a single study.
Fewest barriers and/or facilitators were reported in the category ‘characteristics of the
socio-political context’. Only two (9%) studies reported a barrier or facilitator in this category
(these studies are listed in table 4). The largest number of barriers and/or facilitators (n=17)
was found in the category ‘characteristics of the organization’. Management support was
found most frequently, having been mentioned in eight studies (36%). Strong management
support was described as a facilitator, whereas unbalanced or lack of management support
was found to be a barrier. The definition of management support varied widely. Another
frequently reported barrier was the lack of financial, staffing or material resources (n=5
studies; 26%). In the category ‘characteristics of the intervention’ fifteen barriers and/
or facilitators were identified, with ‘compatibility of intervention with the organization’
being the most commonly reported facilitator in eight studies (42%). The facilitator
‘relative advantage of the intervention’ was reported in five (26%) studies. In the category
‘characteristics of the implementer’ twelve barriers and/or facilitators were identified; the
factor ‘available time of the implementer’ was most commonly reported (n=5; 26%). In the
category ‘characteristics of the user’, eight barriers and/or facilitators were identified with
‘time constraints’ was frequently reported as a barrier to participation. Furthermore, we
found that high work demands and a high workload were also a barrier to participation.
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
63
Systematic review of implementation studies
3
Table 4. Reported barriers (B) and/or facilitators (F) in the studies included in this review
Main categoriesDescription of the determinants for implementation B/F
Characteristics of the socio-political context
1. Compatibility of program with societal developments (attention for health in society) [74]
F
2. Competitive business environment [53] BCharacteristics of the organization
3. Organizational reorganization: reorganization due to take over by another company [68]
B
4. Lack of resources: financial, personnel, material (e.g., equipment, facilities) resources or lack of space or facilities [54, 57, 66, 72, 74]
B
5. Organizational culture:(a) Senior leaders emphasized the need to implement the intervention keeping the organizational culture in mind [53] (b) Intervention did not fit the organizational culture [55, 72] (c) The organizational culture emphasized goal setting and tracks progress towards achieving goals [53] (d) Worksite culture supported social interaction among workers and between workers and managers [55]
F B F F
6. Organizational size:(a) In a large organization (1000+ employees) there were numerous competing priorities and it was challenging to maintain visibility [60] (b) In a small organization (<500 employees) it is challenging to assemble a critical mass of potential participants for participation in the intervention [60] (c) Small organizations tend to receive more intervention components per employee than larger organizations [61]
B
B
F
7. Amount of company locations: Different company locations at which the intervention needs to be delivered [74]
B
8. Organization’s awareness of perceived benefits of investment [74], and awareness of relevance and economics of health and employee wellness [53, 60]
F
9. Company image: the program gives the organization a positive image since it shows that the organization cares about their employees [66]
F
10. Perceived responsibility of employer towards workers health and wellbeing [74] F11. High staff turnover rate among employees made it difficult to provide adequate
exposure to the intervention [68, 69]B
12. Good collaboration between persons/ structures/ services/ collaborative partners within or outside departments and organizations [54, 66, 72]
F
13. Conflicting relationship between management and researchers[68] B14. General good organizational support for health promotion [53] F15. Poor psychosocial work environment consisting of the following the
subcomponents: influence at work, work pace quantitative work demands, interpersonal relations [70]
B
16. History of social interaction: Worksite has a history of bringing employees together for social activities and a history of positive social interaction between worker and management [55]
F
17. Management support: (a) Strong (upper) management support for intervention and general health promotion efforts at the organization [55, 60, 68, 72, 73] (b) Unbalanced management support for intervention [55, 68] (c) Managers encouraging workers to attend intervention [55] (d) Experienced management support are different for junior employees and senior employees [64] (e) formal approval of upper management before start of intervention [57] (f) Lack of perceived management support by implementers on site [74] (h) Management commitment and willingness to provide employees with release time from their usual duties to attend intervention [55]
F B F B F BF
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
64
Chapter 3
Characteristics of the organization
18. Management participation and engagement: (a) Active management participation and involvement alongside and with workers [55, 73] (b) Active management engagement in planning [55]
F F
19. Relationship between management and employees: Respectful relationship between management and worker [55]
F
Characteristics of the implementer
20. Job position of implementer:[74] (a) Self-employed (advantage of managing his or her own time) (b) Internal position (facilitating in scheduling appointments) (c) external position
F F B
21. (Perceived) Support for implementers:[74](a)Poor support from co- implementers (b) Support for implementers to change their routines (applicable when implementer is an occupational physician) [74]
B F
22. Collaboration between implementers: lack of possibility to exchange experiences between implementers [74]
B
23. Available time of implementer: (a) Sufficient time available to implement intervention [56, 66, 72, 74] (b) The intervention involved extra work on top of the heavy workload of the regular duties of the implementer [66] (c) planning difficulties of implementers with planning al contacts in the intervention period [59]
F B B
24. Expectations of implementer: implementers expectations were met [74] F25. Absence of a project leader/ leading person/ ambassador[72] B26. Implementers’ compliance with intervention protocol [52] F27. Staff turnover among implementers: drop out of implementers (without replacing
them) [69, 72]B
28. Absence of decision maker among implementers: among the implementers there lacked a person who was entitled to make decision at department level [72]
B
29. High perceived Level of control for intervention delivery by provider/implementer [60]
F
30. Low level of engagement of implementers in planning, promoting and providing feedback on intervention activities [55]
B
31. Personnel characteristics of implementer: sufficient skills, knowledge and competence to implement guideline or intervention correctly [55, 59, 74]
F
Characteristics of the intervention program
32. Degree of rewards: either financial reimbursement or other incentives [53, 68] F33. Compatibility and alignment of intervention with:
(a) organizations mission statement/business goals/ institutional policy change [53, 60, 68, 74](b) policy, culture, norms and current practices of organization [56, 58, 66, 72] (c) Ease of integration of intervention in working live [64]
F F F
34. The intervention fit implementers current work [74] F35. Intervention is part of the worksites integral health policy and seen as a pilot for
future health promotion policy instead of independent project [57]F
36. Relative advantage: intervention is advantageous compared to the current situation and no negative consequences were observed and the company, managers, implementers and participants benefit from participation [54, 56, 66, 72, 74]
F
37. Time: Project took more time than expected due to high workload of administration and planning[74]
B
38. Complexity: Intervention was not too difficult or complex to implement and execute [56, 59, 72, 74]
F
39. Observability of positive results of the intervention [74] F
Table 4. Continued
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
65
Systematic review of implementation studies
3
Characteristics of the intervention program
40. Risk and uncertainty level/Triability: the degree to which an innovation can be adopted/implemented with minimal risk [56]
F
41. Conflicting interest between worksite and intervention [66] B42. Timing of intervention activities: intervention activities coincide with scheduled
breaks [68]F
43. Technical problems (e.g., equipment breaks down) [54, 69] B44. Degree of incorporation of program communication and interventions into already
established communication channels or existing worksite events/meetings [53, 55]F
45. Presence of advisory board: well-functioning advisory board [55] F46. Ease of access to the program by bringing the program to participants and making
participation free or inexpensive [53]F
Characteristics of the participant
47. Needs of participants: [a] Positive personal preferences for program [63][b] No need for intervention (e.g., already being healthy) [74] [c] Positive program expectation [71], [d] Prior failed attempts to maintain a healthy lifestyle [62]
F B F B
48. Current workload and work structure/schedules: volume of daily tasks, overtime work, shift work, part-time work, irregular work schedules, shifts of different lengths, time-pressures [53, 60, 64, 68]
B
49. Work demands: Workers were unable to participate since they could not leave their work due to work demands, obligations and limited free time and flexibility to leave immediate work area [55, 57, 60, 64]
B
50. Time constraints of participants: lack of time, time constraints and willingness to make time to participate at work [53, 54, 57, 62, 63, 74]
B
51. Amount of peer leaders: Few peer leaders due to geographically separated worksites made it difficult to establish group cohesion[68]
B
52. Lack of social support: [a] No interaction with the entire workforce to build worksite-wide social norms and social support) [68] [b] Peer support: difficult to engage in behavior not considered normal by peers [64]
B B
53. Lack of motivation of workers to participate in intervention[54] B54. Participants self-efficacy: Low to medium self-efficacy is a barrier for participation
[70]B
Degree of implementation and program effectiveness
Only seven (31.8%) studies evaluated the association between implementation and program
outcomes [52, 58-62, 67]. These studies generally found that the level of implementation
in terms of high fidelity and dose was positively associated with a positive change in their
primary outcome measures (body weight, waist circumference, Body Mass Index, fruit and
vegetable intake, physical activity levels, smoking cessation). This was analyzed by means
of linear or logistic regression analysis [52, 58-62, 67], latent growth modeling [52], dose or
as-treated analysis [52, 58-62, 67], analysis of variance [52, 58-62, 67], mixed model logistic
regression analysis or linear mixed model regression analysis [52, 58-62, 67], chi square
tests [52, 58-62, 67], and multilevel linear regression analysis [52, 58-62, 67]. Of the seven
studies, three studies found that higher participation levels (dose received) significantly and
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
66
Chapter 3
positively impacted their primary outcome measures (body weight, waist circumference,
physical activity levels and smoking cessation) [52, 58, 62].
In the study on personal and team goal-setting with the aim of increasing leisure-
time physical activity, Dishman and Wilson found that high implementation groups had a
greater increase in vigorous physical activity over the three time points than did the low
implementation sites. For dose received no significant change over time was observed for
any of the outcome measures (walking, moderate physical activity, vigorous physical activity)
[52]. The results of the study of Volpp and Kim on improving smoking cessation rates by
offering financial incentives for smoking cessation, the results showed that the attendance
levels to the cessation programs were higher among quitters than among non-quitters
in the intervention group. Additionally, quitters in the intervention group attended more
than two times as many sessions as non-quitters in the intervention group [52, 58-62, 67].
Finally, in the study of Verweij on preventing weight gain by implementing an occupational
health guideline, it was found that employees with higher attendance and satisfaction levels
significantly reduced their waist circumference and body weight compared to employees
with lower attendance and satisfaction levels[52, 58-62, 67].
Discussion
The aims of this review were to (1) further our understanding of the quality of process
evaluations alongside effect evaluations for WHPPs, (2) identify barriers/facilitators
affecting implementation, and (3) explore the relationship between effectiveness and the
implementation process.
Prior to discussing the main findings, it should be noted that only a small number of
studies that evaluated the effectiveness of a WHPP included a process evaluation relating
to the implementation of that WHPP. Of the 307 effect evaluations identified in this review,
only twenty-two (7.2%) published an additional process evaluation. With respect to the first
aim we can conclude that the quality of process evaluations alongside effect evaluations of
WHPPs was generally poor to average and a systematic approach was lacking. This makes it
difficult to draw firm conclusion about reliably identifying implementation barriers and/or
facilitators (second aim) or about the relation between effectiveness and implementation
(third aim). Murta et al. (2007) found the same difficulties in a systematic review. They
concluded that process evaluations alongside workplace stress-management interventions
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
67
Systematic review of implementation studies
3
were mainly poor to average, also making it difficult to identify reliable determinants of
effective intervention implementation [22].
The process evaluations covered by our review lacked a theoretical basis, and the
most frequently measured process components were dose delivered and dose received.
This reflects the primary interest of researchers in actual intervention delivery and in
participation levels (in other words, quantitative outcomes), rather than in how an
intervention is delivered, the quality of delivery (fidelity), maintenance, or the reasons
respondents participated or not. Possible explanations may be that researchers are more
trained in quantitative methods of research and less in qualitative methods and analysis.
Furthermore, good qualitative research takes a lot of time and energy and is therefore more
expensive than quantitative research [75].
The included process evaluations tended to operationalize the measured process
components in different ways, even when using the same framework for evaluation. For
instance, four studies used Steckler and Linnan’s framework as a guideline for their process
evaluation [51, 57, 58, 69]. However, in some of these studies, actual attendance levels were
placed under reach [51, 57, 58], whereas another study included attendance levels as part of
the dose received [69]. The definitions for ‘reach’ and ‘dose received’ as defined by Steckler
and Linnan are: ‘proportion of intended target audience that participates in an intervention’
and ‘the extent to which participants actively engage with the intervention’ respectively [14].
The different approaches to operationalization in studies of these components could be
explained by these somewhat ambiguous descriptions in Steckler and Linnan’s framework.
What is the difference between ‘participating’ and ‘actively engaging with’ an intervention?
We can assume that ‘participating in an intervention’ involves a focus on participation and
non-participation regardless of the frequency, duration and intensity of participation and that
these aspects may actually be taken into account when measuring active engagement with
the intervention. Since several studies had different approaches to the operationalization
of process components, it was a challenge to translate the definitions used into general
terms. We therefore adopted a working definition for each process component based on
a previously published framework in order to interpret our findings [11]. For example, we
defined reach as ‘the proportion of the target audience that is aware of the intervention’ and
dose received as ‘the proportion of participants that actually participates in the intervention’,
including frequency, duration and intensity [11]. We therefore distinguished between
awareness and actual participation, the latter including level of participation. However, due
to the operationalization ambiguity in Steckler and Linnan’s framework, others might define
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
68
Chapter 3
these components, and therefore interpret the findings, differently. This limitation was also
mentioned in a comparable review by Durlak and Dupre (2008), revealing that researchers
still do not take the time and effort necessary to develop consensus about the best way to
perform process evaluations in different settings [13].
Our review of the studies provided evidence of various barriers and/or facilitators that
could influence the implementation process. However, partly as a result of the relatively
low quality of the process evaluations, no systematic examination of barriers and/
or facilitators affecting implementation was possible. Barriers and/or facilitators were
often not systemically measured and only observed and documented on the basis of the
researchers’ experience. This raises the question of how reliable the results of this review
are with respect to the identified barriers and/or facilitators (table 4). Since the barriers
and/or facilitators identified in this review clearly overlap with other reviews focusing on
these factors, we can conclude that our results should be reliable [10, 12, 13, 22, 24, 25]. For
example, Sangster-Gormley also identified ‘active management participation’ as a facilitator
for implementation [25].
Another notable finding was that researchers evaluated barriers and/or facilitators
mostly after implementation and only a few studied them during implementation. However,
interventions are more likely to be successful if potential barriers and/or facilitators are
assessed beforehand so they can be anticipated, facilitating implementation. Although
most studies failed to observe barriers and/or facilitators systematically, some barriers and/
or facilitators were more frequently reported in several studies. This suggests that these
are important factors which researchers, practitioners and implementers need to take
into account during a WHPP. In the category ‘characteristics of the socio-political context’,
only one barrier (‘competitive business environment’) and one facilitator (‘compatibility
of program with societal developments’) were reported in all twenty-two studies. This
could suggest that this category is virtually disregarded, despite its importance for a
thorough understanding of these socio-political factors prior to implementation, since the
intended user of the intervention is part of an organization, which in turn is part of a wider
environment [12]. However, it is also possible that intervention developers and researchers
already anticipate on socio-political issues and try to take these into account throughout the
development and implementation process.
Most barriers and/or facilitators were reported within the category ‘characteristics of
the organization’. This is not surprising since most implementation research focuses at this
level as this is essential for continuation. Moreover, the studies in this review are selected
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
69
Systematic review of implementation studies
3
because they focus on organizations. Frequently mentioned implementation barriers were:
‘lack of resources’, ‘no fit of the intervention with organizational culture’ and ‘unbalanced
or lack of manager support’. For example, when the intervention program requires a fitness
center but there is no center nearby, this intervention is not the best fit with the company
and other options, such as sports activities (possibly outdoors) in the immediate vicinity of
the company, should be explored. Frequently mentioned organizational facilitators for WHPP
implementation were: ‘organizations awareness of perceived benefits and relevance’; ‘good
collaboration with all persons involved’; ‘strong and formal management support’; and
‘active management participation and engagement’. The researcher or implementer could
work on these factors with the aim of optimizing implementation, for example by organizing
manager meetings before the start of the intervention so that everyone is informed. However,
another frequently mentioned facilitator for implementation was: ‘an organizational culture
that emphasizes goal setting and supports social interaction’. Nevertheless, it is unclear what
researchers and implementers can do when the organization culture does not fulfill this
criterion before the start of the study, especially since organizational culture is a complex
phenomenon, which is not easily changed [76]. We suggest that the best way to deal with
this ‘problem’ is to adapt the intervention where possible to ensure an optimal fit with the
current organizational culture. In addition we would advise researchers and implementers
to take the time to assess the organizational culture and include organizational determinants
in their process evaluation [77].
Turning to the category ‘characteristics of the implementer’, the results of this review
indicate that the most important facilitators for implementers are: ‘sufficient time’, ‘skills’,
‘knowledge’ and ‘competence’. When implementers experienced ‘planning difficulties’ or
a ‘heavy workload’, this represented a barrier to implementation. However, these barriers
can be overcome by making sure that the implementer receives enough support from
stakeholders within the organization and enough administrative support. In the category
‘characteristics of the intervention program’, fifteen barriers and/or facilitators were
reported. The three most frequently reported facilitators in this category were: ‘relative
advantage’, ‘compatibility’ and ‘complexity’. These three factors are part of Rogers’
Diffusion of Innovations theory, and they are therefore known facilitating factors which
could explain why researchers focused on these factors [78]. However, we also identified
barriers relating to the intervention characteristics, including: ‘conflicting interests between
worksite and interventions’ and ‘technical problems’ which are not part of a known theory.
Fewest barriers and/or facilitators were reported in the category ‘characteristics of the
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
70
Chapter 3
participant’. The most commonly reported barrier to participation was ‘time constraints of
participants’, suggesting that employers should give employees time off to participate in
the interventions. This facilitator is also a reflection of the facilitator ‘management support
and engagement’ which was mentioned earlier. Two frequently reported perceived barriers
for participants were ‘work demands’ and ‘current workload’. However, as for many of the
other found barriers, it is difficult to address these two factors. A possible suggestion to
overcome these barriers could be that at onset of the study, employers should stress out
that employees are allowed to participate under working hours and that if they experience
difficulties in participating due to work demands and workload they should discuss this with
their manager.
The reported barriers and/or facilitators in this review are comparable with the results of
other reviews mentioned in this paper [10, 12, 24, 25]. This suggests that the findings of this
review are generalizable to other settings. It would be beneficial to explore all the barriers
and/or facilitators listed here before implementation so that researchers and implementers
can anticipate possible barriers, incorporate possible facilitators and perhaps adjust their
program accordingly. A new Dutch instrument was proposed recently that makes it possible
to map barriers and/or facilitators beforehand, and therefore possibly help to facilitate
the actual implementation of WHPPs. It also serves as a monitoring instrument enabling
program adjustments before and during implementation (TNO R10625; Fleuren et al. 2012).
Unfortunately, it was not possible to rank the 54 reported barriers and/or facilitators
in order of importance because some determinants were only identified once. This could
suggest that the factor was specific to the type of intervention. In addition, the relation
between the 54 reported barriers and/or facilitators is unclear. This limitation was also
experienced by Fleuren et al. (2004), showing that more research is needed into barriers
and/or facilitators affecting the implementation process [12].
Despite the low quality of most included process evaluations, the findings of this
review do suggest that higher levels of implementation are associated with better program
outcomes. However, the few studies that investigated this association showed a narrow
focus on implementation, mainly addressing the link between dose received and/or fidelity
in relation to program outcomes. Whereas it is so important to use all aspects of process
evaluations for the interpretation of the (lack of) effects of a program [18, 19].
A limitation, which is inherent to writing a systematic review, is publication bias or
studies being overlooked. In other words, our overview of the literature may not be
complete. We tried to avoid this pitfall by selecting four different databases, both medical
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
71
Systematic review of implementation studies
3
and psychological, using broad search terms and checking each lifestyle (i.e. physical activity,
nutrition, smoking, alcohol use and relaxation) separately and by checking the references in
the studies we included. Clearly, our decision to restrict our search to process evaluations in
combination with effect evaluations in a (randomized) controlled trial design for WHPP has
caused a number of process evaluations to fall out of scope for this review. Although we do
acknowledge this, our aim was to gain insight into the relationship between implementation
quality and effectiveness. Importantly, this is also a tendency that has been suggested by
implementation journals, such as Implementation Science.
A best-evidence synthesis falls outside the scope of this review. Moreover, it could not be
performed since we included only the studies that could be paired with a process evaluation.
This means that not all known and relevant controlled trials in the field of WHPPs focusing
on healthy lifestyle are included in this review.
Conclusion and implications for future research
Given the fact that relatively few process evaluations were found by comparison with the
high number of RCT’s, and taking into account that not all process evaluations are published,
it is safe to conclude that process evaluations are still not as high on the research agenda as
effect evaluations. The importance of conducting an effect evaluation as well as a process
evaluation is increasingly being advocated [22, 79]. It does appear that lately more process
evaluations are being conducted, but this review shows that they are in general of low to
average quality due to the lack of a systematic approach.
Durlak and Dupre (2008) noted that “implementation matters” and “science cannot
study what it cannot measure accurately and cannot measure what is does not define” [13].
Our findings support this observation, and moreover, suggest that we should be asking:
what are we measuring, when we measure implementation at all? Without a standardized
approach to the operationalization of process evaluation components, it is difficult to
faithfully replicate studies, identify implementation barriers and/or facilitators, or assess the
methodological quality of process evaluations. This indicates that a general framework for
process evaluations is required that researchers, implementers, reviewers and practitioners
can use to evaluate and assess the quality of the implementation of a WHPP [80, 81]. In
order to create a general framework, it is essential that all relevant stakeholders subscribe to
a consensus about the terminology and operationalization of relevant process components
by developing a taxonomy for process evaluations. This taxonomy could constitute a
fundamental first step towards the standardization of process evaluations in multiple fields,
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
72
Chapter 3
and lead to program reproducibility [80]. Furthermore, it will allow researchers to compare
and assess the quality of process evaluations, systematically identify implementation
barriers and/or facilitators and possibly link implementation to program outcomes.
In short, this review demonstrates the need for a systematic approach to process
evaluation as a way of improving WHPP implementation. In order to set the first steps into
this direction, we suggest that future process evaluations need to apply the framework
Wierenga et al. 2012 we proposed in the introduction of this review and should also use
this framework to assess the methodological quality of the studies. This framework allows
mapping the complete implementation process and takes into account determinants of
implementation and provides the necessary explicit operationalization of each component
[11].
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
73
Systematic review of implementation studies
3
Appendices 1
Search strategies used for PubMed
Lifestyle behavior Strategy
Physical activity ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR physical activity OR exercise OR physical fitness) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))
Nutrition ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR Nutrition OR healthy eating OR food OR diet OR food services OR nutrition policy) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))
Smoking ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR smoking OR Tobacco OR smoking cessation OR tobacco use) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))
Alcohol use ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR Alcohol drinking OR Alcohol intake OR alcohol consumption) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))
Relaxation ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR “stress management” OR “job stress” OR “work stress” OR “psychological stress” OR “life stress” OR “emotional stress” OR distress OR “mental suffering” OR “relaxation therapy” OR relaxation OR meditation OR mind-body OR “mind-body therapies” OR “leisure activities” OR mindfulness OR “mental healing” OR “exercise movement techniques” OR “breathing exercises” OR yoga OR “tai ji” OR pilates OR engagement OR vitality OR detachment) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress OR relaxation)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
74
Chapter 3
Search strategies used for EMBASE
Lifestyle behavior Strategy
Physical activity ‘controlled study’/exp OR ‘randomized controlled trial’/exp OR ‘intervention study’/exp AND (‘worker’/exp OR ‘employee’/exp OR ‘personnel’/exp OR ‘workplace’/exp OR ‘work environment’/exp) AND (‘health promotion’/exp OR ‘lifestyle modification’/exp OR ‘physical activity’/exp OR ‘exercise’/exp OR ‘fitness’/exp) AND (‘absenteeism’/exp OR ‘sickness absence’/exp OR ‘medical leave’/exp OR ‘body mass’/exp OR ‘body weight’/exp OR ‘lifestyle’/exp OR ‘life style’/exp OR ‘quality of life’/exp OR ‘health’/exp OR ‘health behavior’/exp OR ‘blood pressure’/exp OR ‘cholesterol blood level’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py
Nutrition ‘controlled study’/exp OR ‘randomized controlled trial’/exp OR ‘intervention study’/exp AND (‘worker’/exp OR ‘employee’/exp OR ‘personnel’/exp OR ‘workplace’/exp OR ‘work environment’/exp) AND (‘health promotion’/exp OR ‘lifestyle modification’/exp OR ‘nutrition’/exp OR ‘food’/exp OR ‘diet’/exp OR ‘catering service’/exp OR ‘nutrition policy’/exp) AND (‘absenteeism’/exp OR ‘sickness absence’/exp OR ‘medical leave’/exp OR ‘body mass’/exp OR ‘body weight’/exp OR ‘lifestyle’/exp OR ‘life style’/exp OR ‘quality of life’/exp OR ‘health’/exp OR ‘health behavior’/exp OR ‘blood pressure’/exp OR ‘cholesterol blood level’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py
Smoking ‘controlled study’/exp OR ‘randomized controlled trial’/exp OR ‘intervention study’/exp AND (‘worker’/exp OR ‘employee’/exp OR ‘personnel’/exp OR ‘workplace’/exp OR ‘work environment’/exp) AND (‘health promotion’/exp OR ‘lifestyle modification’/exp OR ‘smoking’/exp OR ‘tobacco’/exp OR ‘smoking cessation’/exp OR ‘tobacco dependence’/exp) AND (‘absenteeism’/exp OR ‘sickness absence’/exp OR ‘medical leave’/exp OR ‘body mass’/exp OR ‘body weight’/exp OR ‘lifestyle’/exp OR ‘life style’/exp OR ‘quality of life’/exp OR ‘health’/exp OR ‘health behavior’/exp OR ‘blood pressure’/exp OR ‘cholesterol blood level’/exp OR ‘stress’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py
Alcohol use ‘controlled study’/exp OR ‘randomized controlled trial’/exp OR ‘intervention study’/exp AND (‘worker’/exp OR ‘employee’/exp OR ‘personnel’/exp OR ‘workplace’/exp OR ‘work environment’/exp) AND (‘health promotion’/exp OR ‘lifestyle modification’/exp OR ‘drinking behavior’/exp OR ‘alcohol’/exp OR ‘alcohol consumption’/exp) AND (‘absenteeism’/exp OR ‘sickness absence’/exp OR ‘medical leave’/exp OR ‘body mass’/exp OR ‘body weight’/exp OR ‘lifestyle’/exp OR ‘life style’/exp OR ‘quality of life’/exp OR ‘health’/exp OR ‘health behavior’/exp OR ‘blood pressure’/exp OR ‘cholesterol blood level’/exp OR ‘stress’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py
Relaxation trial OR randomized AND controlled AND trial OR controlled AND trial AND (‘worker’/exp OR workers OR ‘employee’/exp OR employees OR workforce OR worksite OR ‘work site’ OR ‘work environment’/exp) AND (‘health’/exp AND promotion OR worksite AND ‘health’/exp AND promotion OR ‘lifestyle’/exp AND intervention OR ‘stress management’/exp OR ‘job stress’/exp OR ‘work stress’/exp OR ‘psychological stress’/exp OR ‘life stress’/exp OR ‘emotional stress’/exp OR ‘distress’/exp OR ‘mental suffering’ OR ‘relaxation therapy’/exp OR ‘relaxation’/exp OR ‘meditation’/exp OR ‘mind body’ OR ‘mind-body therapies’/exp OR ‘leisure activities’/exp OR mindfulness OR ‘mental healing’/exp OR ‘exercise movement techniques’/exp OR ‘breathing exercises’/exp OR ‘yoga’/exp OR ‘tai ji’/exp OR ‘pilates’/exp OR engagement OR vitality OR detachment) AND (‘absenteeism’/exp OR ‘sickness’/exp AND ‘absence’/exp OR sick AND leave OR body AND ‘mass’/exp AND index OR body AND ‘weight’/exp OR ‘lifestyle’/exp OR ‘life’/exp AND style OR quality AND of AND ‘life’/exp OR ‘health’/exp AND ‘behavior’/exp OR ‘blood’/exp AND ‘pressure’/exp OR ‘cholesterol’/exp AND level OR ‘stress’/exp OR ‘relaxation’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
75
Systematic review of implementation studies
3
Search strategies used for PsycINFO
Lifestyle behavior Strategy
Physical activity (KW=(trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or physical activity or exercise or physical fitness or ‘active living’) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level)) AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)
Nutrition KW=(trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or nutrition or food or diets or ‘healthy eating’ or ‘food services’ or ‘nutrition policy’) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level) AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)
Smoking KW = (trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or smoking OR Tobacco or Tobacco smoking or smoking cessation or tobacco use) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress) AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)
Alcohol use KW = (trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or ‘drinking behavior’ OR alcohol OR ‘alcohol consumption’ OR ‘alcohol drinking’) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress) AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)
Relaxation KW = (trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or “stress management” OR “job stress” OR “work stress” OR “psychological stress” OR “life stress” OR “emotional stress” OR distress OR “mental suffering” OR “relaxation therapy” OR relaxation OR meditation OR mind-body OR “mind-body therapies” OR “leisure activities” OR mindfulness OR “mental healing” OR “exercise movement techniques” OR “breathing exercises” OR yoga OR “tai ji” OR pilates OR engagement OR vitality OR detachment) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress OR relaxation)AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
76
Chapter 3
Search strategies used for Cochrane Central Register of Controlled trials
Lifestyle behavior Strategy
Physical activity (Trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or “work site” or “work environment”) and (health promotion or worksite health promotion or lifestyle intervention or physical activity or exercise or physical fitness) and (absenteeism or sickness absence or sick leave or body mass index or body weight or lifestyle or life style or quality of life or health or health behavior or blood pressure or cholesterol level):ti,ab,kw from 2000 to 2012 (Word variations have been searched)
Nutrition (Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR Nutrition OR healthy eating OR food OR diet OR food services OR nutrition policy) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level):ti,ab,kw from 2000 to 2012 (Word variations have been searched)
Smoking (Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR smoking OR Tobacco OR smoking cessation OR tobacco use) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress):ti,ab,kw from 2000 to 2012 (Word variations have been searched)
Alcohol use (Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR Alcohol drinking OR Alcohol intake OR alcohol consumption) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress):ti,ab,kw from 2000 to 2012 (Word variations have been searched)
Relaxation (Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR “stress management” OR “job stress” OR “work stress” OR “psychological stress” OR “life stress” OR “emotional stress” OR distress OR “mental suffering” OR “relaxation therapy” OR relaxation OR meditation OR mind-body OR “mind-body therapies” OR “leisure activities” OR mindfulness OR “mental healing” OR “exercise movement techniques” OR “breathing exercises” OR yoga OR “tai ji” OR pilates OR engagement OR vitality OR detachment) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress OR relaxation):ti,ab,kw from 2000 to 2012 (Word variations have been searched)
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
77
Systematic review of implementation studies
3
References 1. Kirsten W: Making the link between health and productivity at the workplace--a global perspective.
Ind Health 2010, 48(3):251-255.
2. Jans MP, van den Heuvel SG, Hildebrandt VH, Bongers PM: Overweight and Obesity as Predictors of Absenteeism in the Working Population of the Netherlands. J Occup Environ Med 2007, 49(9):975-980.
3. Robroek SJ, van den Berg TI, Plat JF, Burdorf A: The role of obesity and lifestyle behaviours in a productive workforce. Occup Environ Med 2011, 68(2):134-139.
4. Alavinia SM, Molenaar D, Burdorf A: Productivity loss in the workforce: Associations with health, work demands, and individual characteristics. Am J Ind Med 2009, 52(1):49-56.
5. Engbers LH, van Poppel MN, ChinA.Paw MJ, van Mechelen W: Worksite health promotion programs with environmental changes: a systematic review. Am J Prev Med 2005, 29(1):61-70.
6. Ni Mhurchu C, Aston LM, Jebb SA: Effects of worksite health promotion interventions on employee diets: a systematic review. BMC Public Health 2010, 10(10):62.
7. Jepson RG, Harris FM, Platt S, Tannahill C: The effectiveness of interventions to change six health behaviors: a review of reviews. BMC Public Health 2010, 8(10):538.
8. Proper KI, Koning M, van der Beek AJ, Hildebrandt VH, Bosscher RJ, van Mechelen W: The effectiveness of worksite physical activity programs on physical activity, physical fitness, and health. Clin J Sport Med 2003, 13(2):106-117.
9. Kristensen TS: Intervention studies in occupational epidemiology. Occup Environ Med 2005, 62:205-210.
10. Robroek SJ, van Lenthe FJ, van Empelen P, Burdorf A: Determinants of participation in worksite health promotion programmes: a systematic review. Int J Behav Nutr Phys Act 2009, 6:26.
11. Wierenga D, Engbers LH, van Empelen P, Hildebrandt VH, van Mechelen W: The design of a real-time formative evaluation of the implementation process of lifestyle interventions at two worksites using a 7-step strategy (BRAVO@Work). BMC Public Health 2012, 12(1):619.
12. Fleuren M, Wiefferink K, Paulussen T: Determinants of innovation within health care organizations: literature review and Delphi study. Int J Qual Health Care 2004, 16(2):107-123.
13. Durlak JA, DuPre EP: Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 2008, 41:327-350.
14. Steckler A, Linnan L: Proces Evaluation for Public Health Interventions and Research: San Francisco: Jossey-Bass; 2002.
15. Paulussen TH, Wiefferink K, Mesters I: Invoering van effectief gebleken interventies. In Gezondheidsvoorlichting en gedragsverandering. Edited by Brug,J.,Van Asseman,P.& Lechner,L. Assen: van Gorcum: 2007:.
16. Craig P, Dieppe P, Macintyre S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008, 337(1655):979-983.
17. Saunders RP, Evans MH, Joshi P: Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health promotion practice 2005, 6(2):134-147.
18. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, RIPPLE Study Team: Process evaluation in randomised controlled trials of complex interventions. BMJ 2006, 332(7538):413-416.
19. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks PA: The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med 2004, 27(1):3-12.
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
78
Chapter 3
20. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp N, Smith JL: The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med 2006, Feb(21 Suppl 2):S1-8.
21. Mittman BS: Creating the evidence base for quality improvement collaboratives. Ann Int Med 2004, 140(11):897-901.
22. Murta SG, Sanderson K, Oldenburg B: Process evaluation in occupational stress management programs: a systematic review. Am J Health Promot 2007, 21(0890-1171; 0890-1171; 4):248-254.
23. Glasgow RE, Vogt TM, Boles SM: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999, (89):9-1322.
24. Koppelaar E, Knibbe JJ, Miedema HS, Burdorf A: Determinants of implementation of primary preventive interventions on patient handling in healthcare: a systematic review. Occup Environ Med 2009, 66(6):353-360.
25. Sangster-Gormley E, Martin-Misener R, Downe-Wamboldt B, Dicenso A: Factors affecting nurse practitioner role implementation in Canadian practice settings: an integrative review. J Adv Nurs 2011, 67(6):1178-1190.
26. Furlan AD, Pennick V, Bombardier C, van Tulder M, Editorial Board, Cochrane Back Review Group: 2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group. Spine (Phila Pa 1976) 2009, 34(18):1929-1941.
27. Proper KI, Staal BJ, Hildebrandt VH, van der Beek AJ, van Mechelen W: Effectiveness of physical activity programs at worksites with respect to work-related outcomes. Scand J Work Environ Health 2002, 28(2):75-84.
28. van Tulder M, Furlan A, Bombardier C, Bouter L, Editorial Board of the Cochrane Collaboration Back Review Group: Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine (Phila Pa 1976) 2003, 28(12):1290-1299.
29. French SA, Harnack LJ, Hannan PJ, Mitchell NR, Gerlach AF, Toomey TL: Worksite environment intervention to prevent obesity among metropolitan transit workers. Prev Med 2010, 50(4):180-185.
30. Dishman RK, DeJoy DM, Wilson MG, Vandenberg RJ: Move to Improve: a randomized workplace trial to increase physical activity. Am J Prev Med 2009, 36(2):133-141.
31. Yap TL, Davis LS, Gates DM, Hemmings AB, Pan W: The effect of tailored E-mails in the workplace. Part I. Stage movement toward increased physical activity levels. AAOHN J 2009, 57(7):267-273.
32. Goetzel RZ, Baker KM, Short ME, Pei X, Ozminkowski RJ, Wang S, Bowen JD, Roemer EC, Craun BA, Tully KJ, Baase CM, DeJoy DM, Wilson MG: First-year results of an obesity prevention program at The Dow Chemical Company. J Occup Environ Med 2009, 51(2):125-138.
33. Lemon SC, Zapka J, Li W, Estabrook B, Rosal M, Magner R, Andersen V, Borg A, Hale J: Step ahead a worksite obesity prevention trial among hospital employees. Am J Prev Med 2010, 38(1):27-38.
34. Sorensen G, Barbeau E, Stoddard AM, Hunt MK, Kaphingst K, Wallace L: Promoting behavior change among working-class, multiethnic workers: results of the healthy directions--small business study. Am J Public Health 2005, 95(8):1389-1395.
35. Beresford SA, Thompson B, Feng Z, Christianson A, McLerran D, Patrick DL: Seattle 5 a Day worksite program to increase fruit and vegetable consumption. Prev Med 2001, 32(3):230-238.
36. Sorensen G, Barbeau EM, Stoddard AM, Hunt MK, Goldman R, Smith A, Brennan AA, Wallace L: Tools for health: the efficacy of a tailored intervention targeted for construction laborers. Cancer Causes Control 2007, 18(1):51-59.
37. Sorensen G, Stoddard A, Quintiliani L, Ebbeling C, Nagler E, Yang M, Pereira L, Wallace L: Tobacco use cessation and weight management among motor freight workers: results of the gear up for health study. Cancer Causes Control 2010, 21(12):2113-2122.
38. Stoddard AM, Fagan P, Sorensen G, Hunt MK, Frazier L, Girod K: Reducing cigarette smoking among working adolescents: results from the SMART study. Cancer Causes Control 2005, 16(10):1159-1164.
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
79
Systematic review of implementation studies
3
39. Volpp KG, Troxel AB, Pauly MV, Glick HA, Puig A, Asch DA, Galvin R, Zhu J, Wan F, DeGuzman J, Corbett E, Weiner J, Audrain-McGovern J: A randomized, controlled trial of financial incentives for smoking cessation. N Engl J Med 2009, 360(7):699-709.
40. Driessen MT, Proper KI, Anema JR, Knol DL, Bongers PM, van der Beek AJ: Participatory ergonomics to reduce exposure to psychosocial and physical risk factors for low back pain and neck pain: results of a cluster randomised controlled trial. Occup Environ Med 2011, 68(9):674-681.
41. Groeneveld IF, Proper KI, van der Beek AJ, van Mechelen W: Sustained body weight reduction by an individual-based lifestyle intervention for workers in the construction industry at risk for cardiovascular disease: results of a randomized controlled trial. Prev Med 2010, 51(3-4):240-246.
42. Gilson N, McKenna J, Cooke C, Brown W: Walking towards health in a university community: a feasibility study. Prev Med 2007, 44(2):167-169.
43. Andersen LL, Saervoll CA, Mortensen OS, Poulsen OM, Hannerz H, Zebis MK: Effectiveness of small daily amounts of progressive resistance training for frequent neck/shoulder pain: randomised controlled trial. Pain 2011, 152(2):440-446.
44. Haukka E, Pehkonen I, Leino-Arjas P, Viikari-Juntura E, Takala EP, Malmivaara A, Hopsu L, Mutanen P, Ketola R, Virtanen T, Holtari-Leino M, Nykanen J, Stenholm S, Ojajarvi A, Riihimaki H: Effect of a participatory ergonomics intervention on psychosocial factors at work in a randomised controlled trial. Occup Environ Med 2010, 67(3):170-177.
45. Steenhuis I, Van Assema P, Van Breukelen G, Glanz K, Kok G, De Vries H: The impact of educational and environmental interventions in Dutch worksite cafeterias. Health Promot Int 2004, 19(3):335-343.
46. Hasson D, Anderberg UM, Theorell T, Arnetz BB: Psychophysiological effects of a web-based stress management system: a prospective, randomized controlled intervention study of IT and media workers [ISRCTN54254861. BMC Public Health 2005, 5:78.
47. Vermeer WM, Steenhuis IH, Leeuwis FH, Heymans MW, Seidell JC: Small portion sizes in worksite cafeterias: do they help consumers to reduce their food intake? Int J Obes (Lond) 2011, 35(9):1200-1207.
48. Strijk JE, Proper KI, van der Beek AJ, van Mechelen W: A worksite vitality intervention to improve older workers’ lifestyle and vitality-related outcomes: results of a randomised controlled trial. J Epidemiol Community Health 2012, 66(11):1071-1078.
49. Verweij LM, Proper KI, Weel AN, Hulshof CT, van Mechelen W: The application of an occupational health guideline reduces sedentary behaviour and increases fruit intake at work: results from an RCT. Occup Environ Med 2012, 69(7):500-507.
50. Jorgensen MB, Faber A, Hansen JV, Holtermann A, Sogaard K: Effects on musculoskeletal pain, work ability and sickness absence in a 1-year randomised controlled trial among cleaners. BMC Public Health 2011, 11:840.
51. Driessen MT, Proper KI, Anema JR, Bongers PM, van der Beek AJ: Process evaluation of a participatory ergonomics programme to prevent low back pain and neck pain among workers. Implement Sci 2010, 5:65.
52. Wilson MG, Basta TB, Bynum BH, DeJoy DM, Vandenberg RJ, Dishman RK: Do intervention fidelity and dose influence outcomes? Results from the move to improve worksite physical activity program. Health Educ Res 2010, 25(2):294-305.
53. Wilson MG, Goetzel RZ, Ozminkowski RJ, DeJoy DM, Della L, Roemer EC, Schneider J, Tully KJ, White JM, Baase CM: Using formative research to develop environmental and ecological interventions to address overweight and obesity. Obesity (Silver Spring) 2007, 15 Suppl 1:37S-47S.
54. Pehkonen I, Takala EP, Ketola R, Viikari-Juntura E, Leino-Arjas P, Hopsu L, Virtanen T, Haukka E, Holtari-Leino M, Nykyri E, Riihimaki H: Evaluation of a participatory ergonomic intervention process in kitchen work. Appl Ergon 2009, 40(1):115-123.
55. Hunt MK, Barbeau EM, Lederman R, Stoddard AM, Chetkovich C, Goldman R, Wallace L, Sorensen G: Process evaluation results from the Healthy Directions-Small Business study. Health Educ Behav 2007, 34(1):90-107.
R1R2R3R4R5R6R7R8R9
R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
80
Chapter 3
56. Vermeer WM, Leeuwis FH, Koprulu S, Zouitni O, Seidell JC, Steenhuis IH: The process evaluation of two interventions aimed at portion size in worksite cafeterias. J Hum Nutr Diet 2012, 25(2):180-188.
57. Strijk JE, Proper KI, van der Beek AJ, van Mechelen W: A process evaluation of a worksite vitality intervention among ageing hospital workers. Int J Behav Nutr Phys Act 2011, 8:58.
58. Verweij LM, Proper KI, Hulshof CT, van Mechelen W: Process evaluation of an occupational health guideline aimed at preventing weight gain among employees. J Occup Environ Med 2011, 53(7):722-729.
59. Groeneveld IF, Proper KI, Absalah S, van der Beek AJ, van Mechelen W: An individually based lifestyle intervention for workers at risk for cardiovascular disease: a process evaluation. Am J Health Promot 2011, 25(6):396-401.
60. Estabrook B, Zapka J, Lemon SC: Evaluating the implementation of a hospital work-site obesity prevention intervention: applying the RE-AIM framework. Health Promot Pract 2012, 13(2):190-197.
61. Beresford SA, Shannon J, McLerran D, Thompson B: Seattle 5-a-Day Work-Site Project: process evaluation. Health Educ Behav 2000, 27(2):213-222.
62. Kim A, Kamyab K, Zhu J, Volpp K: Why are financial incentives not effective at influencing some smokers to quit? Results of a process evaluation of a worksite trial assessing the efficacy of financial incentives for smoking cessation. J Occup Environ Med 2011, 53(1):62-67.
63. Yap TL, Busch James DM: Tailored e-mails in the workplace. AAOHN J 2010, 58(10):425-432.
64. Gilson N, McKenna J, Cooke C: Experiences of route and task-based walking in a university community: qualitative perspectives in a randomized control trial. J Phys Act Health 2008, 5 Suppl 1:S176-82.
65. Hunt MK, Harley AE, Stoddard AM, Lederman RI, MacArthur MJ, Sorensen G: Elements of external validity of tools for health: an intervention for construction laborers. Am J Health Promot 2010, 24(5):e11-20.
66. Steenhuis I, van Assema P, Reubsaet A, Kok G: Process evaluation of two environmental nutrition programmes and an educational nutrition programme conducted at supermarkets and worksite cafeterias in the Netherlands. J Hum Nutr Diet 2004, 17(2):107-115.
67. Quintiliani L, Yang M, Sorensen G: A process evaluation of tobacco-related outcomes from a telephone and print-delivered intervention for motor freight workers. Addict Behav 2010, 35(11):1036-1039.
68. Hunt MK, Fagan P, Lederman R, Stoddard A, Frazier L, Girod K, Sorensen G: Feasibility of implementing intervention methods in an adolescent worksite tobacco control study. Tob Control 2003, 12 Suppl 4:IV40-5.
69. Jorgensen MB, Faber A, Jespersen T, Hansen K, Ektor-Andersen J, Hansen JV, Holtermann A, Sogaard K: Implementation of physical coordination training and cognitive behavioural training interventions at cleaning workplaces--secondary analyses of a randomised controlled trial. Ergonomics 2012, 55(7):762-772.
70. Andersen LL: Influence of psychosocial work environment on adherence to workplace exercise. J Occup Environ Med 2011, 53(2):182-184.
71. Hasson H, Brown C, Hasson D: Factors associated with high use of a workplace web-based stress management program in a randomized controlled intervention study. Health Educ Res 2010, 25(4):596-607.
72. Driessen MT, Groenewoud K, Proper KI, Anema JR, Bongers PM, van der Beek AJ: What are possible barriers and facilitators to implementation of a Participatory Ergonomics programme? Implement Sci 2010, 5:64.
73. DeJoy DM, Bowen HM, Baker KM, Bynum BH, Wilson MG, Goetzel RZ, Dishman RK: Management support and worksite health promotion program effectiveness. Ergonomics and Health Aspects 2009, (5624):13-22.
74. Verweij LM, Proper KI, Leffelaar ER, Weel AN, Nauta AP, Hulshof CT, van Mechelen W: Barriers and Facilitators to Implementation of an Occupational Health Guideline Aimed at Preventing Weight Gain Among Employees in the Netherlands. J Occup Environ Med 2012, 54(8):954-960.
R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39
81
Systematic review of implementation studies
3
75. Pope C, Ziebland S, Mays N: Qualitative research in health care. Analysing qualitative data. BMJ 2000, 320:114-116.
76. Cameron KS, Quinn RE, Robert E: Diagnosing and changing organizational culture: Based on the Competing Values Framework: San Fransico: Jossey-Bass; 2006.
77. Weiner BJ, Lewis MA, Linnan LA: Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. Health Educ Res 2009, 24(2):292-305.
78. Rogers E.M.: Diffusion of innovations: New York: Free Press; 2003.
79. Nielsen K, Randall R, Holten A, Gonzálezc ER: Conducting organizational-level occupation health interventions: What works? Work & Stress 2010, 24(3):234-259.
80. Abraham C, Michie S: A taxonomy of behavior change techniques used in interventions. Health Psychol 2008, 27(3):379-387.
81. Pedersen LM, Nielsen KJ, Kines P: Realistic evaluation as a new way to design and evaluate occupational safety interventions. Safety Science 2012, 50(1):48-54.