© 2012 | WORK, FAMILY & HEALTH NETWORK

Evaluating Workplace Health and Wellbeing Interventions

Georgia T. Karuntzos, Ph.D

Jeremy Bray, Ph.D

Jesse M. Hinde

RTI International

Steps for Conducting Effective Evaluations

2

Centers for Disease Control

Standards

UtilityFeasibilityProprietyAccuracy

Step 1: Identify Need and

Engage Stakeholders

Step 2: Describe the

Program and it’s intended effects (Logic Model)

Step 4: Gather credible

evidence

Step 3: Focus theEvaluation

design

Step 5:Justify

conclusions

Step 6:Ensure useand share

lessons learned

Types of Evaluations

Adapted from:Norland, E. (2004, Sept). From education theory.. to conservation practice Presented at the Annual Meeting of the International Association for Fish & Wildlife Agencies, Atlantic City, New Jersey.

Theoretical/Logic Model

■ A diagram of the theory of how a program is supposed to work—a graphic depiction of relationships between activities and results (conceptual/theoretical model)

■ A logical chain of connections showing what a program intends to accomplish

■ Increases intentionality and purpose

■ Guides prioritization and allocation of resources

■ Helps to identify important variables to measure; use evaluation resources wisely

■ Supports replication

WFHN Evaluation Design

■ Single protocol■ Work redesign, supervisor training and self-monitoring

■ Multiple industries and worksites■ Healthcare (blue collar) ■ Telecommunications (white collar)

■ Group randomized field experiment■ Intervention and comparison group assignment at the

worksite or work group level■ Adaptive randomization to balance covariates

Outcome Evaluation Design

■ Nested cohort design

■ Worksite partners are randomized within industry to intervention or control group

■ Outcomes evaluated at multiple levels, employees, workgroups and worksites

■ Allows for multiple levels of clustering (e.g., workgroups within worksites), and a variety of outcomes (e.g., discrete, continuous, count)

Formative(Process) Evaluation Framework

8

Implementation Outcomes

Acceptability Adoption

AppropriatenessCost

FeasibilityFidelity

PenetrationSustainability

-

*IOM Standards of Care

What?

QIsESTs

How?

ImplementationStrategies

Implementation Research Methods

ServiceOutcomes*Efficiency

SafetyEffectiveness

EquityPatient-

centerednessTimeliness

Individual Outcomes

SatisfactionFunction

Health status/symptoms

Proctor’s Model of Implementation Outcomes

Process Evaluation Data■ Document Reviews – provide information to build an “a priori”

understanding of the program content, operations, context, and program stakeholders

■ Review reports, instruments, protocols, promotional materials, patient materials, resource lists, organizational documents (org charts, flow charts, operation manuals), web sites.

■ Observational Studies – provide empirical evidence to assess program fidelity, and generate service flow and timing data to inform outcome and cost analysis

■ Key Informant Interviews – provide contextual information related to program utility, contextual factors that influence program implementation, service delivery, dispersion, and sustainability

■ Practioner and Consumer Surveys -- provide systematic data related to service delivery experiences and program related perceptions

Qualitative Analysis Methods

■ Recursive abstraction (Document Summaries)

■ Iterative process that generates summaries, classifications, lists, rates, or groupings

■ Deductive and Inductive Analysis (data coding)

■ Deductive “a priori” framework

■ Inductive “grounded theory” analysis

■ Results in taxonomies, themes, categories, orders

■ Comparative analysis

■ Documented vs observed processes, behaviors or outputs

■ Document changes over time (e.g., model migration)

■ Variation between processes or outputs

■ Mixed Methods (triangulation and convergence)10

Process Evaluation Results

Comprehensive description of program components

Performance indicators and proficiency scores for program delivery

Common barriers and facilitators across worksites

Descriptive taxonomy of program settings

Construction of moderator variables for use in outcome and economic analyses

Program delivery protocols that are used at each worksite site

Outcome Evaluation Questions■ Program Outputs

■ To what extent is the program actually performed as measured by (e.g. the number of health risk appraisals completed, percentage of employees participating in workshops, number of follow-up services delivered)

■ Proximal and Distal Outcomes

■ What is the effectiveness of the program on outcomes of interest? (changes in sleep quality, changes in bio-measures)?

■ What stakeholder and employee-level characteristics moderate the willingness and ability of performance sites and practitioners to adopt the program

WFHN Study Data

Outcome Evaluation Methods

■ Descriptive statistics generating classifications or groupings of the participating worksites, provider characteristics, employee characteristics

■ Multivariate regression examining the relationship between groups

■ Statistical modeling measuring changes on outcomes over time

Outcome Evaluation Results

■ Rates and frequencies of program participation

■ Performance site descriptive characteristics (including staff characteristics) that are associated with outcomes of interest

■ Within group pre to post changes in health and wellness outcomes

■ Between group comparative changes in health and wellness outcomes at multiple time points

Intervention Effect on Work Environment

FSWE Hypotheses

■H1. There are changes in work environment for the

■ H1a. TOMO intervention group

■ H1b. TOMO control group

■ H1c. LEEF intervention group

■ H1d. LEEF control group

■H2. There are baseline differences in environment between intervention and control groups for the

■ H2a. TOMO industry

■ H2b. LEEF industry

■H3. There are differences in the rate of change in climate between intervention and control groups for the

■ H3a. TOMO industry

■ H3b. LEEF industry

FSWE Factor Analysis

Family Supportive Work Environment

Family Supportive Supervisor Behaviors

Organizational Climate Time Adequacy Schedule Control/

Decision Authority

0

11.4 8.3 10.7 11.5

1.465 1.142 0.655 0.844

Family Supportive Work Environment-9.9

Family Supportive Supervisor Behaviors

Organizational Climate Time Adequacy Schedule Control/

Decision Authority

1.465 1.142 0.655 0.844

11.4 8.3 10.7 11.5

Growth Model Results by Industry

Hypothesis Industry Group Random Effect

Estimate Standard Error p

H1a TOMO Intervention Slope 0.042 0.009 <0.0001

H1b TOMO Control Slope 0.014 0.008 0.081

H1c LEEF Intervention Slope 0.003 0.007 0.657

H1d LEEF Control Slope 0.001 0.006 0.905

H2a TOMO InterventionVs. Control

Intercept 0.033 0.038 0.393

H2b LEEF InterventionVs. Control

Intercept 0.013 0.027 0.619

H3a TOMO InterventionVs. Control

Slope 0.028 0.012 0.016

H3b LEEF InterventionVs. Control

Slope 0.002 0.009 0.804

Economic Evaluation Questions

■ What is the program cost to worksites and to other stakeholders?

■ What is the cost-effectiveness of the program?

■ What characteristics moderate the cost-effectiveness of the program?

Translational Study

An effective intervention is only useful if it is communicated to and adopted by workplaces

Complementary methods: Use participant feedback to inform

post-study messaging Analyze process data to study

employee perception for developing portrayals of the intervention

Assess potential dissemination channels

Stimulate market demand for effective intervention

Thank You

For more information on the Work, Family, and Health Network Study

http://projects.iq.harvard.edu/wfhn/people