FUTURE SUCCESS: WHAT IS

REQUIRED?A Case for

Collaborative Multisite Evaluation

National Institute forOccupational Safety and Health

Presenters:

Vicky Buchan, Ph.D.

Sue Tungate, MSW

NIOSH Agricultural Centers By Regional Boundaries

MULTISITE EVALUATION TEAM Past and Present

Vicky Buchan - HI-CAHSBarbara Brooner - Pacific Northwest CenterHenry Cole - Southeast CenterCarol Donnelly - Southeast CenterGiulia Earle-Richardson - Northeast CenterTara Erb - Northeast CenterVirginia Fischer – National Farm Medicine CenterKaren Gilmore - Southwest CenterSusan Hewitt - HI-CAHSAdrienne Hidy - Pacific Northwest CenterBarbara Kennedy - Deep-South CenterRose Krebill-Prather - UC Davis CenterMarian McDonald - Southeast Center

Marian McDonald - Southeast Center Sharon Morris - Pacific Northwest CenterDeb Multerer - National Farm Medicine CenterPat O’Connor-Marer - UC Davis CenterBarbara Pies - Great Plains CenterRisto Rautiainen - Great Plains CenterPete Rentos - Deep-South CenterMary Lynne Thames - Southwest CenterDavid Tipton - Southeast CenterChristine Townsend - Northeast CenterSue Tungate - HI-CAHSKatheryn Wilson – Southeast CenterNancy Young - National Farm Medicine Center

A special thanks to HI-CAHS support staff and interns

Angi BuchananHelen-Holmquist JohnsonTammy Brannen-Smith

Chris BartholomewMarilyn DavisMary Gardner

Where we came from

NIOSH focus on agriculture1989, Agriculture at Risk: A Report to the Nation

Based on 1988 Iowa conference to develop agricultural occupational and environmental health policy strategies for the future

NIOSH Agricultural Initiative

Original Extramural programsAgricultural Safety & Health CentersFarm Family Health and Hazard SurveillanceO. H. Nurses in Agricultural CommunitiesAgricultural Health Promotion SystemsCancer Control Demonstration Projects

Kennedy ReportThe first recommendation

addressed the need for evaluation.

NIOSH and the Centers need to work together to develop an evaluation scheme for:

The various component programs and projects [within site] andThe Centers overall [multi- or cross-site]

Issues Important to ACE

Congressional intent that Initiative have “a significant and measurable impact”BSC recommendation for overall strategic planning processBSC recommendation for better internal and external communication

The Beginnings of ACE

NIOSH Agricultural Center Initiative

Objectives1. Conduct research2. Implement model education outreach3. Evaluate control technologies4. Implement model programs5. Evaluate6. Provide consultation and training7. Develop linkages and

communications8. Dissemination

DISEASE & INJURY

WORK ENVIRONMENT & WORKFORCE

RESEARCH TOOLS & APPROACHES

Allergic and Irritant DermatitisAsthma and Chronic Obstructive Pulmonary

DiseaseFertility and Pregnancy AbnormalitiesHearing LossInfectious DiseasesLow Back DisordersMusculoskeletal Disorders of the Upper

ExtremitiesTraumatic Injuries

Emerging TechnologiesIndoor EnvironmentMixed ExposuresOrganization of WorkSpecial Populations at Risk

Cancer Research MethodsControl Technology and Personal Protective

EquipmentExposure Assessment MethodsHealth Services ResearchIntervention Effectiveness ResearchRisk Assessment MethodsSocial & Economic Consequences of Workplace

Illness & InjurySurveillance Research Methods

NORA CATEGORY PRIORITY RESEARCH AREAS

GOAL OF COLLABORATIVE

MULTISITE EVALUATION

To coordinate center evaluation efforts seeking standardization of methodology to enable continual

monitoring of progress and impact of the whole INITIATIVE without losing

individual uniqueness of each center

OBJECTIVESDocument Initiative progressFacilitate Inter-Center/NIOSH collaborationIncrease Initiative effectivenessAssist Initiative program planning

Short Term

Measures

Mid Term Measures

Long Term Measures

NIOSH Objectives

--

NORA

Indicators and Key Variables

Initiative PR

Report

Annual Multisite Initiative Report

Information for Center Reporting

MULTISITE EVALUATIONNIOSH Agricultural Initiative

MissionHEALTH SAFETY PREVENTION

Initiative Multisite Database

Database Variables

Identifying InformationCenter & PHRTitle & DescriptionCenter or Project ObjectivesPI or Contact PersonStatus

Key VariablesStrategy (Research, Intervention, Education, Evaluation)Topic or Key WordActivitiesProfessional Education or TrainingProductsTarget Group Demographics & Number of Contacts

IndicatorsCollaborationDisseminationLeveragingRegionalizationReplicationResearch NeedSpecial Sector

NIOSHNIOSH Objectives (number)NORA – Priority Research

Area (for Research)

MULTISITE CHALLENGES

After-the-fact evaluation projects(or retrospective evaluations)Variation in program implementation (even if similar objectives across sites)Contextual differences between sites lead to further adaptationMultiple theoretical perspectivesDifferential “buy-in” to need for and purpose of evaluation

The ProcessCommunication

Conference CallsE-mails

MeetingsPlanningTravelAgendaTimelinesDocumentation

Database Design and Documentation

Orientation Data AggregationCollate Team OutputOrganizationArchiving

Annual Reports

RESEARCH QUESTIONS1. Target population contacts - By NIOSH Objective? By strategy?2. Research projects - By NORA3. Special Sector activities4. Products achieved5. Target groups of Initiative6. Collaborative efforts – NIOSH & Centers or others7. Education & Training under Center Auspices8. Value of leveraged funds & services9. Regionalization – States where Center activity occurred

0

5000

0

1000

00

1500

00

2000

00

2500

00

Nu

mb

er o

f A

g I

nit

iati

ve

Co

nta

cts

1. R

esea

rch

2. O

utre

ach

3. C

ontro

l Tec

h

4. P

reve

ntion

5. E

valu

atio

n

6. T

rain

ing

7. L

inka

ges

8. D

issem

inatio

n

NIOSH Objectives

Target Population Contacts by NIOSH Objective

Number of Initiative Contacts by Region: Research Strategy, FY 2000

These represent the total number of contacts reported by each Agricultural Center for fiscal year2000 that occurred within research projects or activities. The extent of data collection across Centers varies, therefore reported contacts are most likely undercounted. Contact intensity varies,also. For example, some contacts represent direct face-to-face intervention or training, while others could indicate a mass mailing or publication circulation.

Research Contacts<300<2500<5000<10000<35000<170000

Intervention Contacts<300<1500<3000<5000<10000<25000

Number of Initiative Contacts by Region:Intervention Strategy, FY 2000

These represent the total number of contacts reported by each Agricultural Center for fiscal year2000 that occurred within intervention projects or activities. The extent of data collection across Centers varies, therefore reported contacts are most likely undercounted. Contact intensity varies,also. For example, some contacts represent direct face-to-face intervention or training, while others could indicate a mass mailing or publication circulation.

Education Contacts<2000<5000<10000<15000<55000<150000

These represent the total number of contacts reported by each Agricultural Center for fiscal year2000 that occurred within education projects or activities. The extent of data collection across Centers varies, therefore reported contacts are most likely undercounted. Contact intensity varies,also. For example, some contacts represent direct face-to-face intervention or training, while others could indicate a mass mailing or publication circulation.

Number of Initiative Contacts by Region: Education Strategy, FY 2000

Number of Database Records that Address NORA Category I: Disease and Injury

0 5 10 15 20 25

Dermatitis

Asthma & COPD

Reproductive Abnormalities

Hearing Loss

Infectious Diseases

Low Back Disorders

Upper Musculoskeletal Disorders

Traumatic Injuries

Pri

ori

ty R

es

ea

rch

Are

as

Number of Records in the Database

Number of Database Records that Address NORA Category II: Work Environment and Workforce

0 10 20 30 40 50

EmergingTechnologies

IndoorEnvironment

Mixed Exposures

Organization ofWork

SpecialPopulations

Pri

ori

ty R

es

ea

rch

Are

as

Number of Records in the Database

Number of Database Records that Address NORA Category III: Research Tools and Approaches

0 5 10 15 20 25

Cancer Research

Control Tech & PPP

Exposure Assessment

Health Services

Intervention Effectiveness

Risk Assessment

Social & Economic

Surveillance Research

Pri

ori

ty R

es

ea

rch

Are

as

Number of Records in the Database

Database Records with Special Sector Groups in the Primary Target Population

Children38%

Ethnic/Minority Workers29%

Migrant/Seasonal Workers

15%

Women11%

Combination Groups (adult)

5%

Low Income2%

Count of Center Initiative Products FY 2000

138

108

67

49

48

35

25

24

23

20

20

19

17

0 20 40 60 80 100 120 140 160

Presentations, PowerPoint, slides, overheads

Article published, professional (juried publication)

Reports to NIOSH

Poster

Article published, feature (not juried)

Report, unpublished (research, evaluation)

Data collection instrument

Database

Fact Sheet

Thesis or dissertation

Presentation / speech, written

Exhibit material

Website or webpage established

Pro

du

ct T

ypes

Frequency

THE FUTURE