institute of medicine obesity evaluation toolkit: resources for evaluating community-level obesity...

INSTITUTE OF MEDICINE

Obesity Evaluation Toolkit: Resources for Evaluating Community-Level Obesity

Prevention EffortsWebinar

August 25, 2015

INSTITUTE OF MEDICINE

INSTITUTE OF MEDICINE

2

1. Introduction

Leslie Sim, M.P.H. Institute of Medicine of The National Academies of Science, Engineering, and Medicine

2. Overview of report, Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress

Lawrence W. Green, Dr. P.H., M.P.H. University of California, San Francisco

INSTITUTE OF MEDICINE

3

3. Steps in evaluating community-level obesity prevention efforts

Stephen Fawcett, Ph.D.Work Group for Community Health and Development, University of Kansas

4. Web-based resources to support your efforts

Christina Holt, M.A.Community Tool Box, Work Group for Community Health and Development, University of Kansas

5. Questions

INSTITUTE OF MEDICINE

4

Submit questions - toolbox@ku.edu

Issues connecting? - 1-866-770-8162

INSTITUTE OF MEDICINE

5

Evaluating Obesity Prevention Efforts:

A Plan for Measuring Progress

Lawrence W. Green, Chair; and 13 members of

The Committee on Evaluating Progress of Obesity Prevention Efforts

INSTITUTE OF MEDICINE

6

Committee on Evaluating Progress of Obesity Prevention Efforts

INSTITUTE OF MEDICINE

7

AN URGENT

NEED FOR

EVALUATION

INSTITUTE OF MEDICINE

8

What we need to know from evaluation…

• Where are we in making progress and with whom? (current status)

• How are we doing in making progress? (trends over time in assessing needs and implementation of policies and strategies)

• What works in which populations?• What are the unintended consequences?

INSTITUTE OF MEDICINE

9

Committee on Evaluating Progress of Obesity Prevention Efforts

Study Charge: “to develop a concise and actionable plan for

measuring progress in obesity prevention efforts for the nation and adaptable

guidelines for community assessments and evaluation.”

INSTITUTE OF MEDICINE

10

INSTITUTE OF MEDICINE

11

The Concise and Actionable Plan

• Enhanced Leadership• Enhanced Roadmap• Enhanced Capacity and Infrastructure

Included resources: • Indicators for the evaluation plans• Tools and methods for assessing progress

in populations at greater risk for obesity• Community health assessment,

surveillance, and monitoring of interventions

INSTITUTE OF MEDICINE

12

Interdependence of National and Community Obesity Evaluation

Plans

Community Obesity

Evaluation Plan

National Obesity

Evaluation Plan

Core indicators, Data sources, Resources,

Methodologies

Contextual data, Feasibility, Local

innovation

INSTITUTE OF MEDICINE

13

Recommendations

1. Improve Leadership and Coordination

2. Improve Data Collection

3. Provide Common Guidance

4. Improve Access to and Dissemination of Information

5. Improve Workforce Capacity

6. Address Disparities and Health Equity

7. Support a Systems Approach

INSTITUTE OF MEDICINE

14

To access the full report and related dissemination materials(url: http://iom.nationalacademies.org/Reports/2013/Evaluating-Obesity-Prevention-Efforts-A-Plan-for-Measuring-Progress.aspx)

4-page report brief

Pull-out summary of indicators

Interactive indicator widget

INSTITUTE OF MEDICINE

15

Indicators of Progress (Excerpt from Table 4-1)

Indicator Topic ObjectiveOVERARCHING/SYSTEM-LEVEL INDICATORSObesity-adult Reduce the proportion of adults who are obese (body mass

index (BMI) ≥ 30)

APOP GOAL AREA 1: PHYSICAL ACTIVITY ENVIRONMENT

Adult physical activity Increase the proportion of adults who meet current federal

physical activity guidelines for aerobic physical activity and for muscle-strengthening activity

…

INSTITUTE OF MEDICINE

16

National and State Level Indicators (excerpt, Table 6-4)

INSTITUTE OF MEDICINE

17

Community Level Indicators (excerpt, Table 7-2)

Indicator Topica Data Source Current Availability by Community Sizeb

Larger Smaller

Overarching/System-Level

Obesity-adult BRFSS

Overweight-adult BRFSS

Obesity-adolescent YRBSS, School reports

Goal Area 2: Food and Beverage Environment

Sugar-sweetened beverage policies in schools (school district)

SHPPS

Sugar-sweetened beverage consumption YRBSS (adolescent)

School policies to facilitate access to clean drinking water SHPPS

Note: Larger – population >50,000; Smaller – population < 50,000

INSTITUTE OF MEDICINE

18

Some Steps in Evaluating Community-Level Obesity

Prevention Efforts

Stephen Fawcett, Work Group for Community Health and Development, University of Kansas

INSTITUTE OF MEDICINE

19

Vision for IOM Report

Assure collection and analysis oftimely and meaningful data

to inform and improve obesityprevention efforts at national,

state,and community levels

INSTITUTE OF MEDICINE

20

CONTEXT for Evaluation

• Multi-component, multi-sector, and multi-level interventions

• Indicators/measures of varying quality & utility

• Varying capacity, capabilities, leadership, and resources

INSTITUTE OF MEDICINE

21

ACTIVITIES: Develop Resources

for Training, Technical Assistance,

and Dissemination

INSTITUTE OF MEDICINE

22

Steps/Components of a Community Evaluation Plan (Box

8-1)1. Design stakeholder involvement. 2. Identify resources for the monitoring and

summative evaluation.3. Describe the intervention’s framework,

logic model, or theory of change.4. Focus the monitoring and summative

evaluation plan.5. Plan for credible methods.6. Synthesize and generalize.

INSTITUTE OF MEDICINE

23

1. Design stakeholder involvement• Identify stakeholders

• Consider the extent of stakeholder involvement

• Assess desired outcomes of monitoring and summative evaluation

• Define stakeholder roles in the monitoring and summative evaluation

INSTITUTE OF MEDICINE

24

Participatory Evaluation (CBPR)

INSTITUTE OF MEDICINE

25

2. Identify resources for monitoring and summative

evaluation• Person-power resources• Data collection resources

INSTITUTE OF MEDICINE

26

Some Web-Based Data Sources(Table 7-5)

• Community Commons & CHNA.org

• County Health Rankings

• USDA Food Atlas

• CDC Diabetes Interactive Atlas

• Census ACS and County/Zip Business Patterns

• HHS Community Health Status Indicators

INSTITUTE OF MEDICINE

27

What indicators available at local level?Available for all/ many communities:

• Adult obesity/overweight

• Activity: Active transport by walking, bicycling, density of recreational facilities, leisure-time PA

• Nutrition: Adult F/V, food outlet density, farmers’ market density, food deserts, SNAP/WIC store density

Some larger communities also have:• Youth obesity/overweight

• Activity: Youth PA, screen time, school PA participation

• Nutrition: SSB consumption, youth F/V

INSTITUTE OF MEDICINE

Data Availability (Table 7-2 – Example)

Large Small

INSTITUTE OF MEDICINE

29

3. Describe the intervention’s framework or logic model, or

theory of change.• Purpose or mission• Context or conditions• Inputs: resources and barriers• Activities or interventions• Outputs of activities• Intended effects or outcomes

INSTITUTE OF MEDICINE

30

Generic logic model for community obesity prevention (Figure 8-1, adapted).

INSTITUTE OF MEDICINE

31

4. Focus the monitoring and summative evaluation plan.

• Purpose or uses: What does the monitoring and summative evaluation aim to accomplish?

• Set priorities by end-user questions, resources, context

• What questions will the monitoring and summative evaluation answer?

• Ethical implications (benefit outweighs risk)

INSTITUTE OF MEDICINE

32

End-User Focused Evaluation Questions—Some examples:

• How fully was intervention implemented?

• Did the intervention have desired effects?

• What was the impact on participants/ population?• With whom? • Under what conditions?

INSTITUTE OF MEDICINE

33

5. Plan for credible methods.

• Stakeholder agreement on methods• Indicators of success• Credibility of evidence

INSTITUTE OF MEDICINE

34

Some Emerging Methods forData Collection

Environmental change data• Documentation of initiatives• Unobtrusive observations• Secondary data (e.g., GIS)

Policy change data• Documentation of initiatives• Surveillance

Systems change data• Mapping changing relationships

INSTITUTE OF MEDICINE

35

Indicators of Success• Translate expected effects (logic model) into specific

measurable units • Examples include:

– Program Outputs—units of activities delivered– Intermediate Outcomes—changes in communities and

systems (program, policy, environment)– Behavioral Outcomes—changes in diet and physical

activity – Population-level Outcomes—reduced prevalence of

obesity

INSTITUTE OF MEDICINE

36

Evaluation Designs – Match to Goal/Context

Qualitative methods: interviews, focus groups, photo-voice, etc.

INSTITUTE OF MEDICINE

37

6. Synthesize and generalize.

• Disseminating and compiling studies• Learning more from implementation• Ways to assist generalization• Shared sense-making and cultural

competence• Disentangling effects of interventions

INSTITUTE OF MEDICINE

38

Logic Model Design and Shared Sensemaking

Obesity

INSTITUTE OF MEDICINE

39

Obesity Evaluation Toolkit:Web-based Resources

for Community Evaluation

CONTEXT: Distributed evaluation workforce • People we will never see• In places we will never be

TOOLKIT: Just-in-time resources for:• Training• Technical Assistance

INSTITUTE OF MEDICINE

40

Web-based resources to support your efforts

Christina Holt, M.A.Community Tool Box, Work Group for Community Health and Development, University of Kansas

INSTITUTE OF MEDICINE

41

Navigating to the Toolkit

The Obesity Evaluation Toolkit is available online:http://iom.nationalacademies.org/activities/nutrition/obesityprevprogress/resources-evaluating-community-level-obesity-prevention-efforts

INSTITUTE OF MEDICINE

42

INSTITUTE OF MEDICINE

43

What you will find

INSTITUTE OF MEDICINE

44

INSTITUTE OF MEDICINE

45

INSTITUTE OF MEDICINE

46

INSTITUTE OF MEDICINE

47

Evaluation Toolkit

INSTITUTE OF MEDICINE

48

Community Tool Box – Example Resource

INSTITUTE OF MEDICINE

49

Troubleshooting Guide

INSTITUTE OF MEDICINE

50

INSTITUTE OF MEDICINE

51

INSTITUTE OF MEDICINE

52

INSTITUTE OF MEDICINE

53

INSTITUTE OF MEDICINE

54

INSTITUTE OF MEDICINE

55

INSTITUTE OF MEDICINE

56

INSTITUTE OF MEDICINE

57

Questions/ Discussion