1

A non-profit service and advocacy organization © 2011 National Council on Aging

Evidence-Based Programs 101 - Webinar Instructions

Thank you for joining today, please wait while others sign in.

The audio portion of this call will be heard through your

computer speakers.

Please make sure your speakers are on and the volume is

turned up!

You have joined the call in listen-only mode.

If you want to ask a question, please type your question into

the chat box.

Evidence-Based Programs 101 Tips for Selecting, Implementing, and Evaluating EBPs

Marcia Ory, PhD, MPH Ashley Wilson, MPH

Program on Healthy Aging Texas A&M Health Science Center

School of Rural Public Health

July 24th, 2012

Topic for Discussion

EBPs definitions and criteria

Advantages and disadvantages to EBPs

Steps in choosing an EBP for your organization

Texas Example

Program Evaluation Tips

Conclusions

Questions & Answers

Why the Recent Interest in EBPs?

Magnitude and seriousness of health and social problems in our communities

Awareness of preventability of many problems

Emergence of evidence-based practices and programs

Spend limited resources more efficiently

Our Assumptions

Most of you have probably delivered self-management programs to

adults.

Some of these programs have been ―evidence-

based.‖

What are Evidence-based Programs?

• Evidence-Based Programs are interventions based on evidence that is generated by scientific studies published in peer-reviewed journals.

• Model Programs are a type of evidence-based program

What are evidence-based programs?

• Previously implemented

• Found to make positive differences in the lives of participants

Evidence-based programs have been:

Tiered Set of Criteria for Defining EBPs Implemented Through OAA

FY2012 Congressional appropriations

require Title IIID funding to be

Evidence-Based

http://www.aoa.gov/AoARoot/AoA_Programs/HPW/Title_IIID/index.aspx

Highest-level Criteria

Dissemination products have been developed and are available to the public.

Full translation has occurred in a community site

Undergone Experimental or Quasi-Experimental Design

Adapted from Brownson et al, 2003

Type 1:

•Evidence that a health issue exists

Type 2:

•Evidence that a program is effective

Type 3: • Evidence about

design, context, and attractiveness of program

Three Types of Evidence

Target Population

Goals

Rationale

Benefits

Program Structure & Timeframe

Staffing

Facilities & Equipment

Program Evaluation

Fidelity Checklist

Elements Common to EBPs:

Practice Question

What are some advantages and disadvantages of evidence-based programs versus ―home grown‖ ones?

Perceived Disadvantages of EBPs

Requires knowing where to find and how to understand/judge the evidence

Feels like standardization of program rather than site-specific tailoring

Tools and processes are unfamiliar

Difficult to build community support—many prefer ―home grown‖ to ―off the shelf‖

Can be expensive

Online Training Modules: Evidence-based Health Promotion Programs for Older Adults

Increase in the likelihood of positive outcomes

Lead to efficient use of resources

Facilitate the spread of programs

Facilitate the use of common performance measures

Support continuous quality improvement

Help to establish partnerships

Make it easier to justify funding

Online Training Modules: Evidence-based Health Promotion Programs for Older Adults

Perceived Advantages to EBPs

How Does One Chose a EBP?

Choosing an EBP

Step 7: Sustain the program

Step 6: Evaluate the program

Step 5: Translate the intervention into a program

Step 4: Select an intervention

Step 3: Establish broad-based partnerships

Step 2: Identify effective interventions

Step 1: Identify an important health issue among the Seniors you serve

The tasks do not always follow a sequential progression. The list provides a guide, not a strict set of steps.

Step 1: Identify an important health issue among the Seniors you serve

Review health statistics and other data to identify relevant health conditions and risk factors

Ask questions of seniors in your constituency

Determine if the identified health issue affects all of the Seniors you serve or only a subset

Step 2: Identify effective interventions

National Council on Aging

• http://www.healthyagingprograms.org/content.asp?sectionid=32

National Cancer Institute

• http://rtips.cancer.gov/rtips/index.do

Substance Abuse and Mental Health Services Administration

• http://nrepp.samhsa.gov/Search.aspx

Aging Texas Well

• http://www.agingtexaswell.org/ebased/index.cfm

Partnering for Success

Identify organizations with a common mission

• Align goals

• Tap into the organizational cultures and core values of partners

Communicate

•Encourage honesty

• Clarify expectations

• Make sure everyone is given the same information

Work together

• Share praise, glory, and recognition

• Realize that no partner is perfect

• Think creatively

Share Resources

•Enhance participant recruitment

•Maximize program-to-program referrals and dissemination

•Share facilities and materials

•Cross train staff and volunteers

•Increase data collection and

•data analysis capacity

Step 3: Establish broad-based partnerships

Step 4: Select an intervention

Program Match

• Does the program fit your mission?

• Is leadership supportive of the program?

• Are clients interested in the program?

• Can you implement the program as designed?

Costs and Resources

• Is there a licensing fee?

• What type of facilitator is needed?

• Is facilitator training required?

• What equipment and supplies are needed?

• Is the program ongoing or periodic?

Cultural and Contextual Relevance

• Is the program culturally appropriate?

• Are materials written at the correct literacy level and in the correct language?

• Does the program have regional appeal?

Looking for Evidence-Based Programs

Based on best practices and principles

Subjected to rigorous research design

Program is efficacious on outcomes of interest

Program is manualized for replicability

Tested in multiple settings and populations

Published in peer review literature

Endorsed by professional body

Feasible to move to scale

Is Your Community Ready to Implement Evidence-Based Health Promotion Programs?

There is buy-in from leadership

There is access both to personnel with the expertise to do these programs and to the population that needs these programs

There is funding for the program

Agency/partner-ship is willing to do evidence-based health programs and stay true to the model being implemented

National Council on Aging Organizational Readiness Checklist

Step 5: Translate the intervention into a program

• The process of taking a program originally implemented in a controlled ―laboratory-like‖ setting and making it suitable for implementation in the community

Implementation Suitability

• The faithful and accurate adherence to the core elements of an intervention, must be maintained

• Preserves the evidence-base

Maintaining Fidelity

Step 5: Translate the intervention into a program (cont.)

• Program components that generated the positive outcomes

• Goal setting, identifying barriers, monitoring behavior change, etc.

Core elements

• Attributes that make the program ―fit‖ the population

• Size of fonts and reading level of materials, marketing techniques, etc.

• Key characteristics can be adapted, but core elements cannot be adapted (jeopardizes fidelity)

Key characteristics

Step 6: Evaluate the program

Use findings in future program planning

Evaluate program delivery and outcomes to assess program success

Monitor program operations and make adjustments to evaluation plan (if needed)

Create or find the instruments and protocols needed for data collection

Plan goals for process and outcome evaluation before program implementation

Step 7: Sustain the program

Elements of a sustainable infrastructure and delivery system

1 • Effective leadership

2 • Adequate delivery infrastructure

3 • Partnerships

4 • Centralized and coordinated logistical processes

5 • Business planning and financial sustainability

6 • Quality assurance and fidelity to interventions

Targets & Outcomes

Recruiting Partners and Participants

Quality Assurance and Fidelity

Challenges and Solutions

Texas Example

Texas Goals

Dates

April 1, 2010 –

March 31, 2012

Goals

Enroll 4,098 participants

72.5% Completion (2,975 Completers)

*Received extension through June 30th, 2012

Texas Healthy Lifestyles

Bexar AAA -San Antonio Area Central Texas AAA -Temple, Belton, Killeen East Texas AAA -45 counties Rio Grande AAA -El Paso Area Tarrant Count AAA -Fort Worth

Region Target Enrollees Target

Completers

Tarrant County 960 697

Central Texas 900 653

Rio Grande 888 645

East Texas Coalition 750 544

Bexar County 600 436

Total 4,098 2,975

Texas Targets by Region

Region Target Enrollees Target

Completers

Tarrant County 1,255 912

Central Texas 623 416

Rio Grande 985 739

East Texas Coalition 685 477

Bexar County 913 642

Total 4,461 3,186

Final Numbers by Region

Reach

• Questions to ask:

• Are we reaching our intended population?

• How do we attract underserved populations?

• How do we minimize dropout?

One of the first ways we evaluate programs is by their reach and population representativeness.

Recruiting: What Works?

Quality Leaders & Classes

Go to where seniors WORK, PLAY, PRAY, LIVE & EAT

Perseverance

Incentives

Food

Recruiting Program Participants

Networking

• Get out in your community

• Presentations

• Word of mouth

Senior Luncheons

Senior Expos, Health Fairs

City Senior Department

How to Recruit New Partners

Identify common organizational mission

• MOU

Make expectations clear

• Go beyond usual set of partners

• Identify various contributions

Think creatively

Keep a log of partner contacts

Recruiting Community Partners

Approaching Potential Partners

• Networking

• Program can be mutually beneficial to existing organizations

What to expect from a partner

• Space

• Participants

• Leaders

What to provide partner

• Program

• Leaders

• Materials

Quality Assurance & Program Fidelity

Quality Assurance refers to the entire scope of a state’s/organization’s program implementation.

• National Level

• State Level

• Regional Level

• Organizational Level

Fidelity refers very specifically to the degree to which a particular program is delivered as intended.

• Program developers

Both include: planning, monitoring, evaluating and making corrective actions

• Have a plan

Steps to Program Fidelity 4. Create a communications mechanism for feedback

3. Provide copies of plan and set training schedule

2. Prepare state-specific fidelity plan

1. Read Implementation Manual for each program

For reference, view NCOA Module 4: “Assuring Program Quality” at www.ncoa.org/chamodules

Example Fidelity Plan

Challenges

Participant Attrition

Leader Attrition

Program Fidelity

Data Collection

Solutions •Recruit & provide quality classes in community

•Provide transportation

•Know community trends

•Rural communities = fewer participants

Participant Attrition

•Provide leaders classes to teach immediately after training

•Involve leaders in scheduling classes

•Stipends if available Leader Attrition

• Identify and operationalize essential components of original program

• Know if adaptations are consistent with original study

• Dedicated Program Coordinator

• Protocol for fidelity checks

Program Fidelity

•Identify key research questions

•Simplify data collection

•Consider academic partners

•Document successes

Data Collection

Program Evaluation

• What do we want to know?

• Create instruments

• Data collection process?

• Who collects & manages the data?

• Database creation

• How to provide feedback to community & stakeholders?

Why is this

important?

Example Evaluation Manual

Program Evaluation Tips

Do you have other data collection requirements?

•AAA Client intake forms

Know target population’s abilities

•Age

•Reading Level

•Eye sight

Plan for time for program assessment

•Orientation class

•Partnering with University

Listen to and provide feedback to Key Stakeholders

Texas Size Successes

Exceeded our goal of 4,098 enrollees

Developed training infrastructure

Sustainability mechanisms

•Diabetes accreditation

•Partnering with health care organizations

Offering a menu of EBPs

Conclusions

EBPs have many benefits for improving population health

There are inventories of EBPs—no need to start from scratch

Partnerships are critical for recruitment, adoption, maintenance

For maximum impact, plan for sustainability early

Want to Learn More?

Take the online training modules: www.ncoa.org/CHAModules

Visit the Center’s website: www.ncoa.org/CHA

For more information:

Program on Healthy Aging

1266 TAMU

College Station, TX

77843-1266

Phone: 979-458-4202

Fax: 979-458-4264

www.programonhealthyaging.org