using program evaluation to drive outcomes & funding for sustainability andrea m. jensen, ches...
TRANSCRIPT
![Page 1: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/1.jpg)
Using Program Evaluation to Drive Outcomes &
Funding for Sustainability
Andrea M. Jensen, CHESEnvironmental Health Educator
![Page 2: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/2.jpg)
• Participants will understand components of a strategic evaluation plan and individual evaluation plan
• Participants will understand how evaluation can be used for quality improvement and to demonstrate outcomes
• Participants will learn specific examples of how evaluation was used in the Utah Asthma Program to improve quality of asthma interventions and demonstrate program results
![Page 3: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/3.jpg)
Evaluation critical for:• Quality improvement
o Look critically and objectively at program processeso Identify strengths and weaknesseso Make course corrections as necessary
• Demonstrating outcomes o Look objectively at program outcomeso Determine if programs are meeting their purposeo Make course corrections as necessary
Providing measurable outcomes to a funder builds the case for continued
funding
![Page 4: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/4.jpg)
“Conversations for Advancing Action” Podcast
Series
• “Success builds success and if people understand that you are building something successfully and you’re doing good things and you can articulate where your successes are, and how you can achieve a way to build on those challenges. Everybody wants to bet on a winner and that’s the way to show it to them.”
• Maureen Wilce, Air Pollution and Respiratory Health Branch at CDC’s National Center for Environmental Health
![Page 5: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/5.jpg)
We will discuss:
• How to develop a strategic & individual evaluation plan
• How the Utah Asthma Program used evaluation
to:o Improve quality of asthma interventionso Demonstrate program outcomes
![Page 6: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/6.jpg)
Utah Asthma Program
Established in 2002
![Page 7: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/7.jpg)
Utah County Asthma Program
• 13 LHD’s
• 2 funded LHD’s
• Large state o (8 hrs x 5 hrs)o Many rural areas
http://freepages.genealogy.rootsweb.ancestry.com/~tolleygenealogy/utahmap.html
![Page 8: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/8.jpg)
Backgrounds?• Government?
• Nonprofit?
• Nurses?
• Respiratory therapists?
• Health Educators? (Public Health)
![Page 9: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/9.jpg)
Personal interest
Interior Designer Health Educator
![Page 10: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/10.jpg)
12 Hospitalizations
![Page 11: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/11.jpg)
Previous work plan
![Page 12: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/12.jpg)
More projects
![Page 13: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/13.jpg)
More projects
Asthma listserv
![Page 14: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/14.jpg)
CDC’s Framework for Evaluation
• Can be used for:
o Programs/Interventions
o Partnerships/Coalitions
o Surveillance systems
o Communication effortso Anything else you want
to evaluate
![Page 15: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/15.jpg)
http://www.cdc.gov/asthma/program_eval/guide.htm
![Page 16: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/16.jpg)
Introduction: State Asthma Program Evaluation Guide
Module 1: Evaluation and Your State Asthma Program
Module 2: Implementing Evaluations
Module 3: Evaluating Partnerships
![Page 17: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/17.jpg)
Module 4: Evaluating Asthma Surveillance
Module 5: Evaluation of Services and Health Systems Interventions
Module 6: Economic Evaluation for State Asthma Programs
![Page 18: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/18.jpg)
OverviewStrategic Evaluation
PlanIndividual Evaluation
Plan
• Overarching plan that shows evaluation strategies over a period of time
• Provides an overview and timeline of evaluations to be conducted
• Detailed plan for conducting a specific evaluation
![Page 19: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/19.jpg)
Strategic Evaluation Plan (SEP)
• Explains how a strategic approach to evaluation will assist your program in meeting its purpose & goals
• Components include:o Program background and purpose of SEPo Methods for developing and updating the SEPo Proposed evaluations
• Make a prioritized list• Include a summary of each evaluation’s purpose• Include a timeline for conducting the evaluations
o Communication plan• How will information on the SEP results be shared?
![Page 20: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/20.jpg)
Individual evaluation plan
• What are you going to evaluate?
• How are you going to evaluate it?
![Page 21: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/21.jpg)
Components of an individual evaluation plan
• Evaluation purpose
• Description of what you are evaluatingo Not how you will evaluate, but what you are evaluating
• e.g. a certain intervention, partnership efforts, communication efforts, etc.
![Page 22: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/22.jpg)
Components of an individual evaluation plan,
cont.
• Evaluation designo Stakeholder needso Evaluation questions
• What 3-5 major questions do you intend to answer through this evaluation?
o Description and rationale for evaluation design/methods
• Gather credible evidenceo Outline data collection methods for each evaluation
question
![Page 23: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/23.jpg)
CDC Evaluation Checklist
http://www.cdc.gov/asthma/program_eval/AssessingEvaluationQuestionChecklist.pdf
![Page 24: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/24.jpg)
Components of an individual evaluation plan,
cont.• Data analysis and interpretation
o Indicators and standardso What constitutes success?o Who will be involved in drawing and justifying conclusions?
• Use and communication of evaluation findingso How will evaluation findings be used? By whom?o How will evaluations findings be disseminated?
• Evaluation managemento Identify the evaluation teamo Define roles and responsibilities for everyone on the team
![Page 25: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/25.jpg)
Using Evaluation for Quality Improvement:
Recess Guidance• Identify if the Recess
Guidance is being used by schools
• Are schools implementing it correctly?
• Identify strengths & weaknesses
• Make it more effective
![Page 26: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/26.jpg)
Partnership Evaluation: Local Health Departments
(LHDs)• Identify strengths &
weaknesses
• How to improve the LHD grant process and required grant content for future LHD grant endeavors
![Page 27: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/27.jpg)
Surveillance Systems Evaluation
• Critical review of the asthma surveillance system
• Assess strengths & weaknesses
• Make recommendations for system improvement
• Focus special interest in simplicity, flexibility, data quality, and usefulness of the system
![Page 28: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/28.jpg)
Communication Evaluation: Asthma
Listserv
• Understand why people take part in the Utah Asthma Program Listserv
• What changes would make it more useful to participants?
![Page 29: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/29.jpg)
Asthma Listserv Evaluation
• Evaluation Questions:o What are the barriers to partner using listserv for sharing
information?o What improvements should be made to increase
communication on the listserv (should it become a more traditional listserv)?
o What types of messages sent on the listserv are most useful?o Why are members involved in the listserv?
• Data sourceso Literature reviewo Listserv membership excel sheet and recent listserv messageso Online questionnaire sent to listserv participants
![Page 30: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/30.jpg)
Asthma Listserv Evaluation - Recommendations
• Continue the listserv in the current format of 1-way• Email is the preferred communication medium
• Prefer 1-4 messages per month • Having large gaps between emails is ineffective • Find ways to increase the number of people who
follow links/look at materials outside of the basic email message
![Page 31: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/31.jpg)
Process and Outcome Evaluation: Asthma
Telehealth• Online asthma education program that offers CE
to nurses and respiratory therapists
• Two part evaluation o Process evaluation in 2011o Outcome evaluation in 2014
![Page 32: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/32.jpg)
Asthma Telehealth Process Evaluation,
2011• The purpose of this evaluation was to identify
course corrections for the Asthma Telehealth Program. The evaluation looked at fidelity of implementation, reach, and dose delivered
• Recommendations to improve the program were implemented over a period of two years
• Implementation of evaluation results paved the way for a future outcome evaluation
![Page 33: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/33.jpg)
Recommendations• Timing
• Continue quarterly sessions (as opposed to monthly)
• Presentation content and delivery:
• Work with presenters on presentation skills
• Include slides with takeaway points and a call to action to reinforce key messages
• Summarize content half way through • Enhance the question-and- answer section following the
webinar
![Page 34: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/34.jpg)
Recommendations
• Topics• Topic selection must include criteria from EPR-3 guidelines• Use healthcare provider focus groups to identify relevant
topics
• CE and Assessments• Expand CE credit to physicians and pharmacists• Collect data on participant change or patient level for
outcome evaluation• Evaluate skill/ability change based on web-based education
![Page 35: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/35.jpg)
Using Evaluation to Demonstrate Outcomes: Asthma Telehealth Series
• Outcome Evaluation – 2014
o Identify effectiveness of educating health providers on NAEPP guidelines-based care (in order to improve treatment and care for people with asthma)
o Identify how well the Telehealth program has succeeded in achieving its ultimate goals
![Page 36: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/36.jpg)
Evaluation DesignEvaluation Questions• Why do some participate in only one webinar?• What changes have participants made? • How has learning increased? • Is the right audience staying engaged?• How are topics keeping professionals on the forefront
of information for their profession?
Data Collection• Registration forms, pre- and post-test data, and
evaluation sheets completed by participants• Google form email survey 2 weeks after program
participation to assess practice changes
![Page 37: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/37.jpg)
Outcome Evaluation Results
• More RNs than RTs participated
• Increase in participants registering for more than one program over the last 3 years o (30.1% – 43.7%)
• Process evaluation worked to increase numbers!
![Page 38: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/38.jpg)
70% showed improvement from pre-post test
![Page 39: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/39.jpg)
Behavior Change Resulting from Telehealth
Participation
• 50% intended to make a practice change
• 50% selected ‘confirmed my current practice’
• 57% of reported implemented the call to action practice changeo This was a much higher percentage than the criteria
for success (set at 35%)
![Page 40: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/40.jpg)
Recommendations:• Continue with the call to action for participants
• Link the learning objectives of the program to skill-level measureso Participants would indicate his or her skill level before and
after
• Add instant messenger feed
• Promote Telehealth sessions to rural providers
![Page 41: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/41.jpg)
New funding - 2014• Comprehensive Asthma Control through
Evidence-based Strategies and Public Health - Health Care Collaboration grant
• Started over with new 5 year strategic plan……
![Page 42: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/42.jpg)
School-based Education
![Page 43: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/43.jpg)
Home-based Education
![Page 44: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/44.jpg)
Linkages to care• Prescription drug
assistance
• Home remediation
• Insurance navigator
• Referral to free/low cost clinics
![Page 45: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/45.jpg)
Already Evaluating!
![Page 46: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/46.jpg)
Planned Evaluations• System Scan for Expansion of Asthma Services in
Target Areas • GHHI and Asthma Home Interventions • Coverage and Reimbursement for Asthma
Services • Community Health Workers (CHWs) Asthma
Training & Utilization of CHWs • Asthma Feedback and Referral Mechanism • Asthma Business Case Utilization • Care Coordination Mechanism and Partnerships
![Page 47: Using Program Evaluation to Drive Outcomes & Funding for Sustainability Andrea M. Jensen, CHES Environmental Health Educator](https://reader036.vdocument.in/reader036/viewer/2022062805/5697bfc51a28abf838ca6c4a/html5/thumbnails/47.jpg)
Questions?Andrea M. Jensen, CHES
Environmental Health EducatorUtah County Health Department