The Big PictureThe Role of Evaluation in
Public Health
Luann D’Ambrosio, MEd
Activity
Think of your favorite quote or sage advice that helps guide you through your professional work.
Wise Advice
I hear and I forget.
I see and I believe.
I do and I understand.
—Confucius (551–479 BC)
Standards
Utility
Accuracy
Propriety
Feasibility
CDC Evaluation Framework
• Describe the relationship between performance management, program planning and evaluation
• Describe the steps of program planning and evaluation process
• Identify key stakeholders in an evaluation
• Develop the components and elements of a program logic model
• Create a basic evaluation plan including data collection and analyzing methods
• List three ways to use evaluation data to draw conclusions about a program
• Define ROI, cost benefit ratios and social return on investment.
Objectives
Accreditation
CDC GrantsCDC awards grants for public health performance management systems
•$42.5 million allocated for 94 projects in state, local, tribal, and territories health departments
•Distributed in 49 states, 16 tribes and territories and nine local jurisdictions
•Five-year cooperative agreement awards are being funded by the health care reform law
Benefits of Accreditation
Accountability & credibility
Tool for improvement
Greater collaboration
Recognition & validationBetter
understanding of public health
Team building
Highlights HD strengths
Accreditation+
Phase 1
EvaluationMonitoring
Implementation
BudgetsAction Plans
Strategies
Strategic Issues Goals & Objectives
Mission ValuesVision
SWOCAnalysis &
Environmental Scan
Mandate Analysis
StakeholderAssessment
Plan the
Plan
Readiness Assessment
Phase 2
Phase 3
Phase 4
Strategic Planning
Performance Management (PM)
A systematic process aimed at helping achieve an
organization’s mission and strategic goals by
improving effectiveness, empowering employees,
and streamlining the decision-making process.
PerformancePerformance BehaviorBehavior ResultsResults= +
They Are Not the Same
Quality assurance Quality improvement
Reactive Proactive
Works on problems after they occur
Works on processes
Regulatory usually by state or federal law
Seeks to improve (culture shift)
Led by management Led by staff
Periodic look-back Continuous
Responds to a mandate or crisis or fixed schedule
Proactively selects a process to improve
Meets a standard (pass/fail) Exceeds expectations
They Are Not the Same
Evaluation Quality improvement
Assess a program at a moment in time
Understand the process that is in place
Static Ongoing
Doesn’t include identification of the source of a problem or potential solutions
Entails finding the root cause of a problem and interventions targeted to address it
Doesn’t measure improvements
Focused on making measurable improvements
Program-focused Customer-focused
A step in the QI process Includes evaluation
Value of Performance Management
• Disciplined approach
• Systemic
• Systematic
• Engaged customers and staff
• Results focused
• Proven method
Culture of QITopic Organization-wide Program/unit
Improvement System focus Specific project focus
QI planning Tied to the strategic plan
Program/unit level
QI goals Strategic plan Individual program/unit level plans
Approaches Baldrige ProgramOrganization QI Council
Lean Six SigmaIndividual QI TeamsRapid Cycle PDCA
Performance Management Frameworks
• Baldrige
• Six Sigma
• Lean
PDSA Cycle
What are we trying to accomplish?What are we trying to accomplish?
How will we know that a change is an improvement?How will we know that a change is an improvement?
What changes can we make that will result in improvement?What changes can we make that will result in improvement?
Memory Jogger
ToolsLogic models and work flow charts
Data and analysis tools
• Customer-supplier relationships
• Client flow, information flow
• Root cause tools: fishbone diagram, Pareto chart
• Force field analysis• Interrelationship digraph
QI PROCESS
to improve policies, programs, & outcomes•Manage changes•Create learning organization
Turning Point Performance Management Collaborative, 2003
PERFORMANCE MANAGEMENT
SYSTEM
PERFORMANCE STANDARDS•Identify relevant standards•Select indicators•Set goals & targets•Communicate expectations
PERFORMANCE MEASUREMENT•Refine indicators & define measures•Develop data systems•Collect data
REPORTING OF PROGRESS•Analyze data•Feed data back to managers, staff, policy makers, constituents•Develop a regular reporting cycle
QI PROCESS
•Use data for decisions
Questions?