february 2009 program evaluation: a pseudo-case study juliana m. blome, ph.d., mph office of program...
TRANSCRIPT
February 2009
Program Evaluation: A Pseudo-Case Study
Juliana M. Blome, Ph.D. , MPHOffice of Program Analysis and Evaluation
National Institute of General Medical Sciences
MORE Program Directors MeetingColorado Springs, Colorado
June 12, 2009
We’re looking for……
But we often get…..
NIGMS
February 2009
An evaluation plan should include…..
• Program description• Purpose & rationale for evaluation • Evaluation Design • Data Collection & Analyses • Products of evaluation & their use• Project Management • Budget estimate
NIGMS
February 2009
Program Description
Include program goals and baseline data
Program goals are the intended effects of a program
Activities should be organized to achieve specific goals Types of goals: Process, intermediate, long-term
Long-term Goal Example
• Weak: To train a diverse biomedical workforce
• Strong: To significantly increase the # of URMs graduating with baccalaureate STEM degrees and persisting through to graduate study
NIGMS
February 2009
Purpose & Rationale of Evaluation
Type of Evaluation
Needs assessment, Feasibility study, Process evaluation
Outcome evaluation?
Timing
Why is right now the time to conduct an evaluation?
Program Maturity
Is it reasonable to expect certain levels of output or measurable changes at this stage?
NIGMS
February 2009
Evaluation Design
An evaluation design should include…
Study questions Target population Key variables Conceptual framework if applicable
NIGMS
February 2009
Study Questions
What are the key questions the evaluation is designed to answer?
Key questions link to stated purpose of evaluation and program activities
Include any hypotheses that will be tested
Examples
How is the training program being implemented? (process)
What factors have inhibited the achievement of goals? (process)
NIGMS
February 2009
Examples
What has been the impact of the training program on the participants? (outcome)
What is the quality and character of the mentorship that is being provided in the program? (outcome)
How and to what extent does the program increase student skills and knowledge about laboratory research? (outcome)
NIGMS
February 2009
Institutional Impact Questions How has the training program affected your institution?
Institutions have structures which are defined by formal rules (laws, regulations, policies) and informal rules (culture, tradition, trust, implied codes of conduct) that shape people’s behavior
Where might we see institutional change?
• Curriculum development• Policies and practices• Services and support offered to students and faculty• Increased faculty awareness and responsibility for diversity• Impact on students not supported by the training program
NIGMS
February 2009
Target Population
What groups or groups do you need information about in order to answer the study questions?
People or institutions? Size, general characteristics, any subgroups, etc. ?
Examples• Trainees and students
• Project managers
• Academic coordinators
• Faculty
• High-ranking administrators
NIGMS
February 2009
Key Variables What specific information is needed to answer the study questions?
Examples
• Program resources – funding, staffing, infrastructure
• Population characteristics – demographics
• Program activities – operations, processes, other activities
• Program goals, performance measures, comparison measures
Program goal: Provide training opportunity for participants
Performance measure: Minimum # workshops held per year
Comparison measure: At least 4 workshops held per year (recognized standard of performance)
• External factors – factors beyond control of the program that may influence program success
NIGMS
February 2009
Conceptual Framework Consider developing a conceptual framework (logic model) to
illustrate how the program is supposed to achieve its goals
What is a logic model?
A logic model is a systematic and visual way to present and share your understanding of the relationships among the resources your have to operate your program, the activities you plan, and the changes or results you hope to achieve
-W.K. Kellogg Foundation Logic Model Development Guide (www.wkkf.org/Pubs/Tools/Evalaution/Pub3669.pdf)
NIGMS
February 2009
Model of a Training Program
Resources Activities Impact
Research base
Workshops & Seminars
Mentoring by faculty member
Training in scientific methods
Short term KnowledgeSkills Attitudes
IntermediateBehaviorsPractices
Long term
Enter PhD Program
Faculty & Staff
Money
Equipment & Technology
What is invested?
(Inputs) (Outputs) (Outcomes)
What is invested? What is done? What are the changes or benefits?
NIGMS
February 2009
Conceptual Framework: Why should we use one?
Increases understanding of program Provides a common language & framework Links activities to results Helps identify variables to measure Reflects group process and shared understanding Strengthens case for program investment
The bane of evaluation is a poorly designed program.
Ricardo Miller, Director, Evaluation Unit
WK Kellogg Foundation
NIGMS
February 2009
Data Collection and Analysis
Will you use new data or secondary data? Will it be quantitative, qualitative or mixed? Are there appropriate comparison groups? How will you collect the data? Are there ethical or IRB considerations? What are the limitations of the data?
NIGMS
February 2009
Bibliometric analysis
Quantitative; useful in aggregate as tool to assess quality of medical research
Measures only quantity; can be artificially influenced
Case studies Provides understanding of interaction of various influences on research process
Cases not necessarily representative within or across programs
Database extractions, Document reviews
Useful for analyzing archival data: databases, program records, literature review, etc
Records incomplete
Typical Data Collection Strategies
Method Pro Con
NIGMS
February 2009
Expert panel Useful in research fields, especially when few quantifiable indicators exist
Difficult to obtain systematic, objective assessment
Focus groups
Provides understanding of attitudes and thoughts on subject; group dynamic can help elicit honest responses
Results cannot be statistically generalized to larger populations; not quantifiable
Interviews Offer insight from perspective of specific program roles and expertise
Limited perspective; time-intensive
Surveys
Generate statistically reliable data – rating services, behavior, demographics, etc
Requires statistically representative sample & adequate response rate
Typical Data Collection Strategies (Cont.)
Method Pro Con
NIGMS
February 2009
Project Management: Who Participates?
Role Contributions Challenges
Program manager and staff
•Program knowledge Vested interest
Evaluator •Evaluation expertise•Independence
Limited program knowledge
Evaluation Advisory Committee
•Program familiarity•Evaluation expertise•Organizational context
Limited program knowledge
Senior Leader/ Decision-maker
•Organizational context•Resources
Vested interest
NIGMS
February 2009
Budget Estimate
“Rule of thumb” – 10% of project’s total budget
Common Pitfalls Failure to consider in up-front planning Lack of resources – for analysis & interpretation Lack of time – be realistic & consider time for each step Qualitative evaluation – more costly to implement
NIGMS
February 2009
Products of Evaluation
What reports or products are planned? • Executive summary & final report
• Briefings – for students, faculty, & administrators
• How will results be used?
NIGMS
February 2009
The Evaluation Design Matrix: A Tool for Discussion
Key Question(s)
Information Required
Information Source(s)
Data Collection Methods
Data Analysis Methods
Limitations Conclusions
WHAT DO YOU WANT TO KNOW?
WHAT DO YOU NEED TO ANSWER THE QUESTION?
WHERE ARE YOU GOING TO GET IT?
HOW ARE YOU GOING TO GET IT?
WHAT WILL YOU DO WITH IT ONCE YOU GET IT?
WHAT CAN'T YOU DO (CAVEATS)?
WHAT CAN YOU SAY?
•Clear and specific•Measurable•Doable•Key terms defined•Scope•Timeframe•Population
•Program goals •Evidence•Program criteria•Participant rates•Cost information•Funding levels
•Program officials or participants •External stakeholders•Program documents•Databases•Journals
•Structured interviews•Focus groups•Structured surveys•Case studies•Data extractions•Document retrieval
•Descriptive
statistics•Inferential statistics (T-test, regression)•Cost/ benefit analysis•Qualitative analysis
•Data quality or reliability•Access to records•Staffing/ funding constraints•Generalize
•Unexpected finding•Anecdotal information•Precise statements about sample•Generalize•Impact of program changes
NIGMS
February 2009
Why do we ask for program evaluation? ….................Because we’re accountable
What gets measured get’s done If you don’t measure success, you can’t reward it If you can’t reward success, you’re probably rewarding failure If you can’t see success, you can’t learn from it If you can’t recognize failure, you can’t correct it If you can demonstrate results, you can win public support
Osborne & Gabler (1992) in Reinventing Government As summarized by Ellen Taylor Powell -U of Wisc Extension