creating and using evaluation tools for program success
DESCRIPTION
Creating and Using Evaluation Tools for Program Success. Greater Richmond Association for Volunteer Administration July 10, 2014. What is Evaluation?. - PowerPoint PPT PresentationTRANSCRIPT
Creating and Using Evaluation Tools for Program Success
Greater Richmond Association for Volunteer Administration
July 10, 2014
What is Evaluation?
Program evaluation is carefully collecting and analyzing information about a program or some aspect of a program in order to make necessary decisions.
Continual improvement is an unending journey.
2
Types of Simple and Meaningful Tools
SurveysPersonal interviewsFocus groupsOther data collections tools Existing databases
3
What Can Evaluation Tools Capture?
Designing a good tool requires more time and thought than you may think.
Questions can tap into:BehaviorsOpinionsFeelingsKnowledgeExperiencesCharacteristics
4
Who Can Be Helpful?
Depends on the purpose – who can answer your questions?If not apparent, ask for guidance from key informants.If potential population is large, generate a list of candidates and key characteristics then randomly select.
5
What is a Survey?
A quantitative research project in which a relatively large number of people are interviewed, each being asked a standard set of questions, posed in the same way each time.
6
Why Use Surveys?
To collect specific, standardized data across respondents.To collect data across many people as efficiently as possible. To increase understanding.To collect data quickly, even across multiple groups. Open-ended quotes may add impact and credibility.
7
Types of SurveysMethod
PhoneSnail MailWeb-BasedBlended Approach
Question TypesOpen-endedClosed-endedCombination
8
The Survey ToolDO:
Keep the survey as short as possible.Balance white space and length.Keep questions neutral.Be specific in your wording.Pretest your questions.Address only one construct per question.Use a blend of closed and open-ended questions, if reasonable.Mix types of questions throughout.Provide an opportunity for additional comments.
9
How to Enhance ParticipationDO:
Provide a reason for the survey.Use clear instructions.Explain confidentiality protections.Give a realistic sense of the time involved.Track respondents.Use reminders.Provide your contact information.Request contact information from the respondent, if appropriate.Establish an end date.
10
What is an Interview?
An interview is the collection of data by asking people questions and following up or probing their answers.
11
Why Use Interviews?To increase understanding.As an exploratory first step to creating quantitative tools.Enhance understanding of interesting findings which emerged from other processes.Real-world quotes may add impact and credibility.To collect specific, standardized data across respondents.
12
Types of InterviewsMethod
Face-to-faceTelephone
StyleConversational: go with the flowInterview guide
13
Conversational Approach
Questions emerge through conversationHighly individualizedMay provide greater insightsRequires interviewer with strong content and interpersonal skillsNot systematicCan be more difficult to analyze
14
Interview Guide Approach
Outline of topics exists, but wording and order can varyTopics covered is more systematicInterview setting remains fairly informalAlso required considerably skilled interviewerInhibits spontaneous discussion of new topics
15
Open-Ended Structured Approach
Strict script for question, but free-response format for answersMost efficient of qualitative techniquesReduces bias and guides less experienced interviewers
16
The Interview ToolDO:
Keep the interview as short as possible.Keep questions neutral.Be specific in your wording.Pretest your questions.Address only one construct per question.Use a blend of closed and open-ended questions, if reasonable.Provide an opportunity for additional comments.
17
Recruitment Techniques
DO:Provide a reason for the interview.Give a realistic sense of the time involved.Explain how your interview candidate fits into the bigger goal.
18
Preparing for the Interview
DO:Plan an appropriate interview location.Exchange contact information with the interviewee.Prepare to capture information. Create response cards, if needed.
19
Building RapportDO:
Be on time.Review the purpose of the interview.Discuss how the results will be used.Reiterate confidentiality protections.Acknowledge the time frame you have set aside.Provide your contact information on paper.Ask if they have questions before you begin.
20
General TipsDon’t be judgmental!Use appropriate non-verbal cues.Attend to non-verbal cues.Repeat and clarify when needed.Practice active listening. Let the interviewee know when you are switching gears.Keep the interviewee focused.Don’t make assumptions.
21
What is a Focus Group?
A focus group is a group discussion. Participants are brought together in a neutral location for the specific purpose of discussing and issue or responding to ideas or materials of interest.
22
Why Use Focus Groups?To increase understanding.As an exploratory first step to creating quantitative tools.Enhance understanding of interesting findings which emerged from other processes.Real-world quotes may add impact and credibility.To collect specific, standardized data across respondents.
23
Types of Focus Groups
MethodExploratoryIssue-Focused
24
Focus Group ModelDO:
Plan on a 2 hour sessionSecure 8-12 participants and a few alternatesIdentify 5-7 key questions beforehandProvide background information to group. Keep questions neutral.Record all input.Have subject matter experts available to answer questions. Use an objective facilitator.
25
Recruitment Techniques
DO:Provide a reason for the interview.Give a realistic sense of the time involved.Explain how your interview candidate fits into the bigger goal. Establish potential participant listInvite participants by letter followed by phone callConsider offering incentives
26
Preparing for the Session
DO:Secure a roomy, centrally-located venueSelect a facilitatorExchange contact information with the participantsPrepare to capture information.
27
Building RapportDO:
Be on time.Review the purpose of the interview.Discuss how the results will be used.Reiterate confidentiality protections.Acknowledge the time frame you have set aside.Provide your contact information on paper.Ask if they have questions before you begin.
28
Other Tools
ObservationsChecklists (process documentation)Attendance logsCase recordsExisting databases
29
Parting Tips
Keep it simpleBalance simplicity with precisionFocus on interpretationBlend methods if appropriate
30
