![Page 1: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/1.jpg)
Creating and Using Evaluation Tools for Program Success
Greater Richmond Association for Volunteer Administration
July 10, 2014
![Page 2: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/2.jpg)
What is Evaluation?
Program evaluation is carefully collecting and analyzing information about a program or some aspect of a program in order to make necessary decisions.
Continual improvement is an unending journey.
2
![Page 3: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/3.jpg)
Types of Simple and Meaningful Tools
SurveysPersonal interviewsFocus groupsOther data collections tools Existing databases
3
![Page 4: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/4.jpg)
What Can Evaluation Tools Capture?
Designing a good tool requires more time and thought than you may think.
Questions can tap into:BehaviorsOpinionsFeelingsKnowledgeExperiencesCharacteristics
4
![Page 5: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/5.jpg)
Who Can Be Helpful?
Depends on the purpose – who can answer your questions?If not apparent, ask for guidance from key informants.If potential population is large, generate a list of candidates and key characteristics then randomly select.
5
![Page 6: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/6.jpg)
What is a Survey?
A quantitative research project in which a relatively large number of people are interviewed, each being asked a standard set of questions, posed in the same way each time.
6
![Page 7: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/7.jpg)
Why Use Surveys?
To collect specific, standardized data across respondents.To collect data across many people as efficiently as possible. To increase understanding.To collect data quickly, even across multiple groups. Open-ended quotes may add impact and credibility.
7
![Page 8: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/8.jpg)
Types of SurveysMethod
PhoneSnail MailWeb-BasedBlended Approach
Question TypesOpen-endedClosed-endedCombination
8
![Page 9: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/9.jpg)
The Survey ToolDO:
Keep the survey as short as possible.Balance white space and length.Keep questions neutral.Be specific in your wording.Pretest your questions.Address only one construct per question.Use a blend of closed and open-ended questions, if reasonable.Mix types of questions throughout.Provide an opportunity for additional comments.
9
![Page 10: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/10.jpg)
How to Enhance ParticipationDO:
Provide a reason for the survey.Use clear instructions.Explain confidentiality protections.Give a realistic sense of the time involved.Track respondents.Use reminders.Provide your contact information.Request contact information from the respondent, if appropriate.Establish an end date.
10
![Page 11: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/11.jpg)
What is an Interview?
An interview is the collection of data by asking people questions and following up or probing their answers.
11
![Page 12: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/12.jpg)
Why Use Interviews?To increase understanding.As an exploratory first step to creating quantitative tools.Enhance understanding of interesting findings which emerged from other processes.Real-world quotes may add impact and credibility.To collect specific, standardized data across respondents.
12
![Page 13: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/13.jpg)
Types of InterviewsMethod
Face-to-faceTelephone
StyleConversational: go with the flowInterview guide
13
![Page 14: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/14.jpg)
Conversational Approach
Questions emerge through conversationHighly individualizedMay provide greater insightsRequires interviewer with strong content and interpersonal skillsNot systematicCan be more difficult to analyze
14
![Page 15: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/15.jpg)
Interview Guide Approach
Outline of topics exists, but wording and order can varyTopics covered is more systematicInterview setting remains fairly informalAlso required considerably skilled interviewerInhibits spontaneous discussion of new topics
15
![Page 16: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/16.jpg)
Open-Ended Structured Approach
Strict script for question, but free-response format for answersMost efficient of qualitative techniquesReduces bias and guides less experienced interviewers
16
![Page 17: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/17.jpg)
The Interview ToolDO:
Keep the interview as short as possible.Keep questions neutral.Be specific in your wording.Pretest your questions.Address only one construct per question.Use a blend of closed and open-ended questions, if reasonable.Provide an opportunity for additional comments.
17
![Page 18: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/18.jpg)
Recruitment Techniques
DO:Provide a reason for the interview.Give a realistic sense of the time involved.Explain how your interview candidate fits into the bigger goal.
18
![Page 19: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/19.jpg)
Preparing for the Interview
DO:Plan an appropriate interview location.Exchange contact information with the interviewee.Prepare to capture information. Create response cards, if needed.
19
![Page 20: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/20.jpg)
Building RapportDO:
Be on time.Review the purpose of the interview.Discuss how the results will be used.Reiterate confidentiality protections.Acknowledge the time frame you have set aside.Provide your contact information on paper.Ask if they have questions before you begin.
20
![Page 21: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/21.jpg)
General TipsDon’t be judgmental!Use appropriate non-verbal cues.Attend to non-verbal cues.Repeat and clarify when needed.Practice active listening. Let the interviewee know when you are switching gears.Keep the interviewee focused.Don’t make assumptions.
21
![Page 22: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/22.jpg)
What is a Focus Group?
A focus group is a group discussion. Participants are brought together in a neutral location for the specific purpose of discussing and issue or responding to ideas or materials of interest.
22
![Page 23: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/23.jpg)
Why Use Focus Groups?To increase understanding.As an exploratory first step to creating quantitative tools.Enhance understanding of interesting findings which emerged from other processes.Real-world quotes may add impact and credibility.To collect specific, standardized data across respondents.
23
![Page 24: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/24.jpg)
Types of Focus Groups
MethodExploratoryIssue-Focused
24
![Page 25: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/25.jpg)
Focus Group ModelDO:
Plan on a 2 hour sessionSecure 8-12 participants and a few alternatesIdentify 5-7 key questions beforehandProvide background information to group. Keep questions neutral.Record all input.Have subject matter experts available to answer questions. Use an objective facilitator.
25
![Page 26: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/26.jpg)
Recruitment Techniques
DO:Provide a reason for the interview.Give a realistic sense of the time involved.Explain how your interview candidate fits into the bigger goal. Establish potential participant listInvite participants by letter followed by phone callConsider offering incentives
26
![Page 27: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/27.jpg)
Preparing for the Session
DO:Secure a roomy, centrally-located venueSelect a facilitatorExchange contact information with the participantsPrepare to capture information.
27
![Page 28: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/28.jpg)
Building RapportDO:
Be on time.Review the purpose of the interview.Discuss how the results will be used.Reiterate confidentiality protections.Acknowledge the time frame you have set aside.Provide your contact information on paper.Ask if they have questions before you begin.
28
![Page 29: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/29.jpg)
Other Tools
ObservationsChecklists (process documentation)Attendance logsCase recordsExisting databases
29
![Page 30: Creating and Using Evaluation Tools for Program Success](https://reader035.vdocument.in/reader035/viewer/2022062222/568161d8550346895dd1dec7/html5/thumbnails/30.jpg)
Parting Tips
Keep it simpleBalance simplicity with precisionFocus on interpretationBlend methods if appropriate
30