creating and using evaluation tools for program success

Creating and Using Evaluation Tools for Program Success

Greater Richmond Association for Volunteer Administration

July 10, 2014

What is Evaluation?

Program evaluation is carefully collecting and analyzing information about a program or some aspect of a program in order to make necessary decisions.

Continual improvement is an unending journey.

2

Types of Simple and Meaningful Tools

SurveysPersonal interviewsFocus groupsOther data collections tools Existing databases

3

What Can Evaluation Tools Capture?

Designing a good tool requires more time and thought than you may think.

Questions can tap into:BehaviorsOpinionsFeelingsKnowledgeExperiencesCharacteristics

4

Who Can Be Helpful?

Depends on the purpose – who can answer your questions?If not apparent, ask for guidance from key informants.If potential population is large, generate a list of candidates and key characteristics then randomly select.

5

What is a Survey?

A quantitative research project in which a relatively large number of people are interviewed, each being asked a standard set of questions, posed in the same way each time.

6

Why Use Surveys?

To collect specific, standardized data across respondents.To collect data across many people as efficiently as possible. To increase understanding.To collect data quickly, even across multiple groups. Open-ended quotes may add impact and credibility.

7

Types of SurveysMethod

PhoneSnail MailWeb-BasedBlended Approach

Question TypesOpen-endedClosed-endedCombination

8

The Survey ToolDO:

Keep the survey as short as possible.Balance white space and length.Keep questions neutral.Be specific in your wording.Pretest your questions.Address only one construct per question.Use a blend of closed and open-ended questions, if reasonable.Mix types of questions throughout.Provide an opportunity for additional comments.

9

How to Enhance ParticipationDO:

Provide a reason for the survey.Use clear instructions.Explain confidentiality protections.Give a realistic sense of the time involved.Track respondents.Use reminders.Provide your contact information.Request contact information from the respondent, if appropriate.Establish an end date.

10

What is an Interview?

An interview is the collection of data by asking people questions and following up or probing their answers.

11

Why Use Interviews?To increase understanding.As an exploratory first step to creating quantitative tools.Enhance understanding of interesting findings which emerged from other processes.Real-world quotes may add impact and credibility.To collect specific, standardized data across respondents.

12

Types of InterviewsMethod

Face-to-faceTelephone

StyleConversational: go with the flowInterview guide

13

Conversational Approach

Questions emerge through conversationHighly individualizedMay provide greater insightsRequires interviewer with strong content and interpersonal skillsNot systematicCan be more difficult to analyze

14

Interview Guide Approach

Outline of topics exists, but wording and order can varyTopics covered is more systematicInterview setting remains fairly informalAlso required considerably skilled interviewerInhibits spontaneous discussion of new topics

15

Open-Ended Structured Approach

Strict script for question, but free-response format for answersMost efficient of qualitative techniquesReduces bias and guides less experienced interviewers

16

The Interview ToolDO:

Keep the interview as short as possible.Keep questions neutral.Be specific in your wording.Pretest your questions.Address only one construct per question.Use a blend of closed and open-ended questions, if reasonable.Provide an opportunity for additional comments.

17

Recruitment Techniques

DO:Provide a reason for the interview.Give a realistic sense of the time involved.Explain how your interview candidate fits into the bigger goal.

18

Preparing for the Interview

DO:Plan an appropriate interview location.Exchange contact information with the interviewee.Prepare to capture information. Create response cards, if needed.

19

Building RapportDO:

Be on time.Review the purpose of the interview.Discuss how the results will be used.Reiterate confidentiality protections.Acknowledge the time frame you have set aside.Provide your contact information on paper.Ask if they have questions before you begin.

20

General TipsDon’t be judgmental!Use appropriate non-verbal cues.Attend to non-verbal cues.Repeat and clarify when needed.Practice active listening. Let the interviewee know when you are switching gears.Keep the interviewee focused.Don’t make assumptions.

21

What is a Focus Group?

A focus group is a group discussion. Participants are brought together in a neutral location for the specific purpose of discussing and issue or responding to ideas or materials of interest.

22

Why Use Focus Groups?To increase understanding.As an exploratory first step to creating quantitative tools.Enhance understanding of interesting findings which emerged from other processes.Real-world quotes may add impact and credibility.To collect specific, standardized data across respondents.

23

Types of Focus Groups

MethodExploratoryIssue-Focused

24

Focus Group ModelDO:

Plan on a 2 hour sessionSecure 8-12 participants and a few alternatesIdentify 5-7 key questions beforehandProvide background information to group. Keep questions neutral.Record all input.Have subject matter experts available to answer questions. Use an objective facilitator.

25

Recruitment Techniques

DO:Provide a reason for the interview.Give a realistic sense of the time involved.Explain how your interview candidate fits into the bigger goal. Establish potential participant listInvite participants by letter followed by phone callConsider offering incentives

26

Preparing for the Session

DO:Secure a roomy, centrally-located venueSelect a facilitatorExchange contact information with the participantsPrepare to capture information.

27

Building RapportDO:

Be on time.Review the purpose of the interview.Discuss how the results will be used.Reiterate confidentiality protections.Acknowledge the time frame you have set aside.Provide your contact information on paper.Ask if they have questions before you begin.

28

Other Tools

ObservationsChecklists (process documentation)Attendance logsCase recordsExisting databases

29

Parting Tips

Keep it simpleBalance simplicity with precisionFocus on interpretationBlend methods if appropriate

30

31

Questions?Please contact:

Trina WillardPrincipalKnowledge Advisory Group

804-564-6969Trina@KnowledgeAdvisoryGroup.com