may 5, 2015 strategies for evaluation data collection eric graig, ph.d
TRANSCRIPT
![Page 1: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/1.jpg)
April 18, 2023
Strategies for Evaluation Data Collection
Eric Graig, Ph.D.
![Page 2: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/2.jpg)
Slide 2Data Collection for Evaluation
Session Objectives
Participants will understand the factors to consider when selecting data collection methods
Participants will become familiar with the data collection methods most often used in evaluation studies
Participants will leave with an understanding of the resources Innovation network and LSC are providing to help develop effective data collection tools
![Page 3: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/3.jpg)
Slide 3Data Collection for Evaluation
More Information
http://www.innonet.org/services/LSC_Train
![Page 4: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/4.jpg)
Slide 4Data Collection for Evaluation
Starting Points
Evaluation questions Evaluation plan
![Page 5: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/5.jpg)
Slide 5Data Collection for Evaluation
Choosing a Data Collection Strategy
Process evaluation Outcome evaluation Organizational capacity
Budget Expertise Staff time
Desired / Required N
Experiments Surveys Standardized instruments Interviews Focus groups Review of existing records
![Page 6: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/6.jpg)
Slide 6Data Collection for Evaluation
Integrate Data Collection into Operations
The best data collection strategy is one that is seamlessly integrated into you day-to-day operations.
Think of your organization as an information system. Where in your workflow can program evaluation information be best captured?
![Page 7: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/7.jpg)
Slide 7Data Collection for Evaluation
Be Systematic
Effective evaluation requires a close attention to detail. You needn’t be scientific but you should strive to be systematic.
Record keeping is primary. Encourage your left-brain to create forms and
protocols. Write everything down.
![Page 8: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/8.jpg)
Slide 8Data Collection for Evaluation
Example 1
![Page 9: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/9.jpg)
Slide 9Data Collection for Evaluation
Example 2
![Page 10: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/10.jpg)
Slide 10Data Collection for Evaluation
When to Use Surveys
When your evaluation questions are well defined and the range of possible answers is limited
When you need to collect data from a fairly large number of people and you have access to them and can reasonably expect them to cooperate with the survey
When you have the ability to process and analyze the relatively large amount of data you will collect
![Page 11: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/11.jpg)
Slide 11Data Collection for Evaluation
Kinds of Survey Questions
Knowledge questions Attitude questions Behavior questions Subject characteristics
![Page 12: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/12.jpg)
Slide 12Data Collection for Evaluation
Generating Survey Questions
Evaluation QuestionsEvaluation Sub-questionsOperational QuestionsIndividual Survey Items
![Page 13: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/13.jpg)
Slide 13Data Collection for Evaluation
Generating Survey Questions
Evaluation Questions
Evaluation Sub-questions
Operational Questions
Individual Survey Items
•What was learned?•How was identity affected?
•Were students satisfied?
•Satisfied w/ registration proc.•Satisfied with support received•Satisfied w/ course component
•Relevance of content to visits•Quality of the instructors
•Convenience of class schedule
•Right days•Right time
•Right class length
![Page 14: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/14.jpg)
Slide 14Data Collection for Evaluation
Writing Survey Questions
Question formats Multiple choice Ordinal scales Multi-punch questions
Number of choices provided Even number for a scale
Strive for clarity Use appropriate language Avoid words with multiple meanings Be careful around dates and amounts
![Page 15: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/15.jpg)
Slide 15Data Collection for Evaluation
Creating Neutral Questions
Dealing with obviously sensitive topics Dealing with less obviously sensitive topics
How often do you attend religious services?
1. Once a week or more2. Several times a month3. About once a month4. Several times a year5. Never
Some people attend religious services once or several times each week. Others attend only several times a year or not at all. Thinking back over the last year, how often would you say you attended religious services?
![Page 16: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/16.jpg)
Slide 16Data Collection for Evaluation
Survey Delivery
In-person paper and pencil Mail paper and pencil Internet Telephone
![Page 17: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/17.jpg)
Slide 17Data Collection for Evaluation
Questions
![Page 18: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/18.jpg)
Slide 18Data Collection for Evaluation
When to Use Interviews
When you are trying to understand someone else’s subjective experience.
When your evaluation questions are more open-ended and the range of possible answers is large.
When you don’t need to generalize your findings to a larger group.
![Page 19: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/19.jpg)
Slide 19Data Collection for Evaluation
Interview Questions
Avoid questions that lead to ‘motive talk’ Try to use:
Could you tell me about that Please tell me more about that How did that work / come about / feel
Ask interviewees to provide a narrative account of what happened or how they felt
![Page 20: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/20.jpg)
Slide 20Data Collection for Evaluation
Tips for Interviews
Spend time on small talk in order to make the interviewee feel comfortable
Use a written set of questions with probes Follow the protocol but remain open to
exploring other themes as they arise Record if at all possible
![Page 21: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/21.jpg)
Slide 21Data Collection for Evaluation
When to Use Focus Groups
When you are trying to understand subjective experience
When your evaluation questions are more open-ended
When you sense that respondents may be uncomfortable in a one-on-one interview
When your evaluation questions require you to understand a group dynamic
![Page 22: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/22.jpg)
Slide 22Data Collection for Evaluation
Tips for Focus Groups
Same as for interviews Plan on 8 to 10 participants and accept that one
or two may not show up Get a good mix of participants Learn how to prevent particular participants from
dominating the group and how to encourage wall flowers to speak up
Consider an introductory exercise to get the people comfortable with one another
![Page 23: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/23.jpg)
Slide 23Data Collection for Evaluation
Questions
![Page 24: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/24.jpg)
Slide 24Data Collection for Evaluation
When to Program Records
When you have them and can access them When you can vouch for their accuracy and
reliability and when you know they are complete
When you can understand them
![Page 25: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/25.jpg)
Slide 25Data Collection for Evaluation
Tips for Analyzing Program Records
Very their accuracy Find out what, if anything, is missing Be systematic Plan on making recommendations about how
to improve them to make them more useful for evaluation
![Page 26: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/26.jpg)
Slide 26Data Collection for Evaluation
Questions
![Page 27: May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D](https://reader035.vdocument.in/reader035/viewer/2022062713/56649cc05503460f9498688d/html5/thumbnails/27.jpg)
Slide 27Data Collection for Evaluation
Additional Resources
InnoNet website www.innonet.org Asynchronous material Synchronous course One-on-one technical assistance