community-based evaluation methods
TRANSCRIPT
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 1/56
Community-Based
Evaluation Methods& Data Analysis
Amy CarrollApril 22, 2005
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 2/56
Different Types of Data
Opinions, priorities
Aspirations,motivations
Level of awareness,knowledge, attitudes orbeliefs
Behaviors, practices
Assets, skills
Networks, associations
Needs, fears, problems,concerns
Demographic
Services or resourcesprovided
Resident utilization of
services or resourcesprovided
Numbers or rates of disease, illness,disability, injuries
Sales transactions,purchases
Policies
Pictures, other visuals
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 3/56
Quantitative vs. QualitativeData
Quantitative: Collected in the form of
numbers or percentages
Closed-ended questions Answers who?, what?,
when? and where?
Can demonstrate causeand effect
Can “represent” apopulation
Cannot collect newideas or responses, only
those considered aheadof time
Qualitative: Collected in the form of
words, concepts,
themes, or categories Open-ended questions
Answers how?, why?
Can provide richer,more in-depth data
Can provide data in arespondent’s ownwords
Can explore new ideasin a dynamic and
unstructured way
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 4/56
Common Community-BasedCommon Community-Based
Evaluation MethodsEvaluation Methods
• Secondary dataSecondary data
• Activity trackingActivity tracking
• Client trackingClient tracking
• Mapping/ PhotovoiceMapping/ Photovoice
•
SurveysSurveys• Focus groupsFocus groups
• Key informant interviewsKey informant interviews
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 5/56
Which of these methodsWhich of these methods
collect quantitative data?collect quantitative data?
Which collect qualitativeWhich collect qualitative
data?data?
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 6/56
Secondary DataSecondary Data
• County health departmentCounty health department• Vital statisticsVital statistics Birth & death certificatesBirth & death certificates• City or county governmentCity or county government• Schools (including school nurse records)Schools (including school nurse records)• Hospital or clinic recordsHospital or clinic records• Local health promotion, health ed or advocacyLocal health promotion, health ed or advocacy
agenciesagencies
• Private foundations which fund communityPrivate foundations which fund communityprograms Universities or local communityprograms Universities or local communitycollegescolleges
• Local surveys or research efforts going on inLocal surveys or research efforts going on inyour community??your community??
• Data you already collect!Data you already collect!
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 7/56
Activity TrackingActivity Tracking
• Tracking and documenting program Tracking and documenting program
activity implementationactivity implementation
• Tracking and documenting number of Tracking and documenting number of people reachedpeople reached
• Process evaluation method, butProcess evaluation method, but
important for outcome evaluationimportant for outcome evaluation
• May include: weekly/monthly programMay include: weekly/monthly program
implementation log, sign-in sheets atimplementation log, sign-in sheets at
classes or health fairs, counts of classes or health fairs, counts of
materials/ brochures distributed, etc.materials/ brochures distributed, etc.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 8/56
Client TrackingClient Tracking
• Tracking and documenting encounters Tracking and documenting encounterswith clients; describing the encounterwith clients; describing the encounter
and client characteristicsand client characteristics• More than just countingMore than just counting
• Can be used for processCan be used for process and and outcomeoutcome
evaluationevaluation• May include: outreach encounter form,May include: outreach encounter form,
home visit form, clinic intake form, etc.home visit form, clinic intake form, etc.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 9/56
Mapping/ PhotovoiceMapping/ Photovoice
• Data can also include maps and photosData can also include maps and photos →→ don’tdon’t
just have to be numbers or words just have to be numbers or words
• Maps and photos can provide visual imagery toMaps and photos can provide visual imagery to
give meaning to quantitative datagive meaning to quantitative data
• They can show graphically what cannot be They can show graphically what cannot be
described with numbers or words, such as thedescribed with numbers or words, such as the
conditions of a neighborhood or programconditions of a neighborhood or program
facilities, or a community’s proximity to parkfacilities, or a community’s proximity to parkspace or health services.space or health services.
• Make sure your maps and photos are clear andMake sure your maps and photos are clear and
illustrate the points you would like to makeillustrate the points you would like to make
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 10/56
SurveysSurveys
• Surveys are composed of carefullySurveys are composed of carefullycrafted questions on a topic or issue forcrafted questions on a topic or issue forthe purpose of revealing informationthe purpose of revealing information
about community residents or servicesabout community residents or servicesthey utilize.they utilize.
• The most commonly used survey The most commonly used survey
methods are:methods are:
1.1. Mailed SurveysMailed Surveys
2.2. Hand-out SurveysHand-out Surveys
3.3. Face to Face SurveysFace to Face Surveys
4.4. Telephone Interview Surveys Telephone Interview Surveys
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 11/56
Planning A SurveyPlanning A Survey
• Determine your purposeDetermine your purpose• Determine if the survey data collectionDetermine if the survey data collection
is appropriateis appropriate• Select your target audienceSelect your target audience• Select a survey methodSelect a survey method• Design survey questionnaireDesign survey questionnaire• Pilot test surveyPilot test survey• Publicize and distribute surveysPublicize and distribute surveys
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 12/56
Determine the Data YouDetermine the Data You
NeedNeed
• Draft your research questions – whatDraft your research questions – what
you want to learn from theseyou want to learn from these
interviews.interviews.• Identify the data you will need toIdentify the data you will need to
collect in order to answer thesecollect in order to answer these
questions.questions.
• The type of data you are looking for The type of data you are looking for
will help you identify the best personswill help you identify the best persons
in the community to survey, and thein the community to survey, and the
best survey questions to ask them.best survey questions to ask them.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 13/56
Target PopulationTarget Population
The following are two steps in The following are two steps indetermining your survey targetdetermining your survey targetpopulation.population.
1)1) Define your target population --Define your target population --identify the group of individuals fromidentify the group of individuals fromyour community from who you want toyour community from who you want tocollect information (single mothers,collect information (single mothers,high school students, the uninsured,high school students, the uninsured,etc.)etc.)
2)2) Identify the geographic area of your Identify the geographic area of your target population.--target population.--Where is your groupWhere is your groupof interest located?of interest located? What are theWhat are the
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 14/56
Select A Survey MethodSelect A Survey Method
Self Administered:Self Administered: These surveys are filled These surveys are filled
out by respondents themselves without theout by respondents themselves without the
assistance of trained interviewers.assistance of trained interviewers.
A. Mail SurveysA. Mail Surveys
B. Drop-off SurveysB. Drop-off Surveys
Administered by Interviewer:Administered by Interviewer: These surveys These surveys
are filled out with the assistance of trainedare filled out with the assistance of trained
interviewers.interviewers.
A.A. Face to Face SurveysFace to Face Surveys
B. Telephone SurveysB. Telephone Surveys
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 15/56
Design SurveyDesign Survey
QuestionnaireQuestionnaire
• Your survey questions should be in direct Your survey questions should be in direct
relationship with what you want to learn from yourrelationship with what you want to learn from your
target population.target population.
• Asking unrelated questions will interrupt the flow of Asking unrelated questions will interrupt the flow of the survey and tire out your respondents.the survey and tire out your respondents.
• Be brief and strategic when designing questions!Be brief and strategic when designing questions!
• Open-ended questions are great for answeringOpen-ended questions are great for answering
“why” “why” questionsquestions, but only include one or two., but only include one or two.
• Motivating people to fill out your survey will helpMotivating people to fill out your survey will help
you collect the data you need – this can include ayou collect the data you need – this can include a
benefits statement or incentives such as entry intobenefits statement or incentives such as entry into
a lottery or drawing.a lottery or drawing.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 16/56
Pilot Test SurveyPilot Test Survey
• It is always a good idea to pilot testIt is always a good idea to pilot testthe survey with a small group of the survey with a small group of people from your target population.people from your target population.
• The purpose for piloting the survey is The purpose for piloting the survey isto get feedback on the survey questionto get feedback on the survey questionand structure before implementing it.and structure before implementing it.
• This will ensure you identify potentially This will ensure you identify potentiallyconfusing, offensive, leading, or loadedconfusing, offensive, leading, or loadedquestions from your survey.questions from your survey.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 17/56
Publicize and DistributePublicize and Distribute
SurveysSurveys
Advertise your survey a couple of weeksAdvertise your survey a couple of weeks
before distribution in the community.before distribution in the community.
Publicizing what you are trying to do andPublicizing what you are trying to do andhow the information collected will benefithow the information collected will benefit
the community may help increase yourthe community may help increase your
response rate.response rate.
Create a survey distribution timeline andCreate a survey distribution timeline and
stick to it - postponing survey distributionstick to it - postponing survey distribution
date may limit the time you have setdate may limit the time you have set
aside for data analysis.aside for data analysis.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 18/56
Advantages andAdvantages and
Disadvantages of SurveysDisadvantages of Surveys
AdvantagesAdvantages Data can be collectedData can be collected
from a lot of from a lot of
respondents easier thanrespondents easier thanany other methodany other method
Can get a large enoughCan get a large enough
sample that can besample that can be
representative of therepresentative of the
larger populationlarger population Findings can beFindings can be
generalized to the largergeneralized to the larger
populationpopulation
Can cover a lot of topicsCan cover a lot of topics
Can easily compareCan easily compare’’
DisadvantagesDisadvantages Survey instrument mustSurvey instrument must
be carefully constructedbe carefully constructed
to avoid leadingto avoid leadingquestions, and to makequestions, and to make
sure the appropriatesure the appropriate
responses are availableresponses are available
Response rates can beResponse rates can be
low for self-administeredlow for self-administeredsurveys, especiallysurveys, especially
mailed onesmailed ones
Response will be low if Response will be low if
survey is too longsurvey is too long
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 19/56
Focus GroupsFocus Groups
Focus group are a qualitativeFocus group are a qualitative
research method designed toresearch method designed to
learn more about how peoplelearn more about how people
think, feel, or make decisionsthink, feel, or make decisions
(attitudes, perceptions, opinions,(attitudes, perceptions, opinions,
experiences) through focusedexperiences) through focuseddiscussions.discussions.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 20/56
Focus Groups (cont.)
A series of discussions involving 8-12 people,selected to share their perceptions of adefined topic
Stimulates participants to share theiropinions openly in a group discussion
Captures rich data in participants’ own words
Flexible to capture new ideas and issues Can be difficult to summarize and interpret
results across groups
Need to be creative when recruiting busypeople
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 21/56
Planning a Focus GroupPlanning a Focus Group
• Determine the data you needDetermine the data you need
• Determine and select your targetDetermine and select your target
audienceaudience• Develop a focus group discussionDevelop a focus group discussion
guideguide
•
Schedule focus groupsSchedule focus groups• Recruit focus group participantsRecruit focus group participants
• Recruit moderator and note-takerRecruit moderator and note-taker
• Plan and conduct focus groupsPlan and conduct focus groups
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 22/56
Determine the Data YouDetermine the Data You
NeedNeed
• Draft your research questions – whatDraft your research questions – what
you want to learn from these focusyou want to learn from these focus
groups.groups.• Identify the data you will need toIdentify the data you will need to
collect in order to answer thesecollect in order to answer these
questions.questions.
• The type of data you are looking for The type of data you are looking for
will help you identify the best personswill help you identify the best persons
to recruit for your focus groups, as wellto recruit for your focus groups, as well
as the best questions to develop foras the best questions to develop for
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 23/56
Target AudienceTarget Audience
• To whom do you want to talk? To whom do you want to talk?• What types of experiences andWhat types of experiences and
perspectives you are hoping to captureperspectives you are hoping to capturein your discussion?in your discussion?• Design and use aDesign and use a screening toolscreening tool toto
recruit participants: arecruit participants: ask questions thatsk questions that
qualify people to participate.qualify people to participate.• It is important that you includeIt is important that you includeparticipants who will feel comfortableparticipants who will feel comfortabletalking to each other.talking to each other.
• Participants should not know eachParticipants should not know each
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 24/56
How Many Focus Groups?How Many Focus Groups?
• Conduct 2-3 focus groups for each sub-Conduct 2-3 focus groups for each sub-population of interest (i.e. gender, age,population of interest (i.e. gender, age,race/ethnicity, language, etc.)race/ethnicity, language, etc.)
• 2-3 ensures that the perspectives and2-3 ensures that the perspectives andexperience you capture are moreexperience you capture are morerepresentative of the larger populationrepresentative of the larger population
and not just by “chance”.and not just by “chance”.• Length: 1.5 hrs.-2 hrs.Length: 1.5 hrs.-2 hrs.
• Focus Group Size: 6-12 participantsFocus Group Size: 6-12 participants
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 25/56
Focus Group DiscussionFocus Group Discussion
GuideGuide
• Discussion guide:Discussion guide: An outlined script toAn outlined script to
help guide the moderator and focus thehelp guide the moderator and focus the
discussiondiscussion• Organization:Organization:
IntroductionIntroduction
PurposePurposeGround rulesGround rules
Focus group questionsFocus group questions
Closing statementsClosing statements
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 26/56
Discussion Guide QuestionsDiscussion Guide Questions
• Introductory questionsIntroductory questions: getting people to: getting people tothink about the general topic.think about the general topic.
• Transition questionsTransition questions: questions that get: questions that getparticipants to logically transition between theparticipants to logically transition between the
introduction topics and the ideas contained inintroduction topics and the ideas contained inthe key questions.the key questions.
• Key questionsKey questions: 2-5 questions important to: 2-5 questions important togetting the information you have set out togetting the information you have set out tocollect.collect.
• Ending questionsEnding questions: questions that provide: questions that provideclosure and reflection.closure and reflection.
• Summary questionSummary question: note-taker summarizes: note-taker summarizesthe major themes heard throughout thethe major themes heard throughout thediscussion in a short oral summary (2-3discussion in a short oral summary (2-3
minutes), then will ask the participants if theminutes), then will ask the participants if thesummar covered all the ma or oints.summary covered all the major points.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 27/56
Scheduling Focus GroupsScheduling Focus Groups
• Location:Location: The focus groups should The focus groups shouldtake place in atake place in a convenient andconvenient andaccessible location. It is also importantaccessible location. It is also important
to select a neutral environment.to select a neutral environment.• Time of day:Time of day: most convenient formost convenient for
your target audience. Set time earlieryour target audience. Set time earlierthan intend to start.than intend to start.
• Incentives:Incentives: Food or refreshments,Food or refreshments,money or gifts, drawings, child care,money or gifts, drawings, child care,transportation, anything that istransportation, anything that ismeaningful to those communitymeaningful to those community
members.members.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 28/56
Conducting & ModeratingConducting & Moderating
• The discussion guide will help guide the The discussion guide will help guide themoderator with the questions you wantmoderator with the questions you wantanswered.answered.
• Moderator Responsibilities:Moderator Responsibilities: setting upsetting upground rules, keeping the discussion focusedground rules, keeping the discussion focusedand on track, drawing out comments fromand on track, drawing out comments fromquiet participants, and asking for clarificationquiet participants, and asking for clarificationor additional information from participantsor additional information from participantswhen needed.when needed.
• Note-taker Responsibilities:Note-taker Responsibilities: takestakescomprehensive notes, operates audiocomprehensive notes, operates audioequipment (tape-recorder), keeps track of equipment (tape-recorder), keeps track of
time, handles environmental conditions andtime, handles environmental conditions and
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 29/56
Skills of the ModeratorSkills of the Moderator
Extremely important to select moderators basedExtremely important to select moderators based
on skills and experience:on skills and experience: • Comfortable/familiar with leading meetings orComfortable/familiar with leading meetings or
groupsgroups• Able to exercise mild control over the groupAble to exercise mild control over the group
• Able to guide conversation back on targetAble to guide conversation back on target
• Maintain group enthusiasm and interestMaintain group enthusiasm and interest
• Curious about the topicCurious about the topic• Respects participants and their commentsRespects participants and their comments
• Creates and maintains a comfortableCreates and maintains a comfortableenvironmentenvironment
• Adequate background knowledgeAdequate background knowledge
• Good oral and written skillsGood oral and written skills
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 30/56
Advantages andAdvantages and
Disadvantages of FocusDisadvantages of Focus
GroupsGroups AdvantagesAdvantages FlexibleFlexible
Captures rich, in-Captures rich, in-
depth datadepth data Immediate resultsImmediate results
Encourages andEncourages and
stimulates individualsstimulates individuals
to share more openlyto share more openly Data can be combinedData can be combined
with quantitative datawith quantitative data
to provide a completeto provide a complete
picture about an issuepicture about an issue
DisadvantagesDisadvantages May be challenging toMay be challenging to
recruit participantsrecruit participants
Need to schedule atNeed to schedule atleast 2-3 focus groupsleast 2-3 focus groups
to capture diversityto capture diversity
Difficult to generalizeDifficult to generalize
results to the largerresults to the largerpopulation because of population because of
small numbers of small numbers of
participantsparticipants
Difficult to compareDifficult to compare
results across groupsresults across groups
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 31/56
Key InformantKey Informant
InterviewsInterviews• Key informant interviews are qualitative, in-Key informant interviews are qualitative, in-
depth interviews with key informantsdepth interviews with key informants
• Key informants are individuals who knowKey informants are individuals who know
what is going on in the communitywhat is going on in the community
(community leaders, professionals,(community leaders, professionals,
residents, advocates, elected officials,residents, advocates, elected officials,
doctors, etc.)doctors, etc.)
• The following are two common techniques The following are two common techniques
used to conduct key informant interviews:used to conduct key informant interviews:
1.1. Telephone interviews Telephone interviews
2.2.
Face-to face interviewsFace-to face interviews
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 32/56
Planning Key InformantPlanning Key Informant
InterviewsInterviews
• Determine the data you needDetermine the data you need• Determine target population andDetermine target population and
brainstorm possible key informantsbrainstorm possible key informants
• Select key informantsSelect key informants• Select interview techniqueSelect interview technique• Develop an interview guideDevelop an interview guide• Determine documentation methodDetermine documentation method• Select designated interviewer(s)Select designated interviewer(s)• Conduct key informant interviewsConduct key informant interviews• Compile and organize key informantCompile and organize key informant
interview datainterview data
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 33/56
Determine the Data YouDetermine the Data You
NeedNeed
• Draft your research questions – whatDraft your research questions – what
you want to learn from theseyou want to learn from these
interviews.interviews.• Identify the information you will needIdentify the information you will need
to collect to answer these questions.to collect to answer these questions.
• The type of data you are looking for The type of data you are looking forwill help you identify the best personswill help you identify the best persons
in the community to interview, and thein the community to interview, and the
best questions to ask them.best questions to ask them.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 34/56
Target Audience and KeyTarget Audience and Key
InformantsInformants
• Determine your target population (programDetermine your target population (programclientele, racial/ethnic minority groups,clientele, racial/ethnic minority groups,adolescents, women, the elderly, HIV+adolescents, women, the elderly, HIV+persons, etc.)persons, etc.)
• Brainstorm and create a list of possible keyBrainstorm and create a list of possible keyinformants who are knowledgeable andinformants who are knowledgeable andclosely involved with your target populationclosely involved with your target population..
• Select key informants:Select key informants:
• Who have first-hand knowledge about the targetWho have first-hand knowledge about the targetpopulation, the program, and the issues you arepopulation, the program, and the issues you aretrying to investigatetrying to investigate..
• Who represent different experiences, backgrounds,Who represent different experiences, backgrounds,and viewpoints.and viewpoints.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 35/56
Select Interview TechniqueSelect Interview Technique
Telephone interviews (15-25 min): Telephone interviews (15-25 min):• May be the most convenient and least timeMay be the most convenient and least time
intensive way to interview busy key informants.intensive way to interview busy key informants.
• Disadvantage: not having the personalizedDisadvantage: not having the personalizedinteraction otherwise possible through a face-to-interaction otherwise possible through a face-to-
face interview.face interview.
Face-to-face interviews (20-30 min):Face-to-face interviews (20-30 min):• Most frequently used format. But time intensive.Most frequently used format. But time intensive.• Advantages: provides a free-exchange of ideasAdvantages: provides a free-exchange of ideas
and lends itself to ask more complex questions.and lends itself to ask more complex questions.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 36/56
Develop Interview GuideDevelop Interview Guide
IntroductionIntroduction- Introduce yourself and your- Introduce yourself and your
purpose and/or benefits statement.purpose and/or benefits statement.
Key questionsKey questions- Draft 5-10 questions- Draft 5-10 questions
important to getting the information youimportant to getting the information youhave set out to collect.have set out to collect.
Probing questions –Probing questions – Probing questionsProbing questions
encourage participants to reflect moreencourage participants to reflect more
deeply on the meaning of their comments.deeply on the meaning of their comments.Closing questionClosing question—Provide an opportunity for—Provide an opportunity for
the key informant to give any additionalthe key informant to give any additional
information or comments.information or comments.
SummarySummary—Quickly summarize the major—Quickly summarize the major
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 37/56
Determine DocumentationDetermine Documentation
MethodMethod
Note-taking:Note-taking: manually write the keymanually write the key
informant’s comments whileinformant’s comments while
conducting the interview, on a copy of conducting the interview, on a copy of the interview guide.the interview guide.
Tape-recording:Tape-recording: use a tape recorderuse a tape recorder
to document what key informants say.to document what key informants say.
This approach may allow the This approach may allow theinterviewer to freely engage in theinterviewer to freely engage in the
conversation without worrying aboutconversation without worrying about
note-taking.note-taking.
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 38/56
Conduct InterviewsConduct Interviews
• Thank respondents for their time and Thank respondents for their time and
stress the importance of the interview.stress the importance of the interview.
• Interviewers should listen carefully forInterviewers should listen carefully forrecurring and new opinions or beliefs.recurring and new opinions or beliefs.
• In order to compare with otherIn order to compare with other
interviews and identify emerginginterviews and identify emergingthemes, it is important to get answersthemes, it is important to get answers
fromfrom every every person interviewed on keyperson interviewed on key
questions.questions.
Ad t dAd t d
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 39/56
Advantages andAdvantages and
Disadvantages of KeyDisadvantages of Key
Informant InterviewsInformant Interviews AdvantagesAdvantages Rich data can beRich data can be
gathered relativelygathered relatively
cheaply & easilycheaply & easily
Allows interviewer toAllows interviewer to
establish rapport with theestablish rapport with the
respondent, clarifyrespondent, clarify
questions, and draw outquestions, and draw out
responsesresponses
Allows for discussion of Allows for discussion of
topic without grouptopic without group
dynamic of focus groupsdynamic of focus groups
Provides an opportunityProvides an opportunity
to build relationshipsto build relationships
with importantwith important
DisadvantagesDisadvantages Selecting the “right” keySelecting the “right” key
informants may be difficultinformants may be difficult
so they represent diverseso they represent diverse
backgrounds and viewpointsbackgrounds and viewpoints May be challenging to reachMay be challenging to reach
and schedule interviews withand schedule interviews with
busy and/or hard-to-reachbusy and/or hard-to-reach
respondentsrespondents
Difficult to generalize resultsDifficult to generalize results
to the larger populationto the larger population
unless interviewing manyunless interviewing many
key informantskey informants
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 40/56
Things to Consider Before You Select Your Methods
• What data collection method(s) would bestcollect the type of data you need from yourtarget population
• What data collection method(s) would helpyou to best answer your researchquestion(s)?
•
Think of your available resources: time,staffing, skills, funding, computer technology
• Prior experience
• Potential benefits
• Potential challenges
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 41/56
How to Understandthe Data You Collect
(Data Analysis)
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 42/56
Why It’s a Good Idea to PlanAhead for Data Analyses
To determine if the data you collectare practical for analyses
To review the appropriateness of yourchosen data collection method(s)
To inform the data collection
instrument you develop To decide if you will need help with
analyses
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 43/56
1: Check the Data
Make sure it’s all there
Make sure it makes sense
Catch any mistakes that happened whilefilling out the survey or entering the data
Perform data quality checks throughout yourdata collection process
Common mistakes:– A missing or incomplete response
– An impossible response
– A response unrelated to the question
– A response that contradicts an earlier response
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 44/56
2: Go Back to theResearch Questions
Be sure to analyze only the data that willhelp you answer your research question(s)
Don’t get side-tracked into analyzing otherpieces of data with your limited time andresources
It is normal to collect more data than you willneed for your evaluation
You can return to any other interesting dataafter completing your evaluation
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 45/56
3: Reduce the Amount of Data
Save your data in its original form sothat you can return to it if you need to
Look at it in small chunks at a time Eliminate any irrelevant data from
your analysis
Create summary documents to help
you “eyeball” the data more easily tolook for patterns or themes:
– Numeric spreadsheet
– Qualitative summary document
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 46/56
4: Analyze the DataQuantitative Data Analysis: Looking
for Patterns in the Data Calculate averages
– Used with “continuous” data: infinite numberof values
– Add numbered responses– Divide by number of responses
– 2 + 4 + 6 + 10 = 22 22/4 = 5.5
Count frequencies
– Used with “discrete” data: set number of response categories
– Count number of responses
– Number “yes”, number “no”
– Number of choice “A”, Number of choice “B”,
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 47/56
4: Analyze the DataQuantitative Data Analysis: Looking
for Patterns in the Data Calculate proportions
– Is standardized– Equation: % = # of responses/ total # of
respondents X 100– Example: 45 females/ 100 respondents X 100= 45% of the respondents were female
Calculate rates– Similar to proportions; also standardized
– Easier to compare– Equation: Rate = freq of event in pop/ total pop
X 100,000– Example: 15 cases of lung cancer/ population of
3,500 X 100,000 = 428 persons per 100,000have lung cancer
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 48/56
4: Analyze the DataQuantitative Data Analysis: Looking
for Patterns in the Data Compare averages, frequencies, proportions,
rates
Compare data from different populations:
– County vs. County– County vs. State
– State vs. US
Compare data from different segments of a
population:– Male vs. female
– African American, white, Latino, Asian, AmericanIndian, etc.
– Children, teens, adults, seniors
– Clientele of your program vs. those who are not
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 49/56
4: Analyze the DataQuantitative Data Analysis: Looking
for Patterns in the Data
Present the data in different ways to seeadditional patterns and relationships:– Chart/ table – shows averages, counts, proportions,
or rates side-by-side– Pie graph – demonstrates percentages of the whole– Bar graph – compares quantities– Line graph – shows trends over time
Determine your findings– Interesting results?
– Interesting patterns or relationships?
– These are your findings
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 50/56
Exercise 5.4A(Using Step 5 of Performing a
Community Assessment )
Look at sample survey and
sample spreadsheet in AppendixC
Answer questions on page 51 and
52
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 51/56
Step 4: Analyze the DataQualitative Data Analysis: Looking
for Themes in the Data Qualitative data analysis can be deceptively
trickier– Qualitative data is by nature “bigger”
– Analysis requires more analytical thinking andinterpretation
Qualitative data is more open to “bias”– Numbers don’t lie– Analysts bring their values, assumptions and
opinions– Analysts may think they “know” how people feel
Include at least 2, preferably 3 people ineach stage of data analysis to avoid bias
Create as structured of a rocess as ossible
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 52/56
Step 4: Analyze the DataQualitative Data Analysis: Looking
for Themes in the Data1. Read through all of the data at least twice
– Stay “close” or “grounded” in the data
2. Create categories– Focus group or interview questions
– Or specific assessment health issues
3. List themes that emerge from discussion– Sub-categories
– Ideas
4. Rank order according to frequency of appearance
5. Summarize discussion around each theme
6. Determine our findin s
Exercise 5 4B
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 53/56
Exercise 5.4B
(Using Step 5 of Performing a
Community Assessment )
Look at sample focus group
discussion transcript in AppendixC
Answer questions on page 56
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 54/56
5: Verify Findings
You now have:
– Some patterns and relationships you found in yourquantitative data
– Some themes and categories you found in yourqualitative data
Based on the results you tallied and summarized,pull out the main findings of each method youused
Verify these findings by re-tallying and re-summarizing the data to make sure you get thesame results
You can also verify your findings by comparing
the results reached by 2 analysts independently
8/14/2019 Community-Based Evaluation Methods
http://slidepdf.com/reader/full/community-based-evaluation-methods 55/56
6: Interpret Findings andDraw Conclusions
Determine what interpretations can be drawn fromeach finding
Invite different perspectives from different Partners and
community members, as this will make yourconclusions stronger
Are the results similar to what you expected? If not,discuss why you think they are different
Brainstorm alternative explanations for your results tomake sure you have considered all possibilities
Make sure the conclusions answer the originalassessment questions
Draw conclusions and recommendations that can be
shared with external audiences
Exercise 5 6