how to conduct surveys - home | 2017 - uoc exams | · pdf file · 2017-01-28how to...
TRANSCRIPT
HowToConductSurveys6Edition
ThisbookisdedicatedtotheonesIlove:
JohnC.BeckandIngvard
HowToConductSurveysAStep-by-StepGuide
6Edition
ArleneFinkUniversityofCaliforniaatLosAngeles,TheLangleyResearch
Institute
FORINFORMATION:
SAGEPublications,Inc.
2455TellerRoad
ThousandOaks,California91320
E-mail:[email protected]
SAGEPublicationsLtd.
1Oliver’sYard
55CityRoad
LondonEC1Y1SP
UnitedKingdom
SAGEPublicationsIndiaPvt.Ltd.
B1/I1MohanCooperativeIndustrialArea
MathuraRoad,NewDelhi110044
India
SAGEPublicationsAsia-PacificPte.Ltd.
3ChurchStreet
#10-04SamsungHub
Singapore049483
Copyright©2017bySAGEPublications,Inc.
Allrightsreserved.Nopartofthisbookmaybereproducedorutilizedinanyformorbyanymeans,electronicormechanical,includingphotocopying,recording,orbyanyinformationstorageandretrievalsystem,withoutpermissioninwritingfromthepublisher.
PrintedintheUnitedStatesofAmerica
LibraryofCongressCataloging-in-PublicationData
Names:Fink,Arlene,author.
Title:Howtoconductsurveys:astep-by-stepguide/ArleneFink.
Description:Sixthedition.|LosAngeles:SAGE,[2017]|Includesbibliographicalreferencesandindex.
Identifiers:LCCN2015035702|ISBN9781483378480(pbk.:alk.paper)
Subjects:LCSH:Socialsurveys.|Educationalsurveys.
Classification:LCCHN29.F532017|DDC300.72/3—dc23LCrecordavailableathttp://lccn.loc.gov/2015035702
Thisbookisprintedonacid-freepaper.
AcquisitionsEditor:VickiKnight
eLearningEditor:KatieBierach
EditorialAssistant:YvonneMcDuffee
ProductionEditor:JaneHaenel
CopyEditor:KatharineBlankenship
Typesetter:C&MDigitals(P)Ltd.
Proofreader:TriciaCurrie-Knight
Indexer:KathyPaparchontis
CoverDesigner:MichaelDubowe
MarketingManager:NicoleElliott
BriefContentsPrefaceAbouttheAuthorChapter1.ConductingSurveys:EveryoneIsDoingItChapter2.TheSurveyForm:Questions,Scales,andAppearanceChapter3.GettingItTogether:SomePracticalConcernsChapter4.SamplingChapter5.SurveyDesign:EnvironmentalControlChapter6.AnalyzingandOrganizingDataFromSurveysChapter7.PresentingtheSurveyResultsIndex
DetailedContentsPrefaceAbouttheAuthorChapter1.ConductingSurveys:EveryoneIsDoingIt
OverviewWhatIsaSurvey?
Example:SurveystoMeetPolicyorProgramNeedsExample:SurveysinEvaluationsofProgramsExample:SurveysforResearch
WhenIsaSurveyBest?Example:FromanOverlyAmbitiousSelf-AdministeredQuestionnaireExample:SurveysCombinedWithOtherInformationSources
Self-AdministeredQuestionnairesandInterviews:TheHeartoftheMatterQuestionsandResponsesExample:Forced-ChoiceQuestionExample:Open-EndedQuestionSurveySampleandDesignPlanningforDataAnalysisPilotTestingResponseRateReportingResultsExample:TheLookofSurveyResults(1)Example:TheLookofSurveyResults(2)Example:TheLookofSurveyResults(3)
TheFriendlyCompetitionReliabilityandValidityUsefulnessorCredibilityofResultsCostsTheSpecialCaseofOnlineSurveysChecklistforDecidingtheBestUsesforOnlineSurveysGuidelinesforSurveyorsWhoWorkWithCommercialSurveyCompaniesTheSpecialCaseofCellorMobilePhonesMakingtheDecision
ASurveyContinuum:FromSpecifictoGeneralUseExample:SurveyWithaSpecificUseExample:SurveyWithaGeneralUse
Ethics,Privacy,andConfidentialityInformedConsentContentsofanInformed-ConsentFormTheInternetandEthicalSurveysExampleQuestionnaire:MaintaininganEthicallySoundOnlineSurveyExample:Informed-ConsentFormforanOnlineSurvey
ChildrenandSurveyEthicsExample:ChildAssentForm
InternationalSurveysFormalStandardsforSurveyEthicsSummingUpThinkAboutThisReferences
Chapter2.TheSurveyForm:Questions,Scales,andAppearanceOverviewTheContentIstheMessageDefinetheTermsSelectYourInformationNeedsorHypothesesMakeSureYouCanGettheInformationDoNotAskforInformationUnlessYouCanActonIt
Example:PlanforSurveyofSatisfactionWiththeYouthCenterWritingQuestions
Open-EndedandClosedQuestionsExample:Open-EndedQuestionExample:ClosedQuestionExample:Open-EndedQuestionforElementarySchoolTeachingProgramExample:ClosedQuestionforElementarySchoolTeachingProgramMakingtheDecision:Open-EndedVersusClosedQuestions
OrganizingResponsestoOpen-EndedSurveyItems:DoYouGetAnySatisfaction?Step1:AskingRespondents’OpinionsStep2:CodingLB/LLDataExampleLB/LL:ResponseCategoriesExampleLB/LL:ParticipantResponsesStep3:LB/LLDataExampleLB/LL:NumberofResponsesforEachCodeExampleLB/LL:Participants’ResponsePatternExampleLB/LL:SummaryofResponses
RulesforWritingClosedSurveyQuestionsExample:Item-WritingSkills—Length,Clarity,Abbreviations,andJargonExample:Item-WritingSkills—ConcreteQuestionsExample:Item-WritingSkills—SpecificityofQuestionsExample:Item-WritingSkills—HiddenBiasesExample:Item-WritingSkills—HiddenBiasesExample:Question-WritingSkills—VeryPersonalQuestionsExample:Question-WritingSkills—OneThoughtperQuestion
ResponsesforClosedQuestionsYesandNoExample:Yes-and-NoResponses
RatingScalesCategoricalorContinuous?WhatAboutOrdinal?Example:CategoricalRatingScaleExample:OrdinalRatingScaleOrdinalScalesExample:OrdinalScalesExample:SelectingtheNumberofCategories
Example:RankOrderScaleExample:ComparativeRatingScaleChecklistExample:ChecklistResponsesinWhichRespondentMustChooseOneFromaListofSeveralExample:ChecklistResponsesThatRespondentsAnswerYes,No,orDon’tKnowforEachIteminaList
ChildrenandSurveysOnlineSurveys
Example:SurveyProgressBarExample:ResponsetoOneQuestionDictatestheResponsetotheNextExample:ExplanationforChangingOnlineResponses
PlainandSimpleSurveyQuestionsandResponsesScaling
AdditiveScalesExample:ASurveyWithanAdditiveScaleExample:ASurveyofForeignLanguageSkillsDifferentialScalesExample:ScoringaDifferentialScaleSummatedScalesExample:CreatingaSummatedScaleforaSelf-EsteemSurveyExample:ScoringaSummatedScale
SummingUpThinkAboutThis
Chapter3.GettingItTogether:SomePracticalConcernsOverviewLengthCounts
Example:HowaSurvey’sCircumstancesCanInfluenceItsLengthGettingtheSurveyinOrder
Example:AnIntroductiontoaTelephoneSurveyandItsFirstQuestionExample:OrderingSurveyQuestions(1)Example:OrderingSurveyQuestions(2)Example:ProvidingTransitionsChecklisttoGuideQuestionOrder
QuestionnaireFormat:AestheticsandOtherConcernsResponseFormatExample:ResponseFormats
BranchingQuestions,ortheInfamous“Skip”PatternExample:SkipPatternsorBranchingQuestions
Administration:WhoGivesWhattoWhom?Self-AdministeredQuestionnairesChecklistforUsingSelf-AdministeredQuestionnairesInterviews
ReliabilityandValidity:TheQualityofYourSurveyIstheSurveyReliable?Example:InternalConsistencyCountsExample:InternalConsistencyDoesNotCount
IstheSurveyValid?SelectingandAdaptingSurveys
Example:ExcerptFromaRealOnlineServiceAgreementforFreeSurveyItems(NamesChanged)GuidelinesforFindingUseableandUsefulSurveysintheResearchLiteratureExample:SearchforSchoolDropoutSurveys:SampleFindingsFromtheWebofScienceExample:SampleReportsofReliabilityandValidityintheResearchLiterature
FindingSurveysontheWebExample:SearchforEducationSurveys:ASamplePageFromtheNationalCenterforEducationStatistics
TheSurveyIsPutonTrial:GuidelinesforPilotTestingAFar-ReachingWorld:Surveys,Language,andCulture
GuidelinesforTranslatingSurveysExample:QuestionAboutEthnicity:U.S.DepartmentofHealthandHumanServicesOfficeofMinorityHealth
SummingUpThinkAboutThisReferences
Chapter4.SamplingOverviewSampleSizeandResponseRate:WhoandHowMany?
Example:RandomandConvenienceSamplingRandomSamplingMethods
Example:SimpleRandomSampling(1)Example:NotRandomSamplingExample:SimpleRandomSampling(2)MakingtheDecision
StratifiedRandomSamplingExample:StratifiedRandomSamplingMakingtheDecision
SimpleRandomClusterSamplingExample:SimpleRandomClusterSamplingMakingtheDecision
SystematicSamplingMakingtheDecision
ConvenienceSamplesMakingtheDecision
OtherConvenienceSamplingMethodsExample:OtherConvenienceSamples
FindingtheSample:WhoIsIn?WhoIsOut?HowLargeShouldYourSampleBe?
TheStandardErrorStatisticalMethods:SamplingforTwoGroupsandanIntervention
Example:SampleSizeCalculationsforSamplingTwoGroupsandanInterventionSubgroups,Measures,Resources,andScheduleFiveQuestionstoAskWhenDeterminingSampleSizeExample:CalculatingSampleSizeinaSurveyofEmployeesinanExperimentaland
ControlGroupExample:PowertoDetectDifferencesExample:80%PowerandEffectExample:SampleSize,Effect,andPower
ResponseRateTipsforImprovingResponseRate
WeightingExample:CalculatingPoststratificationWeights
MarginofErrorandConfidenceLevelSampleSizeandtheMarginofError
SummingUpThinkAboutThisReferences
Chapter5.SurveyDesign:EnvironmentalControlOverviewWhichDesignsAreAvailable?
Example:SurveysWithDifferingDesignsCross-SectionalSurveyDesigns
Example:Cross-SectionalDesignMakingtheDecision
LongitudinalSurveysExample:LongitudinalSurveys:CohortDesignMakingtheDecision
ExperimentalSurveyDesignsExample:ANonrandomizedControlledTrialorQuasi-ExperimentalDesignExample:ANonrandomizedControlledTrialWithaLongitudinalDesignExample:ARandomizedControlledTrialorTrueExperimentExample:ARandomizedControlledTrialWithaLongitudinalDesignFactorialDesigns:SpecialCasesMakingtheDecision
OtherSurveyDesigns:NormativeandCaseControlNormativeSurveyDesignExample:NormativeDesignExample:NormativeDesign—ComparisontoaModelMakingtheDecision
CaseControlDesignMakingtheDecisionExample:CaseControlDesign
SurveyDesignValidityInternalValidityExternalValidity
Surveys,ResearchDesign,andInternalandExternalValidityExample:Cross-SectionalSurveyandThreatstoValidityExample:LongitudinalDesignandThreatstoValidityExample:RandomizedControlledTrialsandValidity
SurveysWithQualitativeData:ThreatstoInternalandExternalValidityExample:QualitativeStudiesandValidity
SummingUpThinkAboutThisReferences
Chapter6.AnalyzingandOrganizingDataFromSurveysOverviewWhatIsTypicalAnyway?SomeCommonlyUsedMethodsforAnalyzingSurveyData
DescriptiveStatisticsExample:PreschoolPurposesQuestionnaireExample:FrequencyofQuestionnaireResponsesExample:GroupedRatingsofPreschoolPurposesby50DirectorsAverages:Means,Medians,andModesExample:ComputingtheMedianforanEvenNumberofScoresExample:ComputingtheMedianforanOddNumberofScoresVariation:Range,Variance,andStandardDeviationCorrelationandRegressionExample:RankOrderCorrelationDifferencesBetweenGroupsStatisticalDifferencesStatisticalSignificance
SurveyingDifferences:UsualMethodsChi-SquareExample:Chi-SquareThetTestExample:tTestTheMann-WhitneyUTestExample:Mann-WhitneyUTestANOVARisksandOddsExample:OddsRatioandRelativeRisk
ToBeorNottoBe:StatisticianorQualitativeAnalyst?ContentAnalysis,Open-EndedResponses,andComments
Example:HypotheticalContentAnalysis:TeasingBoysandGirlsPuttingtheCartinFrontoftheHorse:SelectingAnalysisMethods
SevenQuestionstoAnswerBeforeChoosinganAnalysisMethodDataManagement
CreatingaCodeBookorOperationsManualExample:ExcerptFromtheCARPS,aSurveytoDetectBingeDrinkinginCollegeStudentsExample:ExcerptFromtheCodeBookfortheCARPS,aSurveytoDetectBingeDrinkingEstablishingReliableCodingMeasuringAgreement:TheKappaMeasuringAgreementBetweenTwoCoders:TheKappa(κ)StatisticReviewingSurveysforMissingDataEnteringtheDataExample:SurveyResponsesFromSixPeopleCleaningtheDataValidatingSurveyData
SummingUpThinkAboutThis
Chapter7.PresentingtheSurveyResultsOverviewReproducingtheQuestionnaire
Example:ReportingResultsWiththeQuestionnaireExample:OnlineSurveyStatisticsinRealTime
UsingTablesExample:ShellTableDescribingChildreninTwoSchoolsExample:ShellTableforComparingChildreninTwoSchoolsSomeTablePreparationRules
DrawingPieDiagramsUsingBarGraphsUsingLineGraphs
Example:DownloadedResultsofaCustomerSatisfactionSurveyExample:TransferringtheResultsofOneQuestionIntoBarandLineGraphs
DrawingDiagramsorPicturesExample:WordsandDiagramsinSurveyReports
WritingtheResultsofaSurveyOrganizingtheReportExample:StructuredAbstractofaSurveyReport
SurveyReportingChecklistsandGuidesClear-WritingTips
TheOralPresentationSlidePresentationsOralVersusWrittenReports:ADifferenceinConversation
Example:TableUsedinaWrittenandanOralReportPosters
SummingUpThinkAboutThisReferences
Index
Preface
Thesixtheditionofthisbooksharesthesamegoalsasthefirst,second,third,fourth,andfiftheditions:toguidereadersindevelopingtheirownrigoroussurveysandinevaluatingthecredibilityandusefulnessofsurveyscreatedbyothers.Thisedition,likepreviouseditions,alsogivespracticalstep-by-stepguidanceonhowtoachievethegoals.Theguidancecentersonchoosingtheappropriatetypeofsurvey,writingsurveyquestionsandresponses,formattingthesurvey,decidingonthecharacteristicsandnumbersofrespondentstoincludeandhowoftentheyshouldbesurveyed,andanalyzingandreportingtheresults.
Thiseditionalsosharesthesamephilosophyaspreviouseditions.Itemphasizestheneedforcarefulplanningandtestingbeforeyoupress“send,”mailthefirstsurvey,orconductthefirstinterview.Butthesimilaritiesamongtheeditionsendhere,andeachchapterinthesixtheditionhassomethingthatyouwon’tfindintheothereditions.
Thefiftheditionplacedastrongemphasisonweb-basedsurveys.Atthattime,web-basedsurveyswerebeginningtheirclimbtothetop.Ithinkwecansaywithoutcontradictionthatinternetsurveyshavemadeittothetop,andsothiseditionaddsmoreabouthowtodothesesurveysandhowtohandleprivacy.Theuniversalityofwebsurveyshasraisedsomenewconcernsamongresearchers,amongthem,howtohandleinternationalsurveys.Afterall,withtheweb,youcanreachuntoldnumbersofpeopleandtheycanbelivinganywhereontheplanet.Butculturalbeliefsandexpectationsvarywidely,andthesurveyorcannotassumethatquestionsthat“work”inoneculturewillnecessarilybeeffectiveinanother.Inthiseditionofthebook,weaddressthepracticalandethicalissuesofgoingbeyondourbordersforsurveyinformation.
Also,sinceyoungpeoplearesocomfortablewithsurveysofallsorts,thisneweditionofthebookprovidesinformationonhowtoquestionchildrenofallages,anditaddressestheethicalconcernsthatarisewhenworkingwithyoungpeople.
Thesixtheditionalsorecognizesanewtrendinreportingresearchresults.Surveyorsarenowrequiredtoadheretoveryspecificguidelinesconcerningthecontentsofareport.Peoplewhoreadandusesurveyresultswanttoknowmorethanjustwhatyoufound.Theywanttobesurethattheinformationcomesfromreliablesources,andtheywanttoknowhowconfidenttheycanbeinyourresults.Theywanttransparency.Thesixtheditionofthisbookgivesyoureportingcheckliststofollow,anditdiscusseshowtodescribehowmuchconfidencesurveyuserscanplaceinyoursurveyresults.
Herearesomeofthenewtopicsweexploreinthesixthedition.
ChildrenandSurveys.
Childrenofallagesarenowincludedinsurveys.Wewanttoknowhowchildrenspendtheirsparetime,whethertheyarehappyathomeandschool,andwhethertheydrink,smoke,oruseillicitdrugs.Childrenvaryintheirageandabilitytodealwithabstractquestions.Theneweditionofthisbookanswersquestionslike:Howdoesasurveyreachchildrentogetreliableandvalidinformation?Howdoesasurveyforyoungchildrendifferfromdoingoneforteens?
ChildrenandSurveyEthics.
Ethicsrequirementsforchildrenareoftenstricterthanthoseforadults.Thiseditionofthebookaddresseshowtoaccommodateethicsboardsanddiscussesthedifferencebetweenassentandconsentinchildren.
QualitativeSurveyData:ValidityandReporting.
Mostsurveysproducestatisticaldata,butothersmayincludeopen-endedquestionsthatneedinterpretation.Theymayrelyonsemistructuredorevenunstructuredinterviews,forexample.Manyproblemsarebettersolvedwithacombinationofsurveydesignsandmethods.Thiseditiontothebookdiscussesthevaluesofand“threats”tothevalidityofqualitativestudiesandprovidesguidanceintransparentlyreportingtheirpurposesandmethods.
InternationalSurveys.
WiththecomingofsmartphonesandWi-Fi,surveysareincreasinglybecominginternational.Inthiseditionofthebook,weaddresshowtoputintopracticeourrespectforculturaldifferencesindesigningthesurveyandinaskingquestions.
MarginofErrorandConfidenceLevels.
Allsurveyshaveerrorsbecauseofimperfectsampling.Thelargerthemarginoferror,thelessconfidencewecanhavethatthesurvey’sresultsareclosetothe“true”figures.Thiseditiondiscusseshowtothinkaboutandcalculatethemarginoferrorandconfidencelevelforyoursurvey.
SurveyReportChecklists.
Onceuponatime,surveyorscouldissuetheirreportsinanywaytheysawfit.Now,thatisnolongertrue,especiallyforpublishedsurveys.Manyjournals,businesses,andeducationalinstitutionsrequiresurveyorstoprovethattheyhavecompletedstandardchecklists.Adheringtothesechecklistsisthoughttobetterensurethetransparencyandaccuracyoftheirmethods.Thiseditionofthebookdiscussesthesechecklistsandtheircontents.
Thesixtheditionofthisbookalsohashadafacelift.Wehaveenlargedthefontstomakeforeasierreadingandretooledthegraphics.Ihaveleftsomefeaturesofthesixtheditionexactlyastheyareintheothereditionsbecausereadersandreviewersfindthemuseful.Theseincludenumerousexamplesandchecklistsandthe“SummingUp”sectionateachchapter’sendtomakesurethatthereaderhasaccesstothemostimportantconcepts.Ihavekeptthesummariesbrief,butIthinktheyareworthwhileandthatitishelpfulforpeopletoknowthebottomline:WhatshouldIfocuson?Toreinforceallmajorpoints,eachchapterstillhaspracticeexercises,manyofthemnewtothesixthedition.Theexerciseswillleadyoutosurveyreportsandtoresearchthatusessurveyssothatyoucantestyourskillsinunderstandingandevaluatingthem.Mostimportantly,Ihavegearedthepracticeexercisestoengageandpromotethereader’sskillsofanalyzing,evaluating,andproducinginformation.Thismeansavoidingexercisesthatfocusonjustregurgitatingfacts.Ihavealsoexpandedandupdatedthechapterbibliographiesandaddedadditionalwebsites,surveyreports,andresearchstudiescontainingsurveys.Ihavelistedsitesforyoutoexploreincaseyouwanttoadaptquestionsfromothersurveys.Finally,thesurveyormustmakechoicesthroughouttheentiresurveyprocess.Inresponsetothissometimesdifficultsurveyactivity,Ihavekeptthe
sectionsfoundthroughoutthebookcalled“MakingtheDecision.”Thepurposeofthesesectionsistohelpthereadertomakeinformedchoicesbycitingtheadvantagesanddisadvantagesateachwaypoint.Youcancontinuetoexpectguidanceinchoosingsurveytypes,questiontypes,samplingstrategies,researchdesigns,anddata-analyticmethods.
Ihaveorientedthisbooksothatitcanbeusedbyeveryonewhoneedstolearnhowtodoasurvey,regardlessofbackground.Thebookislongonpracticeandshortontheory.Itisintendedtohelpyoudoasurvey.Ifyouwantsurveytheory,pleaseusethebook’sbibliography.Ithinkthebookisusefulforself-learningandinworkshops,seminars,andformalclassrooms.
Onapersonalnote,Iwouldliketothankmyeditor,VickiKnight,forsupportingmethroughouttherevisionprocess.ShesawtoitthatweassembledalargepanelofreviewerstowhomIamdeeplyindebted:
ColleenL.Casey,UniversityofTexasatArlingtonSalvatoreFalletta,DrexelUniversityRheaFayeD.Felicilda,MissouriStateUniversityMichaelA.Guerra,LincolnUniversityHowardLune,HunterCollege,CUNYB.J.Moore,CaliforniaStateUniversity,BakersfieldDawnM.Pickard,OaklandUniversityMarcS.Schwerdt,LipscombUniversityJulieSlayton,UniversityofSouthernCalifornia
Theirsuggestionswereoutstanding,andbecauseIamtrulygratefulfortheirtimeandeffort,Ireallydidtrytoincorporatetheirsuggestions.WhenIdidn’t,itwasusuallyjustamatterofconvenience(mine),andIamsurethebooksuffersbecauseofit.IwouldalsoliketothanktheSAGEProductionteamfortheirhelpinreformattingandrecastingthebook.Itismuchbetter,andtheyareresponsiblefortheimprovement.
AbouttheAuthor
ArleneFink(PhD)isProfessorofMedicineandPublicHealthattheUniversityofCalifornia,LosAngeles,andpresidentoftheLangleyResearchInstitute.Hermaininterestsincludeevaluationandsurveyresearchandtheconductofresearchliteraturereviewsaswellastheevaluationoftheirquality.Dr.Finkhasconductedscoresofevaluationstudiesinpublichealth,medicine,andeducation.SheisonthefacultyofUCLA’sRobertWoodJohnsonClinicalScholarsProgramandisascientificandevaluationadvisortoUCLA’sGamblingStudiesandIMPACT(ImprovingAccess,Counseling&TreatmentforCalifornianswithProstateCancer)programs.SheconsultsnationallyandinternationallyforagenciessuchasL’institutdePromotiondellaPréventionSecondaireenAddictologie(IPPSA)inParis,France,andPeninsulaHealthinVictoria,Australia.ProfessorFinkhastaughtandlecturedextensivelyallovertheworldandistheauthorofmorethan130peer-reviewedarticlesand15textbooks.
1CONDUCTINGSURVEYSEveryoneIsDoingIt
OverviewSurveysareeverywhere.Youwillfindthemindoctors’offices,schools,airplanes,andhotelrooms.Surveysareusedtocollectinformationfromoraboutpeopletodescribe,compare,orexplaintheirknowledge,feelings,values,andbehavior.Surveystypicallytaketheformofself-administeredquestionnairesandinterviews.Self-administeredquestionnairescanbecompletedbyhand(“paperandpencil”)oronline(Internetorweb-basedoncomputersandtablets).Interviewsmaytakeplaceinperson(“face-to-face”),onthetelephone(landlineandmobile/cell),andinchatrooms.
Surveydataareusedbyprogramplanners,evaluators,administrators,managers,researchers,marketers,andpolicyleadersindiversefields,includingbusiness,health,education,socialwelfare,andpolitics.Theyareusedbecausetheygetinformationdirectlyfrompeople.
Surveyorsmustdecideonthesurvey’soverallpurposesandspecificquestions.Theyalsoneedtoknowwhoandhowmanypeoplewillbecontacted(sampling)andwhenandhowoftenthesurveywilltakeplace(design).Surveyorsmustalsoprocess,analyze,andinterpretdata.
Choosingamongsurveytypes(self-administeredquestionnairesorinterviews)andadministrationmethods(mail,telephone,orinternet)requires(1)identifyingthecombinationmostlikelytoproducecredibleandaccurateresultsand(2)balancingthedesiredsurveytypesandadministrationmethodsagainstavailableresources.
Surveypurposesandmethodsfallonacontinuum.Somesurveyscanhavefar-reaching,generalizableeffects,andtheirmethodsmustbescientific.Surveysofthepopulation’shealthconductedbytheU.S.governmentareexamplesofscientificsurveys.Othersurveysareconductedtomeetspecificneeds;theirmethodsmaynotalwaysachievethehigheststandardsofscientificrigor,buttheymuststillproduceaccurateresultsandsomustusereliableandvalidtechniques.Pollingstudentsinaparticularschooltoidentifytheirsummerreadingchoices,soastobesurethelibraryiswellstocked,isanillustrationofasurveydesignedtomeetaspecificneed.
Surveyorsmustbeconcernedwithprotectingrespondents’privacyandassuringconfidentialityofresponses.Mostinstitutionshaverulesforprotecting“humansubjects”andensuringthattheirconsenttorespondisaninformedconsent.Onlinesurveyshavespecialrulesforguardingconfidentiality.Thesesurveysmaybevulnerabletooutsidersandneedprotectionacrossthreecommunicationarms:surveyortorespondent,respondenttowebserver,andwebservertosurveyor.
WhatIsaSurvey?Surveysareinformationcollectionmethodsusedtodescribe,compare,orexplainindividualandsocietalknowledge,feelings,values,preferences,andbehavior.Asurveycanbeaself-administeredquestionnairethatsomeonefillsoutaloneorwithassistance,orasurveycanbeaninterviewdoneinpersonoronthetelephone.Somesurveysareonpaperoronline,andtherespondentcancompletethemprivatelyathomeorinacentrallocation—say,atahealthcenter.Therespondentcaneitherreturnthecompletedsurveybysnailmailorfillitoutonline.Surveyscanbeinteractiveandguidetherespondentthroughthequestions.Interactivesurveysalsomayprovideaudiovisualcuestohelp.
Hereareatleastthreegoodreasonsforconductingsurveys:
Reason1:Apolicyneedstobesetoraprogrammustbeplanned.
Example:SurveystoMeetPolicyorProgramNeeds
TheYMCCorporationwantstodeterminewhichhourstobeopeneachday.Thecorporationsurveysemployeestofindoutwhich8-hourshiftstheyarewillingtowork.ThenationalofficeoftheHealthVoluntaryAgencyisconsideringprovidingdaycareforitsstaff’schildren.Howmanyhaveveryyoungchildren?Howmanywouldusetheagency’sfacility?Tenyearsago,theBartleySchoolDistrictchangeditslanguageartscurriculum.Sincethen,somepeoplehavearguedthatthecurriculumhasbecomeoutofdate.WhatdotheEnglishteachersthink?Ifrevisionsareneeded,whatshouldtheylooklike?
Reason2:Youwanttoevaluatetheeffectivenessofprogramstochangepeople’sknowledge,attitudes,health,orwelfare.
Example:SurveysinEvaluationsofProgramsTheYMCCorporationhascreatedtwoprogramstoeducatepeopleabouttheadvantagesanddisadvantagesofworkingatunusualhours.Oneprogramtakestheformofindividualcounselingandspeciallyprepared,self-monitoredDVDs.Thesecondprogramisconductedinlargegroups.Asurveyisconducted6monthsaftereachprogramiscompletedtofindoutwhethertheemployeesthinktheygottheinformationtheyneeded.Thesurveyalsoaimstofindoutwhethertheywouldrecommendthatothersparticipateinasimilarprogramandhowsatisfiedtheyarewiththeirworkschedule.TheHealthVoluntaryAgencyistryingtwoapproachestochildcare.Oneisprimarily“childcentered,”andthechildrenusuallydecidefromalistofactivitieswhichonestheywouldliketododuringthehourstheyareintheprogram.Theotherisacademicandartistic.Childrenaretaughttoread,playmusicalinstruments,anddanceatsettimesduringtheday.Whichprogramismostsatisfactoryinthattheparents,children,andstaffareactiveparticipantsandpleasedwiththecurriculum’scontent?Theagencysurveysparents,children,andstafftogetanswers.TheBartleySchoolDistrictchangeditslanguageartscurriculum.Asurveyisconductedtofindoutwhetherandhowthechangehasaffectedparents’andstudents’opinionsofthehighschoolprogram.
Reason3:Youarearesearcherwhousesasurveytogetinformationabouthowtoguidestudiesandprograms.
Example:SurveysforResearchBecausetheYMCCorporationhassomanyeducationalprograms,itwantstoresearchhowadultslearnbest.Dotheypreferself-learningorformalclasses?Arereadingmaterialsappropriate,orarefilmsandDVDsbetter?Howdotheyfeelaboutcomputer-assistedlearningorlearningdirectlyfromtheInternet?Aspartofitsresearch,andtomakesureallthepossibilitiesarecovered,thecorporationconductsasurveyofasampleofemployeestolearntheirpreferences.TheHealthVoluntaryAgencyisconsideringjoiningwithalocaluniversityinastudyofpreschooleducation.Theagencyconductsasurveyoftheparentsparticipatinginthenewdaycareprograms.Thesurveyasksabouttheparticipants’educationandincome.Researchersneeddatasuchasthesesotheycantestoneoftheirmajorassumptions—namely,thatparentswithhighereducationandincomesaremorelikelytochoosethemoreacademicofthetwopreschoolprograms.TheBartleySchoolDistrictispartofaU.S.government–fundednationalstudyoftheteachingoftheEnglishlanguage.Thestudy’sresearchershypothesizedthatclassroomteachingdependsmoreontheirteachers’educationalbackgroundsandreadingpreferencesthanontheformalcurriculum.Asurveyisconductedtofindoutteachers’educationalbackgroundsandreadinghabitssothatthosedataareavailablefortestingtheresearchers’hypothesis.
WhenIsaSurveyBest?Manymethodsexistforobtaininginformationaboutpeople.Asurveyisonlyone.Considertheyouthcenterthathasasitsmajoraimtoprovideavarietyofservicestothecommunity.Itoffersmedical,financial,legal,andeducationalassistancetoresidentsofthecitywhoarebetween12and21yearsofage,regardlessofbackground.Theprogramisparticularlyproudofitscoordinatedapproach,arguingthatthecenter’seffectivenesscomesfrommakingavailablemanyservicesinonelocationtoall
participants.Nowthatthecenteris10yearsold,asurveyistobeconductedtofindoutjusthowsuccessfulitreallyis.Areparticipantsandstaffsatisfied?Whichservicesdoyoungpeopleuse?Isthecenterreallyamultiserviceone?Arepeoplebetteroffwiththeirhealthandotherneedsbecauseoftheirparticipationinthecenter?Amailedself-administeredquestionnairesurveyisdecidedontohelpanswertheseandotherquestions.Herearesomeexcerptsfromthequestionnaire.
ThequestionnairewasshowntoareviewerwhoseadvicewastoeliminateQuestions5,7,and11,andkeeponlyQuestion10.Thereviewerstatedthatsurveysarenotbestforcertaintypesofinformation.Hereisthereasoning:
Question5asksforareportofaperson’sbloodpressure.Isitnormal?Ingeneral,informationofthiskindismostaccurateifitisobtainedfromothersources—say,amedicalrecord.Manypeoplemighthavedifficultyrecallingtheirbloodpressurewithprecisionandalsowouldbeatalosstodefinenormalbloodpressure.
Question7maybeallrightifyoufeelconfidentthattheperson’srecallwillbeaccurate.Otherwise,therecordsofthecenterareprobablyabettersourceofinformationaboutwhichservicesareused.
Question11asksthepatienttotellhowmuchtimethedoctorspendslisteningratherthantalking.Ifyouareinterestedinthepatient’sperceptions,thenthequestionisfine.If,however,youwantdataontheactualtimethedoctorlistenedratherthantalkedtothepatient,observationbyanimpartialobserverisprobablybest.
Question10isappropriate.Onlyparticipantscantellyouhowsatisfiedtheyare.Noothersourcewilldosoaswell.
Surveysarebynomeanstheonlysourceofinformationformakingdecisions,noraretheynecessarilythemostrelevant.Someothersourcesofinformationarethefollowing:
Observationsoreyewitnessreports;filmed,videotaped,andaudiotapedaccountsPerformanceteststhatrequireapersontoperformatask(suchasteachingalessontoaclass);observersassesstheeffectivenessoftheperformanceWrittentestsofabilityorknowledgeReviewsthatrelyonexistingdocumentation,suchasexaminationofmedicalandschoolattendancerecords;analysisofthecontentofpublishedandunpublishedarticlesanddiaries;interpretationoftheactivitiesofonlinechatandsupportgroups
Surveyscanbeusedindecidingpolicyorinplanningandevaluatingprogramsandconductingresearchwhentheinformationyouneedshouldcomedirectlyfrompeople.Thedatatheyprovidearedescriptionsoffeelingsandperceptions,values,habits,andpersonalbackgroundordemographiccharacteristicssuchasage,health,education,andincome.
Sometimessurveysarecombinedwithothersourcesofinformation.Thisisparticularlytrueforevaluationsandresearch.
Example:FromanOverlyAmbitiousSelf-Administered
Questionnaire5.Isyourbloodpressurenownormal?
Yes 1No 2
7.Whichofthefollowingsocialserviceshaveyouusedinthelast12months?(Pleaseindicateyesornoforeachservice.)
10.Howsatisfiedareyouwitheachofthefollowingservices?Pleaseindicateyoursatisfactionforeachservice.
11.Howmuchtimeina5-minuteperioddoesthedoctorspendlistening(ratherthan,say,talking)toyou?(Pleasecheckone.)Lessthan1minuteAbout1or2minutesMorethan2minutes
Example:SurveysCombinedWithOtherInformationSourcesAspartofitsevaluationofchildcareprograms,theHealthVoluntaryAgencysurveyedparents,children,andstaffabouttheirdegreeofparticipationandsatisfaction.Also,theagencyreviewedfinancialrecordstoevaluatethecostsofeachprogramandstandardizedtestsweregiventoappraisehowreadychildrenwereforschool.TheYMCCorporationisresearchinghowadultslearn.Achievementandperformancetestsaregivenatregularintervals.Inaddition,asurveyprovidessupplementaldataonhowadultsliketolearn.
Self-AdministeredQuestionnairesandInterviews:TheHeartoftheMatter
Allsurveysconsistof(1)questionsandresponses.Togetaccuratedata,youmustaccountforasurvey’s(2)samplinganddesign,(3)dataprocessingor“management”andanalysis,(4)pilottesting,and(5)responserate.Surveyresultsarepresentedaswritten(printedonpaperorreproducedelectronically)andoralreports.
QuestionsandResponsesInformationfromsurveysisobtainedbyaskingquestions.Questionsaresometimesreferredtoasitems.Thequestionsmayhaveforced-responsechoices.
Theselection,wording,andorderingofquestionsandanswersrequirecarefulthoughtandareasonablecommandoflanguage.
Example:Forced-ChoiceQuestionWhatisthemainadvantageofmultiple-choiceoveressayquestions?
CanbescoredobjectivelyArebestatmeasuringcomplexbehaviorsCanhavemorethanoneanswerAretheleastthreateningofthequestiontypes
Questionsonsurveysmaybeopenended.
Example:Open-EndedQuestionWhatisthemainadvantageofmultiple-choiceoveressayquestions?
Answerhere:
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
SurveySampleandDesignSurveysaredatacollectionmethodsusedtoobtaininformationfromandaboutpeople:Fromandaboutwhichpeople,howoften,andwhen?Assoonasyouraisequestionssuchasthese,youmustbecomeconcernedwiththesampleanddesignofthesurvey.Thesampleisthenumberandcharacteristicsofpeopleinthesurvey.Thedesignreferstohowoftenthesurveytakesplace(justonce,orcross-sectional;overtime,orlongitudinal),whethertheparticipantsareselectedatrandomorarechosensomeotherway,andhowmanyseparategroupsareincluded.
Considerthesethreesurveys:
Survey1:Whatdograduatesfromtheclassof2015knowaboutphysicalfitness?
Surveymethod:Onlinequestion-naireSample:All1,000graduatesfromStateCollege’sclassof2015Howoftensurveytakesplace:Justonce—atgraduationHowparticipantsareselected:AllgraduatesareeligibleHowmanygroups:Justone—theclassof2015Design:Cross-sectional
Survey2:Doesknowledgeaboutphysicalfitnesschangeovera12-monthperiodamonggraduatesoftheclassof2015?
Surveymethod:Onlinequestion-naireSample:All1,000graduatesfromStateCollege’sclassof2015Howoftensurveytakesplace:Twice—atgraduationand12monthslaterHowparticipantsareselected:AllgraduatesareeligibleHowmanygroups:Justone—theclassof2015Design:Longitudinalcohort
Survey3:Overtime,dodifferencesexistamonggraduatingclassesintheirknowledgeofphysicalfitness?
Surveymethod:Onlinequestion-naireSample:A75%randomlyselectedsampleofgraduatesfromtheclassesof2015,2017,and2019toequal2,250graduatesHowoftensurveytakesplace:Threetimes—atgraduationand12and24monthslaterHowparticipantsareselected:RandomlyHowmanygroups:Three—theclassesof2015,2017,and2019Design:Longitudinalandcomparative
Survey1asksforaportraitoftheclassof2015’sknowledgeofphysicalfitness,andamailedquestionnaireistobeused.Thisportraitiscalledacross-sectionalsurveydesign.Survey2wantstoknowaboutchangesinknowledgeofphysicalfitnessovera1-yearperiod:fromgraduationforward12months.Thedesignislongitudinal.Theentireclassisa“cohort”ofpeople.
Survey3islongitudinalbecausesurveydataarecollectedfromeachofthethreegraduatingclassesoverthreepointsintime:atthetimeofgraduationand1and2yearslater.Thedesignalsoiscomparativebecauseknowledgecanbecomparedbetweenanytwoandamongallthreeclassesatgraduation,1yearlater,2yearslater,oracrossallthreetimes.AnillustrationofthedesignforSurvey3cantaketheformofTable1.1.
Survey3differsfromSurveys1and2inhowthegraduatesareselectedforparticipation.InSurvey3,a75%sampleofgraduateswillberandomlyselectedtoparticipate.Intheothertwosurveys,allgraduates,notjustasample,areeligible.Randomselectionmeansthateachgraduatehasanequalchanceofbeingincluded.
Allthreesurveysareonline,buttheirsamplesanddesignsvary.
PlanningforDataAnalysisRegardlessofyoursurvey’sdesignorsize,youmustthinkaheadtohowyouplantoanalyzethesurvey’sdata.
Willyoucomputepercentagessothatyourresultslooklikethis?
Ofthetotalsample,50%reportedthattheywereRepublicans,42%wereDemocrats,5%wereIndependent,1%belongedtotheGreenParty,and3%hadnopartyaffiliation.
Willyouproduceaveragestoappearthisway?
Theaverageageoftherespondentsis56.4years.Themedianeducationallevelis13years.
Willyoucomparegroupsandreportsomethinglikethis?
Atotalof60%ofthemen,butonly20%ofthewomen,wereRepublicans.
Respondentsdonotdiffersignificantlyinsatisfactionwiththepresentgovernment.
Willyoulookforrelationshipssuchasthis?
Thesurveyfoundnoassociationbetweenhowliberalorconservativepeoplewereandtheireducationalattainments.
Highschoolgraduateswhowere30yearsofageorolderweresignificantlymorelikelytovoteinthelastelectionthanwereolder,lesseducatedrespondents.
Willyoulookforchangesovertime?
Since2007,statisticallysignificantdifferenceshavebeenfoundinthenumberofmenparticipatingin2ormorehoursofchildcareperday.
PilotTestingApilottestisatryout,anditspurposeistohelpproduceasurveyformthatisusableandthatwillprovideyouwiththeinformationyouneed.Allsurveysmustbepilottestedbeforebeingputintopractice.Self-administeredquestionnairesdependheavilyontheclarityoftheirlanguage(itdoesnotmatterifitisawrittenoronlinequestionnaire),andpilottestingquicklyrevealswhetherpeopleunderstandthedirectionsyouhaveprovidedandiftheycananswerthesurveyquestions.Apilottestofa
face-to-faceinterviewalsowilltellyouaboutinterviewers.Cantheyfollowtheinterviewformeasily?Arethespacesonprintedsurveyslargeenoughforrecordingresponses?Dointerviewersknowwhattodoifthecomputer“freezes”whiletheyareinthemidstofacomputer-assistedinterview?Doestherespondentunderstandhowtomovebackandforwardthroughanonlinesurvey?Pilottestsalsocantellyouhowmuchtimeittakestocompletethesurvey.
Testinghelpsmakethesurveyrunsmoothly.Wheneverpossible,youshouldtrytoduplicatetheenvironmentinwhichthesurveyistotakeplace.Thatmightmeanobtainingpermissionfrompeoplejusttobeinthetryouts,butnotinthesurvey,althoughtheyareeligibleforfullparticipation.
ResponseRateThesurveyorwantseveryonewhoiseligibletorespondtoallquestions.Pilottestinghelpsimprovetheresponseratebecauseitcaneliminateseverepotentialsourcesofdifficulty,suchaspoorlywordedquestionsandnoplacetorecordanswersonprintedquestionnaires.Furthermore,iftheentiresetofsurveyproceduresiscarefullytested,thenthis,too,canhelptheresponserate.Beforeyoudoatelephoneinterview,askthesequestions:Doyouhaveavailableacurrentlistoftelephonenumbers?Areyouwillingtomaketelephonecallsatthetimethesurveyrespondentsareavailable?Doyouhaveaplanforreachingrespondentswhodonotreturncallsleftonvoicemail?Foronlinesurveysaskthefollowing:Doyouhaveavailableacurrentlistofemailaddresses?Doyouknowhowtoensureprivacyofresponses?Otherwaysofensuringgoodresponseratesexist,regardlessofsurveytype,suchaskeepingsurveysshortandsimpleandprovidingincentivesforparticipating.
Howhighshouldtheresponseratebe?Ifyouareconductingalarge,complexsurvey,youwillwanttousestatisticalprocedurestoanswerthisquestion.Ifyoursurveyisrelativelysimple(say,apoolofteachersinaschoolornursesinthreehospitals),thenyouhavetodecidehowmanypeopleyouwillneedfortheresultstobebelievable.If20peopleareeligibleforcompletingamailed,self-administeredquestionnaireandonly10respond,youmayfeeldifferentfromthewayyouwillfeelif,atanothertime,200of400respond.Bothsurveyshavea50%responserate,butreportingontheviewsof10of20peoplemayappeartobelessconvincingthantellingabout200of400.Exceptwhendonestatistically,thedesiredresponseratetendstobeentirelysubjective,andthegeneralruleis“higherisbetter.”
ReportingResultsSurveyresultsarereporteddailyontheInternetandTVandinnewspapersandmagazines.Tomany,asurveyisapollusuallyofsome,butnotall,peopleaboutanissueofimmediatepolitical,social,oreconomicconcern.Surveyresultstypicallylooklikethis.
Example:TheLookofSurveyResults(1)Question:Iftheelectionwereheldtoday,wouldyouvoteforCandidateX?
Example:TheLookofSurveyResults(2)
Source:Takenfromwww.questionpro.com.
Example:TheLookofSurveyResults(3)
Source:ASurveyofApplicants—SenateCommittee.
MoreinformationonstandarderrorsisfoundinChapter5,andmoreinformationonconfidenceintervalsandstandarddeviationsisprovidedinChapter6.
Togetresultssuchastheserequiresmanysteps,andallsurveysfollowthem:
Decidingonthetypeofsurvey(mailedoronlinequestionnaire;telephoneorface-to-faceinterviews)Selectingthesurvey’scontentandwritingquestionsandtryingouttheformDecidingwhoshouldparticipate(Everyone?Asampleofpeople?)andhowoften(Justonce?Eachyearfor5years?)Administeringthesurvey(Whoshouldconducttheinterview?Bywhenmusttheonlinequestionnairebesubmitted?)Processingthedata(Howwilldatabeentered:manuallyorelectronicallyfromsurveytodatabase?)Analyzingandinterpretingtheresults(Wastheresponserategoodenough?Whatdothenumbersordifferencesbetweenpeoplemean?JusthowdopeoplefeelaboutCandidateX?Haveopinionschangedovertime?)Reportingtheresultsorallyorinwritingusingtext,charts,tables,andgraphs(Whoistheaudience?Howlongshouldthereportbe?Whichdatashouldbepresented?)
Nocrediblesurveycanomitanysinglestep,although,dependingonitspurposesandresources,somestepswillreceivemoreemphasisinanygivensurveythaninanother.
TheFriendlyCompetitionHowdoyouchoosebetweenself-administeredquestionnairesandinterviews?Isamailedoranonline
surveybetter?Whenisaninterviewthebestchoice?Herearesomecriteriaforselectingamongthedifferentsurveytypes.
ReliabilityandValidityAreliablesurveyresultsinconsistentinformation.Avalidsurveyproducesaccurateinformation.Reliableandvalidsurveysareobtainedbymakingsurethedefinitionsandmodelsyouusetoselectquestionsaregroundedintheoryorexperience.Nosinglesurveytypestartsoutwithbetterreliabilityandvaliditythananother.Choosethesurveymethodthatismostpreciseandaccurateforyourspecificpurposes.Forexample,ifyouareworriedthatthepeopleyouaresurveyingcannotreadwell,anoral(face-to-faceortelephone)interviewislikelytoproducefarbetterresultsthanawrittenone.
Focusgroupsandpilottestshelpyoudecidewhichtypetouseandwhetheryouhavedoneagoodjobofdesigningthesurveyandmakingituser-friendly.Respondentsorsurveyadministrators(thepeoplewhodotheinterviewingorhandoutthequestionnaires)whohavetroublewiththesurveywilluseitincorrectly,introducingbias,andthatinturnreducestheaccuracyoftheresults.Awelldesigned,easy-to-usesurveyalwayscontributestoreliabilityandvalidity.
UsefulnessorCredibilityofResultsTheresultswillbeusefuliftheyarevalidandifthesurveymethodisonethatusersacceptascorrect.Findoutbeforeyoustartwhichmethodistheonepeoplewant.Sometimesthepeoplewhowillusetheresultshavestrongpreferences.
CostsThisreferstothefinancialburdenofdevelopingandadministeringeachtypeofsurvey.Thecostsassociatedwithwrittenquestionnaires(on-siteandmailed)includepaper,reproduction,andincentives.Mailedquestionnairesrequireanup-to-dateaddresslist(whichyoumayhavetopurchase),postage,andenvelopes.Sometimesyouhaveseveralfollow-upmailings,addingtothecosts.
Thecostsofface-to-faceandtelephoneinterviewsincludephonechargesor,inlargesurveys,thecostsofpurchasingatelephonesystemandpayingformiscalledandout-of-datetelephonenumbersaswellashang-ups.Youalsoneedtopayforwritingascriptfortheinterviewer,trainingtheinterviewers,monitoringthequalityoftheinterviews,andprovidingincentivestorespondents.
Onlinesurveysrequireextensivedevelopmentandtesting.Onewaytosaveondevelopmentistopurchaseuser-friendlyonlinecommercialsurveyprograms.Youwillhavetolearnhowtousethedevelopers’software,however,andthiscanbetime-consumingandcostly.Andeventhebestsoftwarecannotmakeupforasloppysurvey.
TheSpecialCaseofOnlineSurveysOnlinesurveysareself-administeredquestionnaires.Respondentscompleteonlinesurveysonlaptops,desktops,notebooks,tablets,andcellormobilephones.Surveyorslikeonlinesurveys,andrespondents
arebecomingusedtothem.Surveyorslikeonlinesurveysbecausetheycaneasilyreachverylargenumbersofpeopleacrosstheworldandbecauseonlinesurveysoftwareisaccessibleandrelativelyinexpensive.
Onlinesurveyssometimesrunintotroublebecausesurveyorsdonotplanaheadadequately.Surveynoticesaresentouttounsuspectingrespondents,whothendeletethemasspam.Orthesurveyissenttoeveryoneonalistofemailaddresses,andmanyoftheaddressesareincorrect.Otherproblemsincludefailuretoprovideincentivestooverloadedrespondentswhoareoverwhelmedwithsurveyrequestsortoomanyorhard-to-understandquestions.Aswithanyothersurveytype,thesurveyorhasmanycarefuldecisionstomakeaboutthecontentandformatofthequestionsandresponses,thesamplesize,whowillbeincludedinthesurveyandwhowillbeexcluded,andhowtoguaranteethattheresultsareaccurateandsecure.Onlinesoftwarecannotmakeupforapoorlydesignedandimplementedsurveythatisnotusedappropriately.
Hereisachecklisttouseindecidingwhentouseonlinesurveys.
ChecklistforDecidingtheBestUsesforOnlineSurveysThesurveyneedswidegeographiccoverage.Youarehopingtoreachaverylargenumberofpeople.Youhaveaccesstovalidemailaddressesandpermissiontousethem.Unsolicitedsurveysmayappearasspamorjunkmail.Youoryouragencyorinstitutionconductssurveysrelativelyfrequentlysothatanyinvestmentintechnologicalandsoftwareinfrastructureisworthwhile.Developingsoftwaretocreatesurveyquestionscanbecostly.Commercialcompaniesaboundwhocanhelpdesignandimplementsurveysandanalyzedata.That’stheeasypart.Thereisalwaysalearningcurve,however,andlearningnewsoftwaretakestime,andtimeusesfinancialresources.Youhavedesignedsomesurveyquestionstobeansweredbyallrespondents,butotherquestionswillonlybeansweredbysomerespondents.Forinstance,ifyouwanttoexplorepeople’sexercisehabits,butnotallpeoplereportthattheyexercise,youonlywanttoasktheexercisersabouttheiractivities.Onlinesurveyscanbeprogrammedtodistinguishbetweenwhoistoanswerquestionsandwhoistoskipthem.Thissavesthesurveyortheneedtoresorttocomplexinstructions.Youplantousemultimediasuchasvideoclipsandvoice-overindescribinganewidea,product,orservice.Youneedresultsinahurry.Onlinesurveyscanproducereal-timedataaseachpersoncompletesthequestionnaire.Itisimportanttorememberthatthestatisticalresultsproducedbymostsurveysoftwarepackagesarerelativelysimple:numbers,averages,andsoon.Thegraphicsthatareproducedarealsosimpleandincludepiechartsandlinegraphics.Morecomplexanalysesandreportstakemoretimeandexpertise.Youwanttocomparesurveydataovertime.Onlinesurveydatacaneasilybestoredinadatabaseforfutureuse.Newsurveydatacanbeaddedtothedatabasefromtimetotimefromthesameorfromdifferentgroupsofpeople,andyoucancomparedataovertimeforoneormoregroupsofrespondents.Youplantostudysurveybehavior.Withonlinesurveys,thedateandtimeofsurveycompletionandthetimespentoneachquestionareknown.Thisinformationcanbeusedtoseeiftherearedifferencesamongpeopleinhowmuchtimetheyneedtocompletesomeorallofthesurvey.Thedatacanbeusedtohelpyouimprovethesurveyquestions.
Atleastfourpotentialconcernsaccompanymostwebsurveys.Theyarethefollowing:
1.Multipleaddresses.Manypeoplehavemanyemailaddressesorpointsofonlinecontactthroughsocialmedia.Althoughsomepotentialrespondentsmayconsolidatetheiraddressessothattheycanviewmessagesfromallsourceswhenevertheycheck,othersdonot.Somepeopleonlycheckoneortwositesfrequently,andotherschecktheiremailinfrequently.Becertainthattheaddressesyouhaveareaccurateandcommonlyused.
2.Privacyandconfidentiality.Manypeoplearesuspiciousofsendinginformationovertheweb.
Surveyorsmustlearnhowtoguardrespondents’privacyandconvincinglycommunicatethestrengthoftheireffortstorespondents.Itisimportanttonotethatnoteveryoneknowsiftheircomputerortabletissecureorhowtomaximizesecurity,noraretheyawareofwhohasaccesstotheirbrowsinghistory.Thesurveyor,ofcourse,doesnothavecontroloverprivacylapsesthatmayoccurbecauseofapoorlysecureddevice.
3.Identityoftherespondents.Thesurveyorhasnorealwayofknowingwhoisactuallycompletingthequestionnaire.
4.Technologicaldifferences.RespondentsvaryintheirInternetconnectionsandbrowsersandmaysettheirowntextsizeandcolorsettings.Somesurveysdonotpermituseofthebackfunctiononthebrowserandrelyinsteadonaspecialbackarrow.Anydeviationfromexpectationmaybeconfusingtorespondents,andtheymaygiveup.Thesurveyshouldbetestedonallcommonlyusedbrowsers.Ifnecessary,respondentsshouldbeinstructedonhowtogobackandforwardinthesurvey.Computers(desktops,laptops,tablets)losetheirconnections,anditisagoodideatosetupasystemsothat,incaseoflossofpoweroraconnection,therespondentswillbeabletoreturntothesurveywithoutlosingtheirplace.
Manywebsurveysaredonethroughonlinemerchantswhosellsoftwareandsurveyprocessingservices.Manyfirmsdopracticallyeverythingforthesurveyor.Theywillsendoutthesurvey,trackrespondents,analyzedata,andproducereports.Theydon’tusuallycreatethesurvey,buttheymayprovidereferralstoconsultantswhodo.
Ifyoudecidetopurchaseonlinesurveysoftware,bepreparedtospenduptoaweeklearninghowtouseit.Mostvendorsprovideexcellenttutorialsinhowtoselectquestiontypes,numberpagesandquestions,anddecideonfeaturessuchasbackgroundcolorsandlogos.Itisveryimportantthatyoureadthefineprintregardinghowmuchassistanceyouwillhavefromthesurveyvendor.
Surveyorsaretemptedtorelyontheirvendorstoupdateandmaintainthesurveyandthesurveydata.Butservicecontractstendtobelimited,andsotheseguidelinesshouldbefollowed.
Youcangoonlineforinformationonhowtoevaluatemerchants.Gotoanysearchengineandusethesekeywords:“compareonlinesurveyservices.”
GuidelinesforSurveyorsWhoWorkWithCommercialSurveyCompanies
Downloadyourdatafrequently,andstoreyourfilessafely.Youraccesstoyourdatamaydisappearwhenyoudiscontinueyourservice.Createofflinecopiesofeveryonlinesurveysothatyoucanrecreatetheinstrumentinapaperandpencilformatoruseitwithanotheronlineservice.Makesurethatyoucansavethesurveyonyourcomputer.Bewaryofthecompany’slistofsuggestedsurveyquestions.Aretheyappropriateforyourrespondents?Whatevidencehasbeenprovidedthattheyhaveeverbeenusedwithanyone?Bywhomhavetheybeenused?Makecertainthatyouknowwhoownsthesurveydata.Doyouownthedata,ordoesthevendor?Canthevendorsellthedata?Useitforpurposesunrelatedtoyoursurvey?Whathappenstotherespondents’emailaddresses,ifyouusethem,asiscommontoinvitepeopletotakethesurvey?Aretheresponsescollectedoversecured,encryptedconnections?Whereistheinformationsystem’sinfrastructure(servers,networkingequipment)located?Whoownsandmanagestheequipmentinthedatacenters?Arethedatacentersstaffedandmonitoredallday,everyday?Whohasaccesstothedatacenters?Theanswerstothesequestionsshouldbereadilyavailableonthevendor’swebsite.Lookforthevendor’sprivacypolicyand
readit.
TheSpecialCaseofCellorMobilePhonesManylargesurveysthathavehistoricallyreliedonlandlinesamplingarenowalsousingcellphonesamples.Unfortunately,surveyorsarefindingthatcellphonesurveyscanbemoreexpensivethantheythought.U.S.federalregulationsprohibittheuseofautomaticdialingtocellphonelines,andhanddialingincreasescosts.Also,screeningforeligiblerespondentscanraisethecosts.Atthepresenttime,manyyoungerpeopleusecellphones,butolderpeoplepreferlandlines.Reachinganeligiblerespondentsuchastheheadofahouseholdoranolderadultcanmeanextraeffortandexpense.
Cellphonesurveysarealsocomplicatedbythefactthattheyareattachedtoindividualsratherthangeography,soanadditionalscreenmustbeaddedtoensurethattherespondentlivesintheareaforwhichthesurveyorwantstocollectdata.Theprocessbecomesevenmorecomplicatedwhenyouconsiderthatsomepeoplehavelandlines,somehavelandlinesandcellphones,andsomeonlyhavecells.Noonereallyknowsifthesegroupsdifferfromoneanother,andiftherearedifferences,whethertheyaffectthesurvey’sresults.
Cellphonesurveysalmostalwaysrequireincentivesforparticipationbecausetherespondentotherwisepaysforthecalldirectlyorindirectly.
MakingtheDecisionTohelpyoudecideamongthedifferenttypesofsurveysthatmayberelevanttoyourneeds,Table1.2comparestheadvantagesanddisadvantagesofthemajorsurveytypesandremindsyouoftheirspecialneedsandcosts.
ASurveyContinuum:FromSpecifictoGeneralUseSurveyshavebecomeamajormeansofcollectingdatatoanswerquestionsabouthealthandsocial,economic,andpoliticallife.Howextensiveandscientificmustasurveybe?
Comparethesetwosurveys.
Example:SurveyWithaSpecificUseThedirectorsoftheNeighborhoodHalfwayHouseswanttoprovideservicesthatareappropriateforresidents.Atpresent,manycomplaintshavearisenoverthelackofadequatefitnessfacilities.Asurveywillbeconductedtopollthe5healthcareproviders,100residents,and10full-andpart-timestafftofindoutwhatfacilitiesaredesirableandaffordable.
Example:SurveyWithaGeneralUseTheCountyHealthDepartmentisconcernedwiththeeffectivenessofits10halfwayhouses.Together,the10houseshave20,000residentsand220full-andpart-timestaff.TheCountyHealthDepartmenthasnegotiatedarrangementsforhealthcareservicesfromanumberofprovidersinthepublicandprivatesectors.Aspartofitseffectivenessstudy,theCountyHealthDepartmentissurveyingarandomsampleofresidents,staff,andprovidersatallhouses.NextDoorCountyisinterestedinadoptingtheCountyHealthDepartment’shalfwayhousemodelandisanxiouslywaitingfortheresultsofthesurveyandevaluation.
Thejustificationforthefirstsurveyisonehalfwayhouse’sconcernwithitsownneeds.ThereasonforthesecondistheCountyHealthDepartment’sinterestintheeffectivenessofallitshalfwayhouses.Also,NextDoorCountyisinterestedinthesurvey’sresults.Survey1,withitslimitedimpact,canberelativelyinformalinitsmethods.Survey2,incontrast,mustberigorousinitssamplingplan,questionnaireconstruction,anddataanalysisandinterpretation.Survey1isconcernedprimarilywithusefulness.
Survey2alsoisconcernedwithvalidityandgeneralizability:Ifadaptedinanotherplace(NextDoorCounty),willtheCountyHealthDepartment’shalfwayhousemodelbeequallyeffective?
Eachtimeyoudoasurvey,youmustevaluatewhereitspurposesfallonacontinuumthatgoesfromspecifictogeneraluse.Youhavesomeleewaywithasurveydesignedtomeetspecificneeds.Allsurveysthataimtobegeneralizableintheirfindingsmustbeconductedwithrigor.
Ethics,Privacy,andConfidentialitySomepeoplehavebecomesuspiciousofsurveys.Theyfearthattheinformationtheyprovidewillbeusedinappropriately.Manytechniquesexistforprotectingeachperson’sprivacyandensuringthatinformationwillbeusedonlywiththeperson’sknowledgeandforclearlystatedpurposes.Thesurveyorneedstoreassurepotentialrespondentsthatthesetechniqueshavebeenincorporatedintoeachsurvey.Itistheethical—rightthing—todo.
Allcompletedprintedorwrittensurveysshouldbekeptinlockedfiles,andonlyalimitednumberofstaffshouldhaveaccesstothemonaneed-to-knowbasis.Attheconclusionofdataanalysis,thesurveysshouldbeshredded.Furthermore,youcanseparateidentifyinginformation(e.g.,names,birthdates,SocialSecurityorgovernment-issuedidentificationnumbers)fromsurveyresponsesbyassigningcodesto
individualsandusingthecodestolinkthemtotheirresponses.Onlinesurveytakerscanbepermittedtoassigntheirownusernamesandpasswordswhenloggingintotakeasurvey.
Theuseofsurveysandconcernforethicalissuesarecompletelyinterwoven.Surveysareconductedbecauseoftheneedtoknow;ethicalconsiderationsprotecttheindividual’srighttoprivacyorevenanonymity.
IfyoursurveyisforapublicorprivateagencythatisreceivingU.S.governmentfunds,youshouldknowthatthefederalgovernmenthasspecifiedthelegaldimensionsofinformedconsent,privacy,andconfidentiality.Thesedimensionsincludethefollowing:
AfairexplanationoftheprocedurestobefollowedandtheirpurposesAdescriptionofanyrisksandbenefitsAnoffertoansweranyinquiriesAninstructionthatthepersonisfreetowithdrawconsentanddiscontinueparticipationwithoutprejudice
Confidentialityisprotectedbythe“ProtectionofHumanSubjects”guidelinesoftheCodeofFederalRegulationssometimescalledtheCommonRules.Confidentialityreferstothesafeguardingofanyinformationaboutonepersonthatisknownbyanother.Asurveyorwhohasthenamesandaddressesofpeople,evenincodeorwithoutidentification,“deidentified,”maynotusethisinformationtorevealidentities.Inmanysurveys,confidentialityisarealconcernbecausecompleteanonymityispracticallyimpossible.Acodenumber,anemailaddress,anInternetServiceProvider(ISP)address,orevensometimesjustaziporpostalcodemayleadtothesurveyrespondent’sidentity.
Ifyouworkforaprivateagency,organization,orbusiness,youshouldchecktherulesofinformedconsentandconfidentiality.Isthereahumansubjects’protectioncommitteeorInstitutionalReviewBoard(IRB)whoseapprovalyoumustget?Ifyouareastudent,checktoseewhetheryoucanaskthequestionsyouareplanning.Also,youmaybepartofalargerprojectthathasalreadyreceivedapprovalforitsactivitiesaslongasitconformstocertainstandards—amongthem,theinformedconsentofrespondents.
InformedConsentTheconsentformgivespotentialrespondentssufficientwritteninformationtodecidewhethertocompleteasurvey.Hereisalistofcontentstoincludeinaninformed-consentform:
ContentsofanInformed-ConsentFormAtitlesuchas“ConsenttoParticipateinSurvey.”Thenameofthesurvey.Thepurposeofthesurvey.Procedurestobefollowed,includingwherethesurveywilltakeplaceanditsduration.Potentialrisksanddiscomforts,includingansweringquestionsthatarepersonalorbeinginaclosedroomfor2hours.Potentialbenefitstorespondentsandsociety,includingnewknowledgeorbetterinformationtodevelopprogramsorpolicies.Sometimesthebenefitsarenotyetreallyknown.Paymentforparticipation.Sayhowmuchparticipantswillbepaid;ifnopaymentisprovided,sayso.Confidentiality.Iftherespondent’snameistobekeptconfidential,describecodingprocedures,whowillhaveaccesstothesurveys,andwherethecompletedsurveyswillbekept.Ifinformationistobesharedwithanyone,statewithwhom.Youmayberequiredbylawtorevealsurveyresults.
Participationandwithdrawal.Cantheparticipantswithdrawatanytime?Whathappenstothemiftheydo?(Forexample,dotheystillretainanyincentives?Willtheystillreceivethesameeducation,socialbenefits,orhealthcaretheycamefor?)Identificationofsurveyors.Whoshouldbecalledifquestionsariseaboutthesurvey?
TheInternetandEthicalSurveysAnonlinesurveyinvolvesawebofcomputersthatinteractwithoneanother.Communicationstakeplacebetweenthesurveyorandtherespondent,therespondentandthewebserver,andthewebserverandthesurveyor.Securitybreachesarepossibleanywherewithinthewebunlessyouputprotectionsinplace.
CommunicationBetweentheSurveyorandtheRespondentItisnotuncommonforasurveyortocontactarespondentbyemail.TheemailwilldiscussthesurveyandinvitetheparticipanttoclickonaURLorpasteitintoabrowsersuchasMozilla’sFirefox,Google’sChrome,Apple’sSafari,orMicrosoft’sInternetExplorer.Unfortunately,emailisnotalwayssecureorprivate.Manypeopleareunawareofwhethertheircomputersaresecureorevenhowtosecurethem.Emailprogramsmaintainedbyemployersoftenarenotprivate.Ifpeopledonotlogofforarecarelessaboutpasswords,theirprivacycanbecompromisedeasily.Also,inadequatepasswordsareeasytocrack.Ifyouconductasurveythatrequirespeopletouseapassword,youmustensurethatthepasswordsetupissecure.
CommunicationBetweentheRespondentandtheWebsiteWhenarespondententerssensitivedataintheblankspacesofaweb-basedquestionnaire,itissimilartoashopperprovidingacreditcardnumberwhenshoppingonline.OnlinemerchantsuseaSecureSocketsLayer(SSL)protocolthatallowssecurecommunicationsacrosstheInternet.AnSSLprotocolencrypts(convertsintocode)theuser’ssurveyinput,anditdecryptsitwhenitarrivesatthewebsite.Manypotentialsurveyrespondentsarebecomingawareofhoweasilytheirresponsescanbeinterceptedunlesstheyaresecured,andwithoutguaranteesthatresponsesareencrypted,someofthemmayrefusetotakethesurvey.YoumustdecideinadvancewhethertouseSSLandhowtoexplainyoursecuritychoicestorespondents.
CommunicationBetweentheWebsiteandtheResearcherSensitiveidentifiabledataneedtobeprotectedintransitbyusingeitheranSSLprotocolorasecurefiletransferprotocol.
DataProtectionSomepeoplearereluctanttocompleteonlinesurveysorevenconnecttosurveysitesforfearthattheirprivacywillbecompromised.Alldatabasesstoringsensitiveandidentifiableinformationmustbeprotected,regardlessofwhethertheyarecreatedandmaintainedbycommercialfirmsorbyindividuals.Encryptingthedatabasesprobablyprovidesthemostsecurity.
Allreputableorganizationsdeveloporadaptrulesforreassuringrespondentsthatprivacywillberespected.Hereisaminimumsetofrulesforaprivacypolicy:
ThefollowingistheCentersforDiseaseControl’sprivacypolicy(http://cdc.gov/privacy.html).AlthoughtheCDC’ssiteisusedformanypurposes(notjustforsurveysandsurveyreports),itsprivacypolicyiseasytoreadandillustrateshowtocomplywiththeminimumsetofcriteriaforaprivacypolicy.
Lookatthefollowingexcerptfromaprivacystatement.Thestatementcomesfromaverylargecorporationthatconductssurveys.Asyoucansee,thecompanyistruthfulaboutthepotentialforothercompaniestotrackcustomers’activities.However,consumersareleftwiththeobligationto(1)beawarethatunwantedcookiesmaybeplacedontheirharddriveand(2)iftheyprefer,todosomethingaboutitbycontactingtheprivacyofficer.
Thisexcerptraisesseveralquestions:Willtherespondentsactuallyknowifcookiesareontheirharddrive?Howdoestherespondentgetintouchwiththeprivacyofficer?Informationisavailableinthe“ContactUs”portionofthesite,buttherespondenthastolookforit.Itmakessensethatthepublicisincreasinglysuspiciousofonlinesurveysandhowtheirdataareused.
Youcanhelpavoidsomeoftheseproblemsbybeingcertainyouhaveconsideredallthepitfallsofsendingsurveysandsurveyinformationintocyberspace.
IfyouplantousetheInternet(includingemail)to(1)communicatewithstudyparticipantsor(2)sendparticipantinformationtoacollaboratororcontractor,youshouldbeabletocompletethefollowingquestionnaireformaintaininganethicallysoundonlinesurvey.
Eachsurveyhasdifferentlimitsonwhatitneedstocollectandfromwhom.Somesurveysamplesaremorevulnerablethanothersandneeddifferentsafeguards.Thefollowingisaninformed-consentformtypicalofonethatcouldbeusedinanonlinesurveyofteachersinalargeschooldistrict.Thesurvey’spurposeistoidentifyneedsforimprovementintheworkplace.
Somelargeinstitutionsandcompanieshaveethicsboardsandprivacyofficerswhocanhelpyouensureanethicalsurvey.Manysurveyorsandsurveycompanies,however,arenottechnicallysophisticatedregardingprivacy,noraretheytrainedinonlinesurveyethics.Youcanlearnmoreaboutethicalsurvey
researchbygoingonlinetotheNationalInstitutesofHealth’sguidelinesonethicalresearch(http://grants.nih.gov/grants/policy/hs/ethical_guidelines.htm)andtheCollaborativeInstitutionalTrainingInitiative,whichprovidestraininginethicalresearchwithhumansubjects(https://www.citiprogram.org/Default.asp).
ExampleQuestionnaire:MaintaininganEthicallySoundOnlineSurvey
1. DescribethemeasuresthatwillbetakentoensurethatthewebserverhostingtheInternetsiteisprotected.Inthedescription,provideinformationonphysicalsecurity,firewalls,softwarepatches/updates,andpenetrationdrills.
2. Ifapasswordorothersecureauthorizationmethodistobeusedtoallowaccesstothewebsite,askthesetwoquestions:Howwilluserpasswordsbedistributed?Howwillpasswordsandwebaccessbeterminated?
3. Iftheusersessionisencrypted,describethemethodofencryptionthatwillbeused.4. Explainwhowillhaveadministrativeaccesstodataonthewebserver.Givenames,studyroles,andorganizational
affiliations.5. Explainindetailtheadministrativesafeguardsputinplacetorestrictunauthorizedandunnecessaryaccess.6. Describehowtheinformationwillbeused.Willyougive,sell,ortransferinformationtoanyone?7. Givethenameandaddressoftheapplicationowner,thatis,thepersonsorpersonwhomaintainstheapplication.8. Ifemailisusedtocontactrespondents,describethemeasurestakentoassurerespondentsthatthecommunicationisfrom
anauthorizedperson.9. Ifrespondentsareaskedtocontactthesurveyorsusingemail,describehowtherespondentswillbeauthenticatedto
adequatelyensurethesourceoftheemailcommunication.10. Explainhowthestudyconsentformdescribesthepotentialriskstoprivacyassociatedwiththeuseofemail.11. Ifemailistobeusedtosendstudydatatoinvestigators,vendors,orothers,explainifandhowtheemailwillbeencrypted.12. Ifrespondentsaretosendyouattachmentsbyemail,tellthemiftheattachmentswillbeencryptedorpasswordprotected.13. Ifautomatedemailroutingsystemsareused,describethesecuritycontrolsthatwillbeinplace.Specifically,describethe
testinganddisasterrecoveryprocedures.14. Ifcontractorsorvendorshaveaccesstosurveyrespondents’personalidentifiableorconfidentialinformation,
describethelanguagethatisincludedinthecontracttoprotectrespondentprivacyanddescribethesecurityrequirementsthatwillbeprovidedtocontractorsorvendorswhoaredesigningorhostingweb-basedservicesfortheproject.
15. Givethenameofthepersononthesurveyprojectresponsibleforensuringthatthesurveyorganization’spoliciesandproceduresforconfidentialityandsecurityarefollowedforthisproject.Providehisorherprofessionalpositionandaffiliation.
16. Givethenameofthepersonresponsibleforthegeneralsecurityadministrationfortheinformationtechnologyassociatedwiththisparticularsurvey.Providehisorherprofessionalpositionandaffiliation.
Example:Informed-ConsentFormforanOnlineSurveyYourindividualresponsestosurveyquestionswillbekeptconfidentialbyTheSurveyProjectanditssurveycontractor,OnlineSystems,Inc.Confidentialdataareinformation,suchasanindividual’sorschool’sidentification,thatmaynotbereleasedoutsideofTheSurveyProject,exceptwithpermissionfromtherespondent.IndividualsmaygrantTheSurveyProjectpermissiontoreleaseconfidentialdatathatdescribethemselves.AnauthorizedrepresentativeofaSurveyProjectmemberschoolmaygrantTheSurveyProjectpermissiontoreleaseconfidentialdatathatdescribehisorherschool.[Comment:Thisdefinesanddescribesthelimitsofconfidentiality.]
OnlineSystems,Inc.willgenerateaggregatereportsthatcontainschoolwideanddepartmentalinformationtohelpyourschoolidentify,prioritize,andimplementimprovementsintheschoolworkplacethatwillincreasestudentengagement.Informationwillnotbereportedininstanceswhererespondentgroupscontainlessthanfiveindividuals.[Comment:Itmaybepossibletoidentifyindividualviewsinverysmallgroups.Thiswouldviolateprivacy.]Datafromopen-endedquestionswillbeprovidedtoyourschoolindeidentified,redactedform.OnlydeidentifiedrecordleveldatawillberetainedbyTheSurveyProject,andonlydeidentifiedaggregateanalyseswillbesharedinpublicationsandresearchpresentationswiththeacademiccommunity.[Howthedatawillbeused]TheSurveyProjectmayreleasedeidentifiedresponsestoindividualswhoagreetoprotectthedataandwhoagreetoTheSurveyProject’sconfidentialitypolicies.OnlineSystems,Inc.willstoredataonsecureserversandwilldestroyall
identifieddatawithin2yearsofsurveyadministration.Byparticipating,youwillbecontributingvaluableinformationtoyourschool.[Comment:Serverswillbesecured.Thevendormustdestroyidentifiabledatawithin2years.]TheSurveyProjectandOnlineSystems,Inc.havetakennumerousstepstoprotectparticipantsintheSurveyProject.EthicsBoardrequirementsrequirethatyouareinformedthatiftheinformationcollectedweretobecomepublicwithindividualidentificationitcouldprovepersonallyuncomfortable.[Comment:Thisisariskofparticipation.]
ThissurveyhasbeenreviewedandapprovedaccordingtoTheSurveyProject’spoliciesandprocedures.Bycontinuing,youacknowledgethatyouhavereadandunderstoodtheaboveinformationandagreetoparticipateinthissurvey.[Comment:Thisisanonlinesurvey,andtherespondentisnotaskedto“sign”toindicatewillingnesstoparticipate.Signingsoftwareisavailable,butmostsurveysacceptsurveycompletionasinformedconsent.]Ifyouhaveanyquestionsaboutthesurvey,contact....Ifyouhaveanyquestionsaboutyourrightsasaresearchparticipant,contact...[Comment:Whomtocontactwithquestions].
ChildrenandSurveyEthicsWhoisachild?Thedefinitionofachilddiffersacrossresearchjurisdictionssuchascountriesandindividualstates.IntheUnitedStates,achildusuallybecomesanadultby18yearsofage.Thatmeansthata17-year-oldintheUnitedStatesmaystillbeconsideredachild.IntheUnitedStates,theChildren’sOnlinePrivacyProtectionAct(COPPA)requiresverifiableconsentfromaparentorlegalguardianforinterviewingchildrenbelowtheageof13years.Therulesforethicalsurveysareoftenstricterforchildrenthanforadults.
Whendoingsurveyswithadults,thefirststepistomakesuretheyknowwhattheyaregettingintoandhowtogetoutofitiftheywantto.Thatmeanswithadults,thefirststepforparticipationistoprovidethemwiththemeansofprovidinginformedconsent.
Theinformedconsentmodelwithchildrenrequiresparentsorlegalguardianstoprovidepermissionfor(1)theirchildrentotakethesurveyand(2)thesurveyortocontactthechildren.Oncepermissionisobtained,thechildrenareaskedfortheirassenttobecomeparticipants.
IntheUnitedStates,federalregulationsspecifywhatmustbeincludedinanadultconsentprocess,andtheserequirementsalsoapplytotheparentalpermissionprocess.Therearenoregulationsthatrequirespecificelementsordefinethecontentorformatofthechildassentprocess.Thismeansthatsurveyorsareleftontheirowntopreparetheadultandthechilddocuments.
Researchaboutchildren’sdecision-makingskillssupportsthecommonpracticeofusingdifferentassentprocessesforchildrenandforadolescents,withthelevelofdisclosureincreasingaschildrengrowolder.Surveyorsshouldprobablyoffermatureadolescentsinformationaboutthesurveythatiscomparabletoinformationthattheyprovidetoadults.
Don’tforgettoconsiderthenationality,ethnicity,andsocioeconomicstatusofyourpotentialrespondentswhenputtingtogetherparentalpermissionandchildassentprocessesanddocuments.Also,rememberthatculturalassumptionsabouttherightsofchildrenvarywidely.Insomecountriesorsubgroupsitmaybeinappropriateandperhapsoffensivetoaskchildrentocompletequestionnaireson-oroffline.
Whengoingthroughtheassentprocess,makesurechildrenknowthefollowing:
Theyarebeingaskedtoparticipateinaresearchstudyorinalocalefforttounderstandmoreaboutaproblem(howtoimproveourschool;howtomaketheinternetsafer).Thepurposeofthesurvey(tofindouthowchildrenliketolearn;tounderstandhowmuchtime
childrenspendonsocialnetworks).Howmuchtimeisinvolvedinparticipating(nomorethan5minutes).Whatwillhappentothemiftheyagreetoparticipate(youwillanswerquestionsonline).Theforeseeablerisksand/ordiscomforts(youmayfindsomequestionshardtoanswer)andanybenefitstheymayexperience(youmayhelpotherchildrenyourage).Thattheycanasktheirparentsorsurveyoranyquestionstheyhaveaboutparticipating.Thattheirparticipationisvoluntary(entirelyuptothem)andthattheymaystopatanytime.Thattheirfather/mother/guardianknowstheyhavebeenaskedtobeapartofthestudy.
Hereisasampleassentform,onethatismostappropriateforchildrenbetween10and15.
Example:ChildAssentFormHello,
Wearedoingastudytolearnaboutpeoplewhoarehonestandpeoplewhoarenothonest.Weareaskingyoutohelpbecausewedon’tknowmuchaboutwhetherkidsyouragecomeacrossmostlyhonestpeopleormostlydishonestpeople.Wewillusethisinformationtoeducateparentsandteacherstobecomemoreunderstandingoftheworldtheirchildrenlivein.
Ifyouagreetobeinourstudy,wearegoingtoaskyousomequestionsaboutpeople.Wewanttoknowifyouthinktheyareusuallyhonestornot.Forexample,wewillaskyouifyourfriends,peopleyouknow,yourteachers,andfamilymembersareusuallyhonestorusuallynothonest.Wewillnottellyouranswerstoanyone.
Youcanaskquestionsaboutthisstudyanytime.Ifyoudon’twanttoansweraquestion,youdonothaveto.Ifyouwanttostopansweringquestions,youcanleavethestudy.Absolutelynothingwillhappentoyouifyoudecidetoleavethestudy.Noonewillbeupset,andwewillnottellanyone.
Ifyoudon’twanttobeinthisstudy,donotsignthispaper.Beinginthisstudyisuptoyou.
Yoursignature:
Signatureofpersonobtainingassent:
InternationalSurveysIfyouareinvolvedinaninternationalsurvey,keepinmindthatthecontentofaquestionnairereflectsnotonlythelanguageoftheoriginatingcountrybutalsothestandards,expectations,values,andpreoccupationsofthesurveyorsandanylaypeopleinvolvedinhelpingtodevelopandtestthequestions.Linguistsandanthropologistsoftenpointoutthatlanguageandculturearenotthesame.Forexample,bothEnglandandtheUnitedStatesareprimarilyEnglish-speakingcountries,buttheircitizensdonotnecessarilysharethesamevalueswithrespecttoeducation,health,andsocialwell-being.
ThinkaboutthewordfamilyinEnglish.ManyAmericans,forexample,interpretfamilytoincludeonlyrelativeslikemotherorsisterormaybeauntanduncle.Butinothercultures,familyincludesamuchwidergroup.Aquestionsuchas“Hasanyoneinyourfamilyhadheartdisease?”requirestheinternationalsurveyortomakecertainthattheresponsechoicesareinclusiveandmeaningfultotherespondents.Anotheroptionistobespecificandask:Haveanyofthesepeopleinyourfamilyhadheartdisease(mother,father,grandmother,grandfather)?
English-speakingsurveyorsmayfindthatsometimesitisimpossibletofindequivalenttranslationsinlanguagessuchasArabic,Cantonese,Punjabi,orSwahili.Theselanguageshavedifferentrootsfrom
English.Theterms“checkup”and“Papsmear,”forexample,havenoconceptualequivalentinanyChineselanguage.
Ethicalresearchininternationalsettingsmeansthatculturallyappropriateproceduresmustbefollowedtoprotectsurveyparticipants.ManyIRBsandethicsboardsrequiresurveyorstosubmitproofthatthecustomsandlanguageoftheparticipatingcountrieshavebeenrespected.
Considerthisifyouaredoinganinternationalsurvey:
1. Besurethesurveyisavailableintheoriginallanguage(English)andintranslation.Besureyoucanprovideadetailedexplanationofthetranslationprocesstojustifyitsappropriatenessforthecountryinwhichitistobeused.
2. Ifthesurveyincludeschildren,youneedtobeabletoverifytheageatwhichparticipantshavetheabilitytoconsentorassenttoparticipationinsurveys.
3. Iflocalcustomsandregulationsaresuchthatactiveparentalpermissionwouldbeculturallyinappropriate,thesurveyorshouldcompileproofthatsuchpermissionisnotculturallyappropriate.Examplesofsuchproofwouldbedocumentswithspecificregulations(inEnglishandcertifiedtobeaccurate)thatindicatethatsuchpermissionisnotrequiredoranofficialletterfromarankingofficialinthecountryofinterestindicatingthatsuchpermissionisnotculturallyappropriate
4. Ifthesurveyoriseligibleforawaiverofactiveparentalpermission,heorsheshouldbepreparedtoprovideparentswithaletterdescribingthestudy.Thelettershouldbewrittenintheparent’slanguageandatanappropriateliteracylevel.
5. Thesurveyoralsoneedstogetlettersofagreementfromtheappropriateofficials(governmentofficials,schoolofficials,communityofficials)thatstatethatthesurveyitselfhasbeenreviewedandisacceptable.Besurethattheletterisonletterheadandhasanoriginalsignature.
FormalStandardsforSurveyEthicsTwomajorsurveyresearchorganizationscanprovideyouwithstandardsforconductingsurveyresearchtocoverallsituations.TheAmericanAssociationforPublicOpinionResearch(AAPOR)hasdevelopedacodetodescribetheobligationsthatallprofessionalshavetoupholdthecredibilityofsurveyandpublicopinionresearch.ToreadthecodeandtogetmanyotherresourcesincludinghowtoworkwithanIRB,gotohttp://www.aapor.org/AAPORKentico/Standards-Ethics.aspx.
AAPORalsoprovidesalistofsurveypracticesthatitcondemns.Thisincludesrequiringamonetarypaymentorsolicitingmonetarycontributionsfrommembersofthepublicaspartofaresearchprocess,offeringproductsorservicesforsale,usingparticipantcontactsasameansofgeneratingsalesleads,andrevealingtheidentityofindividualrespondents.
TheCouncilofAmericanSurveyResearchOrganizations(CASRO)haspreparedaCodeofStandardsandEthicsforSurveyResearch(http://www.casro.org/).CASROwasestablishedtorepresenttheU.S.researchindustryandorganizationsengagedintheconduct,support,oreducationofmarket,opinion,andsocialresearch,oftendescribedasdataanalytics,consumerinsights,orbusinessintelligence.CASRO’sethicsdocumentcoverstopicslikeconfidentiality,privacy,theavoidanceofharassment,internetresearch,responsibilitiestoclientsandresponsibilitiestooutsidecontractorsandinterviewers.Inthepreambletothecode,CASROstatesthatrespondentsshouldbewillingparticipantsinsurveyresearch,appropriatelyinformedaboutthesurvey’sintentionsandhowtheirpersonalinformationandsurvey
responseswillbeusedandprotected,sufficientlysatisfiedwiththeirsurveyexperience,andwillingtoparticipateagaininsurveyresearch.
SummingUpSurveysareinformationcollectionmethodsusedtodescribe,compare,orexplainindividualandsocietalknowledge,feelings,values,preferences,andbehavior.Surveysarebestwhenyouneedinformationdirectlyfrompeopleaboutwhattheybelieve,know,andthink.Asurveycanbeaself-administeredquestionnairethatsomeonefillsoutaloneorwithassistance.Self-administeredquestionnairescantaketheformofwrittenoronlinesurveys.Writtensurveysmaybecompletedbymailoronlocation.Asurveycanbeaninterviewdoneinpersonoronthetelephone.Allsurveysconsistofquestionsthatincludetheopportunitytorespond.Thatiswhythetermquestionnaireisoftenusedinterchangeablywiththetermsurvey.Togetaccuratesurveydata,youmusttakeintoaccountthesurvey’squestions,responsechoices,samplingmethods,responserate,design,anddataanalysis.Surveyresultsarepresentedaswritten(printedonpaperorreproducedelectronically)andoralreports.Areliablesurveyproducesconsistentinformation,whileavalidoneresultsinaccurateinformation.Mailsurveysareoftenusedbecausepeoplearefamiliarwiththem;however,theresponserateisoftendismalwithoutagreatdealoffollow-upandincentives.Interviewersneedtraining,whichmaybetime-consumingandcostly.TheInternetisanefficientmethodofreachingawideaudience,buttechnicalexpertiseisneededwhetheryoudoitonyourownoruseacommercialfirmtoassist.Onlinesurveys,likeallsurveys,requirecarefuladvanceplanning.Cellphonesurveyscanbecostly,anditisdifficulttosampleamongpeoplewhojusthavelandlines,thosewithlandlinesandcellphones,andthosewithonlycellphones.Surveysmaybedoneforspecificorgeneralpurposes.Surveyfindingsthatareneededformanypeopleandplaceswillrequirespecialattentiontohowtheyaredesigned.Allcompletewrittensurveysshouldbekeptinlockedfiles,andonlyalimitednumberofstaffshouldhaveaccesstothemonaneed-to-knowbasis.Attheconclusionofdataanalysis,thesurveysshouldbeshredded.Separateidentifyinginformation(e.g.,names,birthdates,SocialSecuritynumbers)fromsurveyresponsesbyassigningcodestoindividualsandusingthecodestolinkthemtotheirresponses.Onlinesurveytakersmaybegiventheoptionofchoosingtheirownidentificationnamesandpasswordswhenloggingintoasurvey.Ifemailaddressesareusedtoinformparticipantsaboutandprovidealinktothesurvey,theseshouldbemadeavailableonlytoaselectgroupofsurveyors.IfasurveyisforapublicorprivateagencythatisreceivingU.S.governmentfunds,therearestringentlegaldimensionsofinformedconsent,privacy,andconfidentiality.Thesedimensionsincludethefollowing:
Afairexplanationoftheprocedurestobefollowedandtheirpurposes.Adescriptionofanyrisksandbenefits.Anoffertoansweranyinquiries.Aninstructionthatthepersonisfreetowithdrawconsentandtodiscontinueparticipation
withoutprejudice.Informedconsenttoparticipateinasurveyimpliesawillingnesstocompleteitbecauserespondentsunderstandthesurvey’spurposeandprocedures,thepotentialrisksandbenefitsofresponding,whetherthereispaymentforparticipation,howconfidentialitywillbehandled,andwhetherwithdrawalwithoutpenaltyispossible.Asurvey’sprivacypolicyshoulddothefollowing:
Describethedatathatwillbestoredinthesurvey’sdatabase.Explainwhythedataarebeingstored.Explainwhethertheorganizationgives,sells,ortransfersinformationandifitdoes,towhomandunderwhichcircumstances.Informhowunauthorizedattemptstochangethesite’scontentsaremonitored.Discusswhomaintainsthesite,andifrelevant,explainhowcookiesareused.Aretheysessionspecific?Ifnot,canusersoptoutofthewebpagefeaturethatstoresthecookiesbeyondthesession?Haverulesforethicalresearchandsurveysthatareoftenstricterforchildrenthanforadults.
Theinformedconsentmodelwithchildrenrequiresparents(orlegalguardians)toprovidepermissionfortheirchildrentotakethesurveyandtoallowthesurveyortocontactthechildren.Childrenthenprovidetheirassenttobecomeparticipants.
Ifyouconductsurveysinternationally,culturallyappropriateproceduresmustbefollowedtoprotectsurveyparticipants.Ifyouarenotsurewhatisandwhatisnotethical,gototheAmericanAssociationforPublicOpinionResearchortheCouncilofAmericanSurveyResearchOrganizations.
ThinkAboutThisReadthedescriptionofeachsurveybelowandthenanswerthesequestions:
1. Whatisthesurveyabout?2. Namethesurveymethod.3. Isthisthemethodyouwouldhavechosentoachievethesurvey’sobjectives?Explain.4. Listthreeethicalconcernsthatmustbeaddressedineachofthesurveys.
Evaluatetheadequacyofthefollowingsectionofasurveyorganization’sprivacypolicy.Whichcriteriadidyouuseinmakingyourevaluation?
SurveyDescriptions
PredictingHeartProblemsfromtheStressoftheSeptember11,2001,TerroristAttacksontheUnitedStatesHealthresearcherswantedtoexaminethedegreetowhichstressresultingfromtheSeptember11,2001,terroristattacksontheUnitedStatesforecastedheartproblemsinthenation3yearsafter.Theyenlistedanationalsampleofadultsinaweb-basedsurveyoftheirstress.Thepeoplewhoparticipatedhadcompletedahealthsurveybeforetheattackssotheinvestigatorshad“baseline”informationonthem.Within2weeksoftheattacks,theparticipantscompletedwebsurveys,andtheycontinuedtodoso1,2,and3yearsafter.Theresearcherscontactedthesurveyparticipants’physicianstofindoutwhethertheyhaddiagnosedcardiovascular(heart)ailmentsoverthe3-yearperiod.TheresearchersfoundthatacutestressresponsestotheSeptember11
attackswereassociatedwitha53%increasedincidenceofcardiovascularailmentsoverthe3subsequentyears.
SchoolFurnitureandLower-BackPainThissurveywasdesignedtofindoutwhethersometypesofschoolfurniturepreventorcauselower-backpaininchildren.Fivehundredforty-sixschoolchildrenages14to17yearsansweredawrittenquestionnaireaboutsittingpositionsduringschoolhoursandthepresenceandseverityoflower-backpain.Thedimensionsandtheweightofthechildren’sschoolbagsweremeasured,aswerethetypesanddimensionsoftheschoolfurniture.Thefindingswerethatmorethanhalfoftheadolescentsexperiencedlower-backpainduringthepreceding3months,andaboutone-quarterreportedreduceddailyfunctionorcareseekingbecauseoflower-backpain.Lower-backpainoccurrencewasnotfoundtobeassociatedwiththetypesordimensionsoftheschoolfurnitureorbodydimensionsbutwaspositivelyassociatedwithcarryingtheschoolbagononeshoulder.
MaleandFemaleParticipationinJazzImprovisationThreehundredthirty-twosurveysweregiventostudentsenrolledinmiddleschool,juniorhighschool,highschool,college,andcommunityjazzprogramswithin60milesofamajormidwesternuniversity.Thesurveywasadministeredduringjazzbandrehearsalsandtookapproximately10minutestocomplete.Jazzbanddirectorsand/orstudentteachersadministeredthesurveytomiddleschoolandhighschoolparticipants,whowereinstructedtotakethesurveyhomeforaparent’ssignatureindicatingapprovalforthechild’sparticipationinthestudy.Thefindingsindicatedthatfemalesaresignificantlylessconfident,aremoreanxious,andhavelessself-efficacy(attitude)towardlearningjazzimprovisation.
SatisfactionWithPaidPersonal-AssistanceServicesTraditionalpublichomecareprogramsintheUnitedStatesrelyonpublicorprivateagenciestohireandfirehomecareworkers,scheduleanddirectservices,monitorqualityofcare,disciplineworkersifnecessary,andpayworkersandapplicablepayrolltaxes.Intheagency-directedmodel,clientscanexpresspreferencesforservicesorworkersbuthavenoformalcontroloverthem.Thissurveywasconcernedwithcomparingconsumer-directedversusagency-directedhomecareonsatisfactionwithpaidpersonalassistanceservicesamongMedicaidbeneficiariesinWashingtonstate.Thesurveywasprimarilyconductedthroughtelephoneinterviewsusingacomputer-assistedtelephoneinterviewingsystem,buttherealsowereafewin-personinterviews.Thesurveyfoundthat,amongtheolderpopulation,butnotyoungerpeoplewithdisabilities,beneficiariesreceivingconsumer-directedservicesweremoresatisfiedthanindividualsreceivingagency-directedcare.
TeenDrivers’LicensingPreferencesandPracticesThissurveyusedprobabilitysamplingmethodstoidentify1,38315-to18-year-oldsfromanationallyrepresentativesampleofU.S.households.Theteenscompletedanonlinesurveysothatthesurveyorscouldlearnaboutnationallicensingrates,interestinearlylicensure,andreasonsfordelay.Someoftheteenscamefromcellphone–onlyhouseholds;Internetaccesswasprovidedtothosewithoutit.Thesurveyfoundthatat16years,teenswereaboutequallydividedamongthoseinthelearnerstageandthosewitharestrictedorfulllicense.Forteensoldenoughtostartthelicensingprocess,lackofacar,costs,parentavailability,abilitytogetaroundwithoutacar,andbeingbusywithotheractivitieswereleadingreasonsfordelay.
Parents’ChoiceofTreatmentforTheirChildrenWithAutismParentsofchildrenwithautismspectrumdisordershavealimitednumberofevidence-basedtreatmentsfromwhichtochoose.Ninehundredandseventyparentsrespondedtoanonlinesurveythataskedaboutthetreatmentscurrentlyinuse,thosediscontinued,andreasonsfordiscontinuation.Thesurveyfoundthatmostfamiliesadoptmultipletreatmentapproaches.Parentsweremostlikelytodiscontinuenonevidence-basedtreatmentswhentheydidnotseeimprovementintheirchild’sfunctioning.
InformationSecurityTheSurveyOrganization’semployeesunderstandtheneedforuserprivacy,andwemaintainstrictsecurityprocedurestoprotectyourinformation.TheOrganizationhasappointedaPrivacyPolicyAdministratortomonitorprivacypractices.Accesstouserdataisstrictlylimitedtospecificindividualswhoaretrainedtorespectuserprivacy.Theaccessgiventotheseemployeesisrestricted
totheirneedofsuchinformationforbusinessreasons.Alogofthosewhoaccessedthedataismaintainedandmonitoredtopreventsecuritybreaches.
REFERENCESBowker,A.,D’Angelo,N.,Hicks,R.,&Wells,K.(2011).Treatmentsforautism:Parentalchoicesandperceptionsofchange.JournalofAutismandDevelopmentalDisorders,41(10),1373–1382.
Courtney,M.,Zinn,A.,Johnson,H.,&Malm,K.(2011).EvaluationoftheMassachusettsAdolescentOutreachProgramforYouthsinIntensiveFosterCare:Finalreport.
Holman,E.A.,Silver,R.C.,Poulin,M.,Andersen,J.,Gil-Rivas,V.,&McIntosh,D.N.(2008).Terrorism,acutestress,andcardiovascularhealth:A3-yearnationalstudyfollowingtheSeptember11thattacks.ArchivesofGeneralPsychiatry,65(1),73–80.
Skoffer,B.(2007).Lowbackpainin15-to16-year-oldchildreninrelationtoschoolfurnitureandcarryingoftheschoolbag.Spine,32(24),E713–E717.
Wehr-Flowers,E.(2006).Differencesbetweenmaleandfemalestudents’confidence,anxiety,andattitudetowardlearningjazzimprovisation.JournalofResearchinMusicEducation,54(4),337–349.
Wiener,J.M.,Anderson,W.L.,&Khatutsky,G.(2007).Areconsumer-directedhomecarebeneficiariessatisfied?EvidencefromWashingtonState.Gerontologist,47(6),763–774.
Williams,A.F.(2011).Teenagers’licensingdecisionsandtheirviewsoflicensingpolicies:Anationalsurvey.TrafficInjuryPrevention,12(4),312–319.doi:10.1080/15389588.2011.572100.
2THESURVEYFORMQuestions,Scales,andAppearance
OverviewTodecideonasurvey’scontent,youhavetodefinetheattitude,belief,value,orideabeingmeasured.Forexample,whatismeantbyfear?Byaliberalperspective?Byself-efficacy?Alsoaskthis:WhichquestionsmustbeaskedinavalidsurveyifIwanttomeasurefear(oraliberalperspectiveorself-efficacy)?CanIgettheinformationIneedfromasurvey,orissomeotherdatacollectionmethodabetterwaytogo?
Theresponsestoclosedquestionscantaketheformofyes-or-noanswers,checklists,andratingscales.Checklistsgiverespondentsmanychoices(e.g.,“checkallthatapply”).Ratingscalesrequirerespondentstorank(e.g.,“1=top,10=bottom”)ororder(e.g.,“1=definitelyagree,2=agree,3=disagree,4=definitelydisagree”)ideas.Respondentsmayalsobeaskedtogiveanumberforananswer(e.g.,“Whatisyourage?”).
Responsestoquestionsproducedatathatarecalledcategorical(e.g.,“yesorno”),ordinal(e.g.,“big,bigger,biggest”),andnumericalorcontinuous(e.g.,“20,21,22yearsold”).Thesurveyorhastoconsiderthetypeofdatathesurveyproducesinordertocomeupwithanappropriatemethodofanalyzingandinterpretingitsfindings.
Surveyorsareofteninterestedinresponsestoindividualitems,suchasthenumberofpeoplewhowillvoteforCandidateXorhowoftenwomenbetween65and80yearsofagevisitadoctorina3-monthperiod.Butsometimestheyareconcernedwithascoreonagroupofitemsthatcollectivelyrepresentrespondents’views,healthstatus,orfeelings.
Onlinesurveyshavemanyusefulfeaturessuchasdrop-downmenusandtheabilitytobuildontheanswertoonequestiontocreatethenext.Youcanalsoreachlargenumbersofpeoplewithoneclick.Considerthecosts,whichincludepurchasefeesandtimetolearntousenewsoftware.
TheContentIstheMessageOnceyouhavedecidedthatasurveyisthemethodyouwanttousetogatherdata,youmustconsiderthecontentortopicsitwillinclude.
Supposeyouareevaluatingayouthcenter’sprogressandyourmaintaskistofindoutwhethertheprogram’sobjectiveshavebeenachieved.Saythatoneoftheobjectivesistoraiseyoungpeople’sself-esteembyprovidingthemwitheducation,jobs,financialhelp,andmedicalandmentalhealthassistance.Supposealsothatyoudecidetosurveytheyoungprogramparticipantstofindoutabouttheirself-esteem.Howwouldyoudeterminewhichcontenttoinclude?
Toselectthecontentofasurvey,youhavetodefineyourtermsandclarifywhatyouneedandcangetfromaskingpeopleabouttheirviews.
DefinetheTermsManyhumanattitudesandfeelings,suchasself-esteem,aresubjecttoarangeofdefinitions.Doesself-esteemmeanfeelinggoodaboutoneselfand,ifso,whatdoesfeelinggoodmean?Thesurveyorcananswerquestionssuchasthisbyreviewingtheresearch(inthiscase,theliteratureonself-esteem)tofindoutwhatisknownandtheorizedaboutaconceptsuchasself-esteem,consultingwithexperts,ordefining
theconceptforhimselforherself.Theproblemwithusingyourowndefinitionisthatothersmaynotbeconvincedofitsvalidity.Whenusingatheoreticalconceptsuchasself-esteem,itisprobablybesttoadoptarespectedpointofviewand,ifpossible,analreadyexistingandtestedsurveyform.
Ofcourse,formanysurveys,youwillnotbemeasuringtheoreticalideas,butevensoyoumustdefineyourterms.Supposeyouareassessingacommunity’sneedsforhealthservices.Thetermsneedsandhealthserviceswouldcertainlyrequiredefinitionbecauseyoucandefinethemwithrespecttothetypeandnatureofservicesrequired(outpatientclinics?hospitals?homevisits?)andhowconvenient(timeofdaywhendoctorsshouldbeavailable)orhowcontinuoustheyshouldbe(mustthesamedoctoralwaysbeavailable?).
SelectYourInformationNeedsorHypothesesSupposetwosurveyorschoosethesamedefinitionofself-esteemfortheirevaluationstudyoftheyouthcenter.Dependingonthecircumstances,Surveyor1mightdecidetofocusonself-esteeminrelationtofeelingsofgeneralwell-being,whereasSurveyor2maybeconcernedwithfeelingsofself-esteemonlyastheymanifestthemselvesinschooloratwork.CertainlySurveyors1and2,withtheirdifferentorientations,willaskdifferentquestions.Theresultswillyielddifferentkindsofinformation.Surveyor1,withaconcernforgeneralself-esteem,maynotevencoverworkorschoolandwillnotbeabletoprovidedataonthesetopics.Surveyor2,withhisorherspecialinterestsinschoolandwork,probablywillnotprovideinformationonparticipantself-esteemwithrespecttopersonalrelationships.Themessagesrevealedbyeachsurveywillclearlybedifferent.
Sayyouareinterestedinwhetherparticipantsintheyouthcenterhadtheirgeneralself-esteemenhancedafter2yearsofparticipationintheprogramandthatyouhavedefinedyourtermstoconformtoanacceptedtheoryofadolescentpersonality.Tomakesureyougetallthedatayouneed,youmustaskthisquestion:WhatinformationdoIwantandmustthereforemakecertainIcollect?Remember,ifyoudonotaskforit,youcannotreportitlater!
Herearesometypicalquestionsthattheevaluatoroftheyouthcentercouldask:
Istherearelationshipbetweengeneralfeelingsofself-esteemandwhethertheparticipantisaboyorgirl?Doesparticipants’self-esteemdifferdependingonhowlongtheyhavebeenintheprogram?
Theevaluatorcanalsocomeupwithhypotheseslikethese:
1. Boysandgirlsdonotdifferintheirgeneralfeelingsofself-esteem.2. Participantswhohavebeenintheprogramlessthan12monthswilldifferintheirself-esteemfrom
thosewhohavebeenintheprogram12monthsormore.
Thesequestionsandhypothesessuggestthatthesurveymustgetdataonthreetopics:
1. Generalfeelingsofself-esteem2. Sexofparticipant3. Lengthofparticipationintheprogram
Ifanyofthesetopicsisomitted,thesurveyoroftheyouthcentercannotanswerthetwoevaluationquestions.
MakeSureYouCanGettheInformationInsomecases,peoplemaybereluctanttorevealtheiropinions.Theevaluatoroftheyouthcenter,forexample,maydiscoverthatmanyyoungpeoplearereluctanttoanswerquestionsabouttheirfeelings.Inothercases,potentialsurveyrespondentsmaysimplybeunabletoprovideanswers.Supposeyouwanttoaskparticipantswhohavebeenattendingtheyouthcenterforabout6monthsabouttheirattitudestowardschooljustbeforeenteringthecenter’sprogram.Manymayhaveforgotten.Aninabilitytoanswerquestionsisnotuniqueandisamajorproblemwithaccuratelypredictingvoters’preferencesinnationalelections.Itoftentakestimeforpeopletosettleonacandidate,andsomepeoplechangetheiropinionsseveraltimesoverthecourseofacampaign.Thatisonereasonthatpollsproduceresultsthatdifferamongthemselvesandfromonepointintimetoanother.
Ifyouarenotcertainyoucangettheinformationyouneedfromasurvey,removethetopicandfindanotherdatasource,suchasobservationsorreviewsofrecordssuchasdiariesandreportsofschoolattendanceorhospitaladmissions.
DoNotAskforInformationUnlessYouCanActonItInasurveyofacommunity’sneedsforhealthservices,itwouldbeunfairtohavepeopleratetheirpreferenceforatraumacenterifthecommunityisunabletosupportsuchaservice.Rememberthatthecontentofasurveycanaffectrespondents’viewsandexpectations.Whyraisehopesthatyoucannotorwillnotfulfill?
Onceyouhaveselectedthecontentandsetthesurvey’sboundaries,yournexttaskistoactuallywritethequestions.Writemorequestionsthanyouplantousebecauseseveralwillprobablyberejectedasunsuitable.Firstdraftsoftenhavequestionsforwhicheveryonegivesthesameanswerornoonegivesanyansweratall.Beforedecidingonthenumberandsequenceofquestions,youmustbesurethatyoucoverthecompletedomainofcontentyouhaveidentifiedasimportanttothesurvey.Youmaywanttokeepalistsuchasthefollowingoneusedbythesurveyorofparticipantsatisfactionwiththeyouthcenter.Asyoucansee,thissurveywillnotcoverstaffsensitivitybutwillfocusinsteadonconsideration,accessibility,andavailabilityofservices.
Example:PlanforSurveyofSatisfactionWiththeYouthCenter
WritingQuestions
Open-EndedandClosedQuestionsSurveyitemsmaytaketheformofquestions.
Example:Open-EndedQuestion1. Howcourteousarethepeoplewhomakeyourappointments?
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
Ortheymaybewordedasstatements.
Example:ClosedQuestionCircleyouragreementordisagreementwiththefollowing:
2.Thepeoplewhomakemyappointmentsarecourteous.
Mark(X)onechoice:
Sometimessurveyitemsareopen-ended,meaningthattherespondentsagreetoanswerthequestionorrespondtothestatementintheirownwords.Question1isopen-ended.Atothertimes,surveyitemsforcetherespondenttochoosefrompreselectedalternativesasinQuestion2.
Theoverwhelmingmajorityofsurveysrelyonmultiple-choicequestionsbecausetheyareefficientandareoftenmorereliablethanotherquestiontypes.Theirefficiencycomesfrombeingeasytouseand
score.Also,theirreliabilityisenhancedbecauseoftheuniformdatatheyprovide;everyonerespondsintermsofthesameoptions(e.g.,“agreeordisagree”or“frequentlyorinfrequently”).
Open-endedquestionscanofferinsightintowhypeoplebelievethethingstheydo,butinterpretingthemcanbeextremelydifficult,unlesstheyareaccompaniedbyanelaborateclassificationsystemandpeoplearetrainedtocategorizethedatatheygetwithinthesystem.
Considerthesetwoanswerstoaquestionfromasurveyofparticipantsinanelementaryschoolteachingprogram.
Example:Open-EndedQuestionforElementarySchoolTeachingProgramQuestion:Whatwerethethreemostusefulpartsoftheprogram?
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
Answers:
RespondentAInstructor’slecturesThefieldexperienceTextbook
RespondentBInstructorTeachingintheclassroom
Themostusefulpartwastheexcellentatmosphereforlearningprovidedbytheprogram.
ItisnoteasytocompareRespondentA’sandRespondentB’sresponses.RespondentBliststheinstructorasuseful.Doesthismeanthattheinstructorisausefulresourceingeneral,andhowdoesthiscomparewithRespondentA’sviewthattheinstructor’slectureswereuseful?Inotherwords,areRespondentsAandBgivingthesameanswer?RespondentAsaysthetextbookwasuseful.Ifonlyonetextwasusedintheprogram,thenRespondentsAandBgavethesameanswer.Butbecausethetworecordedresponsesaredifferent,someguessingorinterpretationofwhatismeantisnecessary.
RespondentsAandBeachmentionedsomethingthattheotherdidnot:fieldexperienceandlearningatmosphere.Ifthesewereequallyimportant,thentheycouldbeanalyzedindividually.Butsupposeneitherwasparticularlysignificantfromtheperspectiveofthesurvey’susers.Wouldtheythenbeassignedtoacategorylabeledsomethinglikemiscellaneous?Categoriescalledmiscellaneoususuallyareassignedallthedifficultresponsesandbeforeanalysisandinterpretationarequitecomplicated.Beforeyouknowit,miscellaneouscanbecomethelargestcategoryofall.
Althoughitmayberelativelyeasyforarespondenttoansweranopen-endedquestion,analysisandinterpretationarecomplicated.Thefollowingclosedquestioncouldhavebeenusedtoobtainthesameinformationwiththeaddedresultofmakingtheresponseseasytointerpret.
Example:ClosedQuestionforElementarySchoolTeachingProgram
1. Pleasetellusifyouaresatisfiedordissatisfiedwitheachofthesecomponentsofourprogram.
MakingtheDecision:Open-EndedVersusClosedQuestionsChooseopen-endedquestionswhenyouwanttogivetherespondentstheopportunitytoexpressopinionsintheirownwordsandyouhavetheinterestinandresourcestointerpretthefindings.Open-endedquestionstendtoberelativelyeasytocompose.
Chooseclosedquestionsfortheirrelativeeaseofscoring,analysis,andinterpretation.Closedquestionscanbedifficulttopreparebecausetheyrequirethatallrespondentsinterpretthemthesamewayandthatallrelevantchoicesareincluded,mutuallyexclusive,andsensible.
OrganizingResponsestoOpen-EndedSurveyItems:DoYouGetAnySatisfaction?Acommonuseofasurveyistofindoutwhetherpeoplearesatisfiedwithanewproduct,service,orprogram.Theiropinionsprovideimportantinsightsintowhynewideasorwaysofdoingthingsdoordonotgetused.
Oneopen-endedsetofquestionsthatisparticularlyappropriateforgettingatsatisfactionrequirescollectinginformationaboutwhatpeoplelikebest(LB)abouttheproductorserviceandwhattheylikeleast(LL).
HereishowtheLB/LLtechniqueworks:
Step1:AskingRespondents’OpinionsAskrespondentstolistwhatisgoodandwhatisbad.Alwayssetalimitonthenumberofresponses;forexample,“Listatleastonething,butnomorethanthreethings,youlikedbestabouttheconference.”Ifparticipantscannotcomeupwiththreeresponses,theycanleaveblankspacesorwrite“none.”Iftheygivemorethanthree,youcankeepordiscardtheextrasdependingontheinformationyouneed.
Insteadofaskingabouttheconferenceasawhole,youmaywanttofocusonsomeparticularaspect;for
example,“Listatleastonething,butnomorethanthreethings,youlikedbestabouttheworkshops.”
Step2:CodingLB/LLDataOnceyouhavealltheresponses,thenextstepistocategorizeandcodethem.Todothis,youcancreatecategoriesbasedonyourreviewoftheresponsesoryoucancreatecategoriesbasedonpastexperiencewithsimilarprograms.
Trytokeepthecategoriesaspreciseaspossible—thatis,morecategoriesratherthanfewer—becauseitiseasiertocombinethemlaterifnecessarythanitistobreakthemup.
Supposethesearetypicalanswersparticipantsgavetothequestiononwhattheylikedleastabouttheconference:
Somepeopledidallthetalking.Theinstructordidn’talwayslisten.Icouldn’tsayanythingwithoutbeinginterrupted.Therewastoomuchnoiseandconfusion.Someparticipantswereignored.Theinstructordidn’ttakecontrol.Ididn’tgetachancetosayanything.SmithandJonesweretheonlyoneswhotalked.Theinstructordidn’tseemtocare.Icouldn’thearmyselfthink.
Youmightcategorizeandcodetheseanswerasfollows:
ExampleLB/LL:ResponseCategories
Nowmatchyourcodesandtheresponses:
ExampleLB/LL:ParticipantResponses
Tomakesureyouassignedthecodescorrectly,youmaywanttoestablishtheirreliability.Aretheyclearenoughsothatatleasttworaterswouldassignthesamecodeforagivenresponse?
Step3:LB/LLDataWhenyouaresatisfiedaboutreliability,thenextstepistocountthenumberofresponsesforeachcode.
Hereishowtodothisstepfor10participants:
ExampleLB/LL:NumberofResponsesforEachCode
Lookatthenumberofresponsesineachcategory.The10participantslisted30thingstheylikedleastaboutthesmallgroupdiscussion.Seventeenof30(morethan50%)wereassignedtothesamecategory,
Code2,andthesurveyorcouldjustlyarguethat,basedonthedata,whattheparticipantstendedtolikeleastabouttheworkshopswasthatsomepeoplemonopolizedthediscussionsandothersdidnotgetachancetosayanything.
Next,countthenumberofparticipantswhoseanswerswereassignedtoeachcode.Forexample,onlyParticipantsAandBgaveanswersthatwerecoded1.
ExampleLB/LL:Participants’ResponsePattern
Lookatthenumberofparticipantswhoseresponsesfiteachcategory.Because8or9ofthe10participantsgaveresponsesthatfellintothesametwocategories(Codes2and3),theiropinionsprobablyrepresentthoseoftheentiregroup.Itissafetoaddthatparticipantsalsodislikedthedisorderlyatmospherethatprevailedduringtheworkshops.Theycomplainedthatthenoisemadeithardtothinkclearlyandtheinstructordidnottakecontrol.
Whenrespondentsagreewithoneanother,therewillbeonlyafewtypesofanswersandthesewillbelistedbymanypeople.Ifrespondentsdisagree,manydifferentkindsofanswerswillturnupontheirlistsandonlyafewpeople(fewerthan10%)willbeassociatedwitheachtype.
InterpretingLB/LLdatagetsmorecomplexwhenyouhavemanyparticipantsandresponsestocategorize.Suppose,forexample,youask100participantstoindicatewhichaspectsofahealtheducationprogramtheylikedbest.
First,youmustdecideonyourresponsecategoriesandassigneachoneacode.Thentrythis:
1. Putthecodesinrankorder.Thatis,ifthelargestnumberofparticipantschoseresponsesthatareassignedtoCode3,listCode3first.
2. Calculatethepercentageofstudentsassignedtoeachcode.If40of100studentsmaderesponsesthatwereassignedacodeof3,thenthecalculationwouldbe40%.
3. Countthenumberofresponsesassignedtoeachcode.4. Calculatethepercentageofresponsesassignedtoeachcode.If117responsesfrom400were
assignedtoCode3,then29.25%or117/400ofresponseswereforCode3.5. Calculatethecumulativepercentageofresponsesbyaddingthepercentagestogether:29.25%+
20.25%=49.50%.
Thefollowingisatablethatsummarizesthesestepswithsomehypotheticaldata.Thetableisthentranslatedintoalinegraphusingstandardsoftware.Thegraphisshownafterthetable.
ExampleLB/LL:SummaryofResponses
Asyoucansee,theresponsecategoriesarerankorderedalongthexaxisaccordingtothenumberofparticipantsassignedtoeachcode.Theyaxisrepresentspercentages.
Foreachresponsecategory,youshouldlookfortwopointsonthexaxis:thepercentageofparticipantsandthecumulativepercentageofresponses.First,thecumulativepercentagesofresponsesarejoinedwithasquare(□).Next,someofthepointsrepresentingpercentagesofparticipantsarejoinedwithadiamond().Thegraphshowsthatresponsescodedas3,4,and7seemtobecloseinresponsefrequency.Theyarethecategoriestobeemphasizedbecausethehighestpercentagesofparticipantswereassignedtothesecodesandtheyaccountfor69%ofallresponses.
Items8,10,and1formasecondintuitivegroupingthatrepresents23.5%ofallresponses.Takentogether,responsescodedas3,4,7,8,10,and1accountfor92.5%ofthetotal.
RulesforWritingClosedSurveyQuestionsMultiple-choice,closedsurveyquestionsconsistofastem,whichpresentsaproblem(typicallyintheformofastatement,question,briefcasehistory,orsituation),followedbyseveralalternativechoicesorsolutions.Herearerulesfortheirconstruction.
1.Eachquestionshouldbemeaningfultorespondents.Inasurveyofpoliticalviews,thequestions
shouldbeaboutthepoliticalprocess,parties,candidates,andsoon.Ifyouintroduceotherquestionsthathavenoreadilyobviouspurpose,suchasthoseaboutageorgender,youmightwanttoexplainwhytheyarebeingasked,forexample,“Weareaskingsomepersonalquestionssothatwecanlookforconnectionsbetweenpeople’sbackgroundsandtheirviews....”
2.Usestandardlanguagerules.Becauseyouwantanaccurateanswertoeachsurveyitem,youmustuseconventionalgrammar,spelling,andsyntax.Avoidspecializedwords(unlessyouaretestingpeople’sknowledgeofthem)andabbreviationsandmakesurethatyouritemsarenotsolengthythatyouareactuallytestingreadingorvocabulary.
Example:Item-WritingSkills—Length,Clarity,Abbreviations,andJargon
LengthPoor:Thepaucityofpsychometricscaleswithhighdegreesofstabilityandconstructvalidityismostbothersometosurveyorswhenmeasuringpeople’s
1. economiccharacteristics2. feelings3. knowledge4. health
Better:Thelackofreliableandvalidmethodscausessurveyorsthemostproblemswhenmeasuringpeople’s
1. economiccharacteristics2. feelings3. knowledge4. health
ClarityPoor:Whatremedydoyouusuallyuseforstomachaches?
Better:Whichbrandofmedicinedoyouusuallyuseforstomachaches?
AbbreviationsPoor:Whichpoliticalpartyisresponsiblefortheexpandingsizeofthegrossdomesticproduct(GDP)?
1. Republican2. Democratic
Better:WhichpoliticalpartyisresponsibleforthediminishingsizeoftheGDP?
1. Republican2. Democratic
JargonPoor:Inyourview,whichdyadismostresponsibleforfeelingsoftrustinearlychildhood?
1. Motherandfather2. Fatherandsibling3. Motherandsibling
Better:Inyourview,whichfamilycombinationismostresponsibleforfeelingsoftrustinearlychildhood?
1. Motherandfather2. Fatherandbrotherorsister3. Motherandbrotherorsister
3.Makequestionsconcrete.Questionsshouldbeclosetotherespondent’spersonalexperience.
Example:Item-WritingSkills—ConcreteQuestionsLessconcrete:Doyouthinkotherpeoplewouldenjoythebook?
Moreconcrete:Haveyourecommendedthebooktoanyoneelse?
Askingrespondentswhethertheythinkotherswouldenjoyabookismoreabstractthanaskingwhethertheyrecommendedittoothers.Whoaretheothers?Canyoubesuretherespondentknowsaboutotherpeople’senjoymentwithrespecttothebook?Thefurtheryouremoveaquestionfromtherespondent’sdirectexperience,thecloseryoucometotheproblemsassociatedwithremembering.
Considerthis:
Example:Item-WritingSkills—SpecificityofQuestionsFivecompaniesinasmallcityweresurveyedtofindoutabouttheirattitudestowardhiringwomenformanagerialpositions.Onesurveyquestionasked,“Doyouthinkwomenhaveasgoodachanceasmenformanagerialpositions?”Afriendofthesurveyorpointedoutthatabetterwayofaskingthequestionwas“At(fillinnameofcompany),dowomenhaveasgoodachanceasmenformanagerialpositions?”
Becarefulnottoerrontheconcreteside.IfyouaskpeoplehowmanyhoursofTVtheywatchedeachdayforthepastweek,youshouldbesurethatnoreasonexistsforbelievingthatthepastweekwasunusualsothatthedatawouldnotberepresentativeofatrueweek’sworthofTVviewing.Amongthefactorsthatmightaffectviewers’habitsareTVspecialssuchastheOlympicsandcataclysmssuchasplanecrashes,earthquakes,floods,andfires.
4.Avoidbiasedwordsandphrases.Certainnames,places,andviewsareemotionallycharged.Whenincludedinasurvey,theyunfairlyinfluencepeople’sresponses.Wordssuchaspresident,abortion,terrorist,andalcoholicareexamples.
Supposeyouaresurveyingpeoplewhohadjustbeenthroughadietprogram.Whichwordsshouldyouuse:thinorslender;portly,heavy,orfat?
Rememberthis?
Iamfirm.Youarestubborn.
Heisapig-headedfool.
Lookatthesequestions.
WouldyouvoteforRogerFields?WouldyouvoteforDr.RogerFields?WouldyouvoteforRogerFields,aliberal?
AlthoughRogerFieldsappearstobethemostneutraldescriptionofthecandidate,itmaybeconsideredtheleastinformative.YettheintroductionofDr.orliberalmaybiastheresponses.
5.Checkyourownbiases.Anadditionalsourceofbiasispresentwhensurveywritersareunawareoftheirownpositiontowardatopic.Lookatthis:
Example:Item-WritingSkills—HiddenBiasesPoor:Doyouthinktheliberalsandconservativeswillsoonreachagreaterdegreeofunderstanding?
Poor:Doyouthinktheliberalsandconservativeswillcontinuetheirpresentpoorlevelofunderstanding?
Whenyouareaskingquestionsthatyoususpectencouragestrongviewsoneitherside,itishelpfultohavethemreviewed.Askyourreviewerwhetherthewordingisunbiasedandacceptabletopersonsholdingcontraryopinions.Forasurveyofpeople’sviewsontherelationshipbetweentheliberalsandtheconservatives,youmightaskthefollowing:
Example:Item-WritingSkills—HiddenBiasesBetter:Inyouropinion,inthenext4years,howistherelationshipbetweentheliberalsandtheconservativeslikelytochange?
MuchimprovementSomeimprovementSomeworseningMuchworseningImpossibletopredict
6.Usecautionwhenaskingforpersonalinformation.Anothersourceofbiasmayresultfromquestionsthatmayintimidatetherespondent.Questionssuchas“Howmuchdoyouearneachyear?”“Areyousingleordivorced?”and“Howdoyoufeelaboutyourteacher,counselor,ordoctor?”arepersonalandmayoffendsomepeoplewhomightthenrefusetogivethetrueanswers.Whenpersonalinformationisessentialtothesurvey,youcanaskquestionsintheleastemotionallychargedwayifyouprovidecategoriesofresponses.
Example:Question-WritingSkills—VeryPersonalQuestionsPoor:Whatwasyourannualincomelastyear?
$____________
Better:Inwhichcategorydoesyourannualincomelastyearfitbest?
Below$10,000Between$10,001and$20,000Between$20,001and$40,000Between$40,001and$75,000Over$75,001
Categoriesofresponsesaregenerallypreferredforsensitivequestionsbecausetheydonotspecificallyidentifytherespondentandappearlesspersonal.
7.Eachquestionshouldhavejustonethought.Donotusequestionsinwhicharespondent’struthfulanswercouldbebothyesandnoatthesametimeorallowarespondenttoagreeanddisagreeatthesametime.
Example:Question-WritingSkills—OneThoughtperQuestionPoor:ShouldtheUnitedStatescutitsmilitaryordomesticspending?
YesNoDon’tknow
Better:ShouldtheUnitedStatessubstantiallyreduceitsmilitaryspending?
YesNoDon’tknow
or
ShouldtheUnitedStatesallocatemoremoneytodomesticprograms?
YesNoDon’tknow
or
IftheUnitedStatesreduceditsmilitaryspending,shoulditusethefundsfordomesticprograms?
YesNoDon’tknow
ResponsesforClosedQuestions
YesandNoTheresponsesinasurveywithclosedquestionscantakeseveralforms.
Example:Yes-and-NoResponsesHaveyougraduatedfromcollege?
YesNo
Doesyourcarhavefour-wheeldrive?
YesNoDon’tknow
Yes-and-noresponsesaresimpletouseandscore.Butaslightmisinterpretationmeansthattheanswerwillbeexactlyoppositefromwhattherespondentreallymeans.Also,insomecases,askingforabsolutelynegativeorpositiveviewsmayresultintheparticipant’srefusaltoanswerorchoiceof“don’tknow.”
RatingScales
CategoricalorContinuous?WhatAboutOrdinal?Aratingscalerequirestherespondenttoplacetheitembeingratedinanyoneofanorderedseriesofcategoriesoratsomepointalongacontinuum.Anumericalvalueisassignedtotheorderedcategoriesorpoints.Therearethreetypesofratingormeasurementscales.
1.Categorical.Thesearesometimescallednominalresponsescales.Theyrequirepeopletoaffirmornamethegroupstowhichtheybelong:gender,religiousaffiliation,school,orcollegelastattended.
Example:CategoricalRatingScaleWhatisthenewborn’sgender?
2.Ordinal.Thesescalesrequirethatrespondentsplaceanswersinorderofimportance.Aperson’seconomicstatus(high,medium,orlow)wouldprovideanordinalmeasurement.
Example:OrdinalRatingScaleWhatisthehighestlevelofeducationthatyouachieved?
3.Numericalscales.Numericalscalestaketwoforms:discreteandcontinuous.Adiscretescaleproducesaprecisenumber,whereasacontinuousscaleproducesanumberthatfallsonacontinuum.
Hereisadiscretescale:
Ondaysthatyoudrankalcoholduringthepast12months,howmanydrinksofalcohol(beer,wine,and/orhardliquor)didyouusuallydrink?
Theanswerisdiscretebecauseyougetanactualnumber.Contrastthistoanumericalscalethatproducesanumberonacontinuum:
Whatisyourbloodpressure?____/____mm/HgHowtallareyou(incentimeters)?________cm
Continuousdatacanhavealmostanynumericvalueandcanbemeaningfullysubdividedintofinerandfinerincrementsdependingontheprecisionofthemeasurementsystem.
Youcanalsogographictogetcontinuousdata.Graphicscalesareakindofratingscaleinwhichthecontinuumofresponsesisvisual.Perhapsthemostwell-knowngraphicscaleisusedindescribingpain(seeFigure2.1).Apersonisaskedtoplaceamarkonascaleortocitethenumberthatbestillustrateshisorcurrentpain.
Thedistinctionsamongcategorical,ordinal,andcontinuousscalesareimportantbecausetheydeterminethekindsofstatisticaltreatmentsyoucanuse.Consideraverages.Ifyouwanttheaveragescoreforcontinuousdata,youwouldusethearithmeticaverageormean.Forordinaldata,youmightconsidercalculatingthemedianorthenumberseparatingthehigherhalfofresponsesfromthelowerhalf(moreonstatisticalapproachesandsurveydatacanbefoundinChapter6).
Figure2.1PainMeasurementScale
Source:©1983Wong-BakerFACES®Foundation,www.WongBakerFACES.org.Usedwithpermission.OriginallypublishedinWhaley&Wong’sNursingCareofInfantsandChildren.©ElsevierInc.
Source:©1983Wong-BakerFACES®Foundation,www.WongBakerFACES.org.Usedwithpermission.OriginallypublishedinWhaley&Wong’sNursingCareofInfantsandChildren.©ElsevierInc.
OrdinalScalesWhenratersuseordinalscales,theyselectoneofalimitednumberofcategoriesthathavesomeordertothem:
Example:OrdinalScalesFrequentlySometimesAlmostneverVeryfavorable
FavorableNeitherfavorablenorunfavorableUnfavorableVeryunfavorableStronglyapproveApproveNeitherapprovenordisapproveDisapproveStronglydisapproveDefinitelyagreeProbablyagreeNeitheragreenordisagreeProbablydisagreeDefinitelydisagree
Ordinalscalesareeasytouseandinterpret.Howmanycategoriesshouldtherebe?Somepeopleuseasmanyas9categoriesandothersasfewas3(high,medium,andlow).Anevennumberofchoices—say,4—forcestherespondentawayfromthemiddleground(neitheragreenordisagree).Buttheneedsofthesurveyandskillsoftherespondentmustdeterminethenumberofcategories.Ifpreciseinformationisneeded,therespondentsarewillingandabletogiveit,andyouhavetheresourcestocollectit,usemanycategories(between7and9);otherwise,usefewer.
Considerthesetwosituations:
Example:SelectingtheNumberofCategories1.A4-minutetelephoneinterviewwasconductedtofindouthowoftenfamilieswithworkingmothersatedinnerinrestaurants.Thequestionwasasked:“Inatypicalmonth,howoftendoesyourfamilyeatdinnerinarestaurant?”Theresponsechoicesweretwoormoretimesaweek ,onceaweek ,orlessthanonceaweek .
2.Physicianswereaskedtoratetheappropriatenessofacomprehensivelistofreasonsforperformingselectedsurgicalproceduressuchascoronaryarterybypassgraftsurgeryandgallbladderremoval.Anappropriatereasonwasdefinedasoneforwhichbenefitstopatientsoutweighedrisks.Ascaleof1to9wasused,where1=definitelyinappropriate,whereas9=definitelyappropriate.
InSituation1,the4-minuteinterviewdictatedshortresponses.InSituation2,physicianswereaskedtousetheirexpertisetogivefairlyrefinedratings.
Anoften-usedordinalscaleistheLikert-typescaleinwhichrespondentsareaskedtotellhowcloselytheyagreeordisagreewithastatement.Thenumberatoneendofthescalerepresentstheleastagreement,orstronglydisagree,andthenumberattheotherendofthescalerepresentsthemostagreement,orstronglyagree.Likertscalestypicallyhave5points:
StronglydisagreeDisagreeNeitheragreenordisagreeAgreeStronglyagree
Sometimesa4-pointscaleisused;thisisaforced-choicemethodbecausethemiddleoptionofneitheragreenordisagreeor,byconvention,neutralisnotavailable.Dependingonthesurvey’sneeds,thelist
cangofromstronglydisagreetostronglyagreeortheotherwayaround.Ifthescaleincludesotherwordsateitherendtofurtherclarifythemeaningofthenumbers,itisknownasaLikert-stylequestion.
ResponsestoLikertitemsaresometimesanalyzedasordinaldata,particularlywhenthesurveyorcannotassumethatrespondentsperceivethedifferencebetweenadjacentratingpointsasequidistant.Whenregardedasordinaldata,Likertresponsescanbecollatedintobarchartsandsummarizedbythemedianorthemode(butnotthemean)andvariationsummarizedbytherange(butnotthestandarddeviation)oranalyzedusingnonparametrictests(e.g.,c2testorMann-Whitneytest).Ifthesurveyordecidestoregardthescaleascontinuous,thenmeansandstandarddeviationsareappropriatestatistics.Treatingthedataascontinuouscanoccurifthesurveyordecidestocompareaverageratings.DatafromLikertscalesaresometimesreducedtothecategoricallevelbycombiningallagreeanddisagreeresponsesintotwocategoriesofacceptandreject.Thec2andMcNemartestsarecommonstatisticalproceduresusedafterthistransformation(seeChapter6formoreonthesetechniques).
Forced-choicequestionsareoftenusefulwhenyouwanttodiverttherespondentfromtakingthepathofleastresistancebychoosingamiddlecategory(somewherebetweenagreeanddisagree—neutral)orclaiminguncertainty(notsure,don’tknow).Butforcingrespondentstochoosemayannoythemandmaynotuncoverthetruthabouttheirviews.
Someordinalscalesrelyonrelativejudgmentsorranks,asinthefollowingexample.
Example:RankOrderScaleSource:SurveyMonkey®
Thedownarrowstotheleftofeachnameconsistofthenumbers1to5.Therespondentcanselectaresponseonlyonce.IfFay
Grossisranked5,thentheothersmustberanked1,2,3,or4.
Rankinginvolvescomparingonefactortoanother.Anotherformthatanordinalscalecantakeisthecomparativescale.Thisoneaskstherespondenttocontrastasinglespecificobjectintermsofageneralsituation.
Example:ComparativeRatingScalePleasecomparetheImperialThaiRestauranttoothersinLosAngeles.
Checkone:
__Itisbetterthanmost.__Itisaboutthesameasmost.__Itisnotasgoodasmost.
Toensurethatrankordersandcomparativeratingscalesprovideaccurateinformation,youhavetobecertainthattherespondentsareinapositiontomakecomparisons.Dotheyhaveexperienceinjudgingwritingskills?Ifnot,howcantheyassignranks?AretheyfullyacquaintedwithrestaurantsinLosAngeles?Ifnot,howcantheymakecomparisons?
ChecklistAchecklistprovidesrespondentswithaseriesofanswers.Theymaychoosejustoneormoreanswersdependingontheinstructions.
Example:ChecklistResponsesinWhichRespondentMustChooseOneFromaListofSeveralWhichofthefollowingmedicinesdoyouprefermostfortreatingaheadache?
(Oneansweronly)
Example:ChecklistResponsesThatRespondentsAnswerYes,No,orDon’tKnowforEachIteminaListCheck(ü)oneboxoneachline.
Inthepast12months,hasadoctororotherhealthworkertoldyouthatyouhaveanyofthefollowing:
Checklistshelpremindrespondentsofsomethingstheymighthaveforgotten.Ifyousimplyaskedpeopletolisttheirmedications,chancesarethatsomewouldnotknow.Checklistsalsohelpwiththespellingofdifficultwords.Aproblemisthatrespondentsmightthinkachoicelooksfamiliarandcheckthewrongitem.Didtheytakepenicillinorampicillin?Wasitthismonthorlast?Also,itissomewhatdifficulttoformatandinterpretresponsestochecklistswheremultipleanswerscanbegiven.Supposeinthefirstexamplethatapersonchecksaspirinbutfailstoindicatewhethertheothermedicinesweretaken.Canyouassumethattheotherswerenottaken,orisitpossiblethattheywere,butthepersondidnotbothertocompletetheitem?
ChildrenandSurveysWhatdothesethreequestionshaveincommon?
1. Inthepast12months,howmanytimesonschoolgroundshaveyoubeenafraidofbeingbeatenup?Wouldyousay...
2. Haveyoudoneanyofthesethingsinthelastmonth?
3. Howhappyareyouwith...
Thesethreequestionswereallcreatedforchildrenorteens.Theyarebasedonquestionsthatareusedinrealsurveys.ThefirsttwocomefromsurveysledbytheU.S.OfficeofPlanning,ResearchandEvaluation,oftheAdministrationforChildrenandFamilies,andthethirdisbasedonaquestionfrom“Children’sWorlds,”aninternationalsurveyofchildren’swell-being.
Manygoodreasonsexistforsurveyingchildren.Asweallknow,childrenandadolescentsliveinadifferentworld,andweadultsmaynotreallyknoworunderstandwhattheythink,believe,orvalue.Surveyorsgenerallyagreethatifyouwanttolearnaboutchildren,youshouldgotothemdirectly.Also,manypeopleagreethatchildrenareactorsandparticipantsinsociety,notjustpassiveonlookers,soitmakessensetoincludetheminsurveys.Finally,surveyingchildrenandteenscanbeeducationalandfunforthemandforthesurveyor,too.
Herearesometipsforcreatingwrittenandonlinesurveysforchildrenandadolescents:
1. Keepquestionsshort.2. Withveryyoungchildrenbespecificandavoidabstractions.
Poor:Doyouusuallylikeyourschoollunch?
Comment:“Usually”isanabstractionanditmaybeadifficultconceptforyoungchildrentograsp.
Poor:Doyoulikethefoodatyourschool?
Comment:Achildmaylikethefoodatbreakfast,butdislikethefoodatlunch.Theword“food”istoovague.Specifyifyoumeanbreakfast,snacks,orlunch.Thequestionalsoneedstospecifyatimeperiod.
Better:Didyoulikeyourlunchatschooltoday?
or
Didyouliketheappleyouhadasasnacktoday?
3.Tryoutallquestionsbeforeyouusethem.Atrialwilltellyouifyouneedtoalterthesentencestructureorvocabularytomakeitmoreunderstandabletotheparticipants.Youmayfindthatyouneedtorevisethequestionsaltogether.Forexample,supposeatrialofthesurveyshowsthatfewerthanhalfofthechildreneatlunchatschool.Ifso,thequestion“Didyoulikeyourlunchatschooltoday?”willbeleftunansweredbyhalfthechildren.Ifyouwanttheinformation,youneedtorevisethesurveysothatyouprecedethequestionwith“Didyoueatlunchatschooltoday?”Ifyes,“Didyoulikeyourlunch?”
4.Askforresponsestopositivestatements.Itisdifficultconceptuallyforanyoneatanyagetodisagreewithanegativestatement.
Good:
Poor:
Supposethechildfeelssafeathome.Theresponse“false”translatesto:“ItisfalsethatIdonotfeelsafeathome.”Ifthechilddoesn’thaveagoodtimetogetherwiththefamily,theanswer“false”translatesto:“Itisfalsethatwedon’thaveagoodtimetogetherinmyfamily.”
5.Makecertainthatchildrenunderstandthattherearenowrongorrightanswers.Childrenandyoungpeoplemustbetoldwellinadvancewhytheyarebeingsurveyed(tohelpteachersknowhowyouliketolearnbest;tofindoutwhatkidsyouragedoafterschool)andhowtheinformationwillbeused(toseeifweneedtospendmoreclasstimeworkinginsmallgroups;tohelpusplanafterschoolactivities).
6.Donotforcechildrentoansweraquestioniftheydonotwantto.Thismeansthatyoumustensurethatinanonlinesurvey,ifachildwantstoskipQuestion1,heorshecanstillmoveontoQuestion2.Testingthesurveyinadvancewilltellyouiftherearequestionsthatoffendorstumpmostpotentialrespondents.
Ifso,thesequestionscanbeeliminatedfromthesurveyorhandledinanothermanner.Duringthesurvey’spreliminarytrial,youcanaskrespondentsiftheywouldanswerthequestionifitwereaskedindifferentformat(e.g.,inaprivateinterview).
7.Reinforceyourrespectfortherespondent’sprivacy.Tellthechildthatyouwillneverrevealindividualresponsestoanyonewithoutpermission.Allresponsesshouldbeanonymousor,failingthat,confidential.Olderchildrenmayunderstandthatallonlinedataareprobablytraceable.Theymayknowthatifyouusecommercialsurveysoftware,youarelikelytohavetheirIPaddresses,sotellthemhowyouplanonprotectingtheiridentity.Ifyouconductinterviews,makecertainthatyoudosoinaprivateplace.
OnlineSurveysOnlinesurveysarealmostalwaysself-administeredquestionnaires,asyoucanseefromtheexampleinFigure2.2.
Therulesforcreatingonlinesurveysandchoosingquestionsareexactlythesameasforotherself-administeredquestionnaires.Thesurveymustbedesignedtogettheinformationyouneedfromthepeoplewhoaremostlikelytogivethemostaccurateanswers.Thequestionsmustbegrammaticallycorrectwithareasonablerangeofoptionsforresponses.
Figure2.2Self-AdministeredQuestionnaire
Thedrop-downlistisaquestionformatthatisuniquetoonlinesurveys.Itisoftenusedtosavespacewhenalonglistofresponsesispossible.Forinstance,ifyouwanttoknowthecountryinwhichtherespondentlives,youcansaveagreatdealofscreenspacebyprovidingadrop-downlistofcountries,ratherthanprintingthelistitself.
Hereisaquestionaskingfortherespondent’sjobtitleinafictitiouscompanywithhundredsofemployees.Adrop-downlistisoftenusedtosaverespondentstimeandtoensurethestandardizationofresponses(seeFigure2.3).
Onlinesurveysoftencontainaprogressbar.Thebartellstherespondenttheproportionofthesurveytheyhavecompleted.Hereisanexample.
Example:SurveyProgressBar
Theprogressbargivesrespondentsinformationonhowmuchtimetheywillrealisticallyneedtocompletethesurvey.Ifrespondentsused10minutestocomplete20%ofthesurvey,thentheywillneed50minutestocomplete100%.
Someonlinesurveysareprogrammedtoallowrespondentstologoffandonatwill.Ifrespondentsneed50minutestocompletethesurvey,buttheyonlyhave30minutesatthistime,alog-on-and-offfeaturemayencouragethemtoreturntothesurveyatsomeothertime.GivingparticipantstheabilitytologonandoffatwillmaybeagoodideaifthereisapossibilitythattheirInternetconnectionmaybedisrupted.Bepreparedtosendemailreminderstorespondentswhologoffandthenforgetthattheyhavenotcompletedthesurvey.
Figure2.3OnlineDrop-DownList
Onlinesurveyquestionscanbetailor-madesothattheresponsetoonequestiondictatestheresponsetothenextasillustratedinthisexample.
Example:ResponsetoOneQuestionDictatestheResponsetotheNextSource:http://www.questionpro.com/help/162.html
Onlinesurveysshouldexplainhowtogobacktochangeanswersandthenmoveforward.Considertheseinstructions:
Example:ExplanationforChangingOnlineResponsesPleasereadthefollowinginstructionsbeforebeginningthissurvey.
MovingForward:ClickontheNEXTbuttonlocatedatthebottomofthepagetosaveyourresponsesandcontinuetothenextpage.
MovingBack:UsethePREVIOUSbuttonlocatedatthebottomofthepagetoviewyourresponsesonapreviouspage.Youmaychangeyouranswerstopreviouslyenteredresponses.DonotusetheBACKbuttonofyourbrowsertoreturntothepreviouspage.Usingyourbrowser’sBACKbuttonmaycauseyoutoexitthesurvey.Shouldthisoccur,youwillneedtowait10minutestoreaccessthesurveyusingthelinkprovidedinyouremail.
PlainandSimpleSurveyQuestionsandResponsesSurveyquestionsandresponsechoicesshouldbewrittenata6th-to8th-gradereadinglevel(basedontheU.S.schoolgradesystem).Why?StudiesshowthattheaverageAmericanismostcomfortablereadingatthatlevel.Infact,theevidencesuggeststhatmostpeopleprefersimplewordsandsentencesinsurveys.
TherearemanyformulasavailabletohelpdeterminetheapproximatereadinglevelofadocumentandyoucanfindawealthofinformationabouteachofthemontheInternet.Mostreadabilityformulasprovideagrade-levelscoreandarebasedontheaveragenumberofsyllablesperwordandwordspersentence.Themoresyllablesthereareinawordandthemorewordsthereareinasentence,theharderitistoreadandunderstandthetext.
AmongthemostcommonlyusedmethodstodeterminereadabilityaretheFryandFlesch-Kincaidformulas.Eachhasitsadvantagesanddisadvantages,andexpertsdonotagreeonwhichisbetter.Youcangoonlinetolearnhowtouseeach.Usethesekeywordstogetstarted:“readabilitystatistics.”
TheFlesch-KincaidmethodisconsideredlessaccuratethantheFry,buttheFryistimeconsuming.YoucaneasilycalculatetheFlesch-ReadingeasescorebyusingthereadabilitytoolinMicrosoftWord.Herearetheinstructions.
1. Gotothe“Review”tab.2. Gotothe“Spelling&Grammar”tabandclick“Options.”3. Check“Showreadabilitystatistics”underthe“Grammar”heading.
Eachtimeyoucheckspellingandgrammar,youwillbegivenseveralreadabilitystatistics,includingtheFlesch-Kincaidgrade-levelscore.
Scaling
AdditiveScalesMostsurveysaredesignedsothateachindividualitemcounts.Inasurveyofpeople’sattitudestowardlivinginatrailerpark,youmightask12questions,eachofwhichisdesignedtobeusedtoanalyzeattitudesandthereforeisscoredseparately.Supposeyoucollectedinformationlikethis:
LengthoftimelivinginthistrailerparkWhetherpersoneverlivedinatrailerparkbeforeSatisfactionwithpark’saccom-modationsSatisfactionwithqualityoflifestyleAmenitiesintrailerAgeoftrailerAgeofcarTypeofcarAgeofrespondentGenderofrespondentAnnualincome
Withthisinformation,youcouldreportoneachfactindividuallyoryoucouldlookforrelationships:
BetweenageofrespondentandsatisfactionwithqualityoflifestyleBetweengenderandlengthoftimelivinginthetrailerpark
Othersurveysaredifferent,however,inthattheitemsdonotcountindividually;theymustbecombinedtogetascore.
Considerthis:
Example:ASurveyWithanAdditiveScaleDoctorsatUniversityMedicalCenterobservedthatmanyoftheirveryillpatientsappearedtofunctionquitewellinsociety.Despitetheirdisabilities,theyhadfriendsandwenttothemovies,shopping,andsoon.Otherpatientswithsimilarproblemsremainedathome,isolatedfromfriendsandfamily.Thedoctorshypothesizedthatthedifferencebetweenthetwogroupsofpatientswasintheirpsychologicalfunctioningandaccesstotheresourcesthatmakelifealittleeasierforeveryone.Aspartoftestingtheirhypothesis,theyplantogivethetwogroupstheFunctionalStatusInventorydevelopedjointlybytheHerbertMedicalSchoolandCaliforniaUniversity.After5yearsofstudyandvalidation,researchersatthetwouniversitieshavepreparedasurveyonfunctioningforusewithchronicallyillpatients.Highscoresmeangoodfunctioning;lowscoresmeanpoorfunctioning.
Themethodsusedtoproduceanadditivescalerequiresophisticatedsurveyconstructionskillsbecauseyouhavetoproveconclusivelythathighscorersareinactualitydifferentfromlowscorerswithrespecttoeachandeveryitem.
DefiningAdditiveScalesSurveyorsusethetermscaleinatleasttwoways.Thefirstreferstothewaytheresponsesareorganized:
1. Doyoueatsixormoreservingsoffruitsorvegetableseachday?
Theresponsescaleisthefollowing:Yes □No □
2. Howsatisfiedareyouwiththeexamplesinthisbook?
Theresponsescaleisthefollowing:
Theseconduseofthetermscalereferstoonequestionoracollectionofquestionswhosescoresaremeaningful.Youhaveaquality-of-lifescalewhenasurveywith10questionscanbescoredsothatascoreof1meanslowqualityandascoreof10meanshighquality.
Example:ASurveyofForeignLanguageSkills
Circlethecategorythatbestdescribesyourabilitytospeakeachofthefollowinglanguages.
Foreachitem—French,German,andsoon—anordinalratingscaleisusedtoorganizeresponses.Atthesametime,byaddingallfiveitemstogether,ascaleoflanguageabilitycanbederived.Arespondentwhoisfluentinalllanguageswouldbeatoneendofthescaleandonewhoisnotfluentinanylanguageswouldbeattheother.Supposeyouassignedtwopointsforeachlanguagemarkedfluent,onepointeachforthosemarkedsomewhatfluent,andnopointsfornotfluent.Apersonfluentinallfivelanguagescouldgetamaximumscoreof10points,whereassomeonewhowasfluentinnonewouldbeassignedtheminimumscoreof0.Inthisbook,thistypeofscaleiscalledadditive(becauseindividualresponsestoitemsareaddedtogether,thatis,combined).Amongthemostcommonlyusedadditivescalesarethedifferential,summated,andcumulative.
DifferentialScalesDifferentialscalesdistinguishamongpeopleintermsofwhethertheyagreeordisagreewiththeoryorresearch.Tocreateadifferentialscaleforanideasuchasequalityofopportunity,forexample,meansassemblingmanystatements(e.g.,“qualifiedmenandwomenshouldreceiveequalpayforequalwork”)andhavingexpertsrateeachstatementaccordingtowhetheritwasfavorabletotheidea.Nextyoucomputetheexperts’averageormedianratingsforeachstatement.Thenyouaskrespondentstoagreeordisagreewitheachstatement.Theirscoreisbasedonjustthoseitemswithwhichtherespondentagrees.Togettherespondent’sscore,youlookattheexperts’averagescoreforeachstatementchosenbytherespondent,addtheexperts’averages,andcomputethearithmeticmean.
Typically,thedirectionstousersofdifferentialscalesgosomethinglikethis:
Pleasecheckeachstatementwithwhichyouagree.
or
Pleasechecktheitemsthatareclosesttoyourposition.
Scoringadifferentialscalemighttakethisform.
Example:ScoringaDifferentialScaleStudentAwasadministeredthePhysicalFitnessInventoryandaskedtoselectthetwoorthreeitemswithwhichshemostcloselyagreed.Thesearethetwoitemsshechoseandthejudges’scores.
StudentA’sscorewas3.6(theaverageof3.2and4.0),whichwasconsideredtobesupportiveofphysicalfitness.(Thebestpossiblescorewas1.0andtheworstwas11.0.)
Aretheredisadvantagestodifferentialscales?Perhapsthemostobviousoneisintheamountofworkneededtoconstructthem.Also,youmusttakeintoaccounttheattitudesandvaluesofthejudgeswhoseratingsareusedtoanchorthescaleandinterprettheresponses.Thejudgesmaybequitedifferentfromthepeoplewhomightusethescale.
SummatedScalesAsummatedscalealignspeopleaccordingtohowtheirresponsesaddup.Supposeaself-esteemquestionnairehasaseriesofitemsthatusethesameratingscale(agree,neutral,disagree).
Example:CreatingaSummatedScaleforaSelf-EsteemSurvey
Directions:Checkifyouagreeordisagreewitheachofthefollowingstatements.
Howwouldyoucomputeasummativescaleforthisquestionnaire?First,decidewhichitemsarefavorable(inthiscase,b,d,ande)andwhicharenot(aandc).Next,assignanumericalweighttoeachresponsecategory.Youmightdosomethinglikethis:
favorable=+1pointinbetweenorneutral=0pointsunfavorable=–1point
Aperson’sscorewouldbethealgebraicsumofhisorherresponsestofiveitems.TheanswersthatPersonXgaveareshownintheexamplethatfollows.
Example:ScoringaSummatedScale
PersonXdisagreedwithItemA,whichisfundamentallyunfavorable,andgotascoreof+1.ForItemB,thepersonwasneutralandsoearnedascoreof0.ItemCproducedagreement,butitwasfundamentallyunfavorable;PersonXgotascoreof–1.TherewasagreementwithItemD,resultinginascoreof+1andaneutralresponsetoItemE,producingascoreof0.PersonX’ssummatedscalescorewas+1outofapossibletotalof+5.(Aperfectscoreof+5wouldhavecomeaboutifPersonXanswered:ItemA=disagree;ItemB=agree;ItemC=disagree;ItemsDandE=agree.)
SummingUpThecontentofasurveydependsoncleardefinitionsoftermsbecausehumanattitudesandfeelingsvaryacrosstheories.Becertainrespondentsareinapositiontoanswerthequestions.Aretheyinformedaboutthetopic?Donotaskforinformationthatyouwillnotusebecausethatisawasteofrespondents’andyourtime.Categoricalratingscalesaskforpeopletoaffirmtheirgroupaffiliations,includingsex,religiousaffiliation,school,orcollegelastattended.
Surveyquestionsmaybeclosedoropen-ended.Closedquestionswithseveralchoicesareeasiertoscorethanareopen-endedquestions.Open-endedquestionsgiverespondentsanopportunitytostateapositionintheirownwords;unfortunately,thesewordsmaybedifficulttointerpret.Whenwritingquestions,usestandardgrammarandsyntax.Keepquestionsconcreteandclosetotherespondents’experience;becomeawareofwords,names,andviewsthatmightautomaticallybiasyourresults;checkyourownbiases;donotgettoopersonal;anduseasinglethoughtineachquestion.Considerusingareadabilityformula.Numericalscalestaketwoforms:discreteandcontinuous.Adiscretescaleproducesaprecisenumber,whereasacontinuousscaleproducesanumberthatfallsonacontinuum.Ordinalscalesrequirethatrespondentsplaceanswersinorderofimportance.Anoften-usedordinalscaleistheLikertscaleinwhichrespondentsareaskedtotellhowcloselytheyagreeordisagreewithastatement.Onlinesurveysofferspecialquestion-and-responseformatssuchasdrop-downmenus.Itisimportanttomakesurethattherespondentknowshowtonavigatethroughanonlinesurvey,especiallyhowtomoveforwardandback(tochangeanswersorreview).Followtheseguidelineswhensurveyingchildren:keepquestionsshort;beconcreteandavoidabstractions;tryoutthesurvey;keepresponsechoicespositive;ensurethatchildrenunderstandtherearenorightorwronganswersandthattheydonothavetoanswerquestions;reinforceandproveyourrespectfortheirprivacy.
ThinkAboutThis1. Readthequestionsbelow.Tellwhethertheyusecategorical,ordinal,orcontinuousscales.
1. QuestionsaboutInternetuseforhealthinformation:
2. Prepareaself-administeredquestionnairewith10orfewerquestionsaskingstudentsabouttheir
satisfactionwiththecontentofthischapter.Useatleastonequestioneachthatwillyieldcategorical,ordinal,andnumericaldata.Justifythechoiceandclarityofeachquestion.
3. Createaself-administeredorinterviewquestionnaireforteensbetween13and17yearsthatasksthemaboutoneormoreofthesetopics:bullyingatschool,useofsocialmediaduringclasstime,andlifestylepractices(alcoholuse?Otherdrugs?Exercise?Diet?).
4. Explaintheseterms:open-endedquestion,closedquestion,forced-choicequestion,rankorderscale,comparativeratingscale,Likertscale,drop-downmenu,summativescale,anddifferentialscale.
3GETTINGITTOGETHERSomePracticalConcerns
OverviewHowlongshouldasurveybe?Asurvey’slengthdependsonwhatyouneedtoknow,whenyouneedtoknowit,theamountoftimerespondentsarewillingtospendansweringquestions,andyourresources.
Thefirstquestiononasurveyshouldbeclearlyconnectedtoitspurpose,objectivequestionscomebeforesubjectiveones,movefromthemostfamiliartotheleastandfollowthenaturalsequenceoftime,keepquestionsindependenttoavoidbias,putrelativelyeasyquestionsattheend(particularlyinlongsurveys)butput“sensitive”questionsinthemiddle,avoidmanyitemsthatlookalike,andplacequestionslogically.
Toboosttheresponseratesforself-administeredquestionnaires,planinadvanceandmonitorresults.Onlinesurveysystemscanletyoudothisinrealtime.Considersendingrespondentsaletterbeforethesurveybeginsandalsoatthetimeofthesurvey.Youcandothisviasnailmailoremail.Proposetosendsurveyresults,keepquestionnairesshort,andthinkaboutofferingincentives.Interviewsrequirepreparationandplanning.Interviewersneedsystematic,intensivetraining.Setupasystemformonitoringthequalityoftheinterviewsovertime.
Youshouldpilottestyoursurveytoseethatitcanbeaccessedandadministeredeasilyandaccordingtoplan.Yourmaingoalisreliableandvalidsurveydata.Reliabilityreferstotheconsistencyoftheinformationyouget,andvalidityreferstotheaccuracyoftheinformation.
Onewaytopromotethereliabilityandvalidityofyoursurveyistobaseitonanalready-validatedsurvey,butbecautious.Askforevidenceofthesurvey’sreliabilityandvalidity.
Apilottesthelpsyoudesignareliablesurvey.Whenpilottesting,anticipatetheactualcircumstancesinwhichthesurveywillbeconductedandmakeplanstohandlethem.Chooserespondentssimilartotheoneswhowilleventuallycompletethesurvey.Forreliability,focusontheclarityofthequestionsandthegeneralformatofthesurvey.
Inaworldthatencompassesmanylanguagesandcultures,surveyorsneedtolearnhowtotranslatesurveysbyrelyingonfluentspeakerstoassistandtotestthequestionsandresponsesformeaning.
Pilottestingbolstersreliabilityandvaliditybecauseitcanhelpyouseethatalltopicsareincludedandthatsufficientvarietyintheresponsesisavailable—ifpeopletrulydiffer,yoursurveywillpickupthosedifferences.
Mostpublicandprivateagenciesthatconductsurveysasstand-alonestudiesoraspartoflargerresearchactivitieshaverulesaboutprotectingtherespondents’privacyandconfidentiality.Bepreparedtoexplaintherisksandbenefitstorespondentsiftheycompletethesurvey.
LengthCountsThelengthofasurveydependsonwhatyouneedtoknowandhowmanyquestionsarenecessarysothattheresultinganswersarecredible.Anotherconsiderationistherespondents.Howmuchtimedotheyhaveavailable,andwilltheypayattentiontothesurvey?Relativelyyoungchildren,forexample,maystayputforonlyafewminutes.Youmustconsideryourresources.A10-minutetelephoneorface-to-faceinterviewwillcostlessthansimilarinterviewslasting20minutes.Onlinesurveyscanbegiventotensofthousandsofpeoplewithoneclick.Thetechnicalexpertiseneededtodesigninternetsurveysiscostly,however,andifyousignupwithacommercialvendor,youwillhavetopayalicensefeeandpossiblysetupspecialprivacyarrangements.Herearetwosituationsillustratinghowthecircumstancesunderwhichasurveyisconductedinfluenceitslength.
Inthefirstsituation,thelibraryisdevotingconsiderabletimetoconductinterviewstoidentifycommunity
needs.Inthesecond,thesurveyisshortenedconsiderably.
Example:HowaSurvey’sCircumstancesCanInfluenceItsLengthSituation1:Thelocallibrarywantstobesurethatitcontinuestomeettheneedsofachangingcommunity.Inrecentyears,manymoreofitspatronsaremorethan65yearsofageandasubstantialpercentagespeaksEnglishasasecondlanguage.Amongthelibrary’sconcernsaretheadequacyandrelevanceoftheirexhibits,newspapers,magazines,andotherperiodicals;programsfeaturingnewbooksandwriters;andspecial-interestcollectionsconcernedwithissuessuchashealth.Abilingualvolunteerwillbedevotingtwomorningsandoneafternoonaweekfor8weekstoa45-minuteface-to-faceinterviewwithusersofthelibrary.A50-itemsurveyformhasbeendesignedforthepurpose.
Situation2:Theneighborhoodlibraryalsowantstobesurethatitsservicesareappropriateforapopulationthatisincreasinglyolderandnon-Englishspeaking.Thecityhasdecidedthattheneighborhoodlibraryisnottheonlylibraryinthecitythatischanginginitsneedsandhasagreedtosponsorasurveyofitslibrarypatrons.Tominimizetheamountoftimethatlibrariansandpatronswillhavetospendonthesurvey,a10-minute,6-item,self-administeredquestionnaireispreparedbythecentrallibraryoffice.Fourquestionsareaskedabouttheadequacyofthelibrary’scollectionofbooksandmagazines,computeraccess,andspecial-interestcollections;andtwoquestionsaskabouttherespondent’seducationalbackgroundandincome.Tofacilitatethesurvey’sefficiency,questionnairesareobtainableatthecheckoutdesk,completedatthelibrary,andleftwiththelocalbranchlibrarianwhothensendsthemtothecentralofficeforanalysis.
GettingtheSurveyinOrderAllsurveysshouldbeprecededbyanintroductionandthefirstsetofquestionsshouldberelatedtothetopicdescribedinit.Lookatthisintroductionandfirstquestionforatelephoneinterview.
Example:AnIntroductiontoaTelephoneSurveyandItsFirstQuestionHello.IamcallingfromtheCaliforniaUniversity.Wearesurveyingpeoplewholiveinstudenthousingtofindoutwhetheritisasatisfactoryplacetolive.Yournamewasselectedatrandomfromthehousingregistry,adirectoryofstudentswhohavevoluntarilylistedtheirtelephonenumbers.Ourquestionnairewilltakenomorethan4minutes.Youcaninterruptmeatanytime.MayIaskyouthequestions?
[IfYES,continue.IfNO,saythankyou,expressconcernforbotheringtheperson,andhangup.][Continuehere.]
Thefirstquestionasksyouaboutyouroverallsatisfactionwithyourapartment.Doyouconsiderit[readchoices]
DefinitelysatisfactoryProbablysatisfactoryProbablynotsatisfactoryDefinitelynotsatisfactory[DONOTSAY]noopinionordon’tknow/wronganswer
Theinterviewerstartsoffbysayingthatquestionswillbeaskedaboutsatisfactionwithstudents’housing,andthefirstquestioncallsforaratingofsatisfaction.Peoplesometimesrespondbestwhenthefirstquestionsaskforobjectivefacts.Oncetheybecomeusedtothesurveyandmorecertainofitspurposes,theyaremorelikelytoprovidetheanswerstorelativelysubjectivequestions.Supposeyouwanttoknowaboutthesuccessofasummercitycleanupprogram,forexample.Youmightfirstbeginbyaskingparticipantshowtheyfirstheardabouttheprogramandhowlongtheyhadbeeninit(twoquestionsoffact)andthenaskhowwelltheylikedtheirjob.
Questionsshouldproceedfromthemosttotheleastfamiliar.Inasurveyofneedsforhealthservices,itemscanfirstbeaskedabouttherespondent’sownneedsforservices,thenthecommunity’s,thestate’s,andsoon.
Questionsofrecallshouldalsobeorganizedaccordingtotheirnaturalsequence.Donotaskgeneralquestions,suchas“Whendidyoufirstbecomeinterestedinjogging?”or“Whydidyouchoosejoggingoverotherphysicalexercise?”Instead,prompttherespondentandask,“Whenyouwereinhighschool,didyouhaveanyinterestinjogging?Incollege?”
Sometimestheanswertoonequestionwillaffectthecontentofanother.Whenthishappens,thevalueofthequestionnaireisseriouslydiminished.Lookatthis:
Example:OrderingSurveyQuestions(1)Whichquestionshouldcomefirst?
1. Howmuchhelpdoesyouradvisergiveyou?2. Whichimprovementsdoyouwantinyoureducation?
Answer:QuestionBshouldcomebeforeQuestionA.Ifitdoesnot,thenadviser–studentrelationsmightbeemphasizedundulysimplybecausetheyhadbeenmentioned.
Here’sanotherexample.Whichquestionshouldcomefirst?
Example:OrderingSurveyQuestions(2)1. Howsatisfiedareyouwiththepresident’seconomicpolicy?2. Whatisthequalityofthepresident’sleadership?
Answer:QuestionBshouldprecedeQuestionAbecauseapersonwhoisdissatisfiedwiththepresident’seconomicpolicy(andperhapsnothingelse)mightratethequalityofthepresident’sleadershiplowerthanotherwise.
Placerelativelyeasy-to-answerquestionsattheendofasurvey.Whenquestionnairesarelongordifficult,respondentsmaygettiredandanswerthelastquestionscarelesslyornotanswerthematall.Youcanplacedemographicquestions(age,income,gender,andotherbackgroundcharacteristics)attheconclusionbecausethesecanbeansweredquickly.
Avoidmanyitemsthatlookalike.Twentyitems,allofwhichasktherespondenttoagreeordisagreewithstatements,mayleadtofatigueorboredomandtherespondentmaygiveup.Tominimizelossofinterest,groupquestionsandprovidetransitionsthatdescribetheformatortopic,asinthefollowingexample.
Example:ProvidingTransitionsThenext10questionswillaskwhetheryouagreeordisagreewithdifferentplanksoftheDemocraticPartyplatform.
Questionsthatarerelativelysensitiveshouldbeplacedtowardtheend.Topicssuchasgroominghabits,religiousviews,andpositionsoncontroversialsubjectssuchasabortionandguncontrolmustbeplacedfarenoughalongsothereisreasontobelievetherespondentiswillingtopayattentionbutnotsofarthatheorsheistoofatiguedtoanswerproperly.
Finally,questionsshouldappearinlogicalorder.Donotswitchfromonetopictoanotherunlessyouprovideatransitionalstatementtohelptherespondentmakesenseoftheorder.
Hereisachecklistofpointstoconsiderinselectingtheorderforthequestionsinyoursurvey:
ChecklisttoGuideQuestionOrderForanygiventopic,askrelativelyobjectivequestionsbeforethesubjectiveones.Movefromthemostfamiliartotheleast.
Followthenaturalsequenceoftime.Seetoitthatallquestionsareindependent.Relativelyeasy-to-answerquestions(includingdemographicquestionssuchasageoroccupation)shouldbeaskedattheend.Avoidmanyitemsthatlookalike.Sensitivequestionsshouldbeplacedwellafterthestartofthesurveybutalsowellbeforeitsconclusion.Questionsshouldbeinlogicalorder.
QuestionnaireFormat:AestheticsandOtherConcernsAquestionnaire’sappearanceisvitallyimportant.Awrittenself-administeredquestionnairethatishardtoreadcanconfuseorirritaterespondents.Theresultislossofdata.Apoorlydesignedinterviewformwithinadequatespaceforrecordinganswerswillreducetheefficiencyofeventhemostskilledinterviewers.
Herearesomedosanddon’ts:
Do:Putjustonequestiononaline.Leaveplentyofspaceforresponses.Don’t:Squeezeseveralquestionstogether.Donotabbreviatequestions.
ResponseFormatThegeneralruleistoleaveenoughspacetomaketheappropriatemarks.Hereareseveralresponseformats.
Example:ResponseFormats
FormatBorCisprobablysafer.
BranchingQuestions,ortheInfamous“Skip”Pattern
Whathappenswhenyouareconcernedwithgettinganswerstoquestionsthatyouknowareappropriateforonlypartofyourgroup?Supposeyouaredoingasurveyofyoungpeople’sparticipationinafterschoolactivities.Youknowthatonemajoractivitymightbesportsandanothermightbemusic,butyoualsoknowonlysomeparticipateineither.
Ifyouwanttoaskaboutatopicthatyouknowinadvancewillnotberelevanttoeveryoneinthesurvey,youmightdesignaformsuchastheoneinthefollowingexample.
Example:SkipPatternsorBranchingQuestions3.Doyouparticipateinsports?
1. No(GOTOQUESTION4)2. Yes
IFYES,whichsportsdoyouperform?
or
Doyouparticipateinsports?Yes(CompleteSectionA)
or
No(GotoSectionB)
Youmustbeextracarefulinusingbranchingquestions(or,astheyareoftencalled,skippatterns)inwrittenself-administeredquestionnaires.Infact,somesurveyorsthinkthatskippatternsareconfusingtomostpeopleandshouldnotbeusedinwrittenquestionnairesatall.Remember,respondentscanskipimportantquestionsiftheyarebaffledaboutwheretogonextonthesurveyform.Interviewersmustbetrainedtofollowthebranchesorelsetheymightbeunabletoadministerthesurveycorrectly.
Onlinesurveysareeffectivevehiclesforbranching(sometimescalled“piping”)becauseyoucandesignthesoftwaretoguidetherespondentandeverycommercialvendormakesiteasytouseskip-patternlogic.Forinstance,ifthesurveylogicis“IfrespondentanswersnotoQuestion3,thensendrespondenttoQuestion6,”therespondentwhoanswersnotoQuestion3willautomaticallyandpainlessly(fromtherespondent’sperspective)bemovedtoQuestion6.
Administration:WhoGivesWhattoWhom?
Self-AdministeredQuestionnairesSelf-administeredquestionnairesrequireagreatdealofpreparationandmonitoringtogetareasonableresponserate.Thesequestionnairesaregivendirectlytopeopleforcompletionandusuallylittleassistanceisavailableincasearespondentdoesnotunderstandaquestion.Asurveyquestionnaireaskingaboutteachers’availabilityandneedsforin-servicetrainingmaybeplacedinhisorherofficemailbox,forexample,withaweek’sreturnrequested.Ofcourse,teacherswhohavedifficultywiththeformmightdiscusstheproblemamongthemselves,butnoguaranteesexistthatthesolutionwillbejustascorrectorincorrectasiftheindividualactedalone.
Advancepreparation,intheformofcarefuleditingandtryouts,willunquestionablyhelpproduceaclear,readable,self-administeredquestionnaire.Tofurtherensurethatyougetwhatyouneed,youshouldreviewthereturns.Areyougettingtheresponserateyouexpected?Areallquestionsbeinganswered?Hereisachecklistforusingself-administeredquestionnairesthatcanhelpyougetwhatyouneed.
ChecklistforUsingSelf-AdministeredQuestionnairesSendrespondentsanadvanceletterviaregularmailoremailtellingthemthepurposeofyourquestionnaire.Thelettershouldnotifypeoplewhenthesurveyiscoming,explainwhytherespondentsshouldanswerthequestions,tellthemwhytheywereselected,howlongthesurveywilltake,andwhethertheywillbereimbursed.Prepareashort,formalexplanationtoaccompanythequestionnaireform.Ifyouhavealreadyinformedtherespondentaboutthesurveyinaletterorviaemail,youcankeepthisexplanationshort.Itshouldagaindescribethesurveyaims,respondents,andresponseburden.Considersendingrespondentsasummaryofthefindings.Ifyouaskquestionsthatmaybeconstruedaspersonal—suchasgender,race/ethnicity,age,orincome—explainwhythequestionsarenecessary.Keepthequestionnaireproceduressimple.Providestamped,self-addressedenvelopesformailsurveys.Keeppagefoldingtoaminimumsorespondentsdonotfeeltheyareinvolvedincomplicatedphysicalactivities.Keepquestionnairesshort.Askonlyquestionsyouaresureyouneedanddonotcrowdthemtogether.Inwrittensurveys,giverespondentsenoughroomtowriteandbesureeachquestionissetapartfromthenext.Considerincentives.Thismayencouragepeopletorespond.Incentivesmayrangefrommoneyandstampstopens,food,andfreeparking.Somesurveysenrollparticipantsinalotteryinwhichoneormore“winners”receiverelativelylargegiftslikecomputersandelectronicreaders.Bepreparedtofollowuporsendreminders.Theseshouldbebriefandtothepoint.Donotforgettobudgetmoneyandtimeformailingsandfollow-upphonecalls.Ifyouareusingonlinesurveysoftware,makesureyourcontractincludesafollow-upoption.Ifyouaredoinganonlinesurvey,herearesomespecialconsiderations.
Prepareallsurveyquestionsandresponsesbeforeyougoonlinetocreatethesurvey.Thesurveysoftwarewillguideyoutothetypeofquestion(suchasmultiplechoicewithoneanswerormultiplechoicewith“checkallthatapply”),butdonotletyourselfbeinfluencedbythesoftwareforthesakeofconvenience.Ifnecessary,provideinstructionsforcompletingeachquestion.Decideonwhethertokeepthesurveyasonecontinuousdocumentortodivideitintooneormorequestionsperscreen.Ifyouchoosetohaveoneormorequestionsperscreen,checkoneachscreentomakesurethatitisreadable.Lookateachquestiononline.Ifitisdifficulttoreadoruse,editit.Onceyouhaveallquestionsinadocument,reviewthedocumenttoensurethatthequestionsareproperlysequencedandeasytocomplete.Keeprespondentsuptodateontheirprogress.Onlinesurveysoftwareallowsyoutoprovideprogressgraphs.
Interviews
Findinginterviewers
Interviewersshouldfitinaswellaspossiblewithrespondents.Theyshouldavoidflamboyantclothes,haircuts,andsoon.Sometimesitisagoodideatoselectinterviewerswhoaresimilartorespondents.Ifyouwanttoaskadolescentgirlswhytheysmoke,forexample,youmighthireyoungwomentodothequestioning.
Itisalsoimportantthattheinterviewersbeabletospeakclearlyandunderstandably.Unusualspeechpatternsoraccentsmayprovokeunnecessarilyfavorableorunfavorablereactions.Theinterviewer’swayoftalkingisanextremelyimportantconsiderationinthetelephoneinterview.Youshouldbekeenlyawareofthepossibilitythattheinterviewer’sattitudetowardthesurveyandrespondentwillinfluencetheresults.Iftheinterviewerdoesnotexpectmuchandsendsthismessage,theresponseratewillprobablysuffer.Tomakesureyouaregettingthemostaccuratedatapossible,youshouldmonitortheinterviewers.
Traininginterviewers
Thekeytoagoodtelephoneorface-to-faceinterviewistraining,whichshouldensurethatallinterviewersknowwhatisexpectedofthemandthattheyaskallthequestionsinthesameway,withinthesameamountoftime.
Whetheryouaretrainingtwointerviewersor20,itisimportanttofindatimetomeettogether.Theadvantageofmeetingsisthateveryonecandevelopastandardvocabularyandshareproblems.Ifthetraineeshavetotraveltoreachyou,youmayhavetothinkaboutpayingforgasolineorothermeansofprivateorpublictransportation.
Onceatthetrainingsite,traineesmusthaveenoughspacetositandwriteorperformanyotheractivitiesyouwillrequireofthem.Ifyouwantthemtointerviewoneanotheraspracticefortheirrealtask,besuretheroomislargeenoughsothattwoormoregroupscanspeakwithoutdisturbingtheothers.Youmayevenneedseveralrooms.
Iftrainingtakesmorethananhourandahalf,youshouldprovidesomeformofrefreshment.Ifyoucannotaffordtodothat,atleastgivetraineestimetoobtaintheirown.
Traineesshouldbetakenstepbystepthroughtheirtasksandbegivenanopportunitytoaskquestions.Itisalsoessentialtotellthemsomeofthereasonsfortheirtaskssotheycananticipateproblemsandbepreparedtosolvethem.Themostefficientwaytomakesurethetraineeshavealltheinformationtheyneedtoperformtheirjobistoprepareamanual.Hereyoucanexplainwhattheyaretodoandwhen,where,why,andhowtheyaretodoit.Considersettingupagroupemailordesignatingasectionoftheproject’swebsiteasaplacetorecordchangesinsurveyproceduresandpolicies.
Conductinginterviews
Herearesometipsonconductinginterviewsthatshouldbeemphasizedinyourtrainingsessions:
1. Makeabriefintroductorystatementthatwilldescribewhoisconductingtheinterview(“MaryDoeforArmstrongMemorialMedicalCenter”),tellwhytheinterviewisbeingconducted(“tofindouthowsatisfiedyouarewithourhospitalityprogram”),explainwhytherespondentisbeingcalled(“We’reaskingarandomsampleofpeoplewhoweredischargedfromthehospitalinthelast2months”),andindicatewhetheranswerswillbekeptconfidential(“Yournamewillnotbeusedwithoutyourwrittenpermission”).
2. Trytoimpressonthepersonbeinginterviewedtheimportanceoftheinterviewandoftheanswers.Peoplearemorelikelytocooperateiftheyappreciatetheimportanceofthesubjectmatter.Donottrytodealwitheverycomplaintorcriticism,butsuggestthatallanswerswillreceiveequalattention.
3. Prepareyourselftobeflexible.Althoughitisimportanttostayonscheduleandaskallthequestions,afewpeoplemayhavetroublehearingandunderstandingsomeofthequestions.Ifthathappens,slowdownandrepeatthequestion.
4. Interviewpeoplealone.Thepresenceofanotherpersonmaybedistractingoraviolationofprivacyandthereforemayalterresults.
5. Ifusingaprintedinterviewsurvey,besuretoaskquestionsastheyappearintheinterviewschedule.Itisimportanttoaskeveryonethesamequestionsinthesamewayortheresultswillnotbecomparable.Ifusingalaptop,youwon’tneedtoworryabouttheorderofquestions.Youwillhavetolearnwhattodoifthecomputerfreezesorsomeothertechnicalglitchoccurs.
6. Interviewersshouldfollowallinstructionsgivenatthetrainingsessionanddescribedontheinterviewform.
7. Makecertaininterviewersunderstandtheinformedconsentprocess,andthattheyadheretoit.Someorganizationsandtheirethicsboardsrequiretheuseofspecificlanguage,whichtheinterviewermustusewithoutmodification.Forexample:
Ifyouhaveanyquestionsorconcernsregardingyourrightsasasubjectinthisstudy,youcancontacttheInstitutionalReviewBoard(IRB)at000-200-5138oraccesstheirwebsiteathttp://www.irb.wecare.org.
Monitoringinterviewquality
Tomakesureyouaregettingthemostaccuratedatapossible,youshouldmonitorthequalityoftheinterviews.Thismightmeansomethingasinformalashavinginterviewerscallyouonceaweekorsomethingasformalashavingthemsubmittoyouastandardizedchecklistofactivitiestheyperformeachday.Ifpossible,youmayactuallywanttogowithaninterviewer(ifitisaface-to-faceinterview)orspendtimewithtelephoneinterviewerstomakesurethatwhattheyaredoingisappropriateforthesurvey’spurposes.Topreventproblems,youmightwanttotakesomeorallofthefollowingsteps:
Establishahotline—someoneavailabletoansweranyquestionsthatmightoccurimmediately,evenatthetimeofaninterview.Providewrittenscriptsfortheinterviewer.Ifinterviewersaretointroducethemselvesorthesurvey,givethemascriptorsetoftopicstocover.Makesureyougiveoutextracopiesofallsupplementarymaterials.Ifdatacollectorsaretomailcompletedinterviewsbacktoyou,forexample,makesuretogivethemextraformsordisksandenvelopes.Asanoption,considerhavingdatacollectorsemailtheresultsorenterthemdirectlyontoasecurewebsite.Prepareaneasy-to-readhandoutdescribingthesurvey.Provideascheduleandcalendarsothatinterviewerscankeeptrackoftheirprogress.Numeroussoftwareprogramsareavailabletohelpyoudothis.Considerprovidinginterviewerswithvisualaids.Visualaidsmaybeextremelyimportantwheninterviewingpeoplewhoseabilitytospeakthelanguagemaynotbeasexpertasisnecessarytounderstandthesurvey.Visualaidsarealsousefulinclarifyingideasandmakingsurethateverybody
isreactingtosimilarstimuli.Forexample,supposeyouwanttofindhowpeopleperceivetheeconomy:Isitimproving,remainingstable,orindecline?Toensurethateverybodyhasaccesstothesamesetofdataontheeconomy,youmightshowgraphsandtablesdescribingtheeconomytakenfromthelocalnewspaperforthelast1or2years.Anotheruseofavisualaidmightbeinasurveyofpeople’sidealexpectationsforaplannedcommunity.Youmightshowseveraldifferentplansandaskpeopletodescribetheirreactionstoeach.Thepreparationofaudiovisualaidsforuseinaninterviewisrelativelyexpensiveandrequiresthattheinterviewersbespeciallytrainedinusingthem.Considerthepossibilitythatsomeinterviewersmayneedtoberetrainedandmakeplanstodoso.
ReliabilityandValidity:TheQualityofYourSurveyArulerisconsideredtobeareliableinstrumentifityieldsthesameresultseverytimeitisusedtomeasurethesameobject,assumingthattheobjecthasnotchanged.Ayardstickthatshowsthatyouare6feet1inchtalltodayandalso2weeksfromtodayisreliable.
Peoplechange,ofcourse.Youmaybemoretired,angry,andtensetodaythanyouwereyesterday.Peoplealsochangebecauseoftheirexperiencesorbecausetheylearnedsomethingnew,butmeaningfulchangesarenotsubjecttorandomfluctuations.Areliablesurveyprovidesaconsistentmeasureofimportantcharacteristicsdespitebackgroundfluctuations.Itreflectsthe“true”score—onethatisfreefromrandomerrors.
Arulerisconsideredtobeavalidinstrumentifitprovidesanaccuratemeasure(freefromerror)ofaperson’sheight.Buteveniftherulersaysyouare6feet1inchtalltodayand6monthsfromnow(meaningitisreliable),itmaybeincorrect.Thiswouldoccuriftherulerwasnotcalibratedaccuratelyandyouarereally5feet6inchestall.
Ifyoudevelopasurveythatconsistsofnothingmorethanaskingahospitaladministratorhowmanybedsareinagivenwardandyougetthesameansweronatleasttwooccasions,youwouldhaveaninstrumentthatisreliable.Butifyouclaimthatthesamesurveymeasuresthequalityofmedicalcare,youhaveareliablesurveyofquestionablevalidity.Avalidsurveyisalwaysareliableone,butareliableoneisnotalwaysvalid.
IstheSurveyReliable?Inreviewingapublishedsurvey(alsoinassessingthequalityofahomemadeform),youshouldaskthefollowingquestionsaboutthreetypesofreliability:test-retest,equivalence,andinternalconsistency.
First,doesthesurveyhavetest-retestreliability?Onewaytoestimatereliabilityistoseewhethersomeonetakingthesurveyanswersaboutthesameonmorethanoneoccasion.Test-retestreliabilityisusuallycomputedbyadministeringasurveytothesamegroupontwodifferentoccasionsandthencorrelatingthescoresfromonetimetothenext.Asurveyisconsideredreliableifthecorrelationbetweenresultsishigh;thatis,peoplewhohavegood(orpoor)attitudesonthefirstoccasionalsohavegood(orpoor)attitudesonthesecondoccasion.
Second,arealternativeformsequivalent?Iftwodifferentformsofasurveyaresupposedtoappraisethe
sameattitude,youshouldmakesurethatpeoplewillscorethesameregardlessofwhichonetheytake.IfyouwanttouseFormAofthesurveyasapremeasure,forexample,andFormBasapostmeasure,checktheequivalenceofthetwoformstomakesureoneisnotdifferentfromtheother.
Equivalencereliabilitycanbecomputedbygivingtwoormoreformsofthesamesurveytothesamegroupofpeopleonthesamedayorbygivingdifferentformsofthesurveytotwoormoregroupsthathavebeenrandomlyselected.Youdetermineequivalencebycomparingthemeanscoreandstandarddeviationsofeachformofthesurveyandbycorrelatingthescoresoneachformwiththescoresontheother.Ifthevariousformshavealmostthesamemeansandstandarddeviationsandtheyarehighlycorrelated,thentheyhavehighequivalencereliability.Equivalencereliabilitycoefficientsshouldbehigh;lookforcoefficientsthatareasclosetoperfectaspossible.
Third,howinternallyconsistentarethesurveyquestionsinmeasuringthecharacteristics,attitudes,orqualitiesthattheyaresupposedtomeasure?Totestforinternalconsistency,youcalculateastatisticcalledcoefficientalpha(orCronbach’salpha,namedforthepersonwhofirstreportedthestatistic).Coefficientalphadescribeshowwelldifferentitemscomplementeachotherintheirmeasurementofthesamequalityordimension.
Manysurveyorsarenotatallconcernedwithinternalconsistencybecausetheyarenotgoingtobeusingseveralitemstomeasureoneattitudeorcharacteristic.Instead,theyareinterestedintheresponsestoeachitem.Decidewhetheryoursurveyneedstoconsiderinternalconsistency.Herearetwoexamples.
Whatisadequatereliability?Thecriteriondependsonthepurposeofthesurvey.Tocomparegroups(say,employeesatCompanyAwithemployeesatCompanyB),reliabilitycoefficientsof.50oraboveareacceptable.Tomakedecisionsaboutindividualeducationalorhealthneeds,youneedcoefficientsof.90.
Example:InternalConsistencyCountsA10-iteminterviewisconductedtofindoutpatients’satisfactionwithmedicalcareinhospitals.Highscoresmeanmuchsatisfaction,whereaslowscoresmeanlittlesatisfaction.Towhatextentdothe10itemsmeasurethesamedimensionofsatisfactionwithhospitalcare?
Example:InternalConsistencyDoesNotCountA10-iteminterviewisconductedwithpatientsaspartofastudytofindouthowhospitalscanimprove.Eightitemsaskaboutpotentialchangesindifferentservices,suchasthetypeoffoodthatmightbeserved;theavailabilityofdoctors,nurses,orotherhealthprofessionals;andsoon.Twoitemsaskpatientsfortheirage.Becausethissurveyisconcernedwithviewsonimprovingeightdifferentservicesandprovidingdataonageandeducationofrespondents,eachitemisindependentoftheothers.
IstheSurveyValid?Asurveyisvalidiftheinformationitprovidesisanaccuratereflectionofrespondents’knowledge,attitudes,values,andbehavior.Herearesomequestionstoaskaboutapublishedsurvey’svalidity:
Doesthesurveyhavepredictivevalidity?Youcanvalidateasurveybyprovingthatitpredictsanindividual’sabilitytoperformagiventaskorbehaveinacertainway.Forexample,amedical
schoolentranceexaminationhaspredictivevalidityifitaccuratelyforecastsperformanceinmedicalschool.Onewaytoestablishpredictivevalidityistoadministerthesurveytoallstudentswhowanttoentermedicalschoolandcomparethesescoreswiththeirperformanceinschool.Ifthetwosetsofscoresshowahighpositiveornegativecorrelation,thesurveyorinstrumenthaspredictivevalidity.Doesthesurveyhaveconcurrentvalidity?Youcanvalidateasurveybycomparingitagainstaknownandacceptedmeasure.Toestablishtheconcurrentvalidityofanewsurveyofattitudestowardmathematics,youcouldadministerthenewsurveyandanalreadyestablished,validatedsurveytothesamegroupandcomparethescoresfrombothinstruments.Youcanalsoadministerjustthenewsurveytotherespondentsandcomparetheirscoresonittoexperts’judgmentoftherespondents’attitudes.Ahighcorrelationbetweenthenewsurveyandthecriterionmeasure(theestablishedsurveyorexpertjudgment)meansconcurrentvalidity.Remember,aconcurrentvaliditystudyisvaluableonlyifthecriterionmeasureisvalid.Doesthesurveyhavecontentvalidity?Asurveycanbevalidatedbyprovingthatitsitemsorquestionsaccuratelyrepresentthecharacteristicsorattitudestheyareintendedtomeasure.Asurveyofpoliticalknowledgehascontentvalidity,forexample,ifitcontainsareasonablesampleoffacts,words,ideas,andtheoriescommonlyusedwhendiscussingorreadingaboutthepoliticalprocess.Contentvalidityisusuallyestablishedbyreferringtotheoriesaboutpersonality,emotions,andbehaviorandbyaskingexpertswhethertheitemsarerepresentativesamplesoftheattitudesandtraitsyouwanttosurvey.Doesthesurveyhaveconstructvalidity?Surveyscanbevalidatedbydemonstratingthattheymeasureaconstructsuchashostilityorsatisfaction.Constructvalidityisestablishedexperimentallybytryingthesurveyonpeoplewhomclinicalexpertssaydoanddonotexhibitthebehaviorassociatedwiththeconstruct.Ifthepeoplewhomtheexpertsjudgetohavehighdegreesofhostilityorsatisfactionalsoobtainhighscoresonsurveysdesignedtomeasurehostilityorsatisfaction,thenthesurveysareconsideredtohaveconstructvalidity.
SelectingandAdaptingSurveysTherearetensofthousands(maybemore)ofsurveysandsurveyquestionsthatyoucangetforfreeontheweb.Keepinmindthatthesehavebeendevelopedforgeneralaudiencesandmaynotbeappropriateforyoursurvey.Also,rememberthatyouhavenowayofknowingifthefreequestionshaveeverbeentriedoutoranalyzedand,ifso,withwhom.Perhapsmostimportantly,onceyouaccessasite’ssurveys,youmaybeaskedtosignaserviceagreementorbewarnedthat“cookies”willbeinstalledonyourbrowser.Thecostofthefreeitemsmaynotbeworthittoyou.
Hereisanexcerptfromarealserviceagreementthatmustbesignedinordertodownloadasurveytemplate.
Example:ExcerptFromaRealOnlineServiceAgreementforFreeSurveyItems(NamesChanged)Inordertooperateandprovidetheservice,wecollectcertaininformationaboutyou.Aspartoftheservice,wemayalsoautomaticallyuploadinformationaboutyourcomputer,youruseoftheservice,andserviceperformance.WeuseandprotectthatinformationasdescribedintheExtrasoftOnlinePrivacyStatement(http://go.extrasoft.com/fwlink/?LinkId=74170).Inparticular,wemayaccessordiscloseinformationaboutyou,includingthecontentofyourcommunications,inorderto(a)complywiththelaworrespondtolawfulrequestsorlegalprocess;(b)protecttherightsorpropertyofExtrasoftorourcustomers,includingthe
enforcementofouragreementsorpoliciesgoverningyouruseoftheservice;or(c)actonagoodfaithbeliefthatsuchaccessordisclosureisnecessarytoprotectthepersonalsafetyofExtrasoftemployees,customers,orthepublic.
TheserviceisaprivatecomputernetworkthatExtrasoftoperatesforthebenefitofitselfanditscustomers.Extrasoftretainstherighttoblockorotherwisepreventdeliveryofanytypeofemailorothercommunicationtoorfromtheserviceaspartofoureffortstoprotecttheservice,protectourcustomers,orstopyoufrombreachingthiscontract.Thetechnologyorothermeansweusemayhinderorbreakyouruseoftheservice.
GuidelinesforFindingUseableandUsefulSurveysintheResearchLiteratureOnewaytomakesurethatyouareusingareliableandvalidsurveyistorelyononethathasbeencarefullytestedbyothersurveyorsandresearchersinascientificstudy.Findingscientificallytestedmeasurescanbeparticularlyimportantifyouwanttosurveyattitudes,emotions,healthstatus,qualityoflife,andmoralvalues.Thesefactors,andotherslikethem,areelusiveanddifficulttomeasure,andproducingvalidmeasuresrequiresresearchexpertiseandfunding.Unlessyouarepreparedtodosuchastudy,youwillneedtofindasurveythatisreliableandvalidandappropriateforyourneeds.
Youcanfindsurveysinthepublishedliteratureandonlineonsitesdevotedtosurveysofpublicopinionorthosethatmaintaindataonthepublic’svalues,health,education,andwell-being(seeexampleonnextpage).
Manyresearchersinfieldslikehealth,education,andthesocialsciencesdevelop,validate,andusesurveysandpublishtheirfindingsinpeer-reviewedjournals.Thepublishedliteratureisanoutstandingsourceofready-madesurveysandsurveyquestions.Buthowdoyoufindthesesurveysburiedintheworld’svastandever-growingliterature?Andifyoufindthem,howdoyouselectamongthem?Answeringthesequestionsmeansknowinghowtosearchtheliteratureandevaluateitsqualityandappropriateness.Herearesevenguidelinestohelpyouthroughtheprocess:
1.Selectoneormoreonlinedigitallibrariesthatarerelevanttoyoursurvey.Onlinelibrariescontainmuchoftheworld’spublishedresearcharticles.CommonlyusedonlinelibrariesincludePubMed(medicineandhealth),ERIC(education),andtheWebofScience(socialscienceandscience).Thesedigitallibrariesprovidefreeaccesstocompletearticlesforsomejournals.Mostuniversitiesandsomebusinessesandpubliclibrariessubscribetooneormoreonlinereferencedatabases,andthroughthem,students,employees,andresearcherscanaccesstheworld’sjournals.ThePublicLibraryofScience(PLoS)consistsofopen-accessjournalsthatarefreetoeveryone.GoogleScholaralsoprovidesaccesstojournalarticlesthatcontainsurveys.GoogleScholarisnotspecificallydesignedforacademicandsurveypurposes,sofindingwhatyouneedmaytakelongerthanifyouuseadatabaseliketheWebofScience.
2.Specifythetopicsorvariablesyouwantyoursurveytocover.Comparethesetwoscenarios:
Surveyor1wantstofindoutwhystudentsinanewafterschoolprogramdropout.Surveyor1developstheAfterschoolDropoutSurvey.Thesurveyasksquestionsaboutfamilyincome,numberofchildreninthefamily,andeducationallevelofallfamilymembers.Surveyor1hypothesizesthatstudentswhodropoutaremorelikelytocomefromfamilieswithlowerincomesandlesseducationthanthosewhoremainintheprogram.
Surveyor2isalsointerestedinfindingoutwhythestudentsinthenewafterschoolprogramdropout.Surveyor2developstheAfterschoolDropoutSurvey.Thesurveyasksquestionsabouttheamountoftimestudentsspendeachdayreadingforpleasure,playingvideogames,orgoingonlinetocheckontheirfriends.Surveyor2hypothesizesthatthestudentswhospendthemosttimeplayingvideogamesorgoingonlinearemorelikelytodropoutthanarethosewhospendthemosttimereadingforpleasure.
Example:SearchforSchoolDropoutSurveys:SampleFindingsFromtheWebofScienceSource:ThomsonReutersWebofKnowledge.
Ifyouareinterestedinstudyingwhetherornotfamilyincomeandeducationareassociatedwithretentioninafterschoolprograms,thenSurvey1isagoodchoice.IfyouareinterestedintherelationshipbetweendroppingoutofanafterschoolprogramandhowmuchtimestudentsspendonactivitieslikevideogamesandFacebook,thenSurvey2istheoneforyou.NotethatbothSurveyor1andSurveyor2nametheirsurveythe“AfterschoolDropoutSurvey.”Surveyswiththesameorsimilartitlesmayactuallymeasurequitedifferentconcepts.Youneedtospecifyinadvancewhatyouwanttoknowandcarefullyevaluatetheappropriatenessofeachavailablesurvey.
3.Learnhowtosearchtheappropriateelectroniclibraries.Alldigitallibrariesprovidetutorialsforsearchingtheirdatabasesandarerelativelyuser-friendly.HereisanexampleofthesearchresultsforsurveysofschooldropoutsintheWebofScience.Youcanrefineyoursearchbychoosingthedisciplines,typeofarticle,languages,yearsofpublication,andsoon.
Onceyouhavethelistofpotentialarticles,checkeachonetobesureitcontainsasurvey.Ifyouneedto
continuethesearchbecauseyoudidnotfindappropriatesurveys,studythekeywordsinrelevantarticles.Youcanoftenfindthekeywordsnexttotheabstractorsummaryofthearticle.Eachdigitallibrarycataloguesitsarticlesusingsubjecttermsanddescriptors.Youcanlearnhowtousethesefromthelibrary’stutorials.
4.Makecertainthatyoucanadministerthesurveythewayitwasintendedorthatyouhavetheresourcestoadaptit.Supposeyoufindasurveyintheliteraturethatcoversthetopicsyouplanonaddressing.Youalsofindthatithasfaceandcontentvalidity.However,thesurveyisaface-to-faceinterviewandyouwantanonlinesurvey.Youneedconsiderableexpertisetotranslateaninterviewquestionnairetoanonlinesurvey.Oncedone,itmustbepilottestedextensively.Translation,pilottesting,andrevisionsbasedonthepilotcandelaythestartofyoursurveyandbecostly.So,eventhoughthesurveylooksgood,itmaynotbeappropriateforyourimmediateneeds.Youhavetolookfurther.
5.Checkonthecharacteristicsofthesurveyrespondents.Supposeyoufindanarticlewithasurveythatlooksreallygood.Asyoulookatthearticleinwhichthesurveywasused,askyourselfiftherespondentsaresimilartoyourplannedsample.Asurveythatisdesignedtofindouthowchildrencopewithviolence,forinstance,maybequitedifferentinlanguageandconceptfromonethatismeanttostudyadultcoping.
6.Selectthesurveythathasthebestavailableevidenceofreliabilityandvalidity.Thebestavailableevidencecomesfrommethodologicallyrigorousstudies.Thesearestudiesthatarecarefullycontrolled(say,randomizedcontrolledtrials),withjustifiableandmeaningfulsamples,appropriatestatistics,andsoundconclusions.Theydescribeandjustifytheirchoiceofallmeasures,includingsurveys.Itisoftenagoodideatogototheoriginalsourcelistedinthereferencestocheckonhowthesurveywasvalidated.
Thefollowinggivesyouthreesamplesofwrite-upsintheresearchliteratureregardingreliabilityandvalidity.
Example:SampleReportsofReliabilityandValidityintheResearchLiterature
1.Measuringloveinromanticrelationships:Ameta-analysisThisstudy(Graham,2011)aimedtoidentifythehigher-orderfactorsunderlyingthemostpopularmeasuresofloveinusetoday.Theresearchersusedamethodtheycalled“meta-analyticfactoranalysis.”Eighty-onestudiesrepresenting103samplesand19,387individualsprovideddataforthemeta-analysis.Thereportedcorrelationsbetweenseveralcommonlyusedmeasuresoflovewereaggregatedacrossstudiestoformameta-analyticcorrelationmatrix,whichwasthensubjectedtoprinciplecomponentsanalysis.Factorssuggestinggenerallove,romanticobsession,andpracticalfriendshipemerged.Lovewaspositivelyandobsessionwasnegativelyassociatedwithrelationshipsatisfactionandlength.Theresultscorroboraterecentconceptualizationsoflove(concurrentvalidity)andpointtowardtheneedformeasurementmethodsotherthanglobalself-reportinthefieldofloveresearch.
2.ReliabilityandConstructValidityofPsychopathyChecklistYouthVersionScoresamongIncarceratedAdolescentGirlsAlthoughrecentstudieshavedemonstratedtheutilityofassessingpsychopathictraitsinadolescentmales,thereissubstantiallylessevidenceontheutilityofassessingpsychopathictraitsinadolescentfemales.Thisstudy(Bauer,Whitman,&Kosson,2011)investigatedthereliabilityandconstructvalidityofthePsychopathyChecklist:YouthVersion(PCL:YV)amongasampleof80
incarceratedadolescentfemales.ReliabilityanalysesindicatedhighinterrateragreementandinternalconsistencyforPCL:YVscores.Consistentwithresearchonadultsandadolescentmaleoffenders,psychopathictraitswereassociatedwithagreaternumberofconductdisorderandalcoholdependencesymptoms,earlieronsetofcriminalbehavior,apropensitytowardinstitutionalviolence,andinterpersonalbehaviorsassociatedwithpsychopathy(constructvalidity).Consistentwithsomepriorstudiesofyouth,therewerenonegativerelationshipsbetweenPCL:YVscoresandinternalizingdisorderdiagnosesandsomepositiveassociationsbetweenpsychopathyratingsandnegativeaffectivity(constructvalidity).Resultssuggestthatassessingpsychopathictraitsinadolescentfemalesprovidesmeaningfulinformationregardingcriminalbehavior,patterns,andpersonalitytraitsinyouth.
3.Determinantsoffast-foodconsumption:AnapplicationofthetheoryofplannedbehaviorThisstudy(Dunn,Mohr,Wilson,&Wittert,2011)examinedthefactorsthatinfluencefast-foodconsumptioninAustralia.TheresearchersusedtheTheoryofPlannedBehavior(contentvalidity)tosurvey404peopleabouttheirattitudes,intentions,andbehaviors.Toexaminethefactorsthatpredictedfast-foodconsumption(predictivevalidity)theresearchersusedatechniquecalledstructuralequationmodeling.Theyusedothermethods,includingonecalledfactoranalysis,torefinethedefinitionsoftheconstructsthatcompriseattitudestowardfastfood.Theresearchersfoundthatfast-foodconsumptionmaybeinfluencedbyconvenience,satisfaction,familyandfriends,busylifestyles,experiencingcravingsforfastfood,notknowinghowtocook,workinglonghours,andeatingalone.People’sconcernforlong-termhealthisprobablyoverriddenbytheshort-termbenefitsoffastfood.
7.Tryoutthesurveywithasmallgroupofpeopletoseeifitisuseableanduseful.Thenpilottestitonalargergrouptobesure!
FindingSurveysontheWebSomeorganizationsoffersurveyquestionsfreeonlinetothepublic.Herearejustafewexamples.
GallupOrganization(http://www.gallup.com):
Onthissite,youcanfindrecentpoliticalandeconomicpollingquestionsandresults,aswellas“specialreports”thatexamineattitudestowardsocialissuesandconsumerbehavior.
GeneralSocialSurvey(http://www.norc.org/Research/Projects/Pages/general-social-survey.aspx):
TheGeneralSocialSurveyhasbeenconductedsince1972bytheNationalOpinionResearchCenter.Itmeasuresattitudestowardsocialandpublicpolicyissues,economicstatus,politicalevents,work,andfamilylife.
NationalOpinionResearchCenter(http://norc.uchicago.edu/homepage.htm):
NORCconductssurveyresearchonpublicpolicyissuesforavarietyofagenciesandcorporations.Seethe“Projects”sectionforcopiesofrecentsurveysandtheirreports.
ThePewCharitableTrusts(http://www.pewtrusts.org/):
Pewprovidesaccesstosurveysandsurveyquestionsconcerningthepubliconmediaandjournalism,religionandpubliclife,societyandtheInternetandglobalattitudes,andHispanicsinAmericaandpublicopinion.
TheAgencyforHealthcareResearchandQuality(AHRQ)(http://www.ahrq.gov/data/dataresources.htm):
Atthesite,youcanfindinformationthatwillhelpyouwithsurveysonhealthcaredisparities,qualityandsafety,andHIVAIDS,tonameafewtopics.
TheU.S.Census(http://www.census.gov/):
Thisisamajorsourceofsurveyquestions.Regardlessofhowlargeorsmallasurveyis,andevenifittakesplaceoutsidetheUnitedStates,thisisanexcellentsitetoguidethedevelopmentandadaptationofsurveyquestionsaboutage,ethnicity,income,education,andoccupation.TheU.S.Censusalsoprovidessurveyquestionsthatarerelevanttocriminaljustice(http://www.census.gov/govs/cj/).Samplesurveysincludetheannualparoleandprobationsurveyandsurveysofsexualviolence.
TheInstituteofEducationSciencesthroughtheNationalCenterforEducationStatistics(http://nces.ed.gov/datatools/index.asp?DataToolSectionID=3):
Thissitehasanextensivelibraryofsurveys.Anexampleofthesurveysyouwillfindifyougotothesiteappearsonthenextpage.Asyoucansee,internationalsurveysareincluded.
Remember,buyer,beware!Becarefultochoosesurveysandindividualquestionsthatofferprooftheyhavebeenusedwithrespondentslikeyours.Someonlinesurveycompaniesprovidefreequestionsonarangeoftopicsfromcustomersatisfactiontohealthstatus.Manyofthesequestionshaveneverbeentestedorusedwithanyone,sotheirvalueisnotknown.Thebestsafeguardagainstpoorsurveysandsurveyquestionsistopilottestthemortrythemout.
Example:SearchforEducationSurveys:ASamplePageFromtheNationalCenterforEducationStatistics
TheSurveyIsPutonTrial:GuidelinesforPilotTestingHerearesomebasicrulesforpilottestingasurvey:
1.Trytoanticipatetheactualcircumstancesinwhichthesurveywillbeconductedandmakeplanstohandlethem.Forinterviews,thismeansreproducingthetrainingmanualandallforms;formailedquestionnaires,youhavetoproduceanycoverletters,returnenvelopes,andsoon.Foronlinesurveys,youhavetotesttheeasewithwhichrespondentscanlogontoasecurewebsiteandaccessthesurvey.Needlesstosay,thisrequiresplanningandtimeandcanbecostly,somakesureyoursurveybudgetprovidesforthissetofactivities.
2.Youcanstartbytryingoutselectedportionsofthesurveyinaninformalfashion.Justthedirectionsonaself-administeredquestionnairemightbetestedfirst,forexample,orthewordingofseveralquestionsin
aninterviewmightbetested.Youmayalsowanttotryoutthesurveyprocessinitiallybyusingadifferentmethodfromtheoneyoueventuallyintendtouse.Soifyouareplanningtohandoutquestionnairestoconferenceparticipants,thetrialmayinvolveaninterviewsothatanyproblemswiththequestionsontheformcanbediscussed.Intheend,youshouldgivethesurveylogisticsandformafairpretrial.
3.Chooserespondentssimilartotheoneswhowilleventuallycompletethesurvey.Theyshouldbeapproximatelythesameage,withsimilareducation,andsoon.Donotincludepilottestrespondentsintheactualsurvey.Pilottestrespondentsmayfeelinvestedinthesurveybecausetheyhelpedtocreateit,and,becausetheyarefamiliarwiththesurvey’spurposeandlanguage,theymayhavepreconceivedideasabouthowtogivethe“best”answers.
4.Enlistasmanypeopleinthepilotasseemsreasonablewithoutwastingyourresources.Probablyfewerpeoplewillbeneededtotesta5-itemquestionnairethana20-itemone.Also,ifyouseethatthesurveyneedslittleimprovement,stop.Aruleofthumbistostartwithfivepeopleandcontinueonwithsmallgroupsuntilyounolongerreceivenewinformation.
5.Forreliability,focusontheclarityofthequestionsandthegeneralformatofthesurvey.Hereiswhattolookfor:
FailuretoanswerquestionsGivingseveralanswerstothesamequestionWritingcommentsinthemarginsofwrittensurveys
Anyoneoftheseisasignalthatthequestionnairemaybeunreliableandinneedofrevision.Arethechoicesinforced-choicequestionsmutuallyexclusive?Haveyouprovidedallpossiblealternatives?Isthequestionnaireorinterviewlanguageclearandunbiased?Dothedirectionsandtransitionsmakesense?Haveyouchosentheproperorderforthequestions?Isthequestionnairetoolongorhardtoread?Doestheinterviewtaketoolong?(Forinstance,youplannedfora10-minuteinterview,butyourpilotversiontakes20minutes.)
6.Tohelpbolstervalidity,youshouldensurethatallrelevanttopicshavebeenincludedinthesurvey(givenyourresources).Forasurveyofpoliticalattitudes,haveyouincludedallpoliticalparties?Controversialissues?Whatelsemustbeincludedforyoursurveytohavecontentvalidity?Ifyouarenotcertain,checkwithotherpeople,includingthetrial-runrespondents.Doesthesurveyhaveroomfortheexpressionofallviews?Supposeyouaresurveyingpeopletofindouthowreligioustheyare.Ifyouhadproofinadvancethatallareveryreligious,youwouldnotneedasurvey.Unlessyoucanshowthatatleastintheoryyoucandistinguishthereligiousfromthenonreligious,noonewouldbelievetheresults.Howdoyoufixthis?Inthetrials,choosepeopleyouknowarereligiousandthoseyouknowarenotandgivethemthesurvey.Dotheirresponsesdiffer?
7.Testyourabilitytogetarangeofresponses.Ifpeopletrulydifferintheirviewsorfeelings,willthesurveycapturethosedifferences?Supposeyoursurveyisofasuburbanneighborhood’sattitudetowardaproposednewhigh-risebuildingdevelopment.Youshouldadministerthesurveytopeoplewhoarebothforandagainstthebuilding.Thisapproachwillhelprevealyourownbiasesinhowquestionsarewordedand,forclosedquestions,mighthelpyouidentifychoicesthatpeoplewhofeelstronglyonewayperceiveasmissingbutthatyoumightnothavethoughtofpreviously.
Considerthis:Inapilotofasurveyofchildren’sself-reporthealthbehaviors,respondentswereasked
howoftentheywashedtheirhandsaftereating.Allsixchildrenbetween8and10yearsofageansweredalwaysafterbeinggiventhechoicesalways,never,andIdon’tknow.Thechoiceswerechangedtoalmostalways,usually,andalmostnever.Withthenewcategories,thesamesixchildrenchangedtheiranswerstotwoalmostalwaysandfourusually.
AFar-ReachingWorld:Surveys,Language,andCultureManysurveysaretranslatedintodifferentlanguages.Ifyouplanontranslatingyoursurvey,donotassumethatyoucanautomaticallyrewordeachquestionintothenewlanguage.Betweentheoriginallanguageandthenextlanguageoftenliesaculturalgap.Youmayneedtorewordeachsurveyquestion.
Toavoidconfusingpeopleandeveninsultingthembecauseyoumisunderstandtheirlanguageorculture,youshouldfollowafewsimpleguidelines.Thesestrategiesinvolveenlistingtheassistanceofpeoplewhoarefluentinthelanguage(anditsdialects)andpilottestingthesurveywithtypicalrespondents.Followtheseguidelines.
GuidelinesforTranslatingSurveysUsefluentspeakerstodothefirsttranslation.Ifyoucan,usenativespeakers.Ifyoucanaffordit,findaprofessionaltranslator.Theartoftranslationisinthesubtleties—wordsandphrasesthattakeyearsandculturalimmersiontolearn.Ifyouusefluentspeakers,youwillminimizethetimeneededtorevisequestionwordingandresponsechoices.Tryoutthetranslatedsurveywiththreetofivenativespeakers.Askthemthesequestions:Whatisthisquestionaskingyou?Canyouthinkofabetterwaytoaskthisquestion?Revisethesurveywiththehelpoftheoriginaltranslator.Translatethesurveybackintotheoriginallanguage.Useadifferenttranslatorforthistask.Doesthis“back-translated”surveymatchtheoriginalversion?Ifnot,thetwotranslatorsshouldworktogethertomakethemmatch.Trytheresultingsurveyonasmallgroup(5–10)oftargetrespondents.Ifthetwotranslatorscouldnotagreeonwording,letthegroupdecide.Revisethesurvey.Pilottestthesurvey.Produceafinalversion.Ifyouwanttofindouttherespondents’backgroundsortheirethnicity,youshouldprobablyrelyonexistingquestionsratherthancreatingyourown.ThefollowingquestionaboutethnicitycomesfromtheU.S.DepartmentofHealthandHumanServices(http://minorityhealth.hhs.gov/templates/content.aspx?ID=9227&lvl=2&lvlID=208).
Example:QuestionAboutEthnicity:U.S.DepartmentofHealthandHumanServicesOfficeofMinorityHealthAreyouHispanic,Latino/a,orSpanishorigin?(Oneormorecategoriesmaybeselected)
1. No,notofHispanic,Latino/a,orSpanishorigin2. Yes,Mexican,MexicanAmerican,Chicano/a3. Yes,PuertoRican4. Yes,Cuban5. Yes,anotherHispanic,Latino,orSpanishorigin
Whatisyourrace?(Oneormorecategoriesmaybeselected)
1. White2. BlackorAfricanAmerican3. AmericanIndianorAlaskaNative
4. AsianIndian5. Chinese6. Filipino7. Japanese8. Korean9. Vietnamese10. OtherAsian11. NativeHawaiian12. GuamanianorChamorro13. Samoan14. OtherPacificIslander
ThesequestionsaboutethnicityandracearerelevanttotheU.S.population.IntheUnitedKingdom,othercategoriesareused,suchasWhite,Asian(Indian,Pakistani,Bangladeshi),Black(Caribbean,African),mixedrace,andChinese(http://www.communities.gov.uk/publications/corporate/statistics/citizenshipsurvey200809equality).
SummingUpThelengthofasurveydependsonwhatyouneedtoknowandhowmanyquestionsarenecessarysothattheresultinganswersarecredible.Anotherconsiderationistherespondents.Howmuchtimedotheyhaveavailable,andwilltheypayattentiontothesurvey?Questionsshouldproceedfromthemostfamiliartotheleast.Placerelativelyeasy-to-answerquestionsattheendofasurvey.Avoidmanyitemsthatlookalike.Avoidskippatternsinwrittenself-administeredquestionnaires.Onlinesurveysareeffectivevehiclesforbranchingifyoumustusethembecauseyoucandesignthesoftwaretoguidetherespondent,andeverycommercialvendorprovidesskip-patternlogic.Thekeytoagoodtelephoneorface-to-faceinterviewistraining,whichshouldensurethatallinterviewersknowwhatisexpectedofthemandthattheyaskallthequestionsinthesameway,withinthesameamountoftime.Interviewsshouldbemonitoredtoensureinterviewersareaskingquestionsinthesamewayandmaximizeconsistencyofresponses(reliability).Pilottestingbolstersreliabilityandvaliditybecauseitcanhelpyouseethatalltopicsareincludedandthatsufficientvarietyintheresponsesisavailable.Areliablesurveyprovidesconsistentinformation.Avalidsurveyprovidesaccurateinformation.Onewaytoensurethatyouhaveareliableandvalidsurveyistouseonethatsomeoneelsehaspreparedanddemonstratedtobereliableandvalidthroughcarefultesting.Test-retestreliabilityisusuallycomputedbyadministeringasurveytothesamegroupontwodifferentoccasionsandthencorrelatingthescoresfromonetimetothenext.Equivalencereliabilityisdeterminedbygivingtwoormoreformsofthesamesurveytothesamegroupofpeopleonthesamedayorbygivingdifferentformsofthesurveytotwoormoregroupsthathavebeenrandomlyselected.Anothermeasureofreliabilityishowinternallyconsistentthequestionsonasurveyareinmeasuringthecharacteristics,attitudes,orqualitiesthattheyaresupposedtomeasure.Totestforinternalconsistency,calculateastatisticcalledcoefficientalpha(orCronbach’salpha).Predictivevaliditymeansthatthesurvey’sdataforecastrespondents’abilitytoperformagiventaskorbehaveinacertainway.Concurrentvaliditymeansthatthedatafromanewsurveycorrelatetothedatafromapreviouslyestablishedone.Asurveyhascontentvalidityifexpertsagreethatitsitemsorquestionsaccuratelyrepresentthe
characteristicsorattitudestheyareintendedtomeasure.Constructvalidityisestablishedexperimentallybytryingthesurveyonpeoplewhomclinicalexpertssaydoanddonotexhibitthebehaviorassociatedwiththeconstruct.Ifyouplanontranslatingasurvey,enlisttheassistanceofpeoplewhoarefluentinthelanguage(anditsdialects)andpilottestthesurveywithtypicalrespondents.Choosepublishedsurveyswithcare.Checktoseethattheyarereliable,valid,andappropriateforthepeopleinyoursurvey.
ThinkAboutThis1. Locatethearticlesthatfollowandanswerthesequestions:
1. Namethesurveyorsurveysdiscussedinthearticle.2. Describethepurposeofeachsurvey.3. Discussthereliabilityandvalidityofeachsurvey.4. Areyousatisfiedthatthesurveysarereliableandvalid?Explain.
2. Selectanythreesurveysthatinterestyou,andanswerthesamequestionsasinQuestion1.3. Readthecasestudybelow.Isitokayforthethirdprofessortohaveaccesstothesurveyevenifheis
responsibleformailingtherespondentstheirincentivechecks?
ThreeprofessorsreceiveagrantfromtheNationalLibraryofEducationtodoanonlinesurveyof60schoolstofindoutaboutthesuccessoftheirdataexchangesystem.Eachprofessorteachesatadifferentcollege.TwooftheprofessorsreceivepermissionfromtheirrespectiveIRBstogainaccesstorespondents’emailaddressesinordertosendoutthesurvey,followupwithnonrespondents,andperformthestatisticalanalysis.Thethirdprofessordoesnothaveexplicitpermissiontodoanyofthesethings,butheisinchargeofsendingoutthefinancialreimbursementsforparticipationinthesurvey.ThethreeprofessorsusedtheE-LineOnlineSurveyCompanytobuildtheirsurvey,sowiththeproperusernameandpassword,anyone,includingthethirdprofessor,canlogontothesurveysiteandviewtheresultsbyindividualandbyquestion.Thethirdprofessorneedstoobtainemailaddressestoaskformailingaddresses,whicharenotincludedinthesurvey.Helogsontothesurveyandrecordstheemailaddresses.HedoesnotlookatanyresponsestothesurveybecausehedoesnothaveIRBclearancetodoso.Heemailstheparticipantsand,immediatelyuponreceivingtheirmailingaddresses,sendsoutthereimbursementchecks.
ReferencesAsgeirsdottir,B.B.,Sigfusdottir,I.D.,Gudjonsson,G.H.,&Sigurdsson,J.F.(2011).Associationsbetweensexualabuseandfamilyconflict/violence,self-injuriousbehavior,andsubstanceuse:Themediatingroleofdepressedmoodandanger.ChildAbuse&Neglect,35(3),210–219.
Bauer,D.L.,Whitman,L.A.,&Kosson,D.S.(2011).Reliabilityandconstructvalidityofpsychopathychecklist.CriminalJusticeandBehavior,38(10),965–987.
Dunn,K.I.,Mohr,P.,Wilson,C.J.,&Wittert,G.A.(2011).Determinantsoffast-foodconsumption.Anapplicationofthetheoryofplannedbehaviour.Appetite,57(2),349–357.doi:10.1016/j.appet.2011.06.004.
Flynn,K.E.,Smith,M.A.,&Freese,J.(2006).WhendoolderadultsturntotheInternetforhealthinformation?FindingsfromtheWisconsinLongitudinalStudy.JournalofGeneralInternalMedicine,21(12),1295–1301.
Graham,J.M.(2011).Measuringloveinromanticrelationships:Ameta-analysis.JournalofSocialand
PersonalRelationships,28(6),748–771.
Holman,E.A.,Silver,R.C.,Poulin,M.,Andersen,J.,Gil-Rivas,V.,&McIntosh,D.N.(2008).Terrorism,acutestress,andcardiovascularhealth:A3-yearnationalstudyfollowingtheSeptember11thattacks.ArchivesofGeneralPsychiatry,65(1),73–80.
O’Connor,T.M.,Yang,S.-J.,&Nicklas,T.A.(2006).Beverageintakeamongpreschoolchildrenanditseffectonweightstatus.Pediatrics,118(4),e1010–e1018.
Overby,M.,Carrell,T.,&Bernthal,J.(2007).Teachers’perceptionsofstudentswithspeechsounddisorders:Aquantitativeandqualitativeanalysis.Language,Speech,&HearingServicesinSchools,38(4),327–341.
Richardson,J.,Coid,J.,Petruckevitch,A.,Chung,W.S.,Moorey,S.,&Feder,G.(2002).Identifyingdomesticviolence:Crosssectionalstudyinprimarycare.BritishMedicalJournal,324(7332),274.
Skoffer,B.(2007).Lowbackpainin15-to16-year-oldchildreninrelationtoschoolfurnitureandcarryingoftheschoolbag.Spine,32(24),E713.
Wolfson,M.,Champion,H.,Rogers,T.,Neiberg,R.H.,Barker,D.C.,Talton,J.W.,etal.(2011).Evaluationoffreetogrow:Headstartpartnershipstopromotesubstance-freecommunities.EvaluationReview,35(2),153–188.
4SAMPLING
OverviewShouldyousurveyeveryoneorjusttakeasample?Theanswerdependsonhowquicklyyouneedresults,ifthecredibilityofthefindingswillsufferifsomeoneorsomegroupisleftout,andyourfinancialandtechnicalresources.
Samplingcanbedividedintotwocategories.Arandomsampleisselectedbyanobjectivemethod(suchasdrawingnamesatrandomfromahat),andyoucanalsocalculateeachperson’schancesofselection.Nonrandomsamplesareconvenient:Youselectonlythoserespondentswhoarenearbyandwillingandavailabletocompletethesurvey.
Howlargeshouldasamplebe?Relativelylargersamplesreducesamplingerrors.Ifyouwanttoevaluatetheperformanceoftwo(ormore)groups,statisticalmethodscanguideyouinselectingsamplesizeslargeenoughtodetectdifferencesbetweenthegroups—ifdifferencesactuallyexist.Statisticalmethodsmayalsotellyouhowlargeadifferenceyoucanobserve(andifyouwillbeabletoobservedifferences)withthesamplesizeyouhave.
Theresponserateconsistsofthenumberofcompletedsurveysdividedbythenumberofsurveyseligibleforcompletion.Toimproveresponserate,makesurethatrespondentscaneasilycompleteandsubmitthesurvey,trainsurveyorstoadministersurveysandfollowup,monitorthequalityofsurveyadministration,keepresponsesconfidential,andproviderespondentswithincentivesandrewardswhenpossibleandethical.
SampleSizeandResponseRate:WhoandHowMany?Whenyouconductasurvey,doyouhavetoincludeeveryone?Theanswerisprobablynot.Often,thetargetpopulation—everyone—consistsofalargenumberofpeople.Surveyingthemisnotpractical.
Ifyoudecidetosample,youmustaskthefollowing:Howmanypeopleshouldbeincluded?Ifyourcompanyhas1,000employeesandyouwanttosurveyjustsomeofthem,howdoyoudecidehowmanypeopletoinclude?Sayyouwanttocomparelong-andshort-termjobsatisfaction.Statisticalmethodscanhelpyoudecidehowmanypersonsshouldbeincludedeachtime(shortversuslongterm)sothatifadifferenceinsatisfactionactuallyoccursovertime,youwillbeabletodetectit.Sayyouhave3monthstoconductthesurveyandyouthinkyoucansurveyabout300ofthecompany’semployees.Statisticalmethodscanhelpyoudeterminehowmuchofaneffectyoucandetect,giventhesizeofthesampleyourresourcesallowyoutoassemble.
Supposeyouwanttofindoutwhetheryourneighborswillsupportacommunityvigilanceprograminwhicheachhouseholdtakesresponsibilityforwatchingatleastoneotherhousewhentheownersareaway.Consideralsothatyoudefineyourcommunityashaving1,000homes.Doyouneedtoincludeallhouseholds?Ifyoudo,willtheprogrambemorelikelytoworkthanotherwise?Herearesomequestionsyoushouldanswer.
1. Howquicklyaredataneeded?Supposearecentincreaseinhouseburglariesisthemotivationforthesurveyandyouwanttogetstartedimmediately.Ifyouwaittosurveyall1,000homesinyourneighborhood,youmaywasteprecioustime.
2. Whattypeofsurveyisplanned?Ifyouaregoingtouseatelephoneoronlinesurvey,yoursurveywillprobablytakelesstimethanifyouplantointerviewpeopleintheirhomesorwaitforthemtoreturntheirsurveysinthemail.
3. Whatareyourresources?Iftheyarelimited,youhavetoselectasurveymethodsuchastelephoneinterviewingratherhomeinterviewing.
4. Howcrediblewillyourfindingsbe?Ifall1,000homesparticipate,thenyouwillhavenoproblemarguingthatyoursurveyisrepresentativeoftheneighborhood.Ifonly10homesparticipate,youwillrunintoacredibilityproblem.
5. Howfamiliarareyouwithsamplingmethods?Samplingmethodscanberelativelysimpleorcomplex.Nationalpollsandlargeresearchstudiesusesophisticatedtechniquesthatdependontheskillsofstatisticiansandothertrainedexperts.Othermethodsmaybeeasiertoimplement,buttheresultingsamplemaynotbeanaccuratereflectionofthelargerpopulation.
Howdoyouselectasample?Youcanusearandomorconveniencesamplingmethod.Arandomsampleisselectedobjectively.Anexamplewouldbetheuseofspecialsoftwaretorandomlyselectwinninglotterynumbers.Statisticalmethodsareavailabletocalculatetheprobabilitythateachpersonhasofbeingchosen(orwinningthelottery).Aconveniencesampleincludespeoplewhoareavailableandwillingtotakethesurvey.Thesampleselectionprocessisnotconsideredobjectivebecausenoteveryeligiblepersonhasanequalchance.Ifyouareillonthedayofthesurvey,youwillnotbeabletoparticipateevenifyoumeetallotherrequirements.
Considerthesetwocases.
Example:RandomandConvenienceSamplingCase1.AsurveyaimstofindouthowteachersintheLoebSchoolfeelaboutcertainschoolreforms.All100teachers’namesareputintoahat,thenamesarejumbled,andtheprincipalselects50forthesurvey.
Case2.AsurveyisconductedtofindoutLosHadasSchoolDistrictteachers’viewsoncertainschoolreforms.Tenteachersarechosentobeinterviewedfromeachofthedistrict’ssixelementaryschools,15areselectedfromitsfourintermediateschools,and60arechosenfromitsonehighschool.Participatingteachersarevolunteerswhowererecommendedbytheirprincipal.Theymeetthefollowingcriteria:Theyhavebeenteachinginthedistrictfor5yearsormore,theybelongtooneofthreeteachers’associationsorunions,andtheyhaveparticipatedinatleastonemeetingduringthepastyearonthedistrict’sschoolreform.
Inthefirstsurvey,asampleof50of100teachersischosenfromahat.Thistypeofsamplingisconsistentwithrandomsamplingbecausetheselectionmethodisobjectiveandyoucanuseamathematicalformulatocalculatetheprobabilityofeachpersonbeingselected.Also,youcanusuallyexpectthatthetwogroupsofteacherswillnotbesystematicallydifferentfromoneanother.Ineachgroupof50,youarelikelytohavesimilarnumbersofpeoplewhoaresmart,healthy,andgenerousandsimilarnumberswhoarenotsmart,unhealthy,andmiserly.
Inthesecondsurveyexample,principalsmakerecommendationsandeligibleteachersvoluntarilychoosetoparticipate.Principalsmayhavetheirfavorites.Teacherswhovolunteermaysystematicallydifferfromthosewhodonot.Theymaybemoreenthusiasticaboutthesurveyorhavemoretimetocompleteit,forexample.Thissamplingstrategyisnotobjective.Itcomesupwithanonrandomsample.
RandomSamplingMethodsThreeofthemostcommonlyusedrandomsamplingmethodsarethefollowing:
Simplerandomsampling
StratifiedrandomsamplingSimplerandomclustersamplingStratifiedrandomclustersampling
Asimplerandomsampleisoneinwhicheachpersonhasanequalchanceofbeingselectedfromapopulation.Thepopulationcontainseveryonewhoiseligibleforthesurvey.
Thefollowingisasimplerandomsampling.
Example:SimpleRandomSampling(1)Youwanttoselect100peoplefromphilanthropicfoundationstosurveythemaboutthetypesofgrantstheysponsor.Atotalof400peoplecanprovidethisinformation.Youplacetheirnamesinanyorder.Eachnameisgivenanumberfrom001to400.Then,usingatableofrandomnumbers(foundinstatisticstextbooks),youselectthefirst100peoplewhosenumbersshowuponthetable.
Eachpersoninthisscenariohasanequalopportunityforselection.Thepopulationconsistsof400people.Chancealonedecideswhichofthemissampled.
Thefollowingisnotrandomsampling.
Example:NotRandomSamplingYouwanttosample100peoplefromphilanthropicfoundationstosurveythemaboutthetypesofgrantstheysponsor.Atotalof400peoplecanprovidethisinformation.Youselect25peopleineachoffourareasofthecountry.
Somepeopleinthisscenariohavenochanceofselection:thosewhodonotliveinyourfourchosengeographicareas.
Hereisanotherexampleofsimplerandomsampling.
Example:SimpleRandomSampling(2)Twohundrednurses,therapists,andsocialworkersemployedbyaMidwestcitysignedupforanadultdaycareseminar.Thecityonlyhadenoughmoneytopayfor50participants.Theseminardirectorusedtherandomassignmentfeatureinherstatisticsprogramsoftwaretoselect50names.
Tofacilitatesimplerandomsamplingfortelephonesurveys,somesurveyorsuseatechniquecalledrandomdigitdialing.Inoneofitsvariations,calledtheplus-oneapproach,adigitisaddedtothetelephonenumberthatisactuallyselected.Iftheselectedtelephonenumberis(311)459-4231,thenumbercalledis(311)459-4232.Thistechniquehelpstomakeupforthefactthatinmanyareasofthecountry,particularlyinurbanareas,peopledonotlisttheirtelephonenumbers.
MakingtheDecision
Whenshouldyouchoosesimplerandomsampling?Theadvantagesofsimplerandomsamplingarethese:
Itisthesimplestofallrandomsamplingmethods.Aidsareavailabletoassistyou.Moststatisticstextbookshaveeasy-to-usetablesfordrawingarandomsample.Analternativeistousetherandom-numberfeaturefoundinallstatisticalsoftware.
Amajordisadvantageofsimplerandomsamplingisthatitcannotbeusedtodividerespondentsintosubgroupsorstrata(e.g.,60%maleand40%female).Tomakesureyouhavetheproportionsyouneedinasample,youneedtostratify.
StratifiedRandomSamplingInsimplerandomsampling,youchooseasubsetofrespondentsatrandomfromapopulation.Instratifiedrandomsampling,youfirstsubdividethepopulationintosubgroupsorstrataandselectagivennumberorproportionofrespondentsfromeachstratumtogetasample.
Youcan,forexample,usestratifiedrandomsamplingtogetanequalrepresentationofmalesandfemales.Youdothisbydividingtheentiregroupintosubgroupsofmalesandfemalesandthenrandomlychoosingagivennumberofrespondentsfromeachsubgroup.Thismethodofsamplingcanbemoreprecisethansimplerandomsamplingbecauseithomogenizesthegroups,butonlyifyouchoosethestrataproperly.Thatis,donotsamplemenandwomenunlessyouareplanningtomakecomparisonsbetweenthem.Youshouldplantomakecomparisonsonlyifyouhavesomereasonstobelieve,inadvance,thatthosecomparisonsmightbemeaningful.Inasurveyofvoterpreference,forexample,ifyouhavesomeevidencethatmenandwomenvotedifferently,thenitmakessensetobesurethatyoursurveyincludesenoughmalesandfemalestocomparethem.Withrandomsamplingalone,youmightfindthatbychanceyouhaveasurveysamplethatconsistsmainlyofmenormainlyofwomen.
Hereishowstratifiedrandomsamplingworks.
Example:StratifiedRandomSamplingTheUniversityHealthCenterisconsideringtheadoptionofanewprogramtohelpyoungadultsloseweight.Beforechangingprograms,theadministrationcommissionedasurveytofindout,amongotherthings,howtheirnewprogramcomparedwiththecurrentoneandhowmaleandfemalestudentsofdifferentagesperformed.Previousexperiencehadsuggestedthatolderstudentsappearedtodobetterinweightreductionprograms.Therefore,thesurveyorsplannedtogetasampleofmenandwomenintwoagegroups—17to22yearsand23to28years—andcomparetheirperformanceineachoftheprograms.
About310undergraduatessignedupforthehealthcenter’sregularweightreductionprogramforthewinterseminar.Ofthe310participants,140werebetween17and22yearsold,and62oftheseweremen.Some170studentswerebetween23and28years,and80oftheseweremen.Thesurveyorsrandomlyselected40personsfromeachofthefoursubgroups(male,female,ages17–22,andages23–28)andrandomlyassignedeveryotherstudenttothenewprogram.Thesamplelookedlikethis:
MakingtheDecisionAnadvantageofstratifiedrandomsamplingisthatthesurveyorcanchooseasamplethatrepresentsthevariousgroupsandpatternsofcharacteristicsinthedesiredproportions.
Thedisadvantagesofstratifiedrandomsamplingarethatitrequiresmoreeffortthansimplerandomsamplinganditoftenneedsalargersamplesizethandoesarandomsampletoproducestatisticallymeaningfulresults.Remember,foreachstratumorsubgroup,youprobablyneed20to30personstomakemeaningfulstatisticalcomparisons.
Ifyouhavedifficultyselectingastratifiedrandomsample,keepinmindthatthesameincreaseinprecisionobtainedwithstratificationcangenerallybeproducedbyincreasingthesamplesizeofasimplerandomsample.Increasingsamplesizemaybeeasierthanimplementingastratifiedrandomsample.
SimpleRandomClusterSamplingSimplerandomclustersamplingisusedprimarilyforadministrativeconvenience,nottoimprovesamplingprecision.Sometimesrandomselectionofindividualssimplycannotbeused.Forinstance,itwouldinterrupteveryhospitalwardtochoosejustafewpatientsfromeachwardforasurvey.Sometimesrandomselectionofindividualscanbeadministrativelyimpossible.
Onesolutiontotheproblemofusingindividualsasasamplingunitistousegroupsorclustersofrespondents.
Insimplerandomsampling,yourandomlyselectasubsetofrespondentsfromallpossibleindividualswhomighttakepartinasurvey.Clustersamplingisanalogoustorandomsampling,exceptthatgroupsratherthanindividualsareassignedrandomly.Thismethodpresupposesthatthepopulationisorganizedintonaturalorpredefinedclustersorgroups.Hereishowitworks.
Example:SimpleRandomClusterSamplingTheCommunityMentalHealthCenterhas40separatefamilycounselinggroups,eachwithabout30participants.Thecenter’sdirectornoticedadeclineinattendanceinthelastyearanddecidedtotryoutanexperimentalprograminwhicheachindividualwouldbetestedandinterviewedseparatelybeforebeginningtherapy.Theprogramwasexpensiveandthecentercouldonlyaffordtofinancea150-personprogramatfirst.
Randomlyselectingindividualsfromallgroupmemberswouldhavecreatedfrictionanddisturbedtheintegrityofsomeofthegroups.Instead,asimplerandomclustersamplingplanwasused,inwhich5ofthe30-membergroups—150peoplealltogether—wererandomlyselectedtotakepartintheexperimentalprogram.Eachgroupwastreatedasacluster.Attheendofthe6months,theprogressoftheexperimentalprogramwascomparedwiththatofthetraditionalone.
MakingtheDecisionTheadvantagesofsimplerandomclustersamplingarethese:
Itcanbeusedwhenselectingindividualsrandomlyisinconvenientorunethical.Itsimplifiessurveyadministration.
Thedisadvantageofsimplerandomclustersamplingisthatitrequirescomplexstatisticalmethodstoreconcilesamplingunits(thehospital,street,school)andanalyticunits(patients,homeowners,students).Thatis,youaresamplingbycluster(suchasschools),butyouareanalyzingdatafromindividuals(suchasstudents).
Althoughintheexampleyouhave150peopleinthesurvey,youreallyhavejust5units(the5groupsof30personseach)tostudy.Whycan’tyoustudyeachofthe150personsindividually?Whenpeopleareinspecialgroups—classes,clubs,organizations,neighborhoods—theytendtosharesimilarcharacteristicsandviews.Studyingeachindividualmayberedundantbecauseonepersonmaybesimilartothenext.Youneedarelativelylargenumberofpeopleforeachclusterforthisproblemtobecomelessnoticeable.Justhowlargeanumberyouneedisastatisticalissue.
Complexsamplingstrategiesrequireanunderstandingofsamplingstatistics.
SystematicSamplingInsystematicsampling,youpickanumber,say5,andselectevery5thnameonalistthatrepresentsthepopulation.Ifalistcontains10,000namesandthesurveyorwantsasampleof1,000,heorshemustselectevery10thnameforthesample.
Supposeyouhavealistof500namesfromwhichyouwanttoselect100people.Youcanrandomlyselectanumberbetween1and10.Ifyouchosethenumber3,youwouldbeginwiththe3rdnameonthelistandcountevery5thnameafterthat.Yoursampleselectionwillresultinthe3rdname,8th,13th,andsoonuntilyouhad100names.Ifyouselectthe“start”numberatrandom,systematicsamplingresemblessimplerandomsampling.
MakingtheDecisionThereisadangerinsystematicsampling.Listsofpeoplearesometimesarrangedsothatcertainpatternscanbeuncovered;ifyouuseoneoftheselists,yoursamplewillbesubjecttoabiasimposedbythepattern.Forinstance,relativelyfewpeopleintheUnitedStateshavelastnamesbeginningwithW,X,Y,andZ,andtheymaybeunderrepresentedinasystematicsample.Hereisanotherexample.Supposeyouaresamplingclassroomssothatyoucansurveystudentstofindoutabouttheirattitudestowardschool.Sayalsothattheclassroomsarearrangedinthisorder:
Supposefurtherthatyouselecteverythirdclassstartingwith1a.Thesamplewillconsistofclassrooms1a,4a,7a,andsoon,toNa.Thesurveyofattitudestowardschoolcanbebiasedifeach“a”correspondstoalocationwithintheschoolthatfacesthelawnandisquietandthe“b”and“c”classroomsfacethesportsarenaandarenoisy.
Inconsideringtheuseofsystematicsampling,carefullyexaminethelistofpotentialrespondentsfirst.Ifyoususpectbiasbecauseoftheorderorplacementofsamplingunits(people,classrooms),useanothersamplingmethod.
ConvenienceSamplesAconveniencesampleisonethatyougetbecausepeoplewhoarewillingtocompletethesurveyarealsoavailablewhenyouneedthem.Sayyouwanttofindoutwhetherthestudenthealthservicewasanygood.Youplantointerview50students.Ifyoustandneartheclinicentranceduringtheday,youcanrecruiteachpersonwalkingin.Whenyouhaveacompletesetof50interviews,youhaveaconveniencesample.Hereareseveralsourcesofbiasinthissample:
Studentswhoarewillingtobeinterviewedmaybemoreconcernedwiththehealthservicethanthosewhorefuse.Studentswhousetheserviceatthetimeofyourinterviewmaybegoingforconvenience;sickerstudentsmayusetheserviceatnight.Studentswhotalktoyoumayhaveagripeandwanttocomplain.Studentswhotalktoyoumaybethemostsatisfiedandwanttobrag.Studentsmaywanttotalkbutmayhavenotimeatthemoment;thesemaybeworkingstudents.Perhapsworkingstudentsaredifferentfromotherstudents(older?)intheirneedsandviews.
MakingtheDecisionBecauseofbias,conveniencesamplesareunconvincingunlessyouproveotherwise.Here’showyoumightimproveonthecredibilityofyourconveniencesample:
Askrefusersandparticipantshowconcernedtheyarewiththeirhealthandcomparetheresponses.Youmayfindnodifferences;ifso,thenyourconveniencesample’sfindingsaresupported.Visittheclinicatnighttofindoutwhetherstudentsusingthehealthservicethenaredifferentfromthedaystudentsintheirhealthstatus.Youmayfindnodifferences.Again,ifso,thenyoursampleissupported.Askstudentsiftheyhaveagripe.Askstudentsabouttheirsatisfaction.Askstudentswhytheyrefusetoparticipate.Isitbecausetheypresentlydonothavethetime?Comparestudentswhoparticipatewiththosewhodonotintermsofage,gender,andsoon.Ifnodifferencesarefound,yourconveniencesample’sfindingsareeasiertosupportthanotherwise.If
differencesarefound,youneedtorethinkyoursurveymethodology.
OtherConvenienceSamplingMethodsConsiderthesecommonsituations.
Example:OtherConvenienceSamples1. Asurveyof100deansoflawschools,seniorpartnersinlargelawfirms,andjudgesisconductedtofindouttowhich
lawyerstheygotosolvetheirownlegalproblems.TheresultsarepublishedinGlobalNewsandWorldReportsunderthetitle“TheWorld’sBestLawyers.”
2. Whatmakescollegestudentsliberalorconservative?Familybackground?Regionofthecountryinwhichtheywereborn?Currentreligiouspractices?Educationalattainment?Income?AsurveyisconductedofmembersoftheYoungConservativeAssociationandtheYoungLiberalSociety.Anassessmentoftheresultsrevealsthereasonsforstudents’views.
3. Apopularmoviedownloadingservicehassubstantiallyraiseditsmonthlyfee.Howwilltheincreasedpriceaffectsubscribers?Willmostremainwiththeserviceorgosomewhereelse?Atotalof80,000peoplearecontactedtocompleteanonlinesurvey.AllpotentialrespondentshaveparticipatedpreviouslyinsurveysconductedbySurveyPanels,anInternetsurveycompany.
Inthefirstexample,the“top”lawyersmayprovidetheservicesneededbydeans,seniorlawfirmpartners,andjudges.Howapplicablearethetoplawyers’servicestotypicallegalproblems?Inthesecondexample,studentsinonlytwoorganizationsarebeingsurveyed.Canwetrustthattheyrepresentallstudents,includingthosewhobelongtoothergroupsorchoosenottojoinany?Inthethirdexample,thesurveyissentto80,000peoplewhohaveparticipatedinotheronlinesurveysconductedbySurveyPanels.AllrespondentsarewillingtocompleteanInternetsurvey.Dotheyrepresenttheaverageuserofthefilmdownloadingservice?
Youuseconveniencesamplesbecausetheyareavailableandaccesssavesthetimeandexpertiseneededtodevelopasamplingplanandimplementit.Conveniencesamplesareusefulifthesurveyoriscertainthattheresponsesapplytothetargetpopulation.Theybecomeproblematiciftheresponsescomefrompeoplewhodiffersubstantiallyinimportantways(e.g.,age,gender,Internetsavvy)fromthetargetpopulation.
Thesearestandardconveniencesamplingtechniques:
Snowballsamples
Previouslyidentifiedmembersofagroupidentifyothermembers.Forexample,youselectCEOsandaskthemtonominateothers.
Quotasamples
Thegroupisdividedintosubgroupsinspecificproportions.Thisissimilartostratifiedsampling.
Purposivesamples
Apurposivesample,alsocommonlycalledajudgmentalsample,isonethatisselectedbasedontheknowledgeofapopulationandthepurposeofthestudy.Therespondentsareselectedbecauseofsomecharacteristic.Supposeyouareinterestedinfindingoutwhypeopleshopinaparticularstore.Youmightspendadayinterviewingshoppers.Or,youmightinterviewpeoplewhodonotplantoshopinthatstore.Inthiscaseyouareusingapurposivesamplebecauseyouwantinformationonthecharacteristicsofpeoplewhodo(ordonot)shopinthestore.
Focusgroups
Tento20peoplearebroughttogethertoanswerspecificquestions.Atrainedleaderconductsthesessions.Atranscriberisusuallypresenttosummarizethediscussion.
Panels
Peoplearerecruitedtoparticipateinsurveysonanas-neededbasis.Manyonlinesurveycompaniesrecruitpanelsofrespondentsfromthemillionsofpeoplewhoparticipateintheirsurveys.Large,reputablecompaniesscreenrespondentssothattheyhaveaccesstoadiversegroupofpeople,say,forinstance,savvysmartphoneusers,workingmoms,videogameplayers,orarandomselectionofUnitedStatesadults.Usually,thesurveyordesignsthequestionnaireandspecifieswhoshouldbeincludedinthesurvey.Thesurveycompanydoestherest.Paymentisbyresponse.Forinstance,ifthesurveyorneeds300responses,andthesurveycompanycharges$3.00foreachresponse,thenthecostofthesurveyis$900.Itusuallytakesaboutaweektogetacompletesetofresponses.Toencourageresponses,surveycompaniesusuallyprovideincentivestothepanelmembers.Forexample,onepopularonlinecompanydonates$0.50toacharitychosenbytheindividualpanelistandalsogiveshimorherchancetowina$100giftcard.Thecompanydoesnotprovidedirectpaymentorcreditssoastodiscouragesurveytakersinterestedonlyincollectingrewards.
FindingtheSample:WhoIsIn?WhoIsOut?Howdoyoufindasample?Beforeyoulook,youmustdecidewhoshouldbeincluded(andexcluded)fromthesurvey.SupposeyouwanttoevaluatetheeffectivenessofCOMPULEARN,anonlineprogramtokeepemployeesuptodateonhowtomarketproductselectronically.Youdecidetosurveyasampleofusersandsetthesestandardsforrespondentstobeincludedinthesurvey:
Participateintheprogramfor1monthormoreBewillingtolearntouseFREENetsoftware(about30minutes)Be30yearsofageorolderReadEnglish
Theseeligibilitycriteriahelpyounarrowyoursample.Buttheydosomethingelseaswell:Theyrestrictyourfindingsjusttootherpeoplewhoalsomeetthestandards.Here’swhy:Afteryoursurveyiscomplete,youwillbeabletotellabouttheeffectivenessofCOMPULEARNonlyforpersonswhohavebeenintheprogramfor1monthormore,whocanusethespecifiedsoftware,whoare30yearsofageorolder,andwhoareEnglishspeaking.Apersonwhois29andmeetsallotherstandardsis,bydefinition,ineligibleandyouwillnotbeabletoapplythesurvey’sfindingsto29-year-olds.(Afterall,youchose30yearsofageasaneligibilitycriterionbecauseitpresumablymeantacriticaldemarcationpoint.)
Everytimeyoudoasurvey,youmustdecidethis:DoIwanttosampleeveryoneorjustpeopleorplaceswithcertaincharacteristics?Usuallytheansweris“justpeopleorplaceswithcertaincharacteristics(suchasschoolswith350ormorestudents).”
HowLargeShouldYourSampleBe?Somesurveystakeplacewithjustonegroup.Apollofasampleofvotersisthistypeofsurveysample.Thetrickistoselectasamplethatisrepresentativeofallvoterswhointerestyou.Othersurveysareoftwoormoregroups.Forexample,asurveycomparingthecareerplansofstudentsintheJOBSprogramwithstudentsintheCAREERprogramisthissecondsample.Whencomparingstudents’careerplans,youmaywantrepresentativesamplesofstudentsinbothgroups.Inaddition,youhavetothinkaboutthenumberofstudentsyouneedineachofthetwogroupssothatifadifferenceexistsbetweenthemyouhaveenoughpowertodetectit.
Considertheseexamples.
1. Onegroup,nointerventionorprogram:Youworkfortheparksystemandwanttofindoutwhereyoungpeopledotheirmountaincycling.
Objective:Tosurveyyoungmountainbicyclists’preferencesPopulation:Mountainbicyclists21yearsandyoungerSamplingquestion:Howmanymountainbicyclistsshouldbesurveyedtomakesurethatthesampleisafairrepresentationofmountainbikeriders21yearsandyounger?
2. Twoormoregroupsandanintervention:Youhavetwogroups.Youwanttocomparethemfordifferencesafteroneofthetwohasbeenpartofanewactivity.
Objective:ToevaluateachessprogramPopulation:ChildrenwhoareinanexperimentalchessprogramandacontrolgroupofchildrenwhoarenotSamplingquestion:Howmanychildrenhavetobeineachgrouptodetectapositivedifferenceifoneoccurs?
Whenyouthinkaboutsamplesize,youmustalsothinkaboutthestandarderror,astatisticusedtodescribesamplingerrors.Errorexistsbecause,whenyousample,youselectfromalargerpopulationandthesampletypicallydiffersfromthepopulation.Thisdifferenceisarandomresultofsampling.Youcancontrolitbutprobablynoteliminateitentirely.Ifyoudrewaninfinitenumberofsamples,themeansoraverageswouldformadistributionthatclustersaroundthetruepopulationmean.Thisdistribution,whichhasabellshape,istheso-callednormaldistribution.Ingeneral,largersamplesaremorelikelytocollectaroundthetruepopulationmeanandbeamoreaccurateestimationofthepopulationmean.
TheStandardErrorLargersamplestendtoreducesamplingerrorswhenthesamplesarerandomlyselected.Thestatisticusedtodescribesamplingerroriscalledthestandarderrorofthemean.Itisthestandarddeviationofthedistributionofsampleestimatesofmeansthatcouldbeformedifaninfinitenumberofsamplesofagivensizeweredrawn.
Trythis:Inasurveyof300randomlyselectedrespondents,70%answeryestothequestion,“Ismountain
bikingoneofyourfavoritesports?”Thesurveyorreportsthatthesamplingerroris9.2percentagepoints.Ifyouadd9.2to70%,orsubtract9.2from70%,yougetaninterval—calledaconfidenceinterval—thatrangesbetween60.8%and79.2%.Usingstandardstatisticalmethods,thismeansthatthesurveyorcanestimatewith95%confidence(the95%isaconvention)thatthe“true”proportionofmountainbicyclerswhoansweryesfallswithintheinterval.
Thetrickiskeepingtheconfidenceintervalsmall.Inpractice,largersamplesusuallyreducesamplingerrorsinrandomsamples.Butaddingtothesamplereducestheerroragreatdealmorewhenthesampleissmallthanwhenitislarge.Also,differentsamplingmethods(suchassystematicsampling)mayhavedifferenterrorratesfromrandomsampling.
Rememberthatnotallerrorscomefromsampling.Althoughyouwantalargeenoughsampletokeeptheerrorlow,youdonotwantsamplesizepressurestodistractyousothatothersourcesoferrorblightyoursurvey.Othersourcesoferrorincludeambiguouseligibilitycriteria,badlydesignedandadministeredsurveys,andpoorreturns.
StatisticalMethods:SamplingforTwoGroupsandanInterventionSupposeyouwanttocomparetwogroups.First,dividethepopulationintwo(say,randomly).Then,usestatisticalcalculations(suchastheonesthatfollow)tofindoutwhethereachgroup’ssamplesizeislargeenoughtopickupadifference,ifoneispresent.Ifthepopulationislarge,youmaywanttoselectasample(say,atrandom)andthenassignpersonstothetwogroups.Ifyouselectasampleatrandom,youhaverandomsampling.Ifyouassignpeopletogroupsatrandom,youhaverandomassignment.Ifyouselectallfiveschoolsinacityandrandomlyassignallstudentsineachoftheschoolstogroups,youhavenonrandomclustersamplingandrandomassignment.If,however,yourandomlyselectfiveschoolsinacity,assignthreetoanexperimentandtwotoacontrol,andputallstudentsintheexperimentalschoolsintheexperiment,youhaverandomclustersamplingandnonrandomassignment.
Usethefollowingchecklisttogetorevaluateasamplesizewhenyouhavetwogroupsandanintervention.
Example:SampleSizeCalculationsforSamplingTwoGroupsandanInterventionAssembleandclarifysurveyobjectivesandquestions.
Decidethesurvey’spurposes.Considerthese:
Survey1:QualityofLife
Objective:TodeterminewhetheryoungerandolderwomendifferintheirqualityoflifeaftersurgeryforbreastcancerQuestion:Doyoungerandolderwomendifferintheirqualityoflifeaftersurgeryforbreastcancer?
Survey2:AnxietyinSchool
Objective:TodeterminethenatureandtypeofanxietyassociatedwithschoolQuestion:Doboysandgirlsdifferintheiranxieties?Howdoyoungerandolderstudentscompare?
Eachobjectiveorquestioncontainsindependentanddependentvariables.Independentvariablesareusedtopredictorexplainthedependentvariables.Theyoftenconsistofthegroups(experimentalorcontrol,menorwomen)towhichrespondentsbelongortheircharacteristics(under50yearsoldand51yearsofageandolder).Takethequestion,“Doboysandgirlsdifferintheiranxieties?”Thegroupingorindependentvariableissex.
Thedependentvariablesaretheattitudes,attributes,behaviors,andknowledgethesurveyismeasuring.Instatisticalterms,theyarethevariablesforwhichestimatesaretobemadeorinferencesdrawn.Inthequestion,“Doboysandgirlsdifferintheiranxieties?”thedependentvariableisanxieties.
Identifysubgroups.
Thesubgroupsrefertothegroupswhosesurveyresultsmustbeobtainedinsufficientnumbersforaccurateconclusions.Inthetwoprevioussurveys,thesubgroupscanbeidentifiedbylookingattheindependentvariables.Survey1’ssubgroupsareolderandyoungerwomen.Survey2’ssubgroupsareolderandyoungerboysandgirls.
Identifysurveytypeanddatacollectionneeds.
Thedependentvariablestellyouthecontentofthesurvey.Forexample,Survey1’sspecificquestionswillaskrespondentsaboutvariousaspectsoftheirqualityoflife.Survey2’swillaskaboutanxiety.SupposeSurvey1isaface-to-faceinterviewandSurvey2isaself-administeredquestionnaire.
Checkthesurvey’sresourcesandschedule.
Asurveywithmanysubgroupsandmeasureswillbemorecomplexandcostlythanthosewithfewsubgroups.Considerthis:
Thenumberofsubgroupsrangesfromtwotofour.Administering,scoring,andinterpretingthesurveyforonegroupisdifficultenough;withmorethanone,thedifficultiesmount.
Calculatesamplesize.
Supposeasurveyisconcernedwithfindingoutwhetheraflexible-timeworkprogramimprovesemployeesatisfaction.Supposealsothatonesurveyobjectiveistocomparethegoalsandaspirationsofemployeesintheprogramwithothernonparticipatingemployees.Howlargeshouldeachgroupofworkersbe?Toanswerthisquestion,fiveotherquestionsmustbeanswered.
FiveQuestionstoAskWhenDeterminingSampleSize1.Whatisthenullhypothesis?Thenullhypothesis(Ho)isastatementthatnodifferenceexistsbetweentheaverageormeanscoresoftwogroups.Forexample,onenullhypothesisforthesurveyofemployeesatisfactionisthis:
Ho=Nodifferenceexistsbetweengoalsandsatisfaction(asmeasuredbyaveragesurveyscores)betweenemployeesparticipatingintheprogramandnonparticipatingemployees.
2.Whatisthedesiredlevelofsignificance(alevel)relatedtothenullhypothesisinvolvingthemeaninthepopulation(mo)?Thelevelofsignificance,whenchosenbeforethetestisperformed,iscalledthealphavalue(denotedbytheGreekletteralpha:α).Thealphagivestheprobabilityofrejectingthenullhypothesiswhenitisactuallytrue.Traditionkeepsthealphavaluesmall—.05,.01,or.001—toavoidrejectinganullhypothesiswhenitistrue(andnodifferenceexistsbetweengroupmeans).Thepvalueistheprobabilitythatanobservedresult(orresultofastatisticaltest)isduetochance(ratherthantoparticipationinaprogram).Itiscalculatedafterthestatisticaltest.Ifthepvalueislessthanalpha,thenthenullisrejected.
Whendifferencesarefoundtoexistbetweentwogroups,butinrealitytherearenodifferences,thatis
calledanalphaorTypeIerror.Whennodifferencesarefoundbetweengroups,althoughinrealitythereisadifference,thatistermedabetaorTypeIIerror.
3.Whatchanceshouldtherebeofdetectinganactualdifference?Poweristheabilitytodetectadifferenceofagivensizeifthedifferencereallyexists.Itiscalculatedas1–β(Greekletterbeta).Itisdefinedastheprobabilityofrejectingthenullhypothesiswhenitisfalseoracceptingthealternativehypothesiswhenitistrue.Youwanthighpower.
4.Whatdifferencesbetweenmeansareimportant?Thatis,whatisameaningfulm1–m2?SupposethesurveyusestheGoalsandSatisfactionScale(GASS).Thishypotheticalscalehas50points.Ifthescaleisvalid,youwillhaveaccesstopublishedscoringrulesanddatadescribingwhatthescoresmean.Askthesequestions:Arehigherscoresbetter?Howmany“points”makeapractical(educationalorclinical)difference?Theanswerstoquestionssuchasthesewillhelpyoudecidehowmuchdifferenceyouwanttodetectinyourtwogroups.
5.Whatisagoodestimateofthestandarddeviationsinthepopulation?Thestandarddeviation(s,lowercaseGreeklettersigma)isacommonmeasureofdispersionorspreadofdataaboutthemean.Ifthedistributionofvaluesorobservationsisabell-shapedornormaldistribution,then68%oftheobservationswillfallbetweenthemean±1standarddeviation;95%oftheobservationsbetween±2standarddeviations;and99%oftheobservationsbetween±3standarddeviations.Lookatthis:
Estimatesofthestandarddeviationcancomefrompreviouslydonesurveys.Checkthatthesampleusedtoderivethestandarddeviationissimilartoyourown.Ifitisnot,thestandarddeviationinyourgroupislikelytobedifferentandsoisyourgroup.
Anotherwaytoestimatethestandarddeviationinyourgroupistoconductasmallpilottestusingabout25to50people.Youcanalsohaveexpertsgiveyouestimatesonthehighestandlowestvaluesorscoresasthebasisforcalculatingthestandarddeviation.
Whatfollowsisoneformulaforcalculatingsamplesizeforcomparingthemeansfromtwoindependentgroups(suchastwogroupsofemployees).Group1isinaprogramtoimprovesatisfaction,butGroup2isnot.Thisformulaassumesthatthetwogroups’samplesizesandstandarddeviationsareequal:(zα−zβ)σ2μ1−μ2
where
μ1–μ2isthemagnitudeofthedifferencetobedetectedbetweenthetwogroupszαtheuppertailinthenormaldistribution)andzβ(thelowertail)aredefinedas
zαX–μ1σ/nandzβX–μ2σ/n
Hereisanexampleofhowtoapplytheformula.
Example:CalculatingSampleSizeinaSurveyofEmployeesinanExperimentalandControlGroupSurveysituation.Devsoft’semployeesareparticipatinginaprogramtoimprovetheirjobsatisfaction.Attheconclusionofthe3-yearprogram,participantsintheexperimentalandcontrolgroupswillbesurveyedtofindoutabouttheirgoalsandaspirations.Thehighestpossiblescoreonthesurveyis100points.TheType1oralphalevelissetat.05.Theprobabilityofdetectingatruedifferenceissetat.80.Apanelofexpertsinsatisfactionmeasuressaysthatthedifferenceinscoresbetweentheexperimentalandcontrolgroupsshouldbe10pointsormore.Previousemployeesurveysfoundastandarddeviationof15points.
Thecalculations.Forthecalculation,assumethatastandardnormaldistributionorzdistributionisappropriate.Thestandardnormalcurvehasameanof0andastandarddeviationof1.(Formoreaboutthestandardnormaldistribution,one-andtwo-tailedtests,andzvalues,seeChapter6.Actualzvaluesareobtainableinstatisticsbooksandprogrammanuals.)Thetwo-tailedzvaluerelatedtoα=.05is+1.96.Forα=.01,thetwo-tailedzvalueis2.58;forα=.10,thezvalueis1.65;andforα=.20,thezvalueis1.28.Thelowerone-tailedzvaluerelatedtoβis-.84(thecriticalvalueorzscoreseparatingthelower20%ofthezdistributionfrom80%).Applyingtheformula,wehavethefollowing:(1.96+0.84)(15)2=2(4210)2=2(17.64)orabout36
Atleast36employeesareneededineachgrouptohavean80%chanceofdetectingadifferenceinscoresof10points.
Sometimes—forpracticalreasons—youcanassembleonlyacertainnumberofpersonsforyoursurvey.Howdoyouknowwhetherthenumberislargeenoughtofinddifferences?Again,statisticalmethodstotherescue.Lookatthis.
Example:PowertoDetectDifferencesTheAlcoholRehabilitationUnithasaprogramtoreducerisksfromalcoholuse.Itsetsthisstandardofsuccess:
Attheendoftheprogram,20%oftheharmfuldrinkersinthetreatmentgroupwillreducetheirrisks,whereas10%oftheharmfuldrinkersinthecontrolgroupwilldoso.
Theunithopestobeabletosurvey150personsineachgroupregardingtheirrisks.Isthissamplelargeenough?Astatisticianiscalledintoanswerthequestion.Thestatisticianproducesthistableforcomparingpercentagechangesusingatwo-sidedtestwithα=.05.
Youinterpretthistableasfollows:
Ifwehave150persons(finalanalyticsamplesize)intheexperimentalgroupand150inthecontrolgroup,wewillhave69%powertodistinguishashiftof20%intheexperimentalgroupfromharmfultolessriskyfromashiftof10%inthecontrolgroup.
Youcanuseastatisticalapproachtodistinguishdifferenteffectsizes.Theeffectsizeisthedifferencedividedbythestandarddeviation.Lookatthis.
Example:80%PowerandEffectTheAlcoholRehabilitationUnitwillhavecompletesurveydataon150personsinanexperimentalgroupand150inacontrolgroup.Thesepersonswillhavecompletedasurveybeforetheyparticipateintheunitandimmediatelyafter.Howmuchpowerwillthissamplesizeyield?Poweristheabilitytodetectadifferenceoreffect.Putanotherway,itistheabilityofastatisticaltesttodetectanalternativehypothesisofdifference(betweengroups)ofaspecifiedsizewhenthealternativeistrue(andthenullisrejected).Astatisticianprovidesthistable.Whatdoesitmean?
Youinterpretthistableasfollows:
Ifwehave150(finalanalyticsamplesize)inourexperimentalgroupand150inourcontrolgroup,wewillbeabletodistinguisha.32effectsizebetweenthedifferenceovertimeintheexperimentalgroupversusthedifferenceovertimeinthecontrolgroup.
Becauseyouestimateeffectbydividingthemeanoraveragedifferenceinscoresbythestandarddeviation,youcansee,forexample,whethertheexperimentalgroup’smeanimprovementhadastandarddeviationof.50andthecontrolgroup’smeanimprovementhadoneof.10.Ifso,thentheeffectsizewouldbe50-10=40>.32.Inthiscase,youwillhaveatleast80%powertodetectthedifferenceof.40betweenthetreatmentandcontrolgroups.
Theprecedingdiscussionistohelpyoulearnterminologyandtoaidyouinevaluatingtheusefulnessofyoursamplingmethodsandoutcomes.Samplesizes,effects,andpowerarestatisticalterms.Unlessyouplantolearnstatistics,callinanexperttohelp.Alternatively,considerusingthesample-sizecalculatorsfoundinstatisticalprogramsorontheweb.Gotoyourfavoritesearchengineandtypein“samplesize”tofindfreesample-sizecalculators.
Thefollowingillustrateshowtowriteupandjustifysamplesize,effect,andpower.
Example:SampleSize,Effect,andPower
1.YogaforChronicBackPainTheUnitedKingdomBEAM(BackPainExerciseandManipulation)trialfoundthatachangeintheRolandMorrisDisabilityQuestionnairescoreof1.57pointswasacost-effectivedifference.Assumingastandarddeviationof4points(asintheUnitedKingdomBEAMsamplesize),thisresultsinaneffectsizeof0.39[1.57÷4].Todetectthiseffectsize,assuming80%powerand20%attrition,werequiredatotalof262participants(131pergroup).
Ifyouwouldliketolearnmoreaboutthisstudy,gotoTilbrook,H.E.,Cox,H.,Hewitt,C.E.,Kang’ombe,A.R.,Chuang,L.-H.,Jayakody,S.,etal.(2011).Yogaforchroniclowbackpain.AnnalsofInternalMedicine,155(9),569–578.
2.EffectivenessofTelephoneCounselingforSmokingCessationinParentsBasedonsimilarstudies,weexpecteda6%differencein7-daypointprevalenceabstinenceratesbetweenthetelephonecounselingconditionandthecontrolconditionat12-monthsassessment(13%versus7%,respectively).Astatisticalpowerof.80wastargeted.Hypotheseswillbetestedatatwo-sidedsignificancelevelof.05.Thecalculatedsamplesizewascorrectedforparticipantswhowillbelosttoattrition.Additionally,thesamplesizewascorrectedtoallowforsupplementaryanalysesofmediationandmoderation.
Ifyouwouldliketolearnmoreaboutthisstudy,gotoSchuck,K.,Otten,R.,Kleinjan,M.,Bricker,J.,&Engels,R.(2011).Effectivenessofproactivetelephonecounsellingforsmokingcessationinparents:Studyprotocolofarandomizedcontrolledtrial.BMCPublicHealth,11(1),732.
ResponseRateTheresponserateisthenumberofpersonswhorespond(numerator)dividedbythenumberofeligiblerespondents(denominator).If100peopleareeligibleand75completedsurveysareavailableforanalysis,theresponserateis75%.
Allsurveyshopeforahighresponserate.Nosinglerateisconsideredthestandard,however.Insomesurveys,between95%and100%isexpected;inothers,70%isadequate.
Herearesometipstoimprovetheresponserate.
TipsforImprovingResponseRateKnowyourrespondents.Makecertainthequestionsareunderstandabletothem,tothepoint,andnotinsensitivetotheirsocioculturalvalues.Usetrainedpersonneltorecruitrespondentsandconductsurveys.Setupaqualityassurancesystemformonitoringqualityandretraining.Identifyalargernumberofeligiblerespondentsthanyouneedincaseyoudonotgetthesamplesizedesired.Becarefultopayattentiontothecosts.Usesurveysonlywhenyouarefairlycertainthatrespondentsareinterestedinthetopic.Keepsurveyresponsesconfidentialoranonymous.Sendreminderstononresponders.Provideoptionsforcompletingthesurvey.Forinstance,somepeoplepreferonlinesurveys,whereasotherspreferpaper-and-pencilsurveysthattheycompleteandreturnbymailorfax.Providegiftorcashincentives.Berealisticabouttheeligibilitycriteria.Anticipateinadvancetheproportionofrespondentswhomaynotbeabletoparticipatebecauseofsurveycircumstances(suchasincorrectaddresses)orbychance(theysuddenlygetill).Tellrespondentshowyouwillusethesurvey’sdata.Bepreciseindescribinghowprivacyissafeguarded.
WeightingSurveysrarelyobtaininformationfromeveryonewithinthepopulationofinterest.Tomakeupforthediscrepancy,andtoensurethatthesurveyresultsarerepresentativeofthepopulation,surveyorsoftenassignweightsorvaluestoeachcase.Supposeyousurvey100peoplewhoarebetween15to64yearsofageandyouobtainresponsesfrom60femalesand40males.Forthatagerange,youshouldhaveresponsesfrom50malesand50females.Tocorrectforthisdifference,youassignaweighttoeachmale’sanswerssothattheycontributemore(moreweight)thandoeachfemale’s.
Thetwomostcommontypesofweightsaredesignweightsandpoststratificationornonresponseweights.Designweightsareusedtocompensateforoversamplingorundersamplingofspecificcasesorfordisproportionatestratification.Sayyouwanttocompare50membersofGroupAand50membersofGroupB.YoulearnthatGroupAhasmanypeoplewhooftenrefusetotakesurveys.YoudecidetosamplemorepeopleinGroupA(oversample)thaninGroupB,hopingthatevenwitharelativelylargerefusalrate,youwillendupwithequalnumbersofcompletedsurveysinbothgroups.IfyouoversampledGroupAatarate4timesgreaterthantherateforGroupB,thanthedesignweightforGroupAis¼,andforGroupB,itis1.
Poststratificationornonresponseweightsarealsousedtocompensateforthefactthatpersonswithcertaincharacteristicsarenotequallyaslikelytorespond.Mostgeneral-populationsurveyshavesubstantiallymorefemalethanmalerespondents(often60/40),althoughthereareoftenmoremalesinthepopulation.Becausethesurveywilloverrepresentfemalesandunderrepresentmales,youassignaweighttomakeupforthisbias.
Therearemanyrespondentcharacteristicsthatarelikelytoberelatedtothetendencytorespond,includingage,education,race/ethnicity,sex,andplaceofresidence.Itisusuallymoredifficulttocomeupwithpoststratificationthandesignweightsbecauseyouneedsupplementaryinformationaboutthepopulation.Thismeansidentifyingandanalyzingverylargedatabases(suchasacountry’scensusorauniversity’senrollmentrecords)togetestimatesofthedemographicvariablesthatinterestyou.Thesedatabasesprovideanswerstoquestionslikethefollowing:Whatistheactualproportionofmenandwomeninthepopulation?Whatistheprevalenceofthisdiseaseinyoungerversusolderpeople?Whatpercentageofpeopleliveinsingle-familyhomescomparedtootherlivingarrangements?
Example:CalculatingPoststratificationWeightsHereisahypotheticalexampleofhowpoststratificationweightsarecalculated:
MarginofErrorandConfidenceLevelIfyoureadsurveyreports,youmayhavenoticedastatementlike:Samplingerrorforthetotalsampleof1,066respondentsisplusorminus3.2percentagepointsatthe95%levelofconfidence.Samplesizesandsamplingerrorsforothersubgroupsareavailableuponrequest.
Marginoferrorandconfidencelevelaretwotermstypicallyassociatedwithpolls.Apollisasurveyofpeople’sopinionsataparticularpointintime.Mostpeoplearefamiliarwithpoliticalpollsinwhichthepublicisaskedaboutapoliticalparty’sorcandidate’sperformanceandwhotheyintendtovotefor.Marginoferrorandconfidencelevelareimportantinpollsbecauseofthesurveyor’sconcernthatthesampleofrespondentsmaynotaccuratelyrepresentthepopulation.Forinstance,ifyoupoll1,000studentstofindoutwhethertheyplantovoteinthiselectionforStudentBodyPresidentand700(70%)saytheyplantovote,youneedtobeconfidentthatthe70%islikelytorepresentthe1,000.Youwantthemtobesimilartothe1,000inage,experience,andmotivation.
Technically,themarginoferrorisastatisticexpressingtheamountofrandomsamplingerrorinasurvey’sresults.Thelargerthemarginoferror,thelessconfidenceonehasthatthepoll’sreportedresultsareclosetothe“true”figures,whicharethefiguresforthewholepopulation.
Howwellthesamplerepresentsthepopulationisgaugedbytwoimportantstatistics:thesurvey’smarginoferrorandtheconfidencelevel.Asurveymayhaveamarginoferrorofplusorminus3%ata95%levelofconfidence.Thismeansthatifthesurveyisconducted100times,thedatawillbewithin3percentagepointsaboveorbelowthepercentagereportedin95ofthe100surveys.
Supposeyouwanttofindoutifparentsaresatisfiedwithopportunitiesforinvolvementinschoolactivities.Yousurveyparentsandfindthat50%are“definitelysatisfied.”Youmightthenreportthattheresultsareaccurateplusorminus3percentagepointsatthe95%confidencelevel.Youcouldalsostatetheresultsthisway:“IfIweretoconductthesurvey100times,thepercentageofparentswhosaytheyaresatisfiedwithopportunitiesforinvolvementwillrangebetween47and53%95%ofthetime.”
Assumingyouwantaconfidencelevelof95%,considerthis:
SampleSizeandtheMarginofErrorLookingatthischart,youcanseethatthemarginoferrordecreasesasthesamplesizeincreases,butonlytoapoint.Averysmallsample,suchas50respondents,hasabouta14%marginoferrorwhileasampleof1,000hasamarginoferrorof3%.Thesizeofthepopulation(thelargergroupfromwhichthesampleistaken)doesnotmatter.Thereisapointofdiminishingvalue,however.Bydoublingthesampleto2,000,themarginoferroronlydecreasesfromplusorminus3%toplusorminus2%!
Althougha95%levelofconfidenceisastandardinbusiness,health,politics,andeducation,a90%levelmaysufficeinsomeinstances.A90%levelcanbeobtainedwithasmallersample,whichusuallytranslatesintoalessexpensivesurvey.Toobtaina3%marginoferrorata90%levelofconfidencerequiresasamplesizeofabout750.Fora95%levelofconfidence,theneededsamplesizeisabout1,000.
Determiningthemarginoferroratvariouslevelsofconfidenceisrelativelyeasy.Gotothewebforthemarginoferrorforvariousconfidencelevels.Youcanask:Whatisthemarginoferrorfora90%(95%)confidencelevel?Youcanalsogotothewebtofindsitesforcalculatingthesamplesizeneededtoobtainaspecificmarginoferror.Ifthesurveyorisinterestedinamarginoferrorof3%,forinstance,thecalculatorwillsaywhatthesamplesizeshouldbe.Gotothewebandask:Whatisthesamplesizeneededfora90%(95%)confidencelevelanda3%marginoferror?
SummingUpThreeofthemostcommonlyusedrandomsamplingmethodsarethefollowing:1. Simplerandomsampling2. Stratifiedrandomsampling3. Simplerandomclustersampling
Randomsamplinggiveseveryonewhoiseligibletoparticipateinthesurveyafairchanceofselection.Youcansampleindividualsorlargerunitssuchasschools,offices,andhospitals.Theselargersamplingunitscontainclusters(ofstudents,employees,nurses,physicians,patients),sothetechniqueiscalledclustersampling.Nonrandomsamplesareofteneasiertoassemblethanrandomsamples.Butgainsineasecanbemetwithlossesingeneralizability.
Withsystematicsamples,everynth(e.g.,every5thor500th)unit(individuals,schools,factories)isselectedfromalistofunits.Ifnisrandomlyselected,systematicsamplingbecomeslikerandomsampling.Inconveniencesampling,youselecteveryonewhoisavailablewhenyouneedthemiftheymeetthecriteriaforyoursurvey(rightageorreadinglevel,votedinthelastelection,havelivedinthecommunityforatleast1year,etc.).Othernonprobabilitysamplingmethodsincludesnowballandquotasampling.Othernonrandomsamplingstrategiesincludethefollowing:
Snowballsamples.Previouslyidentifiedmembersofagroupidentifyothermembers.Forexample,youselectCEOsandaskthemtonominateothers.Quotasamples.Thegroupisdividedintosubgroupsinspecificproportions.Thisissimilartostratifiedsampling.Focusgroups.Tento20peoplearebroughttogethertoanswerspecificquestions.Atrainedleaderconductsthesessions.Atranscriberisusuallypresenttosummarizethediscussion.Panels.Respondentsagreetocompletesurveysiftheyarecontacted.Therespondentsareprescreenedfortheirdemographicandothercharacteristics(e.g.,frequentInternetshoppers,peoplewhoneveruseTwitter).
Largersamplestendtoreducesamplingerrorswhenthesamplesarerandomlyselected.Thestatisticusedtodescribesamplingerroriscalledthestandarderrorofthemean.Itisthestandarddeviationofthedistributionofsampleestimatesofmeansthatcouldbeformedifaninfinitenumberofsamplesofagivensizeweredrawn.Surveysrarelyobtaininformationfromeveryonewithinthepopulationofinterest.Tomakeupforloss,andtoensurethatthesurveyresultsarerepresentativeofthepopulation,surveyorsassignweightsorvaluestoeachcase.Therearetwotypes:designandpoststratification,ornonresponse,weights.Themarginoferrorisastatisticexpressingtheamountofrandomsamplingerrorinasurvey’sresults.Thelargerthemarginoferror,thelessconfidenceyoushouldhavethatthepoll’sreportedresultsareclosetothe“true”figures,whicharethefiguresforthewholepopulation.
ThinkAboutThis1. Locatethesurveysinthearticlesbelowandthendothefollowing:
1. Discussthetypeofsamplingthatthesurveyorsuse.2. Describethesurvey’seligibilitycriteriabylistingthecharacteristicsofrespondentswhoare
includedandexcluded.2. Findanythreesurveysthatarepartoflargerstudiesanddiscusstheirsamplingplans.Arethe
samplingplansdiscussedinadequatedetail?Thatis,doyouthinkyouhaveenoughinformationaboutthesamplingstrategyandsamplingsizetoredothesurveyifnecessary?Whatadditionalinformationwouldyouneed?
3. Listatleastfivetipsforimprovingasurvey’sresponserate.4. Explainthefollowingtermsastheyareusedwhentalkingaboutsurveys:standarderror,marginoferror,confidencelevel,confidenceinterval,power,effectsize,levelofsignificance,andnormaldistribution.
5. InSeptember2014,thePewResearchInternetProjectsponsoredasurveytostudytechnology’seffectsontheAmericanworkplace.Gotothesite(http://www.pewinternet.org/2014/12/30/technologys-impact-on-workers/?
utm_source=Pew+Research+Center&utm_campaign=5db76b36bf-Work_Productivity_123014&utm_medium=email&utm_term=0_3e953b9b70-5db76b36bf-399429165)andanswerthesequestions:1. Howwasthesurveyadministered?2. Whichsamplingmethodwasused?3. Whatweretheeligibilityrequirements?4. Describethefinalsampleintermsofitssizeanddemographics.5. Whoconductedthesurvey?Whataretheirqualifications?6. Whatisthemarginoferror(samplingerror)?Whatisthelevelofconfidence?7. Describethesurvey’smainfindings.
ReferencesHolman,E.A.,Silver,R.C.,Poulin,M.,Andersen,J.,Gil-Rivas,V.,&McIntosh,D.N.(2008).Terrorism,acutestress,andcardiovascularhealth:A3-yearnationalstudyfollowingtheSeptember11thattacks.ArchivesofGeneralPsychiatry,65(1),73–80.
O’Connor,T.M.,Yang,S.-J.,&Nicklas,T.A.(2006).Beverageintakeamongpreschoolchildrenanditseffectonweightstatus.Pediatrics,118(4),2005–2348.
Schuck,K.,Otten,R.,Kleinjan,M.,Bricker,J.,&Engels,R.(2011).Effectivenessofproactivetelephonecounsellingforsmokingcessationinparents:Studyprotocolofarandomizedcontrolledtrial.BMCPublicHealth,11(1),732.
Skoffer,B.(2007).Lowbackpainin15-to16-year-oldchildreninrelationtoschoolfurnitureandcarryingoftheschoolbag.Spine,32(24),E713–E717.
Tilbrook,H.E.,Cox,H.,Hewitt,C.E.,Kang’ombe,A.R.,Chuang,L.-H.,Jayakody,S.,etal.(2011).Yogaforchroniclowbackpain.AnnalsofInternalMedicine,155(9),569–578.
Wehr-Flowers,E.(2006).Differencesbetweenmaleandfemalestudents’confidence,anxiety,andattitudetowardlearningjazzimprovisation.JournalofResearchinMusicEducation,54(4),337–349.
5SURVEYDESIGNEnvironmentalControl
OverviewAsurveycanbegiventojustonegrouportomanygroupsofpeopleonceorseveraltimes.Forexample,youcansurveypreschoolchildreninoneschooljustbeforeandimmediatelyaftertheyparticipateinanewmusicprogram,oryoucansurveypreschoolchildrenin10schoolsjustbeforetheyparticipateintheprogramandevery2yearsafteruntiltheycompletehighschool.Asurvey’sdesignreferstoitsfrequency(onetimeormoreoften),sequence(justbeforeandimmediatelyafteranewprogram),thenumberofgroupsinvolved(onegroupconsistingofallstudentsinaschooldistrictortwoormoregroups,oneofwhichispartofanexperimentornewprogram),andhowthegroupswereassigned(atrandomornotatrandom).
Somedesignsareobservationalandothersareexperimental.Across-sectionalsurveydesignprovidesaportraitorobservationofthingsastheyareatasinglepointintime.Apollofvoters’preferences1monthbeforeanelectionandasurveyoftheincomeandageofpeoplewhovoteinaparticularelectionbothusecross-sectionaldesigns.
Longitudinalsurveysareobservationaldesignsthatareusedtofindoutaboutchange.Ifyoutakeasampleofchildrenwhoweresixthgradersin2015andfollowthemwithasurveyeveryyearforthenext5years,youareusingalongitudinaldesign.Othersurveydesignsinvolvecomparisonsbetweenoramonggroups.Theseareexperimentaldesigns.Ifrespondentsareassignedtogroupsonarandombasis,andeveryrespondenthasanequalchanceofassignmenttoagroup,youhavearandomizedcontrolledtrialortrueexperiment.Nonrandomizedcontrolledtrialsorquasi-experimentaldesignsarecomparisongroupdesignsinwhichtherespondentsareassignedtogroupsinanon-randomway.
Validsurveyfindingscomefromvalidsurveys(measurementvalidity)andvalidstudydesigns(internalandexternalvalidity;minimalbias).Validdesignsaredependentonfactorssuchastheobjectivityofthesurveyparticipants’selection,thestabilityovertimeoftheirresponses,therespondents’psychologicalandphysicalgrowth,andeventsintheoutsideworld.Thesefactors,ifnotunderstoodandaddressed,threatentheinternalvalidityofthesurvey’sdesignandbiastheresults.Whenthefactorsaffecttheapplicationofthefindingstoothersettingsandotherrespondents,thentheexternalvalidityorgeneralizabilityofthesurvey’sfindingsisthreatened.
WhichDesignsAreAvailable?Surveysmaytakeplaceontheirown.Acompanypollofitsemployees’satisfactionisanexampleofastand-alonesurvey.Butsurveysareoftenincludedaspartofalargerstudy,andthatstudymayincludeotherdatacollectionmethodsaswell.Supposeacompanyisinterestedinimprovingemployeeperformance.Thecompanyinstitutessomenewprogramsandrandomlyassignssomeemployeestothemwhiletheotherscontinueasusual.After6months,thecompanyinterviewsbothgroupsofemployeestofindoutabouttheirsatisfactionwithbothprograms,observesthematwork,andreviewstheiron-the-jobperformance.Onlyoneofthemeasures—theinterviews—isactuallyasurvey.Nevertheless,thesurveyorneedstobesurethatthedesignofthestudytoevaluatethenewprogramsdoesjusticetotheneedsofthesurvey.
Surveydatacanbeusedtodescribethestatusofthings,showchange,andmakecomparisons.Thesurvey’sdesignreferstothewayinwhichthe“environment”iscontrolledororganized.Theenvironmentreferstothenumberandcharacteristicsofrespondentsandhowoftenandwhentheywillbesurveyed.Themoreenvironmentalcontrolyouhave,themoreaccurateyourresultswillbe.
Theenvironmentalvariablesoverwhichsurveyorshavethemostcontrolareasfollows:(a)whenthesurveyistobegiven(e.g.,immediatelyaftergraduation),(b)howoften(e.g.,twice:immediatelyaftergraduationand12monthsafterthat),(c)thesamplesize(e.g.,10or1,000graduates),(d)thenumberof
groups(e.g.,justone,allgraduates;ormorethanone,graduatesofpublicschoolsandgraduatesofcharterschools),and(e)howthegroupswereassigned(atrandomornotrandomly).
LookatthesesurveysplannedbytheHave-A-HeartAssociation.
Example:SurveysWithDifferingDesigns1. TheHave-A-HeartAssociationoffers10educationalprogramstopeopleinthecommunity.InJune,itisconductinga
surveyofarandomsampleofpeopletofindoutanddescribewhichofthe10programstheyselectandwhy.2. TheHave-A-HeartAssociationwantstoknowwhetherpeoplewhoparticipateinitseducationalprogramsgainknowledge
abouthowtolessentheirriskofaheartattack.Theassociationconductedsurveyswith300volunteerparticipantsfromtheclassof2015.Thesurveyswereconductedonceayearfor3yearsandtheresultswerecomparedfromtimetotimetoseewhetherknowledgegainsendure.
3. TheHave-A-HeartAssociationhasconductedanevaluationoftheeffectivenessoftwocompetingprogramsthataimtoreducerisksforheartdiseasethrougheducation.Participantswereselectedatrandomfromtheassociation’sdatabase.Theywerethenrandomlyassignedtoparticipateinoneofthetwoprograms.Participants’risksforheartdiseaseweresurveyedbeforeprogramparticipationand6and12monthsafter.
ThefirstHave-A-HeartAssociationsurveyistobeconductedinJunetodescribetheprogramsthatthecommunityselects.Thedesign,inwhichdataarecollectedatonepointintime,iscalledcross-sectional.
ThesecondHave-A-HeartAssociationsurveycallsforcollectinginformationovera3-yearperiodfromasingleclass.Thisisalongitudinal,time-series,orcohortdesign.
ThethirdHave-A-HeartAssociationsurveyisusedaspartofanevaluationoftheeffectivenessoftwoprograms.Itusesacomparisongroupdesign.Thedesignisexperimental.
Table5.1showstherelationshipamongthepurposes,samplinganddesignconcerns,results,andtypeofdesignforthethreeHave-A-HeartAssociationsurveys.
Cross-SectionalSurveyDesignsWiththisdesign,dataarecollectedatasinglepointintime.Thinkofacross-sectionalsurveyasasnapshotofagroupofpeopleororganizations.
SupposetheHave-A-HeartAssociationwantstoknowwhichofitseducationalprogramsthecommunityprefers.Considerthisquestionanditsanswer:
Example:Cross-SectionalDesignQuestion:Ifonlyoneprogramwerepossible,whichwouldyouchoose?Sample:Across-sectionof500people,randomlyselected,whoattendedaneducationprogramthisyearDesign:Cross-sectionalMethod:TelephoneinterviewsAnswer:DineOutwins.TheevidenceisseeninTables5.2and5.3.
Assumingthatasampleofparticipantshasbeenwiselychosenbyarandomsamplingtechniqueandtherightquestionshavebeenasked,thetablesintheexamplerevealthatDineOutisthewinner.Thisiswhy:
1. Regardlessofsexorage,DineOutisahead.2. MorementhanwomenpreferFeelFitandEmergencyCare.ButwhenitcomestoDineOut,menand
womenhavenearlythesamepreference.3. Peopleover65yearsofagepreferFeelFit,buttherearenotasmanyofthemaspeopleintheother
twocategories.
Ofcourse,youmightwanttouseonlyTable5.2orjustTable5.3tomakeapointaboutyoursurvey.Youcanalsocombinethemintoonelargetable.Butacarefullyplannedcross-sectionaldesignwillgiveyouavarietyofwaysforanalyzingandpresentingyoursurveydata.
Withthisprogrampreferencedata,forexample,youmightalsohaveconsideredprofession:Howdopeopleinbusinessandtheprofessionscompare?Doesretirementmakeadifference?Oryoumighthaveconsideredresidence:Howdopeopleinonepartofthecitycomparewithpeopleinsomeotherpart?
MakingtheDecisionCross-sectionalsurveyshaveimportantuses.Theydescribethingsastheyaresothatpeoplecanplanforchange.Iftheyareunhappywiththepicturethatacross-sectionalsurveyreveals,theycanchangeit.
Cross-sectionalsurveysarealsorelativelyeasytodo.Theyarelimited,however,inthatifthingschangerapidly,thesurveyinformationwillpossiblybecomeoutdated.
LongitudinalSurveysLongitudinalsurveydesignreferstothecollectionofsurveydataovertime.Theyaresometimescalledtime-seriesdesigns.Acohortdesignreferstosurveyingaparticulargroup(e.g.,sixthgraders)morethanjustonce(say,onceayearfor3years).LookatthisexamplewithparticipantsinaprogramsponsoredbytheHave-A-HeartAssociation.
Example:LongitudinalSurveys:CohortDesignQuestion:Whatdoparticipantsknowaboutheartdisease?Sample:Avolunteersampleof300participantswhoattendedDineOut,whichaimstoteachpeoplethecausesofheartdisease,goodnutrition,andhowtopreventheartdiseaseevenwhendiningoutMethod:Onlineandmailedself-administeredquestionnairesAnswer:Inall3yearsofthesurvey,participantsconsistentlyknowlittleaboutdiseaseprevention.By2009and2010,theyhavesomeknowledgeaboutnutritionand,by2011,theyknowmuchaboutthecausesofheartdisease.
Proofisdisplayedinthefollowingtable.
MakingtheDecisionCohortsprovidethesurveyorwithdataonwhethereffects(say,ofaprogramlikeDineOut)lastovertime.Theproblemwithcohortdesignsisthat,unlesstherespondentsaremotivated,theymaydropoutandtheremaindermaynotbetypicaloftheentiregroup.Also,thegroupmaybeatypicalinthatitiswillingandhasthetimetoparticipateintheprogramandcompletethesurveys.
ExperimentalSurveyDesignsWiththesedesigns,peoplearedividedintotwoormoregroupsandtheirsurveyresultsarecompared.Theclassicalcomparisongroupdesigncontrastsan“experimental”groupwitha“placebo”or“usualcare”group.
Experimentaldesignsinsomefields(education,psychology)aresometimesdividedintoquasi-andtrueexperimentaldesigns.Inquasi-experimentaldesigns,creationandassignmenttogroupsareusuallydeliberateandnotrandom.Intrueexperimentaldesigns(randomizedcontrolledtrials),anindividual’sassignmenttogroupsisachanceresult.Trueexperimentsarethemorepowerfulofthetwobecause,ifproperlyimplemented,youcanconfidentlysayProgram1causedOutcome1.
Example:ANonrandomizedControlledTrialorQuasi-ExperimentalDesignYouhavebeenaskedtoevaluatetwoprogramsfortheelderly.Eligibleparticipantsareassignedtotheprogramsonafirst-come,first-servedbasis,resultingin60peopleinProgram1and59inProgram2.Oneoftheissuesaddressedbythesurveyiswhethertheparticipantsaresatisfiedwiththestaffoftheirprogram.Toanswerthisquestion,youaskparticipantsinbothgroupstocompleteaquestionnaireattheendof3monthsoftreatment.Thedesignforthissurveylookslikethis:
Howvalidisthequasi-experimentalcomparisongroupdesignusedforthesurveyoftheattitudesoftheelderlyintwoprogramstowardtheirstaff?Considerthesepossibilities:
1. Participantsinthetwogroupsmaybedifferentfromoneanotheratthebeginningoftheprogram.Forexample,olderpersonsmaychooseoneprogramovertheother.
2. Participantswhotrulydislikethestaffinoneoftheprogramsmayhavedroppedoutoftheprograms.
Example:ANonrandomizedControlledTrialWithaLongitudinalDesignAnotherquestionposedfortheevaluationofthetwoprogramsfortheelderlyiswhetherparticipantshavelearnedaboutcommonlyprescribeddrugs.Toanswerthisquestion,participantshavebeeninterviewedatthreetimes:atthebeginning,attheendofthefirstmonth,andattheendofthefirstyear.Thissurveydesignstrategycanbedepictedasfollows:
Thevalidityofthisdesignmaybethreatenedifpersonswithserioushealthproblemsarebychancemoreoftenassignedtooneprogramovertheotherorbyadifferentdropoutratefromonegroupcomparedwiththeother.
Example:ARandomizedControlledTrialorTrueExperiment
Thegovernmentcommissionedasurveytodeterminewhichofthreeprogramsfortheelderlywasthemosteffective.Amongtheconcernswasthecostoftheprograms.AcomparisongroupdesignwasusedinwhichpeopleattheFredaSmithCenterwererandomlyassignedtooneofthreeprogramsandtheaveragecostsofthethreeprogramsovera12-monthperiodwerecompared.
Thisdesignisrelativelystrong.Becausepeoplewererandomlyassignedtoeachprogram,anysourcesofchangethatmightcompetewiththeprogram’simpactwouldaffectallthreegroupsequally.However,rememberthat,althoughpeoplewereassignedrandomlytotheprogramswithintheFredaSmithCenter,othercentersmaydifferandthereforethefindingsfromthesurveymaynotbeapplicabletootherplaces.
Thistrueexperimentalandlongitudinaldesignisamongthemostsophisticatedandwillenableyoutomakesoundinferences.
Example:ARandomizedControlledTrialWithaLongitudinalDesignPrograms1and2intheFredaSmithCenterprovedtobeequallycost-effective.Thegovernmentthencommissionedastudytodeterminewhichprogramwasconsideredbyparticipantstodeliverbettermedicalcare.Tomakethedetermination,acomparisongroupdesignwasselectedinwhichcarewasassessed(usingascaleof10=excellentcare;1=poorcare)fromthebeginningto
theendofthe5-yearprogramandcomparedamongpeopleinPrograms1and2.The(hypothetical)surveythatwasusediscalledTheABYQualityofCareReviewSystemanditisasurveyofdoctors,nurses,andpatients.Thedesignisdepictedbythefollowingdiagram.
FactorialDesigns:SpecialCasesFactorialdesignsenableresearcherstoevaluatedifferentmethodsforofferinganewprogram,treatment,orpracticetoseewhichcombinationworksbest.Supposeyouareworkingonastudytofindoutiftheresponseratetoonlinesurveyscanbeimprovedbynotifyingprospectiverespondersinadvancebyemailand/orpleadingwiththemtorespond.Yourecommendusingatwo-by-two(2×2)factorialdesigninwhichsomerespondentsarenotifiedaboutthesurveyinadvancebyemail,whilesomearenot.Also,somepeoplearepleadedwithtorespond,whileothersarenot.Thefactors(theyarealsoindependentvariables)arepleading(Factor1)andnotifying(Factor2).Eachfactorhastwolevels:pleadversusdon’tpleadandnotifyinadvanceversusdon’tnotifyinadvance.
Ina2×2design,therearefourstudygroups:(1)prenotificationemailandpleadinginvitationemail,(2)prenotificationemailandnonpleadinginvitation,(3)noprenotificationemailandpleadinginvitation,and(4)noprenotificationandnonpleadinginvitation.Inthediagramabove,theemptycellsareplaceholdersforthenumberofrespondentsineachcategory(e.g.,thenumberinthegroupplead×notifyinadvancecomparedtothenumberinthegroupplead×don’tnotifyinadvance).Iftherespondentsarerandomlyassignedtoeachofthegroups,youwillhavearandomizedcontrolledtrialortrueexperiment.
Withthisdesign,theresearcherscanstudymaineffects(pleadversusdon’tplead)orinteractiveeffects(prenotificationandpleading).Theoutcomeinthisstudyisalwaystheresponserate.
Factorialdesignsmayincludemanyfactorsandmanylevels.Itisthenumberoflevelsthatdescribesthenameofthedesign.Forinstance,inastudyofpsychotherapyversusbehaviormodificationinoutpatient,inpatient,anddaytreatmentsettings,therearetwofactors(treatmentandsetting),withonefactorhavingtwolevels(psychotherapyversusbehaviormodification)andonehavingthreelevels(inpatient,daytreatment,andoutpatient).Thisdesignisa2×3factorialdesign.
Surveyparticipantsinfactorialdesignsmaybeselectedeitheratrandomorbyconvenience.
MakingtheDecisionRandomizedcontrolledtrialsarecharacterizedbyrandomselectionandassignment.Manyresearchersassertthatthebestrandomizedcontrolledtrialsarethoseinwhichparticipantsandinvestigatorsdonotknowtheirgroupassignment.Theyare“blinded.”Implementingrandomizedcontrolledtrialsmaybedifficultintherealworldofsurveysandsocialresearch.Theyarerelativelycostlyandrequiretimeforimplementationandanalysis.Nevertheless,ifyouwanttoknowwhetheranexperimenthasresults,therandomizedcontrolledtrialisthewaytogo.
Nonrandomizeddesignsareeasiertoimplement.However,becausegroupsarenotrandomlyconstituted,theirinitialdifferencesmayberesponsiblefororbecomeconfoundedwiththeoutcome.Thesurveyormustdoeverythingpossibletopreventtheseinitialdifferences,saybymatchingthegroupsonfactorsor
variablesthatmayinterferewithoutcomes.Thevariablesmayincludeeducation,motivation,income,andsoon.Statisticalmethodsareavailablethathelpcontrolthesedifferences.
OtherSurveyDesigns:NormativeandCaseControlTwolesser-knownsurveydesignsarethenormative(sometimescalledhistorical)andcasecontrol.Bothoffersomecontroloverthesurvey’senvironmentbymakinguseofspecialcomparisongroups.
NormativeSurveyDesignAnormisastandardforcomparinggroups.Normsthatarehistoricalcomefromexistingdataorlargedatabasessuchasthosemaintainedingovernmentregistries.Forinstance,normsareusedwhenaparticularschool’sreadingscoresarecomparedtothenationalaveragesfoundinacountry’seducationdatabase.Lookatthese.
Example:NormativeDesignParticipantsinLosAngeles’YouthProgramusedtheEFFICACYSurvey,aninstrumenttomeasureself-efficacythathadbeenvalidatedonanationalsampleof5,000people.ThenationalsamplewasusedasanormbecausetherewasnoreasontobelievethattheL.A.groupwoulddifferfromtherestofthenationinitsself-efficacy.YouthProgramscoreswerethuscomparedtothoseobtainedbythenationalsample.
Example:NormativeDesign—ComparisontoaModelArephysiciansinanewacademicmedicalcenterassatisfiedwiththeirworkasphysiciansinanolderone?Thenewcenterhasthesamemissionastheolderone.Toanswerthequestion,physiciansinbothcentersaresurveyedandthenewcenteriscomparedwiththemodeloldercenter.
MakingtheDecisionNormativesurveydesignscanbelessexpensiveandtime-consumingthanothercomparisondesignsbecausethesurveyordoesnothavetofindorcreateagrouptoactasthecontrol.Thesedesignsdependontheavailabilityofappropriatedatabases.Remember,yourgroupandthe“normal”onemayactuallydifferinimportantrespects,resultinginless-than-validfindings.SupposetheparticipantsintheYouthProgramofLosAngeleswereyoungerthanthenationalsample,forexample.Ifageandself-efficacywererelated,withmaturityassociatedwithbettersurveyscores,thentheYouthProgramwouldhavetoworkhardertobeasuccess.Ifyouusenormativedesigns,bepreparedtodefendyournormorstandardforcomparison.
CaseControlDesignAcasecontroldesignisoneinwhichgroupsofindividualsareselectedbecausetheyhave(thecase)ordonothave(thecontrol)theconditionbeingstudiedandthegroupsarecomparedwithrespecttoexistingorpasthealthcareconditions,attitudes,habits,beliefs,ordemographicfactorsjudgedtobeofrelevance
tothecausesofthecondition.
Casecontroldesignsaregenerallyusedbyresearcherswhoaretestingaspecifichypothesis—forexample,thataconnectionexistsbetweenlungcancerandcigarettesmokinghabits.
Acasecontroldesignneedstwogroups:acaseandthecontrol.Ideally,thetwogroupswilldifferonlyinthatthecasehasthecharacteristicsorconditionbeingstudiedandthecontroldoesnot.
Mostoftencasecontroldesignsmeanselectingacontrolthatislikethecaseinwaysthatarestronglysuspectedtoaffectthecondition.Ifyouhaveevidencethatgenderaffectsfrequencyofheadaches,thenastudyofpeoplewithheadachesshouldhavecasesandcontrolswithsimilarproportionsofmalesandfemales.Casecontroldesignsaremostoftenobservational.
MakingtheDecisionThemajorweaknessofthecasecontroldesignisthatthetwosurveygroupsmaynotbealikeatallnomatterhowselectedormatchedbecauseofimpossibilityincontrollingforallcharacteristicsthatmayaffectthecondition.Somematchingcriteriamightbeincorrect,forexample,orothersmaybeexcluded.Hereishowasurveycouldbeusedwithacasecontroldesign.
Example:CaseControlDesignTheUniversityMedicalClinicrandomlydivided10,000patientsbetween21and55yearsofageintotwogroups:thosewhohadhadamigraineinthepast12monthsandthosewhohadnot.Theyusedastratifiedrandomsamplingtechniquetoselect100peoplefromeachgroup,halfofwhomwerefemale.Asurveywasconductedtofindoutaboutthefollowinginrelationtoeachgroup:
TypicaldailyactivitiesPotentialsourcesofstressFamilyhistoryandbackgroundMedicalhistoryDiet
SurveyDesignValidityValidsurveyinformationcomesfromreliableandvalidsurveyinstruments(measurementvalidity)andfromthecontextinwhichthesurveytakesplace(designvalidity).Thetwotypesofvaliditydifferinthatmeasurementvalidityreferstothecharacteristicsofthesurveyinstrument,whereasdesignvalidityreferstotheselectionofsurveyrespondentsandthetimingandfrequencyofadministration..Measurementvalidityasksthefollowing:Isthesurvey’scontentcomprehensive?Doesgoodperformancepredictsuccesslateron?Doesameaningfuldifferenceexistbetweenhighandlowscorers?Designvalidityasksthis:Howoftenwillthesurveybegiven?When?Howwillrespondentsbeselected?
Becausesurveyorsworkinanimperfectworld,theircarefullydesignedstudiesareoftenthreatenedbyfactorsoverwhichtheyhavenocontrol.Somepeoplesaytheywilltakethesurvey,buttheydon’tortheydropoutmidway.Thesurveyoraimsforatrueexperimentbutdoesn’tquitemanageit.Thelistgoeson,andthethreatshavetheeffectofbiasingorprejudicingtheresults.Thesethreatsorbiaseslimitthe
survey’saccuracy.
CampbellandStanley(1963)identifiedandsummarizedthreatstoresearchdesigns.Theydividedthethreatsintothosethatunfavorablyaffecttheinternalvalidityofthedesignandthosethatunfavorablyaffecttheexternalvalidity.Adesignhasinternalvalidityifastudy’soutcome(say,animprovementinattitudetowardschool)isduetoorcausedbythevariablesthatarecontrolledormanipulatedinthestudy(suchasanexperimentalprogram).Itisexternallyvalidifthefindingsapplytootherpeopleandothersettings.
InternalValidityHereisareviewofthethreatstointernalvaliditythatCampbellandStanleyassembled.
ExternalValidityThreatstoexternalvalidityaremostoftentheconsequenceofthewayinwhichrespondentsareselectedandassignedtostudygroups.Forexample,respondentsinanexperimentalsituationmayanswerquestionsatypicallybecausetheyknowtheyareinaspecialexperiment.ThisiscalledtheHawthorneeffect.Externalvalidityisalsothreatenedwheneverrespondentsaretested,surveyed,orobserved.Theymaybecomealerttothekindsofbehaviorsthatareexpectedorfavored.HerearesomeofthethreatstoexternalvaliditythatCampbellandStanleyidentified.
Surveys,ResearchDesign,andInternalandExternalValidityWhenyouchooseandimplementasurveydesign,youalmostalwaysautomaticallyinheritthreatstointernalvalidity.Theoneexceptionistherandomizedcontrolledtrialsinwhichrespondentsarerandomlyselectedandassignedwithoutknowingwhethertheyareinanexperimentaloracontrolgroup(thedouble-blindrandomized-controlledtrial).Perfectrandomizedtrialsrequirevastresources.Althoughtheyareconsideredthegoldstandardformanyresearchstudies,theyaredifficultandcostlytoimplement,soyouprobablywillnotfindmany.
Lookatthisexampleforacross-sectionalsurvey,oneofthemosttypicalstand-alonesurveys.
Example:Cross-SectionalSurveyandThreatstoValidityAsurveyorwasaskedtodescribetheamountoftimephysiciansspendwiththeirolderpatients,notcountingofficevisits.Thesurveyorsurveyed16physiciansinanacademicgeriatricmedicalsettingduringthreerandomlysampled1-weekperiods.Usingastructuredform(aself-administeredquestionnaire),theresearchersfoundthatphysiciansspendaconsiderableamountoftimeprovidingcareoutsideofofficevisitsforpatientswithchronicillnesses.Theinvestigatorspointoutthatthestudy’svaliditymayhavebeenlimitedbythefactthatdatawereself-reported(measurementvalidity)andwereobtainedfromanacademicgeriatricmedicine(internalvalidity:selection).
Ifyouwouldliketolearnmoreaboutthissurvey,seeFarber,J.,Siu,A.,&Bloom,P.(2007).Howmuchtimedophysiciansspendprovidingcareoutsideofofficevisits?AnnalsofInternalMedicine,147(10),693–698.
Example:LongitudinalDesignandThreatstoValidityTheGreatSmokyMountainsStudyincludesarepresentativesampleofchildrenandadolescentsages9to16yearsfromthesoutheasternUnitedStates.Childrenandtheirparentsinthestudywereinterviewedonmultipleoccasionsfrom1993to2000aboutthesubjects’recentpsychiatricandsuicidalhistory.Thepurposesoftheinterviewsweretocharacterizerisksforsuicidalityassociatedwithanxiety,depressive,disruptivebehavior,andsubstanceusedisordersandtotestwhethertherearecriticalcombinationsofdisordersthatdiscriminateat-riskyouthindependentofseverityofpsychopathology.
Interviewerswereresidentsofthestudyareaandhadatleastabachelor’s-leveldegree.Theyreceived1monthoftrainingandconstantmonitoringforqualitycontrol.Interviewsusuallytookplaceathome.Theinvestigatorsfoundthattheseverityofsymptom-relatedimpairmentandtotalsymptomloadexplainedmostoftheriskforsuicide.Onlydepressionplusanxietydisorderdiscriminatedat-riskyouthindependentofseverityofpsychopathology.
TheinvestigatorsnotethatthesamplewasderivedfromarelativelysmallgeographicareainthesoutheasternUnitedStatesand
maynotberepresentativeofotherareas(internalvalidity:selection).Also,theydidnotmedicallyverifyreportedsuicideattempts(measurementvalidity:concurrentvalidity).
Ifyouwouldliketolearnmoreaboutthisstudy,seeFoley,D.L.,Goldston,D.B.,Costello,E.J.,&Angold,A.(2006).Proximalpsychiatricriskfactorsforsuicidalityinyouth:TheGreatSmokyMountainsStudy.ArchivesofGeneralPsychiatry,63(9),1017–1024.
Example:RandomizedControlledTrialsandValidityPeoplewhoareoncevictimsofinterpersonalviolenceoftenbecomevictimsagain,withthefrequencyofrecurrencerangingfrom6%to44%.Thisstudyhypothesizedthatthishighrecurrencerateamongyouthcanbereducediftheirunmetpsychosocialneedsaremet.Consentingyouthandtheirguardianswererandomlyassignedtoanexperimentalorcontrolgroup.Anumberofsurveyquestionnairesandscalesweredeveloped,includingParentingPracticesQuestionnaire,FamilyRelationsScale,andTheChildSelf-ReportofDelinquency.
Theinvestigatorsnotemanythreatstothevalidityofthestudythatlimitconfidenceinthefindings.Forexample,thestudydesignwasnotblinded,soparticipantsmayhavebehaveduncharacteristically(Hawthorneeffect).Almostallofthecomponentsofthelargersurveywerevalidated,butthecombinationwasnot(measurementvalidity).
Thestudywaslimitedbythoseyouthwhowereexcluded,losttofollow-up(internalvalidity:selectionandattrition),orhadincompletedocuments(measurementvalidity).
Ifyouwouldliketolearnmoreaboutthisstudyandthesurveysthatwereused,seeZun,L.,Downey,L.,&Rosen,J.(2006).TheeffectivenessofanED-basedviolencepreventionprogram.AmericanJournalofEmergencyMedicine,24(1),8–13.
SurveysWithQualitativeData:ThreatstoInternalandExternalValidityQualitativesurveyorsoftendifferfromtheirmorequantitativelyorientedcolleaguesinestablishingclose,sometimespersonal,relationshipswithrespondents.Qualitativesurveyorsmayspendmonthsorevenyearslivingincommunitiestofindoutfirst-handaboutinhabitants’needs,practices,andvalues.Theclosenessbetweensurveyorandrespondentgivesthesurveyorthepotentialforcollectingin-depthinformationonculturallyimportantandoftensensitivetopics.Atthesame,astrongconnectionmayintroducebiasintothesurveybecausetherespondentsknowtheyarepartofastudy(Hawthorneeffect),theymaybehavedifferentlyasaresultofansweringquestions(reactiveeffectsoftesting),andtheymayrespondtothesurveyor(appearance,perceivedattitude)ratherthanthesurvey.Atthesametime,thesurveyors’attachmenttothecommunitycanalsoaffectwhichquestionstheyask,theirchoiceofpeopletointerview,andtheirunderstandingandexplanationofthefindings.
Herearefivepotentialthreatstothevalidityofqualitativesurveys:
1. Thedynamicsofthesurvey-respondentrelationshipcaninfluencewhoisselectedandwhorespondstothesurvey,anditcanaffectthetruthfulnessoftheresponses.Thisissimilartoselectionbias,whichaffectsbothinternalandexternalvalidity.
2. Thesurveyors’preferenceforcertaintheories,designs,orissuescaninfluencethechoiceofsurveytopicsandtheirexplanationofthefindings.
3. Therecanbebiasduetohowandwhichevents,places,andpeoplearesampled.4. Financialandhumanresourcescanaffecttheavailabilityandreliabilityofinformants.5. Thesurveyors’personalandprofessionalqualities,includingtheirexperienceandrelationshiptothe
communityandindividualpeople,canaffectallcomponentsofthesurveyincludinghowitis
formatted;thesamplingmethodsusedandthequestionsasked;whocollects,analyzes,andreportsthefindings;whenthefindingsarereleased,towhom,andinwhatform(written?oral?online?).
Example:QualitativeStudiesandValidityForoveradecade,improvingAfghanwomen’shealthandreducinghighmaternalmortalityhasbeenapriorityoftheAfghanMinistryofPublicHealthanditsinternationalpartners.Aqualitativeethnographicstudyinonematernityhospitalaimedtounderstandtheperspectivesofhealthcareprovidersontheirroles,experiences,values,andmotivationsandtheimpactofthesedeterminantsonthecareofperinatalwomenandtheirbabies.
Theresearchersspent6weeksofobservationfollowedby22semi-structuredinterviewsandfourinformalgroupdiscussionswithstaff,twofocusgroupdiscussionswithwomen,and41backgroundinterviewswithAfghanandnon-Afghanmedicalandculturalexperts.
Thestudyfindings,accordingtotheresearchers,werelimited[biased]bythefactthatthatpeoplewhowerewillingtotalktotheinvestigatorswerepossiblythemorevulnerablestaffandthiscanhaveskewedthefindings(selection).
Thestudywasconductedwiththeassistanceofatranslator.Theresearchersnotethatalthougheveryeffortwasmadetoreducethenegativeimpactoftranslationandensurequality,someofthedepth,richness,andculturalnuanceswerelost.Somequestionsweresimplifiedandsomesensitivequestionswerenotaskedtoavoidmisunderstanding(measurementvalidityiscompromised).
Afinallimitationwasthatholdinginterviewsinhospitalwasnotidealassomeinterviewsweredisturbedbythedemandsofthehospital.Itwasnotpossibletoconductinterviewsatadifferentlocationortime,duetothebusylivesandfamilyrestrictionsonthemovementsofmanyparticipants(measurementvalidityiscompromised).
Ifyouwouldliketolearnmoreaboutthisstudyandthesurveysthatwereused,seeArnold,R.,vanTeijlingen,E.,Ryan,K.,&Holloway,I.(2014).UnderstandingAfghanhealthcareproviders:AqualitativestudyofthecultureofcareinaKabulmaternityhospital.BJOG:AnInternationalJournalofObstetrics&Gynaecology.
SummingUpAsurvey’sdesignreferstothefrequencyofitsadministration(onetimeormoreoften),itssequence(justbeforeandimmediatelyafteranewprogram),andthenumberofgroupsinvolved(allstudentsinoneschooldistrictorallstudentsineachof10districts)andhowgroupsareassigned(atrandomorconveniently).Surveydesignsareobservationalorexperimental.Cross-sectionalsurveysareobservationalstudiesthatproducedatadescribingthingsastheyareandarerelativelyeasytoimplement.Ifthingschangerapidly,thesurveyinformationwillpossiblybecomeoutdated.Longitudinalsurveysprovidethesurveyorwithdataonwhethereffectslastovertime.Unlesstherespondentsaremotivated,theymaydropout,andtheremaindermaynotbetypicaloftheentiregroup.Also,thegroupmaybeatypicalinthattheywerewillingandhadthetimetoparticipateintheprogramandcompletethesurveys.Trueexperimentsarecharacterizedbyrandomselectionandassignment.Manyresearchersassertthatthebesttrueexperiments—theleastbiased—arethoseinwhichparticipantsandinvestigatorsdonotknowtheirgroupassignment.Implementingtrueexperimentsmaybedifficultinthe“real”worldofsurveysandsocialresearch.Theyarerelativelycostlyandrequiretimeforimplementationandanalysis.Nevertheless,ifyouwanttoknowwhetheranexperimenthasresults,thetrueexperimentorrandomizedcontrolledtrialisthewaytogo.Quasi-experimentaldesignsareeasiertoimplement.However,becausegroupsarenotrandomly
constituted,theirinitialdifferencesmayberesponsiblefortheoutcome.Normativeorhistoricalsurveydesignscanbelessexpensiveandtime-consumingthanothercomparisondesignsifyoucanfindappropriatedatabasesforcomparisonpurposes.Validsurveyinformationcomesfromreliableandvalidsurveyinstruments(measurementvalidity)andfromthevalidityofthesurvey’sdesign(designvalidity).Thebestsurveydesignsminimizesourcesofbiasbycontrollingselection,usingrandomassignmenttogroup,andblindingparticipantsandresearchers.Measurementanddesignvaliditydifferinthatthefirstreferstothecharacteristicsofthesurveyinstrument,whereasthesecondreferstothecontextinwhichthesurveytakesplace.Measurementvalidityasksthefollowing:Isitscontentcomprehensive?Doesgoodperformancepredictsuccesslateron?Doesameaningfuldifferenceexistbetweenhighandlowscorers?Designvalidityasksthis:Howoftenwillthesurveybegiven?When?Howwillrespondentsbeselected?Adesignhasinternalvalidityifastudy’soutcomeisduetoorcausedbythevariablesthatarecontrolledormanipulatedinthestudy.Itisexternallyvalidifthefindingsapplytootherpeopleandothersettings.Threatstointernalvalidityincludeselectionofparticipants,history,maturation,testing,instrumentation,andstatisticalregression.Threatstoexternalvalidityincludetheinteractioneffectsofselectionbiasesandtheexperimentaltreatment,thereactiveeffectsoftesting,thereactiveeffectsofexperimentalarrangementortheHawthorneeffect,andmultipleprograminterference.Qualitativesurveysmayexperiencethreatstotheirvalidityresultingfromthesurveyors’closenesstothecommunitiesandindividualsbeingsurveyedaswellasfromthesurveyors’choiceofevents,people,andquestions.
ThinkAboutThis1. Findthefollowingarticles(oranyfourcontainingsurveys),nametheresearchdesignusedforthe
survey,anddiscussthedesign’sinternalandexternalvalidity.ArticlesArnold,R.,vanTeijlingen,E.,Ryan,K.,&Holloway,I.(2014).UnderstandingAfghanhealthcareproviders:AqualitativestudyofthecultureofcareinaKabulmaternityhospital.BJOG:AnInternationalJournalofObstetrics&Gynaecology.Gardner,F.,Burton,J.,&Klimes,I.(2006).Randomisedcontrolledtrialofaparentinginterventioninthevoluntarysectorforreducingchildconductproblems:Outcomesandmechanismsofchange.JournalofChildPsychologyandPsychiatry,47(11),1123–1132.Gatto,N.M.,Ventura,E.E.,Cook,L.T.,Gyllenhammer,L.E.,&Davis,J.N.(2012).LASprouts:Agarden-basednutritioninterventionpilotprograminfluencesmotivationandpreferencesforfruitsandvegetablesinLatinoyouth.JournaloftheAcademyofNutritionandDietetics,112(6),913–920.Patrick,K.,Calfas,K.J.,Norman,G.J.,Zabinski,M.F.,Sallis,J.F.,Rupp,J.,etal.(2006).Randomizedcontrolledtrialofaprimarycareandhome-basedinterventionforphysicalactivityandnutritionbehaviors:PACE+foradolescents.ArchivesofPediatricAdolescentMedicine,160(2),128–136.
2. Explainhoweachoneofthesetermsisusedwhentalkingaboutsurveys:researchdesign,measurementvalidity,designvalidity,internalvalidity,externalvalidity,cross-sectionalsurvey,longitudinalsurvey,cohortdesign,casecontroldesign,trueexperiment,randomizedcontrolled
trial,andquasi-experiment.
ReferencesCampbell,D.,&Stanley,J.(1963).Experimentalandquasi-experimentaldesignsforresearch.Chicago,IL:Rand-McNally.
Gardner,F.,Burton,J.,&Klimes,I.(2006).Randomisedcontrolledtrialofaparentinginterventioninthevoluntarysectorforreducingchildconductproblems:Outcomesandmechanismsofchange.JournalofChildPsychologyandPsychiatry,47(11),1123–1132.
Patrick,K.,Calfas,K.J.,Norman,G.J.,Zabinski,M.F.,Sallis,J.F.,Rupp,J.,etal.(2006).Randomizedcontrolledtrialofaprimarycareandhome-basedinterventionforphysicalactivityandnutritionbehaviors:PACE+foradolescents.ArchivesofPediatricAdolescentMedicine,160(2),128–136.
6AnalyzingandOrganizingDataFromSurveys
OverviewAnalyzingdatameansusingstatisticalandqualitativemethodstodescribeandinterpretrespondents’answerstothesurvey’squestions.
Surveyorsalmostalwaysreportdescriptivestatistics,suchastherespondents’averageage,educationallevel,andgradeorscore.Sometimestheysummarizethedataasacross-tabulation(cross-tab)todescribetherelationshipbetweentheanswerstotwodifferentsurveyitems.Supposeasurveyorintendstodescribehow100boysand100girlsrespondtoasurveyoftheirsatisfactionwithanewschool-basedsportsprogram.Across-taboftheresultscouldtakethisform:
Thetableshowsthat55%ofgirlsaresatisfied,butonly42%ofboysare.Arethe13percentagepointsameaningfuldifference?Onewayoffindingoutistoapplyastatisticaltestsuchasachi-squaretest.Chi-squaretestsevaluatestatisticallysignificantdifferencesbetweentwoormoregroupswhenthedataareproportions.Ifthedataarecontinuous,thesurveyorusesotherstatisticalteststoevaluatesignificance.Onesuchisthettest.Choosingastatisticaltestalwaysnecessitatesunderstandingwhetherthedataarecategoricalorcontinuous.Categoricalandcontinuousdataresultfromthetypeofresponsescalethataccompanieseachquestion.
Statisticalmethodsallowsurveyorstomakestatementsaboutstatisticalsignificance,onemeasureofthemeaningfulnessofsurveyresults.Afindingofstatisticaldifferencesamongtwogroups’averagescoresimpliesthatthereisindeedadifferencebetweenthemandthat,mostimportant,thedifferencesare“real”becausetheyareunlikelytohaveoccurredbychance.Butsurveyorsarealsointerestedinfindingoutwhetherthefindingsaremeaningfultothepeoplewhousetheresults.Confidenceintervalsprovidearangeofvaluesandhelpindecidingwhetherdifferencesarepracticallymeaningful.
Inadditiontodescriptionsofresponsesanddifferencesinresponses,surveyorsarealsoconcernedwiththerelationshiporcorrelationamongtheanswerstoquestions.Ahighcorrelationbetweenheightandweight,forexample,suggeststhattallerpeopleweighmoreandthatheavierpeoplearetaller.Regressionsareastatisticaltechniquethatusescorrelationsasthebasisforpredictingthevalueofonevariablefromtheother.Forexample,supposeyouwanttoknowwhetherbeinginvolvedinhighschoolathleticsisassociatedwith(orpredicts)participationincollegeathletics.Tofindout,youwoulduseregressionanalysiswithyoursurveydata.
Othertechniquesthatsurveyorsuseincluderisksandodds.Thesestatisticaltechniquesproduceinformationonthelikelihoodthataparticularoutcomewilloccurwithinagroupandcanalsobeusedtocompareoutcomesamonggroups.
Contentanalysisisaqualitativemethodthatcanbeusedtofindcommonideasorthemesintheanswerstoopen-endedsurveyresponses.Contentanalysisrequirescodingcommentsandresponsestoopen-endedquestions.Ifalargeenoughnumberofpeopleconsistentlywriteinthattheyareconcernedwiththesameproblemorexpressaparticularpointofview,itbecomesatheme.
Datamanagementisanintegralcomponentofdataanalysis.Managingdatameansorganizingdataintovariablesandnamingandcodingthemsothatoncetheyareinadatabasetheycanbeanalyzed.Assigningcodestodataisanessentialanalyticactivity.In
largesurveys,youmustverifythereliabilityofthecoding.Twocodersareoftenused.Tofindoutwhethercodingisreliable,considercomputingastatisticcalledthekappa.Thekappaprovidesameasureofagreementbetweentwocoders.
Youmustreviewsurveyreturnsformissingdataanddecidehowtohandlethem.Missingdataresultfromunansweredquestionsorentiresurveys.
Dataentryistheprocessofgettingthesurvey’sresponsesintoadatabase.Onlinesurveydataisautomaticallyenteredintoadatabase,withtheluxuryofreceivingreportsinrealtime.Surveydataalsomaybescannedinorhand-enteredintospreadsheetsforsubsequentconversionintoadatabase.Regardlessofhowthedataareentered,theresponsesneedtobecheckedfirstforvalidity.Thesurveyordoesnotusuallywantanswersfrommenaboutthephysicaljoysofgivingbirth,forexample.Mostonlinesurveyprogramshavetheoptionforautomaticvaliditychecking.Acleandatasetisonethatcanbeusedbyanyonetogetresultsthatareidenticaltoyours.Databecome“dirty”foranumberofreasons,includingpoorquestions,misleadingresponses,miscoding,incorrectdataentry,andmissingresponses.Toavoiddirtydata,makesurethatcodersordataenterersareexperienced,welltrained,andsupervised.Ifyouareusingonlinesurveysoftware,youcanfindavalidationfunctionthatwillallowyoutoguidetheresponsessothattheyareuniformandmeetyourstandards.Thiswillhelpensureacleandataset.
WhatIsTypicalAnyway?SomeCommonlyUsedMethodsforAnalyzingSurveyDataThedayhasarrived.Allthesurveyshavebeenreturned.Theresponserateishigh.Allthequestionsandalltheformshavebeenfilledout.Nowisthetimetofindoutwhatthesurveyshows:Howmanymenresponded?Howmanywomen?Dotheydiffer?Havetheirviewschangedinthepast5years?Surveyorsanswerquestionslikethesebyanalyzingsurveyresponsestoobtainresponsefrequenciesandaverageresponsesandtolookforrelationships,differences,andchanges.Someanalysismethodscommonlyusedinsurveysincludethefollowing.
DescriptiveStatisticsThesearethebasicunitsfordescribingasurvey’srespondentsandresponses.Descriptivestatisticsprovidesimplesummariesaboutthesampleandtheresponsestosomeorallquestions.Togetherwithsimplegraphicsanalysis(suchaspiechartsandbargraphs),theyformthebasisofpracticallyeveryquantitativeanalysisofsurveydata.Descriptivestatisticsforsurveysincludefrequenciesorfrequencydistributions(numbersandpercentages),measuresofcentraltendency(themean,median,andmode),andmeasuresofvariation(rangeandstandarddeviation).
Frequencies
Afrequencyisacomputationofhowmanypeoplefitintoacategory(e.g.,20aremorethan55yearsofage;150haveacold)orchoosearesponse(e.g.,32saiddefinitelyveryimportant;8saidmorethanthreetimesaweek).
Considerthis:50preschooldirectorsaresurveyed.Allschoolsarepubliclysupported.Thesurveyfollows.
Example:PreschoolPurposesQuestionnaireBelowaresomepossibleobjectivesofpreschooleducation.Circlehowimportanteachisinguidingyourschool’sprogram.(Number1beingdefinitelyimportantandNumber5beingdefinitelynotimportant.)
Forthepreschoolpurposesquestion,youcansummarizetheresponsesasshowninthefollowingexample.
Example:FrequencyofQuestionnaireResponses
Sometimesyouwanttogrouptheresponsestogether(or,intechnicalterms,prepareafrequencydistributionofgroupedresponses),asshowninthenextexample.
Example:GroupedRatingsofPreschoolPurposesby50Directors
Inthistable,theresponsesaredividedintotwocategories(dichotomized):important(includingtwocategoriesofresponse:definitelyimportantandimportant)andnotimportant(includingtwocategoriesofresponses:definitelyunimportantandunimportant).Whygroupresponsestogether?Ifonlyafewrespondentsselectoneofthechoices(sayonlythreeanswerdefinitelyimportant),thenthecategorymayloseitsmeaningtoyou.
Averages:Means,Medians,andModesThemean,median,andmodeareallmeasuresofaverageortypicalperformance.
Themean
Thearithmeticaverage,themean,requiressummingunitsanddividingbythenumberofunitsyouhaveaddedtogether.Herearefivescores:50,30,24,10,and6.
Theaverageforthesefivescoresis50+30+24+10+6dividedby5=24.
Theformulaforthemeanismx=SxN.
μisthesymbolforthemean,and∑standsforthesum,so∑xmeanstoaddallthenumericalvaluessuchasx.Nstandsforthenumberofx’s.
Supposeyouwantedtocomputetheaverageratingforthefourpreschoolpurposes.First,youwouldhavetoknowhowmanydirectorsassignedeachratingtoeachpurpose.LookatTable6.1.
Tocomputetheaverageratingforeachpurpose,youwouldmultiplythenumberofdirectorswhochoseeachpointonthescaletimesthevalueofthescale.Forcreativity,13directorschosea1,soyouwouldmultiple13×1,addtheresultstogether(13×1)+(2×2)+(5×3)+(20×4)+(10×5),anddividebythenumberofdirectors:13+4+15+80+50dividedby50=3.24,theaverageratingforcreativity.
Inthiscase,theclosertheaveragesareto1,themoreimportantthepurpose.Lookatthistable.
Themedian
Themedianisthepointonascalethathasanequalnumberofscoresaboveandbelowit.Anotherwayofputtingitisthatthemedianisatthe50thpercentile.Becausethemedianalwaysfallsinthemiddle,itisusedwhenyouwanttodescribetypicalperformance.
Whydoyouneedtypicalperformance?Supposeyouhadasetofscoreslikethis:5,5,6,6,6,8,and104.Theaverageis20,andthemedianis6.Howcanthishappen?Itmayoccurifthegroupyouaresamplingwasdividedinitsattitude(orknowledge,health,etc.),withmostpeoplefeelingonewayandsomefeelingmuchdifferent.Itcanalsohappenifyouareunabletocollectallthedatayouplanto,andmanyofthepeoplewithoneviewarenotrepresentedintheresponses.
Hereishowtocomputethemedianifyouhaveanequalnumberofscores:
1. Arrangethescoresinorderofsize.2. PlacethemedianbetweentheN/2scoreandthe(N/2)+1score(whereNequalsthenumberof
scores),usingthearithmeticaverage.
Example:ComputingtheMedianforanEvenNumberofScoresTakethesescores:−2,0,6,7,9,and9.
TherearesixscoressoN=6,withN/2=3and(N/2)+1=4.Thethirdscoreinorderequals6andthefourthequals7,sothemedianis6.5.
Takethesescores:2,4,5,8,9,and11.
Again,N=6,sothemedianisbetweenthethirdandfourthscores,5and8.Thistime,however,thereisagapofthreeunitsbetweenthethirdandfourthscores.Addingthetwoscores5+8anddividingby2givesavalueof6.5forthemedian.
Whatasetofdataissmallandoddinnumber,dothis:
1. Arrangethescoresinorderofsize.2. Placethemedianatthescorethatisthe(N+1)/2fromthebottomofthedistribution.
Example:ComputingtheMedianforanOddNumberofScoresTrythesenumbers:−9,−8,−6,−6−4,−3,−2,0,and2.
Themedianoftheseninescoresis(9+1)/2orthefifthscore,andsothemedianis−4.
Butsupposethatineitheraneven-numberedorodd-numberedsetofcases,youhaveseveralidenticalscoresatthemedianorborderingit,aswiththisset:3,6,7,8,8,8,9,9,10,and12.
WhenN=10,asinthiscase,themedianwouldusuallyfallbetweenthefifthandsixthscore.Calling8themedianthistimewillnotworkbecauseyouneedfivedifferentscoresaboveandfivedifferentscoresbelow.
Butifyouconsiderascoreof8aspartofacontinuumthatisevenlyspreadthroughouttheintervalbetween7.5and8.5,youcaninterpolateandcomeupwithanintermediatescore.
Thinkofitthisway:Youhavethreecases(3,6,and7)untilyoucometotheintervalof7.5to8.5,whichcontainsthree8s.Withinthatinterval,youwillneedtwomorecasestomakeatotalof5.So,youaddtwothirdsofthedistance,or6.7to7.5,andyougetamedianof8.17.
Themode
Themodeisascore(orapointonthescorescale)thathasahigherfrequencythanotherscoresinitsvicinity.Lookatthesescores.
DistributionAhasasinglemodeof29,with18peoplegettingthatscore.Thisdistributioniscalledunimodal.DistributionBhastwomodes,at25and31,sothedistributioniscalledbimodal.(Althoughthefrequencyof11associatedwiththescoreof31isthesameorlowerthanthatofotherscores,thescoreisamodebecauseithasahigherfrequencythantheothersnearit.)
Themode,withitsconcentrationofscores,describestheprevailingview.Youmightusethemodewhenyoususpectthatyouhaveagroupwithwidelydifferingcharacteristics.Forexample,ifyoususpectthatthereasonpeopleinProgramDdidbetterthanthoseinProgramCisbecausetheywereeconomicallybetterofftobeginwith,youmightcomparetheirincomesbeforetheyenteredeachprogram.
Ifyoufindsomethinglikethis,youcouldconcludethatyouareprobablyright.
PeopleinProgramDareonthehigh-incomepartofthescale,whereasthoseinProgramCareonthelow-incomeside.Together,GroupsCandDgiveyouabimodaldistribution.
Variation:Range,Variance,andStandardDeviationWhenyoucomputeanarithmeticaverage,everynumbercounts.Ifoneortwopeoplehaveveryhighorlowscores,theycanmaketheaverageseemartificiallyhigherorlower.Usingthemedianhelps,butittoocanbemisleading.Considerthissetofscores:2,3,4,5,6,7,and8.
Themeanandmedianare5.Ifyouweretochangethelasttwoscoresto17and18,themedianwouldstaythesameat5,withthreecasesaboveandbelowthescore.Themeanwouldriseto55/7=7.86.
Ifeverynumbercounts,itissometimesimportanttostudythespreadofscores—theirvariation—toshedlightonhowthemeancametobewhatitis.Areallthescoresnearthearithmeticaverage?Aresomeveryhighorverylow?Lookatthistable.
Ifyoulookattheaveragenumberofclassesattendedatthefourschools,youfindthatitis47.5.Lookingfurther,youseethat,atsomeschools,asfewas15classeswereattended,whereasatothers,all60were.Infact,itishardnottobestruckbytherangeofattendance.Atleasttwoinferencescanbemadebecauseyouhaveaccesstotherange:
1. Variationexistedinthreeofthefourschools(1,3,and4),withsometeachersattendingmanyclassesandothersattendingveryfew.
2. Inonlyoneschool(2)didalmosteveryoneattendaboutthesamenumberofclassesaseveryoneelse.
Insomecases,variationisconsideredanasset.Aprogramtotrainpeopletothinkindependentlyorcreativelymightexpectasurveytorevealavarietyofperspectives.Youalsoneedvariationtomakecomparisons.Ifeveryoneperformsequallywellorsharesthesameviews,youcannotselectthebest,thestrongest,themostliberalorconservative,andsoon.Inothercases,however,variationisadisappointmentbecauseyouwanteveryonetosharethesamevieworachieveaskillatthesamelevelofproficiency.Ifthedistrictthatissponsoringthecontinuingeducationbelievesitsprogramsareworthwhile,thewiderangeofattendancewillbedisappointing.
Anothermeasureofvariationiscalledthevariance,anditssquarerootiscalledthestandarddeviation.Thelatterisastatisticaltermbasedonascore’sdistancefromthemean.Infact,thestandarddeviationistheaveragedistancetheaveragescoreisfromthemean.Sometimes,insteadofthestandarddeviation,thevarianceisused.Thevarianceissimplythesquareofthestandarddeviation.
Youcanonlycomputethevarianceandstandarddeviationfromcontinuousdatasuchasaveragescores.Usetherangewhenyouwanttostudyvariationincategoricaldata(frombothcategoricalandordinalscales).
CorrelationandRegressionDatathatarecorrelatedarerelated.Ahighcorrelationbetweenheightandweight,forexample,suggeststhattallerpeopleweighmoreandthatheavierpeoplearetaller.Correlationsarereportedwithinarangeof+1(perfectpositivecorrelation)to−1(perfectnegativecorrelation).
Whenhighvaluesononevariableoccursimultaneouslywithhighvaluesonanother,thetwovariablesaresaidtobepositivelycorrelated,andwhenhighvaluesononevariableoccurwithlowvaluesonanother,thetwovariablesaresaidtobenegativelycorrelated.Thecorrelationcoefficientissymbolizedasrandisusuallyreportedintwodecimalplaces.
ASpearmanrankordercorrelationisusedwithcategoricaldata—thatis,datathatcomefromcategorical
orordinalscales.Arankordercorrelationprovidesameasureforthedegreeofassociationorequivalencebetweentwosetsofranks,asinthisexample.
Example:RankOrderCorrelationAclassof50collegestudentstakestwoattitudesurveys.Thefirstsurveypollstheirviewsonuniversalhealthcare,andthesecondsurveyasksabouttheirpoliticalpreferences.Johnscoreshighest(bestattitude)amongtherespondentsononemeasureandaverageonthesecond,Jane’sscoresarethe5thand8thhighest,Bill’sscoresarethe14thand13thhighest,andsoon.Arankordercorrelationcoefficientiscomputedtoseewhetherthetwosurveysagree:Dopeoplewhorankhighononesurveyalsorankhighontheother?
Whichstatisticalprocedureisanappropriateonetouseifthesurveyproducescontinuousdatasuchasheightandweight?ThePearsonproduct-momentcorrelationisusedtoestablishrelationshipsbetweentwosetsofcontinuousdata.Herearetwosituationsinwhichthesecorrelationsareappropriatelyused:
Therelationshipbetweengradepointaverages(GPAs)andscoresonanattitudetowardschoolsurveyiscorrelated.Therelationshipbetweenliberalandconservativeviews(1=liberalto10=conservative)andfamilyincome(from$5,000to$50,000)iscorrelated.
Regressionsusecorrelationsasthebasisforpredictingthevalueofonevariablefromtheother.Forexample,supposeyouwanttoknowwhetherstudentswhoreportbeinginvolvedinhighschoolathleticsareassociatedwith(orpredict)participationincollegeathletics.Tofindout,youcoulduseregressionanalysiswithyoursurveydata.
Asanotherexample,supposeyouwanttofindoutwhetherstudentswhohavehighgradesinhighschoolarealsolikelytohavehighgradesincollege.Inotherwords,youwanttoknowwhetherhighgradesinhighschoolpredicthighgradesincollege.Tofindout,youwouldfirstsurveycollegefreshmentogettheirhighschoolandfreshmanGPAs.Thenyouwouldexaminethecorrelationbetweenthetwoaverages.Finally,youwouldsurveyanewsetofhighschoolstudents(whoarecollegeboundbutnotthereyet)and,usingregressiontechniques,predicttheirfreshmanGPAbasedontheoriginalcorrelation.
Warning:Youcanusecorrelationstoidentifyrelationshipsbetweenvariables,butyoucannotusethemtoestablishcausation.Supposeyouinterviewasampleofpeopletofindouttheirhighestlevelofcompletededucation(Variable1).Youalsoaskthemtotellyouhowmanymagazinestheysubscribetothisyear(Variable2).Ananalysisofthecorrelationbetweeneducationandsubscribingpracticemayrevealthatthepeopleinyoursamplewhocompletedmanyyearsofschoolingalsosubscribetomanymagazines,butitcannotprovethatcompletingmanyyearsofschoolcausedthepracticeofsubscribing.Itmaybethatpeoplewhocompletethemostschoolingarealsothewealthiestinthesampleandthatenablesthemtosubscribe(theycanaffordit),oryoursamplemaybeatypicalortoosmalltodrawconclusions.
Multipleregressiontechniquesletyouconsidermorethanonepredictor.SupposeyoureviewtheeducationliteratureandfindthatstudentswhohavehighGPAsinhighschoolcomefromhomeswithrelativelyhighhouseholdincomes.Thatis,theliteraturesupportsthehypothesisthatahighcorrelationexistsbetweenhouseholdincomeandstudents’highschoolGPA.Youcannowaskthis:Whatarethecontributionofgradesinhighschool(Predictor1)andthecontributionofhighhouseholdincome(Predictor2)tocollegeGPA?Amultiple-regressionanalysismaybeusedtohelpyouanswerthisquestion.
DifferencesBetweenGroups
Descriptivedifferences:Thecross-tabulation
Across-tabulation(oftenabbreviatedascross-tab)isamethodusedtodescribetwovariablesatthesametime.Itisoftendisplayedasatable.Thetable’snameisusuallydefinedbythenumberofitsrowsandcolumns.
Thefollowingisafictitiousexampleofa3(rows)×2(columns)cross-tabdescribingtheresultsofasurveytofindoutabouttheuseofvariouselectroniccommunicationmethods.Inthisexample,thesurveyorscreateatablethatdescribestheanswerstotwosurveyquestions:
1. Onatypicalday,howmuchtimedoyouspendontheInternet?Theresponsechoicesarelessthan1hour,2−3hours,andmorethan2hours.
2. Whatisyourfeelingaboutsurveys?Theresponsechoicesarelovesurveysandhatesurveys.
Eachcellinthetablegivesthepercentageofsubjectsthatsharethatcombinationoftraits.Forinstance,inthissurvey,70%ofpeoplewhosaytheylovesurveysspend1hourorlessontheInterneteachday.
Cross-tabsarefrequentlyusedbecausetheyareeasytounderstandandmayprovideinsightintothemeaningofasurvey’sresults.Fromthetable,itappearsthat60%ofpeoplewhoreportspendingmorethan3hourseachdayontheInternethatesurveys.Thesurveyorwhoisinterestedinconductinganonlinesurveymayfindthisaninterestingresult.
Nowconsiderthis.AmarketingresearchfirmisinterestedinsurveyingcustomersoftheMacWindowCompanytofindoutwhichproductsitscustomersarelikelytopurchaseinthenext12months.Thisinformationistobeusedtoplanthecompany’sinventory.Thatis,MacWindowwillstocktheproductsthatmostpeopleplantopurchase.MacWindowconductsanonlinesurveytofindoutaboutitscustomers’plans.AportionofthesurveyisreproducedhereinFigure6.1.
Thesurveyorsprepareacross-tabulationoftheresultstotwoquestions:“Doyouplantopurchaseanyoftheseinthenext12months?”and“Whatisyourage?”showninTable6.2.
Fromthesecross-tabbeddata,MacWindowcanseecertainpatterns.Forinstance,thelargestnumberofrespondentsisbetween24and29yearsofage,andaboutone-thirdofthemareplanningtobuyabikeinthenext12months.Thatisasubstantialproportionofpotentialcustomers,andsoMacWindowmaydecidetostockacompletevarietyofbicyclestomeetthosecustomers’needs.Thenextlargestrespondentgroupisbetween30and35yearsold.About23%ofthisgroupisplanningtobuyacamcorder,with32%indicatingtheywillpurchaseasnowboard.Infact,almostone-thirdofallrespondentsover30areplanningtobuyasnowboardinthenext12months.
Figure6.1MacWindow’sOnlineSurvey
TheMacWindowCompanycancreateadditionalcross-tabstoanswerotherquestionsthatmayconcernthecompany.Forinstance,thecompanymaydecidetolookattherelationshipbetweenageandincome.Doesincomerisewithage,oraretheiryoungercustomersfromwealthierhouseholdsthantheirolderones?Domalesandfemalesdifferintheirhouseholdincome?Inpurchaseplans?Isthereadifferenceinexpectedpurchasesbetweenhigh-andlow-incomehouseholds?Thesearegoodquestions,butunfortunately,across-tabcanonlydescribetheincomeandpurchaseplans.Itcannottellwhetheranyobserveddifferencesaremeaningfulinstatisticalterms.Tofindout,thesurveyormustrelyonstandardstatisticalmethods.
StatisticalDifferencesSupposeyouwanttoknowwhetheronegroupofsurveyrespondentsisdifferentfromanother.Aretheyhealthier?Morelikelytovoteinthenextelection?Moreemployable?Betterspellers?MorelikelytopurchaseMacWindowCompany’sbikesordigitalcameras?Methodsusedtotestfordifferencesbetweengroups’surveysincludestatisticaltechniquessuchasthechi-squaretest,ttest,andanalysisofvariance(ANOVA;therearemanyothers).
Statisticaltestsactuallyhypothesizethatthesurveyoutcomesarethesameacrossgroups.Forinstance,ifyouwanttocompareattitudescoresbetweenGroupsAandBandyouwanttouseastatisticaltestsuchasattest,youhypothesizethatthemeanscoresarethesame.Theideaistosetupastrictenvironmentthatputsthisassumptiontothetest.
Thetestofequality—therespondentgroupsarethesame—iscalledthenullhypothesis.Afterstatingthehypothesis,thestatisticalanalysisisperformed,astatisticisproduced,andthisservesasaguideinacceptingorrejectingthenullhypothesis.Ifthenullisrejectedinfavorofanalternativehypothesis(the
groupsaredifferent),thentheresultsareconsideredstatisticallysignificant.
Supposeyousurveypeopleonacruise,halfofwhomarepartofanexperimentalprogramcombiningtourismwitheducation.Youarefairlycertainthattheexperimentisworking,andtheexperimentalgroupismuchmoresatisfiedthanthecomparisongroup.Nevertheless,asascientificsurveyor,youstartoutwiththeassumptionthatbothgroupsareequallysatisfied.Then,youcomparethetwogroups’averagesatisfactionscoresusingastatisticalmethodcalledanindependentttest.Youfindthatyouarecorrect,soyourejectthehypothesisofnodifference.Thereisastatisticallysignificantdifferenceafterall.Becausetheaveragesatisfactionscoresoftheexperimentalgroupwerehigher,youconcludethattheexperimentisworking,youwererightinthefirstplace,andnowyouhavestatisticstobackyouup.Thekeytoyourconclusionistheideaofstatisticalsignificance.Theresultsarestatisticallysignificantiftheyareunlikelytohaveoccurredbychance.
StatisticalSignificanceSupposeyousurveytheattitudesoftwogroupsofstudents,oneofwhichisinanexperimentalreadingprogram.Alsosupposethattheexperimentalgroup’sscoresarepoorerthantheothergroup’s—say,by10points.Aretherelativelypoorerscoresduetochance,oristhenewreadingprogramresponsible?Aresultthatisunlikelytohappenbychancecanbecalledstatisticallysignificant.Howmuchofadifferencebetweenthetwogroupsisnecessarybeforeyoucaneliminatechanceasthemotivation?
Todeterminestatisticalsignificance,youmustrelyonsamplingtheory.Forexample,youaskaquestionsuchasthis:Whatistheprobabilitythatmytworandomsamplesofstudentsfromthesamepopulationwillproducemeanscoresthatdifferbyasmuchas,say,10points?Twentypoints?
Supposeyoudecidethatachancehappeningof1timein100isanacceptablerisk.Thispredefinedprobability(p<.01)iscalledthelevelofsignificance.Ifthedifferencesyouobserveoccurnomorethan1outof100times,youcanrejectthenullhypothesisofnodifferencebetweengroups.
Byconvention,surveyorsusuallyusethe.05,.01,.001,andsoonsignificancelevel,meaningthattheobserveddifferenceintheexperimentalandtraditionalprogramswillbeconsideredstatisticallysignificantifthedifferenceof10pointswouldoccurbychance(assumingthetwogroupsarerandomsamplesfromthesamepopulation)only5timesin100,1timein100,or1timein1,000.
Intestingstatisticalhypotheses,youmustestablishrulesthatdeterminewhenyouwillacceptorrejectanullhypothesis.Take,forexample,astatisticaltestofanexperimental(A)andcontrol(B)readingprogram,wherethenullhypothesisisthatthemeanreadingscoresforbothgroupsareequal.Whenyouapplyastatisticaltest(suchasthettest)tothedata,youdonotexpecttofindzerodifferencesinaveragescoresbetweenthetwogroups.Instead,therealquestioniswhetherthedifferencesaresosmallthattheycouldhaveoccurredsimplybychance.Whenyouselecttworandomsamplesfromthesamepopulation,youcanexpecttheirmeanscorestobeclose,butnotexactlythesame.
Itisuptoyoutodecidehowfarapartthescoresmustbebeforeyouaresatisfiedthatthedifferenceisnotjustanaccident.Youcouldchoose.10,.05,.01,or.001asthelevelofsignificancedependingontheamountoferroryouarewillingtotolerateinrejectingthenullhypothesis.Whythesefractions?Thesearetheonestypicallyusedbysurveyandresearchanalystsbytradition.
Ifyouselectthe.05levelofsignificance,thenabout5timesin100youwillrejectthenullhypothesiswhenitis,infact,correct.Thishappensbecauseyouarecomparingtworandomsamplesfromthesamepopulation,andtheprobabilitythattheywilldifferbychancealoneis5%(p<.05).ThissituationisknownasaTypeIerror.Itistheprobabilityofrejectingthenullhypothesiswhenitistrue.Thatis,youcommitaType1errorwhenyousaythatthereisadifferencewhen,infact,noneexists.
Ifthelevelofsignificanceis.01,thentheprobabilityofaTypeIerrorisonly1in100,or1%.YoucanselectalevelofsignificancethatwouldvirtuallyeliminatethechanceofaTypeIerror,butthereareseriousconsequences.ThelesslikelyyouaretomakeaTypeIerror,themorelikelyyouaretomakeaTypeIIerror.
ATypeIIerroriswhenyouacceptanullhypothesisthatis,infact,incorrect.Yousay,forinstance,thereisnodifferencebetweentwogroups,butactuallythereis.WhenyoumakeaTypeIIerror,thedifferencebetweenthetwogroups’meanscoresdoesnotfallwithintherejectionregion(say,p<.05).
Thepowerofastatisticaltestistheprobabilitythatthetestwillrejectafalsenullhypothesis(thatitwillnotmakeaTypeIIerror).Anotherwaytothinkaboutthisisastheabilityofatesttodetectaneffectgiventhattheeffectactuallyexists.Aspowerincreases,thechancesofaTypeIIerrordecrease.TheprobabilityofaTypeIIerrorisreferredtoasthefalse-negativerate(β).Therefore,powerisequalto1−β.
Poweranalysiscaneitherbebefore(apriori)orafter(posthoc)surveydataarecollected.Aprioripoweranalysisistypicallyusedtodetermineanappropriatesamplesizetoachieveadequatepower.Posthocpoweranalysisusestheobtainedsamplesizeandeffectsizetodeterminewhatthepowerwasinthestudy.
Aneffectsizehelpstodeterminewhetherastatisticallysignificantdifferenceisadifferenceofpracticalconcern.Itisagenerictermforanestimateoftheeffectofbeinginastudyandisoftendefinedasthedifferenceinmeansbetweenanexperimentandcontrolgroupdividedbythestandarddeviationofthecontrolorofbothgroups.Givenasufficientlylargesamplesize,itispossibletoshowthatthereisadifferencebetweentwomeansbeingcomparedouttosomedecimalposition.Theeffectsizehelpsinknowingwhetherthedifferenceobservedisadifferencethatmatters.
Surveyorswhoareinterestedinanalyzingdifferencesbetweengroupsoftenexpressthosedifferencesbyusingconfidenceintervals(CIs)ratherthanapvalue.ACIisanestimateofapopulationparameterlikeamean,butitisdescribedwithinanintervalorrange.
Supposeyousurvey300randomlyselectedstudentsatamajoruniversity.Youaskthemhowmuchtheyspentononlinepurchasesinthepastmonth.Theaverageamountforthesampleturnsouttobe$100permonth.Butitjustmaybethatthese300studentsaremuchsmallerInternetspendersthanaretheotherstudentsintheuniversity.Thepopulationmean(theaverageforalltheuniversity’sstudents)couldbecloserto$500,forexample.Asmallnumberofuniversitystudentsmaybespendingthousandseachmonth,andthatwillraisethepopulationmeanconsiderably.
Thesample’smeanissimplynotonethesurveyorcanbeabsolutelycertainof(unlessthesampleisthewholepopulation).Thelargerthesamplesize,themoreconfidentthesurveyorcanbethatthepopulationmeanliesfairlyclosetothesamplemean.
CIscometotherescuebecausetheyarebasedontheconceptofrepeatedsampling.IfsamplesofthesamesizearedrawnrepeatedlyfromapopulationandaCIiscalculatedfromeachsample,thensomefraction—say,95%—oftheseintervalsshouldcontainthepopulationmean.
LookatTable6.3.
Theresultsareintheformofatablewithdescriptivestatistics:themean,standarddeviation,standarderror,andtheCI.Group1hasameanscoreof11.TheCIforthatscoreis7.67−14.33.IfsamplesofthesamesizeweredrawnrepeatedlyfromthepopulationfromwhichGroup1wasdrawnandaCIwascalculatedforeachsample,then95%oftheseintervalsshouldcontainthepopulationmean,whichwouldbebetween7.67and14.33.
YoucanplottheCIsonagraph.Ifthemeansdonotoverlap,differencesexist.Ifthemeanofonegroupiscontainedintheintervalofthesecond,differencesdonotexist.Iftheintervalsoverlap,butnotthemeans,youcannottell.
LookatthisgraphfortheCIs.
Group2’smeanscoreiswithinGroup1’sCI.Group3’sintervaldoesnotoverlapwitheithergroup.Differencesinthemeanscanbeseen,andyoucanrejectthenull(thatthemeansarethesame).Asanaside,ifyouweretouseastatisticalmethodlikeANOVA,youwouldfindthatthepvalueandCIsagreeintheiroutcome.TheCIandparerelated.Iftheintervalcontains0,thenthepisnotsignificant.
CIsareregularlyusedtodescribedifferences.Theyarepreferredinmanyplacesbecausetheyenableyoutoseetherangeofvalues.
SurveyingDifferences:UsualMethodsHerearesomecommonlyusedtechniquesforevaluatingdifferencesamonggroupsor,moreprecisely,testingwhethertheyarethesame.
Chi-SquareThechi-squaretestisusedwithcategoricaldata.Itteststhehypothesisthatsurveydataexpressedasproportionsareequal.Aproportioniswhatyougetwhenyoufindouthowmanyofallpossiblepeopleansweracertainwayorhaveaspecificcharacteristic.Supposeyousample1,000peopleandfindthat
250haveblueeyes.Theproportionofpeoplewithblueeyesis250/1,000.Supposethat,ofthe250peoplewithblueeyes,60arewomen.Supposeyouwanttocomparetheproportionofmen(190/250)andwomen(60/250)withblueeyes.Youhaveanideathatproportionsaresignificantlydifferentfromoneanother.Inotherwords,theyaretrueandnotchancedifferences.Thestatisticaltestyouwouldusetocomparetheproportionisthechi-square.Thesymbolforchi-squareisχ2.Thefollowingisanexampleofhowachi-squareisused.
Example:Chi-SquareThePrisonOrganizationsurveys60imprisonedmenand40imprisonedwomentofindoutwhethertheyperceivetheprisonsystemasfairorunfair.Theresultsareputinatable.
Thetableshowsthat57menand32womenbelievethesystemisfair,and3menand8womenbelievethatitisunfair.Thechi-squarecanbeusedtotestthenull(nodifferences)hypothesisthattheproportionsofmenandwomenarethesameintheirviewsoffairness.Ifthenulliskept(retained),youconclude“nodifferences.”Thetableiscalleda2×2table.Itlooksjustlikeacross-tab,butinthiscase,astatisticaltestisbeingdoneonthedata.Thefirst“2”consistsofthecolumnsformenandwomen,andthesecond“2”istherowsforfairandunfair.
ThetTestThettestisalsousedtotestfordifferences.Itallowsyoutocomparethemeansoftwogroupstodeterminetheprobabilitythatanydifferencesbetweenthemarerealandnotduetochance.Statisticsbookswilltellyoutohaveatleast20to30respondentspergroupandcontinuousdatatouseattest.Youneedcontinuousdatatocalculatethearithmeticaverageormean.
Example:tTestHopeHospitalhasinitiatedagourmetmealplan.CharityHospitalsaysgourmetmealscosttoomuch.Dothetwohospitalsdifferintheirpatients’satisfactionwithmeals?Thesurveyorssurvey50patientsatHopeand60atCharity.TheyusetheSatisfactionScale,whichaskspeopletomarkwhereona10-pointcontinuousscaletheirsatisfactionlies.Thesurveyorscomputetheaveragesatisfactionateachhospitalanduseattesttocomparetheresults.
TheMann-WhitneyUTestTheMann-WhitneyUtest(alsocalledtheWilcoxonranksum)enablesyoutocomparetwoindependentgroupswhenyoucannotusethettest—say,becausethesamplesizeistoosmall.Thisstatisticalmethodisatestoftheequalityofthemedians.
Example:Mann-WhitneyUTestPartridgeElementarySchoolhad10fourth-gradestudentswithseverehearingimpairments.Fourofthestudentswereinanewprogramtoteachthemtospeakmoreclearly.Attheendofonesemester,studentsinthenewprogramwerecomparedwiththesixstudentsinthetraditionalprogram.Aspecialeducationexpertratedeachchild’sabilitytospeakonascalefrom1to20,with20representingclearspeech.Ifmorestudentshadbeenavailable,thesurveyormighthavechosenthettesttocomparethetwogroups,butafterconsideringthesmallsamplesize,hedecidedtousetheMann-WhitneyUtest.
ANOVAGroupmeansoraveragescanalsobetestedwithANOVA.Thismethodletsyoutestaveragesachievedbytwoormoregroups.Forinstance,youcantestfordifferencesinaveragescoresonastatewidetestforstudentsinExperimentalSchoolAversusExperimentalSchoolBversusControlSchoolC.
RisksandOddsRisksandoddsareusedtodescribethelikelihoodthataparticularoutcomewilloccurwithinagroup,ortheycanbeusedtocomparegroups.Supposethat,forevery100personswhohaveacold,20peoplealsohaveacough.Theriskofacoughwithacoldis20/100or.20.Theoddsofhavingacoughwithacoldarecalculatedbycomparingthenumberofpersonswith(20)andwithout(80)coughs,or20/80,whichis.25.
Whenyouuserisksandoddstocomparegroups,youcomparetherelativelikelihoodthatanoutcomewilltakeplace.Therelativeriskexpressestheriskofaparticularoutcomeintheexperimentalgrouprelativetotheriskoftheoutcomeinthecontrolgroup.Theoddsratioisadescriptionofthecomparisonoftheoddsoftheoutcomeintheexperimentalgroupwiththeoddsinthecontrolgroup.Youcanusetherelativeriskortheoddsratiowith2×2tables.Lookatthefollowingtableoutline.
Example:OddsRatioandRelativeRisk
Youcanaskthefollowing:Whataretheoddsofstillsmoking(orquitting)intheexperimentalgroupcomparedtothecontrol?Whatistheriskofstillsmoking(orquitting)intheexperimentalgrouprelativetotheriskofstillsmoking(orquitting)inthecontrol?
Therelativeriskandtheoddsratiowillbelessthan1whenanoutcomeoccurslessfrequentlyintheexperimentalthaninthecontrolgroup.Similarly,bothwillbegreaterthan1iftheoutcomeoccursmorefrequentlyintheexperimentalthaninthecontrolgroup.
Whyusetheoddsratioorrelativerisk?Whynotstaywiththechi-square?TheansweristheCI.CIscanbeplacedaroundoddsratiosandrelativerisks(butnotchi-squares),accomplishingthesameobjectiveasasignificancetest.Thatis,youcandetermine(bylookingattheintervals)iftheresultsarestatisticallysignificant.LookatTable6.4describingtheresultsofasurveyofInternetuseforhealthinformation.
Thetablehasfivecolumnsandfiverows.Thefirstcolumnconsistsofdemographics.Becausethetabledisplaysoddsratioresults,eachcharacteristicisassumedtobecomparedtoitsopposite:maletofemale,marriedtonotmarried,andsoon.EachcellcontainsanoddsratioandaCI.Inthefemalerowand“EverUsedtheInternetforHealthInfo”column,youwillfindthreebitsofinformation:1.23istheoddsratioand1.06−1.43istheCI.Lookingatthesenumbers,youcanseethatfemalesweresignificantlymorelikelythanmales(oddsratiogreaterthan1andno0intheCI)toreporteverusingtheInternetforhealthinformationwhencomparedwithnotusingtheInternet.Inaddition,moreeducationandhighercognitivescorescorrespondedtoincreasedoddsofonlineuse.
Surveyorshaveanarsenalofstatisticalmethodsthatmaybeappropriatefortheirsurveys.Supposeasurveyorisinterestedincomparingthesatisfactionofpeopleinanewcommunicationsprogramandthoseinatraditionalone.Thesurveyorhasanideathatpeoplewhohaveparticipatedinthetraditionalprogramforatleast3yearswillbelesssatisfiedthanparticipantswhohavejoinedmorerecently.Thatis,lengthofparticipation,calledacovariate,canaffectsatisfactioneventhoughitisnotrelatedtothenewprogram.ThesurveyorcandiminishtheeffectsofthecovariatebyusingastatisticalprocedurecalledANOVA.
Hundredsofsuchusefulstatisticaltechniquesareavailableforsurveyors.Specialformsofttestsand
ANOVAscanbeusedtomeasurechangeovertime.AdependentttestmeasureschangeinasinglegroupfromTime1toTime2.Arepeated-measuresANOVAcanbeusedtodetectchangesinoneormoregroupsattwoormoretimes.
WiththeMcNemartest,eachpersonactsashisorherowncontrol,andsmallsamplesandcategoricaldataareused.Forexample,youcanuseaMcNemartesttofindoutwhetheradifferenceexistsinthenumberofstudentswhodoanddonot(thecategory)choosecareersinsoftwareengineeringafter20haveparticipatedinacounselingprogram.
ToBeorNottoBe:StatisticianorQualitativeAnalyst?Surveyorsdonothavetobestatisticianstoconductsurveys.However,theymustunderstandenoughaboutstatisticsandstatisticalmethodstoaskastatisticianforhelpandtoproperlyusethefunctionsprovidedbyonlinesurveyvendors.Surveyorsshouldalsobeabletoevaluatetheadequacyoftheanalytictechniquesrecommendedandusedbyothers.
Notallsurveydataareamenabletoroutinestatisticalanalysis.Somesurveyinformationconsistsoftheanswerstoopen-endedquestionsandrespondents’comments.Theseanswersandcommentsmaybebriefortheymaybeminiessays.Thesedataareoftencalledqualitativeandarecontrastedwiththestatisticalorquantitativedatathatresultfromtheclosedquestionsassociatedwithmostsurveys.
Qualitativedataanalysiscanbecomplexbecausetheinterpretermustfiltertheresponsethroughhisorherownunderstandingoftherespondent’sintentions.Qualitativeanalysisisoftendonebytwoormorepeopletoeliminateanysingleinterpreter’sperspectiveandcomeupwiththetruth.Largesurveyswithmanycommentscanusequalitativeanalysissoftwaretohelp.
ContentAnalysis,Open-EndedResponses,andCommentsContentanalysisisamethodofanalyzingqualitativedataforthepurposeofdrawinginferencesaboutthemeaningofrecordedinformationsuchastheopen-endedresponsesandcommentsmadebysurveyrespondents.Forthesurveyor,itmeanscarefullyreadingthecommentstoidentifythepresenceofcertainwords,concepts,themes,phrases,characters,orsentencesinordertoquantifythem.Acontentanalysisoftheresultsofahypotheticalexampleofasurveytofindoutaboutthecomparativeeffectsofteasingboysandgirlsmighttakethefollowingform.
Example:HypotheticalContentAnalysis:TeasingBoysandGirlsAllparticipantswereaskedaseriesofquestions,including“Whatwereyouteasedabout?”“Howdidyoufeelaboutbeingteased?”“Howdidyourespondtobeingteased?”and“Didyouchangeanythingaboutyourappearanceasaresultofbeingteased?Ifyes,whatdidyouchange?”
Theresponsesweredividedbysex.Responseswithineachsexwerearrangedbyquestionnumber(e.g.,responsestothequestion,“Howdidyoufeelaboutbeingteased?”werearrangedtogetherforanalysis).Thethemesthatemergedfromresponsestoaspecificquestionwereexaminedtogetherandcomparedbetweenthetwogroups.Differencesinsubstancewerenotedtoassesspossibledifferencesbetweenthegroups.Asthecontentemergedfromthedata,participants’commentsonspecificissueswereincludedtosupportthefindings.
Girlsfrequentlyreportednegativeopinionsandattitudestowardbeingteased(41.3%),followedbynegativeemotions(28.3%),residualeffects(19.6%),socialaspectsofteasing(8.7%),andactiontakeninresponsetoteasing(2.2%).
Boysmorefrequentlythangirlsreportedsocialaspectsofteasing.Theirexperiencesweredescribedasplayfuliftheywereteasedbyfriends.Forexample,oneparticipantstated,“Itdepends;ifit’smyfriend,thenItookitasajoke,butifit’snotmyfriend,Ithinkit’smeanandhurtful.”Anothersaid,“Withmyfriends,IamfinewithbeingteasedbecauseIteasedthem.”
PuttingtheCartinFrontoftheHorse:SelectingAnalysisMethodsTheappropriateanalysismethodforsurveydatadependsonsamplesize,thesurvey’sresearchdesign,andthecharacteristicsandqualityofthedata.Ifsomeoneasks,aspeoplesometimesdo,“Whatisthebestmethodforanalyzingsurveydata?”replywithsevenquestionsthatmustbeansweredfirst.
SevenQuestionstoAnswerBeforeChoosinganAnalysisMethodQuestion1.Howmanypeopleareyousurveying?Samplesizeisanimportantconsiderationinselectingananalyticstrategy.Somestatisticalmethodsdependonrelativelylargersamplesthandosomeothermethods.Lookatthesesamples:
1. All500teachersintheschooldistrict2. Astratifiedsampleof50teachersintheschooldistrict.Thestrataordivisionsaremenandwomen,
aswellashighschool,juniorhighschool,andelementaryschoolteachers.3. SixteachersatHartSeniorHighSchoolandfiveatJamesSeniorHighSchool
Askthefollowing:Doyouhavesufficientdatatocomputemorethantallies,averages,andthevariation?YouwouldforExampleA,butyouprobablycouldnotforExampleC.
WhataboutExampleB?Supposeyouweresurveyingapopulationofteachersofwhomabout60%werewomenand40%weremen.Ifyoutookasampleof50andyourstratificationmethodwasasuccess,youwouldhave30(60%women×50)womeninonegroupand20(40%×50)menintheother.Accordingtothestatisticaltheoryonwhichthettestisbased,groupsof30and20arejustaboutlargeenough.
Considerthefollowingtable.
Ifyouwanttocomparemenandwomenteachersinelementaryschool,youwillhaveonly2menand15womenforyouranalysis(althoughforthetotalyouwillhave20menand30women).Thisleadstothesecondofthesevenquestionstoaskwhenselectinganalysismethods.
Question2.Areyoulookingforrelationshipsorassociations?
Example:RelationshipsandAssocia-tions
1. Whatistherelationshipbetweenvoters’politicalviews(asexpressedonthePoliticalViewSurvey)andnumberofyearsofformalschooling?ThePoliticalViewSurveygivescontinuousscores
rangingfrom10(liberal)to100(conservative).2. ArehighscoresonthePoliticalViewSurveyassociatedwithhighscoresontheAuthoritarian-
LibertarianInventory?Howaboutmediumandlowscores?
Inthefirstexample,thedegreeofassociationbetweenthetwovariables,politicalviewscoresandnumberofyearsofformalschooling,canbecomputedbyusingthePearsonproduct-momentcorrelationbecausecontinuousdataareavailable.Forthesecondexample,withranksofhigh,medium,andlow,theSpearmanrankordercorrelationisappropriate.
Question3.Willyoubecomparinggroups?
Example:ComparingGroups
1. FifteenteachersatHartSeniorHighwillbecomparedwith20atJamesSeniorHigh.2. Atotalof100maleandfemalesteacherfromthedistrict’shighschools,juniorhighschools,and
elementaryschoolswillbesurveyedinDecemberandJuneandcomparedatbothtimes.
Ifyouarecomparinggroups,statisticaltestssuchasthechi-square,ttest,ANOVA,oddsratioorrelativerisks,andMcNemartestscanhelpyoudecidewhetheranyobserveddifferencesareduetosomerealoccurrenceorchance.
Question4.Willyoursurveybeconductedonceorseveraltimes?Thisquestionistofindoutwhetherthesurveydesigniscross-sectionalorlongitudinal.
Example:LookingforChange
1. All500teacherswillbesurveyedinDecember.2. Asampleof100teacherswillbesurveyedinDecemberandagaininJune.
Figure6.2isawaytodisplaytheresultsofsurveysconductedseveraltimes.
Thegraphshowsthatin2002,GroupBhadlowerscoresthanGroupA.By2004,GroupBhadpassedGroupA.Bothgroupsincreasedtheirscoresovertime.Arethedifferencesbetweengroupssignificant?Arethedifferencesovertimesignificant?Thegraphcannottellus.Forstatisticalsignificance,youneedstatisticalmethods.
Figure6.2SurveyResults
Hereisanotherwaytodisplaychangesovertimeusingthesamedata:
Thislinegraphenablesyoutovisuallyinspecttheslopeatwhichthescoreschanged.Butlikethebargraph,youcannottelljustfromlookingatthegraphwhether,andtowhatextent,thechangesare
significant.
Question5.Arethedatarecordedasnumbersandpercentagesorscoresandaverages?Lookatthis:
1. Onesetofsurveydataconsistsofthenumberandpercentageofteacherswhoagreeordisagreewithstatementsaboutthedistrict’seducationalpolicies.
2. Onesetofdataconsistsofscoresonameasureofattitudes(liberalvs.conservative)tovariousapproachestowardeducation.
Figure6.3ChangesinAttitudesOverTime
Surveydatacanbeinnumbersandpercentages(categoricaldata)orscoresthatareamenabletothecomputationofaverages(continuousdata).Theanalytictechniquesyouusewitheacharedifferent.LookatTable6.5andTable6.6.
Comparingnumbersofpeoplewhowereacceptedintolawschoolfromdifferentprograms(categoricaldata)andcomparingthemeanscoresonanattitudeinventory(continuousdata)requiredifferentanalysistechniques.Thefirstcomparisonmightusethechi-square,forexample,whereasthesecondmightuseattest.
Question6.Howmanyindependentanddependentvariablesinterestyou?
Avariableisacharacteristicofinteresttoyoursurvey.Healthstatus,attitudetowardschool,self-efficacy,group(experimental,control),gender,jobsatisfaction,andqualityoflifeareallvariables.Independentvariablesareusedtopredictorexplainfindings.Theyusuallyincludedemographics(suchasage,ethnicity,gender,income,andwhethertherespondentisinanexperimentaloracontrolgroup).Dependentvariablesarewhatyouarelookingforinyoursurvey:knowledge,attitudes,andbehavior.
Supposeyouwanttoknowwhethermenandwomendifferintheirqualityoflife.Theindependentvariableisgender(menandwomen),andthedependentvariableisqualityoflife.Ifyouwanttocomparemenandwomenintheirqualityoflifeandhealthstatus,youwillhaveoneindependentvariable(gender)andtwodependentvariables(qualityoflifeandhealthstatus).Youaddanindependentvariablewhenyouaskthis:Howdomenandwomenintheexperimentalandcontrolteamscompareintheirqualityoflifeandhealthstatus?Theaddedindependentvariableis“team”(experimentalandcontrol).
Youneedtoknowthenumberofindependentanddependentvariablestoselectanappropriateanalysismethod.Considerthefollowingdialogue:
Surveyor:Iwanttocomparetwogroupsofcollegeseniorstofindouttheirlifestylepreferences.Therearemorethan300ineachgroup.
Analyst:Whatdoyoumeanbylifestylepreferences?Areyouplanningtocomparenumbersofseniorswhoprefercertainstylesoflife—say,ruralversusurban?Orareyoucomparingtheirscoresonasurvey?
Surveyor:SeniorsinbothgroupshavecompletedLIFEQUEST,a100-itemsurvey.Itproducesscoresfrom1to100.Higherscoresrepresentmorespirituallifestyles,whereasloweronesmeanmorematerialones.
Analyst:Ifyouhavescoresonacontinuum,youhavecontinuousdata.Withtwogroupsandcontinuousdata,attestsoundsright.
Surveyor:WhatmethodwouldbecorrectifIdecidedtocomparenumbers?
Analyst:Probablyachi-square.
Surveyor:SupposeIhadmorethantwogroupsandcontinuousdata?
Analyst:WemightconsiderANOVA.
Question7.Arethedataofhighquality?
Dataareofhighqualityifthesurveyisreliableandvalid,administeredtotherightnumberoftherightpeople,andrecordedaccurately.Ifthedataareincompleteanduntruthful,itprobablydoesnotmatterwhichanalysismethodyouusebecausetheresultswillnotmeanmuch.
DataManagementWhatisthebestwaytomakesurethedataarecomplete,reliable,andreadyforanalysis,interpretation,andreporting?Answeringthisquestionmeanslearningaboutdataorganizationormanagement.
Organizingormanagingsurveydataisanessentialpartofdataanalysis.Itisthepartoftheanalyticprocessinwhichdataareenteredintoadatabasesothatyoucangetresults.Butbeforedatacanbeentered,youhavetodecidehowtopreparethemsothatyougettheinformationyouneedinauseableform.
CreatingaCodeBookorOperationsManualThecodesaretheunitsthat“speak”tothestatisticalsoftware.Suppose5,200collegestudentscompleteanalcoholusesurveycalledtheCollegeAlcohol-RelatedProblemsSurvey(CARPS).Onequestionofconcerntoyouishowmanymalesreportdrinkingfivedrinksatonesittingeveryday.Tofindthisout,youhavetotellthecomputerwhichvariablestolookfor(e.g.,genderandquantityandfrequencyofalcoholuse).Youalsohavetotellthecomputerhowtodistinguishmalesfromfemalesandhowtocomeupwithanumberthatrepresentshowmucheachpersondrinksandhowoften.
Computerprogramsreadaboutvariablessuchasgenderandquantity/frequencythroughcodes.LookatthisexcerptfromtheCARPS.
Example:ExcerptFromtheCARPS,aSurveytoDetectBingeDrinkinginCollegeStudents
1. Areyoumaleorfemale?[SEX]Male □0Female □1
2. Howofteninthepast12monthshaveyouhadfourormoredrinksofalcoholatonesitting?[QFDRINK]
Chooseoneanswer
Dailyoralmostdaily □1Fourorfivetimesaweek □2Twoorthreetimesaweek □3Twotofourtimesamonth □4Onetimeamonthorless □5Never □0
Thecodesarethenumberstotherightandjustbelowtheresponseboxes.Youusestatisticalprogramstotellyouhowmanypeoplewhoanswered0toQuestion1alsoanswered1toQuestion2.Todothis,thecomputermustbetoldthenameofthevariables(Question1=SEXandQuestion2=QFDRINK)andtheirvalues(0=maleand1=femaleand1=dailyoralmostdailyto0=never).
The“words”inbrackets(SEXandQFDRINK)correspondtothevariablesrepresentedbyeachquestion.Allcodebookscontaindescriptionsofthequestions,codes,andvariablesassociatedwithasurvey.Thecodebookmustbeassembledfortheanalysis.LookatthisexcerptfromthecodebookfortheCARPS.
Example:ExcerptFromtheCodeBookfortheCARPS,aSurveytoDetectBingeDrinking
Allvariablesarebrokendowntodiscreteunitscalledvalues,whichcorrespondtothecodesforthatvariable.Forinstance,thefrequencywithwhichapersonfeelsguiltyorsorryforsomethingheorshedidbecauseofalcoholusehasfivevalues:1=dailyoralmostdaily;2=atleastonceaweekbutlessthandaily;3=atleastonceamonthbutlessthanweekly;4=lessthanonceamonth;0=never;leaveblankifnodata.Thecodesare4,3,2,1,and0.Ifnoinformationfromtherespondentisavailable,nothingisfilledin.
Althoughstatisticalprogramsvaryinterminology,theyarefairlyconsistentinrequiringvariablestobenamedincapitalletters(PROJIDorGENDER)andtoavoidtheuseofspecialcharacters,suchascommasorsemicolons.Someprogramshavelimitsonthenumberofcharacters(usuallyabouteight)thatyoucanuse.Variablelabelsaretheactualnamesofthevariable(e.g.,“frequencyoffeelingguiltyorsorry”istheactualnameofthevariablenamedFGUILT).Tounderstandyourdata,thecomputerneedstoknowthenamesofthevariables(suchasFGUILT),thevariablelabels(frequencyoffeelingguiltyorsorry),andvaluelabels(dailyoralmostdaily=1;atleastonceaweekbutlessthandaily=2,etc.).(Pleasenote:Yourstatisticalprogrammayuseslightlydifferentterms,buttheideawillbeexactlythesame.)
Inlargesurveyprojects,thecodebookistheproject’sofficialrecord.Thecodebookcontainsvariablenames,labels,andvaluesorcodes;surveymethodsandcharacteristics;findings;andthesurveyinstrument(includingscoringsystem,ifrelevant)andsurveyteammembers.Ifyouoranyofyourcolleaguesandassociateseverwanttousethedatafromthesurvey(say,todoadditionalanalyses)or
repeatthesurvey(say,onadifferentsample),youwillfindthecodebooktobeessential.Becauseoftheirdetail,thistypeofcodebookissometimesreferredtoasanoperationsmanual.
Samplecodebooksintheirentiretycanbefoundontheweb.Tofindthem,gotoyourfavoritesearchengineandenter“surveysANDcodebook”or“+survey+codebook.”
EstablishingReliableCodingToensurereliabledatainsmallsurveys—say,withjustonepersondoingthecoding—youshouldrecodeallorasampleofthedatatocheckforconsistency.Thesecondcodingshouldtakeplaceaboutaweekafterthefirstcoding.Thisisenoughtimeforthecodertoforgetthefirstsetofcodessothattheyarenotjustautomaticallyreproduced.Afterthedataarecodedasecondtime,thetwosetsofcodesshouldbecompared.
Inlargeorverylargesurveys,asecondpersonshouldindependentlycodeasampleofthedata.Toensurereliabilitybetweencoders,providedefinitionsofalltermsandformallytrainallcoders.
Despiteyourbesteffortsatsettingupahigh-qualitycodebookanddatamanagementsystem,thecodersmaynotalwaysagreewithoneanother.Tofindoutabouttheextentoftheiragreement—intercoderorinterraterreliability—youcancalculateastatisticcalledthekappa.Kappameasureshowmuchbetterthanchancetheagreementisbetweenapairofcoders.Hereistheprinciplebehindthekappa.
MeasuringAgreement:TheKappaSupposetworeviewersareaskedtoindependentlyreview100interviewswithmembersofacommunitywhoareabouttovoteonwhethertheywantanewmalltobebuilt.Thereviewersaretostudythetranscriptsoftheinterviewstofindouthowmanypeoplementiondisplacementofsingle-ownerstoresbynationalchainsduringthediscussion.Thereviewersareaskedtocode1for“no”ifapersondoesnotmentiondisplacementatleastonceand2for“yes”ifheorshedoesmentiondisplacement.Thefollowingtableshowsthereviewers’codes.
Reviewer1saysthat30(A)ofthe100interviewsdonotcontainreferencetodisplacementofsingle-ownerstores,whereasReviewer2saysthat35(B)donot.Thetworeviewersagreethat20(C)studiesdonotincludementionofdisplacement.
Whatisthebestwaytodescribetheextentofagreementbetweenthereviewers?Twentyof100or20%(C)isprobablytoolowbecausethereviewersalsoagreethat55%(D)ofstudiesincludementionofdisplacement.Thetotalagreement(55%+20%)isanoverestimatebecause,withonlytwocategories(yesandno),someagreementmayoccurbychance.
ThisisshowninthefollowingformulainwhichOistheobservedagreementandCisthechanceagreement.
MeasuringAgreementBetweenTwoCoders:TheKappa(κ)Statisticκ=O−C(Agreementbeyondchance)1−C(Agreementpossiblebeyondchance)
Hereishowtheformulaworkswiththepreviousexample.
1. CalculatehowmanyinterviewsthereviewersmayagreethatbychanceDONOTincludementionofdisplacementofsmallstores.Thisisdonebymultiplyingthenumberofnoanswersanddividingby100becausethereare100interviews:30×35/100=10.5.
2. CalculatehowmanyinterviewstheymayagreethatbychanceDOincludementionofdisplacementbymultiplyingthenumberofinterviewsthateachreviewerfoundtoincludemention.Thisisdonebymultiplyingthenumberofyesanswersanddividingby100:70×65/100=40.5.
3. Addthetwonumbersobtainedinsteps1and2anddivideby100togetaproportionforchanceagreement:(10.5+45.5)/100=0.56.
Theobservedagreementis20%+55%=75%or0.75.Therefore,theagreementbeyondchanceis0.75−0.56=0.19,thenumerator.
Theagreementpossiblebeyondchanceis100%minusthechanceagreementof56%or1−0.56=0.44,thedenominator.k=0.190.44k=0.43
Whatisa“high”kappa?Someexpertshaveattachedthefollowingqualitativetermstokappas:0.00−0.20=slight;0.20−0.40=fair;0.40−0.60=moderate;0.60−0.80=substantial;and0.80−0.10=almostperfect.
Howdoyouachievesubstantialoralmostperfectagreement—reliability—amongreviewers?Youdothisbymakingcertainthatallreviewerscollectandrecorddataonexactlythesametopicsandthattheyagreeinadvanceonwhateachimportantvariablemeans.The“fair”kappaof0.43obtainedbythepreviousreviewerscanbeduetodifferencesbetweenthereviewers’andsurveyors’definitions,poortrainingintheuseofthedefinitions,andmistakesincoding.
ReviewingSurveysforMissingDataMissingdataresultfromunansweredquestionswithinasurveyorfailuretocompletethesurveyatall.Youshouldreviewthefirstcompletedsurveysassoonasyoureceivethem,beforeyouenteranydata.Inself-administered,writtensurveys,especiallymailedquestionnaires,expecttofindunansweredquestions.Respondentsmaynotanswerquestionsbecausetheydonotwantto,theymissthequestions(donotseethem),ortheydonotunderstandthem.Peoplemaynotunderstandthedirectionsforcompletingthesurveyorthequestionsbecausetheamountofreadingistoogreatorthereadinglevelistoohigh.Respondentsmaybeunsureofhowtorespond(completelyfillintheboxes,circlethecorrectanswer,etc.).Theymayfindtheformatdifficulttouse.
Confusingquestionformatsleadtomissingdatabecauserespondentsdonotknowhowtoanswerthe
question.Extensivecognitivepretestsandpilottestingmayovercomesomeproblemsthatcauserespondentstomisunderstandsurveyquestions.Cognitivepretestsareinterviewswithpotentialrespondentsthataskthemtointerpreteachquestionandresponse.Pilottestsaretestsofthesurveyintheplannedsurveysetting.Thesetwoactivitieswilltellyou,beforeyougotothefield,thataparticularformatisunusableorthatsomequestionsdonotmakesense.Pleasenotethatyoushouldpilottestallsurveytypes,includingcommerciallyproducedonlinesurveys.Thequestionformatmaybeinappropriateformanyrespondents,thesurveymaycontainseeminglyirrelevantquestions,thesitemaybedifficulttonavigate,andsoon.Itisaxiomatic,however,thatnomatterwhatyoudo,youcanexpectsomemissingdata:Thatisasurething.
Amajorproblemfacedbysurveyorsishowtohandlemissingdata.Sayyoumail100surveysandget95back.Youproudlyannouncethatyouhavea95%responserate.Aftercloserexamination,youdiscoverthathalftherespondentsdidnotanswerQuestion5andthatnot1ofthe25surveyquestionswascompletedbyeveryrespondent.Withallthatmissinginformation,youcannotreallyclaimtohavea95%responserateforallquestions.
Whatshouldbedoneaboutmissingresponses?Insomecases,youmaybeabletogobacktotherespondentsandaskthemtoanswerthequestionstheyomitted.Insmallsurveys,wheretherespondentsareknown—say,inanofficesettingorwithinaschool—therespondentsmaybeeasilycontacted.Butcollectinginformationasecondtimeisusuallyimpractical,ifnotimpossible,inmostsurveys.Somesurveysareanonymous,andyoudonotevenknowwhotherespondentsare.Ininstitutionalsettings,youmayhavetogobacktotheInstitutionalReviewBoardtogetpermissiontocontacttherespondentsasecondtime.Thisprocessmaytaketimeanddelayyoursurvey’scompletion.Onceagain,statisticalmethodsareavailablethatcanbeusedtoaccountformissingdata.Somearebasedonassumptionsabouthowtheaveragerespondentwouldhaveresponded.
Onlinesurveyscanbeprogrammedsothattherespondentmustansweronequestionbeforeproceedingtothenext.Somerespondentsmayfindthisapproachfrustrating,however,andrefusetocompletethesurvey.Althoughcompellingrespondentstoanswerallquestionsistoutedasamajoradvantageofonlineandothersurveys,somesurveyorsthinkthatforcingrespondentstoanswereveryquestioniscoerciveandunethical.Thesesurveyorsarguethat,inmanysurveys,participationisvoluntary;respondentscanquitwhentheywant.Asurveythatforcespeopletoansweraquestioneveniftheyprefernottomaybeconstruedbysomepeopleasviolatingtheethicalprincipleofautonomyorrespectforindividuals.Moreover,thesurveymayresultinunreliableinformationbecausesomepeoplemayenterameaninglessanswerjusttobeabletomoveontothenextquestion.
EnteringtheDataDataentryistheprocessofgettingthesurvey’sresponsestothestatisticalsoftwareinareadableformat.Fromthere,itgoestoadatabase.Itusuallytakesthreeforms.Inthefirst,datafromacodedsurveyareenteredintoadatabasemanagementprogramorspreadsheet.ThedataarethensavedintextorASCIIfilessotheycanbeexportedintoastatisticalprogramsuchasSPSS,SAS,orStata.AsecondtypeofdataentryinvolvesenteringdatadirectlyintoastatisticalprogramsuchasSPSS,SAS,orStata.Inthethirdformofdataentry,therespondentorinterviewerentersresponsesdirectlyintothecomputer.Dataentryofthistypeisassociatedwithcomputer-assistedinterviewing,onlinesurveys,andscannedsurveys.Theresponsesareautomaticallyenteredintodatabasemanagementsystemsorstatisticalprograms.Programsarealsoavailablethatwillautomaticallyconvertonefileformatintoanother(say,fromSAStoStata).
Eachdataentry,databasemanagement,andstatisticalprogramhasitsownconventionsandterminology.Someprogramswilltellyouthatwhenenteringdatayouaresettinguprecordsforeachrespondent.Therecordconsistsoftheperson’suniqueID(identificationcode)andtheperson’sobservations(responsechoices,scores,comments,etc.).Otherprogramsconsidertheunitofanalysis(suchastheperson)astheobservationandthedatacollectedoneachobservationasthevariablesorfields.Theexamplethatfollowsisasimpledatasetforsixpeople.
Example:SurveyResponsesFromSixPeople
Inthisexample,thetableisorganizedsothattherowsareforeachpersonandthecolumnsaretheirdata.Thatis,Person2’sdataare2,4,1,3,and2.RespIDisanabbreviationfor“respondentidentificationnumber.”Manystatisticalprogramsrequireyoutotellthecomputerwhereonthedatalineavariableislocated.Forinstance,intheexample,theperson’sgenderiscalledGENDER,anddataongendercanbefoundincolumn2.BingedrinkingiscalledBINGE,andthedataforthisvariablecanbefoundincolumn5.
Databasemanagementprograms,statisticalprograms,andcomputer-assistedsurveyswithautomaticdataentrycanfacilitateaccuracybybeingprogrammedtoallowtheentryofonlylegalcodes.Forinstance,ifthecodesshouldbeenteredas001−010,thenyoucanwriterulessothatanentryof01or10isnotpermitted.Ifyoutrytoenter01or10,youwillgetanerrormessage.Withminimumprogramming,theprogramcanalsocheckeachentrytoensurethatitisconsistentwithpreviouslyentereddataandthatskippatternsarerespected.Thatis,theprogramcanmakesurethatthefieldsforquestionsthataretobeskippedbysomerespondentsarecodedasskipsandnotasmissingdata.Designingacomputer-assistedprotocolrequiresskillandtime.Noprotocolshouldberegardedaserror-freeuntilithasbeentestedandretestedinthefield.
CleaningtheDataOncethedataareentered,theyneedtobecleaned.Acleandatasetcanbeusedbyanyonetogetthesameresultsyoudowhenyouruntheanalysis.Databecome“dirty”foranumberofreasons,includingmiscoding,incorrectdataentry,andmissingresponses.
Onegoodwaytoavoiddirtydataistorelyonscannedoronlinesurveys.Ifyouenterdatabyhand,however,makesurethatcodersordataenterersareexperienced,welltrained,andsupervised.
Runfrequenciesonyourdataassoonasyouhaveabout10%oftheresponsesin.Runthemagainand
againuntilyouaresurethatthesurveyisrunningsmoothly.Frequenciesaretabulationsoftheresponsestoeachsurveyquestion.Ifyourdatasetisrelativelysmall,youcanvisuallyscanthefrequenciesforerrors.Forlargedatabaseswithmanytablesorrecords,variables,skippatterns,andopen-endedtextresponses,youmayneedtodoasystematiccomputerizedcheck.Allleadingstatisticalprogramsprovideforcleaningspecificationsthatcanbeusedduringdataentryandlaterasaseparatecleaningprocess.
Severalotherproblemsmayrequireyoutocleanupthedata.Theseincludehavingtodealwiththecompleteabsenceofdatabecausesomesurveyshavenotbeenreturned,missingdatafromsurveysthathavebeenreturned,andsurveysthatcontaindatathatareverydifferentformtheaveragerespondents.
Tominimizelowresponserates,makesurethesurveyismeaningfultotheintendedresponders.Isthetopicimportantenoughsothattheyfeelcompelledtoanswerthequestions?Besuretoprovetotherespondentthatyouwillrespecttherulesofconfidentiality.Checkthesurvey’sreadabilitylevel,format,andaptnessofthelanguagethroughcognitivepretestingandscientifictranslation.Ifthereadinglevelistoohigh,theformatiscluttered,orthetranslationispoor,thenyoucancountonpeopletoignorethesurvey.Ifyouareconductinganinterview,spendthetimeandmoneytotrainandmonitorinterviewers.Youcanlosepeopleiftheydonotunderstandorlikethesurveyorinterviewer.Itoftenhelpstoofferfinancialorotherincentives(suchasgiftsorgiftcertificates)toencouragerespondentstorespond.
ValidatingSurveyDataOnlinesurveysoftwarealmostalwaysletsyouvalidatetheresponses.Forexample,ifaquestionasksforadate,youcandecideonhowthatdateisexpressed.Supposeyoursurveyasksforaperson’sbirthdate.Youcanprogramtheresponseoptionsothattherespondenthasnochoicebuttorecordthedataasmonth,day,andyear.Individualswhouseanothersystem,say,day,month,andyear,willnotbeallowedtorecordtheirresponse.Theywilleithergetamessagesuchas“invalidresponse,”theircomputerwillgiveawarningsound,orboth.Thevalidationfeaturealsoguaranteestheuniformityofresponsesthatisessentialtoquantitativeanalysis.
Accesstothevalidationfeatureusuallyrequiresthatyoupayforahigherlevelofthesurveysoftwarethanjustthebasic,sometimesfree,version.
Somevalidationinstructionsandoptionscommonlyfoundinonlinesurveysoftwareincludethefollowing:
don’tvalidatecommenttextmustbeaspecificlengthmustbeawholenumbermustbeadecimalnumbermustbeadate(MM/DD/YYYY)mustbeadate(DD/MM/YYYY)mustbeanemailaddress
SummingUpSurveydataanalysisusesstatisticalandqualitativemethodstodescribe,compare,andinterpret
respondents’attitudes,preferences,values,andbehavior.Whenchoosinganappropriateanalysismethodforyoursurvey,youmustconsideranumberoffactorssimultaneously.Thesefactorsarethesurvey’ssamplesizeandresearchdesign,aswellasthecharacteristicsandthequalityofthedata.Tochooseamethod,answerthesesevenquestions:
Howmanypeopleareyousurveying?Areyoulookingforrelationshipsorassociations?Willyoubecomparinggroups?Willyoursurveybeconductedonceorseveraltimes?Arethedatarecordedasnumbersandpercentagesorscoresandaverages?Howmanyindependentanddependentvariablesinterestyou?Arethedataofhighquality?Thatis,aretheyreliableandvalid?
Contentanalysisisusedtoanalyzequalitativedataforthepurposeofdrawinginferencesaboutthemeaningofrecordedinformationsuchastheopen-endedresponsesandcommentsmadebysurveyrespondents.Allsurveyreturnsshouldbereviewedformissingdataandaplanshouldbedevisedforhandlingthem.Missingdataresultfromunansweredquestionsorentiresurveys.Dataentryistheprocessofgettingthesurvey’sresponsesfromtherespondenttothestatisticalsoftwareinareadableformatsotheycanbeputintothesurvey’sdatabase.Onlinesurveydatacanbeautomaticallyenteredintoadatabase,withtheluxuryofreceivingreportsinrealtime.Surveydataalsomaybescannedinorhandentered.Regardlessofhowthedatagetintodatabases,theresponsesneedtobecheckedforaccuracy.Acleandatasetisonethatcanbeusedbyanyonetogetresultsthatareidenticaltoyours.Onlinesurveysoftwareusuallyhasavalidationoptionthathelpsstandardizedatacollectionandmakessurethattheinformationthatrespondentsprovideistheinformationaskedforbythesurvey.
ThinkAboutThis1. Findthethreearticleslistedbelow.Nameanddiscussthedata-analyticmethodsthatareusedto
interpretthesurveyresultsdiscussedineacharticle.YoucanlocatethearticlesthroughPubMed(http://www.ncbi.nlm.nih.gov/)throughanysearchenginebyenteringthenameofthejournal.
Article1.Mazzucato,V.,Cebotari,V.,Veale,A.,White,A.,Grassi,M.,&Vivet,J.(2014).Internationalparentalmigrationandthepsychologicalwell-beingofchildreninGhana,Nigeria,andAngola.SocialScience&Medicine,132(2),215–224.
Article2.O’Connor,T.M.,Yang,S.-J.,&Nicklas,T.A.(2006).Beverageintakeamongpreschoolchildrenanditseffectonweightstatus.Pediatrics,118(4),e1010–e1018.
Article3.Richardson,J.,Coid,J.,Petruckevitch,A.,Chung,W.S.,Moorey,S.,&Feder,G.(2002).Identifyingdomesticviolence:Crosssectionalstudyinprimarycare.BritishMedicalJournal,324(7332),274–278.
2. AnswerthesequestionsabouteachofthetwosurveyslistedinQuestion1(oranyothers):Howmanypeopleweresurveyed?Didthesurveyinvestigatorsdeterminerelationshipsormakepredictions?
Didtheinvestigatorscomparegroups?Wasthesurveytheydiscussedconductedonceorseveraltimes?Werethedatanominal?Continuous?Both?Howmanyindependentanddependentvariablesdidthesurveyaddress?Werethedataofhighquality?Thatis,didthesurveyorsdemonstratethattheywerereliableandvalid?
7PRESENTINGTHESURVEYRESULTS
OverviewSurveyresultscanbeshownonthesurveyformandintables,graphs,diagrams,orpictures.Surveyreportsoftenincludesummarytables.Theirpurposeistodescriberespondentsandshowrelationshipsandchanges.Followtherulesifyouplantousetables.Forexample,usethetable’scolumnstoshowtheindependentvariables,suchasgroupandtimingofthesurvey(e.g.,baseline,6monthslater,and3yearslater).Selectatableformatanduseitconsistently,relyonverticallinessparingly,andpresentdatainalogicalorder,suchasfrommosttoleastfrequent.
Piediagramsshowtheproportionoccupiedbyeachresponsecategory.Eachsliceofthepiemustbeequalinproportiontothenumberorpercentageofresponsesitrepresents.
Bargraphsarecommonlyusedtodisplaysurveydatabecausetheyprovideanoverviewofseveralkindsofinformation—comparisonsandchanges—atoneglance.Linegraphsalsoshowchangesanddifferencesingroups.Becarefulnottooversellandmakeaone-pointchangeinscorelooksignificantunlessitreallyis.
Manysurveysresultinwrittenreports.Considerincludingthesesectionsinyours:abstract,summary,tableofcontents,figures,glossaryofterms,statementofpurposesorobjectives,methods,results,conclusions,recommendations,references,index,appendix,andacknowledgments.
Slidepresentationsmaybethebestchoiceifyouplantomakelast-minutechangesandifyouwantimmediatefeedbackfromtheaudience.Anotheroptionistheposter,whichprovidesreaderswithacondensedandvisualexplanationofthesurvey’sobjectives,methods,results,andfindings.
ReproducingtheQuestionnaireSurveyresultscanbeshownonthesurveyformorintables,graphs,diagrams,orpictures.Lookatthis.
Example:ReportingResultsWiththeQuestionnaireSource:SurveyMonkey®
Inthepast4weeks,howmuchofthetimehaveyoufeltdepressed?
Theadvantageofusingthesurveyformtogiveresultsistoletthereaderorlistenerseethequestionsandresponsechoices.Thisapproachshowsthesurvey’srawdata.Itprovidesthedatawithoutanyanalysisorinterpretation.
Onlinesurveysystemsprovideinformationonareal-timebasis.Asresponsescomein,thesystemautomaticallygivesyouinformation,suchasthenumberofpeoplewhocompletedthesurveysofarandhowtheresponsesaddup.HereisanexamplefromQuestionPro.
Example:OnlineSurveyStatisticsinRealTime
UsingTablesSurveyreportsnearlyalwayshavetables.Theirpurposeistodescriberespondentsandshowrelationshipsandchanges.Lookattheseshelltables.
Example:ShellTableDescribingChildreninTwoSchools
Example:ShellTableforComparingChildreninTwoSchools
Thefirsttableprovidestheformatforlistingthepercentageofchildrenintwoschoolswhohavecertaincharacteristics(suchasageandgradepointaverage).Thesecondtableletsyouputinstatisticsusedtotestwhetherthedifferencesaremeaningful.
Table7.1isanotherexamplethatshowsstatisticallysignificantdifferences(indicatedbyoneandtwoasterisks).Thistablealsohasasourcelisted:TheNationalChildren’sHealthSurvey,CenterforHealthStatistics.Includethesourceoftheinformationwhenitisnotobvious.Thesourceisnotobviousifitcomesfromanyonebutthereporter.Notice,too,thatthescoresareexplainedinthetablefootnote.
Source:Self-administeredsurveyadministeredbyTheNationalChildren’sHealthSurvey,CenterforHealthStatistics.
aScoresof75to100indicatehealthylifestyles.Scoresof26to74indicateneitherhealthynorunhealthylifestyles.Scoresof1to25indicateunhealthylifestyles.
*p=.003betweenhealthyandunhealthywithinschool;
**p=.002betweenhealthyandunhealthywithinschool.
LookatTables7.2and7.3.
Whichtabletellsyoumore,Table7.2orTable7.3?Iftheemphasisofthesurveyisonchangesbetweengroups(youngerthan65yearsvs.65yearsandolder),thenTable7.2isbetter.Ifyouwanttoemphasizethechangesthattookplaceovera1-yearperiod,thenTable7.3isbetter.
Source:TheSelf-EfficacyScale(NewYork:NationalPress)Note:Scaleis1to9,withscoresof1to3=high;scoresof7to9=low.
Source:TheSelf-EfficacyScale(NewYork:NationalPress).Note:Scaleis1to9,withscoresof1to3=high;scoresof7to9=low.
SomeTablePreparationRules
1. Tablesdisplaycolumnsandrowsofnumbers,percentages,scores,andstatisticaltestresults.Decidehowmanycolumnsandrowsyoucanincludeandstillkeepthetablereadable.
2. Eachtableshouldhaveatitlethatsummarizesitspurposeandcontent.3. Whenthesourceofatable’sdataisnotimmediatelyobvious,itshouldbegiven.4. Whenyouuseatermthatmaybeconfusing,defineit.Setoffdefinitionswithasterisksor
superscripts.5. Columnsaretheindependentvariables,suchasgroupandtimingofthesurvey(e.g.,baseline,6
monthslater,and3yearslater).6. Selectatableformatanduseitconsistently.Thetablesinthepreviousexamplesusecaptionsin
whichthefirstletterofeachword(exceptprepositionsandarticles)inthecolumnsiscapitalized.Intherows,onlythefirstwordiscapitalized.Manypeopleuseonlyhorizontallinesintables.Useverticallinessparingly,especiallyifyoualsousehorizontallines.
7. Presentdatainsomelogicalorder.Onecommonlyusedorderisfrommostfrequenttoleastfrequent,althoughthereversemaybeappropriate,too.Theideaistobelogicalsothatthereadercanfollow.
8. Includethesamplesizeanddifferentiateamongnumbers,percentages,andotherstatistics.
Allwordprocessingprogramscanhelpyouformattablessotheyareeasytoread.Ifyouusetablesfrequently,learnhowtouseyourprogram’stablefunctions.
DrawingPieDiagramsPiediagramsshowvisuallytheproportionofthewholethateachresponsecategoryoccupies.Supposeyouwereconductingasurveyof80libraryusers’needsandwantedtodistinguishtheneedsofpeopleofdifferentages.Supposealsothatyoufoundthat40respondentswererelativelyyoung—say,between18and25yearsofage—andonly10werebetween25and35,withtheremaining30peopleolderthan45.YoucandescribeyourfindingseffectivelybypresentingthemasshowninFigure7.1.
Figure7.1LibraryUsers:N=80
Figure7.2LibraryUsers:N=80
LookatFigure7.2.Thispiecharthasthesameproportionofusersaspreviously.Bycuttingthepie,youemphasizethedifferentsizesofeachslice.
Ifyouplantousepiediagramsfrequently,youcanrelyongraphicsprograms.Spreadsheet,wordprocessing,andotherprogramsincludegraphicsfunctionsthatenableyoutodrawpiesfromtablesofdata.Thekeytoanaccuratepiediagramisitsscale.Eachslicemustbeequalinproportiontothenumberorpercentageofresponsesitrepresents.Thatis,50%ofresponsesishalfthepie,25%isonequarter,andsoon.Remembertokeeptheslicestonomorethanaboutsixorthepiewillbetoocluttered.
UsingBarGraphsBargraphsarecommonlyusedtodisplaysurveydatabecausetheyprovideanoverviewofseveralkindsofinformationatoneglance.LookatFigure7.3,agraphofchangesinbehaviorbetweenboysandgirlsfrom1985to2005.Thisgraphtellsyoutwothingsatonce:
1. Boys’andgirls’behaviorchangedovertime.2. Boyshadconsistentlyhigherscoresthandidgirls.
NowlookatFigure7.4.Thisgraphofthesameinformationconcentratesonboysandgirls,althoughyoustillgettwokindsofinformationatoneglance.
Figure7.3Boys’andGirls’BehaviorSurvey
Figure7.4Boys’andGirls’BehaviorSurvey
Figure7.5isagraphofthesameinformationfocusingonscores,andyoucanseetwokindsofdataatonce.
Bargraphsshouldalwayshaveatitle,alegendorkeytothebars,andanyotherexplanationsneededtokeeptheresultshonest.Rememberthatseeingcanbedeceiving.Justbecausethegraphsuggestsdifferencesdoesnotmeantheyarereal(significantandpractical).
Figure7.6showsthesamedatapresentedinalinegraph.Linegraphsarebetterthanbarsatshowingtheflowofchangeovertime.Mostgraphicsprogramswillallowyoutoautomaticallyswitchfromonetypeofgraphtoanothersothatyoucanactuallyseewhichonebestdescribesyoursurveydata.Theobjectiveisaccuracy.
UsingLineGraphs
Linegraphsaredrawingsthatallowyoutoshowchangesandcomparegroups.Becarefulnottooversellandmakeaone-pointchangeinascorelooksignificantunlessitreallyis.Lookatthesescores:
Figure7.5Boys’andGirls’BehaviorSurvey
Figures7.7and7.8aretwographs,eachofwhichhasbeenplottedasalinegraphtorepresentthescoresandtheirfrequency(number).Whichgraphismoreaccurate?Figure7.7isthecomputer’schoice.Figure7.8isnot.Inthesecondgraph,thedifferenceinscorescertainlylookslessimpressive.
Figure7.6Boys’andGirls’BehaviorSurvey
Figure7.7
Whichispreferred,abaroralinegraph?Ifindoubt,experiment.Toshowcomparisons,tablesandbargraphsseemtoworkbest.Forchangesovertime,linegraphswin.
Onlinesurveysystemshavetheadvantageoflettingyouquicklyexplorethebestwaytopresenttheresults.Thefollowinggivesapartialexampleoftheresultsofacompanysupportsatisfactionsurvey.Followingthatisanillustrationofanonlinesurveysystem’sreportofacustomersatisfactionsurveyandabarandlinegraphdepictingtheresults.
Figure7.8
Thesurveyorcreatedthegraphsbytransferringtheresponsesfromthesurveytoaspreadsheetprogramandusingits“wizard”function.Thebargraphisusefulinseeinghoweachratingchangedovertime.Thelinegraphisbetteratshowingmovement.Itshouldbenotedthatthegraphsareeasiertounderstandincoloronthescreenorwhenprinted.
Example:DownloadedResultsofaCustomerSatisfactionSurveySource:Takenfromwww.QuestionPro.com.
Example:TransferringtheResultsofOneQuestionIntoBarandLineGraphs
DrawingDiagramsorPicturesUsepicturesordiagramstogetyourpointacross.Supposeyouwanttodescribeyoursurvey’sresearchdesign.Comparethewordswiththediagram.
Allmajorwordprocessingprogramsandspreadsheetshavefunctionsthatallowyoutoeasilyconvertatableofdataintoachart.Ifyouuseflowchartsoften,youshouldconsiderbuyingagraphicsprogram.
Example:WordsandDiagramsinSurveyReportsWeusedadesigninwhichschoolswerethesamplingunits.Fourteenschoolswith2,000totalstudentswereeligibletoparticipateinaninjurypreventionprogram.Werandomlyselected10schoolswithatotalof945students.Fiveschoolswithatotalof443studentswererandomlyassignedtoreceivethecurriculum.Ofthefiveschoolsthatreceivedthecurriculum,425studentscompletedthebaselinesurvey.Ofthe502studentsnotreceivingthecurriculum,450completedthebaselinesurvey.Weobtained12-monthfollow-updataon400studentsinthefiveschoolsreceivingthecurriculum.Oneschoolnotreceivingthecurriculumdroppedout,andwecollected12-monthfollow-upsurveydataintheremainingfourschoolson213students.
Nowlookatthisdiagram.
Adiagramlikethisishelpfulinexplainingcomplexresearchdesigns.Thediagramreliesontheprinciplesusedtomakeanorganizationalorflowchart.
WritingtheResultsofaSurveyAfairlytypicalconclusiontoasurvey’sactivitiesisawrittenreportofitspurposes,methods,results,conclusions,andrecommendations.Onequestionalwaysseemstocomeupwhenyousitdowntoorganizethereport:Howdetailedandtechnicalshouldyoube?Ifyouaretootechnical,youmayreduceyourreadershipsubstantially;however,ignoringthetechnicaldetailsmaysubjectyourreporttocriticism.
Threeaimsshouldguideyourwriting:
1. Becomprehensive.2. Organizecarefully.3. Writeasclearlyasyoucan.
Soundlikeoneofyourearliestschoolteacherswarningyouaboutabookreport?Perhaps.Butifthesethreecriteriaareapplied,yourreportismorelikelytobereadthanotherwise.
Achievingcomprehensivenessmeansincludingasmuchaspossiblesothatanyonewhowantstocanunderstandthereport.Youmustorganizethereportsothatanyonecanfindoutwhatheorshewantstoknow.IfIaminterestedinwhyyouchoosetheXYZSelf-EfficacySurvey,Ishouldbeabletofindthatinformationeasily.Writingclearlyisalwaysadesirableaim.Insurveyreports,claritymeansusingstandardlanguageconventionsandmakingsurethatallambiguoustermsaredefined.
OrganizingtheReportConsiderincludingallthese:
AbstractSummaryTableofcontentsListoftablesandfiguresGlossaryoftermsRoadmap:WhattolookforinthereportandwheretofinditStatementofpurposesorobjectivesOutcomesMethodsResultsConclusionsRecommendationsReferencesIndexAppendixAcknowledgments
Abstract
Theabstractisusuallyabout150to250words.Considerusingastructuredabstract.Lookatthis.
Thisabstracthas152words,countingtheheadings.Yourwordprocessingprogramislikelytohaveafunctionforcountingwords.
Example:StructuredAbstractofaSurveyReportPurpose:ToidentifycommunityattitudestowardintegratingboysandmenintoWomenForward(WF),atraditionallyall-femalegrassrootsorganizationaimedatencouraginggirlsandwomentoenterpolitics
Researchdesign:Across-sectionalsurveyof150people
Survey:Fifteen-minutetelephoneinterviewsconductedduringJuly2004
Respondents:An82%responseratewasobtained.Oftherespondents,70%weremen,and62%ofrespondentswere35yearsofageoryounger.Respondentswererandomlyselectedfromlocalpoliticalorganizations.
Mainoutcome:Supportforintegration
Results:Eighty-fourmen(97.6%)and32women(86%)supportedintegration.Thesedifferencesarenotsignificant.Respondentswhoare35yearsofageandyoungeraresignificantlylesseagerthanolderrespondents(p=.04)tointegratemenintoWF.
Conclusions:AlthoughmenandwomenareaboutequalintheirsupportofintegratingmenintoWF,youngerrespondents—regardlessofgender—arelessenthusiastic.Interpretthesefindingscautiouslybecause70%ofrespondentsweremale.
Summary
Thesummaryisadistillationofallthereport’skeycomponents(objectives,methods,findings).Itshouldbeaboutthreepageslongandtake5minutestoread.Youcanprovidemoredetailthanyoudointheabstractandaddinformationonwhoconductedthesurvey.Youcanalsoincludeoneortwoimportanttablesorfigures.
Tableofcontents
Thetableofcontentsshouldlistallmajorsectionsofthereportwiththeassociatedpagenumbers.
Listoftablesandfigures
Listeachtableandfigureandgiveitscompletenameandthepagenumberonwhichitappears.Insomereports,tablesandfiguresgoattheendinaseparatesection.
Glossaryofterms
Alltechnicalterms(e.g.,randomsampling),abbreviations(e.g.,CBS=CenterforBusinessandScience),andambiguousconcepts(e.g.,healthy,smart,hostile)canbeincludedintheglossary.
Roadmap:Whattolookforinthereportandwheretofindit
Somereportsarecomplex,andthereaderneedsaroadmaptofollowthem.Forinstance,supposeyourreporthastwomajorcomponents:Part1andPart2.Tellthereaderhowtofindeachpart(Separatelybound?Clickonaword?Anicon?).Tellwhatisineach(e.g.,Part1hasadescriptionofthesurveyproject;Part2discussesthevalidation).
Statementofpurposesorobjectives
Thepurposesarethesurvey’sobjectives.Discussinthereportwhyyoudidthesurvey.Didyoudothesurveytofindouthowsatisfiedparticipantsarewiththeirbenefits?Tofindoutwhichbenefitswillsatisfythem?
Outcomes
Onwhichmaindependentvariablesoroutcomesdoesthesurveyfocus?Thesemaybechangesinknowledge,attitudes,behavior,health,orqualityoflife.
Methods
Whatwasdone?Withwhom?Wastheeffortworthwhile?Toanswerthesequestions,discussthefollowing:
Thetypeofsurvey(awritten,self-administeredquestionnaire,telephoneinterview,etc.)andthelimitationsthatresultedbecauseyouchosethattypeofsurvey.Forexample,supposethatinatelephonesurveyoftheteensinCountyX,youusetheirhomelandlinestoreachthem.Youuselandlinesbecauseyoucanconvenientlygetthosenumbersfrompublicrecords.However,almostallteensinCountyXaremorecomfortableusingtheirpersonalmobilephones.Ifyoupersistinusinglandlines,howwillthataffectthesurvey’scredibility?Surveyquestionsasked:Giveexamples;includethesurveyintheappendixifyoucannotincludeitinthetext.Surveylogistics:Duringwhichmonths,overwhatperiod,andhowfrequentlywasthesurveyadministered?Bywhomwasitadministered?Howweretheadministratorstrained?Howwasthequalityoftheadministrationmonitored?Whatarethelimitationsthatresulted?Forexample,sayyoutrainedfivepeople,butonewasnotadequate.Whatmightthishavedonetotheresults?Surveyconstruction:Givetheoriginofquestions,describepilottesting,anddiscussreliabilityandvalidity.Discussthelimitationsthatresultedduringconstruction.Forexample,thevalidationstudymayrevealthatthesurveyisnotusefulwithpeoplewhodonotreadEnglishorSpanishattheninth-gradelevel.Samplingandresponserate:Whatweretheeligibilitycriteria?Howdidyoucalculatetheresponserate?Howadequatewastheresponserate?Didyouuseincentives?Discussthelimitations.Becauseyoudidnotgetsufficientnumbersofeligiblerespondents,whatdidthisdototheresults?Forexample,supposeyouwantedhalfmenandhalfwomen,butrespondentswereprimarilyfemale?Surveyresearchdesignanditslimitations.Forexample,supposeyoursurveyusedcomparisongroupsthatwerenotrandomlyconstituted;whattypesofselectionbiascanyouanticipate?Howseriouslydoesthisaffectyourfindings?Analysis:Foreachmainsurveyoutcomeofinterest,describetheanalysismethod.Ethics:Wererespondentsproperlyinformedoftheirrightsandresponsibilities?Howwasconfidentialitymaintained?
Results
Concerningeachoutcomeofinterest,whatdidyoufind?Usuallythesearegivenasstatistics.
Conclusions
Whatdothefindingsmean?
Recommendations
Whatdoyourecommendbasedonthedata?Warning:Noteverysituationwarrantsrecommendations.Somepeoplewhocommissionsurveysjustrequirethedata;theywillmaketherecommendations.
References
Citethoseyouhaveusedinthetext.Putthereferencesinalphabeticalorderorlistthemintheorderinwhichtheyappearinyourtext.
Index
Includeanindexonlyinlongreports.
Appendix
Includeanyrelevantmaterialthatistoocumbersometobecontainedinthereport.Thismaymeanputtingalongsurveyintheappendixratherthaninthebodyofthereport.Asaruleofthumb,15questionsorfewercanbeincludedinthereport.Keepinmindthatsomesinglequestionscanhavemanychoices,andthesemighthavetobeputintheappendix.Forexample,“Inthepast12months,haveyouhadanyoneofthefollowinghealthproblems:headaches,stomachaches,sprainedankleorwrist,cold?”Eachchoiceissimilartoasinglequestion:“Inthepast12months,haveyouhadheadaches?”“Haveyouhadstomachaches?”“Haveyouhadasprainedankleorwrist?”“Haveyouhadacold?”
Acknowledgments
Thanktheindividualswhoprovidedanyguidanceinthedesignofthesurveyoritsinterpretation.Amongthepersonstoconsiderarethesurvey’sparticipants,statistician,anddatacollectors.
SurveyReportingChecklistsandGuidesNeedsomehelpindecidingexactlywhattoincludeinyourreport?Guidanceintheformofchecklistsisavailable.Forexample,considertheChecklistforReportingResultsofInternetE-Surveys(CHERRIES),whichcanbefoundat:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1550605/table/table1.CHERRIESrequiresthatinternetsurveyreportsgivedetailedinformationonthesurvey’sresearchdesign,IRBprocess,developmentandpretesting,recruitmentandadministration,andcalculationofresponserates.CHERRIESalsocallsforinformationonhowmultipleentriesfromthesameindividualwererecognized,andrequiresthesurveyornotonlytoexplainhowdatawereanalyzedbutalsotodescribehowincompleteresponseswerehandledintheanalysis.Hereisanexampleofhowthechecklistrequiresreportingofthesurvey’sresponserate:
GototheAmericanAssociationforPublicResearch’swebsite(http://www.aapor.org)forhelpwithdefinitionsandpointersformakingsureyouhaveacompleteandtransparentsurveyreport.
Transparentsurveyreportingandsurveyethicsgohandinhand.AstheCouncilofAmericanSurveyResearchOrganizations(CASRO)advises,surveyorsareobligatedtoinsurethatthefindingstheyreleaseareanaccurateportrayalofthesurveydataanddemonstratethattheyhaveputinplacecarefulchecksontheaccuracyofallstatistics.CASROrecommendsthatasaminimum,anygeneralpublicreleaseofsurveyfindingsshouldincludethesurvey’ssponsorshipaswellasadescriptionofthesurvey’spurposes,sample,datesofdatacollection,namesofthesurveyors,exactquestionwording,andanyotherinformationthatalaypersonwouldneedtomakeareasonableassessmentofthereportedfindings.
Clear-WritingTipsHerearesometipsonhowtowriteclearly.
1.Usetheactivevoicewheneverpossible.
Poor:Thereportisrelativelysimpleandisobviouslywrittenforthenonexpert,forthereareveryfewstatisticaltablesgiven.(Twentywords,threeareformsoftheverbtobe.)Better:Therelativelysimplereportisobviouslywrittenforthenonexpert,foritgivesveryfewstatisticaltables.(Seventeenwords,twoverbs)
Avoidthereisandthereare.
Poor:Thereareveryfewstatisticaltablesinthereport.Better:Thereporthasveryfewstatisticaltables.
2.Donotsprinklesentenceswithprepositionalphrases.Whenpossible,avoid,“inordertogiveareason”andreplaceitwith“togiveareason.”Convertaprepositionalphrasetoaparticiple:“Intheefforttogetreliableattitudemeasures”canbecome“tryingtogetreliableattitudemeasures.”Convertaprepositionalphrasetoanadjective:“Itisaquestionofimportance”canbecome“Itisanimportantquestion.”
Nearlyallwordprocessingprogramshaveagrammarfunction.Theyareexcellentinpointingoutpassivesentences.Theyalsoprovidealternativesandcanremindyouofthegrammaryoumayhaveforgotten.However,theyarenotperfect,andtheyarenotinterestedinstyle.
3.Tryareadabilityformula.Readabilitytestingpredictsthegradelevelofwrittenmaterial.Manypeoplefeelcomfortablereadingbelowgradelevel,butpeoplewillignorematerialthatisaboveit.Allmajorwordprocessingprogramswillgiveyoureadabilitystatisticsforsomeorallofyourreport.
Thisisasimpleformulathatgivesyouagoodideaofhowtoestimatereadability.TheresultofusingtheformulaiscalledtheFOGindex:
1. Takea100-wordsampleofyoursurveyreport.2. Computetheaveragenumberofwordspersentence.Ifthefinalsentenceinyoursamplerunsbeyond
100words,usethetotalnumberofwordsattheendofthatsentencetocomputetheaverage.3. Countthenumberofwordswithmorethantwosyllablesinthe100-wordsample.Donotcount
propernounsorthree-syllablewordformsendingin-edor-es.4. Addtheaveragenumberofwordspersentencetothenumberofwordscontainingmorethantwo
syllablesandmultiplythesumby0.4.
Supposea100-wordpassagecontainsanaverageof20wordspersentenceand10wordsofmorethantwosyllables.Thesumofthesewordsis30.Multiplying30by0.4givesaFOGindexof12thgrade.
Lookatthestructuredabstractfoundearlierinthischapter.Thereadabilitystatisticssuggestthattheabstractisofaveragedifficultyrequiringareadinglevelfrom8thto10thgrade.Mostreadabilityformulasonlylookatwords;theydonotincludenumbers.Ifyouaretalkingtoalayaudience,keepnumberssimple.Use76%or76.2%,ratherthan76.17%.
TheOralPresentationOralpresentationsfollowmanyoftherulesofwrittenreports.Speakclearlyandslowly,andpaceyourpresentationtotheaudience’sneeds.Youmustbeselectiveinoralreporting—often,adifficulttask.Makesureyougiveaudiencesusefulresults.Policymakerswanttoknowthebottomlinewhiletheoristswanttoknowaboutvalidity.Layaudiencesareprimarilyconcernedwithfindingsandparticipants.Ifyouhavetime,youcantellthemaboutthesurvey’smethodsandlimits.Allaudiencesexpectexamplesofthesurveyquestions.
SlidePresentationsUsethefollowingguidelinestoprepareslides.
OralVersusWrittenReports:ADifferenceinConversationOralreportsandwrittenreportsaredifferentinatleastoneimportantway.Oralreportsdependonyourabilitytospeakthedetails.Writtenreportsrelyonproseandpicturestospeakforyou.Comparethesetwo.
Example:TableUsedinaWrittenandanOralReport
WrittenReport
WrittenInterpretationoftheTableThetablegivesthebaselineandfollow-upmeansandtheobservednetdifferencesinscoresforfoursurveyoutcomes.Weusedanindependentttesttocomparechangesinmeanscoresfrombaselinetofollow-upfortheexperimentalandthecontrolgroups.Significantdifferenceswerefoundfavoringtheexperimentalstudentsinmoralvalues,religiousbeliefs,andsocialresponsibility.Differencesinsocialresponsibilityapproachedsignificancebutdidnotreachthep≤0.05,thelevelofsignificancechosenforthettest.
OralInterpretationoftheTableEventhetablelooksdifferent.Ithasbeensimplified(seepage190).
Thetablecomparesexperimentalandcontrolschools.(Pointtotheappropriatecolumns.)Weusedanindependentttesttocomparethedifferencesinmeanscoresfrombaselinetofollow-up.(Pointtotheappropriaterows.)Wefoundstatisticallysignificantdifferencesfavoringstudentsinexperimentalschoolsinmoralvalues,religiousbeliefs,andsocialresponsibility.Differencesinethicalbehaviorapproachedsignificancebutdidnotreachthep≤.05,thelevelofsignificancechosenforthettest.
Source:www.apua.org.
PostersPostersareanimportantwaytopresentsurveyresults.Aposterisasummaryofyourworkthatisdesignedtobereadandunderstoodwithoutoralexplanation.Usingarelativelylargeprintout,yourelyonshorttextsectionsandgivetheresultsasbulletpoints.Mostsurveypostersalsoincorporategraphicalelementstohelpillustratekeypointsoftheresearch.Asampleposterisonpp.188–189.Inreallifeitis36inches(91.44cm)by48inches(121.92cm).
Apostershouldpresentanoverviewofthestudyincludingadescriptionofthesurveyorsurveys.It’snot
athesisorjournalarticle,sodon’ttrytocramallthedetailsontotheposter.Acasualviewershouldbeabletogetyourmessagein3to5minutesandreadallthetextinnomorethan10minutes.
Includethestandardsectionsofajournalarticle.Theseincludethefollowing:introductionandstudyobjectives,survey,methods(includingresearchdesign,sampling,anddataanalysis),results,andconclusions.Youmayalsowanttoincludeanabstract,acknowledgments,andreferences.
Apostershouldhaveamaintitlethat’sreadablefrom25feetaway.Peoplewillbewanderingthroughthepostersession,soyouneedtocatchtheireyefromadistance.Ageneralruleistousea72-pointtypeandacommonfontsuchasTimesNewRomanorArialforyourpostertitleandtouseasmallersizeofthesamefontforthesectiontitles.Useasimplecolorscheme.Don’tdistractpeoplebyusingtoomanydifferentcolors,fonts,andfontsizes.
Thinkabouthowyouplantoprintyourposterbeforeyoudesignit.Becausenoteveryprintingoptionoffersthesamepaperdimensionsandbecauselargerpostersizesgenerallycostmoretoprint,firstchoosethepapersizeforprintingandthendesignyourposteraccordingly.Checkwithyourprintingvendortofindoutwhetheryoushouldbeawareofanyspecificlimitationsorguidelines.
MicrosoftPowerPointisarelativelyeasy-to-usetoolforcreatingposters.AdobeIllustratorhasmorefeaturesandcanprovideveryprofessionalresults,especiallyforpostersincludinglotsofhigh-resolutionimages,butitismorecomplexandexpensive.Withalittlesearching,youcanfindfree(andofcoursethoseyoupayfor)onlinetemplatesforsettingupyourposter.Forinstance,oneonlinecompanyhasacollectionoffreePowerPoint(.pptand.pptxnativeformats)researchpostertemplates.YoudownloadtheappropriatePowerPointpostertemplate,addtext,images,andgraphicsandsenditbacktothecompanyforprinting.
SUMMINGUPTheadvantageofusingthesurveyformtogiveresultsistoletthereaderorlistenerseethequestionsandresponsechoices.Onlinesurveysystemsprovideinformationonareal-timebasis.Eachtableofsurveystatisticsshouldhaveatitlethatsummarizesitspurposeandcontent.Whenthesourceofatable’sdataisnotimmediatelyobvious,itshouldbegiven.Whenyouuseaterminatablethatmaybeconfusing,defineit.Setoffdefinitionswithasterisksorsuperscriptletters.Tablecolumnsaretheindependentvariablessuchasgroupandtimingofthesurvey(e.g.,baseline,6monthslater,and3yearslater).Piediagramsshowtheproportionofthewholethateachresponsecategoryoccupies.Bargraphsarecommonlyusedtodisplaysurveydatabecausetheyprovideanoverviewofseveralkindsofinformationatoneglance.Linegraphsaredrawingsthatallowyoutoshowchangesandcomparegroups.Usetheactivevoicewheneverpossibleinsurveyreports.Donotsprinklesentenceswithprepositionalphrases.Tryareadabilityformulaonyourreport’scontent.Limiteachslidetoonemainconcept.Allowthelistener1to2minutesperslide.Toemphasizepointsinaslidepresentation,underlinetitles,usebulletsorcheckmarks,numbereachpoint,orusecontrastingcolorstoseparatepoints.
Ingeneral,uppercaseandlowercaselettersareeasiertoreadthanalluppercase.Postersareawaytopresentasummaryofasurveysothatitisvisuallyaccessible.Postershavethesamecontentasresearchpapers,buttheypresentitsothatthereadergetsthemessageinunder10minutes.
THINKABOUTTHIS1. Prepareaslidepresentationforthecontentsofthischapter.2. Writeaparagraphdescribingthesurveyresultsshowninthetableandlinegraphbelow.
LineGraph:
3.Describehowyouwouldpresenttheresultsofthetableandlinegraphinanoralpresentation.Inwhatwayswouldthewrittenandoralpresentationsbesimilar?Different?
REFERENCESBurns,K.E.,Duffett,M.,Kho,M.E.,Meade,M.E.,Adhikari,N.K.,etal.(2008).Aguideforthedesignandconductofself-administeredsurveysofclinicians.CMAJ,179,245–252.
Draugalis,J.R.,Coons,S.J.,&Plaza,C.M.(2008).Bestpracticesforsurveyresearchreports:Asynopsisforauthorsandreviewers.AmJPharmEduc,72,11.
Eysenbach,G.(2004).ImprovingthequalityofWebsurveys:TheChecklistforReportingResultsofInternetE-Surveys(CHERRIES).JMedInternetRes,6,e34.
Huston,P.(1996).Reportingonsurveys:Informationforauthorsandpeerreviewers.CMAJ,154,1695–1704.
Kelley,K.,Clark,B.,Brown,V.,&Sitzia,J.(2003).Goodpracticeintheconductandreportingofsurveyresearch.IntJQualHealthCare,15,261–266.
Index
Abstractinsurveyreport,180–181Acknowledgmentsinsurveyreport,182Adapting/selectingsurveys,79–87Additivescales,60–62Administrationofsurveys
interviews,74–75self-administered,1,72–73
Aesthetics,survey,71AgencyforHealthcareResearchandQuality(AHRQ),84Alphavalue,106AmericanAssociationforPublicOpinionResearch(AAPOR),29AmericanAssociationforPublicResearch,183Angold,A.,130ANOVA,148,150,152,154Appendixinsurveyreport,182Arnold,R.,132Audiotape,asinformationsource,5Audiovisualaids,76Averages,138–141
Bargraphs,172–174,177Bias,11,47–48
sampling,99–100Bimodaldistribution,140,141Bloom,P.,129Branchingquestion,71–72Bricker,J.,109
Campbell,D.,127Casecontrol(historical)design,126Categoricaldata,155–156Categoricalscales,49Cellphonesurveys,14CentersforDiseaseControlandPrevention(CDC),22–23Checklist,53–54ChecklistforReportingResultsofInternetE-Surveys(CHERRIES),182–183Childrenandsurveys,54–57
ethics,26–28Children’sOnlinePrivacyProtectionAct(COPPA),26TheChildSelf-ReportofDelinquency,130Chi-square,149Chronbach’salpha,77Chuang,L.-H.,109
Cleaningdata,163–164Closedquestions,38,39–40
responsesfor,48–49writing,38,44–48
Clustersampling,simplerandom,98–99Codebook(operationsmanual),157–159CodeofFederalRegulations,19Coding
establishingreliable,159–161intercoderagreement,159–161LB/LLdata,40–44
Coefficientalpha,77Cohorts,6–7,129–130CollaborativeInstitutionalTrainingInitiative,26Commentsinsurvey,152Commercialsurveycompanies,13–14CommonRules,19Comparativeratingscale,53Comparisongroupsurveydesign,7Concurrentvalidity,78,83,130Confidenceinterval(CI),103,147–149Confidentiality,13,19–20Consistency,internal,77–78,83Constructvalidity,78–79,83Contentanalysis,152–153Contentvalidity,78,83Continuousdata,155–156Continuousscales,50Controlgroups,107–109,129Conveniencesampling,94,95,99–102Cookies,23,79Correlations,142–143,154Costello,E.J.,130Costs,survey,11,68CouncilofAmericanSurveyResearchOrganizations(CASRO),29,183Covariate,152Cox,H.,109Credibility,11,29,94,100,181Cross-sectionalsurveydesign,6,7,129Cross-tabs,143–145Culturalbeliefs/expectations
childsurveysand,27,29internationalsurveysand,28–29
Datacategorical,155–156cleaning,163–164
continuous,155–156entering,162–163missing,161–162organizing/managing,157–164protectionof,21–25validatingsurvey,164
DataanalysisANOVA,150,152averages,138–141chi-squaretest,149contentanalysis,152–153correlations,142–143,154cross-tabs,143–145descriptivestatistics,137–138frequencies,137–138,163Mann-WhitneyUtest,150mean,138–139median,139–140mode,140–141nullhypothesis,106,145–146overview,135–136planningfor,7–8poweranalysis,106,108,147range,141regression,142–143risksandodds,150–152selectingmethodsfor,153–157standarddeviation,106–107,141–142statisticalsignificance,146–149summingup,164–165surveyingdifferences,149–152ttest,146,149–150variance,141–142SeealsoData;Results
Datamanagementcodebook/operationalmanualfor,157–159establishingreliablecoding,159–161Kappaand,159–161
Dependentttest,152Dependentvariables,156–157Descriptivedifferences,143–145Descriptivestatistics,137–138Design.SeeSurveydesignDesignweights,110–111Diagrams
drawing,178–179pie,172
Differentialscales,62–63Dirtydata,163Discretescales,49–50Dispersion,106Downey,L.,130Drop-downlist,58
Effectsize,108–109,147Engels,R.,109Environmentalcontrol,surveydesignand,118Equivalencereliability,77ERIC,80Error
marginof,112,113standard,103TypeI,106,147TypeII,106,147
Ethics,survey,19–20childrenand,26–28formalstandardsfor,29onlinesurvey,20–26SeealsoConfidentiality;Culturalbeliefs/expectations;Informedconsent;Privacy
Ethnicity,questionsabout,88Externalvalidity,128–129Eyewitnessreport,5
Face-to-faceinterview,costofconducting,11,68Factorialdesign,124–125FamilyRelationsScale,130Farber,J.,129Figures,listof,insurveyreport,181Film,asinformationsource,5Flesch-Kincaidmethod,60Flowchart,179Focusgroup,11,101Foley,D.L.,130Force-choicequestions,6,52Frequencies,137–138,163Frymethod,60
GallupOrganization,84GeneralSocialSurvey,84Generalusesurvey,18,19Glossaryoftermsinsurveyreport,181Goldston,D.B.,130GoogleScholar,80Graphicscales,50
Graphsbar,172–174,177line,174–178
Hawthorneeffect,128,129,130,131Hewitt,C.E.,109Holloway,I.,132Hypothesis,36–37
Incentives,8,11,12,73,101–102,110,164Independentttest,146,187Independentvariables,156–157Indexinsurveyreport,182Informationneedsselection,36–37Informedconsent,20
forchild,27–28foronlinesurvey,25–26
In-personinterview,telephoneinterviewsvs.,15–18InstituteofEducationSciencesthroughtheNationalCenterforEducation,84InstitutionalReviewBoard(IRB),20,28,29Internalconsistency(IC),77–78,83Internalvalidity,127–128,129,130Internationalsurveys,28–29Internet.SeeOnlinesurveysInterrateragreement,83Interviewers,trainingof,74Interviews,1,5–10
administration,74–75costofconducting,11,68monitoringqualityof,75–76telephone(seeTelephoneinterviews)visualaidsfor,76
Jargon,46Jayakody,S.,109
Kang’ombe,A.R.,109Kappa,159–161Kleinjan,M.,109
Languageandsurveys,28–29Length,survey,68Levelofsignificance,106,146–147Likebest/likeleast(LB/LL)technique,40–44Likert-typescale,52Linegraph,174–178Literaturesearchesandsurveys,79–84
Longitudinal/cohortdesign,6–7,129–130
Mailedinterview,costofconducting,11Mann-WhitneyUtest,150Marginoferror,112,113McNemartest,52,152,154Mean,138–139Measurementvalidity,126,129,130Median,139–140Methodsinsurveyreport,181–182Missingdata,161–162Mobilephonesurvey,14Mode,140–141Multiple-choicequestions,39,44–45
NationalCenterforEducationStatistics,85NationalInstitutesofHealth,26NationalOpinionResearchCenter(NORC),84Nominalresponsescale,49Nonrandomizeddesign,125Nonresponseweights,111Normaldistribution,103,106Normativesurveydesign,125Nullhypothesis,106,145–146Numericalscale,49–50
Observations,5Oddsratio,150–152Onlinelibraries,80,81,82Onlinesurveys
advantagesof,11–12branching,72checklistfor,73commercialsurveycompanies,13–14comparedtomailedsurvey,15–18concernsabout,13–14costs,11,68ethicsand,13,20–26freesurveys,examples,84–85informedconsentand,25–26missingdataand,161–162real-timeinformationand,169–170rulesforcreating,57–60validity,164whentouse,12–13
Open-endedquestions,6,39organizingresponsesto,40–44
writing,38Operationsmanual(codebook),157–159Oralpresentations,184Orderofsurvey,examples,68–71Ordinalscales,49,51–53Otten,R.,109Outcomesinsurveyreport,181
Panels,101–102ParentingPracticesQuestionnaire,130Pearsonproduct-momentcorrelation,142,154Performancetests,5PewCharitableTrusts,84Piediagrams,172Pilottest,8,11,84
guidelinesfor,85–87reliability,86validity,86–87
Piping,72Plus-oneapproach,96Policy/programsneeds,surveysfor,2Posters,188–191Poststratificationweights,111Poweranalysis,106,108,147Predictivevalidity,78,83Presentations,oral,184Privacy,19
childrenand,57onlinesurvey,13,21–25SeealsoEthics
Programevaluationsurvey,2–3,5Progressbar,58Progressgraph,73“ProtectionofHumanSubjects,”19PublicLibraryofScience(PLoS),80PubMed,80Purposivesample,101Pvalue,106,147,148
Qualitativeanalyst,statisticianvs.,152Qualitativesurveydata,validityof,130–132Quality,survey.SeeReliability;ValidityQuasi-experiments,121–122Questionnaires
administrationof,1,72–73format,71overlyambitious,4–5
ParentingPractices,130reproducing,167–169self-administered,4–10,15–18
QuestionPro,169Questions
branching,71–72closed,38,39–40,44–49forced-choice,6,52multiple-choice,39,44–45open-ended,6,38,39,40–44ordering,68–71plainandsimple,60recall,69reliability,39transitions,70writing,44–48
Quotasampling,101
Randomdigitdialing,96Randomizedcontroltrial,124–125,129,130Randomsampling,94–99
defining,7simple,95–96simplerandomcluster,98–99stratified,96–98
Range,141Rankordercorrelation,142Rankorderscale,53Ratingscales,49–53Recommendationsinsurveyreport,182Referencesinsurveyreport,182Regression,142–143Relativerisk,150–152Reliability,76–78
equivalence,77internalconsistency,77–78,83likebest/likeleast(LB/LL)technique,41–42ofpilottest,86ofquestions,39ofsurveyinresearchliterature,82–83test-retest,77validityand,11
Research,surveysfor,3,5Researchliterature,findingsurveysin,79–84Respondents
characteristicsof,82forpilottesting,86
incentivesforparticipating,8,11,12,73,101–102,110,164Response
coding,40–44,159–161formatting,71plainandsimple,60
Responserate,8–9,100,110Results
bargraphs,172–174,177diagrams/pictures,178–179linegraphs,174–178oralpresentationof,184piediagrams,172posters,188–191reporting,9–10reproducingquestionnairesfor,167–169summingup,191tables,169–172usefulness/credibilityof,11writing,179–182SeealsoDataanalysis
Reviews,5Risksandodds,150–152Roadmapinsurveyreport,181Rosen,J.,130Ryan,K.,132
Sample,defining,6.SeealsoSamplingSampling
confidenceleveland,112,113convenience,94,95,99–102finding,102focusgroup,101marginoferrorand,112,113onegroup,nointervention,102–103panel,101–102powerand,106,108purposive,101quota,101random,94–99responserate,tipsforimproving,100simplerandom,95–96simplerandomcluster,98–99sizeofsample,93–95,102–103,106–109snowball,101standarderror,103stratifiedrandom,96–98summingup,113–114
systematic,99,103twogroupsandintervention,103–105weighting,110–111
Scales,rating,49–53Scaling,60–64Schuck,K.,109SecureSocketsLayer(SSL)protocol,21Selecting/adaptingsurveys,79–87Self-administeredquestionnaires,5–10,15–18
administrationof,1,72–73Serviceagreement,79Shelltables,170Simplerandomclustersampling,98–99Simplerandomsampling,95–96Siu,A.,129“Skip”pattern,71–72Slidepresentations,184–186Snowballsampling,101Spearmanrankordercorrelation,142Specificusesurvey,18,19Standarddeviation,106–107,141–142Standarderror,103Stanley,J.,127Statementofpurposes/objectivesinsurveyreport,181Statisticaldifferences,145–146Statisticalsignificance,146–149Statistics,descriptive,137–138Stratifiedrandomsampling,96–98Structuredabstract,180–181Subgroups,identifying,104,105Summaryinsurveyreport,181Summatedscales,63–64Surveydesign
casecontrol,126comparisongroup,7cross-sectional,6,7,129factorial,124–125longitudinal/cohort,6–7,129–130normative,125questionwriting,44–48sampleand,6–7summingup,132–133validityof,126–132
Surveyformavailabilityofinformation,37children,54–57closedquestions,responsesfor,48–54
closedquestions,writing,44–48content/topicfor,35–36informationneeds/hypothesis,36–37onlinesurveys,57–60open-endedquestions,organizingresponsesto,40–44overviewof,35plainandsimplequestions/responses,60questionwriting,38–40,44–48ratingscales,49–53relevanceofinformation,37–38scaling,60–64summaryof,64–65termdefinitions,36
Surveyreport,179–182checklistfor,182–184oralvs.written,187
Surveysadministration,1,72–75cellphone,14childrenand,26–28combinedwithotherinformationsources,5commercialsurveycompanies,13–14comparingtypesof,15–18continuumof,18–19costs,11,68dataanalysis(seeDataanalysis)defining,2–3ethics(seeEthics)format,71generaluse,18,19guidelinesforfindinginresearchliterature,79–84international,28–29language/cultureand,28–29lengthcounts,68online(seeOnlinesurveys)orderof,68–71overviewof,1phone(seeTelephoneinterviews)pilottesting,8,11,84,85–87programevaluation,2–3,5reasonstoconduct,2–3reliability(seeReliability;Validity)reportingresultsof,9–10research,3responserate,8–9selecting/adapting,79–87selectioncriteria,11–18
self-administered,1,5–10,15–18,72–73specificuse,18,19summaryof,30–31tomeetpolicy/programneeds,2translating,87–88validity(seeValidity)whentouse,3–5SeealsoQuestions;Sampling;Surveydesign;Surveyform
Systematicsampling,99,103
Tableofcontentsinsurveyreport,181Tables,169–172
inwrittenandoralreports,187,190listof,insurveyreport,181preparationrules,171–172
Telephoneinterviewsadministrationof,74cellphones,14costs,11,68in-personinterviewvs.,15–18orderingquestionsfor,69samplingfor,96
Test-retestreliability,77Tilbrook,H.E.,109Training,interviewer,74Transitions,70Translatingsurveys,87–88Trueexperiments,122–124Ttest,145,146,149–150,152,153,154,187Two-by-threefactorialdesign,124Two-by-twofactorialdesign,124TypeIerror,106,147TypeIIerror,106,147
Unimodaldistribution,140U.S.Census,84
Validityconcurrent,78,83,130construct,78–79,83content,78,83external,128–129internal,127–128,129,130measurement,126,129,130ofonlinesurvey,164ofpilottest,86–87ofqualitativesurveydata,130–132
ofsurveydesign,126–132ofsurveyinresearchliterature,82–83predictive,78,83reliabilityand,11surveydata,130–132,164threatsto,127–131
VanTeijlingen,E.,132Variables,156–157Variance,141,142,145Variation,141–142Videotape,asinformationsource,5Visualaids,76
WebofScience,80,81Websurveys.SeeOnlinesurveysWeighting,110–111Wilcoxonranksum.SeeMann-WhitneyUtestWriting
oralvs.writtenreports,187questions,44–48surveyresults,179–182tips,183–184
Writtentests,5
Zun,L.,130