fully exploiting qualitative and mixed methods data from online surveys
TRANSCRIPT
FULLY EXPLOITING QUALITATIVE AND MIXED METHODS DATA
FROM ONLINE SURVEYSSIDLIT OF C2C
JULY 2015
PRESENTATION DESCRIPTION
• A WIDE RANGE OF CONTEMPORARY RESEARCH USES ONLINE SURVEYS. THIS PRESENTATION PROVIDES AN OVERVIEW OF WAYS TO EXPLOIT SURVEY-CAPTURED DATA FOR ANALYSIS. THERE WILL BE A SUMMARY OF BASIC SURVEY AND ITEM ANALYSIS THAT MAY BE ACHIEVED WITH SURVEY DATA RESULTS. THERE WILL ALSO BE A RANGE OF TIPS FOR EXTRACTING, CLEANING, STRUCTURING, AND PRESENTING BOTH QUANTITATIVE AND QUALITATIVE DATA FOR DATA-CONSUMER SENSE-MAKING. THE PLATFORM THAT WILL BE USED AS AN EXEMPLAR WILL BE THE QUALTRICS SURVEY PLATFORM, AND TWO SUPPORTING TOOLS USED FOR ANALYSIS ARE EXCEL 2013 AND NVIVO 10. REAL-WORLD PROJECTS ARE USED TO DEMO THESE APPROACHES—WITH PRINCIPAL INVESTIGATOR (PI) PERMISSION.
2
OVERVIEW
• LIGHT DEFINITION OF “EXPLOIT”
• A REFRESHER REVIEW OF SURVEYS [EXCERPTED FROM “REVIEWING SURVEYS, INTERVIEWS AND FOCUS GROUPS” (JAN. 30, 2015)]
• (GENERIC) RESEARCH DESIGN; BASIC PURPOSES OF SURVEYS; THE SURVEY INSTRUMENT; SURVEY RELIABILITY; SURVEY VALIDITY; THE CREDIBILITY OF SURVEY FINDINGS; THE TIME FACTOR; SAMPLING OF SURVEY RESPONDENTS; ONLINE SURVEYS; DATA FORMS
• QUANTITATIVE / QUALITATIVE / MIXED METHODS / MULTI METHOD /
• SOME PRINCIPLES OF DATA ANALYSIS
• QUALTRICS
• ABOUT THE ONLINE SURVEY SYSTEM
• WITHIN-QUALTRICS DATA ANALYTICS 3
OVERVIEW (CONT.)
• THE USES OF OTHER SOFTWARE• EXCEL• MICROSOFT WORD • NOTEPAD • IBM SPSS• NVIVO 10• AUTOMAP AND ORA NETSCENES • NODEXL, UCINET • TABLEAU (PUBLIC), ARCMAP / ARCGIS PRO• RAPIDMINER STUDIO
4
OVERVIEW (CONT.)
• THE RESEARCHER INTERPRETIVE LENS; THE PROBLEM OF HUMAN MANIPULATION OF DATA
• SOME REAL-WORLD CASES
• CASE 1: A MULTI-COUNTRY MULTI-GRAIN LONGITUDINAL SURVEY
• CASE 2: INFORMATION TECHNOLOGY (IT) SATISFACTION SURVEY
• FULL EXPLOITATION OF SURVEY DATA
• REALITY CHECKS & CAVEATS
• CONTACT AND CONCLUSION
5
LIGHT DEFINITION OF “EXPLOIT”
6
“EXPLOIT”
• VERB: MAKE FULL USE OF A RESOURCE; TAKING FULL ADVANTAGE OF A RESOURCE
• NOUN: A DARING FEAT
7
WHY EXPLOIT?...NEW INSIGHTS
• DATA POVERTY / DATA RICHNESS
• RECOUPING HIGH EXPENSE IN COLLECTING SOME SURVEY DATA • GRANT(S); INSTITUTIONAL COSTS; TECHNOLOGIES; PEOPLE TIME; PEOPLE EXPERTISE
• SPACE TO REPURPOSE CAPTURED DATA THROUGH CROSS-REFERENCING AND COMPARING AND CONTRASTING DATA
• AGAINST PUBLICLY AVAILABLE DATASETS
• AGAINST SOCIAL MEDIA DATA
• AGAINST SPATIAL DATA
• AGAINST PUBLIC INFORMATION (OFTEN PUBLIC MEDIA-BASED)
• AGAINST COMPARABLE CASES 8
WHY EXPLOIT?...NEW INSIGHTS (CONT.)
• MACHINE-ENABLEMENTS FOR ANALYZING HETEROGENEOUS DIGITAL DATA (VARIOUS SOURCES, VARIOUS FORMATS, VARIOUS STRUCTURES, VARIOUS MEDIA)
• AUTOCODING BY EXISTING PATTERN, TEXT NETWORK ANALYSIS, GEOSPATIAL MAPPING, AND OTHER APPROACHES
• RICH DATA VISUALIZATIONS
• DATA MINING
• MODEL EXTRACTION FROM DATA
• NOT “EXPLOIT” AS TO CAUSE OR ALLOW ANY POTENTIAL HARM TO RESEARCH PARTICIPANTS; NOT GOING BEYOND THE APPROVED USES OF THE INFORMATION
9
A REFRESHER / REVIEW OF SURVEYS(AS A RESEARCH TOOL)
10
(GENERIC) QUAL / MIXED / MULTI METHODS RESEARCH DESIGN
• CONCEPTUALIZATION: RESEARCH OBJECTIVES, RESEARCH QUESTIONS, HYPOTHESES, POTENTIAL IMPLICATIONS
• THOROUGH REVIEW OF THE LITERATURE (ANNOTATION AND WRITE-UP)
• RESEARCH DESIGN (MIXED METHODOLOGY / SYNTH, MULTI-METHOD / SEQUENTIAL)
• INSTRUMENTATION (DESIGN, PILOT-TESTING, REVISION)
• SAMPLING (SELECTION OF RESPONDENTS: RANDOM, STRATIFIED RANDOM, NON-RANDOM, CONVENIENCE, OTHER)
• RESEARCH
• DATA COLLECTION (SOMETIMES MULTI-METHOD; MIXED METHOD; FORM OF DATA COLLECTION AFFECTS ANALYSIS)
11
(GENERIC) QUAL / MIXED / MULTI METHODS RESEARCH DESIGN (CONT.)
• FOLLOW-UP (IF NEEDED)
• DATA VISUALIZATION
• DATA ANALYSIS (QUANTITATIVE AND QUALITATIVE METHODS)
• “DISCUSSION” SECTION
• REPORTING OUT
• POTENTIAL IMPLICATIONS
• FOLLOW-ON RESEARCH
• FUTURE RESEARCH
12
BASIC PURPOSES OF SURVEYS
• COLLECT DATA ABOUT PEOPLE’S EXPERIENCES, SITUATIONS, ATTITUDES, BELIEFS,OPINIONS, AND OTHER FACTORS AT A PARTICULAR POINT-OF-TIME, OR OVER TIME
• COMPLEMENT VARIOUS OTHER TYPES OF RESEARCH, INCLUDING EXPERIMENTAL RESEARCH (RANDOM SAMPLING, CONTROL GROUP VS. EXPERIMENTAL GROUP)
• MAY BE USED AT ANY TIME IN THE RESEARCH PROCESS FOR VARYING PURPOSES
• IDENTIFY TRENDS OVER TIME FOR PARTICULAR POPULATIONS (IN PARTICULAR CONTEXTS)
• USUALLY INVOLVES BOTH QUALITATIVE AND QUANTITATIVE DATA (MIXED-METHODS / MULTI METHODS) DATA COLLECTION AND ANALYSIS
13
THE SURVEY INSTRUMENT• IS DESIGNED FOR PARTICULAR PURPOSES
• IS WRITTEN IN AN UNDERSTANDABLE WAY (“STANDARD LANGUAGE”); IF IN A FOREIGN LANGUAGE, ACHIEVED BY A PROFESSIONAL TRANSLATOR OR NATIVE SPEAKER (NOT MACHINE-TRANSLATION)
• USES CLOSE-ENDED QUESTIONS APPROPRIATELY WITH A FULL RANGE OF CHOICES (NO FALSE LIMITS)
• IF SCALED RESPONSES, PROPER SCALING (LIKE LIKERT-LIKE SCALES AND CONSISTENCY OF ORDER; OR FORCED-CHOICE 4-POINT LIKERT SCALES WITH NO FENCE-SITTING NEUTRALITY); IF SCALED RESPONSES, PROPER CONSISTENCY IN TERMS OF DIRECTION (HIGHEST-TO-LOWEST FOR ALL QUESTIONS; OR LOWEST-TO-HIGHEST FOR ALL QUESTIONS)
• USES OPEN-ENDED QUESTIONS APPROPRIATELY, WITH SUFFICIENT DIRECTION AND SPACE FOR A FULL TEXTUAL RESPONSE
• IS ACCESSIBLE FOR ALL THOSE WITH A RANGE OF SPECIAL AND OTHER NEEDS (TRANSCRIPTIONS AND TIMED TEXT FOR VIDEOS, ALT TEXT FOR IMAGES, ETC.) 14
THE SURVEY INSTRUMENT (CONT.)
• ALIGNS THE QUESTIONS WITH THE APPROPRIATE DATA TYPES [CATEGORICAL, ORDINAL (RANK ORDER), NUMERICAL (DISCRETE, CONTINUOUS), TEXT- / AUDIO- BASED / VIDEO-BASED , AND OTHERS)
• INCLUDES INFORMED CONSENT AT THE BEGINNING; ENABLES OPT-OUT AT ANY TIME; NO COLLECTION OF EXCESS INFORMATION; NO DECEPTION (UNLESS APPROVED BY THE INSTITUTIONAL REVIEW BOARD / IRB)
• IS INFORMED BY THE RESEARCH LITERATURE (EXPLORED TO “SATURATION”)
• IS STRATEGICALLY SEQUENCED
• AVOIDS FORCING RESPONSES BECAUSE OF THE PARTICIPANT OPT-OUT ISSUE PER IRB GUIDELINES (DEBATABLE) 15
THE SURVEY INSTRUMENT (CONT.)
• AVOIDS ANY BIASING DESIGN OR LEADING LANGUAGE
• IS PILOT-TESTED WITH BOTH EXPERTS AND WITH PEOPLE WHO ARE SIMILAR TO RESPONDENTS, WITH CHANGES MADE TO ENSURE LANGUAGE CLARITY; COMPREHENSIVENESS OF THE SURVEY; CLEAR TRANSITIONS; ACCESSIBILITY; AND CORRECTIONS OF ALL KNOWN ERRORS (AND CONTINUING TESTING UNTIL NO OTHER ERRORS ARE FOUND)
• IS TESTED FOR RELIABILITY (THAT IT IS DEPENDABLE AND CONSISTENT)
• IS TESTED FOR VALIDITY (THAT IT MEASURES WHAT IT PURPORTS TO MEASURE)
• ALIGNS WITH DOMAIN’S PROFESSIONAL RESEARCH STANDARDS AND EXPECTATIONS
• MAY BE VERSIONED FOR DIFFERENT GROUPS, OR MAY BE BRANCHED FOR CERTAIN GROUPS
• MUST MAINTAIN COMPARABILITY IF STUDIED FOR TREND DATA FOR LONGITUDINAL RESEARCH
16
SURVEY RELIABILITY
• ACHIEVING THE SAME RESULTS EVERY TIME THE INSTRUMENT IS USED, SUCH AS THROUGH TEST-RETEST RELIABILITY WITH THE SAME PERSON OR GROUP (OVER TIME); CONSISTENCY OF PERFORMANCE
• RELIABILITY ACROSS DIFFERENT INSTRUMENTS OR “EQUIVALENCE RELIABILITY”
17
SURVEY RELIABILITY (CONT.)
• INTERNAL CONSISTENCY OF MEASURE [CRONBACH’S ALPHA Α / COEFFICIENT ALPHA; THE COMPLEMENTARITY OF QUESTIONS IN RELATION TO EACH OTHER IN MEASURING ONE DIMENSION OR A SINGLE CONSTRUCT (UNIDIMENSIONAL); INTER-CORRELATIONS AMONG THE TEST ITEMS; NOT ROBUST UNDER CONDITIONS OF MISSING DATA; VARIABLES AND THE DEGREE TO WHICH THEY MEASURE THE SAME THING IN AN INTER-ITEM CORRELATION WAY AS EXPRESSED IN A MATRIX AND COMPARISONS DONE BY REMOVING VARIABLES TO SEE WHAT CHANGES OCCUR IN THE MEASURING OF THE CONSTRUCT; A LATENT CONSTRUCT MAY AFFECT THE ALPHA; Α < 1 ]
18
SURVEY VALIDITY
• ACCURATE MEASUREMENT OF WHAT IT WAS DESIGNED TO MEASURE; VALID TO THE TASK
• DIFFERENT TYPES OF VALIDITY:
• PREDICTIVE VALIDITY (PROJECTS TO THE FUTURE)
• CONCURRENT VALIDITY (AGAINST AN ACCEPTED MEASURE)
• CONTENT VALIDITY [REASONABLE SAMPLE OF RELATED INFORMATION AND PROPER TERMS FOR WHAT THE SURVEY WANTS TO SAMPLE(FINK, 2013, P. 67)]
• CONSTRUCT VALIDITY (USING THE INSTRUMENT ON RESPONDENTS WHO’VE BEEN ESTABLISHED BY EXPERTS TO RATE A PARTICULAR WAY ON A PARTICULAR SCALE ON A PARTICULAR CONSTRUCT TO SEE IF THE TARGET SURVEY COMES UP WITH THE SAME RESULTS)
19
THE CREDIBILITY OF SURVEY FINDINGS• “RELIABILITY” AND “VALIDITY” ARE DEVELOPED TO SUPPORT THE “MEASUREMENT
VALIDITY” AND ULTIMATELY THE CREDIBILITY OF SURVEY FINDINGS
• ALSO NEED TO CONTROL FOR “ERROR,” WHICH COMES FROM MANY SOURCES:• REPRESENTATIVE SAMPLING, ELIGIBILITY CRITERIA OF THOSE TAKING THE SURVEY, LOW
RESPONSE RATES, ATTRITION OF PARTICIPANTS (PARTICULARLY IN LONGITUDINAL RESEARCH)
• RESEARCHER EFFECTS: COGNITIVE BIASES, INCENTIVES, WEAKNESSES
• RESEARCH DESIGN
• INSTRUMENT DESIGN
• ADMINISTRATION
• FOLLOW-ON SURVEY WITHOUT SUFFICIENT PASSAGE OF TIME (AND THE EFFECT OF THE FIRST SURVEY’S RESULTS ON THE LATTER)
• EXCLUSION / INCLUSION OF OUTLIER DATA POINTS
• INSUFFICIENT ANALYSIS AND REFINEMENT 20
THE TIME FACTOR
• CROSS-SECTIONAL OR SLICE-IN-TIME SURVEYING
• MULTIPLE-SEQUENTIAL SURVEYING
• LONGITUDINAL (OR PERIODIC OVER-TIME SURVEYING)
21
SAMPLING OF SURVEY RESPONDENTS
• RANDOM (AND SUFFICIENT) SAMPLING THE “GOLD STANDARD” FOR GENERALIZING TO A POPULATION
• STRATIFIED RANDOM SAMPLING TO SELECT MEMBERS OF PARTICULAR GROUPS AS RESPONDENTS
• SIMPLE RANDOM CLUSTER SAMPLING (CONVENIENCE SAMPLING, ASSUMPTION OF PRE-DEFINED CLUSTERS IN THE POPULATION)
• CONVENIENCE SAMPLING (LIKE SNOWBALL SAMPLING, WHICH BIASES TOWARDS HIGHLY CONNECTED ACTORS/AGENTS); NON-RANDOM; GOLD STANDARD AS “REPRESENTATIVE” SAMPLING FOR QUALITATIVE RESEARCH
22
SAMPLING OF SURVEY RESPONDENTS (CONT.)
• SYSTEMATIC (STRATIFIED RANDOM) SAMPLING (LIKE EVERY 5TH PERSON…, MAY HAVE HIDDEN IF UNINTENTIONAL BIASES, WITH A COMMON EXAMPLE AS A-Z SAMPLING BUT WITH FEWER INDIVIDUALS WITH NAMES IN THE W-Z RANGE)
• OPEN-CALL SAMPLING WITH AN ONLINE SURVEY
• BIAS IN TERMS OF THOSE WHO SELF-SELECT IN OR OPT-IN, HAVE INTEREST, HAVE TECHNO ACCESS AND SAVVY, ARE MORE ACTIVIST (MAY NOT REPRESENT QUIETER VOICES)
• POTENTIAL DIFFICULTY IN VERIFYING IDENTITY (EXCEPT INTERNET PROTOCOL OR “IP” ADDRESSES)
• POTENTIAL BROADER GEOGRAPHIC REACH THAN OTHERWISE
23
SAMPLING OF SURVEY RESPONDENTS (CONT.)
• CASE CONTROL: CASE GROUP (“EXTANT” CONDITION) AND CONTROL GROUP (ABSENCE OF “EXTANT” CONDITION) FOR COMPARISON AND CONTRAST AND POTENTIAL GENERALIZING
24
ONLINE SURVEYS
• RESEARCHER NEEDS TO KNOW AND DEPLOY THE TECHNOLOGY WELL
• MUST PROTECT THE DATA WELL TO MEET ALL LEGAL GUIDELINES (GOING WITH A TRUSTED SURVEY COMPANY)
• MUST PROTECT PARTICIPANT PRIVACY AND CONFIDENTIALITY
• MUST DE-IDENTIFY DATA / ANONYMIZE BEFORE DATA INGESTION INTO AN ANALYSIS TOOL (OR DATASET SHARING THROUGH REPOSITORIES OR “REPRODUCIBLE RESEARCH” ARTICLES)
• MUST OFFER OPT-OUT FUNCTION AT ANY TIME (FOR IRB STANDARDS)
• MUST ANTICIPATE POTENTIAL HARM AND MITIGATE
25
ONLINE SURVEYS (CONT.)
• ONLINE SURVEY MUST BE FULLY COMPREHENSIBLE WITHOUT SURVEY TAKER INTERVENTION (DESIGNED TO HEAD OFF POTENTIAL MISINTERPRETATION WITH ADDITIONAL OPT-IN DATA AS NEEDED)
• MAY HAVE “ENUMERATORS” OR ASSISTANTS STAND IN THE LITERACY / NUMERACY GAP
• DATA USUALLY A MIX OF QUANTITATIVE AND QUALITATIVE DATA
• MAY BE EXPORTED AS .CSV, .DOCX, .PDF, AND OTHER FILE TYPES (MANY OF WHICH ARE TRANSCODABLE WITH LITTLE EFFORT)
• MAY BE PARTIALLY EXPORTED IN PRE-MADE TABLES, CHARTS, AND GRAPHS
26
DATA FORMS
• .XLSX DATA TABLES (FOR QUANT DATA)
• .CSV TEXT FILES, .DOC AND .DOCX TEXT FILES
• SOME PRE-EXTRACTED BAR CHARTS FROM THE ONLINE SURVEY SYSTEMS
• AUDIO
• STILL IMAGERY
• VIDEO
27
QUANTITATIVE / QUALITATIVE / MIXED METHODS / MULTI METHOD
28
GENERAL PRINCIPLES AND PRACTICES OF DATA TYPES
QUANTITATIVEREPRODUCIBLE
LAB-BASED, PRE- AND POST-
PLACEBO APPROACH
EMPIRICALLY OBSERVABLE / MEASUREABLE
GENERALIZABLE
MANAGING SUBJECTIVITY THROUGH METHODS, OVERSIGHT, TOOLS, AND PEER
REVIEW
“IN THE GLASS,” IN THE LAB (IN VITRO)
QUALITATIVE“ISOLATE” AND CASE BASED
TRIANGULATION
SATURATION
NON-GENERALIZABLE
MANAGING HUMAN SUBJECTIVITY THROUGH RESEARCHER AWARENESS AND
DISCLOSURE, RESTRICTION OF CLAIMS
“IN THE BODY,” IN THE WORLD (IN VIVO)
MIXED METHODSSYNTHESIZED RESEARCH USING QUANT AND QUAL METHODS, DATA, THEORIES,
AND PARADIGMS
SEQUENCE OF RESEARCH COMBINING QUANTITATIVE AND QUALITATIVE
RESEARCH METHODS (EACH TREATED MORE DISCRETELY)
29
MULTI METHODS
SOME AFFORDANCES OF ONLINE SURVEYS
• ADAPTABLE TO VARIOUS TYPES OF RESEARCH APPROACHES AND METHODS
• MAY BE STAND-ALONE OR COMPLEMENTARITY TO OTHER RESEARCH SOURCES / DATA STREAMS
• ENABLES… • A BROAD GEOGRAPHICAL BREADTH OF RESEARCH AND ACCESS
• BROAD INTEGRATION OF MULTIMEDIA (AUDIO, VIDEO, INTERACTIVE MAPS, SIMULATIONS, AND OTHERS)
• MULTI-LINGUAL APPROACHES
• CONDITIONAL BRANCHING
• CAPTURING A BROAD RANGE OF DATA
• SOME SURVEILLANCE AGAINST HACKING AND MIS-USE
• BUILT-IN DATA PROTECTIONS
30
SOME AFFORDANCES OF ONLINE SURVEYS (CONT.)
• RICH QUESTION TYPES IN QUALTRICS:
• TEXT/GRAPHIC QUESTIONS
• MULTIPLE-CHOICE QUESTIONS
• MATRIX TABLE QUESTIONS
• TEXT ENTRY QUESTIONS
• SLIDER QUESTIONS
• RANK ORDER QUESTIONS
• SIDE-BY-SIDE QUESTIONS
• CONSTANT SUM QUESTIONS
• PICK, GROUP, AND RANK QUESTIONS
• HOT SPOT QUESTIONS
• HEAT MAP QUESTIONS
• GRAPHIC SLIDER QUESTIONS
• GAP ANALYSIS QUESTIONS
• DRILL DOWN QUESTIONS
• INVISIBLE QUESTIONS: TIMING, META INFORMATION, FILE UPLOAD, AND CAPTCHA VERIFICATION
31
SOME AFFORDANCES OF ONLINE SURVEYS (CONT.)
• OTHER ADVANCED AFFORDANCES:
• BRANCHING LOGIC
• PIPED TEXT (CUSTOMIZED TEXT, SEQUENCES, AND INTERACTIONS)
• E-MAIL AND PANEL TRIGGERS
• QUOTAS
• GOOGLE TRANSLATE INTEGRATION
• VIEWING PANELS, AND OTHERS
32
SOME PRINCIPLES OF DATA ANALYSIS
33
DATA HANDLING
• DOWNLOADING RAW DATA AND KEEPING A PRISTINE SET UNTOUCHED BY ANALYSTS OR ANYONE • LEAST LOSSY DATA MAINTENANCE (OFTEN IN NATIVE STRUCTURE, SUCH AS THE MOST HIGH RESOLUTION
FOR IMAGERY AND THE MOST DETAILED AUDIO-VISUAL FILE TYPES)
• CLEAR DATA PROVENANCE: RECORD-KEEPING ABOUT WHERE DATA COME FROM AND THE PARAMETERS FOR THE DATA COLLECTION / EXTRACTION
• PRESERVATION OF DATA AT ALL STAGES, FROM RAW TO PROCESSED (NOT RETROACTIVELY CHANGING UP DATA); FUTURE-PROOFING AGAINST POTENTIAL DATA INACCESSIBILITY IN THE FUTURE
• CONSISTENT FILE NAMING PROTOCOLS • README FILES
34
DATA HANDLING (CONT.)
• PROTECTING AGAINST DATA CORRUPTION AND DATA TAMPERING
• PROTECTING AGAINST DATA LEAKAGE (AND POTENTIAL MIS-USE)
• LEAST-PRIVILEGE PROTECTIONISM
• METADATA SCRUBBING / STRIPPING OFF FOR THE WORKING FILES
• ANONYNIMIZATION (IDENTITY INVISIBLE EVEN TO THE RESEARCHER)
• PSEUDONYMIZATION (IDENTITY REASSIGNED FOR PUBLICATION, BUT IDENTITY VISIBLE TO THE RESEARCHER)
• NO PREMATURE RELEASE OF DATA; NO UNAUTHORIZED RELEASE OF DATA
35
DATA HANDLING (CONT.)
• PROTECTION OF ALL PARTICIPANTS IN THE RESEARCH (INCLUDING THEIR DATA AND PRIVACY)
• FOLLOWING FORMAL GUIDANCE IN TERMS OF DATA HANDLING (INSTITUTIONAL REVIEW BOARD RULES)
• RESEARCHER INTIMACY WITH THE DATA
36
DATA ANALYSIS
DATA CODING: • THE USES OF THEORIES, MODELS, AND / OR FRAMEWORKS TO UNDERSTAND DATA AND
INFORMATION: CODING BASED ON EXPECTATIONS FROM THEORIES, MODELS, AND / OR FRAMEWORKS (~ TO AN A PRIORI APPROACH)
• EMERGENT INTERPRETATION: CODING BASED ON EXTRACTED INFORMATION AND SEEING WHERE IT GOES (~ TO A GROUNDED THEORY APPROACH)
• MIXED CODING APPROACHES: COMBINING BOTH A PRIOR AND EMERGENT CODING METHODS
• IF TEAM CODING, FOCUSING ON VARIOUS ASPECTS OF CONSENSUS (INTERRATER RELIABILITY) OR DISSENSUS…AND OTHER FACTORS
37
DATA ANALYSIS (CONT.)
• MAINTENANCE OF RESEARCH JOURNALING / MEMO-ING / ANNOTATIONS, AND COLLECTION OF THINKING ABOUT DATA ANALYSIS
• NOT OVER-ASSERTING AND NOT GENERALIZING FROM THE INFORMATION
• APPLYING DIFFERENT METHODS TO ANALYZING DATA (WITH CLEAR DOCUMENTATION OF EACH APPROACH, EACH TOOL, AND THE OVERALL DATA PROCESSING AND ANALYTICAL SEQUENCE); A PREFERENCE FOR SIMPLE METHODS WITH GOOD DATA
• JOURNALING / RECORD-KEEPING OF INSIGHTS AT EACH STAGE
38
DATA VISUALIZATIONS
• USING DIFFERENT DATA VISUALIZATIONS FOR DIFFERENT KNOWLEDGE EXTRACTIONS
• LABELING DATA AND OFFERING LEGENDS
• INFORMATIVE LEAD-UP AND LEAD-AWAY TEXT TO DATA VISUALIZATIONS
• SUPPORT FOR INFORMATION CLARITY; AVOIDANCE OF NEGATIVE LEARNING FROM THE DATA VISUALIZATIONS (CONTROLLING FOR MIS-IMPRESSIONS)
39
DATA VISUALIZATIONS (CONT.)
• PROVISION OF ACCESS TO UNDERLYING (EVEN RAW) DATA (SUCH AS IN TABLES) AND DATASETS
• NO MANIPULATIONS OF THE DATA CONSUMERS (NO POLITICAL OVERRIDE, NO IMPRESSION MANAGEMENT, NO MANIPULATION OF AFFECT / EMOTION OR COGNITION)
• REVELATION OF SUBJECTIVITIES
40
ABOUT THE ONLINE SURVEY SYSTEM
• A CLOUD-BASED TOOL: INTERFACE “TOUCHINESS,” DATA STORAGE STATESIDE, SOPHISTICATED TOOL DESIGN, MULTIMEDIA INTEGRATION CAPABILITY
• LIBRARIES: AVAILABLE SURVEY TEMPLATES AND TEMPLATE LIBRARIES
• HELP: PROFESSIONAL, FRIENDLY, AND READY HELP SUPPORTS
• SURVEY REPORTING FEATURE: SURVEY STATISTICS, SURVEY DURATIONS, COMPLETION PERCENT, QUESTION RESPONSE RATES, “DROP OUTS” OR LAST ANSWERED QUESTION COUNTS
• ADDITIONAL CAPTURING OF “TRACE” DATA
42
ABOUT THE ONLINE SURVEY SYSTEM (CONT.)
• PILOT TESTING TO CATCH SOME ISSUES WITH THE SURVEY INSTRUMENT:
• AVOIDANCE OF BIAS
• QUESTION PHRASING AND RESPONDENT COMPREHENSION
• LOW QUESTION RESPONSE RATES
• SURVEY RESPONDENT DROPOUT RATES
• TECHNOLOGY BEHAVIOR (PARTICULARLY IN THE ON-GROUND CONTEXT)
• DESIGN FLAWS
• … AND OTHERS
43
RESPONDENT DEMOGRAPHIC DATA
• INFORMATION ABOUT AGE, GENDER, BACKGROUND, LANGUAGE, REGION, EARNINGS (SES), AND OTHER INFORMATION
• MAY BE USED TO CLUSTER RESPONDENTS ACCORDING TO GROUPNESS IN ORDER TO EXPLORE POTENTIAL PATTERNING AND TO POSE PARTICULAR QUESTIONS
• MAY CREATE GROUPNESS ON NON-DEMOGRAPHIC FACTORS LIKE ATTITUDES TOWARDS PARTICULAR ISSUES, CERTAIN STATES-OF-BEING LIKE HEALTH STATUS, AND OTHERS
44
WITHIN-QUALTRICS DATA ANALYTICS
• CROSS-TABULATION ANALYSIS (AKA “CONTINGENCY TABLE”)
• CAPTURES MULTIVARIATE FREQUENCY DISTRIBUTIONS (INCLUDING WITH CATEGORICAL DATA)
• DEFINED “BANNERS” (COLUMNS) AND “STUBS” (ROWS)
• ENABLES USAGE OF DATA FROM MULTIPLE-CHOICE QUESTIONS, MATRIX QUESTIONS, AND EMBEDDED DATA; ACCESS TO ORIGINAL QUESTIONS AND DATA
• INCLUDES CALCULATION OF CHI-SQUARE MEASURE, DEGREES OF FREEDOM (DF), AND P-VALUE OR ALPHA (FOR SUFFICIENT STATISTICAL SIGNIFICANCE < .05 OR < .01 TO REJECT THE NULL HYPOTHESIS)
45
46
WITHIN-QUALTRICS DATA ANALYTICS (CONT.)
• ALSO THE NET PROMOTER SCORE™ (“CONSUMER LOYALTY SCORE” FOR MARKETING)
47
DATA EXPORT FOR PROCESSING AND ANALYSIS IN OTHER TOOLS
OTHER OFF-PLATFORM ANALYSES • DESCRIPTIVE ANALYSIS ABOUT RESPECTIVE SUB-
POPULATIONS BASED ON CROPS, ON LOCATIONS, ON GENDER
• INFERENTIAL STATISTICS ABOUT RESPECTIVE SUB-POPULATIONS
• CONTENT ANALYSES OF RESPONSES (BASED ON TEXT)
• TESTS OF HYPOTHESES BASED ON THE AVAILABLE EXTRACTED DATA (PARTICULARLY PREDICTOR AND OUTCOME VARIABLES, FROM OBSERVED DATA)
SOME REQUIREMENTS • EXPORT FOR ANALYSIS IN OTHER TOOLS (USUALLY
AS TEXT OR AS DATASETS)
• DATA RE-STRUCTURING (AND CLEANING) FOR VARIOUS TYPES OF QUERYING
• SETUP OF DUMMY VARIABLES
• SETUP OF DIFFERENT DATASETS
• DATA NORMALIZING FOR COMPARABILITY ACROSS SETS
• FULLY-AUTOMATED MACHINE-BASED DATA EXTRACTIONS
48
THE USES OF OTHER SOFTWARE
49
EXCEL: DATA PROCESSING AND VISUALIZATION
• ERASE ALL BLANK CELLS: HIGHLIGHT THE CELLS IN A. IN THE HOME TAB IN EXCEL, GO TO THE EDITING AREA. CLICK ON SORT & FILTER. SORT A TO Z. CLICK F5. CLICK THE SPECIAL BUTTON. CHOOSE THE RADIO BUTTON NEXT TO BLANKS. CLICK “OK.” THEN, CLICK CTRL + -, AND SELECT “SHIFT CELLS UP.” THIS DELETES ALL EMPTY CELLS IN EXCEL.
• ALPHABETIZE CELLS: TO PROCESS THE MIXED TEXTUAL, NUMBER, AND DATE DATA, FIRST, DELETE THE QUESTION AND QUESTION LABEL (OFTEN IN CELLS A1 AND A2). (IF THIS INFORMATION IS STILL NEEDED, PASTE THE DATA IN TO SOME OTHER CELLS LIKE J1 AND J2).
• FILTERING DATA: SELECTING OUT PARTICULAR INFORMATION FOR ATTENTION OR PROCESSING
• DATA VISUALIZATIONS: FREQUENCY BAR CHARTS
50
EXCEL: DATA PROCESSING AND VISUALIZATION (CONT.)
• MORE BASIC PROCESSING IN EXCEL
• DATA EXTRACTIONS FROM THE WEB: “POWER QUERY” ADD-IN TO EXCEL 2013 (AND BACKWARDS COMPATIBLE TO 2010)
• INTEGRATIONS OF DATA FROM DATABASES, AZURE CLOUD, HADOOP (HDFS) DATA THROUGH POWER QUERY TAB
• DATA MAPPING AND VISUALIZATIONS: “POWER VIEW” ADD-IN TO EXCEL 2013 (AND BACKWARDS COMPATIBLE TO 2010)
51
MICROSOFT WORD: SIMPLE TEXT PROCESSING
• TEXT COUNTS FROM THE LARGEST PHRASES FIRST…AND THEN THE SMALLER WORDS (USING SEARCH + REPLACE)
• PROCESSING NUMERICAL RESPONSES TO IMAGE SELECTION AS A SURVEY QUESTION RESPONSE (WITH NUMBERED LABELS FOR THE IMAGES)
52
NOTEPAD OR OTHER TEXT EDITOR: CLEANING TEXT
• CLEANING TEXT BETWEEN SOFTWARE PROGRAMS
• ENABLING READABILITY OF HTML OR XML FILES
53
SPSS (IBM): DESCRIPTIVE STATISTICS, INFERENTIAL STATISTICS, NON-PARAMETRIC TESTS AND ANALYSES, TIME-
SERIES ANALYSES, REGRESSION ANALYSES, AUTOMATED MODELING
• BASIC DESCRIPTIVE STATISTICS
• FREQUENCIES FOR SCALE VARIABLES
• BASIC INFERENTIAL STATISTICS
• META-VARIABLE (CROSS-VARIABLE) ANALYSES
• NON-PARAMETRIC TESTS AND ANALYSES
• TIME-SERIES ANALYSES
• REGRESSION ANALYSES
• AUTOMATED MODELING
• … AND OTHERS
54
NVIVO 10: WORD FREQUENCY COUNTS, TEXT SEARCHES, WORD PROXIMITY EXPLORATION, TEXT
MAPPING• DATA CODING (BY BOTH HUMAN AND
MACHINE)
• DATA PROCESSING
• DATA COLLECTION (FROM SOME SOCIAL MEDIA PLATFORMS, OR OTHER COMPLEMENTARY STREAMS OF DATA)
• WORD FREQUENCY COUNTS
• WORD TREES / TEXT FINDS
• WORD PROXIMITY EXPLORATION AND ANALYSES
• TEXT MAPPING AND VISUALIZATIONS
• GEOGRAPHICAL MAPPING (WITH RELATED GEOGRAPHICAL INFORMATION)
55
AUTOMAP AND ORA NETSCENES: EXTRACTION OF CONTENT NETWORKS FROM TEXTS
AND TEXT CORPUSES
AUTOMAP• TEXT-BASED CONTENT NETWORKS
• APPLICATION OF CUSTOMIZED THESAURUSES
• LEXICAL NETWORKS, AND OTHERS
ORA NETSCENES• VISUALIZING CONTENT NETWORKS AS
GRAPHS
56
NODEXL, UCINET: NETWORK GRAPHING
• TEXT NETWORK GRAPHING
• NETWORK VISUALIZATION AND ANALYSIS, AND OTHERS
57
TABLEAU (PUBLIC), ARCMAP/ARCGIS PRO: GEOGRAPHICAL MAPPING
• MAPPING AND VISUALIZING INFORMATION (TABLEAU)
• USEFUL FOR FAST PROOF-OF-CONCEPTS
• MAPPING INFORMATION TO PHYSICAL LOCATIONS (TABLEAU AND ARCMAP/ ARCGIS PRO)
58
RAPIDMINER STUDIO: DATAMINING
• DATA DESCRIPTIONS: PATTERN IDENTIFICATION VIA SCATTER MATRIX VISUALIZATIONS (BASED ON VARIABLES); CLASSIFICATION MODELS (NAÏVE BAYES, AND OTHERS)
• VARIABLE ASSOCIATIONS: LINEAR REGRESSIONS; LOGISTIC REGRESSIONS (WITH BINARY TARGET VARIABLES)
• MACHINE LEARNING ALGORITHMS FOR LATENT DATA: ARTIFICIAL NEURAL NETWORKS (EXTRAPOLATED INDEPENDENT VARIABLES TO DEPENDENT VARIABLE); GENETIC ALGORITHMS; DECISION TREES, AND OTHERS
59
RAPIDMINER STUDIO: DATAMINING (CONT.)
• TEXTUAL ANALYSIS: INDUCTIVE CLUSTERING (VIA K-MEANS CLUSTERING) FOR TEXTUAL AND QUANTITATIVE VARIABLE DATA (AND OUTPUT AS CORRELATION MATRICES AND CENTROID PLOT VIEWS); DOCUMENT VECTOR MODELS
• SENTIMENT ANALYSIS IN TEXT CORPUSES
ADDITIONAL GENERAL FEATURES
• EASY GRAPHICAL USER INTERFACE (GUI)
• END-TO-END SEQUENCE FROM DATA PROCESSING TO MODELING TO FINAL ANALYTICAL OUTPUTS
• MODEL CROSS-VALIDATION AND PERFORMANCE METRICS
• VARIOUS INTERMEDIATE AND FINALIZED DATA VISUALIZATIONS
60
THE RESEARCHER INTERPRETIVE LENS
61
RESEARCHER SUBJECTIVITY
• EVEN MACHINE-COLLECTED AND (PARTIALLY) MACHINE-PROCESSED INFORMATION HAS TO ULTIMATELY BE ANALYZED BY PEOPLE
• THE IMPORTANCE OF DOMAIN EXPERTISE; THE IMPORTANCE OF TRAINED NAIVETÉ
• RESEARCHERS ENGAGE IN…
• SOME MANUAL “CLOSE READING” INTERPRETATION OF TEXTUAL DATA
• DECIDING ASSERTABILITY (BASED ON THE EVIDENCE)
62
MITIGATIONS FOR HUMAN RESEARCHER SHORTCOMINGS
(EXPECTATIONS, EGOS, COGNITIVE BIASES)
• STRENGTH OF THE SURVEY DESIGN, RANDOM SAMPLING, MULTIPLE DATA VISUALIZATIONS
• RESEARCHER TRAINING: HEALTHY SKEPTICISM, NOT FINALIZING THE DATA UNTIL EVERYTHING HAS BEEN ANALYZED
• PEER REVIEW PROCESSES FOR PRESENTATIONS AND PUBLICATION
• BROAD (OR LIMITED) PUBLICATION OF DATASETS FOR EXTERNAL CHALLENGE AND VERIFICATION
• RESEARCH OVERSIGHT
63
THE PROBLEM OF HUMAN MANIPULATION OF DATA
• FREEZING DATA AND DATA RESULTS (AGAINST CHANGE)
• THE “UNTHINKING DRIVE-BY-GPS APPROACH”: THE “3 DAYS INTERPRETATION” (AND THE INSIDIOUSNESS OF PRE-EXISTING MENTAL MODELS)
• THE DELETION OF DATA RECORDS (WITH SURVEY ADMIN ACCESS)
• INCORRECT USE OF TECHNOLOGY SYSTEMS
• INCORRECT APPLICATION OF PROCESSES
• MISREADING DATA (GIVEN ITS COMPLEXITY)
• THE TWEAKING OF DATA IN MISLEADING WAYS
• THE NOT WANTING TO SEE
64
SOME REAL-WORLD CASESWITH SOME LIVE DEMOS
65
CASE 1: A MULTI-COUNTRY MULTI-GRAIN MULTI-YEAR SURVEY
MAIZE, SESAME, WHEAT, AND CHICKPEA DATA
2015
66
AN OVERVIEW
• THE PRINCIPAL INVESTIGATORS (PI’S) FROM MULTIPLE COUNTRIES
• TOPICS INVOLVING CROPS, CROPPING METHODS AND TECHNOLOGIES, ECONOMIC AND BUDGETARY ISSUES, FARMER LIFESTYLE ISSUES, SOURCES OF DATA AND TRAINING, AND OTHER DETAILS
• ENUMERATORS AND TRANSLATORS
• A TARGET POPULATION OF THOSE WHO FARM IN A COUNTRY IN EAST AFRICA
67
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS
ENHANCING THE SURVEY AND SURVEY PROCESSES
• USING COORDINATED UNIVERSAL TIME (UTC) TO DATE-STAMP RESPONSES AND RECODED TO EASTERN AFRICA TIME (EAT)
• CLARIFYING AGROECOLOGICAL POSITIONS OF RESPONDENTS
• ANALYZING COMPLETION RATES; POINTS AT SURVEY DROPOUT
• ANALYZING QUESTIONS WITH LOW RESPONSE RATES
• LANGUAGE CLARITY ISSUES WITH QUESTIONS
• ENSURING THAT THE QUESTION ASKED AND THE STRUCTURED QUESTION ON QUALTRICS ALIGNED (NOT AN OPEN-ENDED QUESTION WITH A CLOSE-ENDED AND NON-COMPREHENSIVE DROP-DOWN SELECTION; TEXT-BASED QUERIES WITHOUT THE TEXT ENABLEMENT;
68
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS (CONT.)
ENHANCING THE SURVEY AND SURVEY PROCESSES (CONT.)
• ENHANCING CLARITY ABOUT SEQUENCES OF ACTIVITIES FOR WHICH THERE ARE OVERLAPS (NON-MUTUAL EXCLUSIVITY)
• REPORTING NOT INTERPRETING: NOT MAKING ASSUMPTIONS ABOUT TEXTUAL DATA (REGIONS, NAME BRANDS, FOODS, DIALECTS) THAT MAY BE SYNONYMOUS (CLOSE SPELLINGS, CLOSE PRONUNCIATIONS) BUT ENABLING THE EXPERTS TO DO THE SORTING
• AVOIDING HIDING OR MASKING DATA, ERRING ON THE SIDE OF LEAVING DATA RAW
• SUGGESTING DROPDOWNS FOR SOME OF THESE…WITH AN ADDED TEXT FIELD FOLLOW-ON TO CAPTURE ANY OTHER POSSIBLE MISSED DATA
69
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS (CONT.)
ENHANCING THE SURVEY AND SURVEY PROCESSES (CONT.)
• NUMERACY, LITERACY, GEOGRAPHICAL, BUDGETARY, AND OTHER CHALLENGES: MIXED NUMERICAL PARAMETERS (SUCH AS ML PER HECTARE, ML PER QUINTAL AND THEN OTHER VARIATIONS)
• USING THE LOWEST COMMON DENOMINATOR TO REPRESENT MEASURES (AND OFFERING CONVERSIONS AND DEFINED TERMS AND UNDERSTANDINGS)
• CLARIFYING WHEN SOMETHING IS A RATE (SUCH AS A PARTICULAR FOOD CONSUMPTION AMOUNT OVER TIME VS. A ONE-OFF)
• NOT ENABLING MUTUAL EXCLUSIVITIES
70
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS (CONT.)
ENHANCING THE SURVEY AND SURVEY PROCESSES (CONT.)
• PARTICIPANT PRIVACY: THE HIDING OF RESPONDENT AND ENUMERATOR NAMES (AND THEIR CONTACT TELEPHONE NUMBERS) IN THE REPORT (BY POINTING ALL PI’S TO THE ARCHIVED SURVEY FOR SENSITIVE DATA)
• PILOT-TESTING SURVEYS: ENCOURAGING AWARENESS OF THE NEED TO PILOT TEST AND TO IMPROVE THE SURVEY WITH EACH ITERATION (WITHOUT CHANGING SUBSTANCE, IF TRENDLINE ANALYSIS IS NEEDED); MAINTAINING A PRISTINE MASTER, COLLECTING DATA ABOUT CHALLENGES WITH A SURVEY AND ITS USE IN THE WORLD
71
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS (CONT.)
THE APPROACH
• COLLECTION OF ALL RAW DATA (STRUCK THROUGH AND LABELED WITH “FOR POSSIBLE RESEARCH REFERENCE ONLY” AND HIGHLIGHTED WITH SIGNALING COLORS) AND MAKING THESE AVAILABLE FOR DOUBLE-CHECKING AND CONTENT ANALYSIS (INCLUDING EVALUATION IN OTHER DATA ANALYSIS TOOLS)
• CONNECTING RAW DATA WITH PROCESSED DATA (USING Q1, Q2, NAMING PROTOCOLS THROUGH ALL FILES AND DOCUMENTS FOR EASY LINKING AND SEARCHABILITY)
• AVAILABILITY OF ALL SUPPORT FILES (SUCH AS THOSE FROM WHICH TABLES WERE MADE) TO THE PI-S FOR CLARITY
• SCREENSHOTS FROM WITHIN QUALTRICS FOR QUICK SUMMARY DATA (OF SURVEY RESULTS FEATURES) BUT ACTUAL ANALYZED DATA FROM DOWNLOADS FOR ALL OTHER TABLES AND CHARTS (NON-TRIVIAL LIMITATIONS TO PROCESSED DATA USED ALONE, WITHOUT CONSIDERING THE UNDERLYING RAW DATA)
72
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS (CONT.)
• FULL DOCUMENTATION ABOUT ALL DATA HANDLING (WITH STEP-BY-STEPS FOR TRANSFER TO PIS); CLEAR DATA PROVENANCE
• ALIGNING THE DATA ANALYSIS WITH THE SEQUENCE OF THE ORIGINAL SURVEYS (FOR COHERENCE) • ANALYSIS OF ALL ELECTRONIC DATA, NOT THE ONE SET OF MANUALLY COLLECTED DATA FOR CHICKPEA
FARMING (HANDLED BY THE LOCAL PI) • NON-RELEASE OF PARTIAL SURVEY DATA UNTIL FULL SETS EXTRACTED AND ANALYZED
• SUMMARIZING NUMBERS AND VISUAL DATA IN WORDS; SUMMARIZING TEXTUAL DATA QUANTITATIVELY AS WELL (FREQUENCY COUNTS) AND IN TABLE AND CHART FORMAT
• ENABLING CLASSIC ELEMENTS (MIN-MAX DATA RANGES, AVERAGES/MEANS, MEDIANS, MODES, FREQUENCY COUNTS, CLASSIC DATA DISTRIBUTION VISUALIZATIONS LIKE LINE CHARTS, BAR CHARTS, AND OTHERS); ENABLING PHYSICAL MAPPING AND LESS COMMON DATA REPRESENTATIONS 73
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS (CONT.)
DATA AGGREGATION AND DISAGGREGATION
• BREAKING APART MULTI-PART QUESTIONS FOR ANALYSIS IN SEPARATE PARTS
• REFRAMING COMPLEX QUESTIONS INTO MULTIPLE PARTS FOR FULL DATA EXPLOITATION • EXTRACTING LARGE MIXED DATASETS FROM SINGLE QUESTIONS
• BREAKING APART MENTIONED FOODS (TO ATOMISTIC ELEMENTS)
• MIXING PERSONAL PROTECTIVE CLOTHING AS FULL SETS (FROM HEAD TO TOE) *AND* AS INDIVIDUAL ELEMENTS (FROM HEAD TO TOE) (BECAUSE THE DATA MEAN DIFFERENT THINGS)
• OFFERING SIDE-BY-SIDES OF RELATED PROCESSED AND RAW DATA TABLES
74
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS (CONT.)
• HANDLING TEXTUAL DATA: FAST COUNTING FROM OPEN-ENDED TEXT RESPONSES (IN EXCEL) FOR FREQUENCY BAR CHARTS; BULLETED SUMMARIES IN SOME ELEMENTS
• AS LITTLE DATA “LOSSINESS” AS POSSIBLE
• FOCUS ON FACTS, NOT INTERPRETATION
• CLEAN LINES OF LOGIC, AVOIDANCE OF CONFLATION (OF VARIABLES, OF CONCEPTS, OF TERMS)
• LOOKING FOR VERBATIM QUOTABLES (SIC), ANOMALIES, AND UNUSUAL CASES; LOOKING FOR AMBIGUOUS ELEMENTS AND THE NEED TO DISAMBIGUATE (WITHOUT DOING SO INTRUSIVELY)
• EXPLORING FOR CATEGORICAL DATA ANALYSIS POSSIBILITIES (INCLUDING MANY MACHINE-ENHANCED ONES, SUCH AS DATAMINING AND TEXT MINING TOOLS)
75
A NON-EXPERT AND OUTSIDER APPROACH TO FIRST-RUN DATA ANALYSIS (CONT.)
• REDUNDANT DATA STORAGE: MULTI-SITE DATA STORAGE
• ADDITIONAL DATA EXPLOITATION: PROVIDING ADDITIONAL SUGGESTIONS FOR DATA EXPLOITATION (MORE ON THIS IN THE CONCLUDING SUMMARY)
• BEING CLEAR ABOUT THE ADDITIONAL DATA CLEANING, DATA STRUCTURING, AND OTHER WORK REQUIRED
• BEING CLEAR ABOUT THE QUALIFIERS THAT HAVE TO BE APPLIED TO THESE ASSERTIONS
76
SOME LESSONS LEARNED (+)
• GO AMBITIOUS. GO FOR BREADTH AND DEPTH. GO FOR LEARNING ON MULTIPLE RELATED DIMENSIONS.
• BE GENEROUS WITH DATA WHERE POSSIBLE. USE SURVEYS TO BENEFIT THE PROFESSIONAL WORK OF ALL THE PRINCIPAL INVESTIGATORS (PI-S) ON THE PROJECT. LEARN FROM EACH OTHER.
• USE DATA TO IMPROVE POLICIES, MARKETS, PRACTICES, AND PEOPLE’S AWARENESS.
• WELL DESIGNED SURVEYS MAY BE APPLIED ACROSS A RANGE OF CONTEXTS AND LOCALITIES. THEY ARE NOT JUST MADE FOR SINGLE CONTEXTS. THEY MAY BE USED COLLABORATIVELY IN INTERNATIONAL CONTEXTS.
• BE SENSITIVE TO ON-GROUND REALITIES. DEVELOP AN EAR FOR HEARING WHAT IS BEING SAID.
77
SOME LESSONS LEARNED (+) (CONT.)
• THERE ARE BENEFITS TO BEING AN ANONYMOUS (BUT FULLY RESPONSIBLE) DATA ANALYST:
• NO BENEFIT TO ME TO HAVE BYLINE CREDIT IN HIGHLY DISPARATE FIELDS (HAVING THE APPLIED SKILLSET IS MORE IMPORTANT)
• LIMITS RISK IF SOME DATA ARE NOT PROPERLY PROVENANCED OR REPRESENTED IN THE FINAL PRODUCTS (PUBLICATIONS, PRESENTATIONS, DATASETS)
• MINE IS A SUPPORT POSITION WITH WORK PAID OUT BY THE FUNDER (USAID, IN THIS CASE)
• MAY HINDER THE WORK WITH THE PRINCIPAL INVESTIGATORS (PI’S) AND MY OWN SUPERVISOR (CLEARS THE AIR IF BYLINE CREDIT IS OFF-THE-TABLE)
• SOME PI’S SEE BYLINE CREDIT AS A FORM OF PAYMENT (IT’S NOT, AND IT’S ALSO NOT GENERALLY MONETIZABLE…AND IS NOT GENERALLY FUNGIBLE)
• BYLINE CREDIT SOMETIMES BRINGS EGOTISM INTO PLAY (NO THANKS!)
78
SOME LESSONS LEARNED (+) (CONT.)
• AVOID UNNECESSARY FETISHIZING OF SECRECY AND OVER-PROTECTION OF INFORMATION WHERE THAT IS NOT NECESSARY; USE PROPER JUDGMENT
79
SOME LESSONS LEARNED (-) (CONT.)
• ENUMERATOR TRAINING: TRAINING OF ENUMERATORS (NEUTRALITY, NORMING, NOT USING “COPY AND PASTE” RESPONSES, PHONETIC SPELLINGS INSTEAD OF ACTUAL SPELLINGS), PARTICULARLY FOR ENUMERATORS AT A DISTANCE
• SURVEY DESIGN:
• FOLLOW-ON QUESTIONS TREATED AS CENTRAL ONES, RESULTING IN SOME QUESTIONS WITH LOW RESPONSE RATES
• FAILURE TO USE SURVEY DESIGN TO MITIGATE FOR SOME CHALLENGES IN LITERACY / NUMERACY / MULTI-LINGUAL APPROACHES / SPELLING
• FAILURE TO USE SURVEY DESIGN TO DEAL WITH PLACE AND ORGANIZATION NAMES (AND DISAMBIGUATION)
80
SOME LESSONS LEARNED (-) (CONT.)
• ATTENTION TO DETAILS: DOUBLE-CHECK ALL WORK. • SLEEP ON IT, AND THEN REVISIT ASSUMPTIONS AND UNDERSTANDINGS AND DETAILS.
• ENGAGING ANOMALIES: RUN ANOMALOUS FINDINGS TO GROUND. CORRECT FOR MISUNDERSTANDING OF DATA.
• PROPER USE OF TOOLS: MAGNIFY ACCURACY AND EFFICIENCY USING VARIOUS TECHNOLOGIES (COUNTS, SORTS, FILTERING, ORDERING, AND OTHERS).
• TIME AND DATA CRUNCHING: PROPER DATA ANALYSIS TAKES TIME AND EFFORT. THAT SAID, IT CAN BE DONE FAIRLY QUICKLY AND ACCURATELY, WITH PRACTICE. (KEEP THE CLIENT IN THE LOOP. PRACTICE. WORK TO IMPROVE.)
81
CASE 2: INFORMATION TECHNOLOGY (IT) SATISFACTION SURVEYS
82
AN OVERVIEW
• A SURVEY DERIVED FROM ONE CREATED AT STANFORD UNIVERSITY AND FOUND ONLINE (BY A PRIOR TEAM MEMBER); REVISED, LOCALIZED, AND RE-SEQUENCED (TO NEUTRALLY INTRODUCE TOPICS FOR RESPONDENTS, TO DEFINE TERMS)
• MULTIPLE OBJECTIVES: IMPROVING CUSTOMER SERVICE, INFORMING IT LEADERSHIP, AND PUBLICIZING IT SERVICES TO THE BROAD PUBLIC
• STRATIFIED RANDOM SAMPLING
• SURVEY PILOT-TESTING AND REVISION
• ACCESSIBILITY ENABLED (TIMED-TEXT TRANSCRIPTION, ALT-TEXTING)
83
AN OVERVIEW (CONT.)
• VERBATIM QUOTES / STORIES (FOR LATER USE IN PUBLICATIONS AND PRESENTATIONS)
• WHAT EXPERTS BRING TO THE DATA (SUCH AS KNOWING WHAT IS / IS NOT PROBABLE; CONTEXTUAL INSIGHTS AND SALIENT POINTS; RELATED IN-FIELD RESEARCH; PROFESSIONAL PERSPECTIVE, AND OTHERS)
• WHAT NOVICES BRING TO THE DATA (SUCH AS FRESH INSIGHTS AND NEW INTERPRETATIONS; QUESTIONS)
84
SOME LESSONS LEARNED (+)
• BRING IN ALL CLIENTS / STAKEHOLDERS, AND MAKE IT A COLLABORATIVE EFFORT. BE REALLY AND TRULY OPEN.
• CAPTURE A BROAD VIEW OF THE CAMPUS INFORMATION TECHNOLOGY (IT).
• USE THE SURVEY PARTIALLY TO EDUCATE ABOUT THE WIDE SERVICES AVAILABLE.
• TAKE THE LEARNING FROM THE SURVEY INSTRUMENT, AND LEARN FROM IT, AND IMPROVE SERVICES.
85
SOME LESSONS LEARNED (-) (CONT.)
• AVOID POLITICIZING DATA • NOT REMOVING NON-RESPONSE RATES ON QUESTIONS FROM CHARTS
• NOT REMOVING NUMBERS OF RESPONSE COUNTS AND GOING WITH PERCENTAGES ALONE (WHICH ARE MEANINGLESS WITHOUT A COUNT BASELINE)
• NOT SELECTIVELY RELEASING ONLY LITTLE PARTS OF THE SURVEY FINDINGS; NOT FRAMING FINDINGS TO THE POINT OF WRITING A PRESS RELEASE FOR PUBLIC RELATIONS (PR) INFORMATION
• NOT DISMISSING WHAT ONE DOESN’T WANT TO SEE / HEAR / KNOW / ADDRESS
• NOT RELEASING INFORMATION TO PLEASE ADMINISTRATORS
• KEEP QUESTION AND SURVEY CONSISTENCY IN TERMS OF TRENDLINE DATA• AVOID QUESTION INCONSISTENCIES YEAR-TO-YEAR
• WHEN QUOTING, QUOTE VERBATIM AND QUOTE COMPREHENSIVELY 86
SOME LESSONS LEARNED (-) (CONT.)
• IN THE IT SURVEY FOR FACULTY / STAFF / ADMIN, DO NOT SELECTIVELY OMIT IT PERSONNEL BECAUSE THEY DO HAVE INSIGHTS THAT SHOULD BE SHARED. THEY SHOULD BE INCLUDED IN THE FULL SET FROM WHICH THE RANDOM STRATIFIED SAMPLE IS DRAWN.
• DO OMIT RECOGNIZABLE DATA FROM THE RESULTS THAT ARE DRAWN FROM THE SURVEY RESPONSES (TO PROTECT PRIVACY).
• SAMPLE MORE BROADLY IN ORDER TO BE MORE REPRESENTATIVE IN TERMS OF DATA COLLECTION, AND BE INCLUSIVE.
• ALLOW SPACE (TECHNOLOGICAL, MENTAL, AND OTHERS) FOR A WIDE RANGE OF POTENTIAL RESPONSES.
87
SOME LESSONS LEARNED (-) (CONT.)
• WORK TO INFORM RESPONDENTS ABOUT WHAT IS GOING ON TECHNOLOGICALLY. PROVIDE SUFFICIENT DETAILS FOR QUESTIONS. ALLOW INFORMED FEEDBACK.
• USE CORRECT TECHNOLOGICAL PHRASING (TO BE ACCURATE, TO AVOID RIDICULE).
• CREATE OPEN-ENDED QUESTIONS, EVEN AS IT ALLOWS VENTING, INCLUDING SOME F-BOMBS
• ACTUALLY USE THE DATA COLLECTED. JUSTIFY SURVEY TAKER TIME AND EFFORT (10 – 15 MINUTES).
• USE THE PROPER BALANCE TO INCENTIVIZE PARTICIPATION IN SUCH SURVEYS.
• AVOID SKEW IN THE DESIGN. DO NOT DRIVE TRAFFIC TO DESIRED RESPONSES.
88
SOME LESSONS LEARNED (-) (CONT.)
• LEARN FROM ITEM ANALYSIS OF THE SURVEY QUESTIONS. RECORD CRITIQUES OF THE SURVEY, SO THE INSTRUMENT MAY BE IMPROVED FOR THE NEXT ROUNDS.
• KEEP A PRISTINE MASTER SURVEY.
• ARCHIVE SURVEYS FOR WHICH DATA HAS BEEN COLLECTED. DO NOT WRITE OVER A SURVEY WHICH ALREADY HAS DATA COLLECTED.
89
FULL EXPLOITATION OF SURVEY DATA
90
FULL EXPLOITATION
• PURPOSEFUL DATA COLLECTION (AND TRACE AND METADATA): DESIGN THE SURVEY INSTRUMENTS AND METHODS IN ORDER TO COLLECT AS MUCH RELEVANT DATA AS POSSIBLE. STAY ATTENTIVE TO WHAT IS KNOWABLE, WHETHER BY INTENTION OR ACCIDENT. STAY ATTENTIVE TO “TRACE” DATA, INFORMATION THAT IS A BYPRODUCT OF PROCESSES.
• BE AWARE THAT YOU WILL ALWAYS RETRIEVE MORE INFORMATION THAN YOU INTENDED (OR WERE AWARE OF). SOME OF THAT INFORMATION WILL BE PROFESSIONALLY AND ETHICALLY EXPLOITABLE. WHAT IS NOT PROFESSIONALLY / ETHICALLY EXPLOITABLE SHOULD BE ARCHIVED SECURELY AND LEFT ALONE. THERE ARE LIMITS TO EXPLOITATION.
• MULTIPLE INFORMATION STREAMS: CONSIDER HOW TO CROSS-REFERENCE TRUSTED INFORMATION IN ORDER TO COME UP WITH FRESH INSIGHTS AND GET CLOSER TO “GROUND TRUTH”. 91
FULL EXPLOITATION (CONT.)
• BROADER INTERPRETABILITY AND CONTENT ANALYSIS: ENCOURAGE VARIANT POINTS-OF-VIEW, DISSENSUS, AND WIDE-RANGING INTERPRETATIONS OF THE DATA. TRY TO DEBUNK THE WORKING THEORIES. SEE HOW THE EMPIRICAL DATA LINES UP. RETHINK THE “GO-TO” INTERPRETATIONS. ANALYZE ANOMALIES WITH FRESH INTERPRETATIONS.
• APPLY CONTENT ANALYSIS TO RELATED TEXTS.
• SURFACE INSIGHTS FROM STYLOMETRY.
• VARYING UNITS OF ANALYSIS / GRANULARITY: CONSIDER “ISOLATES” FROM THE DATA, AND INTERPRET WHAT THESE COULD MEAN. EXTRACT CASES. EXTEND THE DATA BY DISAGGREGATING PARTS AND PIECES. CONSIDER SYSTEMS AND CONTEXT ISSUES.
92
FULL EXPLOITATION (CONT.)
• TRENDS OVER TIME: CONSIDER HOW TO EXPLOIT PRIOR, RELATED SURVEY DATA IN ORDER MAKE ASSERTIONS OF TRENDS OR CHANGES-OVER-TIME. (THERE IS ALWAYS A TIME ELEMENT TO DATA.)
93
FULL EXPLOITATION (CONT.)
• BROADER WAYS OF KNOWING: CONSIDER OTHER RESEARCH ANALYSIS METHODS AND WAYS OF KNOWING…SUCH AS SOCIAL NETWORK GRAPHING, MATRIX QUERIES, MAPPING TO PHYSICAL LOCATIONS, INTRA-RESPONDENT INSIGHTS, DATA MINING, AUTOMATED (UNSUPERVISED) TEXT ANALYSIS, AND SO ON.
• SEPARATE THE DIFFERENT RESEARCH APPROACHES.
• DOCUMENT ALL STEPS. SAVE ALL INTERMEDIATE VERSIONS OF FILES AND DATASETS.
• MAKE SURE THAT CLARITY AND COHERENCE ARE NEVER LOST.
• BE CLEAR WHERE EVERY PIECE OF DATA BEING USED COMES FROM…AND HOW TRUSTWORTHY THAT DATA IS. (CONFIDENCE MATTERS. UNDERCLAIM RATHER THAN OVERCLAIM.)
• ADDITIONAL LEADS: PURSUE ADDITIONAL LEADS FROM THE INFORMATION BY FORMULATING NEW HYPOTHESES AND METHODS FOR RESEARCH. EXPLORE WHAT IS KNOWABLE. 94
FULL EXPLOITATION (CONT.)
• SECOND- AND THIRD-ORDER EFFECTS: BE MINDFUL OF IMPLICATIONS OF THE DATA AND HOW THAT DATA MAY BE PERCEIVED, USED, AND FRAMED. ALL DATA HAVE POLITICAL IMPLICATIONS, AND IT IS IMPORTANT TO BE AWARE OF THOSE WHILE BEING DISCIPLINED ENOUGH NOT TO CHANGE UP THE FINDINGS TO FIT A POLITICAL MOTIVE. (WORK BEYOND THE RESEARCHER’S OWN LIMITS AND SUBJECTIVITY BY PUTTING INTO PLACE CHECKS AND METHODS…)
• IMPROVED RESEARCH TOOLS AND METHODS: USE THE SURVEY FINDINGS TO SHARPEN RESEARCH TOOLS (AND METHODS) FOR MORE ACCURATE DATA COLLECTION IN FUTURE ITERATIONS.
95
FULL EXPLOITATION (CONT.)
• GETTING MORE EYES ON THE DATA: IF THE RESEARCH CONTEXT ALLOWS, SHARE THE DATASET…AND SEE WHAT OTHERS (TRAINED RESEARCHERS AND THE BROADER PUBLIC—OCCASIONALLY) MAY FIND.
• USE PROPER NAMING PROTOCOLS FOR ALL DATA. MAINTAIN RECORDS AND THE PROPER RELEASES. ARCHIVAL THE DATA IN A FUTURE-PROOFED WAY FOR POSSIBLE FUTURE RE-USE IN OTHER RESEARCH CONTEXTS.
• EXPLOIT THE INSIGHTS OF BOTH EXPERTS AND AMATEURS / NOVICES.
• NOT UNETHICAL, UNPROFESSIONAL, UNAPPROVED, PARTICIPANT-UNINFORMED, NON-COMPLIANT (PER HUMAN SUBJECTS REVIEW RESEARCH STANDARDS) EXPLOITATION OF DATA
96
REALITY CHECKS & CAVEATS
• PRINCIPAL INVESTIGATOR (PI) COMFORT LEVELS: MOST RESEARCHERS WILL NOT MOVE BEYOND WHERE THEY’RE COMFORTABLE IN TERMS OF DATA ANALYSIS AND PROCESSING.
• ALSO, CUSTOMERS FOR THE DATA ARE POLITICAL INDIVIDUALS AND WILL NOT GENERALLY TAKE RISKS WITH DATA. POLITICAL SURVIVAL AND COMFORT ARE TOP-OF-MIND OFTENTIMES.
• COSTS / REQUIRED INVESTMENTS: ADDITIONAL ANALYTICS WILL REQUIRE HYPOTHESIZING AND SETTING UP TEST VALUES. ALL TYPES OF DATA PROCESSING AND ANALYSIS REQUIRE NEW WAYS OF STRUCTURING THE CAPTURED DATA FOR QUERYING, ANALYSIS, AND VISUALIZATION. THE COST OF ANALYSIS IS GENERALLY NON-TRIVIAL. ADDITIONAL WORK REQUIRES PI INTEREST AND FOCUS.
• RELEVANCE AND APPLICABILITY: IT IS IMPORTANT TO TIE NEW FINDINGS TO APPLICABLE DECISIONS AND ACTIONS. IT IS IMPORTANT TO GET PAST THE “SO WHAT?”
97
REALITY CHECKS & CAVEATS (CONT.)
• CONFERENCES & PUBLISHING VENUES: MOST VENUES FOR PRESENTATION AND PUBLISHING ACCLIMATE TO CERTAIN TYPES OF DATA ONLY; ANYTHING OUT-OF-THE-ORDINARY WILL REQUIRE MORE TEXTUAL SETUP AND EXPLANATION (AND DEFENSE). NOVELTY DOES HAVE SOME INHERENT VALUE THOUGH.
98
CONTACT AND CONCLUSION
• DR. SHALIN HAI-JEW, INSTRUCTIONAL DESIGNER, ITAC, K-STATE
• 212 HALE / FARRELL LIBRARY
• 1117 MID-CAMPUS DRIVE NORTH, MANHATTAN, KS 66506-0110
• 785-532-5262
THANKS TO DR. SUBRAMANYAM BHADRIRAJU (K-STATE) FOR THE INVITATION TO WORK ON HIS PROJECT AND HIS GENEROSITY IN ALLOWING ME TO USE SOME PROJECT-RELATED LEARNING TO INFORM THIS PRESENTATION. THE ACTUAL PRESENTATION IS ACHIEVED WITH LIVE ACCESS TO VARIOUS SYSTEMS AND DATA; THE ONLINE VERSION HERE LACKS THE VISUALS AND SEQUENCING FROM THE F2F PRESENTATION. 99