a wealth of information creates a poverty of attention―idries shah, reflections. prepared by...
TRANSCRIPT
Data Analysis and Reporting
“...a wealth of information creates a poverty of attention...”
-Herbert A .Simon
Melissa Chapman Haynes, Ph.D.
Senior Evaluator, Professional Data Analysts, Inc.
MESI Spring Training 2017
2Prepared by Professional Data Analysts, Inc.
“PEOPLE TODAY ARE IN DANGER OF DROWNING IN INFORMATION; BUT, BECAUSE THEY HAVE BEEN TAUGHT THAT
INFORMATION IS USEFUL, THEY ARE MORE WILLING TO DROWN THAN THEY NEED BE.
IF THEY COULD HANDLE INFORMATION, THEY WOULD NOT HAVE TO DROWN AT ALL.”
― IDRIES SHAH, REFLECTIONS
3Prepared by Professional Data Analysts, Inc.
WHAT PART OF THE PROCESS DOES DATA ANALYSIS AND REPORTING COME INTO PLAY?
“Justify conclusions”
Analyze the evaluation data
Determine what the findings say about your
program
“Use and share lessons learned”
Ensure stakeholders are engaged in the
evaluation process, are aware of the results
Use of evaluation results in decision-making
Those who participate in the evaluation should
have a beneficial experience
Minnesota Evaluation Studies Institute Spring Training 2017
4Prepared by Professional Data Analysts, Inc.
WHERE DOES DATA ANALYSIS FIT INTO THE EVALUATION PROCESS?
Minnesota Evaluation Studies Institute Spring Training 2017
5Prepared by Professional Data Analysts, Inc.
EVALUATION PLAN MATRIX
Minnesota Evaluation Studies Institute Spring Training 2017
6Prepared by Professional Data Analysts, Inc.
7Prepared by Professional Data Analysts, Inc.
SCOPE – WHAT ARE WE DOING TODAY?
Developing Qualitative Data Analysis Categories 5
3
4
2Analyzing and Interpreting Qualitative Data
Quantitative Data Analysis
Innovations and Technologies
Planning for Data Analysis 1
Reporting6
Section 1 – Planning for Data Analysis
9Prepared by Professional Data Analysts, Inc.
Data Analysis is an art…
Context matters
Process matters
Asking questions is essential!!
…What can be abstracted across different
contexts about data analysis?
Minnesota Evaluation Studies Institute Spring Training 2017
10Prepared by Professional Data Analysts, Inc.
Epicycles of Analysis – The Model
11Prepared by Professional Data Analysts, Inc.
Mapping the Five Core Activities of Data Analysis (Quant) to the Epicycles
1. Ask questions
2. Explore the data
3. Build statistical
models
4. Interpret results
5. Communicate
results
1. Set expectations
2. Collect data and
compare to
expectations
3. Revise data or
expectations as
needed
12Prepared by Professional Data Analysts, Inc.
Mapping the Five Core Activities to the Epicycles of Analysis
13Prepared by Professional Data Analysts, Inc.
Epicycles of Analysis – Cancer Example
Set Expectations
Collect Information
Revise Expectation
Question
EDA
Formal modeling
Interpretation
Communication
14Prepared by Professional Data Analysts, Inc.
Quantitative
o Descriptive
o Correlational
o Inferential
o Multivariate
Qualitative
o Thematic coding
o Case studies
o Success case method
o Content Analysis
o Grounded Theory
Systems Methods
o Outcome Mapping
o Social Network Analysis
o Process Monitoring of
Impacts
o Causal Loop Diagrams
Types of Data Analysis
15Prepared by Professional Data Analysts, Inc.
Mixed Methods
16Prepared by Professional Data Analysts, Inc.
Data Preparation Process
17Prepared by Professional Data Analysts, Inc.
Basic Data Cleaning Troubleshooting
18Prepared by Professional Data Analysts, Inc.
Activity: Planning for Data Analysis
Survey Data Analysis
Question or item numbers
What data analysis methods might you use?
Why would you use this analysis? What would you hope to learn?
20Prepared by Professional Data Analysts, Inc.
Activity: Planning for Data Analysis Debrief
What insights into data analysis did you gain from this activity?
If you were going to administer a survey to two different groups,
what other analysis might you consider?
What might be the impact on this evaluation if there were no
data analysis plan?
What implications does planning for data analysis have on an
evaluation budget?
Section 2 – Developing Qualitative Data Analysis Categories
23Prepared by Professional Data Analysts, Inc.
How to Develop Categories and Themes
What challenges did you experience in trying to sort the candy according to pre-determined categories?
After sorting the candy to what extent did you reach different conclusions about the data even though the data were the same? What implications are there for evaluation practice?
When might you use a predetermined set of categories for analyzing qualitative data? Where might the predetermined categories come from? What implications might using predetermined categories have on the evaluation’s findings?
How would you go about deriving categories or themes form the data if there were no predetermined categories?
What are the advantages and disadvantages of these two approaches to analyzing qualitative data?
24Prepared by Professional Data Analysts, Inc.
Coding Examples Excerpt
What are the negative aspects
of having your service member
deployed?
25Prepared by Professional Data Analysts, Inc.
Coding Examples Excerpt – Simple Coding
What are the negative aspects
of having your service member
deployed?
26Prepared by Professional Data Analysts, Inc.
Coding Examples Excerpt – Thematic Coding
What are the negative aspects
of having your service member
deployed?
Section 3 – Analyzing and Interpreting Qualitative Data
28Prepared by Professional Data Analysts, Inc.
Group A = Predetermined
categories applied to analysis
Review categories and
agree on definitions
Code data
Complete handout
Summarize data and
develop recommendations
Group B = Emergent categories
developed and applied
Review data and develop
categories based on themes
and patterns
Apply coding to Question 1
Complete handout
Summarize data and
develop recommendations
Pre-determined versus emergent categories
29Prepared by Professional Data Analysts, Inc.
Pre-determined versus emergent categories: Debrief
How similar or different were the findings among those in
Group A? In Group B?
What were some of the similarities and differences
between the results from Group A and Group B?
How would you describe your experience in analyzing
these data?
What surprised you about this work?
What insights did you gain from this activity that will be
useful the next time you collect and analyze qual data?
Section 4 – Quantitative Data Analysis
31Prepared by Professional Data Analysts, Inc.
Descriptive analysis
Correlational analysis
Inferential analysis
Small Group Activity
Section 5 – Innovations and Technologies
34Prepared by Professional Data Analysts, Inc.
Technologies for Qualitative Analysis
35Prepared by Professional Data Analysts, Inc.
Technologies for Quantitative Analysis
SPSS
SAS
Stata
R
Excel
Dedoose (mixed methods)
Section 6 – Reporting and Data Viz
37Prepared by Professional Data Analysts, Inc.
“Standard” Report Format
38Prepared by Professional Data Analysts, Inc.
Other Qualitative Report Formats
39Prepared by Professional Data Analysts, Inc.
Report and Support Use – Better Evaluation
http://www.betterevaluation.org/plan/reportandsupportuse
40Prepared by Professional Data Analysts, Inc.
CHART CHOOSER
Minnesota Evaluation Studies Institute Spring Training 2017
41Prepared by Professional Data Analysts, Inc.
Pop Culture meets Data Viz…
42Prepared by Professional Data Analysts, Inc.
Data viz can excite…
43Prepared by Professional Data Analysts, Inc.
And one more…
45Prepared by Professional Data Analysts, Inc.
Questions?