data analysis & interpretation - …€¢ build & practice data analysis &...

DATA ANALYSIS & INTERPRETATION

August 25, 2017 October 20, 2017

PERFORMANCE MANAGEMENT SERIES Data Management Focus

• Structure

o Workshops & Trainings

• Content

o Data Collection

o Data Analysis & Interpretation

o Data Communication

OBJECTIVES • Enthuse evaluative thinking

• Encourage performance managament

OBJECTIVES • Build & practice data analysis &

interpretation skills

o Cleaning data

o Quantitative & Qualitative analysis processes

o Finding meaning in analyzed data

• Work with existing program data

o Clean data to prepare for analysis

o Analyze program data

o Identify 2-3 key findings from program data

AGENDA

9:00 – 9:10 Intro & Overview

9:10 – 9:30 Cleaning/auditing Data Content

9:35 – 9:50 Cleaning/auditing Data Practice

9:50 – 10:00 Data Analysis Content

10:00 – 10:35 Data Analysis Practice

10:35 – 10:40 Data Interpretation Content

10:40 – 11:25 Data Interpretation Practice

11:25 – 11:30 Closing

BEST PRACTICES FOR

EFFECTIVE PERFORMANCE MANAGEMENT PERFORMANCE MANAGEMENT

PERFORMANCE MEASUREMENT & MANAGEMENT

Processes and systems to:

GATHER

MEANINGFUL DATA

MONITOR IMPLEMENTATION

& IMPACT

WHAT IS PERFORMANCE MANAGEMENT?

What went well?

What didn’t go well?

How do we improve?

Repeat

Good Intentions

Counting Outputs

Measuring Outcomes

Managing Performance

PERFORMANCE MANAGEMENT CONTINUUM WHAT IS PERFORMANCE MANAGEMENT? PERFORMANCE MANAGEMENT CONTINUUM

Good Intentions

Counting Outputs

Measuring Outcomes

Managing Performance

DATA ANALYSIS & INTERPRETATION

D R I V E S

BEST PRACTICES FOR

EFFECTIVE PERFORMANCE MANAGEMENT CLEANING DATA

Why collect data?

Why clean data?

DATA QUALITY & INTEGRITY

Cleaning data ensures it is

ACCURATE & FIT FOR ITS INTENDED USE

UNCLEAN DATA • Varying formats

o Jan 12, 2016 v. 1/2/16 v. 01/02/2016

o Caucasian v. white

• Errors in data

o Negative ages

• Blanks/missing information

• Large jumps in data

o Savings of $12,000 to $150,000 in 6 months

PREPARES DATA FOR ANALYSIS

Cleaning data prepares you for analysis

• Ensures data is free of errors

o Confirms or corrects discrepancies

• Identifies missing data

• Aligns formatting

• Determines process for unknowns/errors

How to clean data

DATA CLEANING PROCESS Reviewing and cleaning data

• Spot check random sample

• Sort/filter data

o Missing values

o Outliers (overly high/low values)

o Check feasibility (errors & discrepancies)

• Address abnormalities

EXAMPLES

Never modify the raw data.

• Use several worksheets when creating a spreadsheet.

• Always copy the raw data to a new worksheet and make the modifications on the new worksheet. If the new worksheet needs to be fixed or the process needs to start over, the data will not need to be extracted again.

• When finished, hide worksheets that don’t need to be seen by other users.

TIP

If your agency uses account numbers, clients numbers, program numbers, social security numbers, or zip codes with a leading zero, make sure the column is formatted as text. Always double check to make sure the leading zero is not dropped. If wrong, this may affect the data being analyzed.

PRACTICE 25-35 minutes

Clean your program data ensuring:

1. Aligned formatting is used

2. Identify & address missing data

a. Determine standard approach

3. Identify & address discrepancies/jumps in data

a. Are there obvious errors?

4. Write standard approach for

a. Missing data & errors

BEST PRACTICES FOR

EFFECTIVE PERFORMANCE MANAGEMENT DATA ANALYSIS

Why analyze data?

What variables do you analyze?

HOW TO “SLICE & DICE” DATA

• What question are you trying to answer?

o Examine variables that will answer or

influence that question

o Consider key variable relationships

o Identify appropriate calculations

How to analyze data

TYPES OF DATA

Quantitative Qualitative

• Numbers & statistics

• Often objective

• Often answers “what” questions

• Words/text & concepts

• Often subjective

• Often answers “why” questions

Qualitative Data Analysis

QUALITATIVE DATA ANALYSIS BASICS

• Less structured than quantitative analysis

• Not guided by universal rules

• Data reduction is key

o Leads to identifying key themes

QUALITATIVE DATA ANALYSIS BASICS

• Predefined codes

• Emergent codes

• Can make numerical

Categorize/ code

responses

• What themes emerge that answer evaluation questions?

Identify trends and

themes

Often helps answer “why” questions

QUALITATIVE DATA ANALYSIS EXAMPLES

Why didn’t clients enact healthy behaviors they

learned about?

1. Because my partner does it

2. I just need to smoke to calm down

3. I get too stressed without smoking

4. My friends said it was lame to not smoke

5. I freak out if I don’t smoke

QUALITATIVE DATA ANALYSIS EXAMPLES

Quantitative Data Analysis

QUANTITATIVE DATA ANALYSIS BASICS Key Quantitative Calculations

Mean Median Mode

Variability Frequency

Distributions

QUANTITATIVE ANALYSIS PROCEDURES

Data Tabulation • Frequency & Percent Distributions

Descriptives Data • Mean, median, mode, range, etc.

Data Disaggregation • Break down across subcategories

Moderate/Advanced Analysis • Correlation, Regression, ANOVA

OUTCOME COMPARISONS

Data disaggregation among the change or benefit clients experience

OVER TIME

AGAINST TARGETS

WITH BENCH-MARKS

BY CLIENT GROUPS

BY SERVICE

Examine findings across all indicators

QUANTITATIVE ANALYSIS APPROACHES

TIP

As soon as you start manipulating data there is the potential of inaccurate data.

Always! Always! Always! Double check your data

Have another person check your data

Lisa Emily Julie

PRACTICE 25-35 minutes

Analyze your program data

*You can use Analysis Decision Tree if helpful

1. Identify evaluation questions to answer

2. Identify variables that influence those questions

3. Identify appropriate analysis approach and

calculations

4. Conduct analysis using qualitative coding or

quantitative calculations

BEST PRACTICES FOR

EFFECTIVE PERFORMANCE MANAGEMENT DATA INTERPRETATION

If you torture the data long enough, it will confess to anything. Ronald Coarse, Economics Nobel Laureate

Why interpret data?

FINDING MEANING IN DATA

How to interpret data

INTERPRETING DATA Interpretation is where meaning is found; consider

o Patterns, themes & deviations

o If results make sense

o Surprising findings & potential causes

o Focus areas for improvement

o Additional questions that arise

INTERPRETING TIPS

• Start with analysis of clean data

• Ask

o So what?

o What does this mean/tell me about my program?

o How do we use these findings?

• Try visualizations

• Allot time for interpretation

DATA INTERPRETATION MEETINGS • Include key stakeholders

• Present data

o Data parties/placemats

o Consider key variables & relationships

• Pose key questions around findings

o What surprises you/stands out?

o What factors may explain findings?

o What action should we take?

o Any new questions?

REMEMBER:

What questions are you trying

to answer?

CAUSATION & CORRELATION

Mutual relationship between variables

(positive or negative)

Causation Correlation

One factor leads to/causes another

PRACTICE 15-20 minutes

Interpret your analyzed program data

*Use Data Interpretation Guide if helpful

1. Identify evaluation questions to answer

2. Identify 2-5 key findings/answers from analyzed

data

3. Identify other stakeholders to involve

4. Consider ideas for presenting data

Why are data analysis & interpretation important to performance management?

THANK YOU!!! • Thank you for your time

• Please share feedback

• Other opportunities

o Program Impact Reviews

• Feel free to contact us

Emily Uzzle Lisa Goodman

314-539-4256 314-539-4217

emily.uzzle@stl.unitedway.org lisa.goodman@stl.unitedway.org