2013 swc beyond toc 2
TRANSCRIPT
-
7/27/2019 2013 Swc Beyond Toc 2
1/41
1
Beyond Theory of
Change
Tips and Strategies forCollecting, Aggregating, and
Reporting PM Outcomes
SW National Service Training ConferenceJuly 24, 2013
Susan Hyatt, Core Thought
www.core-thought.com
-
7/27/2019 2013 Swc Beyond Toc 2
2/41
2Session Description Knowing WHAT data to collect to demonstrate the
change your program creates is important.
However, data collection, aggregation and reporting toolsand processes are critical to insure you have valid and
reliable data on which to base your findings either makes
or breaks your efforts.
This session will provide you the step-wise information tohelp you insure your programs current performance
measurement approach is in alignment with your theory
of change and can yield the information you need to
clearly and confidently show your outcomes.
-
7/27/2019 2013 Swc Beyond Toc 2
3/41
3Learning Objectives
By the end of the session, you will:
Know the four types of outcomes and identify your optionsfor collecting data based on the outcome to be measured;Be able to identify the most appropriate data source and
cost effective data collection methodology to use to
measure your intended outcome; and
Know strategies for aggregating and summarizingperformance measurement data to make powerful
statements that make your program more attractive for
grants and other resources.
-
7/27/2019 2013 Swc Beyond Toc 2
4/41
4
Making the Case for
SupportCNCS funds program interventions
with a high likelihood of creating an
intended change in a communityneed
o Evidence-based interventions: Will
the proposed intervention (design,
dosage) likely lead to the desiredoutcome?
o Whats the cause and effect?
o What to measure to demonstrate
change?
-
7/27/2019 2013 Swc Beyond Toc 2
5/41
5
easur ng rogramPerformance
(Outcomes)
What is
Measure
-ment
Question?
Who
Would
Know?
TOC to
ClarifyDesiredOutcome How to
Collect
Info?
Then DoIt!
What
DoesInfo Tell
You? Tweak
Program
& Tell
YourStory
-
7/27/2019 2013 Swc Beyond Toc 2
6/41
6
Measure theAMOUNT of service
provided (Did you do what you said?)Number of people served (usually best)
Number of products/events/etc. created
What are Outputs?
Bean Counting
-
7/27/2019 2013 Swc Beyond Toc 2
7/41
7
Measure the CHANGES orBENEFITS
that occur because of service provided(individuals, organizations, communities, or theenvironment)
Increase - DecreaseMore - Less
Something is different
What are Outcomes?
The So What?
-
7/27/2019 2013 Swc Beyond Toc 2
8/41
8
Keep It Simple....You Cant
Measure EVERYTHING
Can read at grade
level
3rd Grade Reading Tutoring:
Whats the Best Outcome
to Measure?
Graduates from HS
-
7/27/2019 2013 Swc Beyond Toc 2
9/41
9
How Clear Are You
About What toMeasure?
Problem/Need
DesiredOutcome
ProgramActivities/Inte
rvention
Theory of Change:
Tool to Identify
Cause and Effect
Relationship
-
7/27/2019 2013 Swc Beyond Toc 2
10/41
10
What Are You Trying to
Change? (K-ABC)
KnowledgeSkill
Learning
new
information
or how to do
something
-
7/27/2019 2013 Swc Beyond Toc 2
11/41
11
What Are You Trying to
Change? (K-ABC)
AttitudeBelief
KnowledgeSkill
Learning
new
information
or how to do
something
Thoughts or
feelingsabout
something
-
7/27/2019 2013 Swc Beyond Toc 2
12/41
12
What Are You Trying to
Change? (K-ABC)
AttitudeBelief
KnowledgeSkill
Behavior
Learning
new
information
or how to do
something
Thoughts or
feelingsabout
something
Actions,
conduct,
or habits
-
7/27/2019 2013 Swc Beyond Toc 2
13/41
13
What Are You Trying to
Change? (K-ABC)
AttitudeBelief
KnowledgeSkill
Behavior Condition
Learning
new
information
or how to do
something
Thoughts or
feelingsabout
something
Actions,
conduct,
or habits
Circumstance
s or status
-
7/27/2019 2013 Swc Beyond Toc 2
14/41
14
What to Measure?
Teen Health Program
AttitudeBelief
KnowledgeSkill
Behavior Condition
Increased
knowledge of
healthy food
More willing
to eatvegetables
More frequent
exercise
Improved
physical
condition
-
7/27/2019 2013 Swc Beyond Toc 2
15/41
15
What Do TheseQuestions
Measure?IdentifyingOutputs and Outcome TypesACTIVITY
-
7/27/2019 2013 Swc Beyond Toc 2
16/41
16Now What?
You identified the desired change
(outcome) through your theory ofchange.
Are you clear about WHAT to measure?
AND
Are you measuring it the BEST way?
-
7/27/2019 2013 Swc Beyond Toc 2
17/41
17
Asking the Right
Question is CriticalIntervention: ABS driving school gives a 10 week
course that meets twice a week for 60 minutes that
includes classroom-based and on road lessons on
driving skills
Desired Outcome: Students have basic driving
proficiency.
-
7/27/2019 2013 Swc Beyond Toc 2
18/41
18
Asking the Right
Question
Question: Do you like driving?
Answer: I LOVE driving!!
Do you like driving? gets information about an
attitude not skill level.
-
7/27/2019 2013 Swc Beyond Toc 2
19/41
19
Asking the Right
Question
Question: Are you a skilled driver?
Answer: I think I am a GREAT driver!
Asking, Are you a skilled driver gets information
about self perception, a thought not actual skilllevel.
Self-ratings are subjective NOT objective.
-
7/27/2019 2013 Swc Beyond Toc 2
20/41
20
Asking the Right
Question
Question: Do you know the Colorado
driving laws?Answer: I got 100% correct on my
written drivers test!
Knowing state driving laws reflects knowledge
not actual skill level even though it is objective.
-
7/27/2019 2013 Swc Beyond Toc 2
21/41
21
Asking the Right
Question
Question: Have you demonstrated
you are a skilled driver?Answer: I passed my road test!
An on-road driving test DOES measure skill level or
proficiency and is objective.
-
7/27/2019 2013 Swc Beyond Toc 2
22/41
22Qualitative Data
Describes or characterizes through words
Focuses on meaning, experience or attitudesCollected through focus groups, interviews,
opened ended questionnaire items, and other
less structured situations.
Not the same as anecdotal Information
Qualitative Quality
-
7/27/2019 2013 Swc Beyond Toc 2
23/41
23Quantitative Data
Focus on numbers and frequencies.
Data which can be measuredLength, height, area, volume, weight, speed,
time, temperature, humidity, sound levels, cost,
ages, etc.
Quantitative Quantity
-
7/27/2019 2013 Swc Beyond Toc 2
24/41
24
Whats the Best Way to
Ask the Question?
Qualitative Quantitative
Please describe how you
have been feeling lately.
Thinking about the past week, how
depressed would you say you have
been on a scale from 0 to 5, where 0
means "not at all" and 5 means "the
most possible?
I feel great! I love my new job.
Ive lost 20 pounds and feel
healthier than I have in years.
2
-
7/27/2019 2013 Swc Beyond Toc 2
25/41
25
Whats Your Outcomeand Measurement
Question?ACTIVITY
-
7/27/2019 2013 Swc Beyond Toc 2
26/41
26
Whos Your Best
Source of Information? Data Source = The person, group or organization
that has information on whether the intended
outcome occurred.
Can you access this data source? How?
What are the pros and cons of relying on this data
source? What alternatives are available? What tradeoffs come
with relying on an alternative data source?
Once youve chosen an appropriate data source, make
sure youre asking them the right questions!
-
7/27/2019 2013 Swc Beyond Toc 2
27/41
27
School Attachment
Who Would Know?Students
(preferred data
source)
Teachers
Tutors
Parents
-
7/27/2019 2013 Swc Beyond Toc 2
28/41
28
Who is Your DataSource?
ACTIVITY
-
7/27/2019 2013 Swc Beyond Toc 2
29/41
29
What Method Will You
Use?
Data Collection Method: the process used to
systematically collect data
Match method with:
Measurement question for desired outcome
Data source
Resources (time, dollars, etc.)
-
7/27/2019 2013 Swc Beyond Toc 2
30/41
30Surveys
Types of Questions:
Open ended
Closed ended
Scales
RankingMultiple Choice
-
7/27/2019 2013 Swc Beyond Toc 2
31/41
31
Interviews & Focus
Groups
-
7/27/2019 2013 Swc Beyond Toc 2
32/41
32Observations
-
7/27/2019 2013 Swc Beyond Toc 2
33/41
33
Other Commonly Used
MethodsLogs and Tally Sheets
Standardized Tests
Program Developed Pre/Post Tests
Rubrics
Secondary Data
Other?
-
7/27/2019 2013 Swc Beyond Toc 2
34/41
34
Choosing or Designing
an Instrument
Which ever method you select, what
instrument will you use? Borrow vs. Develop
Does it ask the right questions to get at yourdesired outcome?
Does it have all the necessary components?
What information will each question yield?
How will you use information, if not related tooutcome?
-
7/27/2019 2013 Swc Beyond Toc 2
35/41
35Instrument MappingLook at each question on your data collection tool and
ask:
Does this help us measure the desired outcome? Is there one question? More than one?
What kind of data will we get? Subjective? Objective?
Quantitative? Qualitative? If it doesnt measure the outcome, do we really need to ask
it? How will we use the answer? Nice to know or internal
use?
How will we analyze this?
-
7/27/2019 2013 Swc Beyond Toc 2
36/41
36
Instrument Mapping
ACTIVITY
-
7/27/2019 2013 Swc Beyond Toc 2
37/41
37Data Collection StepsAfter identifying a data source, method and
instrument(s):
1.Identify who you need to work with to collect data
2.Set a schedule for collecting data
3.Train your data collectors
4.Implement data collection
Good practice is to pilot test your tool.
FOR BEST RESULTS make key decis ion s about how to
imp lement data col lect ion BEFORE your p rogram b egins!
-
7/27/2019 2013 Swc Beyond Toc 2
38/41
38Program PM TargetsTarget describes amount of change you anticipate
achieving. Based on indicator specific, measurable
amount of change that will signify success
Apply target to your specific instrument(s)
If instruments vary by site - which questions get at
the desired outcome?
Need to apply program target to data collection toolsused by each site
Need to report aggregate outcome for program - not
each site separately
-
7/27/2019 2013 Swc Beyond Toc 2
39/41
39
Data Aggregation andReporting
ACTIVITY
-
7/27/2019 2013 Swc Beyond Toc 2
40/41
40Summary of Key Points Use your theory of change as a guide to outcome
measurement.
The type of outcome you want to measure (attitude,knowledge, behavior, condition) influences your
choice of data source, method, and instrument.
Be very clear about the measurement question to
ask Choose methods based on ability to measure your
desired outcome and ease of use.
Map your instrument
Remember: Continuous Improvement NOT
-
7/27/2019 2013 Swc Beyond Toc 2
41/41
41Ways to Connect
Susan [email protected]
303-512-0808
@Susan_Hyatt
http://www.facebook.com/CoreThoughtinc
www.core-thought.com
mailto:[email protected]://www.core-thought.com/http://www.core-thought.com/http://www.core-thought.com/http://www.core-thought.com/mailto:[email protected]:[email protected]:[email protected]