![Page 1: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/1.jpg)
Part 5 of 8 AEA Coffee Break Webinars 2013
![Page 2: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/2.jpg)
Outcomes Impacts
![Page 3: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/3.jpg)
As a profession, we often either
oversimplify causation
or we overcomplicate it!
![Page 4: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/4.jpg)
senior
leadership &
management
development
program
![Page 5: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/5.jpg)
too soon?
![Page 6: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/6.jpg)
leadership
development
program
enhanced
knowledge
& skills
improved
performance
as a leader
appointable
candidate for
senior roles
immediate/soon
after learning
weeks to
months later
months to
years later
check timing
of outcomes
v. 1
![Page 7: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/7.jpg)
entry into
leadership
development
program
visibility as
emerging
leader
being asked
to apply for
senior roles
appointable
candidate for
senior roles
On enrolment
(or soon after)
Weeks to
months later
Months later
identification
of needed
experience
gaining
experience in
senior roles
check timing of outcomes
v. 2
![Page 8: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/8.jpg)
leadership
development
program
identification
of needed
experience
gaining
experience in
other roles
appointable
candidate for
senior roles
match content …
… to outcomes
![Page 9: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/9.jpg)
changed job?
career advancement?
did the program help?
ask participants
![Page 10: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/10.jpg)
40%
13%
13%
13%
2%
19%
Changed roles, definite program impact
Changed roles, partial program impact
No change, but enhanced aspirations
Changed roles, but no program impact
Changed roles, negative impact
No change in roles or aspirations
More than half said the program
helped them get a promotion
![Page 11: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/11.jpg)
“firstly, actually
being accepted
for the course
is rated highly,
secondly I understand
that my interview
went well because
of my ability to relate
program learnings to
real life issues.”
![Page 12: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/12.jpg)
3 messages
![Page 13: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/13.jpg)
1. all outcome/impact
evaluation needs
causal inference
3 messages
![Page 14: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/14.jpg)
1. all outcome/impact
evaluation needs
causal inference
2. going qualitative
doesn’t let you off
the causal hook!
3 messages
![Page 15: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/15.jpg)
1. all outcome/impact
evaluation needs
causal inference
2. going qualitative
doesn’t let you off
the causal hook!
3. the real “gold
standard” is sound
causal reasoning!
3 messages
![Page 16: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/16.jpg)
![Page 17: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/17.jpg)
![Page 18: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/18.jpg)
Check the results support causal attribution 1
modus operandi
timing of outcomes
intermediate outcomes
match content to
outcomes
ask key informants
we
use
d
![Page 19: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/19.jpg)
Check the results support causal attribution 1
modus operandi
timing of outcomes
intermediate outcomes
match content to
outcomes
ask key informants
process tracing
comparative case studies
dose-response patterns
fit with a statistical model
fit with expert predictions
statistical control of
extraneous variables
realist analysis
qualitative comparative
analysis
we
use
d
![Page 20: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/20.jpg)
Compare the results to the counterfactual 2
we
use
d
ask key informants
expert-estimated
counterfactual
![Page 21: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/21.jpg)
Compare the results to the counterfactual 2
we
use
d
ask key informants
expert-estimated
counterfactual
statistically created
counterfactual
logically constructed
counterfactual
regression discontinuity
sequential allocation
control group
matched comparisons
judgemental matching
instrumental variables
difference in difference
(or double difference)
qualitative comparative
analysis
![Page 22: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/22.jpg)
Investigate possible alternative explanations 3
general elimination
methodology &
modus operandi
searching for
disconfirming
evidence
asking key
informants
we
use
d
![Page 23: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/23.jpg)
Investigate possible alternative explanations 3
general elimination
methodology &
modus operandi
searching for
disconfirming
evidence
asking key
informants
we
use
d
force field analysis
process tracing
RAPID outcomes
assessment
ruling out technical
explanations
statistically
controlling for
extraneous variables
![Page 24: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/24.jpg)
![Page 25: Part 5 of 8 AEA Coffee Break Webinars 2013...Check the results support causal attribution 1 modus operandi timing of outcomes intermediate outcomes match content to outcomes ask key](https://reader033.vdocument.in/reader033/viewer/2022041721/5e4ec699b98d517b627a62bc/html5/thumbnails/25.jpg)
Jane Davidson
http://RealEvaluation.com
Further resources
http://betterevaluation.org/plan/understandcauses