module 8 monitoring observed outcomes
TRANSCRIPT
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 1/15
Module 8 Module 8
Monitoring ObservedMonitoring ObservedPolicy OutcomesPolicy Outcomes
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 2/15
Social systems Lorenz curve
accounting Gini Index
Regression discontinuity Purchasing power Social experimentation Random innovation
Social auditing Quasi-experimentation
Research and practice Evaluability assessment
synthesis Internal validity
Threats to validity External validity
Interrupted time-series Control-seriesCurrent Euros Constant Euros
Key Terms and Concepts
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 3/15
POLICYPERFORMANCE
EXPECTEDOUTCOMES
PREFERREDPOLICIES
OBSERVEDOUTCOMES
ForecastingEvaluation
Monitoring Recommendation
problem
structuring
problem
structuring
POLICYPROBLEMS
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 4/15
The Importance of TimeThe Importance of TimeRETROSPECTIVE:What happened? Wasit worthwhile?
PROSPECTIVE:What will happen?Will it be worthwhile?
OBSERVEDOUTCOMES
EXPECTEDOUTCOMES
Monitoring
POLICIES
Forecasting
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 5/15
Why Monitoring Is ImportantWhy Monitoring Is Important
It is not that we hav e so many It is not that we hav e so many
well well- -designed policies. R ather, wedesigned policies. R ather, we
hav e more well hav e more well- -designed policiesdesigned policiesthan we hav e w ays t o moni t or than we hav e w ays t o moni t or
them. Wi thou t moni t oring, wethem. Wi thou t moni t oring, we
c annot know a good policy from a c annot know a good policy from a bad onebad one²²or w hether the policy is a or w hether the policy is a
policy at all.policy at all.
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 6/15
An Unmonitored Policy MayAn Unmonitored Policy May
Conceal A Disabled VehicleConceal A Disabled Vehicle
Failing to monitor Failing to monitor the outcomes of athe outcomes of apolicy is likepolicy is like
counting thecounting theamount of amount of gasoline a car gasoline a car has consumedhas consumedwithout seeingwithout seeinghow far it hashow far it hastraveled.traveled.
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 7/15
Four Functions of MonitoringFour Functions of Monitoring
C ompli anceC ompli ance²²Are laws on local governmentAre laws on local government
consistent with EU requirements?consistent with EU requirements?
Audi t ing Audi t ing ²²Are revenues intended for localAre revenues intended for local
communities reaching them?communities reaching them?
Account ing Account ing ²²Are policies on educationalAre policies on educational
reform producing qualified students?reform producing qualified students?
Ex pl anat ionEx pl anat ion²²Are outcomes of a policyAre outcomes of a policycaused by the policy, or by other factors?caused by the policy, or by other factors?
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 8/15
Approaches to MonitoringApproaches to Monitoring
Social Systems AccountingSocial Systems Accounting
Social AuditingSocial Auditing
Research and Practice SynthesisResearch and Practice Synthesis
Policy ExperimentationPolicy Experimentation
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 9/15
Social Systems AccountingSocial Systems Accounting
HousingHousing²²Area per person (square meters)Area per person (square meters)
Average Life ExpectancyAverage Life Expectancy
Quality Adjusted Life YearsQuality Adjusted Life Years
Income Distribution (Gini Index)Income Distribution (Gini Index)
Air Pollution Index (parts per million)Air Pollution Index (parts per million)
Lead Concentration Index (bloodLead Concentration Index (blood
concentration)concentration) Persons in Mental HospitalsPersons in Mental Hospitals
Persons Below Poverty LinePersons Below Poverty Line
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 10/15
Social Auditing With User SurveysSocial Auditing With User Surveys
PolicyPolicy--Program SpecificationProgram Specification²²What goals,What goals,objectives, and resources constitute theobjectives, and resources constitute thepolicy?policy?
Collection of Available InformationCollection of Available Information²²WhatWhatinformation is available on inputs, processes,information is available on inputs, processes,
outputs and impacts?outputs and impacts? Policy ModelingPolicy Modeling²²What causal mechanismsWhat causal mechanisms
link inputs and processes to outputs andlink inputs and processes to outputs andimpacts?impacts?
EvaluabilityEvaluability AssessmentAssessment²²
Is the policy clear Is the policy clear enough and unambiguous to know what toenough and unambiguous to know what tomonitor?monitor?
Collection of New InformationCollection of New Information²²What newWhat newinformation needs to be collected?information needs to be collected?
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 11/15
Research and Practice SynthesisResearch and Practice Synthesis
Synthesis of research on planned change,Synthesis of research on planned change,
communication of innovations, socialcommunication of innovations, social
marketing strategies (journals and books)marketing strategies (journals and books)
Synthesis of published and unpublishedSynthesis of published and unpublishedpolicy documents (memos, reports,policy documents (memos, reports,
statistics)statistics)
Synthesis of reported cases of change,S
ynthesis of reported cases of change,innovation, and reform (case surveyinnovation, and reform (case survey
analysis)analysis)
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 12/15
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 13/15
Threats to Validity (Rival Hypotheses)Threats to Validity (Rival Hypotheses)
When Conducting Policy ExperimentsWhen Conducting Policy Experiments
Statistical Conclusion ValidityStatistical Conclusion Validity
Internal ValidityInternal Validity
External ValidityExternal Validity
Construct ValidityConstruct Validity
Context ValidityContext Validity
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 14/15
Interrupted TimeInterrupted Time--SeriesSeries
Sequence number
987654321
16
14
12
10
8
6
4
2
0
Probable and
Durable
Probable and
Non-Durable
Improbable
Regression to Mean
Improbable Constant
Change
Improbale Non-Linear
Change
Policy Intervention
8/8/2019 Module 8 Monitoring Observed Outcomes
http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 15/15
Some Major Threats to ValiditySome Major Threats to Validity
HistoryHistory
MaturationMaturation
InstabilityInstability InstrumentationInstrumentation
TestingTesting
MortalityMortality
SelectionSelection
Regression towardRegression toward
the meanthe mean
ViolatedViolated
assumptionsassumptions
of statistical testsof statistical tests