procedures for the comparison of policy options: scryer the ex-ante evaluation of policies: the case...
TRANSCRIPT
![Page 1: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/1.jpg)
Procedures for the comparison of policy options: Scryer
The ex-ante evaluation of policies: The case of food safety regulations
Corso per dottorandi Economia e Statistica Agro-alimentare
Maddalena RagonaDipartimento di Scienze Statistiche, Università di Bologna
Bologna, febbraio-marzo 2012
![Page 2: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/2.jpg)
Contents
• Fuzzy sets
• Scryer
![Page 3: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/3.jpg)
Fuzzy logic
Classifying statements in «true» and «false» may be too restrictive…
Any statement may have a certain degree of truth• E.g. is your coffee bitter or sweet?
• 0.8 sweet
• 0.2 bitter
When linguistic variables are exploited, there are specific functions to manage different degrees of truth
![Page 4: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/4.jpg)
Fuzzy logic - Coffee example
Truth
Quantity of sugar
1
0
Bitter Sweet Very sweet
Note that the fuzzy membership functions may have very different shapes, which also depend on how large they are (how uncertain is the judgement)
![Page 5: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/5.jpg)
Fuzzy vs. Probabilistic Logic
The distinction is philosophical• Fuzzyness as «degree of belonging» to different
sets• (Subjective) probability: how much it is probable
that the element belongs to that set (it belongs to one set only, but there are different degrees of perception)
• Probability as a sub-set of fuzzy logic?• Fuzzy probability?
![Page 6: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/6.jpg)
Scryer (MoniQA socio-economic evaluation tool)
Fuzzy multi-criteria tool to support decision-making Steps
1. Qualitative assessment of each impact for each policy option (coding/scoring procedure based on expert(s) judgement)
2. Feasibility filter (data availability, time, costs) to evaluate the possibility of quantitative assessment
3. Quantitative assessment of some impacts accounting for statistical error
4. Fuzzy multi-criteria comparison of options
Computer-based• Currently Excel spreadsheet, to be implemented into web-application
![Page 7: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/7.jpg)
Characteristics Analysis of impacts based on the directions of the EC Impact
Assessment Guidelines (2009) Ranking of policy options based on NAIADE software, developed at JRC-
EC for environmental impact assessment It allows for both synthesis of quantitative (model-based) and qualitative
assessments without the need for monetisation It may take into account public sensitiveness It accounts for uncertainty in outcomes evaluations (including lack of data
/ external uncertainty like weather / expert internal uncertainty) Weighting of impacts is allowed for Sensitivity analysis of the policy ranking
Advantages of a fuzzy multicriteria approach (Scryer)
![Page 8: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/8.jpg)
qualitative assessment – data entering
![Page 9: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/9.jpg)
qualitative assessment – data entering
![Page 10: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/10.jpg)
qualitative indicator X on a 1-9 scale
uncertainty indicator U on a 1-5 scale
qualitative assessment – coding procedure
1 5 9
No impactStrong negative impact
Strong positive impact
1 5
Very good informationVery low or no uncertainty
No informationHigh uncertainty
![Page 11: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/11.jpg)
case study – qualitative assessment
![Page 12: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/12.jpg)
case study – feasibility filter
Is quantitative evaluation needed / feasible?
![Page 13: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/13.jpg)
Step 2 - Feasibility filter
![Page 14: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/14.jpg)
Step 3: quantitative assessment
For each policy option, insert:• Estimated impact• Standard error of estimate
![Page 15: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/15.jpg)
user weight
![Page 16: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/16.jpg)
The scoring system tends to privilege impacts with high probability of occurrence and high level of information certainty.
One prominent impact on the benefit side & several important impacts on the cost side
case study – final ranking
![Page 17: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/17.jpg)
Fuzziness in Scryer
What’s the impact of a specific regulation on public healt?• Negative and weak?• Neutral?- Positive and weak?- Strong and positive?
- The qualitative evaluation may belong to a single statement or to several ones with different degree of membership
- It depends on uncertainty- Qualitative fuzzy evaluations may be aggregated with
quantitative statistical (probabilistic, model-based) evaluations
![Page 18: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/18.jpg)
The starting impact matrix for fuzzy multi-criteria calculations
dimension 14*(2n) X 14n matrix
whose elements xij : ordinal values (between 1 and 9) which measure the impact of policy j for the impact i
U a 14n matrix whose elements uij : corresponding uncertainty assessments (values between 1 and 5)
SUMMARY TABLEType of
assessmentImpact
relevance User weightX(Y) U(E) X(Y) U(E) X(Y) U(E)
PUBLIC HEALTH Quantitative 10 0.133 45.0 4.0 56.0 5.0 44.0 7.0FIRM COMPETITION Quantitative 3 0.040 56.0 67.0 24.0 6.0 499.0 6.0CONDUCT OF BUSINESSES/SMEs Qualitative 5 0.067 4.5 2.3 2.8 3.0 2.5 3.0ADMINISTRATIVE BURDENS ON BUSINESSES Qualitative 5 0.067 5.0 5.0 1.0 4.0 1.0 4.0PUBLIC AUTHORITIES Qualitative 7 0.093 5.0 3.5 1.0 3.5 1.0 3.5INNOVATION AND RESEARCH Qualitative 5 0.067 5.0 3.3 7.0 3.3 7.7 3.3CONSUMERS Qualitative 6 0.080 5.0 3.0 5.0 3.0 5.0 3.0INTERNATIONAL TRADE Qualitative 3 0.040 5.0 3.2 5.5 2.7 5.5 2.7MACROECONOMIC ENVIRONMENT Qualitative 6 0.080 5.0 3.0 5.0 3.0 5.0 3.0LABOUR MARKETS Qualitative 7 0.093 5.0 3.0 5.0 3.0 5.0 3.0ENVIRONMENT Qualitative 6 0.080 5.0 3.0 4.0 2.0 3.0 3.0DISTRIBUTIVE EFFECTS negative Qualitative 4 0.053 5.0 4.0 4.0 4.0 4.0 4.0DISTRIBUTIVE EFFECTS positive Qualitative 5 0.067 5.0 4.0 6.0 4.0 6.0 4.0SOCIAL SENSITIVITY Qualitative 3 0.040 5.0 4.0 5.0 4.0 5.0 4.0
Policy option 1 Policy option 2 Policy option 3
policy j (j: 1,…,n) impact category i (i: 1,…,14)
![Page 19: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/19.jpg)
Steps
1) Transform qualitative variables into Gaussian fuzzy sets
2) Compute distances between pairs of policy options for each specific impact category (distance between two fuzzy sets or stochastic variables)
3) Produce a pairwise comparison between policy options based on the above distances and the weights assigned to impact categories
4) Rank the policies based on their performance in pairwise comparisons
![Page 20: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/20.jpg)
Gaussian fuzzy sets
2
2
( )( ) exp
2k
ijijS
k
k xx q
If element xij is a qualitative score, it needs to be transformed into a fuzzy set
Gaussian fuzzy sets Fuzzy sets defined through a membership function for each of its elements A degree of membership is needed for each of the 9 values of X Fuzzy set Sk where k: 1,…,9 are the potential values that xij may assume q actual assessment of xij, where q is a single value between 1 and 9 The membership function is defined as follows:
K centre of the fuzzy set Sk
k width of the fuzzy set Sk (i.e. a measure of dispersion around the centre) The Gaussian membership functions return a value between 0 and 1, where
when q=k
![Page 21: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/21.jpg)
The «variance» (uncertainty)
function of the centre k of each fuzzy set and of the stated uncertainty level u
assumption dispersion is larger for assessments around 5 and for smaller values of uij
standard deviation for a continuous uniform distribution ranging from 1 to 9 is 2.58 we adopt this value as the maximum variability level
with k=1,…,9
![Page 22: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/22.jpg)
Example
xij=3 score for a given impact uij=4 level of uncertainty The membership function is computed for all sets Sk with k ranging
from 1 to 9, considering the relative dispersion value k. Consider the first fuzzy set S1, for which k=1
1
2
2
(1 3)( 3) exp 0.04
2 0.774ijSx
![Page 23: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/23.jpg)
Distance between two fuzzy sets
o xi1 qualitative impact of the first policy for the i-th category of impact
o xi2 impact of the second policy for same category
o The comparison depends on two fuzzy sets S(xi1=q) and S(xi2=h)
1)Rescale the membership functions through a constant c so that their integral equals to 1, for example, for S(xi1=q)
2) Compute the distance
9 9
1 1 11 1
( ) 1 ( ) 1 ( )k k ki i iS S Skk k
c x q c x q c x q
9 9
1 1 2 2 1 1 2 21 1
( ) ( ) ( ) ( )MAX MAX
l m l mMIN MIN
l m
i i i iS S S Sl ml m
D l m c x q c x h dldm l m c x q c x h
weighted average of all potential distances between the linguistic values, weighted by their
membership functions
![Page 24: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/24.jpg)
Distance - quantitative
when the impact is quantitative distance between two impacts assuming a normal
distribution and exploiting the Hellinger distance
s1 and s2 standard errors of the estimated impacts xi1 and xi2
![Page 25: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/25.jpg)
Pairwise comparison (by impact)
Credibility values are computed for a set of preference relations between 2 options for each impact category2 policy options P1 and P2 6 statements:
• P1 is much better than P2 (according to criterion i)• P1 is better than P2
• P1 is more or less like P2
• P1 is identical to P2
• P1 is worse than P2
• P1 is much worse than P2
range between 0 (not credible at all) and 1 (maximum confidence)
![Page 26: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/26.jpg)
Computation of credibility values (1)
elements needed
(a) semantic distances (also considering the “sign” of the relationship);
(b) cross-over values • parameter which
indicates the distance for which credibility is set at 0.5 (i.e. the confidence that the statement is credible equals the confidence that it is not credible)
• must be fixed (or left to the user)
2, 1 2
2
0 if
1 if
( , ) 2 11
i
x y
x yc P P
D
, 1 2 2
2
0 if
1 if ( , )
1i
x y
x yc P P
D
ln 2
, 1 2( , )D
ic P P e
![Page 27: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/27.jpg)
Computation of credibility values (2)
22
ln 2
, 1 2( , )D
ic P P e
, 1 2 2
2
0 if
1 if ( , )
1i
x y
x yc P P
D
2, 1 2
2
0 if
1 if
( , ) 2 11
i
x y
x yc P P
D
![Page 28: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/28.jpg)
Pairwise comparison (across impacts – aggregation)
wi [0,1] (with i:1,…,c) weights assigned to each criterion
Aggregate preference intensity index for each of the 6 preference statements
![Page 29: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/29.jpg)
entropy
29
*, 1 2
*, 1 2*, 1 2 *, 1 2
0 if ( , ) ( , )
( , ) if ( , )i
ii i
c P PP P
c P P c P P
* 1 2 *, 1 2 *, 1 2 *, 1 2 *, 1 21
1( , ) [ ( , ) ( , ) (1 ( , )) ( , )]
cA B
i i i ii
H P P P P P P P P P Pc
*,
*, 1 22 *,
0 if 0( , )
log otherwise iA
ii
P P
*,
*, 1 22 *,
0 if 1( , )
log (1 ) otherwise iB
ii
P P
preference intensity indices may hide very heterogenous situations, in terms of consistency across the credibility indices for the various criteria entropy measure, to ‘weigh’ the preference intensity indices in the final policy ranking step
Adjusted membership function for each policy comparison, considering a threshold to rule out very small preference intensities:
increases as the basic credibility values concentrate around 0.5 (i.e. uncertainty)
tends to 0 when most of the basic credibility values are 0 or 1 (i.e. certainty)
extremes : • H=0 when all basic credibility values are 0 or 1,
•H=1 when all basic credibility values are 0.5
![Page 30: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/30.jpg)
Ranking of policy optionsThe final step consists in producing two overall indices to rank the various policy alternatives. These two indices + and - aggregate – respectively – the pairwise indices B and W for a given policy option I relative to all policy alternatives.
( , ) [1 ( , )] ( , ) [1 ( , )]
( )2 ( , ) ( , )
p
i j i j i j i jj i
i p
i j i jj i
P P H P P P P H P P
PH P P H P P
( , ) [1 ( , )] ( , ) [1 ( , )]
( )2 ( , ) ( , )
p
i j i j i j i jj i
i p
i j i jj i
P P H P P P P H P P
PH P P H P P
All policy alternatives can now be ranked according to + and -, which are included between 0 and 1. They can be interpreted as a degree of membership to the statements that “Policy alternative i is the best policy option” and “Policy alternative j is the worst policy option”.
entropy can be considered distance between two impacts assuming a normal
distribution and exploiting the Hellinger distance
without entropy omit terms in square brackets and uss 2(p-1) as denominator.
For each policy option, the equations aggregate the much better (much worse) and better (worse) preference intensity indexes, to generate an aggregate preference index for the best/worst policy option.
degree of membership to the statements that ‘Policy alternative i is the best policy option’ and ‘Policy alternative j is the worst policy option’
range between 0 and 1
![Page 31: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/31.jpg)
Multi-Criteria Analysis vs. Cost-Benefit Analysis
MCA CBA
more comprehensive approach less comprehensive approach (only monetary values)
based on experts’ preferences (subjectivity)
measures individual preferences (objectivity), even though biased by income
objectives and criteria are more clearly stated
objectives and criteria are often implicitly assumed
has not a rigorous approach to include time discounting
has a rigorous approach to include time discounting (but difficult to choose appropriate discount factor)
distributional impacts are more clearly considered
distributional impacts are less clearly considered
![Page 32: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/32.jpg)
Final considerations
There is no optimal procedure:• Scale of measurement of impacts• Decision aim
![Page 33: Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia](https://reader035.vdocument.in/reader035/viewer/2022062409/56649ec65503460f94bd16a8/html5/thumbnails/33.jpg)
References
Figueira, J., Greco, S., and Ehrgott, M., 2005. Multiple criteria decision analysis: state of the art surveys. International Series in Operations Research and Management Science. Springer
Munda, G., Nijkamp, P., and Rietveld, P., 1992. Comparison of fuzzy sets: A new semantic distance. Serie Research Memoranda. Free University, Amsterdam
-----, 1995. Qualitative multicriteria methods for fuzzy evaluation problems: An illustration of economic-ecological evaluation. European Journal of Operational Research 82, 79-97
Ragona, M., Mazzocchi, M., Zanoli, A., Alldrick, A.J., Solfrizzo, M., and van Egmond, H.P. (2011). Testing a toolbox for impact assessment of food safety regulations: Maximum levels for T-2 and HT-2 toxins in the EU. Quality Assurance and Safety of Crops & Foods, 3(1):12-23
Zadeh, L.A., 1965. Fuzzy sets. Information and Control 8, 338-353