techniques

TECHNIQUES/TECHNIQUES/TOOLS IN POLICY TOOLS IN POLICY

STUDYSTUDYCarlos C. Rabago, Jr.Carlos C. Rabago, Jr.

What is….?What is….?

ToolTool TechniqueTechnique DataData Qualitative DataQualitative Data Quantitative DataQuantitative Data Data AnalysisData Analysis Qualitative AnalysisQualitative Analysis Quantitative AnalysisQuantitative Analysis

Qualitative vs. Qualitative vs. Quantitative Quantitative

“ “ All research All research ultimately has a ultimately has a qualitative qualitative groundings” D. groundings” D. Campbell Campbell

““There’s no such There’s no such thing as qualitative thing as qualitative data. Everything is data. Everything is either 1 or 0” F. either 1 or 0” F. Kerlinger Kerlinger

The aim is complete, The aim is complete, detailed description detailed description

The aim is to classify The aim is to classify features, count them, features, count them, and construct and construct statistical models in statistical models in an attempt to explain an attempt to explain what is observed what is observed

Data is in the form of Data is in the form of words, pictures or words, pictures or objects objects

Data is in the form of Data is in the form of number and number and statistics. statistics.

Techniques/ToolsTechniques/Tools

Bar ChartBar Chart

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

Benefit-effectiveness Benefit-effectiveness AnalysisAnalysis

This is used to break something This is used to break something down into its components parts, down into its components parts, show trends and make comparisons show trends and make comparisons between the items, represented by between the items, represented by the barsthe bars

BrainstormingBrainstorming

Rules: Rules: Withhold Judgment of IdeasWithhold Judgment of Ideas Encourage Whacky IdeasEncourage Whacky Ideas Quantity CountsQuantity Counts Build on the Ideas of OthersBuild on the Ideas of Others Every Person and Every Idea has Equal Every Person and Every Idea has Equal

Worth.Worth.

Generic Brainstorming Generic Brainstorming ProcessProcess

Provide Background Info Provide Background Info Problem Definition Problem Definition Idea Generation Idea Generation Idea Selection Idea Selection Advantages and Disadvantages Advantages and Disadvantages Critical Concerns Critical Concerns Action Plan & Implementation Action Plan & Implementation

Case StudiesCase Studies

Descriptive evaluation of the life of Descriptive evaluation of the life of community, nature of organization or community, nature of organization or events unfolding. The principal events unfolding. The principal investigator gathers available investigator gathers available documents, holds formal and documents, holds formal and informal conversation with informal conversation with informants. Observes ongoing informants. Observes ongoing activities, and develops an analysis of activities, and develops an analysis of data and interpretation of findings.data and interpretation of findings.

Cause and effect diagram Cause and effect diagram (fishbone diagram)(fishbone diagram)

Steps:Steps:

1. Develop statement of the problem supported 1. Develop statement of the problem supported with data.with data.

2. Construct the fishbone diagram. Place the 2. Construct the fishbone diagram. Place the effect in a box on the right. Draw horizontal effect in a box on the right. Draw horizontal arrow leading into the effect arrow leading into the effect

3. Identify three to six major bones, alternate the 3. Identify three to six major bones, alternate the arrows above and below the center arrow.arrows above and below the center arrow.

4. Draw the major bones as slanting arrows 4. Draw the major bones as slanting arrows leading into the center arrow. To convey a leading into the center arrow. To convey a sense of order sense of order

Problem

5. Identify first-level causes related to 5. Identify first-level causes related to each major boneeach major bone

Machine

Manpower

Problem

Methods

Materials

Problem

Machine Materials

Manpower Methods

6. Identify second-level causes for each 6. Identify second-level causes for each first-level cause and so on.first-level cause and so on.

7. Identify potential root causes. Select 7. Identify potential root causes. Select causes occur more than once, or appear causes occur more than once, or appear to have significant impact on the effect.to have significant impact on the effect.

8. Verify each potential root cause. 8. Verify each potential root cause. Analyze the data available, or gather Analyze the data available, or gather additional data to make certain additional data to make certain assumptions are correct.assumptions are correct.

Check SheetCheck SheetDaily Loan ApplicationDaily Loan Application

FrequencyFrequency

FactorFactor

04/1304/13 04/1404/14 04/1504/15 04/1604/16 04/1704/17

Incomplete Incomplete Borrower Borrower datadata

Incomplete Incomplete co-maker’s co-maker’s datadata

No StatusNo Status

DelinquentDelinquent

TotalTotal 1212 44 88 44 99

Common Sense Common Sense

Instinctive reaction characterized Instinctive reaction characterized with negative logic and mind with negative logic and mind openness.openness.

Cost-benefit analysisCost-benefit analysis

The cost and benefits of policy The cost and benefits of policy options are quantified, usually by options are quantified, usually by assigning amount. Assists in the assigning amount. Assists in the evaluation and comparison of evaluation and comparison of different policy options different policy options

Cost-effectiveness Cost-effectiveness analysisanalysis

Similar to cost-benefit It is used Similar to cost-benefit It is used when the outcome is predetermined when the outcome is predetermined and different ways of reaching that and different ways of reaching that outcome need to be costed for the outcome need to be costed for the purpose of evaluation.purpose of evaluation.

Cross-cutting methodsCross-cutting methods Use at nearly stage of policy analysis.Use at nearly stage of policy analysis.

Identifying and gathering dataIdentifying and gathering dataLibrary search methodsLibrary search methodsInterview for policy dataInterview for policy dataQuick surveysQuick surveysAssessing information qualityAssessing information qualityBasic data analysis, andBasic data analysis, andCommunicating the analysisCommunicating the analysis

Delphi methodsDelphi methods

A very slow and an interactive A very slow and an interactive process for consensus-building process for consensus-building among the panel of experts who are among the panel of experts who are anonymous to one another, using anonymous to one another, using series/rounds of extensive series/rounds of extensive questionnaire to the panel, experts questionnaire to the panel, experts do not communicate each other but do not communicate each other but only provide responses to the Delphi only provide responses to the Delphi administrator administrator

Environmental ScanningEnvironmental Scanning

Systematic gathering of information Systematic gathering of information from outside organization or locus of from outside organization or locus of the problem. This lessens the problem. This lessens randomness of information flowing randomness of information flowing into the organization and provide into the organization and provide warnings for managers of changing warnings for managers of changing external conditions.external conditions.


The Role of External Analysis in The Role of External Analysis in Policy MakingPolicy MakingExternal Analysis

ScanningMonitoringForecastingAssessing

Internal AnalysisVision

MissionStrength

Weaknesses

Strategic Directions Strategic Plan

Flowchart Flowchart

A graphical illustration of the A graphical illustration of the activities in a process (using activities in a process (using symbols). Used in analyzing, symbols). Used in analyzing, documenting or managing a process documenting or managing a process or a program or a program

Flowchart SymbolsFlowchart Symbols

NameName SymbolSymbol UseUse

Oval Oval (Terminator)(Terminator)

Start or EndStart or End

Flow lineFlow line Direction Direction FlowFlow

ParallelogramParallelogram Input/OutputInput/Output

RectangleRectangle Process/StepProcess/Step

DiamondDiamond Decision Decision (Yes/No)(Yes/No)

DocumentDocument DocumentDocument

CircleCircle ConnectorConnector

Focus GroupFocus Group Combine elements of both interview and Combine elements of both interview and

observation. Eight to twelve people gathered observation. Eight to twelve people gathered from similar backgrounds or experiences to from similar backgrounds or experiences to discuss an issue or a specific topic of interest. discuss an issue or a specific topic of interest.

It is used in answering the same type of It is used in answering the same type of questions, exploring the meaning or survey questions, exploring the meaning or survey findings that cannot be explained statistically, findings that cannot be explained statistically, the range of views/opinions on a topic or the range of views/opinions on a topic or interest, providing insights into different interest, providing insights into different opinions among different parties involved in opinions among different parties involved in the change process, and a good method use the change process, and a good method use before designing questionnaire.before designing questionnaire.

Forced Field Analysis Forced Field Analysis

Used to inform Used to inform decision-making/policy making and decision-making/policy making and planning and implementation change planning and implementation change management programs in management programs in organization, method in gaining organization, method in gaining overviews of different forces acting overviews of different forces acting on a potential policy issue and for on a potential policy issue and for assessing their source of strengths assessing their source of strengths and weaknesses.and weaknesses.


StepsSteps

1.1. Participant agree on the area of Participant agree on the area of change. Desired Policy goal or change. Desired Policy goal or objective.objective.

2.2. All the forces in support of the change All the forces in support of the change (facilitating forces) are listed in one (facilitating forces) are listed in one side (driving the change forward) while side (driving the change forward) while all forces working against the change all forces working against the change (restraining forces) are listed on the (restraining forces) are listed on the other side (holding it back)other side (holding it back)


Steps Cont.Steps Cont.

3. The driving and restraining forces 3. The driving and restraining forces are sorted around common themes are sorted around common themes and scored acording to their and scored acording to their ‘magnitude’ ranging from 1 (weak) ‘magnitude’ ranging from 1 (weak) to 5 (strong). The score may not to 5 (strong). The score may not balance on either side.balance on either side.


----------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------------------------------

______________________________________________________________

______________________________________________________________

PLAn

Restraining Forces

Facilitating Forces

Existing Condition

Desired Condition

Grading methodGrading method

The consequences of each The consequences of each alternative on each criterion are alternative on each criterion are considered. considered.

A grade of “pass” or “fail” is A grade of “pass” or “fail” is assigned to each alternative on each assigned to each alternative on each criterion. criterion.

Only those alternatives which “pass” Only those alternatives which “pass” on al criteria are retained, those on al criteria are retained, those which have any ”fails” are rejected.which have any ”fails” are rejected.

Identifying and Identifying and Gathering DataGathering Data

Identifying and Gathering DataIdentifying and Gathering Data Sources of Data (Documents People, Sources of Data (Documents People,

Observations and other sources of info), Observations and other sources of info), information from multiple sources should be information from multiple sources should be sought on controversial data. Problems with sought on controversial data. Problems with sources of data. Looking for documents that sources of data. Looking for documents that may helpful in doing policy analysis is may helpful in doing policy analysis is important. Three questions must be asked: do important. Three questions must be asked: do such documents exist, can they be obtained such documents exist, can they be obtained in a reasonable time, when is additional in a reasonable time, when is additional searching no longer worthwhile?searching no longer worthwhile?

InterviewInterview Collect data trough an exchange of Collect data trough an exchange of

information utilizing face to face or telephone information utilizing face to face or telephone communication. It gathers fact about the communication. It gathers fact about the problem or issue, assess political attitudes problem or issue, assess political attitudes and resources of major players, get ideas on and resources of major players, get ideas on future trends and generate contacts and future trends and generate contacts and materials.materials.

There are two types of interview: There are two types of interview: - Structured interview-written - Structured interview-written

questionnaire.questionnaire.- In depth interview-captures - In depth interview-captures

respondents’ perception (free and open)respondents’ perception (free and open)

Library Search methods.Library Search methods.

Excellent sources of policy-related Excellent sources of policy-related informationinformation

StrategiesStrategies• Look upLook up• Develop list of searchDevelop list of search• Identify key journals(1-2 years)Identify key journals(1-2 years)• Check Guides Current Lit.Check Guides Current Lit.• Annual ReviewsAnnual Reviews

Line GraphLine Graph

It shows progress towards or It shows progress towards or deviation from, an established deviation from, an established target. It can measure practically target. It can measure practically anything.anything.

List ReductionList Reduction

Reduces a large list of items to a Reduces a large list of items to a manageable few. Used to lessen the manageable few. Used to lessen the number of items identified in an number of items identified in an brainstorming session. The items brainstorming session. The items identified in a brainstorming session. The identified in a brainstorming session. The items remaining can be a discussed and items remaining can be a discussed and evaluated using MATRIX. evaluated using MATRIX.

Steps:Steps:

Present – Vote – Mark – Identify - Present – Vote – Mark – Identify - LimitingLimiting

Review of LiteratureReview of Literature

A body of text aims to review the critical A body of text aims to review the critical points of current knowledge on a points of current knowledge on a particular time. particular time.

To bring the reader current literature on To bring the reader current literature on a topic.a topic.

Good literature has a logical ideas; Good literature has a logical ideas; current, relevant, consistent and current, relevant, consistent and appropriate; proper use of terms; and appropriate; proper use of terms; and unbiased and comprehensive view of unbiased and comprehensive view of previous research on topic.previous research on topic.

MatrixMatrix A grid intersecting horizontal and vertical lines. A grid intersecting horizontal and vertical lines. Matrix is used to evaluate the relationship that Matrix is used to evaluate the relationship that

exists between a set of policy options and a set exists between a set of policy options and a set of criteria; and for selecting one option based of criteria; and for selecting one option based on brainstormed list that has already been on brainstormed list that has already been reduced by list reduction technique.reduced by list reduction technique.

Steps:Steps:1.1. Know the choices policy options to be Know the choices policy options to be evaluated;evaluated;2.2. Identify criteria for evaluationIdentify criteria for evaluation3.3. Construct a two-sided matrix and post Construct a two-sided matrix and post evaluation evaluation criteria across the top and the criteria across the top and the options under options under consideration along the side.consideration along the side.

Problem Solving MatrixProblem Solving Matrix

Problem Problem Under Under ConsiderationConsideration

Evaluation CriteriaEvaluation Criteria

Impact on Impact on Lending Lending ActivitiesActivities

Impact on Impact on RevenuesRevenues

Impact on Impact on Overtime Overtime PayPay

Impact on Impact on Work of Work of StaffStaff

Delinquent Delinquent AccountsAccounts

Investment Investment not booked upnot booked up

Ledger cards Ledger cards not updatednot updated

Legend:Legend: 3- High 2- Medium 1- Low 3- High 2- Medium 1- Low

ObservationObservation

A method that gathers first hand data. A method that gathers first hand data. Observer has the opportunity to gather Observer has the opportunity to gather wider range of data. Things are not wider range of data. Things are not discussed in a interview or focus group discussed in a interview or focus group can be learned.can be learned.

Structured format in ObservationStructured format in Observation

Note: Consent must be obtained from Note: Consent must be obtained from interviewees before gathering any interviewees before gathering any observational dataobservational data

Pareto DiagramPareto Diagram

Study of related subjects to Study of related subjects to determine if one is more significant determine if one is more significant than the others, to identify the most than the others, to identify the most important among and between the important among and between the group of items and to focus problem-group of items and to focus problem-solving activities so that the area solving activities so that the area causing the most difficult is causing the most difficult is addressed first. addressed first.

Pareto DiagramPareto Diagram

Pie ChartPie Chart

A circle graph A circle graph divided into pieces. divided into pieces. Each pie/part Each pie/part represents the represents the subject of the study. subject of the study. Used to show how Used to show how individual parts individual parts relate to the whole relate to the whole and to each other. and to each other. Give percentages of Give percentages of distribution or break-distribution or break-up.up.

Application Loan

Multipurpose Loan

Short Term loan

Regular Loan

Problem Tree analysis/ Problem Tree analysis/ Situational Situational

Analysis/Problem AnalysisAnalysis/Problem Analysis Help to find solutions by mapping out Help to find solutions by mapping out

the anatomy of cause-and-effect the anatomy of cause-and-effect around an issue. Best carried out in a around an issue. Best carried out in a small focus group of about six to eight small focus group of about six to eight people. The heart of the exercise is the people. The heart of the exercise is the discussion, debate and dialogues that discussion, debate and dialogues that is generated as factors are arranged is generated as factors are arranged and re-arranged, often forming sub-and re-arranged, often forming sub-dividing roots and branchesdividing roots and branches


Analysis/Problem AnalysisAnalysis/Problem Analysis


Analysis/Problem AnalysisAnalysis/Problem AnalysisSteps:Steps:

1.1. Discuss and agree the problems or Discuss and agree the problems or issue to be analyzed.issue to be analyzed.2.2. Write the ‘focal problem’ or issue in Write the ‘focal problem’ or issue in the centre of the flip chart (trunk of the the centre of the flip chart (trunk of the tree)tree)3.3. Identify the causes of the focal Identify the causes of the focal problem (roots)problem (roots)4.4. Identify the consequences (branches). Identify the consequences (branches). Arrange in a cause-and-effect logic.Arrange in a cause-and-effect logic.

Quick decision analysisQuick decision analysis

A variation on the technique of A variation on the technique of making decision trees. Decision tree making decision trees. Decision tree are ways of diagramming a problem, are ways of diagramming a problem, when the problem has more than when the problem has more than one solution and several decision one solution and several decision options. options.

A tool to help policy analysisA tool to help policy analysis

Scenario writingScenario writing

Mirroring the future impact of a Mirroring the future impact of a policy, particularly possible failure policy, particularly possible failure or unwanted results. Stimulate or unwanted results. Stimulate thinking about alternatives or thinking about alternatives or improvements. Help to focus on the improvements. Help to focus on the perspective of the affected by the perspective of the affected by the policy and responsible for the policy and responsible for the implementation.implementation.

Sensitivity analysisSensitivity analysis

Used to test the extent to which an Used to test the extent to which an assumption relied upon in the development assumption relied upon in the development of a policy will affect the outcomes of a policy will affect the outcomes achieved by the policy. Relevant where it is achieved by the policy. Relevant where it is recognized that a policy option depends on recognized that a policy option depends on assumption about uncertain factors. assumption about uncertain factors. Mathematical techniques can be used Mathematical techniques can be used which to determine of several uncertain which to determine of several uncertain assumptions is the most important and how assumptions is the most important and how multiple uncertainties interact.multiple uncertainties interact.

SurveySurvey A mean of gathering information using A mean of gathering information using

questionnaires. Used to understand and assess questionnaires. Used to understand and assess client needs, attitudes and satisfaction with public client needs, attitudes and satisfaction with public goods and services; and best applied with large goods and services; and best applied with large number of people to be questioned or hard number of people to be questioned or hard speaking with personally. It has two types, closed- speaking with personally. It has two types, closed- the respondents is asked to choose from available the respondents is asked to choose from available responses prepared before hand; and Open- the responses prepared before hand; and Open- the respondents are free to answer as they wish.respondents are free to answer as they wish.

Can be conducted by mail, in person, or by Can be conducted by mail, in person, or by telephone. Quick and relatively inexpensive to telephone. Quick and relatively inexpensive to administer. Results can be subjected to statistical administer. Results can be subjected to statistical analysis.analysis.

SWOT (Strength, SWOT (Strength, Weaknesses, Opportunities Weaknesses, Opportunities

and Threats) analysisand Threats) analysis It asses the internal capacity of the organization on a It asses the internal capacity of the organization on a

given issue and its external environment. This given issue and its external environment. This provides a simple method to asses how a strategy can provides a simple method to asses how a strategy can e best implemented. Internal capacity assessment e best implemented. Internal capacity assessment identifies where the project or organization is as of identifies where the project or organization is as of now: the resources that can be used immediately and now: the resources that can be used immediately and current problems existing and wont go away. current problems existing and wont go away.

It can identify new resources, skills or allies to be It can identify new resources, skills or allies to be needed. External environment assessment focuses on needed. External environment assessment focuses on what is going on outside the organization, or areas what is going on outside the organization, or areas which are not yet affecting the strategy but can either which are not yet affecting the strategy but can either positively or negatively do. This tool can be returned positively or negatively do. This tool can be returned to any point in may different stages of a project. to any point in may different stages of a project.


and Threats) analysisand Threats) analysis Applied broadly, or a small sub-Applied broadly, or a small sub-

component of the strategy can be component of the strategy can be also be analyzed in detail. SWOT also be analyzed in detail. SWOT analysis is about categorizing this analysis is about categorizing this information into action buckets.information into action buckets.


and Threats) analysisand Threats) analysisInternal CapacityInternal Capacity

StrengthsStrengths WeaknessesWeaknesses

•Skills/abilitiesSkills/abilities•Contacts/PartnersContacts/Partners

•ActivitiesActivities•FundingFunding

•CommitmentsCommitments

External EnvironmentExternal Environment

OpportunitiesOpportunities ThreatsThreats•Other Organization or groupsOther Organization or groups

•Resource (human, finacial, machinery)Resource (human, finacial, machinery)•PoliticalPolitical

•PolicyPolicy•Other forcesOther forces

Summary of Tools for Policy Quality Summary of Tools for Policy Quality ImprovementImprovement

ToolTool HelpHelp UseUse

Bar ChatBar Chat Arrange data for Arrange data for quick and easy quick and easy comparison.comparison.

AnalysisAnalysis

BrainstormingBrainstorming Collect large Collect large number of ideas number of ideas from a group of from a group of peoplepeople

Data gatheringData gathering

Cause-and-effect Cause-and-effect diagramdiagram

Identify a set Identify a set related causes related causes that lead to an that lead to an effect or problem. effect or problem. Good Source of Good Source of analytical analytical information.information.

AnalysisAnalysis

Check sheetCheck sheet Collect data in Collect data in a organized a organized manner.manner.

Data gathering/ Data gathering/ analysisanalysis

Environmental Environmental ScanningScanning

Identify trends Identify trends and events in and events in the the environment.environment.


FlowchartFlowchart Differentiate Differentiate between the between the activities in a activities in a process.process.

AnalysisAnalysis

Focus groupFocus group Obtain variety Obtain variety of viewsof views

Data GatheringData Gathering

HistogramHistogram Determine how Determine how data are data are distributeddistributed

AnalysisAnalysis

InterviewInterview Collect data Collect data from direct from direct conversation conversation


Line graphLine graph Display the Display the output of a output of a process process overtime overtime

AnalysisAnalysis

List ReductionList Reduction Reduce a large Reduce a large list of items to list of items to manageable manageable fewfew

AnalysisAnalysis

MatrixMatrix Make Make comparison comparison between two or between two or more sets more sets informationinformation


ObservationObservation Collects first hand Collects first hand datadata

Data GatheringData Gathering

Pareto ChartPareto Chart Arrange data so Arrange data so that the most that the most significant significant element in a set of element in a set of elements is easily elements is easily identifiable.identifiable.

AnalysisAnalysis

Pie ChartPie Chart Display the Display the volume or volume or quantity of one quantity of one item in relation to item in relation to other.other.

AnalysisAnalysis

Problem treesProblem trees Find solution by Find solution by mapping out mapping out cause-and-effect cause-and-effect of an issueof an issue

AnalysisAnalysis

SurveySurvey Collect data from Collect data from a large number of a large number of peoplepeople


SWOT analysisSWOT analysis Assess internal Assess internal capacity and capacity and external external environmentsenvironments


Steps on Policy Analysis Steps on Policy Analysis and Corresponding and Corresponding Tools/TechniquesTools/Techniques

Policy Analysis StepPolicy Analysis Step Tool/TechniqueTool/Technique

11 Verifying, Defining, and Verifying, Defining, and Detailing VarianceDetailing Variance



InterviewInterview

ObservationObservation

SurveySurvey

List reductionList reduction

MatrixMatrix

Case StudyCase Study

Similar studies and Similar studies and articles donearticles done

22 Establishing Establishing Evaluation CriteriaEvaluation Criteria

Cause-and-effect Cause-and-effect DiagramDiagram

FlowchartFlowchart

Pareto ChartPareto Chart


Check sheetCheck sheet

SurveySurvey

InterviewInterview


Quick Policy AnalysisQuick Policy Analysis

Force FieldForce Field

Fish BoneFish Bone

Delphi MethodDelphi Method

School ReportSchool Report

33 Evaluating Evaluating Alternative PoliciesAlternative Policies


InterviewInterview

SurveySurvey

Comparison MethodComparison Method

Paired Paired ComparisonComparison

ScorecardScorecard

MatrixMatrix

Data AnalysisData Analysis

Action PlanAction Plan

Scenario writingScenario writing

44 Assessing Alternative Assessing Alternative PoliciesPolicies

55 Choosing the Choosing the Alternative to be Alternative to be AdoptedAdopted

77 Implementing, Implementing, Monitoring and Monitoring and EvaluationEvaluation



Pie GraphPie Graph

Bar ChartBar Chart

Histogram Check Histogram Check sheetsheet


77 Implementing, Implementing, Monitoring and Monitoring and EvaluationEvaluation



Pie GraphPie Graph

Bar ChartBar Chart

Histogram Check Histogram Check sheetsheet


Cross Cutting MethodsCross Cutting Methods Identifying and gathering Identifying and gathering datadata

Library search methodsLibrary search methods

Interview for policy dataInterview for policy data

Quick surveysQuick surveys

Assessing information Assessing information qualityquality

Basic data analysis, andBasic data analysis, and

Communicating the Communicating the analysisanalysis

techniques

News & Politics

analysis ofdata

field analysis

stage of policy analysis

costbenefit analysis

role of external analysis

benefiteffectiveness

additional data

data available