multicriteria model to improve the use of performance evaluation instruments with focus on...
DESCRIPTION
Euro 2010 PresentationTRANSCRIPT
Proposal of use the Multicriteria Decision Aid for Improve the use of Assessment Tools with Focus
on Performance Indicators
Msc. Edilson Giffhorn, PMP, IPMA - Speaker
PhD. Leonardo Ensslin
PhD. Sandra Rolim Ensslin
Msc. William Barbosa Vianna
24th European Conference on Operacional Research11. – 14.7.2010, Lisboa - Portugal
Santa Catarina Federal University - Brazil
Multicriteria Decision Aid Laboratory
2
Production Engineering Department
Objective:
The aim of this paper is to present a process for improving the way to use the instruments of performance assessment on their way to identify, organize, measure and use the Performance Indicators.
3
4
1494
18th century
20th century – until 1950
20th century – after 1950
Tratactus de Computis et Scripturis - Summa de arithmetica, geometrica, proportioni et proportionalita(Brudan, 2009; Ensslin; Ensslin, 2009)
Industrial Revolution:Large corporations - production control(Leão, 1998)
Scientific Management:Evaluation of merit (people)(Guimarães et al. 1998)
Evaluate processes (Brandão, Guimarães, 2001)
The use of indicators as a management / control tool
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
5
Until1960
1960 - 1995
After 1995
1st Generation Indicators
2nd Generation Indicators
3rd Generation Indicators
Performance Indicators
Neely et al. (2003)
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
6
1st Generation Indicators
Neely (1999); Ensslin, Ensslin (2009); Oliveira et al (2009)
Emphasis
Control
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
7
Bititci, Suwignjo, Carrie; (2001); Ensslin, Ensslin; (2009)
2nd Generation Indicators
Emphasis
Management
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
8
Adaptation of Ensslin, Ensslin (2009)
Characteristics of instruments
New requirements of the context
Generic context Specific context
Generic decision makers Specific decision makers
Monitoring Continuous Improvement
Ordinal scales Ordinal and Cardinal scales
Does not compare performance indicators
Comparison of performance by integration
Available quantitative properties
Quali-quantitative properties
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
9
Neely et al. (2003), Paranjape (2006)
3rd Generation Indicators
Emphasis
Improve specific contexts in a personalized way
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
10
Adaptation of Ensslin, Ensslin (2009)
3rd Generation Indicators
Paradigm exchange
Isolated measures Integrated measures
Quantitative measures Quali-quantitative measures
Generic measures Specific measures for each context
Measures to control Measures to improve performance
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
11
Traditional historical and
financial orientation
Neely, Powell (2004)
Focus on the measure,
misaligned with the goals
and unbalanced
Leandri (2001)
Generic or pre-existing
Denton (2005), Tangen (2003)
Even with the recognition of new requirements, many applications of performance measurement continue to use the
1st and 2nd generation.Introduction
Instrument of Intervention
Evaluation Model
Conclusions
12
Consequences
Schneiderman (1999), Neely (2000), Bourne et al. (2002)
Denton (2005)
Measures not aligned to strategy.
Performance Evaluations poorly developed, misaligned with the strategy and decision-makers do not understand the impacts of their decisions and actions.
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
13
Research Theme
Explore ways of improving the identification / construction of the
Performance Indicators.
Provide greater chance of achievement the strategic objectives based on the tools
selected.
Introduction
Instrument of Intervention
Evaluation Model
Conclusions
Ensslin, Dutra, Ensslin (2000)
14
MCDA-C Phases
Instrument of Intervention
Introduction
Evaluation Model
Conclusions
15
Ensslin et al. (2010)
Identify the important aspects in accordance with the decision maker.
Expand the understanding of the context of the decision maker.
Allow to take into account the interests of the actors involved with the monitoring of
the decision maker.Allow the decision maker to revise his
views in the course of the construction of their knowledge.
Recognize the ordinal and cardinal properties of the indicators.
Ensure that the decision maker legitimize the process with tools scientifically valid.
MCDA-C selected due:
Instrument of Intervention
Introduction
Evaluation Model
Conclusions
16
Structuring Phase
Identification of the Actors Subsystem
Evaluation Model
Introduction
Instrument of Intervention
Conclusions
17
Label:
Build a model to evaluate Performance Indicators as their contribution to the process of performance
evaluation.
Structuring Phase
Evaluation Model
Introduction
Instrument of Intervention
Conclusions
18
Primary Assessment Elements (PAE):
Structuring PhaseEvaluation Model
Introduction
Instrument of Intervention
Conclusions
Goal clearly defined
Practicality
Key objectives
Mathematical operations
Precision
Total = 63
19
Best possible performance
Worst possible performance Scales admit only the statistical operations of mode,
counting and frequency.
Objective underlying the PAE : Capactity to perform mathematical operations between scales.
Scales admit all statistical operations.
PAE 61: mathematical operations
Intensity: Very Strong
Concept 61: Ensure that mathematical operations can be performed between the scales ... ignore this potentiality of the scales.
Structuring Phase
To Transform the PAEs in Concepts
Evaluation Model
Introduction
Instrument of Intervention
Conclusions
Structuring Phase
Evaluation Model
Introduction
Instrument of Intervention
Conclusions
Label
Areas of Concern
Concepts
Value Functions
Substitution Rates
Scales
Actions of improvement
Dissemination
Measurability
25. Ensure the use of processes for reviewing and updating the indicators ... indicators are static.
28. Ensure that the resources demanded for the use of the indicator meets the expectations of the maker ... stop using the indicators.
51. Ensure the operationalization of the indicators ... the set of measures would be implemented.
62. Ensure that different groups of stakeholders consider there to ease the adoption of the indicators ... no barriers to adoption.
73. Ensure that the different actors understand the importance of this package ... there questioning the necessity of the measures.
77. Decision maker considers acceptable estimate of time to make the measure ... time required to perform the measure exceeds the expected return for information.
Ensure the use of a management process for the set of measures ... use procedures not formalized.
Ensure that different actors have the same understanding of how to realize the extent ... way to measure have different interpretations.
Ensure that the different actors to recognize who is responsible for performing the steps .. there is a lack of commitment.
Operationality
24
Competitive
Excelence
Endanger
Identification and Construction
Mathematical Foundations
Build a model to evaluate Performance Indicators as their contribution to the
process of performance evaluation
PVF 6 – Ordinal Scales
PVF 7 – Cardinal Scales
PVF 3PVF 4 –
Value Function
PVF 5 – Substitution
RatesPVF 1 PVF 2
100
0
100
0
100
0
100
0
100
0
100
0
100
0
Good
Neutral
X% Z%
A% B% C% D% E% F% G%
Impact profile of Indicator1
Impact profile of Indicator2
Impact profile of Indicator3
Legend:
Global Evaluation
Indicator 1
Indicator 2
Indicator 3
72
68
40
Knowledge Increase
W%
Evaluation Model
Introduction
Instrument of Intervention
Conclusions
General view of the final model.
25
Conclusion
The model will serve as a support instrument to make Performance Evaluations, in order to have greater accuracy and alignment between the operational level, tactical and strategic organization.
The model will give ways for improving the identification / construction of the Performance Indicators.
Conclusions
Introduction
Instrument of Intervention
Evaluation Model
26
THANKS
END
Msc. Edilson Giffhorn, PMP, IPMA: [email protected]
PhD. Leonardo Ensslin: [email protected]
PhD. Sandra Rolim Ensslin: [email protected]
Msc. William Barbosa Vianna: [email protected]