meta-evaluation of the performance evaluation system of public research institutes in korea chan goo...
TRANSCRIPT
Meta-evaluation of the Performance Evaluation System of Public Research
Institutes in Korea
Chan Goo Yi(Pukyong National University, Korea ; [email protected])
Jang Jae Lee(Korea Institute of Science and Technology Evaluation and
Planning (KISTEP) ; [email protected])
Yong Soo Hwang(Korea Science and Technology Policy Institute (STEPI) ;
AEA Evaluation 2009 Conference Nov. 11-14, Orlando, Florida
2
Contents
1. Introduction 2. Research Method and Framework 3. Korean Research Institute Evaluation 4. Current Practise & Meta-evaluation 5. Discussion for Future Development 6. Conclusion 7. References
3
1. Introduction
� Background - In Korea, evaluation system of research
institutes introduced in 1999, and transferred into the performance evaluation system in 2005
- Arguments among various stakeholders such as CEO, researchers and evaluation panel, whether evaluation can contribute
• the quality enhancement of R&D results • the development of management
system
4
1. Introduction(con.)
� Research Purpose
- To meta-evaluate (1) the rationality of evaluation system itself and (2) the appropriateness of its current practise
- To discuss policy alternatives for development of the evaluation system itself and its implementation
5
2. Research Method & Framework
� Research Method - In-depth interview with 109 stakeholders • conducted between March and May 2008 • 99 internal stakeholders : CEOs(10), managers in administrative dept.(27), principal investigators(30), researchers(32)
• 10 external stakeholders : evaluation panels from university(4), industry(3) and public research institute(3) - Meta- evaluation approach : Evaluation of
evaluation system and practical process
6
2. Research Method & Framework(con.)
� Research Framework : Components of In-depth Interview & Meta-evaluation
Implementation(4)
Paradigm(2)
Purpose Object
Utilization(2)
Impact Type
Panel Interval
Method Indicator
7
3. Korean Research Institute Evaluation
� Brief History of Institute Evaluation System - Introduction period (1999-2002) • similar system operated among research councils • research achievements < management achievements - Diversification period (2002-2005) • improved representing characteristics of individual
member research institutes • research achievements ≒ management achievements - Development period (2006-current) • transferred performance evaluation system • focusing rather outcome or impact than output • research achievements > management achievements
8
3. Korean Research Institute Evaluation(con.)
� R&D Governance in Korea : 3 Research Councils System
- Korea Research Council for Fundamental S&T (KRCF) • 13 member research institute • under the Ministry of Education, Science &
Technology - Korea Research Council for Industrial S&T (ISTK) • 13 member research institutes • under the Ministry of Knowledge Economy - National Research Councils for Economic,
Humanities and Social Sciences (NRCS) • 23 member research institutes • under the Office of the Prime Minister
9
3. Korean Research Institute Evaluation(con.)
� Framework of Current Evaluation SystemField Part Item Indicator
ResearchAchieve. (70%)
Individual Performance Goals (50%)
Objective 1 Sub Objective 1.1 Sub Objective 1.2
Objective 5 Sub Objective 5.1 Common Items (7) Elective Items (10)
Indicator 1.1.1 Indicator 1.1.2 Indicator 1.2.1 Indicator 1.2.2
Indicator 5.1.1
Selected by each institutes autonomously
Comprehensive Performance Goals (20%)
Manage.Achieve.(30%)
Responsibility & Innovation
R&D & Resource Management
3 indicators
3 indicators
10
4. Current Practise & Meta-evaluation
(1) Evaluation Paradigm� Evaluation Purpose - Formal : future development strategies,
research performance enhancement, accountability, program/management improvement, knowledge transfer etc.
- Actual : R&D program/project development, internal management system improvement
(Meta-evaluation) - Incompatible between formal purposes and
actual - Focused rather short-perspective evaluation
purposes
11
4. Current Practise & Meta-evalu.(con.)
� Evaluation Object - Research Achieve.(70%)vs.Management A.(30%) • Each field divided into sub parts and items - Covering both ‘basic R&D program’(grant fund)
and ‘national R&D program’(competition fund) (Meta-evaluation) - Too many evaluation objects • Not differentiating among evaluation objects - Absent of essential objects for core evaluation
purposes such as ‘future development strategy’ - Lacks of consensus for core evaluation objects
among stake-holders
12
4. Current Practise & Meta-evalu.(con.)
(2) Evaluation Implementation� Evaluation Panel - Panel from university, research institute and
industry, & comprising all domestic experts • Research : individual panel for each institute • Management : common panel for all institutes (Meta-evaluation) - Panel members’ professionalism limited • Professionals in sub performance goals of R&D
project rather than peer reviewer or upper performance goal in R&D program
- A few lacks of considering international excellency
13
4. Current Practise & Meta-evalu.(con.)
� Evaluation Interval - Until 2007 : evaluating all institutes every year - From 2008 : separating research achievement
and management one • Research A. : 3 years ; Management A. : 1 year (Meta-evaluation) - Too often evaluated and burden to institutions • Main factor negative affecting other components
such as evaluation purpose, object, utilization - Resulted in more focusing the visible and short
term outputs rather than long term and comprehensive outcomes or impact
14
4. Current Practise & Meta-evalu.(con.)
� Evaluation Method - External panel conducts full evaluation process - Evaluating 4/5 member institutes as one group in
a comparative perspective with others - Combination of the ‘review of performance
report’ submitted by each institute and the ‘site visit’ for four or five hours in individual institute
(Meta-evaluation) - Focused on literature review rather than site visit - Evaluation in a comparative/relative perspective
not an absolute one, in particular for the research performance
15
4. Current Practise & Meta-evalu.(con.)
� Evaluation Indicator : Focused on Research - Each Program(50%) vs. Comprehensive R&D(20%) • Performance goals -> Objects -> Indicators - Each institute suggests their own goals/indicators • Evaluating performance level targeted in advance
(Meta-evaluation) - Disconnection of performance goal and core R&D
activities - Indicator pool limited for comprehensive R&D
performance goal - Indicators more fit for research program rather than
development one
16
4. Current Practise & Meta-evalu.(con.)
(3) Evaluation Utilization� Evaluation Impact - Evaluation findings directly fed back to the basic
R&D(grant fund), but indirectly did to national R&D(competition fund)
- Feed back to internal management system (Meta-evaluation) - Evaluation findings more effecting management
system rather than R&D management process - Lacks of consensus of impact among
stakeholders • Insiders : low, Outsiders : relatively high
17
4. Current Practise & Meta-evalu.(con.)
� Type of Evaluation Utilization - Formal : future development strategy, R&D/
management improvement, R&D prioritization and budget allocation, best practise, CEO’s annual pay adjustment, policy suggestion
- Actual : budget reallocation of basic R&D, adjustment of CEO’s annual payment, best practise
(Meta-evaluation) - Limited and confined evaluation utilization - Instrumental utilization is further actual type
than conceptual one
18
5. Discussion for Future Development
� Outline of Discussions for Developments - Institutional Approach (IA) • Development/amendment of evaluation
system itself and related systems at the level of research councils or the government
• Long-term and institution-based perspective - Operational Approach (OA) • Improvement of evaluation practise and
process under the current system • short-term and operation-based perspective
19
5. Discussion for Future Development(con.)
(1) Evaluation Paradigm� Evaluation Purpose - Need to transfer from internal
responsibility to external one (OA) • Because current system has more
contributed internal responsibility/ management rather than external one
- More focusing responsibility for external stakeholders (OA)
• In particular, responsibility for citizen
20
5. Discussion for Future
Development(con.)
� Evaluation Object - Rearrangement of current evaluation object
(IA) • Whether competitive national R&D program
be included or not? - Re-setting the evaluation objects in the longer
perspective, such as; (IA) • Long-term vision/strategy of the institution • Future potentials and R&D infrastructures • Risk management in related public sector
21
5. Discussion for Future
Development(con.)
(2) Evaluation Implementation� Evaluation Panel
- Setting up individual/independent panel for each institution, from single panel for all (IA)
- Enhancement of professionalism of panel (OA) • Recruiting more field experts like as industries • Extension of job term from 2 to 4/5 years • Career management system of panel members - Adoption of international experts panel (OA)
22
5. Discussion for Future
Development(con.)
� Evaluation Interval
- Extension of evaluation interval from 1 year to 3 to 5 years (IA)
• Linkage with CEO’s term in office • Same or different evaluation interval
between research results and management results
• Consideration of characteristics of research fields; such as emerging technology vs. long-term basic science
23
5. Discussion for Future
Development(con.)
� Evaluation Method - Enforcement of evaluation method in the way
of absolute perspective (IA) • In particular, evaluation of research results - Focusing in-depth review in research lab (IA) • Extending evaluation period to 3 to 4 days • Reviewing first-hand materials(research
note) • Interview and discuss with researchers - Introduction of cross-cutting review • Among related institutions/organizations in
public sector (IA)
24
5. Discussion for Future
Development(con.)
� Evaluation Indicator - Improvement of method of establishing
performance goals/indicators (OA) • Closer connectivity between performance
goal/indicators and core R&D activities • More changeable and creative
goals/indicators - High linkage between performance goals and
internal performance management system (OA)
• For example, BSC, MBO, ISO 9001, KM etc - Increase of indicator pools for comprehensive
performance goals (OA)
25
5. Discussion for Future
Development(con.)
(3) Evaluation Utilization� Evaluation Impact - Enforcement of feed back system of
evaluation findings to R&D management (OA)
• High collaboration with related other agencies for national R&D management
- Extension of scope and target group of evaluation utilization (OA)
• From top manager to all employees
26
5. Discussion for Future
Development(con.)
� Type of Evaluation Utilization - Activation of long-term and conceptual
evaluation utilization (OA) • Setting up vision/mission • Planning R&D strategy • Disseminating the best practise • Producing policy information/knowledge
etc - Informing the multiple type of evaluation
utilization to all stakeholders (OA) • In particular, in-site researchers
27
6. Conclusion
� Summary of the research - In basic, both internal and external
stakeholders consider the current system somewhat useful for R&D management and organizational management
- They also suggest policy alternatives for the development of certain components of evaluation system and its current practise
• Some are the system itself, others are the common limits of evaluation system of public sector in general in Korea
28
6. Conclusion (con.)
� Implication - Policy alternatives for
developments of evaluation system for research institutes in terms of system itself and current practise
- Policy knowledge/ information for analysis and re-establishment of the governance of public research institutes
29
6. Conclusion (con.)
� Limitations and Further Works - Discussion of policy alternatives a little
bit lacks specific and detailed matters in some individual components
• Scope and depth of policy alternatives in certain components is too broad
- Need for more detailed action plan in each evaluation component, based on the findings of this meta-evaluation/research
30
7. References � Further Information for Korean S&T and
Evaluation System - National Science and Technology Council (
http://www.nstc.go.kr) - Ministry of Education and Science and
Technology (http://www.mest.go.kr) - Korea Research Council for Fundamental
S&T (http://www.krcf.re.kr) - Korea Research Council for Industrial S&T (
http://www.istk.re.kr) - Korea Institute of S&T Evaluation and
Planning (http://www.kistep.re.kr)