introduction of research and development project evaluation system at nedo
DESCRIPTION
Introduction of Research and Development Project Evaluation System at NEDO. 28 October 2005 CES/AEA Conference in the Sheraton Centre, Toronto. Momoko OKADA New Energy and Industrial Technology Development Organization(NEDO). -CONTENTS-. ABOUT NEDO R&D EVALUATION SYSTEM AT NEDO - PowerPoint PPT PresentationTRANSCRIPT
1
Introduction of Research and Development Project Evaluation System at NEDO
Momoko OKADANew Energy and Industrial Technology Dev
elopment Organization(NEDO)
28 October 2005CES/AEA Conference in the Sheraton Centre, Toronto
2
-CONTENTS-
• ABOUT NEDO
• R&D EVALUATION SYSTEM AT NEDO -INTERMEDIATE & EX-POST EVALUATIONS -FOLLOW-UP MONITORING& EVALUATIONS
• CLOSING
3
-ABOUT NEDO-
• Establishment -Established in October 1980 as a semi-governmental organization under
the supervision of METI. - Original activities : ・ national R&D projects on new energy
-Current activities : ・ Support of industrial technology R&D (in 1988), ・ Support of environmental technology R&D (in 1990), and ・ Introduction and promotion of new energy and energy conservation technology( in1993).
-NEDO became an Incorporated Administrative Agency as of October 2003 in the process of recent government reform.
4
• Mission -NEDO’s current missions are: ・ Promote new energy and energy conservation technologies through
R&D and subsidy programs,
・ Conduct and support industrial technology R&D, and
・ Enhance international cooperation concerning energy and other
technologies.
-ABOUT NEDO-
5
• Organization
Staff : approximately 1,000 Budget : US$2.1 billion (FY2005)
-ABOUT NEDO-
Machinery System Technology Development Dept.
Environment Technology Development Dept.
Fuel Cells & Hydrogen Technology Development Dept.
New Energy Technology Development Dept.
Energy Conservation Technology Development Dept.
Coal Projects Dept., Coal Mine Subsidence Dept.
Biotechnology & Medical Technology Development Dept.
Nanotechnology & Materials Technology Development Dept.
Electronic & Information Technology Development Dept.
International Projects Dept.
Energy and Environment Policy Dept.
R&D Promotion Dept.
R&D Project Evaluation Dept.R&D Project Evaluation Dept.
Policy Planning and Coordination Dept
General Management Depts. General Affairs; Accounting; Assets Management; Inspection and Operational Management; and Information and Systems.
Chairman, President & Executive Directors AuditorHeadquarters
6
• R&D support -R&D support is NEDO’s core activity.
・ 58% of the total budget (US$1.2 billion) is allocated for R&D activities in FY 2005.
・ Number of NEDO’s R&D projects in FY2005 is 174.
-ABOUT NEDO-
- Electronics and information - Environment- Nanotechnology and materials - Fuel cells and hydrogen- Biotechnology and medical technology - New energy- Machinery systems - Energy conservation
NEDO’s R&D Technology Areas
7
• Challenge for NEDO -Need more effective and efficient operation through:
・ Improved services, and
・ Flexible resource management
-ABOUT NEDO-
8
• R&D Project Management Cycle
PLANPLAN
DODOSEESEE
R&D Project Evaluation Dept.
Board ofExecutive Directors
Board ofExecutive Directors Technology
Development Depts.
Policy Planning and Coordination Dept.
-R&D EVALUATION AT NEDO-
9
• Purposes of R&D Project Evaluations as an important part of the R&D project management cycle.
-To reflect economic and social needs on R&D activities,
-To implement R&D projects more effectively and efficiently, and
-To improve project management system
-R&D EVALUATION AT NEDO-
10
Transparent -Public disclosure of evaluation results, methods, and reflectionsExplicit -Encourage discussionsEffective -Adopt an evaluation method for self- improvement of managementNeutral -External evaluators Efficient -Simple evaluation procedures
• 5 Principles of Evaluation at NEDO
-R&D EVALUATION AT NEDO-
11
• R&D Project Evaluations
Post-project activityProject implementation
Ex-anteevaluation
(2004-)
At R&D theme selection
Ex-anteevaluation
(2004-)
At R&D theme selection
Intermediateevaluation
(2001-)
In the middle of project term
Intermediateevaluation
(2001-)
In the middle of project term
Ex-postevaluation
(2001-)
1 year followingproject completion
Ex-postevaluation
(2001-)
1 year followingproject completion
▼ ▼ ▼ ▼
Follow-upmonitoring &
evaluation(2004-)
For 5 years afterproject completion
Follow-upmonitoring &
evaluation(2004-)
For 5 years afterproject completion
-R&D EVALUATION AT NEDO-
12
• Intermediate & Ex-post Evaluations
-In all procedures, 5 principles are followed
-Each project is assessed and evaluated from the following areas :
・ purpose and strategy,
・ project management,
・ R&D achievements, and
・ prospect for practical applications and other impacts
-R&D EVALUATION AT NEDO-
13
• Intermediate & Ex-post Evaluations - R&D Evaluation Methodology (Rating Method)
1. Each evaluator grades (A, B, C or D) for each evaluation item. A: Excellent, B: Good, C: Acceptable, D: Not acceptable2. An average of rating is used as an indicator. A=3, B=2, C=1, D=0
Items for evaluation average ratings
1. Purpose and strategy 2.3 A A B B B B
2. Project management 1.8 A B C B C B
3. R&D achievement 1.5 B C C C B B
4. Prospect for practical applications and other impacts 1.2 B B D C C C
-R&D EVALUATION AT NEDO-
【 example】
14
-R&D EVALUATION AT NEDO-
1.0
1.5
2.0
2.5
3.0 12
21
225,10,28
29
14
26
179
18
62
3,4
1
7
8,30
23
16
13,24
15
27
9
11
20
25
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0 0.5 1.0 1.5 2.0 2.5 3.0
R&D achievement (X)Pro
sp
ect
for
pra
cti
cal ap
plicati
on
s (Y
)X+Y=4
X+Y=3Good
Excellent
Score Results for Ex-post Evaluation in FY2004
15
• Intermediate & Ex-post Evaluations
Public (citizens)
NEDO
Chairman
Releasing of evaluation results to the public
Deliberation and approval of the evaluation report (draft)
Reflection of evaluation results in projects, etc. Promoting
DepartmentEvaluation report
Preparation of evaluation report (draft)
Promotion Department
Project Participants
SubcommitteeA
SubcommitteeC
SubcommitteeB
SubcommitteeD
Evaluation Secretariat
Explanation of the project
(4-9 persons)
(13 persons)R&D Project Evaluation Committee
-R&D EVALUATION AT NEDO-
16
-R&D EVALUATION AT NEDO-• Intermediate & Ex-post Evaluations
-In response to evaluation results:• Lessons learned from ex-post evaluations may be reflected
upon other on-going or new projects.
• Intermediate evaluations may result in modification or termination of projects.
Responses to Intermediate Evaluation Results
2001 2002 2003 2004
Accelerated - - 2 13
Continued as planned 4 2 12 12
Modified 16 6 15 6
Terminated 2 5 2 2
total 22 13 29* 29*
*some items are overlapping: all items added up do not match the total
(start)
17
-R&D EVALUATION AT NEDO-
• Follow-up Monitoring & Evaluations (since 2004)
-Objectives of this new initiative are:
・ To understand in what ways specifically the outcome of R&D provides benefits to the public
・ To reflect the outcome of R&D in improvement of the management and planning function at NEDO
18
• Follow-up Monitoring & Evaluations
-R&D EVALUATION AT NEDO-
Tracked with the Follow-up Monitoring
Post-Project Activities
NEDO’s R&D Project(3-5 years)
R&DPractical
ApplicationsCommercialization
Post-Project Activities of Private Companies
19
• Follow-up Monitoring & Evaluations-NEDO initiatives as of October 2005: Subjects: 56 R&D projects , 642 entities
・ These projects subject to follow-up monitoring & evaluations have received ex-post evaluation in FY2002-2003.
・ These entities include private companies, universities, research institutes, and public institutions.
-R&D EVALUATION AT NEDO-
20
• Follow-up Monitoring & Evaluations• Findings:
-64% of the above entities are currently conducting post-project activities.
-22% of the entities have reached a stage of practical applications or commercialization of R&D achievements.
*Detailed information is being collected from 51 entities.• Utilization: -Make suggestions for improvement -Document key points to R&D management as a guideline
-R&D EVALUATION AT NEDO-
21
・ Evaluations will play an increasingly important role in: -Improving performance of NEDO as an Incorporated
Administrative Agency, and -Facilitating effective and efficient implementation of R&D project managements.
CLOSING
22
CLOSING
• NEDO intends to
・ Review the current evaluation system
・ Make appropriate modifications
・ Propose new initiatives
23
Thank you for your kind attention.
To be continued…