prompt evaluation of the follow-up irrs mission to … documents/open shared... · web viewprompt...

IAEA

12/20/2013

2013Prompt Evaluation of the follow-up IRRS Mission to the UK

Prompt Evaluation of the follow-up IRRS mission to the UK Page | 1

Table of Contents1. Introduction...............................................................................................................................2

2. Basic Mission Data.....................................................................................................................2

3. Feedback from the Team Members on the Advance Reference Material of the IRRS Mission..2

Characteristic additional comments..............................................................................................3

Further specific remarks................................................................................................................3

General conclusions.......................................................................................................................3

4. Feedback from the Team Members on the Effectiveness of the Mission..................................3

Characteristic additional comments..............................................................................................4

Further specific remarks................................................................................................................4

General conclusion........................................................................................................................4

5. Feedback from the Host Country on the Findings of the Initial Mission....................................4

6. Feedback from the Host Country on the Effectiveness of the Follow-up Mission.....................5

7. Overall Evaluation of the Effectiveness and Efficiency of the Mission.......................................6

Discussion and conclusions............................................................................................................8

8. Summary....................................................................................................................................8

Appendix 1: Module-wise Coverage in the Mission Report...............................................................9


1. Introduction

The Contribution Agreement ENER/11/NUCL/SI2.588650 between the European Atomic Energy Community (represented by the European Commission) and the IAEA among others foresees as an expected result a performance monitoring based on the evaluation of some key performance indicators of the IRRS missions.

The Nuclear Safety Action Plan of the IAEA has as one of its main objectives the requirement to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States” and in specific, it calls the IAEA Secretariat to review the effectiveness of the IRRS peer reviews.

In reply to these requirements a system of performance indicators have been elaborated in order to measure the efficiency and effectiveness of the IRRS missions. Some of the performance indicators are based on direct feedbacks from the IRRS team members and from the representatives of the country hosting the IRRS mission.

Results of and conclusions from the feedbacks and efficiency and effectiveness evaluations are given in the present Prompt Evaluation Report.

2. Basic Mission Data

Host country: United KingdomHost organization: Office of Nuclear Regulation (ONR)Mission date: 30 September – 9 October 2013Team Leader: Bill BORCHARDT (USA)Deputy Team Leader: Ingemar LUND (SWE) Team Coordinator: Adriana NICIC (IAEA NSNI)Deputy Team Coordinator: Gerard BRUNO (IAEA NSRW)Number of external experts: 12Number of IAEA staff: 4Number of observers: 0Mission type: extended follow-upScope of the mission:

Core modules (No. 1 through 10) Facilities and activities: NPP, fuel cycle facilities, radioactive waste facilities, radioactive

sources facilities, decommissioning activities Additional areas (Module 11): occupational radiation protection; control of discharges and

materials for clearance and chronic exposure, environmental monitoring for public radiation protection

Fukushima module Policy issues: (1) Operating Model and Programme Working; (2) Openness and transparency

3. Feedback from the Team Members on the Advance Reference Material of the IRRS Mission

The Advance Reference Material (ARM) normally includes the results of the self-assessment of the host country (usually performed with the aid of the Self-Assessment Tool – SARIS); the Action Plan for improvement in issues found in the self-assessment, a module-wise summary of the status and activity of the regulatory body reviewed and a number of other documents needed for an objective and well informed peer review.

The team members are requested to offer their opinions on the quality of the ARM by rating from 1 to 5 (5 reflecting the highest satisfaction) three questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:


No. Question Average markQr1 How complete do you consider the ARM? 4.0

Qr2 How realistic picture could you obtain on the area you will be reviewing from the ARM?

3.3

Qr3 What is your overall evaluation on the quality of the ARM? 3.3

Total average 3.5

Characteristic additional comments

Qr1: The ARM was considered generally complete although at some points lacking some

necessary information; The ARM reflects an extensive self-assessment work, although uneven; Some parts were not thorough or clear enough.

Qr2: The ARM gave a realistic picture, yet some details were unsatisfactory; Due to the quality of the questionnaire the picture can be considered realistic at a limited

scope only; The Annual Plan and the Strategic Plan were good background.

Qr3: The quality was different for the different modules; The ARM should be more structured; The structure of the questionnaires in some cases does not follow that of the IAEA safety

standards.

Further specific remarks

The Share Point access was very effective; Somewhat less documents with more summaries might ease the work of reviewers

General conclusions

1) The ARM was generally well received, some information was missing;2) The ARM should be more structured.

4. Feedback from the Team Members on the Effectiveness of the Mission

The team members are requested to offer their opinions on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) five questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No. Question Average markQt1 How effective do you consider the activity of the expert team during the

mission?4.1

Qt2 How effective do you consider the activity of the IAEA staff in the team during the mission

4.6

Qt3 How effective do you consider the activity of the Team Leader? 5.0

Qt4 How effective do you consider the activity of the Deputy Team Leader? 4.6

Qt5 How satisfied are you with the preparations of the mission (enough time for preparing yourself, information provided by the IAEA, etc.)?

3.8

Total average 4.4


Characteristic additional comments

Qt1:

The mix of experts in the team was appropriate, the team was very effective; There was good communication among the team members and during the team-meetings; In some cases the guidance and instructions were not fully followed by the team; A few experts focused on the performance of the licensees.

Qt2:

Good support from the IAEA staff, very good coordination before and during the mission; They were focussed, had experience, perhaps I bit too involved; Smooth progress, good and clear instructions;

Qt3:

The TL has shown good leadership and managing ability, communicated clearly; He did a good job in supporting the team members; The TL was very effective and experienced leader; He was particularly effective in resolving issues related to findings;

Qt4:

The DTL was very good in keeping cohesion of the team and in giving support in interviews.

Qt5:

There was not sufficient time to read the ARM; Some of the practical information arrived late; The site visit made it difficult to prepare the first draft in time; Information on the initial meeting should come 4 weeks before the mission; IAEA provided sufficient time and preparatory information, the use of Share Point is

beneficial.

Further specific remarks

First-time participants should be paired with experienced reviewers; The ARM should be summarized in a comprehensive manner; The specified format of the report for this mission was complicated, particularly for the

Fukushima Module; In extended follow-up missions two reviewers should review a new Module; More advance information would be useful, like the entrance meeting presentation before

the mission and the names of the counterparts.

General conclusion

1) The team was effective and well composed;2) The IAEA staff was well prepared, effective, supportive;3) Both the TL and the DTL acted very well;4) A Summary report of the ARM is required5) In the preparatory phase more instructions, guidance and information is to be provided to

the team;

5. Feedback from the Host Country on the Findings of the Initial Mission

The Liaison Officer of the host country is requested prior to the follow-up mission to offer the opinion of the host on the effectiveness of the findings offered in the initial mission by rating from 1 to 5 (5 reflecting the highest satisfaction) two questions for each of the findings (Recommendations and Suggestions); also expanding in free text their comments related to the findings. The questions, the host country’s ratings and the associated typical comments are given below:


No. Question MarkQf1 How accurate the particular recommendation/suggestion was? 4.4

Qf2 How helpful the given recommendation/suggestion was for the improvement of the regulatory body?

4.2

Average 4.3

Further comments:

Certain findings were thought not necessary for improvement or addressed solved issues Some findings could have been better worded Some of the findings were considered particularly valuable and helpful in addressing complex

problems The addressee of some of the findings was mistaken

6. Feedback from the Host Country on the Effectiveness of the Follow-up Mission

The Liaison Officer of the host country is requested to offer the opinion of the host on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) seven questions and also expanding in free text their comments on the questions. The questions, the host country’s ratings and the associated comments are given below:

No. Question MarkQh1 How effective do you consider the mission in assisting the continuous

improvement of nuclear safety in your country?5

Qh2 How objective was the peer review? 5

Qh3 How has the mission helped the exchange of information, experience and good practice with other countries?

5

Qh4 How consistent was the use of the IAEA safety requirements and guides in the mission?

5

Qh5 How justified are the findings of the peer review? 4

Qh6 How relevant are the findings of the peer review for the future development of your regulatory body?

4

Qh7 How competent were the reviewers in their review and findings? 5

Average 4.7

Further comments:

Previous missions to the UK have been highly effective. The Self-Assessment is a huge part of this. The IRRS Mission itself provides a timing constraint that focuses the delivery of improvement, which is one reason for inviting the mission back in 12 months’ time;

The mission was highly objective and after some initial queries focused solely on the agreed scope;

Some of the previous mission findings, followed up during this mission, were highly relevant to exchange and learning from other countries. The sharing of the Good Practices is a key part of this;

The use of IAEA safety requirements was very consistent, even at the expense of perhaps more relevant requirements, such as EU BSS;

The findings are mostly relevant for development. However, the 2009 outstanding finding and a couple of the suggestions are regarding activities which will be delivered anyway, and so are not relevant for development;

The reviewers were all highly competent on their field. There was some degree of uncertainty in scope (e.g. were hospitals to be reviewed) but this was quickly remedied.


7. Overall Evaluation of the Effectiveness and Efficiency of the MissionThe Performance Indicators developed for the measurement of the effectiveness and efficiency of an IRRS mission were evaluated in the extent as it was made possible by the data available at the time of the present evaluation.

In the next figure the values of the performance indicators as they follow from the evaluation of the mission data1, as well as the overall effectiveness of the mission are presented. The rightmost columns (EFF. INDICATION) present the ranges where the particular PIs fall (green – optimum, yellow – acceptable, red – needing attention), whereas the frame in the right lower part summarizes the overall effectiveness of the mission (green – optimum, white – effective, yellow – acceptable, red – to analyse).

The bar diagram in the figure shows the number ratio of issues covered by the mission report to those foreseen by the Standard Mission Report Template. Details on this coverage are given in Appendix 1.

1 On the Efficiency and Effectiveness of IRRS Missions, Draft v6, Working Material, IAEA, Vienna, 2013

Prompt Evaluation of the IRRS mission to the Slovak Republic P a g e | 7

Efficiency and Effectiveness of an IRRS Mission follow-up

INPUT QUANTITIES valueA priory mission data No(id) Unit Value min.(>) max.(≤) min.(>) max.(≤) col PI

Type of mission (1 - initial or 2 - follow-up) 2 optimum 0.0 C1(T) - 16 11 14 10 15 0.14 Team sizeHost country UK effective 0.1 C2(P) - 111 75 92 67 100 0.21 Report lengthStating time of the mission (yyyy-mm-dd) 2013-09-30 acceptable 0.2 C3(tARM) day 59 45 - 30 45 0.00 ARM review timeEnding time of the mission (yyyy-mm-dd) 2013-10-09 needs att ention C4(NAWC) - 10 9 - 7 9 0.00 Advance commentsNumber of modules (nT) 9 C5(fbARM) - 3.54 4.00 5.00 3.00 4.00 0.12 Feedback on ARMNumber of supplementary modules (in follow-up) (nM+) 2 T 0-first -0.96 C6(Exp) - 0.58 0.66 1.00 0.50 0.66 0.12 Team experienceNumber of facility and activity types (nx) 3 λ T-first 1.48 C7(fbHost) - 4.71 4.00 5.00 3.00 4.00 0.00 Host feedbackNumber of additional technical areas (nT) 2 T 0-fu 3.09 C8(fbTeam) - 4.43 4.00 5.00 3.00 4.00 0.00 Team feedbackNumber of policy discussion issues (nP) 2 λ T-fu 0.71 C9(β0) - - 1.00 - 0.80 1.00 - - Action Plan ExtentNumber of NPP units in the host country (ν) 43 C10(β1) - - 1.00 - 0.80 1.00 - - Action Plan coverage

Pre-mission data P 0-first 17.60 C11(β2) - - 1.00 - 0.80 1.00 - - Beyond AP coverageNumber of Team Members (including IAEA) (T) 16 λ M-first 5.83 C12(ρ1) - - 0.75 1.25 0.50 1.50 - - Balance of findingsNumber of Experts in the Team (without IAEA) (Te) 12 λ F-first 0.63 C13(ρ2) - - 0.75 1.25 0.50 1.50 - - Balance of R & SNumber of Experts with IRRS experience (Tx) 7 P 0-fu 54.20 C14(RCont) - - 0.90 1.00 0.80 0.90 - - Report concisenessARM distribution date (yyy-mm-dd) 2013-08-01 λ M-fu 1.55 C15(RTime) day 96 0 90 90 120 0.07 Report completionNumber of issues in the Action Plan (NAP) 0 λ F-fu 0.64 C16(OpenI) - 0.03 0.00 0.20 0.20 0.40 0.00 Open issues leftNumber of advanced ARM comments from TM's (NAWC) 10 μ 0.90 G1 (InitEff) - 4.28 4.00 5.00 3.00 4.00 0.00 Init. miss. effectiveness

Number of findings (R+S) in the initial mission (for follow-up) 33Mission data (R+S) av 39

Number of Recommendations (R) 13 R av 15.1Number of Suggestions (S) 12 G av 12.3Number of Good Practices (G) 6 (R*G) av 157.4Number of findings also in the Action Plan (N=) 0 (R/S) av 0.89

Number of issues remained open (in follow-up) 1Number of Report Pages (P) 111 Size of nucl.pr. 1Conciseness of the Mission Report (ρ) 0.00 Mission size 12

Feedback data (average marks) Opt.team size 12.6By the Team on ARM quality 3.54 Opt.rep.length 83

By the Host on mission effectiveness 4.71 EFF.& EFF. OF THE MISSION TOBy the Team on mission effectiveness 4.43 Mission type:By the Host on initial mission effectiveness (for follow-up) 4.28 Missions duration:

Post-mission data Average Measure of Deviation (Δ):

Date of isuance of mission report (yyyy-mm-dd) 2014-01-15 fields to input!

2013-09-30

effectiveOverall Effectiveness:November 2013 0.065

2013-10-09

follow-up

CALCULATED DATA

IAEA, Version 5

EFF. INDICATIONRel.Dist from Opt.

Optimum rangePERFORMANCE INDICATORS Acceptable range

UK

UK

LIMITS & PARAMETERS

For average deviation

For team size

For report length

For findings

0.000.100.200.300.400.500.600.700.800.901.00

11 12.1 12.2 12.3

Module-wise report conciseness

Prompt Evaluation of the IRRS mission to the Slovak Republic P a g e | 8

Discussion and conclusions

The values of the effectiveness and efficiency Performance Indicators (upper right table in the figure headed by EFF. INDICATION) suggest the following conclusions:

1) The size of the team and the length of the report were both above the acceptable ranges;2) Although the main body of the ARM was provided two months prior to the mission, several experts

complained of lack of time or mission materials;3) Most of the team members provided advance comments in time;4) 58% of the experts in the team have prior IRRS experience as reviewers; this is somewhat below the

optimum 2/3. The results and comments did not indicate any drawback of this noncompliance;5) The feedback on the ARM was in the yellow range, the actual comments explain the reasons;6) The team in general was highly satisfied with the effectiveness of the mission, yet provided suggestion

on further possible developments;7) The host appreciated rather high the mission effectiveness, also with suggestions;8) The conciseness of the mission report was investigated for the part of the report that contains the

topics by which the follow-up mission was extended. The results are seen on the bar-diagram in the figure above. It is apparent that the coverage of the topics listed in the Standard Mission Report Template was rather poor, this is especially true for Module 12 (Supervision of non-NPP facilities and activities) related to radiation sources, waste management and decommissioning, where the average coverage is below 25%. (For details, see Appendix 1.) This performance indicator value, as pertains only to a part of the mission, is not included into the average effectiveness and efficiency value.

9) For the first time since the prompt evaluation reports are prepared, a follow-up mission was evaluated, including the host feedback on the effectiveness of the initial mission (performance indicator G1 in the table). The host expressed its satisfaction by giving an average mark in the optimum range;

10) The overall effectiveness of the mission is in the white (effective range) and its value is somewhat superior to the average in recent missions.

8. Summary

The data of and feedback from the extended follow-up IRRS mission to the United Kingdom have been analysed. The following conclusions are drawn:

In the preparatory phase more instructions and information are required by the team The ARM was generally well received, some information was missing; A summary report of the ARM is required in the future; The team was effective and well composed; Both the TL and the DTL acted very well; The IAEA staff was well prepared, effective, supportive; The team was somewhat oversized; The team included experienced reviewers in slightly smaller proportion than desirable; The team has shown strength in effectively performing their tasks and providing results in time; Certain parts of the mission report are considerably less comprehensive than they should be; The initial mission proved to be effective in the sense that all but one findings therein were accepted

and resolved by the host prior to the follow-up mission The host appreciated the results of the missions as well as the usefulness of the findings in the initial

missions, and provided valuable suggestions as for increasing further the effectiveness and efficiency of the IRRS missions;

The mission was rated effective by the evaluated Performance Indicators.


Appendix 1: Module-wise Coverage in the Mission Report

Conciseness of the Mission Reportconciseness ratio: 0.27

77 21M No. Report section in scope in report M Module-wise

XI 11.1 Occupational radiation protection - legal and regulatory framework 1 1 - general responsibilites of registrants, licensees and employers 1 1 - general responsibilites of workers 1 1 - requirements for radiation protection programmes 1 1 - monitoring programmes and technical services 1 1

11.1 Control of rad. discharges and materials for clearance; envir. monitoring for public rad. prot. - req.s on exemption & clearance 1 1 - dose constraints in the R&Gs 1 0 - regulatory limits for discharges from facilities and activities 1 0 11 0.75

XII.1 12.1.1 Authorization - roles and responsibilities 1 1 - types of authorization 1 1 - process of authorization 1 1 - safety hazards associated with sources 1 0 - reuse or recycling of rad. material 1 1

- authorization of import and export of sealed sources 1 012.1.2 Review and assessment - graded approach 1 0

- responsibility for R&A 1 0 - to address all radiation risks 1 0 - types and number of documents to R&A 1 0 - availablity of internal guidance 1 0 - R&A plan 1 0 - manpower and organizational arrangements 1 0

12.1.3 Inspection &enforcement - areas and aspects 1 1 - graded approach 1 0 - types of inspections, inspection methods, techniques 1 1 - annual plans 1 0 - 3rd party and joint inspections 1 0 - procedures, guides, reports 1 0 - follow-up and use of inspection results 1 0 - enforcement 1 0

12.1.4 Regulations & guides - process of development 1 0 - graded approach 1 0 - review and updating 1 0 - reflecting IAEA safety requirements and best practices 1 0 12.1 0.24

XII.2 12.1.1 Authorization - roles and responsibilities 1 1 - types of authorization 1 0 - process of authorization 1 1

12.1.2 Review and assessment - graded approach 1 0 - responsibility for R&A 1 1 - to address all radiation risks 1 0 - types and number of documents to R&A 1 0 - availablity of internal guidance 1 1 - R&A plan 1 0 - manpower and organizational arrangements 1 0


12.1.4 Regulations & guides - requirements for national policy and strategy 1 1 - interdependence among predisposal management steps 1 0 - requirements on the identification, control and minimzation of RAW 1 0 - characterization and classification of RAW 1 1 - requirements on waste acceptance criteria 1 0 - reflecting IAEA safety requirements and best practices 1 0 - for RAW predisposal man.: RAW packages; retrievability 0 0 - for RAW disposal: passive means; defence in depth 0 0 - for RAW disposal: siting conditions; closure; institutional control 0 0 12.2 0.25

fields to input!

UKfollow-up

Rad.

Sour

ces

Was

te m

anag

emen

t


XII.3 12.1.1 Authorization - retaining key staff 1 1 - compliance with end state criteria 1 0 - regulatory control if not released 1 0

12.1.2 Review and assessment - graded approach 1 1 - responsibility for R&A 1 0 - to address all radiation risks 1 0 - types and number of documents to R&A 1 0 - availablity of internal guidance 1 0 - R&A plan 1 0 - manpower and organizational arrangements 1 0


12.1.4 Regulations & guides - provisions to develop decommissioning strategy 1 1 - responsibilitites for financial provisions 1 0 12.3 0.15

Deco

omm

isio

ning

prompt evaluation of the follow-up irrs mission to … documents/open shared... · web viewprompt...

Documents