discussion paper on irrs development 2015 v2 documents/open shared...10.23: after the decision on an...

2015 IAEA

[FURTHER IMPROVING THE

IRRS PROCESS] Discussions points for the consultancy by experienced IRRS reviewers, Vienna, 20-22.04.2015

Discussion Paper on IRRS Development 2015_v2.docx

F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 2

Contents

FURTHER IMPROVING THE IRRS PROCESS .......................................................................................... 3

Introduction .................................................................................................................................. 3

I. Preparatory Phase of a Mission .............................................................................................. 4

I.1 Information meeting ...................................................................................................... 4

I.2 Preparatory meeting ...................................................................................................... 4

I.3 Advance Reference Material (ARM) ................................................................................ 5

I.4 Team size, assignment, recruitment, experience and training ......................................... 6

I.5 Expected preparatory activity of the team ...................................................................... 9

II. Conduct of a Mission ........................................................................................................... 10

II.1 Schedule of a mission ................................................................................................... 10

II.2 Observations ................................................................................................................ 10

II.3 Review actions ............................................................................................................. 14

II.4 Report writing .............................................................................................................. 16

II.5 Activity of team lead..................................................................................................... 17

II.6 Policy issues discussions ............................................................................................... 18

II.7 Feedback from the missions.......................................................................................... 18

II.8 Evaluation of team members’ performance .................................................................. 20

III. Post-mission Phase .......................................................................................................... 21

III.1 Finalization of the mission report .................................................................................. 21

III.2 Evaluation of missions .................................................................................................. 21

IV. Follow-up missions ........................................................................................................... 23

IV.1 Follow-up as a compulsory part .................................................................................... 23

IV.2 Timing of follow-up ...................................................................................................... 23

IV.3 Follow-up team composition ........................................................................................ 24

IV.4 Extended follow-up missions......................................................................................... 25

V. Development of the IRRS Process......................................................................................... 26

V.1 Scope of an IRRS mission .............................................................................................. 26

V.2 Possible changes in the IRRS Module structure ............................................................. 26

V.3 Possible changes in the IRRS Module review scope........................................................ 27

References .................................................................................................................................. 28

Appendix I: A typical IRRS preparatory meeting Agenda ............................................................... 29

Appendix II: Team assignment and grouping ................................................................................ 31

Appendix III: Standard Schedule of IRRS Missions ........................................................................ 33


FURTHER IMPROVING THE IRRS PROCESS

Discussions points for the Consultancy of Experienced IRRS Reviewers 20-22 April 2015

Introduction In order to further improve the IRRS process a series of Targeted Consultancies have been

conducted in 2012. For similar reasons of IRRS missions conducted between 2006 and 2013 have

been analysed. The IAEA Nuclear Safety Action Plan requires the secretariat to develop effectiveness

criteria of all peer reviews, including the IRRS. Partly to answer this requirement a set of

performance indicators have been introduced to measure the effectiveness and efficiency of a

mission. To summarize the lessons learned and set the development goals a Technical Meeting (TM)

was organized for team leaders and deputy team leaders of past IRRS missions in January 2013. The

documents summarizing the results of the above consultancies and the IRRS mission reports analysis

contain conclusions and proposals for improving further the IRRS process. Many other documents

related to the topics of the consultancy are available from the Share Point area1.

All these discussions, consultations and analyses have led to rapid recent developments of the IRRS

process on one hand and raised questions that need more thoughts for other developments on the

other hand. The Secretariat decided on inviting the most experienced IRRS reviewers for a

consultancy meant to discuss those questions, assist the Secretariat and to seek for further advice in

the development of the IRRS process and its continuous improvement. The present discussion paper

summarizes the topics suggested for discussions during the consultancy scheduled for 20-22 April

2015.

The issues to discuss are divided into five chapters (Preparatory phase of a mission, Conduct of a

mission, Post-mission phase, Follow-up mission and General development issues). Each chapter

contains as many sections as the number of issues belonging to the subject of the chapter.

For easier orientation among the various references, the Targeted Consultancy (TC)

recommendations as given in the consultancy reports are quoted; similarly the conclusions/

proposals of the IRRS analysis reports ‘Lessons Learned from IRRS Missions in 2006-2013 to

Countries with Operating NPPs’ [1} and ‘IRRS Missions 2006-2013: Analysis from a Radiation Safety

Perspective’ [2] are listed (if applicable). The reports above are available from the IAEA IRRS

website2. Whenever relevant, observations by the Technical Meeting of IRRS team leaders held in

2013 [5] are also quoted.

Each section first describes the actual status, then briefly quotes the related issues, recent

developments, and conclusions, proposals for further developments (if any), finally formulates the

questions to be discussed and answered by the consultancy.

The reader is encouraged to consult the underlying documents referred to footnotes 1 and 2 and

bring up further issues for discussion that are not covered by this report but are considered needing

discussion.

1 The Terms of Reference and the reports of the Targeted Consultancies, the Chairmen Summary of the TM of

IRRS team leaders, the report on the analysis, the report on the effectiveness and efficiency performance

indicators, the reports prepared to the IRRS Lessons Learned Ws in Moscow all are available from the Share

Point Area website:

http://gnssn.iaea.org/NSNI/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2FNSNI%2FShared%20

Documents%2FOPEN%20Shared%20Files%2FDocuments%20for%20the%20Consultancy%20of%20Experienced

%20IRRS%20Reviewers%2C%2020%2D22%20April%202015&InitialTabId=Ribbon%2EDocument&VisibilityCont

ext=WSSTabPersistence

2 http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx


I. Preparatory Phase of a Mission

I.1 Information meeting

Status

The IRRS Guidelines does not mention information meetings as part of the preparatory phase. In

practice information meetings are held at the request of the host country in order to obtain

information on the IRRS process in general and the actual mission in specific. Often the information

meeting is organized in conjunction with a national self-assessment seminar.

Issues, developments and proposals

The TCs have dealt with the information meeting and offered the following suggestions (the number

ahead of dot is the sequence number of the consultancy, the one after dot is the sequence number

of the suggestion in the given consultancy):

10.23: After the decision on an IRRS Mission the first step should be a one-day information meeting

between IAEA and the host country about 18-24 months prior to the mission.

10.25: IAEA should consider to make information meeting compulsory and to modify the agenda of that

meeting and of the Preparatory Meeting accordingly

Question(s) to the consultancy

Q-I.1: Should the information meeting be a standard part of the IRRS process? If so, what would be

its standard agenda/duration?

Q-I.2: Do you think that the information meeting may be performed via video-conference?

I.2 Preparatory meeting

Status

The preparatory meeting is defined in Section 7.2 of the IRRS Guidelines and further detailed in

Appendix VI of the Guidelines. The main characters are: it is to be conducted six to nine months prior

to the mission, lasts two or three days, is to be attended by the IRRS Team Leader, the IAEA

Coordinators and, as necessary, the Deputy Team Leader and other representatives of appropriate

IAEA Divisions. The main purpose is to inform the RB about all important details related to an IRRS

mission. A typical agenda of a preparatory meeting is given in Appendix I.


The TCs formulated the following suggestions related to the preparatory meetings:

1.08 and 10.28: In order to shorten the preparatory meetings the generic information about IRRS missions

to present by IAEA to the host should be provided in a separate event if needed and the meeting should

focus on high level issues.

1.09: The participants of the preparatory meetings from IAEA side should be limited to the persons given in

the IRRS Guidelines (Team Leader, Deputy Team Leader and Team Coordinator).

4.06: The preparatory meeting presentations be recorded and made available to the reviewers in advance

to the mission

10.29: At the preparatory meeting a thorough discussion should be held about the background of the

suggested Policy Issues.

Regardless of the guidelines and the TC proposals, in practice the preparatory meeting is also used as a review

of progress of the host state self-assessment (c.f. in Appendix I).


Q-1.3: What would be the optimum contents of the preparatory meeting agenda?


Q-1.4: What would be the optimum composition of the team participating in preparatory meetings?

Q-1.5: How would you suggest coordinating the information and preparatory meetings?

I.3 Advance Reference Material (ARM)

Status

The ARM is described in Section 7.5 of the IRRS Guidelines while Appendix V of the Guidelines lists

the typical documents to be included into the ARM. The ARM is supposed to include the results of

the self-assessment, a summary report on the self-assessment results and all documents that may

be relevant for the review process. The documents in the ARM are typically related to the national

legal framework, the regulatory framework, and reports describing details of the activity of the

government and the regulatory body related to the scope of the mission.


Previous consultancies and conducted IRRS missions raised the following issues:

a) Timely submission and quality of the ARM by the hosts

b) Preparation of a concise yet informative summary report of the ARM

c) Translation of parts of ARM to English

a) Time for ARM submission

The IRRS Guidelines requires the submission of the

ARM at least two months prior to the missions.

Recent practice is illustrated by Figure 1. It shows

the (time available for ARM review minus 45 days)

in the past missions. It is seen that in many missions

even 45 days were not available for the reviewers

and only in a few missions (values above the blue

line) was 60 days granted.

Figure 1: Time available for ARM review – 45 days in recent missions.

The TCs suggested:

2.05: Staff selected to participate in a mission should be allowed sufficient time for reading the ARM and

other necessary preparation for the mission

In report [1] the following related conclusion is reached:

C22: The ARM should be provided with sufficient time preceding the IRRS mission

b) ARM Summary Template

Since the result of the self-assessment is a very extensive document it is reasonable to request a

concise and informative summary of the ARM. The TCs offered a number of related suggestions:

1.03: The ARM should include summaries for every module highlighting the specific aspects of the

regulatory framework of the host country and give an overall perspective on the regulator. The host

country should prepare the ARM in the same format and template used by the IRRS team to prepare the

mission report

4.01: Summaries in the ARM for each module should be structured so that it helps the reviewer in

evaluating the compliance with the relevant GSR part 1 requirements specific to that module. IAEA should

develop a template for mapping to help the host country in the demonstration

7.06: The host country should provide the reviewers with a short summary for each module in the ARM (list

is given in the consultancy report)


10.16: IAEA should elaborate a template for the ARM, the host country should prepare the ARM using this

IRRS ARM template

The Technical Meeting of IRRS team leaders [5] also suggested the elaboration of such a template.

In report [1] it is concluded:

C24: The quality of ARM needs substantial improvement; for that purpose the hosts may need further

guidance

In report [2] it is proposed:

P22: Having read the Host Country’s ARM, including the self-assessment, IRRS Team members should pre-

draft their sections of the IRRS report, using an improved IRRS report template to facilitate the process and

ensure it is consistent…

P23: The IAEA should develop a template for the consistent and straightforward compilation of ARM,

possibly based on the SARIS self-assessment report and consistent with the IRRS mission report template

In reply to these suggestions the Agency has elaborated a Standard IRRS ARM Summary Template

(available from the Share Point area referred to in footnote 1).

c) Translation

Full translation of the documents included in the ARM is very seldom possible. An agreement needs

to be reached for the reasonable amount of translation and the completeness of information made

available to the reviewers. The related suggestions 5.01 and 6.12 by the TCs list those documents

that need to be translated. Furthermore, it is suggested that

6.02: There is a need for a quality check of English translations of documents submitted to the IRRS team


Q-1.6: What is the optimum timing of submission of ARM?

Q-1.7: Can a mission be postponed if the ARM is too late or of poor quality?

Q-1.8: Do you have any suggestion on modification of the Standard IRRS ARM Summary Template?

Q-1.9: What would you suggest as guiding principles for translating ARM documents to English?

Q-1.10: Is there any document presently not part of ARM that you feel necessary to include?

Q-1.11: How would you stimulate/invite Member States to ensure sufficient time of preparation for

their mission participants?

I.4 Team size, assignment, recruitment, experience and training

Status

The IRRS Guidelines describes the team size (5-20 international experts), composition (TL, DTL, IAEA

C. IAEA DC, IAEA Review Area Facilitator, review team members, Administrator) and the recruitment

in Section 7.6. No guidance is given as for the determination of the optimum size of a mission team

or on the workforce necessary for reviewing actual Modules. Neither are requirements on IRRS

specific training of reviewers. The Guidelines requires that “wherever possible the majority of the

team should have previous IRRS mission experience”. For follow-up missions “it is preferable to

include the reviewers who participated in the initial mission”.

Recruitment conditions of team members are described in full details in Section 7.6.2. of the

Guidelines.


a) Team size


It is observed that the size of the IRRS teams tend to increase as time passes. This is demonstrated in

Figure 2, where the team-sizes are shown for initial missions to countries with large, medium and

small nuclear programs, respectively. The most recent missions are on the left hand side and earlier

missions follow to the right. The linear trend-lines are fitted by Excel.

It should be noted, that the optimum size of a team depends not only on the nuclear programme of

the host country, but also on the scope of the mission. This dependence is correctly taken into

account by a method reported in Ref [3], the increasing tendency remains also after that.

Figure 2: Team-sizes in past initial IRRS missions

This increase should not necessarily be judged negative; it may as well be due to a more concise

conduct of the mission or due to the experience collected from past missions.

In this subject the TCs formulated the following suggestions:

1.07: The usual number of experts in Module 11 exceeds the associated risk

1.14: There should be more flexibility regarding the size of the team in order to allow the team leader to

add additional resources where needed

b) Team assignment

The TCs suggested a standard team assignment scheme, displayed in Appendix II, Table 1, which

serves as a guidance in recruitment of recent missions

A further advantage of a Standard Assignment Scheme is that the reviewers are expected to work in

groups in order to be as effective as possible. Footnotes below the table clarify details of the

assignments. The numbers of experts in the groups in Table 1 were elaborated based on past

experience and common sense and have been tested in recent missions. These numbers are

indicative and averages, they may be adjusted/regrouped according to the specific needs of the

missions.

Further suggestions by the TCs on composition of teams are as below:

3.03: Two reviewers should work together when interviewing and assessing results of the Module 4 review

5.11: Whenever possible a jointly co-ordinated approach should be adopted for assessment of IRRS

modules with potential overlap. In particular consideration should be given to having shared membership

for modules 5 & 6

6.09: It may be an advantage that experts familiar with more than one type of facility/activity are members

of the IRRS team

An alternative assignment table assuming four types of host countries is presented in Table 2 of

Appendix II.

c) Team recruitment

As the number of IRRS missions to be conducted increases from year to year, recruitment of suitable

team members becomes more and more difficult. To alleviate this difficulty the TCs and the analysis

reports proposed the establishment of expert pools. The respective TC suggestions are:


1.17: There should be a pool of designated experts that are competent and available for IRRS missions

1.18: The qualification and performance of pool experts should be tracked. Every country delegating an

expert to the pool should formally assure that the expert is capable and has the needed skills on report

drafting as well as the technical expertise. Another source of information on each expert’s performance

should be the survey feedback provided by the Team Leaders

Report [2] proposes:

P14: Countries are encouraged to form a pool of experts in the IRRS process and regularly update the IAEA

with their details. Countries should ensure their named experts periodically participate in IAEA reviewer

training and maintain a good understanding of the IAEA safety standards

Note that the Agency, based on an Agreement with the European Commission maintains a pool of

European IRRS experts and uses it for recruiting European reviewers.

d) Experience and training of reviewers

The left hand side of Figure 3 shows the ratio of experienced reviewers in past initial missions to

nuclear countries. It is seen that the 50% ratio required by the Guidelines is seldom exceeded. On

the right hand side of the Figure the same ratio is shown for the follow-up missions. Also in some of

these missions it was not possible to ensure participation of optimum number of experienced

reviewers, although they always were in majority.

Figure 3: Ratio of experienced reviewers in IRRS missions

In this respect the TCs and the reports suggested:

1.15: The number of experts in an IRRS mission that have no previous IRRS experience should be limited

1.16: Experts in an IRRS mission that have no previous IRRS experience should not work alone on a topic

C26: When recruiting the IRRS team, more attention should be paid to the inclusion of sufficient number of

experienced reviewers

In the evaluation of effectiveness and efficiency of IRRS missions, there is a Performance Indicator

related to the ratio of experienced reviewers. The optimum range of this Indicator is between 0.5

and 0.66 for initial missions and between 0.66 and 1.0 for follow-up missions.

Organization of a Basic IRRS Training Course was proposed by the TCs and was developed and held

first in 2013, then in 2014 with the participation of more than 120 experts. At the request of the

nuclear regulatory bodies on-site training courses were held in the UK and in the USA. Several

experts having obtained training on the courses have since been invited to missions.

Refresher training material has been developed and refresher training sessions were organized prior

to the initial team meetings in recent missions. In other missions, this material was posted on the

mission’s web site and reviewers were asked to make use of it to save time during the initial team

meetings.



Q-1.12: Do you have any suggestion in general on the size and composition of an IRRS team?

Q-1.13: Which of the Tables in Appendix II (if any) do you consider more suitable? Do you have any

different suggestion on the team assignment (including grouping) for various IRRS mission types?

Q-1.14: What is your suggestion concerning the optimum ratio of experienced reviewers in a team?

Q-1.15: Do you have any specific suggestion on the recruitment technique to be followed by the

Agency?

Q-1.16: Do you have any suggestion concerning training of IRRS reviewers?

I.5 Expected preparatory activity of the team

Status

The IRRS Guidelines requires that the team members provide a short written summary of their

observation based on the ARM (also called “initial impressions”) and present it at the initial team

meeting (Sections 7.7., 8.2., 8.4.). Separately from this, it is required (Sec. 7.5.) that the IRRS

reviewer provides written feedback on any significant issues identified at least two week prior to the

mission.

The activity of the team lead prior to the mission is also discussed by the Guidelines in Sections 7.2.

and 7.8., mentioning mainly the Team Leader and the IRRS Coordinator.


In recent missions it is a standard practice that the Team Leader and the IRRS Coordinator request

the reviewers to submit their initial impressions and list of issues at least two weeks prior to the

missions. It is also suggested that the reviewers record their questions stemming from the study of

ARM prior to the mission and hand them over to their counterparts at their first encounter during

the mission, thus making the discussions and interviews more effective. Note on the other hand,

that direct contact among reviewers and counterparts prior to the mission is to be avoided.

In report [2] the following is proposed:

P19: The IRRS Team’s initial impressions of the Host Country self-assessment and action plan should be

circulated to reviewers and the host immediately pre-mission and should form an annex to the IRRS report.

P22: Having read the Host Country’s ARM, including the self-assessment, IRRS Team members should pre-

draft their sections of the IRRS report, using an improved IRRS report template to facilitate the process and

ensure it is consistent. These ‘initial impressions’ should be shared with other Team members (and the

host) at least one week before the mission, accompanied by a list of any issues of concern, where specific

interviews and observations appear to be necessary. At the initial team meeting, immediately before the

IRRS mission begins, a collated first impressions report would effectively inform the team’s discussions


Q-1.17 What would you suggest to require from the reviewers as preparation for a mission?

Q-1.18: Do you have any suggestion on the activity of the team lead prior to a mission?

Q-1.19: What would be the most effective way of applying the initial impressions by the reviewers?


II. Conduct of a Mission

II.1 Schedule of a mission

Status

The IRRS Guidelines does not contain specific guidance on how to schedule the various activities in a

mission.


The TCs resulted in a Standard IRRS Mission Schedule as presented in Appendix III, which has been

tested and proven to be working well, with some flexibility, in recent missions. A striking feature of

this schedule is that the reviewers need to submit their preliminary findings by the end of the third

day. Some reviewers argue that the time available for interviews and/or external or internal

discussions is not sufficient; others are convinced that with effective preparations and use of time

two and a half days must be sufficient for the interviews and discussions. In addition, it must be

taken into account that some reviewers participate in site visits during this period of time.

Note also that if the deadline of submission of first findings is postponed, so must be the deadline

for the draft report text, which may also affect the programme of the week-end.


Q-2.1 Do you have any suggestion on modifying the Standard IRRS Mission Schedule? Please explain!

II.2 Observations

Status

The definitions of the observations (i.e. recommendation, suggestion and good practice), applied in

the present IRRS practice, are given in Section 9.1 of the Guidelines. The format of the observations

is not fixed by the Guidelines. The host country Action Plan is discussed at several places in the

Guidelines, yet its relation to the mission findings is not mentioned.


a) Formulation

Practical application of the definitions raises concerns from time to time. The TCs offered the

following suggestions:

1.31: R,S & GP should be adequately justified in the text and a general conclusion set out even when there

are no R,S or GPs. The text should be based on fact and evidence (not assumptions) and the conclusions

justified against IAEA’s Standards. The text should be succinct and self-explanatory – not an exhaustive

explanation but should include some examples.

1.32: The IAEA Guidance should pick out and present the text supporting R, S & GPS from some good

examples (from various modules) of previous reports (I.e. to illustrate style, layout and justification).

3.08: In the formulation of the mission findings the team leaders should consider the application of the

principle of graded approach

In order to partly respond to the suggestions above the tabular structure of observations (originally

containing the basis (or bases) and the recommendation, suggestion or good practice) was

supplemented by an entry entitled “Observation” meant to briefly summarize the issue that leads to

the given observation.

Report [2] recently raised a further possible modification:


P28: The context and reasons for each recommendation or suggestion should be clearly set out in the

findings (body) text and summarised in the ‘observations summary’ field. To facilitate this is it proposed

that the fields of the IRRS Report observations box be changed.

Currently, this box includes three fields: one for the ‘observation’, one for the basis (or bases) and one for

the recommendations or suggestions. It is proposed that the observations field be replaced with two fields;

one briefly describing the current practice and another describing the shortcoming relative to the relevant

IAEA requirement

The current format of the observation box is seen in Figure 4:

Figure 4: Format of the observations in the report

b) Good practice

Good practice is the most controversial observation of the IRRS. On one hand, the definition of a

good practice is very strict and should allow for only a few of them in several missions, on the other

hand, experience shows that the reviewers tend to use it for acknowledging good performance of

the host otherwise not at all unique or outstanding.

The TCs have already observed that

3.09: It is important to highlight in the mission report also good achievements which do not formally meet

the definition of a Good Practice.

5.02: Some criteria be introduced for appreciating strong features in regulatory performance (not GP, but

good)

Report [2] also emphasizes the controversy of the present use of good practices and suggests:

P29: IAEA should consider the following:

− Require that all four elements of the definition of a good practice be comprehensively addressed in

the IRRS report when describing a good practice. Unless all four criteria are clearly met, the concept of

good practice should not be applicable;

− Define and encourage the reporting of a second level of practice (e.g. ‘Notable Practice’) which

acknowledges actions and processes notable in the circumstances of the Host Country but which do

not meet IRRS Guidelines criteria for good practice.

P30: IAEA should consider that a good practice should be documented only where it has not been seen

before. This suggests the development of an IAEA searchable database of good practice, easily accessible to

IRRS reviewers and the global radiation safety community. MS should be encouraged to report to the

database if (and how) they adopt the good practices they find there

The TM of team leaders suggested [5]:


To ensure that category of good practice is appropriately used, another category of good performance

should be created

In reply to these suggestions the Agency developed the definition of a Notable Performance:

Where a commendable activity is observed which clearly strengthens regulatory effectiveness and/or

organisational efficiency (but, unlike a good practice, is not definably ‘superior to those generally

observed elsewhere’) it may be referred to as ‘notable performance (NP)’. NP will be worthy of the

attention of other regulatory bodies as a model in the general drive for excellence.

c) Observations and the Action Plan

According to the present practice the fact that an issue is recognised by the host country and is

inserted into the Action Plan (AP), may not preclude the reviewers from offering respective

recommendations or suggestions. In fact on the contrary, whenever a noncompliance is recognised

by the reviewer (notwithstanding if the AP called it to her/his attention) it should call forth the

formulation of a recommendation.

Certain reviewers do not agree with the present practice and argue that there is no added benefit in

recommending something to the hosts when it has already decided to take action. Although this

reasoning seems to be logical, the following arguments stand against it: if the presence of an issue in

the Action Plan precluded its appearance as a finding then

• the basic definition of recommendations (“they are proposed where aspects relative to the

IAEA Safety Requirements are missing, incomplete or inadequately implemented”) is not

followed;

• the follow-up mission will not be in the position to review the progress reached in eliminating

issues, as the relevant recommendations were not formulated in the initial mission;

• it would be sufficient for the host to have a general AP item like e.g. “The Inspection system

needs a thorough and complete revision” to preclude any IRRS mission finding on inspection.

It is important also to note here that even for a same issue, the Action Plan items and the

corresponding recommendation or suggestion are usually different in their details.

The related proposals in report [2] are as follows:

P20: To the extent possible, the IRRS report should clearly document the validation (or otherwise) of the

host’s self-assessment as a preface to the reviewer’s findings. Where the country has properly identified

the issues through self-assessment, this should be acknowledged. Where the IRRS team discovers issues

not addressed in the self-assessment, this should also be stated in the IRRS report, since such issues will

clearly be new to the host country

P25: To the extent possible, recommendations and suggestions should be correlated with the Host

Country’s existing action plan and refined, as appropriate to be realistically in accordance with the action

plan

d) Correlations and balance of observations

Each IRRS mission report is headed by the disclaimer: “The number of recommendations,

suggestions and good practices is in no way a measure of the status of the regulatory body.

Comparisons of such numbers between IRRS reports from different countries should not be

attempted”. This, however, does not mean that the ratios and correlations of these numbers may

have no meaningful information on the particular missions. This issue is discussed here.

In reports [1], [3] and [4] correlations among the number of various observations were examined

and the following rules were established:

• The number of Recommendations and the number of Suggestions are expected to exhibit

similar statistical behaviour, i.e. whenever the number of Recommendations is high (in


missions as well as for specific topics), so is the number of Suggestions and vice versa.

Statistical analysis has shown that indeed, they are strongly positively correlated both mission-

by-mission and over the various subjects. Figure 5 illustrates this for various subjects (so called

Subject Groups are formed within each Module and the number of findings belonging to those

Subject Groups appear in the Figure). Nevertheless, in certain missions the expected positive

correlation is less pronounced.

• The number of Recommendations and the number of Good Practices are expected to change

in opposite way by missions, i.e. in missions where the number of Recommendations is high

only few Good Practices are expected and the other way around. According to the results of

the analyses this negative correlation is present in most missions, yet there are a few

particular missions where such correlation does not exist.

• Similar correlation is expected between the number of Suggestions and the number of Good

Practices as in the case of Recommendations and Good Practices, yet in many missions this

negative correlation is missing (c.f. Figure 6).

The reports [1], [3], and [4] offer reasonable explanations for the lack of the expected correlations:

C17: The lack of distinct positive mission-wise correlations between the numbers of Suggestions and of

Recommendations may be attributed to the observations that in some missions Recommendations are

converted to Suggestions as results of compromises between the team and the hosts; whereas in other

cases (especially when there are a number of serious findings) the team tends to offer Recommendations

although a Suggestion might be more adequate

C18: - The numbers of Suggestions and Good Practices are positively correlated over IRRS Modules as well

as over Subject Groups since there possibly are topics that are particularly important in the regulatory

practice (or are consistently pointed out by the missions);

- another reason (more psychological than technical) may be that reviewers formulating a number

of findings unintentionally feel obliged to compensate by also offering Good Practices

Figure 5: Number of Recommendations and Suggestion on various subjects

Figure 6: Number of Suggestions and Good Practices in various missions

0

5

10

15

20

25

30

35

40

45

a b c d e f g x y a b x y a b c d e f g h x y a b c d e f x y a b c d e f x y a b c d e f g h i x y a b c d e f g h x y a b c d x y a b c d e x y a b c d e x y

Findings/Subject Groups in all missions to NPP countries in 2006-2013

Recommendations Suggestions

Mod.3 Mod.4Mod.1 Mod.7 Mod.8 Mod.9 Mod.102 Mod.5 Mod.6

0

5

10

15

20

25

30

35

O E S R J G T F H U P I N Q L V A M C D K B

Nu

mb

er

of

fin

din

gs

Mission

Correlation of Suggestions and Good Practices

S GP


Based on the expected and partly experienced correlations discussed above two Performance

Indicators characterizing the efficiency and/or effectiveness of a mission were proposed [3]:

(1) Balance of Recommendations and Suggestions (essentially expressed through the ratio of the

respective numbers)

(2) Balance of Good Practices and Recommendations (essentially expressed through the product of

the respective numbers)

Figure 7: Balance of findings in missions Figure 8: Balance of Recommendations and Good

Practices in missions

Figures 7 and 8 show the relative deviations of these balance values from their optimum value (1.0)

in the past missions. The ranges between the green lines show the cases where the deviation from

the optimum value is less than 25%. It is seen from the figures that in most missions the deviation

from optimum is reasonable, yet there are missions where very likely an in-depth analysis might

reveal the reasons for the differences.


Q-2.2: Would you recommend changes in definitions of the presently existing observations, or you

are satisfied with it as of now?

Q-2.3: Do you agree with the introduction of a new type of observation: Notable Performance? If so,

do you agree with the definition given here or you suggest changes in it?

Q-2.4: What is your view on handling issues present in the AP when formulating potential findings?

Q-2.5: Would you consider making use of the statistical (correlational) properties of the

observations? If so, how would you suggest applying them?

II.3 Review actions

Status

The IRRS Guidelines describe the mission activity of reviewers in Chapter 8. The Chapter includes the

use of various review methods (reading documents, interviews, observations, visits). There is no

specific guidance on how to distribute the tasks among reviewers working in groups, neither on how

to co-operate among groups.


a) Interviews

Interviews represent the most important way of collecting information, performing peer review. The

practical question arising relates to the number of persons sitting at the two sides of the table. Very

often at least two members of the team group listen to the counterpart(s). If the host


representatives are considerably less or considerably more than the team members the

effectiveness of the interview may be jeopardized.

b) Site visits

Site visit are usually very time consuming and reviewers participating in visits have considerably less

time for interviews and formulation of findings than other reviewers have. Furthermore, many of the

inspections held during visits are formal, are set-ups or are difficult to follow (usually for language

reasons).

c) Co-operation of reviewers

Table 3 shows the usual way of cooperation among the review groups. This Cross-Contribution Table

is part of the standard mission report template.

Gr. Topics to review, write and provide to Gr. Topics to obtain from Gr.

A

• Governmental responsibilities and

functions (mod 1)

• Global Nuclear Safety Regime (mod 2)

• Interface with security (mod 12)

-

-

-

• Module-wise legal

background

C,D,E,F

B

• Responsibilities of the RB (mod 3)

• Management system (mod 4)

• Development of regulations and guides –

in general (mod 9)

-

-

-

• Contents of regulations and

guides

C,D,E,F

C

• Authorization – in general and for NPPs

(mod 5)

• Review & assessment (R&A) – in general

and for NPPs (mod 6)

• Legal background of authorization and

R&A

• Contents of regulations and guides

related to authorization and R&A

-

-

A

B

• Authorization of facilities and

activities other than NPP

• R&A of facilities and activities

other than NPP

E

E

D

• Inspection – in general and for NPPs

(mod 7)

• Enforcement – in general and for NPPs

(mod 8)

• Legal background of inspection and

enforcement


related to inspection & enforcement

-

-

A

B

• Inspection of facilities and


• Enforcement of facilities and


E

E

E

For facilities and activities (other than NPP)

• Legal background

• Authorization and review & assessment

• Inspection and enforcement


A

C

D

B

F

• Emergency preparedness and response

(EPR) (mod 10)

• Legal background of EPR


related to EPR

-

A

B

G • Other optional thematic area -

Table 3: Standard IRRS Cross-Contribution Table


The first column shows the Group identifier (A through G). In the second column the activities

(review areas) of the groups are given, those areas which shall be contributions (inputs) to other

groups are shown in red. In the third column the addressee groups of the contributions (red items)

are given. The fourth column gives in blue those parts that are received by the current group from

other groups as contributions (inputs), while the last column contains the groups from where the

fourth column contributions are expected.


Q-2.6: Do you have any suggestion on how to make the interviews even more effective?

Q-2.7: What would you suggest for making visits even more efficient and effective?

Q-2.8: What would you suggest for further enhancement of an effective cooperation within the

review team?

Q-2.9: are you happy with using “groups’ or is it easier to just use names of reviewers

II.4 Report writing

Status

The IRRS Guidelines contains no specific requirements on the content of a mission report, yet

Appendix I of the Guidelines summarizes those topics that belong to the various Modules and thus

that need to appear in the final mission report. Nothing is said in the Guidelines on the timing, style

or method of writing the mission report neither on the expected size of a report.


a) IRRS Mission Report Template

The TCs realized the need for an extended table of contents, template and guidance for the

preparation of a mission report and elaborated such a template. It has been developed further,

based on the experience from its application. The actual form of the IRRS Report Template (version

6) is available from the Share Point area referred to in footnote 1.

Further suggestions on the template are given in report [2]:

P31: The IRRS report template should explicitly ensure that every element of each requirement is

addressed in full, even indicating clearly if elements are not applicable. For each element of each IAEA

requirement clear statements of compliance should be included for all regulated facilities and activities

encompassed by the national framework for safety. All parts of any requirement validated by the Team

should be listed or as a minimum, the number of each requirement and its sub-paragraphs should be listed,

clearly indicating compliance or otherwise.

In addition to the above, it is proposed that at the beginning of any review discussion, the applicable

elements of each requirement should be agreed and noted by reviewer and counterpart.

b) Details and length of a report

The needs for completeness and for readability of a report always present antagonism. Thus details

still to be included always compete with the wish to keep it as short as possible. The ‘Observation’

line in the observation box (see in Fig. 4) was introduced for providing a brief description of the issue

and possibly avoiding lengthy and repeated explanations in the main body of the report.

Report [2] proposes some kind of repetition, as already quoted in Section II.2:

P28: The context and reasons for each recommendation or suggestion should be clearly set out in the

findings (body) text and summarised in the ‘observations summary’ field. …

In report [4] analysis of the length of reports of past missions is presented. Based on past values and

on the assumption that past reports were overwritten by about 10%, optimum report lengths were


determined for each mission. This length depends on the scope of the mission, size of the host

country’s programme and the number of observations in the mission. Optimum lengths of the

chapters in the report are also defined and included into the mission report template. The lengths of

the reports of past missions relative to their optimum values are shown in Figure 9.

Figure 9: Mission report lengths relative to their optimum values


Q-2.10: What do you think necessary for developing further the IRRS Report Template?

Q-2.11: What would be the right way of describing the context and reasons for a finding?

Q-2.12: Do you agree that the lengths of mission reports should be kept at minimum?

Q-2.13: What would you suggest as guiding principles for preparation of mission reports?

II.5 Activity of team lead

Status

Responsibilities of the IRRS team participants are described in Appendix IV of the IRRS Guidelines.

Roles and responsibilities of the Team Leader and of the IAEA Coordinator are given in sufficient

details; those of their deputies are much less elaborated.


The TCs concluded in the following suggestions:

1.26: The team leader should not be assigned a function in reviewing a module

1.27: On missions to countries with large nuclear programs the DTL should not have a formal function in

reviewing a module. In smaller missions he can support sub team A

These suggestions are being considered in recent missions.

Report [2] proposes:

P15: The IAEA Deputy Coordinator’s role in explaining and ensuring correct use of IAEA requirements

should be reinforced, with training as necessary. The IRRS Guidelines should clearly assign to the Deputy

Coordinator, responsibility for alerting IRRS reviewers and counterparts where misunderstanding /

misinterpretation of a requirement is evident

P17: The IAEA Deputy Coordinator (DC) should be assigned responsibility for gathering feedback on the

application of IAEA safety standards and formally reporting it to IAEA. Special attention should be paid to

requirements that appear to be interpreted variably.

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

UK

FR

A

JPN

ME

X

SP

A

GF

R

UK

R

CA

N

RU

S

FR

A-f

u

UK

-fu

CP

R

US

UK

R-f

u

SP

A-f

u

GFR

-fu

CA

N-f

u

RO

M

RO

K

SLO

SW

I

SW

E

SLR FIN

BU

L

UK

-fu

2

RU

S-fu

CZ

R

BE

L

US

-fu

Nu

mb

er o

f p

age

s re

lati

ve

to

op

tim

um

Mission

Change of relative report length


P34: The Deputy Team Leader with the Deputy Coordinator should ensure that all observations relevant to

non-compliance with IAEA Safety Requirements lead to the appropriate recommendation and basis


Q-2.14: Do you have any suggestion on clarification/modification/extension of roles and

responsibilities of an IRRS Mission team lead?

II.6 Policy issues discussions

Status

Policy issues discussions are introduced and treated in Section 8.6 of the IRRS Guidelines. In

Appendix III a detailed list of the issues suggested for policy discussions and the key elements

therein are given.


Policy issues discussed in the IRRS missions in 2006-2013 were collected and analysed. Figure 10

shows the frequency of the various issues in these discussions. The subjects highlighted in the table

were novel suggestions by the hosts, the other subjects are from among those listed in Appendix III

of the Guidelines.

Figure 10: Frequency of policy issues discussed in missions in 2006-2013


Q-2.15: Do you deem it necessary to modify the concept and/or the practice of policy issues

discussions so far followed? If so, what would you suggest?

II.7 Feedback from the missions

Status

The IRRS Guidelines do not mention feedback on the missions. It mentions, on the other hand, at

several places that the mission may provide effective feedback for the improvement of IAEA

Standards and Guides.


The TCs suggested that

1.44: Continue the initiative to use surveys of IRRS Team members and the host countries to provide

feedback on the effectiveness of the IRRS mission

2.02: IAEA should develop a consistent approach to collect and evaluate feedback from IRRS host countries

and participants on Missions to learn lessons


2.34: The Consultants recommend IRRS participants give structured feedback on the conduct and

effectiveness of the mission. The Consultants endorse IAEA proposals for developing a process for

collecting and evaluating this information. This is to promote the continuing improvement to the IRRS

process

In report [2] P17 (as quoted in Section II.5) proposes that the DC should gather feedback on the

application of IAEA safety standards.

In report [3] four types of feedback questionnaires were suggested. These will be discussed below.

The questionnaires contain questions that are to be rated by marks between 1 and 5 (5 is the most

favourable), and at every question the answerers have the possibility of offering further comments

or suggestions.

Results of the evaluation of the questionnaires for recent missions are given in report [3]

a) Feedback on the ARM by the team

The team members are requested to evaluate the quality of the ARM by answering the questions:

Qr1: How complete do you consider the ARM?

Qr2: How realistic is the picture which you were able to obtain on the area you

will be reviewing from the ARM?

Qr3: What is your overall evaluation on the quality of the ARM?

b) Feedback on the mission by the team

At the end of a mission every member of the team is requested to answer a number of questions

characterising the effectiveness of the participants and of the mission.

c) Feedback on the mission by the host

Representatives of the host are requested to answer the questions below at the end of the mission:

Qh1: How effective do you consider the mission in assisting the continuous

improvement of nuclear safety in the host country?

Qh2: How objective was the peer review?

Qh3: How has the mission helped the exchange of information, experience and

good practice with other countries?

Qh4: How consistent was the use of the IAEA safety requirements and guides in the

mission?

Qh5: How justified are the findings of the peer review?

Qh6: How relevant are the findings of the peer review for the future development

of your regulatory body?

Qh7: How competent were the reviewers in their reviews and findings?

d) Feedback on the initial mission by the host before the follow-up

Representatives of the host are requested to answer two questions for every findings of the initial

mission prior to the beginning of the follow-up mission:

Qf1: How accurate was the particular recommendation/suggestion?

Qf2: How helpful was the given recommendation/suggestion for the improvement

of the regulatory body?

e) Feedback on the IAEA safety standards

No systematic collection of such feedback has so far been conducted.



Q-2.16: Do you suggest any modification in the question sets in use?

Q-2.17: What questions would you ask from the team to feedback on the mission?

Q-2.18: How would you suggest introducing practical feedback on IAEA safety standards?

II.8 Evaluation of team members’ performance

Status

The IRRS Guidelines does not address performance evaluation of reviewers.


Evaluation of the performance of reviewers is primarily important for ensuring the highest possible

quality and expertise of future IRRS teams. The sensitive nature of this type of evaluations is well

understood, therefore introduction of such procedure shall be thoroughly thought over and, if so

decided, the rules need to be meticulously set. It is assumed that the results of evaluation are kept

strictly confidential by the Agency and are not disclosed in any event.

The TM of team leaders [5] suggested:

Team Leader should give the IAEA an evaluation on the performance of individual Team Members at the

end of the mission


Q-2.19: Do you agree or disagree in principle with the evaluation of reviewers’ performance? Please

explain!

Q-2.20: If you agree in principle, what would be the necessary and sufficient conditions of such a

procedure to introduce?


III. Post-mission Phase

III.1 Finalization of the mission report

Status

The IRRS Guidelines sets the scene for the preliminary and final reports in Sections 9.5. and 9.6.

Accordingly the preliminary report (the version left back by the IRRS team) is commented by the

host, but the comments should be limited to issues relating to factual correctness of information

contained in the report. The comments are assessed by the IAEA C., the TL and, if necessary, by

other team members. The final report is to be released within two months following receipt of host

comments.


Report completion times of past missions are shown in Figure 11

Figure 11: Time (in days) necessary for releasing mission reports

It is seen that the 60 days deadlines was only a few times achieved and in several cases the release

of the reports took more than 120 days.

Report [1] concludes:

C31: The IAEA should investigate possible ways of speeding up report completion


Q-3.1: Do you suggest any modification in the process of finalizing mission reports?

Q-3.2: What would you suggest for decreasing the time needed for releasing the report?

III.2 Evaluation of missions

Status

The IRRS Guidelines does not refer to evaluation of IRRS missions.


Based on the Performance Indicators introduced in report [3] and on the feedback comments and

suggestions discussed in Section II.7 above, Prompt Evaluation Reports were prepared on missions

conducted to countries with operating NPPs. In terms of Performance Indicators the efficiency and

effectiveness of the missions are evaluated, discussed and conclusions are drawn. Examples of such

reports are available from the Share Point area referred to in footnote 1. Cover pages of the reports

prepared during the so called pilot-phase are shown in Figure 12.


Figure 12: Prompt Evaluation Reports of recent IRRS missions


Q-3.3: Do you consider important evaluating the IRRS missions?

Q-3.4: What would you suggest for improving further the evaluation methodology?

Q-3.5: Can you suggest additional items to be included in this evaluation?


IV. Follow-up missions

IV.1 Follow-up as a compulsory part

Status

The IRRS Guidelines does not explicitly state that a follow-up mission is a necessary consequence of

an initial IRRS mission. The implicit statement in the Introduction says: “An IRRS mission addresses

all relevant areas, facilities and activities regulated in the state, with a follow-up IRRS mission no

more than four years later to review progress…”.


Some countries that invited an initial IRRS missions failed to invite a follow-up within a reasonable

time-frame.

The TCs suggest:

4.22: Follow up mission should be made a compulsory part of the mission. The IAEA should not conduct a

mission unless there is a clear statement from the host country to host a follow up mission

A similar suggestion was formulated by the TM of team leaders [5]

Follow-up mission in a given time frame should be compulsory part of the IRRS process


Q-4.1: How should IRRS Guidelines be modified (if at all) to reflect that the follow-up mission be part

of the package when the initial request for an IRRS mission is made?

Q-4.2: What other means do you consider effective in stimulating host countries to invite follow-up

missions?

Q-4.3: Would you agree requesting the countries hosting initial and follow-up missions to submit a

final report on how all recommendations and suggestion of the missions have been implemented

(“closing the IRRS loop”)?

IV.2 Timing of follow-up

Status

The IRRS Guidelines mentions timing of a follow-up mission in the Introduction as quoted above (“no

more than four years later”), and in Section 10.2: “Typically a follow-up mission will take place two

to four years following the initial IRRS”


Some countries that invited an initial IRRS

missions failed to invite a follow-up within a

reasonable time, on the other hand, those

countries which have invited follow-up

missions mostly did it within the time-frame

foreseen by the Guidelines. This is show in

Figure 13.

Figure 13: Time elapsed beweet the initial

an follow-up missions

There is, however, another aspect that may influence the optimum timing of the follow-up missions;

this is the success in closing the issues raised by the initial mission. Figure 14 shows the ratios of

issues remained open in the follow-up as per the number of issues in the initial mission. The average

of this ratio is 0.23, i.e. on average 23% of the issues remain open. Accordingly, report [1] concludes:


C14: The relatively high number of issues that remained open suggests that either the time between the

initial and follow-up missions were sometimes not sufficient to reach compliance with the findings of the

initial missions, or the host country did not or could not place equal emphasis on all improvements

Figure 14: Ratio of open issues in the follow-up missions

There are views that express concern on the functioning of the follow-up system based on the

relatively low number of follow-up missions so far. Report [2] expresses this as

P2: In order to identify why IRRS Guidelines on follow-up missions are not being followed, the IAEA should

consider an anonymous survey of those Member States having hosted IRRS, but not yet requested a follow

up within the four years suggested in the Guidelines

A fast overview of the missions since 2006 shows, that there are 16 countries that had initial

missions but have not yet invited a follow-up. Two of them are nuclear countries two others are

having minor nuclear programmes and 11 are from the Africa region. The initial missions to these

latter countries had very limited scopes and therefore the potential follow-up missions to these

countries need specific considerations.


Q-4.4: Do you see the necessity of specific steps related to the timing of follow-up missions? If so,

what would you suggest?

Q-4.5: Do you consider it necessary to urge MSs to invite follow-up missions?

IV.3 Follow-up team composition

Status

This question is treated in Section 10.3.1. of the IRRS Guidelines as “it is preferable that the follow-

up mission includes the TL TC and reviewers who participated in the initial mission”.


In Section I.4.d) above the experience of the reviewers in past missions was discussed. Figure 3 there

shows that it was not always possible to recruit the same reviewers for a follow-up mission who

participated in the initial one. (In fact it shows that sometimes even it was not possible to recruit

experienced reviewers for certain positions of a follow up.)

The TCs have also recommended that

4.23: The reviewers of the follow-up mission be the same as in the initial mission as much as possible


Q-4.6: Do you see the necessity of specific rules or measures on recruiting a follow-up team? If so,

what would you suggest?


IV.4 Extended follow-up missions

Status

The IRRS Guidelines defines a follow-up extended if it also includes the review of specific topical

areas not covered in the initial mission. The Guidelines does not contain any preference or dis-

preference concerning extended follow-up missions.


The TCs considered this issue and suggested:

7.34: It was the consensus of the consultancy that full scope IRRS missions should be promoted and

practice of adding modules to the follow-up missions (modules which were not covered during the main

mission) should be avoided to the extent possible


Q-4.7: What is your suggestion concerning possible extension of future follow-up missions?


V. Development of the IRRS Process

V.1 Scope of an IRRS mission

Status

The IRRS Guidelines discusses the scope of a mission in Chapter 5 as:

“The minimum scope of an IRRS mission includes the core modules, i.e., Modules 1 to 10, together

with policy issues discussions. In order to be called an IRRS mission, the mission scope should always

cover the minimum scope.

When requesting an IRRS the state is strongly encouraged to include all regulated facilities and

activities in the mission. Without prejudice to the previous paragraph, if the state wishes to exclude

specific facilities and activities, a detailed explanation for the exclusion of these facilities or activities

should be provided. The mission report will reflect this in both the title and content of the relevant

report sections, including the Executive Summary”

“Additional technical issues may be included in the scope of the IRRS. These are incorporated in

Modules 11 (Additional Areas) and 12 (Interfaces with Nuclear Security).”

No definitions of a full-scope or a reduced-scope mission are given in the Guidelines.


According to some views the integrated character of IRRS must be reflected by covering the full

extent of the national legal framework and regulatory infrastructure for nuclear and radiation safety.

In this sense report [2] proposes:

P7: To be defined as an integrated mission, IRRS should review the full extent of the State’s national, legal

and governmental framework and regulatory infrastructure for safety, directly involving all organisations

that comprise the national regulatory body for nuclear and radiation safety and have oversight of all

radiation facilities and activities in the country. As per the IRRS Guidelines, the scope of the IRRS should be

clearly explained to the requesting Government and fully agreed at the IRRS preparatory meeting.

The IAEA should consider revising the modular structure of IRRS, by making the currently optional Module

11 a ‘core’ module or explicitly incorporating all elements of Module 11 into the core modules to ensure

the full extent of the Host’s national framework for safety is addressed

P8: Where the Host Country requests an integrated review as proposed above, then all agencies

collectively comprising the national regulatory body and thereby assigned responsibility and providing the

full scope of regulatory responsibilities and functions (as defined in GSR Part-1) should be subject to the

review, with their various identities and roles confirmed at the preparatory meeting

P9: When requesting a peer review which would not be integrated to the extent proposed above, the Host

Country should formally notify the IAEA of its specific requirements so that a more appropriate alternative

mission can be offered


Q-5.1: Do you consider that a peer-review mission could be called an IRRS mission only if it reviews

the full extent of legal, governmental, and regulatory framework related to nuclear and radiation

safety, or do you favour the present practice (full-scope and limited-scope IRRS missions)?

Q-5.2: Would you recommend defining specific IRRS mission types differing in their scope? If so, what

would you suggest?

V.2 Possible changes in the IRRS Module structure

Status

The IRRS module structure is described in Chapter 5, Section 5.1 and Appendix I of the Guidelines.



Most of the core Modules in the IRRS process cover

approximately equal areas of the national or

regulatory framework, some of them, however, are

substantially smaller. Figure 15 illustrates this

through the example of the number of observations

in the various Modules from all missions to nuclear

countries in 2006-2013.

It is apparent that Module 2 (Global Nuclear Safety

Regime) and Module 8 (Enforcement) represent

considerably less weight in the review than the other

core Modules

Figure 15: Total number of observations from missions to nuclear countries

This feature was also realized in report [1] that concluded:

C8: Recommendations and Suggestions, relating to most of the core regulatory functions from initial

missions demonstrate equal emphasis by the peer review, while Modules 2 and 8 have a considerable

lower rate of findings. The definition of the IRRS Modules may therefore need revision to be better

balanced out


Q-5.3: Do you consider it worthwhile changing the Module structure of the IRRS process? If so, what

would you suggest?

V.3 Possible changes in the IRRS Module review scope

Status

The scope of the IRRS review is defined by the IAEA safety standard Requirements and is described

in broad terms in Appendix I of the Guidelines.


Post-mission round-table discussions often point to certain shortcomings of the review scope. No

systematic investigation of this question has so far been conducted, yet a number of examples are

already worth mentioning:

• the review is often dominated by legal and regulation details while the regulatory practice

does not receive sufficient attention; • a specific review approach may be needed in countries where the regulatory body includes

several independent organizations; • special attention should be given to the top level of those regulatory bodies which are

headed by special commissions, committees or boards with a right of veto; • generally little attention is paid to the inspection techniques applied by the host regulator.


Q-5.4: What are the topics that you would suggest for discussion as potential modifications or new

elements in the review scope?


References

[1] Lessons Learned from IRRS Missions in 2006-2013 to Countries with Operating NPPs,

http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx

[2] IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective,


[3] Efficiency and Effectiveness of IRRS Missions,


[4] Analysis of the IRRS Missions in 2006-2013 to Countries with Operating NPPs


[5] Chairmen’s Summary Report on the Technical Meeting of the Integrated Regulatory Review

Service Team Leaders and Deputy Team Leaders for Sharing Experience and Improving the

Implementation of the Service, 28-31 January 2013, Vienna

Appendix I: A typical IRRS preparatory meeting Agenda

Tuesday, 27 May 2014

9:00 – 9:15 Opening Remarks and Introductions RB*, TL*, IAEA

9.15 – 9:30 Preparatory Meeting Objectives IAEA

9:30 – 10:30

General Presentation on legal and regulatory framework in the

host country

- Governmental and legal authority

- Regulatory body structure (organization, staffing,

resources, responsibilities, etc.)

- Overview of facilities and activities regulated

RB

10:30 – 11:00 Coffee Break

11:00 – 12:00 IRRS Process IAEA, TL, DTL*

12:00 – 13:15 Lunch

13:15 – 16:00

IRRS preparation: self-assessment, ARM, choice of interviews

and of activities to be observed

Presentation of RB’s self-assessment process

For each IRRS module:

- Presentation of self-assessment preliminary findings,

- Discussion regarding:

o ARM,

o choice of interviews or visits to be planned

during the IRRS mission.

RB

16:00 – 16:30 Break

16:30 – 18:15

IRRS mission implementation and logistics

a. finalize dates

b1. finalize scope (including discussion of policy issues)

b2. participant roles and responsibilities

c. schedule

1. team selection including access clearances

2. submittal of Self-assessment

3. submittal of reference material (discuss content)

4. submittal of policy issue write – ups

5. initial team meeting

d. communication prior to mission (identify liaison officer)

e. entrance meeting

f. communication during mission (identify counterparts)

g. mission team activities

1. document review

2. RB presentations

3. interviews

4. direct observation/site visits (facilities, organizations,

meetings, emergency exercise, other government officials, etc.)

5. daily team meetings

6. policy issue discussions

h. development of observations, conclusions,

recommendations, suggestions and good practices

i. draft mission report

j. exit meeting

k. communication following mission

RB, TL, DTL,

IAEA

18:15 Adjourn for the day


Wednesday, 28 May 2014

9:00 – 10:30

Logistical Preparations

a) security – clearances and badging

b) conference space and equipment

c) administrative services

d) lodging and transportation

e) lodging and travel to/from site visit(s)

f) emergency exercise/drills (date(s), duration, venue(s),

participants)

g) initial team meeting (time, location)

h) entrance meeting (time, location, participants)

i) exit meeting (time, location, participants)

j) media relations

k) cultural activities

10:30 – 10:45 Coffee break

10:45 – 12:30 Continue discussions

12:30 – 13:45 Lunch

13:45 – 14:30

Summary of Meeting

a. agreements

b. follow – up items

14:30 Closing remarks

15:00 Meeting adjourned

∗ RB – Regulatory Body

TL – IRRS Team Leader

DTL – IRRS Deputy Team Leader


Appendix II: Team assignment and grouping

Note

s Team activities

a) Small

b) Medium Large Follow-up

c)

1) 2) Team Lead (TL, DTL, TC, DTC) 3 3 3 3

2)

Group A

Governmental R&F (mod 1)

Global nuclear safety regime (mod 2)

Interface with security (mod 12)

3

2 3

2

3)

Group B

Responsibilities of the regulatory body (mod 3)

Management system (mod 4)

Development of regulations and guides (mod 9)

2 3

3)

4)

Group C

Authorization (mod 5)

Review and assessment (module 6) 2

2 2

2

3)

4)

Group D

Inspection (mod 7)

Enforcement (mod 8)

2 2

3)

5)

6)

Group E – Facilities/Activities

Nuclear power plants

Radiation sources applications

Research reactors

Fuel cycle facilities

Waste management facilities

Decommissioning

Transport of radioactive material

- 2 5

3

3) Group F

Emergency preparedness (mod 10)

2

2 2

6)

Group G – Other additional thematic areas (mod 11)

- control of medical exposures

- control of chronic exposures (radon, NORM

and past practices) and remediation

1 1

Administration 1 1 1 1

Total 11 17 22 11 a)

Numbers of experts include IAEA staff, b)

Definitions:

Small: No NPPs, few Facilities/Activities

Medium: Less than 5 NPP units (in construction, operating, to decommission), few Facilities/Activities

Large: 5 or more NPP units, and/or many Facilities/Activities c)

Numbers of experts of follow-up missions are defined here for large nuclear programmes, in other missions

should be adjusted according to the needs.

1) DTC is to be assigned to a Group

2) DTL supports Group A

3) Groups C through F review the respective contents of the regulations and guides (if in the scope) and

report to Group B

4) Groups C and D review the respective regulatory activities in general and the NPP-related activities in

specific

5) Group F reviews all core regulatory activities (authorization, review and assessment, inspection, enforce-

ment) for the specific facilities and activities in the scope of the mission and report to Groups C and D

6) For small nuclear programs the Facilities and Activities are to be reviewed by Groups C , D, F and G. Experts

allocated to Facilities/Activities (Group E) in medium and large missions may need to work together with

the experts in Group G. For large programs the number of reviewers may be less than 5 according to the

needs of the mission

Table 1: Assignment of the modules to groups and guidance on the number of experts in an IRRS

mission as suggested by the Targeted Consultancies


Gr Team activities Cntry

Cat I

Cntry

Cat II

Cntry

Cat III

Cntry

Cat IV

Follow-up

of Cat IV

Team Lead (TL, DTL, TC, DTC)

Team Leader 1 1 1 1 1

Deputy Team Leader 0 0 1 1 1

Team Coordinator 1 1 1 1 1

Deputy Team Coordinator 0 0 1 1 1

A Governmental R&F (mod 1) Global nuclear safety regime (mod 2) Interface with security (mod 12)

1 + TL 1 2 2 1

B Responsibilities of the regulatory body (mod 3) Management system (mod 4) Development of regulations and guides (mod 9a)

TL 1 + TL 2 2 1

C Authorization (mod 5) Review and assessment (mod 6) Review of content of relevant regulations (mod 9 b)

C1: Nuclear power plants 0 0 0 2 2 or 3

(profile of

reviewers

selected

based on

R/S)

C2: Radiation source applications 2 2 2 2

C3: Research reactors 0 0 1 1

C4: Fuel cycle facilities 0 0 1 1

C5: Waste management facilities 0 0 1 1

C6: Decommissioning 0 0 1 1

C7: Transport of radioactive Material = C2 1 1 1

D Inspection (mod 7) Enforcement (mod 8) Review of content of relevant regulations (mod 9 b)

D1: Nuclear power plants 0 0 2

= C1 to

C7

D2: Radiation source applications = C2 = C2 = C2 = C2

D3: Research reactors 0 = C3 = C3

D4: Fuel cycle facilities 0 = C4 = C4

D5: Waste management facilities 0 = C5 = C5

D6: Decommissioning 0 = C6 = C6

D7: Transport of radioactive material = C7 = C7 = C7 = C7

F Emergency preparedness (mod 10) 1 1 2 2 1

G Additional thematic areas (mod 11)

G1: Control of medical exposure 1 1 1 1 1

G2: Occupational radiation protection 1 1 1 1 = G1

G3: Control of chronic exposures (Radon, NORM and past practices) and remediation

= G2 = G2 1 1 = G1

G4: Control of rad. discharges and material for clearance 1 1 1 1 = G1

G5: Environmental monitoring associated with authorized practices for public rad. protection purposes

= G4 = G4 = G4 = G4 = G1

I Administration 1 1 1 1 1

Total 10 12 22 26 12

Categorization of countries

Country Cat I: Countries with very limited types and number of regulated radiation sources

facilities/activities

Country Cat II: Countries with no nuclear facilities/activities but with radiation sources facilities/activities

and with well-established regulatory oversight

Country Cat III: Countries with no NPPs, but with other nuclear facilities/activities in addition to radiation

sources facilities/activities with well-established regulatory oversight

Country Cat IV: Countries with NPPs, other nuclear facilities/activities in addition to radiation sources

facilities/activities well-established regulatory oversight,

Table 2: Alternative assignment of the modules to groups and guidance on the number of experts in

an IRRS mission

Appendix III: Standard Schedule of IRRS Missions Initial Mission First Week

Time SAT SUN MON TUE WED THU FRI SAT SUN

9:00-10:00 A

rriv

al o

f T

ea

m M

em

be

rs

Team building and

Initial Team

meeting:

• 5 minutes/TM

self-intro

• Refresher

training

Entrance

Meeting

Inte

rvie

ws

Vis

its

Inte

rvie

ws

Vis

its

Inte

rvie

ws

Vis

its/

EP

R e

xerc

.

DT

C w

rite

s

intr

od

uct

ory

pa

rts

TM write Report

TL and DTL review

introductory part

• Discussing and

improving Draft

Report

• Cross-Reading

• TL, DTL, TC and

DTC read

everything

Fre

e d

ay

, S

oci

al T

ou

r

Re

ad

ing,

Cro

ss-r

ead

ing

of

the

Re

po

rt

10:00-11:00

11:00-12:00 Draft text to TL

12:00-13:00

Lunch Lunch with

Host

Standing lunch Standing

lunch Standing lunch Standing lunch Standing lunch

13:00-14:00

Inte

rvie

ws

Vis

its

Inte

rvie

ws

+ in

-gro

up

dis

cuss

ion

s

Vis

its

Inte

rvie

ws

+ in

-gro

up

dis

cuss

ion

s

Vis

its/

EP

R E

xerc

ise

DT

C w

rite

s in

tro

du

cto

ry p

art

s Policy Discussions

Finalisation of the

Draft Report

14:00-15:00

Team building and

Initial Team

meeting

continued

• First

observations

• In-Group

discussions

Inte

rvie

ws 15:00-16:00

Secr

eta

riat

ed

its

the

rep

ort

Pre

lim

ina

ry D

raft

Re

po

rt

Re

ad

y

Cro

ss-r

ea

din

g b

y T

M

16:00-17:00

Written

preliminary

findings

delivered

17:00-18:00 Daily Team

Meeting

Daily Team

Meeting

Daily Team

Meeting:

Discussion of

findings

Daily Team

Meeting

Daily Team

Meeting

18:00-20:00 Informal

dinner Team Dinner Dinner Dinner Dinner Dinner Dinner Dinner

20:00-24:00 Writing of

the report

Writing of the

report

Secretariat

edits Report

TM write

Report

Writing of the

report TM Read Draft

Secretariat edits the

report


Initial Mission Second Week

MON TUE WED THU FRI

9:00-10:00

Individual discussions

of Rs, Ss and GPs with

counterparts

Cross-Reading

TL, DTL, TC and DTC read

everything

Finalisation

Common read through

and finalisation by the

Team Discussion with Host

Submission of the Final Draft 9:00-10:00

10:00-12:00 Exit Meeting

Press Conference 10:00-12:00

Submission of the Draft

to the Host

12:00-13:00 Standing lunch Standing lunch Lunch Standing Lunch Lunch 12:00-13:00

13:00-15:00 Policy Discussions

Discussion of the

report by the

team

TC

, D

TC

pre

pa

re E

xecu

tive

Sum

ma

ry a

nd

exi

t p

rese

nta

tio

n

Ho

st r

ead

s D

raft

TL

fin

alis

es

Exe

cuti

ve

Sum

ma

ry a

nd

exi

t

pre

sen

tati

on

TC

Dra

fts

the

Pre

ss

Re

lea

se

Written comments by the

Host

Team meeting for

finalisation of the Report

Departure Home

13:00-15:00

15:00-17:00

Individual discussions

of Rs, Ss and GPs with

counterparts

15:00-17:00

17:00-18:00 Daily Team Meeting Discussion of Executive

Summary

Briefing of the DDG

Finalisation of the press

release

17:00-18:00

18:00-20:00 Dinner Dinner Dinner

Farewell Dinner

18:00-20:00

20:00-21:00 Secretariat includes

changes Secretariat finalises text Free

20:00-21:00

21:00-24:00 Free 21:00-24:00


Follow-up Mission

Time SAT SUN MON TUE WED THU FRI SAT

9:00-10:00

Arr

iva

l of

Te

am

Me

mb

ers

Team building

meeting:

• 5 minutes/TM

self-intro

• Refresher

training

Entrance

Meeting

Interviews

TM write Report

TL and DTL review

introductory part

Finalisation

Discussion by the

Team

10:00-11:00

Interviews Discussion by the

Host

Exit Meeting

&

Press Conference

(alternative) 11:00-12:00 Draft text to TL Submission of the

Draft to the Host

12:00-13:00

Lunch

Standing

Lunch

Standing

lunch Standing lunch Standing lunch Standing Lunch

De

pa

rtu

res

of

Te

am

Me

mb

ers

13:00-14:00

Interviews

Interviews

Policy Discussions

Ho

st r

ead

s D

raft

TL

fin

alis

es

the

pre

sen

tati

on

TC

dra

fts

the

Pre

ss R

ele

ase

Written comments

by the Host

14:00-15:00 Initial Team

Meeting:

• IRRS process

• Main

objectives

• Report writing

• Schedule

• First

observations

Finalisation of the

Report 15:00-16:00

Secr

eta

riat

ed

its

the

rep

ort

Cro

ss-r

ea

din

g

16:00-17:00

Written

preliminary

findings

delivered

Preliminary Draft

Report Ready

Presenting the final

Draft of the Report

to the Host

17:00-18:00 Daily Team

Meeting

Daily Team

Meeting:

Discussion of

findings

Daily Team

Meeting

Discussion of

Executive

Summary

Exit Meeting

&

Press Conference

(optional) 18:00-19:00 Informal

dinner Team Dinner Dinner Dinner Dinner Dinner

19:00-20:00

Farewell Dinner 20:00-24:00

Writing of

the report

Secretariat

edits Report

TM write

Report

TM Read Draft Free