the canadian usoap experience. rdims/sgddi 2 the canadian usoap experience aim: to share canadian...
TRANSCRIPT
![Page 1: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/1.jpg)
The Canadian USOAP Experience
![Page 2: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/2.jpg)
RDIMS/SGDDI
2
The Canadian USOAP Experience
Aim: to share Canadian experience gained from the ICAO audit process to provide other states with an insight that may be helpful in preparing for upcoming audits
Outline• Overview of the ICAO audit process• Canadian preparation • Canada’s Audit experience• Lessons Learned• Pitfalls to avoid
![Page 3: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/3.jpg)
RDIMS/SGDDI
3
Overview of the ICAO audit process
Program Objective (Assembly Resolution A35-6)
Assess State’s implementation of:- Safety oversight, and- Standards and Recommended Practices, associated
procedures and guidance material
![Page 4: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/4.jpg)
RDIMS/SGDDI
4
ICAO auditing Philosophy cont’d
ICAO’s Comprehensive Systems approach:
Phase 1. Review of State Aviation Activity Question and Compliance Checklists to confirm Annex implementation and capability for Safety Oversight
Phase 2. On-site audit to confirm information and assessment of State’s overall capability for Safety Oversight
![Page 5: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/5.jpg)
RDIMS/SGDDI
5
Foundation Documents
o
• Rules of Conduct-Memorandum of Understanding
• Audit documentation•Pre-audit tools
- State Aviation Activity Questionnaire- Compliance Checklist
•Protocol Checklists - On site interview questions
![Page 6: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/6.jpg)
RDIMS/SGDDI
6
Memorandum of Understanding
• Defines obligations for the audit
• Sent to State with notification of ICAO’s scheduled intention- 9 months in advance of audit date
• States are expected to sign and return 90 days before the audit
Foundation Documents cont’d
![Page 7: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/7.jpg)
RDIMS/SGDDI
7
Foundation Documents cont’d
MOU Highlights• Scope to address all annexes except 9 and 17• ICAO team members will have command of English and at least one member will have command of the language chosen by the State• Audit will be conducted in accordance with “Safety Oversight Audit Manual Doc 9735)• ICAO to conduct the audit in accordance within a specified timeline
![Page 8: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/8.jpg)
RDIMS/SGDDI
8
Foundation Documents cont’d
State agrees:- to provide necessary access to staff, documents, state facilities;- to provide transportation within the state;- to provide required translation services
and office facilities;- to submit updated version of SAAQ and CC at least 90 days before the audit- to establish a national coordinator; and- to assist with accommodation arrangements.
![Page 9: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/9.jpg)
RDIMS/SGDDI
9
Documentation toolsPreaudit
•State Aviation Activity Questionnaire
•Compliance Checklists
- Forms are available on ICAO web site
Foundation Documents cont’d
![Page 10: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/10.jpg)
RDIMS/SGDDI
10
State Aviation Activity Questionnaire
Advance information required by ICAO for scheduling and workload planning
Provides overview information on State Activity organized into seven parts:
•General Administrative•Legislative•Organization•Operational Activities•Air Navigation Activities•Aerodromes•Accident and Incident Investigation
Foundation Documents cont’d
![Page 11: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/11.jpg)
RDIMS/SGDDI
11
Compliance Checklists
• Advance information required by ICAO for assessing compliance with SARPs
• Presents State with tool to identify differences to SARPs
• Update required 90 days in advance of an audit
![Page 12: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/12.jpg)
RDIMS/SGDDI
12
Protocol ChecklistsFoundation Documents cont’d
A tool for conducting the on site audit or internal self assessment used to ensure:
•Transparency•Consistency•Standardization
Organized to conduct critical element assessment in eight modules:
• Legislation• State Organization• Personnel Licensing• Operation of Aircraft• Airworthiness• Accident Investigation• Air Navigation• Aerodromes
![Page 13: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/13.jpg)
RDIMS/SGDDI
13
Guidance SourcesICAO intranet website provides access to:
• State Aviation Activity Questionnaire• Compliance Checklists• Protocol Checklists• ICAO Annexes and Documents• Completed audits on member states
ICAO Coordinators course• Also listed on the web
ICAO Team leader for the State audit• Makes a State/ICAO team for meeting both goals
![Page 14: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/14.jpg)
RDIMS/SGDDI
14
Audit timeline - On site activityEntry meeting
• Opportunity for introductions; • ICAO provides explanation of process; and• The State to provide a description of their organization.
Conduct of interviews• Start with group interviews for Legislation and Organization;• Importance of time management - Audit plans are on strict schedule - State coordinators to ensure Protocol questions are organized, assigned and prepared for the interviews - ICAO team continues their work after State hours
- A complex audit will take up to two weeks - Industry visits require State observer
![Page 15: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/15.jpg)
RDIMS/SGDDI
15
Audit timeline - On site activity
Closing meeting• meeting is for ICAO team to present their preliminary findings
• no discussion of these findings is intended
• beginning of timeline for acceptance of findings and the final report.
![Page 16: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/16.jpg)
RDIMS/SGDDI
16
Audit Timeline - Post AuditC
losi
n g m
e eti
ng
ICA
O p
rov i
d es
int e
r im
re p
ort t
o S
tate
90 days
60 days
Sta
te c
lari
fica
tion
s to
IC
AO
30 days
ICA
O c
lari
fies
sta
te r
e spo
nse
30 days
ICA
O tr
ansl
ates
and
pos
ts o
n W
eb
![Page 17: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/17.jpg)
RDIMS/SGDDI
17
Canadian Preparation for the Audit
-Coordination-Communications
-Arrangements for daily debriefs-Training for State interviewees
![Page 18: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/18.jpg)
RDIMS/SGDDI
18
Coordination
• three areas to be involved in coordination – Civil Aviation, Transportation Safety Board and Department of National Defense
• Started work on advance arrangements for:» Accommodations and office facilities» Security clearances» Transportation arrangements for
industry and regional visits
Application of ICAO process in The Pre-Audit phase Cont’d
![Page 19: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/19.jpg)
RDIMS/SGDDI
19
Need for Internal Coordination System
- State organizational structure does not parallel audit protocol organization or ICAO annexes- need to arrange and schedule appropriate interview candidates - need to support ICAO auditor during the audit
Scope for Canadian auditing initiatives contin’d
![Page 20: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/20.jpg)
RDIMS/SGDDI
20
Communications
- Need for close coordination between State Coordinator and ICAO Team
- Resulted in effective coordination for planning and
resolution of Protocol question issues
- Introduction of Daily Audit debrief requirement- Daily feedback for Management and Interviewees- Triggers for providing follow-up clarification- ICAO tool for managing “areas of concern”
Application of ICAO process in
The Pre-Audit phase Cont’d
![Page 21: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/21.jpg)
RDIMS/SGDDI
21
Training for the State’s interviewees
- Improved understanding of their role and the audit process;
- Guidance on role reversal from auditor to interviewee; and
- Guidance on what are suitable questions.
Scope for Canadian auditing initiatives contin’d
![Page 22: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/22.jpg)
RDIMS/SGDDI
22
Preparation by the Interviewee
• Protocol questions for specialty areas are available on ICAO’s web site;
• Critical elements numbers on the protocol questions confused the interviewees during their preparation for the audit
![Page 23: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/23.jpg)
RDIMS/SGDDI
23
Observations on the Canadian Audit Experience
- description of ICAO’s Audit Plan;
- observations on the interviews;
- observations on the audit process; and
- a description of the Findings
![Page 24: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/24.jpg)
RDIMS/SGDDI
24
Page from ICAO Work Planfor OPS Specialist
![Page 25: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/25.jpg)
RDIMS/SGDDI
25
A Description of ICAO’s Audit Plan
![Page 26: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/26.jpg)
RDIMS/SGDDI
26
Observations on the Interviews– ICAO’s lack of organizational understanding
lead to misdirection of questions and inappropriate responses
– Interviewees were frustrated by auditors changing the order of the Protocol questions as well as by questioning from shaded areas
- T.C. frustration over amount of inputs that were not addressed
– No use of non-international inventories– Did not address Annex 10
The Interview Process Contin’d
![Page 27: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/27.jpg)
RDIMS/SGDDI
27
Daily Debriefs– Began on the end of second full day of interviews
• Attended by State coordinator and ICAO Team Lead
– Records provided to coordinators and ICAO• Interviewees verified proper understanding and necessity
to provide additional information• ICAO used to manage their follow-up issues and confirm
accuracy of record
– During the Daily Debriefs, ICAO used the terminology “Area of concern” and at the closing meeting the term was changed to “Preliminary Draft Finding”
![Page 28: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/28.jpg)
RDIMS/SGDDI
28
The closing meeting- Canada had all audit participants attend to ensure full
understanding of results and to undertake immediate attention to corrective action.
- Each ICAO specialist summarized the preliminary findings for their area
- Canada responded with commitment to follow through on Findings.
- ICAO ended the meeting with the request to complete a State Assessment form and the commitment to continue using the “Daily Debrief” as part of their audit process
![Page 29: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/29.jpg)
RDIMS/SGDDI
29
Closing meeting Contin’d
Was the audit a success?
• From the state perspective;– Were the logistics met?
• From the State and ICAO perspective;– Was a proper understanding of the state
program developed to permit a satisfactory “systems” assessment?
![Page 30: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/30.jpg)
RDIMS/SGDDI
30
Findings
• ICAO auditors admitted that they had to dig deep to find non-compliance
• There were “areas of concern” we disagreed with and upon clarification were able to resolve
• Others included: – international conditions on licenses– Cabin crew flight and duty times– PPCs every two years– Notify ICAO of exemptions
![Page 31: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/31.jpg)
RDIMS/SGDDI
31
Lessons Learned1. Being a “Beta” audit did not impact the final results
1. A shortage of time did not prevent preparation for the audit
2. Staff wanted more information on ICAO and its requirements. Feedback was for more information on ICAO role, terminology and a simplified version of the audit process
3. Need for state to adopt more proactive role in managing interviews
1. Ensure assignment of Interviewers to appropriate expertise in the organization
2. Promote requirement for advance preparation of questions
![Page 32: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/32.jpg)
RDIMS/SGDDI
32
Lessons Learned Contin’d
Lessons Learned (continued)
4. State response to “areas of concern” required specific focus
– Too much material prevented clear understanding of the state position
5. Communication plan was essential– Resulted in team building and knowledgeable inputs– Daily debrief supported clear understanding by all
participants– Prevented development of inappropriate “areas of concern”
![Page 33: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/33.jpg)
RDIMS/SGDDI
33
Pitfalls in Advance of an Audit
1. Not using all the resources available to understand and prepare for an audit.
- ICAO web, ICAO team leader, State auditing expertise
2. Allowing communication barriers
- Ensure staff have clear roles that are understood in the program
- Test the process for understanding and effectiveness
![Page 34: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/34.jpg)
RDIMS/SGDDI
34
3. Unprepared interviewees• Provide training on process and expected attitude
– Interviewees to understand the limit of their responsibility to answer questions
– Don’t assume interviewees have knowledge of the process
• Prepare in advance, formal responses to the protocol questions and be prepared with supporting reference documentation
• Address apparent regulatory or organizational protocol issues with the time remaining before the actual audit
Pitfalls in Advance of an Audit Contin’d
![Page 35: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/35.jpg)
RDIMS/SGDDI
35
Potential Pitfalls During an Audit
1. Misunderstandings
– ICAO understanding of state organization impacts their protocol management and accuracy of concerns
– State members must understand the limits of their interview responsibilities
– Purpose of industry visits
![Page 36: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/36.jpg)
RDIMS/SGDDI
36
Potential Pitfalls During an Audit Contin’d
2. Giving too much information. 2. Know when to stop and how to focus the material to the key
point that will be understood
3. Not using the “Daily Debrief” system, 2. the feedback pulse for the audit
4. Accepting General Findings based on limited interviews
![Page 37: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/37.jpg)
RDIMS/SGDDI
37
Summary
1. Strong coordinator essential
2. Send coordinator on ICAO training
3. Select knowledgeable Branch coordinators
4. Short and direct answers to questions
5. Establish good communications with ICAO team leader
6. Prepare interviewees including industry
![Page 38: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/38.jpg)
RDIMS/SGDDI
38
Summary (continued)
7. Insist on daily debriefs
8. Escort auditors everywhere
9. Plan to have staff available at all times during the audit including weekends and late at night
10. Work with Auditors to ensure “Areas of Concern” accurately reflect program
![Page 39: The Canadian USOAP Experience. RDIMS/SGDDI 2 The Canadian USOAP Experience Aim: to share Canadian experience gained from the ICAO audit process to provide](https://reader036.vdocument.in/reader036/viewer/2022081418/56649f445503460f94c6512c/html5/thumbnails/39.jpg)
RDIMS/SGDDI
39
Questions?