ame quality assurance performance - civilavmed.org · ame quality assurance performance stephen...
TRANSCRIPT
Federal AviationAdministrationCAMA Annual Meeting
AMEQuality Assurance
Performance
Stephen Véronneau, MD
Manager, Aerospace Medical Education Division
Regional Flight Surgeon
International, Military, and Federal Region
2Federal AviationAdministration
AME Performance Reporting
• Leah Olson, managing [email protected]
• Frances Parker, secretary, [email protected]
3Federal AviationAdministration
AME Performance Reporting
4Federal AviationAdministration
AME Performance Reporting
Objectives
• Policies/Orders• Why Monitoring is Needed• Methods of Surveillance• AME Performance Reports• Renewal, Termination, Appeal of Termination
5Federal AviationAdministration
AME Performance Reporting
6Federal AviationAdministration
AME Performance Reporting
Requirement by FAA Order
• VS 1100.2, Managing AVS DelegationPrograms
• FAA Order 8520.2G, Aviation MedicalExaminer Program, was subsumed into8000.95 CHG 2 in 2015
7Federal AviationAdministration
AME Performance Reporting
U.S. DEPARTMENT OF TRANSPORTATIONFEDERAL AVIATION ADMINISTRATION
AVS PolicyORDER
VS 1100.2
Managing Designees
•Selection
•Appointment
•Oversight
•Training
•Renewal
•Termination
•Appeal
8Federal AviationAdministration
AME Performance Reporting
Aerospace Medical EducationDivision Manager
AME Responsibilities:• Plan, develop, administer and evaluate
training programs• Monitor AME performance• Provide information to other Regional Flight
Surgeons
9Federal AviationAdministration
AME Performance Reporting
Why Performance Monitoring isNeeded
10Federal AviationAdministration
AME Performance Reporting
Why Monitoring is NeededThe Issued Exam QA Study
• All 323,304 records issued by AMEs in CY2005 met selection criteria
• A random sample of 2000 MID numberswere selected by stratified random sampling
• 2000 records, each one reviewed by 2independent reviewers
• Average review time 10 to 15 minutes/case• 4 months of review
11Federal AviationAdministration
AME Performance Reporting
Did AME issue appropriately?
• Population correlation ifrates are applied to the323,304 airman records sentto file– Issued appropriately 309,402– Issued inappropriately 8083– Incomplete 5819
95.7%
2.5%
1.80%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Total 2000 Exams
YesNoIncomplete
Did AME issued airman medicalcertificate appropriately in CY 2005?
Subtotal 1914 50 36
12Federal AviationAdministration
AME Performance Reporting
Aviation Medical Examiner QA Program
• Initiated July 2007, FAA Order VS 1100.2Managing AVS Delegation Programs
• Based upon procedures and softwaredeveloped during a 2006 QA Study
• Additional means of AME performanceoversight (Rv, Hx, C letters, SPA, site visit)
13Federal AviationAdministration
AME Performance Reporting
2008 Issued Reject Results
81.8%
18.2%
97.7%
2.3% 0.3%
99.7%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
commentappropriately
issueappropriately
possiblefalsification
yes no
3405
1763
44
18911527
14Federal AviationAdministration
AME Performance Reporting
AME Comment Appropriate block 60
15Federal AviationAdministration
AME Performance Reporting
AME Appropriate Issue
16Federal AviationAdministration
AME Performance Reporting
Most common causes for problems with issuedcertificates Jan. to June 2012 compared to prior period
Block Description OccurrencesJuly–Dec 2011
OccurrencesJan–Jun 2012
17 Unacceptable Medication 12 3
58 ECG 9
18h High or low Blood Pressure 9 4
18V DUI
18a Headache 1
18M Mental/Depression
18U Rheumatoid Arthritis 1
18 L Neurological disorder, Parkinsonism 1
18J Kidney stone
18u Admission to Hospital
18x Prostate cancer surgery,parathyroidectomy, cataract surgery
18D Retinopathy /Optic Neuritis 1
18X Sleep Apnea 1
52 Fail color vision 1
•Please note that there can more than one error in a single case
17Federal AviationAdministration
AME Performance Reporting
Most common causes for problems with issuedcertificates Jul to Dec 2014 compared to prior period
Block Description OccurrencesJuly-Dec 2014
Past OccurrencesJan-Jun 2014
17 Unacceptable Medication 16 4
58 ECG 0 0
18h High or Low Blood Pressure 40 75
18u Laryngeal cancer 1
18m Mental disorder 1 1
18f Asthma 4 5
18f Sarcoidosis 1
18u, x Rheum. Arthritis, Thyroiditis 1
18k Diabetes Mellitus
18d Vision trouble
19 Sleep apnea 1
18Federal AviationAdministration
AME Performance Reporting
Trends 2007 to 2014
19Federal AviationAdministration
AME Performance Reporting
AME Error Summary Jul to Dec 2014 compared toprior period Jan to June 2014
• AME error rate of the sample was 6.2% versus 9.1%
• Incomplete evaluation of applicants with HTN (CACI) is
the most common problem
• AMEs issuing certificates with inappropriate medications
• No falsifications 0 of 893; 0 of 898 records examined
• CACI related errors are 90% of total observed
20Federal AviationAdministration
AME Performance Reporting
21Federal AviationAdministration
AME Performance Reporting
FAA Aerospace MedicalCertification Services
Airman CustomerSatisfaction Survey
2012 and 2014
22Federal AviationAdministration
AME Performance Reporting
Survey AdministrationRandom sample of airmen who applied for pilotmedical certificates in 2012 and 2014
2012 2014Sampling frame 10,439 10,299Undeliverable 158 81
10281 10218
Returned and metcriteria for inclusion 4,256 3703
Percent returned andmet criteria for inclusion 41% 36%
23Federal AviationAdministration
AME Performance Reporting
Who reviewed your medical history withyou?
17%10%
85.40%
1% 4%5%
80%
18%
0%
25%
50%
75%
100%
AME Non-AMEPhysician
Non-Physician No Review
2012 2014
24Federal AviationAdministration
AME Performance Reporting
Who performed your physical exam?94.7 96.4
11.2 8.26.1 4.43.1 0.71.6 1.51 0.50
102030405060708090
100
2012 2014
AME
Nurse
PA
Non-AMEPhysicianOther
no one
25Federal AviationAdministration
AME Performance Reporting
Did your AME perform a thoroughexam?
95% 93%
5% 7%
0%10%20%30%40%50%60%70%80%90%
100%
2012 2014
YesNo
26Federal AviationAdministration
AME Performance Reporting
Did your AME
27Federal AviationAdministration
AME Performance Reporting
FAA Aerospace MedicalCertification Services
AME CustomerSatisfaction Survey
2012 and 2014
28Federal AviationAdministration
AME Performance Reporting
Regions Represented
0
5
10
15
20
25
30
AK CE EA GL NE NM SO SW WP IMF
RESPONDENTS (N=1739) ALL AMEs (N=3231)
29Federal AviationAdministration
AME Performance Reporting
Demographics• AME respondents: n=1,613• Average Yrs. worked: 19• Senior AMEs: 71%• AME-related practice: 5%• Applicants seen (median): 50
30Federal AviationAdministration
AME Performance Reporting
Overall Approval 2014
SERVICES %Deferrals Standards and Guidelines 72Aerospace Medical Certification Division 92Aerospace Medical Education Division 93Aerospace Medical Certification System 95Regional Flight Surgeon Office 92MedXPress 84Office of Aerospace Medicine Website 88Digital ECG System 80
31Federal AviationAdministration
AME Performance Reporting
Recommendations for Improvement
•AMEs suggested improvement for FAAservices by ranking their top 3
recommendations from multiple choiceoptions.
32Federal AviationAdministration
AME Performance Reporting
How Can FAA Improve Training?
58.0
45.0
22.0
0 10 20 30 40 50 60 70 80
CACI Theme seminar
More Regional Seminars
Live Video seminars at RFSOffice
33Federal AviationAdministration
AME Performance Reporting
How can FAA Improve AMCD? 2014
66.0
65.0
61.0
0 10 20 30 40 50 60 70 80
Attach supporting documents inAMCS
Attach supporting documents inMedXPress
Electronically check status ofdeferral
34Federal AviationAdministration
AME Performance Reporting
How can FAA Improve RFS? 2014
• Live video seminars at RFS office (21%)
35Federal AviationAdministration
AME Performance Reporting
63.0
61.0
58.0
0 10 20 30 40 50 60 70 80
Electronic notification of changesin deferral status
Feedback on inappropriatedeferrals
Online tracking of deferrals
How can FAA Improve Deferral Process? 2014
36Federal AviationAdministration
AME Performance Reporting
How can FAA improve OAM Website?2014
58.0
48.0
43.0
0 10 20 30 40 50 60 70 80
Easy access to CACIworksheets
Online spec sheets
Consistency between NavAidsand AME online publications
37Federal AviationAdministration
AME Performance Reporting
38Federal AviationAdministration
AME Performance Reporting
39Federal AviationAdministration
AME Performance Reporting
VIRTUAL SITE VISITS
40Federal AviationAdministration
AME Performance Reporting
SITE VISITS
41Federal AviationAdministration
AME Performance Reporting
Judgment Errors
42Federal AviationAdministration
AME Performance Reporting
Error LettersDefinitions
• Critical Error (“Rv” Error)– AME issued certificate when should have deferred– AME deferred when should have issued– Letter Sent to AME and RFS
• Less Critical Error (“Hx” Error)– Not enough information to support AME issuing– “Tic Mark”, Not Letter– Part of annual AME performance report
43Federal AviationAdministration
AME Performance Reporting
Error LettersDefinitions
• CACI Errors (“C” Error)– AME issues but deviates from the CACI worksheet– AME sent in CACI worksheet and supporting
information (Guide states do not send). Note:please remember that this only applies to the CACIcondition(s) there might be other information thatdoes need to be submitted for AASI
– AME deferred when should have issued usingCACI
– AME does not use the correct verbiage in Block 60
44Federal AviationAdministration
AME Performance Reporting
45Federal AviationAdministration
AME Performance Reporting
www.faa.gov/GO/AME/
46Federal AviationAdministration
AME Performance Reporting
http://www.faa.gov/other_visit/aviation_industry/designees_delegations/designee_types/ame/dms/
47Federal AviationAdministration
AME Performance Reporting
https://designee.faa.gov/
48Federal AviationAdministration
AME Performance Reporting
Individual AME Performance Summary Report
Report Date: 9/1/2016
THIS IS A REPORT LISTING YOUR PERFORMANCE FOR THE PERIOD 1/1/2015 THROUGH 9/1/2016. IF YOU HAVE QUESTIONSABOUT THIS REPORT, CONTACT YOUR REGIONAL FLIGHT SURGEON. MILITARY, FEDERAL, AND INTERNATIONALEXAMINERS SHOULD CONTACT THE MANAGER, AEROSPACE MEDICAL EDUCATION DIVISION, AAM-400, P.O. BOX 25082,OKLAHOMA CITY, OKLAHOMA, 73125.
Name and Address
REGION: INTERNATIONALREGION
STATE: UNKNOWN COUNTY: UNKNOWN
AME # : 00000xxxx STATUS: ACT SENIORAME:
Y APPOINTMENTDATE:
01/01/1994
1. INFORMATION REGARDING TRAINING AND DELINQUENCY:If "NEXT TRAINING DUE DATE" is the same as "NEXT SEMINAR DUE DATE" then there is no option fordistance learning. If these two dates differ, then there is a choice of MAMERC or SEMINAR training to becompleted prior to "NEXT TRAINING DUE DATE".
Next Training Due Date: 01/31/2019 Next Seminar Due Date: 01/31/2022 Years Delinquent:
Last Class TrainingDate:
01/31/2016 Last Self Training Date: 02/17/2014 (16.94)
2.TRANSMISSION DELAYS: NUMBER OF EXAMS FOR RANGES OF DELAY
Total ExamsTransmitted:
Under 15 Days 15-30 Days 31-60 Days 60+ Days %Over 60 Days
145 140 3 1 1 3.6
NOTE: Delayed exams may have been performed earlier than this report period, so total numbers in para4 may differ
3. EXAMINATION INFORMATION (BASED ON EXAMINATION TYPE REQUESTED BY THE AIRMAN)
1ST 2ND 3RD Total Exams Deferred Denied % Deferred/Denied
55 54 36 145 4 0 18.1
4. INFORMATION REGARDING EXAMS WITH 'Rv' AND 'Hx' ERRORSNOTE: An Rv error is assigned when an AME issued a certificate when it should have been deferred ordenied, requiring reversal by AMCD. An Hx error is assigned when the medical history was judged to beinadequate.
2.TRANSMISSION DELAYS: NUMBER OF EXAMS FOR RANGES OF DELAY
Total Exams Transmitted: Under 15 Days 15-30 Days 31-60 Days 60+ Days %Over 60 Days
145 140 3 1 1 3.6
NOTE: Delayed exams may have been performed earlier than this report period, so total numbers in para 4 maydiffer
3. EXAMINATION INFORMATION (BASED ON EXAMINATION TYPE REQUESTED BY THE AIRMAN)
1ST 2ND 3RD Total Exams Deferred Denied % Deferred/Denied
55 54 36 145 4 0 18.1
4. INFORMATION REGARDING EXAMS WITH 'Rv' AND 'Hx' ERRORSNOTE: An Rv error is assigned when an AME issued a certificate when it should have been deferred ordenied, requiring reversal by AMCD. An Hx error is assigned when the medical history was judged to beinadequate.
MID Applicant Name Exam Date Error Code Code Description
200007267448 name 04/12/2016 Other Other
200007118275 name 12/22/2015 Rv REVERSAL
200006969115 name 07/20/2015 Hx INADEQUATE HISTORY
200006809733 name 03/16/2015 Other Other
49Federal AviationAdministration
AME Performance Reporting
AME records must be retained by the Examiner as their file copy for atleast 3 years unless exceeded by state law requirements …. (scannedcopies are equivalent to file copies)
All completed applications and medical examinations, unless directed bythe FAA, must be transmitted electronically via AMCS within 14 days aftercompletion to the AMCD.
50Federal AviationAdministration
AME Performance Reporting
Student Pilot
• No longer issued by the AME as of April 1,2016
51Federal AviationAdministration
AME Performance Reporting
Student Pilot no longer an AME Activity
EXAM Certificate
52Federal AviationAdministration
AME Performance Reporting
Examination Submission DelaysConcerns
• No Record in System to Support IssuedCertificate– Hassle With Ramp Checks– Could Result in Assessment of Flying Without
License If Crash and Burn Accident
53Federal AviationAdministration
AME Performance Reporting
Submission Delay > 60 Days
0.2
0.5
0.4
0.2 0.2
0.5
0.1 0.1
0.2
0.10.1
0.1 0.1 0.1
0.1 0.1
0.4
0.10.1 0.1 0.1 0.1
0.1
0.10.1
0.10.10.1
0.0
0.1
0.2
0.3
0.4
0.5
0.63/
31/1
2
6/30
/12
9/30
/12
12/3
1/12
3/31
/13
6/30
/13
9/30
/13
12/3
1/13
3/31
/14
6/30
/14
9/30
/14
12/3
1/14
3/31
/15
6/30
/15
Per
cent
All Domestic Only
54Federal AviationAdministration
AME Performance Reporting
55Federal AviationAdministration
AME Performance Reporting
56Federal AviationAdministration
AME Performance Reporting
Training IS Required Every 3 Years-An AME must attend an AME Seminar, or equivalent training asdetermined by the AMED Manager, every 3 years, as a requirementfor continued designation.
- As an option, an AME may alternate Multimedia Aviation MedicalExaminer Refresher Course (MAMERC) in lieu of attending an FAAseminar, but under no circumstances should more than 6 years (72months) elapse between AME seminar attendance, or more than 3years (36 months) pass between seminar attendance and MAMERCcompletion.
Training Requirements
57Federal AviationAdministration
AME Performance Reporting
AME Performance ReportingDeferral Rates
4.7 4.54.84.84.54.44.54.44.44.24.14.14.03.63.63.93.83.63.8
4.85.0
4.7
0
1
2
3
4
5
6
3/31
/201
0
6/30
/201
0
9/30
/201
0
12/3
1/20
10
3/31
/201
1
6/30
/201
1
9/30
/201
1
12/3
1/20
11
3/31
/201
2
6/30
/201
2
9/30
/201
2
12/3
1/20
12
3/31
/201
3
6/30
/201
3
9/30
/201
3
12/3
1/20
13
3/31
/201
4
6/30
/201
4
9/30
/201
4
12/3
1/20
14
3/31
/201
5
6/30
/201
5
Per
cent
58Federal AviationAdministration
AME Performance Reporting
Renewal of Designation
59Federal AviationAdministration
AME Performance Reporting
Designee Management PolicyOrder 8000.95 CHG 2
Published 6 March 2016– Duration of designation 3 years– 10 exams per year to maintain proficiency– NO exams on immediate family– No appeal for loss of credential, failure to train or
lack of FAA need for your designation– All FAA exams must be conducted in English– AMCS changes
• Timeout• Confirmation of staff q 90 days
60Federal AviationAdministration
AME Performance Reporting
Termination of Designation
61Federal AviationAdministration
AME Performance Reporting
Overview of Designee Lifecycle• Suspension and Termination
– Input: Trigger for suspension and termination are identified inFAA Order 8000.95
– Suspension for not meeting current requirements• The designee is afforded an opportunity to correct the issue within a
specific time frame• If not corrected, the designee may be terminated
– Termination For Cause:• “Bad” termination• Can be appealed by designee, follows Appeal process• Designee cannot re-apply
– Termination Not For Cause:• “Not bad” termination• Cannot be appealed• Designee CAN re-apply
– Output: Suspended or Terminated designee
61
62Federal AviationAdministration
AME Performance Reporting
Reasons for termination• No exams in first 12 months• Less than 10 exams per year• Not following FAA rules• Poor reporting of medical examinations• Failure to train• Practice in unapproved location• Unprofessional office• Unprofessional performance of exams• Failure to mail in exams
63Federal AviationAdministration
AME Performance Reporting
Reasons for termination• Loss, restriction, or limitation of a license to
practice medicine• Any action that compromises public trust or
interferes with the AME’s ability to carry outdesignated responsibilities
• Arrest, indictment or conviction• Any other reason the Federal Air Surgeon or
RFS deems appropriate• Exams not conducted in English
64Federal AviationAdministration
AME Performance Reporting
Risk
Severity of an outcome x likelihood ofoccurrence
65Federal AviationAdministration
AME Performance Reporting
66Federal AviationAdministration
AME Performance Reporting
Voluntary Separations2012 2013 2014
Death 9 8 16
Resigned 105 95 86
Retired 38 48 65
67Federal AviationAdministration
AME Performance Reporting
Voluntary Separations2013 2014 2015
Death 8 16 1
Resigned 95 86 25
Retired 48 65 12
68Federal AviationAdministration
AME Performance Reporting
Involuntary Separations
2012 2013 2014
Failure to Train 75 112 136Low Volume 72 25 32Medical Incapacitation 0 2 6Performance 4 2 6Unable to Locate 9 13 19Unapproved OfficeLocation
4 8 2
Transmission Delays 3 0 0Failed Site Visit 1 1 1
69Federal AviationAdministration
AME Performance Reporting
Involuntary Separations
2013 2014 2015
Failure to Train 112 136 57Low Volume 25 32 2Medical Incapacitation 2 6 1Performance 2 6 1Unable to Locate 13 19 1Unapproved OfficeLocation
8 2 1
Transmission Delays 0 0 0Failed Site Visit 1 1 2
70Federal AviationAdministration
AME Performance Reporting
Appeal of Termination
71Federal AviationAdministration
AME Performance Reporting
U.S. DEPARTMENT OF TRANSPORTATIONFEDERAL AVIATION ADMINISTRATION
AVS PolicyORDER
VS 1100.2
Provides for an appeal panel ofthree physicians with a decisionreturned within 45 days.
72Federal AviationAdministration
AME Performance Reporting
Can Not Appeal . . .
• Lack of need• Separation for failure to train
73Federal AviationAdministration
AME Performance Reporting
New Designee Management System
• November 2015 AAM went into DMS• Electronic
– Application– Renewal– Record Keeping
74Federal AviationAdministration
AME Performance Reporting
New Designee Management System
• Electronic– Application– Renewal– Record Keeping– Interaction with Managing Specialist (Regional staff)– AME performance report
75Federal AviationAdministration
AME Performance Reporting
New in DMS for AMEs• AMEs will be able to view their information
on training and performance• AMEs will be able to submit updates to their
information through DMS• AMEs will receive and be able to respond to
corrective actions through DMS whereneeded
• AMEs will request renewal of theirdesignation through DMS
76Federal AviationAdministration
AME Performance Reporting
•https://www.faa.gov/tv/?mediaId=1180
77Federal AviationAdministration
AME Performance Reporting
Designee Management System
•https://designee.faa.gov
78Federal AviationAdministration
AME Performance Reporting
Designee Registration System
79Federal AviationAdministration
AME Performance Reporting
New in DMS for AMEs• AMEs will be able to view their information
on training and performance• AMEs will be able to submit updates to their
information through DMS• AMEs will receive and be able to respond to
corrective actions through DMS whereneeded
• AMEs will request renewal of theirdesignation through DMS
80Federal AviationAdministration
AME Performance Reporting