time to competency, reliability of flexible transnasal ... · pdf filetime to competency,...
TRANSCRIPT
Original Research—General Otolaryngology
Time to Competency, Reliability ofFlexible Transnasal Laryngoscopy byTraining Level: A Pilot Study
Otolaryngology–Head and Neck Surgery2015, Vol. 152(5) 843–850� American Academy ofOtolaryngology—Head and NeckSurgery Foundation 2015Reprints and permission:sagepub.com/journalsPermissions.navDOI: 10.1177/0194599815572792http://otojournal.org
Christopher D. Brook, MD1, Michael P. Platt, MD, MS1,Kimberly Russell, MD1, Gregory A. Grillone, MD1,Avner Aliphas, MD1, and J. Pieter Noordzij, MD1
Sponsorships or competing interests that may be relevant to content are dis-
closed at the end of this article.
Abstract
Objective. To determine the progression of flexible transnasallaryngoscopy reliability and competency in otolaryngologyresidency training.
Study Design. Prospective case control study.
Setting. Academic otolaryngology department.
Subjects. Medical students, otolaryngology residents, andotolaryngology attending physicians.
Methods. Fourteen otolaryngology residents from PGY-1 toPGY-5 and 3 attending otolaryngologists viewed 25 selected anddigitally recorded flexible transnasal laryngoscopies. The evalua-tors were asked to rate 13 items relating to abnormalities inthe oropharynx, hypopharynx, larynx, and subglottis. The levelof concern and level of comfort with the diagnosis wereassessed. Intraclass correlations were calculated for each topicand by level of training to determine reliability within each classand compare competency versus attending interpretations.
Results. Intraclass correlation of residents compared to attend-ing physicians demonstrated significant improvements by yearfor left and right vocal fold immobility, subglottic stenosis, lar-yngeal mass, left and right vocal cord abnormalities, and levelof concern. Additionally, pooled vocal cord mobility andpooled results in categories with good attending reliabilitydemonstrated stepwise improvement as well. For these cate-gories, resident reliability was found to be statistically similarto attending physicians in all categories by PGY-3. There wereno trends for base of tongue abnormalities, pharyngealabnormalities, and pharyngeal and hypopharyngeal masses.
Conclusions. Resident competency for flexible transnasal lar-yngoscopy progresses during residency to reliability withattending otolaryngologists by the PGY-3 year over keyfacets of the examination.
Keywords
reliability, flexible laryngoscopy, resident training, competency
Received October 20, 2014; revised December 16, 2014; accepted
January 23, 2015.
Introduction
Flexible transnasal laryngoscopy is a commonly used diag-
nostic procedure for evaluation of the upper aerodigestive
tract. Within the inpatient setting, residents are often the first
clinicians to evaluate the airway of patients presenting with
acute complaints. Findings on these procedures can range
from normal to minor abnormalities to critical diagnoses that
necessitate immediate intervention. There is currently no vali-
dated method for determining resident competency for flex-
ible transnasal laryngoscopy. The progression from novice to
competent for each residency training program is a variable
process that each residency program determines internally.
The Next Accredidation System (NAS) implemented by the
Accreditation Council on Graduate Medical Education
(ACGME) uses competency-based milestones as evaluation
tools for all residents.1
Progress in surgical education and mechanical skill can
be difficult to measure due to the complexity of surgical
procedures, need for supervision and attending participation,
and need to provide for patient safety. Previous studies of
otolaryngology residents have used objective assessment
tools modeled after the Objective Structured Assessment of
Technical Skills (OSATS) to study development and assess-
ment of resident technical skills when performing operating
room procedures such as mastoidectomy, pediatric laryngo-
scopy and rigid bronchoscopy, and endoscopic sinus sur-
gery.2-6 The purpose of this study was to determine the time
to competency for otolaryngology residents performing flex-
ible transnasal laryngoscopy, the variability among residents
at different stages of training and attending physicians, and
1Boston University School of Medicine, Boston, Massachusetts, USA
Corresponding Author:
Michael P. Platt, MD, MS, Boston University School of Medicine, 830
Harrison Avenue, Boston, MA 02118, USA.
Email: [email protected]
at SOCIEDADE BRASILEIRA DE CIRUR on May 11, 2015oto.sagepub.comDownloaded from
the ability to implement a competency-based tool for flex-
ible transnasal laryngoscopy evaluation.
Methods
This study was approved by the Institutional Review Board at
our institution (Boston University Medical Center study refer-
ence number H-31313). Twenty-one videos were created from
archived endoscopy videos taken during routine clinical eva-
luations in the otolaryngology outpatient clinic between 2006
and 2013. All endoscopies were performed on Olympus
Rhino-Laryngo Digital Laryngoscopes (type V2; Olympus,
Center Valley Pennsylvania, USA) and accessed from digital
archives. All endoscopies used in the study were of adult
patients. Videos were specifically selected from the archives
by the authors for presence of certain pathology or findings.
The videos were de-identified and edited to representative
examinations of 20 to 50 seconds by the senior author (J.P.N.),
a fellowship-trained laryngologist. None of the authors who
participated in the selection or editing process reviewed the
examinations as subjects, and none of the endoscopies passed
beyond the glottis. The purpose of editing was to eliminate
nondiagnostic portions of the examination at the beginning and
the end of the videos that could bias the results. The videos
were contiguous examinations except for 1 video that had a
long, redundant segment removed. Videos were labeled and
randomly sorted in a folder with 4 videos repeated to deter-
mine intrarater reliability. A total of 25 videos were adminis-
tered to all subjects. All videos were administered without
sound to eliminate bias by recognition of the examiner’s voice
(eg, a specific otolaryngologist who specializes in cancer).
Participants for the study were chosen to provide representa-
tive levels of training. The videos were administered at the start
of the academic year, and the videos were shown to 3 residents
from each training level from new PGY-1s through PGY-5s (5
classes of residents), with the exception of only 2 residents from
the PGY-4 year due to an author conflict. Two medical students
participated as well and 3 attending otolaryngologists. Of the
attending otolaryngologists, 1 was a fellowship-trained rhinolo-
gist, and 2 were general otolaryngologists. A standardized ques-
tionnaire was given to each participant for each video that they
viewed, totaling 25 questionnaires for each participant (see
Figure 1). The information from the questionnaires was then
recorded in a spreadsheet.
Data analysis was performed using Microsoft Excel with
the Real-Statistics add-in.7 Interrater reliability within each
training level was assessed with intraclass correlation coeffi-
cients (ICCs) between observers of the same training level
(ie, PGY-1, PGY-2, etc) for each question asked on the
questionnaire. Interrater reliability between each class of
training was then assessed with intraclass correlation coeffi-
cients between each level of training and a ‘‘gold standard’’
of attending ratings. Additionally, interrater ICCs within
each class and against an attending gold standard were cal-
culated for pooled vocal cord mobility (left and right vocal
cord mobility) and pooled scores for all topics that
approached good reliability for attendings. These topics
included subglottic stenosis, left true vocal cord immobility,
right true vocal cord immobility, left and right vocal cord
abnormalities, laryngeal mass, and level of concern.
Attending ICCs for these topics ranged from 0.58 to 0.92.
Pharyngeal wall abnormalities None Mild Moderate Severe
Base of tongue abnormalities None Mild Moderate Severe
Arytenoid abnormalities None Mild Moderate Severe
Left vocal fold abnormalities None Mild Moderate Severe
Right vocal fold abnormalities None Mild Moderate Severe
Right Vocal Fold immobility None Mild Moderate Severe/Fixed
Left Vocal Fold immobility None Mild Moderate Severe/Fixed
Subglottic edema/stenosis None Mild Moderate Severe
Present AbsentLaryngeal mass/tumorOropharyngeal massHypopharyngeal mass
Please circle the choice that best fits the question:
1. Level of concern:a. No intervention neededb. Medical treatment onlyc. Biopsy warrantedd. Immediate airway concern
2. Level of comfort with my diagnosis:a. Completely sure of diagnosisb. Very sure of diagnosisc. Somewhat sure of diagnosisd. Unsure of diagnosis
Exam number _______
Rater PGY level or training level _______
Fiberoptic Laryngoscopy Evaluation Sheet:
Figure 1. Video questionnaire. A copy of this questionnaire was provided to each participant in the study for each of the 25 videos thatthey viewed.
844 Otolaryngology–Head and Neck Surgery 152(5)
at SOCIEDADE BRASILEIRA DE CIRUR on May 11, 2015oto.sagepub.comDownloaded from
Results
Intraclass correlation coefficients were generated to com-
pare the degree of agreement within each class of training
(PGY-1 through outgoing PGY-5, medical students, and
attending otolaryngologists) as well as the degree of agree-
ment between each class of trainee against the attending
otolaryngologist gold standard. For interpretation, we con-
cluded that 1.0 to 0.8 was very good agreement, 0.79 to 0.6
was good agreement, 0.59 to 0.4 was moderate agreement,
0.39 to 0.2 was fair agreement, and 0.19 to 0 was poor
agreement.
The intraclass correlations for interrater reliability within
each level of training demonstrated large variations in relia-
bility (see Table 1). There were several categories with
notably higher intraclass correlation coefficients when com-
paring participants, including subglottic stenosis, left true
vocal cord immobility, right true vocal cord immobility, left
and right vocal cord abnormalities, laryngeal mass, and
level of concern. All of these topics approached good relia-
bility within the attending class (ICCs, 0.58-0.92). Pooled
analysis of vocal cord mobility revealed a strong stepwise
progression of reliability from MS4 through attending level,
with a clear statistical difference between PGY-2 and PGY-
4 ICCs (Table 1). PGY-3 residents demonstrated nonsigni-
ficant improvement over PGY-2 level ICC but were also
not significantly different from PGY-4 ICCs. Pooled analy-
sis of all topics that approached good reliability for attend-
ings was then performed (ICCs, 0.58-0.92), including
subglottic stenosis, left true vocal cord immobility, true
vocal cord immobility, left and right vocal cord abnormal-
ities, laryngeal mass, and level of concern. Again, there was
a statistical difference between PGY-2 and lower levels of
training when compared with PGY-4, PGY-5, and attending
ICCs (Figure 1).
When comparing interrater reliability with intraclass cor-
relations between different levels of training and the attend-
ing otolaryngology gold standard, again the authors found
that there were large variations in reliability but that the
same categories demonstrated higher reliability. These
included subglottic stenosis, left true vocal cord immobility,
right true vocal cord immobility, left and right vocal cord
abnormalities, laryngeal mass, and level of concern (see
Table 2). Again the data demonstrated significantly higher
ICCs for PGY-4 and PGY-5 level residents compared with
attending than with MS4, PGY-1, and PGY-2 residents
compared with attendings. PGY-3 level residents were sta-
tistically no different than PGY-2 or PGY-4 residents, fall-
ing in between the early levels of training and the later
years (Figure 2). Pooled analysis of all topics that
approached good reliability for attendings was then per-
formed again for trainee versus attending ICCs, including
subglottic stenosis, left true vocal cord immobility, true
vocal cord immobility, left and right vocal cord abnormal-
ities, laryngeal mass, and level of concern. As with interra-
ter ICCs within class, there was a statistical difference
between PGY-2 and lower levels of training versus
attending when compared with PGY-4 and PGY-5 trainees
versus attending ICCs (Figure 3).
Finally, looking at intraclass correlation coefficients on
the repeated flexible transnasal laryngoscopy videos, the
authors found that all raters demonstrated good to very
good agreement (ICCs, 0.73-0.90) on repeated examinations
without significant difference between training levels.
Discussion
Standardization of surgical training and improving educa-
tional outcomes is becoming an ever larger focus of resi-
dency programs as the ACGME places emphasis on the 6
core competencies and standardization of medical educa-
tion through the New Accreditation System unrolled in
July 2013.1 The ACGME has stressed the use of mile-
stones in medical education to measure the progress of
residents through the 6 competencies, medical knowledge,
patient care, professionalism, interpersonal and communi-
cation skills, practice-based learning and improvement,
and systems-based learning.1,8-10 With the increasing
importance of these measurements, objective measure-
ments of resident competency and educational advance-
ment have become increasingly employed in surgical and
otolaryngological literature.2-6
To date, there are no objective measurements of resident
competency in flexible transnasal laryngoscopy, despite this
being one of the most commonly performed resident proce-
dures, especially during the junior resident years.11 The
authors’ data demonstrate that there is a wide variation in
agreement of interpretation of flexible transnasal laryngo-
scopy findings; however, there are certain aspects of the
examination that demonstrate higher levels of agreement
between trainees and attending otolaryngologists. These spe-
cific aspects of the examination are right and left vocal cord
abnormalities, left and right true vocal fold mobility, sub-
glottic stenosis, presence of a laryngeal mass, and level of
concern of airway patency.
When examining the data of intraclass correlation coeffi-
cients within each individual class and for each class com-
pared to the attending gold standard, it became apparent
that there was a significant divide between MS4, PGY-1,
and PGY-2 level residents and the senior reviewers, PGY-4,
PGY-5, and attendings. The data demonstrated a clear step-
wise increase in reliability between these 2 groups for rat-
ings within each class and for each class rated against the
attending gold standard. The analysis also demonstrated that
the PGY-3 level residents were somewhere in between, not
statistically different from the lower level reviewers or the
higher level reviewers.
As these data were collected at the start of the academic
year, the authors concluded that the major improvements in
reliability came during the PGY-2 year, which is intuitive as
the PGY-1 residents have limited otolaryngology experi-
ence. During the PGY-1 year at our institution, the residents
spend 3 months on the otolaryngology service, but the
PGY-2 residents spend the entire year in the specialty,
Brook et al 845
at SOCIEDADE BRASILEIRA DE CIRUR on May 11, 2015oto.sagepub.comDownloaded from
Tab
le1.
ICC
sC
alcu
late
dw
ithin
Eac
hC
lass
(MS4
thro
ugh
Att
endin
g),w
ith
95%
Confid
ence
Inte
rval
s.
MS4
For
PG
Y-1
For
PG
Y-2
For
PG
Y-3
For
PG
Y-4
For
PG
Y-5
For
Att
endin
gsO
vera
llIC
C
Phar
ynab
n0.5
1(0
.15-0
.75)a
0.6
(0.3
8-0
.78)b
0.1
2(–
0.1
to0.4
)0.3
6(0
.11-0
.61)
0.1
8(–
0.2
2to
0.5
3)
0.6
6(0
.46-0
.82)b
0.4
(0.1
4-0
.64)a
0.4
(0.2
8-0
.57)a
BO
Tab
n–0.1
4(–
0.5
3to
0.2
6)
0.3
7(0
.12-0
.62)
0.2
2(–
0.0
2to
0.4
2)
0.1
4(–
0.0
8to
0.4
2)
0.0
6(–
0.1
5to
0.3
4)
0.1
5(–
0.0
8to
0.4
3)
0.0
6(–
0.3
5to
0.4
4)
0.2
2(0
.13-0
.37)
Ary
ten
abn
0.2
1(–
0.1
9to
0.5
5)
0.3
9(0
.14-0
.63)
0.5
7(0
.34-0
.76)a
0.5
5(0
.32-0
.75)a
0.1
2(–
0.2
8to
0.4
8)
0.5
7(0
.34-0
.76)a
0.1
(0.1
2-0
.38)
0.3
6(0
.25-0
.54)
Left
TV
Cab
n0.6
(0.3
8-0
.78)b
0.6
7(0
.47-0
.82)b
0.2
7(0
.03-0
.54)
0.4
3(0
.18-0
.66)
0.6
4(0
.43-0
.81)b
0.7
4(0
.57-0
.86)b
0.8
2(0
.69-0
.91)c
0.5
5(0
.42-0
.71)
Rig
ht
TV
Cab
n0.1
(–0.3
2to
0.4
8)
0.4
1(0
.16-0
.65)a
0.2
6(0
.02-0
.53)
0.3
7(0
.12-0
.62)
0.5
1(0
.14-0
.75)a
0.7
2(0
.53-0
.85)b
0.5
8(0
.35-0
.77)a
0.4
1(0
.28-0
.58)a
Left
TV
Cim
mob
0.2
(–0.0
2-0
.55)
0.4
2(0
.17-0
.66)a
0.6
4(0
.17-0
.81)b
0.8
2(0
.69-0
.91)c
0.8
7(0
.77-0
.94)c
0.8
2(0
.69-0
.91)c
0.8
8(0
.74-0
.98)c
0.6
7(0
.55-0
.8)b
Rig
ht
TV
Cim
mob
0.4
(0.1
5-0
.64)
0.4
9(0
.25-0
.71)a
0.5
8(0
.35-0
.77)a
0.6
6(0
.46-0
.82)b
0.9
4(0
.87-0
.97)c
0.8
3(0
.7-0
.91)c
0.9
(0.8
2-0
.95)c
0.6
(0.4
7-0
.75)b
Poole
dT
VC
imm
ob
0.2
8(0
.01-0
.52)
0.4
5(0
.28-0
.61)a
0.6
3(0
.48-0
.75)b
0.7
5(0
.58-0
.86)b
0.9
(0.8
2-0
.94)c
0.8
3(0
.74-0
.89)c
0.8
9(0
.82-0
.93)c
0.6
4(0
.55-0
.74)b
SGS
–0.2
1(–
0.3
4to
0)
0.1
6(–
0.0
7to
0.4
3)
0.5
5(0
.32-0
.75)a
0.8
5(0
.73-0
.93)c
0.9
8(0
.93-0
.99)c
0.9
3(0
.87-0
.97)c
0.9
2(0
.85-0
.96)c
0.5
7(0
.44-0
.72)a
Lary
nx
mas
s0.6
6(0
.36-0
.84)b
0.3
1(0
.06-0
.57)
0.1
7(–
0.6
to0.4
5)
0.4
8(0
.24-0
.7)a
0.3
7(0
.12-0
.62)
0.7
5(0
.57-0
.86)b
0.6
(0.3
8-0
.78)b
0.4
4(0
.32-0
.62)a
Phar
ynx
mas
s0.1
2(0
.1-0
.4)
0.3
3(0
.08-0
.59)
–0.0
3(–
0.2
1to
0.2
4)
0.0
3(–
0.1
7to
0.3
)0.2
8(0
.04-0
.54)
0.1
9(0
.04-0
.46)
0.2
3(–
0.1
to0.5
)0.2
3(0
.14-0
.38)
Hyp
ophar
ynx
mas
s–0.1
1(–
0.2
7to
0.1
4)
0.2
2(–
0.0
2to
0.4
9)
–0.0
1(–
0.2
to0.2
6)
–0.0
2(–
0.2
1to
0.2
5)
0(–
0.4
1to
0.4
)0.2
3(–
0.1
to0.5
)0.1
2(–
0.1
to0.4
)0.0
5(0
.01-0
.11)
Leve
lofco
nce
rn0.2
7(–
0.1
5to
0.6
)0.2
5(0
.01-0
.52)
0.3
7(0
.12-0
.62)
0.5
7(0
.34-0
.76)a
0.5
(0.1
3-0
.75)a
0.5
7(0
.34-0
.76)a
0.8
3(0
.7-0
.91)c
0.4
8(0
.33-0
.63)a
Leve
lofco
mfo
rt0.0
9(–
0.3
3to
0.4
7)
0.1
(–0.1
2to
0.3
8)
0.1
8(0
.05-0
.46
0.2
9(0
.05-0
.55)
0.1
7(–
0.2
5to
0.5
3)
0.2
6(0
.02-0
.53)
0.2
9(0
.05-0
.55)
0.1
(0.0
5-0
.21)
Poole
d(S
GS,
TV
C
abn,T
VC
imm
ob,
lary
nx,le
velof
conce
rn)
0.6
08
(0.5
06-0
.694)b
0.6
41
(0.5
68-0
.708)b
0.6
75
(0.6
06-0
.737)b
0.7
6(0
.697-0
.813)b
0.8
26
(0.7
68-0
.870)c
0.8
61
(0.8
26-0
.891)c
0.8
93
(0.8
65-0
.917)c
0.7
38
(0.6
96-0
.78)b
Abbre
viat
ions:
abn,ab
norm
al;A
ryte
n,ar
yten
oid
;B
OT,
bas
eofto
ngu
e;IC
Cs,
intr
acla
ssco
rrel
atio
ns;
imm
ob,im
mobili
ty;Phar
yn,phar
ynx;SG
S,su
bgl
ott
icst
enosi
s;T
VC
,tr
ue
voca
lco
rd.
a Moder
ate
relia
bili
ty(I
CC
,0.4
-0.6
).bG
ood
relia
bilit
y(I
CC
,0.6
-0.8
).c V
ery
good
relia
bili
ty(I
CC
,0.8
-1.0
).
846 at SOCIEDADE BRASILEIRA DE CIRUR on May 11, 2015oto.sagepub.comDownloaded from
Tab
le2.
ICC
sofEac
hTr
ainin
gC
lass
Com
par
edto
Att
endin
gs,w
ith
95%
Confid
ence
Inte
rval
s.
MS4
vsA
tten
din
gFo
rPG
Y-1
vsA
tten
din
gFo
rPG
Y-2
vsA
tten
din
gFo
rPG
Y-3
vsA
tten
din
gFo
rPG
Y-4
vsA
tten
din
gFo
rPG
Y-5
vsA
tten
din
g
Phar
ynab
n0.3
7(0
.19-0
.58)
0.4
7(0
.3-0
.66)a
0.3
(0.1
5-0
.51)
0.2
8(0
.14-0
.5)
0.3
9(0
.21-0
.6)
0.5
(0.3
3-0
.69)a
BO
Tab
n0.0
3(–
0.0
7to
0.2
3)
0.2
5(0
.11-0
.46)
0.1
8(0
.05-0
.38)
0.1
7(0
.04-0
.37)
0.1
4(0
-0.3
5)
0.2
2(0
.08-0
.42)
Ary
ten
abn
0.0
7(–
0.0
4to
0.2
8)
0.1
6(0
.04-0
.36)
0.3
(0.1
5-0
.51)
0.2
5(0
.11-0
.46)
0.1
(–0.0
1to
0.3
5)
0.2
5(0
.11-0
.46)
Left
TV
Cab
n0.5
9(0
.42-0
.76)a
0.6
8(0
.53-0
.82)b
0.5
6(0
.39-0
.73)a
0.6
4(0
.48-0
.79)b
0.7
7(0
.65-0
.87)b
0.6
9(0
.54-0
.82)b
Rig
ht
TV
Cab
n0.3
5(0
.19-0
.56)
0.4
8(0
.31-0
.67)a
0.4
1(0
.24-0
.61)a
0.5
2(0
.35-0
.7)a
0.5
7(0
.41-0
.74)a
0.5
2(0
.35-0
.7)a
Left
TV
Cim
mob
0.5
4(0
.37-0
.72)a
0.6
4(0
.47-0
.78)b
0.7
6(0
.67-0
.87)b
0.8
4(0
.74-0
.91)c
0.9
1(0
.85-0
.95)c
0.8
6(0
.77-0
.93)c
Rig
ht
TV
Cim
mob
0.6
6(0
.52-0
.81)b
0.6
5(0
.5-0
.8)b
0.5
7(0
.41-0
.74)a
0.7
4(0
.61-0
.85)b
0.9
2(0
.86-0
.96)c
0.8
9(0
.82-0
.93)c
Poole
dT
VC
imm
ob
0.5
9(0
.47-0
.71)a
0.6
4(0
.53-0
.75)b
0.7
(0.6
-0.7
9)b
0.7
9(0
.7-0
.87)b
0.9
1(0
.87-0
.94)c
0.8
7(0
.81-0
.91)c
SGS
0.3
3(0
.16-0
.55)
0.4
4(0
.26-0
.63)a
0.6
9(0
.54-0
.82)b
0.8
9(0
.85-0
.95)c
0.9
1(0
.85-0
.95)c
0.9
1(0
.85-0
.95)c
Lary
nx
mas
s0.4
2(0
.24-0
.63)a
0.5
(0.3
3-0
.69)a
0.3
6(0
.2-0
.57)
0.5
3(0
.36-0
.71)a
0.5
4(0
.37-0
.72)a
0.6
9(0
.54-0
.82)b
Phar
ynx
mas
s0.0
9(0
.02-0
.27)
0.2
9(0
.14-0
.5)
0.1
4(0
.02-0
.33)
0.1
4(0
.02-0
.33)
0.2
4(0
.10.4
5)
0.2
9(0
.14-0
.5)
Hyp
ophar
ynx
mas
s0.1
1(0
-0.2
9)
0.1
8(0
.05-0
.38)
0.0
3(–
0.0
6to
0.1
9)
0.0
6(–
0.0
4to
0.2
3)
0.1
(–0.0
7to
0.1
6)
0.2
(0.0
7-0
.4)
Leve
lofco
nce
rn0.3
7(0
.19-0
.58)
0.4
6(0
.29-0
.65)a
0.4
9(0
.32-0
.68)a
0.6
8(0
.53-0
.82)b
0.7
2(0
.57-0
.84)b
0.6
8(0
.53-0
.82)b
Leve
lofco
mfo
rt0.0
9(–
0.4
to0.2
9)
0.0
6(–
0.4
to0.2
3)
0.0
8(–
0.0
2to
0.2
6)
0.3
1(0
.15-0
.52)
0.1
6(0
.04-0
.36)
0.2
5(0
.11-0
.46)
Poole
d(S
GS,
TV
C
abn,T
VC
imm
ob,
Lary
nx,le
velofco
nce
rn)
0.7
11
(0.6
58-0
.762)b
0.7
45
(0.6
97-0
.789)b
0.7
6(0
.72-0
.803)b
0.8
27
(0.7
9-0
.861)c
0.8
8(0
.86-0
.905)c
0.8
6(0
.83-0
.888)c
Abbre
viat
ions:
abn,ab
norm
al;A
ryte
n,ar
yten
oid
;BO
T,bas
eofto
ngu
e;IC
Cs,
intr
acla
ssco
rrel
atio
ns;
imm
ob,im
mobili
ty;Phar
yn,phar
ynx;SG
S,su
bgl
ott
icst
enosi
s;T
VC
,tr
ue
voca
lco
rd.
a Moder
ate
relia
bili
ty(I
CC
,0.4
-0.6
).bG
ood
relia
bili
ty(I
CC
,0.6
-0.8
).c V
ery
good
relia
bili
ty(I
CC
,0.8
-1.0
).
847 at SOCIEDADE BRASILEIRA DE CIRUR on May 11, 2015oto.sagepub.comDownloaded from
generating significantly more experience and independent
endoscopy with subsequent decision making.
The discovery of the increase in reliability between the
PGY-2 and PGY-3 year, both within class and when com-
pared with the attending gold standard, validates our institu-
tion’s protocol of a backup call system in which senior
residents supervise endoscopic examinations in person in the
first 2 months of the PGY-2 year. Articles written on the
development of the technical skill of performing flexible
transnasal laryngoscopy demonstrate that acquisition of the
mechanical skills is related to the number of times the proce-
dure has been performed and time training in the technique,
and in this paper the data demonstrate that time spent training
increases the reliability of diagnosis for specific important
findings.12,13 Therefore, it is the authors’ recommendation
that a period of supervised endoscopy be adhered to during
the early months of independent otolaryngology call (which
is the start of the PGY-2 year at our institution). After an ini-
tial period of close supervision, residents will have obtained
a significant degree of reliability with attending diagnostic
findings and can be trusted to make critical diagnoses.
Other studies have also been published about interrater
reliability in the interpretation of findings of flexibile transna-
sal laryngoscopy, demonstrating excellent interrater reliability
for the interpretation of aspiration in fiberoptic endoscopic
evaluation of swallowing (FEES) but poor agreement
between reviewers when assessing for extra esophageal
reflux.14-16 To date, no stepwise increase in diagnostic com-
petency has been demonstrated or been measurable for resi-
dents. In this study, the authors have developed an
assessment questionnaire tool that can be used to determine
resident competency at diagnosis of specific findings on flex-
ible transnasal laryngoscopy. Using this tool, it may be possi-
ble to identify residents struggling with endoscopy and
determine their level of competence with the diagnostic
aspect of the procedure. Additionally, within the Next
Accreditation System Aerodigestive Tract Lesions worksheet,
this tool may be able to determine whether or not a resident
has reached level 3, ‘‘Knows the differential diagnosis of
vocal cord lesion; able to clinically neoplastic from non-
neoplastic etiologies,’’ or level 4, ‘‘Formulates appropriate
treatment plan for a specific vocal cord lesion based on
lesion and patient factors,’’ because the tool can compare the
reliability of the resident’s level of concern with that of the
attending gold standard, and the level of concern question
prompts the reviewer to decide whether the airway is
unstable, the patient needs a biopsy, or no specific interven-
tion is necessary (Figure 1, Figure 4). In the future, the
authors hope to truncate and streamline the tool to the topics
that demonstrated the highest reliability, abnormalities of the
vocal cords, immobility of the vocal cords, laryngeal mass,
subglottic stenosis, and level of concern.
There are several significant limitations to this study.
Many training programs have variations in exposure to oto-
laryngology during the first year of postgraduate training,
resulting in varying levels of experience with fiberoptic
endoscopy. At our institution, the PGY-2 year is the first in
which residents take consult call for the otolaryngology ser-
vice, although they perform endoscopy in the outpatient set-
ting during their otolaryngology rotations as interns.
Therefore, the first 2 months with supervised endoscopy
represent the first significant inpatient endoscopy experience
at our institution, and this may vary significantly in other
programs. Additionally, this study was performed at a single
institution within a single residency program, which makes
it difficult to extrapolate to other residency programs, as the
volume of flexible transnasal laryngoscopies and the
amount of supervision and feedback likely vary significantly
between programs. Also, the study does not take into
account the technical ability and skill required to perform
flexible transnasal laryngoscopy, although previous studies
have demonstrated that this skill is required in relatively
few attempts.18 Finally, within our residency, there are only
3 residents per year, limiting the amount of power of the
statistical calculations and limiting the conclusions that can
be drawn from the data. With pooled data from several
institutions, more subtle trends might emerge and lead to a
00.10.20.30.40.50.60.70.80.9
1
MS4 vsA�ending
For PGY1 vsA�ending
For PGY2 vsA�ending
For PGY3 vsA�ending
For PGY4 vsA�ending
For PGY5 vsA�ending
Intr
acla
s Co
rrel
a�on
Coe
ffici
ent
Training Level Compared with A�ending
Pooled Intraclass Correla�on Coefficients forEach Training Level vs A�ending
Figure 3. Intraclass correlations for pooled variables (left andright vocal cord abnormalities, left and right vocal cord mobility,subglottic stenosis, laryngeal mass, level of concern) for each classof training compared to the attending gold standard with 95% con-fidence intervals.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
MS4 For PGY1 For PGY2 For PGY3 For PGY4 For PGY5 ForA�endings
Intr
acla
ss C
orre
la�
on C
oeffi
cien
t
Training Level
Pooled Intraclass Correla�on Coefficients Within Training Level
Figure 2. Intraclass correlations for pooled variables (left andright vocal cord abnormalities, left and right vocal cord mobility,subglottic stenosis, laryngeal mass, level of concern) for each levelof training with 95% confidence intervals.
848 Otolaryngology–Head and Neck Surgery 152(5)
at SOCIEDADE BRASILEIRA DE CIRUR on May 11, 2015oto.sagepub.comDownloaded from
clearer understanding of resident progression in diagnostic
flexible transnasal laryngoscopy.
Conclusion
Resident competency in the ability to interpret flexible transna-
sal laryngoscopy can be measured objectively within a training
program and occurs in the first year of otolaryngology-specific
training. Abnormalities of the vocal cords, vocal cord mobility,
laryngeal mass, subglottis stenosis, and level of concern are
the strongest parameters in determining competency and can
be used to place residents within the New Accreditation
System framework for competence in management of
Aerodigestive Tract lesions. As resident education shifts
toward competency-based evaluations, objective measures of
competency will allow hospitals to improve safe care with
timely diagnosis of critical airway findings.
Acknowledgments
We would like to thank Howard Cabral for assistance with statisti-
cal analysis.
Author Contributions
Christopher D. Brook, design, acquisition, analysis, drafting, revi-
sion, final approval; Michael P. Platt, design, analysis, revision,
final approval; Kimberly Russell, design, acquisition, analysis,
revision, final approval; Gregory A. Grillone, design, analysis,
revision, final approval; Avner Aliphas, design, analysis, revision,
final approval; J. Pieter Noordzij, design, analysis, revision, final
approval.
Disclosures
Competing interests: None.
Sponsorships: None.
Funding source: This work was supported in part through fund-
ing to the Boston University Clinical and Translational Science
Institute (NIH NCATS: UL1-TR000157), which supported
Howard Cabral, who provided assistance with statistical
analysis.
References
1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The Next GME
accreditation system—rationale and benefits. N Engl J Med.
2012;366:1051-1056.
2. Martin JA, Regehr G, Reznick R, et al. Objective structured
assessment of technical skill (OSATS) for surgical residents.
Br J Surg. 1997;84:273-278.
3. Francis HW, Masood H, Chaudhry KN, et al. Objective assess-
ment of mastoidectomy skills in the operating room. Otol
Neurotol. 2010;31:759-765.
4. Ishman SL, Brown DJ, Boss EF, et al. Development and pilot
testing of an operative competency assessment tool for pediatric
direct laryngoscopy and rigid bronchoscopy. Laryngoscope.
2010;120:2294-2300.
5. Ishman SL, Benke JR, Johnson KE, et al. blinded evaluation
of interrater reliability of an operative competency assessment
tool for direct laryngoscopy and rigid bronchoscopy. Arch
Otolaryngol Head Neck Surg. 2012;17:1-7.
6. Laeeq K, Waseem R, Weatherly RA, et al. In-training assess-
ment and predictors of competency in endoscopic sinus sur-
gery. Laryngoscope. 2010;120:2540-2545.
7. Zaiontz C. Real statistical analysis using Excel. Real statistical
resource pack for Excel 2007. http://www.real-statistics.com.
Accessed October 12, 2014.
8. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the
road to competence: developmental milestones for internal
medicine residency training. J Grad Med Educ. 2009;1:5-20.
Figure 4. Accreditation Council on Graduate Medical Education (ACGME) Otolaryngology Milestones Aerodigestive Tract Lesions work-sheet.17 (�ACGME and the American Board of Otolaryngology. Reproduced with permission.)
Brook et al 849
at SOCIEDADE BRASILEIRA DE CIRUR on May 11, 2015oto.sagepub.comDownloaded from
9. Hicks PJ, Schumacher DJ, Benson BJ, et al. The pediatrics
milestones: conceptual framework, guiding principles, and
approach to development. J Grad Med Educ. 2010;2:410-418.
10. Lee AG, Arnold AC. The ACGME Milestone Project in
ophthalmology. Surv Ophthalmol. 2013;58:359-369.
11. Sethi RK, Kozin ED, Remenschneider AK, et al. Subspecialty
emergency room as alternative model for otolaryngologic care:
implications for emergency health care delivery. Am J
Otolaryngol. 2014;35:758-765.
12. Deutschmann MW, Yunker WK, Cho JJ, Andreassen M,
Beveridge S, Bosch JD. Use of a low-fidelity simulator to
improve trans-nasal fibre-optic flexible laryngoscopy in the
clinical setting: a randomized, single-blinded, prospective
study. J Otolaryngol Head Neck Surg. 2013;42:35.
13. Ossowski KL, Rhee DC, Rubinstein EN, Ferguson BJ.
Efficacy of sinonasal simulator in teaching endoscopic nasal
skills. Laryngoscope. 2008;118:1482-1485.
14. Leder SB, Acton LM, Lisitano HL, Murray JT. Fiberoptic
endoscopic evaluation of swallowing (FEES) with and without
blue-dyed food. Dysphagia. 2005;20:157-162.
15. Musser J, Kelchner L, Neils-Strunjas J, Montrose M. A com-
parison of rating scales used in the diagnosis of extraesopha-
geal reflux. J Voice. 2011;25:293-300.
16. Kelchner LN, Horne J, Lee L, et al. Reliability of speech-
language pathologist and otolaryngologist ratings of laryngeal
signs of reflux in an asymptomatic population using the reflux
finding score. J Voice. 2007;21:92-100.
17. The Otolaryngology Milestone Project. Accreditation Council
for Graduate Medical Education website. http://acgme.org/acg
meweb/Portals/0/PDFs/Milestones/OtolaryngologyMilestones.pdf.
Published October 2013. Accessed February 4, 2015.
18. Laeeq K, Pandian V, Skinner M, et al. Learning curve for com-
petency in flexible laryngoscopy. Laryngoscope. 2010;120:
1950-1953.
850 Otolaryngology–Head and Neck Surgery 152(5)
at SOCIEDADE BRASILEIRA DE CIRUR on May 11, 2015oto.sagepub.comDownloaded from