multi-institutional development of a mastoidectomy ... · multi-institutional development of a...

MULTI-INSTITUTIONALDEVELOPMENTOFAMASTOIDECTOMYPERFORMANCEEVALUATIONINSTRUMENTThomasKerwin1,BradHittle1,DonStredney1,PaulDeBoeck2,GregoryWiet3

1InterfaceLab,OhioSupercomputerCenter,Columbus,Ohio,UnitedStates

2DepartmentofPsychology,OhioStateUniversity,Columbus,Ohio,UnitedStates

3DepartmentofOtolaryngology,OhioStateUniversity,Columbus,Ohio,UnitedStates

CORRESPONDINGAUTHOR:ThomasKerwin1

1224KinnearRd.,Columbus,Ohio,43212,UnitedStates

Emailaddress:kerwin@osc.edu

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

MULTI-INSTITUTIONALDEVELOPMENTOFA1

MASTOIDECTOMYPERFORMANCEEVALUATION2

INSTRUMENT3

ABSTRACT4

OBJECTIVE5

Amethodforratingsurgicalperformanceofamastoidectomyprocedurethatisshowntoapply6universallyacrossteachinginstitutionshasnotyetbeendevised.Thisworkdescribesthe7developmentofaratinginstrumentcreatedfromamulti-institutionalconsortium.8

DESIGN9

UsingaparticipatorydesignandamodifiedDelphiapproach,amulti-institutionalgroupofexpert10otologistsconstructeda15elementtask-basedchecklistforevaluatingmastoidectomy11performance.Thisinstrumentwasfurtherrefinedintoa14elementchecklistfocusingonthe12conceptofsafetyafterusingittoratealargeandvariedpopulationofperformances.13

SETTING14

TwelveOtolaryngologicalsurgicaltrainingprogramsintheUnitedStates.15

PARTICIPANTS16

14surgeonsfrom12differentinstitutionstookpartintheconstructionoftheinstrument.17

RESULTS18

Byusing14expertsfrom12differentinstitutionsandaliteraturereview,individualmetricswere19identified,ratedastothelevelofimportanceandoperationallydefinedtocreatearatingscalefor20mastoidectomyperformance.Initialuseoftheratingscaleshowedmodestrateragreement.The21operationaldefinitionsofindividualmetricsweremodifiedtoemphasize“safe”asopposedto22“proper”technique.Asecondratinginstrumentwasdevelopedbasedonthisfeedback.23

CONCLUSIONS24

Usingaconsensusbuildingapproachwithmultipleroundsofcommunicationbetweenexpertsisa25feasiblewaytoconstructaratinginstrumentformastoidectomy.Expertopinionaloneusinga26Delphimethodprovidesfaceandcontentvalidityevidence,however,thisisnotsufficientto27developauniversallyacceptableratinginstrument.Acontinuedprocessofdevelopmentand28

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

experimentationtodemonstrateevidenceforreliabilityandvaliditymakinguseofalarge29populationofratersandperformancesisnecessarytoachieveuniversalacceptance.30

KEYWORDS31

mastoidectomy,assessment32

COMPETENCIES33

MedicalKnowledge,Practice-BasedLearningandImprovement34

INTRODUCTION35Skillassessmentisessentialtoalltypesoftraining,andotologicsurgeryisnoexception.Inaddition36toprovidingevidencethatabasiclevelofskillproficiencyhasbeenachieved,accuratefeedbackcan37acceleratelearning1.Surgicalresidencyprogramscurrentlyuseavarietyoftoolstoassesstrainees,38andnosingletoolhasemergedasthe"goldstandard".Ataminimum,agoodassessmenttoolmust39bereliable,feasible,fair,objective,andvalid2.Thetime-honoredassessmentcurrentlyusedbythe40AmericanBoardofOtolaryngology(ABOto)andtheAccreditationCouncilforGraduateMedical41Education(ACGME)isbaseduponboththeaccumulationof“adequate”casenumbersduring42trainingandalsotheattestationofthespecificresidencyprogramdirectorwheretheresident43trained.Notwithstanding,thereislittleevidenceofthereliabilityorvalidityofthecurrent44assessmentregimen.45

Auniversallyapplicablesetofmetricsthatcanbeagreeduponandusedforassessmentoftechnical46skillinperformingamastoidectomyhasnotbeendevelopedoradopted.Inordertodevelopsuch47anassessmenttool,caremustbegiventoformulateandvalidatethattooltakingintoaccount48differencesbetweentrainingprograms.Assessmenttoolsmustbedesignedbasedonwhatthe49measurementinstrumentwillbeusedforandwhatspecificinferenceswillbemadebasedonthe50results3.Thereisneedforaninstrumentforbothuserfeedbackintrainingandfordeterminingthe51levelofanindividual’sperformance(novice,intermediateorexpert)intermsoftechnical52performanceofamastoidectomywithfacialrecessapproach.53

Inthiswork,wedescribethecreationandevolutionofasetofmetricsspecificallyfordetermining54thelevelofanindividual'sperformanceinmastoidectomy.Weusedabroad-basedconsortiumof55surgeonsatdifferentinstitutionsinconsecutivefeedbackstepssothattheinstrumentcanbe56universallyappliedtoalltemporalbonedissectionperformancesregardlessofinstitutionor57background.58

PREVIOUSWORK59Ratinginstrumentsformastoidectomyhavebeendevelopedbyothergroups,buttheydonot60includesuchabroadbaseofexpertinput.Arecentreviewofthecurrentinstrumentsformeasuring61mastoidectomyperformancebySethiaetal.discusseseachoftheinstrumentsingreaterdetail4.62

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

AgroupatJohnsHopkinsdevelopedaninstrumentbasedontheworkofMartinetal.5for63mastoidectomyperformancecontainingbothaTask-BasedChecklist(TBC)andGlobalRatingScale64(GRS)6.Bothofthescalesusealistofevaluationitemswithratingsofonetofive.WorkbyLaeeqet65al.7andAwadetal.8showsomevalidityevidenceforthatinstrumentbutinonlyasmallnumberof66institutions.67

TheWellingScale(WS1)usesfinalproductanalysis(FPA)forevaluatingacomplete68mastoidectomywithfacialrecessperformedinthetemporalbonelab9,10.Itdefinesbinaryitems69thataresummedtoprovideanoverallscore.70

AsseeninthesurveyresultsfromButleretal.10,eventhoughasetofcommonevaluationitemsfor71mastoidectomycanbecreated,thereexistmanydifferencesbetweentheimportancegiventothose72itemsbyexpertsfromdifferentinstitutions.Additionalcaremustbegiventodevelopandevaluate73instrumentsthatcanbeusedbroadlyatallinstitutions.Inordertocreatesuchaninstrument,an74attemptwasmadebyWanetal.11touseamodifiedDelphimethodtofindconsensusonwhich75itemsshouldbeincorporatedintoaTBC.TheHopkinsscalewasalsodevelopedusingaDelphi76method,butincludedonlyJohnsHopkinsfacultymembersintheprocess.77

TheWanetal.studyreceivedresponsesfrom88membersoftheAmericanNeurotologySocietyor78AmericanOtologicalSocietyoncriteriaimportanttoasuccessfultemporalbonedissection.Based79onthoseresponses,alistofcriteriaorderedbyimportancewascreatedandusedinthisstudy.80

MATERIALSANDMETHODS81Inordertocreateaconsensus-based,cross-institutionalratinginstrumenttomeasuresurgical82performancewestartedwiththelistofassessmentitemsfromWanetal.11Theseitemswerethen83furtherrefinedusingaDelphimethoddescribedindetailbelowwithanexpertgroupconsistingof8414fellowship-trainedotologistsfrom12differentinstitutions(Table1).Inthisrefinement,the85individualitemsfromtheWanstudyweremoreexplicitlydefinedtoencourageauniform86interpretationfordeterminingsuccessorfailureforeachitem.Thislistwasthenreviewedbyall87individualsinthesamegroupofexpertsbymeansofanonlinesurvey.88

Inafirstround,membersoftheconsortiumwereaskedtorankthe5mostimportantand6least89importantmetricsonthelist.Resultsofthesurveyshowed24metricswithadditionalsuggestions90(Table2).91

Aface-to-facemeetingforactivediscussionregardingeachmetric,itsoverallimportanceandan92agreeduponoperationaldefinitionwasconvenedwiththemembersoftheexpertgroup.Inthis93meeting,eachmetricwaspresentedseparatelyalongwithanycommentsthatweremadewithin94thesurveycontext.AnexampleofametricresultanddiscussionispresentedinFigure1.95

Next,theexpertswereaskedtoassignanimportancemeasuretoeachmetric,asfollows:96

• Pass/Fail(P/F):Criticalmetricsthat,ifanyoneisviolated,thereisanautomaticfailure.97Violationsofthesemetricswillresultinseriousmorbiditytothepatient.98

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

• High:Dangerous,ifviolatedcouldpotentiallyresultinmorbiditytothepatient.99• Medium:Potentialcomplicationthatrequiresinterventionandcouldberectifiedor100

managedwithoutsignificantmorbiditytothepatient.101• Low:Potentialcomplicationwhichdoesnotrequireintervention–poortechnique.102

Then,inasecondround,expertswereaskedtoidentifywhichitemswereneededtobecompetent103inordertobeconsiderednovicelevel(readytooperateonpatientundersupervision),104intermediatelevel(readyforminimalsupervision–PGY4/5level),advancedlevel(practice105independentlyatfellowshiptrainedlevel).Usingthefollowingcriteria:106

• Novicelevel.(competencyoneachofthehighimportanceareasandnoFs).(readyfor107cadavericlab)108

• Intermediatelevel.(competencyonallofhighandmediumitemsandnoFs).(readyfor109SupervisedORexperience).110

• Advancedlevel.(expertonallmetricsandnoFs)(readyforindependentsurgery,doesnot111needsupervision).112

TheresultsoftheabovetworoundsarelistedinTable3asoriginalandfinalrelativeimportance.113TheitemslistedasP/F(Pass/Fail)includethoseitemsforwhichiftheywerenotachieved,the114globalperformanceautomaticallyresultedinafailingscoreregardlessofperformanceonanyother115metric.TheitemslistedasHighprioritywerethoseitemswithconditionstobefulfilledtobe116consideredasanoviceoperator,theitemslistedasMediumareitemswithconditionstobefulfilled117tobeconsideredasanintermediateandtheitemslistedaslowarenecessaryconditionsforan118advancedleveloperator.Byimplicationtheabsenceofmoreimportantviolationsisnecessaryas119wellforeachofthethreelevels.120

Atthispoint,underIRBapprovalfromTheOhioStateUniversityOfficeofResponsibleResearch,we121performedastudyusingourpreviouslydevelopedtemporalbonedissectionsimulator12,13across122the12institutions.Thisresultedinsixty-sixmastoidectomyperformancesforreview.Theycovered123awidedistributionofskilllevels:medicalstudents,PGY(Post-GraduateYear)2-5,fellowsand124attendingphysicians.Thissetcomprised36sessionscollectedfromfacultyand30collectedfrom125residentsandstudents.Eachoftwelveexpertreviewers,allconsideredexpertsinotologicsurgery,126wasassignedelevengradingtasks(individualmastoidectomyperformances).Theywereblindedto127theidentityofthesubjectperformingthedissectionanddidnotreviewtheirownperformances.128Thisresultedintwosetsofratingsusingtheinstrumentforeachvirtualmastoidectomyinthe129testingset.Afterexaminingthestatisticalmeasuresfromthistrial,amoderatelylowlevelof130agreementamongraterswasseen(overhalftheinterclasscorrelation12(ICC)valueswerebelow1310.4,whichisconsideredpooragreement).132

Asaresultoftherelativelyweakinter-rateragreement,weconcludedthatperhapsthismaybedue133topooragreementontheoperationaldefinitionofeachmetricandhowitshouldbescored.Asa134result,anadditionalface-to-faceDelphiprocesswasundertakentodiscussthepooragreement135scores.Itwastheconsensusofthegroupthattheoperationaldefinitionsofeachitemwereasource136ofcontinuedvariabilityinhowtheyshouldbeinterpreted.Thegrouprecommendedfurther137refinementbasedonthepremisethattheywouldbeusedtoidentify"safe"asopposedto"proper"138

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

surgicaltechnique.Itwasrecognizedthattherearevariousopinionsastowhatconstitutes139“proper”technique.Theconsensuswasthattherewouldbegreateragreementiftheoperational140definitionofindividualmetricscouldbejudgedonthebasisofits“safety”.Specifically,ifa141particularstyleoftechniquewasnotonethataparticularraterrecognizedas“proper”,itcouldstill142bejudgedonwhetherornotitwasconsideredhighriski.e.,notsafe.Basedonthisdiscussion,a143secondsetofassessmentitemswasdeveloped.Additionally,atthesuggestionoftheexpertgroup,144thetwoitemsintheoriginallistthatconcernedtheexternalauditorycanalwerecombinedinto145one.Theresultofthisseconddiscussiongroupwasthedevelopmentofasecondsetofmetrics146encompassingalistof15items.Theindividualitemsforbothmetricsetscanbeseenand147comparedinTable4.148

Anoverviewofthestepswetooktoconstructthemetricsandthereasonsbehindthemcanbeseen149inTable5.150

DISCUSSION151Aswithclinicalcare,itisimportantthatclearandrigorousevidenceexiststoobjectivelyappraise152theefficacyofoureducationalprograms.15Subjectivedeterminationsbyprogramdirectorsor153traineeself-reportingofnumberofproceduresmustevolveintomoreevidencebasedassessments.154Thisrequiresaconcertedefforttodevelopoutcomemeasuresthatareagreeduponanduniversally155translatable.Forassessmentstobevalid,theymustaccumulatevalidityevidenceinanumberof156areasincludingcontentevidence,responseprocess,internalstructure,relationswithother157variablesandconsequences.16Ourmetricsdemonstrate“contentevidence”basedonthenatureof158thedevelopmentprocessnotedabove.Thenextvalidationstepsincludethedemonstrationofa159sufficientlyhighintra-rateragreementandtherelationshipwithanexternalcriterionforthe160qualityofaperformance.161

WehavefollowedtheprocessoutlinedbyDauphineeandWood-Dauphineefordeveloping162evidenced-basedmedicaleducation.15Thisinvolvesdefiningtheparameterstobemeasured,163measuringthoseparameters,andbenchmarkingthoseparameterstoassesseducationaloutcomes.164Asnotedbyourwork,theefforttodefineoutcomemeasureswithanacceptablelevelofcontent165validityisinitselfoftenpainstaking,especiallyifthegoalincludesuniversalacceptance.Studies166conductedatoneinstitutionoftenarefraughtwithsubjectivebiasandlowsamplesizes.15This167makesdisseminationofrecommendationsandguidelinesforassessmentproblematic.168

Ourattemptatdevelopingaspecificsetofmetricsforaprocedureasspecializedasmastoidectomy169hasprovenextremelychallenging.Inmastoidsurgery,thereareanumberofassessmenttoolsin170existencetoday,noneofwhichprovidebroadenoughacceptanceanduniversality.4Itisthegoalof171thisresearchtocontinuetheprocessofpainstakinglyrefiningthemetricsestablishedandthe172ratingprocesssothattheycanshowthevalidityevidencenecessarytomakeassessmentsthat173correlatewithclinicalperformance.174

Identifying,definingandapplyingmetricssothattheycanbeuniversallyusefulandstillprovide175sufficientinformationtomakevaliddecisionsbasedontheiruseisdifficultevenattheearlystages.176

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

Forthenextstepswearenecessarilysubjecttomanysourcesofpossibleassessorerror17.These177includepossibledriftinassessorinterpretationofindividualmetrics,individualperformance178expectations,andlackoffamiliarityofbeinganassessorasopposedtoatrainer.Thesesourcesof179ratingerroraremultipliedwiththeexpandednumberofassessors.Thesesourceshowever,canbe180mitigatedinthefuturebymakingaconcertedefforttoprovidegoodoperationaldefinitionsofeach181metric,carefultrainingofassessors(perhapsagroupsessionwhereastandardizedperformanceis182ratedanddiscussedwithinthecontextofthegroup),andmonitoringoftheassessor’sperformance183assuggestedbyGallagheretal.17184

Inthefuture,wewilluseournewsetofmetricstoaccumulateadditionalvalidityevidence.185Emphasizingsafetyastheglobalconceptindefiningandadministeringtheitemsisonewaywecan186makeouroperationaldefinitionsmorewidelyapplicable.Wearecurrentlyinvestigatingdefining187ourmeasurementscalesintermsofthreeseparateaxes:boneremoval,toolcontrolandviolations188ofstructures(Table6).Thesecanfunctionasdistinctsubscales.Measurementscalesforskill189masterywillbebuiltforeachaxissuchthattheperformancesoftraineescanbeevaluatedinterms190ofdescriptiveandnormativemasterylevels.Thedescriptivelevelsarespecificpositionsonthe191measurementscaleswhilethenormativelevelsarelevelsthatmustbereachedtobeconsideredan192independentexpert(expertlevel)oranintermediateleveltrainee(intermediatelevel).The193approachtobeusedisatwo-foldextensionofitemresponsetheory(IRT).18-20IRTisafamilyof194statisticalmeasurementmodelsthathasbecomethestandardforthemeasurementofskillsinan195educationalandtrainingcontext.IRTscoresaremodel-baseddescriptivemasterylevels.196Additionally,wearedesigningamethodologytoeasily“traintheraters”sothatconsistencyin197interpretationandapplicationofthemetricsisplausible.198

CONCLUSION199Ourworkmovesclosertothegoalofdevelopingauniversallyacceptableandapplicablesetof200performancemetricsformastoidsurgery.Wehaveusedanextensiveparticipatoryprocessto201formulatealistofmetricsbasedonliteraturereview,multipleroundsofexpertfeedback,and202continuedrefinement.Basedonourmethodology,wefeelthatourresultsdemonstratesignificant203contentvalidity.Ourresultsdemonstrateconsiderableinputofdiverseexpertopinionbutstillneed204tobesupplementedwithothertypesofvalidityinamulti-institutecontext.205

ACKNOWLEDGEMENTS206ThisworkwassupportedbyTheNationalInstituteforDeafnessandotherCommunication207Disorders,NationalInstitutesofHealth,USA,R01DC011321.208

BIBLIOGRAPHY209210

1. CataniaAC.Learning.2nded.EnglewoodCliffs,N.J.:Prentice-Hall;1984.211

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

2. ShahJ,DarziA.Surgicalskillsassessment:anongoingdebate.BJUInt.2001;88(7):655-660.2123. MichelsonJD,ManningL.Competencyassessmentinsimulation-basedprocedural213

education.AmJSurg.2008;196(4):609-615.2144. SethiaR,KerwinTF,WietGJ.PerformanceAssessmentforMastoidectomy:StateoftheArt215

Review.OtolaryngolHeadNeckSurg.2016.2165. MartinJA,RegehrG,ReznickR,etal.Objectivestructuredassessmentoftechnicalskill217

(OSATS)forsurgicalresidents.BrJSurg.1997;84(2):273-278.2186. FrancisHW,MasoodH,ChaudhryKN,etal.Objectiveassessmentofmastoidectomyskillsin219

theoperatingroom.OtolNeurotol.2010;31(5):759-765.2207. LaeeqK,BhattiNI,CareyJP,etal.Pilottestingofanassessmenttoolforcompetencyin221

mastoidectomy.Laryngoscope.2009;119(12):2402-2410.2228. AwadZ,TornariC,AhmedS,TolleyNS.Constructvalidityofcadaverictemporalbonesfor223

trainingandassessmentinmastoidectomy.Laryngoscope.2015;125(10):2376-2381.2249. FernandezSA,WietGJ,ButlerNN,WellingB,JarjouraD.Reliabilityofsurgicalskillsscores225

inotolaryngologyresidents:analysisusinggeneralizabilitytheory.EvalHealthProf.2262008;31(4):419-436.227

10. ButlerNN,WietGJ.ReliabilityoftheWellingscale(WS1)forratingtemporalbone228dissectionperformance.Laryngoscope.2007;117(10):1803-1808.229

11. WanD,WietGJ,WellingDB,KerwinT,StredneyD.Creatingacross-institutionalgrading230scalefortemporalbonedissection.Laryngoscope.2010;120(7):1422-1427.231

12. WietGJ,BryanJ,DodsonE,etal.Virtualtemporalbonedissectionsimulation.Studiesin232healthtechnologyandinformatics.2000;70:378-384.233

13. WietGJ,StredneyD,KerwinT,etal.Virtualtemporalbonedissectionsystem:OSUvirtual234temporalbonesystem:developmentandtesting.Laryngoscope.2012;122Suppl1:S1-12.235

14. ShroutPE,FleissJL.Intraclasscorrelations:usesinassessingraterreliability.PsycholBull.2361979;86(2):420-428.237

15. DauphineeWD,Wood-DauphineeS.Theneedforevidenceinmedicaleducation:the238developmentofbestevidencemedicaleducationasanopportunitytoinform,guide,and239sustainmedicaleducationresearch.AcadMed.2004;79(10):925-930.240

16. CookDA,ZendejasB,HamstraSJ,HatalaR,BrydgesR.Whatcountsasvalidityevidence?241Examplesandprevalenceinasystematicreviewofsimulation-basedassessment.Advances242inhealthscienceseducation:theoryandpractice.2014;19(2):233-250.243

17. GallagherAG,O'SullivanGC.Fundamentalsofsurgicalsimulation:principlesandpractices.244London:Springer;2012.245

18. LindenWJvd.Handbookofitemresponsetheory:Models,statisticaltools,andapplications246BocaRaton:CRCPress;2016.247

19. DeBoekP,WilsonM.Explanatoryitemresponsemodels.Ageneralizedlinearandnonlinear248approach.NewYork:Springer;2004.249

20. DeChamplainAF.Aprimeronclassicaltesttheoryanditemresponsetheoryfor250assessmentsinmedicaleducation.Medicaleducation.2010;44(1):109-117.251

252

253

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

Table1:ParticipatingTrainingInstitutions254

BaylorUniversity

DukeUniversity

HenryFordHospitalSystem

UniversityofIowa

UniversityofMississippi

Montefiore/AlbertEinsteinCollegeofMedicine

StanfordUniversity

UniversityofCalifornia,Irvine

UniversityofCincinnati

UniversityofTexas,Southwestern

TheOhioStateUniversity

MedicalUniversityofSouthCarolina

255

256

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

257

Table2:Resultsfromsurveyonimportanceofindividualmetrics.13expertslisted5itemsashigh258importanceand6itemsaslowimportance.259

Metric Expertsselectingashigh

importance

Expertsselectingas

lowimportance

Maintainsvisibilityoftoolwhileremovingbone 6 1Selectappropriateburrtypeandsize 4 2Antrumentered 4 1Noviolationoffacialnervecanal 11 0Noviolationofsigmoidsinus 3 1Identifiestympanicsegmentoffacialnerve 0 2Doesnotdrillonossicle 5 1Doesnotuseexcessivedrillforcenearcriticalstructures 6 0

Identifieschordatympani 0 3Drillsinbestdirection(understandingofcuttingedge) 3 3

Canalwallup 1 3Identifiesfacialnerveatcochlearformprocess 0 4Appropriatedepthofcavity 0 3Drillswithbroadstrokes 1 3NoholesinEAC 2 2Completesaucerization 2 4Posteriorexternalauditorycanalwallthinned 2 2Facialrecesscompletelyexposed 2 1Identifiesfacialnerveatexternalgenu 1 2Lowfrequencyofdrill“jumps” 2 6Noholesinthetegmen 3 2Useofdiamondburrwithin2mmoffacialnerve 1 2Nocellsremainonsinoduralangle 0 10Sinoduralanglesharplydefined 0 7Otheradditionalmetric 1 0260

261

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

Table3:Originalandfinaldistributionsofmetricsbasedonlevelofimportanceandwhichmetric262expectedtobeachievedateachperformancelevel.263

Metric Importanceproposedtoexperts.

Final

Maintainsvisibilityofburrwhileremovingbone High High

Excessiveforcewillnotbeusednearcriticalstructures High High

Appropriatedepthofcavity Low LowNoholesintegmen Low LowSelectappropriateburr Medium MediumViolationofthesigmoidsinus Medium MediumIdentificationofchordatympaninerve High MediumDrillinbestdirection Medium MediumExternalauditorycanalwallwillremainup Medium Medium

Noholesinexternalauditorycanalwall Low Medium

Completesaucerization Medium MediumPosteriorexternalauditorycanalwallthinnedappropriately

Medium Medium

Violationofthefacialnerve P/F P/FViolationofthehorizontal(lateral)semi-circularcanal P/F P/F

Drillcontactwithossicles P/F P/FViolationofdura P/F264

265

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

Table4:Textofquestionsaskedduringmastoidectomyperformancereview.Question#10wasremoved266forthesecondinstrument,duetooverlapwithquestion#9.267

Number Instrument1 Instrument2

1 Maintainsvisibilityofburrwhileremovingbone

Maintainssafeviewoftheburrthroughouttheprocedure

2 Excessiveforcewillnotbeusednearcriticalstructures

Maintainssafeforcenearcriticalstructuresthroughouttheprocedure

3 Appropriatedepthofcavity Sufficientremovalofmastoidaircellsforpropervisualizationofdeepstructures

4 Noholesintegmen Maintainsintegrityoftegmen

5 Selectappropriateburr EfficientandSafeburrselection

6 Violationofthesigmoidsinus Maintainsintegrityofsigmoidsinus

7 Identificationofchordatympaninerve Identifieschordatympaninervesufficientlytoperformfacialrecessapproach

8 Drillinbestdirection Efficientandsafedirectionofdrilling(paralleltocriticalstructures)

9 Externalauditorycanalwallwillremainup

Sufficientthinningofposteriorexternalauditorycanalwalltovisualizefacialnerve

10 Noholesinexternalauditorycanalwall

11 Completesaucerization Sufficientsaucerizationforsafedrilling

12 Posteriorexternalauditorycanalwallthinnedappropriately

Avoidsoverthinningorholesinposteriorauditorycanalwall

13 Violationofthefacialnerve Maintainsintegrityoffacialnerve

14 Violationofthehorizontal(lateral)semi-circularcanal

Maintainsintegrityofhorizontalsemi-circularcanal

15 Drillcontactwithossicles Maintainsintegrityofossicles

16 Violationofdura Maintainsintegrityofdura

268

269

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

Table5:Stepstakentodeveloptheinstrument,inorder,withabriefreasonforeachone.270

Step ReasonStartwithlistofitemsfromWanetal.

Includeawidesampleofsurgicalexpertise

Surveytodeterminemostandleastimportantitems

Removeverylowpriorityitemsandestablishbroadlevelsofimportance

Meetingtopresentsurveyresultsanddefinemetrics

Reviseitemtextbasedonconsensusfromexperts

Classificationofmetricsfornovice,intermediate,expertachievementlevel.

Reflectimportancelevelsofitemsinthescoringoftheinstrument

Validationstudyusinginstrument TestinstrumentRevisionofinstrumentfocusingonsafety Attempttoincreaseinterraterreliability 271

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

Table6:MetricsandPerformanceAxisforAssessmentStrategy272

Metrics AxisSufficientremovalofmastoidaircellsforpropervisualizationofdeepstructures

BoneRemoval

Identifieschordatympaninervesufficientlytoperformfacialrecessapproach

BoneRemoval

Sufficientthinningofposteriorexternalauditorycanalwalltovisualizefacialnerve

BoneRemoval

Sufficientsaucerizationforsafedrilling BoneRemovalAvoidsoverthinningorholesinposteriorauditorycanalwall BoneRemovalMaintainssafeviewoftheburrthroughouttheprocedure ToolcontrolMaintainssafeforcenearcriticalstructuresthroughouttheprocedure

Toolcontrol

EfficientandSafeburrselection ToolcontrolEfficientandsafedirectionofdrilling(paralleltocriticalstructures)

Toolcontrol

Maintainsintegrityoftegmen ViolationMaintainsintegrityofsigmoidsinus ViolationMaintainsintegrityoffacialnerve ViolationMaintainsintegrityofhorizontalsemi-circularcanal ViolationMaintainsintegrityofossicles ViolationMaintainsintegrityofdura Violation 273

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017

Figure1:Exampleslideofanindividualmetriclevelofimportanceandoperationaldefinitiondiscussion274basedonsurveytogroupofexperts.275

276

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017