What we should do: 14 evidence-based 6ps for

approaching the challenges of performance-oriented CME

DonMoore,PhD#10ECF,Dublin,Ireland

8November2017

Disclosures – Don Moore, PhD

EmploymentVanderbiltUniversityMedicalCenter•  Director,OfficeforCPD•  Director,VanderbiltMOCPorHolioProgramVanderbiltUniversitySchoolofMedicine•  ProfessorofMedicalEducaKonand

AdministraKon•  DirectorofProgramEvaluaKon,Officeof

UndergraduateMedicalEducaKon

FinancialrelaKonships•  Idonothaveanyfinancialrela0onshipswithanycommercialenKtythatmakesordistributesproductsand/orservicesusedbyoronpaKentsthatarerelevanttothecontentofmypresentaKon.

Learning objec7ves

AQerlisteningtothispresentaKon,youshouldbeabletodescribeanddiscuss:•  AfivestepprocessfordesigningCMElearningacKviKes•  AprojectmanagementstrategyforCMElearningacKviKes•  AmethodtodeterminedesiredresultsandcontentforCMElearningacKviKes•  AnapproachtodesignandorganizeCMElearningacKviKesfortransfer•  AnevaluaKonandassessmentplantodetermineifcontentwaslearnedanddesiredresultswereachieved

•  TheearlystagesofanacKonplanforusinginyourCMEpracKcewhatyouhavelearnedhereatthismeeKng

Tip 1 – Manage learning as a project

•  AlearningintervenKonisaprojecttocreateachangeinperformance.•  AprojectistemporaryinthatithasadefinedbeginningandendinKme,andthereforedefinedscopeandresources.•  AprojectisuniqueinthatitisnotarouKneoperaKon,butaspecificsetofoperaKonsdesignedtoaccomplishasingulargoal.•  Aprojectteamo6enincludespeoplewhodon’tusuallyworktogether.•  Alearningprojectmustbeacollabora0veeffortthatinvolvesplanners,subjectma@erexperts,learners,andadministra0vestaff.•  ProjectmanagementistheapplicaKonofknowledge,skills,tools,andtechniquestoprojectacKviKestomeettheprojectrequirements.

Tip 1 – Manage learning as a project

Projectmanagementprocessesfallintofivegroups:•  IniKaKng• PlanningandDesign•  ExecuKng• MonitoringandControlling• Closing

ProjectManagementInsKtute,2017

Tip 1 – Manage learning as a project

•  IniKaKngphase•  Natureandscopeoftheproject•  Projectcharter:costs,tasks,deliverables,schedules

•  PlanningandDesign-planKme,costandresourcesadequatelyto•  EsKmatetheworkneeded•  EffecKvelymanageriskduringprojectexecuKon

•  ExecuKng(Producing)•  Projectdeliverablesareproduced•  ProperallocaKon,co-ordinaKonandmanagementofhumanresourcesandanyotherresourcesnecessaryto

producedeliverables•  Monitoringandcontrolling

•  Projectperformanceisobserved/measuredregularlytoidenKfyvariancesfromtheprojectmanagementplan.

•  CorrecKveacKontakenwhennecessary.•  Closing

•  FinalizeallacKviKesacrossalloftheprocessgroupstoformallyclosetheproject•  PostimplementaKonreview-learnfromexperiencesandapplytofutureprojects

Tip 2 – Use an ADDIE approach

•  ADDIEisanacronymforaninstrucKonaldesignparadigm:•  Analyze–Design–Develop–Implement–Evaluate

•  ADDIEisanumbrellatermthatrepresentsmostofthemanyapproachesusedforcreaKngperformance-basedlearningacKviKes.•  AnADDIEapproachprovidesaframeworktonavigatethecomplexiKesassociatedwithcreaKngperformance-basedlearningaspersonalizedaspossible.•  FocusesonauthenKctasks,complexknowledge,andgenuineproblems,leadingtohighfidelitybetweenlearningenvironmentsandactualworkse^ngs.•  ConsidersalearningacKvitytobeanintervenKontocreateachangeinperformance.

Tip 2 – Use an ADDIE approach

• Analyze–IdenKfyaperformanceissueanditsprobablecauses• Design–Determinethedesiredperformanceandassessmentmethods• Develop–Generatelearningresources•  Implement–Preparethelearningenvironmentandengagethestudents•  Evaluate–Assesslearnerperformanceandevaluateimpact

Branch,2010

Project management and ADDIE

ProjectManagement ClustersofTasks ADDIE

X IniKaKngtheproject/AnalyzedatatodeterminePPG X

X PlanningandDesignoftheproject

Design–Determinewhatneedstobelearned X

DeveloplearningacKviKes X

X Execute/ImplementlearningacKviKes X

X Monitoring/Controlling

EvaluaKon X

X Closing

Tip 2 – Use an ADDIE approach

Transforma7on of Outcomes Framework

Tip 2 – Use an ADDIE approach

•  Tip1Projectmanagement•  Tip2ADDIEAnalysis•  Tip3Startwiththeendinmind•  Tip4Alignthreeassessments•  Tip5Threelevelsofneedsassessment•  Tip6DeterminecauseDesign-Tip7Whatdotheyknow?Develop•  Tip8Organizelearningfortransfer•  Tip9ProvidemoreKmeandfocusImplement•  Tip10FacultyDevelopment•  Tip11Monitorlearning

Evaluate•  Tip12Assesslearning•  Tip13Weredesiredresultsachieved•  Tip14Whatworked

Tip 3 – Start with the end in mind

•  StartwithadocumentedgapinthehealthofpaKentsinapopulaKon,community,orhealthsystemorpaKentsmanagedbyagroupofspecialists.•  ArKculatethegapintermsofwhathealthstatusisandwhatitcouldorshouldbe.•  Desiredresults:reducethesizeoforcompletelyeliminatethegap.

Tip 4 – Align 3 stages of assessments

•  TherearethreestagesintheconKnuumofassessment:•  Needsassessmentdeterminesthepriorknowledgeoflearnersinthesubjectareaofthecourse,andwhatthelearnersshouldbeabletoknowanddoaQercompleKngthecourse(goalsofthecourse).•  FormaKveassessmentisfeedbackandguidanceprovidedconKnuouslythroughoutcoursestohelpastudentunderstandhowheorsheisprogressingtowardsaccomplishingthegoalsandobjecKvesofthecourseandwhatheorsheneedstodotoconKnuetoprogress.•  SummaKveassessmentdeterminesiflearnersachievetheexpectedlevelofperformance(goalsofthecourse).

• Alignmentnecessarytoreachgoals(desiredresults)ofthecourse.(Squires,2012;Squires,2005).

Tip 4 – Align 3 stages of assessments

Tip 5 – 3 levels of needs assessment

DesignSpecificaKons

ClinicianCompetence

ClinicalPerformance

Pa0entHealth

•  PaKenthealth:ThehealthofpaKentsinapopulaKon,community,orhealthsystemorpaKentsmanagedbyagroupofspecialists•  Clinicianperformance:TheperformanceofatargetaudienceofclinicianswhocareforthesepaKents.•  Thecompetenceand/orknowledgeofatargetaudienceofclinicians.

Tip 6 – Determine cause of sub-op7mal care

•  IdenKfythecauseofthegapinpaKenthealthusingfishboneordriverdiagramprocess.

Fishbone Diagram (cause and effect) Physicians

Pa0entsTechnology

Micro-system

HighHbA1c

Poor process

No team

Disorganized

Inconsistent practice

Knowledge

Low literacy

No Support

Lack of integrated EMR

No medicine reconciliation

Tip 6 – Determine cause of sub-op7mal care

•  IdenKfythecauseofthegapinpaKenthealthusingfishboneordriverdiagramprocess.•  IfacauseisidenKfiedthatisrelatedtoclinicianperformance,workwithsubjectmajerexpertsandQIconsultantsto•  DeterminethespecificbehaviorsthatcliniciansarenotperformingornotperformingatanappropriatelevelbyaudiKngasampleofthecharts.•  DetermineappropriatelevelforeachbehavioridenKfiedbyconsulKngcurrentpracKceguidelines,recentresearchfindings,and/orlocalconsensusbestpracKce(clinicalpathways).•  PrioriKzebehaviorsthataremostdistantfromopKmaland/orarelikelytohavethegreatestimpactonpaKenthealthiftheyarenotperformed.

Step1

Step5Steps2,3,4

Tip 7 – What do they know?

•  Importanceof“priorknowledge”forplanningcontent.•  IntervieworsurveySMEsabouttheprioriKzedbehaviors.

•  TheprioriKzedbehaviorsarecomplextasksinwhichperformancerequirestheintegrateduseofbothcontrolled(conscious,conceptual)andautomated(unconscious,proceduralorstrategic)knowledge.•  SMEswillhelpplannersdeterminetheclinicalandbasicscienceknowledgeaswellasthethoughtprocessesthatunderlietheprioriKzedbehaviors.•  TheoutcomeofworkingwithSMEswouldbeadescripKonofperformanceobjecKvesthatcouldbeusedtoplanscenariostoassessthecompetenceofcliniciansinthetargetaudience.

Ambrose,et.al.,2010Branch,2010

Tip 8 – Organize learning for transfer

Whatistransfer?•  TradiKonaldefiniKon:effecKvelyandconKnuouslyretrievingwhatwaslearnedinonese^ngandapplyingitinanotherse^ng(Perkins,1992).•  EmergingdefiniKon:effecKvelyandconKnuouslyretrievingwhatwaslearnedinoneexperienceandlearninghowtousewhathasbeenlearnedinsimilar(rouKne)ordifferent(adapKve)experiences.(BransfordandSchwartz,1999,2005)•  FacilitatedbypreparaKonforfuturelearning,e.g.teachingclinicalcontentwithunderlyingbasicscienceconcepts(MylopolousandWoods,2016,HatanoandInagaki,1986).•  Retrievingisfacilitatedby(Brown,et.al.,2015;CerveroandGaines,2015)

•  AcKveandvariedlearningmethods•  MulKple,variedexposurestocontent•  PracKceandfeedback•  FrequenttesKngasexploratorylearningexercises•  Challenge

Tip 8 – Organize learning for transfer

•  Therealissueisnotwhybasicscience,orhowmuchofit,isneededduringformalmedicaleducaKon.•  Instead,thequesKonishowthisscienceistobemuchmoreeffecKvelytaught,boththroughouttheenKre7-to10-yearperiodofmedicalschool,internship,andresidency,andintheconKnuingeducaKonrequiredofphysicians.

CookeM,2010.

Tip 8 – Organize learning for transfer

•  PredisposingacKviKestocreateorreinforcea“teachablemoment”.•  AcKvatepriorknowledgewithcasescenarioexercises•  ComparewithsummaryofparKcipantperformancedata

•  EnablingacKviKestofacilitatelearningtoaddressthe“teachablemoment”.•  PresentaKon(contentandconcepts)•  WorkedexampleordemonstraKon(contentandconcepts)•  DeliberatepracKcewithscaffolding(contentandconcepts)•  Expertfeedbackwithcoaching(contentandconcepts)

•  ReinforcingacKviKestoassistinretrievalofwhatwaslearned.•  Reminders•  ConKnuousfeedbackofparKcipantperformancedata•  Commitmenttochangeandfollow-up•  Monthlyscenarioexercises

Tip 9 – Provide more 7me and focus

•  StrengtheningretrievalcapabilityneedsmoreKmethanissetasideinatypicalCMEacKvity.•  AcKveandvariedlearningmethods•  MulKple,variedexposurestocontent•  DeliberatepracKcewithscaffolding–the10,000hourcontroversy(Ericsson,2016)•  Expertfeedbackwithcoaching•  FrequenttesKngasexploratorylearningexercises

•  Typicalday-longCMEacKvity:8didacKclectureswithQ&A•  Suggestednewday-longCMEacKvity:

•  Twotopics:oneinthemorning;oneintheaQernoon•  Inquiry-basedlearningformat

Tip 9 – Provide more 7me and focus

KleinandHarris,2007

ChallengecycleLegacycycle

Tip 10 – Help faculty with new approaches

•  FacultywillbechallengedbythechangesthathavebeensuggestedfortheCME/CPDspace.•  Onesetofthesenewanddifferentapproachesdrawontheprinciplesoftheemergingfieldofthelearningsciences•  FocusmoreonhelpingpeoplelearnthanontradiKonalteacher-directedapproaches(Bransford,Brown,&Cocking,2000;Sawyer,2014).

•  ManyclinicianswhoarecurrentlyinvolvedinhealthprofessionseducaKonhavenotreceivedtraininginthesenewerlearner-basedapproaches(Srinivasanetal.,2011).

•  AnothersetofnewanddifferentapproachesfocusonenhancingpaKentoutcomesthroughimprovingclinicianperformance.•  Emphasisonusingperformancedatainplanning.(Moore,et.al,1994)•  IncreasinguseofQImethodologyandcollaboraKonwithQIstaff.(Moore,1995)

•  FacultymustbepreparedforthenewapproachestohealthcareandeducaKonalpracKce.(Davis,et.al.,2017)

Tip 11 – Monitor: Are they learning?

•  AnimportantcomponentofenablingacKviKesisformaKveassessment.•  FormaKvefeedbackisintendedtogeneratefeedbackonperformancetoimproveandacceleratelearning.(NicolandMcFarlane-Dick,2006)•  ThecombinaKonofdeliberatepracKceandexpertfeedbackwithcoachingishowformaKvefeedbackisaccomplished.•  CharacterisKcsofformaKvefeedbackthatshouldbefollowed(Trumbull,2013):

•  Keepclearcriteriaforwhatdefinesgoodperformance•  Encouragelearners’self-reflecKon•  Givelearnersdetailed,acKonablefeedback•  Encouragefacultyandpeerdialoguearoundlearning•  PromoteposiKvemoKvaKonalbeliefsandself-esteem•  ProvideopportuniKestoclosethegapbetweencurrentanddesiredperformance•  CollectinformaKonwhichcanbeusedtohelpshapeteaching

Tip 4 – Align 3 stages of assessments

Tip 12 - Did they learn?

• Answeredbyassessment.•  Theterms“assessment”and“evaluaKon”areoQenusedinterchangeablyineducaKondiscourse.• Butforprecision…

•  “assessment”referstotheperformanceoflearners•  “evaluaKon”referstotheperformanceofacourse,curriculum,orprogram.

• Assessment-anysystemaKcmethodofobtainingdataanddrawinginferencesfromthedatatodeterminewhatanindividualorgrouphaslearnedasaresultofengaginginalearningacKvity(DowningandYudkowsky,2009).

Tip 4 – Align 3 stages of assessments

Tip 8 – Organize learning for transfer •  PredisposingacKviKestocreateorreinforcea“teachablemoment”.

•  AcKvatepriorknowledgewithcasescenarioexercises•  ComparewithsummaryofparKcipantperformancedata

•  EnablingacKviKestofacilitatelearningtoaddressthe“teachablemoment”.•  PresentaKon(contentandconcepts)•  WorkedexampleordemonstraKon•  Deliberateprac0cewithscaffolding–observeprac0cewithrubric.•  Expertfeedbackwithcoaching–gradebasedonobserva0on.

•  ReinforcingacKviKestoassistinretrievalofwhatwaslearned.•  Reminders•  ConKnuousfeedbackofparKcipantperformancedata•  Commitmenttochangeandfollow-up•  Monthlyscenarioexercises

Whenalearnerisready.

Tip 12 – Did they learn?

•  Acourseassessmentplanshouldbedevelopedassoonastheexpectedresultsofacoursearedetermined.•  BlueprinKngisanapproachthatcouldbeusedtoalignlearningobjecKves,learningacKviKes,andassessment(Coderre,Woloschuk,anMcLaughlin,2009).•  Acourseblueprintistypicallyamatrixwithfourcolumns:onecolumnforlearningobjecKves;oneforlearningacKviKes;oneforcontentandscaffolding)andoneforassessmentmethods.Eachrowinablueprintmatrixshouldinclude•  AlearningobjecKverelatedtothedesiredresult•  LearningacKviKesthatwillmostlikelycontributetothelearner’sopportunitytoaccomplishtheobjecKve

•  ContenttobecoveredandthescaffoldsforpracKceexercises•  TheassessmentmethodthatwillmostlikelyprovideevidencethatthelearneraccomplishedthelearningobjecKve.

Tip 12 – Did they learn?

•  Assessmentatthe“knowswhattodo”levelisaccomplishedbytesKngtherecalloffacts.

•  Assessmentatthe“knowshowtodoit”levelisaccomplishedbycase-basedmulKplechoiceitems,essayquesKonsororalpresentaKons.

•  Assessmentatthe“showshowtodoit”levelisaccomplishedbyrubric-basedobservaKonwithstandardizedpaKentsorsmallgroupcasediscussion.

•  Assessmentatthe“doesit”levelisbestaccomplishedthroughchartaudit.

TherearemanypotenKalwaystoassesslearning.GeorgeMiller’spyramidisausefulwayofcategorizingmethodsofassessment(Miller,1990)

Tip 13 - Were desired results achieved?

Desiredresults:reducethesizeoforcompletelyeliminatethegap.• Didtheclinicianslearn?• Didthecliniciansmeettheperformancegoalsofthecourse?•  Competence(ObservaKon)•  PracKce(ChartAudit)•  DidpaKenthealthstatusimprove?

Tip 14 – What worked? What didn’t work?

•  ProgramevaluaKon:systemaKccollecKonandanalysisofinformaKonrelatedtothedesign,implementaKon,andoutcomesofaprogramforthepurposesofmonitoringandimprovingtheprogram.•  BestusedforaprogramofCMEcourses,butcanbeusedforasinglecourse.•  Aggregateddatafromtheassessmentoflearnersisusefulbutnotenough.•  ThepurposeofprogramevaluaKonistoanalyzetheCME/CPDsystemandallofitsinteracKngparts.• ManyothersourcesofinformaKonarenecessaryformakingjudgmentsaboutthequalityofaCME/CPDprogramandrecommendaKonsforimprovement.

W.K.KelloggFoundaKon,2004

Tip 14 – What worked? What didn’t work?

•  Alogicmodelshouldbeconstructedasearlyinthelearningprojectaspossible,especiallywhenthelearningprojectgoalisdetermined.(Kellogg,2004)•  Usually,itsstartswith“theendinmind”liketheADDIEapproach.•  Alogicmodelisatheory-basedapproach–“Ifthisisavailableandthisisdone,thenthatislikelytooccur”.•  EvaluaKonquesKonsateachlevelofthelogicmodel:

•  Didwhatwassupposedtohappen,happen?•  Didithappenasplanned?•  5whys-fiveiteraKonsofaskingwhyisgenerallysufficienttoidenKfyacause.

•  RealistevaluaKon–whatworkedforwhomunderwhatcircumstances.(Pawsonet.al.,2005)

References

References

Thanks!