implementing an effective academic remediation programapds.org/arcs/arcs 2008/implementing an...
Post on 14-Aug-2018
222 Views
Preview:
TRANSCRIPT
Implementing an Implementing an Effective Academic Effective Academic
Remediation Program: Remediation Program: Results and Results and
ConsequencesConsequences
Karen SharrockKaren SharrockYork Hospital, York, PAYork Hospital, York, PA
Illustration by David Harbaugh1
He never recovered from the most recent site visit
WebsterWebster’’s Dictionary:s Dictionary:EffectiveEffective: producing a decided, decisive, or : producing a decided, decisive, or desired outcomedesired outcomeAcademicAcademic: based on formal study especially at an : based on formal study especially at an institution of higher learninginstitution of higher learningRemediationRemediation: concerned with the correction of : concerned with the correction of faulty study habits and the raising of a pupil's faulty study habits and the raising of a pupil's general competence.general competence.
Background: Standards to Demonstrate Background: Standards to Demonstrate Medical Knowledge CompetencyMedical Knowledge Competency
York Hospital Surgery Residency Program considers poor York Hospital Surgery Residency Program considers poor performance on the ABSITE examination to be an performance on the ABSITE examination to be an indication that a resident has not acquired an adequate indication that a resident has not acquired an adequate knowledge baseknowledge base
A minimum total test score of 67% (on the ABSite) is A minimum total test score of 67% (on the ABSite) is required by each resident to remain in good academic required by each resident to remain in good academic standing. Those residents who score less than standing. Those residents who score less than 67%67%correct as the total test (TT) score will be placed on correct as the total test (TT) score will be placed on academic remediation, which includes a concentrated academic remediation, which includes a concentrated course of study devised by the program director.course of study devised by the program director.
Inclusion CriteriaInclusion Criteria
The total test score value of 67% was chosen The total test score value of 67% was chosen becausebecause
Previous compilation of remediation limits in the Previous compilation of remediation limits in the literatureliterature22
Reports of an increased failure rate on following Reports of an increased failure rate on following ABSite exams for those scoring less than the 30ABSite exams for those scoring less than the 30thth
percentilepercentile3.3.
New RRC requirements New RRC requirements
20052005--2006:2006:All residents: Basic Science Curriculum, as well as other All residents: Basic Science Curriculum, as well as other
interdisciplinary conferences and mock oral conferences.interdisciplinary conferences and mock oral conferences.No remediation plan No remediation plan
20062006--20072007As a result of Jan 2006 ABSite results, a remediation plan was As a result of Jan 2006 ABSite results, a remediation plan was
developed in February 2006 and implemented for 4 residents developed in February 2006 and implemented for 4 residents out of 15 categoricalout of 15 categorical
20072007--20082008The success of the previous plan year indicated that we should The success of the previous plan year indicated that we should
modify the basic science curriculum for all residents to modify the basic science curriculum for all residents to incorporate remediation plan componentsincorporate remediation plan components
Residents (4 of 15) who did not meet the minimum cutoff Residents (4 of 15) who did not meet the minimum cutoff score were assigned to the full remediation program. score were assigned to the full remediation program.
Basic Science CurriculumBasic Science CurriculumCY 2005 (ABSite 2006)CY 2005 (ABSite 2006)
•• This conference covers the material present in a This conference covers the material present in a programprogram--chosen basic science textbookchosen basic science textbook
•• WeeklyWeekly
•• 5 question quiz based on the reading administered to 5 question quiz based on the reading administered to assure that the residents have reviewed the material. assure that the residents have reviewed the material.
•• 100% attendance expected.100% attendance expected.
Results 2006Results 2006
Departmental mean = 60.80, departmental median = 63
Remediation CurriculumRemediation Curriculum
Those residents on remediation will: Those residents on remediation will: Complete Basic Science CurriculumComplete Basic Science Curriculum
In addition, supplemental reading will be required In addition, supplemental reading will be required from three additional textsfrom three additional texts
SESAP 13 must be purchased by the residentSESAP 13 must be purchased by the resident
Supplemental monthly exams.Supplemental monthly exams.
Remediation Requirements:Remediation Requirements:
Must receive a Must receive a ““passpass”” on all rotationson all rotations
In House Tests/Quizzes average >80%In House Tests/Quizzes average >80%
Next YearNext Year’’s ABSite score must be above 67% s ABSite score must be above 67% total test score.total test score.
Remediation Requirements (contRemediation Requirements (cont’’d)d)During this period of academic remediation, the During this period of academic remediation, the resident will be offered the following at no cost:resident will be offered the following at no cost:
General medical examinationGeneral medical examinationPsychiatric evaluationPsychiatric evaluationPsychological testingPsychological testingTestTest--taking skills (learning style) assessment taking skills (learning style) assessment and coaching.and coaching.
Consequences of Remediation:Consequences of Remediation:
Lost PrivilegesLost Privileges
Vacation Scheduling Priority deferredVacation Scheduling Priority deferred
Basic Science CurriculumBasic Science CurriculumCY 2006CY 2006
•• This conference covers the material present in a This conference covers the material present in a programprogram--chosen basic science textbook chosen basic science textbook
•• WeeklyWeekly
•• 5 question quiz based on the reading administered to 5 question quiz based on the reading administered to assure that the residents have reviewed the material assure that the residents have reviewed the material
•• 100% attendance expected.100% attendance expected.
Results: Results: CY 05 (ABSite 06) vs. CY 06 (ABSite 07):CY 05 (ABSite 06) vs. CY 06 (ABSite 07):
40
45
50
55
60
65
70
Res A Res B Res C Res D
ABSite 2006ABSite 2007
Residents on remediation in 2006Residents on remediation in 2006
Basic Science CurriculumBasic Science CurriculumCalendar Year 2007Calendar Year 2007
Basic Science Curriculum was assigned to all Basic Science Curriculum was assigned to all residents in the programresidents in the program
Residents on remediation: monthly 20 question Residents on remediation: monthly 20 question examexam
Addition: biAddition: bi--monthly 20 question exam given to monthly 20 question exam given to all residents.all residents.
404550
5560
6570
758085
Res E Res B Res F Res G
ABSite 2007ABSite 2008
Results: Results: CY 06 (ABSite 07) vs. CY 07 (ABSite 08):CY 06 (ABSite 07) vs. CY 07 (ABSite 08):
Residents on remediation in 2007Residents on remediation in 2007
If all requirements are met If all requirements are met ANDAND the resident the resident scores at or above a scores at or above a 67% TT score67% TT score on the next on the next
ABSITE, then s/he will be removed from ABSITE, then s/he will be removed from academic remediation.academic remediation.
HoweverHowever……
Consequences of NonConsequences of Non--compliance:compliance:
Probation or DismissalProbation or Dismissal
Although the ABSITE examination will not be used Although the ABSITE examination will not be used as the sole measure of promotion, substandard as the sole measure of promotion, substandard scores, scores, despite attempts at remediationdespite attempts at remediation, combined , combined with poor performance in other areas will lead to with poor performance in other areas will lead to academic probation and may be grounds for academic probation and may be grounds for dismissal from the program dismissal from the program
(YH Residency Handbook).(YH Residency Handbook).
From the ACGME:From the ACGME:Surgery RRC, Program Requirements,
Section V.1.A.1.e
Assessment should specifically monitor the resident's knowledge by use of a formal exam such as the American Board of Surgery In Training Examination (ABSITE) or other cognitive exams.
(ABSite) Test results should not be the sole criterion of resident knowledge, and
should not be used as the sole criterion for promotion to a subsequent PG level.
Have the resident repeat a year?Have the resident repeat a year?
DIO approval DIO approval -- fundingfunding
Needs RRC approval Needs RRC approval
ABS ABS ““encourages encourages retention of residents in a program as long as is required to achieve the necessary level of qualifications (ABS booklet, 2008).
Program ComparisonProgram Comparison
2008 2007 2006median 64.50 64.00 63.00average 66.08 64.33 60.80
Comparison by PGYComparison by PGY
n/an/a
n/an/a
n/an/a
52.7552.75
58.558.5
20052005
n/an/a
n/an/a
55.355.3
64.2564.25
59.559.5
20062006New BS New BS
curriculumcurriculum
7676n/an/a11
6767676722
60.3360.3365.365.333
707060.560.544
666665.565.555
2008200820072007New New
remediationremediation
PGY PGY (as of 2008)(as of 2008)
Overall FindingsOverall Findings
All residents on remediation demonstrated All residents on remediation demonstrated improvement, but some still did not meet the improvement, but some still did not meet the minimum score of 67%minimum score of 67%Previous literature showing documented success Previous literature showing documented success in other programsin other programsd/td/t improved individual weekly reading effort, improved individual weekly reading effort, curriculum change, fear of consequences?curriculum change, fear of consequences?Newly discovered Newly discovered -- PGY3 dip.PGY3 dip.
Results 2006Results 2006
Departmental mean = 60.80, departmental median = 63
Results 2007Results 2007
50
55
60
65
70
75
PGY5 PGY4 PGY3 PGY2 PGY1
AVG
Median
Departmental mean = 64.33, departmental median = 64
Results 2008Results 2008
Departmental mean = 66.08, departmental median = 64.5Departmental mean = 66.08, departmental median = 64.5
50
55
60
65
70
75
80
PGY5 PGY4 PGY3 PGY2 PGY1
AVG
Median
Goals:Goals:
Initial purpose of ABSiteInitial purpose of ABSiteMeasure knowledgeMeasure knowledgeIdentify deficiencies and potential weaknessesIdentify deficiencies and potential weaknesses
Ultimate effectiveness of remediation should be Ultimate effectiveness of remediation should be determined by pass rate on QE.determined by pass rate on QE.
Review of ABS QE Pass RateReview of ABS QE Pass RateFirst attempt pass rateFirst attempt pass rate
67%
100% 100%
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
2005 2006 2007
Correlation of Competencies:Correlation of Competencies:
Deficiency in medical knowledge affects other Deficiency in medical knowledge affects other competencies as well:competencies as well:
Patient carePatient careICSICSProfessionalismProfessionalismSystems Based PracticeSystems Based PracticeEvidence Based MedicineEvidence Based Medicine
Going forwardGoing forward……
More stringent reliable/valid tests (online surgical More stringent reliable/valid tests (online surgical database of texts/questions)database of texts/questions)
Continue to monitor 1Continue to monitor 1stst time pass rate, ABS QEtime pass rate, ABS QE
Perhaps development of transitional curriculum (PGY2 Perhaps development of transitional curriculum (PGY2 to PGY3)to PGY3)
Continue remediation program.Continue remediation program.
ReferencesReferences
HarbaughHarbaugh, D. , D. Residency Program Alert . Residency Program Alert . HC Pro HC Pro 2007. HC Pro. 30 March 2008 2007. HC Pro. 30 March 2008 http://www.hcpro.com/content/204657.pdfhttp://www.hcpro.com/content/204657.pdf
HarthunHarthun NL. NL. SchirmerSchirmer BD, BD, SanfeySanfey H. Remediation H. Remediation of low ABSITE scores. of low ABSITE scores. CurrSurgCurrSurg. 2005;62:539. 2005;62:539--542542
1.1. American Board of Surgery. American Board of Surgery. Interpretation of Results of Interpretation of Results of the American Board of Surgery Inthe American Board of Surgery In--Training/Basic Science Training/Basic Science Examination.Examination. Philadelphia, PA: American Board of Philadelphia, PA: American Board of Surgery.; 2005Surgery.; 2005
Questions?Questions?
HUMOR FOR THE DAY HUMOR FOR THE DAY ……
top related