the acgme self-study and 10- year site visit 2015 fall program director retreat september 24, 2015...

The ACGME Self-Study and 10-Year Site Visit

2015 Fall Program Director RetreatSeptember 24, 2015

Alan J. Smith, PhD, MEdAssistant Dean and Director for GMEACGME Designated Institutional Official

© 2015 Accreditation Council for Graduate Medical Education

Disclosure

Adapted from original ACGME Webinar presentation by Susan Guralick, MD, DIO, Winthrop University Hospital, and Ingrid Philibert, PhD, MBA, ACGME Senior Vice President for Field Activities, June 18, 2015.


Objectives• Discuss elements of the self-study and ongoing program

evaluation and improvement on which it is based• Explain the sequence and components of the self-study

and the 10-year site visit• Describe program and institutional practices to assist with

self-study preparation and execution• Explain the components of the 10-year accreditation site

visit and Review Committee review


Self-Study Scope• Assess program performance and ongoing improvement

efforts• Facilitate improvement in areas already in substantial compliance

with the accreditation standards

• Initial period: since last accreditation review• Ultimately a 10-year interval

• Review improvement activities, successes achieved, and areas in need of improvement• Based on successive Annual Program Evaluations, ACGME

Resident and Faculty Survey data, and other relevant information, stakeholder input


Strengths and Areas for Improvement• Strengths

• Important to acknowledge and celebrate

• What should definitely be continued, especially in an environment of limited resources

• Areas for Improvement identified by:• Citations, areas for improvement, and other information from

ACGME

• Annual Program Evaluation

• Other program/institutional data sources

• Expected: Longitudinal tracking of strengths and areas for improvement through successive Annual Program Evaluations


Improvement Viewed in Context: Program Aims, Opportunities and Threats• Aims as a way to differentiate programs

• Set and reassessed as part of the Annual Program Evaluation

• Context: Opportunities and Threats• External attractive factors that, if acted upon, will contribute to the

program flourishing or may have a negative effect

• While the program cannot fully control them, it is beneficial to have plans to mitigate their effect


Defining Program Aims• Aims are set and revisited/revised as part of the Annual

Program Evaluation• Relevant considerations

• Who are our residents/fellows?

• What do we prepare them for?• Fellowship

• Academic practice

• Leadership and other roles

• Who are the patients/populations we care for?


Benefits of a Focus on Program Aims• Suggests a relevant dimension of the program:

• Types of graduates produced for specific community needs, practice settings and roles

• Allows for a more “tailored” approach to creating a learning environment• Focus on specific aims can produce highly desirable graduates

that match patient and healthcare system needs 1

• Enhances the focus on functional capabilities of graduating residents• Fits with a milestones-based approach to assessment

1Hodges BD.”A Tea-Steeping or i-Doc Model for Medical Education?.” Acad Med85(9) Sept 2010, pp. S34-S44.


Defining “Opportunities”• Opportunities are external attractive factors that, if acted

upon, will contribute to the program flourishing• What are capabilities for further evolving the program? How can

the program capitalize on them?

• Has there been recent change in the program’s context that creates an opportunity?

• Are these opportunities ongoing, or is there a narrow window for them? How critical is the timing?


Defining “Threats”• Threats include external factors that affect the program• While the program cannot fully control them, it is

beneficial to have plans to mitigate their effect• What factors beyond the program’s control may place it at risk?

• What are the changes in residents’ specialty choice, regulation, or other factors that may affect the future success of the program?

• Are there challenges or unfavorable trends in immediate context that may affect the program?• E.g., faculty burdened with heavy clinical load that prevents

effective teaching and mentorship


Coordinated Self-Study for Core and Subspecialty Programs• Coordination of curriculum and program resources

• Needs of core and subspecialty programs considered collectively

• Subspecialties can access core resources

• Core oversight of fellowships

• Access common strengths, areas for improvement• Coordinated action plans for areas for improvement

• Increase efficiency• Less time and resources spent, coordinated collection and review

of data


Self-Study Process• Who should conduct the self study?

• Not defined by ACGME

• Members of the Program Evaluation Committee (PEC) are the logical choice

• PEC membership may be expanded (beyond required two faculty members and one resident)

• Data gathering• Many potential sources, including ACGME Resident and Faculty

Surveys, program and institutional data

• Important: stakeholder (residents, faculty and relevant others) participation, input and perspective

• Data as evidence to support conclusions


Self-Study Data Gathering• Annual Program Evaluation data, ACGME Resident and

Faculty Survey data, other program and institutional data

• Focus on data gathering as a learning experience

• Evaluate strengths and areas for improvement

• Explore opportunities and threats

• Reflect stakeholder (residents, faculty, and relevant others) participation, input, and perspective

• Data as evidence to support conclusions


Tracking Improvements• Design and implement solutions

• Identify individual or group that will be responsible

• Identify and secure resources

• Timeline

• Follow-up is key: endure all issues addressed• Documentation to facilitate ongoing tracking

• A simple spreadsheet recording improvements achieved and ongoing (See handout 3)

• Record improvements over multiple years


Resident Participation• Resident participation critical:

• They are the beneficiaries of the educational program

• They have first hand knowledge of areas that need improvement

• Double benefit:• Residents help improve their own education

• Resident participation in “educational QI effort” can be used to meet the requirement for resident involvement in quality and safety improvement


Self-Study and the Self-Study Summary

• The program conducts its self-study• After the self-study, the program uploads the Self-Study

Summary in ADS• Information on areas for improvement identified in the self-study

are not included in the Summary

• Timing consideration:• Programs expected to upload the self-study summary in the month

the Review Committee indicated for their first site visit in NAS

• Self-Study Site Visit scheduled 12-18 months after uploading the Self-Study Summary in ADS


The Self-Study Summary• Brief document prepared by the program, uploaded in ADS

(See handout 1)• ACGME Template: 2300 words (~4-5 pages) for core program,

less for small subspecialty programs• Sections: Key Self-Study dimensions

• Aims

• Opportunities and Threats

• Self-Study process• Who was involved, how were data collected and interpreted

• How is ongoing improvement achieved through sequential Annual Program Evaluations

• Omitted by design: information on strengths and areas for improvement


The 10-Year Site Visit• A full accreditation site visit

• Review of all applicable requirements

• 12-18-month period is by design, to allow programs to implement improvements

• “Summary of Achievements”• ACGME template uploaded through ADS• ~ 1200 words, describing key improvements

accomplished from the self-study• Program provides information ONLY on the improvements that

were realized from their self-study, no information on areas that have not been resolved

• Program may provide an update to its Self-Study Summary


The 10-Year Site Visit• Site visit by a team• Site visit team provides verbal feedback

• Key strengths and suggestions for improvement

• Site visit team submits a written report through ADS for the Review Committee (RC)


Review Committee Review of the 10-Year Site Visit• Available to the Review Committee:

• ADS data

• Program’s self-study summary and any updates

• Program’s summary of improvements achieved as a result of the self-study (which does not include data on areas still in need of improvement)

• The site visit report from the 10-year (accreditation) site visit

• Review of program aims and context from the self-study summary allows the RC to assess self-study effectiveness• Improvements the program reports are paired with program aims

and context from self-study summary

• Data on improvements achieved are one measure of effectiveness


Review Committee Review of the 10-Year Site Visit• RC provides a Letter of Notification from the 10-year (Full

Accreditation) Site Visit• Citations and areas for improvement

• RC feedback on the self-study taken into consideration:• Program aims and context

• Improvements reported and verified during the 10-year visit

• Effectiveness of the self-study, based on the improvements reported by the program

• Formative feedback (no accreditation impact) for the initial RC assessment of self-study effectiveness


University of Utah Program Self-Study Dates

Program Self-Study Date

Anesthesiology + Pain Management October 2021

Dermatology July 2021

Emergency Medicine September 2016

Family Medicine May 2020

Internal Medicine + subspecialty fellowships May 2019

Medicine/Pediatrics May 2019

Medical Genetics May 2019

Neurology + subspecialty fellowships November 2021

Neurosurgery January 2018

OB/GYN December 2021

Occupational Medicine October 2019


University of Utah Program Self-Study DatesProgram Self-Study Date

Ophthalmology May 2020

Orthopedic Surgery + subspecialty fellowships June 2019

Otolaryngology + subspecialty fellowships February 2020

Pathology + subspecialty fellowships April 2019

Pediatrics + subspecialty fellowships October 2022

PM&R February 2020

Psychiatry + subspecialty fellowships October 2022

Radiology + subspecialty fellowships November 2016

Radiation Oncology March 2018

Surgery + subspecialty fellowships June 2020

Urology + subspecialty fellowships December 2020

Plastic Surgery May 2021


Handouts

1. Self Study Summary Template

2. Suggested Annual Program Evaluation Template

3. Suggested Annual Program Evaluation Action Plan and Follow-up Template

4. JGME article: Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):502-505.


Sample Timeline for IM Core and Subspecialty Programs*

• Day 1. ACGME Notification of Self-Study Submission Due Date• Week 1. Recruit Self-Study Planning Committee

• Program Director and Associate PDs

• Program Coordinator

• Fellowship Program Directors and Associate PDs

• Faculty member

• Resident

• Fellow

• GME staff?

• DIO?

* Guralnick S, et al. The ACGME self-study – an opportunity, not a burden. J Grad Med Educ. 2015;7(3):502-505.



• Weeks 2-5. Weekly meetings of the planning committee• Independent work done by committee members between meetings

• Week 2. View ACGME self-study webinar (acgme.org)• Review self-study process

• Develop self-study timeline for the program

• Create a self-study planning committee worksheet

• Week 3. Create/review Self-Study SWOT Template• Add/edit items as relevant to local/individual programs




• Week 4. Create or adapt self-study survey draft documents• Simple surveys: faculty, residents, fellows, coordinators, nurses &

other relevant staff, others

• Week 5. Define the focus group process• Week 6. Present Self-Study Planning work to

GMEC/Discussion



Sample Timeline for IM Core and Subspecialty Programs*• Weeks 6-8. Formation of Program-Specific Self-Study Committees

• Should include PEC members plus others as appropriate (e.g., CCC members, Program Coordinator, Chief Residents, Residents/Fellows, Medical Educator

• Week 7. Surveys Distributed

• Weeks 7-8. Survey Data Collected

• Weeks 8-9. Program-specific committees meet• Program-specific self-study planning

• Define program aims and perform SWOT analysis

• Weeks 9-11. Survey groups participate in focus groups to delve into/clarify information gathered in surveys




• Weeks 14-17. Self-Study Program-Specific Committees Meet Weekly for:• Data analysis

• SWOT analysis review with additional data collected

• Perform full self-self study

• Create self-study document

• Weeks 18-19. Write Self-Study document for submission to ACGME

• Weeks 19-20. Finalize Self-Study documents for submission.


THANKS!

Resident/Fellow Schedules in E*Value


Truth or Consequences . . .

• Providing an accurate resident / fellow training schedule to the institution is a foundational program responsibility

• Accurate, timely training schedules in E*Value are essential for:• Affiliate billing (over $20M yearly)• CMS/Medicare Reimbursement (over $30M yearly)• Information/Data Integrity with Affiliates & Government


The Problem• GME Office must routinely resolve 50-75 missing FTEs at

month-end• Discrepancies and late reporting of Leave of Absences,

vacation days, or changes in training location are rampant• GME dedicating an employee 3-4 full days each month to

audit and resolve schedule omissions, conflicts and mistakes in E*Value

• Unprofessional, embarrassing after-the-fact corrections to affiliate bills are commonplace due to scheduling errors


Program Director Support Needed• Ensure that your Program Coordinator / Manager knows

where your people will be training before the rotation begins

• Ensure that training schedules are confirmed and input in E*Value by the 15th of each month

• Promptly report any changes that occur after the 15th to GME

• Encourage your Program Coordinator / Manager to utilize the monthly schedule verification checklist provided by GME and to double-check schedules before the 15th


We’d really rather not go there, but …

The University currently assesses a charge to departments or individuals for payroll corrections or missing scheduled training classes. Please help us avoid implementing this type of ‘penalty’ system.

Thank You!

the acgme self-study and 10- year site visit 2015 fall program director retreat september 24, 2015...

Documents

program aimssuggests

accreditation council

ongoing program evaluation

accreditation reviewultimately

year accreditation site

acgme selfstudy

acgme resident

relevant information