using an internal gme house staff survey as an early ......introduction of the stanford housestaff...

Using an Internal GME House Staff Survey as

An Early Warning System for the ACGME Resident SurveyApril 30, 2018

Nancy Piro, PhD

Sr Program Manager/Education Specialist, Stanford GME

Ann Dohn, MA

GME Director & DIO Stanford GME

❖ Nancy Piro, PhD❖ Ann Dohn, MA

Neither of the above speakers have any conflicts of interest to report.

Agenda

I. Background - Stanford GME Housestaff Survey

II. Analysis of the Construct of the Stanford GME Housestaff Survey

III. Predictions of ACGME Resident Survey Results based on the Internal GME Housestaff Survey

IV. Group Discussion

V. Stanford Program’s Scorecard System

VI. Discussion & Questions

Background: Graduate Medical Education (GME) House Staff Survey

History

Purpose

Questions & Framework

Face Validity

Delivery

Analysis & Reporting

Background: Stanford Housestaff Survey – Historical Perspective

An institutional-wide internal survey, with IRB approval, used since 2008 which is distributed via email to all housestaff officers.

GME Office delivers the survey annually in mid November to all Stanford programs both ACGME and non-ACGME.

All responses are confidential and anonymous. Programs with >3 responses receive aggregated reports.

Background: Stanford Housestaff Survey – Purpose

Initially designed to gather resident and fellow opinions on the educational quality of their training programs, and to find out about other GME-related issues i.e., how GME needed to function more effectively in meeting our trainees’ needs. “Voice of the Resident”

As the survey has evolved, we saw the potential for it to serve as an Early Warning System for programs to detect potential concerns that could be understood/clarified before the ACGME Resident Survey.

Provides programs with aggregated data for review in their Annual Program Evaluations (APEs) and the Annual Institutional Review (AIR)

Background: Stanford Housestaff Survey – Questions & Framework

Survey questions parallel the six (6) main domains from the ACGME Resident Survey: Work Hours, Educational Content, Faculty, Evaluation, Resources, Patient Safety/Team Work.

Questions also focus on the institutional areas needing improvement delineated in prior Clinical Learning Environment Review (CLER) site visit reports.

Our focus is on critical questions – Total Survey Time ~ 5-7 minutes maximum.

Additional Questions:

−Added in alternate years

−For research and program quality control purpose

−Up to 5 questions related to GME issues, such as wellbeing, mentorship, etc.

Background: Stanford Housestaff Survey – Questions

Core Quantitative Questions (1-6 Likert scale):

1. Overall, how satisfied are you with the training you have received in your current program at Stanford?

2. I would recommend my training program here at Stanford to others.

3. Faculty in my program are successful teachers.

4. Faculty in my program spend sufficient time supervising the residents or fellows in the program.

5. Faculty in my program encourage me to ask questions on a regular basis.

6. Faculty in my program encourage me to be open/honest with them.

7. I have an adequate amount of "protected time" to focus on my educational needs.

8. I think my program appropriately balances the need to fulfill service obligations to the hospital with clinical education.

9. My program is organized to meet my educational needs.

10. My overall patient load (the quantity of patients that I see) is appropriate.

11. The variety of patients I see is sufficient for meeting my educational needs.

12. In your program, have you had (will you have) the opportunity to participate in any projects related to Quality Improvement (QI)? (Yes/No)

Background: Stanford Housestaff Survey – Questions Cont’d

Core Qualitative Questions:

1. Please explain why you disagree with "I would recommend my training program here at Stanford to others".

2. What did we miss? Are there any other areas of concern or suggestions (GME Office, orientation, call rooms, security at training sites, etc.)?

Question Response Depiction in Reports for Programs

GME House Staff Question Coding Scheme (1)

GME House Staff Question Coding Scheme (2)

GME House Staff Survey Process

Questions Determined

Review by IRB

Survey Distributed

Data Analyzed

Research Begins

Reports

& Scorecards Developed & Shared

Background: Stanford Housestaff Survey – Face Validity

The Stanford GME Housestaff Survey was reviewed, revised and approved by program directors from core residency programs.

GME House Staff emails/survey cover letters

IRB Approval Letter

Introduction of the Stanford Housestaff Survey – Delivery

The GME Office sends an email with the cover letter containing a hyperlink to the survey to all Housestaff Officers.

Participants are informed that all of their responses are totally anonymous and do not include IP addresses, locations of completion, or any other identification information.

Introduction of the Stanford Housestaff Survey – Analysis & Reporting

Responses are aggregated by programs for reporting

Only programs with at least 4 responses are reported

1-6 Likert scales are designated as Red, Yellow or Green

Overall, how satisfied are you with the training you have received in your program?

Extremely Unsatisfied

Very Unsatisfied

Unsatisfied

SatisfiedVery Satisfied

Extremely Satisfied

I would recommend my training program here to others.

Disagree Strongly

Disagree Moderately

Disagree Slightly

Agree

Slightly

Agree Moderately

Agree Strongly

I have an adequate amount of "protected time" to focus on my educational needs.

I think my program appropriately balances the need to fulfill service obligations to the hospital with

clinical education.

My program is organized to meet my educational needs.

My overall patient load (the quantity of patients I see) is appropriate.

The variety of patients I see is sufficient for meeting my educational needs.

Faculty in my program are successful teachers.

Faculty in my program spend sufficient time supervising the residents or fellows in the program.

Faculty in my program encourage me to ask questions on a regular basis.

Faculty in my program encourage me to be open and honest with them.

Have you had an opportunity to participate in any projects related to Quality Improvement (QI) during

your current training at Stanford? No Yes

Introduction of the Stanford Housestaff Survey – Analysis & Reporting

Program data is compared to the institutional average for each quantitative question.

Stacked/Colored bars are utilized to visually present the data for easy interpretation.

− Solid Fill: Program Data

−Patterned Fill: Institutional Data


Hypothesized construction of the Stanford GME Housestaff Survey

1. Overall, how satisfied are you with the training you have received in your current program at Stanford?

2. I would recommend my training program here at Stanford to others.

3. Faculty in my program are successful teachers.

4. Faculty in my program spend sufficient time supervising the residents or fellows in the program.

5. Faculty in my program encourage me to ask questions on a regular basis.

6. Faculty in my program encourage me to be open/honest with them.

7. I have an adequate amount of "protected time" to focus on my educational needs.

8. I think my program appropriately balances the need to fulfill service obligations to the hospital with clinical education.

9. My program is organized to meet my educational needs.

10. My overall patient load (the quantity of patients that I see) is appropriate.

11. The variety of patients I see is sufficient for meeting my educational needs.

12. In your program, have you had (will you have) the opportunity to participate in any projects related to Quality Improvement (QI)?

Question: Does the Survey test what it was designed to test? Can the survey questions be grouped into different factors as we hypothesized above?

Overall

Faculty

Resources

Educational Content

Clinical Work

QI

Construct Analysis – Dataset

Surveys: Stanford GME Housestaff Survey (12 questions)

Years: 2014-15, 2015-16 and 2016-17

Programs: 29 (21 Residency + 8 Fellowships)

− Residency: Anesthesiology, Dermatology, Emergency medicine, Internal medicine, Neurological surgery, Neurology, Obstetrics and gynecology, Ophthalmology, Orthopaedic surgery, Otolaryngology, Pathology-anatomic and clinical, Pediatrics, Plastic Surgery – Integrated, Physical medicine and rehabilitation, Psychiatry, Child and adolescent psychiatry, Radiology-diagnostic, Radiation oncology, Surgery, Thoracic surgery – integrated, Urology

− Fellowship: Cardiovascular disease, Gastroenterology, Hematology, Nephrology, Pulmonary disease and critical care medicine, Pediatric critical care medicine, Pediatric cardiology, Pediatric hematology/oncology

Response Metrics:

−Percentage of respondents who chose “Agree Strongly/Moderately” or “Extremely/Very Satisfied” or “Yes” (All the Greens)

Construct Validity – Factor Analysis

A method to describe variability among observed and correlated variables in terms of a potentially lower number of underlying variables (called Factors).

A way to reduce number of variables

Factors GME Survey Question Factor 1 Factor 2 Factor 3 Factor 4 Factor 5

Faculty in my program are successful

teachers.0.907 0.634 0.737 0.478

Faculty in my program spend sufficient time

supervising the residents or fellows in the

program.

0.940 0.531 0.550 0.382

The variety of patients I see is sufficient for

meeting my educational needs.0.693 0.434 0.408 0.486

I would recommend my training program

here to others.0.860 0.755 0.800 0.374

I have an adequate amount of "protected

time" to focus on my educational needs.0.438 0.784 0.327 0.541

I think my program appropriately balances

the need to fulfill service obligations to the

hospital with clinical education.

0.590 0.958 0.359 0.685

My program is organized to meet my

educational needs.0.583 0.874 0.364 0.552 0.446

Overall, how satisfied are you with the

training you have received in your program?0.724 0.784 0.700 0.500

Clinical WorkMy overall patient load (the quantity of

patients I see) is appropriate.0.468 0.553 0.940 0.527

Faculty in my program encourage me to ask

questions on a regular basis.0.854 0.678 0.315 0.941 0.359

Faculty in my program encourage me to be

open and honest with them.0.735 0.668 0.432 0.933

QIIn your program, have you had the

opportunity to participate in any projects

related to Quality Improvement (QI)?

0.995

Resources

Faculty

Educational

Content

Construct Analysis – Factor Analysis Cont’d

Extraction Method: Principal Axis Factoring.

Rotation Method: Promax with Kaiser Normalization.

Construct Validity – Conclusions

The Stanford Housestaff Survey was proven to test 5 domains of program quality

Grouping of the questions are in general as we hypothesized, with 3 exceptions –

− Variety of patients factored in Faculty as opposed to Clinical Work as we originally thought

− The Overall Group (- overall, how satisfied are you and would you recommend) factored with: Educational Content and Faculty

III. Predictions of ACGME Resident Survey Results based on the Internal GME

Housestaff Survey

Stanford GME Housestaff Survey was designed to serve as an Early Warning System for programs to detect areas where there were concerns or confusion ahead of the ACGME Resident Survey.

Early Warning System

Research Question: Does the Stanford GME Housestaff Survey in fact predict/correlate with the results of the ACGME Resident Survey?

Prediction of some ACGME Survey Results– Datasets

Surveys: GME House Staff Survey (12 questions), ACGME Resident Survey (41 questions)

Years: 2014-15, 2015-16 and 2016-17

Programs: 29 (21 residency + 8 fellowships)

− Residencies: Anesthesiology, Dermatology, Emergency medicine, Internal medicine, Neurosurgery, Neurology, OB/GYN, Ophthalmology, Orthopedic surgery, Otolaryngology, Pathology-AP/CP, Pediatrics, Plastic Surgery, Physical medicine and rehabilitation, Psychiatry, Child and adolescent psychiatry, Radiology-diagnostic, Radiation oncology, Surgery, Thoracic surgery, and Urology

− Fellowships: Cardiology, Gastroenterology, Hematology, Nephrology, Pulmonary disease and Critical Care Medicine, Pediatric Critical Care medicine, Pediatric cardiology, and Peds Hem/Onc

Metrics:

−GME Housestaff Survey: Percentage of respondents who chose “Agree Strongly/Moderately” or “Extremely/Very Satisfied” or “Yes” (All the Greens)

−ACGME Resident Survey: % Program Compliance, Program Means

Prediction of some ACGME Survey Results– Overall Correlation

Overall average of % program compliance of GME Housestaff Survey positively and significantly correlated with the overall average of % program compliance in the ACGME Resident Survey

Overall, the number of % program non-compliant in the GME Housestaff Survey positively and significantly correlated with the number of % program non-compliant in the ACGME Resident Survey

GME Survey (12 Questions) ACGME Resident Survey (41 Questions)

Average of % Program Compliance of All Qs

Average of Program Mean of All Qs

Number of Non-Compliance (<80%) Number of Non-Compliance (<80%)

Average of % Program Compliance of

All Qs

Correlation

r=.349, p < .01

r=.422, p < .01

r=.434, p < .01

Prediction of the ACGME Survey – Factor Correlation

80 hours1 day free in 7In-house call every 3rd nightNight float no more than 6 nights8 hours between duty periods (differs by level of training )Continuous hours scheduled (differs by level of training )Sufficient supervisionAppropriate level of supervisionSufficient instructionFaculty and staff interested in residency educationFaculty and staff create environment of inquiryAble to access evaluationsOpportunity to evaluate faculty membersSatisfied that evaluations of faculty are confidentialOpportunity to evaluate programSatisfied that evaluations of program are confidentialSatisfied that program uses evaluations to improveSatisfied with feedback after assignmentsProvided goals and objectives for assignmentsInstructed how to manage fatigueSatisfied with opportunities for scholarly activitiesAppropriate balance for educationEducation (not) compromised by service obligationsSupervisors delegate appropriatelyProvided data about practice habitsSee patients across variety of settingsAccess to reference materialsUse electronic medical records in hospital*Use electronic medical records in ambulatory setting*Electronic medical records integrated across settings*Electronic medical records effectiveProvided a way to transition care when fatiguedSatisfied with process to deal with problems and concernsEducation (not) compromised by other traineesResidents can raise concerns without fearTell patients of respective roles of faculty and residentsCulture reinforces patient safety responsibilityParticipated in quality improvementInformation (not) lost during shift changes or patient transfersWork in interprofessional teamsEffectively work in interprofessional teams

ACGME Survey Factors & Questions

Duty Hours

Faculty

Evaluation

Educational

Content

Resources

Patient

Safety/Team

work

GME Survey Question


teachers.



program.


meeting my educational needs.


here to others.


time" to focus on my educational needs.





educational needs.


training you have received in your program?

My overall patient load (the quantity of

patients I see) is appropriate.


questions on a regular basis.


open and honest with them.

In your program, have you had the



Prediction of the ACGME Survey – Factor Correlation

80 hours1 day free in 7In-house call every 3rd nightNight float no more than 6 nights8 hours between duty periods (differs by level of training )Continuous hours scheduled (differs by level of training )Sufficient supervisionAppropriate level of supervisionSufficient instructionFaculty and staff interested in residency educationFaculty and staff create environment of inquiryAble to access evaluationsOpportunity to evaluate faculty membersSatisfied that evaluations of faculty are confidentialOpportunity to evaluate programSatisfied that evaluations of program are confidentialSatisfied that program uses evaluations to improveSatisfied with feedback after assignmentsProvided goals and objectives for assignmentsInstructed how to manage fatigueSatisfied with opportunities for scholarly activitiesAppropriate balance for educationEducation (not) compromised by service obligationsSupervisors delegate appropriatelyProvided data about practice habitsSee patients across variety of settingsAccess to reference materialsUse electronic medical records in hospital*Use electronic medical records in ambulatory setting*Electronic medical records integrated across settings*Electronic medical records effectiveProvided a way to transition care when fatiguedSatisfied with process to deal with problems and concernsEducation (not) compromised by other traineesResidents can raise concerns without fearTell patients of respective roles of faculty and residentsCulture reinforces patient safety responsibilityParticipated in quality improvementInformation (not) lost during shift changes or patient transfersWork in interprofessional teamsEffectively work in interprofessional teams

ACGME Survey Factors & Questions

Duty Hours

Faculty

Evaluation

Educational

Content

Resources

Patient

Safety/Team

work

1 2 3 4 5

0.907 0.634 0.737 0.478

0.940 0.531 0.550 0.382

0.693 0.434 0.408 0.486

0.860 0.755 0.800 0.374

0.438 0.784 0.327 0.541

0.590 0.958 0.359 0.685

0.583 0.874 0.364 0.552 0.446

0.724 0.784 0.700 0.500

0.468 0.553 0.940 0.527

0.854 0.678 0.315 0.941 0.359

0.735 0.668 0.432 0.933

0.995

GME Survey Question


teachers.



program.


meeting my educational needs.


here to others.


time" to focus on my educational needs.





educational needs.


training you have received in your program?

My overall patient load (the quantity of

patients I see) is appropriate.


questions on a regular basis.


open and honest with them.

In your program, have you had the



"Satisfied with process to deal with problems and concerns" &

"Residents can raise concerns without fear" % Program

Compliancer=.169, p > .1

"Satisfied with process to deal with problems and concerns" &

"Residents can raise concerns without fear" Program Mean r=.216, p < .05

Number of Non-Compliance (<80%) Number of Non-Compliance (<80%) r=.288, p < .01

Average % of Program Compliance of

Resources

Prediction of the ACGME Survey – Factor Correlation Cont’d

Faculty (% Program Compliance) r=.434, p < .01

Faculty (Program Mean) r=.469, p < .01



Faculty

Educational Content (% Program Compliance) r=.286, p < .01

Educational Content (Program Mean) r=.267, p < .05



Educational Content

"Appropriate balance for education" & "Education (not)

compromised by service obligations" (% Program Compliance)r=.438, p < .01

"Appropriate balance for education" & "Education (not)

compromised by service obligations" (Program Mean)r=.385, p < .01


% Program Compliance of

"My overall patient load is appropriate"

GME Survey (12 Questions) ACGME Resident Survey ( 41 Questions) Correlation

Average of % Program Compliance of All Qs r=.349, p < .01

Average of Program Mean of All Qs r=.422, p < .01



All Qs

"Faculty and staff create environment of inquiry" % Program r=.275, p < .05

"Faculty and staff create environment of inquiry" Program r=.340, p < .01


Average % of Porgram Compliance of

Resources

"Participated in quality improvement" % Program Compliance r=.488, p < .01

"Participated in quality improvement" Program Mean r=.485, p < .01


% of Program Compliance of

QI

Prediction of the ACGME Survey – Conclusions

GME Housestaff Survey positively and significantly correlates with the ACGME Resident Survey, in terms of:

−Overall compliance scores

−Overall program means

−Each independent factor, except one correlation as noted on the previous slide

Average % of program compliance of Resources to the average of 2 related questions in the ACGME survey

−The number of non-compliant questions

GME House Staff Survey positively and significantly correlates with

the ACGME Resident Survey… so what can we do with this data?

Early warning system for Programs and the Institution / validation of each survey

Data for evidence-based change to improve program quality

Information/statistics for DIO to allocate resources

May signal need for Program Special Review

Data for Annual Program Evaluation (APE)and the Annual Institutional Review (AIR)

Agenda

I. Introduction of the Stanford GME Housestaff Survey


III. Prediction of the ACGME Resident Surveys by the Stanford GME Housestaff Survey


Group Discussion

1. Customizing an internal survey that is most relevant and useful for your institution?

2. How would you advise programs to make improvements based on the survey results?

Topics






Scorecard System – What Is it?

A visual presentation of a program’s educational quality using both external and internal data sources (simple spreadsheet can be used)

External: ACGME Resident Survey, Number of ACGME Citations, Board Pass Rate

Internal: GME Housestaff Survey, Program Evaluation by Resident and Faculty, MedHub Work Hour Report

Illustrated in Green and Red (conditional formatting may be used)

For both the institution and individual programs

Stanford Program Scorecard Example

Trend Analysis - Stanford ProgramSHC Balanced Report Card

Key MeasuresSOURCE INT/EXT

Data

Source2017-18

Sufficient Supervision

Survey

ACGME %-

COMPLIANT

No Data

Yet

Sufficient Instruction

Survey

ACGME %-

COMPLIANT

No Data

Yet

Faculty/Staff Create

Environment of Inquiry

Survey

ACGME %-

COMPLIANT

No Data

Yet

Satisfied with Process for

Problems and Concerns

Survey

ACGME %-

COMPLIANT

No Data

YetClimate Where Residents

Can Raise Concerns

Without Fear

Survey

ACGME %-

COMPLIANT

No Data

Yet

Overall Eval of the

Program

Survey

ACGME %-

COMPLIANT

No Data

Yet

Current Number of

ACGME Citations ACGME

Board Pass Rates ABMS

Overall Satisfaction with

ProgramRESIDENT GME-Survey

100%

Program Organized to

Meet Educational NeedsGME-Survey

93%

Service Over Education GME-Survey86%

Encouraged to Ask

Questions on a Regular

Basis

GME-Survey100%

Residents Can Be Open

and Honest with FacultyGME-Survey

100%

Residents Would

Recommend ProgramGME-Survey

100%

Faculty Overall Evaluation

ProgramFACULTY

Pgm Eval

Mean

Score/10

No Data

Yet

Resident Overall Program

Evaluation RESIDENT

Pgm Eval

Mean

Score/10

No Data

Yet

>80 Violations / AYMedHub

Duty Hr Rpt

No Data

Yet

# Unreviewed Duty Hr

Periods by PD / AY

MedHub

Detailed Rpt

No Data

Yet

RESIDENT

EXT

PROGRAM

INT

PROGRAM

SHC Balanced Report Card

Key MeasuresSOURCE INT/EXT

Data

Source2013-14 2014-15 2015-16 2016-17 2017-18

Sufficient Supervision

Survey

ACGME %-

COMPLIANT

95% 93% 88%100%

Pending

Sufficient Instruction

Survey

ACGME %-

COMPLIANT

77% 84% 76%97%

Pending


Environment of Inquiry

Survey

ACGME %-

COMPLIANT

70% 81% 73%89%

Pending


Problems and Concerns

Survey

ACGME %-

COMPLIANT

68% 77% 73%97%

Pending

Climate Where Residents

Can Raise Concerns

Without Fear

Survey

ACGME %-

COMPLIANT

66% 72% 71%94%

Pending

Overall Eval of the

Program

Survey

ACGME %-

COMPLIANT

8.5 8.6 8.6 9.5 Pending

Total Number of ACGME

Citations (new) (resolved) ACGME4 4 0 0 Pending

Board Pass Rates ABMS2009-13:

87%

2010-14:

83%

2011-15:

86%

2012-16:

82%Pending

Overall Satisfaction with

ProgramGME-Survey 82% 90% 89%

92%91%

Program Organized to

Meet Educational NeedsGME-Survey 94% 85% 74%

89%91%

Service Over Education GME-Survey 94% 80% 80% 86% 71%

Encouraged to Ask

Questions on a Regular

Basis

GME-Survey 100% 90% 80%86%

86%

Residents Can Be Open

and Honest with FacultyGME-Survey 88% NA 91%

86%86%

Residents Would

Recommend ProgramGME-Survey 94% 95% 88%

97%91%

Resident Overall Program

Evaluation

Pgm Eval

Mean

Score/10

8.4 8.4 8.6 8.4 8.5

Faculty Overall Evaluation

ProgramFACULTY

Pgm Eval

Mean

Score/10

9.1 8.9 8.1 8.5 8.6

>80 Violations / AYMedHub

Duty Hr Rpt29 14 15 24 16

# Unreviewed Work Hr

Periods by PD / AY

MedHub

Detailed Rpt0 0 0 0 9

KEY STRENGTH WEAKNESS

5- Year Trend Analysis of AAA

RESIDENT

EXT

PROGRAM

RESIDENT

INT

PROGRAM

Program Trend Analysis– Example

5-year trend analysis

80% cut-off or 0 for citations and work hours

Program Scorecard / Program Trend Analysis

SHC Balanced Report

Card Key MeasuresSOURCE

INT or

EXT

Data

Source

2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18

Sufficient

SupervisionRESIDENT EXT

Survey

ACGME %-COMPLIANT93% 85% 93% 94% 94% 88% 94% No Data

Sufficient

InstructionRESIDENT EXT

Survey



Environment of

Inquiry

RESIDENT EXTSurvey


Satisfied with

Process for

Problems and

Concerns

RESIDENT EXTSurvey


Climate Where

Residents Can Raise

Concerns Without

Fear

RESIDENT EXTSurvey


Overall Eval of the

ProgramRESIDENT EXT

Survey


Total Number of

ACGME Citations

(new) (resolved)

PROGRAM EXTACGME

0 0 0 0 0 0 0No Data

Board Pass Rates PROGRAM EXT ABMS

56% 56% 56% 56% 100% 100% No Data No Data

Overall Satisfaction

with ProgramRESIDENT INT GME-Survey

90% 100% 83% 75% 100% 100% 100% 100%Program Organized

to Meet Educational

Needs

RESIDENT INT GME-Survey

100% 100% 92% 85% 100% 55% 94% 94%

Service Over

EducationRESIDENT INT GME-Survey

100% 100% 75% 80% 91% 55% 94% 94%Encouraged to Ask

Questions on a

Regular Basis


100% 100% 100% 94% 100% 100% 94% 94%Residents Can Be

Open and Honest

with Faculty


100% 100% 100% 94% No Data 86% 94% 94%Residents Would

Recommend

Program


100% 100% 92% 90% 91% 100% 94% 100%

Faculty Overall

Evaluation ProgramFACULTY INT

Pgm Eval

Mean Score/10

8.21 9.69 7.88 9.13 8.82 8.88 9.40 No Data

Resident Overall

Program Evaluation RESIDENT INT

Pgm Eval

Mean Score/10

8.90 9.38 8.25 8.68 7.62 8.70 No Data

> 80 Violations / AY PROGRAM INT

MedHub

Institution Duty Hr Rpt

0% 0% 0% 0% 0% 0% 0% No Data# Unreviewed Duty

Hr Periods by PD /

AY

PROGRAM INT

MedHub

DetailedRpt

0% 0% 0% 0% 0% 0% 0% No Data

KEY STRENGTH WEAKNESS

Institution-Wide Program Scorecard Rolled-Up

A B C D E

SHC Balanced Report Card

Key MeasuresSOURCE

INT or

EXT

Data

Source 2 3 6 7 9

Sufficient Supervision RESIDENT EXTSurvey

ACGME %-

COMPLIANT 92% 100% 63% 100% 86%

Sufficient Instruction RESIDENT EXTSurvey

ACGME %-

COMPLIANT 84% 100% 38% 100% 75%

Appropriate Balance for

EducationRESIDENT EXT

Survey

ACGME %-

COMPLIANT 57% 100% 63% 94% 69%


Environment of InquiryRESIDENT EXT

Survey

ACGME %-

COMPLIANT 71% 100% 63% 100% 64%


Problems and ConcernsRESIDENT EXT

Survey

ACGME %-

COMPLIANT 75% 33% 63% 100% 75%

Climate Where Residents

Can Raise Concerns Without

Fear

RESIDENT EXTSurvey

ACGME %-

COMPLIANT 75% 67% 50% 94% 83%

Overall Eval of the Program RESIDENT EXTSurvey

ACGME %-

COMPLIANT 8.5 7.7 8.6 9.4 8.4

Total Number of ACGME

Citations PROGRAM EXT

ACGME 0 0 0 0 0

AIR / Annual Program Review Dashboard Example

41

Scorecard System – Usage

Program expansion & funding process

Monitor program quality

Monitor institutional trend

Program Expansion & Funding Process Scorecard

43

Institutional Scorecard Trend Analysis

GME SURVEY 2013-2014 2014-2015 2015-2016 2016-2017 2017-2018

Program Is

Organized to Meet

Educational Needs.

88% 92% 85% 86% 89%

Program Balances

Service with Clinical

Education.

86% 87% 84% 82% 84%

Overall Satisfaction

with Training in

Program

85% 92% 92% 93% 94%

Would Recommend

Training Program91% 94% 93% 94% 95%

Faculty Spend

Sufficient Time

Supervising Trainees

95% 96% 92% 93% 96%

Faculty Are

Successful Teachers95% 97% 93% 94% 95%

Encouraged To Ask

Questions95% 97% 91% 93% 96%

Have Not Been

Personally

Mistreated in

Training Program.

89% 89% 91% 91% 95%

Opportunity To

Participate In QI

Projects

66% 88% 76% 73% 75%

Scorecard System – Database

Our Excel database contains:

−All ACGME-accredited programs

−All data sources

GME Survey, ACGME Survey, Program Evaluation, Work Hours Reports from MedHub, etc.

−Data starting from 2012-13

Headers of the database and an example of data

Over 17,500 rows of data for over 100 ACGME programs at Stanford!

Type Year ProgramName ProgramCode Metric % Program Compliant

ACGME-ResidentSurvey 2012-13 A 111111111 Sufficient instruction 86%

ACGME Citations 2017-18 A 111111111 Current Total # of ACGME Citations 2

GME Survey 2013-14 A 111111111 Faculty are successful teachers. 67%

Program Eval by Residents 2012-13 A 111111111 Aggregated Program Evaluation by Residents 7.8

Program Eval by Faculty 2016-17 A 111111111 Aggregated Program Evaluation by Faculty 8.2

Medhub-80 Hour Violation 2014-15 A 111111111 >80 Hour Violations / AY 2

Medhub-Duty Hours Review 2015-16 A 111111111 # Unreviewed Duty Hr Periods by PD / AY 0

Scorecard System – Database (Cont’d)

Generation of scorecard is automated

−Pivot table

−Paste to a formatted spreadsheet template (scorecard)

Topics






VI. Discussion & Questions

Thank You’s and Questions?

Many thanks to Jie Li, PhD, Stanford GME Program Manager for her ideas, research, analyses, presentation and database development efforts.

Contact Information

Nancy Piro: [email protected]

650-723-5948

Ann Dohn: [email protected]

650-723-5948

mailto:[email protected]

mailto:[email protected]

using an internal gme house staff survey as an early ......introduction of the stanford housestaff...

Documents