Mark A. Mattos, MD,

Michigan Vascular Center-Michigan State University

Flint, Michigan

Advanced Vascular Surgery Surgical Skills and

Simulation Assessment Program

Making A Case for a National Examination,………or Not

Presented at the 2017 APDVS Spring Meeting, March 31, 2017, Rosemont, Illinois

O.W. Brown, M.D.

William Beaumont Hospital - Oakland University

Royal Oak, Michigan

Disclosures

• Nothing to Disclose

Fitness to Practice

Definition

When someone who has the skills, knowledge

and character to practice their profession

safely and effectively.

Fitness to Practice – USA

• Cognitive Competence

– Determined by written and oral examinations

objectively administered by the Vascular

Surgery Board

= Board Certification!!

Certification in Vascular Surgery

= Technical Fitness to Practice?

Cognitive vs.Technical Competence

Cognitive competence is not a guarantee of technical skills competence

Operative case volume alone

may not determine technical competence

• 2 different domains

Fitness to Practice - Technically

Practical Requirements

• Competent open and endovascular skills

• Expert knowledge of conduct of procedure and

technical steps

• Display ability to handle high-stress/high-risk

scenarios without panic and with calm dispassion

• Display ability to create and implement contingency

planning

• Display to ability communicate during procedure

• Display appropriate decision making skills

Fitness to Practice - Technically

• Technical Skills Competence in 2017

– Determined solely by subjective assessment

from faculty members of the vascular trainees

parent institution

Making the Case……

Objective accurate assessment of open and

endovascular skills of senior vascular trainees

is essential for determination of

fitness to practice

Furthermore,………..

We may not be allowed to rely on cognitive

competence (Board Certification) alone to

determine Fitness to Practice

Skills Assessment as Part of Certification

• Interventional Cardiology

• Interventional Radiology

• Vascular Medicine

Are You Ready to Practice Vascular Surgery?

January 7-9, 2016, January 12-14, 2017

Marcia & Eugene Applebaum Surgical Learning Institute

William Beaumont Hospital, Royal Oak, Michigan

AVSSSAP

• Initial results regarding performance, assessment and training potential have been favorable from trainees, faculty and industry

• Herein, we summarize the 2-yr program results

• Determine if the program could identify and assess those specific skills necessary to determine fitness for practice of senior vascular trainees

2016-2017 Participants

***

*

*

*

*

**

**

*

**

**

*

*

**

*

*

*

*

*

*

**

*

*

*

*

**

*

**

*

*

*

*

*

*

• 47 senior vascular trainees

• 60 vascular surgery faculty • 35 institutions

• 28 industry partners

Vascular Surgery Trainees

• 32 Senior Vascular Fellows (PGY-7)

• 14 Senior Vascular Residents (PGY-5)

• 1 Junior Vascular Fellow (PGY-6)

• 33 males, 14 females

2017 Vascular Faculty Proctors

• 35 board-certified vascular surgeons

Program Overview

• 31 assessment stations

• 12-15 simulated procedures

– 30-60 minute sessions

• 522 total simulated procedures

• 2088 individual assessment metrics

– 4 metrics per simulated procedure

Simulation ModelsAAA repair (tube graft) Maquet; W.L. Gore

CEA with patch closure Maquet; W.L. Gore

Femoral-Popliteal artery bypass Maquet; W.L. Gore

Hemodialysis access (AVF) Maquet; W.L. Gore

Mismatched E-S anastomosis Bard Vascular; Cryolife; Co-Author (MAM);

Groin Dissection, Femoral Endarterectomy LifeLike BioTissue

EVAR Endologix; Medtronic; W.L. Gore

TEVAR Medtronic; W.L. Gore

Carotid artery PTA and Stenting Abbott; Cordis; Mentice

Renal artery PTA and Stenting Abbott; BSCI; Cordis; Mentice

Iliac artery PTA and Stenting Abbott; BSCI; Cordis; Medtronic Mentice

SFA PTA, Stenting, Atherectomy Abbott; BSCI; Cordis; Medtronic; Mentice

Percutaneous closure device Abbott; Cordis

Videotaping Sessions B-Line Medical

6 Simulated Open Vascular Procedures

9 Simulated Endovascular Procedures

Assessment Methodology

• 1 faculty to 1 trainee

• Faculty are ‘passive’

1st assistants

• Faculty are ‘silent’

expert assessors

• Immediate feedback

Requirements for Proficiency

1. Global Rating Scale Score:

1. >80% of maximum

2. Global Rating Summary Score

1. Level 4 (proficient) or Level 5 (advanced)

3. Critical Task Performance:

1. 100% completion of all critical skill tasks

Global Rating Scale

(Score >80% maximum)

Global Rating Summary Ranking

Critical Task Performance

(100% achievement)

**

**

Proficiency DataSimulated Procedures Global Rating Scale Global Rating Summary Critical Task Performance

AAA Repair 38/49 (77%) 36/47 (76%) 14/47 (29%) L

CEA with patch closure 42/45 (93%) 31/33 (94%). 22/36 (61%)

Femoral-Popliteal artery bypass 40/43 (93%) 38/42 (90%) 15/46 (33%)

Hemodialysis access (AVF) 38/41 (93%) 30/32 (94%) 27/40 (68%) H

Groin Dissection, Fem Endart. 15/18 (83%) 13/17 (76%) 12/19 (63%)

Mismatched E-S anastomosis 32/34 (94%) H 38/39 (97%) H 12/30 (40%)

EVAR 33/41 (80%) 30/40 (75%) 9/24 (38%)

TEVAR 27/42 (64%) 28/38 (74%) 16/27 (59%)

Carotid artery PTA and Stenting 18/36 (50%) L 11/29 (38%) L 11/31 (35%)

Renal artery PTA and Stenting 20/32 (62%) 21/30 (70%) 11/29 (38%)

Iliac artery PTA and Stenting 33/40 (82%) 31/39 (79%) 17/34 (50%)

SFA/Pop PTA and Stenting 26/33 (79%) 25/30 (83%) 11/30 (37%)

SFA PTA, Stent, Atherectomy 5/7 (71%) 3/4 (75%) 2/5 (40%)

Totals 367/461 (80%) 335/420 (80%) 169/398 (42%)

Identifying Trainee Errors

• Heparin administration

– Dosing errors or omissions in 8%

• Incorrect Clamping/Unclamping Technique

– Incorrect sequencing or application occurred in 10%

• Poor use of first assistant

– 25% received a GRS task score of 1 or 2

• Incorrect or unsafe knot tying technique

– 15% received a GRS task score of 1 or 2

Open Vascular Skills Performance Issues

• Aortic Anastomotic Inefficiency (>15”, >2 leaks)

– 42% proximal aortic anastomoses (end-end)

– 19% of distal aortic anastomoses (end-end)

Open Vascular Skills Performance Issues

• Anastomotic Inefficiency (>15”, >2 leaks)

– 61% of proximal femoral anastomoses (end-side)

– 46% of distal popliteal anastomoses (end-side))

Open Vascular Skills Performance Issues

• Anastomotic Inefficiency (>15”, >2 leaks)

HD access anastomoses (end-side)

5 of 23 (22%)

Mismatched anastomoses (end-side)

13 of 19 (68%)

Open Vascular Skills Performance Issues

• Patch Closure Inefficiency (>25”, >2 leaks)

Overall: 6 of 41 (15%)

Carotid Endarterectomy:

5 of 24 (20.8%)

Femoral Endarterectomy:

1 of 17 (5.9%)

Endovascular Skills Performance Issues

• Loss of Wire Access

– 14 of 88 (16%)

• Excessive Contrast Volume

– 44 of 137 (32%)

• Excessive Fluoroscopy Time

– 13 of 138 (9%)

• Prolonged Procedure Time

– 26 of 38 (19%)

• Improper angiographic views– 25%-33% received a GRS task score of 1 or 2

Reduced Faculty Reporting Errors

• Total Assessment Errors: 20.5% 12.8%

– Global Rating Scale – 8.6%

– Global Rating Summary – 12.6%

– Critical Task Performance – 8.3%

Simulator Performance

• Endovascular simulator

– 10-15% software malfunction reboots

– Inability to record or loss of performance data

– Variability between simulators

– Lack of proper type or variety of equipment

• Open vascular simulator

– 5-10% malfunction

– Insert connective problems

– Tissue integrity dysfunction

– Flow-based related issues noted

Trainee Survey Results

Educational Content and Format

My open vascular skills were accurately assessed 86%

My endovascular skills were accurately assessed 75%

Immediate faculty feedback was helpful and improved learning 100%

Having different faculty perform assessments was valuable 100%

Having a vascular skills assessment program would be useful in

my training program

93%

I would recommend this program to other vascular surgery

residents and vascular surgery fellows

100%

The goals and objectives of the AVSSSAP were clearly explained 100%

The goals and objectives of the AVSSSAP were achieved 100%

Summary

AVSSSAP provided objective assessment of the following:

Open Vascular Skills:

1. Knowledge of procedural conduct/technical steps of procedure

2. Retraction and exposure

3. Vessel preparation

4. Anastomotic performance (vascular forceps handling/needle driving)

end-end, end-side, patch repair

large, medium, small (20mm to 3mm)

surface, shallow, deep (0” to 8”)

5. Knot tying performance

surface, shallow, deep (0” to 8”)

large, moderate, small (3-0 to 7-0)

Summary

AVSSSAP provided objective assessment of the following,

Endovascular Skills:

1. Knowledge of procedural conduct/technical steps of procedure

2. Guidewire, catheter and sheath selection

3. Nonselective/selective guidewire, catheter and sheath placement

3. Performance of diagnostic angiography/contrast utilization

4. Performance of balloon angioplasty

5. Performance of stent deployment

6. Performance of aortic endograft selection and deployment

Summary

• Most senior vascular trainees (but not all) appeared to be technically prepared to perform certain vascular procedures in a proficient and independent manner

• Variable measures of performance by trainees

• Faculty assessment performance improved, most likely due to repeated exposure to the assessment format of the program and increased interest in objective-based skills assessment

• Assessment metrics are not validated

Conclusion

• 2-yr data suggests continued value for this type of advanced skills assessment program.

• Ultimate value to program directors remains to be determined

• A small but defined number of inherent adverse simulator-specific tissue and computer software design flaws continue to occur

• Unclear if current simulators can provide the necessary advanced fidelity for the implementation of a liability-free national vascular skills competency exam.

• Alternative options for objective assessment is offered

Recommendations and Alternative Options

• Simulation Summit– 2-day Meeting

– Division Chairs, Program Directors, SVS, APDVS, Regional Society leadership

– Identify specific advanced skills and metrics to be used to assess technical competence and fitness to practice

– Identify what simulators/skills trainers are available

– Industry leaders/simulation companies invited

– Simulation companies to provide simulators/skills trainers

– Division Chairs, Program Directors, Educational Leaders to perform procedures on simulators

– Create a final document identifying metrics for validation, models to be used for achievement of national vascular surgical competence examination

– Begin discussion with simulation companies to produce simulators and skills trainers

Recommendations and Alternative Options

• Regional Skills Assessment Programs

– Society-based Assessment Programs

– All senior vascular surgical trainees must attend program

– Requirement for trainee to be able to apply for board certification

– Full objective assessment with feedback occurs at program

– Full report of trainee performance provided to Program Director

– Program Director assumes responsibility for ensuring trainee is

competent/proficient for all procedures being tested for

– Letter from Program Director goes in trainees file indicating

completion of objective testing and documentation of competency

Survey Results

Thank You!