What is the Best Way to Track

Surgical Complications?

Jacques X. Zhang, B.Sc.

Diana Song, MD

Julie Bedford, RN, MSN

Douglas J. Courtemanche, MD, MS, FRCSC

Marija Bucevska, MD

Jugpal S. Arneja, MD, MBA, FAAP, FACS, FRCSC

Comparing ACS NSQIP versus Traditional M&M Rounds

Conflicts of Interests

• Dr. Courtemanche is a director and shareholder with Resilience Software, which made T-Res

• T-Res is used to collect data for M&M Rounds at UBC and the database provided some of the data for the research

2

What is M&M and NSQIP-P?

3

M&M rounds ACS NSQIP Pediatrics

What is it? Rounds to discuss medical

complications and deaths

Discussions are protected

under Section 51 of the

Evidence Act.1

- Standardized

- Tracks post-op complications

- Subset of surgical patients

- Allows risk-adjusted

benchmarking between

hospitals.2-4

Background

1. bclaws.ca2. Khuri SF. 2005;138(5):837-43. 3. Khuri SF, Daley J, Henderson W, et al. 1998;228(October):491-507.4. Ingraham AM, Richards KE, Hall BL, Ko CY. Adv Surg. 2010;44(1):251-267. .

How does

it work? Complications are entered

retrospectively into a database.

Some discussed at rounds.

Next Slide…

• NSQIP-P definitions are very strict

4

How NSQIP-P Works

Khuri SF. 2005;138(5):837-43. Khuri SF, Daley J, Henderson W, et al. 1998;228(October):491-507.Ingraham AM, Richards KE, Hall BL, Ko CY. Adv Surg. 2010;44(1):251-267. .

Background

5

M&M rounds ACS NSQIP Pediatrics1-3

Rounds are biweekly or quarterly as organized by the department

Risk-adjusted feedback is provided on a semi-annual basis in the form of a odds ratio for the hospital/department

Used for decades NSQIP P pilot in 2008, BCCH joined in 2011

Surgeon is reviewer Surgical Clinical Reviewer (NSQIP) is reviewer

Looks at all cases Looks at a subset of all cases through rigorous sampling

Cases reviewed quarterly (90 days) Tracks post-op occurrences up to a set 30 days

Complication reported only if surgeon aware Active tracking of patient 30 days post-op

All relevant complications tracked Only NSQIP complications are tracked

Recall and reporting bias Follows strict NSQIP guidelines and definitions

Not risk adjusted Risk-adjusted, allows benchmarking

Low inter-rater reliability High inter-rater reliability4

Low cost and labour High cost ($200,000/yr at BCCH) +labour (2 SCRs/ 2 surgeons champions)

Side-by-Side Comparison

1. Khuri SF. 2005;138(5):837-43. 2. Khuri SF, Daley J, Henderson W, et al. 1998;228(October):491-507.3. Ingraham AM, Richards KE, Hall BL, Ko CY. Adv Surg. 2010;44(1):251-267. .4. Shiloach M, Frencher SK, Steeger JE, et al. J Am Coll Surg. 2010;210(1):6-16

Background

Why this paper?

• Limited studies on:

– Pediatric NSQIP in general

– Pediatric M&M vs NSQIP

– Plastic surgery specific data

• PURPOSE:

– To determine the best way to track pediatric plastic surgeries by comparing complications tracked by NSQIP-P vs traditional M&M rounds, in 2012-2013

6

Purpose

Methods

• For the first 2 full years (2012 and 2013) of NSQIP data: – Extract complications (numerators) for both M&M

and NSQIP– Extract total cases (denominators)

• NSQIP is a subset of M&M – Go through op logs and classify M&M data into

“NSQIP categories” eligible vs ineligible

• Stratify data into major and minor complications

7

5.48

7.69

6.626.4

5.796.11

0

1

2

3

4

5

6

7

8

9

2012 2013 2012 + 2013

Co

mp

licat

ion

Rat

e (

%)

Raw Complication Occurrence Rate of NSQIP vs M&M

NSQIP

M&M

NSQIP and M&M have similar rates

8

… However, M&M data includes cases that NSQIP potentially excludes … Need to remove the M&M data that is NSQIP ineligible

NS - Not statistically significant (2-sample Z test)

Results

9

Results

NSQIP vs M&M: adjusted rates

6.62

6.11

5.71

0

1

2

3

4

5

6

7

8

2012 + 2013

Co

mp

licat

ion

Rat

e (

%)

Apples vs Apples: NSQIP vs M&M (NSQIP eligible)

NSQIP

M&M raw

M&M (NSQIP eligible)

10

NS - Not statistically significant (2-sample Z test)

Results

11

~50% of all cases are not tracked by NSQIP!

Half of all cases are not tracked by

NSQIP!

n= 648or (51.4% of all cases)

n=613or (48.6% or all cases)

Results

What is the concordance and

discordance rate?

2012 + 2013 M&M+ M&M-

NSQIP + 13 27 40

NSQIP - 24 584 608

subtotal 37 611 648

NSQIP ineligible 40 573 613

77 1184 1261

12

Concordance rate for M&M Eligible = 13/37 = 35.1%Concordance rate for NSQIP = 13/40 = 32.5% Discordance rate for M&M = 64/77 = 83.1% (MM+, NSQIP-)Discordance rate for NSQIP = 27/40 = 67.5% (MM-, NSQIP+)

6

2

1

2

2

Types of Occurrence: Patients in both NSQIP and TRES M&M

Dehiscence

Hematoma

Dehiscence +Infection

Bleeding

Complex (multiplecomplications)

Results

What is the discordance rate

between NSQIP and M&M?

40

6

15

300

5

10

15

20

25

30

35

40

45

NSQIP ineligible Eligible but notsampled

Sampled but not aNSQIP complication

Sampled butcomplication > 30

days

M&M caughtsomething thatNSQIP missed

Nu

mb

er

of

Cas

es

2012-2013 MM+ NSQIP- n=64

13

Discordance rate for M&M = 64/77 = 83.1%

Results

What is the discordance rate

between NSQIP and M&M?

16

9

2

0

2

4

6

8

10

12

14

16

18

Documented, not reported Unaware Not a plastics complication

Nu

mb

er

of

Cas

es

2012-2013 MM- NSQIP+ n=27

14

Discordance rate for NSQIP = 27/40 = 67.5%

Results

NSQIP and M&M track different

complications

0

0.5

1

1.5

2

2.5

3

3.5

Co

mp

licat

ion

Rat

e (

%)

Types of Occurrences

NSQIP

M&M

15

* = NSQIP tracked complication

Results

What about the severity of the

complication?

• Stratified into major vs. minor complication– Major complication that leads to:

• Death

• Readmission

• Re-operation

– Minor complication • Anything else

– Results: we find ~50/50 split in both systems• 85% “Major” for matched complications (MM+ NSQIP+)

16

Results

7 major complications (2012-2013)

missed by M&M

2

9

7

2

11

7

2

0

0

2

4

6

8

10

12

14

16

18

Match to M&M Documented, notreported

Unaware Not a plasticscomplication

Nu

mb

er

of

Cas

es

2012-2013 MM- NSQIP+ n=27

Major complication

Minor complication

17

Results

Best way to track pediatric plastics

complications?

A combination of both M&M rounds and NSQIP…

• NSQIP and M&M have similar occurrence rates but each has their benefits over the other. NSQIP provides the strict rate of morbidity whereas M&M provides the description.

18

Discussion

MOH

Hospital

Department

Division

Surgeons

NSQIP

M&M

Feedback and Purpose:

19

M&M rounds ACS NSQIP Pediatrics1-3

Rounds are biweekly or quarterly as organized by the department

Risk-adjusted feedback is provided on a semi-annual basis in the form of a odds ratio for the hospital/department

Used for decades NSQIP P pilot in 2008, BCCH joined in 2011

Surgeon is reviewer SCR (nurse) is reviewer

Looks at all cases Looks at a subset of all cases through rigorous sampling

Cases reviewed quarterly (90 days) Tracks post-op occurrences up to a set 30 days

Complication reported only if surgeon aware Active tracking of patient 30 days post-op

All relevant complications tracked Only NSQIP complications are tracked

Recall and reporting bias Follows strict NSQIP guidelines and definitions

Not risk adjusted Risk-adjusted, allows benchmarking

Low inter-rater reliability High inter-rater reliability4

Low cost and labour High cost ($200,000/yr at BCCH) +labour (2 SCRs/ 2 surgeons champions)

Side-by-Side Comparison

1. Khuri SF. 2005;138(5):837-43. 2. Khuri SF, Daley J, Henderson W, et al. 1998;228(October):491-507.3. Ingraham AM, Richards KE, Hall BL, Ko CY. Adv Surg. 2010;44(1):251-267. .4. Shiloach M, Frencher SK, Steeger JE, et al. J Am Coll Surg. 2010;210(1):6-16

Conclusions

Conclusion

1. NSQIP misses 50% of cases2. Not all complications are tracked3. NSQIP overall rate is accurate

compared to M&M4. NSQIP methods are strictly defined and

rigorous while M&M is subject to recall and reporting bias

20

We suggest expansion of NSQIP-P to include currently excluded cases and an extension of the NSQIP-P study interval.

Limitations

• Limited to the plastics department at a single institution, and only for a 2-year study period

• Rates for the M&M complication might be artificially under-reported due to the errors in M&M data

• Low number of complications combined with a large variety plastic procedures may increase the variability in rate

21

Acknowledgements

Dr. ArnejaDr. CourtemancheDr. Diana SongMarija BucevskaJulie BedfordThe plastics team

ADDITIONAL SLIDES

What is the Best Way to Track Plastic Surgery Outcomes: Comparing ACS NSQIP vs M&M rounds

23

QA timeline

1894 1907 1911 1935 1940s 1991 1994 1999 2001 2004 2008 2010 2011

24

Codman develops anesthesia recordCodman helps create the ACS

End-Result Concept into practiceAnesthesia Mortality Committee

Becomes the Anesthesia Study Commission (Prelude to M&Ms)2000: T-res

NSQIP Peds at BCCHNSQIP Peds to 40 hospitals

NSQIP Peds initiation (4 hospitals)NSQIP becomes open subscription program

VA, ACS Patient Safety in Surgery Study, 14 large non-VA academic hospitals

VA study PSI at 3 academic non-VA surgical dptNSQIP established in all 132 major VA surgical centers

NSQIP inception in 44 VA hospitals in the NVASRS

However, complication rates are still

similar

6.62 6.53

0

1

2

3

4

5

6

7

8

2012 + 2013

Co

mp

licat

ion

Rat

e (

%)

NSQIP vs M&M (NSQIP ineligible)

NSQIP

M&M (NSQIP ineligible)

25

NS - Not statistically significant (2-sample Z test)

Results

NSQIP and M&M obtain similar

severity of complications

3.31 3.33 2.93 3.26

3.312.78

2.78

3.26

0

1

2

3

4

5

6

7

NSQIP M&M raw M&M (NSQIPeligible)

M&M (NSQIPineligible)

2012+2013

Co

mp

licat

ion

Rat

e (

%)

Major vs Minor Complications

major

minor

26

NS - Not statistically significant (2-sample Z test)

Results

Matched cases are mostly major

complications

2

20

3

13

1 0

11

20

3

2

200

5

10

15

20

25

30

35

40

45

Matched NSQIPineligible

Eligible butnot sampled

Sampled butnot a NSQIPcomplication

Sampled butcomplication

> 30 days

M&Mcaught

somethingthat NSQIP

missed

Nu

mb

er

of

Cas

es

2012-2013 MM+ NSQIP- n=64

Major complication

Minor complication

27

Results

NSQIP and M&M serve different

purposes…

• NSQIP-P returns a hospital and department odds ratio (hospital wide view)

• Whereas M&M returns a divisional and surgeon level rate (patient level view)

28

MOH

Hospital

Department

Division

Surgeons

NSQIP

M&M

Discussion

Theoretically: Reality: Feedback and Purpose:

All cases All cases

M&MNSQIP

Conclusions

• Similar rates, however:1. NSQIP misses 50% of all cases.

2. NSQIP morbidity rate is confirmed against M&M, even for NSQIP ineligible cases, as well as severity of complications. Validation of the program.

3. Differences in definitions and subjectivity in M&M led to low concordance rate, with NSQIP being a more rigorous system.

4. We suggest expansion of ACS-NSQIP to include currently excluded cases and an extension of the ACS-NSQIP study interval.

29

How M&M works

• Operative notes (Yellow Slips) for each case given back to surgeon and complication are entered unto T-res

• Very user-dependent and non-standardized data

• Database Errors:– Not surgical complication (eg

lack of equipment, wrong dx) (7)

– Cancelled Operation (1)– Any 2° uneventful procedure to

correct complication from 1°procedure (2)

– Date outside 2012-2013 (4)– Duplicate entries (4) – Missing patient name (1)

30

All cases

Eliminate Database Errors

Complication YES Complication NO

New Complication YES Total

• Numerator

• Denominator

History

• Codman– 1894 Develops anesthesia record – 1911 “End Result” concept and his taxonomy of medical errors, which is what T-res (our M&M) is based off

• 1935 Anesthesia Mortality Committee• 1940 becomes the Anesthesia Study Commission prelude to M&M rounds • 1991 NSQIP inception in 44 Veteran affairs hospitals National VA Surgical Risk Study • 1994 NSQIP established in all 132 major VA surgical centers• 1999 VA study PSI Private Sector Initiative, 3 academic non-VA surgical departments• 2001 VA, ACS study Patient Safety in Surgery Study. 14 non-VA large academic medical centers and

four smaller community hospitals, over 3 years. • 2004 NSQIP open subscription program • 2008 Initiation of NSQIP Peds (4 hospitals) • 2010 NSQIP Peds expanded to 40 hospitals • 2012 NSQIP pediatric at BCCH (only one outside US)

31