video summarization of key events stage i - the critical view michael a. grasso, md, phd university...

Video Summarization of

Key EventsStage I - The Critical View

Michael A. Grasso, MD, PhDUniversity of Maryland School of

MedicineUMBC Computer Science

MichaelGrasso.com

Abstract

Laparoscopic surgery is a minimally invasive technique with unique training requirements. Video-assisted evaluation is one method that surgical residents can use to demonstrate competence. Automated video summarization can increase the efficiency of evaluations by directing the senior surgeon to key portions of a surgical procedure. We are using image classification techniques to segment videos of laparoscopic cholecystectomies to assist with surgical training and evaluation.

Overview

Background Laparoscopic Surgery Image Classification

Methods Discussion

Laparoscopic Surgery

Minimally Invasive Surgery. First performed in 1987. Used in many surgical procedures.

Gall bladder removal (cholecystectomy).

Esophageal surgery (fundoplication). Colon surgery (colectomy). Others.

Laparoscopic Approach Narrow tubes (trocars)

are inserted into the abdomen through small incisions.

www.fda.gov

Laparoscopic Procedure Camera is passed

through trocar. Procedure is often

videotaped. Carbon dioxide is

infused through trocar.

Instruments are passed through the trocars to cut, manipulate, and sew.

Laparoscopic Aftercare Compared with an

open procedure. Smaller scars. Reduced pain. Quicker recovery.

http://www.nlm.nih.gov/medlineplus/ency/presentations/100166_1.htm

Technical Challenges

Access limited to small incisions. Long instruments with only the tips

visible. Two-dimensional video. Limited tactile feedback.

British Journal of Surgery. 2004 Dec;91(12):1549-1558

Laparoscopic Training

Traditional apprenticeship model. Acquire skills during actual procedures. Not sufficient for laparoscopic skills.

Other methods. Box trainer with animal or synthetic

models. Virtual reality simulator. Video-based assessment.

Assessment of Skills

Trainee must demonstrate competency.

Evaluation by a senior surgeon. Direct observation of the trainee. Video-based assessment.

Question: Can we organize video in order to assist in video-based assessment?

American Journal of Surgery. 1991 Mar;161(3):399-403

Objective

Identity key portions of surgical procedure to aid in video-based assessment.

Stage I is to identify the "critical view".

Video

Segments

Frames

Overview

Background Laparoscopic Surgery Image Classification

Methods Discussion

Summary: Organize surgical video to make it easier for expert to review.

The Critical View

Helps ensure that the anatomy has been properly identified.

Occurs after dissecting anatomy. Occurs before clipping the cystic

artery and cystic duct.

The Critical View

Cystic artery

Liver

Cystic duct

Fundus

Netter's Atlas of Human Anatomy

The Critical View

Image Classification - Human

Features a person might use. Spectral features.

Tonal variations. Textural features.

Spatial distribution of tonal variations. Contextual features.

Features from surrounding areas.

Image Classification - Computed

Features extracted from image. Spectral features.

Distribution, size, width. Textural features.

Homogeneity, contrast, correlation. Similarity/distance metrics.

Jaccard coefficient, Jeffrey divergence.Journal of WSCG. 2003; 11(1):269-273

IEEE Transaction on Systems, Man, and Cybernetics. 1973 Nov; 3(6):610-621

Color Histogram Red, green, blue, or gray.

Count number of pixels for each tone. One 28 set for an 8-bit image for each color. Does not vary with translation and rotation. Ignores shape and texture.

4x4 image. 4 gray tones. H = {5, 4, 5, 2}

0 0 1 1

0 0 1 1

0 2 2 2

2 2 3 3

Binary Histogram

Quantize values for each tone to 0 or 1.

Background color given less weight. Subtle changes given more weight.

HB = {1, 0, 1, 1}

0 0 0 0

0 0 0 0

0 2 2 2

2 2 3 3

3D Histogram

Distribution within a 3D color-space. 3D color space (red, green, blue). Used in object recognition & image

retrieval. n3 entries, where n = number of tones.

Example. Quantized to 3 tones

for each color.

Spatial-Dependency Matrix Co-occurrence

matrix. Co-occurring values

(0o, 45o, 90o, 135o). Four 28 x 28

matrices for 8-bit image.

Co-occurring Bits

0 1 2 3Refe

ren

ce B

its

0 4 2 1 01 2 4 0 02 1 0 6 13 0 0 1 2

0 0 1 1

0 0 1 1

0 2 2 2

2 2 3 3

135o 90o 45o

0o Ref 0o

45o 90o 135o

M0 =

Additional Spectral Features

Location of the distribution. Mean = Σ (bin*freq) / Σ (freq). Mode = bin of the max freq.

Size of the distribution. Standard deviation.

Width of the distribution. Max(bin) - Min(bin).

Additional Textural Features

Homogeneity. Number of tone transitions.

Contrast. Amount of local variation.

Correlation. Measure of linear dependencies.

Similarity/Distance Metrics

Jaccard Coefficient. Similarity of two sample sets.

|A B| / |A B| Two binary sets.

M11 / (M01 + M10 + M11)

Jeffrey Divergence. Distance between two vector spaces.

Σ (xi log(xi/avgi) + yi log(yi/avgi))n

i=1

Other Distance Metrics

City Block or Manhattan Distance. Euclidean Distance. Chi-Square. Canberra Distance.

Proceedings ACM SAC. 2008;:1225-1230

Related Efforts - Hysteroscopy Use Jeffrey divergence on color

histogram to identify segments. Relevant segments based on image

redundancy. No understanding

of the content of each segment.

Proceedings 27th IEEE-EMBS. 2005;:5680-5683

Mayo Clinic

Related Efforts - Echocardiogram

Use cosine similarity and edge change ratio to identify video segments.

State-based modeling. Identify states in each

video segment. Diastole (resting). Systole (contracting).

IEEE Transaction on Information Technology in Biomedicine. 2008 May;12(3):366-376

Medline Plus

Overview

Background Laparoscopic Surgery Image Classification

Methods Discussion

Summary: Spectral and textural features compared with similarity metrics.

Methods

Our objective. Identity key portions of surgical

procedure to aid in video-based assessment.

Stage I is to identify the "critical view".

Video

Segments

Frames

Tools

FFmpeg http://ffmpeg.mplayerhq.hu/ Extract JPEG images.

ImageJ http://rsbweb.nih.gov/ij/ Macros and Java plugins.

Work Plan

Identify videos for analysis. Convert videos to JPG. Evaluate ability to identify critical view.

Color histogram. Binary histogram. 3D histogram. Spatial-dependency matrix. Jaccard coefficient, Jeffrey divergence.

Algorithm

Feature ExtractionImageJ Color Histograms

Binary Histograms3D Histograms

Spatial-Dependency Matrices

Similarity Metric

Critical View?

Critical View

Random Image

Image ExtractionFFmpeg

Overview Background

Laparoscopic Surgery Image Classification

Methods Discussion

Summary: Attempt to identify the critical view by comparing image features with similarity metrics.

Discussion

Color and binary histograms do not correlate with the critical view. They do, however, predict when we

are in the abdomen. Currently working on 3D histograms

and spatial-dependency matrices. NIH grant application under

development.

Challenges

Live tissue (vs. solid objects). Deformable. Normal variation. Disease states.

May need to consider. Temporal information. Relevant clinical data of the patient. Critical view "rectangle" (contextual).

Summary

We are comparing image features with similarity metrics to identify the critical view.

This is a first step in automated video summarization, to help with video-assisted evaluation of laparoscopic surgery.

Thank You