Anatomical Sketch Understanding

Peter HaddawyComputer Science & Information Management

Asian Institute of Technology

Anatomical Sketching

• Sketching is ubiquitous in medicine– Patient records– Communication with patients– Consultation– Medical education, particularly PBL

COMET: Collaborative Medical PBL Tutor

Suebnukarn & Haddawy, Clinical-Reasoning Skill Acquisition through Intelligent Group Tutoring, IJCAI-05.

Scenario

Unconsciousness due to car accident

It’s certainly possible to have a contusion in

the frontal lobe, but what about damage to

the brain stem?

Discussion boardRecognize what anatomical structure has been drawn and from what perspective

Parietal view of the brain

Identify anatomical parts of the structure, even if not explicitly drawn

Frontal lobe of the brain

Understand annotations on the sketch

Arrow pointing from the label “contusion” to the frontal lobe

Effectively use the sketch as a medium of communication in a dialog

Recognize what anatomical structure has been drawn and from what perspective

Identify anatomical parts of the structure, even if not explicitly drawn

Why sketch understanding?

• The informalness of sketches is important:– Conveys that the ideas are still rough

and thus invites collaboration and modification.

– Clean, precise-looking diagrams create the impression of more precision than was intended and lead to a feeling of commitment to the sketch as originally drawn.

• Sketching in medical PBL gives students practice in recalling anatomical structure.

The challenge of anatomical sketches

• Previous work (e.g. ASSIST, SILK) starts by recognizing primitive components– circles, lines, rectangles

• This approach works for assemblies of components but not for anatomical sketches

• Anatomical sketches are complex structures that may be drawn with greatly varying degrees of detail.

Sketch Recognition

• By convention, 2-D depictions of anatomy are drawn from one of eight standard views.

• Five external views, corresponding to the sides of a cube:– Anterior– Posterior– Superior– Inferior– Lateral (2 sides)

• Three internal views, corresponding to the cutting planes:– Sagittal– Coronal– Axial

Assumptions

• Sketch drawn from one of the eight standard views

• Sketch contains a single anatomical structure with no extraneous parts

• Sketch is complete: no major parts left out

UNAS Sketch Recognition Algorithm

• Sketch recognition becomes a classification problem: – Identify the anatomical structure and

view• Heart: anterior view

• Naïve Bayes classifier– Features: similarity between sketch

and each of a set of templates– Similarity computed using Shape

Context [Belongie, et al 2002]

Recognition Process

AnatomicalStructure & View

…Template 1 Template m

Naïve Bayes Classifier

Lung External

User Sketch

Template 1 Template m…

Shape ContextMatching

Data Collection

• Collected sample sketches from 3rd – 6th year medical students

• 70 sketches of 6 anatomical structures, 2-3 views per structure: 1050 sketches

Constructing the Classifier

• 70% of the data for training– 49 sketches per view

• Treat each sketch as a potential template

• Calculate 15 conditional Gaussians for each sketch– Distribution of shape context match over

the other sketches conditional on the class (anatomical structure & view)

• Use the Wrapper approach [Kohavi, 1997] to select a small, accurate subset of templates– 32 templates

Evaluation

• 30% of the data for testing– 21 sketches per view

• Asked ten 3rd - 6th year medical students to classify each sketch

• Compared: Student accuracy vs UNAS accuracy

Evaluation: Accuracy

Brain Heart Lung

Parietal Sagittal Basal Anterior Posterior Interior External Internal

Students 92.9 92.9 97.6 87.1 80.5 93.8 93.3 91.0

UNAS 86.0 67.0 57.0 81.0 52.0 57.0 76.0 48.0

Skull Stomach Kidney Overall Accuracy

(%)Anterior Parietal Basal External Internal External Internal

Students 99.0 98.6 97.6 95.7 93.3 97.1 96.7 93.81

UNAS 76.0 81.0 52.0 76.0 67.0 52.0 67.0 66.30

Baseline random classification accuracy = 6.7%

Brain

Heart

• Between internal & external viewsBetween internal & external views

• Between similarly shaped structuresBetween similarly shaped structures

UNAS Common Errors

Kidney External

Kidney Internal

Skull Basal Brain Basal

Sketch Segmentation

• Task: Identify explicit and implicit anatomical parts of the sketch

• Algorithm:– Find the best correspondence

between the sketch and a pre-labeled canonical template (Shape Context)

– Transfer labeled points to the sketch– Connect the points to form the parts

Matching sketch and template

Simple Matching

Sketch

Template

Transformation

• Transform the template to fit the sketch– Thin-plate splines

• Then project labels from transformed template onto sketch

• Curve reconstruction (TSP [Giesen, 1999])

Evaluation

• Selected five qualitatively different sketches from two views of each of four structures– Brain : Basal, Parietal– Lung : External, Internal– Heart : Anterior, Posterior– Skull : Anterior, Parietal

• Asked a physician to segment the sketches

• Compared: Physician segmentations vs UNAS segmentations

Evaluation

• Segmentations of the system and those of the expert are very similar.

• Segmentations of the system have no missing or overlapping regions

sketch system physician sketch system physician

Evaluation: Disagreements

• The system is less tolerant of inaccurate proportions than the physician

sketch system physician

Conclusions

• First application of sketch-based interfaces to intelligent tutoring

• First in a medical domain other than image retrieval

• Can recognize 15 views of 6 anatomical structures– Accuracy 66% vs baseline random

classification accuracy of 6.7%

• System’s segmentations qualitatively similar to those of the physician

Ongoing & Future Work

• Improve recognition accuracy and speed• Improve segmentation quality and speed• Integrate with annotation understanding

component• Incorporate into COMET tutoring system

(using UMLS)

• Sketches with partial and multiple anatomical structures

Demos gladly given on request

Shape Context

• Represent a shape as a set points sampled from the shape’s contours

• Each point represented as a coarse histogram of the points surrounding it.

Shape Context

• Uniformly randomly sample points from the sketch and template.

• Compute the shape context descriptor for each point.

• Normalize by the mean squared distance between points to make scale invariant.

• The similarity between two shape context descriptors is just the χ2 statistic.

• Find the least cost match between all points in the sketch and template.– Weighted bipartite graph matching– Hungarian method: O(N3)

Evaluation: Medical Student Accuracy

Evaluation: UNAS Accuracy

Skull

Lung

Evaluation: Disagreements

• The system’s segmentations are occasionally effected by excessive and unimportant contours in the sketch

sketch system expert