Gaussian Process Based Image Segmentation andObject Detection in Pathology Slides

CS 229 Final Project, Autumn 2013

Jenny HongEmail: jyunhong@stanford.edu

I. INTRODUCTION

In medical imaging, recognizing and classifyingdifferent cell types is of clinical importance. How-ever, this requires a medical expert to perform thearduous task of manually annotating images. There-fore, we seek to better automate the two followingproblems in pathology slides of human tissue.

The first is image segmentation, or categorizingpixels into labeled classes. For our dataset of H&Estained cardiomyopathy slides, our classes are theforeground cells (ie. cardiomyocytes), ”background”cells in the interstitial space, and nuclei in eitherregion.

The second problem is object detection, or iden-tifying instances of some class. In this case, our pri-mary interest is identifying instances of cardiomy-ocyte cells. This greatly aids in medical annotationsand allows experts to focus on annotating only theinterstitial cells, which are the parts that most requireexpert attention to label.

We extend existing methods for object detectionby proposing a new model for shapes of objects. Foreach cell, we develop a radial representation, whichwe call a shape profile. We then learn Gaussianprocesses from these profiles model the prior distri-bution over shapes. Such a model is relatively new inthe literature [5] and the work has not also combinedthis technique with the process of proposing newboundaries for cells in a generative method.

We then further augment this method by buildingoff of existing methods for seed detection, which isimportant for refining the radial representations.

0 500 1000 1500

0

200

400

600

800

1000

1200

1400

Fig. 1. Original H&E Stained Slide with bounding boxes

This project is an extension of Jenny’s workunder David A. Knowles during Stanford CS Under-graduate Research Internship during the summer of2013, where she developed the image segmentationportion of the project and began initial work on theshape profile model for Gaussian processes.

December 13, 2013

II. METHODS

A. Image Segmentation

For multi-label segmentation, we use the ap-proximate graph cuts-based algorithm proposed byBoykov et al. [2], minimizing energies of the form

E(f) =∑

{p,q}∈NVp,q(fp, fq) +

∑p∈P

Dp(fp) (1)

0 500 1000 1500

0

200

400

600

800

1000

1200

1400

Fig. 2. Results of Image Segmentation

where N is a set of four-connected neighboring pix-els and P is the set of pixels. The potentials respec-tively seek smoothness (ie. similarity of neighboringpixels) and fidelity to the original image.

For each class, to calculate the unary potentials,we learn a Gausian mixture model with pixels asindividual data points. Each pixel is represented bya vector consisting of its RGB values in the originalimage.

A small set of hand-labeled pixels (relative tothe image size), serves as the training data forthe Gaussian mixture model. We have labeled twobounding boxes of pixels per class, to simulate thetype and amount of data a human would provide iflabeling an image in an interactive environment. Anexample image with bounding boxes (in green) isshown in Fig. 1.

For the pairwise potentials, we started with po-tentials of larger magnitude from a class to itself,which promotes identifying coherent, connected ob-jects. We also gave small potentials to cardiomy-ocyte and background, and cardiomyocyte and nu-cleus.

The result of the image segmentation process isshown in Fig. 2

0 500 1000 1500

0

200

400

600

800

1000

1200

1400

Fig. 3. Results of Connected Component Labeling

B. Object Detection

Our goal in this stage is to identify componentswith smooth outlines, so that we can generate shapeprofiles as described in the next section. Startingwith the segmentation generated from the previousstep, we reduce the image into binary backgroundvs. non-background classes and use a simple floodfill algorithm to identify connected components ofthe same class. Fig. 3 shows the results of connectedcomponent labeling (each color represents a differ-ent connected component).

For each object, we construct a shape profile,or radial representation, by measuring the radiusfrom the center at 20 evenly spaced angles from0 to 2π. The initial guess of the center is taken tobe the centroid of all pixels assigned to the givenconnected component. However, later iterations canincorporate the centers found by the seed detectioncomponent of the project.

This representation allows us to represent a shapewith fidelity to the exact pixels used in the shapebut in a very compact way. From these shapes, welearn parameters to represent a prior distributionover the shapes via a Gaussian process. For ourcovariance function, we use the following periodicrandom function [4] over x

k(x1, x2) = A exp(−sin2(|x1 − x2|)2l2

) (2)

0 1 2 3 4 5 61520253035404550

0 1 2 3 4 5 60

10

20

30

40

50

60

Fig. 4. Sample Gaussian Process Parameters

A is the amplitude and learns the average radiusor measure of size for each cell. l is the length-scale, or strength of correlation between two pointsas a function of their distance. For example, l candistinguish between an elliptical cell shape, such asthe ground truth cardiomyocyte labeling, a figure-eight cell, such as two cells incorrectly identified asone connected component.

We found the parameters to be influenced verylittle by the prior distributions given over theirvalues. However, they were strongly influenced byinitial values. We used initial values A0 =

√Nπ

where N is the number of pixels in the connectedcomponent and a small constant for l0.

Fig. 4 shows a shape profile and the parametersof the Gaussian process learned from it. The dashedgreen line represents the mean and the shaded grayarea extends to one standard deviation above andbelow.

C. Seed Detection

The Gaussian process technique learns param-eters over the prior distribution of shapes, but itdoes not directly aid in identifying the centers or,more importantly, the boundaries of these shapes.Identifying accurate centers is important in findinga shape profile with high fidelity, and to that end,we use a multiscale Laplacian of Gaussian filtering

Fig. 5. Laplacian of Gaussian filter with σ = 10

method proposed by Al-Kofahi et al. [1].

The multiscale LoG method originally used anexact binary graph-cuts algorithm to classify fore-ground and background for pre-processing. How-ever, because of richer image data and differentneeds, we have binarized our image after a multi-label segmentation via flood fill.

The multiscale LoG method simultaneously givesestimates of the shape and of the center for each cell.The scale-normalized LoG filter is given by

LoGnorm(x, y;σ) = σ2(∂2G(x, y;σ)

∂x2+∂2G(x, y;σ)

∂y2)

(3)where σ is the scale value and G(x, y;σ) is aGaussian with 0 mean and scale σ.

Fig. 5 shows the LoG filter detecting centers wellfor the given σ. In particular, red areas show pixelsthat are good fits of centers whose borders align withthe filter.

The centers of cells in the image are given bypeaks in the response surface

RN(x, y) = argmaxσ{LoGnorm(x, y;σ) ∗ IN(x, y)}(4)

For each connected component C where C is aset of pixels, the estimate of its center is thereforegiven by argmax(x,y)∈CRN(x, y). This is again amodification of Al-Kofahi et al.’s method, which

Fig. 6. Sample centers detected by the Laplacian of Gaussian filter

used watershed flooding to detect all local maximarather than restricting to one per connected compo-nent.

This method allows us to choose exactly onecenter per connected component to generate theshape profile for the Gaussian process part of theobject detection process. It also simplifies the prob-lem by allowing us to use the Guassian processto focus only on under-segmentation (ie. when twoor more ground truth cells are detected as oneconnected component) instead of also addressingover-segmentation, which was not a problem for ourdata set.

However, this means we must rely entirely on thescore assigned by the Gaussian processes to detectwhen we should split components. In the originalmethod, the multiscale LoG method implicitly con-tributed to detecting places to split, because eachlocal maximum indicated one cell.

D. Data Set

Thus far, we have run our methods on trans-verse and longitudinal H&E stained cardiomyopathyslides provided by Andrew Connolly of the StanfordMedical School. (Figures in this paper are takenfrom tranverse slides.) For comparative analysis, weplan to test further using result from H&E stainedbreast cancer histopathology images from Mitko

Veta of the Image Sciences Institute [7], a groundtruth data set. We also plan to data from the UCSBBio-Segmentation Benchmark dataset [3], also H&Estained breast cancer images.

III. CONCLUSION

In this project, we have explored the effectsof two approaches to object detection that run inparallel: Gaussian processes based on shape profilesand multiscale Laplacian of Gaussian filters. Eachmethod gives an independent estimate of two values:(1) the size of the cell (corresponding to a parameterof the Gaussian process and the scale of the LoGfilter) and (2) the fit of a connected component(corresponding to the score assigned to a componentby the Gaussian process and whether the originalmultiscale LoG filtering method by Al-Kofahi et al.yields multiple seeds per connected omponent).

The main goal of immediately further work isto integrate these two estimates iteratively. Thatis, a proposed workflow is to iterate between theGaussian process method and the multiscale LoGmethod instead of running them in parallel. Afterone method completes, its estimates of the twovalues above can be passed to the second one, suchas in the form of prior beliefs of the values. Wethus aim to build an integrated method with betterperformance than either method individually.

Finally, we plan to integrate these techniques intothe Interactive Learning and Segmentation Toolkit(ilastik) [6].

ACKNOWLEDGMENT

This project was advised by David A. Knowles,postdoctoral researcher at Stanford University.

In the in the initial segmentation process, weuse Andrew Mueller’s pygco library, which is aPython wrapper for Boykov’s implementation ofhis multi-label graph cuts method in C++. Ourimplementation of the Gaussian process builds offof the open source PYGP Gaussian process toolboxfor Python.

The author would also like to thank AndrewConnolly and Mitko Veta for personally providingtheir H&E slides for use in our data set.

REFERENCES

[1] Y. Al-Kofahi, W. Lassoued, W. Lee, and B. Roysam. Im-proved Automatic Detection and Segmentation of Cell Nucleiin Histopathology Images. IEEE Transactions on BiomedicalEngineering, 57(4):841–852, April 2010.

[2] Y. Boykov, O. Veksler, and R. Zabih. Fast ApproximateEnergy Minimization via Graph Cuts. IEEE Transactions onPattern Analysis and Machine Intelligence, 23(11):1222–1238,November 2001.

[3] E. D. Gelasca, J. Byun, B. Obara, and B. Manjunath. Evaluationand benchmark for biological image segmentation. In IEEEInternational Conference on Image Processing, Oct 2008.

[4] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes forMachine Learning. The MIT Press, 2005.

[5] T. Shepherd, S. J. Prince, and D. C. Alexander. Interactivelesion segmentation with shape priors from offline and onlinelearning. Medical Imaging, IEEE Transactions on, 31(9):1698–1712, 2012.

[6] C. Sommer, C. Straehle, U. Koethe, and F. A. Hamprecht.”ilastik: Interactive learning and segmentation toolkit”. In 8thIEEE International Symposium on Biomedical Imaging (ISBI2011), 2011.

[7] M. Veta, P. J. van Diest, R. Kornegoor, A. Huisman, M. A.Viergever, and J. P. W. Pluim. Automatic Nuclei Segmentationin H&E Stained Breast Cancer Histopathology Images. PLoSONE, 8, 2013.