nbthesis final digital

8/11/2019 Nbthesis Final Digital

1/185

Departamento de Informtica

U O

Tesis Doctoral

Soft computing and machine learning forimage segmentation by deformable models

Nicola Bova


2/185


3/185

U O

Soft computing and machine learning forimage segmentation by deformable models

Memoria que presenta

Nicola Bova

Para optar al grado de Doctor por la Universidad deOviedo

Septiembre de 2013

Directoresscar Cordn Garcascar Ibez Panizo

Departamento de Informatca


4/185


5/185

UNIVERSIDAD DE OVIEDO

FOR-MAT-VOA-009-BIS

FOR-MAT-VOA-009-BIS

AUTORIZACIN PARA LA PRESENTACIN DE TESIS DOCTORAL

Ao Acadmico: 2013/20141.- Datos personales del autor de la TesisApellidos: Bova Nombre: Nicola

DNI/Pasaporte/NIE: Y1424997-T Telfono: 633211739 Correo electrnico:[email protected]

2.- Datos acadmicosPrograma de Doctorado cursado: Ingeniera Informtica

rgano responsable: Comisin Acadmica del Programa de Doctorado

Departamento/Instituto en el que presenta la Tesis Doctoral: Informtica

Ttulo definitivo de la TesisEspaol/Otro Idioma: Soft computing yaprendizaje automtico para la segmentacinde imgenes con modelos deformables

Ingls: Soft computing and machine learningfor image segmentation by deformablemodels

Rama de conocimiento: Inteligencia Artificial y Soft Computing

3.- Autorizacin del Director/es y Tutor de la tesisD/D:Oscar Cordn Garca

DNI/Pasaporte/NIE:45281118-Y

Departamento/Instituto:Universidad de Granada. Dpto. de Ciencias de la Computacin e Inteligencia Artificial

D/D:Oscar Ibez Panizo

D/D:44432787-F

Departamento/Instituto:European Centre for Soft ComputingAutorizacin del Tutor de la tesisD/D:Oscar Ibez Panizo

DNI/Pasaporte/NIE:44432787-F

Departamento/Instituto:European Centre for Soft Computing

Autoriza la presentacin de la tesis doctoral en cumplimiento de lo establecido en elArt.29.1 del Reglamento de los Estudios de Doctorado, aprobado por el Consejo de

Gobierno, en su sesin del da 21 de julio de 2011 (BOPA del 25 de agosto de 2011)

En Mieres, a 16 de Septiembre de 2013

Director/es de la Tesis Tutor de la Tesis

Fdo.: scar Cordn Garca Fdo.: scar Ibez Panizo Fdo.: scar Ibez Panizo

SR. DIRECTOR DE DEPARTAMENTO DE SR. DIRECTOR DE DEPARTAMENTO DE INFORMATICASR. PRESIDENTE DE LA COMISIN ACADMICA DEL PROGRAMA DE DOCTORADO EN INGENIERIAINFORMTICA


6/185

To those I love


7/185

Acknowledgment

First of all, I would like to point out that this work has been funded by the European Com-mission under the MIBISOCproject (Grant Agreement: 238819, within the action Marie CurieInitial Training Network of the 7FP) and is supported by the SpanishMinisterio de Economa

y Competitividadunder project TIN2012-38525-C02-01 (SOCOVIFI2).Then, I would like to gratefully and sincerely thank my advisors, scar Cordn and scar

Ibez, for letting me be a part of the MIBISOC project1 and for the endless motivation, sup-port, guide, and friendship they gave me throughout all the phases of my doctorate. Theirmentorship was paramount in providing a well rounded experience consistent with my long-term career goals.

I feel the need to express gratitude to Stefano Cagnoni, for giving me the opportunity tostart this journey.

I wish to thank Viktor Gal, for his friendship and for the help in developing the systemdetailed in Chapter3.

Moreover, I cannot forget Li Bai, who made it possible for me to go on a secondment ather department in Nottingham University, Nottingham, UK and for the long talks we had onmany interesting topics.

I express gratitude also to Olivier Colliot and Eric Bardinet, for letting me go on a second-ment at the Brain and Spine Institute, La Piti-Salptrire Hospital, Paris, France.

Besides, I would like to thank Krzysztof Trawiski, for his friendship and for the kindsuggestions about the application of ensemble classifiers.

I show gratitude to Andrea Valsecchi for his friendship and for the support in performingthe statistical analysis corresponding to Sec. 2.6.4.

I am also grateful to Arnaud Quirin, for his friendship and for his helpful suggestionsabout clustering.

I wish to thank David Feltell, for his kind support and for providing a level set implemen-tation.

I cannot forget Jorge Novo, as part of the code related to DE for TAN optimization (Sec.2.6) was provided by him and the VARPA group, University of A Corua, Spain.

Moreover, I am grateful to Luciano Sanchez, for his precious help on the understandingof a machine learning method.

I also show gratitude to Sergio Damas, for having taken care of my needs during the timeI spent at the European Centre for Soft Computing.

1MIBISOC: Medical Imaging Using Bio-inspired and Soft Computing is a Marie Curie Initial Training Net-work granted by the European Commission within the Seventh Framework Program (FP7 PEOPLE-ITN- 2008).

3


8/185

4

I feel the need to thank the University of Oviedo and, above all, the European Centre forSoft Computing, where I had the opportunity to learn part of the techniques that made thisdissertation possible while, at the same time, working in a happy and nice environment.

Besides, I cannot forget the colleagues and, above all, the friends of the MIBISOC projectwith whom I shared many interesting moments. I wish you all the best throughout yourfuture careers.

Moreover, I would like to thank all the new friends I made during my three years inAsturias. I really had an amazing time while spending these moments with you. But I cannotforget the quantity of friends I have in Italy (and abroad). I really miss you all, guys.

Finally, and most importantly, I feel the need to thank all my family, those who are near,those who are distant, and those who are too much far away, now. I love you all.

In particular, I am eternally grateful to my parents, for their endless love and becausewhat I am today is only because of them.

I would like to thank Angie and Ale, I am sure you are as proud of me as I am of you.The last words are for Caterina. Her support, encouragement, quiet patience and unwa-

vering love were the pillars upon which I built the past ten years. Thank you for being sucha sweet companion along this lengthy journey we call life.


9/185

Contents

Contents 5

Resumen 9

Abstract 11

Introduction 13

1 Preliminaries 21

1.1. Deformable models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.1.1. Parametric deformable models . . . . . . . . . . . . . . . . . . . . . . . 231.1.2. Geometric deformable models . . . . . . . . . . . . . . . . . . . . . . . 35

1.2. Soft Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431.3. Evolutionary Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

1.3.1. Conceptual Foundations of Evolutionary Computation . . . . . . . . . 441.3.2. The scatter search template . . . . . . . . . . . . . . . . . . . . . . . . . 461.4. Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

1.4.1. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521.4.2. Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531.4.3. Random Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

1.5. Visual descriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551.5.1. Haralick features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561.5.2. Gabor filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601.5.3. Local binary patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611.5.4. Histogram of oriented gradients . . . . . . . . . . . . . . . . . . . . . . 62

2 New advances in Topological Active Nets: model extension and global optimiza-

tion 65

2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652.2. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

2.2.1. Codification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 672.2.2. Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 672.2.3. Fitness Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 692.2.4. Critical Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

2.3. Extended Topological Active Net . . . . . . . . . . . . . . . . . . . . . . . . . . 712.3.1. Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 712.3.2. Optimization process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

5


10/185

6 CONTENTS

2.3.3. Complementary tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 822.4. Extended Topological Active Net model performance study . . . . . . . . . . 84

2.4.1. Experimental design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 842.4.2. Image datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 862.4.3. Analysis of the obtained results . . . . . . . . . . . . . . . . . . . . . . . 87

2.5. A Scatter Search Framework for Extended Topological Active Nets optimization 922.5.1. Motivation for the use of Scatter Search in ETAN optimization . . . . . 932.5.2. Scatter Search-based framework overview . . . . . . . . . . . . . . . . . 942.5.3. Objective function definition: internal energy terms . . . . . . . . . . . 962.5.4. Diversity function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 982.5.5. Diversification generation method . . . . . . . . . . . . . . . . . . . . . 992.5.6. Solution combination method . . . . . . . . . . . . . . . . . . . . . . . . 100

2.6. Performance study for the optimization of Extended Topological Active Nets

with Scatter Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1062.6.1. Experimental design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1062.6.2. Image dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1072.6.3. Analysis of the obtained results . . . . . . . . . . . . . . . . . . . . . . . 1082.6.4. Statistical analysis of the results . . . . . . . . . . . . . . . . . . . . . . . 110

2.7. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

3 Deformable models supervised guidance: a novel paradigm for automatic seg-

mentation 113

3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1133.2. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

3.2.1. Machine learning-based deformable model initialization approaches . 1163.2.2. Machine learning-based deformable model energy term generation ap-

proaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1183.2.3. Machine learning and deformable models for image recognition and

understanding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1203.2.4. Machine learning-based deformable model guidance approaches . . . 121

3.3. A general framework for deformable models supervised guidance . . . . . . . 1233.4. A specific implementation of the framework for medical image segmentation 127

3.4.1. Localizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1273.4.2. Deformable Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1293.4.3. Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

3.4.4. Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1323.4.5. Integration mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1333.5. Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

3.5.1. SCR database: lungs segmentation in chest radiographs . . . . . . . . 1373.5.2. Allen Brain Atlas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

3.6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

4 Conclusions and future work 149

4.1. Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1494.2. Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

Conclusiones y trabajos futuros 155


11/185

CONTENTS 7

Conclusiones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155Trabajos futuros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

Bibliography 161

Abbreviations 179


12/185


13/185

Resumen

La Segmentacin de Imgenes (SI) es una tarea clave en Visin por Ordenador basada endividir unaimagen en sus regiones constituyentes, las cualescomparten ciertas caractersticasvisuales.Entre las tcnicas de SI, los Modelos Deformables (MDs) son enfoques prometedores

que abordan el problema a travs de la explotacin de los datos de la imagen, junto conconocimientoa priori. En general, los MDs son curvas que detectan caractersticas de intersen imgenes por medio de la minimizacin de una funcin de energa, cuyo mnimo globaldebe corresponder a la posicin ideal del MD. Por lo tanto, el proceso de ajuste del modelose aborda como un problema de optimizacin numrica.

A pesar del xito de los MDs, estas tcnicas de SI an presentan varios problemas. Al-gunos MDs tienen dificultades para manejar cambios topolgicos avanzados y los mtodosde optimizacin comunes pueden provocar imprecisiones, es decir, ptimos locales desde elpunto de vista de optimizacin. Adems, es posible que los MDs tengan problemas relacio-nados con la forma en la que se ajustan. De hecho, la definicin adecuada de la funcin deenerga es crtica para la SI. Expresar la correlacin entre el mnimo global de la funcin yla segmentacin ideal con una formulacin matemtica slida es una tarea difcil, si no im-posible, si queremos que dicha formulacin proporcione resultados precisos en diferentesescenarios de SI. Estos inconvenientes reducen la aplicabilidad de tcnicas de SI basadas enMDs.

En esta tesis, proponemos distintas soluciones a los problemas anteriores. Para ello, me-joramos las tcnicas de SI basadas en MDs mediante la aplicacin de mtodos procedentesde dos reas de investigacin distintas: la Computacin Evolutiva (CE) y el Aprendizaje Au-tomtico (AA). Mientras que la primera constituye una clase de mtodos de optimizacineficientes y bio-inspirados utilizados para proporcionar soluciones a problemas computacio-nalmente difciles, la segunda engloba algoritmos que pueden aprender modelos a partir delos datos que describen el comportamiento de un sistema real. Por un lado, nos centramos enun MD especfico, las Mallas Topolgicas Activas (MTAs). Una MTA es un MD que integracapacidades de SI basadas tanto en regiones como en fronteras. Se define como un conjun-to de nodos relacionados entre s y dispuestos como una malla deformable. A pesar de suscaractersticas prometedoras, la aplicabilidad de las MTAs es reducida debido a limitacionestanto en la gestin de los cambios topolgicos como en el mtodo de optimizacin emplea-do, que puede conducir a ptimos locales de la funcin de energa. Esta ltima cuestin se haabordado mediante la incorporacin de las MTAs en un marco de bsqueda global basadoen CE. Sin embargo, mientras que las propuestas anteriores han sido eficaces en evitar losmnimos locales, han fracasado en disear operadores evolutivos adecuados para combinarde manera efectiva las mallas. En esta tesis, introducimos una versin extendida de la MTAcon el fin de solucionar los problemas del modelo original y abordamos la tendencia de las

9


14/185

10 RESUMEN

MTAs a quedar atrapadas en mnimos locales incorporando el MD en un novedoso marco debsqueda global basado en un algoritmo evolutivo avanzado, la Bsqueda Dispersa.

Por otra parte, proponemos un marco genrico y flexible para la segmentacin automticade imgenes empleando tcnicas de AA que ejercen un control directo sobre el ajuste del con-torno del MD. Adems, ofrecemos una implementacin del sistema propuesto mediante lainteraccin de un conjunto especifico de componentes elegidos con el objetivo de segmentarimgenes mdicas de distinta naturaleza.

Los mtodos propuestos se comparan con mtodos de SI del estado del arte. En todos loscasos nuestros modelos han demostrado ser competitivos e incluso mejores que los compe-tidores en los conjuntos de imgenes empleados.

El trabajo realizado ha dado como resultado dos artculos de revistas JCR y dos artculosde congreso. Est previsto un tercer artculo que ser sometido a una revista JCR prxima-mente.


15/185

Abstract

Among computer vision processes, Image Segmentation (IS) is a key task. It subdividesan image into its constituent regions or objects sharing certain visual characteristics. AmongIS techniques, Deformable Models (DMs) are promising approaches that tackle the problem

by exploiting constraints derived from the image data together with specifica prioriknowl-edge. Generally, DMs are curves that detect features of interest in images by means of theminimization of an energy function. The function is defined such that its global minimumcorresponds to the ideal DM position. Therefore, the model adjustment process is tackled asa numerical optimization problem.

Regardless the extensive and successful use of DMs, these SI techniques are still affectedby several issues. At first, some DMs have difficulties in managing advanced topologicalchanges and common optimization methods can lead to inaccuracies, that is, local optimain the sense of optimization. Besides, it is possible that DMs in general present problemsrelated to the the way they are adjusted. In fact, since the global minimum of the energyfunction should correspond to the ideal segmentation result, the proper definition of this

function is critical to get a good outcome of the segmentation process. However, expressingthis correlation with a robust mathematical formulation is often a daunting task, especiallywhen applied to different IS scenarios. These reasons greatly reduce the applicability of DMin IS.

In this dissertation we aim at providing solutions to the DM problems discussed so far. Todo so, we improve DM-based IS techniques by applying methods from both the EvolutionaryAlgorithms (EAs) and Machine Learning (ML) fields. While the former constitutes a class ofefficient, bio-inspired optimization methods used to provide solutions to computationallyintractable problems, the latter concerns the study of algorithms that can learn models fromdata.

On the one hand, we focus on a specific DM, Topological Active Nets (TANs). A TAN is aDM integrating region-based and boundary-based segmentation capabilities. It is defined bya set of interrelated nodes arranged as a deformable mesh. Despite their promising features,TANs have experienced a limited adoption due to limitations regarding difficulties in man-aging topological changes and the employed local search-based optimization method, whichcan lead to local optima of the optimization function. The latter issue has been addressedby embedding TANs in a EA-based global search framework, able to consider multiple al-ternatives in the segmentation process. However, while previous EA-based proposals wereeffective in avoiding local minima, they failed to design proper evolutionary operators ableto effectively combine nets, thus negating the main advantage of a global search approach.To solve these issues, we introduce an extended version of the TAN model by providing asolution to the problems of the original model. Then, we tackle the tendency of the TAN to

11


16/185

12 ABSTRACT

get stuck in local minima of the energy function by embedding the DM in a novel EA-basedglobal search framework.

On the other hand, we propose an accurate, flexible, and general purpose system for au-tomatic IS employing ML techniques to perform a direct guidance of the DM contour. Weprovide an implementation of the proposed system designed as the interaction of a specificset of components. They were chosen aiming at providing accurate results in different med-ical IS scenarios.

The proposed methods were compared with state-of-the-art IS techniques. They werecompetitive or even outperformed them on various image datasets.

The work carried out resulted in two JCR journal articles and two conference papers. Athird JCR journal article will be submitted in the near future.


17/185

Introduction

Artificial Intelligence (AI)is a branch of computer science and engineering dealing withmaking intelligent machines [145]. It has become an essential part of the technology indus-try and many of the most difficult problems in computer science have been tackled with AI

techniques. AI goals include reasoning, knowledge, planning, learning, communication, percep-tionand the ability to moveand manipulateobjects[138, 158, 181, 191]. The research carriedout in the field is highly technical and specialized. It is deeply divided into several subfieldsfocused on specific problems and applications. Interesting opportunities can arise from theintegration of the abilities and skills coming from different AI subfields.

Among other AI branches,Soft Computing (SC)[22, 236] is employed to tackle reasoning,learning, andplanningtasks. In particular, SC refers to a family of robust, intelligent systems,that in contrast to precise, traditional modes of computation, are able to deal with compu-tationally expensive tasks, such as NP-complete problems, even in case of vague, noisy, andpartial knowledge.

Within SC,Evolutionary Algorithms (EAs)constitute a class of search and optimization

methods with the ability to efficiently provide a solution to difficult and often computation-ally intractable problems. EAs imitate the principles of natural evolution [82,95]. Indeed, anEA maintains and processes a population of individuals, each one describing a set of geneticparameters that encode a candidate solution to the optimization problem. The fitness of anindividual with respect to the optimization task is described by a scalar objective function.Genetic operators such as recombination and mutation are applied to the parents in order togenerate new candidate solutions. The result of this evolutionary cycle is a set of more andmore suitable solutions to the optimization problem, according to the Darwinian principleofsurvival of the fittest.

Along with SC, Machine Learning (ML) is the branch of AI that concerns learning. It stud-ies the design of systems that improve automatically through experience. Hence, ML con-cerns the construction and study of computer algorithms that can learn from data [5]. Coreaspects of ML are representation and generalization. In the first place, representation of datainstances and functions evaluated on these instances are part of all ML systems. Moreover,generalization is a highly desirable property as it is related to the chance to perform well onunseen data instances. The ability to represent and generalize knowledge to provide accuratepredictions over new data is the essence of learning.

Before applyingreasoningand learningon a given set of information items, however, itis necessary to deal with the techniques that perceivethese items, that is acquire and gatherthem. Withinthe large group of sources of information, vision is probably the most importantone. It is the most advanced of human senses, so it is not surprising that images play thesingle most important role in human perception [84].

13


18/185

14 INTRODUCTION

Computer vision is the enterprise of automating and integrating a wide range of processesand representations for visionperception[12]. It includes methods for acquiring, processing,analyzing, and understanding images. When applied within AI, computer vision researchesmethods aiming at duplicating the capabilities of human vision. It does so by electronicallyperceiving and understanding an image [203] considering models constructed with the aidof geometry, physics, statistics, and learning theory [74].

Unlike humans, who are limited to the visual band of the electromagnetic spectrum,imaging machines cover almost the entire electromagnetic spectrum, ranging from gammato radio waves. They can operate on images generated by sources that humans are not accus-tomed to associating with images. Thus, digital image processing encompasses a wide andvaried field of applications.

There are no clear cut boundaries in the continuum from image processing at one end tocomputer vision at the other. However, one useful paradigm is to consider three types of com-puterized processes in this continuum: low-, mid-, and high-level processes [84]. Low-levelprocesses involve primitive operations such as image preprocessing to reduce noise, contrastenhancement, and image sharpening. A low-level process is characterized by the fact thatboth its inputs and outputs are images. Mid-level processing on images involves tasks suchas segmentation (partitioning an image into regions or objects), description of those objects toreduce them to a form suitable for computer processing, and classification (recognition) of in-dividual objects. A mid-level process is characterized by the fact that its inputs are generallyimages, but its outputs are attributes extracted from those images (e.g., edges, contours, andthe identity of individual objects). Finally, higher-level processing involves giving sense ofan ensemble of recognized objects, as in image analysis, and, at the far end of the continuum,performing the cognitive functions normally associated with vision.

Among the numerous processes in the field, Image Segmentation (IS)[84, 198] is a keytask. It is a critical issue as the quality of its outcomes has a strong influence on the poste-rior image understanding task. IS subdivides an image into its constituent regions or objectsby assigning a label to every pixel such that pixels with the same label share certain visualcharacteristics. Its goal is to simplify and/or change the representation of an image intosomething that is more meaningful and easier to analyze [198]. Segmentation accuracy de-termines the eventual success or failure of computerized analysis procedures. For this reason,considerable care should be taken to improve the probability of rugged segmentation [84].

Some of the practical applications of IS are:

Medical imaging, where it is employed for tasks such as tumor or other pathologies lo-

cation, computer-guided surgery, and diagnosis.Machine vision, where IS is used to provide imaging-based automatic inspection, anal-ysis, process control, and robot guidance in the industry field.

Object detection, with targets from a broad range of fields, such as automotive (pedes-trian, road, lights, etc.) and satellite sensing (roads, forests, crops, etc.).

Content-based image retrieval, where IS is used to search for digital images in large databases.In this case the search analyzes the contents of the image (in terms of colors, shapes,textures, or any other information that can be derived from the image itself) rather thanthe metadata (such as keywords, tags, or descriptions associated with the image).


19/185

15

Video surveillance, where IS is employed to automatically monitor sensible areas but alsofor demanding real-time applications such us traffic control systems.

Recognition, among with other applications, is used for biometric authentication to per-form the automated recognition of individuals on the basis of their biological and be-havioral traits (fingerprint, face, iris, palmprint, retina, hand geometry, signature, andgait).

In particular, medical IS is of immense practical importance in medical informatics. Medi-cal images, such as ComputedAxial Tomography(CAT), MagneticResonanceImaging(MRI) ,Ultrasound, and X-Ray, are processed and analyzed to extract meaningful information suchas volume, shape, motion of organs; to detect abnormalities, and to quantify changes infollow-up studies [72]. Manual segmentation of medical images by the radiologist is notonly a tedious and time consuming process, but also it is not very accurate especially with

the increasing medical imaging modalities and unmanageable quantity of images that needto be examined. Automated image segmentation [72], which aims at automated extraction ofobject boundary and region features, plays a fundamental role in understanding image con-tent. This is a challenging task due to regions with boundary insufficiencies, lack of texturecontrast betweenRegions Of Interest (ROIs)and background, non-homogeneous intensitieswithin the same class of tissue, high complexity of anatomical structures, as well as a highvariability. Given its difficult nature and the large number of algorithms competing in thefield, medical imaging is an ideal test terrain for novel generic IS approaches.

On a general basis, the level to which the object subdivision is carried depends on theproblem being solved. That is, IS should stop when the objects of interest in the imagesconsidered for a problem have been isolated. While some applications, such us automated

inspection in assembly lines, often only consider simple image primitives and make use ofstructured environments to perform specific anomalies detection, segmentation of non trivialimages is one of the most difficult tasks in image processing.

IS algorithms are generally based on one of the two basic properties of intensity values:discontinuity and similarity [84]. In the first category, the approach is to partition an imagebased on abrupt changes in intensity, such as edges in an image. The principal approaches inthe second category are based on partitioning an image into regions that are similar accord-ing to a set of predefined criteria. Thresholding, region growing, and region splitting andmerging are examples of methods in this group.

Among IS techniques,Deformable Models (DMs)[147] are promising and actively re-searched model-based approaches to tackle a huge variety of IS problems. Generally, DMsare curves, surfaces or solids that detect features of interest in images by means of the min-imization of an energy function. Ideally, the energy function is defined in such a way thatits value, when calculated for a DM correctly segmenting the target object, corresponds toits global minimum. Therefore, the model adjustment process is tackled as a numerical opti-mization (energy minimization) problem. Thewidely recognizedpotency of DMs stems fromtheir ability to segment, match, and track images by exploiting constraints derived from theimage data (in a bottom-up fashion) together with a priori knowledge about the location,size, and shape of these structures (following a top-down approach) [147].

DMs are split into two families, parametric and geometric DMs [154]. While the formerrepresent the model explicitly and are guided by acting on their control points, the latterrepresent the model implicitly as a level set of a higher-dimensional scalar function. Both


20/185

16 INTRODUCTION

families have advantages over each other. On the one hand, parametric DMs are quick andeasy to deform but havesome issues regarding topological changes and adaptability to highlycomplex shapes. On the other hand, geometric DMs can handle topological changes easilybut are slower than their parametric counterparts as well as it is harder to enforce restrictionson their shape.

Regardless the extensive and successful use of DMs for IS in recent years [ 92, 147], theyare still affected by several issues. On the one hand, some DMs have difficulties in managingadvanced topological changes, heavy local deformations, and the definition of the energyfunction. Moreover, employed optimization methods (often based on a local search) can leadto result inaccuracies, that is, local optima in the sense of optimization. On the other hand,it is possible that DMs present problems related to the the way they are adjusted. As said,the usual approach to develop IS using DMs is by adjusting their models by minimizing thevalue of an associated energy function. Usually, this function is based on image features,prior knowledge, and the state of the model. Since the global minimum of the energy func-tion should correspond to the ideal segmentation result, the proper definition of this functionis critical to the outcome of the segmentation process. Actually, in case of loose or suboptimalcorrelation between energyfunction and segmentation results, even an ideal optimizerwouldyield unsatisfactory outcomes. However, in the general case, expressing this correlation witha robust mathematical formulation is a daunting task. In fact, energy definitions greatly varyas a function of the application, image modality, type of DM in use, and target structure. Inparticular, they generally use a large set of obscure weighting coefficients that are hard andtime-consuming to tune [194]. For these reasons, the need to define an appropriate func-tion reflecting the energy-result greatly reduces the applicability of optimization-based DMsegmentation techniques.

The aim of this dissertation is to provide solutions to the DM problems discussed so farin order they can perform IS in a more accurate way. On the one hand, SC and, in particular,EAs represent a very attractive technique for this task as they can provide effective solutionswithin the standard optimization approach. In fact, they are able to considering multiplesolutions in the search space thus having more chances to avoid local minima of the energyfunction. On the other hand, ML provides a set of appealing techniques to tackle the adjust-ment of DMs from a different perspective. In fact, an alternative approach could be a noveltype of DM guidance process in which the necessary information is derived from a set of ex-ample data obtained from the images to be segmented, removing the need to a prioridefinethe energy function.

In order to do so, we will work with two different kinds of DMs,Topological Active Nets(TANs)andLevel Sets (LSs). While the former will be considered to develop solutions basedon the standard optimization-based approach, the latter will be used to illustrate the appli-cation of an alternative ML-based approach.

TANs [8,9,28] are attractive parametric DMs integrating region-based and boundary-based segmentation capabilities. The model is defined by a set of interrelated nodes arrangedas a deformable mesh. The nodes are divided in two subsets dealing with different tasks.While the external nodes fit the target objects boundaries, the internal ones adapt to the in-ner regions of the targets. Topological changes are handled by cutting links between nodesand, consequently, by commuting the type of the nodes as the model adapts to multiple ob-jects. The TAN is adjusted according to a specific energy function taking into account imagefeatures along with spatial relationships among neighboring nodes. Since each TAN config-uration has a value of the energy function associated, the segmentation process is tackled as


21/185

17

an energy minimization problem.TANs have been applied to different IS problems including iris location [162], stereo

matching [192], and medical CT image segmentation [102, 160, 163, 167] with successful re-sults. However, despite their promising features, TANs have experienced a limited adoptiondue to two strong limitations. On the one hand, the model is complex and has difficultiesin managing advanced topological changes, as well as it has issues related to the externalenergy definition, mesh correctness and integrity, and heavy local deformations. None ofthe works in the TAN literature has addressed these issues, so far. On the other hand, theemployed local search-based optimization method can lead to local optima of the optimiza-tion function. In this case, however, a reduced number of researchers addressed this issueby embedding TANs in a global search framework [102, 160, 163, 167], able to consider mul-tiple alternatives in the segmentation process. In order to do so, they hybridized TANs withan EA, achieving encouraging results. In fact, when dealing with embedding TANs in aglobal search framework considering multiple solutions, EAs represent a very appropriateapproach. On the one hand, EA-based proposals [102, 160, 163, 167]are inherently able toconsider multiple segmentation candidates at the same time, significantly reducing the riskto fall in local minima. On the other hand, evolutionary-based mechanisms provide tools tocombine different solutions, with the capability, in the ideal case, of coalescing meshes withdiverse, or even complementary, characteristics. This fact contributes to enhance the overallquality of the solution while, at the same time, improving the convergence time. However,while previous EA-based works were effective in avoiding local minima, they failed to de-sign proper evolutionary operators able to effectively combine nets. As a consequence, theyrequired very large populations of solutions to operate, thus negating the main advantage ofa global search approach. For these reasons, TANs are ideal candidates for the applicationof advanced, EA-based optimization techniques.

Differently from TANs, LSs are geometric DMs [31, 141] based on curve evolution the-ory [116, 117, 193] and the LS method [173, 195]. In the LS method, when segmenting an N-dimensional image, the curve is represented implicitly as a level set of an (N+1)-dimensionalscalar function. As a result, topology changes can be handled automatically. Thus, the guid-ance of the models does not require to consider the topology while adapting the model, eas-ing the whole process. Given this enticing capability, LSs have been extensively used in med-ical IS [68], a field that demands high accuracy but, at the same time, presents subjectivity inthe ideal segmentation definition attending the particular clinician. For these reasons, LSs areparticularly attractive as the DM of choice for the exploration of novel, ML-based approachesin the DM-based IS field.

Objectives

The aim of this dissertation is to improve DM-based IS techniques employing both stan-dard and alternative design approaches. In particular, the contribution is two-fold. On theone hand, we will focus on a specific parametric DM, TANs, and will aim to enhance theestablished TAN model to overcome the intrinsic model limitations still unaddressed. More-over, we will try to embed the TAN in a different global search framework with specificallydesigned components to improve the whole optimization process in terms of effectivenessand efficiency. On the other hand, we will approach DM-based IS solutions from a differentperspective by exploring ML methods as an alternative to optimization approaches to adapt


22/185

18 INTRODUCTION

the DM. In this case, we aim at proposing a general purpose, ML-based segmentation frame-work able to learn from data, which will be particularly tested considering the use of LSs.Specifically, these objectives are divided into the following ones:

Study the state of the art in TAN optimization.We aim at reviewing all the relevant propos-als in the field, concerning both the model definition and the optimization procedure.We will analyze the discipline to point out current drawbacks in adjusting to complexshapes, external energy definition, and mesh integrity. Moreover, we will detail theexisting EA-based proposals, focusing on the issues related to the population size andthe considered evolutionary operators.

Propose an extension of the TAN model. First of all, we aim at incorporating to TAN anexternal energy definition based on a recent technique performing the convolution ofvector fields. Moreover, we will try to address the ability of the model to tackle topo-

logical changes, such as link cuttings, net division, and hole segmentation to betterdeal with holes or complex shapes of the target objects. Moreover, differently from theoriginal TAN model, we will always consider the integrity and correctness of the meshduring the adjustment procedure. Finally, we aim at endowing the extended TAN witha pre-segmentation phase to perform automatic initialization of the net.

Inclusion of the extended TAN in a convenient global search framework. We aim at address-ing the population size issue of previous TAN evolutionary-based proposals. To do so,we consider embedding TANs in a different EA-based global search framework rely-ing on solution combinations and controlled randomizations. Moreover, we will try tocustomize the framework components intending to develop specific evolutionary op-erators able to effectively deal with the IS problem at hand. To do so, we will considerthe use of the scatter search EA [80, 123], whose flexibility makes it an ideal candidateto be used for the current global optimization task. In particular, we aim at introducingproper solution combination operators able to coalesce meshes in an effective way, aglobal search-suitable internal energy term, an appropriate solution diversity function,and a frequency-memory population generator.

Study the state of the art in ML application to DMs. We aim at reviewing the relevantproposals dealing with the application of ML techniques to the adjustment of DMs. Wewill try to propose a taxonomy according to the different uses given to ML techniques.Special attention will be put on those methods presenting an alternative approach tothe classical energy minimization configuration.

Propose an ML-based generic framework for IS using DMs. We aim at proposing an ISframework able to learn a segmentation model from examples in an automatic fash-ion. Therefore, given a training set of reference images, the system should be able toautomatically locate the target ROI and segment it with minimal human intervention.In particular, the framework will not rely on optimization techniques, removing the ne-cessity to define an energy function. Finally, as the framework will be designed withflexibility in mind, we intend to allow the use of any kind of DM.

Develop an ML-based IS framework implementation for medical imaging. To prove the fea-sibility of a ML-based IS framework, we aim at providing an effective implementationtailored to the medical imaging field. We will choose and design specific components


23/185

19

of the generic framework so that the resulting model becomes competitive with state-of-the-art proposals performing the segmentation of different structures in differentmedical image modalities. To do so, we will consider the LS DM.

Analyze the performance of the proposed methods. We aim at validating the designed ISmethods with a proper experimental study. In particular, we intend to compare themwith other state-of-the-art IS algorithms over various image sets of different typolo-gies. To perform this comparison, we will select appropriate IS metrics from the fieldto quantitatively measure the obtained performance. Finally, with the purpose of ex-plaining and interpreting the obtained results, we intend to examine them in detail,also employing statistical analysis when needed.

Structure

In order to achieve the previously described objectives, the current PhD dissertation isorganized in four chapters. The structure of each of them is briefly introduced as follows.

In Chapter1we introduce the preliminary background information of the wide range oftechniques required for a proper understanding of the work developed. First of all, we de-velop a survey of DMs with a particular interest toward those employed in this work. Then,we introduce SC and evolutionary computation, focusing on the Scatter Search (SS) optimiza-tion framework which will be considered in one of our proposals. We also introduce the basisof ML with a focus on the methods considered along this dissertation. Finally, we considerthe techniques to extract characteristic features from images by reviewing some image de-scriptors commonly adopted in literature.

Chapter2is devoted to the research work carried out to study, enhance and evaluateour TAN-based segmentation algorithm. We start reviewing the TAN state-of-the-art ap-proaches. Then, we present an extended version of the TAN model by describing the novelcomponents in depth. We also propose a specific local search method to optimize the ex-tended TAN model. We compare our proposal to a significant set of parametric and geomet-ric DMs on a mix of both synthetic and real-world images. Later, we introduce a novel globalframework for the optimization of the extended TAN, providing motivation for the chosenSS paradigm and detailing the customized components. Finally, we test the new SS-basedframework against the best performing TAN-based global search optimization proposals.

In Chapter3 we present an ML-based procedure as an alternative to the classical opti-mization formulation for DM guidance. To do so, we first study the relevant literature andcategorize common existing approaches. Then, we present a general purpose DM frame-work able to learn the segmentation model from examples extracted from training images.Differently from optimization-based approaches, the proposed system directly guides theDM evolution through common ML algorithms. To show the feasibility of our alternativeapproach, we provide a reference implementation tailored to the medical imaging field us-ing LSs as DM, an ensemble classifier as ML method, the part-based object detector intro-duced in [78] as localizer, and a proper set of image visual descriptors. We test it against alarge set of state-of-the-art segmentation algorithms over two image datasets with differentcharacteristics.

Finally, in Chapter 4 we draw some conclusions bysummarizing the most relevant achieve-ments. We also suggest future work for further improvements related to the DM-based ISresearch line we dealt with in this dissertation.


24/185


25/185

CHAPTER 1Preliminaries

In this chapter we introduce some concepts that we will use in the rest of this dissertation.First, we deal with DMs, theleitmotif of this dissertation. They are a group of methods widelyemployed in computer vision to perform IS. Then, weintroduceSC,an umbrella of techniquesto provide inexact solutions to computationally hard tasks. These techniques are tolerant ofimprecision, uncertainty, partial truth, and approximation. In particular, we focus on EC, aset of methods to tackle optimization problems in an efficient way. Subsequently, we presentML, a branch of artificial intelligence that concerns theconstruction andstudy of systems ableto learn from data. Finally, we review a set of image descriptors widely employed to describeimage properties in computer vision. The size and diversity of this set of concepts, from the

AI and computer vision research fields, is due to the high degree of interdisciplinarity of thiswork.

1.1. Deformable models

In several real-world applications, images are often corrupted by noise and sampling ar-tifacts, which can cause considerable difficulties when applying classical segmentation tech-niques such as edge detection and thresholding. As a result, these techniques either failcompletely or require some kind of postprocessing step to remove invalid object boundariesin the segmentation results. To address these difficulties, DMs have been extensively stud-ied and widely used in medical image segmentation, with promising results. DMs [147] arecurves or surfaces defined within an image domain that can move under the influence ofinternal forces and external forces. While the former forces are defined within the curve orsurface itself, the latter ones are computed from the image data. The internal forces are de-signed to keep the model smooth during deformation. The external forces are defined tomove the model toward an object boundary or other desired features within an image. Byconstraining extracted boundaries to be smooth and incorporating other prior informationabout the object shape, DMs offer robustness to both image noise and boundary gaps and al-low integrating boundary elements into a coherent and consistent mathematical description.Such a boundary description can then be readily used by subsequent applications. More-over, since DMs are implemented on the continuum, the resulting boundary representationcan achieve subpixel accuracy, a highly desirable property for many image processing appli-

21


26/185

22 CHAPTER 1. PRELIMINARIES

cations. Nowadays, DMs have grown to be one of the most active and successful researchareas in image segmentation [147]. Various names, such as snakes, active contours or sur-faces, balloons, and deformable contours or surfaces, have been used in the literature to referto DMs. Fig.1.1shows an example of a segmentation task by a DM.

(a) Input image (b) DM segmentation (c) Output image

Figure 1.1: Example of segmentation of a medical image by a DM. (a) is the image to be seg-mented, (b) is the final adjustment of the DM (in red), and (c) is the result image identifyingtwo segments, the object (black) and the background (white).

There are basically two types of DMs: parametric DMs[7, 41, 114, 146] andgeometric DMs[31, 32, 141, 225].

Parametric DMs represent curves and surfaces explicitly in their parametric forms dur-ing deformation. This representation allows direct interaction with the model and can leadto a compact representation for fast real-time implementation. Adaptations of the modeltopology such as splitting or merging parts during the deformation, can, however, be diffi-cult using parametric models. Geometric DMs, on the other hand, can handle topologicalchanges naturally. These models, based on the theory of curve evolution[6, 116, 117, 193]and the LS method [173], represent curves and surfaces implicitly as a level set of a higher-dimensional scalar function. Their parameterizations are computed only after complete de-formation, thereby allowing topological adaptivity to be easily accommodated. Despite this

fundamental difference, the underlying principles of both methods are very similar.In spite of the great similarities between the models, it is surprising to check the multi-

plicity of terms refering to practically the same concepts, distinguished in many cases by veryminor aspects: Deformable Models [210], Deformable Templates [107], Active Shape Mod-els [46], Active Contour Models/Deformable Contours/Snakes[114], Deformable Surfaces[40, 148, 154], Active Appearance Models [43], or Statistical Shape Models [93]. We preferredto study DMs properties and models dividing them in two typologies: parametric (Section1.1.1) and geometric (Section1.1.2). Among them, we will pay special attention to Topolog-ical Active Nets (Section1.1.1.3) and Level Sets (Section1.1.2.2), as they are the two DMsemployed as parts of the proposals introduced in Chapter 2 and 3 of this dissertation, respec-tively.


27/185

1.1. DEFORMABLE MODELS 23

1.1.1. Parametric deformable models

Two different types of formulations for parametric DMs exist: an energy minimization

formulation and a dynamic force formulation. Although these two formulations lead to sim-ilar results, the former has the advantage of its simplicity whereas the second formulationhas the flexibility of allowing the use of more general types of external forces. In this sec-tion, we concentrate on the first formulation, above all because this is the formulation usedin TANs. We then present several commonly used external forces that can effectively attractDMs toward the desired image features. Finally, we review a set of notable parametric DMswe employ or compare against along this PhD thesis.

1.1.1.1. Energy minimization formulation

The basic premise of the energy minimization formulation of deformable contours is to

find a parametrized curve that minimizes the weighted sum of internal energy and potentialenergy. The internal energy specifies the tension or the smoothness of the contour. Thepotential energy is defined over the image domain and typically possesses local minima atthe image intensity edges occurring at object boundaries.

To find the object boundary, parametric curves are initialized within the image domain,and are forced to move toward the potential energy minima under the influence of both theseforces.

Mathematically, a deformable contour is a curve, defined as X(s) = (X(s), Y(s)), whichmoves through the spatial domain of an image to minimize the following energy functional[72]:

(X) =

S(X) +

P(X).

The first term is the internal energy functional and is defined to be

S(X) =12

10

(s)

Xs2 + (s)

2Xs22ds.

The first-order derivative discourages stretchingand makes the model behave like an elas-tic string. The second-order derivative discourages bending and makes the model behave likea rigid rod. The weighting parameters(s)and (s)can be used to control the strength ofthe models tension and rigidity, respectively. In practice, (s)and (s)are often chosen tobe constants.

The second term is the potential energy functional and is computed by integrating a po-tential energy functionP(x, y)along the contour X(s):

P(X) = 10

P(X(s))ds.

The potential energy functionP(x, y)is derived from the image data and takes smaller val-ues at object boundaries as well as other features of interest. Given a gray-level imageI(x, y)viewed as a function of continuous position variables(x, y), a typical potential energy func-tion designed to lead a deformable contour toward step edges is

P(x, y) =

we|

[G

(x, y)

I(x, y)]|2,


28/185


where weis a positive weighting parameter, G(x, y) is a two-dimensional Gaussian functionwith standard deviation, is the gradient operator, and is the 2D image convolution op-erator. If the desired image features are lines, then the appropriate potential energy functioncan be defined as follows:

P(x, y) =wl[G(x, y) I(x, y)],wherewlis a weighting parameter. Positivewlis used to find black lines on a white back-ground, while negativewlis used to find white lines on a black background. For both edgeand line potential energies, increasingcan broaden its attraction range.

Regardless of the selection of the exact potential energy function, the procedure for mini-mizing theenergy functional is the same. The problem of finding a curveX(s) that minimizesthe energy functional is known as a variational problem[52]. It has been shown that thecurve that minimizesmust satisfy the following Euler-Lagrange equation [41, 114]:

s

X

s

2s2

2

Xs2

P(X) = 0. (1.1)1.1.1.2. External forces

In this section, we describe several kinds of external forces for DMs. These external forcesare applicable to every type of DM.

Multiscale Gaussian potential force When using the Gaussian potential force describedearlier,must be selected to have a small value in order for the DM to follow the bound-ary accurately. As a result, the Gaussian potential force can only attract the model toward

the boundary when it is initialized nearby. To remedy this problem, Terzopoulos, Witkin,and Kass [114] proposed using Gaussian potential forces at different scales to broaden itsattraction range while maintaining the models boundary localization accuracy. The basicidea is to first use a large value of to create a potential energy function with a broad val-ley around the boundary. The coarse-scale Gaussian potential force attracts the deformablecontour or surface toward the desired boundaries from a long range. When the contour orsurface reaches equilibrium, the value ofis then reduced to allow tracking of the boundaryat a finer scale. This scheme effectively extends the attraction range of the Gaussian poten-tial force. A weakness of this approach, however, is that there is no established theorem forhow to schedule changes in. The availablead hocscheduling schemes may therefore leadto unreliable results.

Pressure force Cohen [41] proposed to increase the attraction range by using a pressureforce together with the Gaussian potential force. The pressure force can either inflate or de-flate the model. Hence, it removes the requirement to initialize the model near the desiredobject boundaries. DMs that use pressure forces are also known as balloons [41]. The pres-sure force is defined as

Fp(X) =wpN(X),

where N(X) is the inward unit normal of the model at the point Xand wp is a constantweighting parameter. The sign ofwpdetermines whether to inflate or deflate the model andis typically chosen by the user. Later, region information has been used to define wpwith aspatial-varying sign based upon whether the model is inside or outside the desired object


29/185


[182,188]. The value ofwpdetermines the strength of the pressure force. It must be carefullyselected so that the pressure force is slightly smaller than the Gaussian potential force atsignificant edges, but large enough to pass through weak or spurious edges. When the modeldeforms, the pressure force keeps inflating or deflating the model until it is stopped by theGaussian potential force. A disadvantage in using pressure forces is that they may cause theDM to cross itself and form loops[209].

Distance potential force Another approach for extending attraction range is to define thepotential energy function using a distance map as proposed by Cohen and Cohen [42]. Thevalue of the distance map at each pixel is obtained by calculating the distance between thepixel and the closest boundary point, based either on Euclidean distance [59] or Chamferdistance [24]. By defining the potential energy function based on the distance map, one canobtain a potential force field that has a large attraction range. Given a computed distancemap d(x, y), one way of defining a corresponding potential energy function, introduced in[42], is as follows:

Pd(x, y) =wded(x,y)2 .The corresponding potential force field is given by Pd(x, y).

Gradient vector flow In the DM literature, the traditional external force is derived fromthe edge map f(x, y)calculated on the image I(x, y)and having the property that it showslarger values near the image edges. This edge map has three important properties. First,its gradientfhas vectors pointing toward the edges, which are normal to the edges at theedges. Second, these vectors generally have large magnitudes only in the immediate vicinityof the edges. Third, in homogeneous regions, where I(x, y)is nearly constant,

fis nearly

zero. While the first property is desirable, the last two are not because the capture range willbe very small and homogeneous regions will have no external forces whatsoever.

In [231], Xu and Prince developed a new external force for active contours to solve theproblems associated with initialization and poor convergence to concave boundaries.

They defined theGradient Vector Flow (GVF)as the vector field

v(x, y) = [u(x, y), v(x, y)]

that minimizes the energy functional

E= [(u2x+ u

2y+ v

2x+ v

2y) + |f|2|v f|2]dxdy.

This variational formulation makes the result smooth when there is no data and keeps vnearly equal to the gradient of the edge map when it is large, but forcing the field to be slowly-varying in homogeneous regions. The parameter is a regularization parameter governingthe trade-off between the first and second terms in the integrand and should be set accordingto the amount of noise present in the image.

Using the calculus of variations[52] it can be shown that the GVF field can be found bysolving the following Euler equations

2u (u fx)(f2x + f2y ) = 0, 2v (v fy)(f2x+ f2y ) = 0

where

2 is the Laplacian operator.


30/185


The snake using the GVF field provides a large capture range and the ability to captureconcavities by diffusing the gradient vectors of an edge map generated from the image. Al-though the GVF field has been widely used and improved in various models, it still showssome disadvantages, such as high computational cost, noise sensitivity, parameter sensitivity,and the ambiguous relationship between the capture range and parameters [129].

An example of the GVF force is shown in Fig. 1.2.

Figure 1.2: An example of distance potential force field. (a) A U-shaped object, (b) a close-upof its boundary concavity, and (c) the distance potential force field within the concavity.

Vector Field Convolution Trying to overcome the previous limitations, theVector FieldConvolution (VFC)[129]is calculated convolving a vector field kernel with an edge mapderived from the image. Given a gray-scale image I(x, y),itsedgemapisf(x, y) =|(GI)|,whereGis a 2-D Gaussian function with standard deviation and the operatordenotesthe linear convolution. The VFC is defined as

fvfc(x, y) = [uvfc(x, y), vvfc(x, y)]

and is calculated by convolving the edge map with a vector field kernel

k(x, y) =m(x, y)n(x, y) (1.2)

wherem(x, y)is the magnitude of the vector at (x, y)and n(x, y)is the unit vector pointingto the kernel origin(0, 0):

n(x, y) =

xr

,y

r

,

(except that n(0, 0) = [0, 0]) and where r =

x2 + y2 is the Euclidean distance from thekernel origin. The kernel field has the property that a free particle placed in the field willmove to the kernel origin. If the kernel origin is considered as an edge point, the particle willmove toward the edge.


31/185


The VFC depends on the magnitude term of the vector field kernel m(x, y). In [129], it isdefined as:

m(x, y) = 1

r.

This term is used in order the force field to have a reduced influence on the particle fartheraway of the edges. is a positive parameter controlling the decrease and has a major roledepending on the application. A higher value means that the force field will notbe influencedby features of interest located far away.

An example withr = 5and= 2is shown in Fig. 1.3.

1 3 5 7 9 11

1

3

5

7

9

11

Kernel vector field

Figure 1.3: The vector field kernel.

The VFC external force is then given by:

fvfc(x, y) =f(x, y) k(x, y) = [f(x, y) uk(x, y), f(x, y) vk(x, y)].

Since the edge map is non-negative andlarger near the edges, these will contribute to the VFCexternal force more than homogeneous regions and the free particles will be attracted by theedges1.To enforce a wide range of attraction for edges, a proper value for the rparameter isaroundl/2, withlbeing the largest of the image dimensions.

A sample image along with the gradient of its edge image and the result of the VFC aredepicted in Fig.1.4, respectively.

1.1.1.3. Topological active nets

The Active Nets model [211] came up as a DM that integrates features of region-based andboundary-based segmentation techniques. This way, the model detects the inner topologyof the objects and fits their contours. The TANs [8, 9, 28] are an extension of the originalactive net model that solves some intrinsic problems to the DMs such as the initializationproblem. It also has a dynamic behavior that allows topological local changes in order toperform accurate adjustments and find all the objects of interest in the scene.

A TAN is a discrete implementation of an elastic two-dimensional mesh with interrelatednodes[9]. The structure of a small TAN is depicted in Fig.1.5. As this figure shows, themodelhas two kinds of nodes: internal and external. Each kind of node represents different features

1Working with gray-scale images, the vectors of the field will be attracted by and will point towards thebrighter areas of the image.


32/185


(a) (b)

1.0 11.5 22.0 32.5 43.0 53.5 64.0

1.

0

11.

5

22.0

32.

5

43.0

53.

5

64.0

Standard VFC

(c)

Figure 1.4: (a) original image; (b) Sobel gradient of (a); (c) standard VFC of (a).

of the objects: the external nodes fit the edges of the objects whereas the internal nodes modelthe internal topology of the object. Therefore, this model allows the integration of informa-tion based on discontinuities and information based on regions in the segmentation process.The former is associated to external nodes and the latter to internal nodes.

Figure 1.5: A 5x5 mesh.

A TAN is defined parametrically as v(r, s) = (x(r, s), y(r, s))where(r, s)[0, 1] [0, 1].The mesh deformations are controlled by an energy function defined as follows:

E(v(r, s)) =

10

10

Eint(v(r, s)) + Eext(v(r, s))drds

whereEintand Eextare the internal and the external energy of the TAN, respectively. Theinternal energy controls the shape and the structure of the mesh whereas the external energyrepresents the external forces which govern the adjustment process.

The internal energy depends on first and second order derivatives which control contrac-tion and bending, respectively. The internal energy term is defined by the following expres-sion:

Eint(v(r, s)) =(|vr(r, s)|2 + |vs(r, s)|2)+(|vrr(r, s)|2) + |vrs(r, s)|2 + |vss(r, s)|2)

where subscripts represents partial derivatives, and andare coefficients that control thefirst and second order smoothness of the net. In order to calculate the energy, the parameter


33/185


domain [0, 1][0, 1] is discretized as a regular grid defined by the internode spacing (k, l) andthe first and second derivatives are estimated using the finite differences technique. On onehand, the first derivatives are computed using the following equations to avoid the centraldifferences:

|vr(r, s)|2 =d+r(r, s)2 + dr(r, s)2

2 |vs(r, s)|2 =d

+s(r, s)2 + ds(r, s)2

2 ,

where d+ and d are the forward and backward differences respectively, which are computedas follows:

d+r(r, s) =v(r+ k, s) v(r, s)

k dr(r, s) =

v(r, s) v(r k, s)k

d+s(r, s) =v(r, s + l) v(r, s)

l ds(r, s) =

v(r, s) v(r, s l)l

.

On the other hand, the second derivatives are estimated by:

vrr(r, s) = v(r k, s) 2v(r, s) + v(r+ k, s)

k2

vss(r, s) = v(r, s l) 2v(r, s) + v(r, s + l)

l2

vrs(r, s) = v(r k, s) v(r k, s + l) v(r, s) + v(r, s + l)

kl .

Theexternalenergyrepresentsthefeaturesofthescenethatguidetheadjustmentprocess.In the original proposal, it was defined by the following expression:

Eext(v(r, s)) =f[I(v(r, s))] +

|(r, s)|

p(r,s)

1

v(r, s) v(p)f[I(v(p))]

where and are weights, I(v(r, s)) is theintensity value of the original image in the positionv(r, s), (r, s)is the neighborhood of the node(r, s)and fis a function, which is different forboth types of nodes since the external nodes fit the edges whereas the internal nodes modelthe inner features of the objects. If the objects to detect are dark and the background is bright,the energy ofan internal nodewill be minimum when it is on a point with a low gray level. Onthe other hand, the energy of an external node will be minimum when it is on a discontinuityand on a light point outside the object. In this situation, function fis defined as:

f[I(v(r, s))] =

h[I(v(r, s))n] for internal nodes

h[Imax I(v(r, s))n+(Gmax G(v(r, s)))]+GD(v(r, s)) for external nodes

(1.3)

where and are weighting terms; Imax and Gmax arethemaximumintensityvaluesofimageIand the gradient imageG, respectively;I(v(r, s))andG(v(r, s))are the intensity values ofthe original image and the gradient image in node position v(r, s); I(v(r, s))nis the meanintensity in an nsquare; andhis an appropriate scaling function. The terms h[I(v(r, s))n]and h[I

maxI(v(r, s))

n] are also called In-Out (IO) energy terms (respectively for internal and


34/185


external nodes), in literature. Since it was proposed for one of the first TAN model extensions[102], the external energy also includes theGradient Distance (GD)term, GD(v(r, s)), this is,the distance from the positionv(r, s)to the nearest edge. This term introduces a continuousrange in the external energy since its value diminishes as the node gets closer to an edge.This way, the gradient distance facilitates the adjustment of the external nodes to the objectboundaries. All subsequent publications employed it in TAN models. Image representationsof the gradient and the distance to gradient for every pixel, along with the original image,are shown in Figure1.6.

(a) Original image (b) Gradient (c) Distance gradient

Figure 1.6: The original image (a) and the representations of the gradient (b) and the distanceto gradient (c) for every pixel.

Finally, in [232] the authors incorporated color similarity into the active net energy func-tion. In particular, they devised a color model for a specific road sign and calculated thecolor similarity to this model for each point in the original image, getting a specific energy

term Eimage. In their formulation, this term is set equal to the active net external energy. Thisterm was introduced for a specific road sign detection application and was not employed insubsequent general purpose proposals.

Regardless of the specific energy terms employed, the adjustment process consists of min-imizing these energy functions. In the original proposals [8, 9], the mesh is placed over thewhole image and then the energy of each node is minimized using a Best Improvement LocalSearch (BILS)2 algorithm [97]. In each step of the algorithm, the energy of each node is com-puted in its current position and in its nearest neighborhood. The position with the lowestenergy value is selected as the new position of the node. The algorithm stops when there isno node in the mesh that can move to a position with lower energy.

The BILS algorithm gets good results in those cases with low presence of noise since ittakes the best local adjustment. Nevertheless, this local adjustment may not be the best globalone in many images because of the presence of noise and/or artifacts. Moreover, the BILSalgorithm does not consider any possible alternatives. This way, if the model reaches a wrongsegmentation, a local minimum from the optimization point of view, it gets stuck in it.

Link cutting procedure The BILS algorithm can perform topological changes, that is, cutsof links between adjacent external nodes after the minimization process. First, external nodesthat are more distant to the object edges are identified using the Tchebycheffs theorem. Thisway, an external node n is badly placed if its gradient distance, GDvext(n), fulfills the inequal-

2The BILS was calledgreedy searchin those previous papers.


35/185


ity:GDvext(n)> GDvext+ 3GDvext

where GDvextand GDvextrepresent the average and the standard deviation of the gradientdistance of all external nodes.

Once the outlier set is identified, the link to remove is selected. It is the node with thehighest gradient distance and its worst neighbor in the outlier set.

After the cutting, some internal nodes become external since they are on the boundariesof the net as figure1.7shows. The increase of external nodes allows a better adjustment toobject boundaries.

An example of one cut and the corresponding transformation of internal nodes into ex-ternal ones is shown in Figure1.8.

Figure 1.7: Link cutting procedure. The figures show the TAN before and after the link cut-ting. After the cut, the neighboring internal nodes become external nodes.

With the link cutting procedure described above, it is possible to solve cases in whichexternal nodes are located outside of the edges of the object, as in Figure 1.8. However, incases like the one shown in Figure1.9,the problem is different, since all the external nodes

are located on the edge of the object, so that the link cutting procedure is unable to properlycut the links.In order to solve this, Ibez et al. [103] developed a complementary cutting procedure.

This new cutting procedure is applied when the former one cannot be applied. It consists ofcutting the link which has the higher number of points on pixels which do not belong to thetarget segmentation object. If this cut is not allowed, due to the fact that the cut could formthreads (see Fig.1.10), it checks the next link with the higher number of points out of theobject whose cut is allowed. The algorithm only considers those links whose length is abovea given threshold.

Automatic Net Division Since the link cutting process breaks the net topology to improvethe adjustment, it should be also desirable to divide the net in order to segment the differentobjects present in the image in case there is more than one object of interest. To this end, a netreconfiguration mechanism must be developed in order to perform multiple object detectionand segmentation. The net division is performed by the link cutting algorithm. However,this algorithm cannot be applied directly to the automatic division. The problems arise incases where a node has only two neighbors. In such a case, no other link can be removed inorder to preserve the TAN topology. Thus, a thread will appear between two subnets asFigure1.10shows.

However, this problem can be overcome if a direction in the cutting process is considered[28]. This way, a cutting priority is associated to each node whose connections are removed.A higher priority is assigned to the nodes in cut direction whereas a lower priority is assigned


36/185


Figure 1.8: Image sequence showing the link cutting procedure based on Tchebycheffs nodeselection. Note how, after the cuts and the corresponding transformations of internal nodes

into external ones, the latter ones, having more freedom due to the cut, end up on the edgesof the object.

to the nodes involved in the cut. Figure 1.10(c) shows the recomputation of the node prioritiesafter several cuts.

The cutting priority weights the gradient distance of each node. Thus, once the set ofbadly placed external nodes is obtained, the link to remove consists of two neighboring nodeswithin this set,n1and n2, that fulfill:

GDvext(n1) Pcut(n1)> GD(n) Pcut(n), n=n1GD

vext(n

2)

Pcut

(n2

)> GDvext

(m)

Pcut

(m),

m=n

2, m

(n

1),


37/185


Figure 1.9: Example of bad segmentation where the link cutting procedure based on Tcheby-cheff external node selection is not appropriate.

Figure 1.10: Threads and cutting priorities. (a) Image segmentation with threads. (b) If linka is removed, no other link can be removed in order to preserve the TAN topology. (c)Recomputation of cutting priorities. When a link is broken in a specific direction, the neigh-boring nodes in this direction increase their priorities.

wherePcut(x)is the cutting priority of nodex,GDvext(x)is the distance from the position of

the external nodexto the nearest edge, and (n1)is the set of neighboring nodes ofn1.

Topological Active Volumes In[13], the authors introducedTopological Active Volumes(TAVs), a 3D extension of the TANs. TAVs thus become a DM focused on segmentation tasksby means of a volumetric distribution of nodes. Parametrically, a TAV is defined as v(r,s,t) =(x(r,s,t), y(r,s,t), z(r,s,t)), where (r,s,t) ([0, 1][0, 1][0, 1]). Similarly to TANs, thestate of the model is governed by an energy function, divided in internal and external energyterms. Being mostly a 3D extension of the TAN, this model has the same behavior, advantagesand disadvantages of its 2D counterpart. Fig.1.11shows a representation of the TAV model,whose basic repeated structure is a cube.


38/185


Figure 1.11: A TAV grid.

1.1.1.4. Other parametric models

Active Contour Models One of the first practical examplesof parametric DMs, calledsnakesor Active Contour Models (ACMs), was first proposed byKass, Witkin and Terzopoulos [114],shortly after than Terzopoulos and his seminal papers. An ACM is a variational method for

detecting object boundaries in images: npointsC0

={p0

1, , p0

n} defining the initial closedcontour are deformed, by minimizing a certain energy function, to lie along the object bound-ary.

LetX(p)be a parameterization of contourCand Ibe a image intensity. Then the energyis

E(C) =

|X(p)|2 +

|X(p)|2

|I(X(p))| (1.4)

The first two terms represent the internal energy while the last term accounts for the externalone. The internal energy is responsible for smoothness, while the external energy is in chargeof attracting the contour toward the object boundary. , andare the free parameters ofthe system and are determined a priori. Smallerreduces the noise but can not capture

the sharp corners while larger can effectively capture boundary but it is sensitive to thenoise. Also, makes the snake more resistant to traction, while makes it more resistantto bending. These two parameters prevent the snake to become non-continuous or to breakduring the iteration process of the optimization problem. The total energy can be written as

E(C) =Einternal(C) + Eexternal(C) (1.5)

It can be shown in thecalculus of variations, the contour should satisfy Eq.1.1.

Active Shape Models ActiveShape Models (ASMs) [49]addmorepriorknowledgetoDMs.These shape models derive aPoint Distribution Model (PDM)from sets of labeled points(landmarks) selected by an expert in a training set of images. In each image, a point, or set


39/185


of points, is placed on the part of the object corresponding to its label. The model considersthe points average positions and the main modes of variation found in the training set, sothe shape models are parameterized in such a way as to allow legal configurations. Whilethis kind of model has problems with unexpected shapes, since an instance of the model canonly take into account deformations which appear in the training set, it is robust with respectto noise and image artifacts, like missing or damaged parts. Therefore, in ASM,PrincipalComponent Analysis (PCA)is usually considered to construct, over a set of landmark pointsextracted from training shapes, a PDM and anAllowable Shape Domain (ASD).

Active Appearance Models Active Appearance Models (AAMs)[43]extend ASMs by con-sidering not only the shape of the model, but also other image properties, like intensity, tex-ture or color. An appearance model can represent both the shape and texture variability seenin a training set, and differ from ASMs in that, instead of searching locally about each model

point, they seek to minimize the difference between a new image and one synthesized bythe appearance model[44]. ASMs only use data around the model points and do not takeadvantage of all the gray-level information available across an object as AAMs do.

1.1.2. Geometric deformable models

Geometric DMs, proposed independently by Caselles et al. [31] and Malladi et al. [141],provide an elegant solution to address the primary limitations of parametric DMs. Thesemodels are based on curve evolution theory [116,117,193] and the LS method [173,195].In particular, curves and surfaces are evolved using only geometric measures, resulting in anevolution that is independent of the parameterization. As in parametric DMs, the evolution iscoupled with the image data to recover object boundaries. Since the evolution is independent

of the parameterization, the evolving curves and surfaces can be represented implicitly as alevel set of a higher-dimensional function [72]. As a result, topology changes can be handledautomatically.

1.1.2.1. Curve evolution theory

The purpose of curve evolution theory [72] is to study the deformation of curves usingonly geometric measures such as the unit normal and curvature as opposed to the quantitiesthat depend on parameters such as the derivatives of an arbitrary parameterized curve. Letus consider a moving curveX(s, t) = [X(s, t), Y(s, t)], wheresis any parameterization andtis the time, and denote its inward unit normal as Nand its curvature ask, respectively. The

evolution of the curve along its normal direction can be characterized by the following partialdifferential equation:

X

t =V(k)N,

whereV(k)is calledspeed function, since it determines the speed of the curve evolution. Theintuition behind this fact is that the tangent deformation affects only the curves parameteri-zation, not its shape and geometry. The most extensively studied curve deformations in curveevolution theory are curvature deformation and constant deformation. Curvature deformation isgiven by the so-calledgeometric heat equation:

X

t

=kN,


40/185


whereis a positive constant. This equation will smooth a curve, eventually shrinking it toa circular point. The use of the curvature deformation has an effect similar to the use of theelastic internal force in parametric DMs.

Constant deformation is given by

X

t =V0N,

whereV0is a coefficient determining the speed and direction of deformation. Constant de-formation plays the same role as the pressure force in parametric DMs. The properties ofcurvature deformation and constant deformation are complementary to each other. Curva-ture deformation removes singularities by smoothing the curve, while constant deformationcan create singularities from an initially smooth curve. The basic idea of the geometric de-formable model is to couple the speed of deformation (using curvature and/or constant de-

formation) with the image data, so that the evolution of the curve stops at object boundaries.The evolution is implemented using the LS method. Thus, most of the research in geometricDMs has been focused on the design of speed functions.

1.1.2.2. Level set method

The LS[72] method is employed for implementing curve evolution. It is used to accountfor automatic topology adaptation, and it also provides the basis for a numerical scheme thatis used by geometric DMs. In the LS method, the curve is represented implicitly as a level setof a N-dimensional scalar function, referred to as the LS function, which is usually definedon the same domain as the image. The LS is defined as the set of points that have the samefunction value. Figure1.12shows an example of embedding a curve as a zero level set. It isworth noting that the LS function is different from the level sets of images. The sole purposeof the LS function is to provide an implicit representation of the evolving curve.

Figure 1.12: An example of embedding a curve as a LS. (a) A single curve. (b) The LS functionwhere the curve is embedded as the zero LS (in black). (c) The height map of the LS functionwith its zero LS depicted in black.

Instead of tracking a curve through time, the LS method evolves a curve by updating theLS function at fixed coordinates through time. A useful property of this approach is thatthe LS function remains a valid function while the embedded curve can change its topology.


41/185


Figure 1.13: From left to right, the zero LS splits into two curves while the LS function stillremains a valid function.

This situation is depicted in Fig 1.13.We now derive the LS embedding of the curve evolutionequation. Given a LS function(x,y ,t)with the contour X(s, t)as its zero level set, we have

[X(s, t), t] = 0.

Differentiating the above equation with respect totand using the chain rule, we obtain

t + X

t = 0,

where

denotes the gradient of.

We assume that is negative inside the zero LS and positive outside. Accordingly, theinward unit normal to the LS curve is given by

N =|| .

Using this fact, we can write

t =V(k)||,

where the curvature at the zero LS is given by

k= || = xx2

y 2xyxy+ yy 2

x(2x+

2y)

3/2 .

1.1.2.3. Level Set Model Approximations

When working with LSs, the definition of the function is essential. One common choiceis the signed distance function d(x), which gives the distance of a point to the surface and thesign. Generallyd >0if the point xis outside and d < 0if it is inside the surface (assumingit is a closed surface). This definition is especially interesting to avoid numerical instabilitiesand inaccuracies during computations. But even with this definition, will not remain asigned distance function all the time and a reinitial

nbthesis final digital

Documents