UNIVERSIDADE TCNICA DE LISBOA INSTITUTO SUPERIOR TCNICO

Crack Detection and Characterization in Flexible Road Pavements using

Digital Image Processing

Henrique Jos Monteiro Oliveira

Supervisor: Doctor Paulo Lus Serras Lobato Correia

Thesis approved in public session to obtain the PhD Degree in Electrical and Computer Engineering

Jury final classification: Pass with Merit

Jury

Chairperson: Chairman of the IST Scientific Board

Members of the Committee: Doctor Mrio Alexandre Teles de Figueiredo Doctor Armando Jos Formoso de Pinho Doctor Jorge dos Santos Salvador Marques Doctor Jaime dos Santos Cardoso Doctor Paulo Lus Serras Lobato Correia Doctor Joo Afonso Ramalho Sopas Pereira Bento

2013

UNIVERSIDADE TCNICA DE LISBOA INSTITUTO SUPERIOR TCNICO

Crack Detection and Characterization in Flexible Road Pavements using

Digital Image Processing

Henrique Jos Monteiro Oliveira

Supervisor: Doctor Paulo Lus Serras Lobato Correia

Thesis approved in public session to obtain the PhD Degree in Electrical and Computer Engineering

Jury final classification: Pass with Merit

Jury

Chairperson: Chairman of the IST Scientific Board

Members of the Committee: Doctor Mrio Alexandre Teles de Figueiredo, Professor Catedrtico do Instituto

Superior Tcnico, da Universidade Tcnica de Lisboa

Doctor Armando Jos Formoso de Pinho, Professor Associado (com Agregao) da Universidade de Aveiro

Doctor Jorge dos Santos Salvador Marques, Professor Associado (com Agregao) do Instituto Superior Tcnico, da Universidade Tcnica de Lisboa

Doctor Jaime dos Santos Cardoso, Professor Auxiliar da Faculdade de Engenharia, da Universidade do Porto

Doctor Paulo Lus Serras Lobato Correia, Professor Auxiliar do Instituto Superior Tcnico, da Universidade Tcnica de Lisboa

Doctor Joo Afonso Ramalho Sopas Pereira Bento Presidente da EFACEC, especialista na rea em que se insere a Tese

2013

i

Abstract This thesis proposes a system for the automatic analysis of road flexible pavement surface

images (CrackIT) structured into three stages, to address the problems of crack detection,

crack type characterization, and the severity level assignment to each detected crack, the

system being able to characterize different crack types in a given image. It follows a set of

guidelines inspired on the Portuguese Distress Catalog and whenever found appropriate

adopting the definitions used by other national distress catalogs, like the USA, French and

Spanish catalogs.

The first stage aims the pre-processing of images, to prepare them for crack detection. Images

are smoothed and six strategies are confronted, to account for their suitability to reduce the

high variance of pixel intensities found in typical road pavement surface images, without

deteriorating crack information on them (essentially associated to connected groups of pixels

darker than their surroundings). Then, they undergo an intensity normalization process to

reduce the problem of non-uniform background illumination, followed by an intensity

saturation process.

The second stage, crack detection, is based on a learning from samples paradigm, where a

subset of the available images is automatically selected and used for supervised and

unsupervised training of the system. Six supervised and six unsupervised classification

strategies are confronted, to account for their suitability for the detection of cracks. The

system classifies non-overlapping image blocks as containing crack pixels or not.

The third stage task deals with crack type characterization, for which another classification

system is constructed, now to characterize the detected cracks connect components. Cracks

are labeled according to the types defined in the Portuguese Distress Catalog, with each

different crack present in a given image receiving the appropriate label. Finally, the

assignment of crack severity levels is based on the estimate for the width of each detected

cracks with.

The proposed automatic system is evaluated over two image databases: the first, composed

by real flexible pavement surface images, acquired by an optical imaging device during a

survey over a Portuguese road (following the visual inspection method), also designated as

optical images; the second, acquired by a laser imaging system developed for high-speed road

pavement surface surveying, during the inspection of a Canadian road, designated as active

images. The evaluation is based on a set of well-known metrics, exploiting the availability of

ground truth data that is manually provided (human labeling) for all the images: optical and

active. Promising results are obtained in both crack detection and characterization tasks,

including the assignment of severity level.

KEYWORDS

Road flexible pavements, crack detection, crack type characterization, crack severity level,

segmentation by thresholding, morphological image processing, pattern recognition,

supervised and unsupervised learning, clustering, one-class classification.

iii

Resumo Esta tese prope um sistema automtico de anlise de imagem da superfcie de pavimentos

rodovirios flexveis (CrackIT) estruturado em trs fases, abordando a deteco de fendilhao, a

caracterizao do tipo de fendilhao, seguida pela atribuio de um nvel de severidade a cada fenda

detectada, sendo o sistema capaz de identificar mais do que um tipo de fendilhao em cada imagem.

So seguidas as linhas orientadoras existentes no Catlogo de Degradaes dos Pavimentos

Rodovirios Flexveis (verso da Ex-Junta Autnoma das Estradas), ou adoptando as definies

existentes em catlogos de degradaes de pavimentos rodovirios flexveis de outros pases quando

considerado apropriado, como por exemplo dos Estados Unidos da Amrica, Frana e Espanha.

A primeira fase do sistema automtico tem como objectivo o pr-processamento de imagens,

preparado-as para a deteco de fendilhao. As imagens so suavizadas e seis estratgias so

discutidas, de modo a averiguar a sua adequao na reduo da varincia da intensidade dos pixeis,

que frequentemente observada em imagens da superfcie dos pavimentos rodovirios flexveis, sem

deteriorar a informao sobre a fendilhao (essencialmente associada a agrupamentos de pixeis mais

escuros que os da sua vizinhana). Depois, as imagens sofrem um processo de normalizao da sua

intensidade, reduzindo os efeitos de uma iluminao no uniforme do seu fundo, seguido por um

processo de saturao da intensidade dos seus pixeis.

A segunda fase, a deteco de fendilhao, baseia-se num paradigma de aprendizagem por exemplos,

em que um sub-conjunto das imagens disponveis para processamento so automaticamente

seleccionadas para treino supervisionado e no supervisionado do sistema. Confrontam-se seis

estratgias de classificao supervisionadas e seis no supervisionadas, de modo a averiguar a sua

adequao relativamente deteco de fendilhao. O sistema classifica blocos de imagem no

sobrepostos contendo fendilhao, ou no.

A terceira fase diz respeito caracterizao do tipo de fendilhao, baseando-se noutro sistema de

classificao, desenvolvido para caracterizar os agrupamentos de componentes conexos que foram

detectados como fendas. As fendas detectadas so classificadas de acordo com os tipos de fendilhao

definidos no Catlogo Nacional de Degradaes dos Pavimentos Rodovirios Flexveis, atribuindo-se

um adequado tipo de fendilhao a cada fenda detectada. A atribuio de nveis de severidade baseia-

se na estimao da largura de cada fenda detectada.

O sistema automtico proposto avaliado recorrendo a duas bases de dados de imagens: a primeira,

composta por imagens da superfcie de pavimentos rodovirios flexveis, adquiridas por um

dispositivo ptico (cmara fotogrfica digital) durante uma inspeco visual de uma estrada nacional

em Portugal, designadas por imagens pticas; a segunda, composta por imagens adquiridas por um

sistema activo (recorrendo a radiao laser) desenvolvido para a inspeco visual rpida da superfcie

de pavimentos rodovirios flexveis, designadas por imagens activas. Todo o processo de avaliao da

metodologia desenvolvida nesta tese baseia-se no clculo de mtricas bem conhecidas, explorando a

existncia de dados criados por um inspector experiente (manualmente produzidos) para todas as

imagens das duas bases de dados: pticas e activas. Resultados promissores so obtidos, quer na

deteco de fendilhao, quer na sua caracterizao, seguida da atribuio de um nvel de severidade.

PALAVRAS-CHAVE

Pavimentos rodovirios flexveis, deteco de fendilhao, caracterizao do tipo de fendilhao, grau

de severidade de fendilhao, segmentao por limiarizao, processamento morfolgico digital de

imagem, reconhecimento de padres, aprendizagem supervisionada e no supervisionada, mtodos de

agregao, classificao baseada numa classe.

v

Acknowledgements This PhD thesis is a corollary of all the work developed along a challenging journey of five

years, in which Professors, Family and Friends have contributed and provided a very strong

support.

Foremost, I will forever be thankful to my supervisor, Professor Paulo Lobato Correia, for its

constant insightful discussions about the research, patience and enthusiasm, always being my

resource for getting my science questions answered and helping me writing this thesis during

the last eight months. In other words, this thesis is a result of a joint work, always under his

close supervision. He has been actively interested in my work since the first day that we met

and has always been available to advise me, even at evening hours and holidays. I am very

grateful for his patience, motivation, enthusiasm, and knowledge in Digital Image Processing,

making him not only a Professor but also a friend.

My special thanks to Professor Fernando Pereira, who always provided me a strong support

to my research work, giving me the opportunity to become a member of the Image Group of

Instituto Superior Tcnico.

My gratefully thanks to Professor Joo Bento and Professor Salvador Marques, both belonging

to the thesis supervising committee (CAT).

I also like to thank to my PhD Professors, Professor Salvador Marques, Professor Mrio

Figueiredo and Professor Ana Fred. With them, my scientific knowledge increased and I felt

privileged to be in their presence and becoming their PhD student.

I also like to thank to my colleagues from the Image Group of Instituto Superior Tcnico, Joo

Ascenso, Catarina Brites, Matteo Naccari and Toms Brando, always providing me a pleasant

working environment, in most cases working until evening hours at my side at the laboratory.

I would like to thank to my colleague and friend Jos Caeiro, for its feedback in relation to

some of the topics developed in this thesis, notably the smoothing of images and in the field of

pattern recognition.

My special thanks to my family, who often found themselves deprived of my presence, due to

my dedication to this research work, but always providing me support and friendship that I

needed.

Finally, I sincerely dedicate this thesis to the memory of my mother Fernanda and my father

Joo, both no longer with us.

vii

Table of Contents PART I ........................................................................................................................... 1

1. INTRODUCTION ................................................................................................................... 5

1.1. CONTEXT AND MOTIVATION .............................................................................................................. 5

1.2. THESIS OBJECTIVES ............................................................................................................................. 8

1.3. SUMMARY OF ORIGINAL CONTRIBUTIONS ....................................................................................... 9

1.4. THESIS OUTLINE ............................................................................................................................... 11

1.5. REFERENCES ...................................................................................................................................... 13

2. PAVEMENT SURFACE MONITORING ..................................................................................17

2.1. INTRODUCTION .................................................................................................................................. 17

2.2. PAVEMENT SURFACE DISTRESS CHARACTERIZATION................................................................ 18

2.2.1. Cracking................................................................................................................................................... 18

2.2.2. Break Up and Loss of Materials .................................................................................................. 23

2.2.3. Movement of Materials.................................................................................................................... 26

2.2.4. Deformations ........................................................................................................................................ 28

2.2.5. Repairs ..................................................................................................................................................... 30

2.3. PAVEMENT SURFACE IMAGE ACQUISITION .................................................................................. 31

2.3.1. Characterization of Real Road Pavement Surface Imagery .......................................... 38

2.3.2. Characterization of Synthetic Test Images ........................................................................... 42

2.4. GROUND TRUTH DATA .................................................................................................................... 49

2.5. REFERENCES ...................................................................................................................................... 53

3. STATE-OF-THE-ART REVIEW AND PROPOSED SYSTEM ARCHITECTURE ............................59

3.1. INTRODUCTION .................................................................................................................................. 59

3.2. STATE-OF-THE-ART REVIEW .......................................................................................................... 60

3.2.1. Pre-processing ..................................................................................................................................... 62

3.2.2. Crack Detection ................................................................................................................................... 63

3.2.3. Crack Class/Type Characterization and Severity Level Assignment ....................... 70

3.2.4. Discussion .............................................................................................................................................. 74

3.3. PROPOSED SYSTEM ARCHITECTURE .............................................................................................. 74

3.4. REFERENCES ...................................................................................................................................... 79

PART II ....................................................................................................................... 85

4. PRE-PROCESSING ..............................................................................................................89

4.1. INTRODUCTION .................................................................................................................................. 89

4.2. IMAGE SMOOTHING ........................................................................................................................... 94

4.2.1. Anisotropic Diffusion ....................................................................................................................... 95

viii

4.2.2. Morphological Smoothing .............................................................................................................. 98

4.2.3. Morphological Erosion followed by Morphological Opening ................................... 102

4.2.4. Two-dimension Stationary Wavelet Decomposition .................................................... 104

4.2.5. Unsupervised Information-theoretic Adaptive Image Filtering (UINTA) .......... 109

4.2.6. Improved UNITA Version (R-UINTA) .................................................................................... 115

4.2.7. Discussion ............................................................................................................................................ 119

4.3. PRELIMINARY LABELING OF CRACK BLOCKS ............................................................................. 126

4.4. IMAGE NORMALIZATION AND SATURATION ............................................................................... 132

4.4.1. Image Normalization ..................................................................................................................... 132

4.4.2. Image Saturation .............................................................................................................................. 134

4.5. REFERENCES .................................................................................................................................... 137

5. CRACK DETECTION: BLOCK BASED ................................................................................ 143

5.1. INTRODUCTION ................................................................................................................................ 143

5.2. AUTOMATIC SELECTION OF TRAINING IMAGES ......................................................................... 145

5.3. FEATURE EXTRACTION .................................................................................................................. 147

5.4. FEATURE NORMALIZATION ........................................................................................................... 151

5.5. BLOCK LABEL ASSIGNMENT: CRACK OR NON-CRACK.......................................................... 157

5.5.1. Supervised Classification Strategies ...................................................................................... 157

5.5.2. Unsupervised Classification Strategies ................................................................................ 176

5.6. DISCUSSION ...................................................................................................................................... 198

5.6.1. System Training ................................................................................................................................ 198

5.6.2. Detection of Crack Image Blocks ........................................................................................... 206

5.7. REFERENCES .................................................................................................................................... 209

6. CRACK DETECTION: PIXEL-BASED REFINEMENT ........................................................... 213

6.1. INTRODUCTION ................................................................................................................................ 213

6.2. SEGMENTATION BY THRESHOLDING ........................................................................................... 215

6.3. RELEVANT CRACK CONNECTED COMPONENTS IDENTIFICATION .......................................... 219

6.4. CONNECTED COMPONENTS LINKAGE .......................................................................................... 222

6.5. IDENTIFICATION OF CRACK BLOCKS ............................................................................................ 230

6.6. MERGING BLOCK-BASED AND PIXEL-BASED DETECTIONS ..................................................... 232

6.7. REFERENCES .................................................................................................................................... 237

7. CRACK TYPE CHARACTERIZATION AND SEVERITY LEVEL ASSIGNMENT ......................... 241

7.1. INTRODUCTION ................................................................................................................................ 241

7.2. CRACK TYPE CHARACTERIZATION ............................................................................................... 242

7.3. SEVERITY LEVEL ASSIGNMENT..................................................................................................... 245

7.4. REFERENCES .................................................................................................................................... 247

ix

PART III ................................................................................................................... 249

8. EXPERIMENTAL RESULTS............................................................................................... 253

8.1. INTRODUCTION ................................................................................................................................253

8.2. PRELIMINARY LABELING OF CRACK BLOCKS .............................................................................254

8.3. OPTICAL IMAGES (IMGSET1) .......................................................................................................255

8.3.1. Crack Detection ................................................................................................................................. 255

8.3.2. Crack Characterization (Type and Severity Level) ......................................................... 257

8.4. ACTIVE IMAGES (IMGSET2)..........................................................................................................265

8.4.1. Crack Detection ................................................................................................................................. 265

8.4.2. Crack characterization (Type and Severity Level) .......................................................... 267

8.5. REFERENCES ....................................................................................................................................276

9. CONCLUSIONS AND FUTURE WORK ................................................................................ 279

9.1. CONCLUSIONS ..................................................................................................................................279

9.2. FUTURE WORK ................................................................................................................................281

9.3. REFERENCES ....................................................................................................................................283

APPENDIX A.CRACK BLOCK SIZE SELECTION ........................................................................... 287

REFERENCES ............................................................................................................................ 295

xi

List of Figures Figure 1.1: Graphical user interface of the automatic road surface imagery analysis software

developed, the CrackIT system. .........................................................................................................9

Figure 2.1: Isolated longitudinal fatigue wheel path crack in asphalt pavement at the wheel path area [left (LCPC, 1998: fiche 07)], and another showing a moderate degradation level in the wheel path area, because of the additional presence of some small cracks paralell to the main one [right (LTPP, 1993: pp. 13)]. Cracks development is parallel to the road axis. The object positioned over the pavement on both images is used to assist the catalog user to have a first approach to the crack dimensions. .................................................................................................................................... 19

Figure 2.2: Fatigue wheel path crack samples: low (left) moderate (middle) and high (right) severity levels (JAE, 1997: pp. 7-8). The red arrows indicate the crack outline. The white object located at the bottom of the images is used to assist the catalog user to have a first approach to the crack dimensions. .................................................................. 19

Figure 2.3: Samples of longitudinal cracks aligned with the road axis and edges: low (left) moderate (middle) and high (right) severity levels (JAE, 1997: pp. 11-12). The red arrow indicates the crack outline. .......................................................................................... 20

Figure 2.4: Samples of transversal cracks: low (left) moderate (middle) and high (right) severity levels (JAE, 1997: pp. 13-14). ......................................................................................... 21

Figure 2.5: Alligator Pattern cracking (FHA, 2003: pp. 5). ......................................................................... 21

Figure 2.6: Samples of alligator pattern cracking: low (right) and moderate (middle). The image on the right shows a high severity level alligator pattern cracking, the arrows indicating the formation of potholes in addition to the presence of degraded edges (JAE, 1997: pp. 9-10). ......................................................................................... 22

Figure 2.7: Miscellaneous cracks due to the simultaneous presence of interconnected transversal and longitudinal cracks............................................................................................... 23

Figure 2.8: Generic layers of a flexible road pavement (Silva, 2005: pp. 10). .................................. 23

Figure 2.9: Raveling (or cat head) distress. The appearance of coarse asphalt mixture materials is visible, due to the loss of some tinny asphalt mixture materials of the top pavement layer (LCPC, 1998: fiche 14). .............................................................................. 24

Figure 2.10: Bare distress examples (JAE, 1997: pp. 22). The loss of the wearing layer and the appearance of the pavement layer underneath are visible: low (right) moderate (middle) and high severity levels (right). ................................................................................... 25

Figure 2.11: Pothole distress (JAE, 1997: pp. 23-24): low (left), moderate (middle) and high (right) severity levels. ........................................................................................................................... 25

Figure 2.12: Climbing of fine aggregates (JAE, 1997: pp. 27-28): low (left), moderate (middle) and high (right) severity levels. ....................................................................................................... 27

Figure 2.13: Water bleeding distress. Cracks and their neighborhoods are darker than the pavement surface shade. Also a high width variation of the wet zones is remarkable (FHA, 2003: pp. 31). ..................................................................................................... 27

Figure 2.14: Example of bleeding distress, characterized by the black shiny glass-like aspect in the wheel path area. It is also possible to observe the loss of coarse texture and the presence of fine texture in that area (JAE, 1997: pp. 26)............................................ 28

xii

Figure 2.15: Rutting distress (JAE, 1997: pp. 19): low (left), moderate (middle) and high (right) severity levels. ........................................................................................................................... 29

Figure 2.16: Example of a depression distress classified with high severity level - the maximum depth is higher than 3 cm (LCPC, 1998: fiche 03). .......................................... 30

Figure 2.17: Deformation distress (JAE, 1997: pp. 17): low (left), moderate (middle) and high (right) severity levels. ........................................................................................................................... 30

Figure 2.18: Repairs of road pavement surface (JAE, 1997: pp. 31): low (left), moderate (middle) and high (right) severity levels. The red arrows (left) indicate the outline of the repaired pavement area. ....................................................................................... 31

Figure 2.19: DESY hardware connected to a portable PC (left) and one of the associate keyboards (right) for inputting distress data [images taken from (Pinto, 2003: pp. 121)]. ............................................................................................................................................................. 32

Figure 2.20: VIZIROAD hardware (left) and the keyboards (right) used for inputting distress data [images taken from (IGM, 2010)]......................................................................................... 33

Figure 2.21: Image of GERPHO system. The photographic camera is positioned at the top of a mechanical support [image taken from (Alves, 2007: pp. 32)]....................................... 33

Figure 2.22: WiseCrax software window with the results of distresses image analysis. The right side of the image shows the results superimposed to the pavement image and the left side shows the parameters used in the image recognition process (WiseCrax, 2013). ................................................................................................................................... 34

Figure 2.23: Representative systems of the ones listed in (VIT, 2002: pp. 5): PAVUE [top-left, image taken from (VIT, 2002: pp. 28)]; ARAN [top-right, image taken from (EPC, 2013)]; HARRIS (bottom-left, image taken from (VIT, 2002: pp. 24); GIE (bottom-right, image taken from (VIT, 2002: pp. 23). ............................................................................ 35

Figure 2.24: Surface Imaging System from Greenwood Engineering [top-row, images taken from (SIS, 2013)], where UCI stands for Unified Crack Index, and the Laser Road Imaging System LRIS from Pavemetrics [bottom-row, images taken from (Laurent, Hbert and Savard, 2013) and (LRIS, 2013)]. Colum-right exhibit images taken by both systems.......................................................................................................... 36

Figure 2.25: Sample image (original and crack detection results depicted in yellow, top-right) acquired by the road cracking vehicle (top-left) developed by CSIRO [images from (CSIRO, 2013)], and Vectras AMAC imaging acquisition system [bottom-row, images from (Nguyen et al., 2010)]. .............................................................................................. 37

Figure 2.26: Optical imaging device used to acquire images during visual survey of surface pavement of a Portuguese road. ...................................................................................................... 38

Figure 2.27: Samples of original optical images, six with cracks and two without cracks. ......... 39

Figure 2.28: Original active sample images, each acquired by one sensor (left images acquired by left sensors and right images acquired by right sensor), three with cracks and one without cracks (bottom-right). ............................................................................................... 40

Figure 2.29: Samples of original active images, formed by the combination of two images (each half image acquired by one sensor). ................................................................................ 41

Figure 2.30: Sample images of SImgSet1 (top-row) and SImgSet2 (bottom-row). ...................................... 42

Figure 2.31: Histograms of all the pixels intensities belonging to all the images composing both sets SImgSet1 (left) and SImgSet2 (right). ................................................................................... 43

Figure 2.32: Crack pixels histograms of sample images: optical (top-left) and active (top-right), where B stands for the most frequent intensity. Bottom-line plots present

xiii

the results of the estimates of probability density functions used, superimposed to the respective histograms. ............................................................................................................ 43

Figure 2.33: A graphical example of how synthetic crack images are created: (A) background image with dimensions 75 75 pixels, cropped from an original pavement surface image without cracks; (B) mask of longitudinal crack with 3 mm of width, also with dimensions 75 75 pixels; (C) crack pixels intensities generated using GEV distribution with a2 = 15.4, b2 = 9.91 and 2 = 0.25, the location, scale and shape parameters respectively; (D) final synthetic crack image.................................... 48

Figure 2.34: Samples of the 150 images that compose each image set with the same crack width: ImgSet3 (top-row, varying the longitudinal crack width from 1 mm to 6 mmfrom left to right, in a total of 6 150 = 900 images) and ImgSet4 (bottom-row, varying the longitudinal crack width from 1 mm to 6 mmfrom left to right, in a total of 6 150 = 900 images). ...................................................................... 49

Figure 2.35: Graphic tool developed using MatLab R2011b, for the creation of crack detection ground truth data. The point resulting from cursor lines interception represents the position where the systems user indicates the presence of an image block exhibiting cracks, while the five blue marks indicate image regions already selected as crack over the optical pavement surface image. .......................................... 50

Figure 2.36: Samples of crack detection ground truth data, showing image blocks over the pavement surface images that are manually classified as crack (left-column) and the resulting binary matrices (right-column), for optical (top-row) and active (bottom-row) images. ........................................................................................................................... 51

Figure 3.1: Overall system architecture of CrackIT system for detection, characterization and severity level assignment of cracks using digital images. .................................................. 75

Figure 3.2: Graphical user interface of the automatic road surface imagery analysis software developed. ................................................................................................................................................... 78

Figure 4.1: Samples of pavement surface images acquired by optical (top-left, with size 1536 2048 pixels) and active imaging systems (top-right, with size 2048 2048 pixels), together with the plot of pixel intensities along the sample line shown in blue color (middle-left and middle-right respectively). The bottom line shows histograms for both images: optical (left) and active (right). .......................................... 90

Figure 4.2: Optical (left) and active (right) binary images computed by thresholding using a threshold value of 30, both exhibiting several and sparse groups of white pixels (cracks), a more pronounced feature for the case of active sample image (right).91

Figure 4.3: Average intensities of image blocks of size 75 75 pixels, along horizontal (middle row) and vertical alignments (bottom-row), highlighting the vignetting effect in sample optical images with cracks (top-right) and without cracks (top-left), showing the decreasing average intensities from the center towards the image edges/corners. ............................................................................................................................ 92

Figure 4.4: Pixel intensity saturation function. ................................................................................................ 93

Figure 4.5: Proposed CrackIT pre-processing architecture....................................................................... 94

Figure 4.6: Anisotropic diffusion filtering results for the original images presented in Figure 4.1. The 1st and 3rd rows present original values, while the 2nd and 4th rows shows anisotropic diffusion filtering results. Left and right columns exhibit plots and histograms concerning sample optical and active images, respectively. ................... 97

Figure 4.7: Morphological smoothing results for the original images presented in Figure 4.1, using a disk-shape se with 3 pixels of radius. The 1st and 3rd rows present original values, while the 2nd and 4th rows show morphological filtering results. Left and

xiv

right columns exhibit plots and histograms concerning the sample optical and active images, respectively.............................................................................................................. 100

Figure 4.8: Morphological smoothing results for the original images presented in Figure 4.1, adopting a series of disk-shape structuring elements of increasing size (from 1 to 3 pixels of radius, with step 1). Left plots correspond to processed pixels of a sample image line using fixed structuring element size of 3 pixels of radius, while the ones at the right column correspond to the alternating sequential filtering.101

Figure 4.9: Morphological erosion followed by morphological opening for the original images presented in Figure 4.1, using disk-shape se with a radius of 3 pixels. The 1st and 3rd rows present original values, while the 2nd and 4th rows show filtering results. Left and right columns exhibit plots and histograms concerning the sample optical and active image, respectively....................................................................................... 103

Figure 4.10: Two dimensional 1-level DWT filter bank (lowpass Lf and highpass Hf), where i: and j: means along row i and along column j, respectively. Frequencies can be manipulated by modifying the Transform coefficients, notably the approximation [A(i,j)] and horizontal, vertical and diagonal details, Dv(i,j), Dh(i,j) and Dd(i,j), respectively. The symbol * stands for the convolution operator and the black circles represent the Addition operator............................................................ 106

Figure 4.11: L-level SWT filter bank (lowpass Lf and highpass Hf), with decomposition step (top) and reconstruction (bottom) steps, where i: and j: means along i and along j, respectively. The current level is denoted by L and Transform coefficients are represented by AL(i,j) (the approximation) and DvL(i,j), DhL(i,j) and DdL(i,j), respectively the horizontal, vertical and diagonal details. The symbol * stands for the convolution operator and black circles between filtering convolutions represent the Addition operator. ................................................................. 107

Figure 4.12: Multilevel two-dimensional stationary wavelet decomposition using an orthonormal symlet, applied to the original images presented in Figure 4.1. 1st and 3rd rows present original values, while 2nd and 4th rows shows filtering results. Left and right columns exhibit plots and histograms concerning the optical and active image, respectively....................................................................................... 108

Figure 4.13: Unsupervised, information-theoretic, adaptive image filtering applied to the original images presented in Figure 4.1. 1st and 3rd rows present original values, while 2nd and 4th rows shows filtering results. Left and right columns exhibit plots and histograms concerning the optical and active image, respectively. ................. 114

Figure 4.14: Proposed unsupervised information-theoretic adaptive image filtering (R-UINTA) applied to the original images presented in Figure 4.1. 1st and 3rd rows present original values, while 2nd and 4th rows shows filtering results. Left and right columns exhibit plots and histograms concerning the optical and active image, respectively. ............................................................................................................................................ 118

Figure 4.15: Evaluation procedure of smoothing techniques. ................................................................. 121

Figure 4.16: Samples of ImgSet3 (top-row) and ImgSet4 (bottom-row) with dimension pixels, exhibiting a longitudinal crack of widths ranging from 1 mm to 6 mm (from left to right, respectively). .................................................................................... 121

Figure 4.17: Smoothed optical images. ............................................................................................................... 122

Figure 4.18: Smoothed active images. ................................................................................................................. 123

Figure 4.19: Average values of metric Fm, computed for the entire ImgSet3. Each plot refers to a crack with a certain width, ranging from 1 mm to 6 mm. ........................................... 124

Figure 4.20: Average values of metric Fm, computed for the entire ImgSet4. Each plot refers to a crack with a certain width, ranging from 1 mm to 6 mm. ........................................... 125

xv

Figure 4.21: Original images divided into non-overlapping blocks with dimension of pixels (top-row), preliminary labeling of blocks with relevant crack pixels (middle-row) and the corresponding ground truth (bottom-row)............................ 129

Figure 4.22: 3D representation of Fm values as a function of k1 and k2 parameters (top-row), and the corresponding isolines (middle-row). Linear fitting for k1 and k2 with the points depicted in brown corresponding to simulations leading to a Fm value higher than 80% and 60% for ImgSet1 and ImgSet2, respectively. Left plots are calculated using ImgSet1, whereas the right ones correspond to ImgSet2. .......... 130

Figure 4.23: Results of average linear fit calculated using both linear fits depicted in blue in bottom-row of Figure 4.22. The average linear fit is depicted at magenta color superimposed with all k1 and k2 values that lead to Fm values greater or equal than 80% and 60% respectively for ImgSet1 and ImgSet2 (depicted in blue in top plot), as well as with the isoline plots (bottom-row). ....................................................... 131

Figure 4.24: .Sample line of image blocks depicted in blue superimposed to the smoothed sample active image using R-UINTA strategy (left), and the corresponding average intensities within each image block (right). Due to the presence of white pixels belonging to white lane lines over the road pavement surface, the three blocks from the left present the highest average of pixels intensities...................... 131

Figure 4.25: .Sample line of image blocks depicted in blue, superimposed to the R-UINTA smoothed active image (left), and the corresponding average intensities within each image block (right). Blocks with white pixels corresponding to white lane lines over the road pavement surface have their average intensities replaced by the image average intensity, in this case the gray level of 130..................................... 132

Figure 4.26: Region average intensity values along the row selected in the sample optical image (top), before (bottom-left) and after (bottom-right) normalization, with taking value 130. .............................................................................................................. 134

Figure 4.27: Normalized optical (top-left) and active (bottom-left) sample R-UINTA smoothed images containing a longitudinal crack before (left column) and after (right column) applying the intensity saturation algorithm. Saturated images become slightly darker when compared to the normalized ones. ................................................ 135

Figure 4.28: Image blocks average intensity values along the row selected in the top of Figure 4.26 after normalization (top-left) and saturation (top-right) and standard deviation of region intensities for the normalized images before (bottom-left) and after applying the saturation algorithm (bottom-right). The amplitude is the difference between .............................................................................................................................. 136

Figure 5.1: Architecture for block-based automatic crack detection stage. ................................... 144

Figure 5.2: Sample optical (top-left) and active (bottom-left) smoothed images, the ground truth (middle-column) and the automatic preliminary labeling of crack blocks (right-column)........................................................................................................................................ 146

Figure 5.3: Skeleton composed of 27 blocks (right) of a connected component object composed of 43 8-connected blocks (left). ............................................................................. 146

Figure 5.4: Feature spaces corresponding to the sample optical (left) and active (right) images of the left-column of Figure 5.2, computed from the original intensity values, with unlabeled data (top-row) and labels manually assigned by a human expert (bottom-row)........................................................................................................................... 148

Figure 5.5: Plots illustrating the effect on the feature space of pre-processing operations: smoothing with R-UINTA (top), block intensity normalization (middle) and pixel intensity saturation (bottom), for the optical and active sample images of the left-column of Figure 5.2, using ground truth labels .................................................................. 150

xvi

Figure 5.6: Two sample optical images (top-row) after smoothing, block normalization and intensity saturation and the respective feature spaces, before feature normalization (middle-left), where blue and red points belong to the image on the left, while cyan and magenta points belong to the image on the right. The decision boundaries are computed using only image blocks preliminarily selected as non-crack (middle-right) and target and outlier labels (resulting from one-class classification) are assigned (bottom-left). Finally, translated points (bottom-right), are represented using ground truth labels. ............................................................. 153

Figure 5.7: Feature space rotation results. The plot on the left exhibits the linear fittings computed using only points labeled as target, as well as each angle between the fittings and the feature space horizontal axis, notably 34.1 and 38.4, for the images on upper-left and upper-right corners of Figure 5.6, respectively. The global reference angle is 36.25. The plot on the right presents the fully normalized feature spaces, exhibiting the ground truth labels assigned to points. ......................................................................................................................................................... 154

Figure 5.8: Two active images (top-row) after smoothing and intensity saturation and the respective feature spaces before feature normalization (middle-left), where blue and red points belong to the image on the left, while cyan and magenta points belong to the image on the right. The decision boundaries are computed using only image blocks preliminarily selected as non-crack (middle-right) and target and outlier labels (resulting from one-class classification) are assigned (bottom-left). Finally, points are translated to align with the global centroid (bottom-right), being represented with ground truth labels. .......................................................... 155

Figure 5.9: Feature space rotation results. The plot on the left exhibits the linear fittings computed using only points labeled as target, as well as each angle between the fittings and the feature space horizontal axis, notably 19.3 and 17.3 for the images on upper-left and upper-right corners of Figure 5.8, respectively. The global reference angle is 18.3. The plot on the right presents the fully normalized feature spaces, exhibiting ground truth labels assigned to points. ............................ 156

Figure 5.10: Diagram of the classification strategies considered. ......................................................... 157

Figure 5.11: Sample linear decision boundaries computed for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, with ground truth labels (top-row). The computed linear decision boundaries are shown in the middle row. Pavement surface images blocks are signaled according to the labels assigned during the crack detection task (bottom-row), where green represents true positives (cracks), and red represents false negatives or missed cracks. ................................... 161

Figure 5.12: Sample quadratic decision boundaries computed for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, with ground truth labels (top-row). The computed quadratic decision boundaries are shown in the middle row. Pavement surface image blocks are signaled according to the labels assigned during the crack detection task (bottom-row), where green represents true positives (cracks), yellow represents false positives and red represents false negatives or missed cracks. ........................................................................................................................................ 164

Figure 5.13: Sample decision boundaries calculated using statistically independent features, for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, with ground truth labels (top-row). The computed quadratic decision boundaries are shown in the middle row. Blocks of pavement surface images are signaled according to the labels assigned during the crack detection task (bottom-row), where green represents true positives (cracks), yellow indicate false positives and red represents false negatives or missed cracks. ....................................................... 166

xvii

Figure 5.14: Sample PZw decision boundary computed for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, with ground truth labels (top-row). The computed decision boundaries are shown in the middle row. Blocks of pavement surface images are signaled according to the labels assigned during the crack detection task (bottom-row), where green represents true positives (cracks), yellow indicate false positives and red represents false negatives or missed cracks. ......................................................................................................................................................... 169

Figure 5.15: Sample k-NN decision boundary computed for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, using ground truth labels (top-row), with 1-nearest neighbor estimated for both cases. Labeled points according to the computed decision region are shown in the middle-row. Pavement surface images overlaid with the crack detection results are shown at the bottom row, where green marks represent true positives (cracks) and red ones represent blocks not detected by the CrackIT system, i.e. a false negative or a missed crack. ........................................................................................................................................................... 171

Figure 5.16: Sample FLC decision boundary computed for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, with ground truth labels (top-row). The computed decision boundaries are shown in the middle row. Blocks of pavement surface images are signaled according to the labels assigned during the crack detection task (bottom-row), where green represents true positives (cracks) and red represents false negatives or missed cracks.................................................................. 174

Figure 5.17: Quadratic decision boundary (qdb), depicted in green, computed with labels resulting from hierarchical clustering, for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, superimposed with HC clustering labels (top-row) and those manually assigned, i.e. the ground truth (bottom-row)............................. 178

Figure 5.18: Relabeling of points according to the computed quadratic decision computed (top-row), following the results shown on Figure 5.17. Detected blocks are signaled on the images (bottom-row), where green marks represent true positive (crack) detections, yellow marks indicate false positives and the red ones represent false negatives or a missed crack. .......................................................................... 179

Figure 5.19: Quadratic decision boundary (qdb) calculated with labels resulting from hierarchical clustering, for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, superimposed with k-means clustering labels (top-row) and those manually assigned, i.e. the ground truth (bottom-row). .................................................. 181

Figure 5.20: Relabeling of points according to the computed quadratic decision computed, following the results shown on Figure 5.19. Blocks of pavement surface images are signaled according to the labels assigned at crack detection task (bottom-row), where green marks represent true positive (crack) detections, yellow marks indicate false positives and the red ones represent false negatives or a missed crack. ........................................................................................................................................... 182

Figure 5.21: Decision boundary computed with labels resulting from GMM estimated using clustering by EM algorithm, for the images shown on upper-left corner of Figure 5.6 and Figure 5.8 (top-row), superimposed with labels manually assigned, i.e. the ground truth (bottom-row). Points within non-crack region of feature space are labeled as non-crack, while those located within crack region are labeled as crack. Pavement surface image blocks are signaled according to the labels assigned at crack detection task (bottom-row): green marks (cracks detected); yellow marks (false positives); red marks representing blocks not detected by CrackIT system (false negative or a missed crack). .......................................................... 185

Figure 5.22: One-class Gaussian decision boundary, for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, superimposed with unlabeled data (top-row) and

xviii

those manually assigned, i.e. the ground truth (bottom-row). Points within non-crack region of feature space (inside the green ellipse) are labeled as non-crack, while those located within crack region (outside the green ellipse) are labeled as crack, as depicted in Figure 5.23................................................................................................ 189

Figure 5.23: One-class Gaussian decision boundary following the results shown on Figure 5.22. Pavement surface image blocks are signaled according to the labels assigned at crack detection task (top-row), where green marks represent true positive (crack) detections, yellow marks indicate false positives and the red ones represent blocks not detected by the CrackIT, i.e. a false negative or a missed crack (bottom-row). ........................................................................................................................... 190

Figure 5.24: MCDG decision boundary calculated for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, superimposed with unlabeled data (top-row) and those manually assigned, i.e. the ground truth (bottom-row). Points within non-crack region of feature space (inside the green ellipse) are labeled as non-crack, while those located within crack region (outside the green ellipse) are labeled as crack, as depicted in Figure 5.25................................................................................................ 193

Figure 5.25: MCDG decision boundary following the results shown on Figure 5.24. Pavement surface image blocks are signaled according to the labels assigned at crack detection task (bottom-row), where green marks represent true positive (crack) detections, yellow marks indicate false positives and the red ones represent blocks not detected by the CrackIT, i.e. a false negative or a missed crack (bottom-row). ........................................................................................................................................ 194

Figure 5.26: One-class Parzen decision boundary for the images shown on upper-left corner of Figure 5.6 and Figure 5.8, superimposed with unlabeled data (top-row) and those manually assigned, i.e. the ground truth (bottom-row). Points within non-crack region of feature space (inside the green curve) are labeled as non-crack, while those located within crack region (outside the green curve) are labeled as crack, as depicted in Figure 5.27................................................................................................ 196

Figure 5.27: One-class Parzen decision boundary computed, following the results shown on Figure 5.26. Pavement surface image blocks are signaled according to the labels assigned at crack detection task (bottom-row), where green marks represent true positive (crack) detections, yellow marks indicate false positives and the red ones represent blocks not detected by the CrackIT, i.e. a false negative or a missed crack (bottom-row). ........................................................................................................... 197

Figure 5.28: The five optical images that compose the TrSImgSet1 set (left-column). Preliminary labeling results are shown in the right-column. .................................................................. 199

Figure 5.29: TrSImgSet1 feature space representing original image pixels intensities without being pre-processing (top-row), the non-normalized feature space representing pre-processed images (middle-row) and normalized features (bottom-row). Results correspond to the images shown in Figure 5.28, exhibiting a fitting angle of about 46.3 (dashed line in bottom-row plots), with unlabeled points (left-column) and with labels assigned manually by a human expert (right-column). ........................................................................................................................................................................ 200

Figure 5.30: Decision boundaries calculated using the classification techniques developed in Section 5.5, in respect with the TrSImgSet1: supervised parametric (top-left) and non-parametric (top-right) as well as unsupervised classifiers based on clustering (bottom-left) and one-class methods (bottom-right). ............................... 201

Figure 5.31: The full pre-processed six active images that compose the TrSImgSet2 set (1st and 3rd columns). Preliminary labeling results are shown in 2nd and 4th columns, with images exhibiting non-processed areas due to the presence of white lane lines painted over road pavement surface. ........................................................................................ 203

xix

Figure 5.32: TrSImgSet2 feature space representing original image pixels intensities (without being pre-processed top-row), the non-normalized feature space representing pre-processed images (middle-row) and normalized features (bottom-row), all of them extracted from images shown on Figure 5.31, exhibiting a fitting angle of about 20.9 (dashed line in bottom-row plots), with unlabeled points (left-column) and with labels assigned manually by a human expert (right-column).204

Figure 5.33: Decision boundaries calculated using the classification techniques developed in Section 5.5, in respect with the TrSImgSet2: supervised parametric (top-left) and non-parametric (top-right) as well as unsupervised classifiers based on clustering (bottom-left) and one-class methods (bottom-right). ................................ 205

Figure 5.34: Crack detection results using Parzen windowing decision boundary: optical TstSImgSet1 (top-row) and active TstSImgSet2 (middle-row) testing images, with respective feature spaces depicted in bottom-row. ........................................................... 207

Figure 6.1: Proposed system architecture for crack detection: pixel-based refinement. ....... 214

Figure 6.2: Diagram detailing the cco linkage procedure. ....................................................................... 215

Figure 6.3: Histograms (right column) of normalized images (left column) and the fitted one-dimensional Gaussian function in each one, for optical (top-row) and active (bottom-row) sample images......................................................................................................... 216

Figure 6.4: Normalized images (top-left); selection of pixels with intensities lower than iimg (middle-row); crack pixels candidates using the proposed method after thresholding the normalized images with (bottom-left) and (bottom-right). ....................................................................................................................................... 218

Figure 6.5: Histograms of eccentricities, widths and lengths values computed for all the cco found in all the optical (left column) and active (right column) pavement surface images without cracks, either from ImgSet1 or ImgSet2. ............................................... 220

Figure 6.6: Removal of non-relevant connected components results: original segmented images (top-row) as shown at bottom-row of Figure 6.4; processed binary images exhibiting only those relevant cco that simultaneously fulfill the three conditions purposed as above-mentioned (bottom-row). ...................................................................... 221

Figure 6.7: Normalized image (top-left); segmentation by thresholding with (top-right); selection of relevant cco (middle-left); expanded cco (middle-right); new pixels added to relevant cco depicted in red (bottom-left); first set of non-relevant cco (depicted in green) linked to relevant ones (depicted in green) bottom-right. ........................................................................................................................................... 223

Figure 6.8: Detailed representation of how values are calculated. The figure shows two cco labeled cco1 and cco2, with orientations of their ellipse major axis being +39.5 and -73.6, respectively identified by the two vectors depicted in cyan. Yellow dots represent the respective centroids, i.e. the center of the ellipses that have the same second-moments as the cco (depicted in red). The vector depicted in magenta that connects both centroids defines a new orientation, notably -40.4. The M and P axis define the referencing system for the calculation of centroids coordinates and orientations, which is centered at the lower-left corner of the binary image. ............................................................................................................................. 225

Figure 6.9: Two cases representing extreme values of : 0 (left) and 2 (right), for parallel and collinear ccos, respectively. ................................................................................. 225

Figure 6.10: Connected components linkage, resulting on the identification of one global crack only (right), following the results shown in Figure 6.7 bottom-right. ...................... 227

Figure 6.11: Connected components linkage results. Each row (from left to right) represents: original image; segmentation result; ccos identified as relevant; the final crack

xx

regions detection after the proposed linkage where each color identifies one global crack. ............................................................................................................................................ 227

Figure 6.12: Connected components linkage results (bottom-row) for an optical image (top-left) randomly chosen from TstSImgSet1, showing the selected non-relevant cco (depicted in yellow and red) that are connected to the relevant ones (depicted in green) and the identification of global cracks, each one depicted in a specific color (bottom-right). ...................................................................................................................................... 228

Figure 6.13: Connected components linkage results (2nd) for an active image (1st) randomly chosen from TstSImgSet2, showing selected non-relevant cco (depicted in yellow and red) that are connected to the relevant ones (depicted in green 3rd) and the identification of global cracks, each one depicted in a specific color (4th). .......... 229

Figure 6.14: Binary image showing the presence of one global crack depicted in cyan, divided into non-overlapping blocks (left). Those blocks considered as crack are labeled 1 (right). .................................................................................................................................................. 230

Figure 6.15: Crack blocks highlighted over the binary (top) and the normalized gray optical images (bottom), with one isolated block being removed (signaled by the red circle). ......................................................................................................................................................... 231

Figure 6.16: Crack blocks highlighted over the binary (left) and the normalized gray optical images (right), with two isolated blocks being removed (each one signaled by the red circle). ................................................................................................................................................ 232

Figure 6.17: Crack detection results: block-based (top); pixel-based (middle); merged crack blocks (bottom). The crosses depicted in cyan represent a very relevant crack, while those depicted in purple indicating less relevant crack image blocks. ..... 233

Figure 6.18: Crack detection results: block-based (top); pixel-based (middle); merged crack blocks (bottom). .................................................................................................................................... 234

Figure 7.1: Architecture for crack type characterization and severity level assignment stage. ........................................................................................................................................................................ 242

Figure 7.2: Longitudinal (top-row) and transversal (bottom-row) crack types taken from (JAE, 1997: pp. 12 and 14), with severity levels 1 (left-column), 2 (middle-column) and 3 (right-column). All the remaining cracking distress types are classified as miscellaneous.............................................................................................................. 242

Figure 7.3: Two-dimensional feature space used to characterize cracks into types, for the four sample cracks shown in Figure 7.4................................................................................... 243

Figure 7.4: Sample crack types (from left to right): longitudinal (1st); transversal (2nd); almost transversal (3rd) and almost longitudinal (4th), using the same color scheme as in Figure 7.3. .................................................................................................................... 243

Figure 7.5: Sample optical (top-row) and active (bottom-row) crack type characterization results, for high relevant cracks only, those signaled by the X mark depicted in cyan. ............................................................................................................................................................ 244

Figure 7.6: Two feature spaces with different decision threshold settings: (left) and (right). ........................................................ 245

Figure 7.7: Samples of Crack width measurement: identified cracks (1st and 3rd) and the computed width in mm units with cracks skeleton depicted in red (2nd and 4th). ........................................................................................................................................................................ 246

Figure 8.1: Overall system architecture of the CrackIT system for detection, characterization and severity level assignment of cracks using digital images. ...................................... 254

xxi

Figure 8.2: Original image (top-left); ground truth (top-right); the merging of block-based and pixel-based crack detection results (middle-left); crack type characterization (middle-right); global crack detection results: pixel-based (bottom-left) and block-based (bottom-right). Only one crack of miscellaneous type is detected according to block-based results, exhibiting a reference width of 4.7 mm, thus an SL2 severity level was assigned. .................................................................................................... 258

Figure 8.3: Original image (top-left); ground truth (top-right); the merging of block-based and pixel-based crack detection results (middle-left); crack type characterization (middle-right); global cracks pixel-based (bottom-left) and block-based (bottom-right). Four cracks of longitudinal type are detected according to block-based results, exhibiting reference widths of 3.6 mm, 2.7 mm, 3.8 mm and 2.5 mm from L1 to L4 respectively, thus an SL2 severity level was assigned to all of them. ...... 259

Figure 8.4: Original image (top-left); ground truth (top-right); the merging of block-based and pixel-based crack detection results (middle-left); crack type characterization (middle-right); global cracks pixel-based (bottom-left) and block-based (bottom-right). Three cracks of longitudinal type are detected according to block-based results, exhibiting reference widths of 3.2 mm, 2.2 mm and 3.9 mm from L1 to L3 respectively, thus an SL2 severity level was assigned to all of them. ......................... 260

Figure 8.5: Original image (top-left); ground truth (top-right); the merging of block-based and pixel-based crack detection results (middle-left); crack type characterization (middle-right); global cracks pixel-based (bottom-left) and block-based (bottom-right). Four cracks, three of longitudinal type and one of miscellaneous type are detected according to block-based results, exhibiting reference widths of 4.2 mm, 3.2 mm, 3.9 mm and 2.5 mm from T1 to T3 and M1, respectively, thus an SL2 severity level was assigned to all of them................................................................................ 261

Figure 8.6: Original image (top-left); ground truth (top-right); the merging of block-based and pixel-based crack detection results (middle-left); crack type characterization (middle-right); global cracks pixel-based (bottom-left) and block-based (bottom-right). Two cracks, one of transversal type and one of miscellaneous type, are detected according to block-based results, exhibiting reference widths of 3.7 mm and 3.6 mm, respectively, thus an SL2 severity level was assigned to all of them. ........................................................................................................................................................................ 262

Figure 8.7: Original image (top-left); ground truth (top-right); the merging of block-based and pixel-based crack detection results (middle-left); crack type characterization (middle-right); global cracks pixel-based (bottom-left) and block-based (bottom-right). Two miscellaneous cracks are detected according to block-based results, exhibiting reference widths of 4.2 mm and 3.3 mm, M1 and M2 respectively, thus an SL2 severity level was assigned to both cracks. .............................................................. 263

Figure 8.8: Original image (top-left) considered by the human expert not containing cracks ground truth (top-right); the merging of block-based and pixel-based crack detection results (middle-left); crack type characterization (middle-right); global cracks pixel-based (bottom-left) and block-based (bottom-right). One crack erroneously characterized as miscellaneous according to block-based results, exhibiting a reference width of 4.0 mm, thus an SL2 severity level was assigned to the crack. The results also shows that the CrackIT system is robust to oil spots, since the one shown in the image (near its center) was not considered as a pavement surface distress. .............................................................................................................. 264

Figure 8.9: Original image (1st from left); ground truth (2nd from left); the merging of block-based and pixel-based crack detection results (3rd from left); one global crack pixel-based (4th from left)............................................................................................................... 268

xxii

Figure 8.10: Original image (1st from left); ground truth (2nd from left); the merging of block-based and pixel-based crack detection results (3rd from left); two global cracks pixel-based (4th from left). ............................................................................................................. 269

Figure 8.11: Original image (1st from left); ground truth (2nd from left); the merging of block-based and pixel-based crack detection results (3rd from left); one global crack pixel-based (4th from left). ............................................................................................................. 270

Figure 8.12: Original image (1st from left); ground truth (2nd from left); the merging of block-based and pixel-based crack detection results (3rd from left); two global cracks pixel-based (4th from left). ............................................................................................................. 271

Figure 8.13: Original image (1st from left); ground truth (2nd from left); the merging of block-based and pixel-based crack detection results (3rd from left); one global crack pixel-based (4th from left). ............................................................................................................. 272

Figure 8.14: Original image (1st from left); ground truth (2nd from left); the merging of block-based and pixel-based crack detection results (3rd from left); three global cracks pixel-based (4th from left). ............................................................................................................. 273

Figure 8.15: Crack type characterization results for cracks detected shown in Figure 8.9, Figure 8.10 and Figure 8.11: M1 (top) presenting a width of 5.3 mm (SL2); L1 and M1 (middle) presenting widths of 5.7 mm and 4.7 m (both SL2), respectively; L1 (bottom) presenting a width of 6.2 mm (Sl2). ....................................................................... 274

Figure 8.16: Crack type characterization results for cracks detected shown in Figure 8.12, Figure 8.13 and Figure 8.14: M1 and L1 (top) presenting widths of 4.1 mm and 4.6 mm (both with SL2); L1 (middle) presenting a width of 4.9 mm (SL2); M1, M2 e L1 (bottom) presenting widths of 4.2 mm, 4.6 mm and 5.0 mm (all with Sl2). 275

Figure A.1: Standard deviation of the mean (left) and std (right) features, for eight images without cracks belonging to ImgSet1, as a function of the block size. Each line represents one image. ........................................................................................................................ 288

Figure A.2: Exponential fitting for the results of Figure A.1. Points represent the average calculated at each label position using Figure A.1 curves, for the mean (depicted in red) and the std (depicted in blue) features................................................................... 288

Figure A.3: First derivative of the mean (left) and std (right) fitting curves of Figure A.2.288

Figure A.4: Average distance between two-dimensional feature points (left) and its first derivative (right), for eight non-crack images, with block sizes from 15 to 205, with step 10............................................................................................................................................. 289

Figure A.5: Crack detection evaluation results obtained for different block sizes of optical images considered. .............................................................................................................................. 289

Figure A.6: Plot of 'mean' and 'std' features for the optical image crack (top-left) and no crack (top-right) blocks with original intensities. The block's size varies from to pixels, maintaining the center of each block types at their same pixels coordinates. ............................................................................................................................................. 290

Figure A.7: Crack detection evaluation results obtained for different block sizes of active images considered. .............................................................................................................................. 291

Figure A.8: Plot of 'mean' and 'std' features for the active image crack (top-left) and no crack (top-right) blocks with original intensities. The block's size varies from to pixels, maintaining the center of each block types at their same pixels coordinates. ............................................................................................................................................. 291

xxiii

List of Tables Table 2-1: Listing of the absolute residuals statistics found in the pdfs estimates plotted in

Figure 2.32 (bottom-line), Values on the left-side of slash character correspond to the optical images while the ones on its right-side correspond to the active images. ........................................................................................................................................................... 47

Table 2-2: Crack type characterization and severity ground truth data resulting from the manual classification of crack image blocks for the two sample images show in Figure 2.36. ................................................................................................................................................. 52

Table 4-1: Metrics computed for the optical image of Figure 4.1, along the marked sample line of pixels and for a sample image block with dimensions of . 119

Table 4-2: Metrics computed for the active image of Figure 4.1, along the marked sample line of pixels and for a sample image block with dimensions of . 120

Table 5-1: values computed for the original and pre-processed optical and active sample images on left-column of Figure 5.2. ............................................................ 149

Table 5-2: values calculated for the optical feature spaces in Figure 5.6 and Figure 5.7, taking into account the ground truth labels. .................................................. 154

Table 5-3: values calculated for the active feature spaces in Figure 5.8 and Figure 5.9, taking into account the ground truth labels. .................................................. 156

Table 5-4: Possible loss function values regarding the detection of image blocks containing crack pixels or not. ............................................................................................................................... 159

Table 5-5: Quantitative evaluation metrics regarding the block label assignment for the bottom-row images of Figure 5.11. ............................................................................................. 162

Table 5-6: Confusion matrix of the number of predicted and true blocks labels, for the optical image. .......................................................................................................................................... 162

Table 5-7: Confusion matrix of the number of predicted and true blocks labels, for the active image (without the blocks containing white lane markings). ....................................... 162

Table 5-8: Quantitative evaluation metric regarding block label assignment for the bottom-row images of Figure 5.12................................................................................................................ 163

Table 5-9: Confusion matrix for the number of predicted and true blocks labels, for the optical image. .......................................................................................................................................... 163

Table 5-10: Confusion matrix for the number of predicted and true blocks labels, for the active image (without the blocks containing white lane markings). ......................... 163

Table 5-11: Quantitative evaluation metrics regarding the block label assignment for the bottom-row images of Figure 5.13. ............................................................................................. 165

Table 5-12: Confusion matrix for the number of predicted and true block labels, for the optical image. .......................................................................................................................................... 167

Table 5-13: Confusion matrix for the number of predicted and true block labels, for the active image (without the blocks containing white lane markings). ....................................... 167

Table 5-14: Quantitative evaluation metric regarding the block label assignment for the images in bottom-row of Figure 5.14. ........................................................................................ 170

Table 5-15: Confusion matrix referring the number of predicted and true image blocks labels, for the optical case. .............................................................................................................................. 170

xxiv

Table 5-16: Confusion matrix referring the number of predicted and true image blocks labels, for the active case, without those image blocks containing the white lane markings. .................................................................................................................................................. 170

Table 5-17: Quantitative evaluation metric regarding the block label assignment for the images in bottom-row of Figure 5.15. ....................................................................................... 172

Table 5-18: Confusion matrix referring the number of predicted and true image blocks labels, for the optical case............................................................................................................................... 172

Table 5-19: Confusion matrix referring the number of predicted and true image blocks labels, for the active case, without those image blocks containing the white lane markings. .................................................................................................................................................. 172

Table 5-20: Quantitative evaluation metric regarding the block label assignment for the images in bottom-row of Figure 5.16. ....................................................................................... 175

Table 5-21: Confusion matrix referring the number of predicted and true image blocks labels, for the optical case............................................................................................................................... 175

Table 5-22: Confusion matrix referring the number of predicted and true image blocks labels, for the active case, without those image blocks containing the white lane markings. .................................................................................................................................................. 175

Table 5-23: Quantitative evaluation metric regarding the block label assignment for the images in bottom-row of Figure 5.18. ....................................................................................... 178

Table 5-24: Confusion matrix referring the number of predicted and true image blocks labels, for the optical case............................................................................................................................... 179

Table 5-25: Confusion matrix referring the number of predicted and true image blocks labels, for the active case, without those image blocks containing the white lane markings. .................................................................................................................................................. 180

Table 5-26: Quantitative evaluation metric regarding the block label assignment for the images in bottom-row of Figure 5.20. ....................................................................................... 181

Table 5-27: Confusion matrix referring the number of predicted and true image blocks labels, for the optical case............................................................................................................................... 182

Table 5-28: Confusion matrix referring the number of predicted and true image blocks labels, for the active case, without those image blocks containing the white lane markings. .................................................................................................................................................. 183

Table 5-29: Quantitative evaluation metric regarding the block label assignment for the images in bottom-row of Figure 5.21. ....................................................................................... 186

Table 5-30: Confusion matrix referring the number of predicted and true image blocks labels, for the optical case............................................................................................................................... 186

Table 5-31: Confusion matrix referring the number of predicted and true image blocks labels, for the active case, without those image blocks containing the white lane markings. .................................................................................................................................................. 186

Table 5-32: Quantitative evaluation metric regarding the block label assignment for the images in bottom-row of Figure 5.23. ....................................................................................... 189

Table 5-33: Confusion matrix referring the number of predicted and t