igarss.pdf

Improved BINARY PARTITION TREE construc8on for hyperspectral

images: Applica8on to object detec8on

Introduc)on BPT construc)on Pruning strategy Conclusions

S.Valero , P. Salembier , J. Chanussot, C.M.Cuadras 1 ,2 2 3

GIPSA-‐Lab, Signal & Image Dept, Grenoble-‐INP, Grenoble, France Signal Theory and Comunic. Dept, Technical University of Catalonia, Spain

University of Barcelona, Spain

2 1

3

1

1

Outline

Introduc)on v  Hyperspectral imagery v  BPT

Binary Par))on Tree Construc)on v  Region Model v  Merging Criterion

2

Pruning Strategy v  Road detec)on v  Building detec)on

3

Conclusions 4


1

Outline



2


3

Conclusions 4

Introduc8on BPT construc8on Pruning strategy Conclusions

Hyperspectral imagery BPT

Hyperspectral Imagery

X

y

1

2

N

Pxy

Radiance

Wavelength

N

1Pxy

v  Each pixel is a discrete spectrum containing the reflected solar radiance of the spa)al region that it represents



v  Different analysis techniques have been proposed in the literature. Most of them process the pixels individually, as an array of spectral data without any spa)al structure

Pixels are studied as isolated

discrete spectra

Hyperspectral Imagery v  The ini)al pixel-‐based representa)on is a very low level and unstructured representa)on v  Instead of working with a pixel-‐based representa)on, Binary Par88on

Trees are an example of a poweful structured region-‐based image representa)on.

v  The construc)on of Binary Par88on Trees for hyperspectral images has

been recently proposed

P. Salembier and L. Garrido. Binary par**on tree as an efficient representa*on for image processing, segmenta*on, and informa*on retrieval. IEEE Transac)ons on Image Processing, vol.9(4), pp.561-‐576, 2000. S.Valero, P.Salembier and J.Chanussot. New hyperspectral data representa)on using Binary Par))on Tree. IEEE Proceedings of IGARSS, 2010

1



1

2

2

BPT v  BPTs can be interpreted as a structured image representa)on containing

a set of hierarchical regions stored in a tree structure v  Each node represen)ng a region in the image, BPTs allow us to extract

many par))ons at different levels of resolu)on






How can BPT be extended to the case of hyperspectral data ?

?



Aim: BPT for HS image analysis

CONSTRUCTION PRUNING Hyperspectral

image Classifica)on

Object detec)on Segmenta)on

v  We propose to construct a BPT in order to represent an HS image with a new region-‐based hierarchical representa)on

BPT representa)on

Hyperspectral image





image Classifica)on


v  In this paper: Pruning strategy aiming at object detec)on is proposed

BPT Search Hyperspectral

image The pruning look for regions characterized by some features



1

Outline



2

3

Conclusions 4


Region Model Merging Criterion



F

E

A B

C D

G G

E

F

C D

E

C D

B A B

v  The BPT is a hierarchical tree structure represen)ng an image v  The tree leaves correspond to individual pixels, whereas the root represents the en)re image v  The remaining nodes represent regions formed by the merging of two children v  The tree construc)on is performed by an itera)ve region merging algorithm



A B

C D

C D

B A B




E

A B

C D C D

E

C D

B A B




F

E

A B

C D

E

F

C D

E

C D

B A B




F

E

A B

C D

G G

E

F

C D

E

C D

B A B




The crea)on of BPT implies two important no)ons

v  Region model MRi It specifies how an hyperspectral region is represented and how to model the union of two regions.

v Merging criterion O(Ri,Rj) The similarity between neighboring regions determining the merging order

Rij

Rj Ri

O(Ri,Rj) O(Ri,Rj)





image Classifica)on


v  How to represent hyperspectral image regions?

v  Which similarity measure defines a good merging order?



Region Model

Nbins

Hλi

B A

We propose a non-‐parametric sta)s)cal region model

consis)ng in a set of N probability density func)ons

where each Pi represents the

probabili)y that the spectra data set has a specific radiance value

in the wavelength λi

Radiance

Wavelength λ λi

A

B

Pixel 1 Pixel 2 Pixel 3

F. Calderero and F. Marques.Region-‐Merging techniques using informa*on theory sta*s*cal measures. IEEE Transac)ons on Image Processing, vol.19, pp.1567-‐1586, 2010.

2

2





image Classifica)on


v  How to represent hyperspectral image regions?

v  Which similarity measure defines a good merging order?



Merging Criterion

Mul8dimensional Scaling


Step 2

Principal Coordinates

Associa8on Measure


Step 1



Merging Criterion



Step 1





Analyze the inter-‐waveband similarity rela)onships for each data via metric scaling to obtain

the principal coordinates

Merging Criterion: Step 2



Step 2



Associa8on Measure

PC1

PC2 PC1= PC2 β +e



Are Regression Coefficients equal to 0 ???

An associa)on measure is defined by considering that PC1 and PC2 form a mul)variate regression

model

Rosis Hyperspectral data

RGB Composi)on

Data Set : Rosis Pavia Center 103 bands

BPT is constructed by using the proposed merging order

103 bands


Nregions=22 Nregions=32 Nregions=42

Rosis Hyperspectral data Ground truth manually created

Hierarchical BPT level RHSEG soiware


NRegions=22 NRegions=22

A symmetric distance for object evalua)on: It is defined as the he minimum number of pixels whose

labels should be changed to achieve perfect matching, normalized by the total number of pixels

of the image minus one

Symmetric distance to ground truth equal to 0.48

Symmetric distance to ground truth equal to 0.227

[Tilton, 2005]

1

Outline



2


3

Conclusions 4


Road detec)on Building detec)on



image Classifica)on


v  Pruning strategy aiming at object detec)on is proposed

BPT Search Hyperspectral

image The pruning look for regions characterized by some features



Object Detec8on Strategy

v  Hyperspectral object detec)on has been mainly developed in the context of pixel-‐wise spectral classifica)on.

v  Objets are not only characterized by their spectral signature.

v  Spa)al features such as shape, area, orienta)on can also contribute significantly to the detec)on.

v  Roads appear as elongated structures having fairly homogeneous radiance values usually corresponding to asphalt.



Besides the informa8on provided by asphalt spectrum, a road

contains important spa8al features

Object Detec8on Strategy

λ2tRi

λ2tRj

are the eigenvalues of BRi and BRj which are pro-

portional to the variances of the corresponding principal axes. Here

Ns is the minimum dimension for which

�Nst=1 λ2

tR�Nt=1 λ2

tR≈ 1 and (u�

tvp)2

is just the correlation coefficient between the t-th and p-th coordi-

nates. Thus the numerator in ck is a weighted average of the rela-

tionships between principal axes. Clearly 0 ≤ c1 ≤, ... ≤ cs ≤, ... ≤ cNs = 1. The dimension s is then chosen such that Cs is

high, for instance Cs = 0.9. At this point, having two regions de-

fined by their standard coordinates URi and URj whose dimensions

are Nxs, the Wilk’s criterion W for testing B=0 in a multivariate

regresion model is given by:

W (Ri, Rj) = det(I − U�Rj

URiU�Ri

URj ) =s�

i=1

(1− r2i ) (4)

where det means the determinant and ri corresponds to the

canonical correlation of each axis. Using Eq. 4, an association

measure can be defined as:

AW (Ri, Rj) = 1−W (Ri, Rj) = 1−s�

i=1

(1− r2i ) (5)

satisfying 0 ≤ AW (Ri, Rj) ≤ 1 and AW (Ri, Rj) = 1 if Ri

is equal to Rj . Thus, this leads us to the definition of the proposed

merging criterion:

OMDS(Ri, Rj) = argminRi,Rj

1 − AW (Ri, Rj) (6)

3. OBJECT DETECTION WITH BPT

Hyperspectral object detection has been mainly developed in the

context of pixel-wise spectral classification. The main problem of

this approach is that objects are not only characterized by their spec-

tral signature. Indeed, spatial features such as shape, area, orienta-

tion, etc., can also contribute significantly to the detection.

In order to integrate the spatial and spectral information, BPT is pro-

posed as a search space for constructing a robust object identification

scheme. The strategy is to analyze the BPT using a set of descrip-

tors computed for each node. The work presented here proposes the

analysis of three different descriptors for each node:

D = {Dshape, Dspectral, Darea} (7)

The proposed shape, spectral and area descriptors are related

to the specficic object of interest. Studying D from the leaves to

the root, the approach consists in removing all nodes that signifi-

cantly differ from the characterization proposed by a referenceDref .

Hence, given this reference, the idea consists in considering that the

searched object instances are defined by the closest nodes to the root

node that have descriptors close to the Dref . model. In order to

illustrate the generality of the approach, we describe two detection

examples: roads and building in urban scenes.

3.1. Detection of roads

Roads appear as elongated structures having fairly homogeneous ra-

diance values usually corresponding to asphalt. Given their charac-

teristic shape, Dshape is the elongation of the region. In order to

compute it, we first define the smallest rectangular bounding box

allowed to have an arbitrary orientation that includes the region cor-

responding to the BPT node under study. Dshape is defined as the ra-

tio between the width and the height of the bounding box. Dspectral

is the correlation coefficient between the mean spectrum of the BPT

node and a reference spectrum of the road material. Darea is defined

as the area of the region which should be higher than the minimum

area required to consider that a region can be a road. This minimum

parameter should be set according to the spatial resolution of the im-

age. Computing D for all BPT nodes, the idea is to select the father

nodes closer to the root which have a low elongation, a high correla-

tion between asphalt material and an area higher than a threshold.

3.2. Detection of buildings

Following the same strategy, we consider that the rectangularity of a

region is an important characteristic for buildings. Then, a rectangu-

larity descriptor is proposed asDshape. It is computed by the ratio of

the area of the BPT node and the area of the smallest bounding box

including the region (as in the previous application, the bounding

box can have arbitrary orientation). Dshape=1 for perfectly rectan-

gular regions. Dspectral also corresponds to the correlation coeffi-

cient measured between the mean spectrum of the BPT node and a

reference spectrum of the building material. Darea is the required

area to consider that a region can be a building.. The procedure of

the detection is the same as the roads.

4. EXPERIMENTAL RESULTS

We first provide an evaluation of the BPT construction proposed in

Section 2. The experiments have been performed using a portion

of a publicy available HYDICE hyperspectral. After removing wa-

ter absorption and noisy bands, the data contain 167 spectral bands.

Fig. 4(a) shows a RGB combination of three of them. The BPT is

computed by the procedure described in Section 2. The number of

bins to represent the histograms depends of the image range (here

Nbins = 256). The first component dimension found by the se-

quence ck is s = 3. To evaluate the quality of the BPT construction,

we extract a segmentation result involving a given number NR of re-

gions by undoing the last NR− 1 mergings over the initial partition.

In this section, we compare the partition extracted from the BPT

with the classical RHSEG [11]. In the case of RHSEG, the similar-

ity criterion used is SAM with spectral clustering weight 0.1[12]. To

evaluate the resulting partitions, the symmetric distance dsym [13]

is used as a partition quality evaluation. Having a partition P and a

ground truth GT , the symmetric distance corresponds to the mini-

mum number of pixels whose labels should be changed in partition

P to achieve a perfect matching with GT , normalized by the total

number of pixels in the image. The manually created GT is shown

in Fig. 4(b).

Fig. 4(c)(d) show the segmentation results obtained with BPT and

RHSEG, respectively. In both cases, the resulting partitions involve

63 regions. Comparing both results, the quantitive dsym and the

visual evaluation corroborates that the partition extracted from the

BPT are much closer to the ground truth than the one computed with

RHSEG. This experiment has been done with several other images

acquired by different hyperspectral sensors, but, due to space limi-

tations, we can only present one data set. However, the conclusions

were the same on the remaining dataset.

A second set of experiments are conducted now for object de-

tection and recognition. We compare the classical pixel-wise clas-

sification against the strategy proposed in section 3 for building and

road detection. The pixel-wise classification consists in detecting

λ2tRi

λ2tRj

are the eigenvalues of BRi and BRj which are pro-

portional to the variances of the corresponding principal axes. Here

Ns is the minimum dimension for which

�Nst=1 λ2

tR�Nt=1 λ2

tR≈ 1 and (u�

tvp)2

is just the correlation coefficient between the t-th and p-th coordi-

nates. Thus the numerator in ck is a weighted average of the rela-

tionships between principal axes. Clearly 0 ≤ c1 ≤, ... ≤ cs ≤, ... ≤ cNs = 1. The dimension s is then chosen such that Cs is

high, for instance Cs = 0.9. At this point, having two regions de-

fined by their standard coordinates URi and URj whose dimensions

are Nxs, the Wilk’s criterion W for testing B=0 in a multivariate

regresion model is given by:

W (Ri, Rj) = det(I − U�Rj

URiU�Ri

URj ) =s�

i=1

(1− r2i ) (4)

where det means the determinant and ri corresponds to the

canonical correlation of each axis. Using Eq. 4, an association

measure can be defined as:

AW (Ri, Rj) = 1−W (Ri, Rj) = 1−s�

i=1

(1− r2i ) (5)

satisfying 0 ≤ AW (Ri, Rj) ≤ 1 and AW (Ri, Rj) = 1 if Ri

is equal to Rj . Thus, this leads us to the definition of the proposed

merging criterion:

OMDS(Ri, Rj) = argminRi,Rj

1 − AW (Ri, Rj) (6)

3. OBJECT DETECTION WITH BPT

Hyperspectral object detection has been mainly developed in the

context of pixel-wise spectral classification. The main problem of

this approach is that objects are not only characterized by their spec-

tral signature. Indeed, spatial features such as shape, area, orienta-

tion, etc., can also contribute significantly to the detection.

In order to integrate the spatial and spectral information, BPT is pro-

posed as a search space for constructing a robust object identification

scheme. The strategy is to analyze the BPT using a set of descrip-

tors computed for each node. The work presented here proposes the

analysis of three different descriptors for each node:

D = {Dshape, Dspectral, Darea} (7)

The proposed shape, spectral and area descriptors are related

to the specficic object of interest. Studying D from the leaves to

the root, the approach consists in removing all nodes that signifi-

cantly differ from the characterization proposed by a referenceDref .

Hence, given this reference, the idea consists in considering that the

searched object instances are defined by the closest nodes to the root

node that have descriptors close to the Dref . model. In order to

illustrate the generality of the approach, we describe two detection

examples: roads and building in urban scenes.

3.1. Detection of roads

Roads appear as elongated structures having fairly homogeneous ra-

diance values usually corresponding to asphalt. Given their charac-

teristic shape, Dshape is the elongation of the region. In order to

compute it, we first define the smallest rectangular bounding box

allowed to have an arbitrary orientation that includes the region cor-

responding to the BPT node under study. Dshape is defined as the ra-

tio between the width and the height of the bounding box. Dspectral

is the correlation coefficient between the mean spectrum of the BPT

node and a reference spectrum of the road material. Darea is defined

as the area of the region which should be higher than the minimum

area required to consider that a region can be a road. This minimum

parameter should be set according to the spatial resolution of the im-

age. Computing D for all BPT nodes, the idea is to select the father

nodes closer to the root which have a low elongation, a high correla-

tion between asphalt material and an area higher than a threshold.

3.2. Detection of buildings

Following the same strategy, we consider that the rectangularity of a

region is an important characteristic for buildings. Then, a rectangu-

larity descriptor is proposed asDshape. It is computed by the ratio of

the area of the BPT node and the area of the smallest bounding box

including the region (as in the previous application, the bounding

box can have arbitrary orientation). Dshape=1 for perfectly rectan-

gular regions. Dspectral also corresponds to the correlation coeffi-

cient measured between the mean spectrum of the BPT node and a

reference spectrum of the building material. Darea is the required

area to consider that a region can be a building.. The procedure of

the detection is the same as the roads.

4. EXPERIMENTAL RESULTS

We first provide an evaluation of the BPT construction proposed in

Section 2. The experiments have been performed using a portion

of a publicy available HYDICE hyperspectral. After removing wa-

ter absorption and noisy bands, the data contain 167 spectral bands.

Fig. 4(a) shows a RGB combination of three of them. The BPT is

computed by the procedure described in Section 2. The number of

bins to represent the histograms depends of the image range (here

Nbins = 256). The first component dimension found by the se-

quence ck is s = 3. To evaluate the quality of the BPT construction,

we extract a segmentation result involving a given number NR of re-

gions by undoing the last NR− 1 mergings over the initial partition.

In this section, we compare the partition extracted from the BPT

with the classical RHSEG [11]. In the case of RHSEG, the similar-

ity criterion used is SAM with spectral clustering weight 0.1[12]. To

evaluate the resulting partitions, the symmetric distance dsym [13]

is used as a partition quality evaluation. Having a partition P and a

ground truth GT , the symmetric distance corresponds to the mini-

mum number of pixels whose labels should be changed in partition

P to achieve a perfect matching with GT , normalized by the total

number of pixels in the image. The manually created GT is shown

in Fig. 4(b).

Fig. 4(c)(d) show the segmentation results obtained with BPT and

RHSEG, respectively. In both cases, the resulting partitions involve

63 regions. Comparing both results, the quantitive dsym and the

visual evaluation corroborates that the partition extracted from the

BPT are much closer to the ground truth than the one computed with

RHSEG. This experiment has been done with several other images

acquired by different hyperspectral sensors, but, due to space limi-

tations, we can only present one data set. However, the conclusions

were the same on the remaining dataset.

A second set of experiments are conducted now for object de-

tection and recognition. We compare the classical pixel-wise clas-

sification against the strategy proposed in section 3 for building and

road detection. The pixel-wise classification consists in detecting

Star8ng from the leaves

We analyse each BPT to look for the nodes having specific spa8al and spectral descriptors



v  Firstly, the spa)al and the spectral descriptors are computed for all BPT nodes

v  Secondly, following a bojom-‐up

strategy, the pruning select nodes closer to the root which have a low elonga)on, a high correla)on between asphalt and an area higher than a threshold

Object Detec8on: Example of Roads

v  Roads appear as elongated structures

Road detec8on Building detec)on


The ra)o between the height and the width of the minimum

bounding box containing the region

Oriented Bounding Box containing the region

width height


Road detec8on Building detec)on


v  Roads have fairly homogeneous radiance values usually corresponding to asphalt

Radiance

Wavelength λ

Pixel 1 Pixel 2 Pixel 3

Mean Spectrum

Correla)on Coefficient

Asphalt reference spectrum

Correla)on coefficient between the mean spectra of the region and the asphalt reference spectrum


27 regions 37 regions 57 regions

Par88ons contained in BPT

Hydice Hyperspectral image

Pixel-‐wise Asphalt detec)on (Spectra whose Correla)on

with asphalt is higher than 0.9)

BPT pruning strategy oriented to object

detec)on



Object Detec8on: Example of Building

27 regions 37 regions 57 regions

Par88ons contained in BPT

Hydice Hyperspectral image

Pixel-‐wise Building detec)on (Spectra whose Correla)on with reference is higher than 0.9)

BPT pruning strategy oriented to object

detec)on



Conclusions


v  A new region merging algorithm has been presented here to construct BPTs as a

hyperspectral region-‐based and hierarchical representa)on.

v  Being a generic representa)on, many tree processing techniques can be formulated as pruning strategies for many applica)ons. Here, as an example of BPT processing, a pruning strategy has been proposed for object detec)on. v  A new similarity measure for merging hyperspectral regions have been proposed

taking into account spa)al and spectral correla)ons. It introduces a local dimension reduc)on which is different from the classical point of view where the reduc)on is applied globally on the en)re image.

v  The processing example has shown the advantage of using region-‐based representa)ons.

igarss.pdf

Technology

hyperspectral imagery

image g v

enre image v

tree structure v

unstructured representaon

spaal structure v

enre image f f v

children c d v