a novel method of medical image fusion based on ... · a novel method of medical image fusion based...

A Novel Method of Medical Image Fusion Based on Bidimensional

Empirical Mode Decomposition

1Kongsen Feng, 2Xiaoli Zhang,3Xiongfei Li*

1Kongsen Feng College of Computer Science and Technology Jilin

University,[email protected] 2Xiaoli ZhangCollege of Computer Science and Technology Jilin

University,[email protected] 3 Xiongfei Li Laboratory of Symbolic Computation Jilin University, [email protected]

Abstract In this paper, a novel method of medical image fusion based on bidimensional empirical mode

decomposition (BEMD) is proposed with the aim to improve the quality of fused medical images.

BEMD is first used in this paper to decompose medical images into bidimensional intrinsic mode

functions (BIMFs) and residue. In this paper m-BIMFs is proposed for the first time, which has larger

scale and better feature than BIMFs. Finally, m-PCNN(Pulse Coupled Neural Network), an improved

PCNN mode, is used to fuse corresponding m-BIMFs. The experimental results demonstrate that our

method is successfully applied to medical image fusion and better than traditional wavelet fusion ,

pyramid fusion and unimproved m-PCNN fusion.

Key Words: Medical Image Fusion, Bidimensional Empirical Mode

Decomposition,m-BIMF,m-PCNN

1 Introduction

In recent years, medical image technology has been widely applied to medical diagnosis; however,

single imaging technology is incapable of providing comprehensive diagnosis basis. For example, CT

is of very high spatial resolution and can show better bone information but unclear soft tissue

information; in contrast, MRI is of lower resolution and can show better soft tissue information but

fuzzy bone information. Therefore, the fusion of two or more medical images with complementary

information can provide more comprehensive information for medical diagnosis[1].

The currently popular methods of medical image fusion can be classified into 2 categories: One

image fusion algorithm is based on spatial domain and the other is based on spatial domain and

frequency domain[2]. The former mainly includes method of weighed and selected coefficient,

pyramid [3], PCA[4], etc. The latter mainly includes Fourier transform, wavelet decomposition [5]

[6]and transform, and others. Wavelet transform is an image fusion method researched widely at

present. Wavelet decomposition and transform are characterized by good time-frequency and scale and

praised as a mathematical microscope of signal processing. But wavelet decomposition depends on the

selection of wavelet function and the predefined filter. For different wavelet functions or different

filters, the same application has a great influence on the quality of fused images.

N. E. Huang et al [7] put forward Empirical Mode Decomposition (EMD) in 1998, which is capable

of good analysis of nonlinear and nonstationary signals. EMD is an analytical tool driven by

non-parameter data, capable of decomposing signals into a series of intrinsic mode functions(IMF) and

a residue. The EMD has been applied to a series of problem which require high resolution but are

separable in the time-frequency domain . such as in faults diagnosis of bearings, ocean engineering[8]

and biomedical signal processing. The idea of EMD can be well applied to bidimensional image

analysis, i.e. bidimensional empirical mode decomposition(BEMD). Nunes et al [9] have succeeded in

applying BEMD to the extraction of texture features of images. In this paper, BEMD is used in medical

image fusion. First, BEMD is used to decompose medical images, and then BIMFs resulted from the

decomposition of BEMD are analyzed to obtain new m-BIMFs. Pulse Coupled Neural Network

A Novel Method of Medical Image Fusion Based on Bidimensional Empirical Mode Decomposition Kongsen Feng, Xiaoli Zhang, Xiongfei Li

Journal of Convergence Information Technology(JCIT) Volume6, Number12, December 2011 doi:10.4156/jcit.vol6.issue12.11

84

(PCNN) achieves good effects in image fusion, so the improved m-PCNN mode is used in this paper to

fuse m-BIMFs, and then the inverse transform is performed on the fused BIMF to reconstruct image.

The experimental results demonstrate that the fused image achieved by BEMD outperforms those

achieved by common fusion algorithms.

2 Related Work

2.1Bidmensional Empirical Mode Decomposition

BEMD can decompose the frequency of original image from high to low into a finite quantity of

BIMFs and a tendency image, of which BIMFs satisfy two constraint conditions: First, being symmetry

for local value of the original bidimensional signal and with its mean value equivalent to 0; second, its

maximum values being positive and its minimum values being negative. BEMD is an adaptive

analytical method driven by parameter-free data, and is a sifting process. The steps of BEMD are as

follows:

1) To initialize, the tendency image res = I, H = I, I is the image to be decomposed;

2) To find out all maximum and minimum value points of H;

3) To interpolate the maximum and minimum value points to obtain H’s upper and lower

envelope surfaces ;

4) To calculate mean value Em of upper and lower envelope surfaces;

5) H(m,n) = H(m,n) - Em(m,n), to judge whether H satisfies the sifting conditions, if not, to

return to Step 2;

6) BIMF(m,n) = H(m,n), res(m,n) = res(m,n) – H(m,n);

7) To repeat Step 2 - 6 until the quantity of extreme points is less than 2;

After N layers of decomposition, the final decomposition process of image can be expressed as

𝐼(𝑚, 𝑛) = ∑ 𝐵𝐼𝑀𝐹𝑗𝑁𝑗=1 (𝑚, 𝑛) + 𝑟𝑒𝑠(𝑚, 𝑛) (1)

of which, BIMFj is BIMF of j , and res is the final residue.

The key problems of BEMD are extreme point detection, interpolation method and stopping

criteria. The correct selection of extreme points exerts direct influence on the construction of envelope

surfaces. The method of morphological operators(morphological reconstruction and watershed) is used

to detect extreme points in this paper. At present there are mainly two methods of envelope surface

interpolation based on triangulation and Radial Basis function (RBF) respectively. RBF that has good

interpolation effect on discrete points distributing sparsely and irregularly is used to interpolate in this

paper. The method of standard deviation is adopted as stopping criteria. The standard deviation of two

consecutive Hi(j−1) and Hij (i means the BIMF of i) in the sifting process is calculated, and the

sifting process stops when the standard deviation is less than the defined threshold value which is

usually defined as 0.2 ~ 0.3. The standard deviation is defined as follows:

𝑆𝐷 = ∑ ∑[𝐻𝑖(𝑗−1)(𝑚,𝑛)−𝐻𝑖𝑗(𝑚,𝑛)]

2

(𝐻𝑖(𝑗−1)(𝑚,𝑛))2

𝑁𝑛=1

𝑀𝑚=1 (2)

2.2 m-PCNN Mode

PCNN, proposed by Eckhorn[10], is a simplified neutral network model of sync-pulse principle

based on brain neurons of cat. Broussard[11] achieved image fusion with the help of PCNN to improve

the recognition rate of target and demonstrated the relationship between the ignition frequency of

PCNN neurons and grey scale of image, confirming the feasibility of the application of PCNN to image

fusion.

The application of PCNN mode to image fusion produces good effects [12], but these PCNN

image fusions have such a common characteristic as the incapability of one PCNN of completing the

fusion process of the whole image. In general, at least two PCNNs have to be used to fuse multi-source

images, which lead to the fact that it is hard for PCNN mode to meet the requirement of time overhead


85

effected by a system with high real-time demand.

In view of such a shortcoming of PCNN mode, Wang [13] proposed an improved PCNN mode, i.e.

m-PCNN, which has solved the problem of singleness of external stimulus of PCNN mode, and only

one PCNN model is needed to fuse multiple images and thus efficiency is improved greatly. Here dual

channel PCNN mode with m=2 is introduced.

Dual channel PCNN mode also consists of three parts: Dendritic tree, information fusion and

pulse generation, which is expressed as follows:

𝐻𝑖𝑗1 [𝑛] = 𝑀(𝑌[𝑛 − 1]) + 𝑆𝑖𝑗

1 (3)

𝐻𝑖𝑗2 [𝑛] = 𝑊(𝑌[𝑛 − 1]) + 𝑆𝑖𝑗

2 (4)

𝑈𝑖𝑗[𝑛] = (1 + 𝛽1𝐻𝑖𝑗

1 )(1 + 𝛽2𝐻𝑖𝑗2 ) + 𝜎 (5)

𝑌𝑖𝑗[𝑛] = {1 𝑈𝑖𝑗[𝑛] > 𝐸𝑖𝑗[𝑛]

0 𝑈𝑖𝑗[𝑛] ≤ 𝐸𝑖𝑗[𝑛] (6)

𝐸𝑖𝑗[𝑛] = 𝑒𝑥𝑝(−𝛼𝐸) 𝐸𝑖𝑗[𝑛 − 1] + 𝑉𝐸𝑌𝑖𝑗[𝑛] (7)

Here i, j represents the location of an image; Hij1 and Hij

2 are the inputs of corresponding

channels; Sij1 and Sij

2 are the corresponding external stimuli, which corresponds to the pixel values of

their images; σ is the level factor to adjust the average level of internal activity; 1 and 2 are the

weighting coefficients of H1 and H2. The bigger the weighting value is, the more important the input

of corresponding channel is. Formula (3) and (4) reflect the influence of neighboring neurons on the

current neuron, of which M and W are feed functions. VE and αE magnification factor and time

constant of dynamic threshold

The application of m-PCNN is able to process multi-source images. Multi-source images can be

processed in parallel manner with multiple external stimuli working in the same neuron, which saves a

lot of time spent in processing multi-source images and reduces the complexity of compute.

Figure 1. The neuromime of m-PCNN

3 Image Fusion Algorithm

3.1 m-BIMF

BEMD is an adaptive decomposition process. Namely, the quantity of BIMFs can be decomposed

into is determined by the data of the image itself, and different images have different quantity of

BIMFs after decomposition. Only images with same decomposed layers can be fused. The general

processing method is to set the quantity of BIMFs in advance, which thus alters the stopping criteria of

the quantity of extreme points of the original BEMD decomposition. However, the quantity of BIMFs

of an image is hard to predefine. If the given quantity of layers is fewer, the decomposed BIMFs may

not manifest the texture of the image; if the given quantity is more, unexpected termination may be

effected to the program, so the stopping criteria of decomposition should not been altered.

In the mode of multi-scale image fusion, the quantity of decomposed layers should be moderate.


86

If there are too many decomposed layers, false information may be introduced in the sub-images, and

the quality of fused image will be influenced in the end; if there are too few decomposed layers, it will

degenerate into the mode of single-scale image fusion, and loss the advantages of multi-scale mode.

Similarly, if there are too many decomposed layers by BEMD, there is energy loss in the image

obtained through inverse transform after fusion and the fusion result is not good. The quantity of

BIMFs should be reduced in fusion when the stopping criteria of decomposition are not altered.

When envelope surfaces are constructed and interpolation is done to discrete extreme points, there

may be maximum value points nearby minimum value points, and vice versa due to their disperse

distribution, and upper and lower envelope surfaces may intersect, which extends the boundary of the

image texture. The extension of boundary alters the distribution of latter extreme points, and thus

exerts influences on the construction of latter BIMFs. The interpolation method does not bring error but

makes the original texture information not properly reflected in each BIMF. For example, BIMF1 in

Figure 2 (b) does not properly reflect the characteristics of CT. Single BIMF sometimes does not

properly reflect the original texture features of an image, so more reasonable method is needed to

process BIMF to make it capable of better reflecting the characteristics of the image.

The purpose of using different decomposition methods in image fusion to obtain different

coefficient matrixes or components is to fuse images in different and corresponding frequency range so

as to achieve optimum fusion effect. After different images are decomposed by BIMF, the

corresponding BIMF may not be the component of its corresponding frequency. Because different

images have different distribution of extreme points and different values of extreme points, which

makes corresponding BIMFs not correspond actually. If BIMFs are directly fused, the final fusion

result will be influenced by frequencies lacking in correspondence.

To solve the above shortcomings of BEMD in image fusion, we put forward the definition of m-BIMF,

i.e. a new BIMF formed by adding several BIMFs together. Since BEMD satisfies the decomposition

process from high frequency to low frequency, m-BIMF is also energy distribution from high

frequency to low frequency. For instance, m- 1 with maximum frequency is formed by adding the

first and second BIMFs together; m- 2 with the second maximum frequency is formed by adding

the third and fourth BIMFs together, ···, so (1) can be expressed as:

𝐼(𝑚, 𝑛) = ∑ 𝑚 − 𝐵𝐼𝑀𝐹𝑗(𝑚, 𝑛) + 𝑟𝑒𝑠(𝑚, 𝑛) 𝑗=1 (8)

L is the quantity of m-BIMFs. m-BIMF meets the definition of BIMF; the mean value of m-BIMF

which is obtained by adding together several BIMFs with mean value equivalent to 0 is still zero;

besides its maximum value points are positive and minimum value points are negative. In this way an

image can be expressed by several m-BIMFs. Such a m-BIMF has larger scale than the original BIMF,

which reduces the quantity of BIMFs and has better texture features than those of the original BIMF.

Another important significance of m-BIMF is that corresponding components can better correspond to

each other through reasonable definition of m-BIMF. And thus better image fusion effect can be

achieved.

(a) BIMF1 (b) BIMF2 (c) BIMF3 (d)m-BIMF

Figure 2. BIMFs and m-BIMF

3.2 Fusion Algorithm

The quantity of BIMFs of two original images may be different after they are decomposed by

classic decomposition of BEMD. Here, it is required that the mode set the quantity of common

m-BIMFs of the original images to n. According to the characteristics of images and BIMFs obtained


87

after decomposition, m-BIMF, representing the process from high frequency to low frequency, can be

obtained through calculation. Residue (res) represents the average tendency of the image. Here the

method of entropy weighted and selected coefficient is used to fuse low frequency.

The detailed algorithm is as follows:

(1) To carry out registration on the two medical images I1, I2 to be fused;

(2) To perform BEMD on the two medical images to be fused; Image I1 thus obtains its

corresponding k BIMFs and res; Image I2 obtains its 1 BIMF and res;

(3) To set the quantity of common m-BIMFs to N according to the characteristics of the two

images to be fused, and calculate their corresponding m- j1 and m- j

2;

(4) To set the parameters of m-PCNN;

(5) To use m-PCNN to fuse N m-BIMFs that I1 and I2 corresponds to so as to obtain N new

m-BIMFs;

(6) To fuse res. The following formulae are used to calculate:

𝑟𝑒𝑠 = 1𝑟𝑒𝑠1 + 2𝑟𝑒𝑠2 (9)

1 = 𝑛 1

𝑛 1 𝑛 2 (10)

2 = 𝑛 2

𝑛 1 𝑛 2 (11)

of which, 1, 2 indicates entropies of 1 and 2.

(7) To carry out inverse transform on N fused m-BIMFs and res to obtain the final fuse image.

4 Experimental Results

In this chapter, BEMD is run to decompose multi-modal medical images firstly, and then

m-PCNN is used to fuse images. The selection of parameters exerts great influence on m-PCNN, so

parameter setup is first to be discussed here. The setup of m-BIMF is also an important factor

influencing the final fusion result. Here the setup of each parameter will be given, and then fused

images obtained by our method and other different fusion methods will be compared. Finally, quality

evaluation of different methods is given.

4.1 Parameter Setup

In the experiments, dual channel PCNN is used to fuse BIMFs. The parameter setup is as follows:

1

=0.45, 2

=0.55, K= [0.1091, 0.1409, 0.1091; 0.1409, 0, 0.1409; 0.1091, 0.1409, 0.1091],

M=W=Y[n-1] K, of which, is convolution operation, level factor =1, time constant E=0.012,

threshold value VE=4500. There should not be too many m-BIMFs. Here, N=2, m- 1 equivalent

to the sum of the first three BIMFs, m- 2 equivalent to the sum of the rest BIMFs.

4.2 Objective Evaluation Criteria

Among the following various evalution indexes, M, N represents the size of images respectively;

IS means the original image; F refers to the fused image.

1) Information Entropy (IE). IE refers to the quantity of average information contained in an

image. The bigger it is, the more information it contains and the better the fusion effect is.

𝐼𝐸 = −∑ 𝑝( ) 𝑜 2𝑝( ) 𝑖= (12)

L is the gray scale of the image and p(i) indicates the probability of gray value equivalent to i.

2) Correlation Coefficient (CC) reflects the degree of correlation of IS with F. The bigger CC is,

the more information can be obtained by F from IS and the better the fusion effect is.


88

（𝐼𝑆, 𝐹） =∑ ∑ ( (𝑖,𝑗)− ) ( (𝑖,𝑗)− )

𝑗 1 𝑖 1

√(∑ ∑ ( (𝑖,𝑗)− )2 𝑗 1

𝑖 1 ) (∑ ∑ ( (𝑖,𝑗)− )2

𝑗 1 𝑖 1 )

(13)

where, S, are the respective average gray value of IS and F.

3) Spectral Distortion (SDi) is one of metrics for measuring distortion of the spectrum of the

image. A lower SDi tends to promise a high quality of image fusion, if the metrics is adopted in this

field.

𝑆𝐷 =1

𝑀 𝑁∑ ∑ 𝑠(𝐼𝑆( , 𝑗) − 𝐹( , 𝑗))𝑁

𝑗=1𝑀𝑖=1 (14)

4.3 Discussion of Fusion Results

In order to illustrate the advantages of the method we proposed, a comparison with such existing

methods has been carried out as dual channel PCNN, morphological pyramid, and wavelet fusion.

Figure 3 shows the fusion effects of the four methods. Subjectively, fuzzy and edge phenomena are

presented in eyes in wavelet fusion, and the fusion result showed new information (background color

became bright) outside the brain. The fusion by the morphological pyramid had better contrast but also

had false edge. The fusion by dual-channel PCNN had relatively clear edge and texture, while BEMD

fusion not only had the advantage of dual-channel PCNN, but also had higher contrast and clearer edge.

In the extreme case of BEMD fusion, all BIMFs were added together (with very small res), and the

fusion result was the same as that by dual channel PCNN. Therefore, the decomposition algorithm in

this paper is valid and has better fusion effect than that obtained by direct use of m-PCNN. Objective

evaluation is shown in Table 1. The experimental results demonstrate that BEMD is very effective in

the fusion of this kind of images.

(a) CT (b) MRI (c) dual-channel PCNN

(d) Morphological pyramid (e) DWT (f) Our method

Figure 3. First group CT and MRI medical image

Table 1 Evaluation on fused medical images

Method IE CC SDi

Dual-channel PCNN 4.0145 0.7186 14.1691

Morphological pyramid 4.2365 0.6621 18.8377

DWT 3.9836 0.6204 41.6147

Our method 4.7758 0.8077 14.9555


89

Figure 4 is the fusion results of another group of multi-modal medical images. Subjectively, the

morphological pyramid had better contrast, but the fused result also had edge problem. There was local

fuzziness in wavelet fusion. For dual channel PCNN, the intracranial part was not clear, while the

texture obtained after decomposition by BEMD was clearer. Objective evaluation is shown in Table 2.

The experimental data demonstrate the advantages of BEMD decomposition method.

(a)CT (b) MRI (c) dual-channel PCNN

(d) Morphological pyramid (e) DWT (f) Our method

Figure 4. The second group CT and MRI medical image

Table 2 Evaluation on fused of medical images

Method IE CC SDi

Dual-channel PCNN 5.6345 0.7710 34.7913

Morphological pyramid 6.4545 0.8030 25.8232

DWT 5.9302 0.8492 39.3677

Our method 6.5736 0.8368 23.9086

5 Conclusion

BEMD is an adaptive decomposition method of multi-scale images, and of better characteristics

than wavelet decomposition. First, BEMD is a kind of data-driven decomposition, and does not require

predefined wavelet function as wavelet decomposition does; second, BEMD is particularly suited to

analyze bidimensional non-linear and nonstationary data. m-PCNN overcomes the shortcoming of the

original PCNN mode that multiple PCNN modes are needed to calculate ignition matrix respectively,

and is highly efficient. In this paper, the definition of m-BIMF is proposed through the analysis of

characteristics of BIMF to repartition BIMF so as to make BIMF more suitable for image fusion. The

experiments demonstrate that m-PCNN image fusion algorithm based on BEMD propose in this paper

achieves better fusion result.

m-BIMF, a concept put forward based on BEMD decomposition in this paper, achieves better

results when applied to image fusion. But when an image is used in fusion, how many m-BIMFs it

should be partitioned into, and each m-BIMF should be the sum of which BIMFs still requires further

research. With the improvement of the decomposition theory and algorithm of bidimensional empirical

mode decomposition, BEMD will certainly play a greater role in medical image fusion.

6 Acknowledgement

This paper is supported by the Technology development plan of Jilin Province (No.201105017).


90

7 References

[1]Nathan Blow,“In vivo molecular imaging: the inside job”, Nature Method, vol.6, no.6, pp.465-469,

2009.

[2] Zheng You-zhi, Qin Zheng, “Medical image fusion algorithm based on bidimensional empirical

mode decomposition”, Journal of Software,vol.20,no.5,pp.1096-1105,2009.

[3]Lori A. Overturf, Mary L. Comer, Edward J. Delp, “Color image coding using morphological

pyramid decomposition” , IEEE Transaction on Image Processing,vol.4.no.2,pp. 177-185,1995.

[4]María González-Audícana, José Luis Saleta, Raquel García Catalán, Rafael García, “Fusion of

multispectral and panchromatic images using improved HIS and PCA mergers based on wavelet

decomposition”,IEEE Transaction on Geoscience and Remote sensing,vol.42,no.6,pp.1291-1299,2004.

[5]Li Fan,Yudong Zhang,Zhenyu zhou,David P.Semanek,Shuihua Wang,Lenan Wu,“An improved

image fusion algorithm based on wavelet decomposition”,Journal of Convergence Information

Technology, vol.5, no.10, pp.15-21, 2010.

[6]Yong Yang,”Performing Wavelet Based Image Fusion through Different Integration Schemes”,

International Journal of Digital Content Technology and its Application, vol.5, no.3, pp.156-166, 2011.

[7]Norden E.Huang,Zheng Shen,Steven R.Long, Manli C.Wu, Hsing H.Shih,Quanan Zheng,

Nai-Chyuan Yen,Chi Chao Tung,Henry H Liu, “The empirical mode decomposition and the Hilbert

spectrum for nonlinear and nonstationary time series analysis ”, In Proceeding the royal of society

pp.903-995,1998.

[8]MarcusDaig,Tostren Schlurmann,“Performance and limitations of the Hilbert-Huang transformation

(HHT) with an application to irregular water waves”, ocean Engineering, vol.31,pp.1783-1834,2004.

[9]J.C.Nunes,Y.Bouaoune,E.Delechelle,O.Niang,Ph,Bunel, “image analysis by bidimensional

empirical model decomposition” Image and Vision Computing ,vol.21,pp.1019-1026,2003.

[10]R.Eckhorn, H.J.Reitboeck, M.Arndt, P.W.Dicke, “Feature linking via synchronization among

distributed assemblies:simulationof results from cat cortex”, Neural Computation, vol.2, pp.293-307,

1990.

[11] R.P.Broussard, S.K.Rogers, M.E.Oxley,Gregory L.Tarr, “Physiologically motivated image fusion

for object detection using a pulse coupled neural network”, IEEE Transactions on Neural

Network,vol.10,no.3,pp.554-563,1999.

[12]Wei Huang, Zhongliang Jing, “Multi-focus image fusion using pulse coupled neural network”,

Pattern Recognition Letters, vol.28, pp.1123-1132, 2008.

[13]Zhaobin Wang, Yide Ma, “medical image fusion using m-PCNN”, information fusion, vol.9,

pp.175-186, 2008.


91

a novel method of medical image fusion based on ... · a novel method of medical image fusion based...

Documents