ieee transactions on geoscience and …naotoyokoya.com/assets/pdf/nyokoyatgrs2017.pdf · ieee...

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 1

Multisensor Coupled Spectral Unmixing forTime-Series Analysis

Naoto Yokoya, Member, IEEE, Xiao Xiang Zhu, Senior Member, IEEE,Antonio Plaza, Fellow, IEEE

Abstract—We present a new framework, called multisensorcoupled spectral unmixing (MuCSUn), that solves unmixing prob-lems involving a set of multisensor time-series spectral images inorder to understand dynamic changes of the surface at a subpixelscale. The proposed methodology couples multiple unmixingproblems based on regularization on graphs between the time-series data to obtain robust and stable unmixing solutions beyonddata modalities due to different sensor characteristics and theeffects of non-optimal atmospheric correction. Atmospheric nor-malization and cross-calibration of spectral response functionsare integrated into the framework as a preprocessing step.The proposed methodology is quantitatively validated using asynthetic dataset that includes seasonal and trend changes on thesurface and the residuals of non-optimal atmospheric correction.The experiments on the synthetic dataset clearly demonstrate theefficacy of MuCSUn and the importance of the preprocessingstep. We further apply our methodology to a real time-seriesdata set composed of 11 Hyperion and 22 Landsat-8 imagestaken over Fukushima, Japan, from 2011 to 2015. The proposedmethodology successfully obtains robust and stable unmixingresults and clearly visualizes class-specific changes at a subpixelscale in the considered study area.

Index Terms—Coupled spectral unmixing, multisensor datafusion, time-series analysis, change detection

NOMENCLATURE

B Number of bands.B0 Maximum number of bands.P Number of pixels.M Number of endmembers.N Number of time-series images.K Number of neighbors.Y Spectral data matrix B × P .A Endmember matrix B ×M .X Abundance matrix P ×M .N Residual matrix B × P .

This work was supported by Japan Society for the Promotion of Science(JSPS) KAKENHI 15K20955, Alexander von Humboldt Fellowship forpostdoctoral researchers, and Helmholtz Young Investigators Group “SiPEO”(VH-NG-1018, www.sipeo.bgu.tum.de).

N. Yokoya is with the Department of Advanced Interdisciplinary Studies,the University of Tokyo, 153-8904 Tokyo, Japan, with the Remote Sens-ing Technology Institute (IMF), German Aerospace Center (DLR), 82234Wessling, Germany, and also with Signal Processing in Earth Observation(SiPEO), Technical University of Munich (TUM), 80333 Munich, Germany(e-mail: [email protected]).

X. X. Zhu is with the Remote Sensing Technology Institute (IMF), GermanAerospace Center (DLR), 82234 Wessling, Germany, and also with SignalProcessing in Earth Observation (SiPEO), Technical University of Munich(TUM), 80333 Munich, Germany (email: [email protected]).

A. Plaza is with the Hyperspectral Computing Laboratory, Department ofTechnology of Computers and Communications, University of Extremadura,E-10003 Caceres, Spain (e-mail: [email protected]).

R Relative spectral response function B ×B0.c0 Residual offset vector B × 1.c1 Residual gain vector B × 1.W Weighting matrix on graph P × P .G Adjacency matrix (K + 1)× (K + 1).

I. INTRODUCTION

A synergetic use of hyperspectral and multispectral imagesis important for upcoming spaceborne hyperspectral imagingprojects [1]–[5]. Spaceborne hyperspectral imaging is capableof monitoring and interpreting detailed dynamic processes ofthe surface on a global scale; however, revisit time is limitedcompared to multispectral imaging satellites. For example, theenvironmental mapping and analysis program (EnMAP) [1],which will record more than 240 spectral bands at 0.4–2.5 µmat a ground sampling distance of 30 m, has a 27-day revisitcycle (off-nadir four days). In contrast, recent multispectralsatellites have higher revisit times, e.g., Landsat-8 has a 16-dayrevisit cycle and Sentinel-2 will exhibit a 5-day revisit cyclewith a constellation of two operational satellites (10 days withone satellite) [6], [7]. Improved understanding of the dynamicson the surface is expected by synergistically analyzing a set oftime-series spaceborne hyperspectral and multispectral images.

In the last decade, research on the analysis of time-serieshyperspectral images has focused on pixel-level change de-tection. One of the first examples of change detection usingbi-temporal hyperspectral images is presented in [8]. Linear-transform-based methods have been investigated as a majorunsupervised approach. Change vector analysis [12], whichwas originally proposed for analyzing multispectral imagesand is now widely used in various applications, has beenextended to unsupervised detection of multiple changes us-ing multitemporal hyperspectral images [13], [14]. Target oranomaly detection techniques have been naturally applied tochange detection using multiple hyperspectral images [15]–[17]. Classification has also been demonstrated to be a power-ful supervised approach to obtaining changes from multitem-poral and multisource spectral data [18].

In recent years, research on time-series hyperspectral un-mixing has been receiving particular attention due to itsability to understand class-specific changes at a subpixel level[19]–[29]. One of the first examples of time-series hyper-spectral unmixing is presented in [19]. Several publicationson multitemporal spectral unmixing have shown its efficacyin a wide range of applications [19]–[23]. Recent advancesfocused on applying spectral-library-based unmixing methods


to multitemporal unmixing problems [22], [26]. A data-driven(unsupervised) approach has also been proposed in [27]–[29].

To the best of the authors’ knowledge, time-series spectralunmixing has been scarcely investigated from the viewpointof synergistically exploiting hyperspectral and multispectralimages. To tackle multisensor time-series spectral unmixing,determining how to use hyperspectral and multispectral imagesin a complementary fashion is a challenging issue: there isno standard framework available. In addition, most of theprevious studies on time-series hyperspectral unmixing do nottake into account possible data mismatches due to non-optimalatmospheric correction, even though it always appears in time-series spectral data [30]–[32].

This paper proposes a novel framework, called multisensorcoupled spectral unmixing (MuCSUn), for time-series analy-sis. Our goal is to understand dynamic changes on the surfaceat a subpixel scale from multisensor time-series data, beyonddata modalities, due to different sensor characteristics andthe effects of non-optimal atmospheric correction. A singlespectral unmixing problem is solved jointly with those relatedto other images that have similar surface conditions to obtain arobust, stable, and accurate solution. Regularization on graphsbetween neighborhood images is used to couple multipleunmixing problems. Our technical contribution is twofold:1) we propose a MuCSUn algorithm based on regularizationon graphs between multisensor and multitemporal spectralimages; and 2) we introduce cross-calibration methods forestimating relative spectral response functions (SRFs) andresidual gains and offsets due to non-optimal atmosphericcorrection, and integrate them into the MuCSUn framework.

The remainder of the paper is organized as follows: SectionII introduces our methodology. Experimental results on syn-thetic and real datasets are presented in Sections III and IV,respectively. Section V concludes the paper with some remarksand hints at plausible future research lines.

II. METHODOLOGY

We first provide an overview of the linear spectral unmixingchain that forms the basis for our methodology. Next, weformulate an optimization problem for MuCSUn based ongraphs between multisensor time-series images, and derive analgorithm using the augmented Lagrangian method. Finally,we introduce cross-calibration methods for estimating relativeSRFs and residual gains and offsets due to non-optimalatmospheric correction.

A. Linear spectral unmixing

A linear spectral mixture model is commonly used forunmixing problems owing to its physical effectiveness andmathematical simplicity. The spectrum at each pixel is as-sumed to be a linear combination of several endmemberspectra. Therefore, a hyperspectral image Y ∈ RB×P , withB bands and P pixels, is formulated as

Y = AXT + N, (1)

where A ∈ RB×M is the spectral signature matrix, withM being the number of endmembers, X ∈ RP×M is the

abundance matrix, N ∈ RB×P is the residual, and the operator()T denotes the transposition operation. Assuming that theresidual is approximated as zero-mean white Gaussian noise,a spectral unmixing problem can be generally formulated as

minA,X‖Y −AXT ‖2F + f(A,X)

s.t.A � 0, X � 0, X1M = 1P ,(2)

where ‖ · ‖F denotes the Frobenius norm and f(A,X) is aregularization term. 1M and 1P are column vectors with allelements being equal to one; the subscripts denote the numberof elements. The abundance nonnegativity constraint (X � 0)and the abundance sum-to-one constraint (X1M = 1P ) arewidely used in the literature [33]. In cases when spectralvariability explained by scaling is considered, it is proper toassume that the abundance fractions sum to less than one(X1M � 1P ). It can be also implemented with the sum-to-one constraint by adding a shade endmember. Researchershave studied many models and algorithms based on geometri-cal, statistical, and sparse regression-based approaches in thesearch for robust, stable, tractable, and accurate unmixing [34].

When the spectral signature matrix A is available, the maintask of spectral unmixing is to determine the abundances X.Here we suppose that the spectral signature matrix is obtainedeither from a spectral library or from the data itself, satisfyingthe condition M ≤ B. When f(A,X) = 0, the abundancescan be obtained by solving constrained least-squares (CLS)problems for all pixels via quadratic programming [35].

Since the number of endmembers at each pixel is usuallyvery limited, sparsity of abundances can be also used as anadditional constraint: ‖Xj‖0 ≤ K, j = 1, ..., P , where Xj

denotes the jth row vector of X. Combinatorial optimizationmethods are useful for solving this problem and commonlyadopted in various applications [36]. In recent years, sparseregression methods have attracted particular attention due tothe concurrent theoretical development of compressed sensing[37]–[41]. These approaches can solve the problem, even if itis underdetermined owing to the presence of a large numberof spectral signatures in A (M > B).

Data-driven endmember extraction methods have beenwidely investigated as a means of handling cases where thespectral signature matrix (A) is unknown. Geometrical-basedapproaches such as pure pixel index (PPI) [42], N-FINDR[43], and vertex component analysis (VCA) [44] are popular,with their assumption that pure pixels of all endmembersare included in the data. Minimum volume based algorithmswere developed to tackle the problem without the pure pixelassumption [45]. After determining A, we can obtain X bythe CLS method.

Nonnegative matrix factorization (NMF) algorithms [46],[47] have emerged as useful unmixing methods because theyovercome the difficulties related to the absence of pure pixelswith straightforward implementation and sufficient mathemat-ical flexibility to consider several constraints [48]–[54]. NMFbased algorithms solve (2) via alternating optimization. NMFalternately minimizes the objective function over A with Xfixed, and over X with A fixed with proper initialization, e.g.,using the above mentioned methods based on the pure-pixel


assumption. Various regularization terms have shown to beeffective, for example, minimum volume regularization [48],L1/2 sparsity regularization [52], and graph regularization[53].

In this work, we formulate spectral unmixing of eachtemporal image based on (2) and adopt alternating optimiza-tion for flexibility, with conventional methods being used forinitialization. The L1/2 sparsity regularization on abundancesis considered for the regularization term owing to its strongsparsity promotion even with the abundance sum-to-one con-straint. The L1/2 regularization term is defined as

f(X) = α‖X‖1/21/2, (3)

where ‖X‖1/21/2 =∑P,Mj,m |(X)jm|1/2 and (X)jm denotes the

(j,m)th entry of the matrix X. The parameter α controls thesparsity of the abundances and can be estimated based on thesparseness criterion in [55], which is defined by

α =1√B

B∑i=1

√P − ‖Yi‖1/‖Yi‖2√

P − 1. (4)

B. Multisensor coupled spectral unmxing with graph regular-ization

To tackle spectral unmixing of multisensor time-series data,we have to handle data modalities due to environmental con-ditions and instrumental configurations. In this work, we takeinto account effects of non-optimal atmospheric correction andSRFs that mainly characterize the data modality. The linearspectral mixture model is thus rewritten as

Y = diag(1B + c1)RAXT + c01TP + N, (5)

where c1 ∈ RB×1 is a vector with each element repre-senting the residual gain for each band (an ideal value is0), R ∈ RB×B0 is the relative SRFs resampled with B0

bands, c0 ∈ RB×1 is a vector with each element representingthe residual offset for each band (an ideal value is 0), andthe spectral signature matrix has B0 bands. In cases wheredata-driven endmember extraction methods are used, B0 isdefined as the number of spectral bands of the highest spectralresolution among multiple spectral imagers. When we use aspectral library for the endmember spectral signatures, B0 isdefined by the spectral resolution of the library.

We first focus on estimating the spectral signatures (A)and the abundances (X), assuming that the residual param-eters (c0 and c1) for refinement of atmospheric correction(or atmospheric normalization) and the relative SRFs (R)are known but contain some errors. The estimation of (c0,c1) and R will be given in Section II-C. If we replacediag(1B+c1)−1(Y−c01

TP ) with Y and RA with A, (5) can

be simplified to the traditional linear spectral mixture modelshown in (1). In other words, the traditional linear spectralmixture model assumes perfect atmospheric correction andknowledge of the SRF, although the former condition maynever happen in real cases [32], particularly when we dealwith multisensor time-series data. Therefore, it is important tounmix multiple spectral images on a coupled basis to achieve

Hyperspectral data

Multispectral data

Fig. 1. Manifold of multisensor time-series spectral data

robustness against residuals in atmospheric correction andrelative SRFs.

To this end, we introduce MuCSUn, which solves an un-mixing problem of a single image jointly with those of itsneighboring images (in the time-series hierarchy) by adoptingan intrinsic manifold of the time-series dataset. Fig. 1 illus-trates the manifold of multisensor time-series spectral data.Each sample represents a single image. Owing to changeson the surface, the multisensor time-series dataset forms themanifold, where neighborhood images have similar conditionsof the surface. We formulate MuCSUn as a decentralizedoptimization problem for multi-agent systems [56], i.e., eachspectral unmixing instance solves a local optimization problembased only on information concerning other spectral unmixingproblems in its neighborhood.

We use regularization on local graphs to couple the multiplespectral unmixing problems. In a local graph, a target imageand its neighbors, which are regarded as nodes, are connectedby edges. Local graphs can be mixtures between undirectedand directed graphs, depending on the difference in the spectralresolution between the two images. We use undirected graphsbetween images with similar spectral resolution so that localspectral unmixing problems mutually influence each other. Theaccuracy of the individual spectral unmixing problems dependshighly on the spectral resolution of the data, e.g., hyperspectralimaging enables more accurate spectral unmixing than multi-spectral imaging. If the target image is a multispectral data set,hyperspectral neighboring images can improve the accuracyof the spectral unmixing for the target image; however, thereverse is not necessarily the case. Therefore, we also considerdirected graphs, where the edges have a direction from higher-spectral-resolution data to lower-spectral-resolution data in themanifold (Fig. 1).

Let us consider K + 1 images {Yk ∈ RBk×P }K+1k=1 , i.e.,

a target image and its K-nearest neighbors. The numbers ofspectral bands satisfy B1 ≤ B2 ≤ ... ≤ BK+1 without lossof generality. For individual spectral unmixing, we adopt theL1/2 sparsity regularization on abundances. The problem of


coupled spectral unmixing can be formulated as

minA,X,B,Z

1

2

K+1∑k=1

‖Yk −AkXTk ‖2F + α

K+1∑k=1

‖Xk‖1/21/2

+1

2β

K∑p=1

K+1∑q=p+1

P∑j,l=1

((G)pq(G)qp‖Xjp −Xl

q‖22

+((G)qp − (G)pq)‖Xjp − Zlq‖22)(Wpq)jl

+γ

K∑p=1

K+1∑q=p+1

((G)pq(G)qp‖RpqAq −Ap‖2F

+((G)qp − (G)pq)‖RpqBq −Ap‖2F )

s.t.Ak � 0, Xk � 0, Xk = Zk, Ak = Bk, Xk1M = 1P ,(6)

where A = [AT1 , ...,A

TK+1]T , X = [XT

1 , ...,XTK+1]T , Ak ∈

RBk×M and Xk ∈ RP×M denote the spectral signature matrixand the abundance matrix of the kth data Yk, respectively.B = [BT

1 , ...,BTK+1]T and Z = [ZT1 , ...,Z

TK+1]T are auxiliary

variables that are introduced to implement the directed-graphregularization. In addition to the L1/2 sparsity constraint foreach unmixing, the term with the parameter β is the graphregularization, which places the restriction such that, if twospectral signatures are similar, the abundance vectors are alsosimilar to each other. Wpq ∈ RPp×Pq is the weight matrixon the graph between the pth and qth data, and (Wpq)jlrepresents the closeness of two points: the jth column ofYp and the lth column of Yq . Using the last penalty termwith the parameter γ, we expect that the neighborhood imagesshare similar endmember signatures. Rpq ∈ RBp×Bq is therelative SRF defined between the sensors that observed thepth and qth data. G ∈ R(K+1)×(K+1) is the adjacencymatrix that represents the local graph between K + 1 images.Gpq = Gqp = 1 if the pth and qth images are connected bythe undirected edge, whereas Gpq = 0 and Gqp = 1 if theyare connected by the directed edge from the qth image to thepth image.

The formula in (6) can be solved by alternating optimiza-tion. The alternating update rules (7)–(12) are obtained bythe augmented Lagrangian method. More details on derivingthese update rules are given in the Appendix. X

−1/2k denotes

a matrix where each element is the reciprocal of the squareroot of the element in Xk � 0. If any entry becomes zeroin Xk, a very small positive number will be added to satisfyXk � 0 [52]. Υk and Θk are the Lagrange multipliers andσ > 0 and ρ > 0 are the penalty parameters. Dpq1 ∈ RPp×Pq

and Dpq2 ∈ RPp×Pp denote diagonal matrices whose entriesare column and row sums of Wpq , respectively.

We detail how to select the K-nearest neighbors for eachtarget image and construct the matrices G and W. We presenttwo strategies for neighbors selection, namely, sequentialcoupling and manifold-based coupling. Sequential couplingselects temporally neighboring K images. Manifold-based cou-pling finds the K-nearest neighbors of a target image basedon the spectral similarity measurement over the whole sceneusing a distance metric, such as the heat kernel, spectral angledistance (SAD), or spectral information divergence (SID).

We calculate the spectral similarity between the target imageand the other images on a pixel-by-pixel basis. The averageof calculated similarity values is used to find the K-nearestneighbors. After finding the neighbors, the adjacency matrix Gis constructed based on the spectral resolution so that imageswith similar spectral resolution are connected by undirectededges and directed edges are drawn from higher-spectral-resolution data to lower-spectral-resolution data.

W is defined by the spectral similarity measurement usingthe same distance metric as the one used for neighbors selec-tion. The choice of a distance metric depends on characteristicsof multisensor time-series images. In this paper, the heatkernel is used for synthetic data in Section III, whereas theSAD is used for real data in Section IV to deal with locallyvariable atmospheric conditions (e.g., thin clouds). Note thatthe spectral similarity measurement is carried out with thelowest spectral resolution for consistency of measured valuesby spectrally downgrading higher-spectral-resolution data us-ing the relative SRF. The simplest way to construct W is tocalculate the spectral similarity between only a pair of spatiallycorresponding pixels. In this case, Wpq = Dpq1 = Dpq2

if Pp = Pq is satisfied. We use this technique to savecomputational costs.

Note that (7)–(12) are the update rules for solving (6)without the sum-to-one constraint. To satisfy the abundancesum-to-one constraint, the data matrix Yk and the spectralsignature matrix Ak are augmented by constants defined byYk = [YT

k δ1P ]T and Ak = [ATk δ1P ]T , where δ controls

the impact of the constraint [35]. A moderate parametervalue relaxes the constraint to some extent. The abundancesum-to-one constraints can also be explicitly included in theoptimization, and the update rules can be derived by theaugmented Lagrangian method. We adopt the former techniqueto avoid strict sum-to-one constraints, which do not properlyreflect reality due to spectral variability.

The iterative process is terminated when the following twoconditions are satisfied. The first one is the condition that thechange ratio of the objective function O achieves a value belowa given threshold εo, i.e., ‖∇O‖2 ≤ εoO, where εo is set to10−3 in our experiments. The second one is the convergence ofresiduals for the equality constraints, i.e., ‖A−B‖F√

P×M×(K+1)≤ εr

and ‖X−Z‖F√P×M×(K+1)

≤ εr, where εr is set to 10−4. The penalty

parameters (σ and ρ) affect the convergence of the residualsand the choice of these parameters is an open question.In the literature, it has been recommended to use possiblydifferent penalty parameters for improving the convergencewhile making performance less dependent on the initial choice.We adopt a parameter adjustment scheme in [63] that increasesthe penalty parameter with a factor τ only if the residual isnot decreased by a factor µ < 1, where τ and µ are set to 10and 0.25, respectively, and the initial parameter is set to 10−5.The Lagrange multipliers (Υ and Θ) are typically initializedas 0. For a practical utility, the maximum number of iterationsis also used as the stopping criteria, which is set to 200 in theexperiments.

The proposed algorithm for MuCSUn is summarized inAlgorithm 1.


(Ak)im ← (Ak)im

(YkXk+γ(k−1∑p=1

(G)pk(G)kpRTpkAp+

K+1∑q=k+1

((G)kq(G)qkRkqAq+((G)qk−(G)kq)RkqBq))+Υk)im

(AkXTk Xk+γ(

k−1∑p=1

(G)pk(G)kpRTpkRpkAk+

K+1∑q=k+1

((G)kq(G)qk+(G)qk−(G)kq)Ak))im

(7)

(Bk)im ← (Bk)im

(γk−1∑p=1

((G)kp−(G)pk)RTpkAp+σAk)im

(γk−1∑p=1

((G)kp−(G)pk)RTpkRpkBk+Υk+σBk)im

(8)

(Υk)im ← (Υk)im + σ(Bk −Ak)im (9)

(Xk)jm ← (Xk)jm

(YTk Ak+β(

k−1∑p=1

GpkGkpWTpkXp+

K+1∑q=k+1

((G)kq(G)qkWkqXq+((G)qk−(G)kq)WkqZq))+Θk)jm

(XkATk Ak+αX

−1/2k +β(

k−1∑p=1

(G)pk(G)kpDpk2Xk+K+1∑

q=k+1

((G)kq(G)qk+(G)qk−(G)kq)Dkq1Xk))jm

(10)

(Zk)jm ← (Zk)jm

β(k−1∑p=1

((G)kp−(G)pk)WTpkXp+ρXk)jm

(βk−1∑p=1

((G)kp−(G)pk)Dpk2Zk+Θk+ρZk)jm

(11)

(Θk)jm ← (Θk)jm + ρ(Zk −Xk)jm (12)

Algorithm 1 MuCSUn based on graph regularizationInput: YOutput: A, X

1: for i = 1 to N do2: calculate W between all combinations between Yi and

the others3: find K nearest neighbors of Yi

4: prepare G5: initialize {Ak,Xk}k∈Vi by individual spectral unmix-

ing (Vi is a set of the ith target and K neighbor images)

6: solve (6) by alternating updates of (7)–(12)7: update {Ai,Xi}8: end for9: return {Ai,Xi}Ni=1

C. Atmospheric normalization and cross-calibration of SRFs

The estimation of the residual parameters and the relativeSRFs represents the key to handling multisensor time-seriesdatasets. In this section, we first introduce two atmosphericnormalization methods to estimate the residual gains andoffsets (c0, c1). One method is based on spectral unmixing andthe other is based on cross-normalization. Next, we present animage-based method for estimating relative SRFs.

1) Atmospheric normalization based on spectral unmixing:We assume that the relative SRFs (R) and the initial endmem-ber spectra (A) are given. For example, R is usually measuredpre-launch and may be updated during in-flight calibrationand monitoring. We suppose that an approximation of A isdetermined using a spectral library or extracted from a masterimage (or images) by means of user interaction. If R and Aare available, c1 and c0 can be obtained by the least squaresmethod after estimating X. Expression (5) is thus rewritten as

Y = RAXT + diag(c1)RAXT + c01TP + N. (13)

Assuming that c0 ∼ N (0, σ20I), c1 ∼ N (0, σ2

1I), andσ0, σ1 � 1, diag(c1)RAXT and c01

TP can be approximated

as Gaussian residuals. Therefore, we approximate X by solv-ing

minX‖Y −RAXT ‖2F

s.t.X � 0.(14)

X is obtained using the CLS method. The residual gain andoffset for the ith band (c1i, c0i) is determined by

[1 + c1i c0i]T = (XT

i Xi)−1XT

i YiT , (15)

where Xi = [(RAXT )iT

1P ]. When the number of spec-tral bands (B) of the target image is large (e.g., Y is ahyperspectral data set), the assumptions (c0 ∼ N (0, σ2

0I),c1 ∼ N (0, σ2

1I)) are satisfied and therefore we can estimatethe coefficients accurately.

2) Image-based atmospheric normalization: The methoddescribed above causes large prediction errors when B is small(e.g., Y is a multispectral data set) because diag(c1)RAXT

and c01TP cannot be approximated as Gaussian residuals. In

this case, we adopt image-based cross-normalization methods,which have been widely used in multitemporal studies [57],[58]. We prepare a reference image (Z ∈ RB0×P ) that isalready accurately corrected by the estimated residual pa-rameters (e.g., hyperspectral data). We assume that there areunchanged pixels between the target and reference images.The relationship between the two images can be written as

YA = diag(1B + c1)RZA + c01TP + N. (16)

Here A denotes a subset of unchanged pixels, which canbe determined by similarity measures of spectral signaturesbetween spatially corresponding pixels. If R is available,c0 and c1 can be obtained by the least squares method.Considering that the reference image can still include errorsdue to non-optimal atmospheric correction, the use of morethan one reference image leads to better estimation of the


coefficients. However, to satisfy the assumption of unchangedpixels, we need to select reference images close to the targetimage in the manifold of the time-series dataset. Note thatwhen A is extracted from a single (master) image, its residualparameters can be regarded as ideal (c0 = 0B and c1 = 0B);therefore this image is suitable for being used as a referencefor cross-normalization purposes.

3) Cross-calibration of SRFs: We consider now the image-based estimation of relative SRFs, particularly those of amultispectral imager spectrally resampled with the numberof hyperspectral bands. Although pre-launch SRFs are usu-ally available for spaceborne spectral imagers, image-basedmethods for estimating relative SRFs have been shown to beeffective for data fusion purposes [59], mainly because a linearapproximation of SRFs of the multispectral imager by usingthose provided by the hyperspectral imager always includeserrors and, furthermore, SRFs can change slightly on-boarddue to launch or secular change issues [60]. Now consider theproblem of estimating the relative SRFs with residual gains(diag(1B+c1)R) and offsets (c0) based on (16). The problemfor each band can be expressed as

min(1+c1i)Ri,c0i

‖YiA − (1 + c1i)R

iZA − c0i1TP ‖22

s.t. (1 + c1i)Ri � 0TB0

,(17)

which can be solved via quadratic programming [61]. Weassume that the residuals due to non-optimal atmosphericcorrection have statistically no pattern, i.e., the mean of theresidual gain for each band of the multispectral imager is equalto zero (c1 = 0B). Therefore the relative SRFs can be obtainedas R = 1

L

∑Lk=1(diag(1B + c1)R)k, where L is the number

of images acquired by the multispectral imager.The method for estimation of the residual parameters and

relative SRFs presented in this section is summarized inAlgorithm 2.

III. EXPERIMENT ON SYNTHETIC DATA

A. Synthetic multisensor time-series spectral images

We use a set of synthetic multisensor time-series spectralimages to numerically validate the proposed methodology. Thesynthetic dataset was simulated using the abundance maps ofthe Fractal 1 synthetic data [62]. The Fractal 1 image iscomposed of nine endmembers; their abundance maps weregenerated from fractal patterns with the size of 100 × 100pixels. To simulate realistic changes on the surface, we mainlyconsider seasonal changes and trend changes, which representtwo major types of changes on the surface. We selected nineendmembers from the U.S. Geological Survey (USGS) spectrallibrary: grass, dry grass, oak, soil, melting snow, water,asphalt, green house, and concrete. The reference abundancemaps of endmembers #1–9 are used for (grass, dry grass), drygrass, (oak, soil), soil, (grass, melting snow), water, asphalt,(green house, soil), and (concrete, soil), respectively, where ( ,) denotes a pair. Three pairs (grass, dry grass), (grass, meltingsnow), and (oak, soil) were used to simulate seasonal changes.We assume that grass and oak are foreground objects and drygrass, melting snow, and soil are background objects. Seasonal

Algorithm 2 Atmospheric normalization and cross-calibrationof SRFsInput: YOutput: Corrected Y, {Ri}Ni=1

1: for i = 1 to N do2: if Bi = B0 then3: Ri = diag(1B0

)4: if initial Ai is given from a master image then5: estimate {c0i, c1i} by the image-based method

with the master image6: else7: estimate {c0i, c1i} by the spectral-unmixing-based

method8: end if9: correct Yi by diag(1B + c1i)

−1(Yi − c0i1TP )

10: else11: estimate {((1B + c1)R)i, c0i} by the image-based

method12: end if13: end for14: for i = 1 to N do15: if Bi < B0 then16: Ri = 1

|S|∑k∈S(diag(c1)R)k where S is a subset

dataset taken by the same sensor as the ith image17: estimate {c0i, c1i} by the cross-normalization

method with the reference images that have B0 bands

18: correct Yi by diag(1B + c1i)−1(Yi − c0i1

TP )

19: end if20: end for21: return {Yi,Ri}Ni=1

abundance changes of foreground objects were simulated asthe multiplication of periodic functions and the reference abun-dances. The expressions (1−sin( 2πt

365 ))/2 and (1−cos( 2πt365 ))/2

were used as periodic functions for grass and oak, respectively,where t denotes the day in which the images were taken. Weused the two different periodic functions to simulate differentseasonal growth cycles of vegetation. The residual abundanceswere added to the abundances of background objects. Othertwo pairs (green house, soil) and (concrete, soil) were selectedfor simulating trend changes defining green house and concreteas foreground objects and soil as background. The monotonicincreasing function defined by t

365×5 was used to generatetime-series abundance maps of green house and concrete. Theresidual abundances were added to the abundances of soil.

The SRFs of AVIRIS (bands 1–32, 36–96, 100–160, and163–224) and Landsat-8 (bands 1–8) were used to simulatehyperspectral and multispectral images, respectively. AdditiveGaussian noise with a signal-to-noise ratio (SNR) of 100was added to each temporal image. Non-optimal atmosphericcorrection is simulated using a Gaussian random vector for theresidual gains as c1 ∼ N (0, σ2

1I). σ1 is set to 0.05 based onexperimental results from real data, which will be shown inSection IV. c0 is set to 0 for simplicity. The revisit cyclesof the hyperspectral and multispectral imagers were set to16 and 27 days, respectively, which correspond to those of


(c)

(b)

(a)

0

1

0

1

Ref

eren

ceIS

UC

SU

1C

SU

2R

efer

ence

ISU

CS

U 1

CS

U 2

Fig. 2. (a) Color composites of ten images in the synthetic dataset and their corresponding abundance maps of (b) grass and (c) concrete. Abundances ofreference and estimated abundances obtained by ISU, CSU 1, and CSU 2 are shown from top to bottom in (b) and (c).

Landsat-8 and EnMAP. We consider two scenarios for dataacquisition: 1) full acquisition, that assumes a clear sky for allimages, and 2) realistic acquisition, that assumes a clear skyfor 30% of the observations. The total observation period isset to five years and therefore the total number of imagesis 68 and 115 for hyperspectral and multispectral images,respectively. Composite color images are shown in Fig. 2(a)for ten examples. The corresponding reference abundances ofgrass and concrete are presented in the first row images ofFig. 2(b)(c).

B. Experimental resultsIn this experiment, we examine the efficacy of the follow-

ing four points: 1) coupled spectral unmixing compared toindividual spectral unmixing; 2) the use of multiple sensorscompared to the use of a single sensor; 3) manifold-based

coupling compared to sequential coupling; and 4) atmosphericnormalization. To investigate the first and third points, we usethree methods: 1) individual spectral unmixing (ISU) based onthe CLS method; 2) coupled spectral unmixing with sequentialcoupling (CSU 1); and 3) coupled spectral unmixing withmanifold-based coupling (CSU 2). ISU can be seen as aspecial case of MuCSUn that performs only the initializationprocess in line 5 in Algorithm 1. CSU 1 and CSU 2 are twovariations of MuCSUn and their difference is defined only bythe preparation process of the adjacency matrix G in line 4in Algorithm 1. To investigate the second point, we performthese methods on three different datasets: 1) only hyperspectralimages; 2) only multispectral images; and 3) hyperspectraland multispectral images. For the last point, we conduct allexperiments with and without atmospheric normalization. Tofairly validate our technical contributions, we simplify our


algorithm by setting α = 0 and γ = 0, and A is given andfixed. The number of neighbors is two and β = 1.

Table I shows the average root-mean-square errors (RM-SEs) of abundances, defined by 1

N

∑Ni=1 ‖Xi − Xi‖F , with

atmospheric normalization. The results were obtained for threedatasets using three considered methods in the two scenarios.For the realistic data acquisition scenario, we show the averageresults of 10 Monte Carlo trials obtained after selecting 30%of the observations. Note that only the unmixing accuracy ofmultispectral images is shown for the third dataset, becauseonly multispectral images can gain benefits, whereas theresults of hyperspectral images are the same as for the firstdataset. The coupled spectral unmixing algorithms show betterperformance compared to individual spectral unmixing. Theunmixing accuracy of multispectral data is highly improvedby solving unmixing problems with hyperspectral data in acoupled basis. This demonstrates the benefit of using multiplesensors for spectral unmixing or, more precisely, spectralunmixing of multispectral data can be greatly improved owingto the support of neighboring hyperspectral data. Coupledspectral unmixing based on manifold-based coupling out-performs estimations based on sequential coupling in manycases, particularly in the realistic data acquisition scenarios.This suggests that manifold-based coupling leads to moreaccurate and robust unmixing results than sequential coupling.Coupled spectral unmixing with sequential coupling performswell in ideal scenarios because sequential neighbors with hightemporal resolution represent neighbors in the manifold oftime-series data well. On the other hand, if temporal resolutionof time-series data is limited to realistic scenarios, manifold-based coupling is required because sequential neighbors donot always represent neighbors in the manifold.

Table II shows the unmixing accuracy obtained withoutatmospheric normalization. The unmixing accuracies greatlydecrease in all scenarios with all datasets and methods.This proves the importance of the estimation and correctionof the residual parameters. Major trends among the threemethods are similar to those observed in Table I, whereasit can be seen that there is no improvement obtained bycoupled spectral unmixing for the multispectral-only dataset.Multispectral unmixing problems are severely ill-posed beforeatmospheric normalization and therefore individual spectralunmixing results in large estimation errors. Since the proposedmethods use the result of individual spectral unmixing forinitialization, they converge to inaccurate local optima possiblyworse than the initial solution. The differences in accuracybetween Tables I and II are larger than those between indi-vidual and coupled approaches presented in the tables. Thisimplies that atmospheric normalization with the estimation ofthe residual parameters has a larger impact than the coupledspectral unmixing algorithm.

Figs. 2(b)(c) present reference abundance maps of grass andconcrete for a subset of time-series data with one trial of therealistic acquisition scenario and the corresponding estimatedabundance maps obtained by the three methods. Grass andconcrete are defined as endmembers with seasonal and trendchanges, respectively, as shown in the changes of abundances.Serial day numbers of the subset are 113, 289, 433, 737, 1009,

1137, 1313, 1473, 1697, and 1825, which are all simulatedas observed by the multispectral imager. In Fig. 2(b), thecoupled spectral unmixing methods shows better resemblanceto the reference compared to individual spectral unmixing. Inparticular, manifold-based coupling (CSU 2) exhibits the bestresults. In contrast, in Fig. 2(c), sequential coupling (CSU1) shows the best resemblance, whereas both CSU 1 andCSU 2 outperform individual spectral unmixing. These resultsimply that manifold-based coupling can better capture seasonalchanges and sequential coupling is more suitable for detectionof trend changes.

Fig. 3 shows time-series plots of abundances at four ex-ample points. The coupled spectral unmixing methods showrobust and stable estimation results. Manifold-based couplingresults in better estimation of seasonal abundance changes, asshown in the abundances of oak and soil of the first point(see Fig. 3(a)). Sequential coupling leads to good detectionof trend changes, as shown in the abundances of soil of thethird point (see Fig. 3(c)) and soil and green house of thefourth point (see Fig. 3(d)). These results also confirm theadvantages of manifold-based and sequential coupling in theproposed methodology.

Finally, we investigate the effect of the parameter β on theperformance in the experiment for the dataset composed ofboth hyperspectral and multispectral images with atmosphericnormalization. Fig. 4(a) and (b) plot the average RMSE ofabundances as a function of β for CSU 1 and CSU 2 under theideal and realistic data acquisition scenarios, respectively. Thechoice of β affects the results and the sensitivity trend dependson temporal resolution of the time-series dataset. In the idealscenario, both CSU 1 and CSU 2 perform well and stablewhen β ≥ 1. A similar sensitivity trend is observed in therealistic scenario when β ≤ 10−1; however, the performancedecreases as β increases larger than 1. This result indicatesthat a moderate use of graph regularization (e.g., β = 1) isappropriate in the realistic scenario because the neighboringimages include surface changes to some extent compared to atarget image.

In the experiments conducted using synthetic data, ourfindings are summarized below.

• Atmospheric normalization with the estimation of resid-ual parameters highly improves the unmixing accuracy;therefore it is required as preprocessing in time-seriesspectral analysis.

• Coupled spectral unmixing outperforms individual spec-tral unmixing.

• The use of multiple sensors can improve spectral unmix-ing of multispectral data.

• Manifold-based coupling is suitable for capturing sea-sonal changes, whereas sequential coupling is better ableto detect trend changes.

IV. EXPERIMENT ON REAL DATA

A. Time-series Hyperion and Landsat-8 images

To validate the proposed methodology using real data, anexperiment was performed on a multisensor time-series dataset


TABLE IAVERAGE RMSES OF ABUNDANCES OBTAINED AFTER ATMOSPHERIC NORMALIZATION FROM THREE DATASETS OF SYNTHETIC TIME-SERIES SPECTRAL

IMAGES, I.E. ONLY HYPERSPECTRAL (HS) DATA (68 IMAGES), ONLY MULTISPECTRAL (MS) DATA (115 IMAGES), AND BOTH HS AND MS DATA (183IMAGES) USING INDIVIDUAL SPECTRAL UNMIXING (ISU), COUPLED SPECTRAL UNMIXING WITH SEQUENTIAL COUPLING (CSU 1), AND

MANIFOLD-BASED COUPLING (CSU 2). RMSES FOR ONLY MS DATA ARE SHOWN FOR THE LAST DATASET.

Dataset HS MS MS (coupled with HS)Acquisition Ideal Realistic Ideal Realistic Ideal Realistic

ISU 0.013588 0.013336 0.06987 0.069512 0.06987 0.069512CSU 1 0.012392 0.013559 0.082818 0.068334 0.019508 0.040315CSU 2 0.012309 0.012407 0.064022 0.064740 0.022227 0.031097

TABLE IIAVERAGE RMSES OF ABUNDANCES OBTAINED BEFORE ATMOSPHERIC NORMALIZATION FROM THREE DATASETS OF SYNTHETIC TIME-SERIES

SPECTRAL IMAGES, I.E. ONLY HYPERSPECTRAL (HS) DATA (68 IMAGES), ONLY MULTISPECTRAL (MS) DATA (115 IMAGES), AND BOTH HS AND MSDATA (183 IMAGES) USING INDIVIDUAL SPECTRAL UNMIXING (ISU), COUPLED SPECTRAL UNMIXING WITH SEQUENTIAL COUPLING (CSU 1), AND

MANIFOLD-BASED COUPLING (CSU 2). RMSES FOR ONLY MS DATA ARE SHOWN FOR THE LAST DATASET.

Dataset HS MS MS (coupled with HS)Acquisition Ideal Realistic Ideal Realistic Ideal Realistic

ISU 0.028181 0.028526 0.11484 0.11719 0.11484 0.11719CSU 1 0.025268 0.026195 0.1161 0.12110 0.086309 0.092507CSU 2 0.025552 0.025896 0.11538 0.11840 0.049207 0.052016

500 1000 1500Days

0

0.2

0.4

0.6

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

Abun

danc

e

500 1000 1500Days

0

0.5

1

Abun

danc

e

500 1000 1500Days

0

0.5

1

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

Abun

danc

e

500 1000 1500Days

0

0.05

0.1

Abun

danc

e

500 1000 1500Days

0

0.05

0.1

0.15

Abun

danc

e

500 1000 1500Days

0

0.05

0.1

0.15

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

Abun

danc

e

Grass Dry grass Oak

Soil Melting snow Water

Asphalt Green house Concrete

500 1000 1500Days

0

0.5

1

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

0.3

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

0.6

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

0.6

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

0.6

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

0.3

Abun

danc

e

500 1000 1500Days

0

0.05

0.1

0.15

Abun

danc

e

500 1000 1500Days

0

0.05

0.1

0.15

Abun

danc

e

Grass Dry grass Oak



(a) (b)

500 1000 1500Days

0

0.2

0.4

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

0.3

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

0.6

Abun

danc

e

500 1000 1500Days

0

0.05

0.1

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

0.6

Abun

danc

e

Grass Dry grass Oak



(c)

500 1000 1500Days

0

0.1

0.2

0.3

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

0.6

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

Abun

danc

e

500 1000 1500Days

0

0.5

1

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

0.3

Abun

danc

e

500 1000 1500Days

0

0.1

0.2

0.3

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

Abun

danc

e

500 1000 1500Days

0

0.2

0.4

0.6

Abun

danc

e

Grass Dry grass Oak



(d)

Fig. 3. Time-series plots of abundances for nine endmembers at four example points (a)–(d) obtained by (blue) ISU, (green) CSU 1, (red) CSU 2, and (black)reference.

composed of Hyperion and Landsat-8 data. The dataset includes 11 Hyperion images and 22 Landsat-8 images acquired


10-2 100 1020.03

0.035

0.04

0.045

0.05

0.055

0.06

0.065

RM

SE

CSU 1 CSU 2

ȕ10-2 100 102

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

0.055R

MS

E

ȕ

CSU 1 CSU 2

(a) (b)

Fig. 4. Sensitivity to the change of the parameter β under (a) ideal and (b)realistic data acquisition scenarios.

0.5 1 1.5 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Wavelength (µm)

Spe

ctra

l ref

lect

ance

man-made 1man-made 2man-made 3man-made 4man-made 5

vegetation 1vegetation 2soilwater

1 km

Fukushima Daiichi nuclear power plant

Fig. 5. Location and spectral signatures of endmembers obtained fromHyperion data taken on April 29, 2012 near the Fukushima Daiichi nuclearpower plant.

from August 13, 2011 to October 9, 2015 over Fukushima,Japan, after the Great East Japan Earthquake on March 11,2011. The study scene covers the Fukushima Daiichi NuclearPower Station (37◦25′16′′, 141◦1′58′′) and its surroundingarea (see Fig. 5 left), where visiting on site is forbidden. Sincethe earthquake, storage tanks have been built to store pollutedwater and many of the abandoned areas have been covered byweeds.

All the images were first co-registered using geocoordinateinformation and further registered using phase-correlation-based image matching in case a misregistration of more thanone pixel is detected [64]. Image registration was performedonly by shifting a whole image at a pixel scale to avoidinterpolation while keeping the same spatial resolution. Weused ATCOR for atmospheric correction [65]. For the Hy-perion data, we used 156 bands (bands 8–57, 79–117, 135–165, 183–185, 188–220) after removing bands that includeonly zero values and strong water vapor absorption. For theLandsat-8 data, we used bands 1–7 for the experiment. Thedata acquisition dates and sensors are summarized in TableIII. Color composite images of a subset of the dataset afteratmospheric normalization are shown in Fig. 6(a).

B. Experimental results

In this work, we considered the Hyperion image takenon April 29, 2012 as the master image. VCA was used toextract endmembers from this image. Spectral smoothing was

performed to reduce spikes due to non-optimal atmosphericcorrection. The spectral signatures were manually labeled,resulting in nine endmembers: water, soil, two kinds of veg-etation, and five different man-made objects. Fig. 5 presentsthe spectral signatures of these nine endmembers and theirlocations in the scene. A was fixed and γ was set to 0 for tworeasons. First, temporal spectral variability in the studied time-series dataset was greatly affected by atmospheric conditionssuch as clouds. By fixing A, we prevented convergence ofendmembers to spectral signatures significantly affected byclouds. Second, temporal spectral variability of surface objectswas well represented by using the extracted endmembers whenour analysis is focused on discriminating coarse categories(e.g., water, soil, vegetation, and man-made objects). α was setto 0.05 as the average value of estimated αs for all the imagesusing the criterion given in (4). We set β = 1 based on theparameter sensitivity analysis in Section III-B. The number ofneighbors was set to two.

Figs. 6(b)(c) show the nine temporal abundance mapsobtained by individual spectral unmixing and the pro-posed methodology with manifold-based coupling, respec-tively. Here, the nine endmembers are classified into fourcategories, i.e., water, soil, vegetation, and man-made. Theproposed method shows more stable results when comparedto individual spectral unmixing. For example, in Fig. 6(b),the abundances of man-made objects are highly influencedby thin clouds (May 6, 2013) and waves (August 3, 2014).These effects are majorly mitigated by the proposed method,as shown in the corresponding abundance maps of Fig. 6(c),where seasonal changes of vegetation and soil in abundancesare clearly exhibited. Although individual spectral unmixingfailed to estimate the abundances of the Landsat-8 data settaken on March 26, 2014, the coupled spectral unmixingmethod provided stable results. In Fig. 6(b), large abundanceerrors corresponding to water false positive estimations appearin the land area as a result of the strict sum-to-one constraint.

Fig. 7 gives an example of class-specific change detectionusing the unmixing results obtained from the four temporalHyperion images. Figs. 7(a)(b) depict the results obtained byindividual spectral unmixing and coupled spectral unmixing,respectively. The abundance changes on May 6, 2013, May25, 2014, and May 2, 2015 (as compared to April 29, 2012)are presented in the grayscale images from the first row to thefourth row for man-made objects, vegetation, water, and soil,respectively. The images in the last row are color compositeimages using abundance difference maps of man-made objects,vegetation, and soil for red, green, and blue, respectively, tovisualize class-specific changes. For example, magenta, green,orange, and purple indicate the changes from vegetation toman-made, from soil to vegetation, from soil to man-made,and from vegetation to soil, respectively.

In Fig. 7(a), it can be seen that there are too many overallabundance changes obtained by individual spectral unmixing.In contrast, Fig. 7(b) shows that the proposed method showsstable and reasonable results for the abundance changes. Forexample, in Fig. 7(a), it can be seen that the estimated abun-dances of vegetation change in evergreen forests, which arealways green in the color images of Fig. 6(a) (despite the fact


May 25, 2014August 13, 2011 April 29, 2012 May 6, 2013 August 16, 2013 March 28, 2014 August 3, 2014 May 2, 2015 August 6, 2015

0

1

(c)

(b)

(a)M

an-m

ade

Veg

etat

ion

Wat

erS

oil

Man

-mad

eV

eget

atio

nW

ater

Soi

l

Fig. 6. (a) Color composite images. Abundance maps of man-made, vegetation, water, and soil obtained by (b) individual spectral unmixing and (c) coupledspectral unmixing.

that the actual abundance values do not significantly change).The proposed method shows stable results in evergreen forests,resulting in no change, and a gradual increase of vegetationcan be visualized in areas (e.g., abandoned farmland) coveredby deciduous plants (e.g., weeds), as illustrated in Fig. 7(b).On the other hand, the coupled spectral unmixing algorithmshows a gradual decrease of soil in these areas, whereas theabundance changes of soil obtained by individual spectralunmixing are hard to interpret. These results indicate thatindividual spectral unmixing results in unstable estimation of

abundances due to spectral variability, including effects of non-optimal atmospheric correction. They further indicate that thecoupled spectral unmixing method is able to mitigate thisinstability by adding regularization on abundances, so thatsimilar pixels in neighborhood images in the manifold of time-series data have similar abundances. In the color compositesrepresenting abundance changes obtained by the proposedmethod, trees that have been cut down to increase the numberof water storage tanks in and around the power plant areclearly visualized as magenta and purple pixels. The increase


TABLE IIIDATA ACQUISITION DATES AND SENSORS

2011 2013 2014 2015Date Sensor Date Sensor Date Sensor Date Sensor

August 13 Hyperion January 29 Hyperion January 7 Landsat-8 February 27 Landsat-8December 18 Hyperion February 24 Hyperion January 23 Landsat-8 March 15 Landsat-8

May 6 Hyperion February 24 Landsat-8 March 31 Landsat-8August 16 Landsat-8 March 28 Landsat-8 May 2 Hyperion

September 1 Landsat-8 April 13 Landsat-8 May 18 Landsat-8September 17 Landsat-8 May 25 Hyperion July 9 Hyperion

2012 October 27 Hyperion May 31 Landsat-8 July 21 Landsat-8Date Sensor November 20 Landsat-8 June 16 Landsat-8 August 6 Landsat-8

April 29 Hyperion July 2 Landsat-8 October 9 Landsat-8July 12 Hyperion

August 3 Landsat-8December 25 Landsat-8

May 6, 2013 May 25, 2014 May 5, 2015 May 6, 2013 May 25, 2014 May 5, 2015

(b)(a)

-0.5

0.5

-0.5

0.5

Man

-mad

eV

eget

atio

nW

ater

Soi

l

Man

-mad

eV

eget

atio

nW

ater

Soi

l

Fig. 7. Abundance changes on May 6, 2013, May 25, 2014, and May 2, 2015 compared to April 29, 2012 obtained by (a) individual spectral unmixingand (b) coupled spectral unmixing. Grayscale images correspond to abundance maps of man-made objects, vegetation, water, and soil in the first row to thefourth row and color composite images in the fifth row are made up of abundance difference maps of man-made, vegetation, and soil for red, green, and blue,respectively.

of vegetation may be attributed to the growth of weeds owingto seasonal differences, as well as the yearly expansion ofweeds in the abandoned farmland.

Fig. 8 presents time-series plots of abundances of fourcategories obtained from all the 33 Hyperion and Landsat-8 images using four example pixels (see Figs. 8(a)–(d)) and

those of average values in the study area (see Fig. 8(e)).Four temporal high-resolution images at each point (30×30m) obtained from Google Earth are also shown in Figs. 8(a)–(d). In Fig. 8(a), seasonal changes of abundances betweenvegetation and soil show the existence of deciduous plants.Fig. 8(b) shows the high abundance of vegetation throughout


Dec 10 May 12 Sep 13 Feb 15 Jun 160

0.2

0.4

0.6

0.8

1A

bund

ance


0.2

0.4

0.6

0.8

1

Abu

ndan

ce


0.2

0.4

0.6

0.8

1

Abu

ndan

ce


0.2

0.4

0.6

0.8

1

Abu

ndan

ce

Man-madeVegetationWaterSoil


0.2

0.4

0.6

0.8

Abu

ndan

ce

(a) (b)

(c) (d)

(e)

Mar 24, 14 Jun 1, 15Mar 12, 13Jan 31, 12

Mar 24, 14 Jun 1, 15Mar 12, 13May 18, 11

Mar 24, 14 Jun 1, 15Mar 12, 13May 18, 11

Mar 28, 14 Jun 1, 15Mar 12, 13May 18, 11

Fig. 8. Time-series plots of abundances for four categories obtained from 33 Hyperion and Landsat-8 images using (a)–(d) four example points and those of(e) average values in the study area. Four temporal high-resolution images at each point obtained from Google Earth are shown in (a)–(d).

the year, which demonstrates that the pixel is covered byevergreens. In Figs. 8(c)(d), we can clearly observe the effectsof deforestation and construction of man-made objects. Inparticular, Fig. 8(d) depicts the change of the surface fromvegetation to soil to man-made objects. All the high-resolutionimages are consistent with the time-series plots of abundances,indicating the validity of the results of MuCSUn. In Fig. 8(e),we can find seasonal changes in abundance between vegetationand soil and relatively stable abundances of water and man-made objects. Minimal abundances of vegetation graduallyincrease during the four years, which implies the increase ofweeds in the abandoned areas. This analysis of time-seriesplots of abundances has the potential to improve understandingof dynamic changes on the surface.

V. CONCLUSIONS AND FUTURE LINES

We proposed a framework for spectral unmixing of mul-tisensor time-series spectral data. The proposed framework,called MuCSUn (multisensor coupled spectral unmixing),provides robust and stable unmixing solutions beyond datamodalities due to the different spectral characteristics of im-agers and the effects of non-optimal atmospheric correction. Asingle unmixing problem is jointly solved with other unmixingproblems in its neighborhood in the manifold of the time-series dataset. Multiple unmixing problems are coupled viaregularization on graphs between the local images. We alsointroduced methods for atmospheric normalization and cross-calibration of SRFs. Quantitative validation was performedwith synthetic time-series data composed of hyperspectral and

multispectral images, including trend and seasonal changesand residual gains. The experiment demonstrated the effi-cacy of the proposed methodology. Atmospheric normalizationshowed large improvements in the accuracy of spectral unmix-ing, which suggests its importance as preprocessing in spectralunmixing of multisensor time-series spectral data. The cou-pled spectral unmixing algorithm also improved the unmixingaccuracy compared to individual spectral unmixing. Spectralunmixing problems involving multispectral images particularlybenefitted from the use of multisensor data in the context ofour proposed methodology. Our methodology was applied toa real dataset composed of 11 Hyperion and 22 Landsat-8images and showed robust and stable results visualizing class-specific changes at a subpixel scale. These findings suggestthat the proposed framework can contribute to the synergeticuse of spaceborne hyperspectral and multispectral imagers forunderstanding dynamic changes on the surface.

In our future research we intend to concentrate on end-member extraction from multisensor time-series spectral datalearning temporal spectral variability. More experiments onreal data and additional applications will be explored to verifythe efficacy of MuCSUn.


APPENDIX

The objective function in (6) can be rewritten as

O(A,X,B,Z) =

1

2

K+1∑k=1

Tr((Yk − AkXTk )(Yk − AkXT

k )T ) + α

K+1∑k=1

1TP X

1/2k 1M

+1

2β

K∑p=1

K+1∑q=p+1

((G)pq(G)qp(Tr(XTp Dpq1Xp)

+ Tr(XTq Dpq2Xq)− 2Tr(XT

p WpqXq))

+ ((G)qp − (G)pq)(Tr(XTp Dpq1Xp)

+ Tr(ZTq Dpq2Zq)− 2Tr(XT

p WpqZq)))

+ γ

K∑p=1

K+1∑q=p+1

((G)pq(G)qpTr((RpqAq − Ap)(RpqAq − Ap)T )

+ ((G)qp − (G)pq)Tr((RpqBq − Ap)(RpqBq − Ap)T ))

(18)

Let (Φk)jm, (Ψk)im, (Θk)jm, and (Υk)im be the La-grange multiplier for constraint (Xk)jm ≥ 0 (nonnegativityof abundances), (Ak)im ≥ 0 (nonnegativity of endmembers),(Xk)jm = (Zk)jm (consistency of auxiliary variables Z withX), and (Ak)im = (Bk)im (consistency of auxiliary variablesB with A), respectively, the augmented Lagrangian withoutthe abundance sum-to-one constraint is given by

Lc(A,X,B,Z,Φ,Ψ,Θ,Υ) =

1

2

K+1∑k=1

Tr((Yk − AkXTk )(Yk − AkXT

k )T ) + α

K+1∑k=1

1TP X

1/2k 1M

+1

2β

K∑p=1

K+1∑q=p+1

((G)pq(G)qp(Tr(XTp Dpq1Xp)

+ Tr(XTq Dpq2Xq)− 2Tr(XT

p WpqXq))

+ ((G)qp − (G)pq)(Tr(XTp Dpq1Xp)

+ Tr(ZTq Dpq2Zq)− 2Tr(XT

p WpqZq)))

+ γK∑

p=1

K+1∑q=p+1

((G)pq(G)qpTr((RpqAq − Ap)(RpqAq − Ap)T )

+ ((G)qp − (G)pq)Tr((RpqBq − Ap)(RpqBq − Ap)T ))

−K+1∑k=1

Tr(ΦkXTk )−

K+1∑k=1

Tr(ΨkATk )

+

K+1∑k=1

Tr(Θk(Zk − Xk)T ) +

K+1∑k=1

ρ

2Tr((Zk − Xk)(Zk − Xk)

T )

+

K+1∑k=1

Tr(Υk(Bk − Ak)T ) +

K+1∑k=1

σ

2Tr((Bk − Ak)(Bk − Ak)

T )

(19)

The partial derivatives of Lc with respect to Ak, Bk, Xk �

0, and Zk are∂Lc

∂Ak=− YkXk + AkXT

k Xk − Ψk − Υk + σ(Ak − Bk)

+ γ(

k−1∑p=1

(G)pk(G)kp(RTpkRpkAk − RT

pkAp)

+

K+1∑q=k+1

((G)kq(G)qk(−RkqAq + Ak)

+ ((G)qk − (G)kq)(−RkqBq + Ak))) (20)

∂Lc

∂Bk= γ

k−1∑p=1

((G)kp − (G)pk)(RTpkRpkBk − RT

pkAp)

+ Υk + σ(Bk − Ak) (21)∂Lc

∂Xk=− YT

k Ak + XkATk Ak + αX

−1/2k − Φk − Θk

+ ρ(Xk − Zk)

+ β(

k−1∑p=1

(G)pk(G)kp(Dpk2Xk − WTpkXp)

+

K+1∑q=k+1

((G)kq(G)qk(Dkq1Xk − WkqXq)

+ ((G)qk − (G)kq)(Dkq1Xk − WkqZq))) (22)

∂Lc

∂Zk= β

k−1∑p=1

((G)kp − (G)pk)(Dpk2Zk − WTpkXp) + Θk

+ ρ(Zk − Xk) (23)

Lc is continuously differentiable under X � 0, and theKarush-Kuhn-Tucker (KKT) conditions are satisfied. Using theKKT conditions, the following equations are obtained

− (YkXk)im(Ak)im + (AkXTk Xk)im(Ak)im − (Υk)im(Ak)im

+ γ(

k−1∑p=1

(G)pk(G)kp(RTpkRpkAk − RT

pkAp)im(Ak)im

+

K+1∑q=k+1

((G)kq(G)qk(−RkqAq + Ak)im(Ak)im

+ ((G)qk − (G)kq)(−RkqBq + Ak)im(Ak)im)) = 0(24)

γ

k−1∑p=1

((G)kp − (G)pk)(RTpkRpkBk − RT

pkAp)im(Bk)im

+ (Υk)im(Bk)im + σ(Bk − Ak)im(Bk)im = 0

(25)

− (YTk Ak)jm(Xk)jm + (XkAT

k Ak)jm(Xk)jm

+ α(X−1/2k )jm(Xk)jm − (Θk)jm(Xk)jm

+ β(

k−1∑p=1

(G)pk(G)kp(Dpk2Xk − WTpkXp)jm(Xk)jm

+

K+1∑q=k+1

((G)kq(G)qk(Dkq1Xk − WkqXq)jm(Xk)jm

+ ((G)qk − (G)kq)(Dkq1Xk − WkqZq)jm(Xk)jm)) = 0(26)

β

k−1∑p=1

((G)kp − (G)pk)(Dpk2Zk − WTpkXp)jm(Zk)jm

+ (Θk)jm(Zk)jm + ρ(Zk − Xk)jm(Zk)jm = 0

(27)

These equations lead to the update rules (7)–(12).


REFERENCES

[1] L. Guanter, H. Kaufmann, K. Segl, S. Foerster, C. Rogass, S. Chabrillat,T. Kuester, A. Hollstein, G. Rossner, C. Chlebek, C. Straif, S. Fischer, S.Schrader, T. Storch, U. Heiden, A. Muller, M. Bachmann, H. Muhle, R.Muller, M. Habermeyer, A. Ohndorf, J. Hill, H. Buddenbaum, P. Hostert,S. van der Linden, P. J. Leitao, A. Rabe, R. Doerffer, H. Krasemann,H. Xi, W. Mauser, T. Hank, M. Locherer, M. Rast, K. Staenz, and B.Sang, “The EnMAP spaceborne imaging spectroscopy mission for Earthobservation,” Remote Sensing, vol. 7, pp. 8830–8857, 2015.

[2] A. Iwasaki, N. Ohgi, J. Tanii, T. Kawashima, H. Inada, “Hyperspectralimager suite (HISUI)-Japanese hyper-multi spectral radiometer,” In Proc.Int. Geosci. Remote Sens. Symp. (IGARSS), Vancouver, Canada, 24–29July 2011, pp. 1025–1028.

[3] S. Pignatti, N. Acito, U. Amato, R. Casa, D. B. Roberto, M. Diani,G. Laneve, S. Matteoli, A. Palombo, S. Pascucci, F. Romano, F.Santini, T. Simoniello, C. Ananasso, G. Corsini, and V. Cuomo, “ThePRISMA hyperspectral mission: science activities and opportunities foragriculture and land monitoring,” In Proc. Int. Geosci. Remote Sens.Symp. (IGARSS), Melbourne, Australia, 21–26 July 2013, pp. 4558–4561.

[4] R. Green, G. Asner, S. Ungar, R. Knox, “NASA mission to measureglobal plant physiology and functional types,” In Proc. AerospaceConference, Big Sky, MT, US, 1–8 March 2008, pp. 1–7.

[5] S. Michel, P. Gamet, M. J. Lefevre-Fonollosa, “HYPXIM - a hyper-spectral satellite defined for science, security and defence users,” InProc. IEEE Workshop Hyperspectral Image Signal Process.: Evolutionin Remote Sens. (WHISPERS), Lisbon, Portugal, 6–9 June 2011, 1–4.

[6] J. R. Irons, J. L. Dwyer, and J. A. Barsi, “The next Landsat satellite:the Landsat data continuity mission,” Remote Sens. Environ., vol. 122,pp. 11–21, 2012.

[7] M. Druscha, U. Del Belloa, S. Carliera, O. Colinb, V. Fernandeza, F.Gasconb, B. Hoerschb, C. Isolaa, P. Laberintia, P. Martimorta, A. Mey-gretc, F. Spotoa, O. Sya, F. Marchesed, and P. Bargellinid, “Sentinel-2:ESA’s optical high-resolution mission for GMES operational services,”Remote Sens. Environ., vol. 120, pp. 25–36, 2012.

[8] M. Frank and M. Canty, “Unsupervised change detection for hyperspec-tral images,” in Proc. 12th JPL Airborne Earth Sci. Workshop, 2003, pp.63–72.

[9] V. Ortiz-Rivera, M. Velez-Reyes, and B. Roysam, “Change detection inhyperspectral imagery using temporal principal components,” in Proc.SPIE 6233, Algorithms Technologies for Multispectral, Hyperspectral,and Ultraspectral Imagery XII, Kissimmee, FL, USA, 2006, Art. ID.623312.

[10] A. A. Nielsen, “The regularized iteratively reweighted MAD method forchange detection in multi- and hyperspectral data,” IEEE Trans. ImageProcess., vol. 16, no. 2, pp. 463–478, Feb. 2007.

[11] C. Wu, B. Du, and L. Zhang, “A subspace-based change detectionmethod for hyperspectral images,” IEEE J. Sel. Topics Appl. EarthObserv. Remote Sens., vol. 6, no. 2, pp. 815–830, Apr. 2013.

[12] W. A. Malila, “Change vector analysis: An approach for. detecting forestchanges with Landsat,” in 6th Annu. Symp. Mach. Process. RemotelySensed Data, West Lafayette, IN, USA, 1980, pp. 326–336.

[13] F. Bovolo, S. Marchesi, and L. Bruzzone, “A framework for automaticand unsupervised detection of multiple changes in multitemporal im-ages,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 6, pp. 2196–2212,Jun. 2012.

[14] S. Liu, L. Bruzzone, F. Bovolo, M. Zanetti, and P. Du, “Sequentialspectral change vector analysis for iteratively discovering and detectingmultiple changes in hyperspectral images,” IEEE Trans. Geosci. RemoteSens., vol. 53, no. 8, pp. 4363–4378, Aug. 2015.

[15] A. Schaum and A. Stocker, “Hyperspectral change detection and su-pervised matched filtering based on covariance equalization,” in Proc.SPIE, 2004, vol. 5425, pp. 77–90.

[16] J. Theiler, C. Scovel, B. Wohlberg, and B. R. Roy, “Elliptically contoureddistributions for anomalous change detection in hyperspectral imagery,”IEEE Geosci. Remote Sens. Lett., vol. 7, no. 2, pp. 271–275, Apr. 2010.

[17] J. Meola, M. T. Eismann, R. L. Moses, and J. N. Ash, “Detectingchanges in hyperspectral imagery using a model-based approach,” IEEETrans. Geosci. Remote Sens., vol. 49, no. 7, pp. 2647–2661, Jul. 2011.

[18] G. Camps-Valls, L. Gomez-Chova, J. Munoz-Marı, J. L. Rojo-Alvarez,and M. Martınez-Ramon, “Kernel-based framework for multitemporaland multisource remote sensing data classification and change detec-tion,” IEEE Trans. Geosci. Remote Sens., vol. 46, no. 6, pp. 1822–1835,2008.

[19] D. Lu, M. Batistella, and E. Moran, “Multitemporal spectral mixtureanalysis for Amazonian land-cover change detection,” Can. J. RemoteSens., vol. 30, no. 1, pp. 87–100, 2004.

[20] D. B. Lobell and G. P. Asner, “Cropland distributions from temp oralunmixing of MODIS data,” Remote Sens. Environ., vol. 93, no. 3, pp.412–422, Nov. 2004.

[21] M. A. Goenaga, M. C. Torres-Madronero, M. Velez-Reyes, S. J. VanBloem, and J. D. Chinea, “Unmixing analysis of a time series ofHyperion images over the Guanica dry forest in Puerto Rico,” IEEEJ. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 6, no. 2, pp. 329–338, Apr. 2013.

[22] B. Somers and G. P. Asner, “Multi-temporal hyperspectral mixture anal-ysis and feature selection for invasive species mapping in rainforests,”Remote Sens. Environ., vol. 136, pp. 14–27, Sep. 2013.

[23] P. Du, S. Liu, P. Liu, K. Tan, and L. Cheng, “Sub-pixel change detectionfor urban land-cover analysis via multi-temporal remote sensing images,”Geo-spatial Inf. Sci., vol. 17, no. 1, pp. 26–38, 2014.

[24] Q. Du, L. Wasson, and R. King, “Unsupervised linear unmixing forchange detection in multitemporal airborne hyperspectral imagery,” inProc. IEEE Int. Workshop MultiTemp, 2005, pp. 136–140.

[25] A. Erturk and A. Plaza, “Informative change detection by unmixing forhyperspectral images,” IEEE Geosci. Remote Sens. Lett., vol. 12, no. 6,pp. 1252–1256, Jun. 2015.

[26] A. Erturk, M.-D. Iordache, and A. Plaza, “Sparse unmixing based changedetection for multi-temporal hyperspectral images,” IEEE J. Sel. TopicsAppl. Earth Observ. Remote Sens., in press.

[27] N. Yokoya and X. X. Zhu, “Graph regularized coupled spectral unmixingfor change detection,” Proc. WHISPERS, Tokyo, Japan, June 2-5, 2015.

[28] S. Liu, L. Bruzzone, F. Bovolo, and P. Du, “Unsupervised multitemporalspectral unmixing for detecting multiple changes in hyperspectral im-ages,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 5, pp. 2733–2748,May 2016.

[29] S. Henrot, J. Chanussot, and C. Jutten, “Dynamical spectral unmixingof multitemporal hyperspectral images,” IEEE Trans. Image Process.,vol. 25, no. 7, pp. 3219–3232, July 2016.

[30] A. Singh, “Digital change detection techniques using remotely-senseddata,” Int. J. Remote Sens., vol. 10, no. 6, pp. 989–1003, 1989.

[31] C. Songh, C. E. Woodcock, K. C. Seto, M. P. Lenney, and S. A.Macomber, “Classification and change detection using Landsat TM data:when and how to correct atmospheric effects?” Remote Sens. Environ.,vol. 75, pp. 230–244, 2001.

[32] P. Coppin, I. Jonckheere, K. Nackaerts, B. Muys, and E. Lambin,“Digital change detection methods in ecosystem monitoring: a review,”Int. J. Remote Sens., vol. 25, no. 9, pp. 1565–1596.

[33] N. Keshava and J. F. Mustard, “Spectral unmixing,” IEEE SignalProcess. Mag., vol. 19, no. 1, pp. 44–57, 2002.

[34] J. M. Bioucas-Dias, A. Plaza, N. Dobigeon, M. Parente, Q. Du, P. Gader,and J. Chanussot, “Hyperspectral unmixing overview: geometrical, sta-tistical, and sparse regression-based approaches,” IEEE J. Sel. TopicsAppl. Earth Observ., vol. 5, no. 2, pp. 354–379, 2012.

[35] D. C. Heinz and C.-I. Chang, “Fully constrained least squares linearspectral mixture analysis method for material quantification in hyper-spectral imagery,” IEEE Trans. Geosci. Remote Sens., vol. 39, no. 3, pp.529–545, 2001.

[36] D. A. Roberts, M. Gadner, R. Church, S. Ustin, G. Scheer, O. R. Green,“Mapping chaparral in the Santa Monica mountains using multipleendmember spectral mixture models,” Remote Sens. Environ., vol. 65,pp. 267–279, 1998.

[37] M. D. Iordache, J. Bioucas-Dias, and A. Plaza, “Sparse unmixing ofhyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 6,pp. 2014–2039, 2011.

[38] M. D. Iordache, “A sparse regression approach to Hyperspectral unmix-ing,” Ph.D. dissertation, School of Elect. Comput. Eng., IST, 2011.

[39] Y. Zhong, R. Feng, and L. Zhang, “Non-local sparse unmixing forhyperspectral remote sensing imagery,” IEEE J. Sel. Topics Appl. EarthObserv. Remote Sens., vol. 7, no. 6, pp. 1889-1909, Jun. 2014.

[40] R. Feng, Y. Zhong, and L. Zhang, “Adaptive non-local Euclidean medi-ans sparse unmixing for hyperspectral imagery,” ISPRS J. Photogram.Rem. Sens., vol. 97, pp. 9-24, Nov. 2014.

[41] J. Bieniarz, E. Aguilera, X. X. Zhu, R. Mueller, P. Reinartz, “Jointsparsity model for multilook hyperspectral image unmixing,” IEEEGeosci. Remote Sens. Lett., vol. 12, no. 4, pp. 696–700, 2014.

[42] J. W. Boardman, “Geometric mixture analysis of imaging spectrometrydata,” in Proc. IEEE IGARSS, Pasadena, CA, 1994, vol. 4, pp. 2369–2371.


[43] M. E. Winter, “N-FINDR: An algorithm for fast autonomous spectralend-member determination in hyperspectral data,” in Proc. SPIE, 1999,vol. 3753, pp. 266–275.

[44] J. M. Nascimento and J. M. Bioucas-Dias, “Vertex component analysis:A fast algorithm to unmix hyperspectral data,” IEEE Trans. Geosci.Remote Sens., vol. 43, no. 4, pp. 898–910, Apr. 2005.

[45] J. Li and J. Bioucas-Dias, “Minimum volume simplex analysis: A fastalgorithm to unmix hyperspectral data,” in Proc. IEEE Int. Conf. Geosci.Remote Sens. (IGARSS), 2008, vol. 3, pp. 250–253.

[46] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegativematrix factorization,” Nature, vol. 401, pp. 788–791, 1999.

[47] D. Cai, X. He, J. Han, and T. S. Huang, “Graph regularized non-negativematrix factorization for data representation,” IEEE Trans. Pattern Anal.Mach. Intell., vol. 33, no. 8, pp. 1548–1560, 2010.

[48] L. Miao and H. Qi, “Endmember extraction from highly mixed datausing minimum volume constrained nonnegative matrix factorization,”IEEE Trans. Geosci. Remote Sens., vol. 45, no. 3, pp. 765–777, Mar.2007.

[49] S. Jia and Y. Qian, “Constrained nonnegative matrix factorization forhyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 47,no. 1, pp. 161–173, Jan. 2009.

[50] A. Huck, M. Guillaume, and J. Blanc-Talon, “Minimum dispersionconstrained nonnegative matrix factorization to unmix hyperspectraldata,” IEEE Trans. Geosci. Remote Sens., vol. 48, no. 6, pp. 2590–2602,Jun. 2010.

[51] X. Liu, W. Xia, B. Wang, and L. Zhang, “An approach based onconstrained nonnegative matrix factorization to unmix hyperspectraldata,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 2, pp. 757–772,Feb. 2011.

[52] Y. Qian, S. Jia, J. Zhou, and A. Robles-Kelly, “Hyperspectral unmixingvia l1/2 sparsity-constrained nonnegative matrix factorization,” IEEETrans. Geosci. Remote Sens., vol. 49, no. 11, pp. 4282–4297, 2011.

[53] X. Lu , H. Wu , Y. Yuan , P. Yan and X. Li , “Manifold regularizedsparse NMF for hyperspectral unmixing,” IEEE Trans. Geosci. RemoteSens., vol. 51, no. 5 , pp. 2815–2826, 2013.

[54] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrixfactorization unmixing for hyperspectral and multispectral data fusion,”IEEE Trans. Geosci. Remote Sens., vol. 50, no. 2, pp. 528–537, 2012.

[55] P. O. Hoyer, “Non-negative matrix factorization with sparseness con-straints,” J. Mach. Learn. Res., vol. 5, pp. 14571469, Nov. 2004.

[56] S. L. Waslander, G. Inalhan and C. J. Tomlin, Decentralized Optimiza-tion via Nash Bargaining, 2004, Kluwer Academic Press.

[57] V. Caselles, and M. J. Lopea Garcıa, “An alternative simple approach toestimate atmospheric correction in multitemporal studies,” Int. J. RemoteSens., vol. 10, no. 6, pp. 1127–1134, 1989.

[58] F. G. Hall, D. E. Strebel, J. E. Nickeson, and S. J. Goetz, “Radiometricrectification: toward a common radiometric response among multi-date,multi-sensor images,” Remote Sens. Environ., vol. 35, pp. 11–27, 1991.

[59] N. Yokoya, N. Mayumi, and A. Iwasaki, “Cross-calibration for datafusion of EO-1/Hyperion and Terra/ASTER,” IEEE J. Sel. Topics Appl.Earth Observ. Remote Sens., vol. 6, no. 2, pp. 419-426, 2013.

[60] T. Wang, G. Yan, H. Ren, and X. Mu, “Improved methods for spectralcalibration of on-orbit imaging spectrometers,” IEEE Trans. Geosci.Remote Sens., vol. 48, no. 11, pp. 3924–3931, 2010.

[61] G. D. Finlayson, S. Hordley, and P. M. Hubel, “Recovering devicesensitivities with quadratic programming,” In IS and T/SID Sixth ColorImaging Conference: Color Science, Systems, and Applications, pp. 90–95. IS and T/SID, November 1998.

[62] J. Plaza, E. M. T. Hendrix, I Garcıa, G. Martın, A. Plaza, “Onendmember identification in hyperspectral images without pure pixels: Acomparison of algorithms,” J. Math. Imaging Vis., vol. 42, pp. 163–175,2012.

[63] D. P. Bertsekas, Constrained Optimization and Lagrange MultiplierMethods (Optimization and Neural Computation Series), Athena Sci-entific, 1996.

[64] S. Nagashima, T. Aoki, T. Higuchi and K. Kobayashi, “A subpixelimage matching technique using phase-only correlation,h in Proceedingsof 2006 International Symposium on Intelligent Signal Processing andCommunication Systems (IEEE, 2006), pp. 701-704.

[65] R. Richter and D. Schlapfer, Atmospheric/Topographic Correction forSatellite Imagery: Atcor 2/3 Users Guide , 2015, German AerospaceCenter (DLR).

Naoto Yokoya (S’10–M’13) received the M.Sc. and Ph.D. degrees inaerospace engineering from the University of Tokyo, Tokyo, Japan, in 2010and 2013, respectively.

From 2012 to 2013, he was a Research Fellow with Japan Society forthe Promotion of Science, Tokyo, Japan. Since 2013, he is an AssistantProfessor with the University of Tokyo. Since 2015, he is also an Alexandervon Humboldt Research Fellow with the German Aerospace Center (DLR),Oberpfaffenhofen, and Technical University of Munich (TUM), Munich,Germany. His research interests include image analysis and data fusion inremote sensing.

Xiao Xiang Zhu (S’10–M’12–SM’14) received the Bachelors degree in spaceengineering from the National University of Defense Technology, Changsha,China, in 2006 and the M.Sc. degree, the Dr.-Ing. degree, and the Habilitationin the field of signal processing from the Technical University of Munich(TUM), Munich, Germany, in 2008, 2011, and 2013, respectively.

Since 2011, she has been a Scientist with the Remote Sensing TechnologyInstitute, German Aerospace Center (DLR), Oberpfaffenhofen, Germany,where she is the Head of the Team Signal Analysis. Since 2013, she hasalso been a Helmholtz Young Investigator Group Leader and appointed asTUM Junior Fellow. In 2015, she was appointed as a Professor with SignalProcessing in Earth Observation, TUM. She was a Guest Scientist or VisitingProfessor with the Italian National Research Council (CNR-IREA), Naples,Italy, Fudan University, Shanghai, China, University of Tokyo, Tokyo, Japan,and University of California, Los Angeles, CA, USA, in 2009, 2014, 2015, and2016, respectively. Her main research interests are the following: advancedInSAR techniques such as high-dimensional tomographic SAR imaging andSqueeSAR, computer vision in remote sensing including object reconstructionand multidimensional data visualization, big data analysis in remote sensing,and modern signal processing, including innovative algorithms such as sparsereconstruction, nonlocal means filter, robust estimation, and deep learning,with applications in the field of remote sensing such as multi-/hyperspectralimage analysis.

Dr. Zhu is an Associate Editor of the IEEE Transactions on Geoscienceand Remote Sensing.

Antonio Plaza (M’05–SM’07–F’15) is the Head of the HyperspectralComputing Laboratory at the Department of Technology of Computers andCommunications, University of Extremadura, where he received the M.Sc.degree in 1999 and the PhD degree in 2002, both in Computer Engineering.His main research interests comprise hyperspectral data processing andparallel computing of remote sensing data. He has authored more than 500publications, including 193 JCR journal papers (over 140 in IEEE journals),23 book chapters, and 285 peer-reviewed conference proceeding papers. Hehas guest edited 10 special issues on hyperspectral remote sensing for differentjournals.

Dr. Plaza is a Fellow of IEEE “for contributions to hyperspectral dataprocessing and parallel computing of Earth observation data.” He is a recipientof the recognition of Best Reviewers of the IEEE Geoscience and RemoteSensing Letters (in 2009) and a recipient of the recognition of Best Reviewersof the IEEE Transactions on Geoscience and Remote Sensing (in 2010), forwhich he served as Associate Editor in 2007–2012. He is also an AssociateEditor for IEEE Access, and was a member of the Editorial Board of theIEEE Geoscience and Remote Sensing Newsletter (2011–2012) and the IEEEGeoscience and Remote Sensing Magazine (2013). He was also a memberof the steering committee of the IEEE Journal of Selected Topics in AppliedEarth Observations and Remote Sensing (JSTARS). He is a recipient of theBest Column Award of the IEEE Signal Processing Magazine in 2015, the2013 Best Paper Award of the JSTARS journal, and the most highly citedpaper (2005–2010) in the Journal of Parallel and Distributed Computing.He received best paper awards at the IEEE International Conference onSpace Technology and the IEEE Symposium on Signal Processing andInformation Technology. He served as the Director of Education Activitiesfor the IEEE Geoscience and Remote Sensing Society (GRSS) in 2011-2012,and is currently serving as President of the Spanish Chapter of IEEE GRSS.He has reviewed more than 500 manuscripts for over 50 different journals.He is currently serving as the Editor-in-Chief of the IEEE Transactions onGeoscience and Remote Sensing journal.

ieee transactions on geoscience and …naotoyokoya.com/assets/pdf/nyokoyatgrs2017.pdf · ieee...

Documents