ieee transactions on signal processing 1 off-grid doa

IEEE TRANSACTIONS ON SIGNAL PROCESSING 1

Off-Grid DOA Estimation Using Sparse BayesianLearning in MIMO Radar With Unknown Mutual

CouplingPeng Chen, Member, IEEE, Zhenxin Cao, Member, IEEE, Zhimin Chen, Member, IEEE,

Xianbin Wang, Fellow, IEEE

Abstract—In the practical radar with multiple antennas, theantenna imperfections degrade the system performance. In thispaper, the problem of estimating the direction of arrival (DOA) inmultiple-input and multiple-output (MIMO) radar system withunknown mutual coupling effect between antennas is investi-gated. To exploit the target sparsity in the spatial domain, thecompressed sensing (CS)-based methods have been proposed bydiscretizing the detection area and formulating the dictionarymatrix, so an off-grid gap is caused by the discretization pro-cesses. In this paper, different from the present DOA estimationmethods, both the off-grid gap due to the sparse sampling and theunknown mutual coupling effect between antennas are consideredat the same time, and a novel sparse system model for DOAestimation is formulated. Then, a novel sparse Bayesian learning(SBL)-based method named sparse Bayesian learning with themutual coupling (SBLMC) is proposed, where an expectation-maximum (EM)-based method is established to estimate all theunknown parameters including the noise variance, the mutualcoupling vectors, the off-grid vector and the variance vector ofscattering coefficients. Additionally, the prior distributions for allthe unknown parameters are theoretically derived. With regardto the DOA estimation performance, the proposed SBLMCmethod can outperform state-of-the-art methods in the MIMOradar with unknown mutual coupling effect, while keeping theacceptable computational complexity.

Index Terms—Compressed sensing, DOA estimation, MIMOradar, sparse Bayesian learning, mutual coupling.

I. INTRODUCTION

UNLIKE the traditional phased-array radar, multiple-inputand multiple-output (MIMO) radar systems can transmit

correlated or uncorrelated signals and improve the degree offreedom, so the recent advancement of radar technology hasdirectly led to MIMO radar systems. Usually, the MIMO radarsystems can be classified into the colocated radar and dis-tributed radar. In the colocated MIMO radar, the space betweenantennas is comparable with the wavelength of transmittedsignals [1]–[3], such that the waveform diversity can be used

This work was supported in part by the National Natural Science Foundationof China (Grant No. 61801112, 61471117, 61601281), the Natural ScienceFoundation of Jiangsu Province (Grant No. BK20180357), the Open Programof State Key Laboratory of Millimeter Waves at Southeast University (GrantNo. Z201804). (Corresponding author: Peng Chen)

P. Chen and Z. Cao are with the State Key Laboratory of MillimeterWaves, Southeast University, Nanjing 210096, China (email: chenpengseu,[email protected]).

Z. Chen is with the School of Electronic and Information, Shanghai DianjiUniversity, Shanghai 201306, China (email: [email protected]).

X. Wang is with the Department of Electrical and Computer Engineering,Western University, Canada (e-mail: [email protected]).

to improve the target estimation performance. In the distributedMIMO radar, the distances between antennas are significant,so the spatial diversity of target’s radar cross section (RCS)provided by the different view-angles of antennas can be usedto improve the target detection performance [4,5]. In general,the operation of distributed MIMO radar could be challengingdue to the coordination and signal exchange among differentantennas. Therefore, in this paper, a colocated MIMO radarsystem is investigated to estimate the directions of arrival(DOAs) for multiple targets.

Traditionally, the DOA estimation can be achieved based onthe discrete Fourier transform (DFT) of the received signal inthe spatial domain [6], but the resolution of such technique istoo low to estimate multiple targets using one beam. The max-imum likelihood-based and the subspace-based methods havebeen proposed to improve the DOA estimation performance,including multiple signal classification (MUSIC) method [7,8],Root-MUSIC method [9], and estimating signal parametersvia rotational invariance techniques (ESPRIT) method [10].Additionally, the beamspace-based methods have also beenproposed for DOA estimation [11]. For example, a beamspacedesign method is proposed in [12] for the DOA estimation inthe MIMO radar with colocated antennas; a two-dimensionaljoint transmit array interpolation and beamspace design forplanar array mono-static MIMO radar is proposed in [13] forDOA estimation via tensor modeling; a transmit beamspaceenergy focusing method is proposed in [14] for MIMO radarwith application to direction finding. Moreover, a combinedCapon and approximate maximum likelihood (CAML) methodis proposed in [15] for the estimation of target locations andamplitudes in MIMO radar. The tensor algebra and multidi-mensional harmonic retrieval are investigated for the DOAestimation of MIMO radar [16], and an iterative adaptiveKronecker beamformer for MIMO radar is proposed in [17].However, in the subspace-based DOA estimation methods,only the power of received signals from targets are exploitedto establish the target and noise subspaces.

To exploit the target sparsity in the spatial domain, com-pressed sensing (CS)-based methods are utilized to estimateDOA [18]–[25]. For example, in [11], an iterative adaptiveapproach (IAA), maximum likelihood-based IAA (IAA-ML)and multi-snapshot sparse Bayesian learning (M-SBL) aregiven for the beamforming design based on the sparsity. TheBayesian approach together with expectation-maximization(EM) is used in M-SBL to realize the user parameter-free

arX

iv:1

804.

0446

0v3

[ee

ss.S

P] 7

Dec

201

8


method. In [26], both the SBL and the relevance vectormachine (RVM) are proposed, and the sparse reconstructiontheory based on SBL is developed. In [27], Bayesian com-pressive sensing (BCS) is developed for the sparse signalreconstruction with the CS measurements. In the CS-basedmethod, the DOA estimation performance can be improvedby the dense sampling grids. However, both the computationalcomplexity and the mutual coherence between the columnsin the dictionary are increased by the dense sampling grids.To improve the DOA estimation performance without thedense sampling grids, the off-grid DOA estimation methodis proposed in [28]. To further improve the sparse estimationperformance, an off-grid sparse Bayesian inference (OGSBI)method is first proposed in [29] for the DOA estimation. Then,by solving a specific polynomial in the off-grid DOA esti-mation problem, a Root-SBL method with low computationalcomplexity is proposed in [30]. In [31], the perturbed SBL-based algorithm is proposed for the DOA estimation. A dic-tionary learning algorithm for off-grid sparse reconstruction isproposed in [32]. In [33], a grid evolution method is proposedto refine the grids for the SBL-based DOA estimation.

However, in the practical MIMO radar system, the mutualcoupling effect between antennas cannot be ignored [34,35].Therefore, the DOA estimation methods with the unknownmutual coupling effect have been proposed [36]–[38]. Usu-ally, the mutual coupling effects among the antennas canbe characterized by a mutual coupling matrix, which is asymmetric Toeplitz matrix [39]–[41]. However, in the presentpapers, the effects of both off-grid in the CS-based methodand the mutual coupling among antennas have not beenconsidered simultaneously, especially, for the methods basedon the Bayesian theory.

In this paper, the DOA estimation problem in the MIMOradar system with unknown mutual coupling effect is inves-tigated. Different from the present methods, a novel sparse-based system model considering both the off-grid gap andthe unknown mutual coupling effect is formulated. Then, anovel estimation method named SBL with the mutual cou-pling (SBLMC) is proposed, where an EM-based method isestablished to iteratively estimate all the unknown parametersincluding the noise variance, the mutual coupling vectors, theoff-grid vector and the variance vector of scattering coeffi-cients. Additionally, we theoretically derive the prior distri-butions for all the unknown parameters including the targetscattering coefficients, the mutual coupling vectors, the off-grid vector and the noise variance. Then, the proposed SBLMCis compared with state-of-the-art methods. To summarize, wemake the contributions as follows:• The Sparse-based model for MIMO radar with un-

known mutual coupling effect: Considering both theoff-grid effect and mutual coupling effect, a novel systemmodel of MIMO radar is formulated by exploiting thetarget sparsity in the spatial domain, so the DOA esti-mation problem is converted into a sparse reconstructionproblem.

• The SBL-based method for DOA estimation withunknown mutual coupling effect: A novel SBL-basedmethod (SBLMC) is proposed for the DOA estimation in

Targets

Colocated ΜΙΜΟ radar system

!dT

θκ

!

θκ

dR

Transmitter Receiver

Fig. 1. The MIMO radar system for DOA estimation.

the MIMO radar system with unknown mutual couplingeffect and off-grid effect. By estimating all the unknownparameters iteratively, the better estimation performancecan be achieved than state-of-the-art methods.

• The theoretical expressions for all unknown parame-ters in the SBL-based method: In the proposed SBL-based method (SBLMC), the estimation expressions forall unknown parameters including the noise variance, themutual coupling vectors, the variance vector of scatteringcoefficients and the off-grid vector are all theoreticallyderived.

The remainder of this paper is organized as follows. TheMIMO radar model for DOA estimation is elaborated in Sec-tion II. The proposed DOA estimation method with unknownmutual coupling, i.e., SBLMC, is presented in Section III.Section IV gives the simulation results. Finally, Section Vconcludes the paper.

Notations: Matrices are denoted by capital letters in bold-face (e.g., A), and vectors are denoted by lowercase lettersin boldface (e.g., a). IN denotes an N × N identity matrix.E · denotes the expectation operation. CN (a,B) denotesthe complex Gaussian distribution with the mean being aand the variance matrix being B. ‖ · ‖F , ‖ · ‖2, ⊗, Tr ·,vec ·, (·)∗, (·)T and (·)H denote the Frobenius norm, the`2 norm, the Kronecker product, the trace of a matrix, thevectorization of a matrix, the conjugate, the matrix transposeand the Hermitian transpose, respectively. CM×N denotes theset of M×N matrices with the entries being complex numbers.Ra denotes the real part of complex value a. For a vectora, [a]n denotes the n-th entry of a, and diaga denotes adiagonal matrix with the diagonal entries from a. For a matrixA, [A]n denotes the n-th column of A, and diagA denotesa vector with the entries from the diagonal entries of A.

II. MIMO RADAR MODEL FOR DOA ESTIMATION

As shown in Fig. 1, we consider a colocated MIMO radarsystem, where M transmitting antennas and N receiving an-tennas are adopted. In the MIMO radar system, the orthogonalsignals are transmitted by the antennas, and the waveform inthe m-th (m = 0, 1, . . . ,M−1) transmitting antenna is sm(t).Assuming that K far-field point targets in the same range cellare detected, we will consider the DOA estimation problemfor these targets. The angle of the k-th (k = 0, 1, . . . ,K − 1)


target is denoted as θk. Therefore, under the assumptionof narrowband signals, the received signals during the p-th(p = 0, 1, . . . , P − 1, and P denotes the number of pulses)pulse can be expressed as

yp(t) =

K−1∑k=0

γk,pCRb(θk) [CTa(θk)]Ts(t− τT − τR) + vp(t)

(τT + τR ≤ t ≤ τT + τR + TP), (1)

where TP denotes the pulse duration, vp(t) ,[vp,0(t), vp,1(t), . . . , vp,N−1(t)

]Tdenotes the additive white

Gaussian noise (AWGN), and γk,p denotes the scatteringcoefficient of the k-th target during the p-th pulse. τT denotesthe signal propagation time delay between the transmitter andthe range cell, and τR denotes the delay between the rangecell and the receiver. The received signals and transmittedsignals are respectively defined as

yp(t) ,[y0(t), y1(t), . . . , yN−1(t)

]T, (2)

s(t) ,[s0(t), s1(t), . . . , sM−1(t)

]T. (3)

The steering vectors of the transmitter and receiver are respec-tively denoted as

a(θ) ,[1, ej2π

dTλ sin θ, . . . , ej2π

(M−1)dTλ sin θ

]T, (4)

b(θ) ,[1, ej2π

dRλ sin θ, . . . , ej2π

(N−1)dRλ sin θ

]T, (5)

where dT and dR denote the distance separation betweentwo neighboring antennas in the transmitter and receiver,respectively, and λ denotes the wavelength. CT ∈ CM×Mand CR ∈ CN×N denotes the mutual coupling matrices in thetransmitter and receiver, respectively. The mutual coupling ma-trix CT is a symmetric Toeplitz matrix, and can be expressedas [41]

CT =

1 cT,1 . . . cT,M−1cT,1 1 . . . cT,M−2

......

. . ....

cT,M−1 . . . cT,1 1

, (6)

where cT,m denotes the m-th entry of a vector cT ,[cT,0, cT,1, . . . , cT,M−1

]T. Alternatively, the entry of CT at the

m-th row and m′-th column can be also written as

CT,m,m′ =

1, m = m′

cT,|m−m′|, otherwise. (7)

Using the same method, we can obtain the expression of CR.Since the orthogonal signals are adopted in the transmitting

antennas, we can use M matched filters corresponding to theM orthogonal signals to distinguish the orthogonal signals. Weestimate the parameters of targets at a specific range cell, sothe delays τT and τR are omitted. Therefore, passing the m-thmatched filter (designed for the m-th signal) [1], the received

signals rp(t) from the same range cell are sampled at TP andobtained as

rp,m ,K−1∑k=0

γk,pCRb(θk) [CTa(θk)]T

∫s0(t)sH

m(t)dt...∫

sm(t)sHm(t)dt

...∫sM−1(t)sH

m(t)dt

︸︷︷︸

eMm

+

∫vp,0(t)sH

m(t)dt...∫

vp,m(t)sHm(t)dt

...∫vp,M−1(t)sH

m(t)dt

︸︷︷︸

np,m

(8)

=K−1∑k=0

γk,pCRb(θk) [CTa(θk)]TeMm + np,m

=

K−1∑k=0

γk,p [CTa(θk)]mCRb(θk) + np,m,

where eMm is a M × 1 vector with the m-th entry being 1and others entries being zeros, and np,m is the additive noise.Collect rp,m into a matrix, and we can obtain

Rp ,

rTp,0

rTp,1...

rTp,M−1

=

K−1∑k=0

γk,pCTa(θk) [CRb(θk)]T

+Np

(9)

where the noise matrix is defined as Np ,[np,0,np,1, . . . ,np,M−1

]T. Vectorizing the receiving signal

matrix into a vector rp , vec Rp, we can obtain

rp =

K−1∑k=0

γk,p vecCTa(θk) [CRb(θk)]

T

+ np (10)

=

K−1∑k=0

γk,p [CRb(θk)]⊗ [CTa(θk)] + np,

where np , vec Np.Alternatively, the received signal rp can be also rewritten

into a matrix form

rp = ∆γp + np, (11)

where γp ,[γp,0, γp,1, . . . , γp,K−1

]T, ∆ ,[

δ0, δ1, . . . , δK−1], and

δk , [CRb(θk)]⊗ [CTa(θk)] (12)= [CR ⊗CT] [b(θk)⊗ a(θk)] .

By defining C , CR ⊗CT and d(θk) = b(θk) ⊗ a(θk), wehave

∆ = C[d(θ0),d(θ1), . . . ,d(θK−1)

]= CD, (13)


where D ,[d(θ0),d(θ1), . . . ,d(θK−1)

]. Therefore, the re-

ceived signal with mutual coupling effect can be formulatedby the following model

rp = CDγp + np. (14)

To simplify the formula with the mutual coupling matrix in(14), we will use the following lemma:

Lemma 1. For complex symmetric Toeplitz matrix A =Toeplitz a ∈ CM×M and complex vector c ∈ CM×1, wehave [41]–[43]

Ac = Qa, (15)

where a is a vector formed by the first row of A, and Q =Q1 + Q2 with the p-th (p = 0, 1, . . . ,M − 1) row and q-th(q = 0, 1, . . . ,M − 1) column entries being

[Q1]p,q =

cp+q, p+ q ≤M − 1

0, otherwise, (16)

[Q2]p,q =

cp−q, p ≥ q ≥ 1

0, otherwise. (17)

Based on Lemma 1, δk can be rewritten as

δk = [CRb(θk)]⊗ [CTa(θk)] (18)= [Qb(θk)cR]⊗ [Qa(θk)cT]

= [Qb(θk)⊗Qa(θk)] c,

where c , cR ⊗ cT, and the m-th entry of cT and the n-thentry of cR respectively are

[cT]m =

1, m = 0

cT,m, otherwise, (19)

[cR]n =

1, n = 0

cR,n, otherwise. (20)

Qa(θk) and Qb(θk) can be obtained as

Qa(θk) = Qa1(θk) +Qa2(θk), (21)Qb(θk) = Qb1(θk) +Qb2(θk), (22)

where the p-th row and q-th column entries of Qa1(θk),Qa2(θk), Qb1(θk) and Qb2(θk) respectively are

[Qa1]p,q =

[a(θk)]p+q, p+ q ≤M − 1

0, otherwise, (23)

[Qa2]p,q =

[a(θk)]p−q, p ≥ q ≥ 1

0, otherwise, (24)

[Qb1]p,q =

[b(θk)]p+q, p+ q ≤ N − 1

0, otherwise, (25)

[Qb2]p,q =

[b(θk)]p−q, p ≥ q ≥ 1

0, otherwise. (26)

Therefore, we have

∆ = Q [IK ⊗ c] , (27)

where

Q ,[Qb(θ0)⊗Qa(θ0), . . . ,Qb(θK−1)⊗Qa(θK−1)

].(28)

Then, the received signal in (14) can be rewritten as

rp = Q (IK ⊗ c)γp + np

= Q(γp ⊗ c

)+ np. (29)

Collect the P pulses into a matrix, and the received signalcan be finally obtained as

R = Q[γ0 ⊗ c,γ1 ⊗ c, . . . ,γP−1 ⊗ c

]+N

= Q(Γ⊗ c) +N , (30)

where R ,[r0, r1, . . . , rP−1

], N ,

[n0,n1, . . . ,nP−1

],

Γ ,[γ0,γ1, . . . ,γP−1

], and the u-th row and p-th column

of Γ is denoted as Γu,p. In this paper, we will estimate theDOAs from R with the unknown mutual coupling vector c,the target scattering coefficients Γ and the noise variance σ2

n.

III. DOA ESTIMATION METHOD WITH UNKNOWNMUTUAL COUPLING

A. The Off-Grid Sparse Model

Discretize the angle of detection area into U grids ζ ,[ζ0, ζ1, . . . , ζU−1

], and the u-th discretized angle are denoted

as ζu. Then, a dictionary matrix can be formulated as

Ψ ,[Φ(ζ0),Φ(ζ1), . . . ,Φ(ζU−1)

]∈ CMN×UMN , (31)

where Φ(ζu) , Qb(ζu) ⊗ Qa(ζu), and the space betweendiscretized angles, also known as grid size, is δ , |ζu+1−ζu|.For the dictionary matrix Ψ, the restricted isometry property(RIP) with constant δ$ is defined as [44]

(1− δ$)‖x‖22 ≤ ‖Ψx‖22 ≤ (1 + δ$)‖x‖22, (32)

for all $-sparse vector x ($ = KMN ). If x is $-sparseand Ψ satisfies δ2$ + δ3$ < 1, then x is the unique `1minimizer [45]. However, the tight RIP constant of a givenmatrix Ψ is difficult to compute, so we calculate the minimumDOA separation for multiple targets. As described in [46], inour scenario (the system parameters are given in Section IV),the minimum DOA separation can be obtained as 9.67.

However, for the k-th target, the DOA is θk and is not at thediscretized grids exactly, so the sub-matrix for the k-th targetcan be approximated by

Φ(θk) = Φ(ζuk + (θk − ζuk))

≈

[Qb(ζuk) + (θk − ζuk)

∂Qb(ζ)

∂ζ

∣∣∣∣ζ=ζuk

]

⊗

[Qa(ζuk) + (θk − ζuk)

∂Qa(ζ)

∂ζ

∣∣∣∣ζ=ζuk

]≈ Φ(ζuk) + (θk − ζuk)Ω(ζuk), (33)


a b

αn

c d

β

X

e1

f1

υT

e2

f2

υR

cT

cR

δ

ν

R

Fig. 2. Graphical model of SBLMC (rectangles are the hyperparameters,circles are the radar parameters and signals).

where ζuk is the discretized grid angle nearest to the targetDOA θk, and we define the first order of derivative as

Ω(ζuk) , Qb(ζuk)⊗ ∂Qa(ζ)

∂ζ

∣∣∣∣ζ=ζuk

+∂Qb(ζ)

∂ζ

∣∣∣∣ζ=ζuk

⊗Qa(ζuk). (34)

By formulating a sparse matrix X ∈ CU×P with thecolumns xp (p = 0, 1, . . . , P − 1) having the same supportset, i.e., X ,

[x0,x1, . . . ,xP−1

], the received signal can be

approximated by a sparse-based model

R ≈ [Ψ + Ξ (diag ν ⊗ IMN )] (X ⊗ c) +N , (35)

where Ξ ,[Ω(ζ0),Ω(ζ1), . . . ,Ω(ζU−1)

], and the u-th sub-

matrix can be also written as Ξu , Ω(ζu) to simplify thenotation. The u-th row and p-th column of sparse matrix X ∈CU×P is

Xu,p =

Γuk,p, u = uk

0, otherwise, (36)

and the u-th entry of the off-grid vector ν ∈ RU×1 is

νu =

θk − ζuk , u = uk

0, otherwise. (37)

Finally, by absorbing the approximation into the additivenoise, the off-grid sparse model in the MIMO radar withunknown mutual coupling effect can be described by a sparsemodel

R = Υ(ν)(X ⊗ cR ⊗ cT) +N , (38)

where Υ(ν) , Ψ + Ξ (diag ν ⊗ IMN ). With the re-ceived signal R, we can estimate the target DOAs θk (k =0, 1, . . . ,K − 1) with the unknown parameters including thesparse matrix X , the off-grid vector ν, and the mutualcoupling vectors cT and cR. The DOAs can be obtained fromthe support sets of X , the target scattering coefficients areobtained from the nonzero entries of X , and the mutualcoupling matrices can be obtained from cT and cR.

B. Sparse Bayesian Learning-Based DOA Estimation Method

In this paper, we propose an SBL-based method to estimatethe target DOAs with unknown mutual coupling effect, andthe proposed method is named as SBL with the mutualcoupling (SBLMC). The graphical model of SBLMC is givenin Fig. 2, where the unknown parameters are determined by thehyperparameters, and the received signal S is determined byradar parameters and signals. To realize the SBLMC algorithm,the distribution assumptions are given as follows.

We assume that the additive noise is white (circularlysymmetric) Gaussian noise with the noise variance being σ2

n,and the distribution of noise can be expressed as

p(N |σ2n) =

U−1∏u=0

CN (np|0MN×1, σ2nIMN ), (39)

where the complex Gaussian distribution is defined as

CN (x|a,Σ) =1

πN det(Σ)e−(x−a)

HΣ−1(x−a). (40)

When the noise variance σ2n is unknown, by defining a

hyperparamter, i.e., the precision, αn , σ−2n , a Gammadistribution can be adopted to describe the inverse of noisevariance

p(αn) = G(αn; a, b), (41)

where a and b are the hyperparameters for αn, and

G(αn; a, b) , Γ−1(a)baαa−1n e−bαn , (42)

Γ(a) ,∫ ∞0

xa−1e−xdx. (43)

Note that the Gamma distribution αn ∼ G(αn; a, b) is aconjugate prior of the Gaussian distribution given mean withunknown variance x ∼ N (x|0, α−1n ), so the posterior distri-bution p(αn|x) also follows a Gamma distribution. Therefore,the assumption of Gamma distribution for the precision αncan simplify the following analysis.

When the scattering coefficients Γ are independent amongsnapshots, we can also assume that the sparse matrix Xfollows a Gaussian distribution

p(X|Λx) =

P−1∏p=0

CN (xp|0U×1,Λx), (44)

where Λx ∈ RU×U is a diagonal matrix with the u-th diagonalentry being σ2

x,u. Usually, the sparseness prior is the Laplacedensity function [26,27], but the Laplace prior is not conjugateto the Gaussian likelihood. Therefore, for simplification, weuse the Gaussian prior for the sparse matrix X and obtain theestimation expressions in closed form. Then, by defining theprecision β ,

[β0, β1, . . . , βU−1

]Tand βu , σ−2x,u, we have

the following Gamma prior for β

p(β; c, d) =

U−1∏u=0

G(βu; c, d), (45)

where c and d are the hyperparmaters for β.


Similarly, when the mutual coupling coefficients are inde-pendent with antennas, we can also assume that the mutualcoupling vectors cT and cR follow Gaussian distributions

p(cT|ΛT) =

M−1∏m=0

CN (cT,m|0, σ2T,m), (46)

p(cR|ΛR) =

N−1∏n=0

CN (cR,n|0, σ2R,n), (47)

where ΛT ∈ RM×M is a diagonal matrix with the m-thdiagonal entry being σ2

T,m, and ΛR ∈ RN×N is a diagonalmatrix with the n-th diagonal entry being σ2

R,n. Define theprecisions ϑT ,

[ϑT,0, ϑT,1, . . . , ϑT,M−1

]T(ϑT,m , σ−2T,m)

and ϑR ,[ϑR,0, ϑR,1, . . . , ϑR,N−1

]T(ϑR,n , σ−2R,n). Then, we

can have the following Gamma distributions

p(ϑT; e1, f1) =M−1∏m=0

G(ϑT,m; e1, f1), (48)

p(ϑR; e2, f2) =

N−1∏n=0

G(ϑR,n; e2, f2), (49)

where both e1 and f1 are the hyperparameters for ϑT, andboth e2 and f2 are the hyperparameters for ϑR. Usually, wecan choose the following values a = b = c = d = e1 = f1 =e2 = f2 = 10−2 as the hyperparameters. As shown in [26], thesmall values for hyperparameters are chosen and not sensitiveto specific values [47].

The off-grid parameter ν follows a uniform prior distribu-tion, and the distribution of the u-th entry νu can be expressedas

p(νu; δ) = Uνu([−1

2δ,

1

2δ

]), (50)

where we have

Ux ([a, b]) ,

1b−a , a ≤ x ≤ b0, otherwise

. (51)

The relationships between parameters are shown in Fig. 2.To estimate the DOAs, we can formulate the following prob-lem to maximize the posterior probability with the receivedsignal

X = arg maxX

p(X|R), (52)

where we use a set X ,X,ν, cT, cR, σ

2n,β

to contain all

the unknown parameters. However, the problem of posteriorprobability cannot be solved directly, so an EM method isadopted to realize the sparse Bayesian learning.

To obtain the posterior distribution of X , we first calculatethe joint distribution for all the parameters

p(R,X) = p(R|X)p(X|β)p(cT|ϑT)p(cR|ϑR)p(αn)

p(β)p(ϑT)p(ϑR)p(ν). (53)

Therefore, with the parameters αn, β, ϑT, ϑR, ν, cT and cR,the posterior for X can be obtained as

p(X|R,ν, cT, cR, αn,β,ϑT,ϑR)

=p(R,X)

p(R,ν, cT, cR, αn,β,ϑT,ϑR)

=p(R|X)p(X|β)

p(R|ν, cT, cR, αn,β,ϑT,ϑR), (54)

where p(R|X) and (X|β) can be calculated as

p(R|X) =

P−1∏p=0

CN (rp|Υ(ν)(xp ⊗ c), α−1n IMN )

=

P−1∏p=0

αMNn

πMNe−αn‖rp−Υ(ν)(xp⊗c)‖22 , (55)

p(X|β) =

P−1∏p=0

CN (xp|0U×1,diagβ−1)

=

P−1∏p=0

(U−1∏u=0

βu

)1

πUe−x

Hp diagβxp . (56)

Since the denominator in (54) is not a function of X , theposterior distribution of X can be simplified as

p(X|R,ν, cT, cR, αn,β,ϑT,ϑR) ∝ p(R|X)p(X|β). (57)

Both p(R|X) and p(X|β) are Gaussian functions, so theposterior for X can be also expressed as a Gaussian function

p(X|R,ν, cT, cR, αn,β,ϑT,ϑR) ∝ p(R|X)p(X|β)

∝P−1∏p=0

e−αn‖rp−Υ(ν)(IU⊗c)xp‖22−xHp diagβxp

,P−1∏p=0

CN (xp|µp,ΣX), (58)

where the mean µp and covariance matrix ΣX are

µp = αnΣX(IU ⊗ c)HΥH(ν)rp, (59)

ΣX =[αn(IU ⊗ c)HΥH(ν)Υ(ν)(IU ⊗ c) + diagβ

]−1.

(60)

and we use µp,u to denote the u-th entry of µp.To calculate ΣX and µp, we need to estimate the mu-

tual coupling vectors cT and cR, the off-grid parameterν, and the precisions αn and β. We can use the max-imum posterior probability (MAP) method to maximizep(ν, cT, cR, αn,β,ϑT,ϑR|R). We have

p(ν, cT, cR, αn,β,ϑT,ϑR|R)p(R)

= p(ν, cT, cR, αn,β,ϑT,ϑR,R), (61)

so maximizing p(ν, cT, cR, αn,β,ϑT,ϑR|R) is equivalent tomaximizing p(ν, cT, cR, αn,β,ϑT,ϑR,R). The EM methodcan be used to solve the MAP estimation by treating X asa hidden variable. Before estimating the parameters, we will


first obtain the likelihood function under the expectation withrespect to the posterior of X

L(ν, cT, cR, αn,β,ϑT,ϑR)

, EX|R,ν,cT,cR,αn,β,ϑT,ϑR ln p(X,ϑT,ϑR,R) . (62)

To simplify the notation, we just use E· to representEX|R,ν,cT,cR,αn,β,ϑT,ϑR·, so the likelihood function can besimplified as

L(ν, cT, cR, αn,β,ϑT,ϑR)

= E

ln p(R|X)p(X|β)p(cT|ϑT)p(cR|ϑR)p(αn)

p(β)p(ϑT)p(ϑR)p(ν). (63)

In the following contents, we will give the expressions forall the remaining unknown parameters.

1) For the mutual coupling vector cT, ignoring terms inde-pendent thereof, we can obtain the following likelihoodfunction

L(cT) = E ln p(R|X,ν, cT, cR, αn)p(cT|ϑT)

= E

ln

P−1∏p=0

CN (rp|Υ(ν)(xp ⊗ c), α−1n IMN )

+ ln

M−1∏m=0

CN (cT,m|0, ϑ−1T,m)

∝ −αnP Tr

(IU ⊗ c)HΥH(ν)Υ(ν)(IU ⊗ c)ΣX

−P−1∑p=0

αn‖rp −Υ(ν)(µp ⊗ c)‖22

−M−1∑m=0

ϑT,m|cT,m|2. (64)

In Appendix A , the details about calculating ∂L(cT)∂cT

aregiven. By setting ∂L(cT)

∂cT= 0, cT can be obtained as

cT = H−1T zT, (65)

where

HT =

P−1∑p=0

αnTHT ΥH(ν)Υ(ν)(µp ⊗ cR ⊗ IM )

+ αnPGHT

(U−1∑p=0

U−1∑k=0

ΥHp (ν)Υk(ν)ΣX,k,p

)H

(cR ⊗ IM ) + diagϑT, (66)

and

zT =

P−1∑p=0

αnTHT ΥH(ν)rp, (67)

T T ,[µp ⊗ cR ⊗ eM0 , . . . ,µp ⊗ cR ⊗ eMM−1

], (68)

GT ,[cR ⊗ eM0 , cR ⊗ eM1 , . . . , cR ⊗ eMM−1

]. (69)

2) For the mutual coupling vector cR, using the samemethod with cT, we can obtain

cR = H−1R zR, (70)

where

HR =

P−1∑p=0

αnTHRΥH(ν)Υ(ν)(µp ⊗ IN ⊗ cT)

+ αnPGHR

(U−1∑p=0

U−1∑k=0


)H

(IN ⊗ cT) + diagϑT, (71)

and

zR =

P−1∑p=0

αnTHRΥH(ν)rp, (72)

T R ,[µp ⊗ eN0 ⊗ cT, . . . ,µp ⊗ eNN−1 ⊗ cT

], (73)

GR ,[eN0 ⊗ cT, e

N1 cT, . . . , e

NN−1cT

]. (74)

3) For the precision β of scattering coefficients, ignoringterms independent thereof, we can obtain the likelihoodfunction

L(β) = E ln p(X|β)p(β)

= E

ln

P−1∏p=0

CN (xp|0U×1,Λx)

+ ln

U−1∏u=0

G(βu; c, d).

(75)

By setting ∂L(β)∂β = 0, the u-th entry of β can be

obtained as

βu =P + c− 1

d+ PΣX,u,u +∑P−1p=0 |µu,p|2

. (76)

4) For the precision αn of noise, ignoring terms indepen-dent thereof, we can obtain the likelihood function

L(αn) = E ln p(R|X,ν, cT, cR, αn)p(αn)

= E

ln

P−1∏p=0

CN(rp|Υ(ν)(xp ⊗ cR ⊗ cT), σ2

nI)

+ lnG(αn; a, b). (77)

By setting ∂L(αn)∂αn

= 0, we can obtain

αn =MNP + a− 1

PN1 + N2 + b, (78)

where

N1 , Tr(IU ⊗ c)HΥH(ν)Υ(ν)(IU ⊗ c)ΣX, (79)

N2 , ‖R−Υ(ν)(µ⊗ c)‖2F , (80)

µ ,[µ0,µ1, . . . ,µP−1

]. (81)

5) For the precision ϑT of mutual coupling vector, ignoringterms independent thereof, we can obtain the likelihoodfunction

L(ϑT) = E ln p(cT|ϑT)p(ϑT)

= E

ln

M−1∏m=0

CN (cT,m|0, σ2T,m)

+ ln

M−1∏m=0

G(ϑT,m; e1, f1). (82)


By setting ∂L(ϑT)∂ϑT

= 0, we can obtain the m-th entry ofϑT as

ϑT,m =e1

f1 + cHT,mcT,m

. (83)

6) For the precision ϑR of mutual coupling vector, usingthe same method, we can obtain the n-th entry of ϑR as

ϑR,n =e2

f2 + cHR,ncR,n

. (84)

7) For off-grid ν, ignoring terms independent thereof, wecan obtain the likelihood function

L(ν) = E ln p(R|X,ν, cT, cR, αn)p(ν) . (85)

By setting ∂L(ν)∂ν = 0, we can obtain

ν = H−1z, (86)

where the entry of the u-th row and m-column in H ∈RU×U is

Hu,m = R

(PΣX,u,m +

P−1∑p=0

µHp,mµp,u

)cHΞH

mΞuc

,

(87)

and the u-th entry of z ∈ RU×1 is

zu =

P−1∑p=0

R[rp −Ψ(µp ⊗ c)

]HΞuµu,pc

−U−1∑m=0

RPΣX,u,mc

HΨHmΞuc

. (88)

The details to obtain ν are given in Appendix B.In Algorithm 1, we show the details about the proposed

method SBLMC to estimate the DOAs with unknown mutualcoupling effect. In the proposed SBLMC algorithm, after theiterations, we can obtain the spatial spectrum P X of the sparsematrix X from the received signal R. Then, by searching allthe values of P X, the corresponding peak values can be found.By selecting positions of peak values corresponding to the Kmaximum values, we can estimate the DOAs of targets, wherewe use ζ + ν as the discretized angle vector.

IV. SIMULATION RESULTS

In this section, the simulation results about the proposedmethod for DOA estimation in the MIMO radar system aregiven, and the simulation parameters are given in Table I.For the proposed SBLMC algorithm, the maximum iterationis Niter = 103 and the stop threshold is λ = 10−3. Allexperiments are carried out in Matlab R2017b on a PC witha 2.9 GHz Intel Core i5 and 8 GB of RAM. Matlab codeshave been made available online at https://sites.google.com/site/chenpengdsp/publications.

First, we show the estimated spatial spectrum of 3 targets.As shown in Fig. 3, 3 present methods including off-gridsparse Bayesian inference (OGSBI) [29], Bayesian compres-sive sensing (BCS) [27] and MUSIC [8], have comparedwith the proposed SBLMC method. With the mutual couplingeffect, the traditional MUSIC method cannot achieve better

Algorithm 1 SBLMC algorithm to estimate the DOAs withunknown mutual coupling effect

1: Input: received signal R, dictionary matrix Ψ, the firstorder derivative of dictionary matrix Ξ, the number ofpulses P , the maximum of iteration Niter, stop thresholdλth.

2: Initialization: cT = ϑT = [1,01×(M−1)]T , cR = ϑR =

[1,01×(N−1)]T , αn = 1, the hyperparameters a = b =

c = d = e1 = f1 = e2 = f2 = 10−2, ν = 0U×1,β = 1U×1, iiter = 1, λ = ‖R‖2F .

3: while iiter ≤ Niter or λ ≤ λth do4: Υ(ν)← Ψ + Ξ (diag ν ⊗ IMN ).5: Obtain µp (p = 0, 1, . . . , P − 1) and ΣX from (59) and

(60), respectively.6: Obtain the spatial spectrum

PX = RdiagΣX+1

P

P−1∑p=0

|µp|2, (89)

where |µp| ,[|µp,0|, |µp,1|, . . . , |µp,U−1|

]T.

7: β′ ← β, and update β from (76).8: Update cT and cR from (65) and (70), respectively.9: Update ϑT and ϑR from (83) and (84), respectively.

10: Estimate ν from (86).11: Update αn from (78).12: if iiter > 1 then13: λ = ‖β−β′‖2

‖β′‖2 .14: end if15: iiter ← iiter + 1.16: end while17: Output: the spatial spectrum PX, and the DOAs (ζ + ν)

can be obtained from the positions of peak values in PX.

-80 -60 -40 -20 0 20 40 60 80

DOA (deg)

10-4

10-3

10-2

10-1

100

101

Sp

atia

l sp

ectr

um

SBLMC

Target DOA

OGSBI

BCS

MUSIC

Fig. 3. The spatial spectrum for DOA estimation.

https://sites.google.com/site/chenpengdsp/publications

https://sites.google.com/site/chenpengdsp/publications


-80 -60 -40 -20 0 20 40 60 80

DOA (deg)

10-4

10-3

10-2

10-1

100

101

Sp

atia

l sp

ectr

um

SBLMC

Target DOA

Capon-like method

MUSIC-like method

CS-based method (L1 norm )

Fig. 4. The spatial spectrum of the proposed method is compared with thatof the present methods in DOA estimation.

-20 -10 0 10 20

SNR (dB)

-80

-60

-40

-20

0

DO

A e

stim

ati

on

err

or

(dB

)

SBLMC (2o)

OGSBI

BCS

MUSIC

CRLB

SBLMC (1o)

Fig. 5. The DOA estimation performance with different SNRs.

-14 -12 -10 -8 -6 -4 -2

Mutual coupling between adjacent antennas (dB)

-55

-50

-45

-40

-35

-30

-25

-20

-15

DO

A e

stim

atio

n e

rro

r (d

B)

SBLMC

OGSBI

BCS

MUSIC

Fig. 6. The DOA estimation performance with different mutual couplingeffects.

TABLE ISIMULATION PARAMETERS

Parameter ValueThe signal-to-noise ratio (SNR) of echo signal 20 dB

The number of pulses P 100

The number of transmitting antennas M 10

The number of receiving antennas N 5

The number of targets K 3

The space between antennas dT = dR 0.5 wavelengthThe grid size δ 2

The detection DOA range [−80, 80]

The hyperparameters a, b, c, d, e1, f1, e2, f2 10−2

The mutual coupling between adjacent antennas −5 dB

2 4 6 8 10

Grid size (deg)

-50

-40

-30

-20

-10

0

10

DO

A e

stim

atio

n e

rro

r (d

B)

SBLMC

OGSBI

BCS

Fig. 7. The DOA estimation performance with different grid sizes.

TABLE IIESTIMATED DOAS

Methods Target 1 Target 2 Target 3

Target DOAs 4.3075 27.0740 49.3603

SBLMC 4.2746 27.2441 49.4521

OGSBI 1.8636 25.6868 50.7775

BCS 2.0000 26.0000 50.0000

MUSIC 2.9929 25.7086 50.1324

performance. Since the present Bayesian methods (OGSBIand BCS) have not considered the mutual coupling effect,the estimation performance cannot be further improved. Theproposed SBLMC considers both off-grid and mutual couplingeffects, can achieve the best spatial spectrum and improvethe DOA estimation performance. In Table II, we give theestimated DOAs with different methods. We use the followingexpression to measure the estimation performance

e , 10 log10 ‖θ − θ‖22 (dB), (90)

where θ denotes the estimated DOA vector and θ is the targetDOA vector. Both θ and θ are in rad. The estimation errors ofOGSBI, BCS and MUSIC methods are −25.20 dB, −26.78 dBand −28.94 dB, respectively. Since the mutual coupling effecthas not been considered, the DOA estimation performance of


these 3 present methods almost the same. However, the DOAestimation error of the proposed SBLMC is −49.31 dB, whichis significantly better than the present methods.

Additionally, the DOA estimation performance of the pro-posed SBLMC method is compared with present methods inFig. 4. The CS-based method (L1 norm) is proposed in [24,48],where the DOA estimation problem is converted into a sparsereconstruction problem via `1 minimization. The MUSIC-like method is proposed in [41], where the mutual couplingeffect is considered in the MUSIC-like method. The Capon-like is proposed in [15] for the DOA estimation in MIMOradar systems. As shown in this figure, the proposed methodachieves the best DOA estimation performance in the scenariowith unknown mutual coupling effect by estimating all theunknown parameters iteratively in the SBL-based method.However, the disadvantage of the proposed method is highercomputational complexity than these present methods.

We also show the DOA estimation performance with differ-ent signal-to-noise ratios (SNRs) in Fig. 5. As shown in thisfigure, when SNR ≤ −10 dB, all the methods cannot workwell, and the performance is almost the same. When SNR >−10 dB, the DOA estimation performance of present methodsincluding OGSBI, BCS and MUSIC cannot be improved,and the estimation errors are around −28 dB. However, withimproving SNR, the estimation performance of SBLMC canalso be improved, and the final estimation error can be lowerthan −50 dB with SNR ≥ 5 dB. With the Cramér-Rao lowerbound (CRLB) in [49], Fig. 5 also shows the correspondingCRLB of DOA estimation. As shown in this figure, whenSNR > 5 dB, the proposed method can approach CRLB.In Fig. 5, the curve “SBLMC (2)” is the SBLMC methodwith the grid size being δ = 2 and the curve “SBLMC (1)”is that with the grid size being δ = 1. From the curves,we can see that the estimation performance of SBLMC canapproach the CRLB with the dense sampling grids (δ = 1).Therefore, the reason why the estimation error cannot befurther reduced (error floor) is that the spatial spectrum isdiscretized to find the peak values and the discretized gridscannot be infinitely small. The proposed SBLMC method canimprove the estimation performance, but the improvement islimited by the grid size.

Then, we also show the mutual coupling effect on the DOAestimation in Fig. 6, where the mutual coupling effect betweenadjacent antennas is from −15 dB to −2 dB. With increasingthe mutual coupling effect, the DOA estimation error of BCSis from −34 dB to −28 dB. Since the grid effect in the BCSmethod, decreasing the mutual coupling effect cannot furtherimprove the estimation performance when the mutual couplingbetween adjacent antennas is less than −8 dB. However, forboth OGSBI and MUSIC methods, decreasing the mutualcoupling effect can decrease the estimation error from around−25 dB to around −50 dB. For the proposed methods, sincethe mutual coupling vectors cT and cR are estimated, themutual coupling has limited effect on the DOA estimationperformance, and the estimation error can be lower than −50dB when the mutual coupling between adjacent antennas isless than −2 dB.

We show the DOA estimation performance with different

TABLE IIICOMPUTATIONAL TIME

Methods Time (one iteration) Number of iterations Total timeSBLMC 4.12 s 139 537.71 sOGSBI 2.66 s 146 374.79 s

BCS 0.17 s 147 17.23 sMUSIC – – 80.31 s

grid sizes in Fig. 7, where the space between adjacent dis-cretized angles δ is from 2 to 10. Since the BCS methodhas not considered the off-grid effect, the worst estimationperformance is achieved in these 3 methods. Both BCS andOGSBI methods have not considered the mutual coupling, sowhen the grid size δ is less than 6, the estimation performancecannot be improved. However, for the proposed SBLMC, withdecreasing the gird size δ from 10 to 2, the estimation errorcan be decreased from −8 dB to −50 dB.

Finally, we compare the computational time of the pro-posed SBLMC method with that of the present 3 methods inTable III. All the methods have not been further optimizedto decrease the computational time. Without the additionalsimplifications, the computational complexities of SBLMC inStep 5, Step 8 and Step 10 are O(PU2MN+UM2N2+U3),O(PU2MN + UM2N2 + U3) and O(U3 + PU2MN), sothe computational complexities of SBLMC can be obtainedas O(U3 + PU2MN + UM2N2) per iteration and an ad-ditional computational workload of order O(U2M3N3) forinitialization. To simplify the representation, with U ≥ MN ,the computational complexity of SBLMC can be approximatedby O(PU3). If U < MN the computational complexity ofSBLMC can be approximated by O(PUM2N2). Therefore,the proposed SBLMC algorithm has the same order of compu-tational complexity with OGSBI [29]. Since MUSIC algorithmis a continue domain method, we estimate the target DOAsby discretizing the detection range [−80, 80] into 1.6× 106

grids. As shown in this table, the BCS method is the fastestamong all methods, since the detection angle is discretized byδ = 2. The proposed SBLMC is comparable with the OGSBImethod, but the DOA estimation performance is much better.The computational time of MUSIC method is determined bythe length of discretized angles, and usually is a methodwith higher computational complexity than BCS. Therefore,the proposed SBLMC method can significantly improve theestimation performance in the MIMO radar system with bothoff-grid and mutual coupling effects with the acceptable com-putational complexity.

V. CONCLUSIONS

We have investigated the DOA estimation problem inMIMO radar system with unknown mutual coupling effectin this paper. The off-grid problem in the CS-based sparsereconstruction method has also been considered concurrentlyto improve the DOA estimation performance. The novel sparseBayesian learning with mutual coupling (SBLMC) methodusing EM has been proposed to estimate target DOAs. Ad-ditionally, we have theoretically derived the prior distributions


∂L(cT)

∂cT= −αnP

[cH

(U−1∑p=0

U−1∑k=0

ΥHp (ν)Υk(ν)Ex,k,p

)[cR ⊗ eM0 , . . . , cR ⊗ eMM−1

]]

+

P−1∑p=0

αn[rp −Υ(ν)(µp ⊗ c)]HΥ(ν)[µp ⊗ cR ⊗ eM0 , . . . ,µp ⊗ cR ⊗ eMM−1

]− cHT diagϑT. (91)

U−1∑m=0

νmR

(PΣX,u,m +

P∑p=0

µHp,mµp,u

)cHΞH

mΞuc

=

P∑p=0

R[rp −Ψ(µp ⊗ c)

]HΞuµp,uc

−U−1∑m=0

RPΣX,u,mc

HΨHmΞuc

. (92)

for all the unknown parameters including the variance vectorof target scattering coefficients, the mutual coupling vectors,the off-grid vector and the noise variance. Simulation resultsconfirm that the proposed SBLMC method outperforms thepresent DOA estimation methods in the MIMO radar sys-tem with the unknown mutual coupling effect. Additionally,the computational complexity of SBLMC is also acceptable.However, with the same characteristic of the sparse-basedsuper-resolution methods, the minimum DOA separation ofthe proposed SBLMC method is limited by the radar aperture.Future work will focus on the optimization of MIMO radarsystem using the SBLMC method for DOA estimation.

APPENDIX ATHE DERIVATION OF LIKELIHOOD FUNCTION L(cT)

The likelihood function L(cT) can be rewritten as

L(cT) ∝ −αnPG1(cT)−P−1∑p=0

αnG2(cT)− G3(cT), (93)

where

G1(cT) , Tr

(IU ⊗ c)HΥH(ν)Υ(ν)(IU ⊗ c)ΣX, (94)

G2(cT) , ‖rp −Υ(ν)(µp ⊗ c)‖22, (95)

G3(cT) ,M−1∑m=0

ϑT,m|cT,m|2. (96)

With the derivations of complex vector and matrix, ∂G1(cT)∂cT

is a row vector, and the m-th entry can be calculated as[∂G1(cT)

∂cT

]m

= Tr

∂(IU ⊗ c)HΥH(ν)Υ(ν)(IU ⊗ c)ΣX

∂cT,m

.

(97)

We can calculate


∂cT,m

=∂(IU ⊗ c)H

∂cT,mΥH(ν)Υ(ν)(IU ⊗ c)ΣX

+ (IU ⊗ c)HΥH(ν)Υ(ν)∂(IU ⊗ c)∂cT,m

ΣX

= (IU ⊗ c)HΥH(ν)Υ(ν)

(IU ⊗

∂cR ⊗ cT

∂cT,m

)ΣX

= (IU ⊗ c)HΥH(ν)Υ(ν)(IU ⊗ cR ⊗ eMm

)ΣX, (98)

where eMm is a M × 1 vector with the m-th entry being 1 andother entries being 0. Therefore, the the m-th entry can besimplified as[

∂G1(cT)

∂cT

]m

= cH

(U−1∑p=0

U−1∑k=0


)(cR ⊗ eMm ), (99)

and we finally have the derivation of G1(cT) as

∂G1(cT)

∂cT= cH

(U−1∑p=0

U−1∑k=0


)(100)[

cR ⊗ eM0 , cR ⊗ eM1 , . . . , cR ⊗ eMM−1].

∂G2(cT)∂cT

can be simplified as

∂G2(cT)

∂cT= −[rp −Υ(ν)(µp ⊗ c)]HΥ(ν)

∂µp ⊗ c∂cT

= −[rp −Υ(ν)(µp ⊗ c)]HΥ(ν) (101)[µp ⊗ cR ⊗ eM0 , . . . ,µp ⊗ cR ⊗ eMM−1

].

∂G2(cT)∂cT

can be simplified as

∂G2(cT)

∂cT= cHT diagϑT. (102)

Finally, with ∂G1(cT)∂cT

, ∂G2(cT)∂cT

and ∂G3(cT)∂cT

, the expressionof ∂L(cT)

∂cTcan be obtained in (91).

APPENDIX BTHE DERIVATION OF LIKELIHOOD FUNCTION L(ν)

The likelihood function L(cT) can be rewritten as

L(ν) ∝P−1∑p=0

T1(ν) + T2(ν), (103)

where

T1(ν) , ‖rp −Υ(ν)(µp ⊗ c)‖22, (104)

T2(ν) , Tr(IU ⊗ c)HΥH(ν)Υ(ν)(IU ⊗ c)ΣX. (105)


∂T1(ν)∂ν can be obtained as

∂T1(ν)

∂ν= −2R

[rp −Υ(ν)(µp ⊗ c)]H

∂Υ(ν)(µp ⊗ c)∂ν

= −2R

[rp −Υ(ν)(µp ⊗ c)]HΞ(diagµp ⊗ c)

. (106)

∂T2(ν)∂ν ∈ R1×U is a row vector, and the u-th entry is[

∂T2(ν)

∂ν

]u

= Tr


∂νu

= Tr

[0, (IU ⊗ cH)ΥH(ν)Ξuc,0

]+ Tr

[0, (IU ⊗ cH)ΥH(ν)Ξuc,0

]HΣX

= 2R

U−1∑m=0

cHΥHm(ν)ΞucΣX,u,m

. (107)

Therefore, ∂T2(ν)∂ν can be simplified as

∂T2(ν)

∂ν= 2R

diag

ΣX(IU ⊗ c)HΥH(ν)Ξ(IU ⊗ c)

T.

(108)

Therefore, with ∂L(ν)∂νu

= 0, we can obtain the equation (92)to obtain ν.

REFERENCES

[1] J. Li and P. Stoica, “MIMO radar with colocated antennas,” IEEE SignalProcess. Mag., vol. 24, no. 5, pp. 106–114, Oct. 2007.

[2] P. Chen, L. Zheng, X. Wang, H. Li, and L. Wu, “Moving target detectionusing colocated MIMO radar on multiple distributed moving platforms,”IEEE Trans. Signal Process., vol. 65, no. 17, pp. 4670 – 4683, Jun. 2017.

[3] M. Davis, G. Showman, and A. Lanterman, “Coherent MIMO radar:The phased array and orthogonal waveforms,” IEEE Aerosp. Electron.Syst. Mag., vol. 29, no. 8, pp. 76–91, Aug. 2014.

[4] A. M. Haimovich, R. S. Blum, and L. J. Cimini, “MIMO radar withwidely separated antennas,” IEEE Signal Process. Mag., vol. 25, no. 1,pp. 116–129, Dec. 2008.

[5] P. Chen, C. Qi, and L. Wu, “Antenna placement optimisation forcompressed sensing-based distributed MIMO radar,” IET Radar, Sonar& Navigation, vol. 11, no. 2, pp. 285–293, Feb. 2017.

[6] B. D. V. Veen and K. M. Buckley, “Beamforming: A versatile approachto spatial filtering,” IEEE ASSP Magazine, vol. 5, no. 2, pp. 4 – 24, Apr.1988.

[7] R. O. Schmidt, “Multiple emitter location and signal parameter estima-tion,” IEEE Trans. Antennas Propag., vol. 34, no. 3, pp. 276–280, Mar.1986.

[8] R. Schmidt, “A signal subspace approach to multiple emitter locationspectrum estimation,” Ph.D. dissertation, Stanford University, Stanford,CA, 1981.

[9] M. Zoltowski, G. Kautz, and S. Silverstein, “Beamspace Root-MUSIC,”IEEE Trans. Signal Process., vol. 41, no. 1, pp. 344–364, Jan. 1993.

[10] R. Roy and T. Kailath, “ESPRIT-estimation of signal parameters viarotational invariance techniques,” IEEE Trans. Acoust., Speech, SignalProcess., vol. 37, no. 7, pp. 984–995, Jul. 1989.

[11] L. Du, T. Yardibi, J. Li, and P. Stoica, “Review of user parameter-freerobust adaptive beamforming algorithms,” Digital Signal Processing,vol. 19, no. 4, pp. 567 – 582, Jul 2009.

[12] A. Khabbazibasmenj, A. Hassanien, S. A. Vorobyov, and M. W.Morency, “Efficient transmit beamspace design for search-free baseddoa estimation in mimo radar,” IEEE Trans. Signal Process., vol. 62,no. 6, pp. 1490–1500, March 2014.

[13] M. Cao, S. A. Vorobyov, and A. Hassanien, “Transmit array interpolationfor DOA estimation via tensor decomposition in 2-D MIMO radar,”IEEE Trans. Signal Process., vol. 65, no. 19, pp. 5225–5239, Oct 2017.

[14] A. Hassanien and S. A. Vorobyov, “Transmit energy focusing for DOAestimation in MIMO radar with colocated antennas,” IEEE Trans. SignalProcess., vol. 59, no. 6, pp. 2669–2682, June 2011.

[15] L. Xu, J. Li, and P. Stoica, “Target detection and parameter estimationfor MIMO radar systems,” IEEE Trans. Aerosp. Electron. Syst., vol. 44,no. 3, pp. 927–939, July 2008.

[16] D. Nion and N. D. Sidiropoulos, “Tensor algebra and multidimensionalharmonic retrieval in signal processing for MIMO radar,” IEEE Trans.Signal Process., vol. 58, no. 11, pp. 5693–5705, Nov 2010.

[17] Y. I. Abramovich, G. J. Frazer, and B. A. Johnson, “Iterative adaptivekronecker MIMO radar beamformer: Description and convergence anal-ysis,” IEEE Trans. Signal Process., vol. 58, no. 7, pp. 3681–3691, July2010.

[18] Z. Yang and L. Xie, “Exact joint sparse frequency recovery via opti-mization methods,” IEEE Trans. Signal Process., vol. 64, no. 19, pp.5145 – 5157, Oct. 2016.

[19] ——, “Enhancing sparsity and resolution via reweighted atomic normminimization,” IEEE Trans. Signal Process., vol. 64, no. 4, pp. 995–1006, Feb. 2016.

[20] Y. Yu, A. P. Petropulu, and H. V. Poor, “Measurement matrix design forcompressive sensing-based MIMO radar,” IEEE Trans. Signal Process.,vol. 59, no. 11, pp. 5338 – 5352, Nov. 2011.

[21] M. Carlin, P. Rocca, G. Oliveri, F. Viani, and A. Massa, “Directions-of-arrival estimation through Bayesian compressive sensing strategies,”IEEE Trans. Antennas Propag., vol. 61, no. 7, pp. 3828 – 3838, Jul.2013.

[22] ——, “Novel wideband DOA estimation based on sparse Bayesianlearning with dirichlet process priors,” IEEE Trans. Signal Process.,vol. 64, no. 2, pp. 275 – 289, Jan. 2016.

[23] Q. Shen, W. Liu, W. Cui, and S. Wu, “Underdetermined DOA estimationunder the compressive sensing framework: A review,” IEEE Access,vol. 4, pp. 8865 – 8878, Nov. 2016.

[24] M. Rossi, A. M. Haimovich, and Y. C. Eldar, “Spatial compressivesensing for MIMO radar,” IEEE Trans. Signal Process., vol. 62, no. 2,pp. 419–430, Jan 2014.

[25] Z. Tan, P. Yang, and A. Nehorai, “Joint sparse recovery method forcompressed sensing with structured dictionary mismatches,” IEEE Trans.Signal Process., vol. 62, no. 19, pp. 4997–5008, Oct 2014.

[26] M. E. Tipping, “Sparse Bayesian learning and the relevance vectormachine,” J. Mach. Learn. Res., vol. 1, pp. 211–244, Sep. 2001.

[27] S. Ji, Y. Xue, and L. Carin, “Bayesian compressive sensing,” IEEE Trans.Signal Process., vol. 56, no. 6, pp. 2346–2356, 2008.

[28] H. Zhu, G. Leus, and G. B. Giannakis, “Sparsity-cognizant total least-squares for perturbed compressive sampling,” IEEE Trans. Signal Pro-cess., vol. 59, no. 5, pp. 2002–2016, May 2011.

[29] Z. Yang, X. Lihua, and Z. Cishen, “Off-grid direction of arrival esti-mation using sparse Bayesian inference,” IEEE Trans. Signal Process.,vol. 61, no. 1, pp. 38–43, 2013.

[30] J. Dai, X. Bao, W. Xu, and C. Chang, “Root sparse Bayesian learningfor off-grid DOA estimation,” IEEE Signal Process. Lett., vol. 24, no. 1,pp. 46–50, 2017.

[31] X. Wu, W. Zhu, and J. Yan, “Direction of arrival estimation for off-grid signals based on sparse Bayesian learning,” IEEE Sensors Journal,vol. 16, no. 7, pp. 2004–2016, Apr. 2016.

[32] H. Zamani, H. Zayyani, and F. Marvasti, “An iterative dictionarylearning-based algorithm for DOA estimation,” vol. 20, no. 9, pp. 1784–1787, Sep. 2016.

[33] Q. Wang, Z. Zhao, Z. Chen, and Z. Nie, “Grid evolution method forDOA estimation,” IEEE Trans. Signal Process., vol. 66, no. 9, pp. 2474–2383, May 2018.

[34] Z. Zheng, J. Zhang, and J. Zhang, “Joint DOD and DOA estimationof bistatic MIMO radar in the presence of unknown mutual coupling,”Signal Processing, vol. 92, pp. 3039 – 3048, Jun. 2012.

[35] B. Clerckx, C. Craeye, D. Vanhoenacker-Janvier, and C. Oestges,“Impact of antenna coupling on 2 × 2 MIMO communications,” IEEETrans. Veh. Technol., vol. 56, no. 3, pp. 1009 –1018, May 2007.

[36] J. Liu, Y. Zhang, Y. Lu, S. Ren, and S. Cao, “Augmented nested arrayswith enhanced DOF and reduced mutual coupling,” IEEE Trans. SignalProcess., vol. 65, no. 21, pp. 5549 – 5563, Nov. 2017.

[37] P. Rocca, M. A. Hannan, M. Salucci, and A. Massa, “Single-snapshotDoA estimation in array antennas with mutual coupling through amultiscaling BCS strategy,” IEEE Trans. Antennas Propag., vol. 65,no. 6, pp. 3203–3213, Jun. 2017.

[38] M. Hawes, L. Mihaylova, F. Septer, and S. Godsill, “Bayesian compres-sive sensing approaches for direction of arrival estimation with mutualcoupling effects,” IEEE Trans. Antennas Propag., vol. 65, no. 3, pp.1357–1367, 2017.

[39] B. Liao, Z.-G. Zhang, and S.-C. Chan, “DOA estimation and trackingof ULAs with mutual coupling,” IEEE Trans. Aerosp. Electron. Syst.,vol. 48, no. 1, pp. 891 – 905, Jan. 2012.

[40] T. Basikolo, K. Ichige, and H. Arai, “A novel mutual coupling com-pensation method for underdetermined direction of arrival estimation in


nested sparse circular arrays,” IEEE Trans. Antennas Propag., vol. 66,no. 2, pp. 909 – 917, Feb. 2018.

[41] C. Zhang, H. Huang, and B. Liao, “Direction finding in MIMO radarwith unknown mutual coupling,” IEEE Access, vol. 5, pp. 4439 – 4447,Mar. 2017.

[42] A. Termos and B. M. Hochwald, “Capacity benefits of antenna cou-pling,” in 2016 Information Theory and Applications (ITA), La Jolla,CA, USA, Apr. 2004, pp. 1–5.

[43] X. Liu and G. Liao, “Direction finding and mutual coupling estimationfor bistatic MIMO radar,” Signal Processing, vol. 92, no. 2, pp. 517 –522, Feb. 2012.

[44] E. J. Candès and T. Tao, “Decoding by linear programming,” IEEETrans. Inf. Theory, vol. 51, no. 12, pp. 4203–4215, Dec 2005.

[45] ——, “Near-optimal signal recovery from random projections: Universalencoding strategies?” IEEE Trans. Inf. Theory, vol. 52, no. 12, pp. 5406–5425, Dec 2006.

[46] E. J. Candès and C. Fernandez-Granda, “Towards a mathematical theoryof super-resolution,” Communications on Pure and Applied Mathemat-ics, vol. 67, no. 6, pp. 906–956, Jun. 2014.

[47] S. D. Babacan, M. Luessi, R. Molina, and A. K. Katsaggelos, “SparseBayesian methods for low-rank matrix estimation,” IEEE Trans. SignalProcess., vol. 60, no. 8, pp. 3964–3977, Aug 2012.

[48] Y. Yu, A. P. Petropulu, and H. V. Poor, “MIMO radar using compressivesampling,” IEEE Journal of Selected Topics in Signal Processing, vol. 4,no. 1, pp. 146–163, Feb 2010.

[49] H. Jiang, J. Zhang, and K. M. Wong, “Joint DOD and DOA estimationfor bistatic MIMO radar in unknown correlated noise,” IEEE Trans. Veh.Technol., vol. 64, no. 11, pp. 5113–5125, Nov 2015.

Peng Chen (S’15-M’17) was born in Jiangsu, Chinain 1989. He received the B.E. degree in 2011 andthe Ph.D. degree in 2017, both from the Schoolof Information Science and Engineering, SoutheastUniversity, China. From Mar. 2015 to Apr. 2016, hewas a Visiting Scholar in the Electrical EngineeringDepartment, Columbia University, New York, NY,USA.

He is now an associate professor at the StateKey Laboratory of Millimeter Waves, Southeast Uni-versity. His research interests include radar signal

processing and millimeter wave communication.

Zhenxin Cao (M’18) was born in May 1976. Hereceived the M. S. degree in 2002 from NanjingUniversity of Aeronautics and Astronautics, China,and the Ph.D. degree in 2005 from the Schoolof Information Science and Engineering, SoutheastUniversity, China. From 2012 to 2013, he was aVisiting Scholar in North Carolina State University.

Since 2005, he has been with the State KeyLaboratory of Millimeter Waves, Southeast Univer-sity, where he is a Professor. His research interestsinclude antenna theory and application.

Zhimin Chen (M’17) was born in Shandong, China,in 1985. She received the Ph.D. degree in infor-mation and communication engineering from theSchool of Information Science and Engineering,Southeast University, Nanjing, China in 2015. Since2015, she has been with Shanghai Dianji University,Shanghai, China. Her research interests include arraysignal pro-cessing and Millimeter-Wave communica-tions.

Xianbin Wang (S’98-M’99-SM’06-F’17) is a Pro-fessor and Tier-I Canada Research Chair at WesternUniversity, Canada. He received his Ph.D. degree inelectrical and computer engineering from NationalUniversity of Singapore in 2001.

Prior to joining Western, he was with Commu-nications Research Centre Canada (CRC) as a Re-search Scientist/Senior Research Scientist betweenJuly 2002 and Dec. 2007. From Jan. 2001 to July2002, he was a system designer at STMicroelectron-ics, where he was responsible for the system design

of DSL and Gigabit Ethernet chipsets. His current research interests include5G technologies, Internet-of-Things, communications security, machine learn-ing and locationing technologies. Dr. Wang has over 300 peer-reviewed journaland conference papers, in addition to 26 granted and pending patents andseveral standard contributions.

Dr. Wang is a Fellow of Canadian Academy of Engineering, a Fellow ofIEEE and an IEEE Distinguished Lecturer. He has received many awardsand recognitions, including Canada Research Chair, CRC PresidentâAZsExcellence Award, Canadian Federal Government Public Service Award,Ontario Early Researcher Award and five IEEE Best Paper Awards. Hecurrently serves as an Editor/Associate Editor for IEEE Transactions onCommunications, IEEE Transactions on Broadcasting, and IEEE Transactionson Vehicular Technology and He was also an Associate Editor for IEEETransactions on Wireless Communications between 2007 and 2011, and IEEEWireless Communications Letters between 2011 and 2016. Dr. Wang wasinvolved in many IEEE conferences including GLOBECOM, ICC, VTC,PIMRC, WCNC and CWIT, in different roles such as symposium chair,tutorial instructor, track chair, session chair and TPC co-chair.

ieee transactions on signal processing 1 off-grid doa

Documents