H. C. So Page 1 Nov. 2015

Outlier-Robust Greedy Pursuit

Algorithms in lp-Space for Sparse Approximation

Hing Cheung So

http://www.ee.cityu.edu.hk/~hcso

Department of Electronic Engineering City University of Hong Kong

H. C. So Page 2 Nov. 2015

Outline Introduction

lp-Correlation and lp-Orthogonality

Robust Greedy Pursuit Algorithms

Numerical Examples

Summary

List of References

H. C. So Page 3 Nov. 2015

Introduction What is Sparse Approximation? Sparse approximation refers to decomposing a target signal into a linear combination of very few elements drawn from a fixed collection.

Consider the following underdetermined linear system:

where is the observed vector, with is known coefficient matrix, is the unknown signal-of-interest and is the noise. The task is to find the sparsest or the minimum l0-norm solution of such that .

H. C. So Page 4 Nov. 2015

Why Sparse Approximation is Important? In real world, many signals-of-interest have a sparse representation in some basis. As a result, this problem is a core issue in numerous areas of science and engineering including statistics, signal processing, machine learning, medical imaging and computer vision. It is dual to sparse recovery whose aim is to retrieve a high-dimensional signal based on a small number of linear measurements. For example, in magnetic resonance imaging, we hope to collect as few observations (i.e., ) as possible because scan time reduction means benefits for patients and health care economics.

H. C. So Page 5 Nov. 2015

How to Perform Sparse Approximation? Finding with minimum number of nonzero entries or the minimum -norm solution of is in fact NP hard. In practice, greedy pursuit and convex optimization are two standard approaches for obtaining an approximate solution. Formulating the sparse approximation problem as:

where is the target sparsity of , the key idea of greedy pursuit is to identify the nonzero components sequentially. At each iteration, one column of that is best correlated with the residual from the previous iteration is chosen, then its contribution to is subtracted.

H. C. So Page 6 Nov. 2015

Representative greedy pursuit algorithms include matching pursuit (MP), orthogonal MP (OMP) and weak MP. On the other hand, the convex optimization approach aims to approximate the -norm by the -norm, and widely-used methods include the least absolute shrinkage and selection operator (LASSO):

basis pursuit (BP):

and l1-regularization:

H. C. So Page 7 Nov. 2015

Motivation of lp-Space Sparse Approximation Derivation of these conventional techniques is based on the l2-norm objective function, which implicitly assumes Gaussian data. In spite of providing theoretical and computational convenience, it is generally understood that the validity of the Gaussian distribution is at best approximate in reality. Non-Gaussian impulsive noise arises in many practical applications. These standard solvers may fail to work properly when the observations contain outliers. We propose to apply greedy pursuit to solve:

H. C. So Page 8 Nov. 2015

lp-Correlation and lp-Orthogonality At the th step of the iterative procedure of -norm based MP, we find a column of that is most strongly correlated with the residual and the column index is determined as:

Updated approximation and residual are then computed as:

and

H. C. So Page 9 Nov. 2015

lp-Correlation To derive the robust MP, we first generalize the inner product or correlation, which is based on -norm, to -space. The ‐correlation of and with

, is defined as:

where the ‐norm is

H. C. So Page 10 Nov. 2015

‐correlation has the following properties: The case of elicits the definition of auto‐ ‐

correlation . If or , then . In other words, any

vector has zero ‐correlation with a zero vector. For any , and .

That is, the ‐correlation is scale‐invariant with respect to the first vector but homogeneous with respect to the second vector .

. In addition, attains its maximum if

and only if there exists a scalar such that , that is, and are collinear.

H. C. So Page 11 Nov. 2015

To make ‐correlation scale‐invariant with respect to both vectors, we define normalized ‐correlation coefficient:

At , is reduced to

That is, generalizes the conventional correlation coefficient.

H. C. So Page 12 Nov. 2015

Normalized ‐correlation has the following properties: auto‐ ‐correlation coefficient of a nonzero vector is

. If or , then .

The ‐correlation coefficient is scale‐invariant. That is to

say, for any nonzero scalars and , we have .

. In addition, attains its maximum 1 if

and only if there exists a scalar such that the nonzero vector , that is, and are collinear.

H. C. So Page 13 Nov. 2015

lp-Orthogonality Two vectors and are ‐orthogonal if . At , this reduces to orthogonality in inner product space as is equivalent to . The relationship between ‐orthogonality and the global minimizer of the residual function

If , then , and and are ‐orthogonal for any value .

H. C. So Page 14 Nov. 2015

Algorithms for lp-Correlation Computation The key step for computing ‐correlation is to solve

For , is twice differentiable and strictly convex for any . The global solution can be easily obtained by Newton’s method which has a complexity of at each iteration.

H. C. So Page 15 Nov. 2015

For , the problem is:

where is considered as positive weight. Defining a new sequence

The optimal is the weighted median of the sequence with weights :

H. C. So Page 16 Nov. 2015

Algorithm for computing weighted median Input: Weighting coefficients and data sequence

. 1. Determine the threshold . 2. Sort the data sequence in ascending order with

the corresponding concomitant weights . 3. Sum the concomitant weights, beginning with and

increasing the order. 4. The weighted median is whose weight leads to the

inequality hold first. Output: The weighted median .

H. C. So Page 17 Nov. 2015

For , the problem is reformulated as:

Assume that has been sorted in ascending order. The function is piecewise with breakpoints at , that is, the domain of can be divided into intervals: , , , and . In each interval, the sign of is known and the absolute operator can be removed. Noting that or is a concave function due to , is concave because the non‐negative combination preserves concavity.

H. C. So Page 18 Nov. 2015

As a concave function attains its minimum at the boundary points, the minimizer of belongs to since

:

The algorithm complexity is . Take an example: , and . To find the global minimum of , we compute the objective function value at the sorted where attains its minimum at .

H. C. So Page 19 Nov. 2015

vs at

H. C. So Page 20 Nov. 2015

Robust Greedy Pursuit Algorithms lp-MP It is the outlier-resistant version of MP. Let , and be the solution, residual, and index set of the nonzero elements of at the th iteration. Algorithm for ‐MP Input: , , error tolerance , and target sparsity Initialization: Initialize , and set the initial solution

, residual , and initial index set .

H. C. So Page 21 Nov. 2015

Repeat

Select the index via

Augment the index set .

Update the solution and with

Update the residual

until or

Output:

H. C. So Page 22 Nov. 2015

Theorem 1 The ‐norm of the residual of the ‐MP algorithm decays exponentially with a rate proportional to :

where

with

being the ‐decay‐factor of and , which is the maximal normalized ‐correlation of and the columns of .

H. C. So Page 23 Nov. 2015

Weak lp-MP It is the outlier-resistant version of weak MP. It does not attempt to find the index associated with the maximal possible ‐correlation but chooses the

index that satisfies

Algorithm for Weak ‐MP Input: , , error tolerance , and target sparsity Initialization: Initialize , and set the initial solution

, residual , and initial index set .

H. C. So Page 24 Nov. 2015

Repeat

Set as first index that satisfies . If there is no such an index, set

Augment the index set .

Update the solution and with

Update the residual

until or

Output:

H. C. So Page 25 Nov. 2015

Theorem 2

The ‐norm of the residual of the weak ‐MP algorithm decays exponentially with a rate proportional to

:

lp-OMP

It is the outlier-resistant version of OMP.

Algorithm for ‐OMP Input: , , error tolerance , and target sparsity Initialization: Initialize , and set the initial solution

, residual , initial index set and .

H. C. So Page 26 Nov. 2015

Repeat

Select the index via

Augment the index set and the matrix of chosen atoms as and .

Solve the ‐norm minimization problem:

to obtain the nonzero coefficients.

Update the residual

until or

Output: Index set and the corresponding coefficients

H. C. So Page 27 Nov. 2015

Algorithm Complexity MP/OMP

‐MP/ ‐OMP ( ) ‐MP/ ‐OMP ( ) ‐MP/ ‐OMP ( )

Complexity of Index Selection

Algorithm Complexity OMP

‐OMP ( ) ‐OMP ( ) (local minimum)

Complexity of Orthogonalization

is number of iterations used in iteratively reweighted least squares in solving ‐norm minimization problem

H. C. So Page 28 Nov. 2015

Numerical Examples Sparse Recovery Unknown with nonzero entries whose magnitudes are uniformly drawn in . Known contains random entries. Noise contains Gaussian mixture model (GMM) or salt-and-pepper variables.

‐MP and ‐OMP with or are compared with MP and OMP.

H. C. So Page 29 Nov. 2015

vs Iteration Number in Noiseless and GMM cases

H. C. So Page 30 Nov. 2015

Sparse Recovery in Noiseless case

H. C. So Page 31 Nov. 2015

Sparse Recovery in GMM noise

H. C. So Page 32 Nov. 2015

Sparse Recovery in Salt-and-Pepper noise

H. C. So Page 33 Nov. 2015

MSE for versus SNR in GMM noise

H. C. So Page 34 Nov. 2015

Harmonic Retrieval The observation vector is now:

The frequency matrix is:

contains unknowns associated with bins:

H. C. So Page 35 Nov. 2015

Probability of Resolution vs Generalized SNR in S S noise

H. C. So Page 36 Nov. 2015

MSE vs Generalized SNR in S S noise

H. C. So Page 37 Nov. 2015

Summary Novel concepts of ‐correlation and ‐orthogonality are

devised and they generalize the standard correlation and orthogonality definitions in the inner product space.

‐correlation provides similarity measure of two vectors in ‐space where , and its computational efficient realizations are developed.

‐space versions of MP, OMP and weak MP are derived

and they outperform the ‐norm based counterparts in the presence of impulsive noise or outliers.

H. C. So Page 38 Nov. 2015

List of References [1] M. Elad, Sparse and Redundant Representations: From

Theory to Applications in Signal and Image Processing, NY: Springer, 2010.

[2] M. Lustig, D. Donoho and J. Pauly, “Sparse MRI: The application of compressed sensing for rapid MR imaging,” Magnetic Resonance in Medicine, vol.58, no.6, pp.1182-1195, 2007.

[3] J.A. Tropp, “Greed is good: Algorithmic results for sparse approximation,” IEEE Transactions on Information Theory, vol.50, no.10, pp.2231-2242, 2004.

[4] J.A. Tropp and A.C. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Transactions on Information Theory, vol.53, no.12, pp.4655-4666, 2007.

H. C. So Page 39 Nov. 2015

[5] R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of the Royal Statistical Society, Series B, vol.58, no.1, pp.267-288, 1996.

[6] S.S. Chen, D.L. Donoho and M.A. Saunders, “Atomic decomposition by basis pursuit,” SIAM Journal on Scientific Computing, vol.20, no.1, pp.33-61, 1998.

[7] M. Figueiredo, R. Nowak and S. Wright, “Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems,” IEEE Journal of Selected Topics on Signal Processing, vol.1, no.4, pp.586-597, 2007.

[8] A.M. Zoubir, V. Koivunen, Y. Chakhchoukh and M. Muma, “Robust estimation in signal processing: A tutorial-style treatment of fundamental concepts,” IEEE Signal Processing Magazine, vol.29, no.4, pp.61-80, 2012.

H. C. So Page 40 Nov. 2015

[9] W.-J. Zeng, H.C. So and X. Jiang, “Outlier-robust greedy pursuit algorithms in -space for sparse approximation,” to appear in IEEE Transactions on Signal Processing, 2016.