inverse problems and sparse representations€¦ · r. gribonval - sparse methods - sisea 2011...

Inverse problems and sparse representations

Rémi Gribonval, DR INRIAEPI METISS (Speech and Audio Processing)INRIA Rennes - Bretagne Atlantique

[email protected]://www.irisa.fr/metiss/members/remi/talks

mercredi 30 novembre 2011

mailto:[email protected]

mailto:[email protected]

http://www.irisa.fr/metiss/members/remi

http://www.irisa.fr/metiss/members/remi

NOVEMBER 2011R. GRIBONVAL - SPARSE METHODS - SISEA 2011

Further material on sparsity

2

• Books with a Signal Processing perspective✦ S. Mallat, «Wavelet Tour of Signal Processing», 3rd edition, 2008✦ M. Elad, «Sparse and Redundant Representations: From Theory to

Applications in Signal and Image Processing», 2009.

• Review paper: ✦ Bruckstein, Donoho, Elad, SIAM Reviews, 2009

• Video lectures✦ E. Candès, MLSS’09✦ F. Bach, NIPS 2009✦ Sparsity in Machine Learning and Statistics SMLS’09

• Slides of this course:✦ http://www.irisa.fr/metiss/gribonval/Teaching/SISEA-2011/


http://www.irisa.fr/metiss/gribonval/Teaching/SISEA-2011/

http://www.irisa.fr/metiss/gribonval/Teaching/SISEA-2011/


Reminder of last session

• Session 1: Introduction✓ sparsity & data compression✓ inverse problems in signal and image processing

✦ image deblurring, image inpainting, ✦ channel equalization, signal separation, ✦ tomography, MRI

✓ sparsity & under-determined inverse problems✦ well-posedness

3



Overview of the course

• Session 2: Complexity & Feasibility✓ NP-completeness of ideal sparse approximation✓ Relaxations✓ L1 is sparsity-inducing and convex

• Session 3: Pursuit Algorithms

• Session 4: Recovery Guarantees

• Session 5: How to choose dictionaries

4


NOVEMBER 2011 - R. GRIBONVAL - SPARSE METHODS - SISEA 2011

Session 2: Complexity & Feasibility

NP-completeness of ideal sparse approximationRelaxed optimization principlesL1 induces sparsity



Ideal sparse approximation

• Input:m x N matrix A, with m < N, m-dimensional vector b

• Possible objectives:find the sparsest approximation within tolerance

find best approximation with given sparsity

find a solution x to

6

arg minx�x�0, s.t.�b−Ax� ≤ �

arg minx�b−Ax�, s.t.�x�0 ≤ k

�b−Ax� ≤ �, and �x�0 ≤ k



Ideal Sparse Approximation

• Brute force search

• Theorem (Davies et al, Natarajan)

It is NP-hard!• Are there other more efficient alternatives ?

7



Ideal Sparse Approximation

• Brute force search

• Theorem (Davies et al, Natarajan)

It is NP-hard!• Are there other more efficient alternatives ?

Many k-tuples to try!

7



Elements of Complexity Theory

• Polynomial algorithm: given input of size N, compute output in cost poly(N)

• A problem is polynomial (is in P) if there exists a polynomial algorithm computing the solution to each instance of the problem

•8



Example

• Problem: find the nearest neighbor to an m-dimensional vector from a collection of N such vectors

✓ input size: m x (N+1)

✓ complexity: O(Nm) [N distances in ]

9

Rm



Complexity: the class NP

• Decision problem: of the type “does there exist x satisfying a given set of constraints”

• A decision problem is «non-deterministic polynomial» (in the class NP) if there exists a polynomial algorithm that checks (for any instance of the problem) if a given candidate solution x satisfies the constraint.✦ warning: the algorithm is not required to find a solution. It merely has

to check if a solution x (given by an “oracle”) is acceptable.

10



Complexity comparisons

• Polynomial reducibility : ✓ Problem A can be reduced to Problem B if every

instance of Problem A can be transformed into an instance of Problem B in polynomial time

11

A “at most as complex” as B

reduction

Problem BProblem A

polynomial

find or checksolution

polynomial



Complexity: NP-complete problems

• Problem B is NP-hard if every Problem A in NP can be polynomially reduced to B.

• NP-complete problems: NP-hard + in NP• Fact: there exists at least one NP-complete problem

(satisfiability problem = SAT)

12

Class NPClass P



Complexity: NP-complete problems

• Problem B is NP-hard if every Problem A in NP can be polynomially reduced to B.

• NP-complete problems: NP-hard + in NP• Fact: there exists at least one NP-complete problem

(satisfiability problem = SAT)

12

Class NPClass P

SAT



NP complete problems

13

• Fact: there exists at least one NP-complete problem (satisfiability problem = SAT)

• A problem in NP is NP-complete if and only if some known NP-complete problem reducible to it

• All NP-complete problems are «as hard»

Class NPClass P

SAT



Complexity of sparse approximation

• Step 1: express it as a decision problem:✓ description of an instance

m x N matrix A, m-dimensional vector b, parameters

✓ size of an instance = approximately mN✓ decision problem: does there exists x such that

• Step 2: prove it is in NP. Indeed, one can check in polynomial time O(mN) whether a given x satisfies the constraints

• Step 3: reduce an existing NP-complete problem to it to show it is NP-complete

14

(�, k)

�b−Ax� ≤ �, and �x�0 ≤ k



NP-completeness of sparse approximation

• Which known NP-complete problem? ✓ 3SET : Exact-cover by 3-sets

✦ Description of an instance: • The integer interval

• A collection of N subsets of size 3

✦ Decision problem:• does there exist an exact cover (=disjoint partition) of E from elements of C ?

15

[Davis & al 1997](other problem chosen in [Natarajan 1995])

E = �1, 3k�C = {Fn, 1 ≤ n ≤ N}, Fn ⊂ E, �Fn = 3

∃?Λ,∪n∈ΛFn = E n �= n� ∈ Λ ⇒ Fn ∩ Fn� = ∅



NP-completeness of Sparse Approximation

• Reduction of 3-SETS to sparse approximation✓ m=3k✓ vector✓ matrix✓ tolerance

• Exact cover implies existence of x such that

• Non-exact cover implies the opposite

A = (ain)1≤i≤m,1≤n≤N ain =�

1, i ∈ Fn

0, otherwise

b = (bi)mi=1 bi = 1,∀i

� < 1

�b−Ax� ≤ �, and �x�0 ≤ k

9gap=1

1 3k

1 3k



Ideal sparse approximation: a summary

• Well-posed if:✓ Kruskal rank large enough

• But: ✓ ... L0 minimization NP-complete to solve✓ ... Kruskal rank NP-complete to compute

• So do we give up ?

17

arg minx�b−Ax�, s.t.�x�0 ≤ k



The End



• Audio = superimposition of structures

• Example : glockenspiel

✓ transients = short, small scale✓ harmonic part = long, large scale

• Gabor atoms

Sparse ApproximationWith Time-Frequency Atoms

�gs,τ,f (t) =

1√sw

�t− τ

s

�e2iπft

�

s,τ,f

19



Ideal Sparse Approximation: overcoming NP-completeness

• Seek approximate solution in polynomial time ?

• Find sub-problems with polynomial time solution ?

• Use randomized algorithm with polynomial time and high probability of correctness of the solution ?

• Use heuristic algorithm (greedy, etc.) ?

20



Overall compromise

• Approximation quality

• Ideal sparsity measure : “norm”

• “Relaxed” sparsity measures

�Ax− b�2

�0

22

0 < p < ∞, �x�p :=� �

n

|xn|p�1/p

�x�0 := �{n, xn �= 0} =�

n

|xn|0



Norms ?

23



Lp norms / quasi-norms

• Norms when

• Quasi-norms when

• “Pseudo”-norm for p=0

24

1 ≤ p <∞

0 < p < 1

Triangle inequality

�x�p = 0 ⇔ x = 0�λx�p = |λ|�x�p,∀λ, x

�x + y�p ≤ �x�p + �y�p,∀x, y

Quasi-triangle inequality�x + y�p ≤ 21/p

��x�p + �y�p

�,∀x, y

�x + y�pp ≤ �x�p

p + �y�pp,∀x, y

�x + y�0 ≤ �x�0 + �y�0,∀x, y

= convex

= nonconvex



Optimization problems

• Approximation

• Sparsification

• Regularization

25

minx�b−Ax�2 s.t. �x�p ≤ τ

minx�x�p s.t. �b−Ax�2 ≤ �

minx

12�b−Ax�2 + λ�x�p



Lp “norms” level sets

• Strictly convex when p>1

• Convex p=1 • Nonconvex p<1

26

{x s.t.b = Ax}Observation: the minimizer is sparse



L1 induces sparsity (1)

• Real-valued case✓ A = an m x N real-valued matrix✓ b = an m-dimensional real-valued vector✓ X = set of all minimum L1 norm solutions to

• Fact 1: X is convex and contains a “sparse” solution

28

Ax = b

∃x0 ∈ X, �x0�0 ≤ m

x̃ ∈ X ⇔ �x̃�1 = min �x�1 s.t. Ax = b



Proof ? Exercice!

29



Convex sets

• Definition:

30

θu + (1− θ)v ∈ X, ∀u,v ∈ X, 0 ≤ θ ≤ 1

Convex Not convex



Convex functions

• Definition: f is convex if

31

f(θu + (1− θ)v) ≤ θf(u) + (1− θ)f(v)

∀u,v, 0 ≤ θ ≤ 1

ConvexNot

convex



Proof ? Exercice!

• Convexity of the set of solutions X: ✓ let✓ convexity of constraint

✓ by definition

✓ convexity of objective function

✓ hence

32

Ax = Ax� = A(θx + (1− θ)x�) = b

x, x� ∈ X, 0 ≤ θ ≤ 1

�x�1 = �x��1 = min �x̃�1 s.t. Ax̃ = b

�(θx + (1− θ)x�)�1 ≤ θ�x�1 + (1− θ)�x��1 = �x�1

θx + (1− θ)x� ∈ X



• Existence of a sparse solution✓ let x satisfy with

✦ support

✦ sub-matrix

✓ existence of nontrivial null space vector✓ other solution ✓ for small

Proof? Exercice!

33

I := supp(x) := {i, xi �= 0}

A AI

� ≥ m + 1

→m

N

AIz = 0

��x + �z�1 =

�

i∈I

|xi + �zi| +�

i/∈I

|xi + �zi|

Ax = b �x�0 ≥ m + 1

m

x� = x + �z




✦ support

✦ sub-matrix


Proof? Exercice!

33

I := supp(x) := {i, xi �= 0}

A AI

� ≥ m + 1

→m

N

AIz = 0

��x + �z�1 =

�

i∈I

|xi + �zi| +�

i/∈I

|xi + �zi|

=�

i∈I

sign(xi)(xi + �zi)

Ax = b �x�0 ≥ m + 1

m

x� = x + �z




✦ support

✦ sub-matrix


Proof? Exercice!

33

I := supp(x) := {i, xi �= 0}

A AI

� ≥ m + 1

→m

N

AIz = 0

��x + �z�1 =

�

i∈I

|xi + �zi| +�

i/∈I

|xi + �zi|

=�

i∈I

sign(xi)(xi + �zi) = �x�1 + ��

i∈I

sign(xi)zi

Ax = b �x�0 ≥ m + 1

m

x� = x + �z




✦ support

✦ sub-matrix


Proof? Exercice!

33

I := supp(x) := {i, xi �= 0}

A AI

� ≥ m + 1

→m

N

AIz = 0

��x + �z�1 =

�

i∈I

|xi + �zi| +�

i/∈I

|xi + �zi|

=�

i∈I

sign(xi)(xi + �zi) = �x�1 + ��

i∈I

sign(xi)zi

Ax = b �x�0 ≥ m + 1

m

x� = x + �zcost function

is not minimum



Convexity of the set of minimizers

• Unique solution • Non unique solution

34



L1 induces sparsity (2)

• Real-valued case✓ A = an m x N real-valued matrix✓ b = an m-dimensional real-valued vector✓ X = set of al solutions to regularization problem

• Fact 2: X is a convex set and contains a “sparse” solution

35

∃x0 ∈ X, �x0�0 ≤ m

L(x) :=12�Ax− b�2

2 + λ�x�1

x̃ ∈ X ⇔ L(x̃) = minxL(x)



Proof ? Exercice!

36



L1 induces sparsity

• A word of caution: this does not hold true in the complex-valued case

• Counter example: there is a construction where✓ A = a 2 x 3 complex-valued matrix✓ b = a 2-dimensional complex-valued vector✓ the minimum L1 norm solution is unique and has 3

nonzero components

37

[E. Vincent, Complex Nonconvex Optimization l_p norm minimization for underdetermined source separation, Proc. ICA 2007.]



Global Optimization : from Principles to Algorithms

• Optimization principle

✓ Sparse representation✓ Sparse approximation

local minima convex : global minimum

NP-hard combinatorial Iterative thresholding / proximal algo.FOCUSS / IRLS Linear

Lasso [Tibshirani 1996], Basis Pursuit (Denoising) [Chen, Donoho & Saunders, 1999]

Linear/Quadratic programming (interior point, etc.) Homotopy method [Osborne 2000] / Least Angle Regression [Efron &al 2002]

Iterative / proximal algorithms [Daubechies, de Frise, de Mol 2004, Combettes & Pesquet 2008, ...]

38

λ→ 0λ > 0

minx

12�Ax− b�2

2 + λ�x�pp

Ax = bAx ≈ b


inverse problems and sparse representations€¦ · r. gribonval - sparse methods - sisea 2011...

Documents