sparse and redundant representations - eth zpeople.math.ethz.ch/~pgrohs/srr/intro.pdf · dhf(i; j)...

Sparse and Redundant RepresentationsPh. GrohsETH Zurich, Seminar for Applied Mathematics

09-27-2012

Introduction

Basic ThemeLet A ∈ Rn×m with n < m.

We would like to solve the linear system

Ax = b, x ∈ Rn, b ∈ Rm.

Clearly not uniquely solvable! Representation of b is redundant!

Ph. Grohs 09-27-2012 p. 2

Introduction

Basic ThemeLet A ∈ Rn×m with n < m. We would like to solve the linear system

Ax = b, x ∈ Rn, b ∈ Rm.


Ph. Grohs 09-27-2012 p. 2

Introduction


Ax = b, x ∈ Rn, b ∈ Rm.

Clearly not uniquely solvable!

Representation of b is redundant!

Ph. Grohs 09-27-2012 p. 2

Introduction


Ax = b, x ∈ Rn, b ∈ Rm.


Ph. Grohs 09-27-2012 p. 2

Introduction

X-Ray CT

Mathematically, measurements are line-integrals

Rf (r , θ) =∫

Lf (x , y)dl

Only available for a few angles θ. Reconstruction of f??

Ph. Grohs 09-27-2012 p. 3

Introduction

X-Ray CT


Rf (r , θ) =∫

Lf (x , y)dl

Only available for a few angles θ.

Reconstruction of f??

Ph. Grohs 09-27-2012 p. 3

Introduction

X-Ray CT


Rf (r , θ) =∫

Lf (x , y)dl

Only available for a few angles θ. Reconstruction of f??Ph. Grohs 09-27-2012 p. 3

Introduction

Projection-Slice TheoremTheorem

FrRf (ω, θ) = f̂ (ω cos(θ), ω sin(θ)).

A measurement Rf (r , θ) for a fixed angle θ gives us access to theFourier transform of f along a ray with polar angle θ.

For 22 different angles we getthe Fourier transform off ∈ R256×256 on the rays in thepicture. This leads to aredundant linear system for f !

Ph. Grohs 09-27-2012 p. 4

Introduction




For 22 different angles we getthe Fourier transform off ∈ R256×256 on the rays in thepicture.

This leads to aredundant linear system for f !

Ph. Grohs 09-27-2012 p. 4

Introduction




For 22 different angles we getthe Fourier transform off ∈ R256×256 on the rays in thepicture. This leads to aredundant linear system for f !

Ph. Grohs 09-27-2012 p. 4

Introduction

Solution Strategies I: PseudoinverseA common strategy to solve Ax = b with A ∈ Rn×m, m < n is to usethe Moore-Penrose pseudoinverse and get a solution

x∗ = A†b

= argminAx=b

‖x‖`2 .

Let’s try it for the X-ray problem.

This is a typical benchmark (Shepp-Logan phantom).

Ph. Grohs 09-27-2012 p. 5

Introduction


x∗ = A†b = argminAx=b

‖x‖`2 .


This is a typical benchmark (Shepp-Logan phantom).

Ph. Grohs 09-27-2012 p. 5

Introduction


x∗ = A†b = argminAx=b

‖x‖`2 .


This is a typical benchmark (Shepp-Logan phantom).Ph. Grohs 09-27-2012 p. 5

Introduction

Solution

Ph. Grohs 09-27-2012 p. 6

Introduction

Solution

Ph. Grohs 09-27-2012 p. 7

Introduction

Solution Strategies II: TV-regularizationPseudoinverse approach seeks for solution with smallest `2-norm.

Why should sought solution have this property?

Define discrete gradient for image f ∈ R256×256

Dhf (i, j) =

{f (i + 1, j) − f (i, j) i < 256

0 i = 256 , Dv f (i, j) =

{f (i, j + 1) − f (i, j) j < 256

0 j = 256 , Df (i, j) =

(Dhf (i, j)Dv f (i, j)

).

vertical gradient of image issparse!

instead of searching for thesolution with minimal `2-norm,look for solution with sparsestgradient!

Ph. Grohs 09-27-2012 p. 8

Introduction

Solution Strategies II: TV-regularizationPseudoinverse approach seeks for solution with smallest `2-norm.Why should sought solution have this property?

Define discrete gradient for image f ∈ R256×256

Dhf (i, j) =

{f (i + 1, j) − f (i, j) i < 256

0 i = 256 , Dv f (i, j) =

{f (i, j + 1) − f (i, j) j < 256

0 j = 256 , Df (i, j) =

(Dhf (i, j)Dv f (i, j)

).

vertical gradient of image issparse!

instead of searching for thesolution with minimal `2-norm,look for solution with sparsestgradient!

Ph. Grohs 09-27-2012 p. 8

Introduction

Solution Strategies II: TV-regularizationFollowing these ideas we solve the sparse optimization problem

x0 = argminAx=b

‖Dx‖0 := # {(i , j) : Dx(i , j) 6= 0} .

/This optimization problem is NP-hard.

Relax, solve instead

x1 = argminAx=b

‖Dx‖1 :=∑i,j

√Dhx(i , j)2 + Dv x(i , j)2 :=

∑i,j

‖Dx(i , j)‖2.

,Can be recast as SOCP (Second Order Cone Program)

mint,x

∑i,j

t(i , j) s.t. Ax = b, ‖Dx(i , j)‖2 ≤ t(i , j)

and solved efficiently.

Ph. Grohs 09-27-2012 p. 9

Introduction


x0 = argminAx=b

‖Dx‖0 := # {(i , j) : Dx(i , j) 6= 0} .


Relax

, solve instead

x1 = argminAx=b

‖Dx‖1 :=∑i,j

√Dhx(i , j)2 + Dv x(i , j)2 :=

∑i,j

‖Dx(i , j)‖2.


mint,x

∑i,j



Ph. Grohs 09-27-2012 p. 9

Introduction


x0 = argminAx=b

‖Dx‖0 := # {(i , j) : Dx(i , j) 6= 0} .



x1 = argminAx=b

‖Dx‖1

:=∑i,j

√Dhx(i , j)2 + Dv x(i , j)2 :=

∑i,j

‖Dx(i , j)‖2.


mint,x

∑i,j



Ph. Grohs 09-27-2012 p. 9

Introduction


x0 = argminAx=b

‖Dx‖0 := # {(i , j) : Dx(i , j) 6= 0} .



x1 = argminAx=b

‖Dx‖1 :=∑i,j

√Dhx(i , j)2 + Dv x(i , j)2

:=∑i,j

‖Dx(i , j)‖2.


mint,x

∑i,j



Ph. Grohs 09-27-2012 p. 9

Introduction


x0 = argminAx=b

‖Dx‖0 := # {(i , j) : Dx(i , j) 6= 0} .



x1 = argminAx=b

‖Dx‖1 :=∑i,j

√Dhx(i , j)2 + Dv x(i , j)2 :=

∑i,j

‖Dx(i , j)‖2.


mint,x

∑i,j



Ph. Grohs 09-27-2012 p. 9

Introduction


x0 = argminAx=b

‖Dx‖0 := # {(i , j) : Dx(i , j) 6= 0} .



x1 = argminAx=b

‖Dx‖1 :=∑i,j

√Dhx(i , j)2 + Dv x(i , j)2 :=

∑i,j

‖Dx(i , j)‖2.


mint,x

∑i,j


and solved efficiently.Ph. Grohs 09-27-2012 p. 9

Introduction

Solution

Original image is reconstructed exactly!

Ph. Grohs 09-27-2012 p. 10

Introduction

Summary

Redundant system Ax = b can be solved exactly for sparse solutionsx .

Ph. Grohs 09-27-2012 p. 11

Introduction

Learning Objectives

Ability to...

◦ understand theoretical analysis of sparse optimizationalgorithms,

◦ critique current research publications,◦ implement basic models and methods in signal processing,◦ summarize and explain research publications in the field of

sparse optimization.

Ph. Grohs 09-27-2012 p. 12

Introduction

Learning Objectives

Ability to...◦ understand theoretical analysis of sparse optimization

algorithms,

◦ critique current research publications,◦ implement basic models and methods in signal processing,◦ summarize and explain research publications in the field of


Ph. Grohs 09-27-2012 p. 12

Introduction

Learning Objectives


algorithms,◦ critique current research publications,

◦ implement basic models and methods in signal processing,◦ summarize and explain research publications in the field of


Ph. Grohs 09-27-2012 p. 12

Introduction

Learning Objectives


algorithms,◦ critique current research publications,◦ implement basic models and methods in signal processing,

◦ summarize and explain research publications in the field ofsparse optimization.

Ph. Grohs 09-27-2012 p. 12

Introduction

Learning Objectives


algorithms,◦ critique current research publications,◦ implement basic models and methods in signal processing,◦ summarize and explain research publications in the field of


Ph. Grohs 09-27-2012 p. 12

Introduction

Main Literature Source

Ph. Grohs 09-27-2012 p. 13

Introduction

Modus Operandi

Interactivity!

Everybody is required to...give a ∼ 50 minutes talk on 1-2 Chapters of Elad’s book...attend the talks of others...critique and discuss the talks of others (each time with adifferent focus, e.g., logical coherence, body language, slides,speech)

Ph. Grohs 09-27-2012 p. 14

Introduction

Modus Operandi

Interactivity!

Everybody is required to

...give a ∼ 50 minutes talk on 1-2 Chapters of Elad’s book

...attend the talks of others

...critique and discuss the talks of others (each time with adifferent focus, e.g., logical coherence, body language, slides,speech)

Ph. Grohs 09-27-2012 p. 14

Introduction

Modus Operandi

Interactivity!

Everybody is required to...give a ∼ 50 minutes talk on 1-2 Chapters of Elad’s book

...attend the talks of others


Ph. Grohs 09-27-2012 p. 14

Introduction

Modus Operandi

Interactivity!

Everybody is required to...give a ∼ 50 minutes talk on 1-2 Chapters of Elad’s book...attend the talks of others


Ph. Grohs 09-27-2012 p. 14

Introduction

Modus Operandi

Interactivity!

Everybody is required to...give a ∼ 50 minutes talk on 1-2 Chapters of Elad’s book...attend the talks of others...critique and discuss the talks of others (each time with adifferent focus, e.g., logical coherence, body language, slides,speech)

Ph. Grohs 09-27-2012 p. 14

Introduction

Hints

When giving a talk

...practice the right speed

...think thoroughly about the balance detail vs. big picture

...keep a “red thread” throughout the talk

...give a “takehome message”

Ph. Grohs 09-27-2012 p. 15

Introduction

Hints

When giving a talk...practice the right speed

...think thoroughly about the balance detail vs. big picture



Ph. Grohs 09-27-2012 p. 15

Introduction

Hints

When giving a talk...practice the right speed...think thoroughly about the balance detail vs. big picture



Ph. Grohs 09-27-2012 p. 15

Introduction

Hints

When giving a talk...practice the right speed...think thoroughly about the balance detail vs. big picture...keep a “red thread” throughout the talk


Ph. Grohs 09-27-2012 p. 15

Introduction

Hints

When giving a talk...practice the right speed...think thoroughly about the balance detail vs. big picture...keep a “red thread” throughout the talk...give a “takehome message”

Ph. Grohs 09-27-2012 p. 15

sparse and redundant representations - eth zpeople.math.ethz.ch/~pgrohs/srr/intro.pdf · dhf(i; j)...

Documents