sparse & redundant signal representation and its role in image processing michael elad the...
Post on 19-Dec-2015
216 views
TRANSCRIPT
![Page 1: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/1.jpg)
xD
Sparse & Redundant Signal Representation and its Role in Image
Processing
Michael Elad The Computer Science Department The Technion – Israel Institute of
technology Haifa 32000, Israel
Mathematics and Image Analysis, MIA'06Paris, 18-21 September , 2006
![Page 2: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/2.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
2/56xD
Today’s Talk is About
Sparsity and
RedundancyWe will try to show today that
• Sparsity & Redundancy can be used to design new/renewed & powerful signal/image processing tools.
• We will present both the theory behind sparsity and redundancy, and how those are deployed to image processing applications.
• This is an overview lecture, describing the recent activity in this field.
![Page 3: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/3.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
3/56xD
Agenda
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
Welcome to Sparse
land
![Page 4: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/4.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
4/56xD
Generating Signals in Sparseland
MK
N
DA fixed Dictionary
•Every column in D (dictionary) is a prototype signal (Atom).
•The vector is generated randomly with few non-zeros in random locations and random values.
A sparse & random vector
αx
N
![Page 5: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/5.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
5/56xD
• Simple: Every generated signal is built as a linear combination of few atoms from our dictionary D
• Rich: A general model: the obtained signals are a special type mixture-of-Gaussians (or Laplacians).
Sparseland Signals Are Special
Multiply by D
αDx
Mα
![Page 6: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/6.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
6/56xD
D
Transforms in Sparseland ?
Nx
MαD
M• Assume that x is known to emerge from . .
• We desire simplicity, independence, and expressiveness.
M• How about “Given x, find the α that generated it in ” ?
KαT
![Page 7: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/7.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
7/56xD
Known
So, In Order to Transform …
We need to solve an under-determined linear system of equations:
xαD
• We will measure sparsity using the L0 norm:
0α
• Among all (infinitely many) possible solutions we want the sparsest !!
![Page 8: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/8.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
8/56xD
-1 +1
1
piif
i
Measure of Sparsity?
k
1j
pj
pp 1
122
1p
pp
0p
pp
As p 0 we get a count of the non-zeros in the vector
0α
![Page 9: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/9.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
9/56xD
M
α̂• Are there practical ways to get ?
Signal’s Transform in Sparseland
α
αα
Dx.t.s
Min0 α̂Multipl
y by D
αDx
A sparse &
random vector
α
4 Major Questions
αα ˆ• Is ? Under which conditions?
• How effective are those ways?
• How would we get D?
![Page 10: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/10.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
10/56xD
Inverse Problems in Sparseland ?
• Assume that x is known to emerge from . M
MαD
Nx
vxy H• Suppose we observe , a “blurred” and noisy version of x with . How will we recover x?
ε2
v
xHMy
Noise
• How about “find the α that generated the x …” again?
Kˆ αQ
![Page 11: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/11.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
11/56xD
M
Inverse Problems in Sparseland ?
Multiply by D
αDxA sparse &
random vector
α “blur” by H
vxy H
v
4 Major Questions (again!)
• How would we get D?
α̂• Are there practical ways to get ?• How effective are those ways?
αα ˆ• Is ? How far can it go?
εα
αα
2
0
y
.t.sMin
HDα̂
![Page 12: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/12.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
12/56xD
Back Home … Any Lessons?
Several recent trends worth looking at:
JPEG to JPEG2000 - From (L2-norm) KLT to wavelet and non-linear approximation
From Wiener to robust restoration – From L2-norm (Fourier) to L1. (e.g., TV, Beltrami, wavelet shrinkage …) From unitary to richer representations – Frames, shift-invariance, steerable wavelet, contourlet, curvelet
Approximation theory – Non-linear approximation
ICA and related models
Sparsity.
Overcompleteness.
Sparsity.
Sparsity & Overcompleteness.
Independence and Sparsity.
Sparsel
and is
HERE
![Page 13: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/13.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
13/56xD
To Summarize so far …
We do! it is relevant to us
Who cares?
So, what are the
implications?
The Sparseland
model for signals is very
interesting
We need to answer the 4 questions posed, and then show that all this works well in practice
![Page 14: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/14.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
14/56xD
Agenda
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
T
Q
![Page 15: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/15.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
15/56xD
Lets Start with the Transform …
known
ααα
Dx.t.sMin:P00
Put formally,
Our dream for Now: Find the sparsest solution of
Known
xαD
![Page 16: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/16.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
16/56xD
Multiply by D
αDx
α
M Question 1 – Uniqueness?
α
αα
Dx.t.s
Min0 α̂
Suppose we can
solve this exactly
αα ˆWhy should we necessarily get ?
It might happen that eventually .00ˆ αα
![Page 17: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/17.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
17/56xD
Matrix “Spark”
Rank = 4Rank = 4
Spark = 3Spark = 3
1 0 0 0 1
0 1 0 0 1
0 0 1 0 0
0 0 0 1 0
Example:
Donoho & E. (‘02)
Definition: Given a matrix D, =Spark{D} is the smallest and and number of columns that are linearly dependent.
* In tensor decomposition, Kruskal defined something similar already in 1989.
*
![Page 18: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/18.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
18/56xD
Uniqueness Rule
ααα
Dx.t.sMin:P00
Suppose this problem has been solved somehow
This result implies that if generates
signals using “sparse enough” , the solution of the above will find it exactly.
M
If we found a representation that satisfy
Then necessarily it is unique (the sparsest).
02α
σ
Uniqueness
Donoho & E. (‘02)
![Page 19: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/19.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
19/56xD
α̂Are there reasonable ways to find ?
M
Question 2 – Practical P0 Solver?
α
αα
Dx.t.s
Min0 α̂Multipl
y by D
αDx
α
![Page 20: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/20.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
20/56xD
Matching Pursuit (MP)
Mallat & Zhang (1993)
• Next steps: given the previously found atoms, find the next one to best fit …
• The Orthogonal MP (OMP) is an improved version that re-evaluates the coefficients after each round.
• The MP is a greedy algorithm that finds one atom at a time.
• Step 1: find the one atom that best matches the signal.
![Page 21: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/21.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
21/56xD
Basis Pursuit (BP)
ααα
Dx.t.sMin0
Instead of solving
ααα
Dx.t.sMin1
Solve Instead
Chen, Donoho, & Saunders (1995)
• The newly defined problem is convex (linear programming).
• Very efficient solvers can be deployed: Interior point methods [Chen, Donoho, & Saunders (`95)] ,
Sequential shrinkage for union of ortho-bases [Bruce et.al. (`98)],
Iterated shrinkage [Figuerido & Nowak (`03), Daubechies, Defrise, &
Demole (‘04), E. (`05), E., Matalon, & Zibulevsky (`06)].
![Page 22: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/22.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
22/56xD
Multiply by D
αDx
α
M Question 3 – Approx. Quality?
α
αα
Dx.t.s
Min0 α̂
α̂How effective are the MP/BP in finding ?
![Page 23: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/23.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
23/56xD
Evaluating the “Spark”
• The Mutual Coherence M is the largest off-diagonal entry in absolute value.
DT
=D
DTD
• Compute
• The Mutual Coherence is a property of the dictionary (just like the “Spark”). In fact, the following relation can be shown:
M1
1
Assume normalized
columns
![Page 24: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/24.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
24/56xD
BP and MP Equivalence
Given a signal x with a representation ,
Assuming that , BP and MP are
Guaranteed to find the sparsest solution.
αDx
M115.00
αDonoho & E. (‘02) Gribonval & Nielsen
(‘03)Tropp (‘03)
Temlyakov (‘03)
Equivalence
MP and BP are different in general (hard to say which is better).
The above result corresponds to the worst-case.
Average performance results are available too, showing much better bounds [Donoho (`04), Candes et.al. (`04), Tanner et.al. (`05),
Tropp et.al. (`06), Tropp (’06)].
![Page 25: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/25.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
25/56xD
What About Inverse Problems?
εα
αα
2
0
y
.t.sMin
HDα̂“blur”
by H
vxy H
v
• We had similar questions regarding uniqueness, practical solvers, and their efficiency.
• It turns out that similar answers are applicable here due to several recent works [Donoho, E. and Temlyakov (`04), Tropp (`04),
Fuchs (`04), Gribonval et. al. (`05)].
Multiply by D
αDxA sparse &
random vector
α
![Page 26: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/26.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
26/56xD
To Summarize so far …
The Sparseland model is relevant to us.
We can design transforms and priors
based on it
Use pursuit Algorithms
Find the sparsest solution?
A sequence of works during the
past 3-4 years gives theoretic justifications for
these tools behavior
Why works so well?
What next?(a) How shall we find
D?
(b) Will this work for applications? Which?
![Page 27: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/27.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
27/56xD
Agenda
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
![Page 28: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/28.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
28/56xD
Problem Setting
Multiply by D
αDx
Mα
L0
α
Given these P examples and a fixed size [NK] dictionary D, how would we find D?
P1jjX
![Page 29: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/29.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
29/56xD
Uniqueness?
If is rich enough* and if
then D is unique.
2
SparkL
D
Uniqueness
Aharon, E., & Bruckstein (`05)
Comments:
M
L
K• “Rich Enough”: The signals from could be clustered to groups that share the same support. At least L+1 examples per each are needed.• This result is proved constructively, but the number of examples needed to pull this off is huge – we will show a far better method next. • A parallel result that takes into account noise is yet to be constructed.
P1jjx
![Page 30: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/30.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
30/56xD
Each example is a linear combination of atoms from
D
Practical Approach – Objective
DX A
Each example has a sparse representation with no more than L
atoms
L,j.t.sxMin0j
P
1j
2
2jj,
DAD
Field & Olshausen (96’)Engan et. al. (99’)
Lewicki & Sejnowski (00’)Cotter et. al. (03’)
Gribonval et. al. (04’)(n,K,L are assumed known, D has norm. columns)
![Page 31: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/31.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
31/56xD
K–Means For Clustering
DInitialize D
Sparse Coding
Nearest Neighbor
Dictionary Update
Column-by-Column by Mean computation over the relevant
examples
XT
Clustering: An extreme sparse coding
![Page 32: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/32.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
32/56xD
The K–SVD Algorithm – General
DInitialize D
Sparse CodingUse MP or BP
Dictionary Update
Column-by-Column by SVD computation
Aharon, E., & Bruckstein (`04)
XT
![Page 33: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/33.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
33/56xD
K–SVD Sparse Coding Stage
D
X
L,j.t.sxMin0j
P
1j
2
2jj
ααDA
For the jth example we solve
L.t.sxMin0
2
2j ααα
D
Pursuit Problem !!!
T
![Page 34: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/34.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
34/56xD
K–SVD Dictionary Update Stage
DGk : The examples in and that use the column dk.
P
1jjX
?dk
The content of dk influences only the examples in Gk.
Let us fix all A apart from the kth column and seek both dk and the kth column to better fit the residual!
![Page 35: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/35.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
35/56xD
dk is obtained by SVD on the examples’ residual in Gk.
K–SVD Dictionary Update Stage
D?dk Residu
alE
kα
We should solve:
2
FkTk,d
dMinkk
Eαα
![Page 36: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/36.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
36/56xD
K–SVD: A Synthetic Experiment
D
Create A 2030 random dictionary with normalized
columns
Generate 2000 signal examples with 3 atoms per each and add
noise
0 5 10 15 20 25 30 35 40 45 500
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Iteration
Rel
ativ
e A
tom
s F
ound
MOD performance
K-SVD performance
Results
D
Train a dictionary using the KSVD and MOD and compare
![Page 37: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/37.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
37/56xD
To Summarize so far …
The Sparseland model for signals is
relevant to us. In order to use it effectively we
need to know D
Use the K-SVD algorithm
How D can be found?
We have shown how to
practically train D using the K-SVD
Will it work well ?
What next?Show that all the above can be deployed to applications
![Page 38: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/38.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
38/56xD
Agenda
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
![Page 39: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/39.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
39/56xD
Application 1: Image Inpainting Assume: the signal x has been created
by x=Dα0 with very sparse α0.
Missing values in x imply missing rows in this linear system.
By removing these rows, we get .
Now solve
If α0 was sparse enough, it will be the solution of the
above problem! Thus, computing Dα0 recovers x perfectly.
x0 D
=x~~
D
D~
x~.t.s0Min
![Page 40: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/40.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
40/56xD
Application 1: Side Note Compressed Sensing is leaning on the very same
principal, leading to alternative sampling theorems.
Assume: the signal x has been created by x=Dα0 with very sparse α0. Multiply this set of equations by the matrix Q which reduces the number of rows.
The new, smaller, system of equations is
x~~
x DQQD If α0 was sparse enough, it will be the sparsest
solution of the new system, thus, computing Dα0 recovers x perfectly.
Compressed sensing focuses on conditions for this to happen, guaranteeing such recovery.
![Page 41: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/41.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
41/56xD
Application 1: The Practice
2
2pp
yArgMinˆ
D
Given a noisy image y, we can clean it using the Maximum A-posteriori Probability estimator by [Chen, Donoho, & Saunders
(‘95)].
What if some of the pixels in that image are missing (filled with zeros)? Define a mask operator as the diagonal matrix W, and now solve instead
ˆx̂ D
2
2pp
yArgMinˆ
WD ˆx̂ D
When handling general images, there is a need to concatenate two dictionaries to get an effective treatment of both texture and cartoon contents – This leads to separation [E., Starck, & Donoho (’05)].
![Page 42: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/42.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
42/56xD
Inpainting Results
Source
Outcome
Predetermined dictionary: Curvelet (cartoon) + Overlapped DCT (texture)
![Page 43: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/43.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
43/56xD
Inpainting Results
More about this application will be given in Jean-Luc Starck talk that follows.
![Page 44: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/44.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
44/56xD
The solution: Force shift-invariant sparsity - on each patch of size N-by-N (e.g., N=8) in the image, including overlaps.
Application 2: Image Denoising
Given a noisy image y, we have already mentioned the ability to clean it by solving
When using the K-SVD, it cannot train a dictionary for large support images – How do we go from local treatment of patches to a global prior?
2
2pp
yArgMinˆ
D ˆx̂ D
![Page 45: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/45.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
45/56xD
Our prior
Extracts a patch in the ij
location
L.t.s
xyx21
ArgMinx̂
0
0ij
ij
2
2ijij2
2}{,x ijij
DR
Application 2: Image Denoising
2
2pp
yArgMinˆ
D ˆx̂ D
Our MAP penalty
becomes
![Page 46: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/46.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
46/56xD
K-SVD
L.t.sxyxArgMinx̂0
0ijij
2
2ijij222
1
,}{,x ijij
DRD
x=y and D known
L.t.s
xMin
00
2
2ijij
DR
Compute ij per patch
using the matching pursuit
x and ij known
ij
2
2ijxMin DR
Compute D to minimize
using SVD, updating one column at a time
D and ij known
ijij
Tij
1
ijij
Tij yIx DRRR
Compute x by
which is a simple averaging of shifted
patches
Application 2: Image Denoising
D?
The dictionary (and thus the image prior) is trained on the corrupted itself!
This leads to an elegant fusion of the K-SVD and the denoising tasks.
![Page 47: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/47.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
47/56xD
Sparse-code every patch of 8-by-8 pixels
Application 2: The Algorithm
Update the dictionary based
on the above codes
D
Compute the output image x
The computational cost of this algorithm is mostly due to the OMP steps done on each image patch - O(N2×L×K×Iterations) per pixel.
![Page 48: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/48.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
48/56xD
Initial dictionary (overcomplete DCT)
64×256
Denoising ResultsSource
Result 30.829dB
The obtained dictionary after 10 iterations
Noisy image
20
The results of this algorithm tested over a corpus of images and with various noise
powers compete favorably with the state-of-the-art - the GSM+steerable wavelets denoising algorithm by [Portilla, Strela,
Wainwright, & Simoncelli (‘03)], giving ~1dB better results on average.
![Page 49: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/49.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
49/56xD
Application 3: Compression
The problem: Compressing photo-ID images.
General purpose methods (JPEG, JPEG2000) do not take into account the specific family.
By adapting to the image-content (PCA/K-SVD), better results could be obtained.
For these techniques to operate well, train dictionaries locally (per patch) using a training set of images is required.
In PCA, only the (quantized) coefficients are stored, whereas the K-SVD requires storage of the indices as well.
Geometric alignment of the image is very helpful and should be done.
![Page 50: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/50.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
50/56xD
Application 3: The Algorithm
Training set (2500 images)
Detect main features and warp the images to a
common reference (20 parameters)
On
the tra
inin
g se
t
Divide the image into disjoint 15-by-15 patches.
For each compute mean and dictionary
Per each patch find the operating parameters (number of atoms L,
quantization Q)
Warp, remove the mean from each patch, sparse
code using L atoms, apply Q, and dewarp
On the test image
![Page 51: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/51.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
51/56xD
Compression Results
Results for 820
Bytes per
each file
11.99
10.83
10.93
10.49
8.92
8.71
8.81
7.89
8.61
5.56
4.82
5.58
![Page 52: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/52.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
52/56xD
Compression Results
Results for 550
Bytes per
each file
15.81
14.67
15.30
13.89
12.41
12.57
10.66
9.44
10.27
6.60
5.49
6.36
![Page 53: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/53.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
53/56xD
Compression Results
Results for 400
Bytes per
each file
18.62
16.12
16.81
12.30
11.38
12.54
7.61
6.31
7.20
?
?
?
![Page 54: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/54.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
54/56xD
Today We Have Discussed
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
![Page 55: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/55.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
55/56xD
Summary
Sparsity and Over- completeness are
important ideas. Can be used in designing better
tools in signal/image processing
This is an on-going work:• Deepening our theoretical understanding, • Speedup of pursuit methods,• Training the dictionary,• Demonstrating applications,• …
There are difficulties in using
them!
Future transforms and regularizations should be data-driven, non-linear, overcomplete, and promoting sparsity.
The dream?
![Page 56: Sparse & Redundant Signal Representation and its Role in Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d3a5503460f94a146d1/html5/thumbnails/56.jpg)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
56/56xD
Thank You for Your Time
More information, including these slides, can be found in my web-
page
http://www.cs.technion.ac.il/~elad
THE END !!