signal processing course : inverse problems regularization
DESCRIPTION
Slide for an image processing courseTRANSCRIPT
![Page 1: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/1.jpg)
Inverse ProblemsRegularization
www.numerical-tours.com
Gabriel Peyré
![Page 2: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/2.jpg)
Overview
• Variational Priors
• Gradient Descent and PDE’s
• Inverse Problems Regularization
![Page 3: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/3.jpg)
J(f) = ||f ||2W 1,2 =
Z
R2
||rf(x)||dxSobolev semi-norm:
Smooth and Cartoon Priors
�|�f |2
![Page 4: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/4.jpg)
J(f) = ||f ||2W 1,2 =
Z
R2
||rf(x)||dxSobolev semi-norm:
Total variation semi-norm:
J(f) = ||f ||TV =
Z
R2
||rf(x)||dx
Smooth and Cartoon Priors
�|�f |
�|�f |2
![Page 5: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/5.jpg)
J(f) = ||f ||2W 1,2 =
Z
R2
||rf(x)||dxSobolev semi-norm:
Total variation semi-norm:
J(f) = ||f ||TV =
Z
R2
||rf(x)||dx
Smooth and Cartoon Priors
�|�f |
�|�f |2
![Page 6: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/6.jpg)
Natural Image Priors
![Page 7: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/7.jpg)
Discrete Priors
![Page 8: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/8.jpg)
Discrete Priors
![Page 9: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/9.jpg)
Discrete Differential Operators
![Page 10: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/10.jpg)
Discrete Differential Operators
![Page 11: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/11.jpg)
Laplacian Operator
![Page 12: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/12.jpg)
Laplacian Operator
![Page 13: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/13.jpg)
Function:
f̃ : x 2 R2 7! f(x) 2 R
f̃(x+ ") = f̃(x) + hrf(x), "iR2 +O(||"||2R2)
rf̃(x) = (@1f̃(x), @2f̃(x)) 2 R2
Gradient: Images vs. Functionals
![Page 14: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/14.jpg)
Function:
f̃ : x 2 R2 7! f(x) 2 R
Discrete image: f 2 RN , N = n2
f [i1, i2] = f̃(i1/n, i2/n) rf [i] ⇡ rf̃(i/n)
f̃(x+ ") = f̃(x) + hrf(x), "iR2 +O(||"||2R2)
rf̃(x) = (@1f̃(x), @2f̃(x)) 2 R2
Gradient: Images vs. Functionals
![Page 15: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/15.jpg)
Function:
f̃ : x 2 R2 7! f(x) 2 R
Discrete image: f 2 RN , N = n2
f [i1, i2] = f̃(i1/n, i2/n)
Functional:
J : f 2 RN 7! J(f) 2 RJ(f + ⌘) = J(f) + hrJ(f), ⌘iRN +O(||⌘||2RN )
rf [i] ⇡ rf̃(i/n)
f̃(x+ ") = f̃(x) + hrf(x), "iR2 +O(||"||2R2)
rf̃(x) = (@1f̃(x), @2f̃(x)) 2 R2
rJ : RN 7! RN
Gradient: Images vs. Functionals
![Page 16: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/16.jpg)
Function:
f̃ : x 2 R2 7! f(x) 2 R
Discrete image: f 2 RN , N = n2
f [i1, i2] = f̃(i1/n, i2/n)
Functional:
J : f 2 RN 7! J(f) 2 R
Sobolev:
rJ(f) = (r⇤ � r)f = ��f
J(f) =1
2||rf ||2
J(f + ⌘) = J(f) + hrJ(f), ⌘iRN +O(||⌘||2RN )
rf [i] ⇡ rf̃(i/n)
f̃(x+ ") = f̃(x) + hrf(x), "iR2 +O(||"||2R2)
rf̃(x) = (@1f̃(x), @2f̃(x)) 2 R2
rJ : RN 7! RN
Gradient: Images vs. Functionals
![Page 17: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/17.jpg)
rJ(f) = �div
✓rf
||rf ||
◆If 8n,rf [n] 6= 0,
If 9n,rf [n] = 0, J not di↵erentiable at f .
Total Variation Gradient||rf ||
rJ(f)
![Page 18: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/18.jpg)
rJ(f) = �div
✓rf
||rf ||
◆If 8n,rf [n] 6= 0,
Sub-di↵erential:
If 9n,rf [n] = 0, J not di↵erentiable at f .
Cu =�↵ 2 R2⇥N \ (u[n] = 0) ) (↵[n] = u[n]/||u[n]||)
@J(f) = {� div(↵) ; ||↵[n]|| 6 1 and ↵ 2 Crf}
Total Variation Gradient||rf ||
rJ(f)
![Page 19: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/19.jpg)
−10 −8 −6 −4 −2 0 2 4 6 8 10
−2
0
2
4
6
8
10
12
−10 −8 −6 −4 −2 0 2 4 6 8 10
−2
0
2
4
6
8
10
12
px
2 + "
2
|x|
Regularized Total Variation
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
![Page 20: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/20.jpg)
−10 −8 −6 −4 −2 0 2 4 6 8 10
−2
0
2
4
6
8
10
12
−10 −8 −6 −4 −2 0 2 4 6 8 10
−2
0
2
4
6
8
10
12
rJ"(f) = �div
✓rf
||rf ||"
◆
px
2 + "
2
|x|
rJ" ⇠ ��/" when " ! +1
Regularized Total Variation
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
rJ"(f)
![Page 21: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/21.jpg)
Overview
• Variational Priors
• Gradient Descent and PDE’s
• Inverse Problems Regularization
![Page 22: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/22.jpg)
f (k+1) = f (k) � ⌧krJ(f (k)) f (0) is given.
Gradient Descent
![Page 23: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/23.jpg)
and 0 < ⌧ < 2/L, then f (k) k!+1�! f?
a solution of min
fJ(f).
If f is convex, C1, rf is L-Lipschitz,Theorem:
f (k+1) = f (k) � ⌧krJ(f (k)) f (0) is given.
Gradient Descent
![Page 24: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/24.jpg)
and 0 < ⌧ < 2/L, then f (k) k!+1�! f?
a solution of min
fJ(f).
If f is convex, C1, rf is L-Lipschitz,Theorem:
f (k+1) = f (k) � ⌧krJ(f (k)) f (0) is given.
Optimal step size: ⌧k = argmin⌧2R+
J(f (k) � ⌧rJ(f (k)))
Proposition: One has
hrJ(f (k+1)), rJ(f (k))i = 0
Gradient Descent
![Page 25: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/25.jpg)
Gradient Flows and PDE’sf (k+1) � f (k)
⌧= �rJ(f (k))Fixed step size ⌧k = ⌧ :
![Page 26: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/26.jpg)
Gradient Flows and PDE’sf (k+1) � f (k)
⌧= �rJ(f (k))Fixed step size ⌧k = ⌧ :
Denote ft = f (k)for t = k⌧ , one obtains formally as ⌧ ! 0:
8 t > 0,@ft@t
= �rJ(ft) and f0 = f (0)
![Page 27: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/27.jpg)
J(f) =R||rf(x)||dx
Sobolev flow:
@ft@t
= �ftHeat equation:
Explicit solution:
Gradient Flows and PDE’sf (k+1) � f (k)
⌧= �rJ(f (k))Fixed step size ⌧k = ⌧ :
Denote ft = f (k)for t = k⌧ , one obtains formally as ⌧ ! 0:
8 t > 0,@ft@t
= �rJ(ft) and f0 = f (0)
![Page 28: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/28.jpg)
Total Variation Flow
![Page 29: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/29.jpg)
@ft@t
= �rJ(ft)
Noisy observations: y = f + w, w ⇠ N (0, IdN ).
and ft=0 = y
Application: Denoising
![Page 30: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/30.jpg)
Optimal choice of t: minimize ||ft � f ||
�! not accessible in practice.
SNR(ft, f) = �20 log10
✓||f � ft||
||f ||
◆
Optimal Parameter Selection
t t
![Page 31: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/31.jpg)
Overview
• Variational Priors
• Gradient Descent and PDE’s
• Inverse Problems Regularization
![Page 32: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/32.jpg)
Inverse Problems
![Page 33: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/33.jpg)
Inverse Problems
![Page 34: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/34.jpg)
Inverse Problems
![Page 35: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/35.jpg)
Inverse Problems
![Page 36: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/36.jpg)
Inverse Problems
![Page 37: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/37.jpg)
Inverse Problem Regularization
![Page 38: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/38.jpg)
Inverse Problem Regularization
![Page 39: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/39.jpg)
Inverse Problem Regularization
![Page 40: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/40.jpg)
Sobolev prior: J(f) = 12 ||rf ||2
f? = argminf2RN
E(f) = ||y � �f ||2 + �||rf ||2
(assuming 1 /2 ker(�))
Sobolev Regularization
![Page 41: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/41.jpg)
Sobolev prior: J(f) = 12 ||rf ||2
f? = argminf2RN
E(f) = ||y � �f ||2 + �||rf ||2
rE(f?) = 0 () (�⇤�� ��)f? = �⇤yProposition:
�! Large scale linear system.
(assuming 1 /2 ker(�))
Sobolev Regularization
![Page 42: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/42.jpg)
Sobolev prior: J(f) = 12 ||rf ||2
f? = argminf2RN
E(f) = ||y � �f ||2 + �||rf ||2
rE(f?) = 0 () (�⇤�� ��)f? = �⇤yProposition:
�! Large scale linear system.
Gradient descent:
(assuming 1 /2 ker(�))
where ||A|| = �max
(A)
�! Slow convergence.
Sobolev Regularization
Convergence: ⇥ < 2/||⇥�⇥� ��||
![Page 43: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/43.jpg)
Mask M , � = diagi(1i2M )
Example: Inpainting
since x � [0, 1]2. This distance corresponds to the Euclidean distance over thecube ��1(M), but since c̃f has a complex convoluted geometry, this distanceis not Euclidean when displayed as a 2D image.
Image f Surface c̃f Distance dMFig. 2. Manifold of smooth images.
4.3 Numerical Experiments
Figure 3 shows iterations of the algorithm 1 to solve the inpainting problemon a smooth image using a manifold prior with 2D linear patches, as defined in16. This manifold together with the overlapping of the patches allow a smoothinterpolation of the missing pixels.
Measurements y Iter. #1 Iter. #3 Iter. #50
Fig. 3. Iterations of the inpainting algorithm on an uniformly regular image.
5 Manifold of Step Discontinuities
In order to introduce some non-linearity in the manifoldM, one needs to gobeyond the Fourier world of uniformly regular functions and consider signalsand images with discontinuities.
13
log10(||f (k) � f (�)||/||f0||)
k k
E(f (k))
M
(�f)[i] =
⇢0 if i 2 M,f [i] otherwise.
![Page 44: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/44.jpg)
Symmetric linear system:
Conjugate Gradient
Ax = b () minx2Rn
E(x) = 1
2hAx, xi � hx, bi
![Page 45: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/45.jpg)
Symmetric linear system:
x
(k+1) = argmin E(x)s.t. x� x
(k) 2 span(rE(x(0)), . . . ,rE(x(k)))
Intuition:
Conjugate Gradient
Ax = b () minx2Rn
E(x) = 1
2hAx, xi � hx, bi
Proposition:
8 ` < k, hrE(xk), rE(x`)i = 0
![Page 46: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/46.jpg)
Symmetric linear system:
Initialization:
x
(0) 2 RN, r
(0) = b�Ax
(0), p
(0) = r
(0)
r
(k) =hrE(x(k)), d(k)ihAd(k), d(k)i
d
(k) = rE(x(k)) +||v(k)||
||v(k�1)||d
(k�1)
v
(k) = rE(x(k)) = Ax
(k) � b
x
(k+1) = x
(k) � r
(k)d
(k)
Iterations:
x
(k+1) = argmin E(x)s.t. x� x
(k) 2 span(rE(x(0)), . . . ,rE(x(k)))
Intuition:
Conjugate Gradient
Ax = b () minx2Rn
E(x) = 1
2hAx, xi � hx, bi
Proposition:
8 ` < k, hrE(xk), rE(x`)i = 0
![Page 47: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/47.jpg)
TV" regularization:
(assuming 1 /2 ker(�))
f? = argminf2RN
E(f) = 1
2||�f � y||+ �J"(f)
Total Variation Regularization
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
![Page 48: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/48.jpg)
TV" regularization:
(assuming 1 /2 ker(�))
f (k+1) = f (k) � ⌧krE(f (k))
rE(f) = �⇤(�f � y) + �rJ"(f)
rJ"(f) = �div
✓rf
||rf ||"
◆
Convergence: requires ⌧ ⇠ ".
Gradient descent:
f? = argminf2RN
E(f) = 1
2||�f � y||+ �J"(f)
Total Variation Regularization
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
![Page 49: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/49.jpg)
TV" regularization:
(assuming 1 /2 ker(�))
f (k+1) = f (k) � ⌧krE(f (k))
rE(f) = �⇤(�f � y) + �rJ"(f)
rJ"(f) = �div
✓rf
||rf ||"
◆
Convergence: requires ⌧ ⇠ ".
Gradient descent:
f? = argminf2RN
E(f) = 1
2||�f � y||+ �J"(f)
Newton descent:
f (k+1) = f (k) �H�1k rE(f (k)) where Hk = @2E"(f (k))
Total Variation Regularization
||u||" =p
||u||2 + "2 J"(f) =P
n ||rf [n]||"
![Page 50: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/50.jpg)
k
Large
"TV vs. Sobolev ConvergeSmall"
![Page 51: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/51.jpg)
Observations y Sobolev
Total variation
Inpainting: Sobolev vs. TV
![Page 52: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/52.jpg)
Noiseless problem: f? 2 argminf
J"(f) s.t. f 2 HContraint: H = {f ; �f = y}.
f (k+1)= ProjH
⇣f (k) � ⌧krJ"(f
(k))
⌘
ProjH(f) = argmin
�g=y||g � f ||2 = f + �⇤(�⇤�)�1(y � �f)
Inpainting:ProjH(f)[i] =
⇢f [i] if i 2 M,y[i] otherwise.
Projected gradient descent:
f (k) k!+1�! f?a solution of (?).
(?)
Projected Gradient Descent
Proposition: If rJ" is L-Lipschitz and 0 < ⌧k < 2/L,
![Page 53: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/53.jpg)
TV
Priors:
Non-quadratic
better edge recovery.
=)
Conclusion
Sobolev
![Page 54: Signal Processing Course : Inverse Problems Regularization](https://reader033.vdocument.in/reader033/viewer/2022052600/55866a97d8b42a36488b46f3/html5/thumbnails/54.jpg)
TV
Priors:
Non-quadratic
better edge recovery.
=)
Optimization
Variational regularization:()
– Gradient descent.– Newton.
– Projected gradient.
– Conjugate gradient.
Non-smooth optimization ?
Conclusion
Sobolev