![Page 1: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/1.jpg)
Some topics on Sparsity
Sangnam Nam
August 12, 2010
![Page 2: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/2.jpg)
Why Sparsity?: randomly generated image
![Page 3: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/3.jpg)
Why Sparsity?: randomly generated image
![Page 4: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/4.jpg)
Why Sparsity?: randomly generated image
![Page 5: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/5.jpg)
Why Sparsity?: randomly generated image
![Page 6: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/6.jpg)
Why Sparsity?: randomly generated image
![Page 7: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/7.jpg)
Why Sparsity?: randomly generated image
![Page 8: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/8.jpg)
Why Sparsity?: DCT of randomly generated image
![Page 9: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/9.jpg)
Why Sparsity?: DCT of randomly generated image
![Page 10: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/10.jpg)
Compression: Image
![Page 11: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/11.jpg)
Compression: Image
![Page 12: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/12.jpg)
Compression: Audio
![Page 13: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/13.jpg)
Denoising
![Page 14: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/14.jpg)
Denoising
![Page 15: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/15.jpg)
Denoising: Simple example
signal x dct transform of x
![Page 16: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/16.jpg)
Denoising: Simple example
signal + noise dct transform of (signal + noise)
![Page 17: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/17.jpg)
Denoising: Simple example
Denoising with hard thresholding at 0.07
![Page 18: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/18.jpg)
Denoising: Simple example
Denoising with hard thresholding at 0.18
![Page 19: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/19.jpg)
Denoising: Simple example
With augmented dictionary:
Transform coefficients Denoised Result
![Page 20: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/20.jpg)
Inpainting: Original Image
![Page 21: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/21.jpg)
Inpainting: letter
![Page 22: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/22.jpg)
Inpainting: letter
![Page 23: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/23.jpg)
Inpainting: letter
![Page 24: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/24.jpg)
Inpainting: lines
![Page 25: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/25.jpg)
Inpainting: lines
![Page 26: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/26.jpg)
Inpainting: lines
![Page 27: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/27.jpg)
Inpainting: random
![Page 28: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/28.jpg)
Inpainting: random
![Page 29: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/29.jpg)
Inpainting: random
![Page 30: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/30.jpg)
Compressed Sensing
![Page 31: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/31.jpg)
Compressed Sensing
![Page 32: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/32.jpg)
Compressed Sensing
![Page 33: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/33.jpg)
Compressed Sensing
![Page 34: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/34.jpg)
Compressed Sensing
![Page 35: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/35.jpg)
Compressed Sensing
![Page 36: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/36.jpg)
Separation
![Page 37: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/37.jpg)
Separation
![Page 38: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/38.jpg)
Super-resolution
![Page 39: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/39.jpg)
Super-resolution
![Page 40: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/40.jpg)
Notation/Terminology
I y ∈ Rd : signal
I φ ∈ Φ: a column of Φ ∈ Rd×K
I φ : φ ∈ Φ spans Rd ⇒ Φ : a dictionary for Rd
I φ ∈ Φ: an atom
I y = Φx ⇒ x : a coefficient vector / representation
![Page 41: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/41.jpg)
Sparse Model
![Page 42: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/42.jpg)
`0-minimization
To find the sparsest representation of y :we solve
minx‖x‖0 subject to y = Mx
How do we know we have the solution?
![Page 43: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/43.jpg)
`0-minimization
To find the sparsest representation of y :we solve
minx‖x‖0 subject to y = Mx
How do we know we have the solution?
![Page 44: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/44.jpg)
Uniqueness of Sparse Solutions
I Let ‖x‖0 := #i : xi 6= 0.
I Let Spark of Φ := minx∈ker(Φ),x 6=0 ‖x‖0.
I Let y = Mx , ‖x‖0 ≤ (Spark of Φ)/2
⇒ x is unique solution of minz ‖z‖0 subject to y = Mz .
![Page 45: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/45.jpg)
Uniqueness of Sparse Solutions
I Let ‖x‖0 := #i : xi 6= 0.I Let Spark of Φ := minx∈ker(Φ),x 6=0 ‖x‖0.
I Let y = Mx , ‖x‖0 ≤ (Spark of Φ)/2
⇒ x is unique solution of minz ‖z‖0 subject to y = Mz .
![Page 46: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/46.jpg)
Uniqueness of Sparse Solutions
I Let ‖x‖0 := #i : xi 6= 0.I Let Spark of Φ := minx∈ker(Φ),x 6=0 ‖x‖0.
I Let y = Mx , ‖x‖0 ≤ (Spark of Φ)/2
⇒ x is unique solution of minz ‖z‖0 subject to y = Mz .
![Page 47: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/47.jpg)
Uniqueness of Sparse Solutions
I Let ‖x‖0 := #i : xi 6= 0.I Let Spark of Φ := minx∈ker(Φ),x 6=0 ‖x‖0.
I Let y = Mx , ‖x‖0 ≤ (Spark of Φ)/2
⇒ x is unique solution of minz ‖z‖0 subject to y = Mz .
![Page 48: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/48.jpg)
`0-minimization
Good, we (kind of) know when we have the solution.
Unfortunately, solving `0-minimization is NP-hard.What to do?
![Page 49: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/49.jpg)
`0-minimization
Good, we (kind of) know when we have the solution.Unfortunately, solving `0-minimization is NP-hard.
What to do?
![Page 50: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/50.jpg)
`0-minimization
Good, we (kind of) know when we have the solution.Unfortunately, solving `0-minimization is NP-hard.What to do?
![Page 51: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/51.jpg)
Alternatives
I Greedy Algorithm/Matching Pursuit
I Convex Optimization/Basis Pursuit
![Page 52: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/52.jpg)
Matching Pursuit
Matching Pursuit [Mallat & Zhang]
1. Given signal y , dictionary Φ.
2. Set k = 0, y0 = 0, r0 = y .
3. Find φ := arg minφ∈Φ〈φ, rk〉.4. Update yk+1 := yk + 〈rk , φ〉φ, rk+1 := rk − 〈rk , φ〉φ.
5. Increment k .
6. If yk uses a specified number of atoms or rk is smaller than aspecified error, stop. Otherwise, go to step 3.
![Page 53: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/53.jpg)
Energy Preservation
Energy Preservation (Pythagoras theorem)
‖rk‖22 = ‖rk+1‖2
2 + |〈rk , φ〉|2
![Page 54: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/54.jpg)
Drawback of Matching Pursuit?
MP can select an atom more than once.
![Page 55: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/55.jpg)
Orthogonal Matching Pursuit
Orthogonal Matching Pursuit (OMP)
1. Given signal y , dictionary Φ.
2. Set k = 0, Λ = , y0 = 0, r0 = y .
3. Find ik := arg mini 〈φi , rk〉 and set Λ := Λ ∪ ik.4. Compute ∆y := (best `2-approximation from span of ΦΛ to
rk), update yk+1 := yk + ∆y , and rk+1 := rk −∆y .
5. Increment k .
6. If yk uses a specified number of atoms or rk is smaller than aspecified error, stop. Otherwise, go to step 3.
![Page 56: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/56.jpg)
Variety of Matching Pursuits
I Morphological Component Analysis [MCA, Bobin et al]
I Stagewise OMP [Donoho et al]
I CoSAMP [Needell & Tropp]
I ROMP [Needell & Vershynin]
I Iterative Hard Thresholding [Blumensath & Davies]
![Page 57: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/57.jpg)
Sampling Issue
To find good approximation of f : [0, 5]→ R, one may sample thefunction values
1. Uniformly at N locations for some large N, or
2. Adaptively sample–more locations near 5 in the figure– atsome large number N locations,
then, use linear interpolation, cubic spline approximation, etc.
![Page 58: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/58.jpg)
What if it is a polynomial of degree 6?
What we know it is a polynomial of degree 6? One needs only 7different samples! Approximation is perfect!
![Page 59: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/59.jpg)
What if it is a sum of 6 monomials?
What if it is a sum of 6 monomials of degree less than 200? Howmany samples do we need?
![Page 60: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/60.jpg)
Think about Economy!
Digital Camera
1. High resolution photo (e.g. size 1024x1024) requires largenumber of samples (e.g. 1024x1024 samples).
2. Image is (immediately) compressed to smaller and lossy JPEGfile.
3. Not a big deal for digital cameras?
![Page 61: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/61.jpg)
Shannon Sampling / Nyquist Rate
Nyquist rate ⇒ 44.1 KHz sampling rate
⇒ large audio file⇒ Compress to lossy MP3 file.
![Page 62: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/62.jpg)
Shannon Sampling / Nyquist Rate
Nyquist rate ⇒ 44.1 KHz sampling rate⇒ large audio file
⇒ Compress to lossy MP3 file.
![Page 63: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/63.jpg)
Shannon Sampling / Nyquist Rate
Nyquist rate ⇒ 44.1 KHz sampling rate⇒ large audio file⇒ Compress to lossy MP3 file.
![Page 64: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/64.jpg)
Fewer Samples
I Can we sample less to begin with?
I How do we recover/approximate the original data?
I Key idea to exploit: Sparsity
![Page 65: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/65.jpg)
Fewer Samples
I Can we sample less to begin with?
I How do we recover/approximate the original data?
I Key idea to exploit: Sparsity
![Page 66: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/66.jpg)
Fewer Samples
I Can we sample less to begin with?
I How do we recover/approximate the original data?
I Key idea to exploit: Sparsity
![Page 67: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/67.jpg)
`1-minimization
Relaxminx‖x‖0 subject to y = Φx
Solveminx‖x‖1 subject to y = Φx
Convex problem. Can be recast into Linear Programming.
![Page 68: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/68.jpg)
`1-minimization
Relaxminx‖x‖0 subject to y = Φx
Solveminx‖x‖1 subject to y = Φx
Convex problem. Can be recast into Linear Programming.
![Page 69: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/69.jpg)
Why does `1-minimization work?
Feasible solutions
![Page 70: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/70.jpg)
Why does `1-minimization work?
`2-balls `1-balls `p-balls, 0 < p < 1
![Page 71: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/71.jpg)
Why does `1-minimization work?
`2-minimizer `1-minimizer `p-minimizer, 0 < p < 1
![Page 72: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/72.jpg)
Why does `1-minimization work?
`2-minimizer `1-minimizer `p-minimizer, 0 < p < 1
When can we guarantee successful recovery?
![Page 73: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/73.jpg)
Null Space Property
Notation:N(Φ) := (Kernel / Null Space of Φ)Λ := (Support of x∗)Λ := Λc
![Page 74: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/74.jpg)
Null Space Property
minx‖x‖1 s.t. Φx = Φx∗ (A)
`1-minimization (A) recovers x∗ if and only if
|〈z , sign(x∗)〉| < ‖zΛ‖1
for all z ∈ N(Φ).
![Page 75: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/75.jpg)
Null Space Property
minx‖x‖1 s.t. Φx = Φx∗ (A)
`1-minimization (A) recovers x∗ if and only if
|〈z , sign(x∗)〉| < ‖zΛ‖1
`1-minimization (A) recovers every x∗ with support Λ if
‖zΛ‖1 < ‖zΛ‖1
for all z ∈ N(Φ).
![Page 76: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/76.jpg)
Null Space Property
minx‖x‖1 s.t. Φx = Φx∗ (A)
`1-minimization (A) recovers x∗ if and only if
|〈z , sign(x∗)〉| < ‖zΛ‖1
`1-minimization (A) recovers every x∗ with support Λ if
‖zΛ‖1 < ‖zΛ‖1
`1-minimization (A) recovers every k-sparse x∗ if
‖zΛ‖1 < ‖zΛ‖1
for all z ∈ N(Φ) and Λ with #Λ ≤ k .
![Page 77: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/77.jpg)
Uniqueness for `1 using NSP
If x∗ is k-sparse, then
‖x∗ + z‖1 ≥ ‖x∗‖1 − ‖zΛ‖1 + ‖zΛ‖1
≥ ‖x∗‖1 .
I.e., x∗ is the unique `1-minimizer.
![Page 78: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/78.jpg)
Coherence
Coherence M(Φ) := maxi 6=j |〈φi , φj〉|.
[Gribonval,Nielsen]
TheoremIf ‖x∗‖0 ≤
12
(1 + 1
M(Φ)
)is a solution of y = Mx, then x∗ is the
unique minimizer of the `1-problem. x∗ is also the uniqueminimizer of the `0-problem!
![Page 79: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/79.jpg)
Coherence
Coherence M(Φ) := maxi 6=j |〈φi , φj〉|.
[Gribonval,Nielsen]
TheoremIf ‖x∗‖0 ≤
12
(1 + 1
M(Φ)
)is a solution of y = Mx, then x∗ is the
unique minimizer of the `1-problem. x∗ is also the uniqueminimizer of the `0-problem!
![Page 80: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/80.jpg)
Derivation of `1-guaranteeing sparsity level
For z ∈ N(Φ),
φjzj = −∑i 6=j
φizi .
|zj | ≤ M(Φ)∑i 6=j
|zi |
(1 + M(Φ))|zj | ≤ M(Φ) ‖z‖1
(1 + M(Φ)) ‖zΛ‖1 ≤ (#Λ)M(Φ) ‖z‖1
⇒ ‖zΛ‖1 ≤‖z‖1
2 if
#Λ ≤ 1
2(1 +
1
M(Φ)).
![Page 81: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/81.jpg)
Derivation of `1-guaranteeing sparsity level
For z ∈ N(Φ),
φjzj = −∑i 6=j
φizi .
|zj | ≤ M(Φ)∑i 6=j
|zi |
(1 + M(Φ))|zj | ≤ M(Φ) ‖z‖1
(1 + M(Φ)) ‖zΛ‖1 ≤ (#Λ)M(Φ) ‖z‖1
⇒ ‖zΛ‖1 ≤‖z‖1
2 if
#Λ ≤ 1
2(1 +
1
M(Φ)).
![Page 82: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/82.jpg)
Derivation of `1-guaranteeing sparsity level
For z ∈ N(Φ),
φjzj = −∑i 6=j
φizi .
|zj | ≤ M(Φ)∑i 6=j
|zi |
(1 + M(Φ))|zj | ≤ M(Φ) ‖z‖1
(1 + M(Φ)) ‖zΛ‖1 ≤ (#Λ)M(Φ) ‖z‖1
⇒ ‖zΛ‖1 ≤‖z‖1
2 if
#Λ ≤ 1
2(1 +
1
M(Φ)).
![Page 83: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/83.jpg)
Derivation of `1-guaranteeing sparsity level
For z ∈ N(Φ),
φjzj = −∑i 6=j
φizi .
|zj | ≤ M(Φ)∑i 6=j
|zi |
(1 + M(Φ))|zj | ≤ M(Φ) ‖z‖1
(1 + M(Φ)) ‖zΛ‖1 ≤ (#Λ)M(Φ) ‖z‖1
⇒ ‖zΛ‖1 ≤‖z‖1
2 if
#Λ ≤ 1
2(1 +
1
M(Φ)).
![Page 84: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/84.jpg)
Derivation of `1-guaranteeing sparsity level
For z ∈ N(Φ),
φjzj = −∑i 6=j
φizi .
|zj | ≤ M(Φ)∑i 6=j
|zi |
(1 + M(Φ))|zj | ≤ M(Φ) ‖z‖1
(1 + M(Φ)) ‖zΛ‖1 ≤ (#Λ)M(Φ) ‖z‖1
⇒ ‖zΛ‖1 ≤‖z‖1
2 if
#Λ ≤ 1
2(1 +
1
M(Φ)).
![Page 85: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/85.jpg)
Coherence gives suboptimal estimates
For Φ ∈ Rd×K , we may suppose M(Φ) = O(
1√d
).
We can recover x∗ if x∗ is 12
(1 +
√O(d)
)= O(
√d) sparse.
To put it differently, in order to recover k-sparse x∗, d = O(k2)measurements are desired.
![Page 86: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/86.jpg)
Coherence gives suboptimal estimates
For Φ ∈ Rd×K , we may suppose M(Φ) = O(
1√d
).
We can recover x∗ if x∗ is 12
(1 +
√O(d)
)= O(
√d) sparse.
To put it differently, in order to recover k-sparse x∗, d = O(k2)measurements are desired.
![Page 87: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/87.jpg)
Coherence gives suboptimal estimates
For Φ ∈ Rd×K , we may suppose M(Φ) = O(
1√d
).
We can recover x∗ if x∗ is 12
(1 +
√O(d)
)= O(
√d) sparse.
To put it differently, in order to recover k-sparse x∗, d = O(k2)measurements are desired.
![Page 88: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/88.jpg)
Recovery of Logan-Shepp phantom using FourierTransform Samples
An Image to recover:
![Page 89: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/89.jpg)
Recovery of Logan-Shepp phantom using FourierTransform Samples
Available samples in Fourier Domain (2.7% (4.3%)):
![Page 90: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/90.jpg)
Recovery of Logan-Shepp phantom using FourierTransform Samples
Reconstructed image—perfect!:
![Page 91: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/91.jpg)
How many samples do we need?
To recover
we (obviously) need k ≤ m ≤ N samples.Need to identify k nonzero positions out of N locations ⇒log2
(Nk
)≈ k log N
k measurements
![Page 92: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/92.jpg)
How many samples do we need?
To recover
we (obviously) need k ≤ m ≤ N samples.
Need to identify k nonzero positions out of N locations ⇒log2
(Nk
)≈ k log N
k measurements
![Page 93: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/93.jpg)
How many samples do we need?
To recover
we (obviously) need k ≤ m ≤ N samples.Need to identify k nonzero positions out of N locations ⇒log2
(Nk
)≈ k log N
k measurements
![Page 94: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/94.jpg)
Random Sampling of Fourier Transform
[Candes, Romberg, Tao]
TheoremLet N = n2, f be an n × n real-valued image supported on T . LetΩ be a subset of 1, . . . , n2 chosen unformly at random. If
#Ω ≥ C (#T ) log N,
then with high probability (1− O(N−M)), the minimizer to
ming‖g‖1 s.t. g |Ω = f |Ω
is unique and is equal to f .
![Page 95: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/95.jpg)
Restricted Isometry Property
For s ∈ N, A matrix Φ satisfies the Restricted Isometry Propertywith the isometry constant δs if
(1− δs) ‖x‖22 ≤ ‖Φx‖2
2 ≤ (1 + δs) ‖x‖22
for all s-sparse signal x .
![Page 96: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/96.jpg)
Perfect Recovery condition in terms of RIP constant
[Candes]
TheoremAssume that δ2s <
√2− 1. Then, the solution x∗ to
minx‖x‖1 s.t. Mx = Mx
satisfies‖x∗ − x‖2 ≤ C0s−1/2 ‖x − xs‖1
where C0 is a constant.
Remark: The sensing matrix M is non-adaptive, but the`1-minimization with M recovers every s-sparse signal.
![Page 97: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/97.jpg)
RIP and NSP
[Candes]
TheoremIf h is in the nullspace of Φ, then
‖hT‖1 ≤ ρ ‖hT c‖1 , ρ :=√
2δ2s(1− δ2s)−1
for every T with #T = s.
![Page 98: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/98.jpg)
Examples of Measurement Matrices with good RIP
I Random matrices with i.i.d. entries. If entries of Φ ∈ Rm×d
are drawn from i.i.d. Gaussian with mean 0 and variance 1/m,then with overwhelming probability the RIC δs is less than√
2− 1 when s ≤ Cm/ log(d/m).
I Fourier ensemble. Φ ∈ Rm×d is constructed by samplingrandom m rows of the discrete Fourier transform.
I General orthogonal measurement ensembles.
![Page 99: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/99.jpg)
Questions
1. ‘Random dictionaries’ are good for compressed sensing. Arethere any deterministic dictionaries that are as good asrandom ones?
2. Any alternative to RIP?
3. What happens when the model is not exactly sparse?
4. What happens when there is noise?
![Page 100: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/100.jpg)
Stable and Robust Recovery
We observey = Φx + n.
We reconstruct x as the solution to
minx‖x‖1 s.t. ‖y − Φx‖2 ≤ ε.
![Page 101: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/101.jpg)
Stable and Robust Recovery
[Candes]
TheoremAssume that δ2s <
√2− 1 and ‖n‖2 ≤ ε. Then, the solution x∗
satisfies‖x∗ − x‖2 ≤ C0s−1/2 ‖x − xs‖1 + C1ε
where C0 and C1 are some constants.
![Page 102: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/102.jpg)
Global Optimization Approach for Noisy Problem
We solve
minx
1
2‖Mx − y‖2
2 + λ ‖x‖1
for some λ > 0.
λ→ 0?
λ→∞?
![Page 103: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/103.jpg)
Global Optimization Approach for Noisy Problem
We solve
minx
1
2‖Mx − y‖2
2 + λ ‖x‖1
for some λ > 0.
λ→ 0?
λ→∞?
![Page 104: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/104.jpg)
Global Optimization Approach for Noisy Problem
We solve
minx
1
2‖Mx − y‖2
2 + λ ‖x‖1
for some λ > 0.
λ→ 0?
λ→∞?
![Page 105: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/105.jpg)
Compressed Sensing
Paper: Compressed Sensing by David Donoho
‖θ − θN‖2 ≤ C (N + 1)1/2−1/p, N = 0, 1, . . . .
![Page 106: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/106.jpg)
We Need Good Dictionaries
Techniques/Algorithms of sparse modeling begin with somedictionary Φ that provides sparse representations of the class ofsignals of interest. What are those Φ’s?
![Page 107: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/107.jpg)
Good Dictionaries
I Smooth functions ⇒ Fourier transform
I Smooth functions with point singularities ⇒ Wavelets
I Singularities along smooth curves ⇒ Curvelets, Shearlets, etc.
I . . .
![Page 108: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/108.jpg)
PCA / SVD?
We want to design/learn dictionaries from data.
Principal Component Analysis ⇒ Not suitable for non-Gaussianmixtures
[Olshausen & Field]Used Sparsity Promoting regularization to learn Dictionary⇒ Localized, oriented, bandpass receptive fields emerge
![Page 109: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/109.jpg)
PCA / SVD?
We want to design/learn dictionaries from data.
Principal Component Analysis ⇒ Not suitable for non-Gaussianmixtures
[Olshausen & Field]Used Sparsity Promoting regularization to learn Dictionary⇒ Localized, oriented, bandpass receptive fields emerge
![Page 110: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/110.jpg)
PCA / SVD?
We want to design/learn dictionaries from data.
Principal Component Analysis ⇒ Not suitable for non-Gaussianmixtures
[Olshausen & Field]Used Sparsity Promoting regularization to learn Dictionary⇒ Localized, oriented, bandpass receptive fields emerge
![Page 111: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/111.jpg)
Dictionary Learning via `1-criterion
If Φ were a good dictionary for a signal x , then we can recover xby solving
minx‖x‖1 s.t. y = Φx .
⇒ Given a data set Y := [y1, . . . , yN ], we may find a gooddictionary Φ by solving
minΦ,X‖X‖1 s.t. Y = ΦX .
![Page 112: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/112.jpg)
Dictionary Learning via `1-criterion
If Φ were a good dictionary for a signal x , then we can recover xby solving
minx‖x‖1 s.t. y = Φx .
⇒ Given a data set Y := [y1, . . . , yN ], we may find a gooddictionary Φ by solving
minΦ,X‖X‖1 s.t. Y = ΦX .
![Page 113: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/113.jpg)
Some issues
I We can decrease ‖X‖1 as small as we wish by scaling because
αΦα−1X = ΦX
for any α ∈ R.
I The problemminΦ,X‖X‖1 s.t. Y = ΦX
is not convex.
The first issue can be dealt with by requiring that each column ofΦ to be of unit length. We will write Φ ∈ U(d ,K ) to mean thatΦ ∈ Rd×K is such a dictionary.
![Page 114: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/114.jpg)
Some issues
I We can decrease ‖X‖1 as small as we wish by scaling because
αΦα−1X = ΦX
for any α ∈ R.
I The problemminΦ,X‖X‖1 s.t. Y = ΦX
is not convex.
The first issue can be dealt with by requiring that each column ofΦ to be of unit length. We will write Φ ∈ U(d ,K ) to mean thatΦ ∈ Rd×K is such a dictionary.
![Page 115: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/115.jpg)
Some issues
I We can decrease ‖X‖1 as small as we wish by scaling because
αΦα−1X = ΦX
for any α ∈ R.
I The problemminΦ,X‖X‖1 s.t. Y = ΦX
is not convex.
The first issue can be dealt with by requiring that each column ofΦ to be of unit length. We will write Φ ∈ U(d ,K ) to mean thatΦ ∈ Rd×K is such a dictionary.
![Page 116: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/116.jpg)
Dictionary Identification
General Question: Given a data set Y , we learned a dictionary Φvia some learning method. Is Φ the ‘right’ one?
![Page 117: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/117.jpg)
Dictionary Identification
Suppose that the training data Y is generated by
Y = Φ0X0.
Under what condition on X0 (and Φ0), can we be sure that theminimization problem
minΦ,X‖X‖1 s.t. Y = ΦX
will recover the pair Φ0 and X0?
![Page 118: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/118.jpg)
Necessary and Sufficient Condition
[Gribonval & Schnass]
TheoremSuppose that K ≥ d. (Φ0,X0) is a strict local minimum of theproblem
minΦ,X‖X‖1 s.t. ΦX = Φ0X0,Φ ∈ U(d ,K )
if and only if
|〈CX0 + V , sign(X0)〉| < ‖(CX0 + V )Λ‖1
for every CX0 + V 6= 0 with diag(Φ∗0Φ0C ) = 0 and V ∈ N(Φ0).
![Page 119: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/119.jpg)
Sufficient Condition for Basis
Notation:
![Page 120: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/120.jpg)
Sufficient Condition for Basis
Theorem(Φ0,X0) is a strict local minimum of the problem
minΦ,X‖X‖1 s.t. ΦX = Φ0X0,Φ ∈ U(d , d)
if for every k = 1, . . . , d, there exists dk ∈ Rd with ‖dk‖∞ < 1such that
Xkdk = Xksk − diag((∥∥x i∥∥)i)
mk
where mk is the k-th column of Φ∗0Φ0.
![Page 121: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/121.jpg)
Geometric Interpretation
Local minimum Not local minimum
uk := Xksk − diag((∥∥x i∥∥)i)
mk , Qk := [−1, 1]rk .
![Page 122: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/122.jpg)
Bernoulli-Gaussian Model
X0 follows the Bernoulli-Gaussian Model with parameter 0 < p < 1if each entry of X0 is the product of independent Bernoulli(p)random variable ξij and Normal random variable gij , i.e.,
xij = ξijgij .
![Page 123: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/123.jpg)
Concentration of Measure Phenomena
If X0 follows the Bernoulli-Gaussian distribution with parameter p,then with high probability XkQk contains the `2-ball of radius
α ≈ Np(1− p)
√2
π,
Xksk is contained in the `2-ball of radius
β ≈√
NK p,
and diag((∥∥x i∥∥)i)
mk is contained in the `2-ball of radius
γ ≈ Np
√2
π.
![Page 124: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/124.jpg)
Concentration of Measure Phenomena
If X0 follows the Bernoulli-Gaussian distribution with parameter p,then with high probability XkQk contains the `2-ball of radius
α ≈ Np(1− p)
√2
π,
Xksk is contained in the `2-ball of radius
β ≈√
NK p,
and diag((∥∥x i∥∥)i)
mk is contained in the `2-ball of radius
γ ≈ Np
√2
π.
![Page 125: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/125.jpg)
Concentration of Measure Phenomena
If X0 follows the Bernoulli-Gaussian distribution with parameter p,then with high probability XkQk contains the `2-ball of radius
α ≈ Np(1− p)
√2
π,
Xksk is contained in the `2-ball of radius
β ≈√
NK p,
and diag((∥∥x i∥∥)i)
mk is contained in the `2-ball of radius
γ ≈ Np
√2
π.
![Page 126: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/126.jpg)
Sufficient Condition under Bernoulli-Gaussian Model
If the coherence M(Φ0) of Φ0 is less than 1− p then for largeenough N, Φ0 is locally identifiable.
Surprise: One needs only
N ≥ CK log K .
![Page 127: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/127.jpg)
Sufficient Condition under Bernoulli-Gaussian Model
If the coherence M(Φ0) of Φ0 is less than 1− p then for largeenough N, Φ0 is locally identifiable.
Surprise: One needs only
N ≥ CK log K .
![Page 128: Some topics on Sparsitykowon.dongseo.ac.kr/~lbg/cagd/kmmcs/201008/NamSangNam.pdfI Morphological Component Analysis [MCA, Bobin et al] I Stagewise OMP [Donoho et al] I CoSAMP [Needell](https://reader034.vdocument.in/reader034/viewer/2022042107/5e87015695ec854a037a2a49/html5/thumbnails/128.jpg)
Some dictionary learning methods
I K-SVD [Elad et al.]
I ISI [Gowreesunker]
I . . .