short-course compressive sensing of videos venue cvpr 2012, providence, ri, usa june 16, 2012
DESCRIPTION
Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012. Organizers : Richard G. Baraniuk Mohit Gupta Aswin C. Sankaranarayanan Ashok Veeraraghavan. Part 2: Compressive sensing Motivation, theory, recovery. Linear inverse problems - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/1.jpg)
Short-courseCompressive Sensing of Videos
VenueCVPR 2012, Providence, RI, USA
June 16, 2012
Organizers:Richard G. BaraniukMohit GuptaAswin C. SankaranarayananAshok Veeraraghavan
![Page 2: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/2.jpg)
Part 2: Compressive sensingMotivation, theory, recovery
• Linear inverse problems
• Sensing visual signals
• Compressive sensing– Theory– Hallmark– Recovery algorithms
• Model-based compressive sensing– Models specific to visual signals
![Page 3: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/3.jpg)
Linear inverse problems
![Page 4: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/4.jpg)
Linear inverse problems• Many classic problems in computer can be posed
as linear inverse problems
• Notation– Signal of interest
– Observations
– Measurement model
• Problem definition: given , recover
measurement noise
measurement matrix
![Page 5: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/5.jpg)
Linear inverse problems
• Problem definition: given , recover
• Scenario 1
• We can invert the system of equations
• Focus more on robustness to noise via signal priors
![Page 6: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/6.jpg)
Linear inverse problems
• Problem definition: given , recover
• Scenario 2
• Measurement matrix has a (N-M) dimensional null-space
• Solution is no longer unique
• Many interesting vision problem fall under this scenario
• Key quantity of concern: Under-sampling ratio M/N
![Page 7: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/7.jpg)
Image super-resolution
Low resolution input/observation
128x128 pixels
![Page 8: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/8.jpg)
Image super-resolution
2x super-resolution
![Page 9: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/9.jpg)
Image super-resolution
4x super-resolution
![Page 10: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/10.jpg)
Image super-resolution
Super-resolution factor D
Under-sampling factor M/N = 1/D2
General rule: The smaller the under-sampling, the more the unknowns and hence, the harder the super-resolution problem
![Page 11: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/11.jpg)
Many other vision problems…• Affine rank minimizaiton, Matrix completion,
deconvolution, Robust PCA
• Image synthesis– Infilling, denoising, etc.
• Light transport– Reflectance fields, BRDFs, Direct global separation, Light
transport matrices
• Sensing
![Page 12: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/12.jpg)
Sensing visual signals
![Page 13: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/13.jpg)
High-dimensional visual signals
Human skinSub-surface scattering
Electron microscopyTomography
Reflection
Refraction
FogVolumetric scattering
![Page 14: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/14.jpg)
The plenoptic functionCollection of all variations of light in a scene
Adelson and Bergen (91)
space (3D)
angle (2D)
spectrum(1D)
time (1D)
Different slices reveal different scene properties
![Page 15: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/15.jpg)
space (3D)
angle (2D)
spectrum(1D)
time (1D)
Lytro light-field camera
High-speed cameras
The plenoptic function
Hyper-spectral imaging
![Page 16: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/16.jpg)
Sensing the plenoptic function
• High-dimensional– 1000 samples/dim == 10^21 dimensional signal– Greater than all the storage in the world
• Traditional theories of sensing fail us!
![Page 17: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/17.jpg)
Resolution trade-off
• Key enabling factor: Spatial resolution is cheap!
• Commercial cameras have 10s of megapixels
• One idea is the we trade-off spatial resolution for resolution in some other axis
![Page 18: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/18.jpg)
Spatio-angular tradeoff
[Ng, 2005]
![Page 19: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/19.jpg)
Spatio-angular tradeoff
[Levoy et al. 2006]
![Page 20: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/20.jpg)
Spatio-temporal tradeoff
[Bub et al., 2010]
Stagger pixel-shutter within each exposure
![Page 21: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/21.jpg)
Spatio-temporal tradeoff
[Bub et al., 2010]
Rearrange to get high temporal resolution video at lower spatial-resolution
![Page 22: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/22.jpg)
Resolution trade-off
• Very powerful and simple idea
• Drawbacks– Does not extend to non-visible
spectrum• 1 Megapixel SWIR camera costs 50-
100k
– Linear and global tradeoffs
– With today’s technology, cannot obtain more than 10x for video without sacrificing spatial resolution completely
![Page 23: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/23.jpg)
Compressive sensing
![Page 24: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/24.jpg)
Sense by Sampling
sample
![Page 25: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/25.jpg)
Sense by Sampling
sample too much data!
![Page 26: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/26.jpg)
Sense then Compress
compress
decompress
sample
JPEGJPEG2000
…
![Page 27: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/27.jpg)
Sparsity
pixels largewaveletcoefficients(blue = 0)
![Page 28: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/28.jpg)
Sparsity
pixels largewaveletcoefficients(blue = 0)
widebandsignalsamples
largeGabor (TF)coefficients
time
frequ
ency
![Page 29: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/29.jpg)
• Sparse signal: only K out of N coordinates nonzero
Concise Signal Structure
sorted index
![Page 30: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/30.jpg)
• Sparse signal: only K out of N coordinates nonzero
– model: union of K-dimensional subspacesaligned w/ coordinate axes
Concise Signal Structure
sorted index
![Page 31: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/31.jpg)
• Sparse signal: only K out of N coordinates nonzero
– model: union of K-dimensional subspaces
• Compressible signal: sorted coordinates decay rapidly with power-law
sorted index
power-lawdecay
Concise Signal Structure
![Page 32: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/32.jpg)
What’s Wrong with this Picture?• Why go to all the work to acquire
N samples only to discard all but K pieces of data?
compress
decompress
sample
![Page 33: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/33.jpg)
What’s Wrong with this Picture?linear processinglinear signal model(bandlimited
subspace)
compress
decompress
sample
nonlinear processingnonlinear signal model(union of subspaces)
![Page 34: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/34.jpg)
Compressive Sensing• Directly acquire “compressed” data
via dimensionality reduction• Replace samples by more general
“measurements”
compressive sensing
recover
![Page 35: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/35.jpg)
• Signal is -sparse in basis/dictionary– WLOG assume sparse in space domain
Sampling
sparsesignal
nonzeroentries
![Page 36: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/36.jpg)
• Signal is -sparse in basis/dictionary– WLOG assume sparse in space domain
• Sampling
sparsesignal
nonzeroentries
measurements
Sampling
![Page 37: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/37.jpg)
Compressive Sampling• When data is sparse/compressible, can
directly acquire a condensed representation with no/little information loss through linear dimensionality reduction
measurements sparsesignal
nonzeroentries
![Page 38: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/38.jpg)
How Can It Work?• Projection
not full rank…
… and so loses information in general
• Ex: Infinitely many ’s map to the same(null space)
![Page 39: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/39.jpg)
How Can It Work?• Projection
not full rank…
… and so loses information in general
• But we are only interested in sparse vectors
columns
![Page 40: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/40.jpg)
How Can It Work?• Projection
not full rank…
… and so loses information in general
• But we are only interested in sparse vectors
• is effectively MxK
columns
![Page 41: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/41.jpg)
How Can It Work?• Projection
not full rank…
… and so loses information in general
• But we are only interested in sparse vectors
• Design so that each of its MxK submatrices are full rank (ideally close to orthobasis)– Restricted Isometry Property (RIP)
columns
![Page 42: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/42.jpg)
Restricted Isometry Property (RIP)• Preserve the structure of sparse/compressible
signals
K-dim subspaces
![Page 43: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/43.jpg)
Restricted Isometry Property (RIP)• “Stable embedding”• RIP of order 2K implies: for all K-sparse x1 and
x2
K-dim subspaces
![Page 44: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/44.jpg)
RIP = Stable Embedding• An information preserving projection preserves
the geometry of the set of sparse signals
• RIP ensures that
![Page 45: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/45.jpg)
How Can It Work?• Projection
not full rank…
… and so loses information in general
• Design so that each of its MxK submatrices are full rank (RIP)
• Unfortunately, a combinatorial, NP-complete design problem
columns
![Page 46: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/46.jpg)
Insight from the 70’s [Kashin, Gluskin]
• Draw at random– iid Gaussian– iid Bernoulli
…
• Then has the RIP with high probability provided
columns
![Page 47: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/47.jpg)
Randomized Sensing• Measurements = random linear
combinations of the entries of the signal
• No information loss for sparse vectors whpmeasurements sparse
signal
nonzeroentries
![Page 48: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/48.jpg)
CS Signal Recovery
• Goal: Recover signal from measurements
• Problem: Randomprojection not full rank(ill-posed inverse problem)
• Solution: Exploit the sparse/compressiblegeometry of acquired signal
![Page 49: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/49.jpg)
CS Signal Recovery• Random projection
not full rank
• Recovery problem:givenfind
• Null space
• Search in null space for the “best” according to some criterion– ex: least squares
(N-M)-dim hyperplaneat random angle
![Page 50: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/50.jpg)
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Closed-form solution:
Signal Recovery
![Page 51: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/51.jpg)
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Closed-form solution:
• Wrong answer!
Signal Recovery
![Page 52: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/52.jpg)
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Closed-form solution:
• Wrong answer!
Signal Recovery
![Page 53: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/53.jpg)
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
Signal Recovery
“find sparsest vectorin translated nullspace”
![Page 54: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/54.jpg)
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Correct!
• But NP-Complete alg
Signal Recovery
“find sparsest vectorin translated nullspace”
![Page 55: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/55.jpg)
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Convexify the optimization
Signal Recovery
Candes Romberg Tao Donoho
![Page 56: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/56.jpg)
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Convexify the optimization
• Correct!
• Polynomial time alg(linear programming)
Signal Recovery
![Page 57: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/57.jpg)
Compressive Sensing
• Signal recovery via optimization [Candes, Romberg, Tao; Donoho]
random measurements
sparsesignal
nonzeroentries
![Page 58: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/58.jpg)
Compressive Sensing
• Signal recovery via iterative greedy algorithm– (orthogonal) matching pursuit [Gilbert, Tropp]– iterated thresholding
[Nowak, Figueiredo; Kingsbury, Reeves; Daubechies, Defrise, De Mol; Blumensath, Davies; …]
– CoSaMP [Needell and Tropp]
random measurements
sparsesignal
nonzeroentries
![Page 59: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/59.jpg)
Greedy recovery algorithm #1• Consider the following problem
• Suppose we wanted to minimize just the cost, then steepest gradient descent works as
• But, the new estimate is no longer K-sparse
![Page 60: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/60.jpg)
Iterated Hard Thresholding
update signal estimate
prune signal estimate(best K-term approx)
update residual
![Page 61: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/61.jpg)
Greedy recovery algorithm #2• Consider the following problem
• Can we recover the support ?
sparsesignal
1 sparse
![Page 62: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/62.jpg)
Greedy recovery algorithm #2• Consider the following problem
• If then gives the support of x
• How to extend to K-sparse signals ?
sparsesignal
1 sparse
![Page 63: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/63.jpg)
Greedy recovery algorithm #2
sparsesignal
K sparse
residue:
find atom:
Add atom to support:
Signal estimate (Least squares over support)
![Page 64: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/64.jpg)
Orthogonal matching pursuit
update residual
Find atom with largest support
Update signal estimate
![Page 65: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/65.jpg)
Specialized solvers• CoSAMP [Needell and Tropp, 2009]
• SPG_l1 [Friedlander, van der Berg, 2008]http://www.cs.ubc.ca/labs/scl/spgl1/
• FPC [Hale, Yin, and Zhang, 2007]http://www.caam.rice.edu/~optimization/L1/fpc/
• AMP [Donoho, Montanari and Maleki, 2010]
many many others, see dsp.rice.edu/csandhttps://sites.google.com/site/igorcarron2/cscodes
![Page 66: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/66.jpg)
CS Hallmarks• Stable
– acquisition/recovery process is numerically stable
• Asymmetrical (most processing at decoder) – conventional: smart encoder, dumb decoder– CS: dumb encoder, smart decoder
• Democratic– each measurement carries the same amount of information– robust to measurement loss and quantization– “digital fountain” property
• Random measurements encrypted• Universal
– same random projections / hardware can be used forany sparse signal class (generic)
![Page 67: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/67.jpg)
Universality• Random measurements can be used for
signals sparse in any basis
![Page 68: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/68.jpg)
Universality• Random measurements can be used for
signals sparse in any basis
![Page 69: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/69.jpg)
Universality• Random measurements can be used for
signals sparse in any basis
sparsecoefficient
vector
nonzeroentries
![Page 70: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/70.jpg)
Summary: CS
• Compressive sensing– randomized dimensionality reduction– exploits signal sparsity information– integrates sensing, compression, processing
• Why it works: with high probability, random projections preserve
information in signals with concise
geometric structures– sparse signals– compressible signals
![Page 71: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/71.jpg)
Summary: CS• Encoding: = random linear combinations
of the entries of
• Decoding: Recover from via optimization
measurements sparsesignal
nonzeroentries
![Page 72: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/72.jpg)
Image/Video specific signal models
and recovery algorithms
![Page 73: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/73.jpg)
Transform basis• Recall Universality: Random measurements can
be used for signals sparse in any basis
• DCT/FFT/Wavelets …– Fast transforms; very useful in large scale problems
![Page 74: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/74.jpg)
Dictionary learning• For many signal classes (ex: videos, light-fields),
there are no obvious sparsifying transform basis
• Can we learn a sparsifying transform instead ?
• GOAL: Given training data learn a “dictionary” D, such that
are sparse.
![Page 75: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/75.jpg)
Dictionary learning• GOAL: Given training data learn a “dictionary” D, such that
are sparse.
![Page 76: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/76.jpg)
Dictionary learning
• Non-convex constraint
• Bilinear in D and S
![Page 77: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/77.jpg)
Dictionary learning
• Biconvex in D and S– Given D, the optimization problem is convex in sk
– Given S, the optimization problem is a least squares problem
• K-SVD: Solve using alternate minimization techniques– Start with D = wavelet or DCT bases– Additional pruning steps to control size of the dictionary
Aharon et al., TSP 2006
![Page 78: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/78.jpg)
Dictionary learning
• Pros– Ability to handle arbitrary domains
• Cons– Learning dictionaries can be computationally intensive
for high-dimensional problems; need for very large amount of data
– Recovery algorithms may suffer due to lack of fast transforms
![Page 79: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/79.jpg)
Models on image gradients• Piecewise constant
images– Sparse image gradients
• Natural image statistics– Heavy tailed distributions
![Page 80: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/80.jpg)
Total variation prior
• TV norm– Sparse-gradient promoting norm
• Formulation of recovery problem
![Page 81: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/81.jpg)
Total variation prior
• Optimization problem– Convex– Often, works “better” than transform
basis methods
• Variants– 3D (video)– Anisotropic TV
• Code– TVAL3– Many many others (see dsp.rice/cs)
![Page 82: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/82.jpg)
Beyond sparsityModel-based CS
![Page 83: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/83.jpg)
Beyond Sparse Models • Sparse signal model captures
simplistic primary structure
wavelets:natural images
Gabor atoms:chirps/tones
pixels:background subtracted
images
![Page 84: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/84.jpg)
Beyond Sparse Models • Sparse signal model captures
simplistic primary structure
• Modern compression/processing algorithms capture richer secondary coefficient structure
wavelets:natural images
Gabor atoms:chirps/tones
pixels:background subtracted
images
![Page 85: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/85.jpg)
Sparse Signals• K-sparse signals comprise a particular set of
K-dim subspaces
![Page 86: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/86.jpg)
Structured-Sparse Signals• A K-sparse signal model comprises a particular
(reduced) set of K-dim subspaces[Blumensath and Davies]
• Fewer subspaces <> relaxed RIP <> stable recovery using
fewer measurements M
![Page 87: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/87.jpg)
Wavelet Sparse
• Typical of wavelet transformsof natural signals and images (piecewise smooth)
![Page 88: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/88.jpg)
Tree-Sparse• Model: K-sparse coefficients
+ significant coefficients lie on a rooted subtree
• Typical of wavelet transformsof natural signals and images (piecewise smooth)
![Page 89: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/89.jpg)
Wavelet Sparse• Model: K-sparse coefficients
+ significant coefficients lie on a rooted subtree
• RIP: stable embedding
K-planes
![Page 90: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/90.jpg)
Tree-Sparse• Model: K-sparse coefficients
+ significant coefficients lie on a rooted subtree
• Tree-RIP: stable embedding [Blumensath and Davies]
K-planes
![Page 91: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/91.jpg)
Tree-Sparse
• Model: K-sparse coefficients + significant coefficients
lie on a rooted subtree
• Tree-RIP: stable embedding [Blumensath and Davies]
• Recovery: inject tree-sparse approx into IHT/CoSaMP
![Page 92: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/92.jpg)
Recall: Iterated Thresholding
update signal estimate
prune signal estimate(best K-term approx)
update residual
![Page 93: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/93.jpg)
Iterated Model Thresholding
update signal estimate
prune signal estimate(best K-term model approx)
update residual
![Page 94: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/94.jpg)
Tree-Sparse Signal Recovery
target signal CoSaMP, (RMSE=1.12)
Tree-sparse CoSaMP (RMSE=0.037)
signal lengthN=1024
random measurementsM=80
L1-minimization(RMSE=0.751)
![Page 95: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/95.jpg)
Clustered Signals
target Ising-modelrecovery
CoSaMPrecovery
LP (FPC)recovery
• Probabilistic approach via graphical model
• Model clustering of significant pixels in space domain using Ising Markov Random Field
• Ising model approximation performed efficiently using graph cuts [Cevher, Duarte, Hegde, Baraniuk’08]
![Page 96: Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012](https://reader036.vdocument.in/reader036/viewer/2022062323/5681672c550346895ddbcbc2/html5/thumbnails/96.jpg)
Part 2: Compressive sensingMotivation, theory, recovery
• Linear inverse problems
• Sensing visual signals
• Compressive sensing– Theory– Hallmark– Recovery algorithms
• Model-based compressive sensing– Models specific to visual signals