sparsity-based signal models and the sparse k-svd algorithm
DESCRIPTION
Sparsity-Based Signal Models and the Sparse K-SVD Algorithm. Ron Rubinstein Advisor: Prof. Michael Elad October 2010. Signal Models. Signal models are a fundamental tool for solving low-level signal processing tasks. Inpainting. Detail enhancement. Deblurring. Tompgraphy reconstruction. - PowerPoint PPT PresentationTRANSCRIPT
Slide 1
Sparsity-Based Signal Models and the Sparse K-SVD AlgorithmRon RubinsteinAdvisor: Prof. Michael Elad
October 2010
1Sparse Coding Stage: Global OMPMotivation: adapt the distribution of coefficients to the complexity of the block
Efficiency is achieve by implementing the computations locally only for the affected blockSelect atom+block that produce maximal inner productAdd atom toblock using a single OMP iterationStop when error for entire image reaches desired target
DictionaryImageSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Signal Models
Smooth
Piecewise smooth
Smooth withpoint singularities
?Signal model: a mathematical description of the behavior we expect from a good (uncontaminated ) signal in our systemSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#3
Proof of Equivalence for akLet X and Y be matrices, and let u and v be vectors.Also, assume that vTv=1. Then we can show thatSo, for, we have
Independent of ak!Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Signal Models are EverywhereInterpolation:Bilinear, bicubic: signals are piecewise-smooth Lanczos: signals are band-limitedCompression:PNG: neighboring pixels have similar intensitiesJPEG: small image patches are smoothJPEG2K: images are smooth except for simple singularitiesSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#5AgendaAnalysis and synthesis signal modelsTwo models are better than one
A few words on dictionary designOn dictionaries and choices
Introduction to sparse representationSome background on sparsity and the K-SVD
The sparse dictionary modelIntroducing an efficient and adaptable dictionary!
Why sparse dictionaries are good for youSome uses and applications
Summary and conclusions
Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#6Transform-Based Signal ModelsTransform-based models compute a vector transform of the signal:
Such signal models promote sparsity of (x), in the sense that we expect the coefficients in to decay rapidly.
Wavelet Coefficient Distribution - BarbaraSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#7
Analysis and Synthesis ModelsAnalysis models use a set of linear filters, assumed to produce sparse inner products with the signal:
Synthesis models use a set of atomic signals, assumed to reproduce the input signal via a sparse linear combination:The analysis dictionary contains the linear filters as its rowsThe synthesis dictionary contains the atoms as its columns
DSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#8Analysis versus Synthesis?Obvious question: are the two models equivalent?
Surprising answer:NO!
(Well, except for the invertible case, where we can use D=-1)Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#9Analysis versus Synthesis in the L1 CaseFor the widely-used L1 case,Given any similar-sized D and , we can find large number of signals where the two will substantially differ[Elad, Milanfar & Rubinstein 07]The analysis model is mathematically a subset of the synthesis model, but the mapping is exponentially complexThere are practical cases where one may outperform the other
Geometric structure in the L1 caseSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#10Sparsity measure: how do we quantify the sparsity of the coefficients in x?Current Research on Analysis ModelsL0 (sparsity)Lp normsHuberCauchyDictionary training: can we use computational learning to infer the dictionary from examples?[Black & Roth 05][Rubinstein & Elad 10]Algorithms: how do we efficiently solve the resulting optimization problems?Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#11AgendaAnalysis and synthesis signal modelsTwo models are better than one
A few words on dictionary designOn dictionaries and choices
Introduction to sparse representationSome background on sparsity and K-SVD
The sparse dictionary modelIntroducing an efficient and adaptable dictionary!
Why sparse dictionaries are good for youSome uses and applications
Summary and conclusions
Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#12
Designing Your DictionaryHarmonic Analysis: Analytic dictionaries
Machine Learning:Trained dictionaries[Rubinstein, Bruckstein & Elad 10]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#13Analytic dictionaries arise from a mathematical model of the signals
Analytic DictionariesAtoms have analytic formulationsKnown mathematical properties (e.g. coefficient decay rates)Fast algorithms for computing the transformsLimited expressiveness: all signals behavethe sameSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#14Some Analytic DictionariesFourierGabor
WaveletsCurveletsSmooth signalsSmooth signals
Smooth + point singularitiesSmooth + curve singularities
[Cands & Donoho 99]
Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#15Trained DictionariesDictionary is learned from actual dataFiner adaptation to the target signalsBetter performance in applicationsNon-structured: higher complexity, single scaleTrained dictionaries arise from a set of examples of the signal dataSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Dictionary Training AlgorithmsOlshausen & Field 96MOD (Method of Optimal Directions)[Engan, Aase & Husy 99]
K-SVD [Aharon, Elad & Bruckstein 06]
Generalized PCA[Vidal, Ma & Sastry 05]
Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#17
Dictionary Design: SummaryAnalytic dictionariesLow complexityOptimal for specific classes of signalsNon-adaptive
Trained dictionariesAdaptable to different signal typesBetter results in many applicationsNon-structured Can we have it allSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#18AgendaAnalysis and synthesis signal modelsTwo models are better than one
A few words on dictionary designOn dictionaries and choices
Introduction to sparse representationSome background on sparsity and the K-SVD
The sparse dictionary modelIntroducing an efficient and adaptable dictionary!
Why sparse dictionaries are good for youSome uses and applications
Summary and conclusionsSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The Sparse Representation ModelWe assumes the existence of a synthesis dictionary whose columns are the atom signals.D
We model natural signals as sparse linear combinations of the dictionary atoms:
We seek exact sparsity of , meaningthat it is assumed to contain mostly zeros.D=xSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Sparse CodingThe equation for is underdetermined:Among all the solutions, we want the sparsest one:
The L0 norm counts the number of non-zeros in a vectorProblem 1 (sparse coding): given a signal x, can we find its representation over D?Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Problem 2 (noisy sparse coding): what if we only have y, a noisy version of x?Noisy Sparse Coding
Use the sparsity assumption to recover and approximate x:
Additive Gaussian noiseSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Sparse Coding AlgorithmsThe sparse coding problem NP-hard in general!Many efficient approximation algorithms:GreedyMP/OMPStOMPLARSCoSAMPIterativeFOCUSSBCRIterated ShrinkageRelaxationBasis PursuitLASSOOthersThresholdingShrinkageRandOMPSuccess bounds available, depending on the sparsity of and the amount of noiseEmpirical performance typically much better than theorySparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Orthogonal Matching Pursuit (OMP)Greedy algorithm selects atoms one at a timeInput: signal y, dictionary DOutput: sparse such that y DInitialize: S = empty atom set residual = y
Stopping condition? = all zeros except on S, non-zeros computed by min-L2NoYesSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#
PSNR = 21.4dB1 atoms / block
PSNR = 24.6dB2 atoms / block
Orthogonal Matching Pursuit (OMP)Block size: 8 x 8Overcomplete DCT dictionary, 256 atoms
Overcomplete DCT = extension of DCT with non-integer wave numbers
3 atoms / blockPSNR = 26.6dB
PSNR = 28.1dB4 atoms / block
5 atoms / blockPSNR = 29.4dB
6 atoms / blockPSNR = 30.5dB
7 atoms / blockPSNR = 31.5dB
8 atoms / blockPSNR = 32.5dBSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD algorithm:Train an explicit dictionary from examples Which Dictionary?Output.InputSet of ExamplesDTrained DictionaryX[Aharon, Elad & Bruckstein 06]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD Training AlgorithmThe examples are linear combinations of the atomsEach representation uses at most T atomsThe target function to minimize:..XD
Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD: Overview..XDInitialize DSparse CodeDictionary UpdateSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#K-SVD: Sparse Coding Stage
For the jth example
Ordinary Sparse Coding !..XDSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#K-SVD: Dictionary Update Stage
For the kth atom
(Residual)..
,XDSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#K-SVD: Dictionary Update StageWe can do better!
..
But wait! What about sparsity?XDSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#
Only some of the examples use atom dkWhen updating k, only recompute the coefficients for those examplesdkkTSolve with SVDWe want to solve:EkK-SVD: Dictionary Update Stage
~~~Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD: Summary..XDInitialize DictionarySparse CodeUsing OMPDictionary UpdateAtom-by-atom + coeffs.Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#The K-SVD: ApplicationsImage denoising [Elad & Aharon 06]Image inpainting [Raboy 07]Tomography reconstruction [Liao & Sapiro 08]Demosaicing [Mairal, Elad & Sapiro 08]Facial image compression [Bryt & Elad 08]Video denoising [Protter & Elad 09]Image scaling [Zeyde, Elad & Protter 10]Many othersSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Limitations of Explicit DictionariesUnstructured: complex to store and transmitInefficient: applied via explicit matrix multiplicationOver-parameterized: many degrees of freedom require a lot of training examplesDSingle scale: adapted to a specific signal size, inter-scale relations are not expressedUnstructured:Inefficient:Over-parameterized:Single scale:Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#35AgendaAnalysis and synthesis signal modelsTwo models are better than one
A few words on dictionary designOn dictionaries and choices
Introduction to sparse representationSome background on sparsity and the K-SVD
The sparse dictionary modelIntroducing an efficient and adaptable dictionary!
Why sparse dictionaries are good for youSome uses and applications
Summary and conclusionsSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Parametric DictionariesxAnalytic dictionariesxTrained dictionaries``Parametric DictionariesAdaptivityStructure
`Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#37Some Existing Parametric DictionariesSemi-multiscale K-SVD[Mairal, Sapiro & Elad, 08]
Iterative LS Dictionary Learning Algorithms (ILS-DLA)[Engan, Skretting & Husy, 07]Signature dictionary[Aharon & Elad, 08]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#38Sub-Atomic Particles?
K-SVD dictionary
Could the trained atoms themselves be sparse over some simpler underlying dictionary?Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#39The Sparse Dictionary Model.=.DBABase dictionaryRepresentation matrixX Base dictionarySparse matrixBASparse matrixDouble Sparsity[Rubinstein, Zibulevsky & Elad 10]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#40The Sparse Dictionary Model
BEfficiency: depends mostly on the choice of base dictionary typically an analytic dictionary.Adaptivity: by modifying the representation matrix A..ASparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Complexity of Sparse DictionariesSparse Dictionaries
Signal is d-dimensionalSignal size: n n n = nd = N
Dictionary size: N L, L=O(N)Base dictionary size: N L, separableCardinality of each sparse atom: pAnalyticDictionaries
Explicit DictionariesSparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#42
Efficiency of Sparse DictionariesOMP-Cholesky61 secSparse OMP-Cholesky13 secBatch-OMP*4.5 secSparse Batch-OMP*1.2 secExample: patches from pirate encoding time per 1,000 blocks.Signal size:n x n = n2 = N
Dictionary size:N x 2N
Base dictionary:2-D Separable
Atom sparsity: n
Encoding target:n/2 atoms
* [Rubinstein, Zibulevsky & Elad 08]Sparsity-based signal models and the sparse K-SVD algorithmRon Rubinstein#Assumptions: base dictionary and sparse dictionary are both NxL. Base dictionary is 2-d separable.t