besov bayes chomsky plato richard baraniuk rice university dsp.rice.edu joint work with hyeokho choi...
TRANSCRIPT
Besov
Bayes
Chomsky
Plato
Richard Baraniuk Rice Universitydsp.rice.edu
Joint work with Hyeokho ChoiJustin RombergMike Wakin
Multiscale Geometric Image Analysis
Low-Level Image Structures
• Smooth/texture regions
• Edge singularities along smooth curves (geometry)
geometry
texture
texture
Computational Harmonic Analysis
• Representation
Fourier sinusoids, Gabor functions, wavelets, curvelets, Laplacian pyramid
basis, framecoefficients
Computational Harmonic Analysis
• Representation
• Analysis study through structure of might extract features of interest
• Approximation uses just a few termsexploit sparsity of
basis, framecoefficients
Multiscale Image Analysis
• Analyze an image a multiple scales
• How? Zoom out and record information lost in wavelet coefficients
info 1 info 2 info 3
…
Wavelet-based Image Processing
• Standard 2-D tensor product wavelet transform
Transform-domain Modeling
• Transform-domain modeling and processing
transform
coefficient model
Transform-domain Modeling
• Transform-domain modeling and processing
transform vocabulary
coefficient model
Transform-domain Modeling
• Transform-domain modeling and processing
• Vocabulary + grammar capture image structure
• Challenging co-design problem
transform vocabulary
coefficient grammar model
Nonlinear Image Modeling
• Natural images do not form a linear space!
• Form of the “set of natural images”?
+ =
Set of Natural Images
Small: N x N sampled images comprise an
extremely small subset of RN2
Set of Natural Images
Small: N x N sampled images comprise an
extremely small subset of RN2
Set of Natural Images
Small: N x N sampled images comprise an
extremely small subset of RN2
Complicated: Manifold structure
RN2
Wedge Manifold
Wedge Manifold
3 Haar wavelets
Wedge Manifold
3 Haar wavelets
Stochastic Projections
• Projection
= “shadow”
Higher-D Stochastic Projections
• Model (partial) wavelet coefficient joint statistics
• Shadow more faithful to manifold
Manifold Projections …
Multiscale Grammars for
Wavelet Transforms
Wavelet Statistical Properties
Marginal Distribution
0
• manysmall Scoefficients
• smooth regions
Marginal Distribution
0
• a fewlarge Lcoefficients
• edgeregions
Wavelet Mixture Model
state: S or L
wc: Gaussian withS or L variance
Magnitude Correlations
• Persistenceof wc’s onquadtree
• S S
• L L
Wavelet Hidden Markov Tree
state: S or L
wc: Gaussian withS or L variance
A
Wavelet Statistical Models
• Independent wc’s w/ generalized Gaussian marginal – processing: thresholding (“coring”)– application: denoising
• Correlated wc magnitudes– zero tree image coder [Shapiro]
– estimation/quantization (EQ) model [Ramchandran, Orchard]
– JPEG2000 encoder– graphical models:
• Gaussian scale mixture [Wainwright, Simoncelli]
• hidden Markov process on quadtree (HMT)• processing: tree-structured algorithms
EM algorithm, Viterbi algorithm, …
– applications: denoising, compression, classification, …
Barbara
Barbara’s Books
Denoising Barbara’s Books
nois
y b
ooks
WT t
hre
shold
ing
DW
T H
MT
Denoising Barb’s Books
Wave
lets a
nd Subband C
oding
Vetterli
and K
ovace
vic
My
Life
as
a D
og
Pari
s H
ilton
Numeric
al Analys
is of W
avelet
Methods
Albert
Cohen
HMT Image Segmentation
HMT Image Segmentation
Zero Tree Compression
• Idea: Prune wavelet subtrees in smooth regions
– tree-structured thresholding
– spend bits on (large) “significant” wc’s
– spend no bits on (small) wc’s…
zerotree symbol Z = {wc and all decendants = 0}
Z
New Multiscale Vocabularies
• Geometrical info not explicit
• Modulations around singularities (geometry)
• Inefficient -large number of significant wc’s cluster around edge contours, no matter how smooth
Wavelet Modulations
• Wavelets are poor edge detectors• Severe modulation effects
Wavelet Modulations
• Wavelet wiggles…so its inner product with a singularity wiggles
L
S
L
S
scale
signal
Wavelet Modulation Effects
• Wavelet transform of edge
Wavelet Modulation Effects
• Wavelet transform of edge
• Seek amplitude/envelope
• To extract amplitude need coherent representation
1-D Complex Wavelets [Grossman, Morlet, Lina, Abry, Flandrin, Mallat, Bernard, Kingsbury, Selesnick, Fernandes, …]
• real waveleteven symmetry
imaginary waveletodd symmetry
• Hilbert transform pair(complex Gabor atom)
• 2x redundant tight frame• Alias-free; shift-invariant• Coherent wavelet representation (magnitude/phase)
2-D Complex Wavelets[Lina, Kingsbury, Selesnick, …]
• Even/odd real/imag symmetry• Almost Hilbert transform pair
(complex Gabor atom)• Almost shift invariant• Compute using 1-D CWT
-75 +75 +45 +15 -15 -45
real
imag
• 4x redundant tight frame• 6 directional subbands
aligned along 6 1-D manifold directions
• Magnitude/phase
Wavelet Image Processing
Coherent Wavelet Processing
real part
imaginarypart
+i
Coherent Wavelet Processing
|magnitude|
x exp(i phase)
Coherent Image Processing [Lina]
magnitude
FFT
Coherent Image Processing [Lina]
magnitude phase
FFT
Coherent Image Processing [Lina]
magnitude phase
FFT
CWT
Coherent Wavelet Processing
feature magnitude phase
1 edge L coherent
“speckle” L incoherent> 1 edge
smooth S undefined
Coherent Segmentation
feature magnitude phase
1 edge L coherent
“speckle” L incoherent> 1 edge
smooth S undefined
Edge Geometry Extraction
r
• Extract 1-D edge geometry from piecewise smooth image
wedgelet
• Goal: Extract r and locally
Edge Geometry Extraction
r• CWT magnitude encodes angle
• CWT phase encodes offset r
Edge Geometry Extraction in 2-Doriginal estimated
Multiscale Edge Geometry Grammarfor Wavelet Transforms
• Inefficient -large number of significant WCs cluster around edge contours, no matter how smooth
Cartoon Processing
2-D Dyadic Partitions for Cartoons
• C2 smooth regions
• separated by edge discontinuities along C2 curves
2-D Dyadic Partitions for Cartoons
• Multiscale analysis
• Partition; not a basis/frame
• Zoom in by factor of 2 each scale
2-D Dyadic Partition = Quadtree
• Multiscale analysis
• Partition; not a basis/frame
• Zoom in by factor of 2 each scale
• Each parent node has 4 children at next finer scale
Wedgelet Representation [Donoho]
• Build a cartoon using wedgelets on dyadic squares
• Choose orientation (r,) from finite dictionary (toroidal sampling)
• Quad-tree structure
deeper in tree finer curve approximation
r
2-D Wedgelets
2-D Wedgelets
Wedgelet Representation
• Prune wedgelet quadtree to approximate local geometry (adaptive)
• Decorate leaves with (r,) parameters
Wedgelet Inference
• Find representation / prune tree to balance a fidelity vs. complexity trade-off
• For Comp(W) – need a model for the wedgelet representation(quadtree + (r,))
• Donoho: Comp(W) = #leaves
#Leaves Complexity Penalty
• Accounts for wedgelet partition size, but not wedgelet orientation
#leaves #leaves
“smooth”“simple”
“rough”“complex”
Multiscale Geometry Model (MGM)
Multiscale Geometry Model (MGM)
• Decorate each tree node with orientation (r,) and then model dependencies thru scale
• Insight: Smooth curve Geometric innovations small at fine scales
• Model: Favor small innovations over large innovations (statistically)
Multiscale Geometry Model (MGM)
• Wavelet-like geometry model: coarse-to-fine prediction
– model parent-to-child transitions of orientations
small innovations
large
MGM
• Wavelet-like geometry model: coarse-to-fine prediction
– model parent-to-child transitions of orientations
• Markov-1 statistical model– state = (r,) orientation of wedgelet – parent-to-child state transition matrix
MGM
• Markov-1 statistical model
• Joint wedgelet Markov probability model:
• Complexity = Shannon codelength = = number of bits to encode
MGM and Edge Smoothness
“smooth”“simple”
“rough”“complex”
MGM Inference
• Find representation / prune tree to balance the fidelity vs. complexity trade-off
• Efficient O(N) solution via dynamic programming
MGM Approximation
• Find representation / prune tree to balance the fidelity vs. complexity trade-off
• Optimal L2 error decay rate for cartoons
single scale multiscale
• Choosing wedgelets = rate-distortion optimization
Wedgelet Coding of Cartoon Images
Shannon code length
to encode
Joint Texture/Geometry Modeling
• Dictionary D = {wavelets} U {wedgelets}
• Representation tradeoff: texture vs. geometry
Zero Tree Compression
• Idea: Prune wavelet subtrees in smooth regions
– tree-structured thresholding
– spend bits on (large) “significant” wc’s
– spend no bits on (small) wc’s…
zerotree symbol Z = {wc and all decendants = 0}
Z
Zero Tree Compression
• Label pruned wavelet quadtree with 2 states
zero-tree - smooth region (prune)significant - edge/texture region (keep)
smooth
smooth
smooth
not smooth
S
S
S
SS
S
ZZ
Z
Z: all wc’s below=0
Wedgelet Trees for Geometry
• Label pruned wavelet quadtree with 3 states
zero-tree - smooth region (prune)geometry - edge region (prune)significant - texture region (keep)
• Optimize placement of Z, G, S by dyn. programming
smooth smooth
texture
geometry
S
S
S
SS
S
ZZ
G
G: wc’s below are wc’s of a wedgelet tree
Optimality of Wedgelets + Wavelets
• Theorem For C2 / C2 images, optimal asymptotic L2 error decay
and near-optimal rate-distortion
C2 contour (1-d)
C2 texture (2-d)
Practical Image Coder• Wedgelet-SFQ (WSFQ) coder
builds on SFQ coder [Xiong, Ramchandran, Orchard]
• At low bit rates, often significant improvement in visual quality over SFQ and JPEG-2k (much sharper edges)
• Bonus: WSFQ representation contains explicit geometry information
Improvement overJPEG-2000PSNR dB
bits per pixel
WSFQ
SFQ
Wet Paint Test Image
JPEG Compressed (DCT)PS
NR
= 2
6.3
2dB
@ 0
.01
03
bpp
JPEG2000 Compressed (Wavelets)PS
NR
= 2
9.7
7dB
@ 0
.01
03
bp
p
WSFQ CompressedPS
NR
= 3
0.1
9dB
@ 0
.01
02
bpp
WSFQ Contour Trees
ori
gin
al
JPEG
20
00
WS
FQS
FQ
Gain Over JPEG2000 – Cameraman
WSFQ
SFQ
40% MSE improvement
ExtensionsS
S
S
SS
S
SZ
W
Z D
W
zerotree
wedgeprint
S S
S S
ExtensionsS
B
S
BB
B
DZ
W
Z D
W
zerotree
wedgeprint
B B
B B
barprint
DCTprint
“Coifman’s Dream”
ExtensionsS
S
S
SS
S
SZ
W
Z D
W
zerotree
wedgeprint
S S
S S
ExtensionsS
B
S
BB
B
DZ
W
Z D
W
zerotree
wedgeprint
B B
B B
barprint
DCTprint
“Coifman’s Dream”
ExtensionsS
B
S
BV
B
DZ
W
Z D
W
zerotree
wedgeprint
B S
B B
Eeroprint
barprint
DCTprint
Bars / Ridges
16x16 image block
Multiple Wedgelet Coding
80 bits to jointly encode 5 wedgelets
“Barlet” Coding
22 bits to encode 1 barlet
(fat edgelet/beamlet)
Periodic Textures
DCTprint
5dB
DCT
wavelets
# terms
PS
NR
32x32 block = 1024 pixels
DCTprint
4x fewer coefficients
DCT
wavelets
# terms
PS
NR
32x32 block = 1024 pixels
Summary
• Wavelets and other bases are vocabularies• Image structure induces coefficient grammar• Statistical models
– extract higher-dimensional joint statistics via quadtrees, wedgelets, wedgeprints, …
• New vocabularies– curvelets, bandelets, complex wavelets, …– random projections [Candes, Romberg, Tao; Donoho]
• Grand challenge– rather than low-dimensional projections, work on the
manifold– potential for image processing, compression, ATR, …