ee565 advanced image processing copyright xin li 2009- 20121 why do we need image model in the first...

EE565 Advanced Image Processing Copyright Xin Li

2009-2012 1

Why do we Need Image Model in the first place? Any image processing algorithm has to

work on a collection (class) of images instead of a single one

Mathematical model gives us the abstraction of common properties of the images within the same class

Model is to hypothesis what images are to observation data In physics, can F=ma explain the

relationship between force and acceleration? In image processing, can this model fit this class of images?

Birdview


2009-2012 2

transform models

patch models

PDE models

synthesis/inpainting Encoding/decoding… …denoising

Pyramid-based /DCT-based

NLM/BM3D

TV, MCD/PMD

Image quilting/BM3D-CS

Wavelet/TI thresholding

EZWSPIHTEBCOT…

Reaction-Diffusion/TV

NLM/BM3D

TV, MCD/PMD


PDE-based Denoising Think of image as a 3D surface: a mapping

from domain (x,y) to range u(x,y) Geometry-driven ideas

How to measure “changes”: e.g., total-variation vs. surface area

Importance of direction: from isotropic diffusion to anisotropic diffusion

Discrete implementation: finite—difference method


2009-2012 3


2009-2012 4

Wavelet-based Denoising

where

Laplacian (signal, heavy-tail)

Gaussian (noise, light-tail)

P: shape parameter: variance parameter

qq

pp

Bilateral Filtering (a Novel idea)

output input

S

IIIGGW

IBFq

qqpp

p qp ||||||1

][rd

pp

reproducedfrom [Durand 02]


2009-2012 6

Patch-based Denoising

WD

T T-1ThresholdingWD =

Noisy patches Denoised patches

Connections Equivalence between TV and Haar-

wavelet thresholding Why BM3D works so much better?

Heuristics: Nonlocal similar patches around edges/textures

Theory: more accurate signal variance estimation

Lesson learned: Markov model is wrong


2009-2012 7


2009-2012 8

TV-Inpainting Example

(Courtesy: Jackie Shen, UMN MATH)


2009-2012 9

Wavelet-domain Histogram Matching


2009-2012 10

Joint PDF of Wavelet Coefficients

Neighborhood I(Q): {Left,Up,cousin and aunt}

X=

Y=

Joint pdf of two correlated random variables X and Y

Can you use this model to interpret why EZW works?


2009-2012 11

Wavelet-DomainParametric Texture Models

original

synthesized


2009-2012 12

Locality Revisited: “relativity theory” for image processing

Input image

),...,|(),...,|( 111 Nkkkkk XXXPXXXP N past samples

The definition of local neighborhood has to be relative


2009-2012 13

Nonparametric Texture Synthesis: Image Quilting


2009-2012 14

Connections Local inpainting vs. global synthesis

Observation of scale: think of inpainting the boundary of a leaf vs. inpainting the occluded region of a bush

Selection of scanning order/window size Deterministic vs. statistical

Regularization vs. prior (no fundamental difference, just different languages)

Variational (energy-based) vs. set theoretic (projection-based)

How do we minimize E(u)?

Image Coding as Painting


2009-2012 15

What (intensity uncertainty) vs. Where (location uncertainty)


2009-2012 16

Projection-based post-processing

● Quantization (observation data) set

}][|{},|{, gfQfCgTffCCCC QTQTQT

WT Quantization

● Regularization (prior) constraint set

NLM/BM3D

TV, MCD/PMD


X0

X1

X2

X∞

C1

C2


2009-2012 17

Summary on Models Image models are at the foundation of any

image processing algorithms To gain a deeper understanding of why xxx

works, try to distill the underlying model “All models are wrong; but some are

useful” What really matters is how well your model

matches with the observation data “Nature is not economical of structures but

of principles” So we don’t need to work out zillions of problems/apps

Birdview


2009-2012 18

transform models

patch models

PDE models

synthesis/inpainting Encoding/decoding… …denoising

Pyramid-based /DCT-based

NLM/BM3D

TV, MCD/PMD

Image quilting/BM3D-CS


EZWSPIHTEBCOT…

Reaction-Diffusion/TV

NLM/BM3D

TV, MCD/PMD


PDE-based Image Processing

We only discussed the application of PDE in image restoration in this class

More successful applications are segmentation-related Active contour model (snake) Active contour without edges

(Chan&Vese’2001) Mumford-Shah model


2009-2012 19

Active Contour


2009-2012 20

Wavelet-based Image Processing

The most successful application is lossy image compression

Why is it less effective on other non-coding applications? Segmentation/Classification: linearity Detection/Recognition: invariance

Beyond wavelets vs. Beyond Hilbert spaceEE565 Advanced Image

Processing Copyright Xin Li 2009-2012 21

Beyond Wavelets


2009-2012 22

Daubechies’ wavelet,1988 Do&Vetterli’s contourlet,2005 Bell&Sejnowski’ICA,1996 Elad&Aharon’K-SVD,2006

Dictionary construction Dictionary learning

Patch-based Image Processing We have focused on the applications at

the low-level vision (e.g., denoising, inpainting, post-processing)

Highly effective for high-level vision tasks (e.g., classification, recognition) too SIFT/SURF/HOG/LBP/CARD/BRIEF… Importance of clustering: kmeans vs. kNN


2009-2012 23

The Emerging Trend


2009-2012 24

Low-level vision:Restoration etc.

High-level vision:Recognition etc.

Bridge: SVM = Sparse coding Biologically-plausible

local responses

global responses

What are the Killer Applications? (Biased View) Computational imaging

Gigapixel cameras, light field cameras Human computer interaction

Kinect, touch-screen, siri Intelligent transportation system

Mobileye, real-time traffic monitoring Healthcare-related

The paygrade of radiologists is ridiculously high Cyberlearning

“A picture is worth a thousand words”EE565 Advanced Image

Processing Copyright Xin Li 2009-2012 25


2009-2012 26

Summary on Practice MATLAB provides a user friendly

platform for testing your ideas You can see what you have done

C/C++ programming skills are a plus Efficient implementation could make a

difference (e.g., SPIHT vs. EZW) Team work is becoming more valued

than the past (e.g., Samsung story)

ee565 advanced image processing copyright xin li 2009- 20121 why do we need image model in the first...

Documents

image processing algorithm

image model

markov model

domain x

single onemathematical

collection class of

patchbased denoisingwdtt

correlated random variables