Primal SketchIntegrating Structure and Texture

Ying Nian WuUCLA Department of Statistics

Keck MeetingApril 28, 2006

Guo, Zhu, Wu (ICCV, 2003; GMBV, 2004; CVIU, 2006)

sketchable image

input image

+

sketch graph

synthesized textures

texture regions

synthesized image

=

A Generative Model for Natural Images

Outline

Sparse coding Markov random field Primal sketch model Sketch pursuit algorithm

Sparse Coding

})(){(min ,2

, i

miii

mim

m cSBcI

Olshausen and Field (1996)

inputimage

500 bases

800 bases

matching pursuit

Matching PursuitMallat and Zhang (1993)

Symbolic representationof 300 bases

Reconstructed image

Primak sketch

Markov Property:

MRF model = Gibbs distribution

One example of neighborhood

Markov Random Fields

)))((|)(())(|)(( xIxIpxIxIp

))(exp(1

)( IUZ

Ip

Besag (1974)Geman and Geman (1984)Cross and Jain (1983)

MRF model & Image ensemble

}h)FI,histogram(:{IΩ(H) kky,x,

Image ensemble Image ensemble (Wu, Zhu & Liu, 2000)(Wu, Zhu & Liu, 2000)

|Ω(H)|1/p(I) Ω(H)I

)}FI,(λexp{)Z(

1λ)F,(I;p ky,x,

ky,x,ktexture

MRF model MRF model (Zhu, Wu & Mumford, 1997)(Zhu, Wu & Mumford, 1997)

Feature statistics: histograms of filter responses (Heeger and Bergen, 1995)

Filtering – convolution

i k

kni,mki,nm,nm, IFFI,J

original image I filter responses Jof the “dy” filter

a set of filters F

Histogram of Filter Responses

Average histogram error

800 bases

50*70 patchA sample of image ensemble with 5*13=65 parameters

observed image

sampled image from image ensemble

Primak sketch

Sparse Coding vs. MRF

MRF models target high complexity patterns.

Sparse Coding models target low complexity patterns.

N

1iii1i

N

1iii BcI Δ;B ε,BcI

constH(p(C))))H(p(I1

const: related to the dictionary

0p*)||KL(qH(q)-H(p*)

p*: fitted MRFq: any distribution

Primal Sketch Model

nsksk DDD

n

i

DxxxBxI1

sk),()()(

Image pixels = Sketchable & Non-sketchable

Sketchable: sparse coding using image primitives

Non-sketchable: feature statistics/Markov random fields

))(|)(( sknsk DIDIp

Integration:• Non-sketchable interpolates sketchable• Non-sketchable recycles failed sketch detections

Sketches

Elder and Zucker, 1998

)cos)(sin)((),( 221121 uxuxfxxB ),(sin)(cos)( 2211 lluxux

),(cos)(sin)( 2211 wwuxux

()*()() 0 sgbfaf

Sketch Graph

Vertices:1,2,3 – corners4,5,6,7 – end points8,9,10 – junctions, etc

Sketch graph

Image Primitives

a) Geometric b) Photometric

Geometric Photometric sketch image

),...,1,( niBG i

|}|exp{)(4

0d

dd VGp

Sketch Graph Model

Sketchs = Gabor clusters

Alignment across spatial and frequency domains

Non-alignment

Pool marginal histograms

Gabor filters

Sketches

Alignment

Sketch Graph

Textures

Integrating structure and texture

Model fitting First: Sketch pursuit aided by Gabors Second: Non-sketchable texturing

Sketchability test

BxNxBxIH

NxIH

),,0()()(:

),0()(:2

1

20

]))()(())([()( 22 xBxIxIBLBx

22/ LLRT

Sketch pursuit objective

||)()(4

01d

dd

n

ii VBLGL

)4,3,2,5,1(

Approximated model

sk1

),()()( DxxxBxIn

ii

nsk),()()( DxxxxI

Sketch Pursuit Phase I

input image Edge/ridge mapedge/ridge strengthProposals: a set of sketches as candidates.

Select the sketches in the order of likelihood gain. 5ΔL

Sketch Pursuit Phase IIRefinement

Evolve the sketch graph by graph operators.

Initialization Refinement

Graph Operators

operators graph change illustration

G1, G1’ create / remove a stroke

G2, G2’ grow / shrink a stroke

G3, G3’ connect / disconnect vertices

G4, G4’extend one stroke and cross

/ disconnect and combine

G5, G5’extend two strokes and cross

/ disconnect and combine

operators graph change illustration

G6, G6’combine two stroke

/ break a stroke

G7, G7’combine two parallel strokes

/ split one into two parallel

G8, G8’merge two vertices

/ split a vertex

G9, G9’ create / remove a blob

G10, G10’switch between a stroke(s) and a blob

Graph Editing

Phase I

Phase II

A

B

A B

Phase II Algorithm

1. Randomly choose a local sub-graph (S0)2. Try all 10 pair of graph operators 1~ 5 steps, to

generate a set of new graph candidates (S4,S2,S3)3. Compare all new graph candidates4. Select the one with the largest posterior gain (e.g.

S4), accept the new graph. If no positive gain, no update.

5. Repeat 1~4 until no updateS0

S1 S2 S3

G1

S4

G3

G4

synthesized texturestexture regions

K-mean clusteringHistograms in 7x7 window7 filters x 7 bins

Primal Sketch Model Result

input image

reconstructed image

sketch graph

sketchable image

input image sketch graph

reconstructed image sketchable image

input image reconstructed imagesketch graph

input image sketch graphreconstructed image

Lossy Image Coding

sketchable image

sketch graph

codes for the vertices: 152*2*9 = 2,736 bits

codes for the strokes: 275*2*4.7 = 2, 585 bits

codes for the profiles: 275*(2.4*8+1.4*2) = 6,050 bits

Total codes for structures (18,185 pixels) 11,371 bits = 1,421 bytes

synthesized textures

texture regions

codes for the region boundaries: 3659*3 = 10,977 bits

codes for the texture histograms: 7*5*13*4.5 = 2,048 bits

Total codes for textures (41,815 pixels) 13,025 bits = 1,628 bytes

Total codes for whole image (72,000 pixels), 3,049 bytes

Scaling

Scaling

Scaling

Wu, Zhu, Bahrami, Li (2006)

JPEG Entropy vs Scale

0

0.1

0.2

0.3

0.4

0.5

0.6

1 2 3 4 5 6 7 8

Scale

JPE

G E

ntro

py p

er P

ixel

Scaled Squares

White Noise