image warping - uc berkeleyvis.berkeley.edu/courses/cs294-69-fa11/www/... · energy minimization...

Image Warping

http://www.jeffrey-martin.com

[Some slides from K. Padalkar, S. Avidan, A. Shamir, A. Efros, S. Seitz, and Y. Wang]

Image  Manipula-on  and  Computa-onal  PhotographyCS294-­‐69  Fall  2011

Robert  Carroll

What is a warp?

What is a warp?

Original

What is a warp?

Original Warped

Warps are spatial transformations:Points map to points, but colors don’t change.

Applications

Panorama Stitching Resizing/Retargeting

Wide-Angle Distortion Shape Manipulation

Video Stabilization, Stereoscopic Depth Editing, Object Removal, ...

[Igarashi  et  al.  09]  

Applications

Panorama Stitching Resizing/Retargeting

Wide-Angle Distortion Shape Manipulation

Video Stabilization, Stereoscopic Depth Editing, Object Removal, ...

[Igarashi  et  al.  09]  

Image Transformations

f

x

f

x

f

x

Tf

x

image warping: change domain of image

g(x) = f(T(x))

image filtering: change range of imageg(x) = T(f(x))

T

Image Transformations

image filtering: change range of imageg(x) = T(f(x))

T

T

f

f g

g

image warping: change domain of image

g(x) = f(T(x))

Types of Warps

rotation aspect perspective cylindrical

Mesh:

Parametric:

Discrete:

-­‐  Globally  defined  by  small  #  of  parameters

-­‐  Smooth  (rubber  sheet)  transformaAon

-­‐  Each  output  pixel  can  be  an  arbitrary  input  pixel

[Barnes  et  al.  09]  

[Wang  et  al.  09]  

The  Image  Resizing/RetargeAng  Problem

The  Image  Resizing/RetargeAng  Problem

Image    Retarge-ng    Objec-ves

1. Change  size

2. Preserve  the  important  content  and  structures3. Limit  ar*facts  created

Tradi-onal    Methods

•Letterbox wastes pixels•Squeeze/Hybrid introduce distortions•Cropping removes important parts

original squeeze crop hybridletterbox

Many    Exis-ng    Resizing    Methods

Shai Avidan and Ariel Shamir, Proc. SIGGRAPH, 2007

Seam Carving for Content-Aware Image Resizing

Seam    Carving    Method

Seam    Carving    Method

• “Seam  Carving  for  Content-­‐Aware  Image  Resizing,”      S.  Avidan  and  A.  Shamir,  Proc.  SIGGRAPH,  2007

Seam    Carving    Method

• “Seam  Carving  for  Content-­‐Aware  Image  Resizing,”      S.  Avidan  and  A.  Shamir,  Proc.  SIGGRAPH,  2007

• In  Photoshop  called  “content  aware  scaling”

Seam    Carving    Method

• “Seam  Carving  for  Content-­‐Aware  Image  Resizing,”      S.  Avidan  and  A.  Shamir,  Proc.  SIGGRAPH,  2007

• In  Photoshop  called  “content  aware  scaling”• Main  idea:    Remove  the  least  noHceable  pixels

– How?    Define  an  “energy  func-on”  that  measures  how  perceptually  no-ceable  each  pixel  is

Seam    Carving    Method

• “Seam  Carving  for  Content-­‐Aware  Image  Resizing,”      S.  Avidan  and  A.  Shamir,  Proc.  SIGGRAPH,  2007

• In  Photoshop  called  “content  aware  scaling”• Main  idea:    Remove  the  least  noHceable  pixels

– How?    Define  an  “energy  func-on”  that  measures  how  perceptually  no-ceable  each  pixel  is

• Remove  the  pixels  with  “low  energy”  and  avoid  removing  pixels  with  “high  energy”– How?    Define  a  criterion  for  picking  which  pixels  to  remove

Possible    Energy    Func-ons

• Edgeness– Gradient  magnitude

• Entropy• HOG  (Histogram  of  Gradient)• Saliency• …Caveat: No single energy function performs well across all images

Possible    Energy    Func-ons

• Edgeness– Gradient  magnitude

• Entropy• HOG  (Histogram  of  Gradient)• Saliency• …Caveat: No single energy function performs well across all images

Pixel    Removal    Criteria• Op#mal:    remove  the  k  pixels  with  lowest  energy• Output  image  no  longer  rectangular  

Input imageEnergy image

Output image

Pixel    Removal    Criteria• Remove  k  pixels  with  lowest  energy  in  each  row• No  visual  coherence  between  adjacent  rows

Pixel    Removal    Criteria• Column:    Remove  whole  column  with  lowest  energy• Frequently  introduces  arHfacts

Seam    Defini-on

Seam    Defini-on• VerAcal  Seam

is  an  8-­‐connected  path  of  pixels  in  an  n  x  m  image  from  top  to  boQom,  containing  one,  and  only  one,  pixel  in  each  row  of  the  image:

Seam    Defini-on• VerAcal  Seam

is  an  8-­‐connected  path  of  pixels  in  an  n  x  m  image  from  top  to  boQom,  containing  one,  and  only  one,  pixel  in  each  row  of  the  image:

Seam    Defini-on• VerAcal  Seam

is  an  8-­‐connected  path  of  pixels  in  an  n  x  m  image  from  top  to  boQom,  containing  one,  and  only  one,  pixel  in  each  row  of  the  image:

Seam    Energy

• Energy  of  a  Seam

• Minimum  Energy  Seam

Pixel    Removal    Criteria• Seam:    Remove  the  verHcal  curve  of  lowest  energy

Pixel    Removal    Effec-veness

How    to    Efficiently    Compute    Best    Seam?

• Use  Dynamic  Programming  to  find  lowest  energy  seam  in  linear  Hme

1. Forward  Pass  (top  row  to  boWom  row  for  finding  verHcal  seam)– Define  M(i,j)  =  cumula-ve  energy  at  (i,j)– M(1,j)  =  e(1,j)– M(i,j)  =  e(i,j)  +  min(M(i-­‐1,j-­‐1),  M(i-­‐1,j),  M(i-­‐1,  j+1))

– Find  minimum  value  in  last  row:  minj  M(n,j)

2. Backward  Pass  (boWom  row  to  top  row)– Trace  back  path  from  pixel  in  boZom  row  with  min  value  to  top  row

Seams

Seams over energy image Seams over input image

Shrink    Image    in    1    Dimension

Shrink    Image    in    1    Dimension

• Change  the  image  from  size  n  ×  m  to  n  ×  mʹ′– assume    mʹ′  <  m

Shrink    Image    in    1    Dimension

• Change  the  image  from  size  n  ×  m  to  n  ×  mʹ′– assume    mʹ′  <  m

• Remove  m-­‐mʹ′  =  c  seams  successively

Shrink    Image    in    1    Dimension

• Change  the  image  from  size  n  ×  m  to  n  ×  mʹ′– assume    mʹ′  <  m

• Remove  m-­‐mʹ′  =  c  seams  successively

Shrink    Image    in    1    Dimension

• Change  the  image  from  size  n  ×  m  to  n  ×  mʹ′– assume    mʹ′  <  m

• Remove  m-­‐mʹ′  =  c  seams  successively

Seam Carving

Shrink    Image    in    1    Dimension

• Change  the  image  from  size  n  ×  m  to  n  ×  mʹ′– assume    mʹ′  <  m

• Remove  m-­‐mʹ′  =  c  seams  successively

Scaling

Shrink    Image    in    Both    Dimensions:Op-mal    Seam    Ordering

Shrink    Image    in    Both    Dimensions:Op-mal    Seam    Ordering

• Change  the  image  from  size  n  ×  m  to  nʹ′  ×  mʹ′– assume    mʹ′  <  m  and  nʹ′  <  n

Shrink    Image    in    Both    Dimensions:Op-mal    Seam    Ordering

• Change  the  image  from  size  n  ×  m  to  nʹ′  ×  mʹ′– assume    mʹ′  <  m  and  nʹ′  <  n

• What  is  the  best  order  for  seam  carving?– Remove  verHcal  seams  first?  – Horizontal  seams  first?  – Alternate  between  the  two?

Op-mal    Seam    Ordering

Op-mal    Seam    Ordering

• Solve  opHmizaHon  problem:

Op-mal    Seam    Ordering

• Solve  opHmizaHon  problem:

Op-mal    Seam    Ordering

• Solve  opHmizaHon  problem:

where  k  =  r+c,  r  =  (m−mʹ′),  c  =  (n−nʹ′)  and  αi  is  a  parameter  that  determines  if  at  step  i  we  remove  a  horizontal  or  verAcal  seam:  α  ∈  {0,1}

Op-mal    Seam    Ordering

Op-mal    Seam    Ordering

• Transport  map– Matrix  of  size  n  ×  m– Each  element  T(r,c)  holds  the  minimal  cost  needed  to  obtain  an  image  of  size  n−r  × m−c  

Op-mal    Seam    Ordering

• Transport  map– Matrix  of  size  n  ×  m– Each  element  T(r,c)  holds  the  minimal  cost  needed  to  obtain  an  image  of  size  n−r  × m−c  

Enlarging    Images

Enlarging    Images� Method  1:    Compute  the  opAmal  verAcal  (horizontal)  seam  s  in  image  and  duplicate  the  pixels  in  s  by  averaging  them  with  their  le`  and  right  neighbors  (top  and  boQom  in  the  horizontal  case)

� O`en  will  choose  the  same  seam  at  each  iteraAon,  producing  noAceable  stretching  arAfact

Enlarging    Images� Method  1:    Compute  the  opAmal  verAcal  (horizontal)  seam  s  in  image  and  duplicate  the  pixels  in  s  by  averaging  them  with  their  le`  and  right  neighbors  (top  and  boQom  in  the  horizontal  case)

� O`en  will  choose  the  same  seam  at  each  iteraAon,  producing  noAceable  stretching  arAfact

Enlarging    Images� Method  1:    Compute  the  opAmal  verAcal  (horizontal)  seam  s  in  image  and  duplicate  the  pixels  in  s  by  averaging  them  with  their  le`  and  right  neighbors  (top  and  boQom  in  the  horizontal  case)

� O`en  will  choose  the  same  seam  at  each  iteraAon,  producing  noAceable  stretching  arAfact

Enlarging    Images

� Method  2:    To  enlarge  width  by  k,  compute  top  k  verAcal  seams  (for  removal)  and  duplicate  each  of  them

Content    Amplifica-on

Content    Amplifica-on� Scale  the  image;  this  will  scale  everything,  “content”  as  well  as  “non-­‐content”

� Shrink  the  scaled-­‐image  using  seam  carving,  which  will  (hopefully)  carve  out  the  non-­‐content  part

Content    Amplifica-on� Scale  the  image;  this  will  scale  everything,  “content”  as  well  as  “non-­‐content”

� Shrink  the  scaled-­‐image  using  seam  carving,  which  will  (hopefully)  carve  out  the  non-­‐content  part

Content    Amplifica-on� Scale  the  image;  this  will  scale  everything,  “content”  as  well  as  “non-­‐content”

� Shrink  the  scaled-­‐image  using  seam  carving,  which  will  (hopefully)  carve  out  the  non-­‐content  part

Object    Removal

Object    Removal� User  marks  the  target  object  to  be  removed� Force  seams  to  pass  through  marked  pixels� To  obtain  the  original  image  size,  use  seam  inserAon

Object    Removal� User  marks  the  target  object  to  be  removed� Force  seams  to  pass  through  marked  pixels� To  obtain  the  original  image  size,  use  seam  inserAon

Object    Removal� User  marks  the  target  object  to  be  removed� Force  seams  to  pass  through  marked  pixels� To  obtain  the  original  image  size,  use  seam  inserAon

Object    Removal

� One  shoe  removed  (and  image  enlarged  to  original  size)

input result

Object    Removal� Object  marking  to  prevent  unwanted  results:  mark  regions  where  seams  must  not  pass

Object    Removal� Object  marking  to  prevent  unwanted  results:  mark  regions  where  seams  must  not  pass

Object    Removal� Object  marking  to  prevent  unwanted  results:  mark  regions  where  seams  must  not  pass

Object    Removal� Object  marking  to  prevent  unwanted  results:  mark  regions  where  seams  must  not  pass

Mul--­‐Size    Images

H VInput

Mul--­‐Size    Images• Methods  menAoned  so  far  are  not  real-­‐Ame

H VInput

Mul--­‐Size    Images• Methods  menAoned  so  far  are  not  real-­‐Ame• We  calculate  best  seam,  remove  it,  calculate  the  next  

seam  based  on  new  image,  etc.

H VInput

Mul--­‐Size    Images• Methods  menAoned  so  far  are  not  real-­‐Ame• We  calculate  best  seam,  remove  it,  calculate  the  next  

seam  based  on  new  image,  etc.• Compute  Index  map,  V,  of  size  n  ×  m  that  encodes,  for  

each  pixel,  the  index  of  the  seam  that  removed  it,  i.e.,  V(i,  j)  =  t  means  pixel  (i,  j)  was  removed  by  the  tth  seam  removal  iteraAon

H VInput

Failures

� Too  much  content� No  space  for  seam  to  avoid  content

Optimized Scale-and-Stretch for Image Resizing

1Yu-Shuen Wang, 2Chiew-Lan Tai, 3Olga Sorkine, 1Tong-Yee Lee

1National Cheng KungUniversity, Taiwan

3New York University2Hong Kong University of Science & Technology

Scale-and-stretch warp

• Allow important regions to uniformly scale• Find optimal local scaling factors by global optimization• Result: preserve the shape of important regions, distort non-important ones

importance map

Importance map

saliency map[Itti et al. 98]

x =

importance map

gradients only importance map

The warping mechanism

• Grid mesh, preserve the shape of the important quads

• Optimize the location of mesh vertices, interpolate image

quads with high importance uniform scaling

quads with low importance allow non-uniform scaling

The warping mechanism

• Grid mesh, preserve the shape of the important quads

• Optimize the location of mesh vertices, interpolate image

Deformation Energy• Per quad

sf – uniform scaling

f

vi

vj

vʹ′i

vʹ′j

Deformation Energy• Total deformation energy

• Quadratic energy in vʹ′, can minimize by solving a linear system of equations

importance weights

Constraints

•Corner vertices•Horizontal/vertical sliding

Deformation energy

• Problem: grid lines bend a lot; warp not so smooth

Deformation energy

• Problem: grid lines bend a lot; warp not so smooth

Grid line bending energy

• Attempt to keep original edge orientations but allow length scaling

note thenonlinearfactor

f

vi

vj

vʹ′i

vʹ′j

Energy minimization

• Total energy is nonlinear in vʹ′

• Iterative minimization with tricks• Keep sf and lij as additional variables• Do alternating minimization steps Fix sf and lij and optimize vʹ′ Compute new sf and lij• Sparse direct solver for the linear system and reuse the matrix factorization to gain speed.

Grid line bending energy

• With the bending term added

Results

Results

original SC indirect SC

Scale-and-Stretch

Mesh can move in 2D, with seam carving pixels only move in 1D

original SC indirect SC

Scale-and-Stretch

Results

original SC indirect SC

Scale-and-Stretch

Results

Optimizing Content-Preserving Projections for Wide-Angle Images

Robert CarrollManeesh Agrawala

University of California, Berkeley

Aseem AgarwalaAdobe Systems, Inc.

Rectilinear Lens

© Keith Cooper

Fisheye

© Flickr user kirainet

Cylindrical Panorama

© Flickr user Seb Przd

Viewing Sphere

Viewing Sphere

Viewing Sphere

Viewing Sphere

Courtesy of Flickr user brokendrum70

Cartography

Linear Perspective Projection

Linear Perspective Projection

Stereographic Projection

Stereographic Projection

Cylindrical Projection

Mercator Projection

Optimized Projection

Optimized Projection

GoalGiven a wide-angle image, produce a projection that preserves

straight lines in the scene and the shapes of objects

Our Approach

Mesh the viewing sphere

De!ne mapping constraints

Optimize weighted energy function

Viewing Sphere Mesh

Viewing Sphere Mesh

Viewing Sphere Mesh

Viewing Sphere Mesh

Three Properties

1. Conformality

2. Straight Lines

3. Smoothness of mapping

h

k

Conformality

k

h

Conformality

Lines

Lines

Lines

Should  be  zero

Lines

Should  be  zero

Lines

Should  be  zero

Smoothness

Smoothness

Smoothness

Smoothness

What’s left?

• Cannot satisfy all constraints exactly

– De!ne weighted least-squares energy terms

• Iterative non-linear optimization

Etotal= w1Econformality + w2Elines + w3Esmoothness

Lines

Should  be  zero

Lines

Should  be  zero

Lines

Should  be  zero

Lines

fixed n

Lines

fixed n

�(1� �)

Lines

Should  be  zero

Lines

fixed �

Lines

fixed �

Lines

fixed n

Lines

fixed n

Lines

fixed �

Lines

fixed �

Implementation Details

• Converges in 8 iterations• Up to 160,000 vertices• 15s - 1m, depending on !eld of view

Results

Input/Output

Input/Output

Perspective Mercator

Our ResultStereographic

Perspective Mercator

Our ResultStereographic

Perspective Mercator

Our ResultStereographic

Image  CourtesyFlickr  user

Aldo

Perspective Mercator

Our Result

Stereographic

ImageCourtesyJeff  Chien

PerspectiveMercator

Our ResultStereographic

Input (© Flickr user Mike Schinkel)

Our ResultZorin

Our Result

Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView

Mercator

Our Result

Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView

Mercator

Our Result

Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView

Mercator

Limitations

• Must  be  cropped

Limitations

• Some  stretching  near  poles

Limitations

• Some  stretching  near  poles

Geodesic  Grid

• Lines  that  stretch  across  the  full  field  of  view

Limitations

© Flickr user Editor B

Thanks!