Optimizing Content-Preserving Projections for Wide-Angle Images

ACM SIGGRAPH 2009Robert Carroll (University of California, Berkeley)Maneesh Agrawal (University of California, Berkeley)Aseem Agarwala (Adobe Systems, Inc.)

Outline

• Introduction• Wide-angle projection• Approach• Results• Future work

Introduction

Perspective Mercator Stereographic Paper result

The space of wide-angle projections

• Wide-angle projections should maintain the following properties:– Shape constancy– Orientation constancy

• There is no wide-angle projection that can simultaneously preserve all of these properties

Perspective projection

• The viewing sphere is projected onto a tangent plane through lines emanating from the center of the sphere.– Orientation constancy– Not conformal projection– Robust for fields of view less than about 40°– Field of view approaches 180° the stretching

becomes infinite

Mercator projection

The Mercator projection is a cylindrical projection that is designed to maintain conformality• Conformal projection• It can handle a complete 360 horizontal field of view,

but stretches to infinity as the vertical field of view approaches 180.

• useful for panoramic images with large horizontal fields of view

Stereographic projection

The viewing sphere is projected onto a tangent plane through lines emanating from the pole opposite the point of tangency.• Conformal projection• Like perspective projection, stereographic projection

stretches objects toward the periphery

Approach

LoadImage

Select lines

Crop result image

Approach

Select lines• Click on the two endpoints of the linear

structure to specify the constrain–general line constraint–fixed orientation line constraint (modify the general line constraint)

endpoints

endpoints

Line in the scene

Drawn line

Approach

• The general line constraint– Keep linear structures in the scene from bending

• The fixed orientation line constraint– Let linear structures map to straight lines at a

user-specified orientation in output images (user can choose vertical or horizontal)

Approach

• Given these line constraints our algorithm computes a mapping from the viewing sphere to the image plane.

Mathematical setup

• Notations Viewing sphere maps to plane

Mapping function:

, vector form:

longitude : latitude :

vector form:

Mathematical setup

• Local properties of this mapping – Differential vector

• Conformal mapping

,

spherical coordinates are non-Euclideanequal steps in travel different distances on the sphere depending on

𝐡=[cos 𝜋2 −sin 𝜋2sin𝜋2cos

𝜋2

]=[𝟎−𝟏𝟏𝟎 ]𝐤

,

Cauchy-Riemann equations for mapping a sphere to a plane [Hilbert and Cohn-Vossen 1952; Snyder 1987]

(1)

(2)

(3)

Mathematical setup

• Discretize the mapping– In our case, we can’t derive a close-form solution– We discretize the mapping by sampling a uniform

grid in () indexed by integers ()

: entire set of vertices () that fall in the field of view of the input image

Quad

Conformality

• We form conformality constraints on the mesh by discretizing the Cauchy-Riemann equations (3), giving

(3) ,

𝑢𝑖 , 𝑗+1−𝑢𝑖 , 𝑗=−(𝑣 𝑖+1 , 𝑗−𝑣 𝑖 , 𝑗)/cos𝜙𝑖 , 𝑗

𝑣 𝑖 , 𝑗+1−𝑣 𝑖 , 𝑗=−(𝑢𝑖+1 , 𝑗−𝑢𝑖 , 𝑗)/cos𝜙𝑖 , 𝑗

(4)

(5)

Conformality

• All quads on the viewing sphere are not equal in size– We weight the constraints by

𝐸𝑐= ∑(𝑖 , 𝑗 )∈𝑉

𝑤 𝑖 , 𝑗2((𝑣 𝑖+1 , 𝑗−𝑣 𝑖 , 𝑗 )+cos𝜙 𝑖 , 𝑗(𝑢𝑖 , 𝑗+1−𝑢𝑖 , 𝑗))

2

+ ∑(𝑖 , 𝑗 )∈𝑉

𝑤𝑖 , 𝑗2( (𝑢𝑖+1 , 𝑗−𝑢𝑖 , 𝑗 )+cos𝜙𝑖 , 𝑗 (𝑣 𝑖 , 𝑗+1−𝑣 𝑖 , 𝑗))

2(6)

Straight lines

• We define : the set of all line constraints marked by user

: orientation line constrain

• Virtual vertex – Midpoint of line-quad intersection– We define a virtual vertex as a bilinear interpolation of the surrounding vertices.

points lie on a line(line is on the viewing sphere)

points are collinear on the image plane

Virtual vertex

Sphere

Straight lines

• We compute the position of a virtual vertex on the sphere, and its bilinear interpolation coefficients (a, b, c, d), as shown in Figure

on which we place our line constraints.

𝑢𝑖+1 , 𝑗

𝑢𝑖 , 𝑗+1

𝑢𝑖 , 𝑗

𝑢𝑖+1 , 𝑗+1

Straight lines

• : two line endpoints

• For the rest of this section we drop the superscript and assume the u variables correspond to virtual vertices for a particular line

Straight lines

• Distance

• We therefore define the line energy for a constrained line as

This energy function is non-linear, so we simplify the line energy in two ways,each of which can be solved linearly

(7)

(8)

Straight lines

• We can express the energy function as another way :

Equation (10) is the normalized length of the projection of () onto ()

Straight lines

• Two ways to simplify the line energy– By fixing the normal vector in equation (8)、 (9)

(8)

Smoothness

(13)

(14)

Spatially-varying constraint weighting

• Line endpoint weights :• Salience weights : • Face detection weights : – face detection algorithm of Viola and Jones [2004], as implemented in OpenCV [Bradski and Kaehler2008]

• Total weight

Total energy and Optimization

• Total energy function

• The quadratic energy function at each iteration of our algorithm results in a sparse linear system Ax = 0

• PARDISO sparse direct solver

+ (16)

Results

Future work

• Developing a completely automatic system that identifies salient linear structures using line detection algorithms

• Improved by using a more sophisticated salience measure