1

Computational Photography:

“Miscellaneous Topics”Part 1

Brown

2

This lecture’s topic

We will discuss the following:

• Seam Carving for Image Resizing

– An interesting new way to consider resizing images

– This paper made a “wave” at SIGGRAPH’07

• Color Harmonization

– An automatic approach to harmonize colors

– With an overview of HSV colorspce

• Image Inpainting

– Seminal paper by Bertalmio et al on image restoration

3

Three Papers DiscussedThree papers in this lecture:

1. Shai Avidan and Ariel Shamir SIGGRAPH 07

“Seam Carving for Content-Aware Image Resizing”

2. Daniel Cohen-Or et al, SIGGRAPH06

“Color Harmonization”

(With a pre-discussion on color)

3. Bertalmio et al SIGGRAPH’00

“Image Inpainting”

4

“Seam Carving”

• SIGGRAPH 07Shai Avidan and Ariel Shamir

Shai was at MERL when paper was published,left for Adobe, then toTel-Aviv University

Ariel is a professor at the Efi Arazi School of Computer Science (Israel)

Both got their PhD’s from Hebrew University in 1999

• IdeaTypical image operation: scale-up or down and image

- Don’t necessarily want the image to be scaled pixel by pixel

- Instead want the content to fit into something smaller (call this re-targeting)

- How can we do this in a clever way?

ArielShai

5

Idea Resizing

Here is an image and a plot of each pixels importance

(blue=less important, yellow=more important)

We want to reduce

the size of this image,

to make it smaller.

Let’s say we want to reduce the width of this

image by K columns.

6

Cropping

Cropping

Current solution – crop the image:

Several automatic techniques exist to crop important parts of the

image (and remove the less important). Crop a region

such that a total of k-columns are remove from the left

and right.

7

Column Removal

Column Removal(note artifacts)

Another option is to remove whole columns, where columns

with less energy are removed first. So, remove the K columns

with minimum energy. This results in strange artifacts.

8

Pixel Removal

Pixel Removal

We can remove the K less important pixels per row.

This produces very ugly results.

9

Seam Carving

Seam Carving

We can carve K seams (where a seam runs from top to bottom)

with the least energy.

10

Comparison

11

Seam Carving

Horizontal seam

A connected

line from left

to right that can

move at most

1-pixel per

row

Vertical seam

(connected line from top to bottom, that can move

at most 1-pixel per column)

12

Seams and Energy

• If we want to reduce by either width or height, remove the seam with the least “energy”

• Optimal horizontal seam (right to left), where e is some energy function

• Define (vertical seam) bottom-to-top similarly

• A pixel (i,j) can only be part of one seam. So the goal then is to final the optimal “seams”, s*, such that:

*

13

What Seam Energy to Use?

• The authors discuss several choices, but

these are the two they found the most

useful:

Simply gradient energy (not this is not magnitude, but an approximation)

Where, HoG is the histogram of oriented gradients. Using an 11x11 window about pixel (x,y), compute

gradients orientation and then build a histogram of the orientation (in this case an 8-bin histogram).

Take the max value of the histogram. A value with a large max means here is a strong edge in the

11x11 window. This energy attracts seams to edges, but not to cross the edge (because of the

numerator).

14

e1 and eHoG example

15

Image Expanding

• What if we want to expand the image?

• Picking the minimum energy seam and

duplicating it, gives a strange effect!

16

Image Expanding

• Say we want to expand by K pixels

• Pick the K minimum seam

• Duplicate these seams by linear-interpolation

17

A little help from the user

A failure case, seam carving is not aware of the

content’s meaning, only energy.

18

A little help from the user

No problem, have the user assign “maximum”

energy to regions.

19

Object Removal

Assign 0 energyAssign max energy

20

No hope cases

Structure of content is not suitable for seam-carving.

OK, go back to regular resizing!

21

Seam Carving Summary

• Very, very cool idea

– So obvious after you have seen it that you wish you had this idea!

• You’d be rich, Adobe would hire you and stuff your pockets with cash

• This paper is part of a move to “re-think” image editing

– “Content-aware” image editing

– Resizing while considering the content

– User assistant for hard cases

22

“Color Harmonization”

• SIGGRAPH 07Daniel Cohen-Or et al

Daniel is a professor at the Tel Aviv University in Israel

He has many SIGGRAPH papers every year.

• Idea

Harmonization is the result of choosing colors that are pleasing to humans

- Can we provide a way to do this for images?

- Retarget colors to be harmonized?

Daniel

Recall

• Our discussion on RGB color

• CIE XYZ perceptual space

• And sRGB color space

• Here, we will discuss HSV/HSI space

23

CIE XYZ to CIE xy Chromaticity

[X, Y, Z]

(X=Y=Z)

CIE XYZ

x = X / (X+Y+Z)

y = Y / (X+Y+Z)

z = Z / (X+Y+Z)

CIE xy

Standard RGB (sRGB)

R

G

In 1996, Microsoft and HP

defined the “standard”

RGB primaries.

26

The HSV/HSI Colospace

• This is a different way at looking at the RGB color cube. Let’s first consider the Hue,

Saturation, and Value (HSV) color space which is a variation on the HLS.

• For HSV, we modify the RGB cube such that the greyscale line is the vertical axis.

We now modify the cube to be a cone. A color in this cone is expressed by three

values: 1) its position along the vertical axis (Value), 2) an angle (Hue) about that axis

(with reference point Hue=0o=red); 3) and the distance to the edge of the cone

(Saturation). We sometimes refer to hue as the “Color Wheel”.

RGB

Transformation Visualization between RGB and HSV

http://www.flashandmath.com/advanced/color/RGBtoHSV.html

Example Applet Showing RBG and HSV

http://dcssrv1.oit.uci.edu/~wiedeman/cspace/me/rgbhsv.html

27

HVS and the related HLS

• Adobe uses the HLS space, which is similar to HSV. Instead of

Value, however, HLS uses the term “Lightness”. In, HSV, a value=1

does not result in a pixel that is completely white, it would only be

completely white if the saturation=0.

• However, for HLS, as the lightness value increases, the range of

colors decreases, thus everything becomes white with a

lightness=1.

28

Color Enhancement via HSV Manipulation

• We can manipulate each component of hue, saturation and intensity for all pixels simultaneously

• Normally for image-editing applications:1. Convert colors in RGB representation to HSI

2. Manipulate HSI components, typically by• Hue transformation involves adding a user-specified

constant to the hues of all pixels (equivalent to rotating chromaticity plane about intensity axis)

• Saturation and intensity transformation involves scaling these values by a constant factor

3. Convert colors in HSI representation back to RGB

29

HSI Manipulation Examples

Hue

Saturation

Intensity

30

Another example

original

Hue modification

Saturation modifications

Lightness modifications

31

Back to Color Harmony Paper

32

What is color harmony?

• Harmonic colors are pleasing to the eye.

• They engage the human observer and

give a sense of order and balance in the

visual experience.

[slides from Cohen-Or’s SIGGRAPH talk]

33

• Mathematical formulation has been developing

together with color theory

– Newton, Goethe, Young, Maxwell

• Itten [1960]: harmony means relationships on

the hue wheel:

2-color harmony:

complementary

colors

3-color harmony:

equilateral triangle

N-color harmony:

equilateral N-gon

Formal definition of color harmony?

34

• Matsuda [1995]: extensive empirical studies,

derived 8 hue templates

• Tokumaru et al. [2002] developed a fuzzy

system to evaluate the harmony of color

schemes

i type V type L type I type

T type Y type X type N type

Formal definition of color harmony?

35

Harmonic scheme

• The templates can be arbitrarily rotated

• Harmonic scheme is template type Tm +

specific orientation α

i type V type L type I type

T type Y type X type N type

Type “N” is not considered in this paper, this is for grayscale

images.

36

Harmony score

• To evaluate the harmony of an input

image X we analyze its hue histogram:

– Every pixel p contributes its saturation S(p) to

the bin of the hue H(p)

37

Harmony function

• The harmony of image X w. r. t. harmonic

scheme (Tm , α) :

( ), ( , ) ( ) ( ) ( )m Tm

p X

F X T H p E p S p

H(p)

ETm(α)(p)

This term is

the closest edge

of the template

(oriented at angle

alpha)

38

Best template

• We compute α that minimizes F(X, (Tm , α))

for each template Tm using Brent’s

algorithm

• The best-fitting harmonic scheme:

0 0

( ,α)

( ,α ) argmin ,( ,α)m mm

T F X T

So, given an image, they compute the best fit template from

the different types (see 49) at the best orientation alpha.

39

Harmonization

• Given (Tm , α) we shift the hues so that the

hue histogram is contained in (Tm , α)

40

Color shifting

• The hue of pixel p is shifted to its associated sector

ETm(α)(p)

• The amount of squeezing is controlled by a Gaussian

fall-off function

41

Color Coherency Problem

The problem, no way

to force neighboring pixels

to similar colors.

Here, similar colors (blue) move to

two different regions (green, purple).

42

Color coherency

• Another example

Problem!

This would be better!

43

Graph-cut optimization

• To make the coloring more coherent we

assign ETm(α)(p) by optimizing the labeling

V

1 2( ) ( ) ( )E V E V E V

Favors short

distance to the

template sector

Favors coherent

labeling of

neighboring pixels

[Back to MRF – each pixel now has a data-cost E1 and a neighbor cost E2.

This is similar to the lazy-snapping MRF formulation. ]

44

Graph-cut optimization

• To make the coloring more coherent we

assign ETm(α)(p) by optimizing the labeling

V

1 2( ) ( ) ( )E V E V E V

1( ) ( ) ( ( )) ( )p

E V H p H V p S p

1

2 max

{ , }

( ) ( ), ( ) ( ) ( ) ( , )p q N

E V V p V q H p H q S p q

45

Results

46

Overcoming segmentation

problems• The graph-cut may fail when an object in

the image has several connected

components

47

Overcoming segmentation problems

• User-assisted fix: scribbling on the erroneously

labeled area

• Re-compute the labeling

48

Results – choosing colors

49

Results – cut and paste

• The background is harmonized according to the best-

fitting harmonic template of the pasted foreground

original harmonized harmonized

50

Results

• Text over a poster

51

Results

• Text over a poster

52

Results

Images harmonized

to different “flags”

colors.

Find the flags template,

force the image to this.

53

Discussion

• Nature is already harmonic

original best-fitting

template

poorly-fitting

template

54

Discussion

• Cannot improve good artwork!

Wassily Kandinsky, Composition VII, 1913

55

Discussion

• Grayish colors will remain such

56

Harmony Summary

• Provides a method to enhances the harmony of colors in a given image

• Operates by fitting the image hues into a given harmonic distribution

– Several different harmonic chooses are pre-defined (based on color theory)

• Especially useful for ‘artificial’ colors, cut-and-paste settings and collages that combine imagery from different sources

57

“Image Inpainting”

• SIGGRAPH’00Marcelo Bertalmio et al

While Marcelo was a PhD student at U. Of Minnesota

Now a professor at Universitat Pompeu Fabra (Spain)

• Idea

Find a good way to fill in missing image information

- The idea comes from how artists fix paintings when the paint chips away:

they “in paint”

Marcelo

58

Real “Inpainting”

http://www.cac-accr.ca/english/e-CAC-publications-CAC-pub-sel.asp

59

Image Inpainting

Assume you have an image that has been

corrupted. Above you want to fill in the “white”

pixels with the surrounding content.

Or, another scenario, you

the user draw the region

you want to correct.

60

The idea

Image I

∂Ω

Ω

Ω

∂Ω

is a “hole” you want to repair

is a the border of the hole

with pixel intensities

Idea: propagate information from ∂Ω inside to

Ω. But, do this in a clever manner.

61

The problem

Problem: Need to

propagate the

information along image

gradients! Otherwise

the result will be

incorrect.

Hole to be filled

Incorrect

62

Solution

The paper write up is very complicated, but can be followed if you read

it slowly. Basic idea is that at each step you try to propagate the boundary

∂Ω pixels in image gradient direction. The boundary will slowly shrink.

This slowly fills in the hole (and propagates gradient) thus maintaining

the direction.

(start) (after many iterations)

(more iterations) (and yet more iterations)

63

Examples

Inpainting result.

The key here is that image

gradient was incorporated.

64

Limitations

Not perfect, because it can’t reproduce “texture”

65

Inpainting Summary

• This idea spawned a great deal of further work– Google Scholar has 1902 citations to this paper

• The idea is simple– Fill in the hole by propagating information

– The approach is clever

• But the problem is good– The user helps by giving the region to correct

– Again, we are seeing user-assistance in the procedure

• The algorithm itself is automated

• Similar ideas:

– Bayesian matting/Possion Matting/PIE/and so on