Structure and Aesthetics in Non-Photorealistic Images

Hua Li, David Mould, and Jim DaviesCarleton University

Artistic or Messed

2/35

Related Work on Evaluating Non-Photorealistic Algorithms

• Performance based on processing speed– ill-suited for stylization

• Side-by-side comparisons – not fully convinced by audience

3/35

Perceptual Evaluation on Non-Photorealistic Algorithms

• Quantitative evaluation– rating scores [Schumann et al. 96, Gooch and

Willemsen 02, Mandryk et al. 2011, Mould et al. 2012]

– response time [Gooch et al. 04]– eye-tracking data [Mandryk et al. 2011, Mould et

al. 2012]• Qualitative evaluation– questionnaire-based

4/35

Motivation of Our StudyTone Structure

Tone-based Structure-based

Halftoning

[Pang et al. 08] [Chang et al. 09] [Ours 10]

Stippling [Secord 02]

[Mould 07] [Martin et al. 11] [Ours 11]

Screening [Ulichney 98]

Abstraction [Kyprianidis 11] [Mould 12, 13]

[Qu et al. 08] [Ours 11]

[Floyd and Steinberg 76]

[Ostromoukhov 01]

5/35

Questions to Answer

• Are structural and aesthetic quality related? • Do images matter for side-by-side

comparisons?

6/35

Participants

• 30 participants– 15 female and 15 male– 11 artists– aged 18 to 33

7/35

Study Overview

• 1 ~ 1.5 hours to complete the experiment• Using the keyboard or the mouse to enter

their responses• Tasks:– rating structural and aesthetic quality– collecting response times for rendered images

8/35

Image Stimuli

• Seven categories – include cars, cats, persons, flowers, buildings, mugs,

and birds.• Each category contains 13 different images

including– one unprocessed image– and 12 rendered images using 12 algorithms.

• Images are black and white, or greyscale to remove the influence of color.

9/36

Procedure

• Step 1: verbal introduction• Step 2: training• Step 3: formal study• Step 4: questionnaire• Step 5: ranking

10/35

Interfaces Used

Interface for collecting the response time 11/35

Interfaces Used

Aesthetic rating Structural rating12/35

Experimental Images -Bird Category

Unprocessed13/35

Experimental Images - Bird Category

Structure-Preserving Stippling (SPS)

Structure-Aware

14/35

Experimental Images - Bird Category

Content-Sensitive Screening (CSS)

Structure-Aware

15/35

Experimental Images - Bird Category

SPS with Exclusion Masks (SPH)

Structure-Aware

16/35

Experimental Images - Bird Category

Line Art using edge tangent field (ETF)

Structure-Aware

17/35

Experimental Images - Bird Category

Artistic Tessellation (AT)

Structure-Aware

18/35

Experimental Images - Bird Category

Line Art from SPS (Drawing)

Structure-Aware

19/35

Experimental Images - Bird Category

Secord’s Stippling Method (Secord)

Tone-based

20/35

Experimental Images - Bird Category

Line Art using edge tangent field (Mmosaics)

Tone-based

21/35

Experimental Images - Bird Category

Contrast-Aware Halftoning (CAH)22/35

Experimental Images - Bird Category

Black and White (BW)23/35

Experimental Images - Bird Category

Adding 50% salt and pepper noise (Noisy)

Reduced Information

24/35

Experimental Images - Bird Category

Gaussian filter (Blurring)

Reduced Information

25/35

Positive Correlation Between Structural and Aesthetic Ratings

26/35

Dot-based Methods (Stippling)

Aesthetic Structure1

1.5

2

2.5

3

3.5

4

4.5

SecordSPS

Mea

n Ae

sthe

tic\S

truc

ture

Rati

ngs

Structure-Aware

Tone-basedStructure-

Aware

Tone-based

27/35

Region-based Methods (Mosaics)

Aesthetic Structure1

1.5

2

2.5

3

3.5

MmosaicsAT

Mea

n Ae

sthe

tic\S

truc

ture

Rati

ngs

Structure-Aware

Tone-basedStructure-

Aware

Tone-based

28/35

Effect of Category on Ratings

Aesthetic Structure 1

1.5

2

2.5

3

3.5

4

4.5CarCatBirdPersonMugFlowerBuilding

Mea

n Ae

sthe

tic\S

truc

ture

Rati

ngs

29/35

Effect of Category on Response Time

Series1

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

CarCatBirdPersonMugFlowerBuilding

Z va

lues

of R

espo

nse

Tim

e

30/35Building < Flower < Bird < Cat < Person < Mug < Car

Artists and Non-Artists

Aesthetic Structure1

1.5

2

2.5

3

3.5

4

4.5

5

ArtistNon-Artist

Mea

n Ae

sthe

tic\S

truc

ture

Rati

ngs

31/35

Overall Ranks after Study

• Participants preferred the AT images (7/30 responses) the most, CAH second (6/30).

• Participants’ least favorite – blurred images most often (20/30 responses), and

with AT second (5/30). • Controversial ranking for stylized images

rendered by the AT method.

32/35

1st Ran

k

2nd Rank

3rd Rank

4th Rank

5th Rank

6th Ran

k

7th Ran

k

8th Ran

k

9th Ran

k

10th Ran

k

11th Ran

k

12th Ran

k0

5

10

15

20

25

CAHATNoisyBlurring

From Most Favourite to Least Favourite

Num

ber o

f Par

ticip

ants

Conclusions

• Considering structure as a possible way to increase aesthetic appeal.

• Considering the choice of the images used.– Generally, bird images were the easiest images to

abstract, while Person images were the most difficult.

33/35

Future Work

• More Participants• More Categories• More NPR Algorithms• Eye tracker

34/35

Thanks for Your Attention.

35/35

Questions?

Effect of Algorithms on Ratings (skip)

Aesthetic Structure1

1.5

2

2.5

3

3.5

4

4.5

5

CAHSPSCSSSPHETFATBWDrawingUnprocessedNoisyBlurring

Mea

n Ae

sthe

tic\S

truc

ture

Rati

ngs

36/41

Effect of Algorithms on Response Time (skip)

Series1

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

CAHSPSCSSSPHETFATBWDrawingUnprocessedNoisyBlurringZ

valu

es o

f Rec

ogni

tion

Tim

e

37/41

Interaction between Categories and Algorithms (skip)

Car Cat Bird Person Mug Flower Building1

1.5

2

2.5

3

3.5

4

4.5

5CAHSecordSPSCSSSPHETFMmosaicsATBWDrawingUnprocessedNoisyBlurring

Mea

n Ae

sthe

tic R

ating

s

Aesthetic scores38/41

Interaction between Categories and Algorithms (skip)

CAHSe

cord SP

SCSS SP

H ETF

Mmosaics AT

BW

Drawing

Unprocesse

dNoisy

Blurring

1

1.5

2

2.5

3

3.5

4

4.5

5

CarCatBirdPersonMugFlowerBuilding

Mea

n Ae

sthe

tic R

ating

s

39/41

Interaction between Categories and Algorithms (skip)

Car Cat Bird Person Mug Flower Building

-1

-0.5

0

0.5

1

1.5

CAHSPSCSSSPHETFATBWDrawingUnprocessedNoisyBlurringZ

valu

es o

f Rec

ogni

tion

Tim

e

40/41

1st Ran

k

2nd Rank

3rd Rank

4th Rank

5th Rank

6th Ran

k

7th Ran

k

8th Ran

k

9th Ran

k

10th Ran

k

11th Ran

k

12th Ran

k0

5

10

15

20

25

CAHATNoisyBlurring

From Most Favourite to Least Favourite

Num

ber o

f Par

ticip

ants