attention-directing composition of ying cao rynson w.h. lau elements look over here: antoni b. chan...

Attention-Directing Composition of

Ying Cao Rynson W.H. Lau

Elements

Look Over Here:

Antoni B. Chan

City University of Hong Kong

• Composition of manga elements (i.e., subjects and balloons )

Motivation

© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.

• Manga artist guides viewer’s eyes through the page

Motivation

© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.

Artist’s Guiding Path (AGP)

Viewer Attention

Motivation

• One of the most difficult steps in manga production

• No existing comic creation programs for automatic composition

Goal

1.Rabbit, I came here for gold,

2. and I'm gonna get it!

3. I gotcha, you rabbit! I'll show you!

Close-upFast

LongMedium

Close-upMedium

Big Close-upMedium

MediumMedium

You can't do this to me!

Eureka! Gold at last!

Eureka! Gold at last!

1

2

3

4

5

Talk

Related Work

• Balloon Placement

[Kurlander et al. 1996]

[Chun et al. 2006]

• Eye-tracking Techniques for Comic Analysis and Creation

[Jain et al. 2012] [Toyoura et al. 2012]

Related Work

• Computational Manga

[Qu et al. 2006]

[Qu et al. 2008][Cao et al. 2012]

Related Work

Overview

Input Storyboard Layout Probabilistic Graphical Model

Input Infer

Learn

Data

𝐆 𝐂 𝐀

Probabilistic Graphical Model

Data Acquisition

• 80 manga pages from three manga series

Annotation

[Motion state][Shot type]

[Panel Shape][Subject][Balloon]

Eye movements of viewers

Probabilistic Graphical Model

• Abstracts artist’s guiding path as a latent variable

• Used for composition synthesis

• Connects artist’s guiding path, composition and viewer attention in a probabilisitc network

𝐆Artist’s Guiding Path

𝐂Composition

𝐀Viewer Attention

Subject Placement

Balloon Placement

Viewer Attention(eye-gaze path)

Local Composition in PanelPanel Properties

Artist’s Guiding Path

Probabilistic Graphical Model

Subject Placement

Balloon Placement

Viewer Attention(eye-gaze path)

Local Composition in PanelPanel Properties

Artist’s Guiding Path

Probabilistic Graphical Model

Artist’s Guiding Path

𝐟

𝐈

mean  of   f ( t )

𝐈(𝑡)

actual AGP:

underlying AGP:

Subject Placement

Balloon Placement

Viewer Attention(eye-gaze path)

Local Composition in PanelPanel Properties

Artist’s Guiding Path

Probabilistic Graphical Model

Panel Properties and Local Composition Model

𝐋 𝐋={𝐱𝐋 ,𝑟 𝐋}

𝐏

𝐱𝐋𝑟𝐋

: geometric shape

long, medium, close-up, big close-up: shot type

fast, medium, slow: motion state

𝑔 ,𝑡 ,𝑚

Subject Placement

Balloon Placement

Viewer Attention(eye-gaze path)

Local Composition in PanelPanel Properties

Artist’s Guiding Path

Probabilistic Graphical Model

Subject Placement

𝐈

𝐭𝐒

𝐋

𝐱𝐏

+

𝐈(𝑡𝐒)𝐒𝐒={𝐱𝐒 ,𝑟 𝐒}

𝐱𝐒

𝑟𝐒

: panel center location

: control point index

Subject Placement

Balloon Placement

Viewer Attention(eye-gaze path)

Local Composition in PanelPanel Properties

Artist’s Guiding Path

Probabilistic Graphical Model

Balloon Placement

𝐒

𝐈

𝐭𝐁 𝐎

𝐱𝐒

𝑟𝐒

𝐱𝐁

𝑟𝐁

𝐈(𝑡𝐁)

𝐁 𝐁={𝐱𝐁 ,𝑟𝐁}

: balloon order index

: control point index

Subject Placement

Balloon Placement

Viewer Attention(eye-gaze path)

Local Composition in PanelPanel Properties

Artist’s Guiding Path

Probabilistic Graphical Model

Viewer Attention

𝐒

𝐁

𝐔

𝐱𝐒

𝑟 S

𝐱𝐁

𝑟𝐁𝐔

: viewer attention transition

Complete Model

Interactive Composition Synthesis

Interactive Composition Synthesis

• Generate a composition, subject to user-specified semantics

• Layout Generation + Composition Synthesis

1.Rabbit, I came here for gold,

2. and I'm gonna get it!

3. I gotcha, you rabbit! I'll show you!

Input:

subject & script

Close-up

Fast

shot type & motion state

Talk

inter-subject constraint

Layout Generation

…

Database of labeled pages

[# of panels ] [# of subjects] [# of balloons] [shot type] [motion state]

Composition Synthesis

Configurations of elements

• A Maximum A Posteriori inference framework

Constraint-based likelihood Conditional prior

Input elements & semantics + Layout

𝐘𝐶Constraints

Constraint-based likelihood

• : overlap term • : order term

• : boundary term

• subject relation term

Constraint-based likelihood

𝑟 𝑖 𝑟 𝑗

𝐯 ij

Interaction “Talk”

shot “medium”

Results and Evaluation

Comparison to Heuristic Method for Balloon Placement

Our approachHeuristic [Chun et al. 2006]

Comparison to Heuristic Method for Balloon Placement

Our approachHeuristic [Chun et al. 2006]

Comparison to Manual Method

• User study: How well does our approach facilitate manga composition, as compared to existing manual tool?

Participant preference voting Time for one composition

Eye-tracking Experiment and Analysis

• Is our approach effective in directing viewer attention, as compared with the alternatives?

• To measure fixation consistency across different viewers

• Similarity metrics: 1. Inlier percent

2. Root Mean Squared Distance (RMSD)

Eye-tracking Experiment and Analysis

1. Inlier percent

Viewer A Saliency Map

Viewer B

InliersClassification

RMSD,

2. RMSD

Viewer A Viewer B

Eye-tracking Experiment and Analysis

Eye-tracking Experiment and Analysis

Comparison to Existing Manga Pages

Original Synthesized© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.

Summary

• A novel model for representing dependency among the artist’s guiding path, composition and viewer attention

• A novel approach for joint composition of subjects and balloons

• Enable easy and quick creation of attention-directing compositions

Limitations & Future Work

• Consider visual appearance of manga elements

• Extend to other graphic design tasks

• Understand readers’ behaviors

Thanks