ying cao antoni b. chanrynson w.h. lau city university of hong kong
TRANSCRIPT
![Page 1: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/1.jpg)
Ying Cao Antoni B. Chan Rynson W.H. Lau
City University of Hong Kong
Automatic StylisticLayout
![Page 2: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/2.jpg)
Background
• Manga layout is crucial for manga production, with unique styles
©AYOYAMA Gosho / Shogakukan Inc.
Manga pages Their layouts
![Page 3: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/3.jpg)
Background
• Effective manga layout can benefit– Storytelling– Attention guidance– Visual attractiveness
• It is a difficult task
![Page 4: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/4.jpg)
Goal
• To create high-quality manga layout with ease
Resulting layout
1
2
2
3
3
3
SemanticsArtworks
![Page 5: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/5.jpg)
Challenge
• Not a well-studied problem
• Our solution: data-driven strategy to learn stylistic aspects from existing manga pages
• No explicit rules
![Page 6: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/6.jpg)
Related Work
• General layout problem: global optimization
[ Yu et al. 2011] [Merrell et al. 2011]
![Page 7: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/7.jpg)
Related Work
• Comic layout: heuristic rules or templates
[Kurlander et al. 1996] [Shamir et al. 2006] [Preu et al. 2007]
![Page 8: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/8.jpg)
Related Work
• Computational Manga
[Qu et al. 2006]
[Qu et al. 2008]
![Page 9: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/9.jpg)
Overview
![Page 10: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/10.jpg)
Overview
![Page 11: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/11.jpg)
Overview
![Page 12: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/12.jpg)
Overview
![Page 13: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/13.jpg)
Manga Database
• 4,000 scanned manga pages from two manga series
Panel annotation Page clustering
One manga series
3-panel pages 10-panel pages4-panel pages …
![Page 14: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/14.jpg)
Overview
![Page 15: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/15.jpg)
Style Models
• Represent stylistic aspects of manga layout
• Learned from manga examples
3) Panel shape
…2) Panel importance (size)
1 2 3
1) Layout structure (i.e., spatial arrangement of panels)
…
![Page 16: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/16.jpg)
• A probabilistic generative model:pdf(gm,[x,y])
• Synthesize novel plausible layout structures
Layout structure Model
![Page 17: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/17.jpg)
Root
©AYOYAMA Gosho / Shogakukan Inc.
Layout structure Model
• Generative process: recursive spatial division
R1 R2 R3
C1C2 C1
R2R1
C3 C2 C1
![Page 18: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/18.jpg)
Layout structure Model
• Parameterization: spatial division instance
: Instance label [Root-R]
: Number of rows [3]
: Splitting configuration [()]
X 1
X 2
X 3
©AYOYAMA Gosho / Shogakukan Inc.
![Page 19: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/19.jpg)
Layout structure Model
: Instance label
: Number of rows
: Splitting configuration
𝓛𝑵
𝐗Probabilistic graphical model
• Parameterization: spatial division instance
![Page 20: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/20.jpg)
Layout structure Model
• Sample splitting configuration
: Instance label
: Number of rows
: Splitting configuration
𝓛𝑵
𝐗Probabilistic graphical model
![Page 21: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/21.jpg)
Layout structure Model
• Sample splitting configuration
: Instance label
: Number of rows
: Splitting configuration
𝓛𝑵
𝐗Probabilistic graphical model
Sample𝐗
![Page 22: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/22.jpg)
Layout structure Model
Layout structures sampled from our modelTraining example
• Sampling: recursive splitting using sampled
![Page 23: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/23.jpg)
• Panel clustering
Width
Hei
ght
Panel Importance
3 12
Size Importance Shape?
A shape-to-importance classifier𝑚= 𝒇 (𝒙 )
![Page 24: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/24.jpg)
Panel Shape Variation Model
• Captures panel shape variability‒ Active Shape Model [Cootes et al. 1995]
…{𝒔2 ,𝐔2}
…{𝒔1 ,𝐔1 }
{𝒔3 ,𝐔3 }…
![Page 25: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/25.jpg)
Overview
![Page 26: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/26.jpg)
Semantic SpecificationSingle-panel semantics
Inter-panel semanticsImage geometry
Group of related panels
3
Importance
![Page 27: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/27.jpg)
Overview
![Page 28: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/28.jpg)
Initial Layout Generation
• A layout structure
• Maximum a posteriori (MAP) inference
Fitness between and Our generative model
Input semantics
Existing ones
matches
resembles
![Page 29: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/29.jpg)
Initial Layout Generation
• Likelihood term
Penalize panel-wise mismatch in aspect ratio & importance
Single-panel Likelihood
𝑃 𝑗
𝐼 𝑗
Image geometry
panel geometry
![Page 30: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/30.jpg)
Initial Layout Generation
• Likelihood termInter-panel Likelihood
![Page 31: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/31.jpg)
Initial Layout Generation
• Likelihood termInter-panel Likelihood
Measure the smoothness of path through panels
![Page 32: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/32.jpg)
Initial Layout Generation
• Likelihood termInter-panel Likelihood
Align group boundary with layout boundary
![Page 33: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/33.jpg)
Initial Layout Generation
• Estimate optimal initial layout‒ Exact MAP inference is computationally expensive
…
pdf(gm,[x,y])
Generative Model Maximum Posteriori
![Page 34: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/34.jpg)
Layout Optimization
Unoptimized
• Fit to and reproduce panel shape irregularity
![Page 35: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/35.jpg)
Layout Optimization
• Energy function
Collinearity constraint
𝐕
Boundary constraint
𝐕
𝑃 𝑖
𝐓𝑖 ∙ 𝐼𝑖
Regularization term
![Page 36: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/36.jpg)
Layout Optimization
• Minimize via an alternating solver
![Page 37: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/37.jpg)
Results
(1)
(2)
(2)
(3)
(2)
(2)
(1)
(2)
(2)
(2)
(3)
(3)
(1)
![Page 38: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/38.jpg)
Comparison with existing manga page
Input Our result Existing manga page
(3)
(3)
(1)
(3)
(3)
(2)
(3)
©AYOYAMA Gosho / Shogakukan Inc.
![Page 39: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/39.jpg)
Layouts of different styles
(1)
(2)
(3)
(1)
(3)
(2)
Input Style of “Fairy Tail” Style of “Detective Conan”
![Page 40: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/40.jpg)
Layouts of Western comic style
![Page 41: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/41.jpg)
User Study
• 10 participants: manual tool + our tool
• 10 Evaluators: pairwise comparison
Manual tool Our tool012345 4.09
1.36Aver
age
time(
min
)
Functionality Visual Appeal Style0%
20%
40%
60%
Manual placement Our toolTie
![Page 42: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/42.jpg)
Summary
• First attempt to computationally reproduce layout styles of manga
• A data-driven approach for automatic generation of stylistic manga layout
• Easy and quick production of professional-looking and stylistically rich manga layouts
![Page 43: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/43.jpg)
Limitations & Future Work
• Story pacing
• Art composition & balloon placement
• Generic framework for other layout problems
![Page 44: Ying Cao Antoni B. ChanRynson W.H. Lau City University of Hong Kong](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c515503460f948f95a8/html5/thumbnails/44.jpg)
Thanks