detection of comic characters in comic books and manga

39
17/10/14 - christophe-rigaud.com Detection of comic characters in comic books and manga

Upload: christophe-rigaud

Post on 27-Jun-2015

317 views

Category:

Science


3 download

DESCRIPTION

State of the art methods for comic character (people) localization in comic book images and manga.

TRANSCRIPT

Page 1: Detection of comic characters in comic books and manga

17/10/14 - christophe-rigaud.com

Detection of comic characters in comic books and manga

Page 2: Detection of comic characters in comic books and manga

217/10/14 - christophe-rigaud.com

Presentation

Advisers● Jean-Christophe Burie & Jean-Marc Ogier (L3i, La Rochelle, France)

● Dimosthenis Karatzas (CVC, Barcelona, Spain)

eBDtheque project

● Comic book image processing and understanding

● Cultural heritage in many countries

● Paper ↔ digital comic book conversion enhancement

● New usages using the new technologies

Applications

● Text/image search, augmented reading, reflowing...

France

Spain

Page 3: Detection of comic characters in comic books and manga

317/10/14 - christophe-rigaud.com

Researchers

What researchers can do for the paper to digital transition?

Digital world Digital world

Paper world

Page 4: Detection of comic characters in comic books and manga

417/10/14 - christophe-rigaud.com

Researchers

Image credits: contributors of the eBDtheque dataset

Page 5: Detection of comic characters in comic books and manga

517/10/14 - christophe-rigaud.com

Balloon classification

Speaker localization

Text timestamps

Tone and pitch

Scene understanding

Layout analysis

Panel retrieval

Reflowing

De-hyphenation

Speech synthesis

Language analysis

Translation assistance

Pose retrieval

Facial expression

Relationship retrieval

Character localization and recognition

Researchers

PANELSPANELS

Page 6: Detection of comic characters in comic books and manga

617/10/14 - christophe-rigaud.com

Balloon classification

Speaker localization

Text timestamps

Tone and pitch

Scene understanding

Layout analysis

Panel retrieval

Reflowing

De-hyphenation

Speech synthesis

Language analysis

Translation assistance

Pose retrieval

Facial expression

Relationship retrieval

Character localization and recognition

Researchers

TEXTTEXT

Page 7: Detection of comic characters in comic books and manga

717/10/14 - christophe-rigaud.com

Balloon classification

Speaker localization

Text timestamps

Tone and pitch

Scene understanding

Layout analysis

Panel retrieval

Reflowing

De-hyphenation

Speech synthesis

Language analysis

Translation assistance

Pose retrieval

Facial expression

Relationship retrieval

Character localization and recognition

Researchers

BALLOONSBALLOONS

Page 8: Detection of comic characters in comic books and manga

817/10/14 - christophe-rigaud.com

Balloon classification

Speaker localization

Text timestamps

Tone and pitch

Scene understanding

Layout analysis

Panel retrieval

Reflowing

De-hyphenation

Speech synthesis

Language analysis

Translation assistance

Pose retrieval

Facial expression

Relationship retrieval

Character localization and recognition

Researchers

CHARACTERSCHARACTERS

Page 9: Detection of comic characters in comic books and manga

917/10/14 - christophe-rigaud.com

Challenges of comics analysis

● Variation of comic book styles

● Printing & scanning method changes over time (press, ink-jet, computer-based)

● Diversity of information to extract

● A lot of semantic relations to retrieve (element's relationships, text meaning)

Page 10: Detection of comic characters in comic books and manga

1117/10/14 - christophe-rigaud.com

Summary of comic character extraction

● Supervised approaches

● Character spotting given an example

● Unsupervised approaches

● Hypothesis from speech balloon tail position and orientation

● Region of interest from simple object removal

● Graph­based approach (Nam)

Page 11: Detection of comic characters in comic books and manga

1217/10/14 - christophe-rigaud.com

Summary of comic character extraction

● Supervised approaches

● Character spotting given an example

● Unsupervised approaches

● Hypothesis from speech balloon tail position and orientation

● Region of interest from simple object removal

● Graph­based approach (Nam)

Page 12: Detection of comic characters in comic books and manga

1317/10/14 - christophe-rigaud.com

Character spotting given an example [5]

● Aim: retrieve all the instances (apparitions) of a given character in all the pages of a comic book title/album/collection

● Method summary:

1) Reduction of the number of colors

2) Get the colors of the query example (user interaction)

3) Compute the user query descriptor

4) Spot similar character instances

[5] C. Rigaud, D. Karatzas, J-C Burie, J-M Ogier Color descriptor for content-based drawing retrieval DAS'14

Page 13: Detection of comic characters in comic books and manga

1417/10/14 - christophe-rigaud.com

1) Color reduction/quantization

N>10000 N=256 N=32

Page 14: Detection of comic characters in comic books and manga

1517/10/14 - christophe-rigaud.com

2) User query

Page 15: Detection of comic characters in comic books and manga

1617/10/14 - christophe-rigaud.com

3) Color selection: occurrence level

Page 16: Detection of comic characters in comic books and manga

1717/10/14 - christophe-rigaud.com

3) Color selection: occurrence + discriminality levels

Top 3 colors

Page 17: Detection of comic characters in comic books and manga

1817/10/14 - christophe-rigaud.com

User input

Image credits: (Prunelle, la fille du cyclope, Ankama, 2010

Descriptor

4) Character spotting: example

Page 18: Detection of comic characters in comic books and manga

1917/10/14 - christophe-rigaud.com

correct detections wrong detections

4) Character spotting: example

Page 19: Detection of comic characters in comic books and manga

2017/10/14 - christophe-rigaud.com

Conclusions

● Contributions

● Content Based Drawing Retrieval

● Color descriptor robust to scale, rotation, translation, deformation and occlusion

● Benefit of redundant information

● Limitations

● Colored comics

● Number of color (quantization)

● Requires one example of each character

Query Correct Wrong

Result examples

Page 20: Detection of comic characters in comic books and manga

2117/10/14 - christophe-rigaud.com

Summary of comic character extraction

● Supervised approaches

● Character spotting given an example

● Unsupervised approaches

● Hypothesis from speech balloon tail position and orientation

● Region of interest from simple object removal

● Graph­based approach (Nam)

Page 21: Detection of comic characters in comic books and manga

2217/10/14 - christophe-rigaud.com

Hypothesis from speech balloon tail position and direction

● Given balloons in a panel, where are the (speaking) characters?

Page 22: Detection of comic characters in comic books and manga

2317/10/14 - christophe-rigaud.com

Example

Page 23: Detection of comic characters in comic books and manga

2417/10/14 - christophe-rigaud.com

Example

Page 24: Detection of comic characters in comic books and manga

2517/10/14 - christophe-rigaud.com

Conclusions

● Preliminary results

● Highly relies on the quality of the tail detection

● Only for “speaking” characters (others can be spotted)

● Implicitly retrieves the relationship between balloon and character

Page 25: Detection of comic characters in comic books and manga

2617/10/14 - christophe-rigaud.com

Summary of comic character extraction

● Supervised approaches

● Character spotting given an example

● Unsupervised approaches

● Hypothesis from speech balloon tail position and orientation

● Region of interest from simple object removal

● Graph­based approach (Nam)

Page 26: Detection of comic characters in comic books and manga

2717/10/14 - christophe-rigaud.com

Region of interest (ROI) from simple object removal

● Given an image from a double page of manga, can you predict where the characters are?

Page 27: Detection of comic characters in comic books and manga

2817/10/14 - christophe-rigaud.com

Region of interest (ROI) from simple object removal

● Given an image from a double page of manga, can you predict where the characters are?

??

??

??

?

??

?

??

Page 28: Detection of comic characters in comic books and manga

2917/10/14 - christophe-rigaud.com

Region of interest (ROI) from simple object removal

● Given an image from a double page of manga, can you predict where the characters are?

??

??

?

??

?

42% correct

?? ?

?

Image credits: Yasuhisa Hara, "Kingdom", Shueisha, since 2006.

Page 29: Detection of comic characters in comic books and manga

3017/10/14 - christophe-rigaud.com

Region of interest (ROI) from simple object removal

● In the same image, knowing the position of panels and balloons, can you predict where the comic characters are?

Page 30: Detection of comic characters in comic books and manga

3117/10/14 - christophe-rigaud.com

Region of interest (ROI) from simple object removal

● Knowing the position of panels and balloons, can you predict where the comic characters are?

??

?

?

??

??

??? ?

Page 31: Detection of comic characters in comic books and manga

3217/10/14 - christophe-rigaud.com

Region of interest (ROI) from simple object removal

● Et voilà !

??

?

?

??

??

??

?? 83% correct

Page 32: Detection of comic characters in comic books and manga

3317/10/14 - christophe-rigaud.com

A first test from automatic panels and balloons extraction...

Page 33: Detection of comic characters in comic books and manga

3417/10/14 - christophe-rigaud.com

Original image

Page 34: Detection of comic characters in comic books and manga

3517/10/14 - christophe-rigaud.com

Image masked

Page 35: Detection of comic characters in comic books and manga

3617/10/14 - christophe-rigaud.com

Image masked + distance transform

Page 36: Detection of comic characters in comic books and manga

3917/10/14 - christophe-rigaud.com

To be continued...

Page 37: Detection of comic characters in comic books and manga

4017/10/14 - christophe-rigaud.com

Graph-based approach

● Presented by Nam Le Thanh 

Page 38: Detection of comic characters in comic books and manga

4117/10/14 - christophe-rigaud.com

Live demo: real time panels and balloons extraction

Page 39: Detection of comic characters in comic books and manga

4217/10/14 - christophe-rigaud.com

https://github.com/crigaud/publication/tree/master/2014/TALK/intelligent_media_processing_group