This document discusses research on detecting comic characters in comic books and manga. It describes supervised and unsupervised approaches, including character spotting given an example, hypothesizing locations from speech balloon tails, and predicting regions of interest by removing simple objects. A graph-based approach is also mentioned. The goal is to help the transition from paper to digital comics and enable new applications like search and augmented reading.
Detection of comic characters in comic books and manga
1. Detection of comic characters in comic books
and manga
17/10/14 - christophe-rigaud.com
2. 2
17/10/14 - christophe-rigaud.com
Presentation
Advisers
● Jean-Christophe Burie & Jean-Marc Ogier (L3i, La Rochelle, France)
● Dimosthenis Karatzas (CVC, Barcelona, Spain)
eBDtheque project
● Comic book image processing and understanding
● Cultural heritage in many countries
● Paper ↔ digital comic book conversion enhancement
● New usages using the new technologies
Applications
● Text/image search, augmented reading, reflowing...
France
Spain
3. 3
What researchers can do for the paper to digital transition?
Researchers
Paper world
Digital world Digital world
17/10/14 - christophe-rigaud.com
4. 4
Researchers
Image credits: contributors of the eBDtheque dataset
17/10/14 - christophe-rigaud.com
5. 5
17/10/14 - christophe-rigaud.com
Balloon classification
Speaker localization
Text timestamps
Tone and pitch
Scene understanding
Layout analysis
Panel retrieval
Reflowing
De-hyphenation
Speech synthesis
Language analysis
Translation assistance
Pose retrieval
Facial expression
Relationship retrieval
Character localization and recognition
Researchers
PPAANNEELLSS
6. 6
17/10/14 - christophe-rigaud.com
Balloon classification
Speaker localization
Text timestamps
Tone and pitch
Scene understanding
Layout analysis
Panel retrieval
Reflowing
De-hyphenation
Speech synthesis
Language analysis
Translation assistance
Pose retrieval
Facial expression
Relationship retrieval
Character localization and recognition
Researchers
TTEEXXTT
7. 7
17/10/14 - christophe-rigaud.com
Balloon classification
Speaker localization
Text timestamps
Tone and pitch
Scene understanding
Layout analysis
Panel retrieval
Reflowing
De-hyphenation
Speech synthesis
Language analysis
Translation assistance
Pose retrieval
Facial expression
Relationship retrieval
Character localization and recognition
Researchers
BBAALLLLOOOONNSS
8. 8
CCHHAARRAACCTTEERRSS
17/10/14 - christophe-rigaud.com
Balloon classification
Speaker localization
Text timestamps
Tone and pitch
Scene understanding
Layout analysis
Panel retrieval
Reflowing
De-hyphenation
Speech synthesis
Language analysis
Translation assistance
Pose retrieval
Facial expression
Relationship retrieval
Character localization and recognition
Researchers
9. 9
Challenges of comics analysis
● Variation of comic book styles
● Printing & scanning method changes over time (press, ink-jet, computer-based)
● Diversity of information to extract
● A lot of semantic relations to retrieve (element's relationships, text
17/10/14 - christophe-rigaud.com
meaning)
10. 11
Summary of comic character extraction
17/10/14 - christophe-rigaud.com
● Supervised approaches
● Character spotting given an example
● Unsupervised approaches
● Hypothesis from speech balloon tail position and orientation
● Region of interest from simple object removal
● Graphbased
approach (Nam)
11. 12
Summary of comic character extraction
17/10/14 - christophe-rigaud.com
● Supervised approaches
● Character spotting given an example
● Unsupervised approaches
● Hypothesis from speech balloon tail position and orientation
● Region of interest from simple object removal
● Graphbased
approach (Nam)
12. 13
Character spotting given an example [5]
● Aim: retrieve all the instances (apparitions) of a given character in all the
pages of a comic book title/album/collection
17/10/14 - christophe-rigaud.com
● Method summary:
1) Reduction of the number of colors
2) Get the colors of the query example (user interaction)
3) Compute the user query descriptor
4) Spot similar character instances
[5] C. Rigaud, D. Karatzas, J-C Burie, J-M Ogier Color descriptor for content-based drawing retrieval DAS'14
15. 16
3) Color selection: occurrence level
17/10/14 - christophe-rigaud.com
16. 17
3) Color selection: occurrence + discriminality levels
17/10/14 - christophe-rigaud.com
Top 3 colors
17. 18
4) Character spotting: example
17/10/14 - christophe-rigaud.com
User input
Image credits: (Prunelle, la fille du cyclope, Ankama, 2010
Descriptor
18. 19
4) Character spotting: example
correct detections wrong detections
17/10/14 - christophe-rigaud.com
19. 20
17/10/14 - christophe-rigaud.com
Conclusions
● Contributions
● Content Based Drawing Retrieval
● Color descriptor robust to scale,
rotation, translation, deformation and
occlusion
● Benefit of redundant information
● Limitations
● Colored comics
● Number of color (quantization)
● Requires one example of each character
Query Correct Wrong
Result examples
20. 21
Summary of comic character extraction
17/10/14 - christophe-rigaud.com
● Supervised approaches
● Character spotting given an example
● Unsupervised approaches
● Hypothesis from speech balloon tail position and orientation
● Region of interest from simple object removal
● Graphbased
approach (Nam)
21. 22
Hypothesis from speech balloon tail position and direction
● Given balloons in a panel, where are the (speaking) characters?
17/10/14 - christophe-rigaud.com
24. 25
17/10/14 - christophe-rigaud.com
Conclusions
● Preliminary results
● Highly relies on the quality of the
tail detection
● Only for “speaking” characters
(others can be spotted)
● Implicitly retrieves the
relationship between balloon and
character
25. 26
Summary of comic character extraction
17/10/14 - christophe-rigaud.com
● Supervised approaches
● Character spotting given an example
● Unsupervised approaches
● Hypothesis from speech balloon tail position and orientation
● Region of interest from simple object removal
● Graphbased
approach (Nam)
26. 27
Region of interest (ROI) from simple object removal
● Given an image from a double page of manga, can you predict where the
17/10/14 - christophe-rigaud.com
characters are?
27. 28
Region of interest (ROI) from simple object removal
● Given an image from a double page of manga, can you predict where the
? ?
17/10/14 - christophe-rigaud.com
characters are?
? ?
? ?
?
? ?
?
? ?
28. 29
Region of interest (ROI) from simple object removal
● Given an image from a double page of manga, can you predict where the
17/10/14 - christophe-rigaud.com
characters are?
? ?
? ?
?
? ?
?
42%
correct
? ? ? ?
Image credits: Yasuhisa Hara, "Kingdom", Shueisha, since 2006.
29. 30
Region of interest (ROI) from simple object removal
● In the same image, knowing the position of panels and balloons, can you
predict where the comic characters are?
17/10/14 - christophe-rigaud.com
30. 31
Region of interest (ROI) from simple object removal
● Knowing the position of panels and balloons, can you predict where the
?
17/10/14 - christophe-rigaud.com
comic characters are?
? ?
?
?
?
?
?
???
?
31. 32
Region of interest (ROI) from simple object removal
?
17/10/14 - christophe-rigaud.com
● Et voilà !
? ?
?
?
?
?
?
??
? ? 83%
correct
32. 33
A first test from automatic panels and balloons
extraction...
17/10/14 - christophe-rigaud.com