Text detection - recognition of memes form social networks

1. P12 - Text detection / recognition of memes
from social networks + sentiment analysis
Lviv Data Science Summer school 2019
Anastasiia Karliuk, Annika Schilk, Denis Smirnov, Oleksandr Trofymenko

2. Pipeline
Sentiment analysis

3. Images
• Organization: Whirl Software
• Data set: 8.216 memes
• Challenges:
• Different kinds of memes
(Tweets, photos,
screenprints, comics,…)
• Mostly English, but not
always
• Variety of fonts, sizes, …

4. Text detection
EAST MODEL CRAFT MODEL
CRAFT → https://github.com/hwalsuklee/awesome-deep-text-detection-recognition
EAST → https://github.com/argman/EAST

5. CRAFT Model

6. Preprocessing

7. Localization

8. Localization

9. deep-text-recognition-benchmark
1. Transformation (Trans.) → normalizes the input text image using the Spatial Transformer Network
(STN [1]) to ease downstream stages.
2. Feature extraction (Feat.) → maps the input image to a representation that focuses on the attributes
relevant for character recognition, while suppressing irrelevant features such as font, color, size, and
background.
3. Sequence modeling (Seq.) → captures the contextual information within a sequence of characters for
the next stage to predict each character more robustly, ratherthan doing it independently.
4. Prediction (Pred.) → estimates the output character sequence from the identified features of an
image.
==================================================================
[1]M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In NIPS, pages 2017–2025, 2015.

10. Text recognition
strangelove
or orchow learned to stop
worrying and love the trump

11. Results
21,76%
40,97%
61,41% 59,83%
50,34%
81,90%
66,18% 64% 64,30% 64,96%
0,00%
20,00%
40,00%
60,00%
80,00%
100,00%
EAST Model +
pytesseract
End-to-End Craft model +
Recogn
Craft model + Prepr Craft model
Postproc
Intersection accuracy Word Error Rate (WER)

12. Thank you for your
attention!
Any questions?
Have a safe trip home ☺
Oleksandr Trofymenko
trofimenko.alexander22
@gmail.com
Denis Smirnov
denis.smirnov.199818
@gmail.com -
Anastasiia Karliuk
Karliukanastasia
@gmail.com
Annika Schilk
annika.schilk@live.de