Project was created during a 3-months full-time data science program called Data Science Retreat. I’ve developed a brand recognition system that detects logos in real-life photos. It uses Deep Learning and SVM on Instagram images uploaded by users. It allows for better brand monitoring in social media independently from text descriptions and hashtags. I’ve used state-of-the-art Deep Learning tools (Theano, Lasagne, Caffe).

1. Brand recognition in real-life photos
Łukasz Czarnecki

2. • Machine Learning Engineer /Data Scientist
(Samsung)
• Human Tech Art founder and organiser
Łukasz Czarnecki

4. Monthly Actives Likes Daily Average Photos
Per Day
300M
Photos shared
30B+ 2.5B 70M
Instagram
Twitter
Facebook
Instagram
0.03%
0.07%
4.21%
Followers engagement on brand
Source: Forrester Research, Inc.

5. Feed noise

6. What’s seen by Instagram?

7. Brand recognition in real-life photos

8. Social media engagement

9. Deep Learning
Deep Learning
Needs big
amount of
examples
Great results
in Image
Recognition
Hard to train
Learns
features

10. VS VS
heavy data
augmentation
&
train your
network
Use pretrained
network
&
SVM
Use pretrained
network
&
adjust last
layers
Possible approaches

14. Sliding window
• 16 windows: width/3
• 9 windows: width/2
• 1 window: width

15. NUMERIC GALLERY SAMPLE
406 420351
340 255301
Initial dataset
Adidas
Nike
Starbucks Heineken
COSTA COFFEE Carlsberg

16. Random class

17. Network propagation time
Testing different
networks
VGG_S Network
Optimizing
the code
Switched for
batch processing
Changing
framework
Caffe
~1 sec
Initial time: 2 sec
~ 200 ms ~20 ms

18. Network output
224 x 224 x 3
4096 x 1Network

19. How do you use the network?

20. How do you use that with SVM?

21. Testing results

22. Getting features
from Network
Training SVM
on features
Testing with
sliding
window
First results
68%Precission
66%Recall

24. Adding counterexamples
70%Precission
65%Recall
+2%
-1%

25. Using different layer
73%Precission
67%Recall
+3%
+1%

27. Adjusting probability cutoff
76%Precission
65%Recall
92%
+3%
-2%

30. Getting more data
82%Precission
68%Recall
1
2
3
4
5
Training with
initial dataset
Getting more
unlabeled data
Collecting windows
with detections
Manual
segregation
Training with
bigger dataset
+500
per class
+6%
+3%

31. Data Augmentation
84%Precission
70%Recall
+2%
+2%

33. Adding normalisation
88%Precission
67%Recall
Getting features
from Network
Training SVM
on features
Testing with
sliding
window
Normalisation
+4%
-3%

35. Examples of filtering

36. Final results
Adidas
Nike
Starbucks
COSTA COFFEE
Heineken
40%
47%
84%
80%
88%
80%
76%
93%
90%
97%
Precission Recall
Carlsberg 66%93%
88%Precission
67%Recall

37. Tough examples

38. Thank you