Startup Safary Budapest (http://budapest.startupsafary.com/) presentation 2016.04.21.

1. Fight against robots with
enbrite.ly data platform
Joe MÉSZÁROS

2. Joe MÉSZÁROS
lead software engineer
@joemesz
joemeszaros

3. Who we are?
Our vision is to revolutionize the KPIs and metrics the online
advertisement industry currently using. With our products,
Antifraud, Brandsafety and Viewability we provide actionable
data to our customers.

4. Ad display fraud
(ad stacking, pixel stuffing)
Ad viewability

5. Brand safety
Detecting traffic that comes from unwanted
categories (e.g. adult), countries and single domains

6. 39%
39%Anti fraud detection

8. DATA
COLLECTION
ANALYZE
DATA PROCESSION
ANTI FRAUD
VIEWABILITY
BRAND SAFETY
REPORT + API
What we do?

9. How we do? DATA PLATFORM
...so we need do analyze vast amount of data
Infrastucture Big Data
technologies
+
enbrite.ly
data
platform
=

10. Amazon Web Services (AWS)
● Most popular cloud service provider
● ~70 services, 13 geographical "regions"
● Amazon Big Data = Elastic Map Reduce
● BUT Do not trust the BIG guy (API problem)
https://aws.amazon.com/

11. Apache Hadoop
● de facto Big Data technology
● open source software
● distributed storage (HDFS) + data processing
(MapReduce)
● ecosystem: many additional softwares
http://hadoop.apache.org/ | https://github.com/apache/hadoop

12. Apache Spark
● large-scale data processing engine
● open source software (popular)
● modules: core, sql, sreaming, graph, ML
● faster than Hadoop MapReduce
http://spark.apache.org/ | https://github.com/apache/spark

13. Data platform in numbers
20+ node cluster
16 services 110 servers
0.5 - 4 TB /day
100+ TB on S3

14. How we do? DATA COLLECTION

15. How we do? DATA PROCESSION

16. Let me tell you a short story...

17. Real world example
You have a simple idea to detect bot traffic, which saves
the world. Let’s implement it!

18. Real world example
THE IDEA: Analyse events which are too hasty and deviate from
regular, humanlike profiles: too many clicks in a defined
timeframe.
INPUT: Collected events on Amazon S3
OUTPUT: Invalid sessions

19. Step 1: sessionize events
How to solve it?
Step 2: detect too many clicks
code: https://github.com/enbritely/startup-safary

20. Step 1: event to session
//configure Spark application
//read events from HDFS
JavaRDD<Event> events = lines.map(Converter::jsonToEvent);
Application code : https://github.com/enbritely/startup-safary
//configure Spark application
//read events from HDFS
JavaRDD<Event> events = lines.map(Converter::jsonToEvent);
JavaRDD<Event> clicks = events.filter(e ->
e.type.equals("click"));
//configure Spark application
//read events from HDFS
JavaRDD<Event> events = lines.map(Converter::jsonToEvent);
JavaRDD<Event> clicks = events.filter(e ->
e.type.equals("click"));
JavaPairRDD<String, List<Event>> grouped = clicks
.groupBy(Event::sessionId);
//configure Spark application
//read events from HDFS
JavaRDD<Event> events = lines.map(Converter::jsonToEvent);
JavaRDD<Event> clicks = events.filter(e ->
e.type.equals("click"));
JavaPairRDD<String, List<Event>> grouped = clicks
.groupBy(Event::sessionId);
JavaRDD<Session> sessions = grouped.mapValues(sessionizer);

21. Step 1: event to session
//Sessionizer
(Function<Iterable<Event>, Session>) unorderedEvents -> {
List<Event> clickOrdered = sortyByTimestamp(unorderedEvents);
Session session = new Session(sessionId);
for (Event event: clickOrdered) {
session.addClick(event.getTimestamp());
}
return session;
}
Application code : https://github.com/enbritely/startup-safary

22. Step 2: apply heuristic
Application code : https://github.com/enbritely/startup-safary
JavaRDD<String> badSessions = sessions
.filter(s -> s.getClickCount() > threshold)
.map(s -> s.sessionId + ":" + s.clickCount);
// save output to HDFS

23. Live demo!
● 4 node EMR (Hadoop) Cluster
● Apache Spark 1.6.1
● 1 GB input events
build app : create-cluster : events S3 -> HDFS : submit app

24. Congratulation!
MISSION COMPLETED
YOU just saved the world with a
simple idea within ~10 minutes.

25. WE ARE HIRING!
working @exPrezi office, K9
check out the company in Forbes :-)
amazing company culture
BUT the real reason ….

26. WE ARE HIRING!
… is our mood manager, Bigyó :)

27. BEYOND enbrite.ly
...our investor and event sponsor
is looking for talented guys

28. Joe MÉSZÁROS
lead software engineer
joe@enbrite.ly
@joemesz
@enbritely
joemeszaros
enbritely
THANK YOU!
QUESTIONS?