Slides for Munich Datageek Meetup

1. User Behaviour Tracking
Track - Store - Process
!
//Florian Pfeiffer - Head of Data&Infrastructure - gutefrage.net
!

3. Vision
„Let’s build our own Google
Analytics“

4. Why
analytics does sampling
we want the (raw) data

5. Ideas,Thoughts&Goals
fast / minimal impact on page loading time
high availability
track user over multiple platforms
storage engine? -> hbase

6. Infrastructure

7. Numbers!
10-20ms Response Time per pixel
record for now: ~2500 concurrent reqs
1,5 billion entries in Hbase
10 Nodes in Hadoop Cluster

8. Serving Infrastructure
Loadbalancers & RR DNS
nginx with empty_gif module (~2ms)
data is written to logfile

9. Storing Infrastructure
every nginx node has flume-ng
flume ingests logfile
AsyncHBaseSink with custom Serializer
direct writes to HBase

10. why flume?
we had it already in production ;)
Storm might be an interesting alternative

11. HBase rowkey design

12. Why?
You can scan through all data and use filters
for selecting specific data
But scanning with start & stop row speeds
things up (a lot)

13. HBase rowkey design
Do I need a fast user or a fast timespan
lookup?
User - clientid,ts<,connectionId>
Timespan - ts,clientid<,connectionId>

14. Inverse Timestamps
Data in HBase is stored lexicographicaly
sorted
Normal TS - scan would yield oldest results
first
Inverse TS - newer entries come first (and you
can cancel the scan if you have enough data)

15. Cross Domain Tracking
(Flash)Cookies
Fingerprinting
Etag
HTML5 Storage

16. The olden times…
or
Cookies
Easy to drop a 3rd party cookie with userId
on different websites
Gets more and more blocked (Safari, FF..)

17. Fingerprinting
Yields interesting results on desktop, difficult
on e.g. iPhone
invisible to user
Last resort if everything else fails?

18. Etag
Quite new, based on browser cache
sounds interesting

19. HTML5 Storage
Store data in local HTML5 storage
Retrieve data with Cross Domain Messaging

20. Store data
e.g. UserId, SessionId, GeoIP, URL, action,
data

21. Batch Processing
Calculate how many users are active on
platform A and also on B
Get Traffic of all Questions belonging to
Channel X sorted by Country

22. Now to something
completely different…

23. demo

24. Recommendations
with Myrrix

25. Myrrix
Evolution: taste -> mahout -> myrrix (-> oryx)
Recommender based on ALS

26. Recommendations @
GF.net
User emit signals on questions
view, like, gives answer, answer is voted best
Application sends signals through RabbitMQ
to recommendation servers

27. YEAH
but what happens, when a new user signs up?

28. ?

29. Fetch data from tracking
and feed it into myrrix

30. Collecting&Storing data
works great
using & processing is another thing ;)