A pretty detailed story of how we built a real-time user monitoring platform, gathering data of millions of users. Using the joint forces of CDN, Cloud and BigData, we created a tool for developers and product owners to guide them towards right (and data-driven) product decisions.

1. Real User Monitoring at scale
Jarosław Bloch Albert Łącki

2. PL
SK
HU
RS
CH

4. 20 mln real users
7 mln requests every minute
150 mln PV daily

5. 3 data centers
3 000 servers and devices
130 Gbps traffic

6. 40 teams
> 250 deployments daily
300 specialists

8. Speed
search +0,4s PV/UU -0,5%
load +35% Revenue -5%
load +250ms PV/UU -1,8%
Revenue/UU -4,3%

10. Component monitoring

11. Service monitoring

12. User experience
Our responsibility
Under our control

13. External monitoring

16. Real User Monitoring

17. ?

18. Real User Monitoring
?
? ?
collection
ingestion processing presentation

19. Data collection

20. <script>
var data = collect();
data.send();
</script>
Data collection

21. Navigation Timing API
onLoad
DOM
Processing
ResponseRequestTCP, DNSRedirects
navigationStart
fetchStart
requestStart
responseStart domLoading
responseEnd domInteractive
domComplete

22. Resource Timing API

23. JS Errors
try {
…
} catch (ex) {
collect(ex);
}

24. Data collection
data

25. DC-1
Accelerator
DC-2
GET

26. HTTP 202
Data collection
data

27. DC-1
Script injection
<js>
Accelerator

28. Data ingestion
5 min

29. Data ingestion

30. Kafka
Partition 1
Partition 2
Partition 3

31. Data replication
DC-2DC-1

32. DATA
DATA EVERYWHERE

33. METRICS
METRICS EVERYWHERE

34. Storm
Topology
metrics
data

35. Presentation

36. Graphite
sum(metric1, metric2)
highestMax(services.*, 10)
…

37. Graphite
3 millions metrics

38. Custom alerts

40. Druid

41. Real User Monitoring

42. Deployments

43. Instant feedback

47. Thank you!