Bolt is one of the fastest-growing companies in Europe and so are our data volumes. Since our launch in 2013, we have reached 25 million customers in over 30 countries. Not only we grow our market share in cities where we operate, but we expand to new regions and launch new business divisions. It all comes with massive amounts of new data. In our day-to-day operations, we heavily rely on data for decision making and machine learning. That’s why we made a decision to switch towards real-time stream processing and event sourcing. It was a long and bumpy ride which has provided us with significant benefits. During this presentation, Ruslan will share his team’s experience migrating all company’s infrastructure to real-time data processing, problems team has encountered, short-cuts they have made, and lessons learned along the way – Kafka cluster management, structured and un-structure data processing, data (de)duplication, reliability guarantees, making migration process as seamless as possible.”

1. Doing real-time
stream processing
in one of the fastest-growing
European tech companies
Ruslan Gibaiev

2. 25 mln
passengers
90+ cities in 30 Countries

3. Bolt growth
in less than 3 years
3M ➟ 25M Riders
100 ➟ 1200 Team
11 ➟ 30 Countries
2014 2016 2018

4. Products launched this year only

5. Top
internet brands
(before IPO)
We do more with less25B
21B
5B
4B 2.3B 23M
9B
200M+
Funding data from Crunchbase

6. ● Requirements to data
● Where to get events from?
● Stream processing
● Problems along the way
Presentation outline

7. Requirements to data

8. Requirements to data
● Consistency
1 = 0

9. Requirements to data
● Consistency
● Ease of querying

10. Requirements to data
● Consistency
● Ease of querying
● Scalability

11. Requirements to data
● Consistency
● Ease of querying
● Scalability
● Speed

12. Where to get events from?

13. Microservices

14. Reality

15. Services have to TALK to each other

16. Services have to TALK to each other

17. Distributed transactions

18. How we do it

19. How we do it
?

20. How we do it
?

21. How we do it

22. ● Messaging system
● Open-source
● High throughput
● Horizontally scalable
● Developed @LinkedIn in 2011
Apache Kafka

23. Linearly scalable

24. Write once read many semantics

25. How we do it

26. Kafka Connect to rescue
Picture from Confluent

27. ● Framework
● To connect Kafka to other
systems
● Distributed
Kafka Connect

28. Stream processing

29. What is stream
● Unbounded
● Real-time
Picture from Confluent

30. Two types of stream processors
● StateLESS
● StateFUL

31. Two ways of doing stream processing
● KSQL
CREATE STREAM my_target_stream
WITH (KAFKA_TOPIC='destination_topic') AS
SELECT
id,
created,
code,
message,
state
FROM my_source_stream

32. Two ways of doing stream processing
● KSQL
● Kafka Streams API
Example data from Confluent

33. User defined functions
● Aggregations
● Filtering
● Anomaly detection
● Fraud prevention
Example from Confluent

34. User defined functions
● Aggregations
● Filtering
● Anomaly detection
● Fraud prevention
Example from Confluent
CREATE STREAM my_target_stream
WITH (KAFKA_TOPIC='destination_topic') AS
SELECT
id,
created,
MY_CODE_TRANSFORM(code),
message,
state
FROM my_source_stream

35. ML models
● Any ML model
● Written in / called from Java
● Can be used

36. Our setup:
● ~2K topics
● ~1K streams
● ~5K events/sec

37. Problems

38. Cluster setup and management
● Broker instances
● Connect clusters
● Stream processors

39. Good luck debugging it

40. JVM mostly

41. Data re-play

42. Data re-play

43. Other problems
● Database schema migrations
● Data deduplication
● Types incompatibility

44. Thank you!
Ruslan Gibaiev
Data Engineering
ruslan.gibaiev@bolt.eu