Streaming Analytics @ Uber

1. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Streaming Analytics @ Uber

2. U B E R | Data
Xiang Fu
Sr Software Engineer II @ Uber
Streaming Platform Team
Quick Introduction

3. LAS VEGAS SUMMIT 2015
Billion to Trillions ~ PB
Messages/day bytes/day
Apache Kafka at Uber

4. LAS VEGAS SUMMIT 2015
Billions 100s of TB - PB
Messages Processed / day Bytes Processed / day
Near Real-Time Analytics at Uber

5. LAS VEGAS SUMMIT 2015
100s Billions 10s of TB
Messages Stored Bytes Stored
Pinot at Uber

6. U B E R | Data
Agenda
● Case Study
● The case for SQL as building block
● Self-service Ecosystem
● Conclusion
● Future work

7. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Case Study

8. U B E R | Data
OLAP (OnLine Analytical Processing)
A / B Tests
See progress of
tests in real-time

9. U B E R | Data
OLAP use case
UberEats Manager
“What is my revenue for
past 90 days?”

10. U B E R | Data
Our integrated platform
● Filter events
● Merge streams
● Decorate with external data

11. U B E R | Data
What’s missing ?
● Cumbersome for data scientists / Ops people
● Redundant code
● Custom backfill pipelines

12. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
SQL as the building block

13. U B E R | Data
SQL + Stream Processing
70-80% of jobs can be
implemented via SQL

14. U B E R | Data
SQL + Stream Processing: Powerful abstraction
Projection, Filtering, Window, Join

15. U B E R | Data
SQL + Stream Processing: Schema Inference
Schema
Service

16. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
New eco-system: Athena X

17. U B E R | Data
New Eco-system: Athena X
HDFS
Kafka
Alerts
Kafka
Streams
Other data
destinations
Database
Streams
HDFS
Cassandra
HTTP
Pinot

18. Athena-X UI
Self Service Job Creation & Management

19. Athena-X UI (Cont)
High-level SQL(ish) expression

20. Athena-X UI (Cont)

21. Athena-X Deployment

22. Athena-X Deployment(Cont)

23. U B E R | Data
Batch World Story
Hive/Spark SQL/oozie
● Bootstrap/Backfill

24. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Conclusion

25. U B E R | Data
Conclusion
● Write SQLs to build streaming applications
○ Productivity: production in hours
● The AthenaX approach
○ Streaming processing as a platform
○ Self-serving production support end-to-end

26. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Future Work

27. U B E R | Data
Future Work
● Auto Scaling
● Multi DC Support

28. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Thank you