KAFKA +
Building the World's Realtime Transit Infrastructure
For Illustration only
SURGE - CIRCA 2013
SURGE - CIRCA 2016
DATA CONSUMERS
Real-time, Fast
Analytics
BATCH PIPELINE
Storm
Applications
Data Science
Analytics
Reporting
KAFKA
VERTICA
...
Product
Features
Predictive
Models
Operational
Analytics
Business
Intelligence
INFRASTRUCTURE ECOSYSTEM
NEAR REALTIME
PRICE SURGING
PRODUCT FEATURES
FRAUD -
ANOMALY
DETECTION
PREDICTIVE MODELS
PREDICTIVE MODELS
ETA
OPERATIONAL ANALYTICS
UberEATs
OPERATIONAL ANALYTICS
XP
OPERATIONAL ANALYTICS
BUSINESS INTELLIGENCE
KAFKA 8KAFKA 7 MIGRATOR
Limited Availability
Difficult to Scale
Not multi-DC Multi-lang incompatibility Multi-DC, multi-la...
Kafka 7
Mirrormaker
2.0
Rest
architecture
Data AuditAutomated
Topic Mgmt
Logs Business events
Async REST library
Data Audit
Local spooling
High throughput
custom protocol
REST ARCHITECTURE
Rest P...
Automated Schema and Topic Management
Mirrormaker 2.0
Robust
Data Audit
Dynamic topics
MIRROR MAKER 2.0
Destination DCSource DC
Msg counts across multiple DCs
End-end latencies across multiple
DCs
DATA AUDIT FOR KAFKA MESSAGES
Mirrormaker
2.0
Rest
architecture
Data Audit Kafka 8Automated
Topic Mgmt
A ROBUST FUTURE
0 data loss messaging system
Data discovery and lineage
Quota management
Self-correcting brokers
Active ac...
Real-time Data
Dynamic SQL(ish)
Real-time decision
THE FUTURE
Real-time Data
Custom Application
Real-time decision
THE PRE...
TELEMATICS
SELF DRIVING CAR
Thank you, Kafka Community!
Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout
Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout
Prochain SlideShare
Chargement dans…5
×

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

5 558 vues

Publié le

How Uber uses Kafka to drive our real-time business.

Publié dans : Ingénierie
  • Soyez le premier à commenter

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

  1. 1. KAFKA + Building the World's Realtime Transit Infrastructure
  2. 2. For Illustration only
  3. 3. SURGE - CIRCA 2013
  4. 4. SURGE - CIRCA 2016
  5. 5. DATA CONSUMERS Real-time, Fast Analytics BATCH PIPELINE Storm Applications Data Science Analytics Reporting KAFKA VERTICA RIDER APP DRIVER APP API / SERVICES DISPATCH (gps logs) Mapping & Logistic Ad-hoc exploration ELK Samza Alerts, Dashboards Debugging REAL-TIME PIPELINE HADOOP Surge Mobile App DATA PRODUCERS KAFKA 8 ECOSYSTEM @UBER
  6. 6. Product Features Predictive Models Operational Analytics Business Intelligence INFRASTRUCTURE ECOSYSTEM
  7. 7. NEAR REALTIME PRICE SURGING PRODUCT FEATURES
  8. 8. FRAUD - ANOMALY DETECTION PREDICTIVE MODELS
  9. 9. PREDICTIVE MODELS ETA
  10. 10. OPERATIONAL ANALYTICS
  11. 11. UberEATs OPERATIONAL ANALYTICS
  12. 12. XP OPERATIONAL ANALYTICS
  13. 13. BUSINESS INTELLIGENCE
  14. 14. KAFKA 8KAFKA 7 MIGRATOR Limited Availability Difficult to Scale Not multi-DC Multi-lang incompatibility Multi-DC, multi-language support 2013 2014 2015 - 2016 KAFKA 7 WORLD Difficult to Operate Producer Scale Issues High Availability High Scalability Kafka 7 + Mirrormaker Deployed everywhere Kafka 7 migrator Deployed everywhere New Kafka 8 pipeline
  15. 15. Kafka 7 Mirrormaker 2.0 Rest architecture Data AuditAutomated Topic Mgmt
  16. 16. Logs Business events Async REST library Data Audit Local spooling High throughput custom protocol REST ARCHITECTURE Rest Proxy
  17. 17. Automated Schema and Topic Management
  18. 18. Mirrormaker 2.0 Robust Data Audit Dynamic topics MIRROR MAKER 2.0 Destination DCSource DC
  19. 19. Msg counts across multiple DCs End-end latencies across multiple DCs DATA AUDIT FOR KAFKA MESSAGES
  20. 20. Mirrormaker 2.0 Rest architecture Data Audit Kafka 8Automated Topic Mgmt
  21. 21. A ROBUST FUTURE 0 data loss messaging system Data discovery and lineage Quota management Self-correcting brokers Active active data pipelines
  22. 22. Real-time Data Dynamic SQL(ish) Real-time decision THE FUTURE Real-time Data Custom Application Real-time decision THE PRESENT
  23. 23. TELEMATICS
  24. 24. SELF DRIVING CAR
  25. 25. Thank you, Kafka Community!

×