Streaming Analytics in Uber for QCon SF 2016

1. Stream Processing &
Analytics with Flink
Danny Yuan, Engineer @ Uber
@g9yuayon

2. Four Kinds of Analytics
- On demand aggregation and pattern detection
- Clustering
- Forecasting
- Pattern detection on geo-temporal data

3. Two Ingredients
Geo/Spatial Time

4. Real-time aggregation and pattern matching

5. Slide title
Complex Event Processing

6. How many cars enter and exit a user defined area in past 5 minutes
Examples

7. It doesn’t have to be within a time or area
CEP with full historical context
Notify me if a partner completed her 100th trip in a given area just now?

8. Patterns in the future
How many first-time riders will be dropped off in a given area in the next 5
minutes?

9. Patterns in the future
How many first-time riders will be dropped off in a given area in the next 5
minutes?

10. Geo: user flexibility is important

11. Geo: user flexibility is important

12. It needs to be scalable

13. It needs to be scalable

14. It needs to be scalable
- Every hexagon
- Every driver/rider

15. CEP Pipeline Built on Samza
- No hard-coded CEP rules
- Applying CEP rules per individual entity: topic, driver,
rider, cohorts, and etc
- Flexible checkpointing and statement management

16. Slide title

17. Slide title

18. Slide title

19. Slide title

20. Slide title

21. Slide title

22. Slide title

23. Slide title

24. We need to evolve our architecture for other analytics

25. Clustering

26. Slide title Manually Created Cluster

27. Slide titleCall for algorithmically created clusters
- Clustering based on key performance metrics

28. Slide titleCall for algorithmically created cluster
- Clustering based on key performance metrics
- Continuously measure the clusters

29. Slide titleCall for algorithmically created clusters
- Clustering based on key performance metrics
- Continuously measure the clusters
- Different clustering for different business needs

30. Slide titleCall for algorithmically created clusters
- Clustering based on key performance metrics
- Continuously measure the clusters
- Different clustering for different business needs
- Create clusters in minutes for all cities

31. Slide titleCall for algorithmically created clusters
- Clustering based on key performance metrics
- Continuously measure the clusters
- Different clustering for different business needs
- Create clusters in minutes for all cities
- Foundation for other stream analytics

32. Slide title
Home-grown Clustering Service

33. Slide title
Home-grown Clustering Service
- All cities under 3 minutes

34. Slide title
Home-grown Clustering Service
- All cities under 3 minutes
- Pluggable algorithms and measurements

35. Slide title
Home-grown Clustering Service
- All cities under 3 minutes
- Easily pluggable algorithms and measurements
- Historical geo-temporal data for clustering

36. Slide title
Home-grown Clustering Service
- All cities under 3 minutes
- Easily pluggable algorithms and measurements
- Historical geo-temporal data for clustering
- Real-time geo-temporal data for measurement

37. Slide title
Home-grown Clustering Service
- All cities under 3 minutes
- Easily pluggable algorithms and measurements
- Historical geo-temporal data for clustering
- Real-time geo-temporal data for measurement
- Shared optimizations

38. Slide title
Home-grown Clustering Service
- All cities under 3 minutes
- Easily pluggable algorithms and measurements
- Historical geo-temporal data for clustering
- Real-time geo-temporal data for measurement
- Shared optimizations. To put things in perspective:
- 70,000 hexagons in SF
- Naive distance function requires at least 70,000 x
70,000 = 4.9 billion pairs!

39. Slide title
Home-grown Clustering Service
- All cities under 3 minutes
- Easily pluggable algorithms and measurements
- Historical geo-temporal data for clustering
- Real-time geo-temporal data for measurement
- Shared optimizations
- Incremental updates
- Compact data representation
- Memoization
- Avoid anything more complex than O(nlog(n))

40. Forecasting
- Every decision is based on forecasting

41. Forecasting
- Forecasting based on both historical data and stream input

42. Forecasting
- Forecasting based on both historical data and stream input

43. Forecasting
- Forecasting based on both historical data and stream input
Anomaly,
or emerging demand?

44. Forecasting
- Spatially granular forecasting - down to every hexagon

45. Forecasting
- Spatially granular forecasting - down to every hexagon

46. Forecasting
- Temporally granular forecasting - down to every minute

47. Forecasting
- Temporally granular forecasting - down to every minute

48. Pattern Detection
- Similarity of different metrics across geolocation and time
- Metric outliers across geolocations and time
- Frequent occurrences of certain patterns
- Clustered behavior
- Anomalies

49. Common Requirements in Pattern Detection
- Not just traditional time series analysis
- Incorporating insights on marketplace data
- Required both historical data and real-time input
- Spatially granular patterns - down to every hexagon
- Temporally granular patterns - down to every minute

50. Example: Anomaly Detection
- Simple time series analysis
- For a single geo area
- Can be noisy

51. A More Realistic Anomaly Detection

52. Example: Anomaly Detection

53. Example: Anomaly Detection

54. What’s the right architecture to support the analytics use cases?

55. Shared abstraction: multi-dimensional geo-temporal data

56. - Time series by event time
Shared abstraction: multi-dimensional geo-temporal data

57. - Time series by event time
Shared abstraction: multi-dimensional geo-temporal data
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

58. - Time series by event time
Shared abstraction: multi-dimensional geo-temporal data
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

59. - Time series by event time
Shared abstraction: multi-dimensional geo-temporal data
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

60. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
Shared abstraction: multi-dimensional geo-temporal data

61. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- e.g. event-based triggers
Shared abstraction: multi-dimensional geo-temporal data

62. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- e.g. event-based triggers
- e.g., triggers of computation results
Shared abstraction: multi-dimensional geo-temporal data

63. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing
Shared abstraction: multi-dimensional geo-temporal data

64. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing. E.g.,
Shared abstraction: multi-dimensional geo-temporal data

65. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing. E.g.,
Shared abstraction: multi-dimensional geo-temporal data

66. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing. E.g.,
Shared abstraction: multi-dimensional geo-temporal data

67. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing. E.g.,
Shared abstraction: multi-dimensional geo-temporal data

68. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing. E.g.,
State
Shared abstraction: multi-dimensional geo-temporal data

69. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing. E.g.,
State per key
Shared abstraction: multi-dimensional geo-temporal data

70. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing
- Unified stream
Shared abstraction: multi-dimensional geo-temporal data

71. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing
- Unified stream
- Real-time streams: unbounded streams
Shared abstraction: multi-dimensional geo-temporal data

72. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing
- Unified stream
- Real-time streams: unbounded streams
- Batch: bounded streams
Shared abstraction: multi-dimensional geo-temporal data

73. - Time series by event time
- Flexible windowing - tumbling, sliding, conditionally triggered
- Stateful processing
- Unified stream
- Real-time streams: unbounded streams
- Batch: bounded streams
- s/lambda/kappa
Shared abstraction: multi-dimensional geo-temporal data

74. - Ordering by event time
- Flexible windowing with watermark and triggers
- Exactly-once semantics
- Built-in state management and checkpointing
- Nice data flow APIs
Apache Flink

75. Mental Picture for Processing Geo-temporal Data

76. Mental Picture for Processing Geo-temporal Data

77. Mental Picture for Processing Geo-temporal Data

78. Mental Picture for Processing Geo-temporal Data

79. Mental Picture for Processing Geo-temporal Data

80. Mental Picture for Processing Geo-temporal Data

81. Mental Picture for Processing Geo-temporal Data

82. A Simple Example: simple prediction
Sources
.fromKafka()
.config(config)
.cluster(aCluster)
.topics(topicList)

83. A Simple Example
assignTimestampsAndWatermarks

84. A Simple Example
keyBy(…)

85. A Simple Example
.timeWindow(…)

86. A Simple Example
.flatMap(…)

87. A Simple Example
.keyBy(…)

88. A Simple Example
.apply(statefulFn)

89. A Simple Example
.addSink(…)

90. High Level Data Flow

91. High Level Data Flow

92. High Level Data Flow

93. High Level Data Flow

94. High Level Data Flow

95. High Level Data Flow

96. High Level Data Flow

97. High Level Data Flow

98. High Level Data Flow

99. High Level Data Flow

100. High Level Data Flow

101. Geotemporal API for efficiency

102. Geotemporal API for efficiency

103. Geotemporal API for productivity

104. Geotemporal API for productivity

105. Geotemporal API for productivity

106. Forecasting as an example

107. Forecasting as an example

108. Forecasting as an example

109. Forecasting as an example

110. Forecasting as an example

111. Forecasting as an example

112. Forecasting as an example

113. Forecasting as an example

114. Forecasting as an example

115. Forecasting as an example

116. Forecasting as an example

117. Forecasting as an example

118. Forecasting as an example

119. Forecasting as an example

120. Forecasting as an example

121. Forecasting as an example

122. Forecasting as an example

123. Lessons Learned
- Make sure you have robust infrastructure support
- Scaling up, namely single-node optimization matters
- Ensure exactly-once by proper data modeling
- Use external state store to avoid too much snapshotting
- Standardize monitoring and data validation

124. Lessons Learned
- Make sure you have robust infrastructure support

125. Lessons Learned
- Make sure you have robust infrastructure support

126. Lessons Learned
- Make sure you have robust infrastructure support

127. Lessons Learned
- Make sure you have robust infrastructure support

128. Lessons Learned
- Make sure you have robust infrastructure support
- Scaling up, namely single-node optimization matters

129. Lessons Learned
- Make sure you have robust infrastructure support
- Scaling up, namely single-node optimization matters
- Ensure exactly-once by proper data modeling

130. Lessons Learned
- Make sure you have robust infrastructure support
- Scaling up, namely single-node optimization matters
- Ensure exactly-once by proper data modeling
- Use external state store to avoid too much snapshotting
- Standardize monitoring and data validation

131. Lessons Learned
- Make sure you have robust infrastructure support
- Scaling up, namely single-node optimization matters
- Ensure exactly-once by proper data modeling
- Standardize monitoring and data validation

132. Choose a Stream Processing Platform

133. Thank You