Realtime Analytics (aka. LeanStream) with Druid

1. 2017.06.08
Realtime Analytics
(aka. LeanStream)
with Druid
KT NexR - Seungwoo Han / bucky.han@nexr.com

2. WHO AM I
• Startup Build Fail x 2
• KT NexR (feat. 믹서기)
• LeanStream 개발

3. Introduction
• History
• Realtime Analytics Flow
• Query Spec
• Used Query
• Realtime Analytics in Lean Stream Demo

4. History

5. History
NiFi1.0
Kafka0.9.0
SchemaRegistry1.2
Phoenix4.9.0/Hbase1.2.3
Druid0.9.0/Pivot
Lean Stream Web 1.4
MySql/MariaDB/etc
Spark 1.6.2
Yarn 2.7.3
HDFS 2.7.3
Zookeeper 3.4.6
Lean App 1.4

6. History

7. History

8. History
NiFi1.0
Kafka0.9.0
SchemaRegistry1.2
Phoenix4.9.0/Hbase1.2.3
Druid0.9.0/Pivot
Lean Stream Web 1.4
MySql/MariaDB/etc
Spark 1.6.2
Yarn 2.7.3
HDFS 2.7.3
Zookeeper 3.4.6
Lean App 1.4
Druid0.9.0/RealtimeAnalytics

9. Realtime Analytics Flow
Knockout
JQuery
request response
Serialize as Data
Generate
JSON
Query
Queries
Front Backend
Echart

10. Front Backend
Realtime Analytics Flow

11. Aggregation Query Spec
• Data Source
• Interval
• Dimensions
• Filters
• Aggregations
• Granularity
• Limit
• 데이터가 있는 저장소에서
• 원하는 시간에
• 원하는 컬럼값을
• 필터링할건 하고
• Dimension의 Metric값을 연산하여
• 어떤 시간(sec,min,hour,day,etc)으로 쪼개서
• 몇개를 조회할것인가!

12. Used Query

13. Used Query
•Time(__timestamp) only grouping
•One string dimension grouping
Generally
Timeseries , TopN > GroupBy

14. Used Query

15. Realtime Analytics in Lean Stream
Dimension : USER_ID, USER_NAME ….
Metric : count
가상의 실시간 사용자 통화데이터

16. Realtime Analytics in Lean Stream
Play
Realtime Analytics

17. Chart Type

18. Chart Type

19. Chart Type

20. Chart Type

21. Chart Type

22. Chart Type

23. Q&A

24. One More Thing