Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar, Stealth Startup

@apachepinot | @KishoreBytes
Look how easy it is to go from events,
to User-facing Realtime Analytics!
Using Apache Kafka and Apache Pinot

About Us
Neha Pawar
Engineer, Stealth Startup
Apache Pinot Committer
Tim Berglund
Developer Advocate
Conﬂuent

User-facing Realtime
Analytics
Analytics for the ALL
Analytics of the NOW

Analytics
Analytics for ALL
end-users
Queries triggered
by Apps
Personalized
analytics

@apachepinot | @KishoreBytes Time
Value
Of
Data
Analytics
Analytics of the
NOW

Who Viewed My Proﬁle
Seunghyun Lee
Senior Software Engineer
LinkedIn
Chinmay Soman
Founding Engineer
Total users 700 Million+
QPS 1000s
Latency SLA < 100 ms p99th
Freshness Seconds

UberEats Restaurant Manager
● Identify surge in realtime
● Detect missed/inaccurate
orders in realtime
Total users 500000+
QPS 100s
Latency SLA < 100 ms p99th
Freshness Seconds - Minutes

Challenges for the underlying system
User-facing
Realtime
Analytics System
Large Volume &
Velocity of Data
Realtime
Ingestion
1000s of QPS
Milliseconds
Latency
Seconds
Freshness
High
Dimensionality
Scalable

Kafka: The perfect solution for the events capturing part
Velocity of
ingestion
Realtime
Ingestion
Seconds
Freshness
High
Dimensionality
Scalable

Apache Kafka

Topics

Partitioning

Partitions Assignment

Producers

Consumers

How to solve low-latency high-throughput analytics part?
Large Volume &
Velocity of Data
Realtime
Ingestion
1000s of QPS
Milliseconds
Latency
Seconds
Freshness
High
Dimensionality
Scalable

Need a specialized analytics database that can..
Ingest from
Kafka & serve
real-time data
Handle high
event rate from
Kafka and
scale with
Kafka
Provide ultra
low latency at
high queries
per second
Handle dynamic query
patterns on highly
dimensional data w/o
exploding storage
1 2 3 4

Options?
Spark SQL
Presto
Big Query
Druid
Elastic Search
Kylin
KV Store
Latency
Flexibility
low
high
low
high

Introducing Apache Pinot

Apache Pinot Community
Slack Users
1100+
Companies
50+
Join our growing community
https://communityinviter.com/apps/apache-pinot/apache-pinot
Events/sec
1M+
Peak QPS
170K+
Query latency
ms

Need a specialized analytics database that can..
Ingest from
Kafka & serve
realtime data
Handle high
event rate from
Kafka and
scale with
Kafka
Provide ultra
low latency at
high queries
per second
Handle dynamic query
patterns on highly
dimensional data w/o
exploding storage
1 2 3 4

Apache Pinot Architecture
Pinot
Controller Zookeeper
Server 2
Server 1
Pinot Servers
Server 3
Pinot
Broker
Pinot
Brokers
Queries
Scatter - gather
Consuming,
indexing, serving

Server 3
Server 2
Server 1
p0 -> Server 1
p1 -> Server 2
p2 -> Server 3
p3 -> Server 1
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Pinot Realtime Ingestion
Queries
Consuming,
indexing, serving
Partition -> Pinot Server

Server 3
Server 2
Server 1
p0 -> Server 1, CONSUMING
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Queries
Consuming,
indexing, serving
State

Server 3
Server 2
Server 1
p0 -> Server 1, CONSUMING, 102
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Queries
Consuming,
indexing, serving
Start Offset
Kafka
Consumers

Server 3
Server 2
Server 1
p0 -> Server 1, DONE, 102, 300
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Queries
Consuming,
indexing, serving

Demo
● Emit events into a Kafka topic
● Create Pinot Schema and Table
● Query!

Indexing
User-facing
Realtime Analytics
Events
Indexes
Sorted
Inverted
Range
Star-tree
JSON
Geospatial
Text

Star-tree Indexing
country browser device os clicks
us chrome ... ... ...
ca firefox ... ... ...
jp ie ... ... ...
us firefox ... ... ...
ca ie ... ... ...
… … ... ... ...
select count(*) from X
where country = us and
browser = chrome
Country
Browser
Star-Tree Index
US CA
IE C IE C

Horizontal scaling by adding new Kafka brokers + Pinot servers
Apps
Kafka
Cluster
Pinot
Cluster
Server
Brokers
Producers
Kafka Consumers
Events

Kafka Producer - Reprise

Partition aware routing
Realtime Analytics
Events
Partitioned
stream
murmur3(memberId)
% numPartitions
Partition-aware
segments
Seg0 - partition #0
Seg1 - partition #1
Seg2 - partition #2
Seg3 - partition #0
Seg4 - partition #1
Seg5 - partition #2
Partition aware
query routing
SELECT SUM(metric)
WHERE memberId = m289
Choose a column as
partitioning key
Person
{
"memberId" : "m809",
"name" : "adam",
"age" : 30,
"addresses" : [ {
"number" : 1,
"street" : "main st",
"country" : "us"
}]
}
murmur3(m289) % numPartitions

User-facing Realtime Analytics System
Large Volume &
Velocity of Data
Realtime
Ingestion
1000s of QPS
Milliseconds
Latency
Seconds
Freshness
High
Dimensionality
Scalable

Takeaway
● Kafka is the key data infrastructure for event-streamed systems
● Kafka has good analytics solutions for operationalized streaming queries
● Pinot is purpose-built for ultra-low latency analytics, at high-throughput
● Pinot is a great solution for user-facing real-time analytics
● It is very easy to go from events in Kafka to analytics in Pinot
● Kafka + Pinot is the perfect combination for user-facing real-time analytics

Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar, Stealth Startup

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar, Stealth Startup

Similaire à Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar, Stealth Startup (20)

Plus de HostedbyConfluent

Plus de HostedbyConfluent (20)

Dernier

Dernier (20)

Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar, Stealth Startup