talk about kafka, storm

1. #kafka
#storm
@FifthEl ‘15
I’m @bhaskerkode aka Bosky
Product Engg at Helpshift

2. you’re here at a
bigdata,
& analytics
conference
to …

3. mobile
web
numbers
insights
predict
iot
internal
?
this talk

4. mobile
web
numbers
insights
predict
iot
internal
your job
your raise
?
this talk really is about sleep
good
sleep

5. Kafka is your glue
to have different teams break down
complex end-to-end problems into smaller
more manageable ones
like a “checkpoint” in a long hike/game/
drive
seperation of concerns helps you design distributed systems

6. kafka & stream
processing
power combo <insert quote to impress friend>

7. developed
world
wired
communication
wireless /
mobile
developing
world
wireless /
mobile

9. the same may happen
with bigdata & analytics
data
ingestion
offline
processing
stream
processing
data
ingestion
stream
processing

10. Linkedin
http://kafka.apache.org/

11. Twitter
https://storm.apache.org/

12. mobile
web
HA
iot
internal
biz
logic
typical analytics ingestion architecture
db
cache
search
infra

13. mo
web
HA
iot
inte
auth logic
task #1
kafka
producer
db
cache
search
infra
kafka
consumer
consumer storm
spark samza

14. how kafka helps me
sleep better
how? let’s talk numbers

15. ~300 million
requests/dayto our events kafka producer; ekaf
served by 22 cores
or 11 servers
running on auto-pilot
for 1+ years little maintenance, only scale servers
zero-downtime support plenty of metrics
(started with 3 servers replacing 14 JVM producers)
separate auth logic
from producer logic
erlang
= peaceful sleep
open source
(scalable)

16. mo
web
iot
inte
modular design
kafboy
ekaf
auth logic
HTTP 500
unauthorized
wrong api’s
metrics
HTTP 200

17. modular design
kafboy
ekaf
auth logic
HTTP 500
unauthorized
wrong api’s
metrics
HTTP 200

18. one thing particularly
distinguishes kafka
secret sauce? let’s take wild guesses

19. community?
NO let’s take more wild guesses at the secret sauce

20. let’s illustrate with some examples

21. You: “Create topic foo with 3 partitions”

22. kafka: “OK! each partition is set on
a different broker. one of the
brokers is an elected leader,
push your messages to the leader
host, port please. if leader goes
down, other broker is elected.”

23. is it the fault-tolerance
you get?
NO let’s take more wild guesses at the secret sauce
You: “create topic foo”
with 3 partitions
partitions
leaders
election

24. You: “hey broker1, where should i push to?”

25. kafka: please dial 1-800-metadata
to any broker. we all maintain this
data. you can make a metadata
request with topic(s) you want, and
we will return it s partitions, and
their broker hosts, ports, leader
info, etc

26. is it the separation of
concerns?
NO let’s take more wild guesses at the secret sauce
You: “hey broker1, where should i push to?”
metadata on all brokers
gives host & port of leader
for partitions + more info

27. You: “hey partition1 on broker1, for topic1
here are 10 messages.”

28. kafka: “thank you for calling partition1.
i will append it to a dir/file called topic1/
partition1.log.
Since its append only like a commit log,
you get ordering within partition free!”

29. is it the producer
speed you get?
NO let’s take more wild guesses at the secret sauce
You: “hey partition1 on broker1, for topic1
here are 10 messages.”
append only commit-log
ordering within partition
make 1 partition for global ordering

30. You: “hey broker2, where should i consume from?

31. kafka: “thank you for connecting to
the right broker. you can now read any
partition you want, and from any
offset. All i care about is which offset
to start reading from in a topic/
partition.log file”

32. You: “hey broker1, topic1, partition1
i want all data from message offset
….10(from ZK) onwards”

33. kafka: “thank you for calling partition1.
i’m going to sendfile the bytes you asked
from kernel-space directly to the socket
of a consumer using zero copy, thus
reducing context switches & minimal
garbage collection. yes, i am badass.”

34. is it the consumer
speed you get?
NO let’s take more wild guesses at the secret sauce
You: “hey broker1, topic1, partition1
i want all data from message offset
….10(from ZK) onwards”
offset bytes
sendfile bytes from offset to socket
kernel space not userspace

35. You: “for topic1,
i want 3 consumers all reading every msg.
for topic2,
i want the data split between 3 consumers”

36. kafka: “3 different pipelines/actions
on the same input topic1? Nice!
I can see your team is growing.
Thank you for grokking the concept of
consumer groups topic2. Make sure all
3 of your use the same group-id, and i’ll
take of the rest!”

37. is it the consumer
parallelism?
NO let’s take more wild guesses at the secret sauce
You: “for topic1, i want 3 consumers all reading every msg.
for topic2, i want the data split between 3 consumers”
can broadcast to all consumers
can split b/w a group of consumers

38. binary protocol.
but importantly
a documented spec of
what goes/comes over-the-wire
the one thing particularly
distinguishes kafka & gives it a
stellar status in the ecosystem is
its

39. the creators of kafka built the brokers & the spec
of how to communicate with it (producers & consumers),
and let the community speak the protocol
a documented spec of
what goes/comes over-the-wire

40. the creators of kafka built the brokers & the spec
a documented spec of
what goes/comes over-the-wire
eg: do you know what is sent
by a namenode/jobtracker/datanode over the wire?
(PS: where is the spec? come meet me after)

41. protocols win over api’s/drivers
the creators of kafka focussed on the spec
If you knew what data a namenode/jobtracker/datanode actually communicates over
the wire. It opens up a new world.
allows diff language clients to express
themselves best
its just data over TCP sockets.
more freedom
more integrations & faster adoption

42. 000300000000000
10007636c69656e7
4310000000100066
576656e7473
0,3,0,0,0,0,0,1,
0,7,99,108,
105,101,110,116,
49,0,0,0,1,0,
6,101,118,101,11
0,116,115
1. open a tcp socket to any kafka 0.8+ broker:port
2. send these 29 bytes that asks for metadata for topic
“events”
these 29 bytes
(or in hex)
and you’ll
get back
metadata
for the topic
“events”
always.

43. 0,3,
0,0,
0,0,0,1,
0,7,
99,108,105,101,110,116,49,
0,0,0,1,
0,6,
101,118,101,110,116,115
[2 bytes] metadata code = 3
[2 bytes] api version = 0
[4 bytes] int id (for replies)
[2 bytes] client id length = 7
[7 bytes] “client1”
[4 bytes] no# of topics = 1
[2 bytes] topic[0] length = 6
[6 bytes] “events”
what this packet means

44. <<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
% 0,0,0,1, % this is a response to req id 1
% 0,0,0,2, % number of brokers
% 0,0,0,1, % broker[0] id
% then broker name, host, port
% 0,0,0,3, % topics len
% 0,0,0,6, % topic[0] name len
% ……………….. % topic[0] name events
% 0,0,0,2, % topic[0] partitions len
% 0,0, % topic[0] partition1 error code
% 0,0,0,0, % topic[0] partition1
% 0,0,0,1 % topic[0] partition1 leaderid
% 0,0,0,1, % topic[0] partition1 replicas len
% 0,0,0,3, % topic[0] partition1 replica1
% 0,0,0,1, % topic[0] partition1 isr len
% 0,0,0,3, % topic[0] partition1 isr1
% another partition data here
% etc
metadata
response
decoded

45. similiarly all
other operations
encode the request as a packet
send over tcp socket
sync produce handle response
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
async produce
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
no response

46. MessageSet => [Offset MessageSize Message]
Offset => int64
MessageSize => int32
Messages
Message => Crc MagicByte Attributes Key
Value
Crc => int32
MagicByte => int8
Attributes => int8
Key => bytes
Value => bytes

47. the joy of knowing how
your data is encoded and
sent over tcp socket
keeps things simple, lets you sleep better
easier to debug, test, add middle-wares to audit, etc
MQTT endoded messages crypto encoded
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
thrift/avro encoded messages
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
AOL/YAHOO/* packets
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>

48. brokers focus on the
commit logs, etc
and delegate several opinionated areas
to the client, onus is on the client to make smart decisions
compression queue’ing
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
detect downtime
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>
load balancing to partitions
<<0,0,0,0,0,0,0,3,0,0,0,3,0,25,118,97,103,114,97,110,116,
45,117,98,117,110,116,117,45,112,114,101,99,105,115,101,
45,54,52,0,0,35,133,0,0,0,1,0,25,118,97,103,114,97,110,
116,45,117,98,117,110,116,117,45,112,114,101,99,105,115,
101,45,54,52,0,0,35,131,0,0,0,2,0,25,118,97,103,114,97,
110,116,45,117,98,117,110,116,117,45,112,114,101,99,105,
115,101,45,54,52,0,0,35,132,0,0,0,3,0,0,0,2,97,49,0,0,0,
2,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,3,0,0,0,1,0,0,0,3,0,
0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,
2,97,50,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,
0,1,0,0,0,1,0,0,0,0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,1,
0,0,0,2,0,0,0,2,97,51,0,0,0,2,0,0,0,0,0,0,0,0,0,3,0,0,0,
2,0,0,0,3,0,0,0,1,0,0,0,2,0,0,0,3,0,0,0,1,0,0,0,0,0,1,0,
0,0,1,0,0,0,2,0,0,0,1,0,0,0,2,0,0,0,2,0,0,0,1,0,0,0,2>>

49. clients
keeps things simple
a subset of what clients need to do include:
1. bootstrap
2. open sockets
3. encode packets
4. decode packets
5. send over tcp
6. route responses
7. handle failures, events
8. state machines

50. distributed systems
need to be built
as state machines
don’t trust a client that blocks
until an operation is complete

51. distributed systems
need to be built
as state machines
// send
// wait for response
BAD
even if this thread/process
is doing nothing until timeout
the idling is not efficient

52. distributed systems
need to be built
as state machines
// listening for socket states
// listening for responses
// send, and continue
// on_recv, route response
// after_timeout, route timeout
GOOD

53. distributed systems
need to be built
as state machines
set concurrency options for
ekaf
set the hostname of a load
balancer over your brokers
http://github.com/helpshift/ekaf

54. distributed systems
need to be built
as state machines
ekaf hits the ground
running with 1 call
// publish(topic, message)
if no state machine, it flows
from
request metadata ->
worker pool creation
-> socket connecting ->
ready state
http://github.com/helpshift/ekaf

55. distributed systems
need to be built
as state machines
if topic state machine has
metadata
it knows which broker for
each partition
if state already has socket,
queue it
http://github.com/helpshift/ekaf

56. distributed systems
need to be built
as state machines
all messages in states
before ready are
queued.
if queue hits size
OR
hits flush timeout. send it
http://github.com/helpshift/ekaf

57. http://github.com/helpshift/ekafGo through docs at

58. tests include broker downtime , adding a
broker, etc & a mini kafka broker for tests
http://github.com/helpshift/kafkamocker

59. flushed queue
worker up
worker down
downtime saved
downtime replayed
time to connect
max downtime q
Callbacks
used for Metrics

60. last year 6k/min
now 6k/sec
only scaled up
servers

61. hello world
driven
development
response time
against endpoint
to just echo

62. ekaf @Layer (ex-Apple engineers)
“The art of powering the Internet’s next messaging system”
https://www.youtube.com/watch?
v=mv2MBYU8Yls#t=33m5s

63. ekaf@ a chinese social network
1 pull request about to be merged
and elsewhere

64. back to pipelines
involving a kafka producer and consumer

65. SDK nginx
auth/api
#erlang
active user analytics pipeline @helpshift
PG
S3
kafka
http
producer
#kafboy (uses ekaf)
to disk
hyperloglog
counts
#clojure
kafka consumer
#clj-kafka
~1 billion
devices
HA
EMR
(internal jobs)
(dashboards)

66. mail delivery @helpshift
actually
sent
#clj-kafka
kafka consumer
#clj-kafka
email

67. [WIP] ES indexing @helpshift
actually
indexed
#clj-kafka
ES bulk index
docs

68. audit/action trails @helpshift
PG
#clj-kafka
kafka consumer
#clj-kafka
old object
new object
diff
emit/ignore
rows
few rules
objects are
namespace
must have id

69. Storm
@helpshift

71. iTunes
Play
reviews
distributed
crawler
#go
#master
#worker farm
#controller
the reviews storm pipeline @helpshift
PG
kafka
producer
#shopify/sarama
deduplication
tokenization
topic extraction
sentiment analysis
storm
kafka spout

72. example storm topology
read up more on spouts and bolts (any Q’s?)

73. • PG and multiple bolts tip
• Metrics at every bolt
• Local statsite -> grafana
• Avoid metric explosion
instrumenting tips

74. [WIP] segment population
query
ES
job
tracking
#clj-kafka
kafka consumer / storm
elasticsearch
query
representing
segment
scheduler
S3
counts
in PG
“find users who match level 2 , who did not find the easter egg”

75. fit your use case
population count
moving average
a note on samza’s state

76. Kafka is your glue
to have different teams break down
complex end-to-end problems into smaller
more manageable ones
like a “checkpoint” in a long hike/game/
drive
seperation of concerns helps you design distributed systems

77. numbers
your job
good sleep
your product
kafka + storm

78. Small Snapshot of Helpshift
Hay Day Boom Beach Clash of Clans Deer Hunter High School
Story
Family GuyFlipboard Circa Wordpress Misfit Microsoft
Outlook
APP + API
DB
MONITORING
OTHER
ROUTING
HAProxy
Our SDK is being embedded in a growing list of popular apps

79. #kafka
#storm
Thanks!
I’m @bhaskerkode aka Bosky
Product Engg @Helpshift
bosky@helpshift.com
Find this talk at
http://bit.ly/fifthel15-kafka-storm