1. End-to-end Streaming
Between gRPC Services via Kafka
John Fallows

2. 2
From Northern Ireland, UK
California, USA
Now
Experience John Fallows
CTO @ Aklivity
Serial Entrepreneur
HTML5 Contributor
WebSocket Pioneer
25+ years Java
Author Pro JSF & Ajax
JMS, Kafka, gRPC, MQTT, …
grpc-kafka who am i?

3. grpc-kafka what is grpc?
❖ open source RPC (Remote Procedure Call) framework
❖ protocol buffers for service definitions
package example;
service EchoService
{
rpc EchoUnary(EchoMessage) returns (EchoMessage);
}
message EchoMessage
{
string message = 1;
}

4. grpc-kafka why grpc?
❖ strong contract for service definitions
❖ code generation for many languages
❖ protobuf over http/2 optimizes communication
❖ unified http/2 connection per client

5. grpc-kafka grpc challenges
❖ load balancing, client vs proxy, connection vs rpc
❖ limited browser support, requires proxy
❖ binary message format is not human readable
❖ lack of edge caching

6. grpc-kafka compare grpc vs kafka
service
service
service
⚙
service
⚙
service
⚙
service
❖ contract
❖ coupling
❖ communication
❖ codec

7. grpc-kafka integrate grpc + kafka (open source zilla)
Every network and application data flow is a stream.
Streams are composed to create efficient protocol translation pipelines.
shared memory
tcpin
tlsin
httpin
grpcin
kafkaout
tcpout
● Garbage free
● Wait free
● Elastic scalability
● Sliding window flow control
● Multi-core CPU friendly
● Extensible by design
⚙

8. grpc-kafka unary rpc
package example;
service EchoService
{
rpc EchoUnary(EchoMessage) returns (EchoMessage);
rpc EchoBidiStream(stream EchoMessage) returns (stream EchoMessage);
}
message EchoMessage
{
string message = 1;
}

9. server
(unary)
grpc-kafka unary rpc
grpc
kafka
kafka
grpc
zilla zilla
kafka
client
(unary)
POST POST
(correlated)

10. grpc-kafka bidi streaming rpc
package example;
service EchoService
{
rpc EchoUnary(EchoMessage) returns (EchoMessage);
rpc EchoBidiStream(stream EchoMessage) returns (stream EchoMessage);
}
message EchoMessage
{
string message = 1;
}

11. server
(bidi)
grpc-kafka bidi streaming rpc
grpc
kafka
kafka
grpc
zilla zilla
kafka
client
(bidi)
POST POST

12. demo
grpc end-to-end streaming via kafka

13. grpc-kafka integration impact (observe)
HEADERS
:method POST
:path /example.EchoService/EchoBidiStream
content-type: application/grpc
1: {"Hello, world"}
topic: echo-requests
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:reply-to echo-responses
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"} 1: {"Hello, again"}
EOS
HEADERS
:status 200
content-type: application/grpc
1: {"Hello, world"}
1: {"Hello, again"}
topic: echo-responses
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"}
TRAILERS
:grpc-status 0
HEADERS
:method POST
:path /example.EchoService/EchoBidiStream
content-type: application/grpc
1: {"Hello, world"}
1: {"Hello, again"}
EOS
HEADERS
:status 200
content-type: application/grp
1: {"Hello, world"}
1: {"Hello, again"}
TRAILERS
:grpc-status 0

14. grpc-kafka integration impact (reach)
topic: echo-requests
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:reply-to echo-responses
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"}
HEADERS
:status 200
content-type: application/grpc
1: {"Hello, world"}
1: {"Hello, again"}
topic: echo-responses
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"}
TRAILERS
:grpc-status 0
HEADERS
:method POST
:path /example.EchoService/EchoBidiStream
content-type: application/grpc
1: {"Hello, world"}
1: {"Hello, again"}
EOS
event
driven
micro
service

15. grpc-kafka integration impact (migrate)
HEADERS
:method POST
:path /example.EchoService/EchoBidiStream
content-type: application/grpc
1: {"Hello, world"}
topic: echo-requests
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:reply-to echo-responses
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"} 1: {"Hello, again"}
EOS
topic: echo-responses
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"}
HEADERS
:status 200
content-type: application/grp
1: {"Hello, world"}
1: {"Hello, again"}
TRAILERS
:grpc-status 0
event
driven
micro
service

16. grpc-kafka beyond request-response
❖ grpc server streaming from kafka (fanout)
❖ grpc server streaming from kafka (fanout, reliable)
❖ grpc client streaming to kafka (oneway)
❖ grpc client streaming to kafka (oneway, reliable)

17. grpc-kafka server streaming (fanout)
package example;
import "google/protobuf/empty.proto";
service FanoutService
{
rpc FanoutServerStream(google.protobuf.Empty) returns (stream FanoutMessage);
}
message FanoutMessage
{
string message = 1;
}

18. grpc-kafka server streaming (fanout)
grpc
kafka
zilla kafka
client
(server streaming)
POST
event
driven
micro
service

19. grpc-kafka server streaming (fanout, reliable)
HEADERS
:method POST
:path /example.FanoutService/…
content-type: application/grpc
EOS
HEADERS
:status 200
content-type: application/grpc
1: {"Hello, world"}
32767: 0xfda8c…
grpc
kafka
zilla kafka
client
1: {"Hello, again"}
32767: 0xe1ac…
✖

20. grpc-kafka server streaming (fanout, recovery)
HEADERS
:method POST
:path /example.FanoutService/…
content-type: application/grpc
last-message-id-bin: base64(0xfda8c…)
EOS
HEADERS
:status 200
content-type: application/grp
grpc
kafka
zilla kafka
client
1: {"Hello, again"}
32767: 0xe1ac…
✔

21. demo
grpc reliable server streaming via kafka

22. grpc-kafka zilla examples
❖ grpc.kafka.echo
Echoes messages sent to a Kafka topic via gRPC from a gRPC client
❖ grpc.kafka.proxy
Correlates gRPC requests and responses over separate Kafka topics
❖ grpc.kafka.fanout
Streams messages published to a Kafka topic, applying conflation
based on log compaction
❖ …
github.com/aklivity/zilla-examples

23. grpc-kafka multi-protocol

24. www.aklivity.io
github.com/aklivity/zilla ★
Q & A
pssst…we have 😎 socks at Booth S11
Stream Data Anywhere.