Exploring the problem of Microservices communication and how both Kafka and Service Mesh solutions address it. We then look at some approaches for combining both.

1. 1
Kafka and the Service Mesh
Gwen Shapira, Principal Data Architect
@gwenshap

2. You Have Microservices
2

3. They need to communicate
3

4. 4
Orders
Returns
?
Invento
ry
Fulfill
Order
Validat
e Order
I know! I’ll use REST APIs

5. Synchronous request-response communication
Leads to
Tight point-to-point coupling
5

6. 6
Making Changes is Risky
Change

7. 7
Adding Services Requires Explicit Calls
Mistak
e
Handle
r

8. All these lines of communication require:
Logging, monitoring, error handling and more
8

9. The most difficult thing about microservices
is data.
Especially joining data.
9

10. We need a way to communicate that is:
Decoupled
Flexible
Operationally Transparent
Data Aware
10

11. Kafka?
11

15. 15
From This:

16. 16
To This:

17. 17
And This: New
Servi
ce

19. Kafka:
- Immutable, ordered, stream (log) of events
- Highly Available
- Scalable
- Load balancing consumers
- HA consumers
- (Potentially) infinite storage

20. Kafka + Schema Registry
20

24. APIs between services are Contracts
In Stream Processing World – Event Schemas ARE the API
…except they stick around for a lot longer

27. We need a way to communicate that is:
Decoupled
Flexible
Operationally Transparent
Data Aware
27

28. Service Mesh?
28

29. It is a proxy.

30. Sidecar:
Components of the application, deployed in a separate
container to provide isolation and encapsulation.
This pattern allows applications to be composed of
heterogeneous components.

31. 31

32. Proxy as sidecar:
Proxy
Proxy
Proxy
Proxy
Proxy

33. Proxy
I have a new
IP now.
YOLO!
Who
cares?
I magically
know all
about it!

34. Proxy
I can recover
from errors
without drowning
Error?
No worries!
Lets retry every
millisecond forever
LOL. I’m
dropping 99%
of the retries.

35. Cool Things proxies do!
- One stable uri for each service
- Service discovery
- Log requests
- Monitor request latency
- Routing - A/B testing, green/blue deployments
- Circuit breaking
- Protocol translation
- SSL Termination

36. They don’t do it alone

37. We need a way to communicate that is:
Decoupled
Flexible
Operationally Transparent
Data Aware
37

38. How do we choose?
38

40. 40
Two different paradigms
Request-Response Events Streams
- Very Low latency
- Often sync
- Point-to-point
- “Bespoke API”
- Continuous processing
- Often async
- Event driven
- General-purpose events

41. Vision #1:
Using Service Mesh to hide Kafka
Introduce

42. Proxy
I am somehow
getting events
from Kafka
I’m using REST to
talk to a service
I’m really re-
directing
events to
Kafka

43. REST Proxy does bits of that.
HTTP on one-side, Kafka on the other.
We are adding more protocols:
- JMS
- MQTT
- gRPC

44. Vision #2:
Kafka as part of control plane

45. Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also
logging stuff to
Kafka

46. Vision #3:
Kafka as a Service in a Mesh

47. 47
Like This:
Proxy
Proxy
Proxy
Proxy
Proxy

51. Vision #4:
Front Kafka

52. 52
Like This:
P
R
O
X
Y
P
R
O
X
Y

53. Lets Sum Things Up
53

54. - REST API point-to-point communication
hits the wall
- Service Mesh adds discovery and operationalibility
- Kafka adds data, compatibility, immutable storage
- You can have both. In many different tastes.
- Coming soon to a proxy near you

55. Resources and Next Steps
https://github.com/confluentinc/cp-demo
https://www.confluent.io/download/
https://slackpass.io/confluentcommunity
https://www.confluent.io/blog
https://www.confluent.io/confluent-operator/
https://github.com/envoyproxy/envoy/issues/2852