http://jeeconf.com/program/how-to-cook-apache-kafka-with-camel-and-spring-boot/

1. 2016

2. HOW TO COOK APACHE KAFKA
WITH CAMEL AND SPRING BOOT
2Java EE conference 2016
Ivan Vasyliev
Playtika Core Services Team

3. AGENDA
Basics of Apache Kafka
Apache Camel
Spring Boot
Demo
Q&A
3Java EE conference 2016
CODE SLIDES

4. WHY APACHE KAFKA?
4Java EE conference 2016
http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf

5. WHY APACHE KAFKA?
Designed for large scale
Widely adopted by top tech companies
Hardened production quality product
Data replication out of the box
5Java EE conference 2016

6. FEATURES
At most once, at least once guarantees
Batching for high throughput cases
Efficient with DEFAULT settings
6Java EE conference 2016

7. EVEN MORE FEATURES
Mirroring between datacenters
Connectors to various DWH
Complex event processing integrations
7Java EE conference 2016

8. HIGH LEVEL VIEW
8Java EE conference 2016
http://kafka.apache.org/documentation.html#introduction

9. HIGH LEVEL VIEW
Publisher/subscriber and point-to-point models
Client which sends message – producer
Client which receives messages - consumer
9Java EE conference 2016

10. WHAT IS NOT INCLUDED - JMS
10Java EE conference 2016

11. WHAT IS NOT INCLUDED - JMS
Not a JMS compliant server
No message headers
Can employ message key
Send in payload
Wait for it, on roadmap
No transactions/JTA support
11Java EE conference 2016

12. WHAT IS NOT INCLUDED - EXACTLY ONCE GUARANTEE
12Java EE conference 2016

13. WHAT IS NOT INCLUDED - EXACTLY ONCE GUARANTEE
No exactly once guarantee
Duplicates because of failures
De-duplication is on roadmap
De-duplication on consumer
With camel EIP, by message ID/body
Consumer can tolerate duplicates
13Java EE conference 2016

14. APACHE KAFKA LANGUAGE
14Java EE conference 2016
http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf

15. APACHE KAFKA LANGUAGE
Topic - represents stream of messages
Contains set of partitions
Partition - subset of messages in stream
Partitioning is done by message key on producer
No “queue” in dictionary
15Java EE conference 2016

16. TOPICS AND PARTITIONS
16Java EE conference 2016
http://kafka.apache.org/documentation.html#intro_topics

17. TOPICS AND PARTITIONS
Partition is smallest unit of storage in kafka
Partition is data file with messages
Producer always append to end of file
Consumers scroll/seek over file
Consumer offset is persisted (zk or kafka)
Strong ordering guarantees for consumer
17Java EE conference 2016

18. QUEUE SEMANTIC IS DONE ON CLIENT
18Java EE conference 2016
http://kafka.apache.org/documentation.html#intro_consumers

19. QUEUE
Consumer offset is persisted by group id/per partition
Queue semantic inside of consumer group
Topic semantic between consumer groups
19Java EE conference 2016

20. CONSUMPTION IS ALL ABOUT OFFSETS
20Java EE conference 2016
https://hadoopabcd.wordpress.com/2015/04/11/kafka-building-a-real-time-data-pipeline/

21. CONSUMPTION IS ALL ABOUT OFFSETS
Consumer polls data from broker
Consumer offset is send (committed) to server
Auto offset commit enabled
By separate thread, periodically
Auto offset commit disabled
By your code, when batch of messages processed
21Java EE conference 2016

22. CONSUMER OFFSET AND AUTO-COMMIT
22Java EE conference 2016

23. CONSUMER OFFSET AND AUTO-COMMIT
With “auto-commit” enabled you can loose messages
Step1: One thread did not finish processing and failed
Step 2: Auto-commit thread does not care
Auto-commit is OK for status heartbeats
Auto-commit is NOT OK if you need “at least once”
guarantee, e.g. payment processing
23Java EE conference 2016

24. DATA REPLICATION
24Java EE conference 2016

25. DATA REPLICATION
Leader receives all reads and writes
Decides when to commit message
Follower syncs messages from leader
Take over if leader is down
Replication controller maintains leader
Zookeper used for coordination
Leader election
Consensus protocol
25Java EE conference 2016

26. APACHE KAFKA PRODUCER
26Java EE conference 2016

27. APACHE KAFKA PRODUCER
Performs load balancing
Uses message key to select partition
Finds appropriate kafka broker leader for partition
Has few configurable acknowledge modes
Can do batching in async mode
27Java EE conference 2016

28. DELIVERY GUARANTEED
28Java EE conference 2016

29. DELIVERY GUARANTEED
Durability with ack levels on producer side
Data replication between brokers
No in-memory state, efficient persistence
Manually committing offset on consumer side
29Java EE conference 2016

30. ISSUES - OPS
Ops is not free
There is Zookeeper on board
Easy to setup with Docker/Rancher
Need to learn basics to setup and monitor
30Java EE conference 2016

31. ISSUES – DATA
Can’t auto-scale existing data
Option 1: Add new partitions, they will go to new nodes
Option 2: Do it manually, move partitions around
Option 3: Wait for it, on roadmap
Mirroring seems to work into one direction
Can’t handle very large number of topics
31Java EE conference 2016

32. WHY APACHE CAMEL?
32Java EE conference 2016

33. WHY APACHE CAMEL?
Message routing DSL (java/scala/grooovy)
Enterprise Integration Patterns
Idempotent consumer (de-duplication)
Aggregator
…
Abstractions for testing
MockEndpoint
Route Advice
33Java EE conference 2016

34. APACHE CAMEL
34Java EE conference 2016
http://camel.apache.org/java-dsl.html

35. APACHE CAMEL
Lightweight and embeddable
Spring boot integration
Connectors to various message and data sources
35Java EE conference 2016

36. SPRING BOOT
36Java EE conference 2016

37. SPRING BOOT
Fat jar/jee containerless deployment
Autoconfiguration and conditionals
Сodeless usage of spring cloud/netflix projects
37Java EE conference 2016

38. 38Java EE conference 2016

39. GOTCHA’S – PRODUCER FASTER THAN CONSUMER, PRECONDITIONS
Its not recommended to have lots of partitions
Each partition is consumed by one consumer thread
Producer X times faster than consumer
39Java EE conference 2016

40. GOTCHA’S – PRODUCER FASTER THAN CONSUMERS, ACTIONS
Monitor kafka lag
Messages not consumed by group
Add intermediate multiplexing queue
See camel “seda” component
Think carefully since in-memory state can lead to data loss
Consider adding more partitions
Will allow more consumption threads
40Java EE conference 2016

41. GOTCHA’S – PRODUCER FASTER THAN CONSUMERS, TOOLS
41Java EE conference 2016
https://github.com/quantifind/KafkaOffsetMonitor

42. GOTCHA’S – AUTO OFFSET RESET
When you start test you do not receive any messages
Producer sends message before consumer is UP
Check auto.offset.reset setting in unit test
Latest (or largest in old api) can lead to consumption of only new messages
Earliest (or smallest in old api) will mean “from beginning”
42Java EE conference 2016

43. GOTCHA’S – CLIENT VERSION MIGHT NEED TO MATCH SERVER
Clients supposed to be “backward compatible”, but …
If you see weird things – you should check classpath
43Java EE conference 2016

44. GOTCHA’S – WATCH THE CLASSPATH
Multiple versions of kafka client
Multiple versions of kafka client dependencies
Multiple versions of zookeper client
44Java EE conference 2016

45. DEPENDENCY MANAGEMENT
Use dependency management to force versions and
exclusions
Use “Maven helper” Intellij plugin to check issues
https://github.com/krasa/MavenHelper
https://plugins.jetbrains.com/plugin/7179
45Java EE conference 2016

46. 46Java EE conference 2016

47. Thank you!
ivasylyev@playtika.com
Join us:
http://goo.gl/LuWMo3
47Java EE conference 2016