Slides of our talk at:
Spring.io Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka.
https://2019.springio.net/sessions/real-time-investment-alerts-using-apache-kafka-spring-kafka-at-ing-bank
By Tim van Baarsen & Marcos Maia
Nowadays our customers are expecting real time interactions with investments products and services we deliver at ING Bank especially because financial market fluctuations can have a direct impact on the investments performance of our customers.
In this talk, we present how we deal with the massive stream of price updates on stocks and how we leverage many features of Apache Kafka and Spring Kafka to improve and simplify our solutions. From simple producer / consumer up to advanced usage of streams we strongly rely on Kafka to solve our functional and non functional requirements delivering high quality, real time stream processing software that is reliable, precise and extremely valuable to our customers.
Main challenges of our project One of the main challenges of this project was to deal with a massive continuous stream of stock price updates and matching them against thousands of alerts created by our customers and still keep up with sending real time alerts to them.
3. ING is active in more than 40 counties
3
https://www.ing.jobs
4. Agenda
4
• Use case @ ING
• Mifid II
• Mobile Investments App
• Overall Architecture
• Demo
• Metrics / Monitoring
• Kafka streams
• Local development / Code walk through
• Lessons learned
• Questions
5. Disclaimer
5
Both the code and high level architecture we show here today is
not the actual product we developed within ING!
This example is inspired by our work
7. Mifid II: “Inform customers when a stock drops -10% in price
at the end of the day!”
7
Mifid II
Regulations
!
At the end of the day?
"No, in real time!Markets in Financial Instruments Directive II =
Regulatory reform financial markets European Union
came into effect on January 3, 2018
8. Screens of our mobile Investments app
8
price going up!
specific price
-10% drop
in price
Recurring alerts
13. Monitoring - Metrics
13
• Micrometer
• Think SLF4J, but for metrics (Jon Schneider)
• Application metrics
• Out of the box metrics
• Business metrics
• Vendor Neutral
15. Kafka Streams Join
15
Matching
Service
raw-stock-quote stock-quote-tick stock-quote-meta
log-compaction
Raw-tick
stock-quote-ticker RocksDB
Key value
ING Raw-tick
GOOGL Raw-tick
AAPL Raw-tick
Key value
ING Meta-data
GOOGL Meta-data
AAPL Meta-data
GlobalKTableKStream
16. Local development
16
• Docker
• Kafka
• Zookeeper
• Kafka manager
• All our Spring Boot applications
• Producers for development
• Custom Kafka command line producer
• Lab project
17. Spring Kafka
17
• Producing:
• ProducerFactory
• KafkaTemplate
• “The producer is thread safe and sharing a single producer instance
across threads will generally be faster
than having multiple instances.”
• Consuming
• Kafka consumers are not thread safe
• @KafkaListener
• MessageListener (many different interfaces)
• Access to ConsumerRecord
• Access to manual control to commit the offset
• Scale up multiple consumer (threads) within the application
18. Spring Kafka
18
• Integration tests
• Junit rule: EmbeddedKafkaRule
• @EmbeddedKafka
• Metrics
• Kafka consumer metrics out of the box (Micrometer)
19. Lessons learned
19
• Use Spring Kafka after you understand the Kafka APIs
• Error handling
• (De)serialization -> Apache Avro
• Think about # of partitions per topic upfront if possible (routing)
• Can your consumers keep up? Can you scale up?
• Monitor your topics / streams in production
• Based on metrics -> Micrometer
20. Lessons learned
20
• Kafka clients are backward compatible with Kafka brokers
• Kafka Streams
• KTables
• GlobalKTables
• Interactive Queries
• RocksDB build for ssd
• Describe
• processing.guarantee=exactly_once
• Kafka topology
• Plan ahead
• small & multiple topologies
