This presentation discusses how Kafka Streams can expose the data from stateful operations, which can enable robust and more streamlined microservices.

1. Join the Confluent
Community Slack
Channel
Subscribe to the
Confluent blog
cnfl.io/community-slack cnfl.io/read
Welcome to the Boston Apache Kafka® Meetup!
5:30pm-6:00pm:
Doors Open, Networking, Pizza and
Drinks
6:00pm - 6:45pm:
Bill Bejeck, Confluent
6:45pm-7:00 pm:
Break/Networking
7:00pm - 7:45pm:
Bill Scott, Jacob Zweifel & Srinivas
of Tribal Scale, Cupcakes, Kafka,
and .NET CoreApache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. The Apache Software Foundation has no affiliation
with and does not endorse the materials provided at this event.

2. 2
Kick Your Database to the Curb
Using Kafka Streams Interactive Queries to Enable
Powerful MicroServices

3. 3
Brief Introduction
• Worked at Confluent (Streams Team) 2 years
• Apache Kafka Committer
• Author Kafka Streams in Action
Special thanks to @gamussa!

4. 4
Agenda
• What is State
• Kafka Streams Overview
• Describe Interactive Queries
• Live Demo!

5. 5
Stateful Stream Processing
What is State?

6. 6
Stateful Stream Processing
What is State?
Information your application needs to remember
beyond the scope of a single record

7. GroupBy Example
public static void main(String[] args) {
int counter = 0;
int sendInterval = 15;
Map<String, Integer> groupByCounts = new HashMap<>();
try(..consumer = new KafkaConsumer<>(consumerProperties());
..producer = new KafkaProducer<>(producerProperties())){
consumer.subscribe(Arrays.asList(”A”,”B”));

8. GroupBy Example
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}

9. GroupBy Example
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}

10. GroupBy Example
if(counter++ % sendInterval == 0) {
for(Entry<String, Integer> groupedEntry:groupByCounts.entrySet()){
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts",
groupedEntry.getKey(),
groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}

11. GroupBy Example
if(counter++ % sendInterval == 0) {
for(Entry<String, Integer> groupedEntry:groupByCounts.entrySet()){
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts",
groupedEntry.getKey(),
groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}

12. 12
Streams GroupBy
...
stream = streamBuilder.stream(Arrays.asList(“A”, “B”))
stream.groupByKey()
.count()
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))

13. 13
Streams GroupBy
...
stream = streamBuilder.stream(Arrays.asList(“A”, “B”))
stream.groupByKey()
.count()
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))

14. 14
Streams GroupBy
...
stream = streamBuilder.stream(Arrays.asList(“A”, “B”))
stream.groupByKey()
.count()
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))

15. 15
Stateful Stream Processing
Streams Stateful Operations
• Joins
• Windowing operations
• Aggregation/Reduce
Using any of these operations, Streams creates a state
store

16. 16
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service

17. 17
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service
Database

18. 18
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service
Database

19. 19
Making Streams Queryable
stream.groupByKey()
.count()
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))
..
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
someService.save(record.key(), record.value())
..
}
..

20. 20
Making Streams State Directly Queryable
...
stream = streamBuilder.stream(Arrays.asList(“A”, “B”))
stream.groupByKey()
.count(Materialized.as(“count-store”))
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))

21. 21
Making Streams State Directly Queryable
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"ks-interactive-stock-analysis-appid");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
"localhost:9092");
properties.put(StreamsConfig.APPLICATION_SERVER_CONFIG,
host+":"+port);
...

22. 22
Making Streams State Directly Queryable
Kafka Streams Application
KAFKA
Embedded RPC

23. 23
What’s with the APPLICATION_SERVER_ID
• A single Streams instance doesn’t contain all keys
• Streams will query other instances for store misses
• A single Streams instance can be proxy for all instances

24. 25
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
KAFKA
Streams app “B”
Host = hostB:4568

25. 26
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
Metadata -> hostB:4568
KAFKA
Streams app “B”
Host = hostB:4568
Metadata -> hostA:4567

26. 27
Topic Partitions and Streams Tasks
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
Topic with four partitions
Four partitions are converted to 4 tasks so
each streams application is assigned 2
partitions/tasks

27. 28
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
{“ENERGY”:”10000”} written to
partition 0 assigned to App A
{“FINANCE”:”11000”} written to
partition 1 assigned to App B

28. 29
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
{“ENERGY”:”10000”} written to
partition 0 assigned to App A
{“FINANCE”:”11000”} written to
partition 1 assigned to App B
http://hostA:4567?key=FINANCE

29. 30
Example of a Streams RPC
Kafka Streams Application
KAFKA
{ JS }
Demo Time!

30. 31
Embedding the Web Server
KafkaStreams kafkaStreams =
new KafkaStreams(builder.build(), streamsConfig);
InteractiveQueryServer queryServer =
new InteractiveQueryServer(kafkaStreams, hostInfo);

31. 32
Embedding the Web Server
.
kafkaStreams.setStateListener(((newState, oldState) -> {
if (newState == KafkaStreams.State.RUNNING
&& oldState == KafkaStreams.State.REBALANCING) {
queryServer.setReady(true);
} else if (newState != KafkaStreams.State.RUNNING) {
queryServer.setReady(false);
}
}))

32. 33
Embedding the Web Server
.
kafkaStreams.setStateListener(((newState, oldState) -> {
if (newState == KafkaStreams.State.RUNNING
&& oldState == KafkaStreams.State.REBALANCING) {
queryServer.setReady(true);
} else if (newState != KafkaStreams.State.RUNNING) {
queryServer.setReady(false);
}
}))

33. 34
Embedding the Web Server
public void init() {
get("/window/:store/:key/:from/:to", (req, res) -> ready ?
fetchFromWindowStore(req.params()) : STORES_NOT_ACCESSIBLE);
get("/window/:store/:key", (req, res) -> ready ?
fetchFromWindowStore(req.params()) : STORES_NOT_ACCESSIBLE);
get("/kv/:store", (req, res) -> ready ? fetchAllFromKeyValueStore(req.params()) :
STORES_NOT_ACCESSIBLE);
get("/kv/:store/:local", (req, res) -> ready ?
fetchAllFromLocalKeyValueStore(req.params()) : STORES_NOT_ACCESSIBLE);
get("/session/:store/:key",
(req, res) -> ready ?
fetchFromSessionStore(req.params()) : STORES_NOT_ACCESSIBLE);
get("/iq",
(req, res) -> {
res.redirect("interactiveQueriesApplication.html");
return "";
});

34. 35
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions> readOnlySessionStore = kafkaStreams.store(store,
QueryableStoreTypes.sessionStore());

35. 36
Embedding the Web Server
getHostInfo(String storeName, String key) {
StreamsMetadata metadata =
kafkaStreams.metadataForKey(storeName, key, stringSerializer);
return metadata.hostInfo();
}

36. 37
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions> readOnlySessionStore = kafkaStreams.store(store,
QueryableStoreTypes.sessionStore());

37. 38
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions>
readOnlySessionStore = kafkaStreams.store(
store,
QueryableStoreTypes.sessionStore());

38. 39
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
// Iterate over readOnlySessionStore and
// store results in a list sessionResults
return gson.toJson(sessionResults);

39. 40
Client View Development
<body>
<h2>Kafka Streams Equities Dashboard Application</h2>
<!–- Other div elements left out for clarity -->
<div id="sessionDiv">
<h3 id="sessionHeader">Customer Session Equity Activity Table</h3>
<table id="sessionTable">
<tr>
<th>Customer Id</th>
<th>Average Equity Transaction Spent Per Session</th>
</tr>
</table>
</div>
</body>

40. 41
Client View Development
<script>
function loadIqTables() {
$.getJSON("/kv/TransactionsBySector", function (response) {
updateTable(response, $('#txnsTable'))
$('#txnsHeader').animate({color:'red'},500).animate({color:'#CCCCCC'}, 500)
})
updateTableWithList("/window/NumberSharesPerPeriod/", symbols,
$('#stockTable'), $('#stockHeader'));
updateTableWithList("/session/CustomerPurchaseSessions/", customers,
$('#sessionTable'), $('#sessionHeader'))
}
setInterval(loadIqTables, 7000);
</script>

41. 42
Client View Development
<script>
function loadIqTables() {
$.getJSON("/kv/TransactionsBySector", function (response) {
updateTable(response, $('#txnsTable'))
$('#txnsHeader').animate({color:'red'},500).animate({color:'#CCCCCC'}, 500)
})
updateTableWithList("/window/NumberSharesPerPeriod/", symbols,
$('#stockTable'), $('#stockHeader'));
updateTableWithList("/session/CustomerPurchaseSessions/", customers,
$('#sessionTable'), $('#sessionHeader'))
}
setInterval(loadIqTables, 7000);
</script>

42. 43
Summary
Interactive Queries is a powerful abstraction that
simplifies stateful stream processing
There are still cases for which external database/storage
might be a better

43. 44
Summary
Kafka Streams in Action Examples: https://github.com/bbejeck/kafka-streams-in-
action/blob/master/src/main/java/bbejeck/webserver/InteractiveQueryServer.java
Music example: https://github.com/confluentinc/examples/blob/master/kafka-
streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/
KafkaMusicExample.java
Streaming Movie Ratings: https://github.com/confluentinc/demo-
scene/tree/master/streams-movie-demo

44. 45
Thanks!
Stay in Touch!
• https://slackpass.io/confluentcommunity
• https://www.confluent.io/blog/
• Twitter @bbejeck
• We are hiring! https://www.confluent.io/careers/

45. NOMINATE YOURSELF OR A PEER AT
CONFLUENT.IO/NOMINATE

46. KS19Meetup.
CONFLUENT COMMUNITY DISCOUNT CODE
25% OFF*
*Standard Priced Conference pass