When Funding Circle needed to scale its lending platform, we chose Kafka and Clojure. More than a programming language, Clojure is an interactive development environment with which you can build up an application function by function in a continuous unbroken flow. Since 2016 we have been developing our lending platform using Clojure and Kafka Streams, and today we process millions of transaction dollars daily. In 2018 we released "Jackdaw", our open-source Clojure library for working with Kafka Streams. In this talk, attendees will learn a radical new approach to building stream processing applications in a highly productive environment--one they can use immediately via Jackdaw or apply to their favorite programming system.

1. Data-Oriented Programming
with Clojure and Jackdaw
Charles Reese

2. Funding Circle
allows investors to
lend to established,
creditworthy
small businesses.
“We’ve made it our mission to help
small businesses across the world
go even further.”
Samir Desai
Co-founder and CEO
2
72,000
Small business
borrowers globally
$4.5 billion
Loans under
management globally
90,000+
Individual investors and
financial institutions

3. Kafka as a streaming platform
3
Kafka
API Front End Metrics Data Lake
Underwriting Servicing Trades Accounting

4. 2
8
2
40
Double-entry bookkeeping in Kafka Streams
4
sources
sinks
state stores
nodes
7
2
1
1
3
18
1
5
2
branch
foreach
flatMap
flatMapValues
map
mapValues
transform
transformValues
merge

5. 5
Why Clojure?
• Data-oriented
• Function composition
• The REPL
• JVM hosted

6. “
6
It's better to have 100 functions
operate on one data structure than
10 functions on 10 data structures.
Perlis, A. J. (1982). Epigrams on Programming.
SIGPLAN Notices 17(9)

7. Only a few data structures...
7
list, vector, map, set

8. 8
Many functions
distinct filter remove for keep keep-indexed cons concat lazy-cat mapcat cycle
interleave interpose rest next fnext nnext drop drop-while nthnext for take take-nth
take-while butlast drop-last for flatten reverse sort sort-by shuffle split-at split-
with partition partition-all partition-by map pmap mapcat for replace reductions map-
indexed seque first ffirst nfirst second nth when-first last rand-nth zipmap into
reduce set vec into-array to-array-2d frequencies group-by apply not-empty some reduce
seq? every? not-every? not-any? empty? some filter doseq dorun doall realized? seq
vals keys rseq subseq rsubseq lazy-seq repeatedly iterate repeat range line-seq
resultset-seq re-seq tree-seq file-seq xml-seq iterator-seq enumeration-seq

9. ((comp str +) 1 2 3 4)
;;=> "10"
(filter (comp not zero?) [0 1 0 2 0 3 0 4])
;;=> (1 2 3 4)
Composable parts
9

10. The REPL
10
interactive
responding to input
incremental
occurring in small steps
iterative
repeated with frequency

11. 11

12. 12
JVM hosted
(def streams-builder
(StreamsBuilder.))
(def kstream
(.stream streams-builder
"input"
(Consumed/with (serde) (serde))))
(.to kstream
"output"
(Produced/with (serde) (serde)))

13. Jackdaw
13
(defn topology-builder
[{:keys [input output] :as topics}]
(fn [builder]
(-> (j/kstream builder input)
(j/flat-map-values split-lines)
(j/group-by (fn [[_ v]] v))
(j/count)
(j/to-kstream)
(j/to output))
builder))
topic: input
topic: output
flatMapValues
groupBy
count
toStream

14. Jackdaw
14
(defn topology-builder
[{:keys [input output] :as topics}]
(fn [builder]
(-> (j/kstream builder input)
(j/flat-map-values split-lines)
(j/group-by (fn [[_ v]] v))
(j/count)
(j/to-kstream)
(j/to output))
builder))
topic: input
topic: output
flatMapValues
groupBy
count
toStream

15. 15
Jackdaw Features
• Create and manage topics
• Produce and consume
• Kafka Streams DSL
• EDN, JSON, and Avro serdes
• End-to-end tests

16. Testing with Jackdaw
16
(def commands
[[:write!
:input
"all streams lead to kafka"
{:key-fn (constantly "")}]
[:write!
:input
"hello kafka streams"
{:key-fn (constantly "")}]
[:watch
(fn [journal] (= 8 (count (get-in journal [:topics :output]))))
{:timeout 2000}]])

17. Testing with Jackdaw
17
(def commands
[[:write!
:input
"all streams lead to kafka"
{:key-fn (constantly "")}]
[:write!
:input
"hello kafka streams"
{:key-fn (constantly "")}]
[:watch
(fn [journal] (= 8 (count (get-in journal [:topics :output]))))
{:timeout 2000}]])

18. Testing with Jackdaw
18
(deftest test-xf-word-count
(jtf/with-fixtures [(jtf/integration-fixture topology-builder test-config)]
(jackdaw.test/with-test-machine (test-transport test-config)
(fn [machine]
(let [{:keys [journal]} (jackdaw.test/run-test machine commands)]
(is (= 1 (word-count journal "hello")))
(is (= 2 (word-count journal "kafka"))))))))

19. 19
["inside every large program"
"is a small program"
"struggling to get out"]

20. Process all the lines, one step at a time.
20

21. 21
Steps
1. Split each line into words.
2. Count the words for each line.
3. Reduce over all the lines.

22. Split each line into words.
22
(["inside" "every" "large" "program"]
["is" "a" "small" "program"]
["struggling" "to" "get" "out"])

23. Count the words for each line.
23
({"inside" 1 "every" 1 "large" 1 "program" 1}
{"is" 1 "a" 1 "small" 1 "program" 1}
{"struggling" 1 "to" 1 "get" 1 "out" 1})

24. Reduce over all the lines.
24
{"inside" 1 "every" 1 "large" 1 "program" 2
"is" 1 "a" 1 "small" 1
"struggling" 1 "to" 1 "get" 1 "out" 1}

25. 25
(defn f
[acc x]
(merge-with + acc x))
(->> coll
(map (fn [x] (split x #" ")))
(map frequencies)
(reduce f))

26. Process all the lines, one step at a time.Process all the lines, one line
26

27. 27
{"inside" 1
"every" 1
"large" 1
"program" 1}
{} {"inside" 1
"every" 1
"large" 1
"program" 2
"is" 1
"a" 1
"small" 1}
{"inside" 1
"every" 1
"large" 1
"program" 2
"is" 1
"a" 1
"small" 1
"struggling" 1
"to" 1
"get" 1
"out" 1}

28. 28
(comp
(partial map (fn [x] (split x #" ")))
(partial map frequencies))

29. 29
Transducers
• Composable transformations
• Take a reducing fn and return another reducing fn
• Decoupled from inputs and outputs
• Reusable
• Fast

30. 30
(map inc [1 2 3 4])
;;=> (2 3 4 5)

31. 31
(map inc)

32. 32
(def count-words
(comp
(map (fn [x] (split x #" ")))
(map frequencies)))
(def f
(completing
(fn [acc x]
(merge-with + acc x))))

33. 33
(transduce count-words f {} coll)

34. 34
{"inside" 1
"every" 1
"large" 1
"program" 2
"is" 1
"a" 1
"small" 1
"struggling" 1
"to" 1
"get" 1
"out" 1}
But...
[["inside" 1]
["every" 1]
["large" 1]
["program" 1]
["is" 1]
["a" 1]
["small" 1]
["program" 2]
["struggling" 1]
["to" 1]
["get" 1]
["out" 1]]
We want...

35. 35
[(["inside" 1] ["every" 1] ["large" 1] ["program" 1])
(["is" 1] ["a" 1] ["small" 1] ["program" 2])
(["struggling" 1] ["to" 1] ["get" 1] ["out" 1])]
[{"inside" 1, "every" 1, "large" 1, "program" 1}
{"is" 1, "a" 1, "small" 1, "program" 1}
{"struggling" 1, "to" 1, "get" 1, "out" 1}]

36. 36
(defn xf-running-total
[]
(fn [rf]
(let [state (volatile! {})]
(fn
([] (rf))
([result] (rf result))
([result input]
(let [next (as-> input %
(vswap! state #(merge-with (fnil + 0) %1 %2) %)
(select-keys % (keys input))
(map vec %))]
(rf result next)))))))

37. 37
(defn xf-running-total
[]
(fn [rf]
(let [state (volatile! {})]
(fn
([] (rf))
([result] (rf result))
([result input]
(let [next (as-> input %
(vswap! state #(merge-with (fnil + 0) %1 %2) %)
(select-keys % (keys input))
(map vec %))]
(rf result next)))))))

38. 38
(defn xf-running-total
[]
(fn [rf]
(let [state (volatile! {})]
(fn
([] (rf))
([result] (rf result))
([result input]
(let [next (as-> input %
(vswap! state #(merge-with (fnil + 0) %1 %2) %)
(select-keys % (keys input))
(map vec %))]
(rf result next)))))))

39. 39
(into [] (xf-running-total) coll')

40. 40
(def count-words
(comp
(map (fn [x] (split x #" ")))
(map frequencies)
(xf-running-total)))
(transduce count-words concat coll)
Putting it all together...

41. 41

42. 42
?

43. 43
transform()
+
flatMap()
transduce
+
concat

44. 44
(defn transduce-kstream
[xf kstream]
(-> kstream
(j/transform (fn [] (transformer xf))
["transducer"])
(j/flat-map (fn [[_ v]] v))))

45. 45
(defn transduce-kstream
[xf kstream]
(-> kstream
(j/transform (fn [] (transformer xf))
["transducer"])
(j/flat-map (fn [[_ v]] v))))

46. 46
(defn transformer
[xf]
(let [ctx (atom nil)]
(reify
Transformer
(init [_ context]
(reset! ctx context))
(transform [_ k v]
(let [store (.getStateStore @ctx "transducer")
v (first (into [] (xf store) [[k v]]))]
(KeyValue/pair k v)))
(close [_]))))

47. 47
(defn transformer
[xf]
(let [ctx (atom nil)]
(reify
Transformer
(init [_ context]
(reset! ctx context))
(transform [_ k v]
(let [store (.getStateStore @ctx "transducer")
v (first (into [] (xf store) [[k v]]))]
(KeyValue/pair k v)))
(close [_]))))

48. 48
(defn xf-running-total
[state swap-fn]
(fn [rf]
(fn
([] (rf))
([result] (rf result))
([result input]
(let [[k v] input
next (as-> v %
(swap-fn state #(merge-with (fnil + 0) %1 %2) %)
(select-keys % (keys v))
(map vec %))]
(rf result next))))))

49. 49
(defn xf-running-total
[state swap-fn]
(fn [rf]
(fn
([] (rf))
([result] (rf result))
([result input]
(let [[k v] input
next (as-> v %
(swap-fn state #(merge-with (fnil + 0) %1 %2) %)
(select-keys % (keys v))
(map vec %))]
(rf result next))))))

50. 50
(defn count-words
[state swap-fn]
(comp
(map (fn [[k v]] [k (split v #" ")]))
(map (fn [[k v]] [k (frequencies v)]))
(xf-running-total state swap-fn)))
Putting it all together...

51. 51
(transduce (count-words (atom {}) swap!) concat coll)

52. 52

53. 53
Example: Simple Ledger
[["1" {:debit-account "tech"
:credit-account "cash"
:amount 1000}]
["2" {:debit-account "cash"
:credit-account "sales"
:amount 2000}]]

54. 54
We want...
(["tech"
{:account-name "tech"
:before-balance 0
:after-balance -1000}]
["cash"
{:account-name "cash"
:before-balance 0
:after-balance 1000}]
["cash"
{:account-name "cash"
:before-balance 1000
:after-balance -1000}]
["sales"
{:account-name "sales"
:before-balance 0
:after-balance 2000}])

55. 55
(->> coll
(transduce (xf-split-entries nil nil) concat)
(transduce (xf-running-balances (atom {}) swap!) concat))

56. Topology
56
topic: entry-pending
topic: transaction-added
transduce-kstream
(xf-split-entries)
transduce-kstream
(xf-running-balances)
topic: transaction-pending

57. 57

58. 58

59. 59
Resources
"Transducers" by Rich Hickey
Strange Loop. (2014, September).
https://www.youtube.com/watch?v=6mTbuzafcII
Kafka and the REPL: Stream Processing, the Functional Way
Reese, C. (2018, November).
http://2018.clojure-conj.org/charles-reese/
Testing Event-Driven Systems
Chambers, A. (2018, April).
https://www.confluent.io/blog/testing-event-driven-systems
Jackdaw
Funding Circle. (2017).
https://github.com/FundingCircle/jackdaw