Serverless functions or FaaS are all the rage. By leveraging well established event-driven microservice design principles and applying them to serverless functions you can build a homogenous ecosystem to run FaaS applications. Kafka’s natural ability to store and replay events means serverless functions can not only be replayed, but they can also be used to choreograph call chains or driven using orchestration. Kafka also means you can democratize and organize FaaS environments in a way that scales across the enterprise. Underpinning this mantra is the use of Cloud Events by the CNCF serverless working group (of which Confluent is an active member).

1. 1
FaaS Composition using Kafka and CloudEvents
Neil Avery,
Office of the CTO,
@avery_neil

2. 22
The world is changing.

3. 33
The New Business Reality
Past
Technology was
Innovation required for growth
Running the business on yesterday’s data was
“good enough”
Today
Technology
Innovation required for survival
Yesterday’s data = failure.
Modern, real-time data infrastructure
is required.

4. 44
Transportation

5. 55
Transportation
ETA
Real-time sensor
diagnostics
Driver-rider match
Banking
Fraud detection
Trading and risk
systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

6. 66
What enables this
transformation?

7. 77
Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

8. 8
is a distributed event streaming platform
Publish & Subscribe
to Events
Store
Events
Process & Analyze
Events

9. 99

10. 1010
Out of the tar pit, 2006

11. 1111
“We believe that the major
contributor to this complexity in
many systems is the handling of
state and the burden that this adds
when trying to analyse and reason
about the system.”
Out of the tar pit, 2006

12. 1212
We like ‘all the things’

13. 1313

14. 1414
What about Microservices?

15. 1515
What are microservices?
Microservices are a software development
technique - a variant of the service-oriented
architecture (SOA) architectural style that
structures an application as a collection of
loosely coupled services.
https://en.wikipedia.org/wiki/Microservices

16. 1616
structures an application as a collection of
loosely coupled services.
https://en.wikipedia.org/wiki/Microservices
this is new!

17. 1717
So what went wrong?

18. 1818
Too many cooks

19. 19
REST anyone? Making changes is risky

20. 20
Handling state is hard Cache?
Embedded?
Route to right instance?

21. 21
Shifts in responsibility, redundancy

22. 22
● Scaling is hard
● Handling state is hard
● Sharing, coordinating is hard
● Run a database in each microservice - is hard
What have we learned about microservices?

23. 2323
Microservices
We had it all wrong (again)
FIX: Make them asynchronous and use
streams of events
We didn’t really
know what they
were anyway

24. 2424
Why?
Inversion of responsibility

25. 25
Typical event → command interaction

26. 26
Event → first interaction

27. 27
Evolvability

28. 2828
Stop!
Isn’t FaaS like a fine-grained, event
driven microservice?

29. 2929
Microservice/FaaS/Stream-Processor
Single atomic unit
does one thing
like

30. What is FaaS?
●
●
●
●
●
●
●
DRIVEN BY EVENTS
{...}

31. Stream processors FaaS (serverless fns)
{...}

32. Kafka cluster
stream
processing
Kafka Streams
KSQL
Producer/Consumer

33. 33
Capture behavior

34. 34
Time travel user experience?
how many users
affected?has it happened
before?
Ask many questions of the same data, again and again
time

35. 35
Evolvability
user experience?
how many users
affected?
has it happened
before?
new old
supports data change, logic change, logic extension, schema
evolution, loose coupling, add processors, A/B path

36. {
user: 100
type: bid
item: 389
cat: bikes/mtb
region: dc-east
}
Kafka partitions give you horizontal scale
/bikes/ by item-id
key#
Key
space
{...}
{...}
{...}
Topic
Partition
Partition
assignment
Stream
processor
FaaS
function

37. 37
Stream processors handle state
Stream
processor
Stream
processor
Stream
processor
Topic: click-stream
Interactive query
CDC events from KTable
CDC Stream
partition
partition
partition
CQRS
Elastic

38. 3838
Rate of adoption
● Serverless is the top-growing extended cloud service for the
second year in a row, with a 50 percent growth over 2018
24 to 36 percent adoption.
● Stream processing is tied for first, increasing from
20 to 30 percent adoption.
Rightscale cloud report 2019

39. 3939
Composition

40. 4040
Composition

41. 4141
Composition styles
1. Manual wiring
2. Chaining
3. Choreography
4. Orchestration / Workflow
(Step functions, Durable Fns,
Zeebee, Apache Airflow)

42. 4242
Composition styles
1. Manual wiring
2. Chaining
3. Choreography
4. Orchestration / Workflow
(Step functions, Durable Fns,
Zeebee, Apache Airflow)

43. 43

44. 44

45. 45

46. 46
payment.incoming.pre
2.
Payment
Inflight
payment.incoming
1.
{...}
Fraudulent
Check
“FaaS Connector”
a bridge between Kafka topic(s) and serverless function(s)

47. {...}
{...}
{...}
{...}
bad stream

48. 48
Bad stream?
As we say in IT...

49. 4949

50. 50
FaaS Connector modes
1. Sync:
● fire and queue response
● preserve order
2. Async:
● fire and forget
● order agnostic
https://www.confluent.io/hub
→ AWS Lambda Connector (preview)
→ Azure Functions Connector (preview)
→ GCP Functions Connector (coming)
<confluent-platform>
../bin/connect-standalone ./etc/json-standalone.properties ./etc/AwsLambdaSinkConnector.properties

51. 5151
Confluent Hub
confluent.io/hub

52. 52
Kafka Streams
● Data driven (dataflow)
● Partition aware
● Scale
● Resilient
● Delivery guarantees
● Replay events and time-travel
[Java, Scala]

53. 53
Stream processing ‘FaaS’ Patterns
● Filter (gate)
● Transform
● Fan-out (one to many)
● Fan-in (many to one - join)
● Branch/Choose (if ‘n’ then)
● Aggregate
● ...more...

54. 54
Kafka Streams: Filter/Gate - payment.isComplete()
{
StreamsBuilder builder = new StreamsBuilder();
KStream<String, Payment> incoming = builder.stream(TASK_TOPIC);
incoming.filter(
(key, payment) -> payment.isComplete()).to(TASK_COMPLETE);
…
Topology gateTopology = builder.build();
streams = new KafkaStreams(gateTopology, streamsConfig);
streams.start();
}

55. 55
Kafka Streams: Fan out
{
StreamsBuilder builder = new StreamsBuilder();
KStream<String, Payment> incoming = builder.stream(TASK_TOPIC);
incoming.flatMapValues(payment -> {
ArrayList<String> fanTasks = new ArrayList<>();
for (int i = 0; i < payment.getAmount(); i++) {
fanTasks.add("fan-" + i);
}
return fanTasks;
}
).to(TASK_FANOUT); flatMapValues: Many V to the same K
Or
flatMap: Many K,V to one K,V pair

56. 56
Kafka Streams: Fan In (join)
builder.stream(TASK_FANOUT)
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.windowedBy(SessionWindows.with(Duration.ofMinutes(1)))
.reduce(new Reducer<String>() {
public String apply(String agg1, String newValue) {
return agg1 + newValue;
}
})
.toStream()
.map((key, value) ->
new KeyValue<>(key.key() + "@" +
key.window().start() + "->" +
key.window().end(), value))
.to(TASK_REDUCE);
or: groupBy(KeyValueMapper)

57. Streaming events to FaaS: Fan-out Fan-in
{...}
{...}
{...}
{ bid:100; sold:1000;}
/bid-history
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
Calculating percentiles using a ‘unit-of-work’ pattern
{
digest.merge(digest)
}
Stream processor with
a session window to
join related events by
key (k, v[1-10])
Stream
processor
fan out is natural (data-driven)
OR
use a Stream processor flatMap()

58. 58
Multiple Bounded contexts
- Chaining
- Layering (Events as APIs)
2.
Logistics
payment.incoming 1.
Payments
payment.complete

59. {faas}
Central nervous system
appappappapp
Payments Department 2
{faas}appappappapp
Department 3 Department 4
Pattern: Central nervous system
Choreography
Choreography
Orchestration
Hierarchical topic naming

60. {faas}
What is the message format James?
appappappapp
Payments Department 2
Cloud Events: bringing order to chaos
I’m so confused!

61. https://cloudevents.io/

62. {
"specversion" : "0.4-wip",
"type" : "com.github.pull.create",
"source" : "https://github.com/cloudevents/spec/pull",
"subject" : "123",
"id" : "A234-1234-1234",
"time" : "2018-04-05T17:31:00Z",
"comexampleextension1" : "value",
"comexampleextension2" : {
"othervalue": 5
},
"datacontenttype" : "text/xml",
"data" : "<much wow="xml"/>"
}

63. ● Specification (an Event envelope)
● Transport Bindings (Kafka, AMQP, HTTP others)
● SDKs: Java, .NET, Go-lang etc
● Still early - nearly version 1.0
● Useful for exposing to external apps (Events as APIs)
● Support is coming for Kafka SerDes and CE.clients (sdk)
CloudEvents is a specification for describing event data in common formats
to provide interoperability across services, platforms and systems.
You will adopt CloudEvents

64. 6464
What about that software crisis that started in
1968?
“We believe that the major contributor to this complexity
in many systems is the handling of state and the burden
that this adds when trying to analyse and reason about
the system.”
Out of the tar pit, 2006

65. Key takeaways
● Event-Streaming apps
● Patterns
● Composition
● Scale

66. 66
Questions?
@avery_neil
“Journey to event driven” blog
1. Event-first thinking
2. Programming models
3. Serverless
4. Pillars of event-streaming ms’s
https://bit.ly/2tFfU84
or @avery_neil twitter profile

67. 67
@avery_neil