As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

1. How to Discover, Visualize,
Catalog, Share and Reuse
your Kafka Event Streams
Jonathan Schabowsky
Senior Architect, CTO, Solace

2. Agenda
2
• Your growing Kafka event streams
• Flying blind
• Event portal: an API portal for event streams
• Demo: How to discover, visualize, catalog, share and reuse
your Kafka event streams
• Summary
© Solace
2

3. Does Your Organization Look Like this?
3
© Solace
Data Stream Flow
Legacy Application Infrastructure
….
?
?
?
JSON, XML, Plain Text…
IoT/Device Management
Producers
Consumers
Connectors
Stream
Processors
Avro

4. You’re Flying Blind…
Where do you discover
event streams for reuse?
Why does a given
event stream exist?
What topic do you use
to subscribe to it?
What changed?
How do you determine the
data structure of its payload?
Who should have access to it?
Who made the
latest revision?
Who can tell you more
about an event?
Is your change backward
compatible?
Who is impacted
by your change?
Does it comply with
security policy?
4
© Solace

5. Some Try Solutions Not Intended
for Event Stream Management
• It's a pain…
– No single place to discover
and share/reuse event streams
– No best practices guidance
• You’re breaking systems and
don’t know who changed
what???
– Suffer from changes with unintended
consequences
– Hard to audit changes when data is
stored in different tools
Brittle,
complex,
manual
solutions
Custom
Developed
Catalog
PowerPoint
API
Management
Spread
sheets
GitConfluence
5
© Solace

6. API Management
Solved this for REST
6
© Solace

7. They answer Who,
What, When,
Where, Why and
How For RESTful
APIs…
but what about
event streams?
Inspired by API Management Platforms
7
© Solace
API
Gateway
Documentation
• Discover
Registration
• Govern
Analysis
• Improve
• Monetize
Community
• Collaborate
Architects
and
Developers
(Internal, Partner
and Third-Party)
Productivity
• Generate Code
API
Portal

8. Sinks
“While most organizations have
basic event processing infrastructure
- such as notification services,
message buses, and event brokers -
many don’t have the high-level
productivity tooling that helps
developers design, develop, test and
manage event-centric applications.
Nor do they have tools for
governing, publishing and managing
event-based interfaces. The market
for these tools is sparse.”
Gartner “Top 3 Trends in Application Architecture That Enable
Digital Business” Anne Thomas, Yefim Natis, Mark O’Neill, 28 Oct 2019
8
© Solace
Event
Broker
Security
Persistence
Protocol
Translation
One-to-Many
Distribution
Source
Runtime
We Need an Event Portal
for Asynchronous Interactions
Design, Develop
and Manage
Event
Portal
Documentation
• Discover
Registration
• Govern
Analysis
• Improve
• Monetize
Community
• Collaborate
Productivity
• Generate Code (code gen)
(spec)

9. 9
© Solace
Single place to design, create, catalog, visualize, discover, share,
secure and manage all events within your ecosystem
Introducing Event Portal
Event Portal
Mainframe/
SAP
Distributed
Apps IoTMicro
services
Server
Less
Event Brokers

10. Event Portal
How Users Interact with Event Portal
10
Event
Management
(Design Time)
© Solace
Your Event-Driven
Enterprise
B2B/B2C
Ecosystem
IoTMicro-
services
Server-
less
Generate
Code
REST
API
Existing Tools
• Schema registries
• Data governance
• CI/CD pipelines
Customers
& Partners
• Discover and
consume events
in real-time
Architects
• Discover, create, update, review
apps, events and payload
schemas
• Map payload schemas to events;
and events to apps
• Govern event consumption
across app domain boundaries
Developers
• Collaborate with
architects on design
• Export AsyncAPI app
contracts for each app
• Generate and implement
event-driven apps that
provide business value
Data Scientists
• Understand
event-driven data
• Discover new insights
by combining events

11. Demo
11
© Solace

12. Benefits of an Event Portal
12
Solace
Your business can…
• Understand who is consuming which
events, how much, for what
• Create value-added services and
gain insights by combining events
• Make it easy for users across app
teams, LoBs and partner ecosystem
to find and reuse events
• Monetize popular or particular
high-value event streams
Reuse/Sharing
Architects & developers can:
• Collaborate on events
and event-driven apps
• Visually see event-driven
interactions to understand
impact of changes
• Low Code: Generate app code by
exporting AsyncAPI definitions
• Benefit from best practices &
consistently apply conventions
Productivity
CDO/data governance can:
• Understand where data is
coming from (lineage) and going
• Track changes and audit for
deviations while promoting apps
and dependencies though Dev,
Test, QA, and Prod
• Use application/event
relationships to create security
policies and validate compliance
with schema
Control

13. So, if you…
Have a lot of event-
driven applications
13
© Solace
You need an event portal!
Have (or need
to define) a lot of
event streams
Want to
manage events
as easily as you
manage your APIs

14. Event Portal = API Portal for Events/Streams
One tool to design,
create, catalog,
visualize, discover,
share, secure and
manage all of your
events and streams
(including Kafka and any
other event broker)
14
© Solace
Sign up for free!
• About data
and apps, NOT
managing Kafka
infrastructure
• High-level
productivity
tooling to help
you fully
embrace EDA
• Governance
of apps, event
and data