NAV handles a lot of applications for our benefits(sickness, unemployment etc.). We are rewriting the systems handling these applications to a stream based approach, placing Kafka-streams in the centre of our architecture. This talk will go through our approach to building these systems. We are mostly using Kotling as our programming language, and are running our applications on Kubernetes. We use Change-data-capture to extract data from our legacy systems, to ease the pain of migration. By treating our data, and our applications as streams we get a lot of benefits. We can rerun our applications with different rules, to simulate the effect of policy changes. We can do real time evaluation of applications, to improve the user experience. We also get better robustness, as we remove the runtime dependency to our core systems, and can receive applications even when other parts of the systems are down. The talk will also present the architecural rules we are implementing for NAV, using streams as the main communication pattern between organization units, and some learnings from our current work on a data platform to support Life is a stream of events.
4. Biggest organisation in the
norwegian welfare system
19 000 employees
Pays out a third of the
national budget in Norway
Responsibilities similar to
15-20 agencies in the US
- Pensions
- Unemployment
benefits
- Parental Leave
- Sickness Benefits
6. LEGACY
Old systems (up to 40
years), that are critical
infrastructure for Norway
Huge potential in
digitalisation
- More efficient
- Personalization
- Faster and fairer
case-management
- Simulations
- (Big) Data
7. Traditional flow
Paper based Applications
Mainframe for manual automation
Electrical paper (PDF)
Some Web Applications with richer interface
Data Security
Authorization is very important
8. Data
Architecture
Data is grouped because they “belong together”
Common Databases holds data from other agencies
Separate teams owns those databases, but data is used by others
IBM MQ for eventing
Service Bus for …
Big Separate Analytics operation, traditional DWH
Slow pace, lots of coordination
10. Early Use
Cases
Replacing MQ for some events
Change Data Capture on old systems
- Golden gate from Oracle
- InfoSphere from IBM
Small POCs
- Asynchronous microservices with
streams
- Pipelines into analytics with connect
11. Security
Teams owns their data and thus the topics and
approves consumers
Some issues with Connect and Streams at first
Azure AD not supported, built our own
autorisation plugin
14. The Fred George Way
Rapids and Rivers
Optimize for Speed
Need-pattern
Tiny microservices
15. Basic Architecture
One topic for everything
Publish a packet
All consumers listen for all
packets and filters based on
content
Choreography based on
packet content
Append data to packet
20. Comparison The Fred George
Way
Smaller autonomous
components
Can easily add new
features
Difficult to
understand the
whole picture
The Traditional way
Easier to reason
about
Good fit for kafka
streams
Shareable data for
free
Tighter coupling
between services
Bigger services
22. Software is policy
Implementing the rules from the books is the easy
part
Getting the data is the difficult part
Traditional integration architecture works against
speed
Software with changeability is a political asset.
24. Sharing all
decisions
Events are created in a distributed
fashion
Applications received
Decisions decisions made
Undecided:
- Common format?
- Producer or consumer
responsible?
25. Reverse Conway
Maneuver
NAV was built for projects
Product Areas
Changing the organization
- Move central registries to domain
teams
- Group some domains into areas
Conway's law.
Organizations which design systems ... are
constrained to produce designs which are copies of
the communication structures of these organizations.
The law is based on the reasoning that in order for a
software module to function, multiple authors must
communicate frequently with each other.
26. Sharing data across
service areas
Service areas are a collection of domains
Freedom inside an service area
Only Communication between service areas
using streams
Avro and unlimited retention
Master Data distributed as streams
Data on the inside, data on the outside
27. Commands vs Events
Events
Describing what has happened
Publish/Subscribe - N consumers
“Application Received”
Command
Expects a change
Probably only one consumer
Seldom across areas
“Execute Payment”
29. GDPR
Data Catalogue
Consumers must have a legal reason for
reading data
Document this reason outside code, but
automatic
Logic on the consumer side to filter out
message types
Log Compaction for deletion of user-data.
30. Sharing data on a
national level
Data originating in one organisation can
be important for another organisation
Kafka, as it is now, is not really suited for
communication between organisations
Alternatives exists like atom-feeds over
http
Future: Separate team that hunts life
events, and makes them available as
streams for everyone
31. CentralPerson
register
Atom feeds
Owned by Norwegian Tax Authorities, a separate organisation
Data used by almost every public organisation, and many
private
Slow rate of change - suitable for caching
33. Key points
Moving from synchronous to
asynchronous is not easy
- Change of mindset!
- Expose the data!
Streams work on both
application and system level
Make it a platfor