2019 Confluent Streaming Event

1. Amsterdam
6th November 2019
London
8th November 2019
Frankfurt
11th November 2019
David M. Walker
Data Management & Warehousing

2. What Does The Enterprise Want From Technology?
More For Less
Greater productivity when
creating or enhancing
technology solutions and
delivered by fewer people and a
shorter time to market
Cost Reduction
Paying less for the technology
stack and support teams
required to run the business
solutions
Risk Management
The reduction and/or mitigation of
risks that the business faces from
technology including fraud
protection, legislative and industry
standard compliance, data security,
platform availability, delivery, etc.
Better Analytics & Reporting
The ability for decision makers to access the data
that they need in order to make informed
decisions and execute strategically rather than
operate tactically. Where possible the operational
must be as automated as possible.
Differentiated Product
Technology that helps create a competitive
advantage in the commercial world or
demonstrates value add in a non-profit
environment

3. We are evolving our approach to building solutions …
… but are we getting the benefits ?
Micro Services
Architecture
Data
Governance
Agile
Development
Automation & Self
Service
Data Science
& AI

4. Where are we struggling to use these tools?
• Digital Native companies have been built this way
• Start ups can adopt these approaches from the outset
• Innovators are probably already cloud native
• But for many organisations the elephant
in the room is the existing infrastructure
of on-premise monolithic platforms
• How do we make the change from large
legacy platforms to modern architectures ?

5. What is a Micro-Service Architecture?
Rest API
Operational User Interface – Any Channel (Web, Mobile App, Third Party, etc.)
Microservice Database
Micro-Application
(e.g. Admin)
Data
Capture
Event
Handlers
Event Streaming based on Distributed, Replicated Event Log – (e.g. Kafka)
Rest API
Microservice Database
Micro-Application
(e.g. Orders & Booking)
Data
Capture
Event
Handlers
Rest API
Search Engine
Micro-Application
(e.g. Search)
Data
Capture
Event
Handlers
User Interface
Corporate Business Platform
Enterprise Database
(e.g. Oracle, MS SQL Server)
Monolithic Architecture Microservices Architecture
Analytic & Data
Science Tooling
Analytic Database(s)
Analytical
User Interface
Event
Handlers
Rest
API
StreamAnalyticTooling
Data
Capture

6. The Micro-Service Architectural Approach …
• A microservice architecture is a style that structures
an application as a collection of services that are:
• These are designed to facilitate the rapid, frequent
and reliable delivery of large, complex applications,
whilst allowing the organization to evolve its
technology stack.
https://microservices.io/
Highly maintainable
and testable
Loosely Coupled
Independently
Deployable
Organized Around
Business Capabilities
Owned By
A Small Team

7. Example: Beneficiaries In A Payment System
• Creates new beneficiary
• POST /v1/customer/beneficiaries
• List all beneficiaries in customer environment
• GET /v1/customer/beneficiaries
• Update existing beneficiary
• PUT /v1/customer/beneficiaries/{beneficiary_id}
• Fetch the beneficiary entity
• GET /v1/customer/beneficiaries/{beneficiary_id}
• Creates new account for beneficiary
• POST /v1/customer/beneficiaries/{beneficiary_id}/accounts
• Return all accounts for given for beneficiary
• GET /v1/customer/beneficiaries/{beneficiary_id}/accounts
• Return account for beneficiary
• GET /v1/customer/beneficiaries/{beneficiary_id}/accounts/{account_id}
• Update existing account for beneficiary
• PUT /v1/customer/beneficiaries/{beneficiary_id}/accounts/{account_id}
• Change default account of beneficiary
• PUT /v1/customer/beneficiaries/{beneficiary_id}/accounts/{account_id}/make-default
https://docs.railsbank.com/api/beneficiaries
Rest API
Operational User
Interface
Microservice Database
Beneficiaries
Micro-Application
Data
Capture
Event
Handlers

8. The Benefits (1)
• Does one job well
• In the example that is managing beneficiaries
• A well defined versioned API that is easy to integrate with
• Has a GDPR benefit of being able to identify and audit all interactions with
beneficiaries data
• Only beneficiary_id and account_id are held outside the service
• Security benefits
• Can secure data at rest and data in motion, tokenisation, authentication and
access control
• Owned by a single team using a technology stack of their choice
and architect the service to handle specific requirements
• For example distributed or local database, CAP requirements, etc.
• Note: There shouldn’t be unbounded choice of technologies
Rest API
Operational User
Interface
Microservice Database
Beneficiaries
Micro-Application
Data
Capture
Event
Handlers

9. The Benefits (2)
• Easy to build an automated Continuous Deployment (CD) and
Continuous Integration (CI) process that operates asynchronously
from other micro-services release cycles
• Can be deployed in a scalable way on premise or in the cloud
• Can be used by multiple legacy platforms
• Can aid performance on the legacy platforms by off-loading work
Rest API
Operational User
Interface
Microservice Database
Beneficiaries
Micro-Application
Data
Capture
Event
Handlers

10. The Challenges
• Stripping Out Functionality from Existing Systems
• Its not the obvious functionality but the implicit functionality – for example “Mandate
Printing” might reference “Beneficiary” tables would no longer be there
• Performance
• Compared to a tightly coupled system a call out to a remote platform is inherently
slower
• For most services this is not a big problem – but becomes significant against core
transactions, where scale-out services need to be used to mitigate
• Management
• As an organisation deploys more services there is an overhead to managing the
number of interactions between services
• The management strategy needs to evolve early – don’t wait until you have a problem
• Watch For Service Scope Creep
• Don’t let your services become the new monoliths
Rest API
Operational User
Interface
Microservice Database
Beneficiaries
Micro-Application
Data
Capture
Event
Handlers

11. Communicating between Services
Transaction Service
TransactionService
CreateTransaction
Saga
Transaction
Create Transaction
Account Service
Account
CreditLimit
CreditReservations
create()
create()
approve()
Account Command Channel
CreateOrder Saga Reply Channel
*This is an orchestration pattern, other patterns e.g. choreography exist
Reserve Credit
Credit Reserved
• The channel provides a method of communication between services
• Serialises events to ensure that they happen in the right order
• Added benefit that it is recording the business events as they happen
This is Kafka

12. On Premise On Premise
Migrating to Micro Services – First Steps
The
Monolith
2001: A Tech
Odyssey
On Premise Cloud
Public ServicesIn House Developed Services
Deploy A New
Service that adds
functionality e.g.
Fraud Detection Experiment with
On-Prem & Cloud
Deployment
Use Third Party
Micro Services
to reduce
development costs
Fraud
Detection
Fraud
Detection
Prospect
Scoring

13. On Premise On Premise
Extract Single Function Micro Services
The
Monolith
2001: A Tech
Odyssey
On Premise Cloud
Extract Stand Alone
Functionality e.g.
’Beneficiary’
Fraud
Detection
Service
Fraud
Detection
Service
Beneficiaries
Service
Beneficiaries
Service
Public ServicesIn House Developed Services
Prospect
Scoring
Service

14. On Premise On Premise
Move To Micro Service Interactions
The
Monolith
2001: A Tech
Odyssey
On Premise Cloud
Public ServicesIn House Developed Services
Account
Service
Customer
Service
Inter Service Channels
Fraud
Detection
Service
Fraud
Detection
Service
Beneficiaries
Service
Beneficiaries
Service
Prospect
Scoring
Service
Progressively
Less Deployments
On Premise
Inter Service Channels

15. On Premise On Premise
Move Core Functions To Cloud Service
The
Monolith
2001: A Tech
Odyssey
On Premise Cloud
Public ServicesIn House Developed Services
Account
Service
Customer
Service
Inter Service Channels
Fraud
Detection
Service
Fraud
Detection
Service
Beneficiaries
Service
Beneficiaries
Service
Prospect
Scoring
Service
Inter Service Channels
Transaction
Service
Authorization
Service
Core Business
Functions and
Innovative New
Product/Services

16. On Premise On Premise
Decommission Your Legacy Monolith & Move To Cloud
On Premise Cloud
Public ServicesIn House Developed Services
Account
Service
Customer
Service
Inter Service Channels
Fraud
Detection
Service
Fraud
Detection
Service
Beneficiaries
Service
Beneficiaries
Service
Prospect
Scoring
Service
Inter Service Channels
The
Monolith
2001: A Tech
Odyssey
Transaction
Service
Authorization
Service
Monetise
Your
Data by
providing
services
to others
• This journey will (probably) take
several years
• There are significant benefits
along the way that make the
journey worthwhile
• Mitigate the risk associated with
migration to a new platform
However there are No Silver Bullets

17. What about Analytics & Data Science capabilities?
Rest API
Operational User Interface – Any Channel (Web, Mobile App, Third Party, etc.)
Microservice Database
Micro-Application
(e.g. Admin)
Data
Capture
Event
Handlers
Event Streaming based on Distributed, Replicated Event Log – (e.g. Kafka)
Rest API
Microservice Database
Micro-Application
(e.g. Orders & Booking)
Data
Capture
Event
Handlers
Rest API
Search Engine
Micro-Application
(e.g. Search)
Data
Capture
Event
Handlers
Analytic & Data
Science Tooling
Analytic Database(s)
Analytical
User Interface
Event
Handlers
Rest
API
StreamAnalyticTooling
• Events can be routed directly
and in real-time to analytical
platforms
• Move from trying to
understand what was
happening after the event
(data warehousing) to
analysing events as they
happen (streaming analytics)
• Opportunity to offer your
analytics as service to others
(true data monetisation)
Data
Capture

18. The Event Stream
Using The Event Stream
The event stream becomes a
‘database’ or immutable business
events
It is the critical channel for moving
events between micro services
It is the source of the data that feeds
data science and analytics
Key Questions To Ask
Deploy anywhere and replicate
between on-premise and cloud?
Are there management tools to
control and secure the
environment?
Is there a managed cloud service in
your preferred cloud provider?
A stream-processing solution that provides a high-throughput, low-latency,
distributed, replicated, highly available platform for handling real-time data
feeds with persistent immutable store.

19. Micro Services helps Data Governance
Data Security by creating multiple loosely coupled
layers each of which is protected in multiple ways
Data Privacy by only holding confidential data in a
small number of well protected places and
restricting access via APIs
Data Quality by tightly defining where and how
data is entered and prohibiting side routes to
updating that information
Compliance e.g. GDPR
by being able to define
where data is held, how
if flows and is retained
in the organisation and
the data models (as
defined by the APIs)
used

20. Micro Services help Agile Development
Self organising
teams take
ownership of well
defined products and
can held made
responsible for
delivery of working
software
Facilitates an
environment that
has short cycles and
sustainable
development and is
focused on technical
excellence
Makes change is
significantly easier, and
changes can be made
without having to co-
ordinate massive
releases across
a monolithic
Platform

21. Micro Services help Automation and Self Service
In Development
Smaller systems are easier to
build continuous development
(CD) and continuous
integration (CI) environments
that also allow continuous
Automated environment builds
allow developers to sand box
and experiment rapidly – fail
fast
In Production
Scale out solutions can be ramped up and down on demand,
reducing the operating costs and increasing platform availability
and resilience
In Analytics
Quicker access to data for the analytics and data science team
Multiple concurrent experiments to be carried out on isolated
platform capacity that doesn’t impact operational systems

22. Micro Services helps Data Science & Artificial Intelligence
Access to the business events as
they happen
Rather than using the data
artefacts after the event
Easier access to the right data
rather than exploring
everything
Facilitates deploying AI tools
into production
Reduces the scope of where
they have to be connected to
the platform
Allows multiple intelligent (AI)
services to be deployed quickly
and easily into the business
process

23. More
For Less
Cost
Reduction
Risk
Management
Better Analytics
& Reporting
Differentiated
Product
Micro Services Architecture
Data Science & AI
Data Governance
Automation & Self Service
Agile Development
Whole >
∑ Parts

24. David M. Walker
davidw@datamgmt.com
+44 7990 594 372
Scan Me

25. Amsterdam
6th November 2019
London
8th November 2019
Frankfurt
11th November 2019
David M. Walker
Thank You