2. Who am I?
• Senior Integration Specialist, Deloitte
• Microsoft Azure MVP
• MCSE, MCT, MCPD, MCTS BizTalk &
Azure
• Pluralsight Author
• www.mindovermessaging.com
• @daniel2me
3. 3
"APIS MAKE DIGITAL SOCIETY
AND DIGITAL BUSINESS WORK;
THEY ARE THE BASIS OF EVERY
DIGITAL STRATEGY."
"The 10 Things CIOs Need to Know About APIs and the API Economy" –
Gartner (G00318859, Jan 2017)
Image courtesy of https://www.instarem.com/blog/the-api-economy-in-2019-hype-or-the-engine-that-fuels-business-growth/
4. "DATA IS THE LIFEBLOOD OF YOUR BUSINESS"
4
"Data is the Lifeblood of Your Business – That's Why You Need a Data Strategy at its Heart" – The Drum (Feb 2020)
5. FORBES: MAKING THE MOST OF YOUR DATA
Leave your data
where it is.
Make sure you can
handle all data types.
Harness data to drive
automation.
Democratize data. Protect your data
5
"Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
6. FORBES: MAKING THE MOST OF YOUR DATA
Leave your data
where it is.
Make sure you can
handle all data types.
Harness data to drive
automation.
Democratize data. Protect your data
6
"Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
7. FORBES: MAKING THE MOST OF YOUR DATA
Leave your data
where it is.
Make sure you can
handle all data types.
Harness data to drive
automation.
Democratize data. Protect your data
7
"Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
8. FORBES: MAKING THE MOST OF YOUR DATA
8
"Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
Leave your data
where it is.
Make sure you can
handle all data types.
Harness data to drive
automation.
Democratize data. Protect your data
9. FORBES: MAKING THE MOST OF YOUR DATA
9
"Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
Leave your data
where it is.
Make sure you can
handle all data types.
Harness data to drive
automation.
Democratize data. Protect your data
10. FORBES: MAKING THE MOST OF YOUR DATA
10
"Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
Leave your data
where it is.
Make sure you can
handle all data types.
Harness data to drive
automation.
Democratize data. Protect your data
12. 12
"How to Justify Strategic Investments in Integration Technology" – Gartner (G00385596, May 2019 )
13. 13
Data
Integration
APIs
BRINGING IT ALL TOGETHER:
Integration provides the link, but…
• APIs are still coupled to the data
sources
• Availability of backend systems
is a dependency
• Complexity & cost of APIs
designed to aggregate data
sources
14. 14
Data
Integration
APIs
BRINGING IT ALL TOGETHER:
DIGITAL INTEGRATION HUB
"An advanced application
architecture that
aggregates multiple back-
end system of record data
sources into a low-latency
and scale-out, high-
performance data store."
"Turbocharge Your API Platform with a Digital Integration Hub – Gartner (G00360082, Jul 2018)
DIH
18. • Consolidates and aggregates data from
multiple sources
• Transforms the data into non-proprietary
semantics, using entity views
• Allows for the access and manipulation of data
without impacting the core business systems
• Provides advanced search capabilities
• Potential components:
• Data warehouse
• Data lake
• Master Data Management (MDM)
• Analytical systems & AI
The heart of the DIH
HIGH PERFORMANCE DATA STORE
18
19. • Captures data at the source (preferably as it
happens)
• Accommodates multiple integration patterns
& styles
• Event brokering / messaging
• Extract Transform & Load (ETL)
• Change Data Capture (CDC)
• Integration Patterns (ESB, iPaaS)
• Stream processing (Spark, Flint)
Gathering your data
EVENT-BASED INTEGRATION LAYER
19
20. • Microservices access the data at the
entity level
• Can be hierarchical and inter-connected
• Generally do not access the Systems of
Record directly
• API Gateway provides management and
access layer:
• Discovery
• Security
• Abstraction
• Monetization
• Analytics
Surfacing/exposing your data
FRONT-END API SERVICES
20
21. BENEFITS
• Responsive user experience
• "Defending" systems of record
• 24/7 support
• Decoupling the front-end layer from the
system of record applications
• Supporting legacy systems replacement
• Normalising the APIs for a certain application
domain
• Providing real-time business insight
22. CHALLENGES
• Complexity of rolling out a high-performance data management
technology (e.g. NoSQL DBMS or in-memory data grid)
• Supporting bidirectional, event-driven synchronization between
the high-performance data store and system or record
applications
• Designing a canonical data model for the DIH business entities
that supports multiple channels
• Implementing appropriate metadata management to support
discovery and introspection of data entities and relationships
represented across multiple data sources
• Designing, building, and managing the complex distributed
architecture of a Digital Integration Hub
22
23. RECOMMENDATIONS
23
Determine if your organisation needs a DIH and
if it has the skills to support it
Know your data and consumer requirements
Understand the integration patterns that will be
required
Design your APIs to be abstracted from the
underlying systems ("API First" approach)
Consider using a technology partner for
implementation
25. AZURE DIH ACCELERATOR
• Template on GitHub
• Jump starts by setting up the engineering system for your application
• Allows you to concentrate on the business logic
• Consists of:
• A pre-configured development environment
• An application
• Build and Deployment automation
25
https://github.com/Azure-Samples/digital-integration-hub
28. GOALS:
• Establish an integration platform
• Integrate core clinical systems
using HL7 while enabling
integration of modern digital
consumers over FHIR APIs
• Enable the interoperability of
internal and external client
systems hosted in a fully
controlled on-premises
infrastructure or in a secure
public cloud
Pathology Services Provider
CASE STUDY
28
https://customers.microsoft.com/en-us/story/1384019799287777382-
nsw-government-health-provider-azure
29.
30. SUMMARY
• API-based access to disparate data services is
costly from a performance and maintenance
perspective
• A DIH enables enhanced performance in
accessing your organisational data, while
ensuring effective protection of your backend
systems
• A DIH architecture also provides increased
scalability, greater flexibility, and better insights
31. REFERENCES
"Turbocharge Your API Platform with a Digital Integration Hub – Gartner (G00360082, Jul 2018)
"The 10 Things CIOs Need to Know About APIs and the API Economy" – Gartner (G00318859, Jan 2017)
"Data is the Lifeblood of Your Business – That's Why You Need a Data Strategy at its Heart" – The Drum (Feb 2020)
"Data Hubs, Data Lakes & Data Warehouses: How They Are Different and Why They Are Better Together" – Gartner
(G00465401, Feb 2020)
"Five Ways to Make the Most of Your Data" – Forbes (Jul 2021)
"How to Justify Strategic Investments in Integration Technology" – Gartner (G00385596, May 2019 )
"Infuse your data hub strategy with Data & Application Integration" – Gartner (G00343327, Dec 2017)
31
APIs are can solve many problems:
Enable systems & applications to interact
Disparate data sources & access protocols
Disparate consumer channels
But… there be challenges:
Security
Performance (consumer & system impact)
Avoiding numerous point-to-point solutions
APIs only as good as the data they can access
Large organisations rely on multiple systems of record
Information needs to be shared on multiple channels
Other internal systems
Partner systems
Client systems
Strategy is important
Connect rather than collect – for real-time insights
structured, unstructured, relational, and geospatial, etc.
Make sure you can analyze and knit together these different types into a unified whole.
modern platform treats data as multimodal and multidimensional.
use it to drive automation and machine-augmented decision making.
Invoice reconciliation, handling basic customer complaints
empower people with self-service analytics tools
take into account any areas of exposure or vulnerability and build in the right security, access controls and governance to protect your data
What does Gartner have to say about integration?
Let's break down the components
High cost: huge number of API calls potentially imply a massive, often low-value workload which ultimately hits the system of record applications and the integration layer. ■
Complexity: The integration macroservices could be extremely complex to develop, deploy, run and manage, especially if the back-end data is highly fragmented across multiple applications and data sources.
Availability: Dependency on backend systems being available
Tight coupling: Changes in the back-end systems imply re-engineering of the integration macroservices or their complete redesign, for example, if a back-end system is replaced with a different solution.
"By storing an aggregated replica of the system of record data needed by the channel applications, the DIH protects the former from excessive workloads while optimizing the data access latency and responsiveness for the latter."
Metadata management – enables metadata-driven development & introspection, e.g to support domain-driven design
Discover, capture, & synchronise metadata models
Analytics – technically not part of the DIH, but often a driver for its implementation
ETL – preferred for initial loads or periodic refreshes
Some exceptions to direct access:
CQRS (Command Query Responsibility Segregation) pattern
Updates to SoR
Would typically use data virtualization, federated views
Responsive user experience: while enabling users access to a consolidated, yet real-time, view of data scattered across multiple system of record back-ends
"Defending" systems of record: from the potentially excessive, often low value, workloads generated by the channels.
24/7 support: Always-on access to the front-end API services, even in situations where the
back-end systems must be put offline for maintenance, upgrade or other reasons. ■
Decoupling the front-end layer from the system or record applications:
Supporting legacy system of record applications modernization:
Normalizing the APIs for a certain application domain: So that with a single set of APIs the channel applications can access data held in multiple systems of record
Providing real-time business insight: Gathering data on user behavior or offering them additional services (including search, predictive analytics
DRIVER: low-latency, real-time, consolidated and accurate access to data is critical for business success.
This repository contains a template of an application built for the Azure Application Platform. This template is built to make it easy to dive straight into implementing business logic without having to spend time on setting up an engineering system for your application. The templates give you a starting point, while providing the option to change and extend any of the pre-configured components to suit your needs.
The front end processing layer are implemented by Azure API Management for providing the discoverability and gateways for all your APIs. The microservices implementation leverage serverless functions which provide a simple CRUD based API.
The data layer is implemented using PostgresDB
For Analytics and getting a single pane of glass view over your disparate data sources this is where you could optionally add on Azure Synapse analytics to extend this solution. This is not included in the sample.
The Integration layer in this case uses Azure Event Grid for the event driven programming model to ensure the system responds to all API events in real time and Logic Apps to react upon these events and process the incoming data. The implementation of the logic app is left blank, leaving you to leverage the 450+ connector ecosystem to implement the syncing functionality between the data layer and your backend systems of record you choose to integration the Digital Integration Hub with.
and Monitoring provided by Application Insights and Log Analytics features of Azure Monitor.
The application is comprised of an Azure Function API implemented written in Typescript, a PostgreSQL 'items' database and an Integration layer implemented by Logic Apps and Event Grid. It uses an Object-Relational Mapper (Sequelize) and implements a single object, items, to get you started.