Integration is as relevant as ever. Change seems to be omnipresent and software architectures are constantly evolving together with technologies, concepts and tooling. As a result of companies leveraging available Cloud technologies we are now dealing with elastic infrastructures and hybrid application ecosystems, which means that on-Premises systems and Cloud services need to be integrated, giving way to added complexity. Moreover, to unlock data and functionality in a scalable and secure manner, integration strategies must be consistent with an API-first design. When it comes to concerns such as decoupling, performance or availability, it must be possible to scale distinct components within the integration infrastructure independently. Ideally, these components should be able to deploy and run isolated from each other to ensure the system’s resiliency and flexibility, since the components are not influencing or dependent on each other.
This presentation will take you on a little time journey while discussing the role and relevancy of integration architectures, key patterns and their evolution (Publish/Subscribe, Event Sourcing, CQRS) as well as the way how those should be implemented today. Are we still talking about monolithic integration platforms or is there a new kid on the block, more lightweight, following a Microservices architectural style?
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
From EAI to Serverless
1. From EAI to Serverless
https://www.redexpertalliance.com
From EAI to Serverless
Evolution of Integration patterns and architectures
Digital Exchange 2019, Gummersbach 2019/05/25
2. From EAI to Serverless
https://www.redexpertalliance.com
Red Expert Alliance - Who we are
2
3. From EAI to Serverless
https://www.redexpertalliance.com
Who are we?
3
Lucas Jellema (AMIS, NL)
Arturo Viveros (Sysco, NO)
Sven Bernhardt (OPITZ, DE)
4. From EAI to Serverless
https://www.redexpertalliance.com
Agenda
1. What is integration?
2. Integration challenges
3. “Traditional” integration approach
4. Recents insights and developments
5. Integration Patterns
6. Out of the Box
7. Conclusion
4
5. From EAI to Serverless
https://www.redexpertalliance.com
What is integration?
5
6. From EAI to Serverless
https://www.redexpertalliance.com
What is integration?
System
A
System
B
XX
Simple definition: enabling systems to talk to each other
6
7. From EAI to Serverless
https://www.redexpertalliance.com
What is integration?
System
A
System
B
Ensure that data is where it is needed when it is needed in the form in which it is needed
7
8. From EAI to Serverless
https://www.redexpertalliance.com
Integration challenges
8
9. From EAI to Serverless
https://www.redexpertalliance.com
Integration Challenges
● Monolithic / aging systems
● Disparate data models
● Architectural complexity
● Heterogeneous ecosystem
● Technical debt
● Bloated projects / overhead
● Visibility / error handling
● Production deployment
9
10. From EAI to Serverless
https://www.redexpertalliance.com
IT organizations are drowning
10
11. From EAI to Serverless
https://www.redexpertalliance.com
“Traditional” integration approach
11
12. From EAI to Serverless
https://www.redexpertalliance.com
12
13. From EAI to Serverless
https://www.redexpertalliance.com
ESB as basic architecture element
13
global internal specific
External Interfaces – Proxy Servics
Integration Logic
Internal Interfaces – Business Interfaces
14. From EAI to Serverless
https://www.redexpertalliance.com
Limited and expensive Scalability
Loadbalancer
ESB 1 ESB 2 ESB 3
14
15. From EAI to Serverless
https://www.redexpertalliance.com
Caught in a technology Trap
Oracle SOA Cloud
Service
Fixed Technologies Set
<XML />
{
„dialect“:“JSON“
}
15
17. From EAI to Serverless
https://www.redexpertalliance.com
Recent insights and developments
17
18. From EAI to Serverless
https://www.redexpertalliance.com
New insights, new context
18
increased scale – because of real -time and web scale demands
introduced by IoT and a further growth of the use of Apps
Need for speed - real time insights from fast data streams
No/less strict delineation between Enterprise (message based)
Integration and ETL/Data Integration
tighter security requirements,
faster change cycle (Agile, DevOps),
24/7 Operations
Scarcity of talent/developers
Is a canonical model/language such a good idea? What about
‘domain/context mapping’ and ‘anti corruption layer’ from
domain driven design
We have learned to ask: How fresh should data be? How crucial is
ACID and instant consistency?
increased usage of standard applications (primarily SaaS)
the advent of cloud (necessity to support hybrid scenarios - cross on
premises and cloud as well as multi vendor cloud,
new architecture patterns such as domain driven design,
microservices, CQRS, event sourcing, “dumb pipes, smart end points”
advances in technology, containers, serverless, REST, API
technology – JSON (vs XML), REST (vs SOAP), containers,
serverless, “embedded HTTP server”, industry standards like Kafka
TCO of Middleware Platforms
Strategic roadmap of middleware vendors
19. From EAI to Serverless
https://www.redexpertalliance.com
The world is getting more complex!
19
20. From EAI to Serverless
https://www.redexpertalliance.com
Shift to hybrid integration solutions
20
21. From EAI to Serverless
https://www.redexpertalliance.com
Integration is not just ESB anymore
Modern Integration platforms consists of diverse components
● Integration (µ)Services
● API Management
● Event Hub
21
https://www.linkedin.com/pulse/when-creating-apis-focus-your-api-gateway-what-does-best-wunderlich
22. From EAI to Serverless
https://www.redexpertalliance.com
Integration, Big Data and Analytics
22
Analytics App 1
Analytics App 2
Analytics App 3
Analytics App 4
Landing
Zone
Data Pipeline /
Data Ingestion
Metadata
Store
Content
Store
Rawdata
Store
Big Data Platform
DataLake
Integration
Services
APIs
23. From EAI to Serverless
https://www.redexpertalliance.com
Integration pattern
23
24. From EAI to Serverless
https://www.redexpertalliance.com
Two dimensions of integration
24
25. From EAI to Serverless
https://www.redexpertalliance.com
● End User ”integration”
● API Gateway as runtime component for
○ API policy enforcement
○ Request routing
● (Near) Real-time, synchronous interactions
● APIs born from business requirements
○ Who is using the API?
○ How is the API being used?
● Use of lightweight, standard protocols
Vertical integration
25https://microservices.io/patterns/apigateway.html
API Gateway
Backend for Frontend
26. From EAI to Serverless
https://www.redexpertalliance.com
API first design approach
26
27. From EAI to Serverless
https://www.redexpertalliance.com
Demo Use Case
27
Customer API
CRM Database
Client
Application● Integration of a DB-based CRM system
● Client application uses API to
○ Find/Read Customer data
○ Create Customer data
○ Update Customer data
○ Delete Customer data
● API ensures secured access of Customer
data
● Integration Service takes care of providing
the data in the correct format according to
the API description
VerticalIntegration
28. From EAI to Serverless
https://www.redexpertalliance.com
Evolution of ‘vertical integration flow’ implementation
28
29. From EAI to Serverless
https://www.redexpertalliance.com
Horizontal integration
29
● Classic System2System integration
● Heavy-weight, proprietary protocols
● Technical interfaces
● Batch processing
● Asynchronous interaction patterns
● New insights: relaxed freshness and consistency demands, data replication
● New context: cloud, hybrid, “dumb pipes, smart endpoints”, Standard Apps & SaaS evolution
31. From EAI to Serverless
https://www.redexpertalliance.com
(Cloud) Adapters & iPaaS platforms
31
32. From EAI to Serverless
https://www.redexpertalliance.com
Challenges with Horizontal Integration
● Handle requirements for speed, consistency, volume & [global] distribution
● Detect Change
● Extract Data
● Publish Change
● Transport
○ deal with limited connectivity, temporary unavailability
● Convert/Transform
● Apply
● Monitor
● Handle exceptions
● Enforce constraints & authorization
32
33. From EAI to Serverless
https://www.redexpertalliance.com
Event-Driven Integration
33
“Life is a Stream of Events”
https://www.slideshare.net/ConfluentInc/life-is-a-stream-of-events
34. From EAI to Serverless
https://www.redexpertalliance.com
Event-Driven Integration
34
● Systems act and react to
internal & external events
● Streams of events represent
information and behavior
● Data should end up in the right
places… eventually
● Popular design patterns:
○ Event Sourcing
○ CQRS
○ Materialized view
35. From EAI to Serverless
https://www.redexpertalliance.com
Event-Driven Integration
35
36. From EAI to Serverless
https://www.redexpertalliance.com
Event-Driven Integration
36
37. From EAI to Serverless
https://www.redexpertalliance.com
“Serverless functions provide a synergistic relationship with event streaming applications; they
behave differently with respect to streaming workloads but are both event driven”
Event-Driven Serverless Architectures
https://confluent.i
o
38. From EAI to Serverless
https://www.redexpertalliance.com
Out of the box
Ahead of Time Integration
Collective Data Set
Event Sourcing
38
39. From EAI to Serverless
https://www.redexpertalliance.com
JIT vs AOT
39
40. From EAI to Serverless
https://www.redexpertalliance.com
JIT vs AOT Data Integration
● JIT
○ Process [synchronous] request for data ‘just in in time’
○ Performance overhead suffered at (every) request time
○ Only data that is requested
○ Fresh as can be
○ All authorization rules applied
○ Single Source of Truth
○ Transformation & Aggregation at request time
○ Hard (runtime) dependencies from consumer on producer
● AOT
○ Pre-process / pre-share data ‘ahead of time’
○ Potentially includes data that will never be requested
○ Data can be outdated
40
41. From EAI to Serverless
https://www.redexpertalliance.com
CQRS => Specialized Query Stores
Cater for different (read) Data Usage Scenarios
Different in:
● Why
● What – grain, aggregation, breadth, type of and variation in queries
● How – format, language, accuracy and freshness, consistency
● When – frequency, time of day | week | month, 24/7
● Where – latency, band width, off line
● Who – scale, different roles & user groups
41
42. From EAI to Serverless
https://www.redexpertalliance.com
CQRS => Optimized Data Set for Special Use Case
● Shape & Format
● Right Grained (aggregation, consolidation)
● Filtered
● Enriched (from multiple sources)
● Location
● Time (refreshed at the right time)
● Search Options
● BASE (eventually – when? – consistent)
● Authorization
● GDPR
42
C Q
BFF API
Frontend
43. From EAI to Serverless
https://www.redexpertalliance.com
Collective Data Set
43
44. From EAI to Serverless
https://www.redexpertalliance.com
Data Sharing Ecosystems
● Multiple independent
organizations
● Mutually benefit from
having access to the same
Data at the same time
● Share data & Have access
to data with minimum of overhead and with equality among the ecosystem partners
● Partners are on different network domains, physical locations, technology stacks, …
● New Partners can join
● Partners can leave
● Consensus on what constitutes correct data
44
45. From EAI to Serverless
https://www.redexpertalliance.com
Collective Data Set
45
46. From EAI to Serverless
https://www.redexpertalliance.com
Collective Data Set
46
47. From EAI to Serverless
https://www.redexpertalliance.com
Collective Data Set == Collective Data Change Events
47
48. From EAI to Serverless
https://www.redexpertalliance.com
Collective Data Set => Chained Distributed Data Store
48
49. From EAI to Serverless
https://www.redexpertalliance.com
Conclusion
49
50. From EAI to Serverless
https://www.redexpertalliance.com
Key Takeaways
50
51. From EAI to Serverless
https://www.redexpertalliance.com
Questions & Answers
51
Thanks for your attention!