Define enterprise integration strategy

1. 1
Define Enterprise Integration strategy
- Author
Bhawani Nandan Prasad
(bhawani_Nandan@yahoo.com)
- ProgramEngagement Management,
Enterprise Architect, Information architect, Data
Architect, BI Architect and Application Architect
• Master of Business Administration
• Diploma in Senior Management
• Bachelor of Engineering in Computer Science

2. 2
Integration strategy statement
Integration of systems and applications is core to
the success of technical interoperability. Encore’s
integration strategy is that developers use tools
and technologies that are “fit for purpose” i.e. are
applicable for the task at hand and are
documented within the Encore EA standards. All
solutions must use approved technologies with
established, repeatable patterns that enable the
Encore technical team to respond in a timely and
well understood fashion.

3. 3
EI Methodology
• Integration of systems and applications is core to the success of technical
interoperability. Our Approach – Define the current state for the tier 1
application. Identify the existing integration points, functional or non-
functional. Identify the current integration characteristic, is it event driven,
real time and synchronous, or is it delayed and in bulk mode. What could
be the functional state, real time, event driven, bulk or one record at a
time, back end data synchronization, exposed reusable services, etc.
• Deliverables: Current integration points, for each application potential
integration points or extensions.
• Steps/Plan
• Define the existing functional/non-functional integration points
• Define the current technical stack for the integration
• Define the end state for functional integration.

4. 4
EI Macro / Micro Level Design

5. Integration
Style/ Criteria
Batch File
Transfer
ETL
Direct Data
Access
Point-to-
Point
Messaging
(Event Driven
Architecture /
Pub-Sub)
Services/ ESB
Transport
Protocols
• FTP
• SFTP
• FTPS
Proprietary
ODBC/JDBC/
Proprietary
JMS /
Websphere MQ
JMS /
Websphere MQ
HTTP/ HTTPS /FTP
Business Logic None
Transforms/
Validates
Duplicated Distributed Distributed
Distributed and
Composable
Data Format
• Industry
• Vendor
• Own BU
• Industry
• Vendor
• Encore
• Stored Procedure
• DAL Pattern
• ORM Framework
• Industry
• Vendor
• Encore
• Industry
• Vendor
• Encore
Industry Standards
• SOAP
• XML
• JSON
Coupling Loose Loose Very Tight Loose Very loose Loose
Topology 1 to 1 Many to Many 1 to 1 Many to Many Many to Many 1 to Many
Vendor /
Product Lock in
Low High Medium Medium to High Medium to High Medium to High
Language /
Platform
Dependency
Low High Medium Medium to High Medium to High Low
5
Integration Method Comparison

6. Integration
Style/ Criteria
Batch File
Transfer
ETL
Direct
Data
Access
Point-to-
Point
Messaging
EDA Services/ ESB
Availability Medium High Medium High High Medium
Flexibility Low Medium Very Low Medium Very High Very High
Configurable QOS No Yes No Yes Yes Yes
Payload Size High High Medium Low Low Medium
Frequency Low Low High High Medium Medium to High
Maturity of Tech
and Tools (1-5)
5 5 5 5 2 3
TCO (maintenance,
risk)
High
High set up
Maintenance
lower
High Medium Medium Low to Medium
Complexity Low Medium
Low to
Medium
Medium To
High
High Medium to High
Assured Delivery/
Reliability
Low High High
Medium To
High
Medium To High Medium to High
6
Integration Method Comparison
(cont.)

7. Integration
Style/
Criteria
Batch File
Transfer
ETL
Direct Data
Access
Point-to-Point
Messaging
EDA Services/ ESB
Capabilities • Datagram
• Datagram
• Fan-In
• Fan-Out
• Queue based
• Data Integration
(ETL) Solution
• Datagram
• Fan-In
• Fan-Out
• JMS
Publish/Subscribe
• Topic based
• Subscriber/Recipi
ent List
• Request/Reply
• Queue based
• Datagram
• Fan-In
• Fan-Out
• JMS
Publish/Subscribe
• Topic based
• Subscriber/Recipie
nt List
• Request/Reply
• Queue based
• Datagram
• Fan-In
• Fan-Out
• JMS Publish/Subscribe
• Topic based
• Subscriber/Recipient
List
• Request/Reply
• Queue based
Tools
Generic
• JScape
• MoveIt
Platform Specific:
• CD Connect
• Transit
• ETL
Decomissioned:
• FileZilla
• CoreLite
• SSH
• Shared Drives
Generic:
• DataStage
• CastIron
Platform Specific:
• DataLoader
(Salesforce)
• SSIS (MSSQL)
• Change Data
Capture
(MSSQL)
Java:
• Websphere
Microsoft:
• IIS
• LinkServer
(MSSQL)
• Websphere MQ • Websphere MQ
ESB:
• Websphere MQ
• CastIron
• DataPower
Services:
• Websphere WS (Java)
• WCF/ IIS (Microsoft)
7
Integration Method Comparison

8. Integration Styles
8
Appendix A

9. File extract produced by Application A extract program.
Transferred physically (media) or via FTP. File read by
Application B load program
Custom code usually written in each application to produce
and process extract files.
9
App A App B
Batch File Transfers (FTP)

10. ETL Software provides set of tools and infrastructure for
extracting data from source system databases or files, apply
transformations and loads resulting data into target system.
Often the extracts are “intrusive” database reads, but may
also use extract files generated by the Application
10
App A App B
ETL
Extract, Transform, Load (ETL)

11. Direct access/update by
Application A to Application B
data
• Usually uses ODBC/JDBC
• Data “owned” by one of the
applications
Shared (Master) Databases
• Multiple applications
• Update a shared database
11
Application
B Data
Application B
Application A
SQL / ODBC /
JDBC etc
Shared
Data
Application BApplication A
SQL/ODBC/
JDBC etc
SQL / ODBC /
JDBC etc
Local Admin
Direct Data Access

12. App A App B
RMI / IIOP
12
Language specific API exposed by Application A called synchronously and directly by
other applications, e.g. RMI
Calls are usually “fine-grained” method level, e.g. retrieving a single data item
Remote Procedure Call (RPC)

13. App A App B
Queue Management Software
13
Point to Point Messaging Integration
– Asynchronous message queuing between applications
– Provides message reliability / guaranteed delivery
– Both endpoints have embedded queuing software
Point to Point Messaging

14. Application A Application B
Application C Application D
Integration
Broker
14
Messaging Hub or hubs with routing and transformation
capability. Usually uses underlying message queuing
software for transport. Often includes pre-built ‘adapters’
to vendor business applications
EAI (Integration Brokers)

15. Application A
Application B
Application C
Application D
Pub-Sub
Manager
Topic A
Topic B
Topic C
15
“Event” published by an application on a channel or “topic” to a
“subscription manager” which distributes the event information
to all of the current open subscribers. Publishers and subscribers
often are not aware of each others identity. Often uses
underlying queue management software for transport
EDA (Event Driven Architecture /
Pub-Sub)

16. Application A
Application B
Application C
Application D
Look up
ESB
Services
Registry
Look up
Orchestration
Engine
Security
Web
Services
Mgmt
16
Service interface exposed by business applications available for use/reuse over
standard protocols.
ESBs provide routing, transformation and other services. Services Registry provides
service look up and location transparency. Standardized security, systems
management etc. Orchestration
SOA (Services / ESBs)

17. Integration Pattern Topics
Review Integration Patterns
• Datagram
• Fan-In
• Fan-Out
• JMS Publish/Subscribe
– Topic based
– Subscriber/Recipient List
• Request/Reply
– Queue based
– RPC
• Data Integration (ETL) Solution
17

18. Datagram Integration Pattern
Requester Source
Integration
Layer
18
Considerations
• When send and forget (asynchronous) integration is sufficient
• Simple pass-thru integration
• Loosely coupled end points
• Limited reusability
• Error handling procedure
• Data retention

19. Fan-In Integration Pattern
Integration
Layer
Source
System
Target
System
Source
System
Source
System
19
Considerations
• Multiple source systems
• Interoperability between heterogeneous systems
• Loosely coupled end points
• Increases reusability
• Central control and management for mediation and transport
• Supports content based routing
• Latency differs from each source systems

20. Fan-Out Integration Pattern
Integration
Layer
Source
System
Target
System
Target
System
Target
System
20
Considerations
• Multiple target systems
• Interoperability between heterogeneous systems
• Loosely coupled end points
• Increases reusability
• Central control and management for mediation and transport
• Supports content based routing

21. Publish/Subscribe – Topic based
Pattern
Integration Layer
Publisher Subscriber
Subscriber
Subscriber
Topic
Publish Subscribe
Subscribe
Subscribe
Acknowledge
Acknowledge
Acknowledge
21
Considerations
• The most appropriate distributed architecture
• Interoperability between heterogeneous systems
• Increases scalability
• Consumption of data flexibility

22. Publish/Subscribe – Subscriber
List Pattern
Integration Layer
Publisher
Recipient
List
Publish
Subscriber
Subscriber
Subscribe
22
Considerations
• The most appropriate distributed architecture
• Interoperability between heterogeneous systems
• Requires routing to a dynamically specified recipients
• Consumption of data flexibility

23. Request / Reply (Queue) Pattern
Integration
Layer
Requester Provider
RequestRequest
Response Response
23
Considerations
• When applications require two-way (synchronous) conversations
• Increases maintenance and support (requires 2 queues per end point)
• Change end points without impacting existing services

24. Request / Reply (RPC) Pattern
Integration
Layer
Requester
RequestRequest
Response Response
Provider
24
Considerations
• Interoperability between heterogeneous systems
• When synchronous integration is required
• Change end points without impacting existing services

25. Data Integration (ETL) Solution
Extract Transform Cleanse Load
Legacy
Systems
Packaged
Application
Other
Internal
Application
ETL Solution
Transient Data
Source
Manhattan
WMS
DataMart
25
Considerations
• When large data integration is required
• Integration occurs at the database and data source level
• A proven industry standard for data integration
• Flexibility of input/output data formats

26. Conceptual Model View
26
Client Access Tier
Presentation Tier
Application Service Tier
Integration Tier
Data Tier
PlatformLayer
Web Browser Client Server Other
Web Server Terminal Server Mail Server Voice Response
Data Warehouse
Application API
Database
Middleware
Message Oriented
Middleware
ETL SOAP
Application Server Package Report ServerPortal Server
HR, Finance,
Planning, Logistics,
Etc.
LDAP
The diagram represents the various
tiers of service based on the N-tier
application.
Client Access Tier: Represents the
type of end-user access.
Presentation Tier: Represents the
client touch points and channels of
services.
Application Service Tier: Represents
that application and business logic to
support the functions and services.
Integration Tier: Represents the touch
points between applications and
systems.
Data Tier: Represents the storage of
information that are needed by
applications.

27. High-Level Application Integration Model
Application ABC
Custom Code Based
Batch
Application XYZ
EAI HUB (SAP XI)
ETL (Ascential)
Fast Asynchronous
Messaging
ERP
Adapter Adapter
Module
Module
Module
Module
Adapter AdapterShort Batch Messaging
Queue Queue
Queue
ApplicationLogic
ETL Based Batch
Staging
Simple DBMS to DBMS Replication
ERP
API
1
2
3
4
1. Pull data from source system
2. Stage data in a persistent store
3. Process staged data and transform
4. Load data into target system
Point to Point (API) Integration
FILE
FILE
FTP
Application Server
Synchronous SOAP
UDDI
Application Logic
27
Legend
Fast Async
Short Batch
Custom Batch
ETL Batch
DB Sync
P2P
FTP
Near Real-Time Interface - CORE
Message based Store & Forward Short Batch - CORE
Custom Developed Batch - CORE
ETL Based Store & Forward Batch - CORE
DBMS to DBMS Replication - SPECIALIZED
Synchronous Point to Point - DECLINING
SOAP
File Transfer Service - CORE
Synchronous SOAP - CORE