Effective Application Development in IBM Integration Bus Presented at InterConnect 2016 conference

1. Effective Application Development
in IBM Integration Bus
Andrew Coleman
Architect, IBM Integration Bus on Cloud
Chair, W3C XML Query Working Group

2. Please Note:
2
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal
without notice at IBM’s sole discretion.
• Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment,
promise, or legal obligation to deliver any material, code or functionality. Information about
potential future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our
products remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in a
controlled environment. The actual throughput or performance that any user will experience will
vary depending upon many factors, including considerations such as the amount of
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and
the workload processed. Therefore, no assurance can be given that an individual user will
achieve results similar to those stated here.

3. Effective Application Development
• What should you consider when building connectivity solutions?
– What technology is provided to help you?
– What design techniques should you know about?
– What should you try to avoid?
• Main points
– Design with IIB features
– Design for performance and scalability
– Design for administration
– Design with interfaces
– Design consistently
• Test

4. 1. Design with IIB facilities

5. Tools of the Trade!

6. • Built-in nodes encapsulate transports, technologies and applications
– Our intent is always to make the common tasks easy, and the rest possible!
– Use the built-in nodes to reduce the amount of custom code required
– Makes best use of built-in facilities like activity trace and resource statistics
Create Your Connectivity Solution

7. • Integration Bus can interact with a huge range of environments
• Transports and protocols
– MQ / SOAP / HTTP / TCPIP / JMS
• Data repositories
– File / Database / IMS
• Applications
– SAP / Siebel / PeopleSoft / JDEdwards
– CICS / IMS
– .Net
• APIs
– SCA
– CORBA
• All input nodes can be configured to work with any message format.
Getting data in and out of IIB

8. • WMB and Integration Bus have several transformation options:
– Mapping
– XSLT
– ESQL
– Java
– .NET
• Reflects the importance of transformation in connectivity solutions
– User-defined nodes supported for Java and C/C++
• Every transformation option has strengths and weaknesses!
– Performance and scalability
– Backend integration
– Skill sets and learning curve
– Developer usability
– Portability and maintenance
• Use a transformation technology appropriate to the problem at hand!
Transformation Options

9. Subflows
• Subflows are simply message flows that are invoked from another flow
– Input and output nodes in the subflow become terminals in the main flow
– Use subflows to break up large problems into smaller more manageable chunks
• Subflows are directly deployable to the runtime
– Shared subflows deployed just once per execution group (or application)
– No need to redeploy message flows after changes to shared routines are made
– Redeployment of a subflow is automatically picked up by any consumers

10. • Models are needed for parsing, validation and transformation
– Models avoid the need to write custom code to parse messages!
Message Modelling
• Graphical mapper requires models to display the message structure
– ESQL editor provides in line validation of code that navigates message trees

11. Message Modelling
• XML documents and messages are modelled using XML schemas
– XSDs and WSDLs are deployed directly to the runtime – no import required!
• Non XML data (both text and binary) is defined using DFDL schemas
– Data Format Description Language standardised through the Open Grid Forum
– DFDL schemas replace message set projects and the MRM message domain
– Input nodes support DFDL alongside XMLNSC, JSON, MRM etc
• New model based tooling to design and test DFDL schema models
– Test parsing and test data generation built into editor!
– As with XSDs, DFDL schemas are also deployed directly to the runtime

13. 2. Design for Performance
and Scalability

14. Design for Performance
• There are several areas that incur a processing cost
– Parsing and serialization of messages as they enter and exit the flow
– Integration logic such as transformations executed during message processing
– Transformation cost includes general complexity, tree navigation and tree copying
– External resources such as interactions with databases and queue managers
• Message flow design is critical to achieve high performance and scalability
– For example using global units of work to achieve transactionality across a flow
• Performance is a large (and important) topic in its own right
– There is a wealth of information and guidance available online!

15. Design for Performance
• Loops in a message flow are exactly like a recursive function call
– So each iteration of the loop consumes some stack space
• Avoid loops unless you are certain that the number of iterations will not
become large
– Stack overflows will bring down the execution group along with all running
flows!
– There is usually an alternative design that avoids looping

16. Design for Scalability
• Large messages can consume large amounts of memory/CPU
• There is an industry trend toward larger data sets
– Already apparent in IIB deployments
• A flow with a large memory overhead can
– slow down other flows by causing swapping
– in the worst case, bring down the execution group
• Recommendations
– Consider whether a message 10 times the current size would be able to be
handled without a problem
– Where appropriate, use large message handling techniques in your flows
( see Large Messaging sample )

17. 3. Design for Administration

18. Applications, Libraries, Services and REST APIs
• Applications package end-to-end connectivity solutions
– The concept of an application is shared between the toolkit and runtime
– Applications are deployed and managed as a single unit of isolation
• Libraries package resources for reuse (flows, scripts, models)
• Resources in an application are not visible to anything else
– Use applications to manage your solutions inside an execution group
• A Service is an Application with a well defined interface (WSDL)
• A REST API is an Application built from a Swagger definition

19. Shared Libraries
• New shared container for reusable artefacts
– Subflows, message models (XSD, DFDL)
– ESQL, maps
– NOT flows!
• Saves memory
– Multiple applications can reference a single copy of a shared lib
• Separately deployable from the application
• Shared libraries have no running state
– Cannot be started or stopped
– No runtime threads assigned
• v8/9 static libraries continue to work as before
V10

20. Static Libraries verses Shared Libraries
• Static Libraries • Shared Libraries (v10)
App1
Lib1 Lib2
App2
Lib1 Lib2
App3
Lib1 Lib2
App1
App2ShLib1
ShLib2
App3

21. Configurable Services
• Use configurable services to externalize deployment time properties
– Allows deployment information to be configured by a system administrator
– For example, credentials for the database nodes and FTP server and port
– Ideal for moving solutions through staging environments (test to production)

22. Monitoring
• Monitoring tools available via Web UI and Integration Explorer
– Statistics monitor resource usage across execution groups
– Range of options to update and visualise the resource statistics
– Activity Logging to focus on resource usage

23. Monitoring
• Monitoring also provided on individual message flows
– Quick and easy way to identify hot spots in your solutions!

24. 4. Design with Interfaces

25. Web Services
• Drag and drop WSDL to generate integration service
– Generates a sub-flow for each SOAP operation
– Framework manages invocation of sub-flows and error handlers

26. REST APIs
• New container type for implementing REST APIs
• Generates framework based on Swagger definition
V10

27. 5. Design consistently

28. Patterns for Simplified Development
• Creates top-down, parameterized
connectivity solutions
• Reduces common problems in flow
development
• Establishes best practices for integration
• Reduces time-to-value for solution
development
• Patterns are easily extended with regular
IIB functionality

29. Pattern Generation
• Pattern generation creates the production ready projects
– Generated projects reflect the configuration choices of the pattern user
– Configuration is saved so that the pattern can be re-generated if required

30. Built-In Patterns
• Integration Bus provides a core set of built-in patterns
• These implement a variety of common scenarios
– Web service front end to a MQ based application
– Processing data stored in a file and routing to one or more queues
– Adding a proxy in front of a web service provider
– Processing data from an SAP system and routing to MQ
– Shredding messages and routing to one or more queues
• Patterns are selected based on client feedback and field experience
• This core set of patterns continues to grow with each release

31. Pattern Authoring
• Pattern becomes even more useful when you can create your own!
– Every organization has their own repeating connectivity patterns!
– Pattern authoring is the name we give to this technology in WMB and
Integration Bus
• We recommend you start with a working solution
– One or more projects
• Pattern authoring is a design activity
– It may be long lived
– It is often not sequential
• Using patterns is a top-down activity driven by a requirement, but:
– Authoring a working solution is (typically) a bottom-up activity
– So pattern authoring bridges these two different approaches
• Patterns have their own development cycle
– Pattern Authoring editor supports this design activity

32. Create Your Working Solution
• No change at all - design your solutions as you do today
– Pattern authoring does not change the tools you use to create solutions
– The key to a good pattern is to create a good working solution!

33. Design Your Pattern
• Straightforward to create patterns using the Pattern Authoring editor!
– Design the user interface which is presented to your pattern users
– Easy to add branding - style sheets, images and other files as required

34. Pattern Refinement
• Pattern authoring supports property changes
– Node, user-defined properties (UDPs) and promoted node properties
• Property variability is the most common type of variability that a pattern
might need to express - there are many others:
– Generate application text files such as ESQL scripts
– Make structural changes to Message Flows
– Create administration files such as MQSC scripts
• It is impossible to try and predict all the possible extensions that a
pattern author might wish to implement
• We provide two ways to extend pattern authoring
– Java code that is invoked when pattern instances are generated
– PHP templates that generate text files in pattern instance projects

35. Packaging a Pattern

36. Integra(on API
• The Integration API is our Java application development API
– Packaged along with the administration API in ConfigManagerProxy.jar
– Provides a complete Java API to manipulate message flows (and subflows)
• The Integration API can be used in standalone applications or in patterns
– Wide variety of applications are now possible such as business data mashups
– Easy to implement mass creation of message flows during migration projects

37. 6. Test!
Flow Exerciser
V10

38. Flow Exerciser
Click to start
recording

39. Flow Exerciser

40. Flow Exerciser
Now recording – the flow
is greyed out (read-only)

41. Flow Exerciser

43. Flow Exerciser

44. Flow Exerciser
Path through the
flow is highlighted
Click to see
message
content
Can re-send the recorded
messages (logical tree assembly)

45. Summary
• Use the features of WMB & IIB to their full extent!
– Message modelling, transformation options, applications and libraries
– Capabilities link in with activity trace, record edit replay, and more
• Libraries and subflows make for simpler designs
– Deployable subflows and libraries extend the benefits to the runtime
• Consider production and performance early on in your design
– In particular, applications and configurable services
– Your system administrator will thank you for this!
• Patterns address time-to-value for your solutions
– It takes just a few minutes to create a pattern!
– Why not create a patterns community in your organisation?
• Developer Edition
– Free edition of IB for use in evaluation, dev and test usage
– Fully functional: all nodes available and no time limitations
– Throughput rate limited to 1TPS per integration flow

46. Notices and Disclaimers
46
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or
transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has
been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors.
IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY
WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM
THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS
OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of
the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without
notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are
presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not
necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither
intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

47. Notices and Disclaimers Con’t.
47
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or
represent or warrant that its services or products will ensure that the customer is in compliance with any law
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or
other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM
products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or
the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL
WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM
patents, copyrights, trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise
Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM
ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®,
MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®,
PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter
Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson,
WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation,
registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A
current list of IBM trademarks is available on the Web at "Copyright and trademark information" at:
www.ibm.com/legal/copytrade.shtml.

48. Thank You
Your Feedback is
Important!
Access the InterConnect 2016
Conference Attendee Portal to complete
your session surveys from your
smartphone,
laptop or conference kiosk.