There’s never been more options -- or more confusion -- about which technologies to use when connecting systems? When is BizTalk Server the right choice? Should I still use WCF? Is the Windows Azure Service Bus ready for production scenarios? In this session, we’ll first do an overview of each core technology in the integration portfolio. Then we’ll review a simple decision framework for down-selecting your choices. Finally, we’ll have an interactive discussion (and demonstration) of real-life scenarios and which technology is the right fit.
When to use What? A look at choosing Integration Technology
1. brought to you by
When to Use What?
A Look at Choosing Integration Technology
RICHARD SEROTER
@rseroter
2. Richard Seroter
Director of Product Mgmt at CenturyLink Cloud
Microsoft MVP for Integration
Technical Trainer at Pluralsight
Cloud Editor for InfoQ.com
3-time Book Author
22. Design
DevOperations
Org
Strategy
Long term solution fit
Vendor support available
Uses existing (software) investments
Budget impact
Build vs. buy philosophy
Risk tolerance
Speed to market
OSS components
Commodity skill sets
Simple upgrades
41. QUICKQUALIFY
“Get multiple loan quotes within minutes!”
BUSINESS NEED COMPANY OVERVIEW
• Accept loan parameters from web customers
• Store loan request in on-premises system
• Reliably provide loan request to volatile list of
external providers
• Track responses and show the customer choices
• High uptime, low latency integration
• Strong development team
• Over-extended Ops team
• Decentralized org, local autonomy
• Microsoft-friendly organization
Previous book (cover)
Technology count: WCF, WF, AppFabric, BizTalk, ESB, SSIS, SSSB, Sync Framework, Azure SB Relay, StreamInsight
Technology count TODAY: WCF, WF, AppFabric, BizTalk, ESB, SSIS, SSSB, Sync Framework, Azure SB relay, Azure SB Brokered, Azure VMs, Azure Notification Hubs, BizTalk Services, StreamInsight
LET’S TALK THROUGH SOME TRENDS THAT EMERGED SINCE THIS BOOK
Source: http://www.flickr.com/photos/davenielsen/5718247676/sizes/l/
Cloud computing
Change to the traditional computing paradigm
Host matters less; elastic, on-demand
Accepted part of an organization’s app portfolio
New types of applications and scenarios
Widely distributed
Mix of hosting providers
Different integration scenarios
Cloud to cloud, ground to cloud, cloud to ground
Source: http://www.flickr.com/photos/mattandbeccasteudelwedding/9067696311/sizes/l/
Micro Services
Something championed by ThoughtWorks a while back, and now something Netflix (and others) do a lot of
Mini applications/services
No heavy, complex services
Very specific services that are easy to consume and weave together
Think of cloud providers who have distinct services for provisioning services, autoscale, etc
Source: http://www.flickr.com/photos/24752578@N02/5589897831/sizes/l/
Lightweight Integration
It’s not just about a powerful, centralized integration server
That’s of course always been the case, but new attention paid to simplified messaging
Something between a basic queue and a full blown ESB
New tools, new way to connect lightweight services together
Source: http://www.flickr.com/photos/anekoho/9638930676/sizes/h/
NoSQL endpoints
New types of endpoints, often outside your direct control
Relational databases are less important now than ever
Still a place for them, but major companies are finding they don’t need them
We’ve replaced our SQL backend with one
To the integrator, this adds new scenarios
Source: http://www.flickr.com/photos/peternijenhuis/6959653638/sizes/h/
Automation
DevOps and the cultural movement to unprecedented synergy between Dev and Ops teams
Automated builds, continuous integration, continuous deployment
Possible to get rid of snowflake servers, and move to immutable servers
Source: http://www.flickr.com/photos/suckamc/5907306188/sizes/l/
We want to be at a place where we have the simplest toolset possible, but one that meets our needs
This continued expansion of available integration services is logical, when considering these trends
There are more and more tools for specific scenarios (micro-integration services), and fewer monolithic tools that serve EVERY purpose
How do we get there?
Walk through a decision framework
Talk through the available technologies
Apply these principles
What’s a decision framework?
Way to analyze a situation and identify the right fit
There cannot be a massive decision matrix that drops the correct solution in your lap
Instead, there are multiple dimensions to consider, and those may be unique to each organization and situation
Steps
First, let’s discuss the sources that describe our situation
Then, we’ll talk about the 4 dimensions to discuss
Input into a decision …
Functional requirements
What is the application supposed to do?
Business requirements
Input into a decision …
Non functional requirements
Describe what the system itself needs to do to support the functional requirements
security, availability, maintainability, compliance, exception handling, etc
You can argue if these are actually functional requirements, and in some cases they are
Input into a decision …
Derived requirements
Ones that aren’t explicit stated, but can be derived from what’s needed
Based on a functional requirement to provide a certain type of data, I might derive a requirement to persist data in a warehouse for long range analytics
Input into a decision …
Organization direction/strategy
This is the least transferrable!
What are the company goals, investments?
These are often fixed, and non-negotiable
NOW, LET’S TALK ABOUT THE DECISION CRITERIA ITSELF
Solution design – design patterns required
Data volumes, data set size, guaranteed delivery, data sources/destinations, latency, sync/async, data processing (workflow, quality operations), SSO, failure handling
MODERN UPDATES … (reliability of endpoint, location of endpoint, constraints imposed by source/destination, circuit breakers)
Solution development – what it takes to build the solution
Developer availability, learning curve, developer setup, IDE and tool integration, support system, source control integration, automated builds
MODERN UPDATES … (works in cloud, web-based IDEs, continuous integration/delivery support)
Solution operations – what it takes to maintain the solution
Event logging, tuning ability, handle bursts and failures, backup/restore routines, support tools, versioning capabilities, scriptability
MODERN UPDATES … (immutable runtimes, rapid releases/rollbacks, works in distributed fashion)
Organizational considerations – enterprise standards and org direction
Long term solution, vendor support, uses existing software investment, limited budget impact, build vs buy compatibility, risk tolerance, speed to market
MODERN UPDATES … (OSS, commodity skill sets, easy/no upgrade)
Product: BizTalk Server
Tag Line: Comprehensive application integration
Most Recent Release: 4/13
Capabilities: Durable messaging, orchestration, technology adapters, trading partner management, BAM, BRE, etc
Core Use Cases: Async messaging-based integration between applications; stateful, long running workflow processes that coordinate interactions between systems
Buy-Hold-Sell: Buy
Product: ESB Toolkit
Tag Line: “Dynamic messaging”
Most Recent Release: 4/13 with BizTalk 2013
Capabilities: Dynamic routing, services for mapping and pipelines
Core Use Cases: Diverse message paths; message flow modeling, supremely loose coupling, centralized exception management
Buy-Hold-Sell: Soft Buy
Product: Windows Server AppFabric
Tag Line: Advanced service hosting
Most Recent Release: 3/12
Capabilities: Caching, advanced WCF and WF hosting; troubleshooting; persistent workflows
Core Use Cases: WCF service management, hosting long running WF services
Buy-Hold-Sell: Hold
Product: Service Bus for Windows
Tag Line: On-premises durable messaging
Most Recent Release: 10/13
Capabilities: Running messaging aspects of Azure Service Bus on-premises; topics and queues; manage with new Windows Azure Pack on Windows
Core Use Cases: Start work on premises and move to cloud; basic pub-sub messaging;
Buy-Hold-Sell: Hold
Product: Workflow Manager
Tag Line: On-premises workflow host
Most Recent Release: 3/13 (CU)
Capabilities: multi-tenancy, tracking and monitoring, instance management, versioning support
Core Use Cases: Host and manage WF apps;
Buy-Hold-Sell: Hold
http://msdn.microsoft.com/library/jj193528
Product: WCF
Tag Line: Sophisticated service framework
Most Recent Release:
Capabilities: Contract first service design for REST and SOAP services; configuration-centric service model that is extreme (overly?) extensible
Core Use Cases: Building SOAP based web services; Sync or async
Buy-Hold-Sell: Soft Buy
Product: ASP.NET Web API
Tag Line: Modern web services
Most Recent Release: 1/14
Capabilities: REST/HTTP service design; extensible model; Odata support
Core Use Cases: web service design
Buy-Hold-Sell: Buy
Product: Windows Workflow
Tag Line: Lightweight, durable processes
Most Recent Release:
Capabilities: declarative models that describe a process; built-in and custom activities; durable workflow; service endpoints
Core Use Cases: long running workflows; exposing business processes as services
Buy-Hold-Sell: Soft Buy
Product: SSIS (also Master Data Services?)
Tag Line: Bulk data transfer
Most Recent Release: SQL 2014 (no big changes)
Capabilities: Data integration and data transformation; lots of data sources (files, FTP, HTTP, SMTP, RDBMS); bulk movement
Core Use Cases:
Buy-Hold-Sell: Buy
Product: StreamInsight
Tag Line: Event stream processing
Most Recent Release: 6/12
Capabilities: high volume pattern matching against event streams; adapter model; Iobservable, Ienumerable support; multiple hosting options; LINQ queries
Core Use Cases: situational awareness from data streams
Buy-Hold-Sell: Soft Hold
Product: Windows Azure Virtual Machines – BizTalk Image
Tag Line: Full BizTalk in the Cloud
Most Recent Release: June 2013
Capabilities: Run full BizTalk Server in the cloud; no HA with SQL clustering
Core Use Cases: Dev/test scenarios
Buy-Hold-Sell: Hold
Product: Windows Azure Service Bus Relays
Tag Line: Services that span boundaries
Most Recent Release:
Capabilities: Traverse networks and securely expose sync and async WCF services
Core Use Cases: partner integration, mobile device integration, remote worker tools, cloud to cloud integration
Buy-Hold-Sell: Strong Buy
Product: Service Bus Brokered Messaging
Tag Line: Durable messaging in the cloud
Most Recent Release:
Capabilities: Durable messaging through queues (single reader), and topics (broadcast with subscriptions)
Core Use Cases: partner integration, mobile integration, cloud to cloud integration
Buy-Hold-Sell: Buy
Product: Service Bus Notification Hubs
Tag Line: Scalable message distribution
Most Recent Release:
Capabilities: Broadcast messages to mobile devices and Windows 8;
Core Use Cases:
Buy-Hold-Sell: Buy
Product: Windows Azure BizTalk Services
Tag Line: Lightweight cloud message broker
Most Recent Release:
Capabilities: Link applications via cloud-hosted broker; source/destination adapters; strong data mapper
Core Use Cases: connecting partners, cloud apps
Buy-Hold-Sell: Strong Hold
MANY other products that work well in a Windows and cloud world
Brokers like Apache Kafka
Queue technologies like RabbitMQ, or Amazon SQS
Cloud-based integration tools like SnapLogic and Informatica Cloud
SaaS-integrated services for message publication and workflow
E.g. Dynamics CRM Online or Salesforce.com
Options …
Use messaging and use BizTalk to send messages to vendor endpoints
PRO: Good tracking, chance for enrichment before sending out; guaranteed delivery
CON: hard to quickly change endpoints; puts responsibility on internal team to manage/scale;
Use messaging and use BizTalk to send to Topics, local DB, and notification Hubs
Use messaging and send loans directly to SB topics (and internal DB) from WF
PRO
CON
Send loans to SB topic and one listener is internal DB
PRO: Simple, offload responsibility for delivery
CON: No coordination; chance of race condition; no notification for vendors of new messages
Send loans to SB topic, one listener is internal DB, and also use Notification Hubs for alerting
Provide REST-ful query API (exposed via SB) on loan DB and use notification hubs to send messages to providers to query and provide their best offer
PRO: Simple; no replication; easy to onboard new vendors; offload availability to 3rd party; durable;
CON: Not event based; need to chain together way to notify vendors; no orchestration or tracking; limited visibility into vendor processing
Let’s consider the option that leverages the Service Bus for all this
Send a message to Notification Hubs so very easy to broadcast info and push alerts
Use Topics for scalable, durable messaging
Subscribe ourselves to update internal systems