There are times when you should consider setting up secure communications between your software components across network boundaries.
Here are just a few:
* Your application is enormous (e.g., the global deployment of a marketing site targeting billions of people)
* Remoteness (e.g., your company has branch office locations around the globe)
* Your network constraints prevent communication (e.g., your machines in Azure Cloud Services are unable to talk to each other directly)
* You don't know the network conditions (e.g., IoT or mobile devices)
Yves Goeleven and Sean Feldman show how to overcome such challenges using Azure Service Bus.
Making communication across boundaries simple with Azure Service Bus
1. Yves Goeleven & Sean Feldman
Solution Architects & Azure MVPs
http://particular.net
Azure Service Bus
Setting up communication across boundaries
2. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Secure & reliable communication
between software components
across network boundaries.
• Inside cloud services
• Between datacenters
• Between branch offices
• Hybrid solutions
• IoT & Mobile devices
When do you need Azure Service Bus ?
3. Introduction to ServiceInsight for NServiceBusAzure Service Bus
• DNS (mynamespace.servicebus.windows.net): Scale Unit allocation
• Load Balancer: High Availability
• Gateway Node: Authorization & entity lookup
• Broker Node: Servicing entity operations store
• Store: SQL Azure Database or Storage Container
High Level Architecture Overview
Scale Unit
Scale Unit
DNS
L
B
L
B
Gateway
Gateway
Gateway
Gateway
Client
Broker
Broker
Broker
Broker
Store Store Store
Store
Store Store
StoreStore Store
Store
Store
4. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Entity: a generic term for constructs in an Azure Service Bus namespace
Entity ?
Queue
Subscription
Subscription
Topic
Sending
Client
Sending
Client
Receiving
Client
Receiving
Client
Receiving
Client
Receiving
Client
Each entity is a SQL Azure Database or
Storage Blob under the hood!
Messaging Entities
• Queue
• Topic
• Subscription
• Rule
Other (out of scope): Relay,
EventHub, Consumer Group, ...
6. Introduction to ServiceInsight for NServiceBusAzure Service Bus
2 Ways to connect
• NamespaceManager: to create & manage entities
• MessagingFactory: to send/receive
Connectivity
7. Introduction to ServiceInsight for NServiceBusAzure Service Bus
NamespaceManager
• Namespace Address
• TokenProvider
• Methods:
• Get, Exists, Create,
Update, Delete
Connecting to manage entities
Namespace
Manager
Additional options
• REST API: https://docs.microsoft.com/en-us/rest/api/servicebus/
• ARM: https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-resource-manager-
overview
Token
Provider
REST API
HTTPS
Creation is really expensive!
C#
8. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Exceptions to deal with:
Connectivity
• Timeouts
• Authorization (manage)
Broker Exceptions
• Transient
• Non-transient
Concurrency conflicts
What can possibly go wrong?
DNS
L
B
Gateway
Client
Broker
timeout
timeout
Exception Exception
Unauthorized
Client
? Concurrency conflict
timeout
C#
9. Introduction to ServiceInsight for NServiceBusAzure Service Bus
MessagingFactory
• Namespace Address
• TokenProvider
• Connection Settings
• Maintains the network
connection
MessageSender/Receiver
• Created by Factory
• Logical connection
Connecting to send/receive messages
Messaging
Factory
Message
Sender
Establishing a physical connection is expensive!
Message
Receiver TCP/WebSocket
Connection
AMQP/SBMP
Messaging
Factory
Message
Sender
Message
Receiver TCP/WebSocket
Connection
AMQP/SBMP
Broker
Gateway
Node
Broker
Gateway
Node
C#
10. Introduction to ServiceInsight for NServiceBusAzure Service Bus
You want to receive/send as long as
your application lives.
Physical connection will drop!
Creating a factory every time is not
an option: establishing connection is
expensive!
Lifecycle management
• Maintain instances/pools
• Manage connection & open new if
closed
Additional Connectivity Challenges
DNS
L
B
Gateway
Factory Local interruption
Hardware failures
Reboot
Factory
C#
12. Introduction to ServiceInsight for NServiceBusAzure Service Bus
2 Ways to receive and process
messages
• Receive And Delete
• Peek Lock
Receiving messages
13. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Message deleted upon receive.
Cannot rollback received
message.
Single shot processing
• Exception occurs = message
lost!
Cloud environment is very
susceptible to exceptions!
• Advise against this mode for
workflow style processing
Receive and Delete
Message
Receiver
Receive And Delete
Delete
C#
14. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Hide message on receive
• Temporal lock
• Explicit complete/abandon
after processing
Multi shot processing
• Exception occurs = Retry
• Retry counter
• Counter exceeded => DLQ
Slow processing = retries!!!
• RenewLock -> more gotcha’s
there, see performance section
Peek Lock
Message
Receiver
Peek Lock
DLQ
Temporal Lock
Delete
C#
15. Introduction to ServiceInsight for NServiceBusAzure Service Bus
What Topology to use?
• Queue per message type
• Queue per receiver
Serialization Challenges
• Internal serialization or not?
• Need to deserialize same way!
• What CLR Type (Q per endpoint)
Concurrency Challenges
• All code must be thread safe
Handling Multiple Message Types
Receiver
Receiver
Sender
ReceiverSender
Is this the right way to construct a message?
new BrokeredMessage(“{json}”)
Sender
Receiver
Receiver
C#
16. Introduction to ServiceInsight for NServiceBusAzure Service Bus
MaxDeliveryCount
• To avoid locking on processing
problematic messages, a
message is automatically
deadlettered after number of
deliveries.
Challenges
• Poisonous vs errored messages
• Building a backing off strategy for
retries (3rd party web service)
• Monitoring for DLQed messages
over multiple entities
Dead Letter Queue vs Business Errors
DLQ
Max
Delivery
Count
failed processing
Receiving
Client
10
.
.
.
failed processing
Receiving
Client
2
failed processing
Receiving
Client
1
C#
18. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Challenges we’ll cover
• Throttling
• Message size
• Flooding entities
• Transactions
Sending messages
19. Introduction to ServiceInsight for NServiceBusPresentation title
Usage exceeding quotas, the
broker will throw a
ServerBusyException.
Mitigation: retry after 10 seconds.
Retries can fail as well.
Can’t retry the same message
object.
Exception Handling (User Code)
succeeded
Sending
Client
.
.
.
throttled
Sending
Client
throttled
Sending
Client
Operations can and will fail. Based on exception, need to be retried.
C#
20. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Message size (including headers)
• Standard tier: 256KB
• Premium tier: 1MB
Headers: up to 64KB
MessageSizeExceededExceptio
n
Message Size
Sending
Client
Size < MAX
Sending
Client
Size > MAX
Warning! Message size is not
known until it’s sent out…
“To get an accurate value for the size of a
BrokeredMessage, you should read the Size
property after completing the Send/Receive
operation on the BrokeredMessage.”
21. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Scenario:
An orphaned subscription queue
receiving messages that are not
processed.
Challenges
• Overflooding subscriptions
• Overall namespace
performance degradation
Overflooding Subscriptions
Subscription
Subscription
Topic
Sending
Client
Receiving
Client
Receiving
Client
Solutions
• Use Forwarding
Gotcha
• TTL on subscription
Forwarding
Receiving
Client
Sending
Client
22. Introduction to ServiceInsight for NServiceBusAzure Service Bus
All messages succeed or fail –
atomic
NOT entirely what you think!
• Single queue only
• No other transactional
resources allowed
Transactions
Queue
Sending
Client
Queue A
Sending
Client
Queue B
C#
23. Introduction to ServiceInsight for NServiceBusAzure Service Bus
All messages succeed or fail -
atomic
Surprise!
A hard limit of 100 messages or
TransactionSizeExceededExcep
tion
Prevents usage of batch
completion, has negative
performance impact
Transactions and Number of Messages
Sending
Client
1 100…
Sending
Client
1 100… 101
C#
24. Introduction to ServiceInsight for NServiceBusAzure Service Bus
All messages succeed or fail –
atomic
NOT entirely what you think!
• Abandon of incoming message
can not be included!
• Complete can be included
• Single queue only
• If complete included, sends
must go to receive queue
• No other transactional
resources allowed
Transactions with Receive Operations
Queue
Client
Queue A
Client
Queue B
Complete
Complete
25. Introduction to ServiceInsight for NServiceBusAzure Service Bus
A hidden gem in a heap of
features
Work around for the single queue
transaction limit!
• Uses receive queue as a
transfer queue
Use limited within namespace
Send Via
myQueue
Receiving
Client
queueA
Sending
Client
queueB
atomic operation
f
o
r
w
a
r
d
i
n
g
transfer queue
C#
27. Introduction to ServiceInsight for NServiceBusAzure Service Bus
DOs and DONTs
• Factories
• Batching
• Coding practices
• Prefetching
Performance improvements
28. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Remember
• Using multiple senders
helps.
• More factories helps more.
Multiple Factories
Messaging
Factory
Message
Sender
Message
Sender TCP/Websocket
Connection
AMQP/SBMP
Messaging
Factory
Message
Sender
Message
Sender TCP/Websocket
Connection
AMQP/SBMP
Broker
Gateway
Node
Broker
Gateway
Node
C#
29. Introduction to ServiceInsight for NServiceBusAzure Service Bus
2 Kinds
• Batch flush interval
• Batched send
Batch size
• Standard tier: 256KB
• Premium tier: 1MB
MessageSizeExceededException
Bonus! Total message size is not
known until it’s sent out…
Extra Bonus! Max 100 messages
when sending inside
transactionscope
Batch Sending
Sending
Client
1 N…
Sending
Client
1 N…
Batch size < MAX
Batch size > MAX
30. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Don’t!
Do!
Coding Practices That Kill Native Batching
foreach(var message in messages)
{
await sender.SendAsync(message)
}
var tasks = new List<Task>()
foreach(var message in messages)
{
tasks.Add( sender.SendAsync(message) )
}
await Task.WhenAll(tasks)
31. Introduction to ServiceInsight for NServiceBusAzure Service Bus
How it works
• Load additional messages when
receiving
• Prefetched messages are locked on
the gateway
• Each client maintains its own in-
memory cache of processing
Prefetching
Messaging
Factory
Message
Receiver
Broker
Gateway
Node
C#
Gotchas
• Cannot renew locks
• Prefetch size depends on the processing code speed
• Unlocked prefetched messages are not evicted from cache
• Prefetched messages with expired TTL are not evicted from cache
32. Introduction to ServiceInsight for NServiceBusAzure Service Bus
Service Bus is a great service!
• With many powerfull
features
Combining those features while
maintaining reliability &
performance is the challenge!
It’s a balancing act