https://www.updateconference.net/en/2019/session/a-guide-through-the-azure-messaging-services A guide through the Azure Messaging services - Update Conference

1. A guide through the Azure Messaging services
Eldert Grootenboer

2. Eldert Grootenboer
@egrootenboer

3. Messaging?

4. Messaging in Azure
Event Hubs
Streaming
data
Service Bus
Enterprise
messaging
Event Grid
Cross cloud
reactive
eventing
Storage Queues
Simple task queues
Relay
Secure
sockets
without
changes to
your
network

5. Why so many choices?

6. Command
Report
Measurement
Job
Query
Handover
Transfer
Update
Assignment
Request
Notification
Trace

7. Facts
Command Report
Measurement
Job
Query
Handover
Transfer
Update
Assignment
Request
Notification
Trace
Intents
EventingMessaging

8. Events
Independent
Report State Change
Actionable
Time Ordered
Context Partitioned
Report Condition
Analyzable
Discrete Series

9. EventingMessaging

10. Let’s have a look at the services

11. Simple Queues

12. Storage queues are task queues
Coordinate work across compute (VMs, Functions, etc.)
Low cost
Pay as you go
Easy to use
Messages are generally homogeneous
Unlimited number of queues
Unlimited number of concurrent connections
Maximum queue size 500TB
Maximum message size 64KB
Few features

13. Few features
No headers/metadata
No pub/sub
No Тime To Live
No scheduling
No dead-letter queue
Peeking no more than 32 messages only
No de-duplication
7 days retention period
No transactional guarantees

14. A simple queue
Sender sends message to queue
Queue ACKs receipt
Receiver connects to queue & retrieves message
Receiver ACKs complete (or other action)

15. Enterprise Messaging

16. Enterprise Messaging as a Service
Queues & Topics
Reliable asynchronous communication
Rich features for temporal control
Message headers
Routing and filtering
Convoys & Sessions (related messages with state)

17. Features of Service Bus
Scheduled delivery
Time To Live
ForwardTo
Defer
Sessions
Batching
Ordering
Auto-delete on idle
OnMessage
Duplicate detection
Actions
Transactions
Poison message handling

18. Features of Service Bus Premium
Dedicated resources
Geo-DR
Availability Zones
Virtual Network Service Endpoints

19. Topics & Subscriptions
Sender only knows about Topic
Receivers only know about Subscriptions
Filters and Actions exist on Subscriptions
Topic
Subscription 1
Subscription 2
type=order
type=quote
Filter
Filter
“type” = “order”
“type” = “quote”

20. Where does Service Bus fit in?
Decoupling and providing durability with transactional consistency
Queue

21. Push based publish-subscribe event routing

22. What is an event?
Something happened
Notification
Triggers a reaction
Based on facts
Can be series based or discrete

23. IoTHub
BlobStorage
ResourceGroups
AzureSubscriptions
EventHubs
CustomTopics
Storage(GPv2)
AzureFunctions
LogicApps
AzureAutomation
WebHooks
EventHubs
Container Registry
StorageQueues
DatalakeStore
Azure Event Grid

24. Concepts
Events: what happened
Event Publishers: where it took place
Topics: where publishers send events
Event Subscriptions: how you receive events
Event Handlers: the app or service reacting to the event
Events

25. Trigger a functionto run
CognitiveAPI when a file
is added to storage
Use a function to run a
compliance checkon each
newly created SQL database
Tag newly provisioned VMs
with AzureAutomation and
add to metadata store
Use custom“drive start” and
“driveend” events to log vehicle
performancemetrics
What is Event Grid For?

26. "topic": "/subscriptions/{id}/resourceGroups/{rg}/providers/Microsoft.Storage/storageAccounts/{storageacc}",
"subject": "/blobServices/default/containers/monitored/blobs/blob.json",
"eventType": "Microsoft.Storage.BlobCreated",
"eventTime": "2017-09-20T01:12:35.8066557Z",
"id":"9ac0de1a-001e-00da-23ad-311d4a06bb2f",
"data":{
"api":"PutBlob",
"clientRequestId": "9ac0de1a-9da0-11e7-9a3b-51cc40a2ca27",
"requestId": "9ac0de1a-001e-00da-23ad-311d4a000000",
"eTag":"0x9AC0FC4ADA5997D",
"contentType": "application/json",
"contentLength": 268,
"blobType": "BlockBlob",
"url":"https://{storageacc}.blob.core.windows.net/monitored/blob.json",
"sequencer": "00000000000019200000000000F12B37",
"storageDiagnostics": {
"batchId": "9ac0de1a-e1a1-44c5-a0a1-a66b2a191fc4“
}
What does an Event look like?

27. Publishing custom events
Custom Topics
Topic: <Name of Topic>
Subject: <whatever you choose>
EventType: <whatever you choose>
mytopic.westus2-1.eventgrid.azure.net
sub1
sub2
mytopic
Topic:mytopic.westus2-1.eventgrid.azure.net
Subject: /products/beverages
EventType: skuCreated
Topic:mytopic.westus2-1.eventgrid.azure.net
Subject: /products/snacks
EventType: skuUpdated
Publisher

28. The challenge with Custom Topics
Having many topics forces you to do routing
This is what Grid was made for
There is a better way!
sub1
sub2
mytopic2
sub1
sub2
mytopic3
sub1
sub2
mytopic4
sub1
sub2
mytopic1
Publisher

29. Event Domains
All events to single endpoint
Sender sets topic
Subscribers are authenticated via
AAD (cross org support)
Subscribers identity not
required to be Azure
Entire service can be
white labeled

30. Advanced Filters
Works on the data portion of the event
Supports ten levels of nesting
Supports one or more values
Up to 10 advanced filters can be combined in
OR
Numbers
NumberLessThan
NumberLessThanOrEquals
NumberGreaterThan
NumberGreaterThanOrEquals
NumberIn–thevaluefordata.keyequalsavalueintheset[0,2.08,3.14]
NumberNotIn–thevaluefordata.keyisnotintheset[1,11,112,1124]
Strings
StringContains–thevaluefordata.keycontains“the”
StringIn–thevaluefordata.keyequalsavalueintheset[“small”,“brown”,“fox”]
StringNotIn–thevaluefordata.keyequalsavalueintheset[“jumped”,“over”,“the”]
StringBeginsWith–thevaluefordata.keybeginswith“lazy”
StringEndsWith–thevaluefordata.keyendswith“dog”
Boolean
BoolEquals

31. Engineered for scale and reliability
Retry intervals
10 seconds
30 seconds
1 minute
5 minutes
10 minutes
30 minutes
1 hour
Once an hour up to 24 hours
Defaults: 30 delivery attempts / 24 hours
Dead-lettering
Requires Storage account + container
Dead-lettered events stored as blobs

32. Storage
Queues
Flexible
Interoperability
Multi-Cloud
Solutions
IoT Hub
Blob Storage
Resource Groups
Azure Subscriptions
Event Hubs
Custom Topics
Storage (GPv2)
Container Registry
Service Bus
Azure Event Grid + CNCF CloudEvents
Azure Functions
Logic Apps
Azure Automation
WebHooks
Event Hubs

33. Distributed log streaming

34. Records a stream
Recording moves forward only
You can play the tape over and over again
Can play from anywhere on the tape
FastFwd or Rewind then Play
Multiple channels with different data
Event streaming

35. Event Hubs conceptual architecture
Kafka
HTTP
AMQP

36. Where does data streaming fit in?
Cloud gateways
(web APIs)
Field
gateways
Applications
LegacyIOT
(customprotocols)
Devices
IP-capabledevices
(Windows/Linux)
Low-powerdevices
(RTOS)
Search and query
Cortana Analytics
PowerBI Dashboards
Devices to take action

37. Event Hubs Capture: slow data stream
Policy based push to own storage
Avro format
Does not impact throughput
Better batching and archival scenarios
Offloads batch processing from your real-time stream (no pressure drop)

38. How does Capture work?

39. How are Event Hubs and Kafka similar?
Both use partitioned consumer
Partitions are independent
Both use a client side cursor concept
Both can scale to very high workloads
Conceptually they are nearly the same

40. Azure Event Hubs for Apache Kafka™
Kafka 1.0 compatible endpoint backed by Event Hubs
Use your existing Kafka application & tools with Event Hubs
Only need to change your connection string
All defaults supported

41. Cloud relayed sockets

42. Cloud relayed communication
Hybrid Connections & Relay
Securely connect to on premises data and services from anywhere
Load balance multiple sites behind a single endpoint
Protect internal services by projecting a cloud endpoint
Does not require any network changes

43. Where does Relay fit in?
When you have scenarios that are too complex or costly for VPN
When VNET integration is not possible
When you have to keep your data where it is

44. How does it work?
Service registers cloud endpoint
Client connects to cloud endpoint
Network magic happens

45. Relay as a gateway
Can front multiple service instances
Round robin controlled in Relay service
Sticky sessions
ServiceInstances

46. Almost done

47. Serverless Streaming Data Enterprise
Key Takeaways
Segment Simple Queuing Eventing PubSub Streaming Enterprise Messaging Cloud Socket
Product Storage Queues Event Grid Event Hubs Service Bus Relay & HC
What do you
care about
• Communication
within an app
• Individual
message
• Queue (polling)
semantics
• Easy to use
• Pay as you go
• Communication
between apps /
orgs
• Individual
message
• Push semantics
• Filtering and
routing
• Pay as you go
• Fan out
• Many messages in a
Stream (think in MBs)
• Ease of use and
operation
• Low cost
• Fan in
• Strict ordering
• Works with other
tools
• Instantaneous
consistency
• Strict ordering
• JMS
• Non-repudiation &
Security
• Geo-Replication &
Availability
• Rich features (de-dupe,
scheduling, etc.)
• Raw TCP socket
that can traverse
firewalls
• Load balanced
live traffic
What you
care less
about
• Ordering of
messaging
• Instantaneous
consistency
• Ordering of
messaging
• Instantaneous
consistency
• Individual message
semantics
• Server-side cursor
• Only Once
• Cost
• Simplicity
• Buffered
messaging
• Queue semantics
Focus point • Single partition
lightweight
queue
• Reliable fan out
push at massive
scale
• High scale distributed
log
• Highly indexed full
featured message broker
• High density TCP
tunneling
Hybrid

48. @egrootenboer
eldert@eldert.net
https://blog.eldert.net
Thank You