The usual proposal and demos of EventHub and IoTHub event processing is done with Azure Functions or Azure Stream Analytics. Now with the great offering of container technology, implemententing the "basical" IEventProcessor interface or the (not so) hidden EventHubClient receiver becomes more practical and scalable. So let's get back to the C# coding roots. In this session you will learn processing and aggregating data for monitoring and real time analytics with C#, when Stream Analytics can be oversized. And then you will learn how to make some useful stateful calculations for your IoT solutions to transform the stream processing service into a more complete IoT backend.

1. #azureconf18
Stateful stream processing
of IoT events with C# and
containers
Marco Parenzan
Solutions Sales Specialist @ Insight
Azure MVP

2. #azureconf18
marco_parenzan
marcoparenzan
marcoparenzan
Marco Parenzan
Solutions Sales Specialist @ Insight
Azure MVP

3. “Avoid state!

4. “Human interaction events
make stateless affordable...
IoT events no...

5. #azureconf18
Avoid state!
• Not always possible
• Store/restore can be
• Aggregation
• Calculations
• Expression evaluation
• Alarm

6. #azureconf18
Which services for (IoT) events
processing?
• Azure Functions
• Stream Analytics
• EventProcessorHost

7. #azureconf18
Azure Function
• Good for low ingestion
• Moving over EventGrid will improve performances
• No event correlation
• Durable Functions not for this...
• EventGrid will change, but it’s not the same

8. #azureconf18
Stream Analytics
• Great semantics
• Hyperscale
• But
• Not cost effective in the mid-low
• No SQL friendly
• Difficult to test
• Slow to start
• shooting fish in a barrel (equivalente di «sparare a un topo con
un cannone»)
• Just to make time window aggregation

9. #azureconf18
EventProcessorHost
• Where it is?
• How does it scale with EventProcessorHost?
• EventHub is «a queue» (not really, but Receive exists!)

10. #azureconf18
Summer testing of...
• Experimenting with IoT protocols directly (AMQP and
MQTT) by the device
• Consuming EventHub with AMQP protocol directly
• Consuming Redis with raw protocol
• Want to experiment containers
• No more VMs for workers
• Never liked WebJobs for that
• No functions…Event Grid destiny…

11. “...and a 1SU (3
subqueries) Stream
Analytics process hanging
with just 40 messages at
the same time every two
minutes...

12. #azureconf18
“Serverless Streaming At Scale with
Cosmos DB” by Davide Mauri
• October 9th, 2018 after my thought...
• https://medium.com/@mauridb/serverless-streaming-at-
scale-with-cosmos-db-e0e26cacd27d
• «…But if you don’t need time-aware features, like Hopping
or Tumbling Window, complex data processing capabilities,
like stream joining, aggregates of streams and the likes, a
more lightweight solution can be an option….”

13. #azureconf18
Containers
• Containers = Love
• IaaS = Hate
• AKS .... Love?  No Melchiori around YEAH!
• Serious Answer
• Containers are the futurepresent
• I don’t implement an AKS cluster in a proposed solution, not
PaaS
• Only if already existing or requested
• Service Fabric is the same

14. #azureconf18
Great news
• Azure Container Instances
• AKS as a Service (PaaS)
• Not completly equivalent with AKS
• No orchestrator
• Fits well for a fixed partition-based model (one container
instance per partition)

15. #azureconf18
Biggest news
• Service Fabric Mesh!
• SF as a Service Fabric
• Still in preview (only container orchestrator, no Reliable
Collections, no Actor) but look promising

16. #azureconf18
Stateful IoT processing: how
does it work?

17. #azureconf18
Architecture
Event Hub
Stream
Analytics
SQL
Database

18. #azureconf18
Replacing Stream Analytics
Event
Processing
Hopping
Window
Accumulating
Events in
windows
Releasing
Windows
Persisting
Windows

19. #azureconf18
Event Processing
• Accessing EventHub API (also for IoT Hub)
• Using AMQPLite lib
• also Microsoft.Azure.EventHub
• But no Microsoft Azure.EventHub.Processor

20. #azureconf18
Hopping Window
• Fixed size time window
• Overlapping window
• Extreme condition: zero
overlapTumbling Window
• Class for window
identification

21. #azureconf18
Accumulating events in windows
• Collections
• High availability?
• Redis!
• Using Lists

22. #azureconf18
Releasing data as time windows
• How long windows stay in memory?
• SA uses the notion of «unordered events»
• No time critical
• Example 15minutes windowafter 20+ minutes
• How?
• Redis caching expiration!

23. #azureconf18
Persisting Time Windows
• Direct SQL pressure? No!
• Using queues
• Then a worker can off load queue and store in SQL
• In the demo it’s just a thread, but it can be another
container...

24. #azureconf18
Hyper-scaling
• All these steps can be implemented in different containers
(and probably SA does the same )
• And remember that you can replicate it for each partition

25. #azureconf18
Building an event processor
• .NET Core
• ASP.NET Core
• IHostedService
• Make it observable (at least in testing)
• MVC
• SignalR

27. #azureconf18
Hosting a process in a
container: Azure Container
Instance

28. #azureconf18
Azure Container Instances (ACI)
Easily run containers without managing servers
Now GA!
Containers as a primitive
billed per second
Secure applications with
hypervisor isolation
Run containers
without managing
servers

29. #azureconf18
What can you build with ACI today
Elastic Bursting
(AKS)
Event Driven AppsData Processing
Jobs

30. #azureconf18
AKS bursting with the ACI Connector
Deployments/tasks

32. #azureconf18
Hosting a process in a
container: Service Fabric
Mesh

33. #azureconf18
Service Fabric
Dedicated Azure clusters
Service Fabric
Cluster
Bring your own infrastructure
Service Fabric
Standalone
On-premisesAny cloud
Dev machine
Dedicated
Service Fabric
Mesh
Fully managed by Azure
Serverless

34. #azureconf18
Service Fabric Mesh
• Fully managed containers and microservices platform
• Elastic scaling
• Billing by the second
• Currently in preview

35. #azureconf18
Service Fabric Mesh Preview
• Deploy multi-container applications
• Networking and ingress
• Azure Files volumes
• Deploy with other Azure resources
• Windows Containers only

36. #azureconf18
Service Fabric Mesh – the road ahead
• Reliable collections
• Service Fabric volume driver
• Secrets (Key Vault)
• Routing rules and Envoy integration
• Ingress traffic routing
• Service-to-service traffic routing
• Partition-aware routing
• DNS for service discovery
• Retry and circuit-breaker logic

37. #azureconf18
Reliable Collections
Replicated transactional store
Data structures C++ STL API
C# Java Go Others
Replication, leader election,
orchestration, clustering, federation,
etc.
Service Fabric runtime
Reliable Collection Libraries
Language-specific data structure APIs

38. #azureconf18
Reliable Collections
• Separate lifecycle
• Reliable Collections handle Service Fabric state machine events
• Your service is stateful, but your code looks stateless
Service Package
Reliable Collections Runtime
Stateful lifecycleReliable Collections APIs
Your Application Code
Service Fabric runtime

39. “No more Redis
(that is nothing about
better or worst)
Just one thing less...

41. #azureconf18
Conclusions

42. #azureconf18
Conclusions
• IoT is not easy, so the event processing itself cannot be
easy!
• Remember the EventHub API (refactored
• Time for containers (and less Functions or WebJobs)
• Service Fabric Mesh look promising (and ACI too)
• W Serverless, W PaaS
• Let SQL live longer