Scalable Event Processing with WSO2CEP 4.0, Describing how it can scale using Apache Storm.

2. ● Complex Event Processing
What ?
Why ?
● WSO2 Complex Event Processor
Scalability
Edge Computing
● Usecases
● What’s new is WSO2 CEP 4.0 ?

6. ● Amount of data you analyse are big
● You don’t need data for long time
● You don’t want to store all the data
● Time critical analysis
○ Finding opportunities before they’re gone
○ Identify threats beforehand

9. • Filters & Transformations
• Windows and Aggregations
• Joins
• Sequences and Patterns
• Event Tables

10. Stateless
• Filters & Transformations
Stateful Inmemory
• Windows and Aggregations
• Joins
• Sequences and Patterns
Stateful Persisted
• Event Tables

12. • Vertical Scaling
• Horizontal Scaling

14. E.g. Calculate Mean

15. E.g. Calculate Mean

16. E.g. Calculate Mean

17. How about scaling median ?

18. How about scaling median ?
If & only if we can partition !

20. Spark Streaming
• Supports distributed processing
• Runs micro batches
• Not supports pattern & sequence detection

21. Spark Streaming
• Supports distributed processing
• Runs micro batches
• Not supports pattern & sequence detection
Apache Storm
• Supports distributed processing
• Stream processing engine

22. Advantages
• Supports distributed processing
• Supports Partitioning
• Extendable
• Open Source
Disadvantages
• Need to write Java code
• Need to start from basic principles ( & data structures )
• Adoption for change is slow
• No support to govern artifacts

23. Advantages
• Supports distributed processing
• Supports Partitioning
• Extendable
• Open Source
Disadvantages
• No need to write Java code (Supports SQL like query
language)
• No need to start from basic principles (Supports high level
language)
• Adoption for change is fast
• Govern artifacts using Toolboxes
• etc ...

28. define stream StockStream
(symbol string, volume int, price double);
@name(‘Filter Query’)
from StockStream[price > 75]
select *
insert into HighPriceStockStream ;
@name(‘Window Query’)
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;

29. define stream StockStream
(symbol string, volume int, price double);
@name(‘Filter Query’)
from StockStream[price > 75]
select *
insert into HighPriceStockStream ;
@name(‘Window Query’)
partition with (symbol of HighPriceStockStream)
begin
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;
end;

30. define stream StockStream
(symbol string, volume int, price double);
@name(Filter Query’)
@dist(parallel= ‘3')
from StockStream[price > 75]
select *
insert into HightPriceStockStream ;
@name(‘Window Query’)
@dist(parallel= ‘2')
partition with (symbol of HighPriceStockStream)
begin
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;
end;

33. Siddhi CEP
Siddhi CEP

34. • System Monitoring (Logs, Servers)
• Finance (Financial trading)
• Banking (Credit card fraud detection)
• Marketing (Location & time based advertising)
• Logistics (Taxi services, Vehicle parking)
• Healthcare
• E-commerce (Online retailers)
• Telecommunication (Mobile offers)
• System automation (Auto scaling)
• IoT (Smart home, Smart buildings)
• ...

35. • Distributed processing with Apache Storm
• Improved High Availability
• Improved Siddhi Query Language
• Toolbox Support
• Realtime Dashboard
• Websocket + MQTT Support
And lots of extensions
• Geo
• Timeseries
• NLP
• PMML & WSO2 ML
Try out CEP 4.0 Alpha : https://github.com/wso2/product-
cep/releases