VLDB 2010 tutorial - "event processing - past present and future". Singapore, September 15, 2010

1. Event processing – past, present, future VLDB 2010 Tutorial, Singapore, September 15 th , 2010 Opher Etzion (opher@il.ibm.com)

2. An example to kick-off this tutorial: Act: Passenger has been re-routed to another destination – send the luggage Act: Bag has reached to the wrong aircraft Notify: Bag has been checked but did not reach the ULD within 20 minutes Notify: Bag has been checked but did not reach the connecting flight THE LUGGAGE PERSPECTIVE: Across the 24 largest airlines more than 5.6 million bags went missing in 2006, this is an average of 15.7 bags per 1,000 travelers. 15% of the bags are never found. BBC News, April 4, 2007 Event processing can help here..

3. Outline of this tutorial What is behind event processing ? How is it related to other computing terms? Where are its roots? Event processing – architecture, building blocks The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications The future: Trends and research challenges

4. What is behind event processing ? How is it related to other computing terms? Where are its roots?

5. What is “event processing” anyway? or Event processing is a form of computing that performs operations on events

6. Event processing and Data stream management? Aliases? One of them subset of the other? Totally unrelated concepts?

7. In computing we processed events since early days Network and System Management

8. Emerging technologies in enterprise computing (Gartner Hype Cycle, Summer 2009)

9. What’s new? The analog: moving from files to DBMS In recent years – architectures, abstractions, and dedicated commercial products emerge to support functionality that was traditionally carried out within regular programming. For some applications it is an improvement in TCO; for others is breaking the cost-effectiveness barrier.

10. What is an event There are various definitions of events Event is an occurrence within a particular system or domain; the word event has double meaning: the real-world Occurrence as well as its computerized representation

11. In daily life we often react to events..

12. Sometime we even react to the occurrence of multiple events I closed the deal with the Canadians We closed two huge deals in a single day, It is a good opportunity to send all the team to Some fun time in Singapore I closed the deal with the Australians

13. Event Patterns Pattern detection is one of the notable functions of event processing

14. What we actually want to react to are – situations TOLL VILOATOR FRUSTRATED CUSTOMER Sometimes the situation is determined by detecting that some pattern occurred in the Flowing events. Toll violation Frustrated customer Sometimes the events can approximate or indicate with some certainty that the situation has occurred

15. Event processing is being used for various reasons

16. Event Driven Architecture Event driven architecture: asynchronous, decoupled; each component is autonomic.

17. Ancestor: active databases On event When condition Do action With coupling mode Composite events were inherited to event processing

18. Ancestor: Data Stream management system Source: Ankur Jain’s website

19. Ancestor: Temporal databases There is a substantial temporal nature to event processing. Recently – also spatial and spatio-temporal functions are being added

20. Ancestor: Messaging – pub/sub middleware

21. Ancestor: Discrete event simulation

22. Ancestor: Network and system management

23. Event processing – architecture, building blocks

24. Fast Flower Delivery Flower Store Van Driver Ranking and Reporting System Bid Request Delivery Bid Assignments, Bid alerts, Assign Alerts Control System GPS Location Location Service Location Driver’s Guild Ranking and reports Delivery confirmation Pick Up confirmation Ranked drivers / automatic assignment Bid System Store Preferences Delivery Request Assignment System Manual Assignment Assignment Assignments, Pick Up Alert Delivery Alert http://www.ep-ts.com/EventProcessingInAction

25. Event Processing Agent Context Event Channel Event Consumer Event Type Event Producer Global State The seven Building blocks

26. Event processing network

27. Example of EPN – part of the FFD example

28. Event type definition Detection time, Occurrence time, source, Certainty… Stock id, quote, volume… Free comments…

29. Producer – State Observer in workflows State observer Push: Instrumentation points; Pull: Query the state

30. Producer – Code instrumentation

31. Producer – syndication

32. Producers – video streams to events

33. Producer – sensors

34. Producer and consumer - Sixth sense

35. Twitter as a producer and consumer

36. Consumer - Performance monitoring dashboard

37. Consumer - Ambient Orb

38. Event Processing Agent Filter Transform Detect Pattern Translate Aggregate Split Compose Enrich Project Event Processing Agents

39. The EPA picture

40. Filter EPA A filter EPA is an EPA that performs filtering only, and has no matching or derivation steps, so it does not transform the input event .

41. Transform EPA sub types

43. Pattern detection example Pattern name: Manual Assignment Preparation Pattern Type: relative N highest Context: Bid Interval Relevant event types: Delivery Bid Pattern parameter: N = 5; value = Ranking Cardinality: Single deferred Find the five highest bids within the bid interval Taken from the Fast Flower Delivery use case

45. Pattern policies Evaluation policy —This determines when the matching sets are produced Deferred Cardinality policy —This determines how many matching sets are produced within a single context partition Repeated type policy —This determines what happens if the matching step encounters multiple events of the same type Every - Override – First – Last – with max/min value of…

46. Pattern policies – cont. Consumption policy —This specifies what happens to a participant event after it has been included in a matching set Consume Reuse Bounded reuse Order policy —This specifies how temporal order is defined By occurrence time – by detection time – by stream position – by attribute

48. Context has three distinct roles (which may be combined) Partition the incoming events The events that relate to each customer are processed separately Grouping events together Different processing for Different context partitions Determining the processing Grouping together events that happened in the same hour at the same location

49. Context Definition A context is a named specification of conditions that groups event instances so that they can be processed in a related way. It assigns each event instance to one or more context partitions . A context may have one or more context dimensions. Temporal Spatial State Oriented Segmentation Oriented

50. Context Types Examples Spatial State Oriented Temporal Context “ Every day between 08:00 and 10:00 AM” “ A week after borrowing a disk” “ A time window bounded by TradingDayStart and TradingDayEnd events” “ 3 miles from the traffic accident location” “ Within an authorized zone in a manufactory” “ All Children 2-5 years old” “ All platinum customers” “ Airport security level is red” “ Weather is stormy” Segmentation Oriented

51. Context Types Context Fixed location Entity distance location Event distance location Spatial State Oriented Fixed interval Event interval Sliding fixed interval Sliding event interval Temporal Segmentation Oriented

52. The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications

54. Existing Styles for EP languages (samples) Inference Rules State oriented Agent Oriented Imperative/ Script Based RuleCore Oracle Aleri Streambase Esper EventZero TIBCO WBE Apama Spade AMiT Netcool Impact * - if we add simple and mediated event processing the picture is even more diversified ECA Rules SQL extension Coral8 Agent Logic Starview XChange EQ Prova

55. StreamBase Studio

56. StreamBase Pattern Matching

57. CCL Studio (Coral8  Sybase)

59. Microsoft Streaminsights var topfive = (from window in inputStream.Snapshot() from e in window orderby e.f ascending, e.i descending select e).Take(5); var avgCount = from v in inputStream group v by v.i % 4 into eachGroup from window in eachGroup.Snapshot() select new { avgNumber = window.Avg(e => e.number) };

60. Esper EPL – FFD Example /* * Not delivered up after 10 mins (600 secs) of the request target delivery time */ insert into AlertW(requestId, message, driver, timestamp) select a.requestId, "not delivered", a.driver, current_timestamp() from pattern[ every a=Assignment  (timer:interval(600 + (a.deliveryTime-current_timestamp)/1000) and not DeliveryConfirmation(requestId = a.requestId) and not NoOneToReceiveMSG(requestId = a.requestId)) ];

62. Amit Terminology Situation Conditions Lifespan e5 e8 Initiator Terminator Event Selection Actions Operation Keys e1 e3 Input events Operator Joining Counting Temporal Absence Aggregation Notifications Messages Definition updates User plug-ins e2

63. Amit - Situation

64. IBM Websphere Business Events

65. Apama EPL – FFD Examples

66. Apama Simulation Studio – cont.

69. Performance objectives

70. Optimizations Optimizations related to EPA assignment: partition, parallelism, Distribution, load balancing. Optimization related to the execution process: scheduling, routing optimizations and load shedding Optimizations related to the coding of specific EPAs: code optimization, state management.

71. Putting derived events in order

75. The future: Trends and research challenges

78. Trend II: Going from monolithic to diversified Variety of functions Variety of Quality of Service requirements: Variety of platforms “ One size fits all” will not work – Instead a collection of building blocks that can fit together

80. Trend IV: Going from programmer centered to semi-technical person centered Source: ebizQ Event processing market pulse

82. Trend VI: Going from reactive to proactive

83. Emerging directions: Four directions to observe Multiple platforms – same look and feel The engineering of constructing EP applications Adding intelligence to Event processing Tailor-made optimizations

84. Emerging direction I: multiple platforms – same look and feel

86. Emerging direction III: Event processing software engineering Best practices Methodologies Modeling & meta-modeling Design Patterns

87. Emerging direction IV: Intelligent event processing Offline and continuous mining of meaningful patterns in event histories Inexact event processing – handling inexact events and also false positives and false negatives Causality – a key for proactive, but also vital for provenance

88. Summary Event processing has emerged from multiple academic disciplines The state of the Practice is the 1 st generation of products – mainly engineering based Going to the next phase – many challenges that require collaboration of research and industry Already attracted coverage of analysts and all major software vendors The Dagstuhl grand challenge