This presentation discusses how WSO2 used complex event processing (CEP) and MapReduce based technologies to track and process data from a soccer match as part of the annual DEBS event processing challenge while achieving throughput in excess of 100,000 events/sec.

1. Last Updated: Jan. 2014
VP Platform Evangelism
@cobiacomm on Twitter
http://blog.cobia.net/cobiacomm/
Chris Haddad
Tracking a Soccer
Game with Big Data

2. Big Data Improves Team Performance
๏ Sports Teams
๏ Your Business Team2

3. Improvement Requires A Feedback Loop
3

4. Internet of Things
(IoT) Bridges
Physical and Virtual
Worlds
๏ Sensors, actuators everywhere
๏ Internet connectivity
Improvement = IoT, Apps, and APIs
๏ Visualization Apps
๏ Control APIs

5. DEBS Challenge
• Bridge game play to Internet of Things
• Sensors in shoes, ball, goalie hands
• Analyze Soccer Game
• Spatial and Temporal Processing
• Visualize Game Play
• Activity Heat maps
• Recommend Performance
Improvements
http://srinathsview.blogspot.com/2013/05/solving-debs-2013-grand-challenge-with.html

6. Bridging Game Play to IoT
6
Temporal Tracking
Spatial Tracking
• Each event includes the
location (x,y,z), time stamp,
velocity and acceleration
• Throughput 15,000 events per
second

7. Analyze Game
7
๏ BALL POSSESSION
STATE DIAGRAM
๏ SHOT ON GOAL
STATE DIAGRAM

8. Analyze Game
8
๏ RUNNING LOCATION STATES

9. 9
Solution Architecture

10. Batch Analytic Foundation
10

11. Business Activity Monitor

12. Realizing Real-time Analytics
๏ Processing Data on the fly, while
storing a minimal amount of
information and responding fast
(from <1 ms to few seconds)
๏ Idea of Event streams
๏ A series of events in time
๏ Enabling technologies
๏ Stream Processing (Storm)
๏ Complex Event processing
(Siddhi)

13. Complex Event Processing

15. Complex Event
Processing
Stream Processing
•SQL like language
•Supports powerful
temporal operators
(e.g. windows, event
patterns)
•Focus on speed
•Harder to scale
•e.g. WSO2 CEP,
Streambase, Esper
•Operators connected
in a network, but you
have to write the logic
•Distributed by design
•Focus on reliability (do
not loose messages),
has transactions
•e.g. Storm, S4

16. Complex Event Processing Operators
๏ Filters or transformations (process a single event)
๏ from Ball[v>10] select .. insert into ..
๏ Windows + aggregation (track window of events: time, length)
๏ from Ball#window.time(30s) select avg(v) ..
๏ Joins (join two event streams to one)
๏ from Ball#window.time(30s) as b join Players as p on p.v
< b.v
๏ Patterns (state machine implementation)
๏ from Ball[v>10], Ball[v<10]*,Ball[v>10] select ..
๏ Event tables (map a database as an event stream)
๏ Define table HitV (v double) using .. db info ..

17. Usecase 1: Running Analysis
๏ Detect when speed crosses threshold limits
define partition player by Players .id;
from s = Players [v <= 1 or v > 11] ,
t = Players [v > 1 and v <= 11]+ ,
e = Players [v <= 1 or v > 11]
select s.ts as tsStart , e.ts as tsStop ,s.id as
playerId ,
‘‘trot" as intensity , t [0].v as instantSpeed
,
(e.ts - s.ts )/1000000000 as unitPeriod
insert into RunningStats partition by player;

18. Usecase 2: Ball possession
๏ Ball possession
๏ Defined as possessing the ball from time you hit it until someone else hit it or ball
leaves the ground)

19. Usecase 3: Heatmap of Activity
๏ Show where actions happened (via cells defined by a grid of
64X100 etc.), need updates once every second
๏ Can solved via cell change boundaries, but does not work if one player stays more than
1 sec in the same cell. So need to join with a timer.

20. Usecase 4: Detect kicks on goal
๏ Main Idea: Detect kicks on
the ball, calculate direction
after 1m, and keep giving
updates as long as it is in
right direction

21. Results for DEBS Scenarios

22. Big Data and IoT Changes the
Game
๏ Real-time Status Dashboard
๏ Player game play
๏ Momentum indicators
๏ Predictions about the next move
๏ Improvement Dashboard
๏ Study of game and players effectiveness
๏ Monitor player health and body functions

23. Success Target
Photo by John Trainoron Flickr
http://www.flickr.com/photos/trainor/2902023575/, Licensed

24. Situational Context: Traffic Example
{incidents: [
{impacting: false,
eventCode: 0,
iconURL: "http://...",
lat: 38.743999,
shortDesc: "Exit ramp closed on I 95...",
type: 1,
severity: 0,
ullDesc: "In NEWINGTON exit ramp closed
on I-95 ...",
startTime: "2010-02-21T00:14:07",
lng: -77.188004,
id: "368598263",
endTime: "2010-02-27T05:04:19"
},
Image source: http://www.directoryofnewyorkcity.com/blog/2009/05/how-to-find-parking-in-new-york-
city/

25. Accelerate
interactions
inside and
outside the
organization
Reduce
interaction
friction and cost
Increase
engagement and
enhance
productivity
Sense
business activity
and
automatically
adapt
Become a More Connected Business
http://wso2.com/landing/enabling-the-connected-business

26. Connected Business
Reference Architecture

27. Conceptual Architecture

28. CEP High Availability

29. Siddhi Storm Bolt (next Major Release)
๏ We have written a Siddhi bolt that would let users run
distributed Siddhi Queries using Storm
SiddhiBolt siddhiBolt1 = new SiddhiBolt( .. siddhi queries .. );
SiddhiBolt siddhiBolt2 = new SiddhiBolt( .. siddhi queries .. );
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("source", new PlayStream(), 1);
builder.setBolt("node1", siddhiBolt1, 1)
.shuffleGrouping("source", "PlayStream1");
..
builder.setBolt("LeafEacho", new EchoBolt(), 1)
.shuffleGrouping("node1", "LongAdvanceStream");
..
cluster.submitTopology("word-count", conf, builder.createTopology());

30. 30

31. Contact us !

32. 32
About the Presenter
๏ Chris Haddad
๏ VP Platform Evangelism
๏ Learn more about me
๏ www.linkedin.com/in/cobiacomm/
๏ Follow me
๏ @cobiacomm on Twitter
๏ http://blog.cobia.net/cobiacomm
๏ On Google+ too