Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c

2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA 
2013 © Trivadis
Echtzeitverarbeitung von Twitter Streams
mit Oracle Event Processing (OEP) 12c
Guido Schmutz
DOAG Konferenz 2014
19.11.2014 – 15:00 Raum Kiev
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
1

2013 © Trivadis
Guido Schmutz
•  Working for Trivadis for more than 17 years
•  Oracle ACE Director for Fusion Middleware and SOA
•  Co-Author of different books
•  Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
•  Member of Trivadis Architecture Board
•  Technology Manager @ Trivadis
•  More than 25 years of software development
experience
•  Contact: guido.schmutz@trivadis.com
•  Blog: http://guidoschmutz.wordpress.com
•  Twitter: gschmutz
19.11.2014
2

2013 © Trivadis
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (OEP)
4.  Visualization with Oracle Business Activity Monitoring (BAM)
5.  Event Processing in Information Architecture
19.11.2014
3

2013 © Trivadis
Big Data Definition (4 Vs)
19.11.2014
4
+ Time to action ? – Big Data + Event
Processing = Fast Data
Characteristics of Big Data: Its Volume,
Velocity and Variety in combination

2013 © Trivadis
The world is changing …
The model of Generating/Consuming Data has changed ….
Old Model: few companies are generating data, all others are consuming
data
New Model: all of use are generating data, and all of us are consuming
data
19.11.2014
5

2013 © Trivadis
19.11.2014
6

2013 © Trivadis
Internet Of Things – Sensors
are/will be everywhere
There are more devices tapping into the
internet than people on earth
How do we prepare our
systems/architecture for the future?
19.11.2014
7
Source: CiscoSource: The Economist

2013 © Trivadis
AGENDA
1.  Introduction
19.11.2014
8

2013 © Trivadis
Retrieve Tweets and Visualize
19.11.2014
9

2013 © Trivadis
Access to Tweets
19.11.2014
10
Quelle
Source Limitations Cost
Twitter’s Search API 3200 / user
5000 / keyword
180 requests / 15 minutes
free
Twitter’s Streaming API 1%-40% of total volume free
DataSift
none
0.15 -0.20$ /
unit
Gnip none On request

2013 © Trivadis
AGENDA
1.  Introduction
19.11.2014
11

2013 © Trivadis
Oracle Event Processing (OEP) - Engine
Lightweight Java Application Server
•  Full environment for running Java applications
•  Module Framework - OSGi
High Throughput
•  Hundreds of thousands of
events/second
Event Processing Infrastructure
Easy-to-use development
environment
•  Service Framework – Spring DM,
POJO
Multiple-choice VM
•  JRockit or WebLogic RealTime
19.11.2014
12

2013 © Trivadis
Oracle Event Processing Application
Adapter
•  Provide a way for EPN to exchange data with components that are external to network
•  Either an Event Source (inbound) or an Event Sink (outbound)
Processor
•  Contain query code written in CQL and consumes event data from a preceding
channel
•  May also generate new events to an output channel
•  CQL is an extension of SQL
19.11.2014
13

2013 © Trivadis
Oracle Event Processing Application
Bean
Bean
•  Provide a place for logic written in Java
•  Java code can receive events (Event Sink) and/or send events (Event Source)
•  Are implemented using the Spring Framework model or the OEP event bean model
Channel
•  Two types of channel exist: Stream and Relation
CQL Processor
•  Executes Continous Query Language (CQL) queries
•  Used to declaratively describe logic (queries) to be applied on the event stream
19.11.2014
14

2013 © Trivadis
Oracle Event Processing - CQL
CQL is an extension to Structured Query Language (SQL)
•  Same keywords and syntax rules, but additional support for streaming data
event conceptually corresponds to a row in a database table, but
•  one event is always before or after another in time
•  stream is potentially infinite and ever-changing
19.11.2014
15
Data Query
Result Tuples Result Tuples
Traditional Database CEP Engine
Query
Data

2013 © Trivadis
Oracle Event Processing – Visual Development Tools
19.11.2014
16

2013 © Trivadis
Oracle Event Processing – Operation & Management
19.11.2014
17

2013 © Trivadis
Implementation – complete picture
19.11.2014
18
Mention
Extractor
Twitter
Adapter
Counter 
Processor
Hashtag 
Extractor
Author
Extractor
BAM
Tweet
BAM
Counter
Working on my OEP & BAM
demo for the session
”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014
#oep12c #bam12c http://
bit.ly/1xafUZJ
gschmutz
#doag2014
#oep12
#bam12c,2
gschmutz,1
@doag2014,5
#oep12c,1
JMS
JMSTwitter
range 30 seconds 
slide 30 seconds
Sentiment 
Enricher
Working on my OEP & BAM demo for
the session ”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
#bam12c

2013 © Trivadis
1) Creating a Twitter Adapter
19.11.2014
19
Twitter
Adapter
bit.ly/1xafUZJ
Twitter

2013 © Trivadis
2) Send Tweets to BAM
19.11.2014
20
Twitter
Adapter
BAM
Tweet
bit.ly/1xafUZJ
JMSTwitter Sentiment 
Enricher
1

2013 © Trivadis
3) Extract interesting information from Tweet
19.11.2014
21
Mention
Extractor
Twitter
Adapter
Hashtag 
Extractor
Author
Extractor
BAM
Tweet
bit.ly/1xafUZJ
gschmutz
#doag2014
#oep12
JMSTwitter Sentiment 
Enricher
1
#bam12c

2013 © Trivadis
4) Count occurrences within period
19.11.2014
22
Mention
Extractor
Twitter
Adapter
Counter 
Processor
Hashtag 
Extractor
Author
Extractor
BAM
Tweet
BAM
Counter
bit.ly/1xafUZJ
gschmutz
#doag2014
#oep12
#bam12c,2
gschmutz,1
@doag2014,5
#oep12c,1
JMS
JMSTwitter
range 30 seconds 
slide 30 seconds
Sentiment 
Enricher
1
#bam12c

2013 © Trivadis
Implementing in Oracle Event Processing
19.11.2014
23
Mention
Extractor
Twitter
Adapter
Counter 
Processor
Hashtag 
Extractor
Author
Extractor
BAM
Tweet
BAM
Counter
JMS
JMSTwitter
range 30 seconds 
slide 30 seconds
Sentiment 
Enricher

2013 © Trivadis
1) Creating Twitter Adapter –
Connecting to Twitter Stream
19.11.2014
24

2013 © Trivadis
Tweet Event
19.11.2014
25

2013 © Trivadis
Adapter Factory
19.11.2014
26

2013 © Trivadis
Assembly
19.11.2014
27

2013 © Trivadis
Export Adapter to server
19.11.2014
28

2013 © Trivadis
Using Twitter Adapter
19.11.2014
29

2013 © Trivadis
2) Sentiment Enrichment
19.11.2014
30

2013 © Trivadis
3) Sending Tweets to BAM
Using Oracle BAM Enterprise Message Sources (JMS) interface
19.11.2014
31

2013 © Trivadis
19.11.2014
32
3) Sending Tweets to BAM –
Convert event to JMS MapMessage

2013 © Trivadis
Architectural Pattern: Standalone Event Stream
Processing
19.11.2014
4343
Event Processing
(ESP / CEP)
State Store /
Event Store
EnterpriseEventBus
(Ingress)
Event
Cloud
Internet of
Things
Social Media
Streams
Enterprise
EventBus
43
Analytical
Applications
DB
Enterprise
ServiceBus
Business Rule
Management
SystemRules
Event Processing
Result
Store

2013 © Trivadis
Hadoop Big Data
Infrastructure
Architectural Pattern: Event Stream Processing as part
of Lambda Architecture
19.11.2014
4444
Event Processing
(ESP / CEP)
State Store /
Event Store
EnterpriseEventBus
(Ingress)
Event
Cloud
Internet of
Things
Social Media
Streams
Enterprise
EventBus
44
Analytical
Applications
DB
Enterprise
ServiceBus
Event Processing
Map/
Reduce
HDFS Result
Store
Result
Store

2013 © Trivadis
Hadoop Big Data
Infrastructure
Architectural Pattern: Event Stream Processing as part
of “Kappa” Architecture
19.11.2014
4545
Event Processing
(ESP / CEP)
State Store /
Event Store
EnterpriseEventBus
(Ingress)
Event
Cloud
Internet of
Things
Social Media
Streams
45
Analytical
Applications
DB
Enterprise
ServiceBus
Event Processing
ReplayHDFS
Result
Store

2013 © Trivadis
Event
Cloud
Event Processing in modern architecture
19.11.2014
46
Enterprise
Applications
WS
REST
JMS
RDMBS
LocalESB
External Cloud
Service Providers
EnterpriseServiceBus(ESB)
EJB
Event Processing
(ESP / CEP)
State Store /
Event Store
BPM and SOA Platform
Event
REST
Business
Logic/Rules
NoSQL
Analytical Applications
DataAnalytics
Internet of
Things
Event Processing
46
Mobile Apps
DB
Rich (Web)
Client Apps
DB
Social Media
Streams
EnterpriseEventBus
(Ingress)
EnterpriseEventBus
Visualization
Biz Logic Rules
WS
Event
Business Rule 
Management
SystemRules
Processes ACM
HumanWF
Oracle Event Processing
for Java Embedded
Coherence
Oracle NoSQL
Business Activity Monitoring (BAM)
Oracle RDBMS
ServiceBus
SOA Suite
BPM Suite
WebLogicJMS
WebLogicJMSEDN
Oracle Rules
Oracle Event Processing (OEP)
Oracle NoSQLOracle RDBMS
Business Activity  
Monitoring (BAM)
APIGateway
EDN
Oracle Real-Time
Decisions (RTD)

2013 © Trivadis
Questions and answers ...
2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA 
Guido Schmutz
Technology Manager
guido.schmutz@trivadis.com
19.11.2014
47

Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c

Similar to Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c (20)

More from Guido Schmutz

More from Guido Schmutz (20)

Recently uploaded

Recently uploaded (20)

Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c