This slides, co-produced by Angelo Corsaro and Thomas Bernhardt introduce stream processing with OpenSplice and Esper. The slides also provide a basic introduction to OpenSplice DDS and the Esper CEP engine.
3. DDS Overview
...from a Stream Processing Perspective
Angelo CORSARO, Ph.D.
Chief Technology Officer
OMG DDS Sig Co-Chair
PrismTech
angelo.corsaro@prismtech.com
4. Stream Processing [1/3]
! Stream Processing is an architectural style for building
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
systems that operate over continuous (theoretically
infinite) streams of data
! Stream Processing is often reified under one of its many
declinations, such as:
! Reactive Systems
! Signal Processing Systems
! Functional Stream Programming
! Data Flow Systems
5. Stream Processing [2/3]
! The Stream Processing Architecture very
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
naturally models systems reacting to streams of
data and events produced by the external
world, such as the data produced by sensors, a
camera or even the data produced by the
stock exchange.
! Stream Processing Systems usually operate in
real-time over streams and generate in turns
other streams of data providing information on
what is happening or suggesting actions to
perform, such as by stock X, raise alarm Y, or
detected spatial violation, etc.
6. Stream Processing [3/3]
! Stream Processing Systems are typically
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
modeled as collection of modules
communicating via typed data
channels called usually streams
Filter
! Modules usually play one of the Filter
following roles:
Filter
! Sources: Injecting data into the System
! Filters/Actors: Performing some
computation over sources Sink
! Sinks: Consuming the data produced by Stream
the system source
8. Defining Streams
! In abstract terms, a stream is an infinite sequence of
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
data samples of a given type T
! Streams can be further classified in continuous and
discrete streams. Sometimes referred as Behaviors/
Signals and Events
! In this presentation we’ll refer to Continuous Streams
as Data Streams and to Discrete Streams as Event
Streams
9. Data Streams
Data Streams Temp
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
! The value of a Data Stream
is always defined, i.e.
continuous.
! Good examples of a Data
Stream are the value
assumed by a real-world
entity, such as temperature, time
pressure, a price, etc.
10. Event Streams
Event Streams OverheatAlarm
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
! The value of the stream is
defined at precise point in
time, i.e. it is discrete
! Good examples of Event
Streams are events in the real
world, such a violation of a
regulatory compliance, the
time
temperature higher than a
given value, etc.
11. [A Stream Perspective]
What is DDS?
! DDS is a high-performance, real-time, highly-
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
available, fully-distributed, messaging technology
that allows you to define data/event streams and
make them dynamically discoverable
! DDS is equipped with a rich set of QoS providing
control on the key temporal and availability
properties of data
13. DDS Topics [1/2]
“org.opensplice.demo.TTempSensor”
! A Topic defines a stream class/
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
category
! A Topic has associated a user
defined type and QoS
! The Topic name, type and QoS
defines the key functional and
non-functional invariants
! Topics can be discovered or
locally defined struct TempSensor {
long Id; DURABILITY,
float temp; DEADLINE,
float hum;
} PRIORITY,
#pragma keylist TempSensor id …
14. DDS Topics [2/2]
“org.opensplice.demo.TTempSensor”
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
! DDS Topic types can have
associated keys
! Each unique key-value
identifies a unique sub-
stream of values -- called
Topic Instance
struct TempSensor {
long id; DURABILITY,
float temp; DEADLINE,
float hum;
} PRIORITY,
#pragma keylist TempSensor id …
15. “Seeing” Streams
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
id =701
struct TempSensor {
id =809 @key long id;
Topic float temp;
float hum;
};
id =977
Instances Instances
17. Anatomy of a DDS Application
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
Domain val dp = DomainParticipant(0)
Domain
Participant
// Create a Publisher / Subscriber
Publisher p = dp.create_publisher();
Publisher Topic Subscriber
Session Subscriber s = dp.create_subscriber();
// Create a Topic
Topic<TempSensor> t = Gives access to a
dp.create_topic<TempSensor>(“com.myco.TSTopic”) DDS Domain
DataWrter DataReader
Reader/Writers
User Defined for Types
// Create a DataWriter/DataWriter
DataWriter<TempSensor> dw = pub.create_datawriter(t);
DataReader<TempSensor> dr = sub.create_datareader(t);
18. Anatomy of a DDS Application
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
Domain val dp = DomainParticipant(0)
Domain
Participant
// Create a Publisher / Subscriber
val pub = Publisher(dp)
Publisher Topic Subscriber
Session val sub = Subscriber(dp)
// Create a Topic
val topic = Topic[TempSensor](dp,
“org.opensplice.demo.TTempSensor”)
DataWrter DataReader
Reader/Writers
User Defined for Types Pub/Sub
// Create a DataWriter/DataWriter
Abstractions
DataWriter<TempSensor> dw = pub.create_datawriter(t);
DataReader<TempSensor> dr = sub.create_datareader(t);
19. Anatomy of a DDS Application
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
Domain val dp = DomainParticipant(0)
Domain
Participant
// Create a Publisher / Subscriber
val pub = Publisher(dp)
Publisher Topic Subscriber
Session val sub = Subscriber(dp)
// Create a Topic
val topic = Topic[TempSensor](dp,
“org.opensplice.demo.TTempSensor”)
DataWrter DataReader
Reader/Writers for User Defined for Types
// Create a DataWriter/DataWriter
val writer = DataWriter[TempSensor](pub, topic)
val reader = DataReader[TempSensor](sub, topic) Reader/Writer for
application
// Write data defined Topic
val t = new TempSensor ts(101, 25, 40)
writer write ts; Types
20. Data & Event Streams
! DDS does not provide different types for Data/Event Streams.
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
The difference between the two can be made through the
DataReader API by using properly using read/take operations
! DataReader::read
! Reads the value of the stream w/o removing it from the stream. As a result
multiple read can see the last known value of the stream
! DataReader::take
! Takes the value available on the stream (if any yet) and removes it from the
stream
22. Content Filtered Topics
! Content Filtered Topics
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
provide a way of defining a Example:
filters over an incoming // Create a Topic (on default domain)
stream associated with a val topic = Topic[TempSensor](“TTempSensor”)
val ftopic =
given topic ContentFilteredTopic[TempSensor](“CFTempSensor”,
topic,
filter,
! Filters are expressed as the params)
“WHERE” clause of an SQL // - filter is a WHERE-like clause, such as:
// “temp > 20 AND hum > 50”
statement // “temp > %0”
// “temp > hum”
// “temp BETWEEN (%0 AND %1)
! Filters can operate on any //
// - params is the list of parameters to pass to the
attribute of the type // filter expression – if any
associated with the topic
23. Filter Expression Syntax
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
! DDS Filters are condition over
a topic type attributes
! Temporal properties or
causality cannot be
captured via DDS filter
expression
24. History
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
Data older than “n samples ago”
get’s out the window
! DDS provides a way of
controlling data future past
windows through the
History QoS now
The window keeps
the last n data samples
25. [Putting it All Together]
TempSensor Moving Average
object MovingAverageFilter {
def main(args: Array[String]) {
if (args.length < 2) {
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
println("USAGE:ntMovingAverageFilter <window> <filter-expression>")
}
val topic = Topic[TempSensor]("TTempSensor")
val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1))
val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt)
val reader = DataReader[TempSensor](ftopic, rqos)
reader.reactions += {
case e: DataAvailable[_] => {
var average: Float = 0
val window = e[TempSensor].reader.history
window foreach (average += _.temp)
average = average / window.length
println("+--------------------------------------------------------")
println("Moving Average: " + average)
}
}
}
27. Product Organization
Commercial Edition
! No Cost Runtime Licenses
!"#$%&'()*+,--.*/%&01234(*5**677*8&'()0*8303%93:
! Your choice of licensing
! LGPL or Commercial
! Subscription or Perpetual
! Complete DDS Implementation
Commercial
Edition
! Comprehensive Developer and
Deployment Support Options with a
range of Service Level Agreements
Commercial Add-Ons
! Individually licensable technologies
Commercial Add-ons
! Rich ecosystem covering tools,
integration, testing, etc.
28. Key Points So Far
!"#$%&'()*+,-,.*/%&01234(*5**677*8&'()0*8303%93:;
! DDS key abstractions for building stream processing
architectures
! DDS provides some event processing capabilities that
facilitate the development of Stream Processing Filters
! What else can you use for “Stream Processing” in combination
with DDS?
! Let’s have Tom introduce us to the world of CEP and Esper!
43. %#
#!
%#
#!
[insert into insert_into_def]
select select_list
from stream_def [as name] [, stream_def [as name]] [,...]
[where search_conditions]
[group by grouping_expression_list]
[having grouping_search_conditions]
[output output_specification]
[order by order_by_expression_list]
%!#
select acctId, sum(amount)
from Withdrawal.win:time(1 minute)
group by acctId
having sum(amount) 1000
order by acctId asc
9
55. -$
!
!
!#
every login=LoginEvent
- logout=LogOutEvent where timer:within(2 sec)
!
every [2] login=LoginEvent
!
every [2..] login=LoginEvent until logout=LogoutEvent
77. OpenSplice + Esper
!#$%'()*+,-,.*/%01234(*5**677*8'()0*8303%93:;
! Esper Provides an EsperIO framework for plugging-in
new stream transports
! Plugging OpenSplice into Esper is trivial even w/o
relying on the EsperIO framework
! Let’s have a look…
79. iShapes Application
Spotted shapes represent subscriptions
! To explore play with OpenSplice and
Pierced shapes represent publications
Esper, we’ll use the simd-cxx ishapes
!#$%'()*+,-,.*/%01234(*5**677*8'()0*8303%93:;
application
! Three Topics
! Circle, Square, Triangle
! One Type:
struct ShapeType {
string color;
long x;
long y;
long shapesize;
};
#pragma keylist Shapetype color
80. Esper Setup
!#$%'()*+,-,.*/%01234(*5**677*8'()0*8303%93:;
Step 1: Register Topic Types
val config = new Configuration
val ddsConf = new ConfigurationEventTypeLegacy
ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC)
config.addEventType(ShapeType,
classOf[org.opensplice.demo.ShapeType].getName,
ddsConf)
val cep: EPServiceProvider =
EPServiceProviderManager.getDefaultProvider(config)
81. Esper Setup
!#$%'()*+,-,.*/%01234(*5**677*8'()0*8303%93:;
Step 2: Register a Listener for receiving Esper Events
val listener = new UpdateListener {
def update(ne: Array[EventBean], oe: Array[EventBean]) {
ne foreach(e = {
! // Handle the event
})
}
}
83. iShapes FrameRate
!#$%'()*+,-,.*/%01234(*5**677*8'()0*8303%93:;
! Let’s suppose that we wanted to keep under control
the iShapes Frame rate for ech given color
! In Esper this can be achieved with the following
expression:
insert into ShapesxSec
select color, count(*) as cnt
from ShapeType.win:time_batch(1 second)
group by color
84. iShapes Center of Mass
!#$%'()*+,-,.*/%01234(*5**677*8'()0*8303%93:;
! Suppose that we wanted to compute the center of
mass of all the shapes currently displayed over the
last second
! The Esper expression for this would be:
select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg
(y),int), shapesize) as NewShape
from ShapeType.win:time(10 sec)
85. References
!#$%'()*+,-,.*/%01234(*5**677*8'()0*8303%93:;
#1 OMG DDS Implementation #1 Java-Based CEP Engine
Open Source Open Source
www.opensplice.org www.espertech.com
Fastest growing JVM Language Scala API for OpenSplice DDS
Open Source Open Source
www.scala-lang.org code.google.com/p/escalier