Benchmarks like LSBench, SRBench, CSRBench and, more recently, CityBench satisfy the growing need of shared datasets, ontologies and queries to evaluate window-based RDF Stream Processing (RSP) engines. However, no clear winner emerges out of the evaluation. In this paper, we claim that the RSP community needs to adopt a Systematic Comparative Research Approach (SCRA) if it wants to move a step forward. To this end, we propose a framework that enables SCRA for window based RSP engines. The contributions of this paper are: (i) the requirements to satisfy for tools that aim at enabling SCRA; (ii) the architecture of a facility to design and execute experiment guaranteeing repeatability, reproducibility and comparability; (iii) Heaven – a proof of concept implementation of such architecture that we released as open source –; (iv) two RSP engine implementations, also open source, that we propose as baselines for the comparative research (i.e., they can serve as terms of comparison in future works). We prove Heaven effectiveness using the baselines by: (i) showing that top-down hypothesis verification is not straight forward even in controlled conditions and (ii) providing examples of bottom-up comparative analysis.
Call Girls in Mayapuri Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Heaven Framework for Comparative RSP Engine Research
1. DEIB - Politecnico di Milano
Riccardo Tommasini, Emanuele Della Valle,
Marco Balduini and Daniele Dell’Aglio
Heaven: a framework for systematic
comparative research approach for RSP
engines
2. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
2
Agenda
• Introduction
• Motivation
• Heaven [Contribution]
• Requirements Analysis
• Test Stand Architecture
• Baselines
• Conclusion and Future Works
3. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
3
It’s a Streaming World
5. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
5
Stream Reasoning
Logical real time reasoning on multiple,
heterogeneous, gigantic and inevitably noisy data
streams.
-- E. Della Valle, S. Ceri, F. van Harmelen and H.
Stuckenschmidt, 2010
10. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
10
State of the art RSP Benchmarking
Benchmark
DataStreams &
Ontologies
Queries Metrics
SR Bench ✔ ✔ Feasibility
LS Bench ✔ ✔ Feasibility, Throughput
CSRBench ✔ ✔
Feasibility, Throughput,
Correctness
CityBench ✔ ✔
Feasibility, Throughput,
Memory
No absolute
winner
11. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
11
Domain Specific Benchmark
The goal of a domain specific benchmark is to
foster technological progress by guaranteeing a
fair assessment.
- Jim Gray, The Benchmark Handbook
for Database and Transaction Systems, 1993
12. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
12
A Well-Known Hypothesis
The incremental maintenance of the
materialisation is faster then full re-materialisation
of ontological entailment when content changes
are small enough (e.g. greater than 10%).
15. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
15
Analysis
A. Qualitatively, is there a solution that always
outperforms the others?
B. If no dominant solution can be found, when
does a solution work better than another one?
C. Quantitatively, is there a solution that
distinguishes itself from the others?
D. Why does a solution perform better than another
solution under a certain experimental
condition?
16. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
16
Comparative Research
• It is natively case driven:
• It considers cases as a combination of known properties
• It defines analysis guidelines through baselines
• It is extensively used to analyse complex systems
• It provides layered frameworks to
• systematically examine cases
• identify similarities/differences enabling us to catch more
insights.
17. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
17
Research Question
Can we enable a systematic comparative
research approach (SCRA) for RSP
engines?
18. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
18
Heaven
• A set of requirements to satisfy.
• An architecture for an RSP engine Test Stand.
• Two baseline RSP engine architectures
• A proof-of-concept implementation (open
source)
19. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
19
Requirement Analysis
An Experimental Environment guarantees
Comparability
Repeatability
Reproducibility
On their definitions we eliciting the the
requirements our framework has to satisfy.
20. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
20
Comparability related requirements
[R1] RSP engine agnostic, i.e. independent from
the tested RSP engine.
[R2] Independent from the measured key
performance indicators (KPIs), i.e., the KPIs set
has to be extensible.
[R3] Identify baseline RSP engines, i.e., the
minimal meaningful approaches to realise an RSP
engine.
21. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
21
Reproducibility related requirements
[R4] Data independent, i.e. allowing the usage
of any data stream and any static data.
[R5] Query independent, i.e. allowing the usage
of any query from users’ domains of interest.
22. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
22
Repeatability related requirements
[R6] Minimise the experimental error, i.e., it has
to affect the RSP engine evaluation as little as
possible and in a predictable way.
23. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
23
RSP Experiment Design
is the RSP engine used as subject in the experiment;
is an ontology and any data not subject to change
during the experiment.
is the description of the input data streams:
is the set of continuous queries registered into the
engine
is the set of key performance indicators (KPIs) to
collect.
The result of the execution of an experiment is a
Report that captures the engine dynamics.
E
T
Q
D
K
R
24. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
24
Test Stands (from aerospace engineering)
• Experimental environment
• Systematic evaluation of
complex system
• Black Box evaluation of
complex system
25. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
RSPEngine
< ,Q>
25
E,D,T,Q,KE
Input outputInterface
Interface
T
T QD
Streamer
D
Receiver
Heaven Test Stand Architecture
K
ResultCollector
K
26. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
26
Heaven Test Stand Architecture
27. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
27
RSP Baselines
Simplified RSP engines cases that combine known
properties, i.e. minimal meaningful approaches to
realise an RSP engine.
Pipeline of a Data Stream Management System
(DSMS) and a Reasoner.
30. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
30
RSP Baselines
• 𝞺DF entailment regime
• they exploit absolute time, i.e. their internal
clock can be externally controlled.
• Ensures results correctness even when
overloaded
• Allows to calculate latency of query
response (responsiveness)
31. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
31
Example of Dynamics Comparison
Incremental Baseline
32. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
32
Conclusion
Top-down hypothesis verification, even when an
RSP engines is extremely simple (i.e. the baselines), is
not straight forward.
There is a growing need of comparative analysis.
Heaven enables the systematic execution of
experiments, paving the road to comparative
investigations.
33. ESWC - 2016 - Riccardo Tommasini - @rictomm - DEIB Polimi
ESWC16
33
Future Works
Systematic analysis of existing solutions
A web-based environment where a users can:
• choose one of existing benchmarks (datasets,
queries)
• design experiment
• run them and consult the results online
• compare the results agains the baselines or
existing integrated RSP engines.