The document discusses quality of data-aware data analytics workflows. It begins with outlining the topics to be covered, which include data analytics workflows structures and systems, issues with quality of data-aware workflows, and quality of data-aware simulation workflows. It then provides examples of different workflow systems and frameworks for data analytics workflows. Key points discussed are the need to understand hierarchical workflow structures, addressing data and service concerns, importance of quality of data for data analytics workflows, and approaches to modeling quality of data metrics and optimizing workflows based on quality of data.

1. Quality of data-aware data analytics
workflows
Hong-Linh Truong
Distributed Systems Group,
Vienna University of Technology
truong@dsg.tuwien.ac.at
http://dsg.tuwien.ac.at/staff/truong
1ASE Summer 2014
Advanced Services Engineering,
Summer 2014
Advanced Services Engineering,
Summer 2014

2. Outline
 Data analytics workflows – structures and
systems
 Issues on Quality of data aware data analytics
workflows
 Quality of data aware simulation workflows
ASE Summer 2014 2

3. Data analytics workflows
ASE Summer 2014 3
Things
People
DaaSDaaS
Computation
Service
Computation
Service
We use the term „workflow“ in a
generic meaning!!!

4. Different views of (data analytics)
workflow systems
4
View
Domain
view
Business
Workflow
Scientific/E-
science
Workflow
Data/Computati
on view
Data
intensive
workflow
Computatio
n intensive
workflow
Human-
intensive
workflow
System
view
Grid
workflow
Enterprise
workflow
Cloud-
based
workflow
Execution
model
view
Service-
based
workflow
Batch job
workflow
Interactive
workflow
ASE Summer 2014

5. Pros and cons of (data analytics)
workflow systems
ASE Summer 2014 5
 Ian J. Taylor, Ewa Deelman, Dennis B. Gannon, and Matthew Shields. 2006. Workflows for E-Science: Scientific
Workflows for Grids. Springer-Verlag New York, Inc., Secaucus, NJ, USA.
 Bertram Ludäscher, Mathias Weske, Timothy M. McPhillips, Shawn Bowers: Scientific Workflows: Business as
Usual? BPM 2009: 31-47
 Mirko Sonntag, Dimka Karastoyanova, Frank Leymann: The Missing Features of Workflow Systems for Scientific
Computations. Software Engineering (Workshops) 2010: 209-216
 Lavanya Ramakrishnan and Beth Plale. 2010. A multi-dimensional classification model for scientific workflow
characteristics. In Proceedings of the 1st International Workshop on Workflow Approaches to New Data-centric
Science (Wands '10). ACM, New York, NY, USA, , Article 4 , 12 pages. DOI=10.1145/1833398.1833402
http://doi.acm.org/10.1145/1833398.1833402
 Jia Yu and Rajkumar Buyya. 2005. A taxonomy of scientific workflow systems for grid computing. SIGMOD Rec. 34,
3 (September 2005), 44-49. DOI=10.1145/1084805.1084814 http://doi.acm.org/10.1145/1084805.1084814
 Ian J. Taylor, Ewa Deelman, Dennis B. Gannon, and Matthew Shields. 2006. Workflows for E-Science: Scientific
Workflows for Grids. Springer-Verlag New York, Inc., Secaucus, NJ, USA.
 Bertram Ludäscher, Mathias Weske, Timothy M. McPhillips, Shawn Bowers: Scientific Workflows: Business as
Usual? BPM 2009: 31-47
 Mirko Sonntag, Dimka Karastoyanova, Frank Leymann: The Missing Features of Workflow Systems for Scientific
Computations. Software Engineering (Workshops) 2010: 209-216
 Lavanya Ramakrishnan and Beth Plale. 2010. A multi-dimensional classification model for scientific workflow
characteristics. In Proceedings of the 1st International Workshop on Workflow Approaches to New Data-centric
Science (Wands '10). ACM, New York, NY, USA, , Article 4 , 12 pages. DOI=10.1145/1833398.1833402
http://doi.acm.org/10.1145/1833398.1833402
 Jia Yu and Rajkumar Buyya. 2005. A taxonomy of scientific workflow systems for grid computing. SIGMOD Rec. 34,
3 (September 2005), 44-49. DOI=10.1145/1084805.1084814 http://doi.acm.org/10.1145/1084805.1084814

6. Hierarchical view of workflows (1)
mProject1Service.java
public void mProject1() {
}
mProject1Service.java
public void mProject1() {
}
WorkflowWorkflow
A();A();
<parallel>
</parallel>
<parallel>
</parallel>
Workflow Region nWorkflow Region n
Activity mActivity m
Invoked Application mInvoked Application m
Code
Region 1
Code
Region 1
Code
Region q
Code
Region q
Code
Region …
Code
Region …
<activity name="mProject1">
<executable name="mProject1"/>
</activity>
<activity name="mProject1">
<executable name="mProject1"/>
</activity>
<activity name="mProject2">
<executable name="mProject2"/>
</activity>
<activity name="mProject2">
<executable name="mProject2"/>
</activity>
while () {
...
}
while () {
...
}
Hong Linh Truong, Schahram Dustdar, Thomas Fahringer:
Performance metrics and ontologies for Grid workflows. Future
Generation Comp. Syst. 23(6): 760-772 (2007)
Hong Linh Truong, Schahram Dustdar, Thomas Fahringer:
Performance metrics and ontologies for Grid workflows. Future
Generation Comp. Syst. 23(6): 760-772 (2007)
ASE Summer 2014 6

7. Representing and programming
data analytics workflows
 Programming languages
 General- and specific-purpose programming
languages, such as Java, Python, Swift
 Programming models, such as MapReduce, Hadoop,
Complex event processing
 Descriptive languages
 BPEL and several languages designed for specific
workflow engines
 They can also be combined
7ASE Summer 2014

8. Data analytics workflow execution
models
ASE Summer 2014 8
Data analytics
workflows
Data analytics
workflows Execution EngineExecution Engine
Local SchedulerLocal Scheduler
jobjob jobjob jobjob jobjob
Web
serviceWeb
serviceWeb
service
Web
service
People

9. Data analytics workflow execution
models
ASE Summer 2014 9
Data analytics
workflows
Data analytics
workflows
Execution EngineExecution Engine
Service
unit
Local
input
data
Analytics
Results
Web service
MapReduce/Hadoop
Sub-Workflow
MPI
Other solutions
Servers/Cloud/Cluster
How data is
transferred among
service units?
How data is
transferred among
service units?

10. Examples of systems and
frameworks for data analytics
workflows
ASE Summer 2014 10
ASKALONASKALON
KEPLERKEPLER
TAVERNATAVERNA
TRIDENTTRIDENT
Apache ODE +
WS-BPEL
Apache ODE +
WS-BPEL
PegasusPegasus
JOperaJOperaADEPTADEPT
MapReduce/HadoopMapReduce/Hadoop
SwiftSwiftRR

11. Some examples (1)
ASE Summer 2014 11
Source: Gideon Juve, Ewa Deelman, G. Bruce Berriman, Benjamin P. Berman, Philip Maechling: An Evaluation of the
Cost and Performance of Scientific Workflows on Amazon EC2. J. Grid Comput. 10(1): 5-21 (2012)
Source: Gideon Juve, Ewa Deelman, G. Bruce Berriman, Benjamin P. Berman, Philip Maechling: An Evaluation of the
Cost and Performance of Scientific Workflows on Amazon EC2. J. Grid Comput. 10(1): 5-21 (2012)

12. Some examples (2)
ASE Summer 2014 12
Source: http://www.dps.uibk.ac.at/projects/brokerage/Source: http://www.dps.uibk.ac.at/projects/brokerage/

13. Some examples (3)
ASE Summer 2014 13
Source: Cesare Pautasso, Thomas Heinis, Gustavo Alonso: JOpera: Autonomic Service
Orchestration. IEEE Data Eng. Bull. 29(3): 32-39 (2006)
Source: Cesare Pautasso, Thomas Heinis, Gustavo Alonso: JOpera: Autonomic Service
Orchestration. IEEE Data Eng. Bull. 29(3): 32-39 (2006)

14. Some examples (4)
ASE Summer 2014 14
Source: Sudipto Das, Yannis Sismanis, Kevin S. Beyer, Rainer Gemulla, Peter J. Haas, and John McPherson. 2010.
Ricardo: integrating R and Hadoop. In Proceedings of the 2010 ACM SIGMOD International Conference on Management
of data (SIGMOD '10). ACM, New York, NY, USA, 987-998. DOI=10.1145/1807167.1807275
http://doi.acm.org/10.1145/1807167.1807275
Source: Sudipto Das, Yannis Sismanis, Kevin S. Beyer, Rainer Gemulla, Peter J. Haas, and John McPherson. 2010.
Ricardo: integrating R and Hadoop. In Proceedings of the 2010 ACM SIGMOD International Conference on Management
of data (SIGMOD '10). ACM, New York, NY, USA, 987-998. DOI=10.1145/1807167.1807275
http://doi.acm.org/10.1145/1807167.1807275

15. Elastic provisioning for workflows
With cloud computing we can
 Provision a computing system
 E.g., a virtual cloud-based cluster
 Provision a workflow execution platform
 E.g., batch job workflow engine, Hadoop runtime
 Deploy and execute workflows
elastic!
ASE Summer 2014 15

16. WHY DO WE NEED TO KNOW THE HIERARCHICAL
STRUCTURES WELL?
ASE Summer 2014 16
WHICH ASPECTS ARE WELL ADDRESSED W.R.T.
„DATA/SERVICE CONCERNS“

17. 17
Hong Linh Truong, Peter Brunner, Vlad Nae, Thomas Fahringer: DIPAS: A distributed performance analysis service for
grid service-based workflows. Future Generation Comp. Syst. 25(4): 385-398 (2009)
Hong Linh Truong, Peter Brunner, Vlad Nae, Thomas Fahringer: DIPAS: A distributed performance analysis service for
grid service-based workflows. Future Generation Comp. Syst. 25(4): 385-398 (2009)
Well-addressed concerns --
performance
ASE Summer 2014 17

18. Well-addressed concerns –
performance/cost
ASE Summer 2014 18
Source: David Chiu, Sagar Deshpande, Gagan Agrawal, Rongxing Li: Cost and accuracy sensitive dynamic workflow
composition over grid environments. GRID 2008: 9-16
Source: David Chiu, Sagar Deshpande, Gagan Agrawal, Rongxing Li: Cost and accuracy sensitive dynamic workflow
composition over grid environments. GRID 2008: 9-16

19. QUALITY OF DATA IN DATA
ANALYTICS WORKFLOWS
ASE Summer 2014 19

20. Performance and Data Quality
Aspects
20
Data Analytics
Data in
Data out
Executed on
Analytics
Models
uses
Execution time?
Performance Overhead?
Memory Consumption?
Is the data good
enough?
How bad data
impacts on
performance?
Is the data good enough
to be stored and shared?
Data quality metrics and models are
strongly domain-specific
Data quality metrics and models are
strongly domain-specific
Which models should be
used?
ASE Summer 2014 20

21. WHY QOD FOR DATA ANALYTICS
WORKFLOW IS IMPORTANT?
ASE Summer 2014 21

22. Very little support
 Qurator workbench
 “Personal quality models” can be expressed and
embedded into query processors or workflows.
 Assume that quality evidence is presented
 Kepler
 A data quality monitor allows user to specify quality
thresholds.
 Expect that rules can be used to control the execution
based on quality.
ASE Summer 2014 22
P Missier, S M Embury, M Greenwood, A D Preece, & B Jin, Managing Information Quality in e-Science: the Qurator
Workbench, Proc ACM International Conference on Management of Data (SIGMOD 2007), ACM Press, pages 1150-
1152, 2007.
Aisa Na’im, Daniel Crawl,Maria Indrawan, Ilkay Altintas, and Shulei Sun. Monitoring data quality in kepler. In Salim Hariri
and Kate Keahey, editors, HPDC, pages 560–564. ACM, 2010.
P Missier, S M Embury, M Greenwood, A D Preece, & B Jin, Managing Information Quality in e-Science: the Qurator
Workbench, Proc ACM International Conference on Management of Data (SIGMOD 2007), ACM Press, pages 1150-
1152, 2007.
Aisa Na’im, Daniel Crawl,Maria Indrawan, Ilkay Altintas, and Shulei Sun. Monitoring data quality in kepler. In Salim Hariri
and Kate Keahey, editors, HPDC, pages 560–564. ACM, 2010.

23. Research questions
 What are main QoD metrics, what are the relationship between QoD
metrics and other service level objectives, and what are their roles
and possible trade-offs?
 How to support different domain-specific QoD models and link them
to workflow structures?
 How to model, evaluate and estimate QoD associated with data
movement into, within, and out to workflows? When and where
software or scientists can perform automatic or manual QoD
measurement and analysis
 How to optimize the workflow composition and execution based on
QoD specification?
 How does QoD impact on the provisioning of data services,
computational services and supporting services?
ASE Summer 2014 23

24. Approach
ASE Summer 2014 24
Core models, techniques and algorithms to allow
the modeling and evaluating QoD metrics
Core models, techniques and algorithms to allow
the modeling and evaluating QoD metrics
QoD-aware composition and executionQoD-aware composition and execution
QoD-aware service provisioning and
infrastructure optimization
QoD-aware service provisioning and
infrastructure optimization

25. Modeling and evaluating QoD
metrics for data analytics
workflows
ASE Summer 2014 25

26. QoD-aware optimization for data
analytics workflow composition
and execution
ASE Summer 2014 26

27. HOW TO INTEGRATE QOD
EVALUATORS? AND WHICH CONCERNS
NEED TO BE CONSIDERED?
ASE Summer 2014 27

28. QoD metrics evaluation
 Domain-specific metrics
 Need specific tools and expertise for determining
metrics
 Evaluation
 Cannot done by software only: humans are required
 Complex integration model
 Where to put QoD evaluators and why?
 How evaluators obtain the data to be evaluated?
 Impact of QoD evaluation on performance of
data analytics workflows
ASE Summer 2014 28

29. WHAT KIND OF OPTIMIZATION CAN BE
DONE?
ASE Summer 2014 29

30. QoD-aware optimization for data
analytics workflows
 Improving quality of results
 Reducing analytics costs and time
 Enabling early failure detection
 Enabling elasticitiy of services provisioning
 Enabling elastic data analytics support
 Etc.
ASE Summer 2014 30

31. EXAMPLE: QOD-AWARE
SIMULATION WORKFLOWS
ASE Summer 2014 31

32. 32
QoD-aware simulation workflows
Michael Reiter, Hong Linh Truong, Schahram Dustdar, Dimka Karastoyanova, Robert Krause, Frank Leymann, Dieter
Pahr: On Analyzing Quality of Data Influences on Performance of Finite Elements Driven Computational Simulations.
Euro-Par 2012: 793-804
Michael Reiter, Uwe Breitenbücher, Schahram Dustdar, Dimka Karastoyanova, Frank Leymann, Hong Linh Truong: A
Novel Framework for Monitoring and Analyzing Quality of Data in Simulation Workflows. eScience 2011: 105-112
Michael Reiter, Hong Linh Truong, Schahram Dustdar, Dimka Karastoyanova, Robert Krause, Frank Leymann, Dieter
Pahr: On Analyzing Quality of Data Influences on Performance of Finite Elements Driven Computational Simulations.
Euro-Par 2012: 793-804
Michael Reiter, Uwe Breitenbücher, Schahram Dustdar, Dimka Karastoyanova, Frank Leymann, Hong Linh Truong: A
Novel Framework for Monitoring and Analyzing Quality of Data in Simulation Workflows. eScience 2011: 105-112
ASE Summer 2014

33. Hybrid resources needed for
quality evaluation
 Challenges:
 Subjective and objective evaluation
 Long running processes
 Our approach
 Different QoD measurements
 Human and software tasks
33ASE Summer 2014

34. 34
Evaluating quality of data in
workflows
Michael Reiter, Uwe Breitenbücher, Schahram Dustdar, Dimka Karastoyanova, Frank Leymann, Hong Linh Truong: A
Novel Framework for Monitoring and Analyzing Quality of Data in Simulation Workflows. eScience 2011: 105-112
Michael Reiter, Uwe Breitenbücher, Schahram Dustdar, Dimka Karastoyanova, Frank Leymann, Hong Linh Truong: A
Novel Framework for Monitoring and Analyzing Quality of Data in Simulation Workflows. eScience 2011: 105-112
ASE Summer 2014

35. QoD Evaluator
 Software-based QoD evaluators
 Can be provided under libraries integrated into
invoked applications
 Web services-based evaluators
 Human-based QoD evaluators
 Built based on the concept human-based services
 Can be interfaces via Human-Task
 Simple mapping at the moment
 Human resources from clouds/crowds
ASE Summer 2014 35

36. Open issues: quality-of-result
(QoR) driven workflows
 How to support QoR driven analytics?
 Some basic steps
 Conceptualize expected QoR
 Associate the expected QoR with workflow activities
 Use the expected QoR
 to match/select underlying services (e.g., data sources,
cloud IaaS, etc
 Utilize the expected QoR and the measured QoR
and apply elasticity principles for
 Refine the workflow structure
 Provision computation, network and data
ASE Summer 2014 36

37. Exercises
 Read mentioned papers
 Discuss pros and cons of descriptive languages
- and programming languages – based data
analytics workflows
 Examine how QoD evaluators can be integrated
into different programming models for QoD-
aware data analytics workflows
 Implement some QoD evaluators for Hadoop
 Develop techniques for determining places
where QoD evaluators are inserted
ASE Summer 2014 37

38. 38
Thanks for
your attention
Hong-Linh Truong
Distributed Systems Group
Vienna University of Technology
truong@dsg.tuwien.ac.at
http://dsg.tuwien.ac.at/staff/truong
ASE Summer 2014