call girls in Kamla Market (DELHI) 🔝 >༒9953330565🔝 genuine Escort Service 🔝✔️✔️
Why Workflows Break
1. Why Workflows Break - Understanding and Combating
Decay in Taverna Workflows
Jun Zhao, Jose Manuel Gomez-Perez, Khalid Belhajjame, Graham Klyne,
Esteban Garcia-Cuesta, Aleix Garrido, Kristina Hettne, Marco Roos,
David De Roure, and Carole Goble
IEEE eScience 2012. Chicago, USA 10 October, 2012
http://www.flickr.com/photos/sheepies/3798650645/ @ CC BY-NC 2.0
2. Reproducibility: Why Bother?
◉ Results produced by scientists not only give insight,
they lead to progress and are built upon
◉ Therefore, the ability to test results is important
◉ In natural sciences, when a scientist claims an
experimental result, then others scientist should be
able to check it.
◉ This should be also possible for experiments carried
out in computational environments.
IEEE eScience 2012. Chicago, USA 10 October, 2012
3. 47 of 53
“landmark”
publications
could not be
replicated
Inadequate cell lines
and animal models
Nature, 483, 2012
Credit to Carole Goble JCDL 2012 Keynote
4. Reproducibility: Why Bother?
◉ Results produced by scientists not only give insight,
they lead to progress and are built upon
◉ Therefore, the ability to test results is important
◉ In natural sciences, when a scientist claims an
experimental result, then other scientists should be
able to check it.
◉ This should be also possible for experiments carried
out in computational environments.
IEEE eScience 2012. Chicago, USA 10 October, 2012
5. Reproducibility: Why Bother?
◉ Results produced by scientists not only give insight,
they lead to progress and are built upon
◉ Therefore, the ability to test results is important
◉ In natural sciences, when a scientist claims an
experimental result, then other scientists should be
able to check it.
◉ This should be also possible for experiments carried
out in computational environments.
IEEE eScience 2012. Chicago, USA 10 October, 2012
6. A famous quote
An article about computational science in a scientific
publication is not the scholarship itself, it is merely
advertising of the scholarship. The actual
scholarship is the complete software
development environment and the complete
set of instructions which generated the figures.
Jon B. Buckheit and David L. Donoho,
WaveLab and reproducible research,
1995
IEEE eScience 2012. Chicago, USA 10 October, 2012
7. Another quote
Abandoning the habit of secrecy in favor of
process transparency and peer review was the
crucial step by which alchemy became
chemistry.
Eric S. Raymond, The art of UNIX
programming, 2004
IEEE eScience 2012. Chicago, USA 10 October, 2012
8. Workflows: A Means for
Preserving Scientific Methods
Fortunately, there is a means that can be used to document
the experiment that the scientist ran, and even re-run it!
chromosome17 chromosome37
Scientific workflows
Kegg pathway
Kegg pathway Kegg pathway
Kegg pathway
query
query query
query
Increasingly adopted in modern sciences.
Transparent documentation of
Detect common
Detect common
pathways
pathways
experimental methods
Common pathways
Repeatable and configurable
IEEE eScience 2012. Chicago, USA 10 October, 2012
9. Workflow Decay
A decayed or reduced ability to be executed or
produce the same results
Our Contributions
An empirical analysis for identifying and
categorizing the causes of workflow decay
A software framework to assess workflow
preservation
10. Storyline
The importance of reproducibility
Workflow as a means for preserving scientific methods
Understanding the causes of workflow decay
Combating decay
Lessons learnt and future work
IEEE eScience 2012. Chicago, USA 10 October, 2012
11. Understanding The Causes of
Workflow Decay
We adopted an empirical approach
To identify the causes of workflow decay
To quantify their severity
To do so, we analyzed a sample of real
workflows to determine if they suffer from
decay and the reasons that caused their decay
IEEE eScience 2012. Chicago, USA 10 October, 2012
12. Experimental Setup
Taverna workflows from Software environment
myExperiment.org Taverna 2.3
Taverna 1
Taverna 2
Experiment metadata
June-July 2012
Selection process 4 researchers
By the creation year
By the creator
By the domain
IEEE eScience 2012. Chicago, USA 10 October, 2012
13. Analyzed Workflows
Number of Taverna 1 workflows from 2007 to 2011
2007 2008 2009 2010 2011
Tested 12 10 10 10 4*
Total 74 341 101 26 13
Number of Taverna 2 workflows from 2009 to 2012
2009 2010 2011 2012
Tested 12 10 15 9
Total 97 308 289 184
IEEE eScience 2012. Chicago, USA 10 October, 2012
14. Profile of Analyzed Workflows
IEEE eScience 2012. Chicago, USA 10 October, 2012
15. The Proportion of Decay
Taverna 1
75% of the 92 tested
workflows failed to be
either executed or
produce the same result
(if testable)
Those from early years
Taverna 2 (2007-2009) had 91%
failure rate
IEEE eScience 2012. Chicago, USA 10 October, 2012
16. The Cause of Decay
Manual analysis
By the validation report from Taverna workbench
By interpreting experiment results reported by Taverna
Identified 4 categories of causes
Missing example data
Missing execution environment
Insufficient descriptions about workflows
Volatile third-party Resources
Other unconsidered possible factors
Changes in the local operating environment (hardware, OS,
middleware, compiler, etc)
IEEE eScience 2012. Chicago, USA 10 October, 2012
17. Decay Caused by Third-Party
Causes
Resources Examples
Refined Causes
Third party resources Underlying dataset, particularly those Researcher hosting the data changed
are not available locally hosted in-house dataset, is no institution, server is no longer available
longer available
Services are deprecated DDBJ web services are not longer
provided despite the fact that they are
used in many myExperiment
workflows
Third party resources Data is available but identified using Due to scalability reasons the input
are available but not different IDs than the ones known to data is superseded by new one making
accessible the user the workflow not executable or
providing wrong results
Data is available but permission, Cannot get the input, which is a
certificate, or network to access it is security token that can only be
needed obtained by a registered user of
ChemiSpider
Services are available but need The security policies of the execution
permission, certificate, or network to framework are updated due to new
access and invoke them hosting institution rules
Third party resources Services are still available by using the The web services are updated
have changed same identifiers but their functionality
have changed IEEE eScience 2012. Chicago, USA 10 October, 2012
18. The Cause of Decay
Manual analysis
By the validation report from Taverna workbench
By interpreting experiment results reported by Taverna
Identified 4 categories of causes
Missing example data
Missing execution environment
Insufficient descriptions about workflows
Volatile third-party Resources
Other unconsidered possible factors
Changes in the local operating environment (hardware, OS,
middleware, compiler, etc)
IEEE eScience 2012. Chicago, USA 10 October, 2012
19. Summary of Decay Causes
50% of the decay was caused by
volatility of 3rd-party resource
Unavailable
Inaccessible
Updated
Missing example data
Unable to re-run
Missing execution environment
Such as local plugins
Insufficient metadata
Such as any required
dependency libraries or
permission information
IEEE eScience 2012. Chicago, USA 10 October, 201
20. Storyline
The importance of reproducibility
Workflow as a means for preserving scientific methods
Understanding the causes of workflow decay
• Combating decay
• Lessons learnt and future work
IEEE eScience 2012. Chicago, USA 10 October, 2012
21. Combating Workflow Decay
• Objective: To provide enough information to
– Prevent decay
– Detect decay
– Repair decay
• Approach: Research Objects + Checklists
– Research Objects [1][2]: Aggregate workflow specifications
t
o jec together with auxiliary elements, such as example data inputs,
Pr annotations, provenance traces that can be used to prevent
ver
f4E decay and/or repair the workflow in case of decay.
W
– Checklists: to check that sufficient information is preserved
along with the workflows
[1] http://wf4ever.github.com/ro/
[2] http://wf4ever.github.com/ro-primer/ IEEE eScience 2012. Chicago, USA 10 October, 2012
22. Checklists
• Checklists are a well established tool
for guiding practices to ensure safety,
quality and consistency in the conduct
of complex operations.
• They have been adopted by the
biological research community to
promote consistency across research
datasets
• In our case, we use checklists to
assess if a research object contains
sufficient information for running the
workflow and checking that its results
are replicable.
IEEE eScience 2012. Chicago, USA 10 October, 2012
23. Cheklist-ing the Reproducibility
of a Workflow
The Minim model used in our approach is an adaptation of the MiM model [1][2].
[1] Matthew Gamble, Jun Zaho, Graham Klyne and Carole Goble. MIM: A Minimum Information Model Vocabulary and
Framework for Scientific Linked Data. eScience 2012
[2] https://raw.github.com/wf4ever/ro-manager/master/src/iaeval/Minim/minim.rdf
IEEE eScience 2012. Chicago, USA 10 October, 2012
24. Use Case
• 4 myExperiment packs
– 2 from genomics, 1 from geography, and 1 domain-neutral
• Experiment process:
– Transform them into RO
– Create checklist descriptions
• Observations
– 2 research objects were found not to contain the necessary
information to run them, 2 others failed because of update to
third party resources and environment of execution.
IEEE eScience 2012. Chicago, USA 10 October, 2012
25. Storyline
The importance of reproducibility
Workflow as a means for preserving scientific methods
Understanding the causes of workflow decay
• Combating decay
• Lessons Learnt and future work
IEEE eScience 2012. Chicago, USA 10 October, 2012
26. Lessons Learnt
1. Dependency is the root enemy of reproducible
workflows
2. Documentation, i.e., annotation, is vital
3. Documentation should be easy to create
IEEE eScience 2012. Chicago, USA 10 October, 2012
27. The Future Work
• Decay detection, explanation, and repair
• Reproducibility and provenance
• Working with scientists is vital for reproducible science
– GigaScience
– BioVel
– 2020 Science
IEEE eScience 2012. Chicago, USA 10 October, 2012
28. Acknowledgement
EU Wf4Ever project (270129)
funded under EU FP7 (ICT- 2009.4.1).
(http://www.wf4ever-project.org)
The principles of provenance. Dagstuhl, March 1, 2012
Notes de l'éditeur
So why do we bother about reproducibility. Because results that scientists reach are not only insights. They are used to ensure progress in practice. Moreover, their results is built upon by other scientists to reach new results. Therefore, testing claimed results is crucial for science to be self-correcting. ============ Important scientific results not only give insight but also lead to practical progress. The ability to test results is crucial for science to be self-correcting. A hallmark of the scientific method is that experiments should be described in enough detail that they can be repeated and perhaps generalized. The idea in natural science is that if a scientist claims an experimental result, then another scientist should be able to check it. Similarly, in a computational environment, it should be possible to repeat a computational experiment as the authors have run it or to change the experiment to see how robust the authors’ conclusions are to changes in parameters or data (a concept called workability).
As an example, In this article, a researcher has found that many basic studies on cancer are unreliable, with grim consequences for producing new medicines in the future. This supports the need for testing research results. http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 "landmark" publications -- papers in top journals, from reputable labs -- for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development. Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature.
In natural sciences, when a scientists claims an experimental result, then another scientists should be able to check it. This should also be possible in a computational environment to test the experiment as the authors have run it, or even change the experiment to see how robust the conclusions reached by the authors.
In natural sciences, when a scientists claims an experimental result, then other scientists should be able to check it. This should also be possible in a computational environment to test the experiment as the authors have run it, or even change the experiment to see how robust the conclusions reached by the authors. To do so, we need information that documents the experiment and specify the computational environment in which it was ran.
I am quoting these two quotes to underline the fact that the experiment results are not enough and that we need information about the process whereby such results were produced.
This is partly witnessed by existing workflow repositories, notably myExperiment and crowdLab, which provide scientists with the means to store, share, publish and reuse workflows. One of the good features of scientific workflows vis a vis reproducibility is that they are repeatable and configurable. Well in principle. Unfortunately, our experience suggests that workflows are likely to suffer over time from decay hindering or reducing the ability to execute them and reproduce the same results.
Analyse a sample of real workflows to determine if they suffer from decay and the reasons that caused their decay
C. Missing execution environment The execution of a workflow may rely on a particular local execution environment, for example, a local R server or a specific version of workflow execution software. Some of our test workflows exhibit this type of decay. Taverna often provides sufficient information about missing libraries, and sometimes workflow descriptions provide a warning about the requirement for a specific library. This type of decay appears to be fixable by installing the missing software, albeit requiring some effort. D. Insufficient descriptions about workflows Sometimes a workflow workbench cannot provide sufficient information about what caused the failure of a workflow run. Additional descriptions in the workflow can play an important role in assisting users reusing the workflows to understand the purpose of the workflow and its expected outcomes.
C. Missing execution environment The execution of a workflow may rely on a particular local execution environment, for example, a local R server or a specific version of workflow execution software. Some of our test workflows exhibit this type of decay. Taverna often provides sufficient information about missing libraries, and sometimes workflow descriptions provide a warning about the requirement for a specific library. This type of decay appears to be fixable by installing the missing software, albeit requiring some effort. D. Insufficient descriptions about workflows Sometimes a workflow workbench cannot provide sufficient information about what caused the failure of a workflow run. Additional descriptions in the workflow can play an important role in assisting users reusing the workflows to understand the purpose of the workflow and its expected outcomes.
Based on our findings, we bundle workflow specifications together with auxiliary information for mitigate its decay, e.g., example data inputs, annotations describing the workflow, provenance traces. The resulting aggregation is term Research Object, which abstractions that are used for workflow preservation. These are designed and tooling is developed around in the context of the Wf4Ever project To verify that a given research object provides the information necessary for preserving a workflow against decay, we adopt a checklist-based approach. Checklists are a well- established tool for guiding practices to ensure safety, quality and consistency in the conduct of complex operations [11], More recently, they have been adopted by the biological research community to promote consistency across research datasets [33] Checklists are well established tool for guiding practices to ensure safety, quality and consistency to conduct complex operations. In our cases, we use them to specify the minimum information needed to prevent workflow decay.
The checklists are expressed using the Minim model