SlideShare une entreprise Scribd logo

Ciel universal distributed execution engine

•
•
Save Manos
Save Manos

Presentation of the CIEL framework as described in the paper: "CIEL: a universal execution engine for distributed data-flow computing"

Technologie
1  sur  40
Télécharger pour lire hors ligne
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . CIEL universal distributed execution engine . .. . . Presenter: Emmanouil Dimogerontakis @{AdvDS} EMDC KTH November 6, 2012 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 1/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 2/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Load Balancing Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Load Balancing Transparent Fault Tolerance Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Limitations Task graphs used up to now: Static Acyclic . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Limitations Limitations : Task graphs used up to now: Limited Expressive Power Static Poor Performance Acyclic Insufficient Fault Tolerance . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Overview Figure: Distributed Execution Engines comparison . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 5/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 6/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL WHY Universal? Support same cluster of algorithms as a TM . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL WHY Universal? Support same cluster of algorithms as a TM HOW ? using Dynamic Task Graphs . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL primitives objects references tasks Figure: A Task Grapha a Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 8/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Dynamic Task Graphs Figure: A Dynamic Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 9/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Master & Worker Figure: CIEL Master . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 10/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Master & Worker Figure: CIEL Master Figure: CIEL Worker . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 10/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Architecture Figure: CIEL Architecture . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 11/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 12/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Creating Tasks with Skywriting Figure: Spawning a new task . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 13/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Creating Tasks with Skywriting Figure: Spawning a new task Figure: Dereferencing futures . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 13/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 14/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Optimizations Globally unique identifiers enable memoization Streaming partially written objects between tasks . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 15/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Fault Tolerance Client (no driver program) Worker (periodic heartbeat) Master (persistent logging, secondary masters, object table reconstruction) . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 16/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 17/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Performance Comparison with production system Figure: DistrubutedGrep on Hadoop and Ciel . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 18/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Perfomance of Iterative Algorithm Figure: K-means on Hadoop and Ciel with 20 workers . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 19/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Overheads Figure: Speedup of Binomial Options Pricing Model on 47 workers . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 20/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Future Work Integrate CIEL with existing programming languages Partition master state Explore use of multiple cores (see [5]) Explore use of non-deterministic parallelism (see [3]) . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 21/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: sharing large amounts of data fine-grain parallelization fully automatic parallelism relation algebra environment distributed operating system . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: are really good for : sharing large amounts of writing iterative algorithms data data-dependent control flow fine-grain parallelization using dynamic task graphs fully automatic parallelism transparent fault tolerance relation algebra environment and automatic distribution distributed operating system scaling across hundreds of machines . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: are really good for : sharing large amounts of writing iterative algorithms data data-dependent control flow fine-grain parallelization using dynamic task graphs fully automatic parallelism transparent fault tolerance relation algebra environment and automatic distribution distributed operating system scaling across hundreds of machines Questions ? . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . [1] D.G. Murray. A distributed execution engine supporting data-dependent control flow. PhD thesis, PhD thesis, Univ. of Cambridge, 2011. [2] D.G. Murray and S. Hand. Scripting the cloud with skywriting. In Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, pages 12–12. USENIX Association, 2010. [3] D.G. Murray and S. Hand. Non-deterministic parallelism considered useful. In HotOS XIII, 13th Workshop on Hot Topics in Operating Systems, 2011. [4] D.G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, and S. Hand. Ciel: a universal execution engine for distributed data-flow computing. In Proceedings of the 8th USENIX conference on Networked systems design and implementation, page 9. USENIX Association, 2011. [5] M. Schwarzkopf, D.G. Murray, and S. Hand. Condensing the cloud: running ciel on many-core. . Proceedings of EuroSys SFMA, 2011. . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 23/23
CIEL Skywriting Experiments Part I . . Appendix . .. . . . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 24/23
CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 1/7
CIEL Skywriting Experiments . Hidden slide 1 Figure: Task and Object table maintained in Master node . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 2/7
CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/7
CIEL Skywriting Experiments . Hidden slide 2 Figure: Spawning Tasks1 1 Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/7
CIEL Skywriting Experiments . Hidden slide 3 Figure: Blocking on futures2 2 Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 5/7
CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 6/7
CIEL Skywriting Experiments . Hidden slide 4 Figure: Primary Master Failure . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/7

Recommandé

Performance boosting of discrete cosine transform using parallel programming ...
Performance boosting of discrete cosine transform using parallel programming ...Performance boosting of discrete cosine transform using parallel programming ...
Performance boosting of discrete cosine transform using parallel programming ...IAEME Publication
 
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...INFOGAIN PUBLICATION
 
Virtual machine placement in a virtualized cloud
Virtual machine placement in a virtualized cloudVirtual machine placement in a virtualized cloud
Virtual machine placement in a virtualized cloudiaemedu
 
Mixed Integer Programming: Analyzing 12 Years of Progress
Mixed Integer Programming: Analyzing 12 Years of ProgressMixed Integer Programming: Analyzing 12 Years of Progress
Mixed Integer Programming: Analyzing 12 Years of ProgressIBM Decision Optimization
 
An overview and survey of simulation models
An overview and survey of simulation modelsAn overview and survey of simulation models
An overview and survey of simulation modelsAnjandeep Rai
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalabilitymabuhr
 
HCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionHCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionGennady Baranov
 
How to reduce hosting costs for Redis based applications on Java
How to reduce hosting costs for Redis based applications on JavaHow to reduce hosting costs for Redis based applications on Java
How to reduce hosting costs for Redis based applications on JavaNikita Koksharov
 

Recommandé

Performance boosting of discrete cosine transform using parallel programming ...
Performance boosting of discrete cosine transform using parallel programming ...Performance boosting of discrete cosine transform using parallel programming ...
Performance boosting of discrete cosine transform using parallel programming ...IAEME Publication
 
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...INFOGAIN PUBLICATION
 
Virtual machine placement in a virtualized cloud
Virtual machine placement in a virtualized cloudVirtual machine placement in a virtualized cloud
Virtual machine placement in a virtualized cloudiaemedu
 
Mixed Integer Programming: Analyzing 12 Years of Progress
Mixed Integer Programming: Analyzing 12 Years of ProgressMixed Integer Programming: Analyzing 12 Years of Progress
Mixed Integer Programming: Analyzing 12 Years of ProgressIBM Decision Optimization
 
An overview and survey of simulation models
An overview and survey of simulation modelsAn overview and survey of simulation models
An overview and survey of simulation modelsAnjandeep Rai
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalabilitymabuhr
 
HCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionHCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionGennady Baranov
 
How to reduce hosting costs for Redis based applications on Java
How to reduce hosting costs for Redis based applications on JavaHow to reduce hosting costs for Redis based applications on Java
How to reduce hosting costs for Redis based applications on JavaNikita Koksharov
 
Constraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingConstraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingEray Cakici
 
Edge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsEdge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsDebmalya Biswas
 
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Mickael Istria
 
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...Edge AI and Vision Alliance
 
Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 GovCloud Network
 
Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab
 
Creating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersCreating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersIRJET Journal
 
Thyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyThyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyAltair ProductDesign
 
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Alessandra Bagnato
 
Photo Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesPhoto Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesIRJET Journal
 
Unit i computer aided design
Unit i computer aided designUnit i computer aided design
Unit i computer aided designmanimaran m
 
FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2GovCloud Network
 
Shading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfShading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfAPSanyal1
 
Model Execution and Meta Layout
Model Execution and Meta LayoutModel Execution and Meta Layout
Model Execution and Meta LayoutChristian-Albrechts-Universität zu Kiel
 
Transforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingTransforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingAMD
 
Cray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge SimulationsCray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge Simulationsinside-BigData.com
 
Agile archiecture iltam 2014
Agile archiecture iltam 2014Agile archiecture iltam 2014
Agile archiecture iltam 2014Dani Mannes
 
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Benoit Combemale
 
Ch1
Ch1Ch1
Ch1Rahul prasad
 
Applitools - Cross Browser
Applitools - Cross BrowserApplitools - Cross Browser
Applitools - Cross BrowserApplitools
 
Software Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSoftware Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSave Manos
 
Lock Service with Paxos in Erlang
Lock Service with Paxos in ErlangLock Service with Paxos in Erlang
Lock Service with Paxos in ErlangSave Manos
 

Contenu connexe

Similaire à Ciel universal distributed execution engine

Constraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingConstraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingEray Cakici
 
Edge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsEdge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsDebmalya Biswas
 
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Mickael Istria
 
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...Edge AI and Vision Alliance
 
Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 GovCloud Network
 
Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab
 
Creating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersCreating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersIRJET Journal
 
Thyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyThyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyAltair ProductDesign
 
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Alessandra Bagnato
 
Photo Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesPhoto Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesIRJET Journal
 
Unit i computer aided design
Unit i computer aided designUnit i computer aided design
Unit i computer aided designmanimaran m
 
FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2GovCloud Network
 
Shading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfShading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfAPSanyal1
 
Transforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingTransforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingAMD
 
Cray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge SimulationsCray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge Simulationsinside-BigData.com
 
Agile archiecture iltam 2014
Agile archiecture iltam 2014Agile archiecture iltam 2014
Agile archiecture iltam 2014Dani Mannes
 
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Benoit Combemale
 
Applitools - Cross Browser
Applitools - Cross BrowserApplitools - Cross Browser
Applitools - Cross BrowserApplitools
 

Similaire à Ciel universal distributed execution engine (20)

Constraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingConstraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in Scheduling
 
Edge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsEdge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare Applications
 
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
 
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
 
Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020
 
Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab Modelica conference 20150921
Scilab Modelica conference 20150921
 
Creating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersCreating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and Autoencoders
 
Thyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyThyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis Technology
 
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
 
Photo Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesPhoto Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted Features
 
Unit i computer aided design
Unit i computer aided designUnit i computer aided design
Unit i computer aided design
 
FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2
 
Shading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfShading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdf
 
Model Execution and Meta Layout
Model Execution and Meta LayoutModel Execution and Meta Layout
Model Execution and Meta Layout
 
Transforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingTransforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud Computing
 
Cray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge SimulationsCray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge Simulations
 
Agile archiecture iltam 2014
Agile archiecture iltam 2014Agile archiecture iltam 2014
Agile archiecture iltam 2014
 
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
 
Ch1
Ch1Ch1
Ch1
 
Applitools - Cross Browser
Applitools - Cross BrowserApplitools - Cross Browser
Applitools - Cross Browser
 

Plus de Save Manos

Software Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSoftware Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSave Manos
 
Lock Service with Paxos in Erlang
Lock Service with Paxos in ErlangLock Service with Paxos in Erlang
Lock Service with Paxos in ErlangSave Manos
 
FOSS Licenses: A first attempt
FOSS Licenses: A first attemptFOSS Licenses: A first attempt
FOSS Licenses: A first attemptSave Manos
 
A boring presentation about social mobile communication patterns and opportun...
A boring presentation about social mobile communication patterns and opportun...A boring presentation about social mobile communication patterns and opportun...
A boring presentation about social mobile communication patterns and opportun...Save Manos
 
Man In The Browser
Man In The BrowserMan In The Browser
Man In The BrowserSave Manos
 
P2P-Tuple: Towards a Robust Volunteer Computing Platform
P2P-Tuple: Towards a Robust Volunteer Computing Platform P2P-Tuple: Towards a Robust Volunteer Computing Platform
P2P-Tuple: Towards a Robust Volunteer Computing Platform Save Manos
 
A survey on modifications for unstructured P2P in WMNs .
A survey on modifications for unstructured P2P in WMNs . A survey on modifications for unstructured P2P in WMNs .
A survey on modifications for unstructured P2P in WMNs . Save Manos
 
Intelligent Placement of Datacenter for Internet Services
Intelligent Placement of Datacenter for Internet Services Intelligent Placement of Datacenter for Internet Services
Intelligent Placement of Datacenter for Internet Services Save Manos
 
Network as a Service
Network as a ServiceNetwork as a Service
Network as a ServiceSave Manos
 
RESTful Web Services
RESTful Web ServicesRESTful Web Services
RESTful Web ServicesSave Manos
 
Distributed systems
Distributed systemsDistributed systems
Distributed systemsSave Manos
 

Plus de Save Manos (14)

Software Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSoftware Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network Testbeds
 
Lock Service with Paxos in Erlang
Lock Service with Paxos in ErlangLock Service with Paxos in Erlang
Lock Service with Paxos in Erlang
 
NaaS
NaaSNaaS
NaaS
 
FOSS Licenses: A first attempt
FOSS Licenses: A first attemptFOSS Licenses: A first attempt
FOSS Licenses: A first attempt
 
A boring presentation about social mobile communication patterns and opportun...
A boring presentation about social mobile communication patterns and opportun...A boring presentation about social mobile communication patterns and opportun...
A boring presentation about social mobile communication patterns and opportun...
 
Man In The Browser
Man In The BrowserMan In The Browser
Man In The Browser
 
Apache Mahout
Apache MahoutApache Mahout
Apache Mahout
 
P2P-Tuple: Towards a Robust Volunteer Computing Platform
P2P-Tuple: Towards a Robust Volunteer Computing Platform P2P-Tuple: Towards a Robust Volunteer Computing Platform
P2P-Tuple: Towards a Robust Volunteer Computing Platform
 
A survey on modifications for unstructured P2P in WMNs .
A survey on modifications for unstructured P2P in WMNs . A survey on modifications for unstructured P2P in WMNs .
A survey on modifications for unstructured P2P in WMNs .
 
Intelligent Placement of Datacenter for Internet Services
Intelligent Placement of Datacenter for Internet Services Intelligent Placement of Datacenter for Internet Services
Intelligent Placement of Datacenter for Internet Services
 
Network as a Service
Network as a ServiceNetwork as a Service
Network as a Service
 
Openflow
OpenflowOpenflow
Openflow
 
RESTful Web Services
RESTful Web ServicesRESTful Web Services
RESTful Web Services
 
Distributed systems
Distributed systemsDistributed systems
Distributed systems
 

Dernier

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfPrecisely
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clashcharlottematthew16
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 

Dernier (20)

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 

Ciel universal distributed execution engine

  • 1. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . CIEL universal distributed execution engine . .. . . Presenter: Emmanouil Dimogerontakis @{AdvDS} EMDC KTH November 6, 2012 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 1/23
  • 2. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 2/23
  • 3. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
  • 4. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
  • 5. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Load Balancing Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
  • 6. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Load Balancing Transparent Fault Tolerance Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
  • 7. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Limitations Task graphs used up to now: Static Acyclic . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/23
  • 8. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Limitations Limitations : Task graphs used up to now: Limited Expressive Power Static Poor Performance Acyclic Insufficient Fault Tolerance . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/23
  • 9. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Overview Figure: Distributed Execution Engines comparison . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 5/23
  • 10. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 6/23
  • 11. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL WHY Universal? Support same cluster of algorithms as a TM . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/23
  • 12. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL WHY Universal? Support same cluster of algorithms as a TM HOW ? using Dynamic Task Graphs . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/23
  • 13. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL primitives objects references tasks Figure: A Task Grapha a Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 8/23
  • 14. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Dynamic Task Graphs Figure: A Dynamic Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 9/23
  • 15. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Master & Worker Figure: CIEL Master . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 10/23
  • 16. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Master & Worker Figure: CIEL Master Figure: CIEL Worker . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 10/23
  • 17. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Architecture Figure: CIEL Architecture . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 11/23
  • 18. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 12/23
  • 19. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Creating Tasks with Skywriting Figure: Spawning a new task . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 13/23
  • 20. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Creating Tasks with Skywriting Figure: Spawning a new task Figure: Dereferencing futures . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 13/23
  • 21. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 14/23
  • 22. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Optimizations Globally unique identifiers enable memoization Streaming partially written objects between tasks . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 15/23
  • 23. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Fault Tolerance Client (no driver program) Worker (periodic heartbeat) Master (persistent logging, secondary masters, object table reconstruction) . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 16/23
  • 24. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 17/23
  • 25. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Performance Comparison with production system Figure: DistrubutedGrep on Hadoop and Ciel . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 18/23
  • 26. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Perfomance of Iterative Algorithm Figure: K-means on Hadoop and Ciel with 20 workers . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 19/23
  • 27. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Overheads Figure: Speedup of Binomial Options Pricing Model on 47 workers . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 20/23
  • 28. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Future Work Integrate CIEL with existing programming languages Partition master state Explore use of multiple cores (see [5]) Explore use of non-deterministic parallelism (see [3]) . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 21/23
  • 29. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: sharing large amounts of data fine-grain parallelization fully automatic parallelism relation algebra environment distributed operating system . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
  • 30. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: are really good for : sharing large amounts of writing iterative algorithms data data-dependent control flow fine-grain parallelization using dynamic task graphs fully automatic parallelism transparent fault tolerance relation algebra environment and automatic distribution distributed operating system scaling across hundreds of machines . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
  • 31. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: are really good for : sharing large amounts of writing iterative algorithms data data-dependent control flow fine-grain parallelization using dynamic task graphs fully automatic parallelism transparent fault tolerance relation algebra environment and automatic distribution distributed operating system scaling across hundreds of machines Questions ? . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
  • 32. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . [1] D.G. Murray. A distributed execution engine supporting data-dependent control flow. PhD thesis, PhD thesis, Univ. of Cambridge, 2011. [2] D.G. Murray and S. Hand. Scripting the cloud with skywriting. In Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, pages 12–12. USENIX Association, 2010. [3] D.G. Murray and S. Hand. Non-deterministic parallelism considered useful. In HotOS XIII, 13th Workshop on Hot Topics in Operating Systems, 2011. [4] D.G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, and S. Hand. Ciel: a universal execution engine for distributed data-flow computing. In Proceedings of the 8th USENIX conference on Networked systems design and implementation, page 9. USENIX Association, 2011. [5] M. Schwarzkopf, D.G. Murray, and S. Hand. Condensing the cloud: running ciel on many-core. . Proceedings of EuroSys SFMA, 2011. . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 23/23
  • 33. CIEL Skywriting Experiments Part I . . Appendix . .. . . . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 24/23
  • 34. CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 1/7
  • 35. CIEL Skywriting Experiments . Hidden slide 1 Figure: Task and Object table maintained in Master node . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 2/7
  • 36. CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/7
  • 37. CIEL Skywriting Experiments . Hidden slide 2 Figure: Spawning Tasks1 1 Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/7
  • 38. CIEL Skywriting Experiments . Hidden slide 3 Figure: Blocking on futures2 2 Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 5/7
  • 39. CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 6/7
  • 40. CIEL Skywriting Experiments . Hidden slide 4 Figure: Primary Master Failure . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/7