SlideShare une entreprise Scribd logo
1  sur  20
Télécharger pour lire hors ligne
Apache Web Services in the
 Real World, an E-Science
       Perspective
            Srinath Perera
        Architect, WSO2 Inc.
  Member, Apache Software Foundation
     Lanka Software Foundation
Outline
●   Linked Environment for Atmospheric
    Discovery Project (LEAD), the Use Case.
●   LEAD Architecture, using SOA to build a
    Large Scale E-Science Project.
●   History: LEAD and Apache Web Service
    Projects.
●   Apache as a Sustainability Model for
    Academic Projects.
E-Science
●   Continuation of High Performance Computing,
    Parallel Computing, and Grid.
●   Cyber-infrastructures to support Scientific
    Research.
●   Build around “Computation” as the third Pillar of
    Science (along with Analysis and
    Experimentation).
●   Characterized by wide range of computing (CPU
    minutes to CPU years) and Data (few KB to Pbs
    of data) requirements.
●   Based on Real life usecases.
Reality is Harder than Fiction
●   E-Science joins Theory with Real life data
●   Real Life Applications often go beyond our
    experiences.
    ●   Most Weather models are calculated much less
        than ideal resolutions, otherwise a 24 hour forecast
        takes more than 24 hours !!!
    ●   Physics Usecases (e.g. Large Hadron Collider),
        Telescopes, Genome Analysis generate Tera bytes
        of data in days if not hours, and moving a 1TB
        takes hours even in a 10 GB networks of TeraGrid.
●   Scale, Geographical Distribution of resources,
    Heterogeneity makes these usecases Complex.
Linked Environments for
Atmospheric Discovery (LEAD)
●   U.S. NSF funded, 10+ Universities, 11M $, 5
    Years.
●   Used for U.S. National Weather forecasts by
    NOAA.
●   Presented to U.S. Congress as an example to
    justify Scientific research spendings by U.S.
    NSF.
●   Have brought the state of the art forecasting
    capabilities to wider audience ranging from
    hardcore scientists to high schools students.
LEAD: Dynamic Weather Analysis in
        U.S. Wide Scale
Why is it Hard?
●   Geographically Distributed Sensors, Computing Power,
    Storage, and Expertise.
●   Handling Failures and Recovery
●   Long Running Jobs (> 1 Hour).
●   Large Scale Jobs (10-1000+ processors).
●   Large Sized Data (KBs to GB of data).
●   Need to serve many Parallel Users.
●   Usage Spikes.
LEAD as an Example
●   Assume a Hurricane developed, and 1000
    scientists across U.S. come to LEAD portal to
    run forecasts.
●   Lets assume,
    ●   Each user run 3 workflows.
    ●   Each Workflow has 6 services, generates about 300
        notifications, moves 50 100MB files, generates 50
        100MB files, and runs for one hour.
    ●   Each Service needs 5 CPUs Hours .
Which Means
●   3000 Parallel workflows
●   Need 90,000 CPUs per Hour
●   250 TPS for messaging System
●   Move 8GB/Sec through the network
●   Generate 15TB data per Hour

    LEAD Can not handle these numbers
    yet, but they give us an idea about the
                   challenge.
SOA, E-Science and LEAD
●   E-Science infrastructures are Distributed, Complex,
    and Heterogeneous.
●   SOA is designed to handle just the like.
●   LEAD is based on many SOA Specs
       –   WSDL, SOAP, WS-Addressing for Communication
       –   WS-BPEL for Workflows
       –   WS-Eventing for Messaging
       –   WSDM for service Management
●   LEAD People have closely worked with and
    contributed to Web Services, pushing its limits to
    apply it to LEAD.
LEAD Architecture
Workflow Subsystem
Data Subsystem
Messaging Subsystem
LEAD & Apache WS History
●   Few People from LEAD has been major contributors for
    Apache Axis, and then Axis2.
●   LEAD is not based on Axis2.
●   LEAD is older than Axis2, and it forked off in Axis era,
    mainly because of Async messaging support.
●   Five years ago LEAD implemented many tools (e.g.
    Registries, Async Messaging, Workflow Engine), that are
    hot topics now.
●   Towards the end, LEAD started looking at Axis2 and other
    Apache Projects from a Sustainability Perspective.
●   Most part are already converted, others are being
    converted.
LEAD with Apache Projects
●   LEAD Switched to Apache ODE for workflow
    execution more than a Year ago.
●   LEAD data subsystems switched to Axis2 about a
    Year ago.
●   Job Submission was switched to Axis2 based solution
    few months back.
●   Service Factory is being converted to Axis2 right now.
●   Conversion of Messaging System is in progress
    (Through a Indiana University and LSF collaboration).
Apache as a Sustainability model
         for Research projects
●   Industry values “People”, we (opensource) value “Code”, and
    Academia values “Ideas”.
●   Most NSF Grants, now, ask for a Sustainability Model as part
    of Proposals.
●   One option is a commercial spin off
●   Doing it in a opensource way, building a community and users
    around a project is also a potential Solution.
●   Many Challenges: ownership, need to renounce control, active
    engagement of the community are the key.
    ●   “Source Open” is not good enough!!
    ●   “Dump and Run” does not work either.
Pros & Cons
             Advantages                          Disadvantages

Reach to a wider Audience. Healthy       You have to let go of the
User Community, world debug your         ownership, at least to a some
project for you.                         extent.
Potential Long Lifetime, Self            Need for community Consent
sustaining community if Successful.      might slow you down.

To take advantage of Apache              You have to learn to listen and
Process throughout Project life cycle    explain. Some arguments are
(Releases, SVN, Jira, Wiki, Culture ).   harder to do in a mailing list.
Better Chances of Attracting external    Have to Time Publications.
Developers, more inputs. Better
chance of avoiding “source open”.
Take advantage of Apache
Infrastructure.
Conclusion
●   Wanted to share a Real Life, Large-Scale SOA
    Usecase
●   Wanted to show LEAD-Apache interactions as
    a real Life Case Study of interactions between
    Apache and an Academic Project.
●   Wanted to Showcase Apache as a
    Sustainability Mechanism, if it is done right.
Questions?

Contenu connexe

Similaire à Apache Web Services in the Real World, an E-Science Perspective

IESL Talk Series: Apache System Projects in the Real World
IESL Talk Series: Apache System Projects in the Real WorldIESL Talk Series: Apache System Projects in the Real World
IESL Talk Series: Apache System Projects in the Real WorldSrinath Perera
 
2016 05 sanger
2016 05 sanger2016 05 sanger
2016 05 sangerChris Dwan
 
Big Data HPC Convergence and a bunch of other things
Big Data HPC Convergence and a bunch of other thingsBig Data HPC Convergence and a bunch of other things
Big Data HPC Convergence and a bunch of other thingsGeoffrey Fox
 
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudData-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudOla Spjuth
 
Architecting Agile Data Applications for Scale
Architecting Agile Data Applications for ScaleArchitecting Agile Data Applications for Scale
Architecting Agile Data Applications for ScaleDatabricks
 
Hadoop @ Sara & BiG Grid
Hadoop @ Sara & BiG GridHadoop @ Sara & BiG Grid
Hadoop @ Sara & BiG GridEvert Lammerts
 
Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Ola Spjuth
 
From the South: building together a high-tech ecosystem
From the South: building together a high-tech ecosystemFrom the South: building together a high-tech ecosystem
From the South: building together a high-tech ecosystemNicolás Erdödy
 
AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...
AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...
AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...Amazon Web Services
 
Big Data Analytics Strategy and Roadmap
Big Data Analytics Strategy and RoadmapBig Data Analytics Strategy and Roadmap
Big Data Analytics Strategy and RoadmapSrinath Perera
 
The Netflix Way to deal with Big Data Problems
The Netflix Way to deal with Big Data ProblemsThe Netflix Way to deal with Big Data Problems
The Netflix Way to deal with Big Data ProblemsMonal Daxini
 
Capacity Management and BigData/Hadoop - Hitchhiker's guide for the Capacity ...
Capacity Management and BigData/Hadoop - Hitchhiker's guide for the Capacity ...Capacity Management and BigData/Hadoop - Hitchhiker's guide for the Capacity ...
Capacity Management and BigData/Hadoop - Hitchhiker's guide for the Capacity ...Renato Bonomini
 
Scientific
Scientific Scientific
Scientific marpierc
 
Cloud Programming Models: eScience, Big Data, etc.
Cloud Programming Models: eScience, Big Data, etc.Cloud Programming Models: eScience, Big Data, etc.
Cloud Programming Models: eScience, Big Data, etc.Alexandru Iosup
 
SKA_in_Seoul_2015_NicolasErdody v2.0
SKA_in_Seoul_2015_NicolasErdody v2.0SKA_in_Seoul_2015_NicolasErdody v2.0
SKA_in_Seoul_2015_NicolasErdody v2.0Nicolás Erdödy
 
(Big) Data (Science) Skills
(Big) Data (Science) Skills(Big) Data (Science) Skills
(Big) Data (Science) SkillsOscar Corcho
 
Cytoscape: Now and Future
Cytoscape: Now and FutureCytoscape: Now and Future
Cytoscape: Now and FutureKeiichiro Ono
 
BISSA: Empowering Web gadget Communication with Tuple Spaces
BISSA: Empowering Web gadget Communication with Tuple SpacesBISSA: Empowering Web gadget Communication with Tuple Spaces
BISSA: Empowering Web gadget Communication with Tuple SpacesSrinath Perera
 
Big Data with IOT approach and trends with case study
Big Data with IOT approach and trends with case studyBig Data with IOT approach and trends with case study
Big Data with IOT approach and trends with case studySharjeel Imtiaz
 
How To Maintain Million Lines Of Open Source Code And Remain Sane or The Stor...
How To Maintain Million Lines Of Open Source Code And Remain Sane or The Stor...How To Maintain Million Lines Of Open Source Code And Remain Sane or The Stor...
How To Maintain Million Lines Of Open Source Code And Remain Sane or The Stor...Radovan Semancik
 

Similaire à Apache Web Services in the Real World, an E-Science Perspective (20)

IESL Talk Series: Apache System Projects in the Real World
IESL Talk Series: Apache System Projects in the Real WorldIESL Talk Series: Apache System Projects in the Real World
IESL Talk Series: Apache System Projects in the Real World
 
2016 05 sanger
2016 05 sanger2016 05 sanger
2016 05 sanger
 
Big Data HPC Convergence and a bunch of other things
Big Data HPC Convergence and a bunch of other thingsBig Data HPC Convergence and a bunch of other things
Big Data HPC Convergence and a bunch of other things
 
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudData-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and Cloud
 
Architecting Agile Data Applications for Scale
Architecting Agile Data Applications for ScaleArchitecting Agile Data Applications for Scale
Architecting Agile Data Applications for Scale
 
Hadoop @ Sara & BiG Grid
Hadoop @ Sara & BiG GridHadoop @ Sara & BiG Grid
Hadoop @ Sara & BiG Grid
 
Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...
 
From the South: building together a high-tech ecosystem
From the South: building together a high-tech ecosystemFrom the South: building together a high-tech ecosystem
From the South: building together a high-tech ecosystem
 
AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...
AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...
AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...
 
Big Data Analytics Strategy and Roadmap
Big Data Analytics Strategy and RoadmapBig Data Analytics Strategy and Roadmap
Big Data Analytics Strategy and Roadmap
 
The Netflix Way to deal with Big Data Problems
The Netflix Way to deal with Big Data ProblemsThe Netflix Way to deal with Big Data Problems
The Netflix Way to deal with Big Data Problems
 
Capacity Management and BigData/Hadoop - Hitchhiker's guide for the Capacity ...
Capacity Management and BigData/Hadoop - Hitchhiker's guide for the Capacity ...Capacity Management and BigData/Hadoop - Hitchhiker's guide for the Capacity ...
Capacity Management and BigData/Hadoop - Hitchhiker's guide for the Capacity ...
 
Scientific
Scientific Scientific
Scientific
 
Cloud Programming Models: eScience, Big Data, etc.
Cloud Programming Models: eScience, Big Data, etc.Cloud Programming Models: eScience, Big Data, etc.
Cloud Programming Models: eScience, Big Data, etc.
 
SKA_in_Seoul_2015_NicolasErdody v2.0
SKA_in_Seoul_2015_NicolasErdody v2.0SKA_in_Seoul_2015_NicolasErdody v2.0
SKA_in_Seoul_2015_NicolasErdody v2.0
 
(Big) Data (Science) Skills
(Big) Data (Science) Skills(Big) Data (Science) Skills
(Big) Data (Science) Skills
 
Cytoscape: Now and Future
Cytoscape: Now and FutureCytoscape: Now and Future
Cytoscape: Now and Future
 
BISSA: Empowering Web gadget Communication with Tuple Spaces
BISSA: Empowering Web gadget Communication with Tuple SpacesBISSA: Empowering Web gadget Communication with Tuple Spaces
BISSA: Empowering Web gadget Communication with Tuple Spaces
 
Big Data with IOT approach and trends with case study
Big Data with IOT approach and trends with case studyBig Data with IOT approach and trends with case study
Big Data with IOT approach and trends with case study
 
How To Maintain Million Lines Of Open Source Code And Remain Sane or The Stor...
How To Maintain Million Lines Of Open Source Code And Remain Sane or The Stor...How To Maintain Million Lines Of Open Source Code And Remain Sane or The Stor...
How To Maintain Million Lines Of Open Source Code And Remain Sane or The Stor...
 

Plus de Srinath Perera

Book: Software Architecture and Decision-Making
Book: Software Architecture and Decision-MakingBook: Software Architecture and Decision-Making
Book: Software Architecture and Decision-MakingSrinath Perera
 
Data science Applications in the Enterprise
Data science Applications in the EnterpriseData science Applications in the Enterprise
Data science Applications in the EnterpriseSrinath Perera
 
An Introduction to APIs
An Introduction to APIs An Introduction to APIs
An Introduction to APIs Srinath Perera
 
An Introduction to Blockchain for Finance Professionals
An Introduction to Blockchain for Finance ProfessionalsAn Introduction to Blockchain for Finance Professionals
An Introduction to Blockchain for Finance ProfessionalsSrinath Perera
 
AI in the Real World: Challenges, and Risks and how to handle them?
AI in the Real World: Challenges, and Risks and how to handle them?AI in the Real World: Challenges, and Risks and how to handle them?
AI in the Real World: Challenges, and Risks and how to handle them?Srinath Perera
 
Healthcare + AI: Use cases & Challenges
Healthcare + AI: Use cases & ChallengesHealthcare + AI: Use cases & Challenges
Healthcare + AI: Use cases & ChallengesSrinath Perera
 
How would AI shape Future Integrations?
How would AI shape Future Integrations?How would AI shape Future Integrations?
How would AI shape Future Integrations?Srinath Perera
 
The Role of Blockchain in Future Integrations
The Role of Blockchain in Future IntegrationsThe Role of Blockchain in Future Integrations
The Role of Blockchain in Future IntegrationsSrinath Perera
 
Blockchain: Where are we? Where are we going?
Blockchain: Where are we? Where are we going? Blockchain: Where are we? Where are we going?
Blockchain: Where are we? Where are we going? Srinath Perera
 
Few thoughts about Future of Blockchain
Few thoughts about Future of BlockchainFew thoughts about Future of Blockchain
Few thoughts about Future of BlockchainSrinath Perera
 
A Visual Canvas for Judging New Technologies
A Visual Canvas for Judging New TechnologiesA Visual Canvas for Judging New Technologies
A Visual Canvas for Judging New TechnologiesSrinath Perera
 
Privacy in Bigdata Era
Privacy in Bigdata  EraPrivacy in Bigdata  Era
Privacy in Bigdata EraSrinath Perera
 
Blockchain, Impact, Challenges, and Risks
Blockchain, Impact, Challenges, and RisksBlockchain, Impact, Challenges, and Risks
Blockchain, Impact, Challenges, and RisksSrinath Perera
 
Today's Technology and Emerging Technology Landscape
Today's Technology and Emerging Technology LandscapeToday's Technology and Emerging Technology Landscape
Today's Technology and Emerging Technology LandscapeSrinath Perera
 
An Emerging Technologies Timeline
An Emerging Technologies TimelineAn Emerging Technologies Timeline
An Emerging Technologies TimelineSrinath Perera
 
The Rise of Streaming SQL and Evolution of Streaming Applications
The Rise of Streaming SQL and Evolution of Streaming ApplicationsThe Rise of Streaming SQL and Evolution of Streaming Applications
The Rise of Streaming SQL and Evolution of Streaming ApplicationsSrinath Perera
 
Analytics and AI: The Good, the Bad and the Ugly
Analytics and AI: The Good, the Bad and the UglyAnalytics and AI: The Good, the Bad and the Ugly
Analytics and AI: The Good, the Bad and the UglySrinath Perera
 
Transforming a Business Through Analytics
Transforming a Business Through AnalyticsTransforming a Business Through Analytics
Transforming a Business Through AnalyticsSrinath Perera
 
SoC Keynote:The State of the Art in Integration Technology
SoC Keynote:The State of the Art in Integration TechnologySoC Keynote:The State of the Art in Integration Technology
SoC Keynote:The State of the Art in Integration TechnologySrinath Perera
 

Plus de Srinath Perera (20)

Book: Software Architecture and Decision-Making
Book: Software Architecture and Decision-MakingBook: Software Architecture and Decision-Making
Book: Software Architecture and Decision-Making
 
Data science Applications in the Enterprise
Data science Applications in the EnterpriseData science Applications in the Enterprise
Data science Applications in the Enterprise
 
An Introduction to APIs
An Introduction to APIs An Introduction to APIs
An Introduction to APIs
 
An Introduction to Blockchain for Finance Professionals
An Introduction to Blockchain for Finance ProfessionalsAn Introduction to Blockchain for Finance Professionals
An Introduction to Blockchain for Finance Professionals
 
AI in the Real World: Challenges, and Risks and how to handle them?
AI in the Real World: Challenges, and Risks and how to handle them?AI in the Real World: Challenges, and Risks and how to handle them?
AI in the Real World: Challenges, and Risks and how to handle them?
 
Healthcare + AI: Use cases & Challenges
Healthcare + AI: Use cases & ChallengesHealthcare + AI: Use cases & Challenges
Healthcare + AI: Use cases & Challenges
 
How would AI shape Future Integrations?
How would AI shape Future Integrations?How would AI shape Future Integrations?
How would AI shape Future Integrations?
 
The Role of Blockchain in Future Integrations
The Role of Blockchain in Future IntegrationsThe Role of Blockchain in Future Integrations
The Role of Blockchain in Future Integrations
 
Future of Serverless
Future of ServerlessFuture of Serverless
Future of Serverless
 
Blockchain: Where are we? Where are we going?
Blockchain: Where are we? Where are we going? Blockchain: Where are we? Where are we going?
Blockchain: Where are we? Where are we going?
 
Few thoughts about Future of Blockchain
Few thoughts about Future of BlockchainFew thoughts about Future of Blockchain
Few thoughts about Future of Blockchain
 
A Visual Canvas for Judging New Technologies
A Visual Canvas for Judging New TechnologiesA Visual Canvas for Judging New Technologies
A Visual Canvas for Judging New Technologies
 
Privacy in Bigdata Era
Privacy in Bigdata  EraPrivacy in Bigdata  Era
Privacy in Bigdata Era
 
Blockchain, Impact, Challenges, and Risks
Blockchain, Impact, Challenges, and RisksBlockchain, Impact, Challenges, and Risks
Blockchain, Impact, Challenges, and Risks
 
Today's Technology and Emerging Technology Landscape
Today's Technology and Emerging Technology LandscapeToday's Technology and Emerging Technology Landscape
Today's Technology and Emerging Technology Landscape
 
An Emerging Technologies Timeline
An Emerging Technologies TimelineAn Emerging Technologies Timeline
An Emerging Technologies Timeline
 
The Rise of Streaming SQL and Evolution of Streaming Applications
The Rise of Streaming SQL and Evolution of Streaming ApplicationsThe Rise of Streaming SQL and Evolution of Streaming Applications
The Rise of Streaming SQL and Evolution of Streaming Applications
 
Analytics and AI: The Good, the Bad and the Ugly
Analytics and AI: The Good, the Bad and the UglyAnalytics and AI: The Good, the Bad and the Ugly
Analytics and AI: The Good, the Bad and the Ugly
 
Transforming a Business Through Analytics
Transforming a Business Through AnalyticsTransforming a Business Through Analytics
Transforming a Business Through Analytics
 
SoC Keynote:The State of the Art in Integration Technology
SoC Keynote:The State of the Art in Integration TechnologySoC Keynote:The State of the Art in Integration Technology
SoC Keynote:The State of the Art in Integration Technology
 

Dernier

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 

Dernier (20)

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 

Apache Web Services in the Real World, an E-Science Perspective

  • 1. Apache Web Services in the Real World, an E-Science Perspective Srinath Perera Architect, WSO2 Inc. Member, Apache Software Foundation Lanka Software Foundation
  • 2. Outline ● Linked Environment for Atmospheric Discovery Project (LEAD), the Use Case. ● LEAD Architecture, using SOA to build a Large Scale E-Science Project. ● History: LEAD and Apache Web Service Projects. ● Apache as a Sustainability Model for Academic Projects.
  • 3. E-Science ● Continuation of High Performance Computing, Parallel Computing, and Grid. ● Cyber-infrastructures to support Scientific Research. ● Build around “Computation” as the third Pillar of Science (along with Analysis and Experimentation). ● Characterized by wide range of computing (CPU minutes to CPU years) and Data (few KB to Pbs of data) requirements. ● Based on Real life usecases.
  • 4. Reality is Harder than Fiction ● E-Science joins Theory with Real life data ● Real Life Applications often go beyond our experiences. ● Most Weather models are calculated much less than ideal resolutions, otherwise a 24 hour forecast takes more than 24 hours !!! ● Physics Usecases (e.g. Large Hadron Collider), Telescopes, Genome Analysis generate Tera bytes of data in days if not hours, and moving a 1TB takes hours even in a 10 GB networks of TeraGrid. ● Scale, Geographical Distribution of resources, Heterogeneity makes these usecases Complex.
  • 5. Linked Environments for Atmospheric Discovery (LEAD) ● U.S. NSF funded, 10+ Universities, 11M $, 5 Years. ● Used for U.S. National Weather forecasts by NOAA. ● Presented to U.S. Congress as an example to justify Scientific research spendings by U.S. NSF. ● Have brought the state of the art forecasting capabilities to wider audience ranging from hardcore scientists to high schools students.
  • 6. LEAD: Dynamic Weather Analysis in U.S. Wide Scale
  • 7. Why is it Hard? ● Geographically Distributed Sensors, Computing Power, Storage, and Expertise. ● Handling Failures and Recovery ● Long Running Jobs (> 1 Hour). ● Large Scale Jobs (10-1000+ processors). ● Large Sized Data (KBs to GB of data). ● Need to serve many Parallel Users. ● Usage Spikes.
  • 8. LEAD as an Example ● Assume a Hurricane developed, and 1000 scientists across U.S. come to LEAD portal to run forecasts. ● Lets assume, ● Each user run 3 workflows. ● Each Workflow has 6 services, generates about 300 notifications, moves 50 100MB files, generates 50 100MB files, and runs for one hour. ● Each Service needs 5 CPUs Hours .
  • 9. Which Means ● 3000 Parallel workflows ● Need 90,000 CPUs per Hour ● 250 TPS for messaging System ● Move 8GB/Sec through the network ● Generate 15TB data per Hour LEAD Can not handle these numbers yet, but they give us an idea about the challenge.
  • 10. SOA, E-Science and LEAD ● E-Science infrastructures are Distributed, Complex, and Heterogeneous. ● SOA is designed to handle just the like. ● LEAD is based on many SOA Specs – WSDL, SOAP, WS-Addressing for Communication – WS-BPEL for Workflows – WS-Eventing for Messaging – WSDM for service Management ● LEAD People have closely worked with and contributed to Web Services, pushing its limits to apply it to LEAD.
  • 15. LEAD & Apache WS History ● Few People from LEAD has been major contributors for Apache Axis, and then Axis2. ● LEAD is not based on Axis2. ● LEAD is older than Axis2, and it forked off in Axis era, mainly because of Async messaging support. ● Five years ago LEAD implemented many tools (e.g. Registries, Async Messaging, Workflow Engine), that are hot topics now. ● Towards the end, LEAD started looking at Axis2 and other Apache Projects from a Sustainability Perspective. ● Most part are already converted, others are being converted.
  • 16. LEAD with Apache Projects ● LEAD Switched to Apache ODE for workflow execution more than a Year ago. ● LEAD data subsystems switched to Axis2 about a Year ago. ● Job Submission was switched to Axis2 based solution few months back. ● Service Factory is being converted to Axis2 right now. ● Conversion of Messaging System is in progress (Through a Indiana University and LSF collaboration).
  • 17. Apache as a Sustainability model for Research projects ● Industry values “People”, we (opensource) value “Code”, and Academia values “Ideas”. ● Most NSF Grants, now, ask for a Sustainability Model as part of Proposals. ● One option is a commercial spin off ● Doing it in a opensource way, building a community and users around a project is also a potential Solution. ● Many Challenges: ownership, need to renounce control, active engagement of the community are the key. ● “Source Open” is not good enough!! ● “Dump and Run” does not work either.
  • 18. Pros & Cons Advantages Disadvantages Reach to a wider Audience. Healthy You have to let go of the User Community, world debug your ownership, at least to a some project for you. extent. Potential Long Lifetime, Self Need for community Consent sustaining community if Successful. might slow you down. To take advantage of Apache You have to learn to listen and Process throughout Project life cycle explain. Some arguments are (Releases, SVN, Jira, Wiki, Culture ). harder to do in a mailing list. Better Chances of Attracting external Have to Time Publications. Developers, more inputs. Better chance of avoiding “source open”. Take advantage of Apache Infrastructure.
  • 19. Conclusion ● Wanted to share a Real Life, Large-Scale SOA Usecase ● Wanted to show LEAD-Apache interactions as a real Life Case Study of interactions between Apache and an Academic Project. ● Wanted to Showcase Apache as a Sustainability Mechanism, if it is done right.