SlideShare une entreprise Scribd logo
1  sur  26
Télécharger pour lire hors ligne
Opal: Simple Web Services Wrappers for ScientificOpal: Simple Web Services Wrappers for Scientific
ApplicationsApplications
Sriram Krishnan*, Brent Stearn, Karan Bhatia, Kim K. Baldridge,
Wilfred W. Li, Peter Arzberger
*sriram@sdsc.edu
ICWS 2006 - Sept 21, 2006
Web PortalsRich Clients
Set of Biomedical Applications
NBCR Computational InfrastructureNBCR Computational Infrastructure
Resources
Telescience Portal
Computational Grid
Web Services
Workflow
Middleware
PMV ADT
Vision ContinuityAPBSCommand
APBS Continuity
Gtomo2
TxBRAutodockGAMESS
QMView
RequirementsRequirements
• Enable access to scientific applications on
Grid resources
– Seamlessly via a number of user interfaces
– Easily from the perspective of a scientific user
• Enable the creation of scientific workflows
– Possibly with the use of commodity workflow
toolkits
Challenges for theChallenges for the Scientific UserScientific User
• Access to Grid resources is still very
complicated
– User account creation
– Management of credentials
– Installation and deployment of scientific
software
– Interaction with Grid schedulers
– Data management
Towards Services Oriented Architectures (SOA)Towards Services Oriented Architectures (SOA)
• Scientific applications wrapped as Web services
– Provision of a SOAP API for programmatic access
• Clients interact with application Web services,
instead of Grid resources
– Used in practice by several scientific communities
• NBCR: http://nbcr.net
• GEON: http://geongrid.org
• GLEON: http://gleon.org/
• CAMERA: http://camera.calit2.net
Talk OutlineTalk Outline
• Motivation for a Services Oriented
Architecture (SOA)
• Overall end-to-end architecture
• The Opal Toolkit: Introduction & technical
details
• Some use cases
• Concluding remarks
Condor pool SGE Cluster PBS Cluster
Globus Globus Globus
Application Services Security Services (GAMA)
State
Mgmt
Gemstone PMV/Vision Kepler
Architecture OverviewArchitecture Overview
Scientific SOA: BenefitsScientific SOA: Benefits
• Applications are installed once, and used by all
authorized users
– No need to create accounts for all Grid users
– Use of standards-based Grid security mechanisms
• Users are shielded from the complexities of Grid
schedulers
• Data management for multiple concurrent job runs
performed automatically by the Web service
• State management and persistence for long running jobs
• Accessibility via a multitude of clients
Possible ApproachesPossible Approaches
• Write application services by hand
– Pros: More flexible implementations, stronger data
typing via custom XML schemas
– Cons: Not generic, need to write one wrapper per
application
• Use a Web services wrapper toolkit, such as
Opal
– Pros: Generic, rapid deployment of new services
– Cons: Less flexible implementation, weak data typing
due to use of generic XML schemas
The Opal Toolkit: OverviewThe Opal Toolkit: Overview
• Enables rapid deployment of scientific
applications as Web services (< 2 hours)
• Steps
– Application writers create configuration file(s) for a
scientific application
– Deploy the application as a Web service using Opal’s
simple deployment mechanism (via Apache Ant)
– Users can now access this application as a Web
service via a unique URL
Opal ArchitectureOpal Architecture
Tomcat Container
Axis Engine
Opal WS Opal WS
Cluster/Grid Resources
Container
Properties
Service
Config
Scheduler,
Security,
Database
Setups
Binary,
Metadata,
Arguments
ImplementationImplementation DetailsDetails
• Implemented as a regular Axis service
– Application behavior specified by the service
configuration
– Configuration passed as a parameter via the auto-
generated deployment descriptor (WSDD)
• Possible to have multiple instances of the same
class for different applications
– Distinguished by a unique URL for every application
• No need to generate sources or WSDL prior to
deployment
Sample Container PropertiesSample Container Properties
# the base URL for the tomcat installation
# this is required since Java can't figure out the IP
# address if there are multiple network interfaces
tomcat.url=http://ws.nbcr.net:8080
# database information
database.use=false
database.url=jdbc:postgresql://localhost/app_db
database.user=<app_user>
database.passwd=<app_passwd>
# globus information
globus.use=true
globus.gatekeeper=ws.nbcr.net:2119/jobmanager-sge
globus.service_cert=/home/apbs_user/certs/apbs_service.cert.pem
globus.service_privkey=/home/apbs_user/certs/apbs_service.privkey
# parallel parameters
num.procs=16
mpi.run=/opt/mpich/gnu/bin/mpirun
Sample Application ConfigurationSample Application Configuration
<appConfig xmlns="http://nbcr.sdsc.edu/opal/types"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<metadata>
<usage><![CDATA[psize.py [opts] <filename>]]></usage>
<info xsd:type="xsd:string">
<![CDATA[
--help : Display this text
--CFAC=<value> : Factor by which to expand mol dims to
get coarse grid dims
[default = 1.7]
...
]]>
</info>
</metadata>
<binaryLocation>/homes/apbs_user/bin/psize.py</binaryLocation>
<defaultArgs>--GMEMCEIL=1000</defaultArgs>
<parallel>false</parallel>
</appConfig>
Application Deployment & UndeploymentApplication Deployment & Undeployment
• To deploy onto a local Tomcat container:
ant -f build-opal.xml deploy -DserviceName=<serviceName>
-DappConfig=<appConfig.xml>
• To undeploy a service:
ant -f build-opal.xml undeploy -DserviceName=<serviceName>
Service OperationsService Operations
• Get application metadata: Returns metadata specified
inside the application configuration
• Launch job: Accepts list of arguments and input files
(Base64 encoded), launches the job, and returns a jobID
• Query job status: Returns status of running job using the
jobID
• Get job outputs: Returns the locations of job outputs
using the jobID
• Get output as Base64: Returns an output file in Base64
encoded form
• Destroy job: Uses the jobID to destroy a running job
MEME+MAST WorkflowMEME+MAST Workflow using Keplerusing Kepler
Kepler Opal Web Services ActorKepler Opal Web Services Actor
Ligand Protein Interaction Using Web ServicesLigand Protein Interaction Using Web Services
• Baldridge, Greenberg, Amoreira, Kondric
• GAMESS Service
– More accurate Ligand Information
• LigPrep Service
– Generation of Conformational Spaces
• PDB2PQR Service
– Protein preparation
• APBS Service
– Generation of electrostatic information
• QMView Service
– Visualization of electrostatic potential file
• Applications:
– Electrostatics and docking
– High-throughput processing of ligand-
protein interaction studies
– Use of small molecules (ligands) to turn
on or off a protein function
ConclusionsConclusions
• We presented Opal, a toolkit for rapidly
exposing legacy scientific applications as
Web services
– Provides features like Job management, Scheduling,
Security, and Persistence, in an easy to use and
configurable manner
– Lowers inertia in the scientific community against the
adoption of Web services and SOAs
– Currently being used by the NBCR community for
Grid-enabling several scientific applications, via a
multitude of interfaces
Future WorkFuture Work
• WSRF Integration
– State management using standard Grid mechanisms
– Asynchronous status notifications via WS-Notification
• Meta-scheduling and resource monitoring
– Use of CSF4 and GFarm
• Alternate mechanisms for I/O staging
– GridFTP, RFT
• Addition of strong data typing for I/O
– Use of XML schemas, DFDL
Thanks for listening!Thanks for listening!
• More information, downloads, documentation:
– http://nbcr.net/services/
• Questions and comments welcome!
– sriram@sdsc.edu
AppendixAppendix
Grid Computing isGrid Computing is ……
• “Co-ordinated resource sharing and problem solving in dynamic
multi-institutional virtual organizations.” [Foster, Kesselman,
Tuecke]
– Co-ordinated - multiple resources working in concert, eg. Disk & CPU,
or instruments & database, etc.
– Resources - compute cycles, databases, files, application services,
instruments.
– Problem solving - focus on solving scientific problems
– Dynamic - environments that are changing in unpredictable ways
– Virtual Organization - resources spanning multiple organizations and
administrative domains, security domains, and technical domains
GridGrid Computing isComputing is …… (Industry)(Industry)
• “About finding distributed, underutilized compute resources (systems,
desktops, storage) and provisioning those resources to users or
applications requiring them.” [The Grid Report, Clabby Analytics]
– Distributed - all the resources laying around in departments or server rooms.
– Underutilized - organizations save money by increasing utilization versus
purchasing new resources.
– Resources - servers and server cycles, applications, data resources
– Provisioning - predict and schedule resource use depending on load.
Scientific Objective: Modeling and Analysis Across ScalesScientific Objective: Modeling and Analysis Across Scales
Tools that Integrate Data, Construct Models
and Perform Analysis across Scales
Organisms
Macromolecular
Molecule
Subcellular
Atom
Cell
Tissue
Organ
Modeling Synaptic
Activity
crossbridge
lattice
multicellular
filament
ventricles
Modeling
the Heart

Contenu connexe

Tendances

Sherlock: an anomaly detection service on top of Druid
Sherlock: an anomaly detection service on top of Druid Sherlock: an anomaly detection service on top of Druid
Sherlock: an anomaly detection service on top of Druid DataWorks Summit
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...DataWorks Summit/Hadoop Summit
 
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksUsing Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksDataWorks Summit
 
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on HiveFaster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on HiveDataWorks Summit/Hadoop Summit
 
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)Spark Summit
 
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...Spark Summit
 
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...Databricks
 
A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...
A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...
A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...Databricks
 
Never late again! Job-Level deadline SLOs in YARN
Never late again! Job-Level deadline SLOs in YARNNever late again! Job-Level deadline SLOs in YARN
Never late again! Job-Level deadline SLOs in YARNDataWorks Summit
 
Improvements to Flink & it's Applications in Alibaba Search
Improvements to Flink & it's Applications in Alibaba SearchImprovements to Flink & it's Applications in Alibaba Search
Improvements to Flink & it's Applications in Alibaba SearchDataWorks Summit/Hadoop Summit
 
Streaming Analytics with Spark, Kafka, Cassandra and Akka
Streaming Analytics with Spark, Kafka, Cassandra and AkkaStreaming Analytics with Spark, Kafka, Cassandra and Akka
Streaming Analytics with Spark, Kafka, Cassandra and AkkaHelena Edelson
 
Rethinking Streaming Analytics For Scale
Rethinking Streaming Analytics For ScaleRethinking Streaming Analytics For Scale
Rethinking Streaming Analytics For ScaleHelena Edelson
 
Sa introduction to big data pipelining with cassandra &amp; spark west mins...
Sa introduction to big data pipelining with cassandra &amp; spark   west mins...Sa introduction to big data pipelining with cassandra &amp; spark   west mins...
Sa introduction to big data pipelining with cassandra &amp; spark west mins...Simon Ambridge
 
Integrating Apache Phoenix with Distributed Query Engines
Integrating Apache Phoenix with Distributed Query EnginesIntegrating Apache Phoenix with Distributed Query Engines
Integrating Apache Phoenix with Distributed Query EnginesDataWorks Summit
 
Fast and Reliable Apache Spark SQL Releases
Fast and Reliable Apache Spark SQL ReleasesFast and Reliable Apache Spark SQL Releases
Fast and Reliable Apache Spark SQL ReleasesDataWorks Summit
 
Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSA
Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSABuilding the Foundations of an Intelligent, Event-Driven Data Platform at EFSA
Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSADatabricks
 

Tendances (20)

Sherlock: an anomaly detection service on top of Druid
Sherlock: an anomaly detection service on top of Druid Sherlock: an anomaly detection service on top of Druid
Sherlock: an anomaly detection service on top of Druid
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
 
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksUsing Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
 
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on HiveFaster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
 
Real Time Machine Learning Visualization with Spark
Real Time Machine Learning Visualization with SparkReal Time Machine Learning Visualization with Spark
Real Time Machine Learning Visualization with Spark
 
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
 
ebay
ebayebay
ebay
 
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...
 
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
 
A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...
A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...
A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...
 
Never late again! Job-Level deadline SLOs in YARN
Never late again! Job-Level deadline SLOs in YARNNever late again! Job-Level deadline SLOs in YARN
Never late again! Job-Level deadline SLOs in YARN
 
Hadoop to spark_v2
Hadoop to spark_v2Hadoop to spark_v2
Hadoop to spark_v2
 
Improvements to Flink & it's Applications in Alibaba Search
Improvements to Flink & it's Applications in Alibaba SearchImprovements to Flink & it's Applications in Alibaba Search
Improvements to Flink & it's Applications in Alibaba Search
 
Next Gen Big Data Analytics with Apache Apex
Next Gen Big Data Analytics with Apache Apex Next Gen Big Data Analytics with Apache Apex
Next Gen Big Data Analytics with Apache Apex
 
Streaming Analytics with Spark, Kafka, Cassandra and Akka
Streaming Analytics with Spark, Kafka, Cassandra and AkkaStreaming Analytics with Spark, Kafka, Cassandra and Akka
Streaming Analytics with Spark, Kafka, Cassandra and Akka
 
Rethinking Streaming Analytics For Scale
Rethinking Streaming Analytics For ScaleRethinking Streaming Analytics For Scale
Rethinking Streaming Analytics For Scale
 
Sa introduction to big data pipelining with cassandra &amp; spark west mins...
Sa introduction to big data pipelining with cassandra &amp; spark   west mins...Sa introduction to big data pipelining with cassandra &amp; spark   west mins...
Sa introduction to big data pipelining with cassandra &amp; spark west mins...
 
Integrating Apache Phoenix with Distributed Query Engines
Integrating Apache Phoenix with Distributed Query EnginesIntegrating Apache Phoenix with Distributed Query Engines
Integrating Apache Phoenix with Distributed Query Engines
 
Fast and Reliable Apache Spark SQL Releases
Fast and Reliable Apache Spark SQL ReleasesFast and Reliable Apache Spark SQL Releases
Fast and Reliable Apache Spark SQL Releases
 
Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSA
Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSABuilding the Foundations of an Intelligent, Event-Driven Data Platform at EFSA
Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSA
 

Similaire à Opal: Simple Web Services Wrappers for Scientific Applications

Creating a Centralized Consumer Profile Management Service with WebSphere Dat...
Creating a Centralized Consumer Profile Management Service with WebSphere Dat...Creating a Centralized Consumer Profile Management Service with WebSphere Dat...
Creating a Centralized Consumer Profile Management Service with WebSphere Dat...Prolifics
 
Building a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for AnalystsBuilding a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for AnalystsAvere Systems
 
Oracle E-Business Suite On Oracle Cloud
Oracle E-Business Suite On Oracle CloudOracle E-Business Suite On Oracle Cloud
Oracle E-Business Suite On Oracle Cloudpasalapudi
 
Paa sing a java ee 6 application kshitiz saxena
Paa sing a java ee 6 application   kshitiz saxenaPaa sing a java ee 6 application   kshitiz saxena
Paa sing a java ee 6 application kshitiz saxenaIndicThreads
 
Geospatial Sensor Networks and Partitioning Data
Geospatial Sensor Networks and Partitioning DataGeospatial Sensor Networks and Partitioning Data
Geospatial Sensor Networks and Partitioning DataAlexMiowski
 
PEARC17: Live Integrated Visualization Environment: An Experiment in General...
PEARC17: Live Integrated Visualization Environment: An Experiment in General...PEARC17: Live Integrated Visualization Environment: An Experiment in General...
PEARC17: Live Integrated Visualization Environment: An Experiment in General...moneyjh
 
A Tour of Azure SQL Databases (NOVA SQL UG 2020)
A Tour of Azure SQL Databases  (NOVA SQL UG 2020)A Tour of Azure SQL Databases  (NOVA SQL UG 2020)
A Tour of Azure SQL Databases (NOVA SQL UG 2020)Timothy McAliley
 
在小學有效運用雲端電腦以促進電子學習(第一節筆記)
在小學有效運用雲端電腦以促進電子學習(第一節筆記)在小學有效運用雲端電腦以促進電子學習(第一節筆記)
在小學有效運用雲端電腦以促進電子學習(第一節筆記)Tsz Wing Chu
 
Architectures, Frameworks and Infrastructure
Architectures, Frameworks and InfrastructureArchitectures, Frameworks and Infrastructure
Architectures, Frameworks and Infrastructureharendra_pathak
 
Designing a Feedback Loop for Event-Driven Data Sharing With Teresa Wang | Cu...
Designing a Feedback Loop for Event-Driven Data Sharing With Teresa Wang | Cu...Designing a Feedback Loop for Event-Driven Data Sharing With Teresa Wang | Cu...
Designing a Feedback Loop for Event-Driven Data Sharing With Teresa Wang | Cu...HostedbyConfluent
 
Open shift and docker - october,2014
Open shift and docker - october,2014Open shift and docker - october,2014
Open shift and docker - october,2014Hojoong Kim
 
Unified Situational Awareness Dashboard for Spacecraft Operations: an inte...
Unified Situational Awareness Dashboard for Spacecraft Operations: an inte...Unified Situational Awareness Dashboard for Spacecraft Operations: an inte...
Unified Situational Awareness Dashboard for Spacecraft Operations: an inte...Haisam Ido
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithMarkus Eisele
 

Similaire à Opal: Simple Web Services Wrappers for Scientific Applications (20)

Creating a Centralized Consumer Profile Management Service with WebSphere Dat...
Creating a Centralized Consumer Profile Management Service with WebSphere Dat...Creating a Centralized Consumer Profile Management Service with WebSphere Dat...
Creating a Centralized Consumer Profile Management Service with WebSphere Dat...
 
Building a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for AnalystsBuilding a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for Analysts
 
Brad stack - Digital Health and Well-Being Festival
Brad stack - Digital Health and Well-Being Festival Brad stack - Digital Health and Well-Being Festival
Brad stack - Digital Health and Well-Being Festival
 
Oracle E-Business Suite On Oracle Cloud
Oracle E-Business Suite On Oracle CloudOracle E-Business Suite On Oracle Cloud
Oracle E-Business Suite On Oracle Cloud
 
Paa sing a java ee 6 application kshitiz saxena
Paa sing a java ee 6 application   kshitiz saxenaPaa sing a java ee 6 application   kshitiz saxena
Paa sing a java ee 6 application kshitiz saxena
 
Cisco project ideas
Cisco   project ideasCisco   project ideas
Cisco project ideas
 
spring-cloud.pptx
spring-cloud.pptxspring-cloud.pptx
spring-cloud.pptx
 
Predix
PredixPredix
Predix
 
Geospatial Sensor Networks and Partitioning Data
Geospatial Sensor Networks and Partitioning DataGeospatial Sensor Networks and Partitioning Data
Geospatial Sensor Networks and Partitioning Data
 
PEARC17: Live Integrated Visualization Environment: An Experiment in General...
PEARC17: Live Integrated Visualization Environment: An Experiment in General...PEARC17: Live Integrated Visualization Environment: An Experiment in General...
PEARC17: Live Integrated Visualization Environment: An Experiment in General...
 
A Tour of Azure SQL Databases (NOVA SQL UG 2020)
A Tour of Azure SQL Databases  (NOVA SQL UG 2020)A Tour of Azure SQL Databases  (NOVA SQL UG 2020)
A Tour of Azure SQL Databases (NOVA SQL UG 2020)
 
在小學有效運用雲端電腦以促進電子學習(第一節筆記)
在小學有效運用雲端電腦以促進電子學習(第一節筆記)在小學有效運用雲端電腦以促進電子學習(第一節筆記)
在小學有效運用雲端電腦以促進電子學習(第一節筆記)
 
Architectures, Frameworks and Infrastructure
Architectures, Frameworks and InfrastructureArchitectures, Frameworks and Infrastructure
Architectures, Frameworks and Infrastructure
 
Designing a Feedback Loop for Event-Driven Data Sharing With Teresa Wang | Cu...
Designing a Feedback Loop for Event-Driven Data Sharing With Teresa Wang | Cu...Designing a Feedback Loop for Event-Driven Data Sharing With Teresa Wang | Cu...
Designing a Feedback Loop for Event-Driven Data Sharing With Teresa Wang | Cu...
 
Cloud Migration
Cloud MigrationCloud Migration
Cloud Migration
 
Open shift and docker - october,2014
Open shift and docker - october,2014Open shift and docker - october,2014
Open shift and docker - october,2014
 
Unified Situational Awareness Dashboard for Spacecraft Operations: an inte...
Unified Situational Awareness Dashboard for Spacecraft Operations: an inte...Unified Situational Awareness Dashboard for Spacecraft Operations: an inte...
Unified Situational Awareness Dashboard for Spacecraft Operations: an inte...
 
Un-clouding the cloud
Un-clouding the cloudUn-clouding the cloud
Un-clouding the cloud
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolith
 
Cloud presentation NELA
Cloud presentation NELACloud presentation NELA
Cloud presentation NELA
 

Dernier

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 

Dernier (20)

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 

Opal: Simple Web Services Wrappers for Scientific Applications

  • 1. Opal: Simple Web Services Wrappers for ScientificOpal: Simple Web Services Wrappers for Scientific ApplicationsApplications Sriram Krishnan*, Brent Stearn, Karan Bhatia, Kim K. Baldridge, Wilfred W. Li, Peter Arzberger *sriram@sdsc.edu ICWS 2006 - Sept 21, 2006
  • 2. Web PortalsRich Clients Set of Biomedical Applications NBCR Computational InfrastructureNBCR Computational Infrastructure Resources Telescience Portal Computational Grid Web Services Workflow Middleware PMV ADT Vision ContinuityAPBSCommand APBS Continuity Gtomo2 TxBRAutodockGAMESS QMView
  • 3. RequirementsRequirements • Enable access to scientific applications on Grid resources – Seamlessly via a number of user interfaces – Easily from the perspective of a scientific user • Enable the creation of scientific workflows – Possibly with the use of commodity workflow toolkits
  • 4. Challenges for theChallenges for the Scientific UserScientific User • Access to Grid resources is still very complicated – User account creation – Management of credentials – Installation and deployment of scientific software – Interaction with Grid schedulers – Data management
  • 5. Towards Services Oriented Architectures (SOA)Towards Services Oriented Architectures (SOA) • Scientific applications wrapped as Web services – Provision of a SOAP API for programmatic access • Clients interact with application Web services, instead of Grid resources – Used in practice by several scientific communities • NBCR: http://nbcr.net • GEON: http://geongrid.org • GLEON: http://gleon.org/ • CAMERA: http://camera.calit2.net
  • 6. Talk OutlineTalk Outline • Motivation for a Services Oriented Architecture (SOA) • Overall end-to-end architecture • The Opal Toolkit: Introduction & technical details • Some use cases • Concluding remarks
  • 7. Condor pool SGE Cluster PBS Cluster Globus Globus Globus Application Services Security Services (GAMA) State Mgmt Gemstone PMV/Vision Kepler Architecture OverviewArchitecture Overview
  • 8. Scientific SOA: BenefitsScientific SOA: Benefits • Applications are installed once, and used by all authorized users – No need to create accounts for all Grid users – Use of standards-based Grid security mechanisms • Users are shielded from the complexities of Grid schedulers • Data management for multiple concurrent job runs performed automatically by the Web service • State management and persistence for long running jobs • Accessibility via a multitude of clients
  • 9. Possible ApproachesPossible Approaches • Write application services by hand – Pros: More flexible implementations, stronger data typing via custom XML schemas – Cons: Not generic, need to write one wrapper per application • Use a Web services wrapper toolkit, such as Opal – Pros: Generic, rapid deployment of new services – Cons: Less flexible implementation, weak data typing due to use of generic XML schemas
  • 10. The Opal Toolkit: OverviewThe Opal Toolkit: Overview • Enables rapid deployment of scientific applications as Web services (< 2 hours) • Steps – Application writers create configuration file(s) for a scientific application – Deploy the application as a Web service using Opal’s simple deployment mechanism (via Apache Ant) – Users can now access this application as a Web service via a unique URL
  • 11. Opal ArchitectureOpal Architecture Tomcat Container Axis Engine Opal WS Opal WS Cluster/Grid Resources Container Properties Service Config Scheduler, Security, Database Setups Binary, Metadata, Arguments
  • 12. ImplementationImplementation DetailsDetails • Implemented as a regular Axis service – Application behavior specified by the service configuration – Configuration passed as a parameter via the auto- generated deployment descriptor (WSDD) • Possible to have multiple instances of the same class for different applications – Distinguished by a unique URL for every application • No need to generate sources or WSDL prior to deployment
  • 13. Sample Container PropertiesSample Container Properties # the base URL for the tomcat installation # this is required since Java can't figure out the IP # address if there are multiple network interfaces tomcat.url=http://ws.nbcr.net:8080 # database information database.use=false database.url=jdbc:postgresql://localhost/app_db database.user=<app_user> database.passwd=<app_passwd> # globus information globus.use=true globus.gatekeeper=ws.nbcr.net:2119/jobmanager-sge globus.service_cert=/home/apbs_user/certs/apbs_service.cert.pem globus.service_privkey=/home/apbs_user/certs/apbs_service.privkey # parallel parameters num.procs=16 mpi.run=/opt/mpich/gnu/bin/mpirun
  • 14. Sample Application ConfigurationSample Application Configuration <appConfig xmlns="http://nbcr.sdsc.edu/opal/types" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <metadata> <usage><![CDATA[psize.py [opts] <filename>]]></usage> <info xsd:type="xsd:string"> <![CDATA[ --help : Display this text --CFAC=<value> : Factor by which to expand mol dims to get coarse grid dims [default = 1.7] ... ]]> </info> </metadata> <binaryLocation>/homes/apbs_user/bin/psize.py</binaryLocation> <defaultArgs>--GMEMCEIL=1000</defaultArgs> <parallel>false</parallel> </appConfig>
  • 15. Application Deployment & UndeploymentApplication Deployment & Undeployment • To deploy onto a local Tomcat container: ant -f build-opal.xml deploy -DserviceName=<serviceName> -DappConfig=<appConfig.xml> • To undeploy a service: ant -f build-opal.xml undeploy -DserviceName=<serviceName>
  • 16. Service OperationsService Operations • Get application metadata: Returns metadata specified inside the application configuration • Launch job: Accepts list of arguments and input files (Base64 encoded), launches the job, and returns a jobID • Query job status: Returns status of running job using the jobID • Get job outputs: Returns the locations of job outputs using the jobID • Get output as Base64: Returns an output file in Base64 encoded form • Destroy job: Uses the jobID to destroy a running job
  • 17. MEME+MAST WorkflowMEME+MAST Workflow using Keplerusing Kepler
  • 18. Kepler Opal Web Services ActorKepler Opal Web Services Actor
  • 19. Ligand Protein Interaction Using Web ServicesLigand Protein Interaction Using Web Services • Baldridge, Greenberg, Amoreira, Kondric • GAMESS Service – More accurate Ligand Information • LigPrep Service – Generation of Conformational Spaces • PDB2PQR Service – Protein preparation • APBS Service – Generation of electrostatic information • QMView Service – Visualization of electrostatic potential file • Applications: – Electrostatics and docking – High-throughput processing of ligand- protein interaction studies – Use of small molecules (ligands) to turn on or off a protein function
  • 20. ConclusionsConclusions • We presented Opal, a toolkit for rapidly exposing legacy scientific applications as Web services – Provides features like Job management, Scheduling, Security, and Persistence, in an easy to use and configurable manner – Lowers inertia in the scientific community against the adoption of Web services and SOAs – Currently being used by the NBCR community for Grid-enabling several scientific applications, via a multitude of interfaces
  • 21. Future WorkFuture Work • WSRF Integration – State management using standard Grid mechanisms – Asynchronous status notifications via WS-Notification • Meta-scheduling and resource monitoring – Use of CSF4 and GFarm • Alternate mechanisms for I/O staging – GridFTP, RFT • Addition of strong data typing for I/O – Use of XML schemas, DFDL
  • 22. Thanks for listening!Thanks for listening! • More information, downloads, documentation: – http://nbcr.net/services/ • Questions and comments welcome! – sriram@sdsc.edu
  • 24. Grid Computing isGrid Computing is …… • “Co-ordinated resource sharing and problem solving in dynamic multi-institutional virtual organizations.” [Foster, Kesselman, Tuecke] – Co-ordinated - multiple resources working in concert, eg. Disk & CPU, or instruments & database, etc. – Resources - compute cycles, databases, files, application services, instruments. – Problem solving - focus on solving scientific problems – Dynamic - environments that are changing in unpredictable ways – Virtual Organization - resources spanning multiple organizations and administrative domains, security domains, and technical domains
  • 25. GridGrid Computing isComputing is …… (Industry)(Industry) • “About finding distributed, underutilized compute resources (systems, desktops, storage) and provisioning those resources to users or applications requiring them.” [The Grid Report, Clabby Analytics] – Distributed - all the resources laying around in departments or server rooms. – Underutilized - organizations save money by increasing utilization versus purchasing new resources. – Resources - servers and server cycles, applications, data resources – Provisioning - predict and schedule resource use depending on load.
  • 26. Scientific Objective: Modeling and Analysis Across ScalesScientific Objective: Modeling and Analysis Across Scales Tools that Integrate Data, Construct Models and Perform Analysis across Scales Organisms Macromolecular Molecule Subcellular Atom Cell Tissue Organ Modeling Synaptic Activity crossbridge lattice multicellular filament ventricles Modeling the Heart