OGF actively collaborates with other standards organizations through cooperative agreements to develop standards for distributed computing. OGF has relationships with groups like DMTF, ISO, SNIA, ETSI, ITU-T, and NIST to jointly develop standards for areas like cloud computing, identity management, and data formats. These collaborations help drive innovation while avoiding duplication of efforts between organizations.
ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing
1. The Conference on the Use of Cloud for
High Performance Computing (HPC)
Taking the Best from
Industry & Academics:
Cross-SDO Projects to
Accelerate HPC Cloud
Innovation
Alan Sill,
Texas Tech
University;
VP of
Standards,
Open Grid
Forum
2. Alan Sill, TTU
September 24, 2013
About the Open Grid Forum:
Open Grid Forum (OGF) is a leading global standards
organization cooperating with many partners in the areas of
cloud, grid and related forms of advanced distributed
computing.
The OGF community and its partners pursue these topics
through an open process for development, creation and
promotion of relevant specifications and use cases.
The central feature of this work is open forum with open
processes to champion architectural blueprints related to cloud
and grid computing.
The resulting specifications and standards enable pervasive
adoption of advanced distributed computing techniques for
business and research worldwide.
2
3. Alan Sill, TTU
September 24, 2013
History and Background
• OGF began in 2001 as an organization to promote the
advancement of distributed computing worldwide.
• Grid Forum --> Global Grid Forum --> GGF + Enterprise Grid
Alliance --> formation of OGF in 2005.
• Mandate is to take on all forms of distributed computing and to
work to promote cooperation, information exchange, best
practices in use and standardization.
• OGF best known for a series of important computing, security
and network standards that form the basis for major science
and business-based distributed computing (BES, GridFTP,
DRMAA, JSDL, RNS, GLUE, UR, etc.).
• Have also been working on cloud and Big Data standards
(OCCI, WS-Agreement, DFDL, etc.) for several years.
• Cooperative work agreements with other SDOs in place.
3
4. Alan Sill, TTU
September 24, 2013
• DRMAA: Distributed Resource Management Application API
Grid Engine, Open Grid Scheduler: (open source); TORQUE and related
products: Adaptive Computing; PBS Works: Altair Engineering; Gridway:
DSA Research; Condor: U. of Wisconsin / Red Hat;
• OGSA® Basic Execution Service Version 1.0 and BES HPC Profile:
BES++ for LSF/SGE/PBS: Platform Computing; Windows HPC Server
2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering;
• JSDL: Job Submission Description Language (family of specs):
BES++ for LSF/SGE/PBS and Platform LSF: Platform Computing;
Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client
only): Altair Engineering;
• WS-Agreement (family of specifications):
ElasticLM License-as-a-Service: ElasticLM; BEinGrid SLA Negotiator,
LM-Architecture and Framework: (Multiple partners); BREIN SLA
Management Framework: (Multiple partners); WSAG4J, Web Services
Agreement for Java (framework implementation): Fraunhofer SCAI.
4
OGF Standards In Use In Industry:
5. Alan Sill, TTU
September 24, 2013
OGF Standards
OGF has an extensive set of applicable standards related to
advanced distributed grid and cloud computing and
associated storage management and network operation:
• Job Submission and Workflow Management (JSDL, BES)
• Federated Identity Management (FedSec-CG)
• Virtual Organizations (VOMS)
• Managing the Trust Eco-System (CA operations, AuthN/AuthZ)
• Network Management (NSI, NML, NMC, NM)
• Secure, fast multi--party data transfer (GridFTP, SRM)
• Service Agreements (WS-Agreement, WS-Agreement
Negotiation)
• Data Format Description (DFDL)
• Cloud Computing interfaces (OCCI)
• Distributed resource management APIs (DRMAA, SAGA, etc.)
• Firewall Traversal (FiTP)
• (Many others under development)
5
6. Alan Sill, TTU
September 24, 2013
Example: Worldwide LHC
Computing Grid
6
~450,000 cpu cores
~430 Pb storage
Typical data transfer
rate: ~12 GByte/sec
Total worldwide grid
capacity: ~2x WLCG
across all grids and
VOs
7. LSN-MAGIC Meeting
February 22, 2012
The Role of Standards for Risk Reduction and
Inter-operation in XSEDE
Andrew Grimshaw
XSEDE: The Next Generation of
US Supercomputing Infrastructure
OGF standards
power some of
the largest
supercomputing
infrastructures
in the world!
XSEDE cloud
infrastructure still
being written
8. LSN-MAGIC Meeting
February 22, 2012
XSEDE Services Layer:
Simple services combined in many ways
8
–Resource
Namespace
Service
1.1
–OGSA
Basic
Execu8on
Service
–OGSA
WSRF
BP
–
metadata
and
no8fica8on
–OGSA-‐ByteIO
–GridFTP
–JSDL,
BES,
BES
HPC
Profile
–WS
Trust
Secure
Token
Services
–WSI
BSP
for
transport
of
creden8als
–…
(more
than
we
have
room
to
cover
here)
Examples – (not
a complete list)
Basic message: XSEDE represents a phase change in the
engagement of OGF standards with US cyberinfrastructure.
9. Alan Sill, TTU
September 24, 2013
9
• OCCI is an API and Protocol
• Sits on the boundary of a Service Provider
and Service Consumer
• No assumptions about the boundary
OCCI® by OGF
Open Cloud Computing Interface®
Many real-world implementations!
10. Task Force – Sept 2013
Members
•70 individuals
•40 institutions
•13 countries
Stakeholders
•23 Resource Providers
•10 Technology Providers
•7 User Communities
•4 Liaisons
Technologies
•OpenNebula.
•StratusLab.
•OpenStack.
•Synnefo.
•WNoDeS.
BSC
CNRS
LMU
OeRC
Masaryk
TUD
IFAE
Cyfronet
100%IT
CESNET
RADICAL
SRCE
DANTE
FZJ
GRNET
GWDG
STFC
SARA
KTH
INFNFCTSG
EGI.eu
Imperial
CESGACETA
IFCA
IGI
IPHC
IN2P3
SZTAKI
IISAS SixSq
Credit: David Wallom
Chair EGI Federated Cloud Task Force
11. Federation Test bed – September 2013
FedCloud Core Services
Information
System
Top-DBII
Monitoring
Nagios
Image
Metadata
Marketplace
Appliance
Repository
VO
VOMS server
User Interfaces
OCCI Clients
rOCCI;WNoDeS-CLI
CDMI Clients
Libcdmi-java
Resource Providers
VOMS proxy
Vmcatcher
Vmcaster
Marketplace
client
SSM
CDMI server LDAP
OpenNebula
OpenStack
StratusLab
WNoDeS
OCCI server
AAI
VOMS Proxy
EGI Production
Infrastructure
Service
Availability
SAM
Accounting
APEL
Configuration
DB
GOCDB
External Services
Certification
Authorities
Credit: David Wallom
Chair EGI Federated Cloud Task Force
12. Alan Sill, TTU
September 24, 2013
12
Collaborating Organizations:
• It is often said that there are “too many standards organizations”. This is
a lot like saying there is “too much software”.
• Each has its own area of specialty, its own contributor base, and its own
method of funding to develop its work products.
• How does OGF relate to these organizations? (Answer: Cooperatively!)
(Some of many...)
13. Interoperability:
Appropriate use of standards permits software processes
to interoperate with other components and
infrastructures, and thus reduces risks including the risk
of unwanted vendor lock-in.
This allows developers, vendors and users to focus more
on higher level capabilities and therefore less on
reinventing common aspects and features of their APIs
and interface modules.
Coupling standards and software innovation is therefore
crucial to economical cloud development at this stage.
14. Alan Sill, TTU
September 24, 2013
14
Why Pursue Standards as a
Means to Interoperability?
•Risk reduction:
•Lessen risk of dead-end product design and orphan
components.
•Lessen or remove risk of vendor service lock-in.
•Mitigate reusability barriers for software and data
access.
•Provide best-of-breed development and methods.
•Mix-and-match for input & output of processing steps.
•Allow innovation/competition at more interesting
layers and development of better internal features.
•Facilitate interoperation with other provider software
services, components and infrastructures.
15. Alan Sill, TTU
September 24, 2013
Business Benefits
• Having an organized set of acceptance criteria
can improve business value to members in the following ways:
• For VENDORS, ensures that their product passes test
acceptance conditions, leading to fewer customer complaints in
the field.
• For PROVIDERS, ensures that products they are hosting are
well-behaved, with fewer unexpected error or service failure
conditions.
• For USERS, ensures that the products and services they use
will be interoperable with the API, framework or standard they
are using as the basis for their purchased services,
• Pre-purchase specification for products and services can be
done on a rational basis.
• Productivity of programmer teams is maximized by having
well-defined workflows for scenario and unit testing.
15
16. Alan Sill, TTU
September 24, 2013
16
OGF Cooperative Agreements
In Place as of Late 2013
OGF and DMTF:
• OGF published the the OCCI Core, Infrastructure and HTTP
Rendering specifications as GFD.183, 184 and 185, and is
working on a JSON rendering. We created a joint work register
with DMTF on OCCI and CIMI in 2011 and continue to
cooperate with them on implementation and testing of the CIMI
specification. DMTF has also joined OGF in the Cloud Plugfests.
OGF and ISO:
• OGF has an ongoing Category A liaison with ISO JTC1 SC38 on
Cloud Computing and is working with ISO on joint activities.
OGF and SNIA (CDMI):
• OGF has cooperative agreement w/SNIA with respect to
CDMI and has co-hosted 10 Cloud Plugfests so far (series
continues).
17. Alan Sill, TTU
September 24, 2013
17
OGF and ETSI:
• Cooperative MoU in place since 2012. OGF contributing to
ETSI CSC effort. ETSI joined the Cloud Plugfest testing as a
co-sponsor in 2012 and continues to support this series.
OGF and ITU-T:
• OGF has an ongoing formal liaison with ITU-T JCA Cloud.
OGF and TM Forum:
• MoU in place since 2005, renewed 2012. OGF contributed to
TM Forum cross-SDO work on on “End-to-End Management
of Cloud Service Agreements”, including SLAs, ongoing.
OGF and CSA:
• Cooperative agreement between OGF & CSA since formation.
OGF and IEEE:
• OGF co-sponsored IEEE Cloudcom conferences in 2011
(Athens) and 2012 (Taipei); ongoing engagements.
OGF Cooperative Agreements
In Place as of Late 2013
18. Alan Sill, TTU
September 24, 2013
18
OGF and XSEDE:
• OGF actively supports the standards process that underlies
XSEDE infrastructure and many of its ongoing projects.
OGF and EGI:
• OGF works closely with EGI and associated projects; recently
co-located OGF39 + Cloud Interoperability Week at EGI TF.
OGF and OCEAN:
• OCEAN joined Cloud Plugfest series in 2013 as a co-sponsor
and as one of the primary sponsors of the upcoming
September 2013 Cloud Interoperability Week.
OGF and OW2:
• OW2 co-sponsored Sep. 2013 Cloud Interoperability Week &
Cloud Plugfest series and supports associated cloud projects.
OGF Cooperative Agreements
In Place as of Late 2013
19. Alan Sill, TTU
September 24, 2013
19
OGF and NIST:
• OGF contributed several talks and input to US National
Institute of Standards and Technology Cloud Standards
Roadmap in 2010-2013 timeframe; AFS co-chairs SAJACC
working group.
OGF and European Commission FP7/H2020 projects:
• OGF technologies underpin the approaches of many
successfully funded FP-series projects and expects to
participate in supporting the Horizon 2020 call.
OGF and Cloud Plugfest series:
• OGF was one of the co-founding members of the ongoing
Cloud Plugfest developer-oriented testing series and co-
sponsored the recent Cloud Interoperability Week. OGF
provides substantial infrastructure to the ongoing series.
OGF Cooperative Agreements
In Place as of Late 2013
20. Alan Sill, TTU
September 24, 2013
http://collaborate.nist.gov/twiki-cloud-computing/bin/view/CloudComputing/SAJACC
NIST SAJACC Public Process
20
21. Alan Sill, TTU
September 24, 2013
European Union “Standards & Interoperability for
e-Infrastructure Implementation” (SIENA) Initiative
SIENA
(2010-‐2012)
is
a
Support
Ac8on
funded
by
the
European
Commission
under
FP7
(2007-‐13)
CapaciDes
programme.
A
coordinated
effort...
...towards
the
delivery
of
a
future
e-‐
Infrastructures
Roadmap....
...aligned
with
the
needs
of
European
and
na=onal
ini=a=ves
and
the
evolving
world.
IEG
–
Industry
Expert
Group
Partners
Interna8onal
Experts
REB
–
Roadmap
Editorial
Board
SLG
–
Special
Liaison
Group
Standards
Development
Organisa8ons
(SDOs)
Distributed
Compu8ng
Infrastructure
(DCI)
Ini8a8ves
Consolidated
Cloud
Compu8ng
Workshops
Final Report:
June 2012
21
24. Alan Sill, TTU
September 24, 2013
Cloud Plugfest Developer Series:
24
Continuing series
co-sponsored by
OGF, DMTF, SNIA,
OASIS, ETSI,
OCEAN and OW2!
9 previous events
held so far!
Developer-oriented in-
person standards and
software testing series
OPEN TO ALL!
Easy to get involved
and join in events as
developers or project
researchers!
Cloud Plugfest 10 just
completed!
Cloud Interoperability
Week Sep. 2013!
http://
cloudplugfest.org
27. Proof that iPhones are
used in space:
Link: Voyager - The Interstellar Mission
28. Alan Sill, TTU
September 24, 2013
Conclusions
• OGF actively engages with partners and participants
throughout the international arena through an open forum with
open processes to promote best practices and standards in
advanced distributed computing.
• OGF and its associated projects occupy an important role in
standards and software development with significant uptake in
advanced distributed computing, including cloud, grid,
networking and large-scale data processing, transfer and
handling through innovative cooperation with many partners.
• TTU is leveraging these standards to support a wide variety of
flexible architectures for advanced scientific and business
uses through the NSF Cloud and Autonomic Computing
Center and is actively seeking international collaborators to
extend this work more comprehensively through the cloud.
28