This talk givens an overview of several multi-SDO and cross-SDO activities to promote and spur innovation in cloud computing. The focus is on API development and standardization, including testbeds, test use cases, and collaborative activities between organizations to create and carry out development and testing in this area. The focus is on work being pursued through the Cloud and Autonomic Computing Center at Texas Tech University, which is part of the US National Science Foundation's Industry/University Cooperative Research Center, and on work being done by standards organizations such as the Open Grid Forum, Distributed Management Task Force, and Telecommunications Management Forum in which the CAC@TTU is involved. A summary is also given of work to produce a new round of more detailed use cases suitable for testing by the US National Institute of Standards and Technology's Standards Acceleration to Jumpstart Adoption of Cloud Computing (SAJACC) working group, with brief mention also given to other related work going on in this area in other parts of the world. Background and other standards work is also mentioned.
MPLS/SDN 2013 Intercloud Standardization and Testbeds - Sill
1. Cross-SDO Projects to Accelerate Cloud Innovation
Alan Sill, Ph.D
Site Director, Cloud and Autonomic Computing Center
Senior Scientist, High Performance Computing Center
Adjunct Professor of Physics
Texas Tech University
Vice President of Standards, Open Grid Forum
2. Standards as a Means to Interoperability
• Risk reduction:
• Lessen risk of dead-end product design and orphan
components.
• Lessen or remove risk of vendor service lock-in.
• Mitigate reusability barriers for software and data access.
• Provide best-of-breed development and methods.
• Mix-and-match for input & output of processing steps.
• Allow innovation/competition at more interesting layers
and development of better internal features.
• Facilitate interoperation with other provider software
services, components and infrastructures.
• Approach must be explicitly cross-SDO and cross-vendor.
Alan Sill, TTU
November 20, 2013
!2
3.
4. CAC Goals and Vision
The Texas Tech site intends to provide a practical work arena for
development and coordination of standards, standards-based
software and reference implementations applicable to cloud
and other forms of advanced distributed computing.
The site will fill a need to organize, classify, develop reference
implementations for and otherwise contribute to standardsbased software in advanced distributed computing.
The vision that underlies these goals is one of harmonious,
coordinated development of software that interoperates across
many boundaries of deployment and implementation, and that
can be repurposed, rescaled and redeployed as needed to solve
a wide variety of user, vendor and supplier problems.
Alan Sill, TTU
November 20, 2013
!4
5. Standards, Clouds and Interoperability:
Our vision at the CAC is that appropriate use of standards
as part of the innovation process permits software and
hardware used in clouds to interoperate with other
components and infrastructures, and thus reduces risks
including the risk of unwanted vendor lock-in.
!
This allows developers, vendors and users to focus more
on higher level capabilities and therefore less on
reinventing common aspects and features of their APIs and
interface modules.
!
Coupling standards and software innovation is therefore
crucial to economical cloud development at this stage.
Alan Sill, TTU
November 20, 2013
!5
14. CMS Global Data Grid
CMS Experiment
Online
System
Tier 0
Tier 1
Tier 2
10-40 Gb/s
Taiwan T1
UK T1
Italy T1
Physics caches
across Tier 2
10 Gb/s
UCSD
FNAL T1
>10 Gb/s
Caltech
Univ. T3
Tier 3
Tier 4
CERN T0
~4 Gb/s
Wisconsin
Univ. T3
Univ. T3
PCs
Florida
Univ. T3
15. Example: Worldwide LHC
Computing Grid
~450,000 cpu cores
~430 Pb storage
Typical data transfer
rate: ~12 GByte/sec
Total worldwide grid
capacity: ~2x WLCG
across all grids and
VOs
Alan Sill, TTU
November 20, 2013
!15
16.
17. EGI in numbers
CPU
cores
361,300
across
53
countries
(1.44
M
job/day)
Storage
Value
(yearly
increase)
Disk
(PB)
235
PB
(+69%)
Tape
(PB)
Value
(yearly
increase)
176
PB
(+32%)
!17
XSEDE/EGI BoF - XSEDE 2013
EGI-‐InSPIRE
RI-‐261323
www.egi.eu
18. Science Virtual Organizations on the OSG
• Astrophysics
• Biochemistry
• Bioinformatics
• Earthquake
Engineering
• Genetics
• Gravitational-‐wave
physics
• Mathematics
• Nanotechnology
• Nuclear
and
particle
physics
19. LSN-MAGIC Meeting
February 22, 2012
XSEDE: The Next Generation of
US Supercomputing Infrastructure
The Role of Standards
for Risk Reduction and
Inter-operation in XSEDE
OGF standards
power some of
the largest
supercomputing
infrastructures
in the world!
20. LSN-MAGIC Meeting
XSEDE Services Layer:
February 22, 2012
Simple services combined in many ways
–Resource
Namespace
Service
1.1
–OGSA
Basic
Execu@on
Service
–OGSA
WSRF
BP
–
metadata
and
no@fica@on
–OGSA-‐ByteIO
Examples – (not
–GridFTP
a complete list)
–JSDL,
BES,
BES
HPC
Profile
–WS
Trust
Secure
Token
Services
–WSI
BSP
for
transport
of
creden@als
–…
(more
than
we
have
room
to
cover
here)
XSEDE represents a phase change in the engagement of
modern computing standards with US cyberinfrastructure.
!20
21. Standardization Benefits
p
Having an organized set of acceptance criteria
can improve business value to members in the following ways:
n For VENDORS, ensures that their product passes test acceptance
conditions, leading to fewer customer complaints in the field.
n For PROVIDERS, ensures that products they are hosting are wellbehaved, with fewer unexpected error or service failure
conditions.
n For USERS, ensures that the products and services they use will be
interoperable with the API, framework or standard they are using
as the basis for their purchased services,
p
Pre-purchase specification for products and services can be done on a
rational basis.
p
Productivity of programmer teams is maximized by having welldefined workflows for scenario and unit testing.
Alan Sill, TTU
November 20, 2013
!21
22. About the Open Grid Forum:
Open Grid Forum (OGF) is a leading global standards
organization cooperating with many partners in the areas of
cloud, grid and related forms of advanced distributed
computing.
The OGF community and its partners pursue these topics
through an open process for development, creation and
promotion of relevant specifications and use cases.
The central feature of this work is open forum with open
processes to champion architectural blueprints related to cloud
and grid computing.
The resulting specifications and standards enable pervasive
adoption of advanced distributed computing techniques for
business and research worldwide.
Alan Sill, TTU
November 20, 2013
!22
23. History and Background
•
•
•
•
•
•
OGF began in 2001 as an organization to promote the
advancement of distributed computing worldwide.
Grid Forum --> Global Grid Forum --> GGF + Enterprise Grid
Alliance --> formation of OGF in 2005.
Mandate is to take on all forms of distributed computing and to
work to promote cooperation, information exchange, best
practices in use and standardization.
OGF best known for a series of important computing, security
and network standards that form the basis for major science
and business-based distributed computing (BES, GridFTP,
DRMAA, JSDL, RNS, GLUE, UR, etc.).
Have also been working on cloud and Big Data standards
(OCCI, WS-Agreement, DFDL, etc.) for several years.
Cooperative work agreements with other SDOs in place.
Alan Sill, TTU
November 20, 2013
!23
25. OGF Standards Strengths
§
OGF has an extensive set of applicable standards related to federated
community grid and cloud computing:
§
§
§
§
§
§
§
§
§
§
§
§
§
Federated Identity Management (FedSec-CG)
Managing the Trust Eco-System (CA operations, AuthZ and AuthN tools)
Virtual Organizations (VOMS) and related authorization tools
Job Submission and Workflow Management (JSDL, BES, HPC Profile)
Network Management (NSI, NML, NMC, NM)
Secure, fast multi--party data transfer (GridFTP, SRM)
Data Format Description Language (DFDL)
Service Agreements (WS-Agreement, WS-Agreement Negotiation)
Cloud Computing interfaces (OCCI family of specifications)
Distributed resource management (DRMAA, SAGA, etc.)
Firewall Traversal (FiTP); Usage Accounting (UR, GLUE)
Others under development through ISOD-RG, DCIFed-WG, etc.
Working to gather this information to form an organized description of OGF
work - an OGF “Cloud Portfolio”.
Alan Sill, TTU
November 20, 2013
!25
26. OGF Standards In Use In Industry:
• DRMAA: Distributed Resource Management Application API
Grid Engine, Open Grid Scheduler: (open source); TORQUE and related
products: Adaptive Computing; PBS Works: Altair Engineering; Gridway:
DSA Research; Condor: U. of Wisconsin / Red Hat;
• OGSA® Basic Execution Service Version 1.0 and BES HPC Profile:
BES++ for LSF/SGE/PBS: Platform Computing; Windows HPC Server
2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering;
• JSDL: Job Submission Description Language (family of specs):
BES++ for LSF/SGE/PBS and Platform LSF: Platform Computing;
Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client
only): Altair Engineering;
• WS-Agreement (family of specifications):
ElasticLM License-as-a-Service: ElasticLM; BEinGrid SLA Negotiator,
LM-Architecture and Framework: (Multiple partners); BREIN SLA
Management Framework: (Multiple partners); WSAG4J, Web Services
Agreement for Java (framework implementation): Fraunhofer SCAI.
Alan Sill, TTU
November 20, 2013
!26
27. OGF Infrastructure Area
The OGF Infrastructure groups explore and define what is needed to
interface physical and virtual resources to higher level constructs. These
include networks and network devices, computers and virtual machines,
storage, visualization devices, instruments, and sensor technologies.
•
•
•
•
•
•
Network Measurements Working Group (NM-WG)
•
•
Firewall Virtualization For Grid Applications Working Group (FVGA-WG)
Network Measurement And Control Working Group (NMC-WG)
Network Mark-Up Language Working Group (NML-WG)
Network Service Interface Working Group (NSI-WG)
Open Cloud Computing Interface Working Group (OCCI-WG)
Infrastructure Services On-Demand Provisioning Research Group
(ISOD-RG)
Grid High-Performance Networking Research Group (GHPN-RG)
Alan Sill, TTU
November 20, 2013
!27
29. InterCloud-Related Standardization Activities
•
NIST Cloud definition (NIST SP 800-145), NIST Special Publication 500-291 version 2, NIST
Cloud Computing Standards Roadmap, July 2013and Cloud Computing Reference Architecture
(CCRA), v1.0 (NIST SP 500-292) http://www.nist.gov/itl/cloud/publications.cfm
•
ITU-T Focus Group on Cloud: Technical Report (Part 1 to 7)
http://www.itu.int/en/ITU-T/focusgroups/cloud/Documents/FG-coud-technical-report.zip
•
IEEE - WGs on InterCloud issues and Cloud Profiles
– IEEE ICWG/2302 WG - Intercloud WG (ICWG) Working Group
http://standards.ieee.org/develop/wg/ICWG-2302_WG.html
– IEEE P2301, P2302 Projects
http://standards.ieee.org/develop/project/2301.html, 2302.html
• OGF ISOD-RG
– On-Demand Infrastructure Services Provisioning Best Practices:
http://ogf.org/documents/GFD.208.pdf
• IETF Internet Drafts
– Cloud Reference Framework. Internet Draft, by B. Khasnabish, J. Chu, S. Ma,Y. Meng, N. So, P.
Unbehagen, M. Morrow, M. Hasan,Y. Demchenko
http://tools.ietf.org/html/draft-khasnabish-cloud-reference-framework-05.txt
– Cloud Service Broker, Internet Draft by Shao Weixiang, Hu Jie, Bhumip Khasnabish.
http://tools.ietf.org/html/draft-shao-opsawg-cloud-service-broker-03.txt
Alan Sill, TTU
November 20, 2013
!29
30. Open Cloud Computing Interface®
OCCI® by OGF
• OCCI is an API and Protocol
• Sits on the boundary of a Service Provider
and Service Consumer
http://ogf.org/documents/GFD.183.pdf
• No assumptions about the boundary
http://ogf.org/documents/GFD.184.pdf
http://ogf.org/documents/GFD.185.pdf
Many real-world implementations!
Alan Sill, TTU
November 20, 2013
!30
31. Federation Test bed – September 2013
FedCloud Core Services
AAI
EGI Production
Infrastructure
Service
Availability
SAM
Configuration
GOCDB
DB
Accounting
APEL
External Services
Certification
Authorities
Image Metadata
Marketplace
VOMS Proxy
User Interfaces
OCCI Clients
rOCCI; WNoDeS-CLI
CDMI Clients
Libcdmi-java
Information
System
Top-DBII
Resource Providers
VOMS proxy
OCCI server
CDMI server
!
OpenNebula
OpenStack
StratusLab
WNoDeS
Marketplace
client
Appliance
Repository
SSM
Vmcatcher
Vmcaster
LDAP
Monitoring
Nagios
VO
VOMS server
Credit: David Wallom
Chair EGI Federated Cloud Task Force
32. Task Force – Sept 2013
Cyfronet
FZJ
OeRC
EGI.eu
CESNET
GWDG
IN2P3
CNRS
KTH
FCTSG
Members
Technologies
•OpenNebula.
•StratusLab.
•OpenStack.
•Synnefo.
•WNoDeS.
•70 individuals
•40 institutions
•13 countries
CETA
Masaryk
INFN
CESGA
SARA
IGI
RADICAL
STFC
TUD
Stakeholders
IFCA
•23 Resource Providers
•10 Technology Providers
•7 User Communities
•4 Liaisons
SZTAKI
BSC
GRNET
Imperial
DANTE
LMU
IPHC
IISAS
SixSq
100%IT
IFAE
SRCE
Credit: David Wallom
Chair EGI Federated Cloud Task Force
33. Multiple Organizations Have Cloud Standards
Multiple organizations have their own APIs, and real work has
been done by standards organizations to catch up and offer
common standards and products that can work across these.
Now is the time to do this! Recent work has brought several
cloud-related standards into frameworks that can be used to
implement them (for example, CloudStack, OpenStack, Open
Nebula, and several others) and are mature enough to apply:
n
n
n
n
n
n
n
Open Cloud Computing Interface (OCCI) from OGF
Cloud Data Management Interface (CDMI) from SNIA
Data Format Description Language (DFDL) from OGF
Open Virtualization Format (OVF) from DMTF and ISO
Cloud Application Management Protocol (CAMP) from OASIS
Cloud Infrastructure Management Interface (CIMI) from DMTF
WS-Agreement and WS-Agreement Negotiation from OGF
Alan Sill, TTU
November 20, 2013
!33
34. Standards Organizations Active in Cloud Computing:
(Some of many...)
•
It is often said that there are “too many standards organizations”. This is a lot like
saying there is “too much software”.
•
Each has its own area of specialty, its own contributor base, and its own method
of funding to develop its work products.
•
How best to work with these organizations? (Our answer: Cooperatively!)
Alan Sill, TTU
November 20, 2013
!34
37. Who is DMTF
• Established in 1993 to enable more effective management of millions of
IT systems worldwide by bringing the IT industry together to collaborate
on the development, validation and promotion of systems management
standards.
• The group spans the industry with 160 member companies and
organizations, and more than 4,000 active participants crossing 43
countries.
• Strong Alliance Partnership with GICTF and other Organizations
• The DMTF board of directors is led by 17 innovative, industry-leading
technology companies. They include; Broadcom Corporation; CA
Technologies.; Cisco; Citrix Systems, Inc.; Fujitsu; HP; Hitachi, Ltd.;
Huawei; IBM; Intel Corporation; Microsoft Corporation; NetApp; Oracle;
Software AG; SunGard Availability Services; Telecom Italia and VMware,
Inc.
38. Open Virtualization Format (OVF)
XML
myapp.ovf
A standard packaging format for virtual machines
A distribution format for VMs
Supports single VM & multiple VM configurations
Optimized for distribution & simple automation
Vendor and platform independent
Now an ANSI and ISO standard
myapp.mf
myapp.cert
An OVF package consists of
One OVF descriptor with extension .ovf
zero or one OVF manifest (w/ extension .mf)
zero or one OVF certificate (w/ extension .cert)
zero or more disk image files
zero or more additional resource files (such as ISO images)
web.xxx
images.iso
OVF Package (myapp.ova)
OVF 2 (Recently Released): brings an enhanced set of networking
capabilities making it applicable to a broader range of use cases that are
emerging as industry enters the Cloud era.
39. Cloud Infrastructure Management Interface (CIMI)
Model Scope: Core IaaS functionality
Deploying and managing: Machines, Volumes, Networks, Monitoring, etc.
Client: VM/application developer, deployer and administrator
Server: IaaS Cloud Provider
Version 1.1 is released and implemented
CIMI REST/HTTP-Based Protocol
Specification currently describes a REST/HTTP binding to the model.
Other bindings are Considered.
Follows REST principles and describes mapping of the HTTP protocol verbs to operations on
the model.
Standard HTTP status codes are used to convey the results of the operations.
Serialization formats for the message body include JSON and XML
Other features:
Cloud Entry Point to manage Systems, Machines, Volumes, and Networks; Grouping of
resources meant to be managed as a single unit; Metering and Monitoring support; Resource
metadata; Events and event logging; Jobs; Dynamic Discovery of a provider’s resources
characteristics; Entity creation using Templates
40. Cloud Auditing Data Federation (CADF) Working Group
Scope / Objectives
Develop Standards for the Federation of Cloud Auditing Data
By Specifying a Normative, Prescriptive Auditing Event Data Format along with
Interface Definitions and a compatible Component and Interaction model.
The Data Model will include support for:
Classification by Extensible Event Taxonomies to categorize cloud provider IT
Resources, event Actions and Outcomes.
Federation of Customized Auditing Reports and Logs - event data will support
federation and be composable into customizable reports and logs.
The Interface Model includes:
Definition of Service Methods to Manage and Federate the Data Model’s Events,
Logs and Reports
Interfaces will support audit data Submission, Import and Export, Query and
Subscription.
The Component and Interaction Model will
Demonstrate how the Interfaces and Data Format can be used by Cloud Providers
and Consumers to Support Cloud Auditing use cases.
Future work may include Profiles that extend the core data and interface specifications
to accommodate particular methods of consumption
41. DMTF NSMWG Focus: Virtualized and Hybrid
Network Environment Management
Apps
and
Services
that
Utilize
Virtualized
L3
Resource/Entities
Apps
and
Services
that
Utilize
Virtualized
Network
Entities
Management
of
Virtualized
Network
Entities
vNE
Management
API
Virtualized
Network
Entities
Network
Entities
Abstraction
Edge/Core/Border
Router Fire
Wall
*
AAA
Server
…
DNS
D
N
S
… Balancer
Load
Physical
and
Virtual
Network
Entities
42. Software Entitlement Working Group
Purpose:
To extend the Common Information Model to capture software entitlement and usage
metrics and to deliver the associated profile.
Based on ISO/IEC 19770-2 Software Identification Tags
Driven by a request for JP Morgan Chase.
1. Update CIM to support Software Entitlement and Usage Metrics – Q4 2013
2. Create CIM Software Entitlement and Usage Metrics Profile - Q1 2014
For more information
www.dmtf.org/cloud
www.dmtf.org/OVF
7
44. Cloud Plugfest Developer Series:
(Multiple Partners)
Developer-oriented inperson standards and
software testing series
OPEN TO ALL!
http://
cloudplugfest.org
Cloud Plugfest 10 just
completed!
Continuing series
co-sponsored by
OGF, DMTF, SNIA,
OASIS, ETSI,
OCEAN and OW2!
9 previous events
held so far!
Easy to get involved
and join in events as
developers or project
researchers!
Cloud Interoperability
Week Sep. 2013!
Alan Sill, TTU
November 20, 2013
!44
45.
46. NIST SAJACC Public Process
http://collaborate.nist.gov/twiki-cloud-computing/bin/view/CloudComputing/SAJACC
Alan Sill, TTU
November 20, 2013
!46
48. NIST SAJACC Phase I Cloud Computing Use Cases
•
•
•
Public project to define testable
use cases that can provide the
basis for independent evaluation
of cloud standards, products and
processes.
Phase I completed and working
group report “SAJACC Working
Group Recommendations to
NIST” delivered Feb. 12, 2013.
Working group continues with
Phase II to define and refine use
cases with greater technical
detail.
49. Example Work In Progress:
Reorganize
and rewrite
previous
SAJACC Use
Cases
50. Example Work In Progress:
Incorporate
input from other
ongoing NIST
cloud computing
working groups
52. Example Work In Progress:
Include diagrams where
appropriate to improve clarity
of the logic sequence and
workflow of a complex
operation, step or procedure.
53. European Union “Standards & Interoperability for e-Infrastructure
Implementation” (SIENA) Initiative (2010-2012)
SIENA
(2010-‐2012)
was
a
Support
AcAon
funded
by
the
European
Commission
under
FP7
(2007-‐13)
CapaciQes
programme.
Partners
“A
coordinated
effort...
Final Report:
June 2012
...towards
the
delivery
of
a
future
e-‐Infrastructures
Roadmap....
InternaAonal
Experts
REB
–
Roadmap
–
Special
Liaison
–
Industry
Expert
SLG
IEG
Editorial
Board
GroupGroup
...aligned
with
the
needs
of
European
and
na>onal
ini>a>ves
and
the
evolving
world.”
Standards
Development
OrganisaAons
(SDOs)
Distributed
CompuAng
Infrastructure
(DCI)
IniAaAves
Consolidated
Cloud
CompuAng
Workshops
Alan Sill, TTU
November 20, 2013
!53
54. European Cloud Standards Coordination (New)
Multi-organization open survey commissioned by EC,
coordinated by ETSI along same lines as GICTF, NIST
ra
D
In
ft
ro
P
re
g
s!
s
Alan Sill, TTU
November 20, 2013
!54
55. Summary
p
Many active projects are underway to document, map and extend the
important role played by standards and software development with
significant uptake in advanced distributed computing, including
cloud, grid, networking and large-scale data processing, transfer and
handling through innovative cooperation with many partners.
p
The CAC actively engages with partners and participants throughout
the international arena to understand and promote best practices
and standards in cloud and advanced distributed computing.
p
TTU is leveraging these standards to support a wide variety of
flexible architectures for advanced scientific and business uses
through the NSF Cloud and Autonomic Computing Center and is
actively seeking international collaborators to extend this work more
comprehensively through the cloud.
Alan Sill, TTU
November 20, 2013
!55