EGI Cloud Compute service for EOSC-hub

1. eosc-hub.eu
@EOSC_eu
EOSC-hub receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 777536.
Run virtual machines on demand with complete
control over computing resources
EGI Cloud Compute

2. 2
Motivation and driving consideration about the service
Service architecture and interfaces: overview
- How the user can access the service
 E.g.: REST, GUI, CLIs, etc.
- Service options and attributes
Acceptable Usage Policy (AUP)
Access policy and business model
Use cases
Documentation/tutorial/information
11/27/2018
Content

3. 3
EGI Cloud Compute provides users with a distributed computing service
to deploy and scale virtual machines on-demand. It offers access to API-
controlled computational resources in a secure and isolated
environment without the overhead of managing physical servers.
Cloud Compute offers the possibility to select pre-configured virtual
appliances (e.g. CPU, memory, disk, operating system or software) from
a catalogue replicated across all EGI cloud providers. With Cloud
Compute you can:
- Execute compute- and data-intensive workloads (both batch and interactive)
- Host long-running services (e.g. web servers, databases or applications servers)
- Create disposable testing and development environments on virtual machines
and scale your infrastructure needs
- Select virtual machine configurations (CPU, memory, disk) and application
environments to fit your requirements
- Manage your resources in a flexible way with integrated monitoring and
accounting capabilities
11/27/2018
Motivation

4. Cloud Management
Framework
IaaS API
Cloud Management
Framework
IaaS API
Direct API
Access
411/27/2018
Service architecture and interfaces
EGI Federation Services:
Accounting, Monitoring, Conf. DB, Info Discovery, AppDB
EGI Resource Center A
AppDB VMOpsGUI Access
IaaS Federated Access Tools
Federated
Access
EGI Resource Center B

5. 5
GUI access:
- AppDB VMOps https://dashboard.appdb.egi.eu/vmops
API/CLI access:
- Discovery: AppDB IS API (REST and GraphQL)
https://wiki.egi.eu/wiki/Federated_Cloud_Discovery#AppDB
- IaaS Federated Access Tools:
https://wiki.egi.eu/wiki/Federated_Cloud_IaaS_Orchestration
- Direct IaaS access, several APIs depending on the provider:
https://wiki.egi.eu/wiki/Federated_Cloud_APIs_and_SDKs
11/27/2018
Service access

6. 6
Service Options:
- General purpose instance: base performance instance type with
up to 8 cores and up to 4 GB RAM per core
- Compute-intensive instance: optimised instance for cpu-bound
tasks. Up to 64 cores (non-overcommitted) and possible low-
latency network
- High-memory instance: optimised instances for tasks with big
memory requirements. Up 120GB RAM per core.
Service attributes:
- Associated storage to each VM (GB)
- Access type: opportunistic (non guaranteed resources) or
reserved
- Start of service (date) and duration (days)
- API of the provider (OpenStack, OCCI)
11/27/2018
Service options and attributes

7. 7
EGI AuP:
- https://documents.egi.eu/public/ShowDocument?docid=2623
Further VO-level AuPs may also apply
11/27/2018
Acceptable Usage Policy

8. 8
Access policies:
- Policy-based
- Wide-access
- Market-drive
Payment model:
- Sponsored (free for a certain quota)
- Wide-access
- Market-driven
11/27/2018
Access policies and Funding models

9. Access to the EGI Cloud
Virtual Organisations = Resource pools
VO 1
(cloud a, b, c)
VO 2
(cloud b, c, d, e, f)
1. Community-specific VOs – e.g. CHIPSTER, Highthroughtputseq, EISCAT, etc. (SLA, OLAs)
2. Training VO = training.egi.eu  To be used today
3. Generic VOs – e.g. fedcloud.egi.eu  Incubator for new users (recommended for follow up)
Browse VOs at http://operations-portal.egi.eu/vo/search (both grid and cloud)
c
e
f
b
a
d

10. 10
Compute and data intensive workloads
- Batch and interactive (e.g. Jupyter notebooks) with
scalable and customized environments
Service Hosting
- Long-running services (e.g. web server, database,
application server)
Datasets repository
- Store and manage large datasets (in a storage volume)
Disposable and testing environments
- Host training environments, test applications
11/27/2018
Use cases - Scenarios

11. Appliances Marketplace
(AppDB)
Clouds of the Chipster
Virtual Organisation
Virtual Appliances
of the Chipster VO
Chipster
VM
VM
VM
VM
VM
Cloud calls
(CMD/API)
VM
Block
storage
Chipster
portal
Push VM image
Prepare,
register
VM image
Chipster
developer
(CSC, Finland)
Bioinformaticians
Analysis software
contains over 300
analysis tools for NGS,
microarray, proteomics
and sequence data.
VM
Block
storage
Read more...
Cache
Cache
Cache
Cache
Use case – Sharing community
applications: Chipster

12. 11/27/2018
Use case - Scalable bioinformatics web-
servers powered by cloud computing: NBIS
A distributed Swedish
research infrastructure
providing bioinformatics
support to life science
researchers.
Query Frontend
via web-page or API
Clouds of the NBIS
Virtual Organisation
VM
VM
VM
shared
storage
Read more…
Scalable handling of
jobs using VMs from
EGI Cloud Compute
NBIS Servers powered by EGI:
Pcons.net
TOPCONS
SCAMPI2
ProQ3
PconsC2/PconsC3
BOCTOPUS2

13. Use case – EMSO Data Management
platform
Data access
DMP
tools
DATA MANAGEMENT PLATFORM
Asynchreal-timeingestion.
Data
ingestion
ENVRI Reference Model v2.0
Collection and Analysis of Ocean Big Data: EGI
Cloud hosts EMSO Data Management Platform
services providing acquisition, processing,
publishing and curation capabilities of all data
collected by deep sea observatories
Read more…
EMSO DMP is powered by Apache Spark and
Apache Zeppelin running on EGI Cloud.

14. Use case – GeoHazards TEP
Read more…
Cloud platform developed and
operated by Terradue.
It provides a rich set of ready
to use EO Data processing
services for geohazards
analysis and monitoring.
EGI Cloud Compute
contributes resources to run
near real-time processing of
satellite data acquisitions.
www.terradue.com
User Communities
4
Image taken from slides at https://indico.egi.eu/indico/event/3455/session/13/contribution/84

15. 15
User documentation:
- https://wiki.egi.eu/wiki/Federated_Cloud_user_support
Providers documentation:
- http://egi-federated-cloud-integration.readthedocs.io/
11/27/2018
Documentation

16. eosc-hub.eu @EOSC_eu