NextGEOSS Webinar - Cloud APIs

1. Cloud APIs NextGEOSS Webinar, November 28th 2019 Hervé Caumont (Terradue) Bjorn Backeberg (EGI.eu)

2. ● Introduction on the use of Cloud APIs within NextGEOSS ○ Hervé Caumont (Terradue) ● EGI.eu Federated Cloud APIs for a uniﬁed UX in a distributed cloud infrastructure ○ Bjorn Backeberg (EGI Foundation) ● NextGEOSS Cloud Bursting Service - Terradue support to application builders for multi-cloud deployment of Pilot Applications ○ Hervé Caumont (Terradue) AGENDA Platform Integration

3. 1 Hervé Caumont, Terradue Introduction on the use of Cloud APIs within NextGEOSS

4. User Management AAA NextGEOSS DataHub Usage Analytics User Management SSO Usage Analytics Catalog Discovery User Management AAA Geospatial User Feedback User Management SSO

8. Jupyter Notebooks NextGEOSS DataHub Usage Analytics Tasks ? Your Service ! Usage Analytics Compute Geospatial tagging User Management AUTHZ Geospatial User Feedback User Management SSO

9. Cloud Integration API ● Offer ○ Make your data processing algorithms scalable & portable across Cloud Providers, and directly accessible in a standard way ○ Get your web applications more robust in exploiting data sources ○ Build applications based on interoperable, distributed systems ○ Benefit from standard integration layers and focus on your core business Infrastructure-as-a-Service for application design and integration Standard APIs to stage data, run jobs & and package Apps eScience tools to build laboratory notebooks and scalable workflows Cloud Sandboxes Application Frameworks

10. User Management API ● Offer ○ Make users of your application authenticated via Single-Sign-On (SSO) ○ Make your API calls mediated via Authentication, Authorization and Accounting (AAA) ○ All of it in a federated environment (including the NextGEOSS Platform Services!) ○ Get intelligent access control to: ■ monitor your “free access” resources usage, ■ enforce other data or service policies, ■ serve your auditing needs ○ Your application is uplifted to the level of effective management & security We establish digital trust with the identity providers (IdP) of your user community Your user community can join your application without changing their sign-in habits You define the access levels for people working with your application authentication authorisation

11. Data Discovery API NextGEOSS DataHub is powered by A unique access point to the wide range of European Earth observation data Applications can remotely access Standard APIs, to systematically feed their data processing needs Data Hub users can explore Earth observation data, share comments, suggestions and rate the contents ● Offer ○ Make use of the NextGEOSS Data Hub standard API to fetch and consume data resources within your applications ○ Get high flexibility and robustness for your data sourcing process. ○ Tap into distributed data repositories from one place, without the burden of maintaining multiple data access endpoints and protocols.

12. Geospatial User Feedback API ● Offer ○ Add user feedback capabilities to resources in your community portal, using recognized OGC standards ○ Get user experiences using your resources ○ Help build a knowledge base and a collaborative platform for your community portal ○ Get your community portal more interactive ○ Supplement the information you already provide about your data ○ Get better community engagement and an increase trust in your services By simply adding a link or button, retrieve previous feedback items Widget integration label Add new elements or edit previous onesWidget functions Widget customization Use CSS to easily change styles and to hide/show desired information

13. Cloud Bursting API ● Offer ○ Make your Cloud-ready application available as a “Cloud appliance” on the Platform repository ■ Decide on your Cloud appliances access conditions: only to your organisation, shared with partner organisations, ... ○ Leverage the “deployers” previously setup for you, according to your existing Cloud provisioning arrangements ■ Your credited accounts for a Cloud Provider, … ○ Adapt your Cloud deployment strategies as new opportunities come to you Ensure Cloud provisioning is performed according to the credit lines in place Get access to the deployer service matching your Cloud provider resources Define configuration and “deploy” ! Check your application service endpoints in production, and run operations Cloud Controller APIs Deployer APIs

14. User Management AAA NextGEOSS DataHub Usage Analytics User Management SSO Usage Analytics Catalog Discovery User Management AAA Geospatial User Feedback User Management SSO Our focus for today’s Webinar

15. 2 Bjorn Backeberg, EGI.eu EGI.eu Federated Cloud APIs for a unified UX in a distributed cloud infrastructure

16. What is the EGI FedCloud? ● Offer ○ Multi-cloud Infrastructure as a Service (IaaS) ○ Harmonized access to participating cloud sites ○ Hybrid Cloud including public and commercial infrastructures ○ Technology agnostic, support OpenStack, OpenNebula and Synnefo ● Key features ○ Single Sign-On ○ Virtual Appliance catalogue and unified GUI dashboard ○ Resource discovery ○ Customer Relationship Management ■ Centralised accounting ■ Service Level Agreement monitoring Cloud Compute Run Virtual Machines on demand similar to AWS EC2/EBS or GCP Compute Engine EGI Services powered by the FedCloud Cloud Container Compute Online Storage Training infrastructure Applications on demand Notebooks

17. ● The EGI CLoud Infrastructure Platform (CLIP): ○ an abstract Cloud Management stack subsystem ○ integrated with components of the EGI Core Information Management System ● CLIP: a thin layer of federation and interoperability services around local deployments and integrations of Cloud Management stacks. ● Cloud Management stack deployments must provide at least one of these IaaS interaction ports preferably using standardised APIs, specifically OCCI for VM management, and CDMI for object storage. The (original) Federation Model

18. Challenges with OCCI ● It turned out that it was difficult to impose OCCI API as the single IaaS interaction port for providers and users ● OCCI lacked support in user tools and had a limited feature list ⇨ Necessitated manual management processes of the IaaS ⇨ Limited possibility for adopting cloud-native approaches where resources are managed automatically via API calls ➠ EGI decision to allow providers to decide for themselves which API standard to implement ● EGI focus shift towards ○ building tools to enhance the user experience → including common interfaces towards a unified system ○ automate resource provisioning

19. The current access layers IaaS Federated Access Tools layer helps users of the cloud to deal with the heterogeneity in the IaaS API and EGI Federation services Providers have their APIs that can be used with EGI Check-in accounts, opening the door to automation of cloud-native applications. EGI Federated Cloud no longer mandates a single API for every provider. OCCI still widely supported but sites are moving to native APIs (mainly OpenStack!)

20. Typical user workflow

21. EGI Applications Database Registry for virtual appliances (VA) ✓ a logical container of versioned image file & metadata bundles VA distribution medium ✓ distributing endorsed VAs to the resource providers/sites Resource providers and VO catalogue ✓ list of the VAs which are available by each site/resource provider for every VO https://appdb.egi.eu/ The base URI of the RESTful API is: http://appdb.egi.eu/rest/1.0

22. EGI VMOps Dashboard AppDB VMOps - User Friendly GUI ● Single Web dashboard to manage VMs in the federation ○ Point-and-click solution to create new VMs ● Powered by Infrastructure Manager ● Integrated with: ○ Single Sign-On, discovery, VM catalogue, monitoring APIs used for the Dashboard: ● Check-in OpenID Connect for federated AAI ● AppDB information system API (GraphQL), discover VA, providers, VOs ● ARGO monitoring information, to query the status of the providers ● OpenStack/OCCI/OpenNebula APIs, via IM (used by VMOps) / Terraform ● Create VMs using information from APIs (send jobs to cloud)

23. The EGI fedcloud today EGI FedCloud consists of 23 distributed IaaS providers Activities coordinated by the EGI Cloud Task Force Users from a broad range of scientific disciplines supported >3,000 computing hours >600,000 VMs deployed

24. ● Expand the capabilities and capacity of the federated cloud: ○ GPGPU, High-memory and compute intensive VMs ○ Improve discovery ● Move towards API-based management of resources ○ EGI involved in SLA setup ○ Users deploy their own resources with the tools of their choice: VMOps, IaaS Orchestration, direct API access, their own tooling ● Simplify providers integration ○ Centralise the operation of the federation features (e.g. connection to AppDB) ○ Liaise with commercial providers Future plans

25. 3 Hervé Caumont, Terradue NextGEOSS Cloud Bursting Service Terradue support to application builders for Cloud bursting (multi-cloud deployment of Pilot Applications)

26. Cloud Bursting: some use cases ● Multiple deployment of a same application ○ For different target user audiences ○ and/or for different production campaigns, requiring to secure dedicated Cloud resources over a given period of time ● Multiple deployment of different versions of a same application ○ According to different funding context ■ e.g. a stakeholder funding a new evolution of the application for its own usage only

27. Ellip Solutions, for EO Application Integration ● Integration environment for the test & validation of EO data processing components developed in several programming language: ○ C/C++, Java, Python, R, Matlab, IDL ● Automated build & packaging, based on workflow code wrappers to ensure scalability ● Cloud bursting (deployer engine) to selected Production environment (Cloud providers)

28. Cloud APIs, for Hybrid Cloud capability Openstack API powered by libcloud CloudFerro IaaS EODC Cloud powered by jclouds Commercial Providers OCCI, & native cloud APIs EGI Federated Cloud OpenNebula Cloud Controller Terradue ONDA DIASSobloo DIAS Openstack API powered by libcloud Openstack API powered by libcloudOpenstack API - powered by libcloud CREODIAS

29. Cloud Bursting: some use cases ● Multiple deployment of a same application ○ For different target user audiences ○ and/or for different production campaigns, requiring to secure dedicated Cloud resources over a given period of time ● Multiple deployment of different versions of a same application ○ According to different funding context ■ e.g. a stakeholder funding a new evolution of the application for its own usage only ● This is supported in NextGEOSS by the “Ellip Launchpads” solution: ○ Currently in beta release ○ Deployments are still operated by the Operations support team at Terradue ■ on behalf of application owners, ■ after agreement on the target Cloud resources to use for the deployment in production

30. Cloud Bursting service - Operations ● Deploy and operate the NextGEOSS Cloud resources in support of “Application Provider” needs: ○ After the setup and validation of your Cloud Appliance on your Cloud Integration environment ■ Based on the configured Cloud Platform repositories per Pilot service ■ Based on the configured Cloud Platform deployers per Pilot service ○ And for the authorised and selected Cloud Production environments per Pilot ■ Appliance deployment on the ‘production’ ICT resources (Cloud bursting) ● Production servers on EGI federated Cloud using the allocated budget ● Or other IaaS as needed ■ Then start of the Operations: ● Partner is in control of the application running (directly, or via the authorised Community Portals)

31. SATCEN Pilot – Application deployed on AWS and Hetzner Reference to the Cloud appliance to be deployed Application owner: Operations Support:

32. SATCEN Pilot – Application deployed on AWS and Hetzner Simple configuration fields to define the “Deployers”

35. SATCEN Pilot – Application deployed on AWS and Hetzner Submit request to the Launchpad service to perform the deployment

36. Cloud Bursting service - State of play ● Resources for developers ○ NextGEOSS Integration Guide (v2) ○ Ellip Solutions user guide ■ How to join / My account ■ Ellip Core services ○ UPCOMING: Ellip Launchpads user guide ● Support activities ○ As part of the NextGEOSS support team ■ On the NextGEOSS Service Desk system ○ As part of the user training sessions ■ Hands-on workshops with project partners ■ Face to face trainings (Sept. 2018 & upcoming July 2019) ■ Webinars (April 2018, Nov. 2019) ○ Dedicated support telcos (on-demand) ○ Support site for business partners ■ https://support.terradue.com ● Currently used for operations by ○ NextGEOSS Platform Services ■ Sentinel-2 Biopar vegetation indices Cloud appliance for crops monitoring and food security policies, by VITO ■ Environmental predictors and RS-EBVs model Cloud appliance for Biodiversity Habitat Monitoring, by WENR ■ CAMS Solar radiation gridded data Cloud appliance for photovoltaic plants production monitoring, by ARMINES ■ Sentinel-1 Change Detection Cloud appliance for humanitarian relief and border security, by SATCEN ■ Sentinel-1 Sea Ice concentration Cloud appliance for Cold Regions monitoring and policy making, by NERSC ■ Sentinel-1 InSAR-based ground velocity mapping Cloud appliance for Disaster Risk Assessment, by NOA ■ Sentinel-2 Leaf Area Index Cloud appliance, by ITC ■ UPCOMING: ■ Sentinel-2 Land Use Cloud appliance, by DEIMOS ■ Objects Drift Model, by CLS ■ Wetlands water management, by FSU Jena ○ Other deployments in operations ■ ESA GEP - Geohazards Exploitation Platform (30+) ■ ESA HEP - Hydrology Exploitation Platform (15+) ■ H2020 INTAROS - integrated Arctic Observing System ■ H2020 Co-ReSyF - Coastal Areas Research Platform ■ H2020 EcoPotential - Protected Areas EO data products

37. Cloud Bursting service ● Offer ○ Make your Cloud-ready application available as a “Cloud appliance” on the Platform repository ■ Decide on your Cloud appliances access conditions: only to your organisation, shared with partner organisations, ... ○ Leverage the “deployers” previously setup for you, according to your existing Cloud provisioning arrangements ■ Your credited accounts for a Cloud Provider, … ○ Adapt your Cloud deployment strategies as new opportunities come to you Ensure Cloud provisioning is performed according to the credit lines in place Get access to the deployer service matching your Cloud provider resources Define configuration and “deploy” ! Check your application service endpoints in production, and run operations Cloud Controller APIs Deployer APIs

38. Data analysis systems built with our solutions Run and monitor data processing jobs OGC Web Processing Service (WPS) interface for management of processing jobs OGC Web Processing Service (WPS) accessed from a Jupyter Notebook

39. Looking forward hearing from you! NextGEOSS Webinar, November 28th 2019 Hervé Caumont (Terradue) Bjorn Backeberg (EGI.eu) Cloud APIs