OGC Interfaces in Thematic Exploitation Platforms

19 Nov 2019

Contenu connexe

Présentations pour vous(20)


OGC Interfaces in Thematic Exploitation Platforms

  1. ® Sponsor OGC Interfaces in Thematic Exploitation Platforms 113th OGC Technical Committee Toulouse, France Pedro Gonçalves, Terradue 18 November 2019 Copyright © 2019 Open Geospatial Consortium
  2. OGC ® Thematic Exploitation Platforms • R&D activities started in 2016 to create an ecosystem of Thematic Exploitation Platforms • Users access a work environment containing the data and resources, as opposed to downloading and replicating the data “at home” The fundamental principle is to move the user to the data and tools Copyright © 2019 Open Geospatial Consortium
  3. OGC ® Fields in each Thematic Domain • Geohazards domain – Earthquakes, Volcanoes, Landslides … • Hydrology domain – Hydrological modelling, Flood mapping, Water quality monitoring … • Data-intensive research – Targeted data – Targeted software Copyright © 2019 Open Geospatial Consortium
  4. OGC ® Data and ICT Resources for Exploitation • Fast access & large volume – EO data – In-situ data • Distributed processing – Hybrid Cloud infrastructure – Commercial, Public and Private Clouds Copyright © 2019 Open Geospatial Consortium
  5. OGC ® Platform Tools and Capacity • Collaborative tools – Integrating resources (data source, processor) – Publishing Results – Referencing Publications – Sharing processing results • Different roles – Community scientists – Data providers – Resource providers Copyright © 2019 Open Geospatial Consortium
  6. OGC ® Geohazard Exploitation Platform • Platform based on virtualization & federation of EO data – Provide services & support to the geohazards community • On-demand & systematic processing services – Cloud Compute power, managing multi-tenant resources • Access to Copernicus Sentinels – Hundred TBs of EO data archives (ERS & ENVISAT), and EO missions (ALOS-2, Cosmo-Skymed and TerraSAR-X) under CEOS WG Disaster and the GSNL agreements Copyright © 2019 Open Geospatial Consortium
  7. OGC ® Geohazard Exploitation Platform • Users from geoscience centres hosting geohazard experts – skills in EO (to process, analyze, validate data) – generating DRM products for decision-makers – decision-makers are not direct platform users • Initially targeting three main scenarios: – EO Data Exploitation on-demand processing services – New EO Service Development Integrates new or improves processing chain – New EO Product Development Executess a global/systematic processing chain to generate a new derived information layer Copyright © 2019 Open Geospatial Consortium
  8. OGC ® Geohazards Services Portfolio • 25+ on-demand services using Optical & SAR data grouped in Thematic Apps organized by Community Managers • New basic services providing full resolution and change detection imagery for rapid online visualization • 8 systematic services delivering continuously updated information • Large scale production of Sentinel-1 InSAR browse images at 100m and 50m resolution over tectonic regions and volcanoes Copyright © 2019 Open Geospatial Consortium
  9. OGC ® Development Software Available • Libraries and toolboxes: – Orfeo Toolbox – Gdal – pandas/geopandas – NumPy • Environments – Jupyter – Xarray – Dask • Processing – YARN, OOZIE – Kubernetes
  10. OGC ® Some Statistics (Sep 2019) • 12 205 processing CPU hours / month consumed solely by users • Products – Generated: 8872 Published: 7796 Downloaded: 10749 • Overall 251 selected users (121 organisations) – Carrying out projects on the platform via the early adopters programme to use data processing services, publish results and provide feedback to improve the platform • 1545 registered users (96 countries) – To search for EO data acquisitions, shared results from processing jobs, and public information layers
  11. OGC ® API driven • Data Discovery – OGC OpenSearch – Geo & Time Extensions – EO Extensions • Data Packaging – OGC OWS Context • Application Packaging – OGC OWS Context + T13/14 • Data Processing Execution – WPS 2.0 API
  12. OGC ® Data Discovery API • Earth Observation Catalogues from a Portal or a Jupyter • Accessing OGC OpenSearch API directly from Python and explore catalogue contents as Panda/Geopanda data structures
  13. OGC ® • Invoke on-demand data processing services from authorized application Portals or a Jupyter Notebooks with OGC WPS 2.0 API • Analyse and exploit data results On-demand Data Processing API
  14. OGC ® Systematic Data Processing API • Invoke systematic data processing services from authorized application Portals or a Jupyter Notebooks • Analyse and exploit data results
  15. OGC ® Data Processing Application Integration • Applications developed in any programming language supported – C/C++, Java, Python, Matlab, IDL … • Continuous Integration, automated build & packaging, deployment to selected Production environment (Cloud providers) Copyright © 2019 Open Geospatial Consortium
  16. OGC ® EO Application Development Implement an EO application using your IDE on your machine using the client APIs Example shown: PyCharm IDE with EO application to process Sentinel-1 GRD data for sigma0 calculation using SNAP
  17. OGC ® EO Application Version Control Use a software repository like GitHub or GitLab to host the EO application code and files → versioning and well tracked releases Our Sentinel-1 sigma0 SNAP application is is hosted by GitLab
  18. OGC ® EO Application build process Continuous Integration/Deployment environment to build, package and dockerize the EO application release(s) Our Sentinel-1 sigma0 SNAP application is now built, packaged and dockerized
  19. OGC ® EO Application - Package The build, package and dockerize process produces an Application package with: - A docker image - Its associated OWS-Context document The docker image is accessible on a docker-hub
  20. OGC ® EO Application Exploitation - Docker To consume the EO Application docker image: - pull the docker image with the EO Application from the docker-hub - run the docker image in a container
  21. OGC ® EO Application Exploitation - Deploy On TEPs, the Application Package is deployed and the EO Application exposed as a Web Processing Service (WPS 2.0 API)
  22. OGC ® EO Application exploitation on a TEP The deployed Application Package as a Web Processing Service (WPS 2.0 API) on a TEP is exploited using Cloud resources (e.g. DIAS)
  23. OGC ® EO Application Exploitation on a Notebook Consume the EO application via WPS API 2.0, from a Jupyter Notebook or desktop application client, to analyse and exploit data results
  24. OGC ® Takeaway Messages • Users benefit from fast access to EO data, storage capacity and processing resources offered by platform-based solutions • A hosted processing platform is a partnership that needs the support of scientist/developer to offer operational services providing value added information to the community • Platform e-collaboration and reproducible knowledge promote innovation and response capacity
  25. OGC ® Platform for Earth Sciences Communities • to share their findings, streamline their creation of new data products and make these accessible, interoperable • communities remain owners of their created assets, and decide how to share these on the platforms, for other users to find and reuse without a platform lock-in • to document their developments as reproducible experiments • with FAIR guiding principles to make data findable, accessible, interoperable and reusable
  26. OGC ® Thank You!
  27. OGC ® accessible by a portal
  28. OGC ®