SlideShare une entreprise Scribd logo

20090701 Climate Data Staging

•Télécharger en tant que PPT, PDF•
•
H
Henning Bergmeyer

This is the presentation version containing animations. Therefore it is not printable. I am going to post a printable version during the next days.

TechnologieActualités & Politique
1  sur  31
A Python Framework for Staging of Geo-referenced Data on the Collaborative Climate Community Grid (C3-Grid) Henning.Bergmeyer [at] dlr.de German Aerospace Center Simulation- and Software Technology (V.09-06-22a) Most up-to-date version of these slides available at: h ttp://tr.im/ep09_pymodest
[object Object],[object Object],[object Object],[object Object],[object Object]
Locations and employees ,[object Object],[object Object],[object Object], Koeln  Lampoldshausen  Stuttgart  Oberpfaffenhofen Braunschweig   Goettingen Berlin -   Bonn Trauen   Hamburg  Neustrelitz Weilheim  Bremen - 
Talk Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
The Heterogeneity Problem with Data in Climate Research ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Scientific Workflow and Use of Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],The Collaborative Climate Community Grid C3-Grid (2005-2009)
Portal-Based Discovery ,[object Object],[object Object],[object Object],[object Object],Advanced Search Browse by Data Set
C3-Grid Layered Architecture (simplified) WDC Climate WDC Mare WDC RSAT DWD DKRZ, PIK, GKSS, AWI, MPI-M IFM-Geomar FU Berlin Uni Cologne Storage Solutions dCache OGSA-DAI Flat Files … heterogeneous! Abstraction: Metadata + Data Request Service ,[object Object],[object Object],[object Object],=> custom staging scripts Portal Workflow Management Data Management Data Information System Archive Users View Grid Middleware Resources
Metadata for Description and Discovery ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data Request Service: Standard Selection Constraint Set ,[object Object],[object Object]
Data Request Service: Data Selection Constraints ,[object Object],[object Object],[object Object],[object Object],log_surface_pressure mole_fraction_of_ozone_in_air …
Data Request Service: Data Selection Constraints ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],log_surface_pressure mole_fraction_of_ozone_in_air …
Delivering Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Becoming a C3-Grid Data Provider ,[object Object],[object Object],[object Object],Data Providers know their data and their tools!
General Implementation of File Stagers for C3-Grid ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
PyModESt Staging Process Skeleton initialize Environment read Stage Request select DataProcessor choose Request Mode handle Cancel Request handle Stage Request handle Estimation Request handle Exceptions tidy Work Space write Responses ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],select DataProcessor ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Necessary Implementation effort for DP when using PyModESt
What the DP does and what is done for her/him ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
PyModESt Modular Extensibility ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
PyModESt: Metadata Update md. removeQuicklook () md. filterContentInfo (self.thesaurus.translateToC3(self.requested_vars, self.object_id)) md. setHorizontalBounds (self.lon_min, self.lon_max, self.lat_min, self.lat_max) md. setVerticalExtent (self.alt_min, self.alt_max, self.alt_verticalcrs) md. updateTimePeriod (self.timeperiod_begin_date, self.timeperiod_stop_date) md. setObjectId (self.object_id + ".“, datetime.utcnow().isoformat().replace(":","-")) md. addLineageProcessStep ( PROCESS_DESCRIPTION, datetime.datetime.utcnow(), self.stage_request.object_ids[0], RESPONSIBLE_PERSON, "http://wis.wmo.int/2006/catalogues/gmxCodelists.xml#CI_RoleCode_distributor", INSTITUTE_IDENTIFIER)
Offer Contracts (Scheduling Agreement) File Size Estimation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Offer Contracts (Scheduling Agreement) Staging Time Estimation ,[object Object],[object Object],[object Object],[object Object]
DLR Example Data Sets ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
DLR Example Data Sets ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Benefits of Using PyModESt ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Open Issues ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusion ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Summary ,[object Object],[object Object],[object Object],[object Object],[object Object]
20090701 Climate Data Staging
20090701 Climate Data Staging
20090701 Climate Data Staging

Recommandé

SomeSlides
SomeSlidesSomeSlides
SomeSlides
guestd60742
 
Recent Upgrades to ARM Data Transfer and Delivery Using Globus
Recent Upgrades to ARM Data Transfer and Delivery Using GlobusRecent Upgrades to ARM Data Transfer and Delivery Using Globus
Recent Upgrades to ARM Data Transfer and Delivery Using Globus
Globus
 
Grid Computing July 2009
Grid Computing July 2009Grid Computing July 2009
Grid Computing July 2009
Ian Foster
 
Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)
Globus
 
Globus: Enabling the Open Storage Network
Globus: Enabling the Open Storage NetworkGlobus: Enabling the Open Storage Network
Globus: Enabling the Open Storage Network
Globus
 
A Data Ecosystem to Support Machine Learning in Materials Science
A Data Ecosystem to Support Machine Learning in Materials ScienceA Data Ecosystem to Support Machine Learning in Materials Science
A Data Ecosystem to Support Machine Learning in Materials Science
Globus
 
Health Sciences Research Informatics, Powered by Globus
Health Sciences Research Informatics, Powered by GlobusHealth Sciences Research Informatics, Powered by Globus
Health Sciences Research Informatics, Powered by Globus
Globus
 
20160922 Materials Data Facility TMS Webinar
20160922 Materials Data Facility TMS Webinar20160922 Materials Data Facility TMS Webinar
20160922 Materials Data Facility TMS Webinar
Ben Blaiszik
 

Recommandé

SomeSlides
SomeSlidesSomeSlides
SomeSlides
guestd60742
 
Recent Upgrades to ARM Data Transfer and Delivery Using Globus
Recent Upgrades to ARM Data Transfer and Delivery Using GlobusRecent Upgrades to ARM Data Transfer and Delivery Using Globus
Recent Upgrades to ARM Data Transfer and Delivery Using Globus
Globus
 
Grid Computing July 2009
Grid Computing July 2009Grid Computing July 2009
Grid Computing July 2009
Ian Foster
 
Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)
Globus
 
Globus: Enabling the Open Storage Network
Globus: Enabling the Open Storage NetworkGlobus: Enabling the Open Storage Network
Globus: Enabling the Open Storage Network
Globus
 
A Data Ecosystem to Support Machine Learning in Materials Science
A Data Ecosystem to Support Machine Learning in Materials ScienceA Data Ecosystem to Support Machine Learning in Materials Science
A Data Ecosystem to Support Machine Learning in Materials Science
Globus
 
Health Sciences Research Informatics, Powered by Globus
Health Sciences Research Informatics, Powered by GlobusHealth Sciences Research Informatics, Powered by Globus
Health Sciences Research Informatics, Powered by Globus
Globus
 
20160922 Materials Data Facility TMS Webinar
20160922 Materials Data Facility TMS Webinar20160922 Materials Data Facility TMS Webinar
20160922 Materials Data Facility TMS Webinar
Ben Blaiszik
 
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Globus
 
Data Orchestration at Scale (GlobusWorld Tour West)
Data Orchestration at Scale (GlobusWorld Tour West)Data Orchestration at Scale (GlobusWorld Tour West)
Data Orchestration at Scale (GlobusWorld Tour West)
Globus
 
What's New in Globus - Internet2 TechEXtra
What's New in Globus - Internet2 TechEXtraWhat's New in Globus - Internet2 TechEXtra
What's New in Globus - Internet2 TechEXtra
Globus
 
Automating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with GlobusAutomating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with Globus
Globus
 
Connecting Your System to Globus (APS Workshop)
Connecting Your System to Globus (APS Workshop)Connecting Your System to Globus (APS Workshop)
Connecting Your System to Globus (APS Workshop)
Globus
 
Globus Portal Framework (APS Workshop)
Globus Portal Framework (APS Workshop)Globus Portal Framework (APS Workshop)
Globus Portal Framework (APS Workshop)
Globus
 
GlobusWorld 2021 Tutorial: Building with the Globus Platform
GlobusWorld 2021 Tutorial: Building with the Globus PlatformGlobusWorld 2021 Tutorial: Building with the Globus Platform
GlobusWorld 2021 Tutorial: Building with the Globus Platform
Globus
 
Gateways 2020 Tutorial - Instrument Data Distribution with Globus
Gateways 2020 Tutorial - Instrument Data Distribution with GlobusGateways 2020 Tutorial - Instrument Data Distribution with Globus
Gateways 2020 Tutorial - Instrument Data Distribution with Globus
Globus
 
Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
Gateways 2020 Tutorial - Automated Data Ingest and Search with GlobusGateways 2020 Tutorial - Automated Data Ingest and Search with Globus
Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
Globus
 
Instrument Data Orchestration with Globus Search and Flows
Instrument Data Orchestration with Globus Search and FlowsInstrument Data Orchestration with Globus Search and Flows
Instrument Data Orchestration with Globus Search and Flows
Globus
 
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with GlobusGateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
Globus
 
Foundations for the Future of Science
Foundations for the Future of ScienceFoundations for the Future of Science
Foundations for the Future of Science
Globus
 
Globus status and publication plans
Globus status and publication plansGlobus status and publication plans
Globus status and publication plans
Ian Foster
 
Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)
Globus
 
GlobusWorld 2020 Keynote
GlobusWorld 2020 KeynoteGlobusWorld 2020 Keynote
GlobusWorld 2020 Keynote
Globus
 
NIH Data Commons Architecture Ideas
NIH Data Commons Architecture IdeasNIH Data Commons Architecture Ideas
NIH Data Commons Architecture Ideas
Ian Foster
 
Gateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to GlobusGateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to Globus
Globus
 
Globus publication demo screenshots
Globus publication demo screenshotsGlobus publication demo screenshots
Globus publication demo screenshots
Ian Foster
 
Architecting An Enterprise Storage Platform Using Object Stores
Architecting An Enterprise Storage Platform Using Object StoresArchitecting An Enterprise Storage Platform Using Object Stores
Architecting An Enterprise Storage Platform Using Object Stores
Niraj Tolia
 
Simplified Research Data Management with the Globus Platform
Simplified Research Data Management with the Globus PlatformSimplified Research Data Management with the Globus Platform
Simplified Research Data Management with the Globus Platform
Globus
 
Edwards climate data detectives - yale 2-2015
Edwards climate data detectives - yale 2-2015Edwards climate data detectives - yale 2-2015
Edwards climate data detectives - yale 2-2015
Paul Edwards
 
Climate data in R with the raster package
Climate data in R with the raster packageClimate data in R with the raster package
Climate data in R with the raster package
CCAFS | CGIAR Research Program on Climate Change, Agriculture and Food Security
 

Contenu connexe

Tendances

Automating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with GlobusAutomating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with Globus
Globus
 

Tendances (20)

Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
 
Data Orchestration at Scale (GlobusWorld Tour West)
Data Orchestration at Scale (GlobusWorld Tour West)Data Orchestration at Scale (GlobusWorld Tour West)
Data Orchestration at Scale (GlobusWorld Tour West)
 
What's New in Globus - Internet2 TechEXtra
What's New in Globus - Internet2 TechEXtraWhat's New in Globus - Internet2 TechEXtra
What's New in Globus - Internet2 TechEXtra
 
Automating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with GlobusAutomating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with Globus
 
Connecting Your System to Globus (APS Workshop)
Connecting Your System to Globus (APS Workshop)Connecting Your System to Globus (APS Workshop)
Connecting Your System to Globus (APS Workshop)
 
Globus Portal Framework (APS Workshop)
Globus Portal Framework (APS Workshop)Globus Portal Framework (APS Workshop)
Globus Portal Framework (APS Workshop)
 
GlobusWorld 2021 Tutorial: Building with the Globus Platform
GlobusWorld 2021 Tutorial: Building with the Globus PlatformGlobusWorld 2021 Tutorial: Building with the Globus Platform
GlobusWorld 2021 Tutorial: Building with the Globus Platform
 
Gateways 2020 Tutorial - Instrument Data Distribution with Globus
Gateways 2020 Tutorial - Instrument Data Distribution with GlobusGateways 2020 Tutorial - Instrument Data Distribution with Globus
Gateways 2020 Tutorial - Instrument Data Distribution with Globus
 
Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
Gateways 2020 Tutorial - Automated Data Ingest and Search with GlobusGateways 2020 Tutorial - Automated Data Ingest and Search with Globus
Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
 
Instrument Data Orchestration with Globus Search and Flows
Instrument Data Orchestration with Globus Search and FlowsInstrument Data Orchestration with Globus Search and Flows
Instrument Data Orchestration with Globus Search and Flows
 
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with GlobusGateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
 
Foundations for the Future of Science
Foundations for the Future of ScienceFoundations for the Future of Science
Foundations for the Future of Science
 
Globus status and publication plans
Globus status and publication plansGlobus status and publication plans
Globus status and publication plans
 
Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)
 
GlobusWorld 2020 Keynote
GlobusWorld 2020 KeynoteGlobusWorld 2020 Keynote
GlobusWorld 2020 Keynote
 
NIH Data Commons Architecture Ideas
NIH Data Commons Architecture IdeasNIH Data Commons Architecture Ideas
NIH Data Commons Architecture Ideas
 
Gateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to GlobusGateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to Globus
 
Globus publication demo screenshots
Globus publication demo screenshotsGlobus publication demo screenshots
Globus publication demo screenshots
 
Architecting An Enterprise Storage Platform Using Object Stores
Architecting An Enterprise Storage Platform Using Object StoresArchitecting An Enterprise Storage Platform Using Object Stores
Architecting An Enterprise Storage Platform Using Object Stores
 
Simplified Research Data Management with the Globus Platform
Simplified Research Data Management with the Globus PlatformSimplified Research Data Management with the Globus Platform
Simplified Research Data Management with the Globus Platform
 

En vedette

Document IT Communicate IT Succeed
Document IT Communicate IT SucceedDocument IT Communicate IT Succeed
Document IT Communicate IT Succeed
Deborah Gleason
 
Andreas Hense: Climate data for our future – acquired, analysed, archived
Andreas Hense: Climate data for our future – acquired, analysed, archivedAndreas Hense: Climate data for our future – acquired, analysed, archived
Andreas Hense: Climate data for our future – acquired, analysed, archived
"Open Access - Open Data" conference, 13th/14th December, 2010
 
ICLR Friday Forum: Climate data in Ontario (November 13, 2015)
ICLR Friday Forum: Climate data in Ontario (November 13, 2015)ICLR Friday Forum: Climate data in Ontario (November 13, 2015)
ICLR Friday Forum: Climate data in Ontario (November 13, 2015)
glennmcgillivray
 
Data recovery saudi arabia
Data recovery saudi arabiaData recovery saudi arabia
Data recovery saudi arabia
Dolphin Data Lab
 
ESRI User Conference 2014 - A Location Aware Mobile Tool for Direct and Indir...
ESRI User Conference 2014 - A Location Aware Mobile Tool for Direct and Indir...ESRI User Conference 2014 - A Location Aware Mobile Tool for Direct and Indir...
ESRI User Conference 2014 - A Location Aware Mobile Tool for Direct and Indir...
Francisco Ramos
 
Climate Data Sharing for Urban Resilience - OGC Testbed 11
Climate Data Sharing for Urban Resilience - OGC Testbed 11Climate Data Sharing for Urban Resilience - OGC Testbed 11
Climate Data Sharing for Urban Resilience - OGC Testbed 11
George Percivall
 
Saudi arabia presentation
Saudi arabia presentationSaudi arabia presentation
Saudi arabia presentation
bpdow12
 
Making Climate Data Sing
Making Climate Data SingMaking Climate Data Sing
Making Climate Data Sing
Scott St. George
 
Saudi Aramco Carbon Management - May 2013
Saudi Aramco Carbon Management - May 2013Saudi Aramco Carbon Management - May 2013
Saudi Aramco Carbon Management - May 2013
Global CCS Institute
 
The Role of DAta for Climate Monitoring and Prediction
The Role of DAta for Climate Monitoring and PredictionThe Role of DAta for Climate Monitoring and Prediction
The Role of DAta for Climate Monitoring and Prediction
NAP Events
 

En vedette (20)

Edwards climate data detectives - yale 2-2015
Edwards climate data detectives - yale 2-2015Edwards climate data detectives - yale 2-2015
Edwards climate data detectives - yale 2-2015
 
Climate data in R with the raster package
Climate data in R with the raster packageClimate data in R with the raster package
Climate data in R with the raster package
 
Contextualizing the Visualization of Climate Data
Contextualizing the Visualization of Climate DataContextualizing the Visualization of Climate Data
Contextualizing the Visualization of Climate Data
 
Document IT Communicate IT Succeed
Document IT Communicate IT SucceedDocument IT Communicate IT Succeed
Document IT Communicate IT Succeed
 
Interpreting Climate Data - Analysing climate vulnerability- online training ...
Interpreting Climate Data - Analysing climate vulnerability- online training ...Interpreting Climate Data - Analysing climate vulnerability- online training ...
Interpreting Climate Data - Analysing climate vulnerability- online training ...
 
Andreas Hense: Climate data for our future – acquired, analysed, archived
Andreas Hense: Climate data for our future – acquired, analysed, archivedAndreas Hense: Climate data for our future – acquired, analysed, archived
Andreas Hense: Climate data for our future – acquired, analysed, archived
 
ICLR Friday Forum: Climate data in Ontario (November 13, 2015)
ICLR Friday Forum: Climate data in Ontario (November 13, 2015)ICLR Friday Forum: Climate data in Ontario (November 13, 2015)
ICLR Friday Forum: Climate data in Ontario (November 13, 2015)
 
Data recovery saudi arabia
Data recovery saudi arabiaData recovery saudi arabia
Data recovery saudi arabia
 
ESRI User Conference 2014 - A Location Aware Mobile Tool for Direct and Indir...
ESRI User Conference 2014 - A Location Aware Mobile Tool for Direct and Indir...ESRI User Conference 2014 - A Location Aware Mobile Tool for Direct and Indir...
ESRI User Conference 2014 - A Location Aware Mobile Tool for Direct and Indir...
 
Climate Data Sharing for Urban Resilience - OGC Testbed 11
Climate Data Sharing for Urban Resilience - OGC Testbed 11Climate Data Sharing for Urban Resilience - OGC Testbed 11
Climate Data Sharing for Urban Resilience - OGC Testbed 11
 
Saudi arabia climate policy report
Saudi arabia climate policy reportSaudi arabia climate policy report
Saudi arabia climate policy report
 
Geography Water
Geography WaterGeography Water
Geography Water
 
Collaborate 2012: Environmental Accounting and Reporting
Collaborate 2012: Environmental Accounting and ReportingCollaborate 2012: Environmental Accounting and Reporting
Collaborate 2012: Environmental Accounting and Reporting
 
Saudi arabia presentation
Saudi arabia presentationSaudi arabia presentation
Saudi arabia presentation
 
Making Climate Data Sing
Making Climate Data SingMaking Climate Data Sing
Making Climate Data Sing
 
BigDataEurope - Big Data & Climate Change
BigDataEurope - Big Data & Climate ChangeBigDataEurope - Big Data & Climate Change
BigDataEurope - Big Data & Climate Change
 
Saudi Aramco Carbon Management - May 2013
Saudi Aramco Carbon Management - May 2013Saudi Aramco Carbon Management - May 2013
Saudi Aramco Carbon Management - May 2013
 
The Role of DAta for Climate Monitoring and Prediction
The Role of DAta for Climate Monitoring and PredictionThe Role of DAta for Climate Monitoring and Prediction
The Role of DAta for Climate Monitoring and Prediction
 
Non-renewable groundwater management in Saudi Arabia
Non-renewable groundwater management in Saudi ArabiaNon-renewable groundwater management in Saudi Arabia
Non-renewable groundwater management in Saudi Arabia
 
Gulf Region
Gulf RegionGulf Region
Gulf Region
 

Similaire à 20090701 Climate Data Staging

DataFinder concepts and example: General (20100503)
DataFinder concepts and example: General (20100503)DataFinder concepts and example: General (20100503)
DataFinder concepts and example: General (20100503)
Data Finder
 
seed block algorithm
seed block algorithmseed block algorithm
seed block algorithm
Dipak Badhe
 
Real Time Java DDS
Real Time Java DDSReal Time Java DDS
Real Time Java DDS
kerush
 
2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed...
2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed...2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed...
2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed...
Rudolf Husar
 
Linking HPC to Data Management - EUDAT Summer School (Giuseppe Fiameni, CINECA)
Linking HPC to Data Management - EUDAT Summer School (Giuseppe Fiameni, CINECA)Linking HPC to Data Management - EUDAT Summer School (Giuseppe Fiameni, CINECA)
Linking HPC to Data Management - EUDAT Summer School (Giuseppe Fiameni, CINECA)
EUDAT
 
DDS Advanced Tutorial - OMG June 2013 Berlin Meeting
DDS Advanced Tutorial - OMG June 2013 Berlin MeetingDDS Advanced Tutorial - OMG June 2013 Berlin Meeting
DDS Advanced Tutorial - OMG June 2013 Berlin Meeting
Jaime Martin Losa
 
Google Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 DayGoogle Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 Day
programmermag
 
GeoShareOverview_Mar11
GeoShareOverview_Mar11GeoShareOverview_Mar11
GeoShareOverview_Mar11
Werner Runge
 

Similaire à 20090701 Climate Data Staging (20)

PyModESt: A Python Framework for Staging of Geo-referenced Data on the Coll...
PyModESt: A Python Framework for Staging of Geo-referenced Data on the Coll...PyModESt: A Python Framework for Staging of Geo-referenced Data on the Coll...
PyModESt: A Python Framework for Staging of Geo-referenced Data on the Coll...
 
DataFinder concepts and example: General (20100503)
DataFinder concepts and example: General (20100503)DataFinder concepts and example: General (20100503)
DataFinder concepts and example: General (20100503)
 
Organizing the Data Chaos of Scientists
Organizing the Data Chaos of ScientistsOrganizing the Data Chaos of Scientists
Organizing the Data Chaos of Scientists
 
DataFinder: A Python Application for Scientific Data Management
DataFinder: A Python Application for Scientific Data ManagementDataFinder: A Python Application for Scientific Data Management
DataFinder: A Python Application for Scientific Data Management
 
Distributed Systems: How to connect your real-time applications
Distributed Systems: How to connect your real-time applicationsDistributed Systems: How to connect your real-time applications
Distributed Systems: How to connect your real-time applications
 
061206 Ua Huntsville Seminar
061206 Ua Huntsville Seminar061206 Ua Huntsville Seminar
061206 Ua Huntsville Seminar
 
Environment Canada's Data Management Service
Environment Canada's Data Management ServiceEnvironment Canada's Data Management Service
Environment Canada's Data Management Service
 
seed block algorithm
seed block algorithmseed block algorithm
seed block algorithm
 
Real Time Java DDS
Real Time Java DDSReal Time Java DDS
Real Time Java DDS
 
Grid1
Grid1Grid1
Grid1
 
Hpdw 2015-v10-paper
Hpdw 2015-v10-paperHpdw 2015-v10-paper
Hpdw 2015-v10-paper
 
060314 Ispra Htap Presentations Husar 060314 Ispra
060314 Ispra Htap Presentations Husar 060314 Ispra060314 Ispra Htap Presentations Husar 060314 Ispra
060314 Ispra Htap Presentations Husar 060314 Ispra
 
2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed...
2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed...2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed...
2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed...
 
Linking HPC to Data Management - EUDAT Summer School (Giuseppe Fiameni, CINECA)
Linking HPC to Data Management - EUDAT Summer School (Giuseppe Fiameni, CINECA)Linking HPC to Data Management - EUDAT Summer School (Giuseppe Fiameni, CINECA)
Linking HPC to Data Management - EUDAT Summer School (Giuseppe Fiameni, CINECA)
 
Enterprise guide to building a Data Mesh
Enterprise guide to building a Data MeshEnterprise guide to building a Data Mesh
Enterprise guide to building a Data Mesh
 
DDS Advanced Tutorial - OMG June 2013 Berlin Meeting
DDS Advanced Tutorial - OMG June 2013 Berlin MeetingDDS Advanced Tutorial - OMG June 2013 Berlin Meeting
DDS Advanced Tutorial - OMG June 2013 Berlin Meeting
 
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
 
Google Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 DayGoogle Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 Day
 
GeoShareOverview_Mar11
GeoShareOverview_Mar11GeoShareOverview_Mar11
GeoShareOverview_Mar11
 
My Other Computer is a Data Center: The Sector Perspective on Big Data
My Other Computer is a Data Center: The Sector Perspective on Big DataMy Other Computer is a Data Center: The Sector Perspective on Big Data
My Other Computer is a Data Center: The Sector Perspective on Big Data
 

Dernier

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMECloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Drew Madelung
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Roshan Dwivedi
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
UK Journal
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Dernier (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMECloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 

20090701 Climate Data Staging

  • 1. A Python Framework for Staging of Geo-referenced Data on the Collaborative Climate Community Grid (C3-Grid) Henning.Bergmeyer [at] dlr.de German Aerospace Center Simulation- and Software Technology (V.09-06-22a) Most up-to-date version of these slides available at: h ttp://tr.im/ep09_pymodest
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20. PyModESt: Metadata Update md. removeQuicklook () md. filterContentInfo (self.thesaurus.translateToC3(self.requested_vars, self.object_id)) md. setHorizontalBounds (self.lon_min, self.lon_max, self.lat_min, self.lat_max) md. setVerticalExtent (self.alt_min, self.alt_max, self.alt_verticalcrs) md. updateTimePeriod (self.timeperiod_begin_date, self.timeperiod_stop_date) md. setObjectId (self.object_id + ".“, datetime.utcnow().isoformat().replace(":","-")) md. addLineageProcessStep ( PROCESS_DESCRIPTION, datetime.datetime.utcnow(), self.stage_request.object_ids[0], RESPONSIBLE_PERSON, "http://wis.wmo.int/2006/catalogues/gmxCodelists.xml#CI_RoleCode_distributor", INSTITUTE_IDENTIFIER)
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.