SlideShare une entreprise Scribd logo
1  sur  34
Télécharger pour lire hors ligne
HDF Update
Mike Folk
National Center for Supercomputing Applications
HDF and HDF-EOS Workshop VII
September 24, 2003
-1-

HDF
Topics
•
•
•
•

Who is supporting HDF?
HDF software in 2003
Activities with EOS tools teams
Other Activities of Interest

-2-

HDF
Who is supporting HDF?
• NASA/ESDIS

– Earth science applications, instrument data
• DOE/ASCI (Accelerated Strategic Computing Init.)
– Simulations on massively parallel machines

• NCSA/NSF/State of Illinois

– HPC and Grid data intensive apps, Visualization, user support
– Atmospheric and ocean modeling environments

• DOE Scientific Data Analysis & Computation Program
– High performance I/O R & D

• National Archives and Records Administration
– Small grant to consider HDF5 as an archive format

-3-

HDF
HDF software in 2003
• Library releases
• Java Products
• Other tools

-4-

HDF
HDF4.2 Release 1
• Planned for October, 2003
• Alpha release available now from NCSA ftp
server
• Bug fixes
• Szip compression
– Fast compression method for EOS data

• Not included: Error detection code in HDF4
– Evaluated, decided not needed in HDF4
– Will address outside the library

-5-

HDF
HDF4.2r1
• New compilers

– Intel
– Portland Group

• New OS
–
–
–
–

Mac OS X
AIX 5.1 64-bit
OSF1
Red Hat 8/9

-6-

HDF
HDF4.2r1
• Tools (per DAAC and Instrument Team requests)
– hdfimport

• Converts float and/or integer data to SDS and/or 8-bit Raster
• Image data can be scaled about the mean value
• Revision of earlier fp2hdf

– hdfdiff

• Compares two HDF4 files
• Revision of earlier hdfdiff tool
• Requested by DAAC & instrument teams

– hdfrepack

• Makes a copy of an HDF4 file
• optionally rewrite objects with compression, uncompression,
and/or chunking

-7-

HDF
HDF5 software milestones in 2003
Q1 ‘03
Base
library
High level
library
Java
products

♦

1

.
.4

Q2 ‘03

Q3 ‘03

5
♦

PIs
LA
♦H
1.2
♦

1

.
.6

♦

Q4 ‘03

0
♦

1

.
.6

1

2.0
1.
1.3
♦

Other
-8-

5
-H rsion
H4 ve .1
♦ on y 1
c ar
libr

HDF
HDF5 1.4.5
• Released in February 2003
• New platforms
– AIX 5.1 (-64 bit)
– Mac OS X

• New compiler support on Linux 2.4
– Portland Group – pgcc. pgf90, pgCC
– gcc and g++ 3.2.

• Added some missing Fortran 90 APIs
• Fixed many bugs
• Some performance improvements
-9-

HDF
HDF5 1.6.0
• Released in July
• Most notable new features
– New filters
• szip compression
• “shuffling”
• checksum

– Properties
• Generic properties to allow users to extend property lists according to
their needs
• Control allocation time and fill value properties

– Compact storage layout for datasets
– Redesigned I/O pipeline for better performance.
– Hyperslab operations

- 10 -

HDF
HDF5 1.6.0
• New tools
– H5diff -- compare two HDF5 files
– h5import
• import ascii and binary data to an HDF5 file

– H5fc & h5c++
• more easily compile Fortran and C++ applications that use
HDF5

• Old tools
– h5toh4 conversion
• upgrade of h5toh4 utility
• updated the HDF4 to HDF5 Mapping specification

- 11 -

HDF
HDF5 High level APIs
• Make HDF5 easier to use
– More operations per call than the normal HDF5 API

• Encourage standard ways to store objects
– Enforce standard representation of objects in HDF5

- 15 -

HDF
HL HDF5: HDF5 Lite
• Higher-level functions that do more
operations per call than the basic API
• Wrap intuitive functions around certain sets
of features in the existing APIs
• Currently covers dataset and attribute
related functions

- 16 -

HDF
HL HDF5: HDF5 Image
• Defines a standard storage scheme for
datasets that are intended to be interpreted
as images
• 2 types of images
– 8-bit indexed to a palette
– 24-bit with 3 color planes (RGB)

• Also palette functions

- 17 -

HDF
HL HDF5: HDF5 Table
• Defines a standard storage scheme for datasets that
are intended to be interpreted as tables
• A “table” is a collection of records with fixedlength fields:

- 18 -

HDF
Parallel HDF5
• A few performance improvements
• MPICH/MPE instrumentation feature added

– performance analysis tools for their MPI programs

• “Flexible parallel HDF5” programming model
– More flexible model for parallel HDF5

• New parallel platforms supported
–
–
–
–
–

Solaris 2.8 (32 & 64 bits)
OSF 5.1
Cray T3E, SV1, T90
HPUX 11.0
FreeBSD
- 19 -

HDF
HDF5 1.6.1
• Bug fixes needed by Aura team
• Due Oct. 15
• Thanks to Cheryl Craig and the Aura team
for finding the bugs and working with us

- 20 -

HDF
HDFView
HDFView – a Java based
visual tool to browse and
edit HDF4 and HDF5 files.
•
•
•
•
•
•
•

Browse objects in hierarchy
Import/export JPEG images
Create and delete objects
Copy/paste between files
Change/delete data content
Display/modify attributes
Save data values to a text file

- 21 -

HDF
Modular HDFView
Modular HDFView – improved
HDFView where I/O and GUI
components are replaceable modules.
• Replaceable modules:
– File I/O (file/data format)
– Tree view (show file structure)
– Table view (spreadsheet-like)
– Text view (view/edit text dataset)
– Image view (view/process image)
– Palette view (view/change palette)
– Metadata (attribute) view
- 22 -

Application
(HDFView)
Interfaces
I/O, TreeView,
TableView, etc

Default

User

Implementation

Implementation

HDF
Other tools work
• H5diff
– Compare the structure and contents of two HDF5 files,
and report differences
– Command line utility like Unix ‘diff’ and older ‘hdiff’
– Report missing objects, inconsistent size, datatype, etc.
– Compare values of numeric datasets
– First beta available January 2003
– See poster

- 23 -

HDF
Other activities with EOS tools teams
• Collaboration with ECS contractor to add
HDF-EOS modules to HDF java tool
• Consultation & collaboration with the Data
Usability team
– XML and other tools
– Poster at AGU

- 24 -

HDF
Other Activities of Interest

- 25 -

HDF
DOE/ASCI*
“ASCI provides the integrating simulation and
modeling capabilities and technologies needed …for
future design assessment and certification of nuclear
weapons and their components”

•
•
•
•

Massively parallel computing and I/O
Complex data models and big data
HDF5 a standard format for ASCI apps
NCSA role
– Library development and maintenance
– Data modeling
– Porting and tuning on big machines

* “Advanced Simulation and Computing Program”

- 26 -

HDF
National Archives and Records Administration
• Pilot project with HDF5
• Explore scientific data format requirements for
long term archiving of electronic records
• Geospatial data archiving and access
– 2-d and 3-d raster data, vector data
– Converting common formats to HDF5 and HDF-EOS
– Exploring scalability, applicability

• See poster:
“HDF5, HDF-EOS and Geospatial Data Archives”
- 27 -

HDF
Extendable Terascale Facility (ETF)
• NSF-sponsored computing and data grid
– Charter members:
NCSA, SDSC, Caltech, Argonne National Lab,
Pittsburgh Supercomputing Center
– Others to join later

•
•
•
•

Terascale computing and data
HDF4 and HDF5 apps common among early users
Parallel HDF5 on Linux clusters, others
Challenging I/O requirements
- 28 -

HDF
NPOESS
• National Polar-orbiting Operational Environmental
Satellite System
– Combine satellite systems of civil and defense programs

• HDF5 to be used to distribute data to users
• See presentations/posters this afternoon

- 29 -

HDF
netCDF-HDF Project
• Enhanced NetCDF-4 Interface to HDF5 Data
• Combine desirable characteristics of netCDF
and HDF5, while taking advantage of their
separate strengths
• Preserve format and API compatibility for
netCDF users
• Demonstrate benefits of this combination in
advanced Earth science modeling efforts
- 30 -

HDF
Atmospheric and Ocean Models
•
•

“Modeling Environment for Atmospheric Discovery”
HDF5 for high performance I/O for atmospheric and
ocean modeling
–
–
–
–

•
•

Weather Research and Forecasting (WRF) model
Regional Ocean Modeling System (ROMS)
Coupling of WRF and ROMS
Potential ETF application

UAH ESML & data mining also involved
See poster:
An HDF5 WRF/IO Module: Lessons Learned
- 31 -

HDF
DOE SciDAC* Program
• “Programming Models for Scalable Parallel Computing”
• High performance I/O R&D
–
–
–
–

Effectiveness of compression on I/O performance
Transformation of data during I/O
Integration of HDF5 with high performance Fortran
Improving parallel I/O performance in HDF5

* Scientific Discovery through Advanced Computing”

- 32 -

HDF
HDF5 Mesh API prototype
• Support for structured and unstructured “mesh” data
• For applications such as computational fluid
dynamics, finite element analysis, and visualization.
• A higher-level API
• Format
– HDF5 groups and datasets to organize the data

• Collaboration involving NCSA, CEI and others
• Documentation still pretty sketchy, but see
• ftp://ftp.ensight.com/pub/HDF_RW/hdf_rw.tgz

• Discussion list
- 33 -

HDF
Information Sources
• HDF website
– http://hdf.ncsa.uiuc.edu/

• HDF5 Information Center
– http://hdf.ncsa.uiuc.edu/HDF5/

• HDF Helpdesk
– hdfhelp@ncsa.uiuc.edu

• HDF users mailing list
– hdfnews@ncsa.uiuc.edu
- 34 -

HDF
Thank you

- 35 -

HDF
Acknowledgements
This report is based upon work supported in part by a Cooperative
Agreement with NASA under NASA grant NAG 5-2040 and NAG
NCCS-599. Any opinions, findings, and conclusions or
yada yada yada …expressed in this material are those of the
recommendations
author(s) and do not necessarily reflect the views of the
National Aeronautics and Space Administration. Other
support provided by NCSA and other sponsors and agencies.
(http://hdf.ncsa.uiuc.edu/acknowledge.html).

- 36 -

HDF
Questions/comments?

- 37 -

HDF

Contenu connexe

Tendances

Hdf5 parallel
Hdf5 parallelHdf5 parallel
Hdf5 parallelmfolk
 

Tendances (20)

HDF5 and Ecosystem: What Is New?
HDF5 and Ecosystem: What Is New?HDF5 and Ecosystem: What Is New?
HDF5 and Ecosystem: What Is New?
 
MODIS Land and HDF-EOS
MODIS Land and HDF-EOSMODIS Land and HDF-EOS
MODIS Land and HDF-EOS
 
GDAL Enhancement for ESDIS Project
GDAL Enhancement for ESDIS ProjectGDAL Enhancement for ESDIS Project
GDAL Enhancement for ESDIS Project
 
HDF Update for DAAC Managers (2017-02-27)
HDF Update for DAAC Managers (2017-02-27)HDF Update for DAAC Managers (2017-02-27)
HDF Update for DAAC Managers (2017-02-27)
 
HDF Status and Development
HDF Status and DevelopmentHDF Status and Development
HDF Status and Development
 
HDF Product Designer
HDF Product DesignerHDF Product Designer
HDF Product Designer
 
HDF & HDF-EOS Data & Support at NSIDC
HDF & HDF-EOS Data & Support at NSIDCHDF & HDF-EOS Data & Support at NSIDC
HDF & HDF-EOS Data & Support at NSIDC
 
HDF5 High Level and Lite Libraries
HDF5 High Level and Lite LibrariesHDF5 High Level and Lite Libraries
HDF5 High Level and Lite Libraries
 
Hierarchical Data Formats (HDF) Update
Hierarchical Data Formats (HDF) UpdateHierarchical Data Formats (HDF) Update
Hierarchical Data Formats (HDF) Update
 
HDF5 In Support of Database Applications
HDF5 In Support of Database ApplicationsHDF5 In Support of Database Applications
HDF5 In Support of Database Applications
 
Hdf5 parallel
Hdf5 parallelHdf5 parallel
Hdf5 parallel
 
NetCDF and HDF5
NetCDF and HDF5NetCDF and HDF5
NetCDF and HDF5
 
Parallel HDF5 Developments
Parallel HDF5 DevelopmentsParallel HDF5 Developments
Parallel HDF5 Developments
 
H5Coro: The Cloud-Optimized Read-Only Library
H5Coro: The Cloud-Optimized Read-Only LibraryH5Coro: The Cloud-Optimized Read-Only Library
H5Coro: The Cloud-Optimized Read-Only Library
 
The HDF Group: Community models and outreach
The HDF Group: Community models and outreachThe HDF Group: Community models and outreach
The HDF Group: Community models and outreach
 
The New HDF-EOS WebSite - How it can help you
The New HDF-EOS WebSite - How it can help youThe New HDF-EOS WebSite - How it can help you
The New HDF-EOS WebSite - How it can help you
 
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
 
HDFCloud Workshop: HDF5 in the Cloud
HDFCloud Workshop: HDF5 in the CloudHDFCloud Workshop: HDF5 in the Cloud
HDFCloud Workshop: HDF5 in the Cloud
 
Status of HDF-EOS, Related Software and Tools
 Status of HDF-EOS, Related Software and Tools Status of HDF-EOS, Related Software and Tools
Status of HDF-EOS, Related Software and Tools
 
HDF Update 2016
HDF Update 2016HDF Update 2016
HDF Update 2016
 

Similaire à HDF Update Highlights Support, Software Releases and Activities

Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...The HDF-EOS Tools and Information Center
 
Hdf5 current future
Hdf5 current futureHdf5 current future
Hdf5 current futuremfolk
 

Similaire à HDF Update Highlights Support, Software Releases and Activities (20)

HDF Updae
HDF UpdaeHDF Updae
HDF Updae
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
HDF5 and The HDF Group
HDF5 and The HDF GroupHDF5 and The HDF Group
HDF5 and The HDF Group
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
H5spark
H5spark H5spark
H5spark
 
HDF-EOS Workshop II Introduction
HDF-EOS Workshop II IntroductionHDF-EOS Workshop II Introduction
HDF-EOS Workshop II Introduction
 
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
HDF Project Update
HDF Project UpdateHDF Project Update
HDF Project Update
 
Hdf5 current future
Hdf5 current futureHdf5 current future
Hdf5 current future
 
HDF-EOS Software Developer/Vendor Workshop Wrapup
HDF-EOS Software Developer/Vendor Workshop WrapupHDF-EOS Software Developer/Vendor Workshop Wrapup
HDF-EOS Software Developer/Vendor Workshop Wrapup
 
ICESat-2 H5-ES Product Development Strategy
ICESat-2 H5-ES Product Development StrategyICESat-2 H5-ES Product Development Strategy
ICESat-2 H5-ES Product Development Strategy
 
Introduction to HDF5 Data and Programming Models
Introduction to HDF5 Data and Programming ModelsIntroduction to HDF5 Data and Programming Models
Introduction to HDF5 Data and Programming Models
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
ESDIS Status (2002)
ESDIS Status (2002)ESDIS Status (2002)
ESDIS Status (2002)
 
HDF4 Mapping Project Update
HDF4 Mapping Project UpdateHDF4 Mapping Project Update
HDF4 Mapping Project Update
 
Support for NPP/NPOESS/JPSS by The HDF Group
 Support for NPP/NPOESS/JPSS by The HDF Group Support for NPP/NPOESS/JPSS by The HDF Group
Support for NPP/NPOESS/JPSS by The HDF Group
 
HDF Cloud: HDF5 at Scale
HDF Cloud: HDF5 at ScaleHDF Cloud: HDF5 at Scale
HDF Cloud: HDF5 at Scale
 
HDF Software Process - Lessons Learned & Success Factors
HDF Software Process - Lessons Learned & Success FactorsHDF Software Process - Lessons Learned & Success Factors
HDF Software Process - Lessons Learned & Success Factors
 
HDF-EOS APIs, tools, etc.
HDF-EOS APIs, tools, etc.HDF-EOS APIs, tools, etc.
HDF-EOS APIs, tools, etc.
 

Plus de The HDF-EOS Tools and Information Center

STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
 

Plus de The HDF-EOS Tools and Information Center (20)

Cloud-Optimized HDF5 Files
Cloud-Optimized HDF5 FilesCloud-Optimized HDF5 Files
Cloud-Optimized HDF5 Files
 
Accessing HDF5 data in the cloud with HSDS
Accessing HDF5 data in the cloud with HSDSAccessing HDF5 data in the cloud with HSDS
Accessing HDF5 data in the cloud with HSDS
 
The State of HDF
The State of HDFThe State of HDF
The State of HDF
 
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance FeaturesHighly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
 
Creating Cloud-Optimized HDF5 Files
Creating Cloud-Optimized HDF5 FilesCreating Cloud-Optimized HDF5 Files
Creating Cloud-Optimized HDF5 Files
 
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance DiscussionHDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
 
Hyrax: Serving Data from S3
Hyrax: Serving Data from S3Hyrax: Serving Data from S3
Hyrax: Serving Data from S3
 
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLABAccessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
 
HDF - Current status and Future Directions
HDF - Current status and Future DirectionsHDF - Current status and Future Directions
HDF - Current status and Future Directions
 
HDF - Current status and Future Directions
HDF - Current status and Future Directions HDF - Current status and Future Directions
HDF - Current status and Future Directions
 
HDF for the Cloud - Serverless HDF
HDF for the Cloud - Serverless HDFHDF for the Cloud - Serverless HDF
HDF for the Cloud - Serverless HDF
 
HDF5 <-> Zarr
HDF5 <-> ZarrHDF5 <-> Zarr
HDF5 <-> Zarr
 
HDF for the Cloud - New HDF Server Features
HDF for the Cloud - New HDF Server FeaturesHDF for the Cloud - New HDF Server Features
HDF for the Cloud - New HDF Server Features
 
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
 
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
 
HDF5 Roadmap 2019-2020
HDF5 Roadmap 2019-2020HDF5 Roadmap 2019-2020
HDF5 Roadmap 2019-2020
 
Leveraging the Cloud for HDF Software Testing
Leveraging the Cloud for HDF Software TestingLeveraging the Cloud for HDF Software Testing
Leveraging the Cloud for HDF Software Testing
 
Google Colaboratory for HDF-EOS
Google Colaboratory for HDF-EOSGoogle Colaboratory for HDF-EOS
Google Colaboratory for HDF-EOS
 
Parallel Computing with HDF Server
Parallel Computing with HDF ServerParallel Computing with HDF Server
Parallel Computing with HDF Server
 
HDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's GuideHDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's Guide
 

Dernier

Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?Rustici Software
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfHCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfROWELL MARQUINA
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Arti Languages Pre Seed Pitchdeck 2024.pdf
Arti Languages Pre Seed Pitchdeck 2024.pdfArti Languages Pre Seed Pitchdeck 2024.pdf
Arti Languages Pre Seed Pitchdeck 2024.pdfwill854175
 
Transport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MITransport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MIRomil Mishra
 
Software Security in the Real World w/Kelsey Hightower
Software Security in the Real World w/Kelsey HightowerSoftware Security in the Real World w/Kelsey Hightower
Software Security in the Real World w/Kelsey HightowerAnchore
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 

Dernier (20)

Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfHCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Arti Languages Pre Seed Pitchdeck 2024.pdf
Arti Languages Pre Seed Pitchdeck 2024.pdfArti Languages Pre Seed Pitchdeck 2024.pdf
Arti Languages Pre Seed Pitchdeck 2024.pdf
 
Transport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MITransport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MI
 
Software Security in the Real World w/Kelsey Hightower
Software Security in the Real World w/Kelsey HightowerSoftware Security in the Real World w/Kelsey Hightower
Software Security in the Real World w/Kelsey Hightower
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 

HDF Update Highlights Support, Software Releases and Activities

  • 1. HDF Update Mike Folk National Center for Supercomputing Applications HDF and HDF-EOS Workshop VII September 24, 2003 -1- HDF
  • 2. Topics • • • • Who is supporting HDF? HDF software in 2003 Activities with EOS tools teams Other Activities of Interest -2- HDF
  • 3. Who is supporting HDF? • NASA/ESDIS – Earth science applications, instrument data • DOE/ASCI (Accelerated Strategic Computing Init.) – Simulations on massively parallel machines • NCSA/NSF/State of Illinois – HPC and Grid data intensive apps, Visualization, user support – Atmospheric and ocean modeling environments • DOE Scientific Data Analysis & Computation Program – High performance I/O R & D • National Archives and Records Administration – Small grant to consider HDF5 as an archive format -3- HDF
  • 4. HDF software in 2003 • Library releases • Java Products • Other tools -4- HDF
  • 5. HDF4.2 Release 1 • Planned for October, 2003 • Alpha release available now from NCSA ftp server • Bug fixes • Szip compression – Fast compression method for EOS data • Not included: Error detection code in HDF4 – Evaluated, decided not needed in HDF4 – Will address outside the library -5- HDF
  • 6. HDF4.2r1 • New compilers – Intel – Portland Group • New OS – – – – Mac OS X AIX 5.1 64-bit OSF1 Red Hat 8/9 -6- HDF
  • 7. HDF4.2r1 • Tools (per DAAC and Instrument Team requests) – hdfimport • Converts float and/or integer data to SDS and/or 8-bit Raster • Image data can be scaled about the mean value • Revision of earlier fp2hdf – hdfdiff • Compares two HDF4 files • Revision of earlier hdfdiff tool • Requested by DAAC & instrument teams – hdfrepack • Makes a copy of an HDF4 file • optionally rewrite objects with compression, uncompression, and/or chunking -7- HDF
  • 8. HDF5 software milestones in 2003 Q1 ‘03 Base library High level library Java products ♦ 1 . .4 Q2 ‘03 Q3 ‘03 5 ♦ PIs LA ♦H 1.2 ♦ 1 . .6 ♦ Q4 ‘03 0 ♦ 1 . .6 1 2.0 1. 1.3 ♦ Other -8- 5 -H rsion H4 ve .1 ♦ on y 1 c ar libr HDF
  • 9. HDF5 1.4.5 • Released in February 2003 • New platforms – AIX 5.1 (-64 bit) – Mac OS X • New compiler support on Linux 2.4 – Portland Group – pgcc. pgf90, pgCC – gcc and g++ 3.2. • Added some missing Fortran 90 APIs • Fixed many bugs • Some performance improvements -9- HDF
  • 10. HDF5 1.6.0 • Released in July • Most notable new features – New filters • szip compression • “shuffling” • checksum – Properties • Generic properties to allow users to extend property lists according to their needs • Control allocation time and fill value properties – Compact storage layout for datasets – Redesigned I/O pipeline for better performance. – Hyperslab operations - 10 - HDF
  • 11. HDF5 1.6.0 • New tools – H5diff -- compare two HDF5 files – h5import • import ascii and binary data to an HDF5 file – H5fc & h5c++ • more easily compile Fortran and C++ applications that use HDF5 • Old tools – h5toh4 conversion • upgrade of h5toh4 utility • updated the HDF4 to HDF5 Mapping specification - 11 - HDF
  • 12. HDF5 High level APIs • Make HDF5 easier to use – More operations per call than the normal HDF5 API • Encourage standard ways to store objects – Enforce standard representation of objects in HDF5 - 15 - HDF
  • 13. HL HDF5: HDF5 Lite • Higher-level functions that do more operations per call than the basic API • Wrap intuitive functions around certain sets of features in the existing APIs • Currently covers dataset and attribute related functions - 16 - HDF
  • 14. HL HDF5: HDF5 Image • Defines a standard storage scheme for datasets that are intended to be interpreted as images • 2 types of images – 8-bit indexed to a palette – 24-bit with 3 color planes (RGB) • Also palette functions - 17 - HDF
  • 15. HL HDF5: HDF5 Table • Defines a standard storage scheme for datasets that are intended to be interpreted as tables • A “table” is a collection of records with fixedlength fields: - 18 - HDF
  • 16. Parallel HDF5 • A few performance improvements • MPICH/MPE instrumentation feature added – performance analysis tools for their MPI programs • “Flexible parallel HDF5” programming model – More flexible model for parallel HDF5 • New parallel platforms supported – – – – – Solaris 2.8 (32 & 64 bits) OSF 5.1 Cray T3E, SV1, T90 HPUX 11.0 FreeBSD - 19 - HDF
  • 17. HDF5 1.6.1 • Bug fixes needed by Aura team • Due Oct. 15 • Thanks to Cheryl Craig and the Aura team for finding the bugs and working with us - 20 - HDF
  • 18. HDFView HDFView – a Java based visual tool to browse and edit HDF4 and HDF5 files. • • • • • • • Browse objects in hierarchy Import/export JPEG images Create and delete objects Copy/paste between files Change/delete data content Display/modify attributes Save data values to a text file - 21 - HDF
  • 19. Modular HDFView Modular HDFView – improved HDFView where I/O and GUI components are replaceable modules. • Replaceable modules: – File I/O (file/data format) – Tree view (show file structure) – Table view (spreadsheet-like) – Text view (view/edit text dataset) – Image view (view/process image) – Palette view (view/change palette) – Metadata (attribute) view - 22 - Application (HDFView) Interfaces I/O, TreeView, TableView, etc Default User Implementation Implementation HDF
  • 20. Other tools work • H5diff – Compare the structure and contents of two HDF5 files, and report differences – Command line utility like Unix ‘diff’ and older ‘hdiff’ – Report missing objects, inconsistent size, datatype, etc. – Compare values of numeric datasets – First beta available January 2003 – See poster - 23 - HDF
  • 21. Other activities with EOS tools teams • Collaboration with ECS contractor to add HDF-EOS modules to HDF java tool • Consultation & collaboration with the Data Usability team – XML and other tools – Poster at AGU - 24 - HDF
  • 22. Other Activities of Interest - 25 - HDF
  • 23. DOE/ASCI* “ASCI provides the integrating simulation and modeling capabilities and technologies needed …for future design assessment and certification of nuclear weapons and their components” • • • • Massively parallel computing and I/O Complex data models and big data HDF5 a standard format for ASCI apps NCSA role – Library development and maintenance – Data modeling – Porting and tuning on big machines * “Advanced Simulation and Computing Program” - 26 - HDF
  • 24. National Archives and Records Administration • Pilot project with HDF5 • Explore scientific data format requirements for long term archiving of electronic records • Geospatial data archiving and access – 2-d and 3-d raster data, vector data – Converting common formats to HDF5 and HDF-EOS – Exploring scalability, applicability • See poster: “HDF5, HDF-EOS and Geospatial Data Archives” - 27 - HDF
  • 25. Extendable Terascale Facility (ETF) • NSF-sponsored computing and data grid – Charter members: NCSA, SDSC, Caltech, Argonne National Lab, Pittsburgh Supercomputing Center – Others to join later • • • • Terascale computing and data HDF4 and HDF5 apps common among early users Parallel HDF5 on Linux clusters, others Challenging I/O requirements - 28 - HDF
  • 26. NPOESS • National Polar-orbiting Operational Environmental Satellite System – Combine satellite systems of civil and defense programs • HDF5 to be used to distribute data to users • See presentations/posters this afternoon - 29 - HDF
  • 27. netCDF-HDF Project • Enhanced NetCDF-4 Interface to HDF5 Data • Combine desirable characteristics of netCDF and HDF5, while taking advantage of their separate strengths • Preserve format and API compatibility for netCDF users • Demonstrate benefits of this combination in advanced Earth science modeling efforts - 30 - HDF
  • 28. Atmospheric and Ocean Models • • “Modeling Environment for Atmospheric Discovery” HDF5 for high performance I/O for atmospheric and ocean modeling – – – – • • Weather Research and Forecasting (WRF) model Regional Ocean Modeling System (ROMS) Coupling of WRF and ROMS Potential ETF application UAH ESML & data mining also involved See poster: An HDF5 WRF/IO Module: Lessons Learned - 31 - HDF
  • 29. DOE SciDAC* Program • “Programming Models for Scalable Parallel Computing” • High performance I/O R&D – – – – Effectiveness of compression on I/O performance Transformation of data during I/O Integration of HDF5 with high performance Fortran Improving parallel I/O performance in HDF5 * Scientific Discovery through Advanced Computing” - 32 - HDF
  • 30. HDF5 Mesh API prototype • Support for structured and unstructured “mesh” data • For applications such as computational fluid dynamics, finite element analysis, and visualization. • A higher-level API • Format – HDF5 groups and datasets to organize the data • Collaboration involving NCSA, CEI and others • Documentation still pretty sketchy, but see • ftp://ftp.ensight.com/pub/HDF_RW/hdf_rw.tgz • Discussion list - 33 - HDF
  • 31. Information Sources • HDF website – http://hdf.ncsa.uiuc.edu/ • HDF5 Information Center – http://hdf.ncsa.uiuc.edu/HDF5/ • HDF Helpdesk – hdfhelp@ncsa.uiuc.edu • HDF users mailing list – hdfnews@ncsa.uiuc.edu - 34 - HDF
  • 32. Thank you - 35 - HDF
  • 33. Acknowledgements This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA grant NAG 5-2040 and NAG NCCS-599. Any opinions, findings, and conclusions or yada yada yada …expressed in this material are those of the recommendations author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration. Other support provided by NCSA and other sponsors and agencies. (http://hdf.ncsa.uiuc.edu/acknowledge.html). - 36 - HDF

Notes de l'éditeur

  1. &lt;number&gt;
  2. Use this as backup
  3. Use this as backup
  4. Use this as backup
  5. This is Ray Milburn and Larry Klein Discussions in Feb, June, Exchanged email in July, Aug. Sent early release of our code 8.29/03 Ray and Larry are said to have made a proposal to Raytheon to let Ray work with us on this. We are waiting to hear.