The presentation will provide an overview of subsetting software development activity at UAH. Updates have been made to all packages, reflecting the latest versions of HDF5 and HE5. The library of tools (HSE) for subsetting HDF-EOS data is up-to-date for SGI, Sun, and Linux platforms. Subsetting software is operational at NSIDC DAAC and GDAAC, in testing at LPDAAC. Ongoing work and plans will also be described, including row/column subsetting and index subsampling.

1. Subsetting at UAH
Bruce Beaumont, Matt Smith,
Helen Conover, Sara Graves
HDF & HDF-EOS Workshop VIII
2004 October 26-28 Aurora, CO

2. Why Subset?
• Goal: to provide a science data user with only the data
they need as quickly as possible.
• Benefits science data users and data centers:
- Reduces analysis time by reducing amount of data
- Reduces time for data delivery
- Reduces resources (network, personnel, media, etc.)
• Steps:
- Locate spatial / temporal / spectral area of interest
- Extract
- Re-assemble for distribution/use

3. What is HSE?
HEW Subsetting Engine
A new packaging option for the HDFEOS Subsetter designed for users that
want subsetting, but do not want to host
the full HEW installation or even the
standalone HEW back-end.

4. What is HEW?
• HDF-EOS Web-based Subsetter
– Prototype software designed to be datasetindependent (HDF-EOS)
– Funded by NASA/ESDIS for EOS-DIS
– To be used within ECS (EOS-DIS Core System)
– Original Front-end/GUI (optional)
• Uses HTML forms and JavaScript
– Original Back-end
• Needed subset criteria and HDF-EOS data
• Performed subsetting as a “batch” job

5. What are HSE’s capabilities?
• Versions available for HDF-EOS 2 and HDF-EOS 5.
• Subsets multiple files in one call.
• Subsets properly-formatted HDF-EOS grid and swath objects.
• Subsets spatially by latitude/longitude or row/column.
• Subsets swaths temporally by date/time range.
• Subsets swaths by full or partial (subscan) lines.
• Subsets spectrally by HDF-EOS field.
• Subsamples along any dimension by stride (repeating interval)
or by discrete index list. (see chart)
• Copies all file and field attributes to the output file and updates
HDF-EOS “core” metadata when possible.

6. Stride Subsampling on a Dimension
•STRIDE = 2
0
1
2
3
4
5
6
7
8
9
7
8
9
Selected
Indexed Subsampling on a Dimension
•INDEXES = (1, 3, 4, 5, 7)
0
1
2
3
4
5
Selected
6

7. Existing HEW Back-End
Architecture
Subset t ing
crit eria
( ODL)
St at us file
End
-user
E-mail
HEW_ DISS
Input
file( s)
Out put
file( s)
Log file
• Front end supplies
subsetting criteria file
• Subsetter writes messages
to status file
• Subsetter writes messages
to log file
• Subsetter sends e-mail to
end-user
Many sites do not want to create ODL files or the hassle of dealing
with message or log files or e-mail.

8. New HSE
Architecture
All subsetting functionality is contained
within a callable function
•
User Applicat ion Code
Subset t ing crit eria Ret urn code
•
Input
file( s)
HEW Subset t ing Engine
funct ion
HSE_ LogMsg
callback
HSE_ St at usMsg
callback
Out put
file( s)
•
User application code
builds subsetting criteria
structure
Subsetting engine
function calls user’s
functions for status and
log messages
No e-mail is sent

9. SPOT
• Subsettability “checker”
– Displays content/structure
of HDF-EOS4 (or HE5) files
– Examines files for subsettability by HSE
– Simple command-line interface
– Stand-alone operation
– Available at subset.org

10. HEW integration with ECS
EDG System
EDG
Order
submission
(HTML)
End
user
ECS
2
ECS
1
7
3
Output data
(Reingested)
4
Data order
and reply
Subset ODL
and reply
Output
data
6
Subsetter
Subsetting System
5
Input
data

11. Product Availability Matrix
HDF-EOS 2
(HDF4)
HDF-EOS 5
(HDF5)
SGI • Sun • Linux
N/A
N/A
SGI • Sun • Linux
HEWBE
SGI • Sun • Linux
Planned
HEW
SGI • Sun • Linux
N/A
Product
HSE
HSE-5
SPOT
SGI • Sun • Linux

12. Currently Available/Planned
Subsetting Applications
•
HDF-EOS Subsetting Engine (HDF-EOS, HE5)Status Deployments
–
–
–
–
–
–
•
Complete System
Subsetting Engine Only
SPOT - Subsettability Checker
HSE Integrated with ECS Data Order System
HSE Integrated with AMSR-E Processing
Subsetting as a Web Service
GHRC
GSFC
many
NSIDC, EDC
AMSR-E SIPS
(ECHO)
available
available
Science teams
GHRC
in work
(various)
Customized Subsetting
– MODIS tools
– Coarse-grain SSM/I Subsetter
•
available
available
available
available
available
planned
General Purpose Customizable Subsetting
– Subsetting Tool using ESML

13. http://subset.org