hdf5 opendap project update and demo
DESCRIPTION
HDF5 OPeNDAP Project Update and Demo. MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.). OPeNDAP Review. A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/1.jpg)
HDF5 OPeNDAP ProjectUpdate and Demo
MuQun Yang and Hyo-Kyung Lee (The HDF Group)
James Gallagher (OPeNDAP, Inc.)
1HDF and HDF-EOS Workshop XII10/17/2008
![Page 2: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/2.jpg)
OPeNDAP Review
• A software framework that allows simple access to remote data
• Data Access Protocol• Client-server model• Widely used by Earth Science Community
2HDF and HDF-EOS Workshop XII10/17/2008
![Page 3: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/3.jpg)
Client Library(libnc-dap)
DAP Protocol (via http)
Server(apache)
Handler(hdf4_handler)
Remote Data(HDF4)
View Data (User)
OPeNDAP Concept
3HDF and HDF-EOS Workshop XII10/17/2008
Visualization Tools
(gradsdap)
Typical Remote Access
FTP/HTTP
Visualization tools
(e.g. grads)
![Page 4: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/4.jpg)
DAP Protocol (via http)
Server(apache)
Remote HDF5View Data (User)
OPeNDAP HDF5 Handler
4HDF and HDF-EOS Workshop XII10/17/2008
HDF5 Handler
![Page 5: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/5.jpg)
Mapping HDF5 Datatypes to DAP Datatypes
• Compound Datatype
• Groups
• Object/Regional References
5HDF and HDF-EOS Workshop XII10/17/2008
Challenges
![Page 6: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/6.jpg)
DAP Protocol (via http)
Server(apache)
View Data (User)
OPeNDAP HDF5 Handler with HDF-EOS5
6HDF and HDF-EOS Workshop XII10/17/2008
HDF5 Handler
Remote HDF5
Client Library(libnc-dap)
Visualization tools
(grads)
Remote HDF-EOS5
![Page 7: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/7.jpg)
Challenges: HDF- EOS5
• Grid with No Geolocation data Clients expect Grid with Geolocation data
• Some Attributes stored as Extremely Long String(s) (e.g. StructMetada.0) Clients expect structured attributes
7HDF and HDF-EOS Workshop XII10/17/2008
![Page 8: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/8.jpg)
Tweaks for HDF-EOS5
Added two default HDF5 handler configuration options:• --enable-eos-grid • --enable-eos-meta
8HDF and HDF-EOS Workshop XII10/17/2008
![Page 9: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/9.jpg)
Challenges: OPeNDAP Clients
• Need special attributes on dataset.• Need shared geolocation variables outside Grid.
9HDF and HDF-EOS Workshop XII10/17/2008
![Page 10: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/10.jpg)
Tweaks for OPeNDAP Clients
Added two optional handler configuration options:• --enable-short-name• --enable-CF
10HDF and HDF-EOS Workshop XII10/17/2008
![Page 11: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/11.jpg)
HDF5 Handler
DAP Protocol (via http)
Server(apache)
View Data (User)
OPeNDAP HDF5 Handler with HDF-EOS5
11HDF and HDF-EOS Workshop XII10/17/2008
HDF5 Handlerw/ tweaks
Remote HDF5
Client Library(libnc-dap)
Visualization tools
(grads)
Remote HDF-EOS5
![Page 12: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/12.jpg)
Day After Server Tweaks
• Finally, Happy Clients!
GrADSNCLFerretMATLAB
ncBrowse
IDVODC
IDL
12HDF and HDF-EOS Workshop XII10/17/2008
![Page 13: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/13.jpg)
Problems of Tweaking HDF5 Handler
13HDF and HDF-EOS Workshop XII10/17/2008
Remember that we added two optional HDF5 handler configuration options:1. --enable-short-name2. --enable-CF
Causes ambiguity among variable names
(e.g. /GroupA/ozone vs. /GroupB/ozone)
Drops some key attributes (e.g. StructMetdata, HDF_ROOT_GROUP)
![Page 14: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/14.jpg)
HDF5-Friendly OPeNDAP Client Library
14HDF and HDF-EOS Workshop XII10/17/2008
DAP Protocol (via http)
Server(apache)
View Data (User)
HDF5 Handler
Client Library(libnc-dap)
Visualization tools
(grads)
Remote HDF5
HDF5 Handlerw/ tweaks
HDF5-FriendlyClient Library(liboc-dap)
HDF5 GroupsView Groups???
![Page 15: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/15.jpg)
Example: Groups in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
15HDF and HDF-EOS Workshop XII10/17/2008
![Page 16: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/16.jpg)
Example: Reference in HDF5
• Important for NPOESS• Object / Regional Reference • Map to DAP URL
16HDF and HDF-EOS Workshop XII10/17/2008
![Page 17: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/17.jpg)
Example: dap2h5
• A test application for the client library• It can construct HDF5 from DAP output
17HDF and HDF-EOS Workshop XII10/17/2008
DAP Protocol (via http)
Server(apache)
View Data (User)
HDF5-F. Client Library App.
(dap2h5)
Remote HDF5
HDF5 Handler
HDF5-FriendlyClient Library
(liboc-dap)
Group/Ref.View Group/Ref.
![Page 18: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/18.jpg)
• One more reason: Help Clients to view Swath
HDF5-Friendly OPeNDAP Client Library
18HDF and HDF-EOS Workshop XII10/17/2008
![Page 19: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/19.jpg)
HDF5 Handler
DAP Protocol (via http)
Server(apache)
View Data (User)
Visualizing HDF-EOS5 Grids
19HDF and HDF-EOS Workshop XII10/17/2008
HDF5 Handlerw/ tweaks
Remote HDF5
Client Library(libnc-dap)
Visualization tools
(grads)
Remote HDF-EOS5
Swath
Visualizing HDF-EOS5 Swath Problem
???
HDF5-FriendlyClient Library(liboc-dap)
Visualization Tools
(gradsoc)
View Swath
![Page 20: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/20.jpg)
Demo: MLS swath via GrADS
• GrADS coupled with our client library
• The client library provides grid mapping from swath data
• GrADS displays swath through HDF-EOS5 specific client library API calls
20HDF and HDF-EOS Workshop XII10/17/2008
![Page 21: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/21.jpg)
Summary of Client Library Prototype
• Finished coding • Tested with the GrADS visualization
client• Documented • Working on a demo DAP to HDF5
tool by using the client library prototype
• Will test with NCL if time allows
10/17/2008 HDF and HDF-EOS Workshop XII 21
![Page 22: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/22.jpg)
Caution
• Our Client Library is ONLY PROTOTYPE! It does NOT support all DAP data types It does NOT support all AURA files It does NOT support all Visualization clients
22HDF and HDF-EOS Workshop XII10/17/2008
![Page 23: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/23.jpg)
Summary
• HDF5 Access via OPeNDAP Is Easy Is used by GES DISC to serve Aura files
• HDF5-Friendly OPeNDAP Client Library Prototype Serves HDF5 better(EOS swath) Benefits visualization clients
23HDF and HDF-EOS Workshop XII10/17/2008
![Page 24: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/24.jpg)
Future Work
• HDF5 to DAP2 Mapping Document• Release HDF5-friendly OPeNDAP Client Library Prototype
• URL: http://hdfgroup.org/projects/opendap
24HDF and HDF-EOS Workshop XII10/17/2008
![Page 25: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/25.jpg)
Credits
Mike Folk (The HDF Group)Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang
(NASA)Jennifer Adams (GrADS)
Dave Brown (UCAR)
25HDF and HDF-EOS Workshop XII10/17/2008
![Page 26: HDF5 OPeNDAP Project Update and Demo](https://reader035.vdocument.in/reader035/viewer/2022070503/568156d9550346895dc47c58/html5/thumbnails/26.jpg)
Acknowledgement
This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.
26HDF and HDF-EOS Workshop XII10/17/2008