integrating hdf5 with srb
DESCRIPTION
Fast partial access to objects from very large files in the SDSC Storage Resource Broker (SRB5) can be extremely challenging, even when those objects are small. The HDF-SRB project integrates the SRB and NCSA Hierarchical Data Format (HDF5), to create an access mechanism within the SRB that is can be orders of magnitude more efficient than current methods for accessing object-based file formats. The project provides interactive and efficient access to datasets or subsets of datasets in large files without bringing entire files into local machines. A new set of data structures and APIs have been implemented to the SRB support such object-level data access. A working prototype of the HDF5-SRB data system has been developed and tested. The SRB support is implemented in HDFView as a client application.TRANSCRIPT
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 1
Integrating HDF5 with SRB
Peter CaoThe HDF Group
November 29, 2006
Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD2
Imagine
10 GB
100 Frames x 10 GB
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD3
Outline
• Introduction to the project• HDF5-SRB model• Server and client application• Demo if time permits
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD4
Motivation
SRB HDF5
Distributed data system Indexing and searching Access control, etc
Large and diverse data High performance I/O Subsetting, etc
High performance distributed data system
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD5
Goal
• Use SRB as middleware to transfer data between the server and client
• Use Object-level access for interactive and efficient access to parts of the file
A client/server system for object-level access to HDF5 files stored at SRB
server
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD6
HDF5 Data Access on SRB
• Normal ways to access SRB:• Get the whole file: large files (100GB)• Use POSIX low level calls: low performance
• New way: • Implement proxy operations to access objects or parts of
objects in one request
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD7
Example: Whole File Access
clientclient
Get the file
Transfer large file – slow!
HDF5HDF5
serverserverI need to see the eye of
Hurricane Bob!
HDF5HDF5
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD8
Example: Object Level Access
clientclient
HDF5HDF5
Get me the eye of hurricane Bob
1 request, small transfer – fast!
serverserverI need to see the eye of
Hurricane Bob!
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD9
HDF5/SRB Model
HDF5-SRB Module(pack/unpack messages)
clientclient serverserver
HDF5 Object(File, Group, Dataset,
Subset, Attribute)
SRB ServerSRB Server
HDF5HDF5
MCAT
HDF5 LibraryHDF5 Library
HDF5-SRB Module(pack/unpack messages)
HDF5 Object(File, Group, Dataset,
Subset, Attribute)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD10
New Data Objects and APIs
• A new set data objects• H5File, H5Group, H5Dataset, H5Datatype, etc• Encapsulated client requests and server results
• Enhanced SRB APIs• Pack/Unpack routines (exchange data between byte
stream and structure) to handle complicated struct – string, pointers, pointers to arrays, arrays of pointers, etc
• New srbGenProxyFunct (general Proxy Function) handles other types of request besides HDF5
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD11
Running Server/Client
• A SRB server that supports HDF5• HDF5 library and other external libraries (SZIP, ZLIB)• A SRB version 3.4.0 or later from http://
www.sdsc.edu/srb/• Follow instruction on how to run SRB server from UG
packed with SRB source release or online at http://www.hdfgroup.org/hdf-srb-html/
• Any client application that implements HDF5-SRB Objects• No HDF5 library is required on the client• Example client application: HDFView 2.3 or above
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD12
Example
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD13
HDFView property file
.hdfview2.3srbaccount0.domain=demosrbaccount0.home=/A/home/srbAdmin.demosrbaccount0.host=kagiso.hdfgroup.uiuc.edusrbaccount0.password=srbDemo05srbaccount0.port=5544srbaccount0.resource=demoRescsrbaccount0.user=srbAdminsrbaccount1.domain=demosrbaccount1.home=/A/home/srbAdmin.demosrbaccount1.host=stanley.ncsa.uiuc.edusrbaccount1.password=srbdemo05srbaccount1.port=5544srbaccount1.resource=demoRescsrbaccount1.user=srbAdmin