Download - Integrating HDF5 with SRB
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 1
Integrating HDF5 with SRB
Peter CaoThe HDF Group
November 29, 2006
Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD2
Imagine
10 GB
100 Frames x 10 GB
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD3
Outline
• Introduction to the project• HDF5-SRB model• Server and client application• Demo if time permits
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD4
Motivation
SRB HDF5
Distributed data system Indexing and searching Access control, etc
Large and diverse data High performance I/O Subsetting, etc
High performance distributed data system
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD5
Goal
• Use SRB as middleware to transfer data between the server and client
• Use Object-level access for interactive and efficient access to parts of the file
A client/server system for object-level access to HDF5 files stored at SRB
server
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD6
HDF5 Data Access on SRB
• Normal ways to access SRB:• Get the whole file: large files (100GB)• Use POSIX low level calls: low performance
• New way: • Implement proxy operations to access objects or parts of
objects in one request
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD7
Example: Whole File Access
clientclient
Get the file
Transfer large file – slow!
HDF5HDF5
serverserverI need to see the eye of
Hurricane Bob!
HDF5HDF5
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD8
Example: Object Level Access
clientclient
HDF5HDF5
Get me the eye of hurricane Bob
1 request, small transfer – fast!
serverserverI need to see the eye of
Hurricane Bob!
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD9
HDF5/SRB Model
HDF5-SRB Module(pack/unpack messages)
clientclient serverserver
HDF5 Object(File, Group, Dataset,
Subset, Attribute)
SRB ServerSRB Server
HDF5HDF5
MCAT
HDF5 LibraryHDF5 Library
HDF5-SRB Module(pack/unpack messages)
HDF5 Object(File, Group, Dataset,
Subset, Attribute)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD10
New Data Objects and APIs
• A new set data objects• H5File, H5Group, H5Dataset, H5Datatype, etc• Encapsulated client requests and server results
• Enhanced SRB APIs• Pack/Unpack routines (exchange data between byte
stream and structure) to handle complicated struct – string, pointers, pointers to arrays, arrays of pointers, etc
• New srbGenProxyFunct (general Proxy Function) handles other types of request besides HDF5
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD11
Running Server/Client
• A SRB server that supports HDF5• HDF5 library and other external libraries (SZIP, ZLIB)• A SRB version 3.4.0 or later from http://
www.sdsc.edu/srb/• Follow instruction on how to run SRB server from UG
packed with SRB source release or online at http://www.hdfgroup.org/hdf-srb-html/
• Any client application that implements HDF5-SRB Objects• No HDF5 library is required on the client• Example client application: HDFView 2.3 or above
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD12
Example
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD13
HDFView property file
.hdfview2.3srbaccount0.domain=demosrbaccount0.home=/A/home/srbAdmin.demosrbaccount0.host=kagiso.hdfgroup.uiuc.edusrbaccount0.password=srbDemo05srbaccount0.port=5544srbaccount0.resource=demoRescsrbaccount0.user=srbAdminsrbaccount1.domain=demosrbaccount1.home=/A/home/srbAdmin.demosrbaccount1.host=stanley.ncsa.uiuc.edusrbaccount1.password=srbdemo05srbaccount1.port=5544srbaccount1.resource=demoRescsrbaccount1.user=srbAdmin
