integrating hdf5 with srb
DESCRIPTION
Integrating HDF5 with SRB. Object-level Access to Remote Files. P eter Cao, NCSA Mike Wan, SDSC. Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration. Outline. Introduction to HDF5 The HDF-SRB model SRB Support in HDFView. Matter & universe. Life & nature. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/1.jpg)
February 2-3, 2006 SRB Workshop, San Diego
Peter Cao, NCSA
Mike Wan, SDSC
Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration
Object-level Access to Remote Files
Integrating HDF5 with SRB
![Page 2: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/2.jpg)
February 2-3, 2006 SRB Workshop, San Diego 2/26
Outline
Introduction to HDF5
The HDF-SRB model
SRB Support in HDFView
![Page 3: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/3.jpg)
February 2-3, 2006 SRB Workshop, San Diego 3/26
Overview of HDF5 Answering big
questions …
Matter & universe
Weather & climate
August 24, 2001 August 24, 2002Total Column Ozone (Dobson)
60 385 610
Life & nature
![Page 4: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/4.jpg)
February 2-3, 2006 SRB Workshop, San Diego 4/26
Overview of HDF5 Involves big
data …
![Page 5: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/5.jpg)
February 2-3, 2006 SRB Workshop, San Diego 5/26
Overview of HDF5 On big
computers …
![Page 6: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/6.jpg)
February 2-3, 2006 SRB Workshop, San Diego 6/26
Overview of HDF5 HDF solution
…
Software & tools
open source & multiple platform
Commonmodels
extensions Standard
APIsconventions & easy use
File formatfor all kinds of data
Efficiencystorage & IO
![Page 7: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/7.jpg)
February 2-3, 2006 SRB Workshop, San Diego 7/26
Overview of HDF5 Exmaple HDF5
![Page 8: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/8.jpg)
February 2-3, 2006 SRB Workshop, San Diego 8/26
Overview of HDF5 HDF Software
HDF I/O Library
Tools & ApplicationsTools & Applications
HDF FileHDF File
![Page 9: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/9.jpg)
February 2-3, 2006 SRB Workshop, San Diego 9/26
Overview of HDF5 Object
model
Primary ObjectsGroupsDatasets
Additional ways to organize dataAttributesSharable objectsStorage and access properties
![Page 10: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/10.jpg)
February 2-3, 2006 SRB Workshop, San Diego 10/26
Overview of HDF5 Groups
“/”
tom dick harry
temp
A mechanism for collections of related objectsEvery file starts with a root groupSimilar to UNIXdirectoriesCan have attributes
![Page 11: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/11.jpg)
February 2-3, 2006 SRB Workshop, San Diego 11/26
Overview of HDF5
Datasets
DataMetadataDataspace
3
RankRank
Dim_2 = 5Dim_1 = 4
DimensionsDimensions
time = 32.4
pressure = 987
temp = 56
AttributesAttributes
Chunked
compressed
Dim_3 = 7
Storage infoStorage info
IEEE 32-bit float
DatatypeDatatype
![Page 12: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/12.jpg)
February 2-3, 2006 SRB Workshop, San Diego 12/26
Overview of HDF5 Data
subsetting
(c) A sequence of points from a 2D array to a sequence of points in a 3D array.
(d) Union of hyperslabs in file to union of hyperslabs in memory.
(b) Regular series of blocks from a 2D array to a contiguous sequence at a certain offset in a 1D array
(a) Hyperslab from a 2D array to the corner of a smaller 2D array
![Page 13: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/13.jpg)
February 2-3, 2006 SRB Workshop, San Diego 13/26
Project Description
Motivation
SRB HDF5
Indexing and searching Distributed data system Access control
Large and diverse data High performance access Interactive and subsetting
High performance distributed data system
![Page 14: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/14.jpg)
February 2-3, 2006 SRB Workshop, San Diego 14/26
Project Description Goals
Working prototype of client/server system forobject-level access to HDF5 stored in the SRB
Use SRB as middleware to transfer data between the server and client Use Object-level access for interactive and efficient access to part of the file
![Page 15: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/15.jpg)
February 2-3, 2006 SRB Workshop, San Diego 15/26
Remote Data Access on SRB
Methods
Normal ways to access SRB:Get the whole file: large files (100TB
SCEC)Use POSIX low level calls: low
performance
New way: Implement proxy operations to access
objects or parts of objects in one request
![Page 16: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/16.jpg)
February 2-3, 2006 SRB Workshop, San Diego 16/26
Normal SRB File Access Architecture
SRB ServerSRB Server
HDF5HDF5
MCAT
clientclient
HDF5 File(whole file or a
sequence of bytes)
![Page 17: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/17.jpg)
February 2-3, 2006 SRB Workshop, San Diego 17/26
Object-level File Access Architecture
SRB ServerSRB Server
MCAT
HDF5 LibraryHDF5 Library
HDF5-SRB Module(pack/unpack messages)
HDF5 Object(File, Group, Dataset,
Subset, Attribute)
HDF5-SRB Module(pack/unpack messages)
ClientClient ServerServer
HDF5 Object(File, Group, Dataset,
Subset, Attribute)
Client Application HDF5 file
![Page 18: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/18.jpg)
February 2-3, 2006 SRB Workshop, San Diego 18/26
Examples of File Access
HDF5HDF5
I need to see the eye of Hurricane Bob!
![Page 19: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/19.jpg)
February 2-3, 2006 SRB Workshop, San Diego 19/26
Examples of File Access Whole file
transfer
clientclient
Get the file
Transfer large image – slow!
HDF5HDF5
![Page 20: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/20.jpg)
February 2-3, 2006 SRB Workshop, San Diego 20/26
Examples of File Access SRB POSIX
API
HDF5HDF5
clientclient
image foundimage open
open image
find imagefile’s open
Open file
Many small messages – slow and complex!
![Page 21: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/21.jpg)
February 2-3, 2006 SRB Workshop, San Diego 21/26
Examples of File Access Object
level
clientclient
HDF5HDF5
Get me the eye of hurricane Bob
1 request, small transfer – fast!
![Page 22: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/22.jpg)
February 2-3, 2006 SRB Workshop, San Diego 22/26
HDF5-SRB Model New
objects/APIs
A new set data objects H5File, H5Group, H5Dataset, H5Datatype, etc Encapsulated client requests and server results
Enhanced SRB APIs Pack/Unpack routines (exchange data between
byte stream and structure) to handle complicated struct – string, pointers, pointers to arrays, arrays of pointers, etc
New srbGenProxyFunct (general Proxy Function) handles other types of request besides HDF5
![Page 23: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/23.jpg)
February 2-3, 2006 SRB Workshop, San Diego 23/26
HDF5-SRB Model Data Flow
Client APIsrbObjRequest(void *obj, int objID)
Server APIsrbObjProcess(void *obj, int objID)
srbGenProxyFunct
1. packMsg()
2. u
npac
kMsg
()
HDF5 Library
HDF5 file
3. H5Obj::op()
4. Access file
5. H5Object
6. p
ackM
sg()
7. unpackMsg()
SRB Server
![Page 24: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/24.jpg)
February 2-3, 2006 SRB Workshop, San Diego 24/26
Running Server/Client
A SRB server that supports HDF5 HDF5 library and other external libraries (SZIP, ZLIB) A SRB version 3.4 or later from
http://www.sdsc.edu/srb/ Follow instruction on how to run SRB server from UG
packed with SRB source release or online at http://hdf.ncsa.uiuc.edu/hdf-srb-html/HDF-SRB-UG.html
Any client application that implements HDF5-SRB Objects No HDF5 library is required on the client Example client application: HDFView 2.3 or above
![Page 25: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/25.jpg)
February 2-3, 2006 SRB Workshop, San Diego 25/26
Short Demo HDFViewSupport Windows and Linux
![Page 26: Integrating HDF5 with SRB](https://reader036.vdocument.in/reader036/viewer/2022081514/56814737550346895db47647/html5/thumbnails/26.jpg)
February 2-3, 2006 SRB Workshop, San Diego 26/26
Question / Comments?