data management for internet backplane protocol
DESCRIPTION
Data Management for Internet Backplane Protocol. by Tang Ming Assoc/Prof. Francis Lee. School of Computer Engineering, Nanyang Technological University, Singapore. 0. Outline. Introduction The Logistical Networking & Globus Replica Catalog - PowerPoint PPT PresentationTRANSCRIPT
1
Data Management for Internet Backplane Protocol
by Tang Ming Assoc/Prof. Francis Lee
School of Computer Engineering,Nanyang Technological University, Singapore
2
0. Outline Introduction The Logistical Networking &
Globus Replica Catalog Data Management for IBP
3
1. Introduction Backgrounds of IBP
Univ. of Tennessee Base layer of the Logistical Networking A technology to share network storages distributed
across the globe Our work
Integrate IBP and Globus Replica Catalog, develop the ExNode File Warehouse, and implement a data management system for IBP.
4
2. Logistical Networking IBP: a middleware to allow the sharing of distributed storage
resources over the network L-Bone: a resource discovery service that maintains a list of
public depots and metadata about those depots exNode: a data structure for aggregation storage allocations LoRS: provide API and command line tools
IBP Depots
Depot
Depot
DepotDepot
Depot
Depot
Depot
Depot
Depot
User
L-Bone Servers
5
2.1 Examples: Upload / Download a File to IBP with LoRS Tools $ lors_upload blues.mp3 –f –c 1 –F 1
File “blues.mp3” is uploaded to IBP and the exNode file “blues.mp3.xnd” is generated.
$ lors_download blues.mp3.xnd –f File “blues.mp3” is downloaded
The data resources are not shareable to public Need a data management service for data sharing!
6
3. Globus Replica Catalog Provide the replica location information for
Data Grid Logical File Name Physical File Name
LFN PFNFile1 gsiftp://ssim1.sce.ntu.edu.sg/dir/File2 ftp://ssim4.sce.ntu.edu.sg/dir2/File3 irm://vista.sas.ntu.edu.sg/tmp/
7
3.1 Motivation of using GRC
Globus is the de facto standard of the Grid system
The GRC has been used by many Data Grid projects
It’s a way to integrate IBP and the existed Data Grids
8
4. Data Management for IBP System architecture Components
The Logistical Networking Globus Replica Catalog (GRC) ExNode File Warehouse (EFW) IBP Data Management (IDM)
Workflow examples (upload & download) Application
9
4.1 System Architecture
IBP Data Management
IBM
Applications
Globus Replica Catalog(GRC)
exNode File Warehouse(s)(EFW)
LoRSInterface
GRCInterface
EFWInterface
Logistical Networking
DepotDepot
Depot
Depot
Depot
Depot
Depot
Depot
Depot
L-Bone Servers
Depot
10
4.2 ExNode File Warehouse (EFW) EFW stores and maintains the exNode files for the data
in IBP Also keeps the metadata about the data files, e.g., the
expiration dates and time stamps EFW interacts with IDM and GRC
client
exNode info.(in memory)
Storage
Main thread
Communication thread
Expiration checking thread
EFW
Communication thread
Communication thread
client client
IBM
Globus RC
…
11
4.3 IBP Data Management (IDM)
IDM interacts with the Logistical Networking, GRC, and EFW, and it provides APIs to applications.
Functions: upload, download, refresh, delete, query a file or a file set
lorsUploadFile(FN, duration, copies, ...)
App. IDM IBP GRC EFW(s)
InitService()
return result
putFile(FN, duration, copies, ...)
return exNode
putExnode(exNode, efw, duration)
return key
regLoc2Coll(efw, coll)
return result
regFile2Coll(FN, coll)
regFile2Loc(FN, efw)
return key
return result
return result
12
4.4 Workflow – Uploading a File
IBP Data Management
(IDM)
IBP
IBM
1
Globus Replica Catalog(GRC)
exNode file warehouse
(EFW)
8
2 3
4
5
6
7iMac
User
13
4.5 Workflow – Downloading a File
IBP Data Management
(IDM)
IBP
IBM
1
exNode file warehouse
(EFW)
8
6 75
4
2
3iMac
UserGlobus Replica Catalog
(GRC)
14
4.6 Support Grid Computing The system provides a global data cache to support Grid
computing
IBP Data Management System
IBM
Globus Replica Catalog(GRC)
exNode File Warehouse(s)(EFW)
Logistical Networking
DepotDepot
Depot
Depot
Depot
Depot
Depot
Depot
Depot
L-BoneServers
Depot
Grid Computing Resources
latigid
latigid latigidlatigid
latigid latigid latigid latigid
latigid
User
Submit jobs
Data files
SG
US
UK
15
5. Future Works Develop data discovery and replica
selection service Support Globus new service: Replica Location
Service (RLS)
For further information:http://www.ntu.edu.sg/home5/p148943492/idm.htm
16
Thanks!