data management for internet backplane protocol

1

Data Management for Internet Backplane Protocol

by Tang Ming Assoc/Prof. Francis Lee

School of Computer Engineering,Nanyang Technological University, Singapore

2

0. Outline Introduction The Logistical Networking &

Globus Replica Catalog Data Management for IBP

3

1. Introduction Backgrounds of IBP

Univ. of Tennessee Base layer of the Logistical Networking A technology to share network storages distributed

across the globe Our work

Integrate IBP and Globus Replica Catalog, develop the ExNode File Warehouse, and implement a data management system for IBP.

4

2. Logistical Networking IBP: a middleware to allow the sharing of distributed storage

resources over the network L-Bone: a resource discovery service that maintains a list of

public depots and metadata about those depots exNode: a data structure for aggregation storage allocations LoRS: provide API and command line tools

IBP Depots

Depot

Depot

DepotDepot

Depot

Depot

Depot

Depot

Depot

User

L-Bone Servers

5

2.1 Examples: Upload / Download a File to IBP with LoRS Tools $ lors_upload blues.mp3 –f –c 1 –F 1

File “blues.mp3” is uploaded to IBP and the exNode file “blues.mp3.xnd” is generated.

$ lors_download blues.mp3.xnd –f File “blues.mp3” is downloaded

The data resources are not shareable to public Need a data management service for data sharing!

6

3. Globus Replica Catalog Provide the replica location information for

Data Grid Logical File Name Physical File Name

LFN PFNFile1 gsiftp://ssim1.sce.ntu.edu.sg/dir/File2 ftp://ssim4.sce.ntu.edu.sg/dir2/File3 irm://vista.sas.ntu.edu.sg/tmp/

7

3.1 Motivation of using GRC

Globus is the de facto standard of the Grid system

The GRC has been used by many Data Grid projects

It’s a way to integrate IBP and the existed Data Grids

8

4. Data Management for IBP System architecture Components

The Logistical Networking Globus Replica Catalog (GRC) ExNode File Warehouse (EFW) IBP Data Management (IDM)

Workflow examples (upload & download) Application

9

4.1 System Architecture

IBP Data Management

IBM

Applications

Globus Replica Catalog(GRC)

exNode File Warehouse(s)(EFW)

LoRSInterface

GRCInterface

EFWInterface

Logistical Networking

DepotDepot

Depot

Depot

Depot

Depot

Depot

Depot

Depot

L-Bone Servers

Depot

10

4.2 ExNode File Warehouse (EFW) EFW stores and maintains the exNode files for the data

in IBP Also keeps the metadata about the data files, e.g., the

expiration dates and time stamps EFW interacts with IDM and GRC

client

exNode info.(in memory)

Storage

Main thread

Communication thread

Expiration checking thread

EFW

Communication thread

Communication thread

client client

IBM

Globus RC

…

11

4.3 IBP Data Management (IDM)

IDM interacts with the Logistical Networking, GRC, and EFW, and it provides APIs to applications.

Functions: upload, download, refresh, delete, query a file or a file set

lorsUploadFile(FN, duration, copies, ...)

App. IDM IBP GRC EFW(s)

InitService()

return result

putFile(FN, duration, copies, ...)

return exNode

putExnode(exNode, efw, duration)

return key

regLoc2Coll(efw, coll)

return result

regFile2Coll(FN, coll)

regFile2Loc(FN, efw)

return key

return result

return result

12

4.4 Workflow – Uploading a File

IBP Data Management

(IDM)

IBP

IBM

1

Globus Replica Catalog(GRC)

exNode file warehouse

(EFW)

8

2 3

4

5

6

7iMac

User

13

4.5 Workflow – Downloading a File

IBP Data Management

(IDM)

IBP

IBM

1

exNode file warehouse

(EFW)

8

6 75

4

2

3iMac

UserGlobus Replica Catalog

(GRC)

14

4.6 Support Grid Computing The system provides a global data cache to support Grid

computing

IBP Data Management System

IBM

Globus Replica Catalog(GRC)

exNode File Warehouse(s)(EFW)

Logistical Networking

DepotDepot

Depot

Depot

Depot

Depot

Depot

Depot

Depot

L-BoneServers

Depot

Grid Computing Resources

latigid

latigid latigidlatigid

latigid latigid latigid latigid

latigid

User

Submit jobs

Data files

SG

US

UK

15

5. Future Works Develop data discovery and replica

selection service Support Globus new service: Replica Location

Service (RLS)

For further information:http://www.ntu.edu.sg/home5/p148943492/idm.htm

Email: mtang@pmail.ntu.edu.sg ebslee@ntu.edu.sg

16

Thanks!