peter berrisford ral – data management group srb services
TRANSCRIPT
![Page 1: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/1.jpg)
Peter Berrisford
RAL – Data Management Group
SRB Services
![Page 2: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/2.jpg)
Peter Berrisford
RAL
Introduction
• An Overview of SRB
• CCLRC and SRB
• Case Study: e-Minerals Mini-Grid
• SRB Production Services
• Questions
![Page 3: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/3.jpg)
Peter Berrisford
RAL
Managing Data
• Historically data has been STORED rather than MANAGED
• Problems arising from this include:– Scaling– Distribution– Access Control, Authentication, Security– Data Migration– Data Curation
![Page 4: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/4.jpg)
Peter Berrisford
RAL
What is SRB?
• Storage Resource Broker (SRB) is a software product developed by the San Diego Supercomputing Centre (SDSC).
• Allows users to access files and database objects across a distributed environment.
• Actual physical location and way the data is stored is abstracted from the user
• Allows the user to add user defined metadata describing the scientific content of the information
![Page 5: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/5.jpg)
Peter Berrisford
RAL
How SRB Works
MCATDatabase
MCATServer
SRB AServer
SRB BServer
SRBClient
a
b
c d
e
f
g
• 4 major components:– The Metadata Catalogue
(MCAT)– The MCAT-Enabled
SRB Server– The SRB Storage Server– The SRB Client
![Page 6: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/6.jpg)
Peter Berrisford
RAL
The MCAT Database
• The MCAT database is a metadata repository that provides a mechanism for storing information used by the SRB system.
• Includes both– Internal system data required for
running the system– Application (user) metadata
regarding data sets being brokered by SRB.
![Page 7: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/7.jpg)
Peter Berrisford
RAL
The MCAT Server• At least one SRB Server must be
installed on the node that can access the MCAT database. This is known as the MCAT-Enabled SRB Server.
• MCAT SRB Server works directly against the MCAT database to provide SRB Services
• All other SRB Servers interact through the MCAT Server
![Page 8: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/8.jpg)
Peter Berrisford
RAL
The SRB Server
• The SRB Server is a middleware application that accepts requests from clients and obtains/queries/manages the necessary data sets.
• It queries the MCAT SRB Server to gather information on datasets and supplies this back to the SRB client.
![Page 9: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/9.jpg)
Peter Berrisford
RAL
SRB Client Tools
• Provide a user interface to send requests to the SRB server.
• 4 main interfaces: – Command line (S-Commands)– MS Windows (InQ)– Web based (MySRB).– Java (JARGON)
• Web Services (MATRIX)
![Page 10: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/10.jpg)
Peter Berrisford
RAL
![Page 11: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/11.jpg)
Peter Berrisford
RAL
Concepts
• Location: A physical node running an SRB Server
• Physical Resource: A storage area managed by an SRB Server
• Logical Resource: One or more Physical Resources – can be distributed
• Collection – Data abstraction of resources
![Page 12: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/12.jpg)
Peter Berrisford
RAL
SRB in Detail
SRBArchives
ADS, HPSS,ADSM,DMF
DatabasesDB2, Oracle,PostgreSQL
File SystemsUnix, NT,Mac OSX
Application
C, C++, Linux I/O
Unix Shell
Resource,User
User Defined
ApplicationMeta-data
RemoteProxies
DataCutter
Third-partycopy
Java, NTBrowsers
WebPrologPython
MCAT
HRM
![Page 13: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/13.jpg)
Peter Berrisford
RAL
Administration
• Users / Locations / Resources must be managed
• Two methods for doing this:
– Java MCAT Admin Tool
– Command line tools
![Page 14: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/14.jpg)
Peter Berrisford
RAL
CCLRC and SRB
• The Data Management Group in CCLRC started working with SRB in November 2002 after a fact finding mission to the USA.
• There was an immediate requirement for a storage based product that allowed the addition of searchable metadata
• Generated lots of internal interest, which led to a number of projects with SRB
![Page 15: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/15.jpg)
Peter Berrisford
RAL
SRB Example: CMS
• Largest project using CCLRC SRB services to date is the CERN CMS experiment.
• SRB chosen for ‘Pre-Challenge Production’, producing data for Data Challenge 2003/2004 (DC03/DC04)
• Need to prove data can be transferred, replicated and stored at LHC rates
• DC04 provided key input to SRB Version 3.2
![Page 16: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/16.jpg)
Peter Berrisford
RAL
SRB Case Study: e-Minerals
• UK e-science project for modelling the atomistic processes involved in environmental issues
![Page 17: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/17.jpg)
Peter Berrisford
RAL
e-Minerals Requirements
• Data Management Requirements
– Scientists want to store input and output files from simulations in different locations
– manage their own files/data via the web
– give access to other project members
– give temporary access to others
![Page 18: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/18.jpg)
Peter Berrisford
RAL
Architecture
Daresbury App Server
Cambridge
SRB Resource
Reading
SRB Resource
Bath
SRB Resource
Eminerals MiniGrid
Daresbury Database server
MCAT
SRB Server
Oracle Client
MySRB
Web Browser
Application server runs SRB software
Database server holds locations of files
![Page 19: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/19.jpg)
Peter Berrisford
RAL
Building on Experience - New Services
• CCLRC SRB Service– Initial service availability: October– Proposed Customers include:
• ISIS Facility, British Atmospheric Data Centre (BADC), AHDS
– ADS interface (with Containers)– Test systems in place
• NGS SRB Service– e-Minerals, e-Materials, Integrative Biology
![Page 20: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/20.jpg)
Peter Berrisford
RAL
SRB Services
• SRB version 3.2
– Performance, scalability and reliability
• Ongoing Service Enhancements
– Automatic failover
– Product Documentation and Training - Collaboration with SDSC
![Page 21: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/21.jpg)
Peter Berrisford
RAL
Summary
• Links established with SRB community and SDSC• Real SRB projects implemented • Creating new generation of SRB Production
Systems• Can help community with:
– SRB Test Systems– SRB Production Systems– SRB Training and Support
• Contributing to future versions
![Page 22: Peter Berrisford RAL – Data Management Group SRB Services](https://reader035.vdocument.in/reader035/viewer/2022062619/551604fd550346a2308b4da1/html5/thumbnails/22.jpg)
Peter Berrisford
RAL
Questions
• ?