data management the gsm-wg perspective. background srm is the storage resource manager a control...
TRANSCRIPT
Data ManagementThe GSM-WG Perspective
Background
• SRM is the Storage Resource Manager
• A Control protocol for Mass Storage Systems
• Standard protocol: OGF GSM-WG– Chairs: A Shoshani (LBNL), J Jensen (STFC
RAL)
Who Is Using It?
• EGEE, OSG,…
• WLCG
• …
• Not only
academic
HPC
TIER 0
Country or ROC
Storage Element Implementations
• dCache (DESY+FNAL)• DPM (CERN)• CASTOR (CERN+RAL)• BestMan (LBNL)• Jasmine (JLAB)• StoRM (INFN/ICTP-EGRID)
1
1
1
1
11
What is a “Storage Element”?
Storage Element
Disk storage
Tape storage
THE (gLite) GRID
•Computing •Data Mgmt Metadata
Disk storageDisk storage
Disk Pools
What is a “Storage Element”
• Interface to mass storage systems
• Part of a loosely coupled grid
• Provide uniform (control) interface
• Protect the storage system against the grid
• Enable grid to efficiently use storage
What is a Storage Element?
DataTransfer
ControlInterface
(SRM)
InformationSystem
StorageElement
Used by EGEE, OSG,…
Six-seven different implementations
An SRM is a “Storage Element” but not vice versaGIN: Use SRB as Storage Element
Tech
• ~6 interoperating implementations– 362 storage elements deployed– 226 are SRM1 implementations– 6 SRM2 implementations (can’t be quite right)
• Some Open Source, some not
• Using SOAP over GSI sockets– Delegation needed for srmCopy– No WS-Anything
Where is the File
GUID
SURL SURL SURL
LFN LFN LFN
PhysicalFile
TURL
PhysicalFile
StorageElements
PhysicalFilePhysical
File
Information View (simplified)
StorageArea
StorageElement
VOInfo
ControlProtocol
Data TransferProtocol
Space?Resource
Accounting
Owner/Access
How is the file?
Capa-bilities
ExpirationMode
RetentionPolicy
AccessLatency
Replica Output CustodialReleaseWhen
expired
WarnWhen
expired
Perma-nent
Online Nearline(Offline)(Lost)(…)
How I reach the file?
File AccessParameters
OverwriteMode
AccessPattern
Connection
Yes NoWhen
DifferentTransfer Processing WAN LAN
Transfer Protocols
• WAN protocols– Must always support GridFTP– Suggestion to support HTTP/HTTPS
• GSI issue though - see tech slide• dCache supports http for reading
• Support LAN protocols– Secure vs insecure
LCG: GFAL
• LCG (will) decide to use GFAL – the “Grid File Access Library”
• It was decided to interface to EDG SE using SRM 1 interface
• For now using EDG RM
POSIX interface
SRM 1 client
EDG 2.1 Storage Element
Mass Storage
EDG SRM
EGEE: DICOM server supportThe GridThe Grid
Storage Element
WP10 DM2
DICOMServer
Metadata
Encrypt, anonymise
Metadata
Store keyStore patient metadata
Access control on metadata required; different ACLs for different types of metadata
Challenges
Services
Capabilities
Scale,Performance
Economy,Sustainability
Middleware
State of the Art
Users
Challenges
Users
Applications
Culture,History
Customermgmt
Usability
Users
Services
Trust
Availability
Accounting
Discovery
Services
State of the Art
WebServices
Virtualisation
Media
State of the Art
Middleware
Stability
Applications
MaintenanceSupport
Ease of installAnd Config
Middleware
Scale, Performance
Staging
Transfer rates Size of files
Number of files
Volume
Scale,Performance
Sustainability, Economy
Scale
Trust Dynamic
Agreement
Cost Model
Economy
Capabilities
Content
Access
Curation
SECURITY
Capabilities