data management: us focus kaushik de, armen vartapetian univ. of texas at arlington us atlas...
TRANSCRIPT
![Page 1: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/1.jpg)
Data Management: US Focus
Kaushik De, Armen VartapetianUniv. of Texas at Arlington
US ATLAS Facility, SLAC
Apr 7, 2014
![Page 2: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/2.jpg)
Introduction
We are at midway point of LS1 Numerous improvements in computing underway Need to be ready for new challenges during Run 2 New systems: ProdSys2, Rucio, Grid->Clouds, HPC
Are we ready for Run 2 Data Management? Rucio migration coming soon – need extensive testing Biggest challenge will be Tier 1 storage shortage Tier 2 storage should be ok – to start with
The most important items? Need new ADC driven data management and data distribution plans Need automated tools for managing US user data
Apr 7, 2014Kaushik De 2
![Page 3: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/3.jpg)
Apr 7, 2014Kaushik De 3
![Page 4: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/4.jpg)
Apr 7, 2014Kaushik De 4
![Page 5: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/5.jpg)
Apr 7, 2014Kaushik De 5
![Page 6: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/6.jpg)
Space Tokens
Maybe we will be able to simplify tokens after Rucio For now they are necessary – for accounting, deletions…
ADC managed: DATADISK/DATATAPE GROUPDISK SCRATCHDISK
Locally managed: PRODDISK USERDISK LOCALGROUPDISK
Apr 7, 2014Kaushik De 6
![Page 7: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/7.jpg)
Lack of Cache Space on DATADISK
Apr 7, 2014Kaushik De 7
![Page 8: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/8.jpg)
BNL DATADISK
Apr 7, 2014Kaushik De 8
![Page 9: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/9.jpg)
US Tier 2 DATADISK
Apr 7, 2014Kaushik De 9
![Page 10: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/10.jpg)
DATADISK at US Tier 2’s
Apr 7, 2014Kaushik De 10
![Page 11: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/11.jpg)
TAPE Will be Crucial for Run 2
Apr 7, 2014Kaushik De 11
![Page 12: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/12.jpg)
SCRATCHDISK
Apr 7, 2014Kaushik De 12
![Page 13: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/13.jpg)
GROUPDISK
Apr 7, 2014Kaushik De 13
BNL-OSG2; 1205.6
AGLT2; 937MWT2_UC; 935.4
NET2; 237.1
SLACXRD; 593.6
SWT2; 504.9
US GROUPDISK Usage 4/5/14
![Page 14: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/14.jpg)
PRODDISK
Migration from pandamover Migration to Rucio
Apr 7, 2014Kaushik De 14
BNL-OSG2; 42.5
AGLT2; 55
MWT2_UC; 73
NET2; 109.7
SLACXRD; 125.9
SWT2; 112.5
![Page 15: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/15.jpg)
USERDISK
Managed locally in the US No change to current policy
Apr 7, 2014Kaushik De 15
BNL-OSG2; 779.3
AGLT2; 215
MWT2_UC; 324.3
NET2; 125.4
SLACXRD;
189.4
SWT2; 192
![Page 16: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/16.jpg)
Proposed US LOCALGROUPDISK Policy
Standard Policy on Total Used Space: Allow 3 TB (TBD) per user per site in US facilities If > 3 TB used, send automated warning emails
Exceptions Policy for Total Used Space: User needs to fill web form if they need more than standard limit Automatic exceptions granted for 20 TB at one site, 30 TB total US Exceptions will expire after duration specified by user
Exceptional cases (outside above policies): Must be approved by RAC
Last Access Time Policy: If data not used for more than 1 year (TBD), send warning email
Multiple Replicas Policy: If more than 7 replicas grid-wide, send warning emails
Group Usage Policy: If data is appropriate for placement on DATA/GROUPDISK, send email
Apr 7, 2014Kaushik De 16
![Page 17: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/17.jpg)
Current Status
BNL-OSG2; 202.87
AGLT2; 199.17
MWT2_UC; 279.11
NET2; 239.61
SLACXRD; 151.99SWT2; 28.77
LOCALGROUPDISK Usage in TB 4/5/14
Total used – 1.1 PB Extensive cleaning done recently (>700 TB) Need to automate management
Quotas per user/site Deletions and exceptions
Apr 7, 2014Kaushik De 17
![Page 18: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/18.jpg)
Current Tools
Apr 7, 2014Kaushik De 18
Developed by H. Ito
![Page 19: Data Management: US Focus Kaushik De, Armen Vartapetian Univ. of Texas at Arlington US ATLAS Facility, SLAC Apr 7, 2014](https://reader036.vdocument.in/reader036/viewer/2022070416/5697c0211a28abf838cd2cae/html5/thumbnails/19.jpg)
Future Tools
Need a new Localgroupdisk management system Under development – extending Hiro’s tools Database backend to keep historical space usage by user Database backend to keep track of allowed exceptions Web frontend for users System to send warning emails Provide summary statistics and monitoring
Apr 7, 2014Kaushik De 19