oxford update hepix pete gronbech gridpp project manager october 2014
TRANSCRIPT
Oxford UpdateHEPix
Pete GronbechGridPP Project Manager
October 2014
Oxford Particle Physics - Overview
• Oxford University has one of the largest Physics Departments in the UK– ~450 staff and 1100 students.
• Particle Physics sub department is the largest in the UK
• We now support Windows, Linux (Ubuntu) and Mac on the desktop across the department.– Ubuntu - Cobbler used to install base system, Cfengine used
for package control and configuration.• Two computational clusters for the PP physicists.
– Grid Cluster part of the SouthGrid Tier-2– Local Cluster (AKA Tier-3)
• A common Cobbler and Puppet system is used to install and maintain all the SL6 systems.
Lincoln - October 20142
UK GridPP Computing Structure
• One Tier-1 center - RAL• 18 University Particle Physics Departments• Each is part of a regional Grid Tier-2 center• Most have some local computing clusters (AKA Tier-3)
• Oxford is part of SouthGrid. • SouthGrid is comprised of all the non London based
sites in the South of the UK.• Birmingham, Bristol, Cambridge, JET, Oxford, RAL PPD,
Sussex.
Lincoln - October 20143
UKI Tier-1 & Tier-2 contributions
The UK the second largest contributor to the WLCG. (~11% cf. 28% for USA)
Accounting for the last year.
Tier-1 accounts for~31%
Tier-2s share as below
4Lincoln - October 2014
UK Grid Job Mix
Lincoln - October 20145
Lincoln - October 20146
UK Tier-2 reported CPU – Historical View to present
Comparing last update in 2012.
Lincoln - October 20147
SouthGrid SitesAccounting as reported by
APEL
Oxford Grid Cluster
• Upgrades over the last year to increase the capacity of the storage and some modest CPU upgrades.
• 15 Dell 720XDs servers (12*4TB raw capacity). Note we used SATA not SAS this time. Unlikely to do this in the future. (SAS becoming the default and support for SATA costs extra). SE is running DPM.
• Three ‘twin-squared’ Viglen Supermicro HX525T2i worker nodes have been installed. Intel E5-2650v2 8 core (16 Hyper-threaded cores each) provides 384 job slots with 2GB RAM.
• Two thirds of the Grid Cluster now running HT Condor behind ARC CE. Remaining third running legacy torque/maui driven by CREAM CE.
Lincoln - October 20148
Current capacity 16,768HS06 1300TB
Intel E5-2650 v2 SL6 HEPSPEC06
Average result 361.429% improvement over 2650 v1on SL5
Lincoln - October 20149
Power Usage – Twin squared chassis
Max 1165W
Idle 310W
Lincoln - October 201410
Oxford’s Grid Cluster
11Lincoln - October 2014
Begbroke Computer Room
12Lincoln - October 2014
Local Computer room showing PP cluster & cold
aisle containment
13
• Very similar h/w to the Grid Cluster.• Same Cobbler and Puppet management
setup.• Lustre used for larger groups• Capacity:- 7192 HS06, 716TB
Lincoln - October 2014
Networking
~900 MB/s = 7.2 Gbps.
• The University had a 10Gbit link to JANET with a 10 Gbit failover link.
• A third link was added and the Grid traffic routed exclusively down that in August 2013.
• Plots from March 2014 (Atlas transfer from BNL)
Lincoln - October 201414
• CMS Tier-3– Supported by RALPPD’s PhEDEx server – Useful for CMS, and for us, keeping the site busy in quiet times– However can block Atlas jobs and during accounting period not so desirable
• ALICE Support– There is a need to supplement the support given to ALICE by Birmingham.– Made sense to keep this in SouthGrid so Oxford have deployed an ALICE VO box
• UK Regional Monitoring– Oxford runs the nagios based WLCG monitoring for the UK– These include the Nagios server itself, and support nodes for it, SE, MyProxy and WMS/LB– Multi VO Nagios Monitoring added two years ago.
• IPv6 Testing– We take leading part in the IPv6 testing, many services enabled and tested by the community.– perfSONAR IPv6 enabled. RIPE Atlas probe also on IPv6.
• Cloud Development– Openstack test setup (Has run Atlas jobs)– VAC setup (LHCb, Atlas & GridPP Dirac server jobs)
Other Oxford Work
Lincoln - October 201415
Lincoln - October 201416
Conclusions
• Recent hardware purchases have provided both storage capacity and CPU performance improvements.
• Good Network connectivity• Solid computer rooms.• Medium sized Grid site but have involvement in many
development projects.
HEPix Spring 2015 is coming to Oxford
17
Other Oxford Attractions!
18
Including Oxford Physics
Lincoln - October 201419