oxford university particle physics unix overview sean brisbane particle physics systems...

Oxford University Particle Physics Unix Overview

Sean BrisbaneParticle Physics Systems Administrator

Room 661Tel 73389

[email protected]

14th October 2015

Graduate Lectures 1

Strategy Local Cluster Overview Connecting to it Grid Cluster Computer Rooms How to get help

14th October 2014Graduate Lectures 2


Particle Physics Strategy The Server / Desktop Divide

Win 7 PC Linux Desktop

Clients

Servers

General Purpose Unix

Server

Group DAQ

Systems

Linux Worker nodes

Web Server

Linux FileServers

Win 7 PC

Ubuntu PC

Virtual Machine Host

NIS Server

torque Server

laptop


Distributed Model

Files are not stored on the machine you are using, but on remote locations

You can run a different operating system by making a remote connection to it

Computing work splits broadly into Office work (local/remote) Writing code (remote) Computation (remote)

Use your favourite desktop/laptop for office files.

Make a remote desktop connection to pplxint8/9 to do computing work on the batch farm

It is better to write code on the remote Linux desktop


Recommended working strategy

We use RDP for remote desktop This means that everyone in particle physics

has access to multiple desktop environments from anywhere

Windows (termserv.physics.ox.ac.uk) Scientific Linux (pplxint8/9) Ubuntu Linux (ubuntu-trusty-ts) MaxOSX (osxts via vnc)


Physics Remote desktops


StorageWindows server

Central Linux file-server

PP file-serverStorage system:

Client Windows Central Ubuntu

PP Linux

Recommended storage

H:\ drive /home folder /home and /data folders

Windows storage

“H:\” drive or “Y:\home”

/physics/home

PP Storage Y:/LinuxUsers/pplinux/data/home

/data/home, /data/experiment

Central Linux Y:/LinuxUsers/home/particle

/network/home/particle

Physics storage .

Windows: map your H:\ drive by typing net use H:

https://winfe.physics.ox.ac.uk/home/yourname /USER:yourname- Where yourname is your physics user name

OSX: http://www2.physics.ox.ac.uk/it-services/connecting-to-physics-file-servers-from-os-x

Linux: http://www2.physics.ox.ac.uk/it-services/access-windows-shares-from-linux


To store files on servers using your laptop

RDP and storage demoH:\ drive on windows

Connecting to Linux on pplxint8 from windows/home and /data on Linux


Unix Team (Room 661): Sean Brisbane – Local Server and User Support Kashif Mohammad – Grid and Local Support Ewan MacMahon – Grid Systems Administrator Pete Gronbech - Senior Systems Manager and GridPP Project Manager

General purpose interactive Linux based systems for code development, short tests and access to Linux based office applications. These are accessed remotely.

Batch queues are provided for longer and intensive jobs. Provisioned to meet peak demand and give a fast turnaround for final analysis.

Our Local Systems run Scientific Linux (SL) which is a free Red Hat Enterprise based distribution. The same as the Grid and CERN

We will be able to offer you the most help running your code on SL6.


Particle Physics Linux

Particle Physics Local Batch cluster

Oxford’s Tier 2 Grid cluster


Current Clusters


PP Linux Batch Farm Scientific Linux 6

pplxint8pplxint9

Interactive login nodes

pplxdatatrans Grid data transfer nodes

pplxwn31

pplxwn32

pplxwn41

pplxwnnn

pplxwn67 16 * Intel cores

16 * Intel 2650 cores




Users log in to the interactive nodespplxint8 & 9, the home directories and all the data disks (/home area or /data/group ) are shared across the cluster and visible on the interactive machines and all the batch system worker nodes.

Approximately 600 cores (430 incl. JAI/LWFA), each with 4GB of RAM memory.

The /home area is where you should keep your important text files such as source code, papers and thesis

The /data/ area is where you should put your big reproducible input and output data

pplxwn68

jailxwn01 64 * AMD cores

16 * Intel cores

jailxwn02 64 * AMD cores

14th October 2014Graduate Lectures13

PP Linux Batch Farm Data Storage

pplxfsn

40TB

pplxfsn

40TB

Data Areas

pplxfsn

19TB

NFS and lustre storage Servers

Home areas

Data Areas

The /data areas are big and fast disks. This is too big to be backed-up but still have some redundancy features and are safer than laptop storage. This does not help if you delete files.

The /home areas are backed up by two different systems nightly.

The latest nightly backup of any lost or deleted files from your home directory is available at the read-only location /data/homebackup/{username}

If you need older files, tell usIf you need more space on /home, tell us

Store your thesis on /home NOT /data.

pplxfsn

30TBData Areas


Two Computer Rooms provide excellent

infrastructure for the future

The New Computer room built at Begbroke Science Park jointly for the Oxford Super Computer and the Physics department, provides space for 55 (11KW) computer racks. 22 of which will be for Physics. Up to a third of these can be used for the Tier 2 centre. This £1.5M project was funded by SRIF and a contribution of ~£200K from Oxford Physics.

The room was ready in December 2007. Oxford Tier 2 Grid cluster was moved there during spring 2008. All new Physics High Performance Clusters will be installed here.


Local Oxford DWB Physics Infrastructure Computer

RoomCompletely separate from the Begbroke Science park a computer room with 100KW cooling and >200KW power has been built. ~£150K Oxford Physics money.

Local Physics department Infrastructure computer room.

Completed September 2007.

This allowed local computer rooms to be refurbished as offices again and racks that were in unsuitable locations to be re housed.

Use a strong password not open to dictionary attack! fred123 – No good Uaspnotda!09 – Much better

More convenient to use ssh with a passphrased key stored on your desktop. Once set up


Strong Passwords etc

Demo1. Plain ssh terminal connection

1. From ‘outside of physics’ 2. From Office (no password)

2. ssh with X windows tunnelled to passive exceed. Single apps.

3. Password-less access from ‘outside physics’

http://www2.physics.ox.ac.uk/it-services/ppunix/ppunix-cluster

http://www.howtoforge.com/ssh_key_based_logins_putty


Connecting with PuTTY to Linux

Oxford Advanced Research Computing A shared cluster of CPU nodes, “just” like the local

cluster here GPU nodes

Faster for ‘fitting’, toy studies and MC generation *IFF* code is written in a way that supports them

Moderate disk space allowance per experiment (<5TB) http://www.arc.ox.ac.uk/content/getting-started

The Grid Massive globally connected computer farm For big computing projects Atlas, LHCb, t2k and SNO please stay at the end!

Come talk to us in RM 661 about these


Other resources (for free)

Now more details of use of the clusters Help Pages

http://www.physics.ox.ac.uk/it/unix/default.htm

http://www2.physics.ox.ac.uk/research/particle-physics/particle-physics-computer-support

ARC http://www.arc.ox.ac.uk/content/getting-started

Email [email protected]

GRID talk at the end


The end of the overview

Atlas, Sno, LHCb and t2k please read this

GRID certificates


SouthGrid Member Institutions

Oxford RAL PPD Cambridge Birmingham Bristol Sussex

JET at Culham


Compute Servers Twin and twin squared nodes

1770 CPU cores

Storage Total of ~1300TB The servers have between 12 and 36 disks,

the more recent ones are 4TB capacity each. These use hardware RAID and UPS to provide resilience.


Current capacity


Get a Grid CertificateMust remember to use the same PC to request and retrieve the Grid Certificate.

The new UKCA page

http://www.ngs.ac.uk/ukca

uses a JAVA based CERT WIZARD

You will then need to contact central Oxford IT. They will need to see you, with your university card, to approve your request:

To: [email protected]

Dear Stuart Robeson and Jackie Hewitt,

I Please let me know a good time to come over to Banbury road IT office for you to approve my grid certificate request.

Thanks.

Log in to pplxint9 and runmkdir .globus

chmod 700 .globus

cd .globus

openssl pkcs12 -in ../mycert.p12 -clcerts -nokeys -out usercert.pem

openssl pkcs12 -in ../mycert.p12 -nocerts -out userkey.pem

chmod 400 userkey.pem

chmod 444 usercert.pem

14th October 2014

When you have your grid certificate…

Save to a filename in your home directory on the Linux systems, eg:

Y:\Linuxusers\particle\home\{username}\mycert.p12

This is the Virtual Organisation such as “Atlas”, so: You are allowed to submit jobs using the

infrastructure of the experiment Access data for the experiment

Speak to your colleagues on the experiment about this. It is a different process for every experiment!


Now Join a VO

Your grid certificate identifies you to the grid as an individual user, but it's not enough on its own to allow you to run jobs; you also need to join a Virtual Organisation (VO).

These are essentially just user groups, typically one per experiment, and individual grid sites can choose to support (or not) work by users of a particular VO.

Most sites support the four LHC VOs, fewer support the smaller experiments.

The sign-up procedures vary from VO to VO, UK ones typically require a manual approval step, LHC ones require an active CERN account.

For anyone that's interested in using the grid, but is not working on an experiment with an existing VO, we have a local VO we can use to get you started.


Joining a VO

Test your grid certificate:> voms-proxy-init –voms lhcb.cern.chEnter GRID pass phrase:Your identity: /C=UK/O=eScience/OU=Oxford/L=OeSC/CN=j bloggs

Creating temporary proxy ..................................... Done

Consult the documentation provided by your experiment for ‘their’ way to submit and manage grid jobs


When that’s done


BACKUP


Puttygen to create an ssh key on Windows (previous slide point #4)

Paste this into ~/.ssh/authorized_keys on pplxint

Enter a secure passphrase then : - Enter a strong passphrase - Save the private parts of the key to a subdirectory of your local drive.

Run Pageant once after login Right-click on the pageant symbol and and

“Add key” for your Private (windows ssh key)


Pageant

oxford university particle physics unix overview sean brisbane particle physics systems...

Documents

remote linux desktop

graduate lectures1 slide

graduate lectures9 slide

graduate lectures2 slide

strategy slide

osx linux

remote desktop connection

physics user