Getting Started on Topsail

Getting Started on Topsail

Mark ReedCharles Davis

ITS Research Computing

2

Compute Clusters Logging In File Spaces User Environment and

Applications, Compiling Job Management

OutlineOutline

3

LogisticsLogistics

Course Format Lab Exercises Breaks UNC Research Computing

• http://its.unc.edu/research Getting started Topsail page

• http://help.unc.edu/6214

What is a compute cluster?What is Topsail?

What is a compute cluster?What is Topsail?

5

What is a compute cluster?

What is a compute cluster?

Some Typical Components Compute Nodes Interconnect Shared File System Software Operating System (OS) Job Scheduler/Manager Mass Storage

6

Compute Cluster Advantages

Compute Cluster Advantages

fast interconnect, tightly coupled aggregated compute resources large (scratch) file spaces installed software base scheduling and job management high availability data backup

7

Initial Topsail ClusterInitial Topsail Cluster

Initially: 1040 CPU Dell Linux Cluster• 520 dual socket, single core nodes

Infiniband interconnect Intended for capability research Housed in ITS Franklin machine room Fast and efficient for large

computational jobs

8

Topsail Upgrade 1Topsail Upgrade 1

Topsail upgraded to 4,160 CPU• replaced blades with dual socket, quad core

Intel Xeon 5345 (Clovertown) Processors• Quad-Core with 8 CPU/node

Increased number of processors, but decreased individual processor speed (was 3.6 GHz, now 2.33)

Decreased energy usage and necessary resources for cooling system

Summary: slower clock speed, better memory bandwidth, less heat, quadrupled the core count• Benchmarks tend to run at the same speed per core• Topsail shows a net ~4X improvement• Of course, this number is VERY application dependent

9

Topsail – Upgraded blades

Topsail – Upgraded blades

52 Chassis: Basis of node names• Each holds 10 blades -> 520 blades total• Nodes = cmp-chassis#-blade#

Old Compute Blades: Dell PowerEdge 1855• 2 Single core Intel Xeon EMT64T 3.6 GHZ procs• 800 Mhz FSB• 2MB L2 Cache per socket• Intel NetBurst MicroArchitecture

New Compute Blades: Dell PowerEdge 1955• 2 Quad core Intel 2.33 GHz procs• 1333 Mhz FSB• 4MB L2 Cache per socket• Intel Core 2 MicroArchitecture

10

Topsail Upgrade 2Topsail Upgrade 2

Most recent Topsail upgrade (Feb/Mar ‘09)

Refreshed much of the infrastructure Improved IBRIX filesystem Replaced and improved Infiniband

cabling Moved cluster to ITS-Manning building

• Better cooling and UPS

11

Top 500 HistoryTop 500 History

Top 500 lists comes out twice a year• ISC conference in June• SC conference in Nov

Topsail debuted at 74 in June 2006 Peaked at 25 in June 2007 Still in the Top 500

12

Current Topsail Architecture

Current Topsail Architecture

Login node: 8 CPU @ 2.3 GHz Intel EM64T, 12 GB memory

Compute nodes: 4,160 CPU @ 2.3 GHz Intel EM64T, 12 GB memory

Shared disk: 39TB IBRIX Parallel File System

Interconnect: Infiniband 4x SDR 64bit Linux Operating System

13

Multi-Core ComputingMulti-Core Computing

Processor Structure on Topsail• 500+ nodes• 2 sockets/node• 1 processor/socket• 4 cores/processor

(Quad-core)• 8 cores/node

http://www.tomshardware.com/2006/12/06/quad-core-xeon-clovertown-rolls-into-dp-servers/page3.html

14

Multi-Core ComputingMulti-Core Computing

The trend in High Performance Computing is towards multi-core or many core computing.

More cores at slower clock speeds for less heat

Now, dual and quad core processors are becoming common.

Soon 64+ core processors will be common• And these may be heterogeneous!

15

The Heat ProblemThe Heat Problem

Taken From: Jack Dongarra, UT

16

More ParallelismMore Parallelism

Taken From: Jack Dongarra, UT

17

Infiniband Connections

Infiniband Connections

Connection comes in single (SDR), double (DDR), and quad data rates (QDR). • Topsail is SDR.

Single data rate is 2.5 Gbit/s in each direction per link.

Links can be aggregated - 1x, 4x, 12x. • Topsail is 4x.

Links use 8B/10B encoding —10 bits carry 8 bits of data — useful data transmission rate is four-fifths the raw rate. Thus single, double, and quad data rates carry 2, 4, or 8 Gbit/s respectively.

Data rate for Topsail is 8 GB/s (4x SDR).

18

Topsail Network Topology

Topsail Network Topology

19

Infiniband Benchmarks

Infiniband Benchmarks

Point-to-point (PTP) intranode communication on Topsail for various MPI send types

Peak bandwidth:• 1288 MB/s

Minimum Latency (1-way):• 3.6 ms

20

Infiniband Benchmarks

Infiniband Benchmarks

Scaled aggregate bandwidth for MPI Broadcast on Topsail

Note good scaling throughout the tested range (from 24-1536 cores)

21

Login to TopsailLogin to Topsail

Use ssh to connect:• ssh Topsail.unc.edu

SSH Secure Shell with Windows For using interactive programs with

X-Windows Display:• ssh –X Topsail.unc.edu• ssh –Y Topsail.unc.edu

Off-campus users (i.e. domains outside of unc.edu) must use VPN connection

File SpacesFile Spaces

23

File SpaceFile Space

Home directories• /ifs1/home/<onyen>• anyone over 15 GB is not backed up

Scratch Space• /ifs1/scr/<onyen>• over 39 TB of scratch space• run jobs with large output in this space

Mass Storage• ~/ms

24

Mass StorageMass Storage

“To infinity … and beyond” - Buzz Lightyear

long term archival storage

access via ~/ms

looks like ordinary disk file system – data is actually stored on tape

“limitless” capacity

data is backed up

For storage only, not a work directory (i.e. don’t run jobs from here)

if you have many small files, use tar or zip to create a single file for better performance

Sign up for this service on onyen.unc.edu

User Environment and Applications, Compiling

Code

User Environment and Applications, Compiling

Code Modules

26

ModulesModules

The user environment is managed by modules

Modules modify the user environment by modifying and adding environment variables such as PATH or LD_LIBRARY_PATH

Typically you set these once and leave them

Note there are two module settings, one for your current environment and one to take affect on your next login (e.g. batch jobs running on compute nodes)

27

Common Module Commands

Common Module Commands

module avail• module avail

apps

module help

module list module add module rm

Login version

module initlist module initadd module initrm

More on modules see http://help.unc.edu/CCM3_006660

28

Parallel Jobs with MPIParallel Jobs with MPI

There are three implementations of the MPI standard installed:• mvapich• mvapich2• openmpi

Performance is similar for all three, all three run on the IB fabric. Mvapich is the default. Openmpi and mvapich2 have more the the MPI-2 features implemented.

29

Compiling MPI programs

Compiling MPI programs

Use the MPI wrappers to compile your program• mpicc, mpiCC, mpif90, mpif77• the wrappers will find the appropriate

include files and libraries and then invoke the actual compiler

• for example, mpicc will invoke either gcc or icc depending upon which module you have loaded

30

Compiling on TopsailCompiling on Topsail

Serial Programming• Intel Compiler Suite for Fortran77, Fortran90, C

and C++ - Recommended by Research Computing icc, icpc, ifort

• GNU gcc, g++, gfortran

Parallel Programming• MPI (see previous page)• OpenMP

Compiler tag: -openmp for Intel -fopenmp for GNU

Must set OMP_NUM_THREADS in submission script

Job Scheduling and ManagementJob Scheduling and Management

32

What does a Job Scheduler and batch system do?

What does a Job Scheduler and batch system do?

Manage Resources allocate user tasks to resource monitor tasks process control manage input and output report status, availability, etc enforce usage policies

33

Job Scheduling Systems

Job Scheduling Systems

Allocates compute nodes to job submissions based on user priority, requested resources, execution time, etc.

Many types of schedulers• Load Sharing Facility (LSF) – Used by

Topsail• IBM LoadLeveler• Portable Batch System (PBS)• Sun Grid Engine (SGE)

34

LSFLSF

All Research Computing clusters use LSF to do job scheduling and management

LSF (Load Sharing Facility) is a (licensed) product from Platform Computing• Fairly distribute compute nodes among users• enforce usage policies for established queues

most common queues: int, now, week, month• RC uses Fair Share scheduling, not first come,

first served (FCFS) LSF commands typically start with the letter

b (as in batch), e.g. bsub, bqueues, bjobs, bhosts, …• see man pages for much more info!

35

Simplified view of LSFSimplified view of LSF

bsub –n 64 –a mvapich –q week mpirun myjob

Login Node

Jobs Queued

job routed to queue

job_Jjob_Fmyjobjob_7

job dispatched to run on available host which satisfies job requirements

user logged in to login node submits job

36

Running Programs on Topsail

Running Programs on Topsail

Upon ssh to Topsail, you are on the Login node.

Programs SHOULD NOT be run on Login node.

Submit programs to one of 4,160 Compute nodes.

Submit jobs using Load Sharing Facility (LSF) via the bsub command.

37

Common batch commands

Common batch commands

bsub - submit jobs bqueues – view info on defined queues

• bqueues –l week bkill – stop/cancel submitted job bjobs – view submitted jobs

• bjobs –u all bhist – job history

• bhist –l <jobID>

38

Common batch commands

Common batch commands

bhosts – status and resources of hosts (nodes)

bpeek – display output of running job Use man pages to get much more info!

• man bjobs

39

Submitting Jobs: bsub Command

Submitting Jobs: bsub Command

Submit Jobs - bsub• Run large jobs out of scratch space, smaller jobs

can run out of your home space

bsub [-bsub_opts] executable [-exec_opts] Common bsub options:

• –o <filename> –o out.%J

• -q <queue name> -q week

• -R “resource specification” -R “span[ptile=8]”

• -n <number of processes> used for parallel, MPI jobs

• -a <application specific esub> -a mvapich(used on MPI jobs)

40

Two methods to submit jobs:

Two methods to submit jobs:

bsub example: submit the executable job, myexe, to the week queue and redirect output to the file out.<jobID> (default is to mail output)

Method 1: Command Line• bsub –q week –o out.%J myexe

Method 2: Create a file (details to follow) called, for example, myexe.bsub, and then submit that file. Note the redirect symbol, <• bsub < myexe.bsub

41

Method 2 cont.Method 2 cont.

The file you submitted will contain all the bsub options you want in it, so for this example myexe.bsub will look like this#BSUB –q week

#BSUB –o out.%J

myexe

This is actually a shell script so the top line could be the normal #!/bin/csh, etc and you can run any commands you would like.• if this doesn’t mean anything to you then

nevermind :)

42

Parallel Job exampleParallel Job example

Batch Command Line Method bsub –q week –o out.%J -n 64 -a mvapich

mpirun myParallelExe

Batch File Method bsub < myexe.bsub where myexe.bsub will look like this

#BSUB –q week

#BSUB –o out.%J

#BSUB –a mvapich

#BSUB –n 64

mpirun myexe

43

Topsail QueuesTopsail Queues

Queue Time Limit

Jobs/User CPU/Job

int 2 hrs 128 ---

debug 2 hrs 64 ---

day 24 hrs 512 4 – 128

week 1 week 512 4 – 128

512cpu 4 days 512 32 – 512

128cpu 4 days 512 32 – 128

32cpu 2 days 512 4 – 32

chunk 4 days 512 Batch Jobs• For access to the 512cpu queue the scalabitly should be

demonstrated

44

Common Error 1Common Error 1

If job immediately dies, check err.%J file err.%J file has error:

• Can't read MPIRUN_HOST

Problem: MPI enivronment settings were not correctly applied on compute node

Solution: Include mpirun in bsub command

45

Common Error 2Common Error 2

Job immediately dies after submission err.%J file is blank Problem: ssh passwords and keys were

not correctly setup at initial login to Topsail Solution:

• cd ~/.ssh/• mv id_rsa id_rsa-orig• mv id_rsa.pub id_rsa.pub-orig• Logout of Topsail• Login to Topsail and accept all defaults

46

Interactive JobsInteractive Jobs

To run long shell scripts on Topsail, use int queue

bsub –q int –Ip /bin/bash• This bsub command provides a prompt

on compute node• Can run program or shell script

interactively from compute node

Totalview debugger can also be run interactively from Topsail

47

MPI/OpenMP TrainingMPI/OpenMP Training

Courses are taught throughout year by Research Computing• http://learnit.unc.edu/workshops• http://help.unc.edu/CCM3_008194

See schedule for next course • MPI• OpenMP

48

Further Help with Topsail

Further Help with Topsail

More details about using Topsail can be found on the Getting Started on Topsail help document• http://help.unc.edu/?id=6214• http://keel.isis.unc.edu/wordpress/ - ON CAMPUS

For assistance with Topsail, please contact the ITS Research Computing group• Email: research@unc.edu• Phone: 919-962-HELP• Submit help ticket at http://help.unc.edu

For immediate assistance, see manual pages on Topsail:• man <command>