introduction to the unity cluster...1. pwd –printcurrentlocation 2. ls...
TRANSCRIPT
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Introduction to the Unity Cluster
Jason Thomas
June 26, 2018
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Introduction
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Motivation: Why use the Unity Cluster
I SpeedI processing speed(?); CPU cache; free up resources on local
machine for long running jobsI Large data setsI Remote accessI Stepping stone for Ohio Super Computer (spark cluster
computing system)
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Tools Needed for Getting up and Running
1. Connect to the Unity Cluster2. Transfer Files3. Commands for navigating the cluster4. Programs for editing and running (Stata/R) code
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Background
I Unity cluster runs linuxI Clusters & NodesI Interactive Sessions & adding jobs to the queue
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Connecting to the Unity Cluster
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
VPN: Virtual Private Network
How do we connect: Step 1
I If connecting from an off-campus location, we need to establisha connection via a VPN
I Software recommended by OSU:I Cisco AnyConnect Secure Mobile Client:I anyconnect-windows-4.5 (.exe for Windows; .dmg for Mac)I https://osuasc.teamdynamix.com/TDClient/KB/ArticleDet?
ID=14542I Ready to connect. vpn.asc.ohio-state.edu -> ConnectI Username & Password: same as emailI Second Password: Duo Mobile
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
X Windows System
How do we connect: Step 2
Enable the Unity Cluster to open new windows on your computer
I X Windows SystemI XWin32 & FastX are alaible for OSU faculty and students
I Xming what we will use today (available from Sourceforge)I popular X Win server for WindowsI free and in the public domain (under MIT, BSD, and other
licenses)
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
SSH: Secure Socket Shell
How do we connect: Step 3
I The Unity cluster uses the linux operating systemI interaction through a terminal (typing commands at a prompt)
I Software for connecting to linux servers: PuTTYwww.chiark.greenend.org.uk/~sgtatham/putty/latest.htmlI (download, install, and run from the start menu: PuTTY
Desktop App)I Configuration:
I Host Name: unity.asc.ohio-state.eduI Saved Sessions: Unity (or whatever you want to name it) ->
SAVEI Connection -> SSH -> X11 -> Enable X11 forwarding (Session
-> SAVE!)
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Transferring Files
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
SFTP: Secure File Transfer Protocol
How do we upload (download) files to (from) the UnityCluster?
I PuTTY also has SFTP Client (secure file transfer protocol)I not as friendly, so we’ll use a different tool. . .
I FileZilla has a graphical user interface (GUI)I (Remember that we need to connect through a VPN)I Configuration:
I Host: : unity.asc.ohio-state.eduI Username & Password: same as OSU emailI Port: 22
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Commands for Navigating the Cluster
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
7(ish) Commands for the linux terminal
Inside the PuTTY window. . .
1. pwd – print current location2. ls – print contents of current folder3. mkdir NAME – make a new folder called NAME in the current
location4. cd PATH – change to new location PATH5. rm Filename – remove the file Filename
I rmdir – remove empty folder6. mv Filename newPATH – move the file Filename to the
location newPATH7. exit – does just that
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Tips for the linux terminal
Inside the PuTTY window. . .
I cd – without a path will take you to your home directoryI TAB – complete file/folder namesI TAB TAB – offer suggestions for possible file/folder namesI cp -r FOLDER newPATH – copy a folder (and its contents)
to newPATHI ls -la – prints more information about folder contents (and
lists files with names starting with a period, e.g., .bashrc)I You can set up aliases in the file .bashrc
I alias data = ’cd /Research/Data/; ls -l’I Up Arrow – cycles through previous commands
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Programs for Editing and Running Code
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Editing Files
Inside the PuTTY window. . .
I gedit – basic text editorI emacs – powerful text editor with a steep learning curve
I great for writing code (ESS – Emacs Speaks Statistics)I other options: vi and nano
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Running Stata
Inside the PuTTY window. . .
List available programs> module avail
I spacebar – page down; q – quit
Start interactive session for 2 hours> qsub -I -l walltime=2:00:00
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Running Stata (cont.)
starts an interactive session (for default 1 hour) using 1 node, 4processors, and 16Gigs of RAM> qsub -I -l nodes=1:ppn=4,mem=16GB}
Once the session starts (it may take a while depending on traffic)> module load stata
> stata
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Running R
First we need to load the intel module> module load intel
List available versions of R then load and start R> module avail R/
> module load R/3.5.0
> R
> 3 + 5> q('no')
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Running R BATCH Job
I Batch files can be submitted using the Portable Batch System(PBS)
I Be sure to avoid DOS/Windows line endingsI lines must end with a linefeed (LF)I tabs must be 4 spaces
I See example PBS script: run_R_job.pbsI Submit the R BATCH job with the following command
> qsub -d "$PWD" run_R_jobs.pbs
the -d "PWD" option starts R in the current directory as opposed tothe home directory (the default)
Jason ThomasIntroduction to the Unity Cluster
Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code
Useful Commands for the qsub system
I qstat -u username.number – querry your current jobsI showq – print the current queueI qdel jobID – delete job with ID jobID
I qstat will list the job IDI for more see OSC’s page
Jason ThomasIntroduction to the Unity Cluster