using eclipse parallel tools platform with bluefern’s...

Using Eclipse Parallel Tools Platform with BlueFern’s HPCs

The BlueFern Team

Module 1: Introduction

• Objective

• To introduce the Eclipse platform and PTP

• Contents

• What is Eclipse?

• What is PTP?

What is Eclipse?

• A vendor-neutral open-source workbench for multi-language development

• An extensible platform for tool integration

• Plug-in based framework to create, integrate and utilise software tools

Eclipse Features

• Full development lifecycle support

• Revision control integration (CVS, SVN, Git)

• Project dependency management

• Incremental building

• Content assistance

• Help

• Language sensitive searching

• Multi-language support

• Debugging

Parallel Tools Platform (PTP)

• The Parallel Tools Platform aims to provide a highly integrated environment specifically designed for parallel application development

• Features include:

• An integrated development environment (IDE) that supports a wide range of parallel architectures and runtime systems

• A scalable parallel debugger

• Parallel programming tools (MPI, OpenMP, UPC, etc…)

• Support for the integration of parallel tools

• An environment that simplifies the end-user interaction with parallel systems

• http://www.eclipse.org/ptp

http://www.eclipse.org/ptp

Module 2: Installation

Objective

To learn how to install Eclipse and PTP

Contents

System Prerequisites

Eclipse Download and Installation of “Eclipse IDE for Parallel Application Developers”

Installation Confirmation

Updating the PTP within your Eclipse to the latest release

Before you continue, insert the disc we prepared for this workshop into your computer’s DVD

System Prerequisites

Local system (running Eclipse)

Linux (just about any distribution)

Mac OS X (10.5 Leopard or 10.6 Snow Leopard)

Windows (XP or above)

Java: Eclipse requires Sun or IBM Java

Only needs Java runtime environment (JRE)

Java 1.5 or higher

Java 1.5 is the same as JRE 5.0

Open JDK, distributed with some Linux distributions, should work successfully (it works for me)

The GNU Java Compiler (GJC), which comes standard on Linux, will NOT work

Eclipse Packages

The version of Eclipse for this workshop is 3.7, also known as “Indigo”

Eclipse is available in a number of different packages for different kinds of development

http://eclipse.org/downloads

With Indigo, there is a package directly relevant for HPC:

Eclipse IDE for Parallel Application Developers

This is recommended for all new installs

The latest packages, 3.7 SR2, are included on the disc you received earlier. They are within the eclipse_packages directory on the disc

PTP can also be added to an existing Eclipse installation



Eclipse Installation

• Download the “Eclipse IDE for Parallel Application Developers” package

• For this workshop, copy the file from the disc you received earlier (<DVD root>/eclipse_packages) and paste it in your home directory

• Make sure you match the architecture with the computer you are using

• For this workshop, please use “eclipse-parallel-indigo-SR2-incubation-macosx-cocoa-x86_64.tar.gz

• If your computer is running Linux or Mac OS X, untar the file

• On Mac OS X, you can just double-click in the Finder

• If your computer is running Windows, unzip the file

• Untarring or unzipping the package file creates an eclipse folder containing the executable as well as other support files and folders

Starting Eclipse

• Linux

• From a terminal window, enter:

“<eclipse_installation_path>/eclipse/eclipse&”

• Mac OS X

• From finder, open the eclipse folder where you installed it

• Double-click on the Eclipse application

• Windows

• Open the eclipse folder

• Double-click on the eclipse executable

Specifying A Workspace

• Eclipse prompts for a workspace location at the start up time

• The workspace contains all user-defined data

• Projects and resources such as folders and files

• Enter /Users/<logon id>/workspace for this workshop

Note: If you don’t want to see this message every time when Eclipse is launched, tick this check box

Eclipse Welcome Page

• The welcome page is displayed when Eclipse is run for this first time

• Select “Go to the workbench” to continue this tutorial

Check Installation Details

• To confirm you have installed Eclipse PTP successfully

• Mac: Eclipse>About Eclipse

• Others: Help>About

• Choose Installation Details

• Confirm you have “Eclipse IDE for Parallel Application Developers” in the installed software list

• Close the dialog: Close, OK

Checking for PTP Updates

• From time to time, there may be newer PTP releases than the Eclipse release

• PTP maintains its own update site with the most recent release

• Bug fix release can be more frequent than Eclipse’s and what is within the package

• You must enable the PTP-specific update site, so PTP updates can be found

Updating PTP (1)

• Enable PTP-specific update site

• Help>Install New Software…

• Click Available Software Sites link

• Ensure the checkbox is selected for the PTP site:

http://download.eclipse.org/tools/ptp/updates/indigo

• Choose OK

• Choose Cancel (to return to Eclipse workbench)

• Now select Help>Check for updates

• If you see “No updates were found”…

• It is only because there are no updates in the “Eclipse IDE for Parallel Application Developers”

• We will update the PTP within it

http://download.eclipse.org/tools/ptp/updates/indigo

Updating PTP (2)

• We will get the latest PTP release (5.0.7), which is more recent than the version that is currently included (5.0.5) in the parallel package

• Update PTP via an archived package

• Download the package, ptp-master-5.0.7-201203231413.zip, from http://www.eclipse.org/ptp/downloads.php (included in the disc that you received earlier)

• Select Help>Install New Software…

• Select Add…

• Give this repository a sensible

name, such as PTP 5.0.7 update

• Select Archive…

• Navigate to and select the archived

package

• Click the OK button

http://www.eclipse.org/ptp/downloads.php

Updating PTP (3)

• Alternatively, download the update from the Internet directly (skip this for this tutorial):

• Select Help>Install New Software…

• In the Work with: dropdown box, select the PTP update site that you confirmed already:

Updating PTP(4)

• Quick and dirty: (this is what we will do for this tutorial)

• Check everything – which updates existing features and adds a few more

• Detailed:

• Open each feature and check ones you want to update

• Icons indicator: Grey plug: already installed

• Double arrow: can be updated

• Colour plug: not installed yet

Updating PTP (5)

• When updating via a local archive,

unselect Contact all update sites during

install to find required software

• Select Next to continue updating PTP

• Select Next to confirm features to install

• Note: if you are NOT on Linux: you may have to go back and unselect, under Fortran Development Tools (Photran), the “Linux Intel Fortran Compiler Support” – only if it complains

• Accept the License agreement and select Finish

• Wait for installation to finish

• Select Restart Now when prompted

• We are Ready to go now!!!

Module 3: Developing with Eclipse

• Objectives:

• Learn basic Eclipse concepts: Perspectives, views, etc…

• Learn how to create and manage a C project

• Learn how to build and run a project

• Learn about navigating, searching, refactoring, etc…

• Contents:

• Basic Eclipse features

• Managing projects in Eclipse

• Creating a C project

• Build and run a C project

• Basic editor features

• Content Assist & Templates

• Navigating, Searching and Refactoring

Basic Eclipse Features

Eclipse Basics

• A workbench contains the menus, toolbars, editors and views that make up the main Eclipse window

• The workbench represents the desktop development environment

• Contains a set of tools for resource management

• Provides a common way of navigating through the resources

• Multiple workbenches can be opened at the same time

• Only one workbench can be open on a workspace at a time

view

view

view editor

Perspectives

• Perspectives define the layout of views and editors in the workbench

• They are task oriented, i.e. they contain specific views for doing certain tasks. For example:

• Parallel Runtime Perspective manages resource managers that are used for managing local or remote parallel runtime environments

• C/C++ Perspective for manipulating C/C++ source code

• Debug Perspective for debugging applications

• If you are still on the Welcome screen, select “Go to Workbench” now

Switching Perspectives

• Three ways of changing perspectives:

1. Choose the Window>Open Perspective menu option then select the target perspective or choose Other…

2. Click on the Open Perspective button in the upper right corner of the screen (hover over it to see names)

3. Click on a perspective shortcut button

• Try to switch to the C/C++ Perspective now.

Which Perspective?

• Which perspective am I in?

It is indicated on the Title Bar

Views

• The workbench window is divided up into Views

• The main purposes of a view are:

• To provide alternative ways of presenting information

• For navigation

• For editing and modifying information

• Views can have their own menus and toolbars

• Items available in menus and toolbars are available only in that view

• Menu actions only apply to the view

• Views can be resized

Stacked Views

• Stacked views appear as tabs

• Selecting a tab brings that view to the foreground

Expand a View

• Double-click on a view/editor’s tab to fill the workbench with its content

• Repeat to return to original size

Double Click

Double Click

Help

• To access help:

• Help>Help Contents

• Help>Search

• Help>Dynamic Help

• Help Contents provides detailed help on different Eclipse features in a browser

• Search allows you to search for help locally, or using Google or the Eclipse web site

• Dynamic Help shows help related to the current context (perspective, view, etc.)

Eclipse Preferences (1)

• Eclipse Preferences allow customization of almost everything

• To open use

• Mac: Eclipse>Preferences…

• Others: Window>Preferences…

• The C/C++ preferences allow many options to be altered

• In this example you can adjust what happens in the editor as you type.

Eclipse Preference (2)

• In this example, the Code Style preferences are shown

• Changing preferences here will allow code to be automatically formatted in different ways

Managing Projects in Eclipse

Project Types

• There are three types of project in Eclipse:

• Local

• Source is located on local machine, builds happen locally

• Synchronized

• Source is local, then synchronized with remote machine(s)

• Building and launching happens remotely and can also happen locally

• Remote

• Source is located on remote machine(s), build and launch takes place on remote machine(s)

• We will concentrate on the Local project for the rest of Module 3 and revisit Synchronized and Remote in later modules.

Project Build Types

• A project’s build type can either be:

• Makefile-based: Project contains its own makefile (or makefiles) for building the application

• Managed: Eclipse manages the build process, no makefile is required

• Time to start our Hello World project !!!

Create a C Project

Creating a Project (1)

• Select File>New>Project…

• Select C Project then press Next>

• You can combine the these two

steps by selecting the shortcut

File>New>C Project


• Select Hello World ANSI C Project under the Executable project type

• Select a Toolchain based

on what’s available on your local computer. (select MacOSX GCC for this workshop because we are using Macs)

• Give the project a name, such as HelloWorld

• Press Next> if you wish to configure basic project settings; otherwise press Finish (press Next> for this workshop)


• Update basic settings as you wish

• Press Next> when you are ready to update the project’s configurations

• Or press Finish if you wish to skip the project configuration

• (Press Next> for this workshop)


• Tick on Configurations that you wish to add to this project. (Tick both Debug and Release for this workshop)

• Select Finish to complete the creation of a project

• Selecting Advanced settings… will allow you to configure more project settings. We will skip this for this workshop.


A Hello World Project in a Workbench

Build and Run a C Project

Build a Project

• To build the current project using the current active build configuration, click on icon

• You can build using a different build configuration by clicking the down-arrow on the icon and selecting the configuration you want to use

(The current active build configuration is ticked)

• The build output is displayed in the Console tab view

Run a Project

• Click icon to run the current project

• If the project has not been built or the current build is not up to date, it will be automatically built using the current active build configuration

• When running the project, all data printed to the standard output will be displayed in the Console tab view (Except if you are using the Cygwin toolchain under a Windows OS, in such a case, you will have to run the produced executable manually within a Cygwin Terminal)

Basic Editor Features

Editor

• An editor for a resource (e.g. a file) opens when you double-click on a resource

• The type of editor depends on the

type of the resource

• .c files are opened with the

C/C++ editor by default

• You can use Open With to use

another editor

• In this case the default editor is fine (double-click)

• Some editors can edit more than just raw text

• When an editor opens on a resource, it stays open across different perspectives

• An active editor contains menus and toolbars specific to that editor

Saving File(s) in an Editor

• When you change a file in the editor, an asterisk (*) on the editor’s title bar indicates unsaved changes

• Save the changes by using

shortcut Command/Ctrl+S

or File>Save or icon on

the tool bar

• Save multiple files by using

shortcut Command/Ctrl+Shift+S

or File>Save All or icon on the tool bar

Editor and Outline View

• Double click on the source file, its default editor will open on the main view

• Outline view is shown for the file in the editor

• Console tab view shows the result of a local build, run etc…

Source Code Editor and Markers (1)

• Modify HelloWorld.c so the

content is the same as shown in

the screenshot

• Save these modifications

(hint: Command/Ctrl+S)

• Click icon on the tool bar to

build the project or

select Project>Build Project

(Yes, there is an error in these modifications and it is intentional)


• A source code editor is a special

type of editor for manipulating

source code

• Language features are

highlighted

• Marker bar for showing

• Breakpoints

• Errors/warnings

• Task Tags, Bookmarks

• Location bar for navigating to interesting

features in the file Icons:


• Resolve the error by adding a semicolon (;) to the end of puts function

• Rebuild the project

• All error and warning markers should be disappear

• When you see bug icons in the editor marker

bar, they are likely suggestions from Codan

• Code checkers can flag possible errors,

even if code is technically correct

• To turn them off, use Preferences

Window>Preferences or

Mac: Eclipse>Preferences

C/C++ > Code Analysis and

uncheck all problems

• Select Ok to close preferences


• Text editors can show line

numbers in the left column

• To turn on line numbering:

• Right-mouse click in the

editor marker bar

• Click on

Show Line Numbers

Adding Existing Files to a Project (1)

• Right click or (Mac: Ctrl+click)

on the src folder in

the Project Explorer and then

select Import…

Or

• Select the src folder in

the Project Explorer and then

select File>Import…

Adding Existing Files to a Project (2)

• Type “file system” in the

Select an import source: text field

• Select , then

click Next> button

• Click Browse… button and navigate to

<DVD root>/tutorial/Module_03

• Tick both Func.c and Func.h

• Click Finish button

Content Assist & Template

Content Assist & Template in Source Editor (1)

• Use Content Assist to include Func.h in HelloWorld.c:

• In the source editor, type

#include “

• Press Ctrl+Space to enable Content Assist

• Available headers should be displayed in a list

• Type Fu, suggestion list shows only “Func.h”

• Press Enter to complete the include statement


• Use Content Assist to call Func(int a) in HelloWorld.c’s main function:

• Press Ctrl+Space to enable Content Assist

• A long suggestion list is shown

• Keep typing Fun, and you will see the suggestion list gets shorter

• Use arrow keys to select the function you wish to call. In this case, select Func(int a):int

• Press Enter to accept the selected function


• Content Assist also helps you to peek at the signature of a function

• As you select Func(int a):int, its signature will be automatically displayed

• If you didn’t use Content Assist to write a function, you can still use it to check its signature by pressing Ctrl+Space after typing the function name


• Use Content Assist &

Template to write a

For loop:

• Type: for then press

Ctrl+Space

• Select for – for loop then press Enter

• A for loop template will be written in the source code

• Enter i to replace the var in the template

and then press Tab

• Enter 10 to replace the max in the template and press Tab to complete the template


• Use Content Assist & Template to modify your HelloWorld.c to match what is shown in the screenshot below:

Navigating, Searching and Refactoring

Source Code Navigation (1)

• Eclipse’s source code editor provides some handy features to navigate through your code

• On demand hyperlink:

• On line 23 in HelloWorld.c, a function Func that is not implemented within the current file is called

• To open the file where Func is implemented and navigate to it

• Hold the Ctrl/Command key and click on the Func

• On demand hyperlink also works on functions and variables in the current file

• Hold the Ctrl/Command key and click on variable msg in line 23, you will see the cursor is now moved to line 18 where msg is declared

(Note: Click = left mouse click on Windows/Linux)

Source Code Navigation (2)

• Open declaration

• Move the cursor to the middle or after a variable or function name

• Right-Click or (Ctrl+Click) to bring up the context menu then select Open Declaration

• or you can just press the shortcut F3

F3

Simple Find and Replace

• Ctrl+F or (Command+F) to bring up the simple

Find/Replace dialog

• It only works within the current active file in the

editor view

• Try to find all references of Func in HelloWorld.c

Language Specific Searching (1)

• Select Search>C/C++… or Ctrl/Command+H and click the “C/C++Search” tab

• It “knows” what can be

declared in C/C++

• Try to search for all

occurrences of function

Func in the current

Workspace

Language Specific Searching (2)

• The outcome of the search will be displayed in the Search tab view

• Expand the tree to see all occurrences

• Double click a matched occurrence to navigate to its location in a source file

Refactoring (1)

• Eclipse features a Refactor tool which helps you to modify your code and preserve its correctness at the same time:

• It only works on specific types of modifications, such as renaming a function, extracting selected lines of code into a function etc…

• Bring up the Refactor menu by selecting

Refactor in the menu bar or

Right Click or Ctrl+Click on the editor to open

the context menu and then select the Refactor

menu

Refactoring (2)

• Hard coded constant within a

function is bad in practice. Let’s

extract it to a constant

• Select the number 10 in line 22 and then bring up the Refactor menu

• Select Extract Constant…

• Give the new constant a meaningful name

then click the Next> button to preview

the outcome

Refactoring (3)

• Check the code comparisons and verify if the Refactor is doing what you are expecting

• Press the Finish button to confirm the change

Refactoring (4)

• The final code should be like the following:

Module 4: Working with MPI C Code in Eclipse

• Objectives

• Learn how to create a MPI project using C

• Learn about MPI features

• Learn how to build an MPI program using C on a local system

• Contents:

• Configure MPI Settings in Eclipse

• Helpful features for using MPI

• Build and Run a MPI Job Locally

• Build and Run a MPI Job Remotely

Configuring MPI Settings in Eclipse (1)

• You need to configure Eclipse to ‘find’ your local MPI install to work with a MPI based project properly. If it is not configured, you won’t be able to build and run your code locally.

• Select Window>Preference or (Eclipse>Preference on MacOS) to bring up the preference dialog

• Expand Parallel Tools, Parallel Language Development Tools and then select MPI

Note:

1. It is assumed that a version of MPI is already installed in your local environment

2. If you are using Eclipse and the Cygwin toolchain in a MS Windows environment, skip this section. BlueFern is not currently supporting MPI in a MS environment at this stage due to technical challenges

Configuring MPI Settings in Eclipse (2)

• Press the New… button to bring up a file chooser and navigate to the directory where MPI’s header is stored

(For this tutorial or on a Mac, navigate to /usr/include)

• Set MPI build Command(C): to mpicc

• Set MPI build Command (C++): to mpiCC

• Press the OK button to close the preference dialog

Creating a MPI Hello World Project (1)

• Select File>New>C Project

• Enter HelloMPI in the Project name

field

• Select MPI Hello World C Project

as the Project Type

• Select the appropriate toolchain that is

available for the local development

environment

(MaxOSX GCC for this workshop)

• Press the Finish button to create the

project

Creating a MPI Hello World Project (2)

• Double click HelloMPI.c in the project’s src folder to open it in the source editor view

Helpful Features for Using MPI

Context Sensitive Help (1)

• Eclipse provides a context sensitive help, and it works for MPI functions too

• Select Help>Dynamic Help

• Click on the Restore shortcut to prevent it from

minimising the view

• Click on line 28, MPI_Comm_Rank...

• Special info for this function is now

displayed in the help view

• Click on the link to open the function’s

help document Click here

Context Sensitive Help (2)

• Hovering the mouse pointer over a MPI function brings up a short description for the function

Parallel Analysis

• Eclipse PTP can parse code in the current selected files or project to provide parallel analysis

• One analysis identifies all MPI artifacts and displays them in a tab view

• Select the HelloMPI project

• Click on the downward arrow on icon and select

Show MPI Artifacts

• The outcome of the analysis is displayed in the

MPI Artifact View, clicking on the artifact will take you the

line where the artifact resides in the source code

Building and Running a MPI Project Locally

(Note: Skip this section if you are using PC running MS Windows OS

and do not have Open MPI installed)

Build a MPI Project Locally

• If the Eclipse has been configured correctly, you should be able to build the project successfully by clicking the icon on the toolbar

• The build output is displayed in the Console tab view

• If the project does not build successfully, check if the project’s properties are set correctly

Checking Project Properties

• Select Project>Properties

• Expand C/C++ build,

• Select Tool Chain Editor, For this workshop

• the Current toolchain should be MacOSX GCC

• the Current builder should be Gnu Make Builder

• Select Settings, For this workshop

• the Command for the MacOS X C Linker should be mpicc

• the Command for the GCC Assembler should be as

• the Command for the GCC C Compiler should be mpicc

(Note: Project properties are build configuration specific, please make sure you check all of them in all available build configurations)

Resource Manager to Control a Local Open MPI Runtime (1)

• Switch to Parallel Runtime perspective

• Select Windows>Open Perspective>Others...

• Select Parallel Runtime, and click the OK button


• Right Click or (Ctrl+Click on Mac)

within the Resource Managers tab

view to bring up its context menu

and select Add Resource Manager...

• Select Open MPI and then click on

the Next> button


• Select Local, for both

Remote service provider: and

Connection name:

• Click on the Finish button to create

a local Open MPI resource manager

using default settings

(Note: default Open MPI settings are

sufficient for this tutorial)


• Select the newly created

resource manager,

Open_MPI@Local (Open MPI)

• Right Click or

(Ctrl+Click on a Mac)

• Select Start Resource Manager to start the resource manager

• If the resource manager is started successfully, its icon should turn to green

• Switch back to C/C++ perspective by using its

shortcut on the toolbar

Run Configuration for a Local MPI Job (1)

• To create a run configuration using the local Open MPI runtime

• Click on the down arrow next to the Run As... icon on the

toolbar

• Select Run Configurations...

• Select Parallel Application as the

configuration type

• Click on the to add a new configuration


• Give the configuration a

name (use HelloMPI_Local

for this tutorial)

• Select Open_MPI@Local

as the Resource Manager:

• Set Number of processes:

to 4

• Click on the Apply button

to save the configuration


• Click on the Application tab

• Click on the Browse button

next to the

Application Program: text

field

• Navigate to the directory

where the built executable is,

select the executable and

click on the OK button

(Note: for this tutorial, select

HelloMPI in <Eclipse Workspace Dir>/HelloMPI/Debug/)

Running a MPI Job Locally (1)

• After you complete the run

configuration, the Run button

should be enabled

• You can click on the Run

button to launch a run using

the current configuration

• For this tutorial, click the

Close button and we will

launch the run from the

workbench instead

Running a MPI Job Locally (2)

• To launch a run using a specific Run Configuration from the workbench

• Click on the down arrow next to the Run As icon on the toolbar

• Select the Run Configuration you wish to launch the run

(For this tutorial, select HelloMPI_Local)

• The standard output generated by the run

will be displayed in the Console tab view on

the workbench

Building a MPI Project on BlueFern’s HPCs

Converting an Existing Project to a Synchronized Project (1)

• Switch to the C/C++ Perspective if your workbench is in another perspective

• Select File>New>Other...

• Type Convert in the Wizard:’s

text filter

• Select

Convert C/C++ or Fortran Project

to a Synchronised Project

• Click on the Next> button


• Select HelloMPI as a candidate

for conversion

• Select Git as the

Synchronization Provider

• Select Remote Tools as the

Remote Provider:

• Click the New... button to

create a connection to the remote

environment


• Give the new connection a Target name:

• Tick the Remote host radio button

• Enter the IP address or host name of the

remote host in Host:

(For this workshop, enter

hpclogin1.canterbury.ac.nz)

• Enter your user name and password

for logging on to the remote host

(they are supplied to you in the beginning

of this workshop)

• Click on the Finish button to create the

connection


• Enter the location where you wish

to store the current project on

the remote host

(you can select the Browse... Button

to open a file browser viewing the

remote file system)

• Click on the Finish button to convert

the current project to a

synchronised project

• A new build configuration should now

be available to this project

Configuring Remote Building Environment (1)

• If you try to build the project using the Remote build configuration, Eclipse will automatically upload your local files to the remote host and then invoke a build

• However, the build will most likely fail. This is because the build environment on the remote host is different from your local environment

• For this tutorial, you need to update:

• Toolchain,

• Settings for compiling and linking

• Environment variables

• Building tool


• Right click or (Ctrl+Click) on the HelloMPI project in the Project Explorer tab view to open its context menu

• Select Properties in the context menu

• Expand C/C++ Build properties

• Select the Tool Chain Editor


• Project properties is build configuration specific, change the current configuration to Remote, if it is not currently active

• Unselect Display compatible toolchains only checkbox

• Change Current toolchain: to XL C/C++

• Change Current builder: to

Sync Builder

• Click on the Apply button to

persist the change

Note: BlueFern encourage users

to use XL C/C++ compilers when

using BlueFern’s HPCs.


• Select Synchronize properties

• Confirm its properties are using the remote

connection created earlier

• If you wish to update the current project to

synchronise and utilise a different remote

environment, you will need to update these settings

(skip this for this tutorial)

• Select the Settings to setup compiler and linker


• Select Local XL C Compiler, and then enter mpcc in the Command:

• Select Local XL C

Executable Linker,

and then enter mpcc in

the Command:

• There are other options for

configuring the compiler

too, but we will skip them

for this tutorial

• Click on the Apply button

(Note: mpcc is a compiler driver to compile and link MPI C code. Using it saves us from entering paths to MPI headers and libraries)


• Select the Environment properties. This is where you can setup environment variables in the Remote Environment in build time

• They default to your local environment settings and are most likely

not compatible in the

remote environment

• For this workshop,

Undefine all

variables. We will

use SSH session’s

default instead

• Click the Apply

button when done


• Finally select the C/C++ Build properties

• Unselect the check box next to Use default command

• Change the

Build command:

to gmake

• Click on the OK

button

Note: Eclipse generates makefiles in GNU format when a project is set as a Managed project. Therefore you need to use GNU make to build the project, which is gmake under AIX

Build a Project in a Remote Environment

• Click on the down arrow on the build icon on the toolbar and select 3 Remote (Build on remote machine)

• It triggers Eclipse to synchronize the local

source code with the configured remote host

and also invoke the build

• The outcome of the build is displayed in the Console tab view on the workbench

Launch a MPI Job on BlueFern’s HPCs

Resource Manager to Submit a Job via LoadLeveler (1)

• Switch the workbench to the Parallel Runtime perspective

using a shortcut on the toolbar or

select Window>Open Perspective>Other...

and select Parallel Runtime

• Click within the Resource Managers view,

Right Click or (Ctrl+Click on a Mac),

and select Add Resource Manager...


• Select IBM LoadLeveler as

the Resource Manager Type:

• Click on the Next> button

(Note: Although it is possible to use

other types of resource manager with

BlueFern’s HPCs, IBM LoadLeveler is

the recommended resource manager)


• Select Remote Tools as the

Remote service provider:

• Select the remote connection created earlier for Connection name:

• Enter

/usr/local/bin/ptp_ibmll_proxy

in Path to proxy executable:

• Click on the Finish button

Note: we will be using default configurations for the LoadLeveler proxy. They can be changed by clicking on the Next> button and following the rest of the wizard


• Right Click (or Ctrl+Click on Mac) the newly created resource

manager in the Resource Managers and select

Start Resource Manager

• Status of the remote cluster

is now displayed in the

Machines tab view

Prepare a LoadLeveler Script (1)

• Switch the workbench back to C/C++ perspective

(hint: shortcut or Windows>Open Perspective>(Others...))

• Right Click (or Ctrl+Click on Mac) on the src folder under HelloMPI project

• Select New>File


• Enter HelloMPI.ll in the File name: field

• Click on the Finish button


• Prepare your LoadLeveler script as shown below:

Note: if you are not sure about what you value need to populate for the initialdir, you can get it from the Root Location: under the synchronize setting in project

properties

Run Configuration for a Remote MPI Job (1)

• To create a run using the local Open MPI runtime

• Click on the down arrow next to the Run As... icon on the

toolbar

• Select Run Configurations...

• Select Parallel Application as the

configuration type

• Click on the to add a new configuration


• Give the configuration a

name

(use HelloMPI_Remote

for this tutorial)

• Select

IBM@BlueFern.....

as the Resource Manager:

• Click on the Browse

button to open a file

browser on the remote file

system, and navigate to

the LoadLeveler script prepared earlier


• Click on the Application tab

• Click on the Browse button

next to the

Application Program: text

field

• Navigate to the remote

directory where the built

executable is,

select the executable and

click on the OK button

• Click the Apply button to

save the current configuration

Running a MPI Job Remotely (1)

• After you complete the run

configuration, the Run button

should be enabled

• You can click on the Run

button to launch a run using

the current configuration

• For this tutorial, click the

Close button and we will

launch the run from the

workbench instead


• To launch a run using a specific Run Configuration from the workbench

• Click on the down arrow next to the Run As icon on the toolbar

• Select the Run Configuration you wish to launch the run

(For this tutorial, select HelloMPI_Remote)

• Unlike using Open MPI to run a local MPI job,

your job will be processed by the LoadLeveler in

batch mode.

• When the job’s run is completed, its output will be stored in the file specified in the LoadLeveler script

• If you wish to be notified when your job’s run is completed, populate your email address in the #@ notification field in your LoadLeveler script


• Because the project has been converted to a Synchronized project, files generated in the remote file system will be automatically copied to your local file system

• When the job’s run is completed, you should be able see its output files displayed in the Project Explorer view on the workbench if these files are generated within the project’s root directory or sub directories

• Double click such files to check the run’s output

Module 5: Working with a Real Software Project

• Objectives

• Learn how to use Eclipse PTP for a real software project

• Contents:

• Create an empty project

• Import existing code into a project

• Set up MPI runs locally and remotely

Create an Empty Synchronized Project (1)

• Create a Synchronized project because the software will be run on a remote HPC mostly

• File>New>Project...

• Select Synchronized C/C++ Project under the Remote category

(hint: use the text filter to find it quickly)

• Give the project a project name

• Select MPI Empty C Project as the project type

• Select a Linux GCC as the project’s tool chain

Note: Although we actually want to use the XLC C/C++ Tool Chain in the remote environment, there are some issues when using it in the project creation wizard. The workaround is to select Linux GCC in project creation and change it later


• Click on the Next> button to skip the MPI Project Settings and Build configuration dialogs

• Select Remote Tools as the Remote Provider

• Select the remote connection created previously as the connection

• Enter the location where you want to store the project on the remote host

(Note: you may want to include the project name in the path here. Eclipse does not crate a new project in the remote location for you)

• Click on the Finish button to complete the creation of an empty project


• The newly created project will have three build configurations:

• Debug – synchronise local files with the remote files and then invoke a debug build in the remote environment

• Release – synchronise local files with the remote files and then invoke a release build in the remote environment

• Workspace– synchronise local files with the remote files and then invoke a debug build in the local environment

Importing Existing Source Code to the Project

• Select the newly created project in the Project Explorer, bring up its context menu and create a new folder called src for the project

(Note: It is a good practice to keep all source code files in a folder under the project’s root directory. It separates source code files from other types of files, such as documentation.)

• Use the import utility to import all files in

/<DVD Dir>/tutorials/Module_5/ to the project’s src folder

• Select the src folder in the Project Explorer

• Bring up the context menu (right click or ctrl+click on a Mac)

• Select Import...

• Follow the wizard to complete this task

Updating Project Properties (1)

• Select the project in the Project Explorer

• Bring up its context menu and select Project Properties

• Project properties are build configuration specific, we need to update the project properties in all configurations

• Update the project properties based on the tables below:

• C/C++ Build>Tool Chain Editor:

Build Configuration

Debug Release Workspace

Toolchain XL C/C++ Tool Chain

XL C/C++ Tool Chain

Linux GCC (MacOSX GCC)

Builder Sync Builder Sync Builder Sync Builder


• C/C++ Build>Settings:

Build Configuration:

Debug Release Workspace

Local XL C++ Compiler

mpCC mpCC GCC C++ Compiler

mpiCC

XL C++ Executable Linker

mpCC mpCC GCC C Compiler

mpicc

GCC C++ Linker

mpiCC


• C/C++ Build>Environment

• Undefine all environment variables for both Debug and Release build configurations

• C/C++ Build

• Change the Build command: to gmake, because Eclipse generates makefiles in GNU format for managed projects

Building the Project

• Try building the project remotely using both Debug and Release configurations and both of them should result in a successful build

• Try building the project locally using the Workspace configuration and it will fail, because the software uses a library called mass, which is not available on your local computer

Fixing The Local Build (1)

• mass (Mathematical Acceleration Subsystem) is an IBM tuned version of the C math library that provides better performance on IBM HPCs

• To run this project locally, we need to use GNU C’s standard math library instead

• Let’s try to use a macro and change the Workspace build configuration to control the build behaviour

• Double click on the build error displayed in the Console tab view on the workbench to open the file and the location where this issue resides


• Modify the beginning of fie agonist_variation_main.cpp as in the following:

(This simple macro basically says: if symbol MATH_H is defined when compiling the source code, include math.h; otherwise include mass.h instead)


• Use the searching tool to find out if mass.h is defined in any other files

• Select Search>File... to search multiple files

• Entering mass.h in the Containing text: and then click on the Search button

• The search outcome will be displayed in the Search tab view on the workbench

• You will find that mass.h is also included in

update.cpp

• Double click update.cpp in the search tab

view to open the file in the editor


• Use the same macro to control the build behaviour when file update.cpp is complied

• Introduce the following changes to file update.cpp:


• Define symbol MATH_H in the build configuration Workspace

• Open the project’s properties

• Change the build configuration to Workspace

• Select C/C++ Build>Settings

• Select Preprocessor under GCC C++ Compiler

• Click on the icon in the panel of Defined Symbols

• Enter USE_MATH_H and then click on the OK button

• Click the Apply button to save the changes

• Click the OK button to close the project properties dialog

• Try building this project using the Workspace configuration again, the project should build successfully now.

Creating a Local Run (1)

• The sample code used in this tutorial hard coded the size of its simulating model by a #define symbol, let’s find it and replace it with a smaller value (otherwise, you will be waiting for quite some time for the run to complete)

• Use the searching utility to find symbol NUM_EC

• Set its defined value to 4

• Save the change

• Rebuild the project


• When running this sample code, it will generate a lot of output files. It would be an unnecessary overhead to synchronise such files with the remote file system. Let’s exclude them from been synchronised:

• Identify this project in the Project Explorer

• Expand its project tree

• Select the Workspace folder

• Right click or (Ctrl+Click on a Mac) on the Workspace folder to bring up its context menu

• Select Synchronization>Exclude Selection


• Open the Run Configuration wizard

• Create a new configuration for a

Parallel Application

• Under the Resources tab

• Select a local Resource Manager for this configuration (Note: use Open_MPI@Local for this workshop)

• Set the number of processes available to this run to 4 (Number of processes field when using a local OpenMPI resource manger)

• Under the Application tab

• Check if the Parallel Project field has been populated correctly, if not correct it

• Enter the full path to the executable built locally in the Application program field

Running the Project Locally

• After setting up a run configuration, click on the Run button to launch it

• the button might not be enabled if the selected Resource Manager is not currently running

• In such a case, save the current configuration and close the wizard

• Switch the workbench to the Parallel Runtime perspective and start the resource manager selected for the run configuration

• Go back to the run configuration and launch the run

• When the run is completed, you will see END OF PROGRAM in last line of the Console tab view

• The files generated by the run will be available in the Workspace folder under the project in the Project Explorer view on the workbench

Creating a Remote Run (1)

• Try to build the project using both the Debug and Release build configurations. The project should build successful in both configurations

• To avoid unnecessary synchronisation overhead, exclude both Debug and Release folders under the project

• For this tutorial, a remote run will be managed by the LoadLeveler. So, we need to prepare a LoadLeveler script first

• Add a new file to the project and call it remote_run.ll

• Open the newly created file in the source code editor by double clicking it in the Project Explorer view

• Create the script based on the following:


# Example MPI LoadLeveler Job file # @ shell = /bin/ksh # # @ job_name = Agonist_Variation # # @ job_type = parallel # # # @ node = 1 # @ tasks_per_node = 4 # # # @ wall_clock_limit = 00:00:05 # # @ group = tutorial # # set initial dir to the path to your project in the remote environment + /Release/ # initialdir = /hpc/home/<path to the project in the remote environment>/Release # @ initialdir = /hpc/home/ysu20/workspace/Agonist_Variation/Release # # @ output = $(job_name).$(jobid).out # @ error = $(job_name).$(jobid).err # @ notification = never # @ class = dev8_1 # @ queue # invoke the executable poe ./Agonist_Variation


• Creating a remote run configuration:

• Open the Run Configuration wizard

• Create a new configuration for a

Parallel Application

• Under the Resources tab

• Select a remote Resource Manager for this configuration (Note: use IBMLL@A_Remote_Connection for this workshop)

• Use Advanced Submit mode

• Enter the path and filename of the Loadleveler script that was created earlier


• Under the Application tab

• Check if the Parallel Project field has been populated correctly, if not correct it

• Enter the full path to the executable built locally in the Application program field

• After the configuration is created, launch the run

• The job will be processed by the LoadLeveler in the remote environment in batch mode

• The run’s standard output will not be displayed in the Console tab view

• The best way for getting a notification is to populate your email address in the notification field in the LoadLeveler script, particularly if the run can last for a long time

(For this tutorial, the run will only take a few minutes. Let’s just wait for a message saying the remote job is terminated in the Console tab view)

Obtaining Output from a Remote Run

• A practical parallel run usually generates a large amount of data, stored in the remote file system as one or more files, therefore it is not practical to use a project’s file synchronisation feature to retrieve such files

• Eclipse uses git for file synchronisation and it is not designed for transferring files large in size

• BlueFern recommends its users to use:

• FileZilla, a tool with an easy to use GUI and supports SFTP

• scp command that is available on most Linux/Unix/MacOS environments (this is available on a Windows environment if you have Cygwin installed)

Getting More Help for Using Eclipse PTP

• Eclipse PTP User Group:

• Go to https://dev.eclipse.org/mailman/listinfo/ptp-user and subscribe to the mail list

• Send your questions to [email protected]

• Contact BlueFern via [email protected]

• We will do our best to answer questions related to how to use Eclipse PTP with BlueFern’s HPCs

https://dev.eclipse.org/mailman/listinfo/ptp-user



mailto:[email protected]




Acknowledgements

• This tutorial was developed based on the tutorial, “A New and Improved Eclipse Parallel Tools Platform: Advancing the Development of Scientific Applications”, which was presented by Greg Watson (IBM), Beth Tibbitts (IBM), Jay Alameda (NCSA), Jeff Overbey (UIUC), Wyatt Spear (U. Oregon) and Alan Humphrey (SCI) in SC11

• The original tutorial can be downloaded at http://download.eclipse.org/tools/ptp/docs/ptp-sc11-slides-final.pdf

• The sample code used in Module 5 is provided by Dr. Mohsin Shaikh and the Brains Trust research group in the University of Canterbury

http://download.eclipse.org/tools/ptp/docs/ptp-sc11-slides-final.pdf







The END and Thank You

using eclipse parallel tools platform with bluefern’s...

Documents