Portable Parallel Performance Tools

Shirley Browne, UTK

Clay Breshears, CEWES MSRC

Jan 27-28, 1998

Overview

• Tool-assisted performance analysis and tuning• Post-mortem analysis of trace files

– AIMS– MPE logging and nupshot– Pablo– VAMPIR

• Run-time analysis using dynamic instrumentation– Paradyn

Schedule - Tuesday, Jan. 27

• 9:00-10:00

– Trace file generation, MPI profiling interface, trace file formats, analysis techniques

• 10:00-12:00

– Overview of AIMS, nupshot, VAMPIR, SvPablo

• 12:00-1:30

– Lunch

• 1:30-5:00

– Introductory hands-on exercises using trace file generation and analysis tools

Schedule - Wed., Jan. 28

• 9:00-10:15

– Pablo TraceLibrary and Analysis GUI

• 10:30-11:30

– Dynamic instrumentation and Paradyn

• 11:30-1:00

– Lunch

• 1:00-2:00

– Hands-on exercise using Paradyn

• 2:00-5:00

– Bring Your Own Code hands-on exercise

Tool-assisted Performance Analysis

Developperformance

model

Instrumentsource code(optional orautomatic)

Link withinstrumentation

library

Generatetracefile during

program execution

Analyze trace file

Compare modelpredictions withmeasurements

Tune code and/orrevise model

Source Code Instrumentation

• MPI profiling interface (MPE logging, Pablo MPI trace library, VAMPIRtrace)

• Automatic parsing and insertion of calls to tracing routines (AIMS, SvPablo)

• By hand insertion of calls to tracing routines (MPE logging, Pablo basic and I/O trace library, VAMPIRtrace)

MPI Profiling Interface

• Conforming implementation of MPI must– Provide a mechanism through which all the

MPI functions may be accessed with a name shift (All MPI functions which normally start with MPI_ should also be accessible with the prefix PMPI_)

– Provide a no-op routine MPI_PCONTROL

MPI Profiling Interface (cont)

• Goal - enable authors of profiling (and other) tools to interface to MPI implementations without access to the underlying implementation

• Enables interception of all MPI calls made by the user program

Example Profiling Routine

static int totalBytes;static double totalTime;int MPI_Send(void *buffer, const int count, MPI_Datatype datatype, int dest, int tag, MPI_comm comm)

{

double tstart = MPI_Wtime(); int extent; int result = PMPI_Send(buffer,count,datatype,dest, tag,comm); MPI_Type_size(datatype,&extent); /* Compute size */ totalBytes += count * extent; totalTime += MPI_Wtime() - tstart; return result;}

Control of Profiling

• MPI_PCONTROL() allows user to– enable and disable profiling during execution– flush trace buffers at non-critical points– add user events to a trace file

• Default state after MPI_INIT has been called is for profiling to be enabled at normal default level

Trace File Formats

• Self-Defining Data Format (SDDF) - Pablo

• Alog - MPE logging

• VAMPIR

• AIMS

• IBM VT format

• PICL

Parallel Tools Library (PTLIB)• http://www.nhse.org/ptlib/• Access to and information about parallel tools• Communication libraries, compilers,

debuggers, performance tools• Initial focus on parallel performance tools• Parallel Tools Digest• Discussion forums

PTLIB Evaluation of Performance Tools for MPI

• Tracefile generation and analysis tools– AIMS– MPE logging and nupshot– Pablo– VAMPIR

• Automated runtime analysis tool– Paradyn

PTLIB Evaluation Criteria

• Robustness

• Usability

• Scalability

• Portability

• Versatility

PTLIB Evaluation Platforms

• UNIX workstations

• IBM SP

• Cray T3E

• SGI Power Challenge Array

• SGI Origin 2000

Parallel Performance Tool Capabilities Post-

mortemanalysis

Automaticrun-timeanalysis

Sourcecode

clickback

State-timediagram

(zooming/scrolling)

Statisticalanalysis

Languages& platformssupported

AIMS x x x xFortran,CSGI , Sun,IBM SP

nupshot x x(x)Language-independentMost MPIplatforms

Paradyn(MPI version) x x

Fortran, C,HPF, C++IBM SP

SvPablo x x xANSI C, HPFSun, SGI

VAMPIR x x(x)Language-independentall platforms

SDDF Example (cont.)

“Seek” {

[2] {

201803857,

0

}, 20.1803857, 70013, 0, 0.0031946, 3, 0, 0, 0 };;