• UPC: a GAS extension to ISO C99• shared types & pointers
• Libraries for parallel ops• Synchronization• Collective data movement• File I/O
• Many open source & vendor implementations• Berkeley UPC, MuPC• HP UPC, Cray UPC
• See UPC Booth #137
• NAS Benchmarks written from scratch• FT, CG, MG, LU - all NAS-compliant • Use some Berkeley UPC extensions• All competitive with best MPI versions,
even on cluster networks
• UPC: a GAS extension to ISO C99• shared types & pointers
• Libraries for parallel ops• Synchronization• Collective data movement• File I/O
• Many open source & vendor implementations• Berkeley UPC, MuPC• HP UPC, Cray UPC
• See UPC Booth #137
• NAS Benchmarks written from scratch• FT, CG, MG, LU - all NAS-compliant • Use some Berkeley UPC extensions• All competitive with best MPI versions,
even on cluster networks
Christian Bell, Dan Bonachea, Christian Bell, Dan Bonachea, KaushikKaushik DattaDatta, Rajesh , Rajesh NishtalaNishtala, Paul Hargrove, Parry Husbands, Kathy Yelick, Paul Hargrove, Parry Husbands, Kathy Yelick
http://upc.http://upc.lbllbl.gov.gov
The Performance and Productivity Benefits of Global Address SpacThe Performance and Productivity Benefits of Global Address Space Languagese Languages
http://titanium.http://titanium.cscs..berkeleyberkeley..eduedu http://gasnet.http://gasnet.cscs..berkeleyberkeley..eduedu
• Global Address Space languages (GAS) support:• Global pointers and distributed arrays• User controlled layout of data across nodes• Communicate using implicit reads & writes
of remote memory
• Languages: UPC, Titanium, Co-Array Fortran• Productivity benefits of shared-memory programming• Competitive performance on distributed memory
• Use Single Program Multiple Data (SPMD) control • Fixed number of compute threads• Global synchronization, barriers, collectives
• Exploit fast one-sided communication • Individual accesses and bulk copies• Berkeley implementations use GASNet
• Global Address Space languages (GAS) support:• Global pointers and distributed arrays• User controlled layout of data across nodes• Communicate using implicit reads & writes
of remote memory
• Languages: UPC, Titanium, Co-Array Fortran• Productivity benefits of shared-memory programming• Competitive performance on distributed memory
• Use Single Program Multiple Data (SPMD) control • Fixed number of compute threads• Global synchronization, barriers, collectives
• Exploit fast one-sided communication • Individual accesses and bulk copies• Berkeley implementations use GASNet
Global Address Space LanguagesGlobal Address Space Languages
LU DecompositionLU Decomposition
Fourier TransformFourier Transform
Conjugate GradientConjugate GradientPerformancePerformance
ProductivityProductivity
GASNet PortabilityGASNet Portability• Native network hardware support:
• Quadrics QsNet I/II (Elan3/Elan4) • Cray X1 - Cray shmem• SGI Altix - SGI shmem• Dolphin - SCI• InfiniBand - Mellanox VAPI• Myricom Myrinet - GM-1 and GM-2• IBM Colony and Federation - LAPI
• Portable network support:• Ethernet - UDP: works with any TCP/IP• MPI 1: portable impl. for other HPC systems
• Berkeley UPC, Titanium & GASNet highly portable:• Runtimes and generated code all ANSI C
• New platform ports in 2-3 days • New network hardware 2-3 weeks
• CPU's: x86, Itanium, Opteron, Athlon, Alpha, PowerPC, MIPS, PA-RISC, SPARC, T3E, X-1, SX-6, …
• OS's: Linux, FreeBSD, NetBSD, Tru64, AIX, IRIX, HPUX, Solaris, MS-Windows/Cygwin, Mac OSX, Unicos, SuperUX, Catamount, BlueGene, …
• Native network hardware support:• Quadrics QsNet I/II (Elan3/Elan4) • Cray X1 - Cray shmem• SGI Altix - SGI shmem• Dolphin - SCI• InfiniBand - Mellanox VAPI• Myricom Myrinet - GM-1 and GM-2• IBM Colony and Federation - LAPI
• Portable network support:• Ethernet - UDP: works with any TCP/IP• MPI 1: portable impl. for other HPC systems
• Berkeley UPC, Titanium & GASNet highly portable:• Runtimes and generated code all ANSI C
• New platform ports in 2-3 days • New network hardware 2-3 weeks
• CPU's: x86, Itanium, Opteron, Athlon, Alpha, PowerPC, MIPS, PA-RISC, SPARC, T3E, X-1, SX-6, …
• OS's: Linux, FreeBSD, NetBSD, Tru64, AIX, IRIX, HPUX, Solaris, MS-Windows/Cygwin, Mac OSX, Unicos, SuperUX, Catamount, BlueGene, …
NAS Parallel Benchmark Code Size
0
100200
300400
500600
700800900
F+MPI Ti F+MPI Ti F+MPI Ti
Prod
uctiv
e lin
es o
f cod
e
DeclarationsCommunicationComputation
CG FT MG
ApplicationsApplications
Mach-10 shock hitting solid surface at oblique angle
• Immersed Boundary Method Simulation• Human Heart• Cochlea
• Adaptive Mesh Refinement (AMR)• AMR Poisson Multigrid Solvers• 3D AMR Gas Dynamics Hyperbolic Solver• AMR with Line Relaxation (low aspect ratio)
• Bioinformatics: Microarray oligonucleotide selection• Finite Element Benchmarks• Tree-structured n-body kernels• Dense Linear Algebra: LU, MatMul
• Immersed Boundary Method Simulation• Human Heart• Cochlea
• Adaptive Mesh Refinement (AMR)• AMR Poisson Multigrid Solvers• 3D AMR Gas Dynamics Hyperbolic Solver• AMR with Line Relaxation (low aspect ratio)
• Bioinformatics: Microarray oligonucleotide selection• Finite Element Benchmarks• Tree-structured n-body kernels• Dense Linear Algebra: LU, MatMul
final Point<2> NORTH = [0,1], SOUTH = [0,-1],EAST = [1,0], WEST = [-1,0];
foreach (p in gridA.domain()) gridB[p] = S0 * gridA[p] +
S1 * ( gridA[p+NORTH] + gridA[p+SOUTH] +gridA[p+EAST] + gridA[p+WEST] );
Unified Parallel CUnified Parallel C• Titanium: GAS dialect of Java
• Compiled to native executable: • No JVM or JIT
• Highly portable & high performance• High productivity language
• All the productivity features of Java• Automatic memory management• Object oriented programming, etc
• Built-in support for scientific computing:• n-D points, rectangles, domains• n-D domain calculus operators• Flexible & efficient multi-D arrays
• High-performance templates • User-defined immutable (value) classes• Explicitly unordered n-D loop iteration• Operator overloading• Efficient cross-language support
• Allows for elegant and concise programs
• Titanium: GAS dialect of Java• Compiled to native executable:
• No JVM or JIT• Highly portable & high performance
• High productivity language• All the productivity features of Java
• Automatic memory management• Object oriented programming, etc
• Built-in support for scientific computing:• n-D points, rectangles, domains• n-D domain calculus operators• Flexible & efficient multi-D arrays
• High-performance templates • User-defined immutable (value) classes• Explicitly unordered n-D loop iteration• Operator overloading• Efficient cross-language support
• Allows for elegant and concise programs
High PerformanceHigh Performance
Shared
Glo
bal
addr
ess
spac
e X
Privatep p p
P4 Z
Thread0 Thread1 … Threadn
P5
yy y
A[0] A[1] A[n] Shared
Glo
bal
addr
ess
spac
e X
Privatep p p
P4 Z
Thread0 Thread1 … Threadn
P5
yy y
A[0] A[1] A[n]
• Global Address Space logically partitioned• Shared data access from any thread• Private data local to each thread
• Pointers are either local or global (possibly remote)
• Global Address Space logically partitioned• Shared data access from any thread• Private data local to each thread
• Pointers are either local or global (possibly remote)
2-d Multigrid stencil code in Titanium2-d Multigrid stencil code in Titanium
AMR Code Size
10 K
20 K
30 K
40 K
50 K
C++/Fortran/MPI Titanium
Line
s of
cod
e
Elliptic PDE solver*
AMR operations
AMR data structures
MG Class D Speedup - Opteron/InfiniBand
020406080
100120140
0 20 40 60 80 100 120 140Processors
Spee
dup
(Bes
t32P
roc:
171.
3sec
) Fortran w/MPITitanium
* C++ code has a more general grid generator
FT Class C Speedup - G5/InfiniBand
01020304050607080
0 10 20 30 40 50 60 70Processors
Spee
dup
(Bes
t 16
Proc
:37.
9sec
) Titanium (non-blocking)Titanium (blocking)Fortran w/MPI
FFTW is used for all serial 1-D FFT's
113.3 sec121.9 secParallel (large)57.5 sec47.6 secSerial (small)
Chombo (C++/F/MPI )Titanium AMRAMR Solver
• High ratio of computation to communication• scales very well to large machines: Top500
• Non-trivial dependence patterns• HPL code uses two-sided MPI messaging,
and is very difficult to tune• UPC implementation of LU uses:
• one-sided communication in GAS model• lightweight multithreading atop SPMD• memory-constrained lookahead to manage
amount of concurrency at each processor• highly adaptable to problem/machine sizes• natural latency tolerance
• Performance is comparable to HPL / MPI• UPC code less than half the size
• High ratio of computation to communication• scales very well to large machines: Top500
• Non-trivial dependence patterns• HPL code uses two-sided MPI messaging,
and is very difficult to tune• UPC implementation of LU uses:
• one-sided communication in GAS model• lightweight multithreading atop SPMD• memory-constrained lookahead to manage
amount of concurrency at each processor• highly adaptable to problem/machine sizes• natural latency tolerance
• Performance is comparable to HPL / MPI• UPC code less than half the size
Network-independent
GAS language GAS language program codeprogram code
Translator Generated C CodeTranslator Generated C CodeLanguage Runtime SystemLanguage Runtime System
GASNet Communication SystemGASNet Communication SystemNetwork HardwareNetwork Hardware
Platform-independent
Compiler-independent
Language-independent
SourceSource--toto--source source TranslatorTranslator
• Solves Ax = b for x, where A is sparse 2-D array• Computation dominated by Sparse Matrix-Vector
Multiplications (SPMV)• Key communication for 2-D (NAS) decomposition:
• team-sum-reduce-to-all (vector&scalar)• point-to-point exchange
• Need explicit synchronization in one-sided model • Uses tree-based team reduction library• Tunable based on network characteristics
• Overlap the vector reductions on the rows of the matrix with the computation of the SPMV
• Outperform MPI implementation by up to 10%
• Solves Ax = b for x, where A is sparse 2-D array• Computation dominated by Sparse Matrix-Vector
Multiplications (SPMV)• Key communication for 2-D (NAS) decomposition:
• team-sum-reduce-to-all (vector&scalar)• point-to-point exchange
• Need explicit synchronization in one-sided model • Uses tree-based team reduction library• Tunable based on network characteristics
• Overlap the vector reductions on the rows of the matrix with the computation of the SPMV
• Outperform MPI implementation by up to 10%
0
10
20
30
40
50
60
70
80
90
100
Opteron/InfiniBand D/256 Alpha/Elan3 D/512 x86/Myrinet D/64
system, problem class / proc count
MFl
ops
/ thr
ead
MPI / Fortran MPI / C MPI / C w/ Overlap UPC UPC w/ Overlap
Opteron/InfiniBand Flood Bandwidth Comparison
0
100
200
300
400
500
600
700
800
900
1 10 100 1,000 10,000 100,000 1,000,000 10,000,000Transfer Size (bytes)
Ban
dwid
th (M
B/s
ec)
GASNet put_nb
MPI (MVAPICH 0.9.5) ISend/IRecv
jacquard.nersc.gov- Dual CPU 2.2Ghz Opteron, 4GB mem- Mellanox InfiniHost PCI-X, IB-4X HCAs
Bandwidth Ratio
1.001.251.501.752.002.252.50
10 1,000 100,000 10,000,000Message Size (bytes)
(MB
= 2
^20
b) .
Flood Bandwidth for 4KB Messages
231 MB/s
815 MB/s 223 MB/s
679 MB/s 714 MB/s
177 MB/s 495 MB/s 152 MB/s
252 MB/s
426 MB/s
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
80 %
90 %
100 %
AlphaElan3
Itanium2Elan4
x86Myrinet
Power5InfiniBand
OpteronInfiniBand
Per
cent
of H
ardw
are
Pea
k Ba
ndw
idth
GASNet MPI
Flood Bandwidth for 512KB Messages
256 MB/s859 MB/s 229 MB/s
796 MB/s 799 MB/s
246 MB/s
851 MB/s225 MB/s
671 MB/s 659 MB/s
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
80 %
90 %
100 %
AlphaElan3
Itanium2Elan4
x86Myrinet
Power5InfiniBand
OpteronInfiniBand
Perc
ent o
f Har
dwar
e Pe
ak B
andw
idth
.
GASNet MPI
Itanium2/Elan4 Roundtrip Latency Comparison
0
1
2
3
4
5
6
7
8
9
10
11
12
1 10 100 1000Transfer Size (bytes)
Rou
nd-tr
ip la
tenc
y (u
sec)
MPI Isend/Irecv ping + 0-byte ackMPI send/recv ping + 0-byte ackGASNet put_nb + syncGASNet get_nb + sync
• Communication micro-benchmarks show GASNet consistently matches or outperforms MPI
• One-sided, lightweight communication semantics• eliminates matching & rendezvous overheads• better match to hardware capabilities• leverages widely-available support for remote
direct memory access• Active Messages support provides extensibility
• Communication micro-benchmarks show GASNet consistently matches or outperforms MPI
• One-sided, lightweight communication semantics• eliminates matching & rendezvous overheads• better match to hardware capabilities• leverages widely-available support for remote
direct memory access• Active Messages support provides extensibility
Datta, Bonachea and Yelick, LCPC 2005Datta, Bonachea and Yelick, LCPC 2005
Wen and Collella, IPDPS 2005Wen and Collella, IPDPS 2005
(up
is g
ood)
(up
is g
ood)
(up
is g
ood)
(up
is g
ood)
(up
is g
ood)
(up
is g
ood)
(up
is g
ood)
(up
is g
ood)
(dow
n is
goo
d)(d
own
is g
ood)
(up
is g
ood)
(up
is g
ood)
(dow
n is
goo
d)(d
own
is g
ood)
• Titanium reduces code size and development time• Communication more concise than MPI, less error-prone• Language features and libraries capture useful paradigms
• Titanium reduces code size and development time• Communication more concise than MPI, less error-prone• Language features and libraries capture useful paradigms
LU performance comparison
10% 20% 30% 40% 50% 60% 70% 80% 90%
Cray X1 (64)
Cray X1 (128/124)
SGI Altix (32)
Opteron 2.2 GHz (64)
Sys
tem
(p
roc
coun
t)
% peak performance
MPI/HPLUPC/LU
• 3-D FFT with 1-D partitioning across processors• Computation is 1-D FFTs using FFTW library• Communication is all-to-all transpose• Traditionally a bandwidth-limited problem
• Optimization approach in GAS languages• Aggressively overlap transpose with 2nd FFT• Send more, smaller msgs to maximize overlap
• Per-target slabs or individual 1-D pencils• Use low-overhead one-sided communication
• Consistently outperform MPI-based implementations• Improvement of up to 2x, even at large scale
• 3-D FFT with 1-D partitioning across processors• Computation is 1-D FFTs using FFTW library• Communication is all-to-all transpose• Traditionally a bandwidth-limited problem
• Optimization approach in GAS languages• Aggressively overlap transpose with 2nd FFT• Send more, smaller msgs to maximize overlap
• Per-target slabs or individual 1-D pencils• Use low-overhead one-sided communication
• Consistently outperform MPI-based implementations• Improvement of up to 2x, even at large scale
(up
is g
ood)
(up
is g
ood)
Fine-grained communication overlap less effective in MPI due to increased messaging overheads and higher per-message costs
Fine-grained communication overlap less effective in MPI due to increased messaging overheads and higher per-message costs
TitaniumTitanium
