a case study of communication optimizations on 3d ...charm.cs.illinois.edu/newpapers/09-24/talk.pdfa...

A CASE STUDY OF COMMUNICATION OPTIMIZATIONS ON 3D MESH INTERCONNECTS

University of Illinois at Urbana-Champaign

Abhinav Bhatele, Eric Bohm, Laxmikant V. KaleParallel Programming Laboratory

Euro-Par 2009

Outline

MotivationSolution: Mapping of OpenAtomPerformance BenefitsBigger Picture:

Resources NeededHeuristic Solutions

Automatic Mapping

August 27th, 2009

2

Abhinav Bhatele @ Euro-Par 2009

OpenAtom

Ab-Initio Molecular Dynamics codeConsider electrostatic interactions between the nuclei and electronsCalculate different energy termsDivided into different phases with lot of communication

August 27th, 2009

3


OpenAtom on Blue Gene/L

August 27th, 2009Abhinav Bhatele @ Euro-Par 2009

4

0

0.05

0.1

0.15

0.2

0.25

0.3

512 1024 2048 4096 8192

Tim

e pe

r ste

p (s

ecs)

No. of cores

w32 Default

Runs on Blue Gene/L at IBM T J Watson Research Center, CO mode

w32 = 32 water molecules with 70 Ry cutoff

The problem lies in …


5

Performance Analysis and Visualization Tool: Projections (part of Charm++) – Timeline View

Solution –


6

Topology Aware Mapping

Processor Virtualization

User View System View

Programmer: Decomposes the computation into objects

Runtime: Maps the computation on to the processors

August 27th, 2009

7


Benefits of Charm++


8

Computation is divided into objects/chares/virtual processors (VPs)Separates decomposition from mappingVPs can be flexibly mapped to actual physical processors (PEs)

Topology Manager API†

The application needs information such asDimensions of the partitionRank to physical co-ordinates and vice-versa

TopoManager: a uniform APIOn BG/L and BG/P: provides a wrapper for system callsOn XT3/4/5, there are no such system callsProvides a clean and uniform interface to the application


9

† http://charm.cs.uiuc.edu/~bhatele/phd/topomgr.htm

Parallelization using Charm++

August 27th, 2009

10

Eric Bohm, Glenn J. Martyna, Abhinav Bhatele, Sameer Kumar, Laxmikant V. Kale, John A. Gunnels, and Mark E. Tuckerman. Fine Grained Parallelization of the Car-Parrinello ab initio MD Method on Blue Gene/L. IBM J. of R. and D.: Applications of Massively Parallel Systems, 52(1/2):159-174, 2008.


Mapping Challenge


11

Load Balancing: Multiple VPs per PEMultiple groups of communicating objects

Intra-group communicationInter-group communication

Conflicting communication requirements

Topology Mapping of Chare Arrays

August 27th, 2009

12

RealSpace and GSpace have state-wisecommunication Paircalculator and

GSpace have plane-wisecommunication


Performance Improvements on BG/L


13

0

0.05

0.1

0.15

0.2

0.25

0.3

512 1024 2048 4096 8192

Tim

e pe

r ste

p (s

ecs)

No. of cores

w32 Defaultw32 Topology

Runs on Blue Gene/L at IBM T J Watson Research Center, CO mode, Year: 2006


Improved Timeline Views


14

Results on Blue Gene/L


15

0

5

10

15

20

25

1024 2048 4096 8192 16384

Tim

e pe

r ste

p (s

ecs)

No. of cores

w256 Defaultw256 TopologyGST_BIG DefaultGST_BIG Topology

GST_BIG = 64 Ge, 128 Sb and 256 Te molecules with 20 Ry cutoff

Runs on Blue Gene/L at IBM T J Watson Research Center, CO mode

Results on Blue Gene/P


16

0

2

4

6

8

10

12

1024 2048 4096 8192

Tim

e pe

r ste

p (s

ecs)

No. of cores

w256 Defaultw256 Topology


Runs on Blue Gene/P at Argonne National Laboratory, VN mode

Results on Cray XT3


17

2

3

4

5

6

7

8

512 1024 2048

Tim

e pe

r ste

p (s

ecs)

No. of cores

w256 Defaultw256 TopologyGST_BIG DefaultGST_BIG Topology

Runs on Cray XT3 (Bigben) at Pittsburgh Supercomputing Center, VN mode(with system reservation to obtain complete 3d mesh shapes)

Performance Analysis


18

0

500000

1000000

1500000

2000000

2500000

3000000

1024 2048 4096 8192

Idle

Tim

e (s

ecs)

No. of cores

DefaultTopology

Performance Analysis and Visualization Tool: Projections – Idle time added across all processors

w256M_70Ry on Blue Gene/L

Reduction in Communication Volume


19

0

200

400

600

800

1000

1200

1024 2048 4096 8192

Band

wid

th (

GB)

No. of cores

DefaultTopology

Data obtained from Blue Gene/P’s Uniform Performance Counters

w256M_70Ry on Blue Gene/P

Relative Performance Improvement


20

0

0.5

1

1.5

2

2.5

512 1024 2048 4096 8192

% Im

prov

emen

t

No. of cores

Cray XT3Blue Gene/LBlue Gene/P

w256M_70Ry

Bigger picture


21

Different kinds of applications:Computation boundCommunication bound

Latency tolerantLatency sensitive

Technique:Obtain processor topology and application communication graphHeuristic Techniques for mapping

Why does distance affect message latencies?

Consider a 3D mesh/torus interconnect

Message latencies can be modeled by

(Lf/B) x D + L/B

Lf = length of flit, B = bandwidth,

D = hops, L = message size

When (Lf * D) << L, first term is negligible


22

But in presence of contention …

Automatic Topology Aware Mapping


23

Many MPI applications exhibit a simple two-dimensional near-neighbor communication patternExamples: MILC, WRF, POP, Stencil, …

Acknowledgements:Shawn Brown and Chad Vizino (PSC)Glenn Martyna, Sameer Kumar, Fred Mintzer (IBM)Teragrid for running time on Bigben (XT3)ANL for running time on Blue Gene/P

DOE Grant B341494 (CSAR), DOE Grant DE-FG05-08OR23332 (ORNL LCF) and NSF Grant ITR 0121357


Funding

E-mail: [email protected]: http://charm.cs.illinois.edu

a case study of communication optimizations on 3d ...charm.cs.illinois.edu/newpapers/09-24/talk.pdfa...

Documents