matlab hpcs extensions
Post on 09-Jan-2016
33 Views
Preview:
DESCRIPTION
TRANSCRIPT
MATLAB HPCS Extensions
Presented by: David PaduaUniversity of Illinois at Urbana-Champaign
Senior Team Members
• Gheorghe Almasi - IBM Research
• Calin Cascaval - IBM Research
• Basilio Fraguela - University of Illinois
• Jose Moreira - IBM Research
• David Padua - University of Illinois
Objectives
• To develop MATLAB extensions for accessing, prototyping, and implementing scalable parallel algorithms.
• To give programmers of high-end machines access to all the powerful features of MATLAB, as a result.– Array operations / kernels.– Interactive interface. – Rendering.
Goals of the Design
• Minimal extension.• A natural extension to MATLAB that is easy to
use. • Extensions for direct control of parallelism and
communication on top of the ability to access parallel library routines. It does not seem it is possible to encapsulate all the important parallelism in library routines.
• Extensions that provide the necessary information and can be automatically and effectively analyzed for compilation and translation.
Uses of the Language Extension
• Parallel libraries user.
• Parallel library developer.
• Input to type-inference compiler.
• No existing MATLAB extension had the necessary characteristics.
Approach
• In our approach, the programmer interacts with a copy of MATLAB running on a workstation.
• The workstation controls parallel computation on servers.
Approach (Cont.)
• All conventional MATLAB operations are executed on the workstation.
• The parallel server operates on a new class of MATLAB objects: hierarchically tiled arrays (HTAs).
• HTAs are implemented as a MATLAB toolbox. This enables implementation as a language extension and simplifies porting to future versions of MATLAB.
Hierarchically Tiled Arrays
• Array tiles are a powerful mechanism to enhance locality in sequential computations and to represent data distribution across parallel systems.
• Our main extension to MATLAB is arrays that are hierarchically tiled.
• Several levels of tiling are useful to distribute data across parallel machines with a hierarchical organization and to simultaneously represent both data distribution and memory layout.
• For example, a two-level hierarchy of tiles can be used to represent:– the data distribution on a parallel system and – the memory layout within each component.
Hierarchically Tiled Arrays (Cont.)
• Computation and communication are represented as array operations on HTAs.
• Using array operations for communication and computation raises the level of abstraction and, at the same time, facilitates optimization.
Semantics
HTA Definition
Array partitioned into tiles in such a way that adjacenttiles have the same size along the dimension of adjacency; each tile being either an unpartitioned array or an HTA
More HTAs
Example of non-HTA
Addressing the HTAs
• We want to be able to address any level of an HTA A– A.getTile(i,j): Get tile (i,j) of the HTA– A.getElem(a,b): Get element (a,b) of the HTA,
disregarding the tiles (flattening)– A.getTile(i,j).getElem(c,d): Get element (c,d)
within tile (i,j)
• Triplets supported in the indexes
HTA Indexing Example
x4 = A(2,9:11)
x1 = A{2}{4,3}(3)
x2 = A{:}{2:4,3}(1:2)
x3 = A{1}(1:4,1:6)
A = hta {1:2}{1:4,1:3}(1:3)
“flattened” access
Assigning to an HTA
• Replication of scalarsA(1:3,4:5)=scalar
• Replication of matricesA{1:3,4:5}=matrix
• Replication of HTAsA{1:3,4:5}=single_tile_HTA
• Copies tile to tileA{1:3,4:5}=B{[3 8 9],2:3}
Operation Legality
• A scalar can be operated with every scalar (or set of them) of an HTA
• A matrix can be operated with any (sub)tiles of an HTA
• Two HTAs can be operated if their corresponding tiles can be operated– Notice that this depends on the operation
Tile Usage Basics
• Tiles are used in many algorithms to express locality
• Tiles can be distributed among processors
• Block-cyclic always
• Operation on tiles take place on the server where they reside
Using HTAs for Locality Enhancement
for I=1:q:n for J=1:q:n for K=1:q:n for i=I:I+q-1 for j=J:J+q-1 for k=K:K+q-1 C(i,j)=C(i,j)+A(i,k)*B(k,j); end end end end endend
Tiled matrix multiplication using conventional arrays
• Here, C{i,j}, A{i,k}, B{k,j} represent submatrices.
• The * operator represents matrix multiplication in MATLAB.
Tiled matrix multiplication using HTAsUsing HTAs for Locality Enhancement
for i=1:m for j=1:m for k=1:m C{i,j}=C{i,j}+A{i,k}*B{k,j}; end endend
A{1,1}B{1,1}
A{1,2}B{2,2}
A{1,3}B{3,3}
A{1,4}B{4,4}
A{2,2}B{2,1}
A{2,3}B{3,2}
A{2,4}B{4,3}
A{2,1}B{1,4}
A{3,3}B{3,1}
A{3,4}B{4,2}
A{3,1}B{1,3}
A{3,2}B{2,4}
A{4,4}B{4,1}
A{4,1}B{1,2}
A{4,2}B{2,3}
A{4,3}B{3,4}
Using HTAs to Represent Data Distribution and Parallelism
Cannon’s Algorithm
A{1,1}B{1,1}
A{1,2}B{2,2}
A{1,3}B{3,3}
A{1,4}B{4,4}
A{2,2}B{2,1}
A{2,3}B{3,2}
A{2,4}B{4,3}
A{2,1}B{1,4}
A{3,3}B{3,1}
A{3,4}B{4,2}
A{3,1}B{1,3}
A{3,2}B{2,4}
A{4,4}B{4,1}
A{4,1}B{1,2}
A{4,2}B{2,3}
A{4,3}B{3,4}
A{1,2}B{1,1}
A{1,3}B{2,2}
A{1,4}B{3,3}
A{1,1}B{4,4}
A{2,3}B{2,1}
A{2,4}B{3,2}
A{2,1}B{4,3}
A{2,2}B{1,4}
A{3,4}B{3,1}
A{3,1}B{4,2}
A{3,2}B{1,3}
A{3,3}B{2,4}
A{4,1}B{4,1}
A{4,2}B{1,2}
A{4,3}B{2,3}
A{4,4}B{3,4}
A{1,2}B{2,1}
A{1,3}B{3,2}
A{1,4}B{4,3}
A{1,1}B{1,4}
A{2,3}B{3,1}
A{2,3}B{4,2}
A{2,4}B{1,3}
A{2,1}B{2,4}
A{3,3}B{4,1}
A{3,4}B{1,2}
A{3,1}B{2,3}
A{3,2}B{3,4}
A{4,4}B{1,1}
A{4,1}B{2,2}
A{4,2}B{3,3}
A{4,3}B{4,4}
a=hta(A,{1:SZ/n:n,1:SZ/n:n},[n n]);b=hta(B,{1:SZ/n:n,1:SZ/n:n},[n n]);c=hta(n,n,[n n]);c{:,:} = zeros(p,p);
for i=2:n a{i,:} = circshift(a{i,:}, [0 -1]); b{:,i} = circshift(b{:,i}, [-1 0]);endfor k=1:n c = c + a * b; a = circshift(a,[0 -1]); b = circshift(b,[-1 0]);end
Cannnon’s Algorithm in MATLAB with HPCS Extensions
Cannnon’s Algorithm in C + MPI
for (km = 0; km < m; km++) { char *chn = "T";
dgemm(chn, chn, lclMxSz, lclMxSz, lclMxSz, 1.0, a, lclMxSz, b, lclMxSz, 1.0, c, lclMxSz);
MPI_Isend(a, lclMxSz * lclMxSz, MPI_DOUBLE, destrow, ROW_SHIFT_TAG, MPI_COMM_WORLD, &requestrow);
MPI_Isend(b, lclMxSz * lclMxSz, MPI_DOUBLE, destcol, COL_SHIFT_TAG, MPI_COMM_WORLD, &requestcol);
MPI_Recv(abuf, lclMxSz * lclMxSz, MPI_DOUBLE, MPI_ANY_SOURCE, ROW_SHIFT_TAG, MPI_COMM_WORLD, &status);
MPI_Recv(bbuf, lclMxSz * lclMxSz, MPI_DOUBLE, MPI_ANY_SOURCE, COL_SHIFT_TAG, MPI_COMM_WORLD, &status);
MPI_Wait(&requestrow, &status); aptr = a; a = abuf; abuf = aptr;
MPI_Wait(&requestcol, &status); bptr = b; b = bbuf; bbuf = bptr; }
Speedups on afour-processor IBM SP-2
Speedups on anine-processor IBM SP-2
The SUMMA Algorithm (1 of 6)
Use now the outer-product method (n2-parallelism)Interchanging the loop headers of the loop in Example 1 produce:
for k=1:n for i=1:n for j=1:n C{i,j}=C{i,j}+A{i,k}*B{k,j} end end
endTo obtain n2 parallelism, the inner two loops should take the form of a block operations:
for k=1:n C{:,:}=C{:,:}+A{:,k} B{k,:};
endWhere the operator represents the outer product operations
The SUMMA Algorithm (2 of 6)
• The SUMMA AlgorithmA BC
b11 b12
a11
a21
a11b11
a21b11
a11b12
a21b12
Switch Orientation -- By using a column of A and a row of B broadcast to all, compute the “next” terms of the dot product
Switch Orientation -- By using a column of A and a row of B broadcast to all, compute the “next” terms of the dot product
The SUMMA Algorithm (3 of 6)
c{1:n,1:n} = zeros(p,p); % communicationfor i=1:n t1{:,:}=spread(a(:,i),dim=2,ncopies=N); % communication t2{:,:}=spread(b(i,:),dim=1,ncopies=N); % communication c{:,:}=c{:,:}+t1{:,:}*t2{:,:}; % computationend
The SUMMA Algorithm (4 of 6)
The SUMMA Algorithm (5 of 6)
The SUMMA Algorithm (6 of 6)
Matrix Multiplication with Columnwise Block Striped Matrix
for i=0:p-1block(i) =[ i*n/p : (i-1)*n/p-1 ];
endfor i=0:p-1
M{i} = A(:,block(i));v{i} = b(block(i));
endforall j=0:p-1
c{j} =M{i}*v{j};endforall i=0:p-1;j=0:p-1
d{i}(j) = c{j}(i)endforall i=0:p-1
b{i} = sum(d{i})end
Flattening
• Elements of an HTA are referenced using a tile index for each level in the hierarchy followed by an array index. Each tile index tuple is enclosed within {}s and the array index is enclosed within parentheses. – In the matrix multiplication code, C{i,j}(3,4) would represent
element 3,4 of submatrix i,j.
• Alternatively, the tiled array could be accessed as a flat array as shown in the next slide.
• This feature is useful when a global view of the array is needed in the algorithm. It is also useful while transforming a sequential code into parallel form.
Two Ways of Referencing the Elements of an 8 x 8 Array.
C{1,1}(1,1)C(1,1)
C{1,1}(1,2)C(1,2)
C{1,1}(1,3)C(1,3)
C{1,1}(1,4)C(1,4)
C{1,2}(1,1)C(1,5)
C{1,2}(1,2)C(1,6)
C{1,2}(1,3)C(1,7)
C{1,2}(1,4)C(1,8)
C{1,1}(2,1)C(2,1)
C{1,1}(2,2)C(2,2)
C{1,1}(2,3)C(2,3)
C{1,1}(2,4)C(2,4)
C{1,2}(2,1)C(2,5)
C{1,2}(2,2)C(2,6)
C{1,2}(2,3)C(2,7)
C{1,2}(2,4)C(2,8)
C{1,1}(3,1)C(3,1)
C{1,1}(3,2)C(3,2)
C{1,1}(3,3)C(3,3)
C{1,1}(3,4)C(3,4)
C{1,2}(3,1)C(3,5)
C{1,2}(3,2)C(3,6)
C{1,2}(3,3)C(3,7)
C{1,2}(3,4)C(3,8)
C{1,1}(4,1)C(4,1)
C{1,1}(4,2)C(4,2)
C{1,1}(4,3)C(4,3)
C{1,1}(4,4)C(4,4)
C{1,2}(4,1)C(4,5)
C{1,2}(4,2)C(4,6)
C{1,2}(4,3)C(4,7)
C{1,2}(4,4)C(4,8)
C{2,1}(1,1)C(5,1)
C{2,1}(1,2)C(5,2)
C{2,1}(1,3)C(5,3)
C{2,1}(1,4)C(5,4)
C{2,2}(1,1)C(5,5)
C{2,2}(1,2)C(5,6)
C{2,2}(1,3)C(5,7)
C{2,2}(1,4)C(5,8)
C{2,1}(2,1)C(6,1)
C{2,1}(2,2)C(6,2)
C{2,1}(2,3)C(6,3)
C{2,1}(2,4)C(6,4)
C{2,2}(2,1)C(6,5)
C{2,2}(2,2)C(6,6)
C{2,2}(2,3)C(6,7)
C{2,2}(2,4)C(6,8)
C{2,1}(3,1)C(7,1)
C{2,1}(3,2)C(7,2)
C{2,1}(3,3)C(7,3)
C{2,1}(3,4)C(7,4)
C{2,2}(3,1)C(7,5)
C{2,2}(3,2)C(7,6)
C{2,2}(3,3)C(7,7)
C{2,2}(3,4)C(7,8)
C{2,1}(4,1)C(8,1)
C{2,1}(4,2)C(8,2)
C{2,1}(4,3)C(8,3)
C{2,1}(4,4)C(8,4)
C{2,2}(4,1)C(8,5)
C{2,2}(4,2)C(8,6)
C{2,2}(4,3)C(8,7)
C{2,2}(4,4)C(8,8)
Status
• We have completed the implementation of practically all of our initial language extensions (for IBM SP-2 and Linux Clusters).
• Following the toolbox approach has been a challenge, but we have been able to overcome all obstacles.
Jacobi Relaxation
while dif > epsilon v{2:,:}(0,:) = v{:n-1,:}(p,:); % communication v{:n-1,:} (p+1,:) = v{2:,:} (1,:); % communication v{:,2:} (:,0) = v{:,1:n-1}(:,p); % communication v{:, :n-1}(:,p+1) = v{:,2:}(:,1); % communication
u{:,:}(1:p,1:p) = a * (v{:,:} (1:p,0:p-1) + v{:,:} (0:p-1,1:p)+ v{:,:} (1:p,2:p+1) + v{:,:} (2:p+1,1:p)) ; %computation
dif=max(max( abs ( v – u ) )); v = u ; end
Sparse Matrix Vector Product
A b
P1
P3
P2
P4(Distributed) (Replicated)
Sparse Matrix Vector Product HTA Implementation
c = hta(a, {dist, [1]}, [4 1]);
v = hta(4,1,[4 1]);
v{:} = b;
t = c * v;
r = t(:);
Implementation
Tools
• MATLABTM allows user-defined classes• Functions can be written in C, FORTRAN (mex)• Obvious communication tool: MPI• MATLABTM currently provides us both the
language and the computation engine both in the client and the servers
• Standard MATLABTM extension mechanism: toolbox
MATLABTM Classes
• Have constructor, methods• They are standard MATLABTM functions (possibly
mex files) that can access the internals of the class• Allow overloading of standard operators and
functions• Big problem: no destructors!!• Solution: register created HTAs and apply garbage
collection
Architecture
MATLABTM
HTA Subsystem
MATLABTM
FunctionsMex
Functions
MPIInterface
MPIInterface
MATLABTM
HTA Subsystem
MexFunctions
MATLABTM
Functions
Client Server
HTAs Currently Supported
• Dense and sparse matrices
• Real and complex double precision
• Any number of levels of tiling
• HTAs must be either completely empty or completely full
• Every tile in an HTA must have the same number of levels of tiling
Communication Patterns
• Client– Broadcasts commands & data– Synchronizes servers sometimes
• Still, servers can swap tiles following commands issued by client
• So in some operations the client is a bottleneck in the current implementation, while in others the system is efficient
Communication Implementation
• No MPI collective communications used
• Individual messages
• Extensive usage of non blocking sends
• Every message is a vector of doubles– Internally MATLABTM stores most of its data
as doubles– Faster/easier development
Conclusions
• We have developed parallel extensions to MATLAB.
• It is possible to write highly readable parallel code for both dense and sparse computations with these extensions.
• The HTA objects and operations have been implemented as a MATLAB toolbox which enabled their implementation as language extensions.
top related