1ITCS 4/5010 CUDA Programming, UNC-Charlotte, B. Wilkinson, Jan 22, 2013MemCoalescing.ppt

Memory Coalescing

These notes will demonstrate the effects of memory coalescing

Use of matrix transpose to improve matrix multiplication performance

2

Memory coalescing is combining separate memory accesses into one combined access – it is done by the GPU when the locations are sequential locations in global memory banks.

Consider setting the elements of two-dimensional array to given data values.

This could be done across rows or down columns

In the following code, we will demonstrate the effects of each approach

3

Load numbers into a two-dimensional array

Flattened global threadID of thread loaded into array element being accessed so one can tell which thread accesses which location when one prints out array

Access done across rows and also across column and time of execution compared. In practice, a problem may dictate the access order

GPU structure -- one or more 2-D blocks in a 2-D grid. Each block is 2-D 32x32 threads fixed (max. for compute cap. 2.x)

Approach

4

__global__ void gpu_Comput1 (int *h, int N, int T) {

int col = threadIdx.x + blockDim.x * blockIdx.x;

int row = threadIdx.y + blockDim.y * blockIdx.y;

int threadID = col + row * N; // thread ID

int index = col + row * N; // array index

for (int t = 0; t < T; t++) // loop to reduce other time effects

h[index] = threadID; // load array with global thread ID

}

One way

5

__global__ void gpu_Comput2 (int *h, int N, int T) {

int col = threadIdx.x + blockDim.x * blockIdx.x;

int row = threadIdx.y + blockDim.y * blockIdx.y;

int threadID = col + row * N; // thread ID

int index = row + col * N; // array index

for (int t = 0; t < T; t++) // loop to reduce other time effects

h[index] = threadID; // load array with global thread ID

}

Another way

6

/* ------------------------- GPU Computation 1 -----------------------------------*/

gpu_Comput1<<< Grid, Block >>>(dev_h, N, T); // launch once kernel outside timing

cudaEventRecord( start, 0 );

gpu_Comput1<<< Grid, Block >>>(dev_h, N, T);

cudaEventRecord( stop, 0 ); // measure end time

cudaEventSynchronize( stop ); // wait for event recording

cudaEventElapsedTime( &elapsed_time_ms1, start, stop );

cudaMemcpy(h,dev_h, size ,cudaMemcpyDeviceToHost); //Results to check

printf("\nComputation with memory coalescing possible\n");

printArray(h,N);

printf("\nTime to calculate results on GPU: %f ms.\n", elapsed_time_ms1);

Computation 2 similar

7

Some results

A grid of one block and one iterationArray 32x32

No speedup recorded because time of other operations dominate execution time

8

A grid of one block and 1000000 iterations

Array 32 x 32

Speedup = 17.16

9

Repeat just to check results are consistent

10

A grid of 16 x 16 blocks and 10000 iterations

Array 512x512

Speedup = 12.08

Different numbers of iterations produce similar results

11

Array size Speedup

32 x 32 16.70

64 x 64 15.11

128 x 128 15.12

256 x 256 11.85

512 x 512 12.03

1024 x 1024 11.75

2048 x 2048 11.80

4096 x 4096 11.90

1000 iterations. Block size 32 x 32. Number of blocks to suit array size

Different Array Sizes

12

Effects of memory access in matrix multiplication

One thread is responsible for computing one result C ij and needs access a row of A and a column of B:

Thread

Each thread access one row of A and one column of BN2 row/column combinations, N2 threads

13

Seen another way, in first time period, each thread accesses the first element in a row of A:

Thread 0, …

Thread I, …

Thread N-1, …

Consider those threads that access different rowsGiven the row-major order of how A is stored, those threads will locations are not in consecutive locations

– Bad cannot do memory coalescing.

Question: how many threads access the same location?

14

Next, each thread accesses the first element in a column of B:

Thread 0, … Thread I, … Thread N-1, …

Consider those threads that access different columnsGiven the row-major order of how A is stored, those threads will locations are in consecutive locations.

– Good! Can do memory coalescing.

Question: how many threads access the same location?

15

How can we get better memory accesses and memory coalcesing?

1. Transpose one array Copy all rows of A to columns and all columns of A to rows before access A and modify program according.

(Not mentioned in course textbook or other NVIDIA book, although appears obvious way – see next about whether works!)

16

Sequential code for a transpose using same array:

for (i=0; i < N; i++) for (j=0; j < i; j++) {

temp = B[i][j];B[i][j] = b[j][i];B[j][i] = temp;

}

(In my code, I use separate arrays)

Could be done on host prior to copying to device.How would the code look like if on device?

17

/* ------ COMPUTATION DONE ON GPU USING A TRANSPOSED ARRAY-----*/

transposeArray(a, a_T, N); // transpose array

cudaEventRecord(start, 0); // here time measured before // host-device copy, but not transpose

// cudaEventSynchronize(start); // Needed?

cudaMemcpy(dev_a, a_T , size ,cudaMemcpyHostToDevice);// cpy transp. AcudaMemcpy(dev_b, b , size ,cudaMemcpyHostToDevice); // copy B

gpu_matrixmult_T<<<Grid,Block>>>(dev_a,dev_b,dev_c,N);

cudaMemcpy(c_T,dev_c, size ,cudaMemcpyDeviceToHost);

cudaEventRecord(stop, 0); // measure end timecudaEventSynchronize(stop);cudaEventElapsedTime(&elapsed_time_ms2, start, stop );

printf("Time to calculate results on GPU with transposed array: %f ms.\n", elapsed_time_ms2); // print out execution time

18

Some results

8 x 8 array

1 blockof 8 x 8 threads

Speedup = 1.62 over not transposing array

19

Some results

32 x 32 array

1 blockof 32 x 32 threads

Speedup = 1.17 over not transposing array

20

Some results

256 x 256 array

8 blocksof 32 x 32 threads

Speedup = 0.89!! over not transposing array

21

Some results

1024 x 1024 array

32 blocksof 32 x 32 threads

Speedup = 0.93!! over not transposing array

Questions