using empirical roofline toolkit and nvidianvprof · 2018. 2. 9. · gsrb_fp 240648192 4566618...
TRANSCRIPT
Using Empirical Roofline Toolkit and Nvidia nvprof
Protonu Basu, Samuel Williams, Leonid OlikerLawrence Berkeley National Laboratory
ERT Results from a SummitDev Node
10
100
1000
10000
100000
0.01 0.1 1 10 100
GFL
OPs
/ se
c
FLOPs / Byte
Empirical Roofline Graph (Results.summitdev.ccs.ornl.gov.02.MPI4/Run.001)
17904.6 GFLOPs/sec (Maximum)
L1 - 6
506.5
GB/s
DRAM - 192
9.7 GB/s
https://bitbucket.org/berkeleylab/cs-roofline-toolkit
Components of the ERT
ERT_RESULTS Results.summitdevERT_DRIVER driver1ERT_KERNEL kernel1.....
User Written ERT Script
ERT Kernel
ERT Driver
10
100
1000
10000
100000
0.01 0.1 1 10 100
GFL
OPs
/ se
c
FLOPs / Byte
Empirical Roofline Graph (Results.summitdev.ccs.ornl.gov.02.MPI4/Run.001)
17904.6 GFLOPs/sec (Maximum)
L1 - 6
506.5
GB/s
DRAM - 192
9.7 GB/s
JSON file
Driver
int main () {#pragma omp parallel private(id){uint64_t n, t;initialize(&A[nid]);for (n = 16; n < SIZE; n *= 1.1) {
for (t = 1; t < TRIALS; t *= 2) {// start timer hereKernel(n, t, &A[nid]);// stop timer here#pragma omp barrier#pragma omp master {
MPI_Barrier(MPI_COMM_WORLD);}
}}}
Driver
Sync
Init
Compute
Kernel
void Kernel (uint64_t size, unit64_t trials, double * __restrict__ A) {
double alpha = 0.5;uint64_t i, j;for (j = 0; j < trials; ++j ) {
for (i = 0; i < nsize; ++i) {A[i] = A[i] + alpha;
}alpha = alpha * 0.5;
}}
Bandwidth
Kernel
void Kernel (uint64_t size, unit64_t trials, double * __restrict__ A) {
double alpha = 0.5;uint64_t i, j;for (j = 0; j < trials; ++j ) {
for (i = 0; i < nsize; ++i) {double bete = 0.8;#if FLOPPERITER == 2beta = beta * A[i] + alpha;
#elif FLOPPERITER == 4beta = beta * A[i] + alpha;beta = beta * A[i] + alpha;#elif FLOPPERITER == 8…#endifA[i] = beta;
}alpha = alpha * 0.5;
}}
GFlops
Configuration Script for SummitDevERT_RESULTS Results.summitdevERT_DRIVER driver1ERT_KERNEL kernel1
ERT_GPU TrueERT_GPU_CFLAGS -x cu
ERT_MPI TrueERT_FLOPS 1,2,4,8,16,32,64,128,256ERT_ALIGN 32
ERT_CC nvccERT_CFLAGS -O3 -gencode code=sm_60 ...ERT_LD mpiccERT_LDFLAGS ...
ERT MPI_PROC 4
ERT_RUN mpirun -np ERT_MPI_PROCS -map-by ppr:4:node --bind-to none -gpu -display-map ERT_CODE
ERT_BLOCKS_THREADS 28672ERT_GPU_BLOCKS 28,56,112,224,448ERT_GPU_THREADS 64,128,256,512,1024
ERT_NUM_EXPERIMENTS 1ERT_MEMORY_MAX 1073741824ERT_WORKING_SET_MIN 128ERT_TRIALS_MIN 1
ERT_GNUPLOT gnuplot
ERT Results from a SummitDev Node
10
100
1000
10000
100000
0.01 0.1 1 10 100
GFL
OPs
/ se
c
FLOPs / Byte
Empirical Roofline Graph (Results.summitdev.ccs.ornl.gov.02.MPI4/Run.001)
17904.6 GFLOPs/sec (Maximum)
L1 - 6
506.5
GB/s
DRAM - 192
9.7 GB/s
AI of using Nvidia nvprof
Nvidia nvprof enables the collection of a timeline of CUDA-related activities on both CPU and GPU,
including kernel execution, memory transfers, memory set and CUDA API calls
AI of using Nvidia nvprof
To plot roofline (AI) data for a kernel, we need:
Number of floating-point operations executed
Data-volume to and from DRAM or cache
The runtime in seconds
Runtime of Kernel
command: nvprof --print-gpu-trace ./build/bin/hpgmg-fv 6 8
output:
Time Calls Avg Min Max2.52256s 1764 1.4300ms 1.4099ms 1.4479ms
Namevoid smooth_kernel<int=7,int=16,int=4,int=16>
(level_type, int, int, double, double, int, double*, double*)
Nvidia nvprof is used in the gpu-trace mode
Using nvprof to Collect Metrics
command: nvprof --kernels "smooth_kernel” --metrics flop_count_dp--metrics dram_read_throughput--metrics dram_write_throughput--metrics dram_read_transactions--metricsdram_write_transactions./build/bin/hpgmg-fv 6 8
Use the nvprof metric summary mode to collect:
Flops
DRAM read/write throughput
DRAM read/write transactions
Using nvprof to Collect Metrics
Kernel: void smooth_kernel<int=7, int=32, int=4, int=16>(level_type, int, int, double, double, int,
double*, double*)
Invocations Metric Name Min Max Avg
1764 flop_count_dp 240648192 240648192 240648192
1764 dram_read_throughput 299.98 GB/s 307.48GB/s 303.72 GB/s
1764 dram_write_throughput 40.102GB/s 41.099GB/s 40.578GB/s
1764 dram_read_transactions 4537918 4599890 4567973
1764 dram_write_transactions 606387 611691 610299
command: nvprof --kernels "smooth_kernel”
--metrics flop_count_dp
--metrics dram_read_throughput
--metrics dram_write_throughput
--metrics dram_read_transactions
--metricsdram_write_transactions
./build/bin/hpgmg-fv 6 8
Computing Arithmetic Intensity
Method 1
FP / ( DR + DW ) * (size of transaction = 32 Bytes)
Method 2
FP / (TR + TW) * time taken by kernel
FP = double precision opsDR/DW= dram read/write transactionsTR/TW= dram read/write throughput
Gauss-Seidel Red-Black (GSRB)
Red Sweep: update red points Black Sweep: update black points
Two Flavors of GSRB (GSRB_BRANCH)
template<intLOG_DIM_I,intBLOCK_I,intBLOCK_J,intBLOCK_K>__global__ void smooth_kernel(…){
int i=ilo+threadIdx.x;int j=jlo+threadIdx.y;
for (k=klow; k<klow+kdim; k++){int ijk = i+j*jStride+k*kStride;if ((i^j^k^color^1)&1){
double Ax=apply_op_ijk(); //7-point VC stencildouble lambda=Dinv_ijk();xo[ijk]=X(ijk)+lambda*(rhs[ijk]-Ax);
}}The if-condition blocks unnecessary computation
Two Flavors of GSRB (GSRB_FP)
template<intLOG_DIM_I,intBLOCK_I,intBLOCK_J,intBLOCK_K>__global__ void smooth_kernel(…){
int i=ilo+threadIdx.x;int j=jlo+threadIdx.y;
for (k=klow; k<klow+kdim; k++){Int ijk = i+j*jStride+k*kStride;double* RedBlack=level.RedBlack_FP
+ghosts*(1+jStride)
+((k^color000)&1)*kStride;double Ax=apply_op_ijk(); //7-point VC stencildouble lambda=Dinv_ijk();xo[ijk]=X(ijk)+RedBlack [ij] lambda*(rhs[ijk]-Ax);
}} RedBlack mask is used to block unnecessary writes
Results from GSRB on SummitDev GPUs
flopsexecuted
Read Transactions
WriteTransactions
Data Movement (Bytes)
GSRB_FP 240648192 4566618 610123 165655712
GSRB_BRANCH 120061952 9306276 613283 317425888
Thread Block (I=32, J=4)
ERT BANDWIDTH : 497 GB/s
GSRB_FP READ+WRITE THROUGHPUT: 371 GB/s
Results from GSRB on SummitDev GPUs
flopsexecuted
Read Transactions
WriteTransactions
Data Movement (Bytes)
GSRB_FP 240648192 4566618 610123 165655712
GSRB_BRANCH 120061952 9306276 613283 317425888
Thread Block (I=32, J=4)
flops(requisite)
Ratio Data Movement (Bytes)Requisite
Ratio
GSRB_FP 119537664 2.013157895 128798208 1.286164727
GSRB_BRANCH 119537664 1.004385965 110398464 2.875274497
Requisite flops computed for GSBR_branch