implementationandoptimizationofacfdsolverusing...

Research ArticleImplementation and Optimization of a CFD Solver UsingOverlapped Meshes on Multiple MIC Coprocessors

Wenpeng Ma 1 Wu Yuan2 and Xiaodong Hu2

1College of Computer and Information Technology Xinyang Normal University Henan 464200 China2Computer Network Information Center Chinese Academy of Sciences Beijing 100190 China

Correspondence should be addressed to Wenpeng Ma mawpxynueducn

Received 2 February 2019 Revised 27 March 2019 Accepted 23 April 2019 Published 27 May 2019

Academic Editor Basilio B Fraguela

Copyright copy 2019WenpengMa et al )is is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

In this paper we develop and parallelize a CFD solver that supports overlapped meshes on multiple MIC architectures by usingmultithreaded technique We optimize the solver through several considerations including vectorization memory arrangementand an asynchronous strategy for data exchange on multiple devices Comparisons of different vectorization strategies are madeand the performances of core functions of the solver are reported Experiments show that about 316x speedup can be achieved forthe six core functions on a single Intel Xeon Phi 5110P MIC card and 59x speedup can be achieved using two cards compared toan Intel E5-2680 processor for two ONERA M6 wings case

1 Introduction

Computing with accelerators such as graphics processing unit(GPU) [1] and Intel many integrated core (MIC) architecture[2] has been attractive in computational fluid dynamics(CFD) areas recent years because it provides researchers withthe possibility of accelerating or scaling their numerical codesby various parallel techniques Meanwhile the fast develop-ment of computer hardware and the emerging techniquesrequire researches to explore suitable parallel methods forapplications in engineering Intel MIC architecture consists ofprocessors that inherit many key features of Intel CPU coreswhich makes the code migrating less expensive and becomepopular in the development of parallel algorithms

Many CFD-based codes or solvers have been studied onIntel MIC architecture Gorobets et al [3] used variousaccelerators including AMD GPUs NVIDIA GPUs andIntel Xeon Phi coprocessors to conduct direct numericalsimulation for turbulent flows and compared the resultsfrom these accelerators Farhan et al [4] utilized native andoffload mode of MIC programming model to parallelize theflux kernel of PETSc-FUN3D and they obtained about 38xspeedup with offload mode and 5x speedup with nativemode by exploring a series of shared memory optimization

techniques Graf et al [5] ran their PDE codes on single IntelXeon Phi Knights Landing (KNL) node and multiple KNLnodes respectively by using MPI +OpenMP programmingmodel with different thread affinity types Cai et al [6] cal-culated the nonlinear dynamic problems on Intel MIC byemploying offload mode and overlapped data transferstrategy and they obtained 17x on MIC over sequentialversion on the host for the simulation of a bus model Sainiet al [7] investigated various numerical codes on MIC to seekperformance improvement with different host-coprocessorcomputingmodes comparisons and they also presented load-balancing approach for symmetric use of MIC coprocessorsWang et al [8] reported the large-scale computation of a high-order CFD code on Tianhe-2 supercomputer that consists ofboth CPU and MIC coprocessors And other CFD-relatedworks on Intel MIC architecture can be found in references[9ndash12] Working as coprocessors GPUs also have beenpopular in CFD Many researchers [13ndash17] have studied GPUcomputing on structured meshes which involved coalescedcomputation technique [13] heterogeneous algorithm[15 17] numerical methods [16] etc Corrigan et al [18]investigated an Euler solver on GPU by employing un-structured grid and gained important factor of speedup overCPUs )en a lot of results included data structure

HindawiScientific ProgrammingVolume 2019 Article ID 4254676 12 pageshttpsdoiorg10115520194254676

optimization [19 20] numerical techniques [21] and ap-plications [22] based on unstructured meshes on the GPUplatform For GPU simulations on overlapped (overset)meshes Soni et al [23] developed a steady CFD solver onunstructured overset meshes by using GPU programmingmodel which accelerated both the procedure of grid (mesh)assembly and the procedure of numerical calculations )enthey extended their solver to unsteady ones [24 25] to maketheir GPU implementation capable of handling dynamicoverset meshes More CFD-related computing on GPUs canbe found in an overview reference [26]

However most of the existing works either used con-sistent structured or unstructured meshes without meshoverlapping over the computational domain on MIC archi-tecture or studied overlapped meshes on GPUs A majority ofCFD simulations involving overlapped meshes were imple-mented or developed on distributed system through messagepassing interface (MPI) [27] without using coprocessors inpast several decades Specifically Djomehri and Jin [28] re-ported the parallel performance of an overset solver using ahybrid programming model with both MPI [27] andOpenMP [29] Prewitt et al [30] conducted a review ofparallel implementations using overlapped mesh methodsRoget and Sitaraman [31] developed an effective overlappedmesh assembler and investigated unsteady simulations onnearly 10000 CPU cores Zagaris et al [32] discussed a rangeof problems regarding parallel computing for moving bodyproblems using overlapped meshes and presented the pre-liminary performance on parallel overlapped grid assemblyOther overlapped mesh-related works can be found in[33ndash36] Although the work in [7] conducted tests by using asolver that is compatible with overlapped meshes on MICcoprocessors the way it accessed multiple MIC coprocessorsis through the native mode or symmetric mode of MIC Inthis paper we focus on the use of offload mode of MIC andinvestigate the parallelization of a solver with overset mesheson a single host node with multiple MIC coprocessors )econtributions of this work are as follows

(i) We parallelize an Euler solver using overlappedmeshes and propose an asynchronous strategy forcalculations on multiple MIC coprocessors within asingle host node

(ii) We investigate the performances of core functionsof the solver by employing offload mode with dif-ferent thread affinity types on the MIC architectureand we make detailed comparisons between theresults obtained by Intel MIC vectorization and thatobtained by Intel SSE vectorization

(iii) A speedup of 59x can be obtained on two MICcoprocessors over a single Intel E5-2680 processorfor two M6 wings case

)e remainder of the paper is as follows We first in-troduce the MIC architecture and programming modelAnd this is followed by equations and numerical algorithmsthat have been implemented in the solver In Section 4we discuss implementation and optimization aspects in-cluding data transfer vectorization and asynchronous data

exchange strategy on multiple MIC coprocessors )e per-formances of core functions by using different thread affinitytypes are reported and comparisons are made in Section 5)e last section summarizes our work

2 MIC Architecture and Programming Model

Many integrated core (MIC) architecture [2] is a processorthat is capable of integrating many times86 cores providing thecomputing power of high parallelism )e architecture usedin the first Intel Xeon Phi product is called Knights CornerKNC )e KNC coprocessors can have many (up to 61)double dispatched in-order executing times86 computing coresEach core has a 512 bit vector processing unit (VPU) whichsupports 16 single or 8 double floating point operations percycle and each core is able to launch 4 hardware threads32KB L1 code cache 32KB L1 data cache and 512KB L2cache are available to each core)e coprocessor used in thiswork is Intel Xeon Phi 5110P [37] which consists of 60 coreseach of which runs at 105GHz It can launch a total of 240threads simultaneously And 8GB of GDDR5 memory isavailable on it

Anyone who is familiar with C C++ or Fortran pro-gramming language can develop codes onMIC coprocessorswithout major revision of their source codes MIC providesvery flexible programming models including native hostmode native MIC mode offload mode and symmetricmode [38] Coprocessors are not used in native host modeand programmers can run their codes on CPUs just like theydo before the MIC architecture was introduced By contrastcodes can be conducted only on coprocessors in native MICmode when they are compiled with ldquo-mmicrdquo optionSymmetric mode allows programmers to run codes on bothCPU cores and coprocessors And offload mode is mostcommonly used on a single coprocessor or multiple co-processors within a single host node)e basic use of offloadmode for programmers is to write offload directives to makethe code segment run on MIC coprocessors To take fulladvantage of computational resources on MIC the codesegment can be conducted in parallel by employing mul-tithreading techniques such as OpenMP [29]

3 Equations and Numerical Algorithms

31 Compressible Euler Equations )e three-dimensionaltime-dependent compressible Euler equations over a con-trol volume Ω can be expressed in integral form as

z

zt1113946ΩW dΩ + 1113929

zΩFcdS 0 (1)

where W [ρ ρu ρv ρw ρE]T represents the vector of con-servative variables and Fc [ρV ρuV + nxp ρvV + nyp

ρwV + nzp ρHV]T denotes the vector of convective fluxeswith V nxu + nyv + nzw

32 Numerical Algorithms )e flux-difference splitting(FDS) [39] technique is employed to calculate the spatialderivative of convective fluxes In this method the flux at theinterface (for example i direction) expressed by Fiplusmn(12)jk

2 Scientific Programming

can be computed by solving an approximate Riemannproblem as

Fi+(12)jk 12

F WL( 1113857 + F WR( 1113857minus Ainv1113868111386811138681113868

1113868111386811138681113868 WR minusWL( 11138571113960 1113961 (2)

where the left and right state of q qL and qR are constructedby Monotonic Upstream-Centered Scheme for Conserva-tion Laws (MUSCL) [40] and min-mod limiter

Equation (1) is solved in time in this work by employingan implicit approximate-factorization method [41] whichachieves first-order accuracy in steady-state simulations

Euler wall boundary conditions (also called inviscidsurface conditions) inflowoutflow boundary conditionsand symmetry plane boundary conditions are applied toequation (1) by using the ghost cell method [42]

33 Mesh Technique In this work we aim to solve 3D Eulerequations on multiple MIC devices by using overlappedmesh technique In this technique [33 42] each entity orcomponent is configured by a multiblock mesh system anddifferent multiblock mesh systems are allowed to overlapwith each other During the numerical calculations over-lapped regions need to receive interpolated informationfrom each other by using interpolationmethods [43 44])eprocess of identifying interpolation information amongoverlapped regions termed as mesh (grid) assembly[31ndash34 45] has to be employed before starting numericalcalculations )is technique has reduced the difficulty ofgenerating meshes for complex geometries in engineeringareas because an independent mesh system with high meshquality can be designed for a component without consid-ering other components However it adds the complexity ofconducting parallel calculations (via MPI [27] for example)on this kind of mesh system As mesh blocks are distributedon separated processors where data can not be shared di-rectly more detailed work should be done on data exchangeamong mesh blocks )ere are two types of data commu-nication for overlapped mesh system when calculations areconducted on distributed computers One is the data ex-change on the shared interfaces where one block connects toanother and the other is the interpolation data exchangewhere one block overlaps with other blocks And the dis-cussion of both types of communication is going to becovered in this work

4 Implementation

41 Calculation Procedure )e procedure of solvingequation (1) depends mainly on the mesh techniqueemployed In the overlapped mesh system the steps ofcalculations are organized in Listing 1 As described inSection 33 performing mesh assembly is a necessary step(Listing 1 line 1) before conducting numerical computingon an overlapped mesh system Mesh assembly identifiescells which need to receive interpolation data (CRI) as well ascells which provide interpolation data (CPI) for CRIs inoverlapped regions and creates a map to record where thedata associating with each block should be sent to or received

from)is data map stays unchanged during the steady-stateprocedure and is used for data exchange (Listing 1 line 8)within each iteration )en the mesh blocks that connectwith other blocks need to share the solutions at theircommon interfaces When the data from blocks that provideinterpolation and from neighbouring blocks are ready aloop (Listing 1 lines 10ndash13) is launched to compute fluxesand update solutions on each block one by one

Figure 1 illustrates how we exploit multithreadingtechnique to perform all operations in Listing 1 on a singlecomputing node with multiple MIC coprocessors Eventhough more than one MIC coprocessor can be assembledon the same node they are unable to communicate with eachother directly )ey must communicate through the hostnode and it is expensive for data to be moved between MICdevices and the host node because of the limited PCI-Ebandwidth )at requires us to work out data transferstrategy in order to achieve better performance on MICcoprocessors as two types of communication occur in everysingle iteration

Our first consideration is to locate each mesh blockcluster (MBC) that is associated with a component or entityon a specific MIC coprocessor )is benefits from the naturefeature of the overlapped mesh system (Section 33)employed in this work Since a MBC consists of one-to-onematched blocks only it does not involve operations of in-terpolation over all mesh blocks it contains So this way ofdistributing computational workload to MIC coprocessorsavoids one-to-one block data exchange across MIC co-processors which reduces the frequency of host-devicecommunication However the data transfer from oneblock to all the blocks it overlaps across MIC devices isinevitable For data transfer over overlapped regions acrossdifferent MIC devices we proposed an algorithm of com-munication optimization which will be introduced anddiscussed in Section 44

As shown in Figure 1 a bunch of OpenMP threads arecreated on the host node and each thread is associated with aMIC coprocessor and responsible for the data movementand computation of at least one MBC )e operations ofphysical boundary conditions (lines 5ndash7) the data exchangeon one-to-one block interfaces (line 9) over a MBC and themost time consuming part (lines 10ndash13) in Listing 1 areconducted on the corresponding MIC device For steadycalculations in this work the host calls the process of meshassembly [46 47] only once to identify the CRIs and CPIs foreach mesh block and prepare the data structure in over-lapped regions At the end of each steady cycle the updatedsolutions of CPIs are used to calculate update the in-terpolation data and then copied to the host memory )ehost collects the data redistributes it for CRIs and copies theCRIs with new interpolated values back into the MICmemory When the interpolated data are ready a cycle ofspatial and temporal calculations can be performed on meshblocks one by one without any data dependence

42 Memory Arrangement A MIC coprocessor has its ownmemory space and independent memory allocation and

Scientific Programming 3

arrangement have to be performed for the computationsthat are offloaded to a MIC device More importantlymanipulating memory frequently and dynamically mighthave a negative effect on the overall efficiency and it is theprogrammerrsquos responsibility to control and optimize thememory usage

In the computational procedure of solving equation (1)the core calculations including boundary conditions spatialdiscretizations and advancements in time just involveupdating the values in arrays that can be used in differentsubroutines during the repeated cycles )erefore the wholeprocedure conducts calculations on allocated variables duringthe cycle and then outputs them at the end of the process)isfact inspires us to use dec$ offload begin target(mici) and dec$ end offload to include all the variables that need to be

allocated on MIC or copied from host before the cycle starts)e clause of in(varnamelength(len)) is used with alloc_if(-true) free_if(false) options to allocate memory for a variableand initialize it with the values on the host by memory copyHowever there are a lot of variables that are used as tem-porary spaces and do not have to be initialized In this case weuse the keyword of nocopy(nocopy(tmplength(len_tmp)alloc_if(true) free_if(false))) to avoid extra memory copiesbetween the host and a MIC coprocessor When all thecalculations are completed out clause is declared with allo-c_if(false) free_if(true) options and placed between offloadbegin and end offload regions to copy the newest conservativeflow variables to the host memory and free the variables in thedevice memory)e memory arrangement described above isillustrated in Listing 2

Configuration fileand overlapped

mesh data

MIC_0 MIC_n

Interpolation data

Thread_0 Thread_i Thread_n

Mesh assembly Host

Initialization

Spatial and temporalcalculation

Data exchange

MBC_n

Update interpolation data

Boundary conditionsReceiving CRIs


Data exchange

MBC_0 MBC_1



Output file

Figure 1 Flow chart of calculations on multiple MIC coprocessors

(1) Conduct mesh assembly to obtain interpolation information over the mesh system (MS)(2) Read flow configuration file and overlapped mesh data(3) Initialization(4) repeat(5) for each iblock in MS do(6) Apply boundary conditions to iblock(7) end for(8) Ws(CRI) f(Wt(CPI)) exchange interpolated data between overlapped block pair (s t) where CRI cells receiving

interpolation data and CPI cells providing interpolation data(9) Wi(ghostcells)⟵Wj andWj(ghostcells)⟵Wi exchange each one-to-one block data at block interface connecting blocki

and blockj

(10) for each iblock in MS do(11) Calculate Fi

c Fjc and Fk

c fluxes on iblock(12) AF time advancement and update solutions on iblock(13) end for(14) Until convergence(15) Output flow data

LISTING 1 Procedure of solving equation (1) on overlapped mesh system


Inside the cycle many MIC kernels are launched by thehost and the nocopy keyword is used again to keep the datadeclared out of the cycle persistent across offloading pro-cedure For example as shown in Listing 2 Fluxj_mic isdeclared as a function that can be called onMIC directly andit reuses the array mbc_n which is declared and allocatedoutside the cycle without any extra memory operationsduring the cycle Andmbc_n can either be copied out to hostmemory or just deallocated on the coprocessor at the endpoint of the cycle )is concept of memory arrangementwhich has been applied in other applications [6 7] preventsfrequent data copies inside the cycle from affecting theoverall efficiency

43 Vectorization Vectorization is an effective way to im-prove computing speed on CPU by using Streaming SIMD(single instruction multiple data) Extension (SSE) Simi-larly MIC coprocessors support 512 bit wide Knights Cornerinstructions which allow 16 single or 8 double floating pointoperations at the same time In this section we explorevector optimization to make full use of MIC computationalresources

Generally there are two different ways to implement theflux calculations )e first method is to perform a nested loopover all mesh cells (or mesh nodes) It calculates all the fluxesthat go in or out of a cellrsquos interfaces and accumulates them forthe cell And the vector operations can be applied to theinnermost loop by using Intel autovectorization or manualSIMD directives However this algorithm involves redundantcomputations in which the flux of an interface which isshared by two cells is computed twice Literatures [18] haveshown that redundant computing was not harmful to GPUimplementation because it can hide the latency of globalmemory accesses on GPUs An alternative way of fluxcomputation which is more popular for CPU computing andemployed in this work is to evaluate the flux of every edgeonly once by performing three nested loops along three di-rections And then the fluxes of edges are accumulated intothe cells that contain them However there are still different

coding strategies for the later techniqueWe investigate two ofthem and compare the effects on vectorizations

One of the coding strategies (CS-1) is to conduct fluxcomputing along different directions For example eachthread is responsible for visiting a piece of the array ofsolution along i direction denoted as w(j k 1 id n) whenFi is considered As discontinuous memory accesses occurwhen higher dimensions of w are accessed loading the pieceof w into a one-dimensional temporary array at the be-ginning is an effective way to reduce cache misses in thefollowing process And the remaining flux calculations canbe concentrated on the temporary array completely )entwo directives dir$ ivdep or dir$ simd can be declared tovectorize all the loops regarding the temporary arrayFigure 2(a) shows the pseudocode of this strategy )eadvantage of this consideration is to keep the code of fluxcomputation along different directions almost same andreduce the workload of redevelopment Another method(CS-2) refers to performing Fj Fk and Fi fluxes along thelowest dimension of w Fluxes along different directions arecomputed by the nested loops with the same innermost loopto guarantee that the accesses of continuous memory holdno matter which direction is under consideration Howeverthe body codes inside the nested loops vary from one di-rection to another In this circumstance vectorization di-rectives are applied to the innermost (the lowest dimension)loop and the two nested loops outside are unpacked man-ually to increase the parallelism [7] or applied to the twoinner loops by merging two loops to make full use of theVPU on MIC )e method merging the two inner loops isemployed in the present paper and the pseudocode is shownin Figure 2(b) It should be noted that the Knights Cornerinstructions [38] which can be applied to perform vecto-rization more elaborately are not involved in this workbecause this assembly-level optimization may result in poorcode readability and make the overhead of code mainte-nance increase

Except for the content of the code inside the loops theprocess of time advancement has the similar structure of

(1) dec$ offload begin target(micn) in(mbc_nlength(len) alloc_if(true) free_if(false))

(2) dec$ end offload(3) repeat(4)

(5) dec$ offload target(micn) nocopy(mbc_nlength(len) alloc_if(false) free_if(false))(6) $omp parallel do private (ik i k)(7) do ik 1 ik_total(8) i (ikminus 1)(kdminus 1) + 1(9) k mod(ikminus 1 kdminus 1) + 1(10) call Fluxj_mic(mbc_n i k )(11) end do(12) $omp end parallel do(13)

(14) until convergence(15) dec$ offload begin target(micn) nocopy(mbc_nlength(len) alloc_if(false) free_if(false))(16) dec$ end offload

LISTING 2 Memory arrangement on MIC


nested loops with fluxes computing)erefore we merge thetwo loops inside to provide larger data set for vectorizationon MIC

44 Host-Device Communication Optimization As de-scribed in Section 41 our task allocation strategy is todistribute workload by MBCs which avoids the data ex-change on one-to-one interfaces across MIC coprocessorsbut inevitably leads to communication across MIC co-processors because of interpolation among MBCs

Although workloads which are located on different MICdevices are performed in parallel they still are dispatched oneby one when they are on the same coprocessor As a MBCcorresponds to a specificMIC device mesh blocks in theMBCare calculated and updated one by one )erefore a meshblock that contains CPIs does not have to wait until thecomputation of other mesh blocks has completed before itneeds to copy data to the host More specifically when thesolution of a mesh block has been updated successfully duringa block loop (Listing 1 line 10) it can start a communicationrequest to copy interpolation data provided by CPIs to thehost immediately And the overhead of copying data from amesh block to the host can be hidden by the flow computingof other mesh blocks in the same device To implement thisidea we take advantage of the asynchronous transfer of theoffload mode Because overlapped mesh that consists of 2MBCs is the most common case we address the pseudocodeof our algorithm about communication optimization in Al-gorithm 1 by using 2 MBCs and 2 MICs

Outside the main procedure of numerical calculations aloop over all MIC devices is performed In order to makeeach device responsible for a part of the whole computationwe use OpenMP directive omp parallel do before the loop tostart a number of threads each of which maps to a deviceAccording to the memory arrangement discussed in Section42 memory allocation and initialization have been done ondevices before the computation starts )erefore by usingthis thread-device mapping each device can access its owndevice memory start computing and use shared hostmemory for necessary data exchange At the end point oftime advancement the clause offload_transfer is used withthe signal option to put a request of data transfer from thedevice to the host and it returns immediately and allows the

device to keep running the next MIC kernels So the datatransfer and the block computing can be conducted at thesame time Once all the MIC kernels in a steady step havefinished the CPU is required to check the signals by usingoffload_wait to make sure all the transfers have been donebefore executing the process of exchanging interpolationdata )en the CPU uses the master thread to distribute theinterpolation data from CPIs to the corresponding positionsof CRIs Vectorization can be employed in this part becausethe operations involve assignment operations only Afterthat the CPU starts data transfer requests to copy data backto each device respectively At the same time the next cyclebegins and the MIC kernel of set_boundary_condition isexpected to run on MIC overlapping with the overhead oftransferring the interpolation data back to MIC devices Insome cases the last mesh block residing on a MIC devicemay have the largest number of CPIs and it is difficult tooverlap the data transfer from device to the host withcomputation because there are no more blocks that need tobe computed As the overlapped region involves only a partof mesh blocks we sort the mesh blocks in the descendingorder in accordance with the number of CPIs to avoidtransferring data with insufficient overlapped computation

5 Experiments

51 Testing Environment Our experiments were conductedon the YUAN cluster at the Computer Network InformationCenter at the Chinese Academy of Sciences )e cluster is ofhybrid architecture that consists of both MIC and GPUnodes )e configuration of MIC nodes is that each node hastwo Intel E5-2680 V2 (Ivy Bridge 28GHz 10 cores) CPUsand two Intel Xeon Phi 5110P MIC coprocessors )ememory capacity for the host and coprocessors is 64GB and8GB respectively Two environmental variables MIC_OMP_NUM_THREADS and MIC_KMP_AFFINITY weredeclared to investigate the performances under differentnumber of OpenMP threads and affinity types Intel Fortrancompiler (version 2013_sp10080) was used with the highestoptimization -O3 and -openmp options to compile thesolver

52 Results We used two ONERA M6 wings (shown inFigure 3) each of which was configured with four 129 times

113 times 105 subblocks )e lower wing and its mesh systemwere formed by making a translation of the upper wingdown along Y-axis by the length of the wing and then thetwo mesh systems overlapped with each other Figure 4shows a closer view of the overlapped mesh system at thesymmetry plane before the mesh assembly Figure 5 shows acloser view of the mesh system after the mesh assembly wasperformed )e mesh assembly procedure automaticallyformed an optimized region where the interpolation re-lationship between the two mesh systems can be identifiedwhich is shown in the middle of the upper and lower wings)e initial condition for this case is as follows Mach numberM 084 angle of attack α 306deg and slip angle β 0deg Asthe host node has two Intel MIC coprocessors we made each

jk_total = (jd ndash 1) lowast (kd ndash 1)do jk = 1 jk_total

k = (jk ndash 1)(jd ndash 1) + 1j = mod(jk ndash 1 jd ndash 1) + 1

dir$ simdtmp() = w(j k n)

dir$ ivdepoperation_1(tmp())


end do

(a)

do k = 1 (kd ndash 1)

tmp() = w( k n)dir$ ivdep

dir$ ivdep


operation_2(tmp())

end do

(b)

Figure 2 Two different vectorization implementations


MIC coprocessor responsible for the workload of one meshsystem )e results presented below start with the perfor-mances of six kernels (three in spatial and three in temporal)running on MIC

Tables 1ndash3 show both the wall clock times for one cycleobtained by different combinations of number of threadsand thread affinity types and that obtained by CPUs )ewall clock times for one cycle were evaluated by averagingthe times of first 100 cycles And the second coding strategyfor vectorization of flux computing was employed in thistest In each table the first column lists the number ofthreads from 4 to 118 which is horizontally followed by twothree-column blocks Each block corresponds to a specifiedMIC kernel and lists wall clock times in different threadaffinity types A column in such a block represents the wallclock times on MIC varying from the number of threadsunder a declared thread affinity type In order to observe thescalability of MIC kernels as the number of threads in-creases we demonstrate the relative speedup as t4tp wheret4 is the wall clock time estimated by using 4 threads and tp is

that estimated by using p threads )e relative speedups arealso filled in each column on the right of wall clock timesFurthermore to compare the performance on the MICarchitecture with that on CPUs we also show the corre-sponding full vectorization optimized CPU time for eachkernel at the last row of each table and calculate the absolutespeedup through dividing the CPU time by the minimumvalue in all three values obtained by 118 threads underdifferent affinity types )e results in all three tables showthat the wall clock times obtained by scatter and balancedmodes have an advantage over that obtained by compactmode )is can be explained by the fact that the compactmode distributes adjacent threads to the same physicalprocessor as much as possible in order to make maximumutilization of L2 cache when adjacent threads need to sharedata with each other but this can result in load imbalanceproblems among physical cores Our implementations havemade each thread load independent data into a temporaryarray and operate the array locally which is more suitable forboth scatter and balanced modes because the local opera-tions in each thread make better use of L1 cache on aprocessor without any intervention from other threads

As the balanced mode distributes threads to processorsin the same way as the scatter mode does when NTle 59 theresults obtained from these two modes are supposed to besame However there exist slight differences between themthrough Tables 1ndash3 because of time collecting errors comingfrom different runs When NT 118 each coprocessor tookresponsibility of two threads in both the balanced and scattermodes but they differed because the balanced mode kepttwo adjacent threads on the same coprocessor whereas thescatter mode kept them on different coprocessor Howeverthe results obtained by these two modes did not show ob-vious gap in this case)is might be caused by the cache raceamong threads on a single MIC processor when it undertookmore than one thread

It is noticed that except Fj the wall clock times of otherfive MIC kernels no longer decrease effectively when more

(1) $omp parallel do private(idev ib )(2) do idev 0 1(3) repeat(4) offload target(micidev) set_boundary_condition for each block(5) if(icycle gt 1) offload target(micidev) wait(sgr(idev)) set_CRI_to_domain(6) offload target(micidev) exchange_interface_data(7) do ib 1 nb(idev)(8) offload target(micidev) spatial_step(9) offload target(micidev) temporal_step(10) offload target(micidev) compute_CPI(11) offload_transfer target(micidev) out(CPIib) signal(sgpidev(ib))(12) end do(13) master thread offload_wait all sgpidev related to each device(14) master thread on CPU exchange_interpolation_data(15) master thread offload_transfer target(micidev) in(CRIimbc) signal(sgr(idev))(16) until convergence(17) $omp end parallel do

ALGORITHM 1 Communication optimization algorithm

X

Y

Z

Figure 3 Two M6 wings


than 59 threads were used in both balanced and scattermodes )is can be explained by the fact that the OpenMPdirectives in these kernels worked at the outmost loop andthe maximum number of workload pieces equals to 128 (or112 104) so it was difficult for each thread to balance theworkload on it when 118 threads were launched And for Fjwe vectorized it along the innermost direction and manuallycombined the two nested looped into one loop to increasethe parallelism by slightly modifying the codes )erefore it

has been showed that Fj scaled better than Fk and Fi did asthe number of threads increased and Fk and Fi have anadvantage in wall clock time due to maximizing the use ofVPU on MIC but at the expense of scalability

A speedup of 25x was obtained on the MIC coprocessorfor TAj which is the lowest speedup among the threekernels )at was mainly caused by the cache misses andpoor vectorization when each thread tried to load the jminus k

plane of the solution array (w(j k i n)) into the temporary

Table 1 Wall clock times for Fj and Fk (in seconds)

NTFj Fk

Scatter Balanced Compact Scatter Balanced Compact

4 180 (10x) 181 (10x) 3256 (10x) 0680 (10x) 0680 (10x) 1668 (10x)8 0932 (193x) 0935 (194x) 1680 (194x) 0352 (193x) 0352 (193x) 0856 (195x)16 0520 (346x) 0519 (349x) 0880 (37x) 0192 (354x) 0194 (351x) 0468 (356x)32 0288 (625x) 0288 (628x) 0488 (667x) 0160 (425x) 0161 (422x) 028 (596x)59 0196 (918x) 0196 (923x) 0296 (110x) 0144 (472x) 0144 (472x) 0224 (745x)118 0144 (125x) 0136 (133x) 0160 (2035x) 0148 (459x) 0148 (459x) 0196 (851x)CPU time 052 (382x) 054 (364x)NT number of threads

Figure 5 Closer view of two overlapped mesh systems after mesh assembly

Figure 4 Closer view of two overlapped mesh systems before mesh assembly


space and reshaped it along k direction)e same reason canexplain the results shown in Figure 6 where the comparisonsbetween two code strategies (CS) for flux computation of Fkand Fi by using balanced mode have been made )e wallclock times for the four kernels were evaluated by per-forming the first 100 steps of the solver CS-1 conducted 10more discontinuous array loadings than CS-2 did whichmade vectorizations far from efficient on MIC coprocessorsAlthough the discontinuous memory accesses were vec-torized by simd clause compulsively they turned out less

effective on MIC coprocessors Also we can observe clearefficiency drops when more than 16 threads were used inCS-1 because we have done 2D vectorization in CS-1 whichmade it hard for MIC to keep balanced workload amongthreads

)en we split each of the original block into 4 subblocksalong both i and j directions and form a mesh system of 16blocks for each wing and report the wall clock times for 500steps by running the case on CPU and two MIC devicesrespectively Figure 7 shows wall clock time comparisons in

4 8 10 16 32 59 118Number of threads

101

102

103

Wal

l clo

ck ti

mes

(sec

onds

)

Fk CS-1Fi CS-1

Fk CS-2Fi CS-2

Figure 6 Comparisons among different implementations in vectorization

Table 2 Wall clock times for Fi and TAj (in seconds)

NTFi TAj



Table 3 Wall clock times for TAk and TAi (in seconds)

NTTAk TAi




different block sizes Solving equation (1) on 8 overlappedmesh blocks using two Intel Xeon Phi 5110P MIC cards canachieve 59x speedup compared to sequential calculationswith full optimizations whereas solving the same problemon 32 overlapped mesh blocks using two MIC cards onlyachieves 36x )is is to be expected because threads areunlikely to keep busy and balanced workload when thedimensions of blocks are relatively small as stated above andthe six core functions can achieve only about 2x speedup ona single MIC device Furthermore it is observed that se-quential calculations using 32 mesh blocks spent about 10more time than that using 8 mesh blocks )is can beexplained by the fact that larger mesh block makes better useof Intel SSE vectorization To show the correctness andaccuracy of the parallel solver on the MIC architecture weplot the pressure contours at the symmetry plane in Figure 8and compare the corresponding pressure coefficients onthe airfoils calculated by MICs with that calculated byCPUs in Figure 9 We can see clearly from Figure 9 that our

solver running on MICs can capture each shockwave whichis expected on each airfoil for this transonic problem andproduce pressure coefficients in agreement with that cal-culated by CPUs

6 Summary

We have developed and optimized an overlapped mesh-supported CFD solver on multiple MIC coprocessors Wedemonstrated and compared different code strategies forvectorizations by using a two M6 wings case and analysedthe results in detail An asynchronous method has beenemployed in order to keep the data exchange from in-terfering the overall efficiency Calculations on the caseachieved 59x speedup using two MIC devices compared tothe case using an Intel E5-2680 processor Our future workincludes extending this solver to cover unsteady flow

100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x

CPU CPU2MICs 2MICs8 blocks 32 blocks

Figure 7 Wall clock times for 500 steps on CPU and MIC

PPinf1495139212881185108209780875077206690565046203590256

Figure 8 Pressure contours at the symmetry plane

ndash05

0

05

1

0 02 04 06 08 1X

Cp

Upper wing MICLower wing MIC

Upper wing CPULower wing CPU

Figure 9 Pressure coefficients comparisons on the airfoils at thesymmetry plane


calculations which involve relative motion among over-lapped mesh blocks on multiple MIC devices

Conflicts of Interest

)e authors declare that they have no conflicts of interest

Acknowledgments

)is work was supported by a grant from the NationalNatural Science Foundation of China (nos 61702438 and11502267) the Key Research Project of Institutions ofHigher Education of Henan Province (no 17B520034) andthe Nanhu Scholar Program of XYNU

References

[1] NVIDIA NVIDIA Tesla V100 GPU Architecture NVIDIASanta Clara CA USA 2017

[2] httpswwwintelcomcontentwwwusenarchitecture-and-technologymany-integrated-coreintel-many-integrated-core-architecturehtml

[3] A Gorobets F Trias R Borrell G Oyarzun and A OlivaldquoDirect numerical simulation of turbulent flows with parallelalgorithms for various computing architecturesrdquo in Pro-ceedings of the 6th European Conference on ComputationalFluid Dynamics Barcelona Spain July 2014

[4] M A A Farhan D K Kaushik and D E Keyes ldquoUn-structured computational aerodynamics on many integratedcore architecturerdquo Parallel Computing vol 59 pp 97ndash1182016

[5] J S Graf M K Gobbert and S Khuvis ldquoLong-time simu-lations with complex code using multiple nodes of Intel XeonPhi knights landingrdquo Journal of Computational and AppliedMathematics vol 337 pp 18ndash36 2018

[6] Y Cai G Li and W Liu ldquoParallelized implementation of anexplicit finite element method in many integrated core (MIC)architecturerdquo Advances in Engineering Software vol 116pp 50ndash59 2018

[7] S Saini H Jin D Jespersen et al ldquoEarly multi-node per-formance evaluation of a knights corner (KNC) based NASAsupercomputerrdquo in Proceedings of the IEEE InternationalParallel amp Distributed Processing Symposium WorkshopChicago FL USA May 2015

[8] Y X Wang L L Zhang W Liu X H Cheng Y Zhuang andA T Chronopoulos ldquoPerformance optimizations for scalableCFD applications on hybrid CPU+MIC heterogeneouscomputing system with millions of coresrdquo Computers ampFluids vol 173 pp 226ndash236 2018

[9] K Banas F Kruzel and J Bielanski ldquoFinite element nu-merical integration for first order approximations on multi-and many-core architecturesrdquo Computer Methods in AppliedMechanics and Engineering vol 305 pp 827ndash848 2016

[10] W C Schneck E D Gregory and C A C Leckey ldquoOpti-mization of elastodynamic finite integration technique onIntel Xeon Phi knights landing processorsrdquo Journal ofComputational Physics vol 374 pp 550ndash562 2018

[11] J M Cebrian J M Cecilia M Hernandez and J M GarcıaldquoCode modernization strategies to 3-D stencil-based appli-cations on Intel Xeon Phi KNC and KNLrdquo Computers ampMathematics with Applications vol 74 no 10 pp 2557ndash25712017

[12] M Lukas Z Jan M Michal et al ldquoEvaluation of the IntelXeon Phi offload runtimes for domain decompositionsolversrdquo Advances in Engineering Software vol 125 pp 46ndash154 2018

[13] S M I Gohari V Esfahanian and H Moqtaderi ldquoCoalescedcomputations of the incompressible NavierndashStokes equationsover an airfoil using graphics processing unitsrdquo Computers ampFluids vol 80 no 1 pp 102ndash115 2013

[14] L Fu K Z Gao and F Xu ldquoAmulti-block viscous flow solverbased on GPU parallel methodologyrdquo Computers amp Fluidsvol 95 pp 19ndash39 2014

[15] W Cao C F Xu Z HWang H Y Liu and H Y Liu ldquoCPUGPU computing for a multi-block structured grid based high-order flow solver on a large heterogeneous systemrdquo ClusterComputing vol 17 no 2 pp 255ndash270 2014

[16] M Aissa T Verstraete and C Vuik ldquoToward a GPU-awarecomparison of explicit and implicit CFD simulations onstructured meshesrdquo Computers amp Mathematics with Appli-cations vol 74 no 1 pp 201ndash217 2017

[17] C Xu X Deng L Zhang et al ldquoCollaborating CPU and GPUfor large-scale high-order CFD simulations with complexgrids on the TianHe-1A supercomputerrdquo Journal of Com-putational Physics vol 278 pp 275ndash297 2014

[18] A Corrigan F F Camelli R Lohner and J Wallin ldquoRunningunstructured grid-based CFD solvers on modern graphicshardwarerdquo International Journal for Numerical Methods inFluids vol 66 no 2 pp 221ndash229 2011

[19] A Lacasta M Morales-Hernandez J Murillo and P Garcıa-Navarro ldquoAn optimized GPU implementation of a 2D freesurface simulation model on unstructured meshesrdquo Advancesin Engineering Software vol 78 pp 1ndash15 2014

[20] P Barrio C Carreras R Robles A L Juan R Jevtic andR Sierra ldquoMemory optimization in FPGA-accelerated sci-entific codes based on unstructured meshesrdquo Journal ofSystems Architecture vol 60 no 7 pp 579ndash591 2014

[21] Y Xia H Luo M Frisbey and R Nourgaliev ldquoA set ofparallel implicit methods for a reconstructed discontinuousGalerkin method for compressible flows on 3D hybrid gridsrdquoin Proceedings of the 7th AIAA Georetical Fluid MechanicsConference Atlanta GA USA 2014

[22] J Langguth N Wu J Chai and X Cai ldquoParallel performancemodeling of irregular applications in cell-centered finitevolume methods over unstructured tetrahedral meshesrdquoJournal of Parallel and Distributed Computing vol 76pp 120ndash131 2015

[23] K Soni D D J Chandar and J Sitaraman ldquoDevelopment ofan overset grid computational fluid dynamics solver ongraphical processing unitsrdquo Computers amp Fluids vol 58pp 1ndash14 2012

[24] D D J Chandar J Sitaraman and D Mavriplis ldquoGPUparallelization of an unstructured overset grid incompressibleNavierndashStokes solver for moving bodiesrdquo in Proceedings of the50th AIAA Aerospace Sciences Meeting Including the NewHorizons Forum and Aerospace Exposition Nashville TNUSA January 2012

[25] D Chandar J Sitaraman and DMavriplis ldquoDynamic oversetgrid computations for CFD applications on graphics pro-cessing unitsrdquo in Proceedings of the Seventh InternationalConference on Computational Fluid Dynamics Big IslandHawaii July 2012

[26] K E Niemeyer and C-J Sung ldquoRecent progress and chal-lenges in exploiting graphics processors in computationalfluid dynamicsrdquo Journal of Supercomputing vol 67 no 2pp 528ndash564 2014


[27] G Edgar E F Graham B George et al ldquoOpen MPI goalsconcept and design of a next generation MPI implementa-tionrdquo in Proceedings of the 11th European PVMMPI UsersGroup Meeting pp 97ndash104 Budapest Hungary September2004 httpwwwopen-mpiorg

[28] M J Djomehri and H Jin ldquoHybrid MPI+OpenMP pro-gramming of an overset CFD solver and performance in-vestigationsrdquo NASA Technical Report NASA Ames ResearchCenter Moffett Field CA USA 2002

[29] B Chapman G Jost and R van der Pass Using OpenMPPortable Shared Memory Parallel Programming )e MITPress Cambridge MA USA 2007

[30] N C Prewitt DM Belk andW Shyy ldquoParallel computing ofoverset grids for aerodynamic problems withmoving objectsrdquoProgress in Aerospace Sciences vol 36 no 2 pp 117ndash1722000

[31] B Roget and J Sitaraman ldquoRobust and efficient overset gridassembly for partitioned unstructured meshesrdquo Journal ofComputational Physics vol 260 pp 1ndash24 2014

[32] G Zagaris M T Campbell D J Bodony et al ldquoA toolkit forparallel overset grid assembly targeting large-scale movingbody aerodynamic simulationsrdquo in Proceedings of the 19thInternational Meshing Roundtable pp 385ndash401 SpringerBerlin Heidelberg October 2010

[33] J Cai F Tsai and F Liu ldquoA parallel viscous flow solver onmulti-block overset gridsrdquoComputers amp Fluids vol 35 no 10pp 1290ndash1301 2006

[34] B Landmann and M Montagnac ldquoA highly automatedparallel Chimera method for overset grids based on theimplicit hole cutting techniquerdquo International Journal forNumerical Methods in Fluids vol 66 no 6 pp 778ndash804 2011

[35] W D Henshaw ldquoSolving fluid flow problems on moving andadaptive overlapping gridsrdquo in Proceedings of the In-ternational Conference on Parallel Computational FluidDynamics Washington DC USA May 2005

[36] W Liao J Cai and H M Tsai ldquoA multigrid overset grid flowsolver with implicit hole cutting methodrdquo Computer Methodsin Applied Mechanics and Engineering vol 196 no 9ndash12pp 1701ndash1715 2007

[37] httpsarkintelcomproducts71992Intel-Xeon-Phi-Coprocessor-5110P-8GB-1-053-GHz-60-core-

[38] E Wang Q Zhang B Shen et al High-Performance Com-puting on the Intelreg Xeon Phitrade How to Fully Exploit MICArchitectures Springer Berlin Germany 2014

[39] P L Roe ldquoApproximate Riemann solvers parameter vectorsand difference schemesrdquo Journal of Computational Physicsvol 43 no 2 pp 357ndash372 1981

[40] A Jameson W Schmidt and E Trukel ldquoNumerical solutionsof the Euler equations by finite volume methods usingRungendashKutta time-stepping schemesrdquo in Proceedings of the14th Fluid and Plasma Dynamics Conference AIAA PaperPalo Alto CA USA 1981

[41] T H Pulliam and D S Chaussee ldquoA diagonal form of animplicit approximate-factorization algorithmrdquo Journal ofComputational Physics vol 39 no 2 pp 347ndash363 1981

[42] J Blazek Computational Fluid Dynamics Principles andApplications Elsevier Amsterdam Netherlands 2nd edition2005

[43] Z Wang N Hariharan and R Chen ldquoRecent developmentson the conservation property of chimerardquo in Proceedings ofthe 36th AIAA Aerospace Sciences Meeting and Exhibit RenoNV USA January 1998

[44] R L Meakin ldquoOn the spatial and temporal accuracy of oversetgrid methods for moving body problemsrdquo in Proceedings of

the 12th Applied Aerodynamics Conference AIAA Paper 1994-1925 Colorado Springs CO USA June 1994

[45] S E Rogers N E Suhs and W E Dietz ldquoPEGASUS 5 anautomated preprocessor for overset-grid computationalfluid dynamicsrdquo AIAA Journal vol 41 no 6 pp 1037ndash1045 2003

[46] W Ma X Hu and X Liu ldquoParallel multibody separationsimulation using MPI and OpenMP with communicationoptimizationrdquo Journal of Algorithms amp ComputationalTechnology vol 13 pp 1ndash17 2018

[47] Y Wu ldquoNumerical simulation and Aerodynamic effect re-search for multi-warhead projectionrdquo Journal of SystemSimulation vol 28 no 7 pp 1552ndash1560 2016 in Chinese


Computer Games Technology

International Journal of

Hindawiwwwhindawicom Volume 2018

Hindawiwwwhindawicom

Journal ofEngineeringVolume 2018

Advances in

FuzzySystems


Volume 2018


ReconfigurableComputing



Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

thinspArtificial Intelligence

Hindawiwwwhindawicom Volumethinsp2018


Civil EngineeringAdvances in


Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications


Hindawi

wwwhindawicom Volume 2018

Advances in

Multimedia


Biomedical Imaging



Engineering Mathematics


RoboticsJournal of



Computational Intelligence and Neuroscience


Mathematical Problems in Engineering

Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018

Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018


Human-ComputerInteraction

Advances in


Scientic Programming

Submit your manuscripts atwwwhindawicom

optimization [19 20] numerical techniques [21] and ap-plications [22] based on unstructured meshes on the GPUplatform For GPU simulations on overlapped (overset)meshes Soni et al [23] developed a steady CFD solver onunstructured overset meshes by using GPU programmingmodel which accelerated both the procedure of grid (mesh)assembly and the procedure of numerical calculations )enthey extended their solver to unsteady ones [24 25] to maketheir GPU implementation capable of handling dynamicoverset meshes More CFD-related computing on GPUs canbe found in an overview reference [26]

However most of the existing works either used con-sistent structured or unstructured meshes without meshoverlapping over the computational domain on MIC archi-tecture or studied overlapped meshes on GPUs A majority ofCFD simulations involving overlapped meshes were imple-mented or developed on distributed system through messagepassing interface (MPI) [27] without using coprocessors inpast several decades Specifically Djomehri and Jin [28] re-ported the parallel performance of an overset solver using ahybrid programming model with both MPI [27] andOpenMP [29] Prewitt et al [30] conducted a review ofparallel implementations using overlapped mesh methodsRoget and Sitaraman [31] developed an effective overlappedmesh assembler and investigated unsteady simulations onnearly 10000 CPU cores Zagaris et al [32] discussed a rangeof problems regarding parallel computing for moving bodyproblems using overlapped meshes and presented the pre-liminary performance on parallel overlapped grid assemblyOther overlapped mesh-related works can be found in[33ndash36] Although the work in [7] conducted tests by using asolver that is compatible with overlapped meshes on MICcoprocessors the way it accessed multiple MIC coprocessorsis through the native mode or symmetric mode of MIC Inthis paper we focus on the use of offload mode of MIC andinvestigate the parallelization of a solver with overset mesheson a single host node with multiple MIC coprocessors )econtributions of this work are as follows

(i) We parallelize an Euler solver using overlappedmeshes and propose an asynchronous strategy forcalculations on multiple MIC coprocessors within asingle host node

(ii) We investigate the performances of core functionsof the solver by employing offload mode with dif-ferent thread affinity types on the MIC architectureand we make detailed comparisons between theresults obtained by Intel MIC vectorization and thatobtained by Intel SSE vectorization

(iii) A speedup of 59x can be obtained on two MICcoprocessors over a single Intel E5-2680 processorfor two M6 wings case

)e remainder of the paper is as follows We first in-troduce the MIC architecture and programming modelAnd this is followed by equations and numerical algorithmsthat have been implemented in the solver In Section 4we discuss implementation and optimization aspects in-cluding data transfer vectorization and asynchronous data

exchange strategy on multiple MIC coprocessors )e per-formances of core functions by using different thread affinitytypes are reported and comparisons are made in Section 5)e last section summarizes our work

2 MIC Architecture and Programming Model

Many integrated core (MIC) architecture [2] is a processorthat is capable of integrating many times86 cores providing thecomputing power of high parallelism )e architecture usedin the first Intel Xeon Phi product is called Knights CornerKNC )e KNC coprocessors can have many (up to 61)double dispatched in-order executing times86 computing coresEach core has a 512 bit vector processing unit (VPU) whichsupports 16 single or 8 double floating point operations percycle and each core is able to launch 4 hardware threads32KB L1 code cache 32KB L1 data cache and 512KB L2cache are available to each core)e coprocessor used in thiswork is Intel Xeon Phi 5110P [37] which consists of 60 coreseach of which runs at 105GHz It can launch a total of 240threads simultaneously And 8GB of GDDR5 memory isavailable on it

Anyone who is familiar with C C++ or Fortran pro-gramming language can develop codes onMIC coprocessorswithout major revision of their source codes MIC providesvery flexible programming models including native hostmode native MIC mode offload mode and symmetricmode [38] Coprocessors are not used in native host modeand programmers can run their codes on CPUs just like theydo before the MIC architecture was introduced By contrastcodes can be conducted only on coprocessors in native MICmode when they are compiled with ldquo-mmicrdquo optionSymmetric mode allows programmers to run codes on bothCPU cores and coprocessors And offload mode is mostcommonly used on a single coprocessor or multiple co-processors within a single host node)e basic use of offloadmode for programmers is to write offload directives to makethe code segment run on MIC coprocessors To take fulladvantage of computational resources on MIC the codesegment can be conducted in parallel by employing mul-tithreading techniques such as OpenMP [29]

3 Equations and Numerical Algorithms

31 Compressible Euler Equations )e three-dimensionaltime-dependent compressible Euler equations over a con-trol volume Ω can be expressed in integral form as

z

zt1113946ΩW dΩ + 1113929

zΩFcdS 0 (1)

where W [ρ ρu ρv ρw ρE]T represents the vector of con-servative variables and Fc [ρV ρuV + nxp ρvV + nyp

ρwV + nzp ρHV]T denotes the vector of convective fluxeswith V nxu + nyv + nzw

32 Numerical Algorithms )e flux-difference splitting(FDS) [39] technique is employed to calculate the spatialderivative of convective fluxes In this method the flux at theinterface (for example i direction) expressed by Fiplusmn(12)jk



Fi+(12)jk 12


1113868111386811138681113868 WR minusWL( 11138571113960 1113961 (2)





4 Implementation












mesh data

MIC_0 MIC_n

Interpolation data


Mesh assembly Host

Initialization


Data exchange

MBC_n




Data exchange

MBC_0 MBC_1



Output file




and blockj


c Fjc and Fk





















5 Experiments









end do

(a)



dir$ ivdep


operation_2(tmp())

end do

(b)










X

Y

Z








NTFj Fk










101

102

103

Wal

l clo

ck ti

mes

(sec

onds

)

Fk CS-1Fi CS-1

Fk CS-2Fi CS-2



NTFi TAj




NTTAk TAi






6 Summary


100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x



PPinf1495139212881185108209780875077206690565046203590256


ndash05

0

05

1

0 02 04 06 08 1X

Cp








Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in





Fi+(12)jk 12


1113868111386811138681113868 WR minusWL( 11138571113960 1113961 (2)





4 Implementation












mesh data

MIC_0 MIC_n

Interpolation data


Mesh assembly Host

Initialization


Data exchange

MBC_n




Data exchange

MBC_0 MBC_1



Output file




and blockj


c Fjc and Fk





















5 Experiments









end do

(a)



dir$ ivdep


operation_2(tmp())

end do

(b)










X

Y

Z








NTFj Fk










101

102

103

Wal

l clo

ck ti

mes

(sec

onds

)

Fk CS-1Fi CS-1

Fk CS-2Fi CS-2



NTFi TAj




NTTAk TAi






6 Summary


100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x



PPinf1495139212881185108209780875077206690565046203590256


ndash05

0

05

1

0 02 04 06 08 1X

Cp








Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in








mesh data

MIC_0 MIC_n

Interpolation data


Mesh assembly Host

Initialization


Data exchange

MBC_n




Data exchange

MBC_0 MBC_1



Output file




and blockj


c Fjc and Fk





















5 Experiments









end do

(a)



dir$ ivdep


operation_2(tmp())

end do

(b)










X

Y

Z








NTFj Fk










101

102

103

Wal

l clo

ck ti

mes

(sec

onds

)

Fk CS-1Fi CS-1

Fk CS-2Fi CS-2



NTFi TAj




NTTAk TAi






6 Summary


100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x



PPinf1495139212881185108209780875077206690565046203590256


ndash05

0

05

1

0 02 04 06 08 1X

Cp








Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in





















5 Experiments









end do

(a)



dir$ ivdep


operation_2(tmp())

end do

(b)










X

Y

Z








NTFj Fk










101

102

103

Wal

l clo

ck ti

mes

(sec

onds

)

Fk CS-1Fi CS-1

Fk CS-2Fi CS-2



NTFi TAj




NTTAk TAi






6 Summary


100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x



PPinf1495139212881185108209780875077206690565046203590256


ndash05

0

05

1

0 02 04 06 08 1X

Cp








Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in











X

Y

Z








NTFj Fk










101

102

103

Wal

l clo

ck ti

mes

(sec

onds

)

Fk CS-1Fi CS-1

Fk CS-2Fi CS-2



NTFi TAj




NTTAk TAi






6 Summary


100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x



PPinf1495139212881185108209780875077206690565046203590256


ndash05

0

05

1

0 02 04 06 08 1X

Cp








Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in









NTFj Fk










101

102

103

Wal

l clo

ck ti

mes

(sec

onds

)

Fk CS-1Fi CS-1

Fk CS-2Fi CS-2



NTFi TAj




NTTAk TAi






6 Summary


100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x



PPinf1495139212881185108209780875077206690565046203590256


ndash05

0

05

1

0 02 04 06 08 1X

Cp








Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in








101

102

103

Wal

l clo

ck ti

mes

(sec

onds

)

Fk CS-1Fi CS-1

Fk CS-2Fi CS-2



NTFi TAj




NTTAk TAi






6 Summary


100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x



PPinf1495139212881185108209780875077206690565046203590256


ndash05

0

05

1

0 02 04 06 08 1X

Cp








Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in






6 Summary


100

101

102

103

104

Wal

l clo

ck ti

mes

(sec

onds

)

59x36x



PPinf1495139212881185108209780875077206690565046203590256


ndash05

0

05

1

0 02 04 06 08 1X

Cp








Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in







Acknowledgments


References
























































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in
































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




implementationandoptimizationofacfdsolverusing...

Documents