report of the cemracs project hpc- ilbios: understanding...
TRANSCRIPT
![Page 1: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/1.jpg)
OLGA ASSAINOVA / XAVIER PORTELL
Report of the CEMRACS project HPC-IlBios: understanding the best HPC strategy for the IlBioS approach.Olga Assainova*, Xavier Portell**
CEMRACS 2016. Numerical challenges in parallel scientific computing
August 19, 2016, Marseille
**UMR ECOSYS, AgroParisTech, INRA, Université Paris-SaclayF-78850 Thiverval-Grignon, France.E-mail: [email protected]
*Université de BourgogneFaculté des Sciences Mirande9 avenue Alain SavaryBP 47870 21078 Dijon CedexE-mail: [email protected]
![Page 2: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/2.jpg)
.0‹N°›
SUMMARY
Introduction and background
2
3
1
An overview of StarPU
Model versions produced in CEMRACS
4 Pending work
![Page 3: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/3.jpg)
.0‹N°›
SUMMARY
Introduction and background
2
3
1
An overview of StarPU
Model versions produced in CEMRACS
4 Pending work
![Page 4: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/4.jpg)
Introduction and background_01
.04
![Page 5: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/5.jpg)
.0‹N°›
The CO2 and N2O as main GHG
The CO2 and N2O as main GHG
![Page 6: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/6.jpg)
.0‹N°›
Riebbek (2011)
An important part of the CO2 emissions are due to the activity of the soil microorganisms
The CO2 and N2O as main GHG
![Page 7: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/7.jpg)
.0‹N°›
0,03 % of the total GHG emissions but with a 300-fold greater potential for global warming (Thomson et al 2012)
Hu et al (2015)
![Page 8: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/8.jpg)
.0‹N°›
The soil matrix
The soil matrix
![Page 9: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/9.jpg)
.0‹N°›
The complex geometry of the pore space can now be characterized to a high level of detail and quantify its connectivity and topology
characterized.
X-ray Computed Tomography
The soil matrix
The morphology of the soil matrix affects microbial activity and the gas emissions dynamics to the atmosphere
![Page 10: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/10.jpg)
.0‹N°›
1 Compute-aided Detection - Fluorescence In Situ Hybridization
CAD-FISH1
Imaging of thin soil sections
Microorganisms in soil tend to be found in microcolonies, this means that “identical organisms” will face different microhabitat
conditions
The individuality of the microorganisms plays a role
The soil matrix
![Page 11: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/11.jpg)
.0‹N°›
Mass transport processes modelling in soil
Mass transport processes modelling in soil
![Page 12: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/12.jpg)
.0‹N°›Mass transport processes modelling in soil
Lattice-Boltzmann modelling Fluid Node (D3Q19)
Solid Sites
Solid Sites
From: www.egr.msu.edu/ ~kutay/Lbsite/
![Page 13: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/13.jpg)
.0‹N°›Mass transport processes modelling in soil
Lattice-Boltzmann modelling
Solid Sites
Solid Sites
From: www.egr.msu.edu/ ~kutay/Lbsite/
AdvantageComplex pore space geometry easy
to handleIssues
Computationally demanding, but easy to parallelize (GPU or
accelerators)
![Page 14: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/14.jpg)
.0‹N°›
IlBioS conceptualization and description
IlBioS conceptualization and description
![Page 15: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/15.jpg)
.0‹N°›
The model ILBioS is built coupling an Individual-based Model of the soil bacteria to a lattice-Boltzmann
model simulating the fluid dynamics and mass transport processes of soluble substrates
IlBioS conceptualization and description
![Page 16: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/16.jpg)
.0‹N°›Model conceptualization and description
Lattice-Boltzmann Nodes
Three dimensional lattice-Boltzmann (D3Q7)
![Page 17: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/17.jpg)
.0‹N°›Model conceptualization and description
Three dimensional lattice-Boltzmann (D3Q7)
Connected Lattice-Boltzmann Nodes
![Page 18: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/18.jpg)
.0‹N°›Model conceptualization and description
Three dimensional lattice-Boltzmann (D3Q7)
![Page 19: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/19.jpg)
.0‹N°›Model conceptualization and description
Three dimensional lattice-Boltzmann (D3Q7)
Fluid nodes
Solid nodes
Dissolved Organic Carbon (DOC)
![Page 20: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/20.jpg)
.0‹N°›Model conceptualization and description
Three dimensional lattice-Boltzmann (D3Q7)
![Page 21: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/21.jpg)
.0‹N°›Model conceptualization and description
Three dimensional lattice-Boltzmann (D3Q7)
Fluid nodes
Boundary Solid nodes
Solid nodes
Boundary Fluid nodes
![Page 22: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/22.jpg)
.0‹N°›Model conceptualization and description
Three dimensional lattice-Boltzmann (D3Q7)
Particulate Organic Matter:
Soil bacteria:
Uptake DOC from the boundary fluid nodes.
Release DOC to the boundary fluid nodes.
![Page 23: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/23.jpg)
.0‹N°›Model conceptualization and description
Three dimensional lattice-Boltzmann (D3Q7)
mj mass of the POM agent.
![Page 24: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/24.jpg)
.0‹N°›
mi mass of the bacterium
Model conceptualization and description
Three dimensional lattice-Boltzmann (D3Q7)
![Page 25: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/25.jpg)
.0‹N°›
Computing time requirements: antecedents
Computing time requirements: antecedents
![Page 26: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/26.jpg)
.0‹N°›Computing time requirements: antecedents
Computing time requirements of lBioS❑3D CT image of 2003 nodes.❑10% of porosity.❑10 bacterial nodes.❑5 to 15 POM nodes.❑DOC as a single soluble lattice-Boltzmann substrate.❑Single desktop computer.
25 hours of processing time.
![Page 27: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/27.jpg)
.0‹N°›
Model complexity of upcoming IlBioS models
Model complexity of upcoming IlBioS models
![Page 28: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/28.jpg)
.0‹N°›Model complexity of upcoming IlBioS models
Microbial regulation of the terrestrial Nitrous oxide formation (Hu et al., 2015)
106 – 109 bacteria / gram of soil
![Page 29: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/29.jpg)
.0‹N°›Model complexity of upcoming IlBioS models
Initial (minimal) conceptualization of a multispecies IbM to reproduce CO2 and N2O flow in natural samples
NH3/NH4 NO2 /NO3
N2O
Denitrifying heterotroph(CO2 producing, aerobic (O2) and anaerobic (NO2)
respiration)
Nitrifying autotroph (Aerobic, CO2 consuming)
![Page 30: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/30.jpg)
.0‹N°›Model complexity of upcoming IlBioS models
Initial (minimal) conceptualization of a multispecies IbM to reproduce CO2 and N2O flow in natural samples
CO2, O2, DOC, NH4, NO2/ NO3, N2O
Required lattice-Boltzmann substrates in the minimal system:
![Page 31: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/31.jpg)
.0‹N°›
Future computing time requirements
Future computing time requirements
![Page 32: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/32.jpg)
.0‹N°›Future computing time requirements
Computing time requirements of llBioS❑3D CT image of 2003 nodes.❑10 % of porosity.❑1 000 000 bacterial nodes❑5 to 25 POM nodes.❑O2, DOC, NH4, NO2/ NO3
125-150 hours of processing time.
In any case, a non practical amount of time to perform
simulation experiments
![Page 33: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/33.jpg)
.0‹N°›
Parallelisation strategy of HPC-IlBioS
Parallelisation strategy of HPC-IlBioS
![Page 34: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/34.jpg)
.0‹N°›
Tiling of the data in subdomains
StarPU
CPU functions
Parallelisation strategy of HPC-IlBioS
![Page 35: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/35.jpg)
.0‹N°›
StarPU
100 xSpeed up for lattice-Boltzmann models using GPU
(Banari et al., 2014)
CUDA functions
CPU functions
Parallelisation strategy of HPC-IlBioS
![Page 36: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/36.jpg)
.0‹N°›
MPIStarPU
CUDA functions
CPU functions
CUDA functions
CPU functions
CUDA functions
CPU functions
Future need of increasing the resolution of the 3D CT images
Parallelisation strategy of HPC-IlBioS
![Page 37: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/37.jpg)
.0‹N°›
SUMMARY
Introduction and background
2
3
1
An overview of StarPU
Model versions produced in CEMRACS
4 Pending work
![Page 38: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/38.jpg)
.0‹N°›
StarPU Introduction
StarPU Introduction
![Page 39: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/39.jpg)
.0‹N°›StarPU Introduction
The basics of StarPU syntax and tools
❑Codelet❑Task❑Buffer❑Handle❑CPU/CUDA functions
![Page 40: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/40.jpg)
.0‹N°›StarPU Introduction
StarPU preparation task routine
Registration of the data starpu_init(NULL);
starpu_data_handle_t f_handle; starpu_data_handle_t fn_handle;
starpu_vector_data_register(&ToV_handle, STARPU_MAIN_RAM, (uintptr_t)ToV, nsite, sizeof(ToV[0]));
starpu_matrix_data_register(&f_handle, STARPU_MAIN_RAM, (uintptr_t)f, Q, Q, nsite, sizeof(f[0][0]));
..=>
starpu_data_unregister(f_handle); starpu_data_unregister(f_handle); starpu_shutdown();
![Page 41: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/41.jpg)
.0‹N°›StarPU Introduction
StarPU preparation task routine
Codelet
struct starpu_codelet bounceback_codelet = { .cpu_funcs = { bounceback_spu }, .cpu_funcs_name = {"bounceback_spu"}, .nbuffers = 2,
.modes = { R, RW } };
Task insert starpu_task_insert(&bounceback_codelet, STARPU_R, ToV_handle, STARPU_RW, f_handle, STARPU_VALUE, &SbD, sizeof(unsigned int), STARPU_VALUE, &nfluidsite, sizeof(unsigned long), STARPU_VALUE, &nsite, sizeof(unsigned long),
STARPU_VALUE, &Q, sizeof(int), 0);
![Page 42: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/42.jpg)
.0‹N°›
void bounceback_spu(void *buffers[], void *cl_arg){
unsigned int SubD=0; unsigned long nfluidsite=0; unsigned long nsite=0; int Q=0;
/* Unpacking arguments to their local copy */ starpu_codelet_unpack_args(cl_arg, &SubD,&nfluidsite, &nsite,&Q);
/* Unpacking buffers to to their local copy */ struct starpu_vector_interface *ToV_v =(struct starpu_vector_interface *) buffers[0]; unsigned long* ToV = (unsigned long *)STARPU_VECTOR_GET_PTR(ToV_v);
struct starpu_matrix_interface **f_m =(struct starpu_matrix_interface **) buffers[1]; double **f= (double **) STARPU_MATRIX_GET_PTR(f_m);
/* Beginning of the computations */ register unsigned long ns; for(ns=nfluidsite;ns<nsite;ns++){ if (SubD!=ToV[ns]) continue; /*Skipping computation of LB Nodes not in the current sub domain*/ register int q; for(q=0;q<Q;q++){ f[q][ns]=0.; } } } // end of bounceback_step_spu() function
StarPUexecutivefunction
StarPU Introduction
![Page 43: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/43.jpg)
.0‹N°›
void bounceback_spu(void *buffers[], void *cl_arg){
unsigned int SubD=0; unsigned long nfluidsite=0; unsigned long nsite=0; int Q=0;
/* Unpacking arguments to their local copy */ starpu_codelet_unpack_args(cl_arg, &SubD,&nfluidsite, &nsite,&Q);
/* Unpacking buffers to to their local copy */ struct starpu_vector_interface *ToV_v =(struct starpu_vector_interface *) buffers[0]; unsigned long* ToV = (unsigned long *)STARPU_VECTOR_GET_PTR(ToV_v);
struct starpu_matrix_interface **f_m =(struct starpu_matrix_interface **) buffers[1]; double **f= (double **) STARPU_MATRIX_GET_PTR(f_m);
/* Beginning of the computations */ register unsigned long ns; for(ns=nfluidsite;ns<nsite;ns++){ if (SubD!=ToV[ns]) continue; /*Skipping computation of LB Nodes not in the current sub domain*/ register int q; for(q=0;q<Q;q++){ f[q][ns]=0.; } } } // end of bounceback_step_spu() function
StarPU Introduction
![Page 44: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/44.jpg)
.0‹N°›
SUMMARY
Introduction and background
2
3
1
An overview of StarPU
Model versions produced in CEMRACS
4 Pending work
![Page 45: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/45.jpg)
.0‹N°›
ILBios
![Page 46: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/46.jpg)
.0‹N°›
Spatial cells (LB Nodes)
NBULK
• Water (DOC) • Air • Solid • (agents)
NBOUND NBOUNDS
0 (NX*NY*NZ)-1LBNode
nfluidsites
Bacteria (many individuals/node)
POM
ILBios
![Page 47: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/47.jpg)
.0‹N°›
ILBios
![Page 48: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/48.jpg)
.0‹N°›
Typedef struct BiologicalAgent { unsigned long Who; /* ID of the individual */ unsigned long LBNode; /* lattice-Boltzmann Node */ double Mass; double Active_Fraction; double Respired_CO2; /* technical variable */
} type_bacteria;
Vector holding the individual bacteria properties
General workflow of the model IlBioS
ILBios
![Page 49: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/49.jpg)
.0‹N°›
Typedef struct POMAgent { unsigned long pWho; /* ID */ unsigned long pLBNode; /* lattice-Boltzmann Node */ double pMass; int pFluidNeigh; /* Number of fluid neighbors */ unsigned long *pFluidNeighID; /* technical variable */
} type_POM;
Vector holding the POM properties
ILBios
![Page 50: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/50.jpg)
.0‹N°›
Initialization of the DOC = population Boltzmann
Solid Sites
Solid Sites
From: www.egr.msu.edu/ ~kutay/Lbsite/
ILBios
![Page 51: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/51.jpg)
.0‹N°›
Initialization of the DOC = population Boltzmann Fluid Node (D3Q19)
Solid Sites
Solid Sites
From: www.egr.msu.edu/ ~kutay/Lbsite/
ILBios
![Page 52: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/52.jpg)
.0‹N°›
double *rho = (double *) calloc(nsite, sizeof(double));
Fluid Node (D3Q19)
Solid Sites
Solid Sites
From: www.egr.msu.edu/ ~kutay/Lbsite/
double **f = (double **) calloc(Q, sizeof(double*)); for (q = 0; q < Q; q++) { f[q] = (double *) calloc(nsite, sizeof (double)); }
Initialization of the DOC = population Boltzmann
ILBios
![Page 53: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/53.jpg)
.0‹N°›
Time step (TS) or main loop
ILBios
![Page 54: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/54.jpg)
.0‹N°›
The State of the bacterial and POM agents and the global out-fluxes are stored in
appropriate data arrays every sampling time.
ILBios
![Page 55: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/55.jpg)
.0‹N°›
ILBios
![Page 56: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/56.jpg)
.0‹N°›
Mass of DOC particles moving in the unitary directions
D2Q5
ILBios
![Page 57: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/57.jpg)
.0‹N°›
ILBios
![Page 58: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/58.jpg)
.0‹N°›
The mass of DOC particles is propagated to the neighbors
ILBios
![Page 59: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/59.jpg)
.0‹N°›
Solid nodes
ILBios
![Page 60: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/60.jpg)
.0‹N°›
The DOC particles bounce back on the solid nodes directions
Solid nodes
ILBios
![Page 61: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/61.jpg)
.0‹N°›
ILBios
![Page 62: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/62.jpg)
.0‹N°›
Increase the DOC content of all fluid neighbors
ILBios
![Page 63: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/63.jpg)
.0‹N°›
Decreases the DOC content of the LBNode
ILBios
![Page 64: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/64.jpg)
.0‹N°›
Writing the sampled output in files.
ILBios
![Page 65: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/65.jpg)
.0‹N°›
ILBios
ILBios SbD
![Page 66: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/66.jpg)
.0‹N°›
ILBios
ILBios SbD
1 2
3 4
![Page 67: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/67.jpg)
.0‹N°›
ILBios
ILBios SbD
ILBios SPU
One task for every function in the time step loop
![Page 68: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/68.jpg)
.0‹N°›
for(t=0;t<TMAX+1;t++) { ..=> starpu_task_insert(&POM_agents_loop_codelet, STARPU_R, tw_handle, STARPU_W, rho_handle, STARPU_RW, f_handle, STARPU_RW, POMPopul_handle, STARPU_VALUE, &POM_ts, sizeof(unsigned long), STARPU_VALUE, &Q, sizeof(int), STARPU_VALUE, &Q2, sizeof(int), STARPU_VALUE, &KPOM, sizeof(double), STARPU_VALUE, &MIN_POS_DBL, sizeof(double), 0);
Examplesofthecodelet
starpu_task_insert(&propagation_codelet, STARPU_R, neighbour_handle, STARPU_RW, f_handle, STARPU_VALUE, &Q, sizeof(int), STARPU_VALUE, &nfluidsite, sizeof(unsigned long), STARPU_VALUE, &nsite, sizeof(unsigned long), 0);
}
ILBios
ILBios SbD
ILBios SPU
![Page 69: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/69.jpg)
.0‹N°›
ILBios
ILBios SbD
ILBios SPU One task for every function and many tasks for collision function
HPC-ILBios (collision)
![Page 70: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/70.jpg)
.0‹N°›
for(int step=0; step<PARTS; step++) { struct starpu_task *task = starpu_task_create(); task->cl = &collision_vSubD_codelet;
task->handles[0] = ToV_handle; task->handles[1] = rho_handle; task->handles[2] = uw_handle; task->handles[3] = c_handle; task->handles[4] = twstar_handle; task->handles[5] = jo2_handle; task->handles[6] = jo_q_handle; task->handles[7] = f_handle; task->handles[8] = j_q_handle; task->handles[9] = evenf_handle; task->handles[10] = oddf_handle; task->handles[11] = fn_handle; // The only one changing parameter param.min = min; param.lenth =lenth; task->cl_arg = ¶m; task->cl_arg_size = sizeof(param); task->synchronous = 1; ret =starpu_task_submit(task)
}
ILBios
ILBios SbD
ILBios SPU
HPC-ILBios (collision)
![Page 71: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/71.jpg)
.0‹N°›
SUMMARY
Introduction and background
2
3
1
An overview of StarPU
Model versions produced in CEMRACS
4 Pending work
![Page 72: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/72.jpg)
.0‹N°›
Domaindecompositionapproach
❑ Partial domain decompositionof the liquid parts of the cub❑Implementing MPI
R1
R2
![Page 73: Report of the CEMRACS project HPC- IlBios: understanding ...smai.emath.fr/cemracs/cemracs16/.../Session1/3_HPCIlBioS_Report_xpc.pdf · Report of the CEMRACS project HPC-IlBios: understanding](https://reader035.vdocument.in/reader035/viewer/2022062914/5e6f2be3488b7b09bc502b84/html5/thumbnails/73.jpg)
.0‹N°›
Thank you very much for your attention.
But, before we finish …
A big thanks to all people that has helped us !!
Alexandre Ancel
Herbert Oberlin
David Coulette
Philippe Helluy and all SCHNAPS members
Matthieu Boileau
Laura Grigori
Camile Coti