george em karniadakis division of applied mathematics the crunch group: cross-site simulations on...
Post on 20-Dec-2015
214 views
TRANSCRIPT
![Page 1: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/1.jpg)
George Em KarniadakisDivision of Applied Mathematics
The CRUNCH group: www.cfm.brown.edu/crunch
Cross-Site Simulations on the TeraGrid
spectral elements Micro / Nano-fluidics parallel computing
![Page 2: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/2.jpg)
Grand-Challenge Problem 1: Turbulence – Drag crisis(Tightly-Coupled Problem)
• Turbulence – Last frontier in classical physics
• Climate, environment, transport, energy,…
• Re=300,000 (CPU ~ Re3) requires 20 Billion DOFs
• Memory 4 TBytes
![Page 3: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/3.jpg)
Wave Propagation in a Model of the Arterial Circulation
(Data of 55 main arteries from J.J. Wang and K. Parker, 1997)
Grand-Challenge Problem 2: Human Arterial Tree(Loosely-Coupled Problem)
![Page 4: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/4.jpg)
First ParallelTeraGrid
Paradigm
NCSA IA64SDSC IA64
in-sitecommunication
Cross-sitecommunication
in-sitecommunication
TG Site TG Site
Whole flowDomain
All-to-all
![Page 5: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/5.jpg)
-5/3
DNS versus Experiments: max Re=10,000
DNSExperiments
(Rockwell, 2004)
Energy Spectrum
Black – simulation
Blue - experiment
RMS velocity
![Page 6: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/6.jpg)
Turbulence: Single-Site Performance
Fixed problem size Fixed workload
• PSC: Compaq Alpha EV68, 1 GHz
• 300 Million DOFs, 2-level MPI
• MPICH-G2 and MPI perform similarly (SDSC/IA-64)
![Page 7: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/7.jpg)
• Half processors from NCSA, half from SDSC
• Intel IA-64 processors (Itanium-2, 1.5 GHz)
• Slow-down factor 1.5SDSC TG
NCSA TGFFT Matrix
transposition
Turbulence: Cross-Site Performance
Fixed problem size Fixed workload
![Page 8: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/8.jpg)
P(t)
W1
W2
Ascending aorta
U(t)
Inflow conditions U(t)
P(t)
Thoracic aorta
Femoral
P(t)
U(t) W1
W2
U(t)
Tibial
P(t)
Outflow conditions(Peripheral resistance)
1D Model – Sherwin et al. / Imperial College
![Page 9: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/9.jpg)
Platelet Aggregation in Arterioles and Platelet Aggregation in Arterioles and VenulesVenules
FLOW
Parameters: Vessel diameter - 50 µm, vessel length - 400 µm, blood velocity - 100 µm/s,platelet diameter - 3 µm, platelet concentration - 300000/mm3, platelet density - 1.03 fluid densitySimulation time - 28 s
venules
platelet aggregate
![Page 10: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/10.jpg)
Growth Rate vs. Blood VelocityGrowth Rate vs. Blood Velocity
Experiments: Begent and Born, Nature, Vol. 227, No. 5261, pp. 926-930, 1970
![Page 11: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/11.jpg)
Second Parallel TeraGrid Paradigm
Multiscale Simulation of Arterial Tree
![Page 12: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/12.jpg)
Arterial-Tree: Cross-Site Performance
(Homogeneous Network)
• Three arteries; 4 Million DOFs per artery
• 1CPU/node on ANL; 2CPUs/node on NCSA/SDSC
• No slown-down, full scalabilitySDSC TG
ANL TG
NCSA TG
Fixed problem size Fixed workload
![Page 13: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/13.jpg)
SDSC TG
NCSA TGPSC TG
Arterial-Tree: Cross-Site Performance
(Heterogeneous Network)
•PSC connects to TG via
application gateway (qsockets)
•Two arteries per site
•PSC proc:2 GF vs 6 GF IA-64
![Page 14: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/14.jpg)
New Unique Capability
• Potentially unlimited salability; Enabling technology
– Integrate “real and virtual” in projects like:– Digital human, digital ocean, digital space, …
• Predictability and Uncertainty – Stochastic simulations– Prediction vs. Postdiction– Risk-based/Reliability-based design– Sensitivity analysis – steering of experiments
(e.g., DDDAS concept)
• Inverse Problems– Engineering design– Biomedical sciences– Geological/Climate Modeling
![Page 15: George Em Karniadakis Division of Applied Mathematics The CRUNCH group: Cross-Site Simulations on the TeraGrid spectral elementsMicro](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d405503460f94a1ad59/html5/thumbnails/15.jpg)
What Users Need
• Debuggers for TG (a la TotalView)• New topology-aware parallel algorithms• Sustained network/cluster performance• TG visualization capability• Middleware
– Robust MPICH-G2– Co-scheduling– Network & Globus diagnostics– Authentication/Security – often in conflict
• Consultants/Referees with TG-Expertise