high performance computer simulations - · pdf filehigh performance computer simulations...
TRANSCRIPT
High Performance Computer Simulations
Borrajo Juan*, Chimenti Martín*, De Vecchi Hugo⁺, Grioni Mauro#, Rojido Martín⁺, Seres Vincent^
* IMPSA HYDRO, ⁺ IMPSA CIT, # IMPSA WIND, ^ IMPSA IT
• The only company with proprietary technology for hydropower and wind power generation in Latin America.
• The largest wind generator manufacturer in Latin America and one of the biggest hydropower manufacturers in the region.
• The largest investor in wind farms in Latin America with assets in Brazil, Uruguay, Argentina and Peru.
• The world’s fastest growing family business according to the Top 50 Global Challengers perform by Ernst & Young.
• A company with a high profit track record for more than 106 years.
• A major source of qualified and sustainable work.
• Local and experienced management teams.
• Worldwide exporter in the Hydropower business and for all Latin America in wind power generation.
• A company with access to international capital markets for over 15 years and financed by multilateral credit banks (BID, CAF, CII).
IMPSA is…
2
IMPSA - Production centers
ARGENTINA MALAYSIA
IMPSA Plant I Mendoza - Argentina
IMPSA Hydro Recife Suape, Brazil
IMPSA Wind Recife Suape, Brazil
IMPSA Hydro Recife Suape, Brazil
BRAZIL
IMPSA Wind Recife Suape, Brazil
IMPSA Plant II Mendoza - Argentina
IMPSA Malaysia Malaysia
IMPSA Malaysia Malaysia
4
* Article published in the Financial Times, “Battle of Speed Machines”, on Wednesday, July 10, 2013.
Each 15 years, speed of the most powerful computers in the world are multiplied by 1000.
HISTORICAL DEVELOPMENT
6
• IMPSA hardware has followed a similar pattern.
• Historically, the number of degrees of freedom (DOF) was used as a reference.
• 15 years ago, on a HP 7000, for FEM analysis, it was possible to run 100,000 (one hundred thousand) DOF.
• Nowadays, we can analyze more than 100,000,000 (one hundred million) DOF.
• In 15 years, the size of the models created and resolved by IMPSA has been multiplied by 1000.
HISTORICAL DEVELOPMENT - IMPSA
7
IMPSA HARDWARE - Workstations
Personal Computer
X1 (26 Users)
Nodes 1 Dell 7500
Processors 2 Xeon® X5647 2.93 GHz
Cores 8 - 2 CPU, 4 cores / CPU
RAM 24 GB
ROM 500GB - 7.2Krpm, SATA 2
Network Ethernet 1 Gbps
FEM Calculation
X2 (1 Workstation)
Nodes 1 Dell 7500
Processors 2 Xeon® X5647 2.93 GHz
Cores 8 - 2 CPU, 4 cores / CPU
RAM 48 GB
ROM 1TB - 7.2Krpm, SATA 2
Network Ethernet 1 Gbps
CFD Calculation
X8 (1 Cluster)
Nodes 8 Dell 7500
Processors 16 Xeon® X5647 2.93 GHz
Cores 64 - 16 CPU, 4 cores / CPU
RAM 18 GB/Node = 144 GB
ROM 4TB - 7.2Krpm, SATA 2
Network Ethernet 1 Gbps
2007 2009/2012
8
IMPSA HARDWARE - Cluster HPC
Cluster Dell HPC Rack : PowerEdge 4220 HPC Nodes : 8 vertical blades, PowerEdge M620 Head Nodes : 2 horizontal blades Storage Bay : 30 HDD, SAS 2.0, 10krpm, de 900GB
= 27 TB
Nodes 8 PowerEdge M620
Processors 16 Xeon® E5-2680 2.70 GHz, 331 Gflops
Cores 128 - 16 CPU, 8 cores por CPU
RAM 32 GB/CPU = 512 GB
ROM 2 x 146 GB - 15Krpm SAS 2.0
Network Infiniband 40 Gbps
2013 (2.77 Tflops) 9
COMPUTER APPLICATIONS AND CHALLENGES
SPEED
ACCURACY
LARGE ASSEMBLIES
FLUID DYNAMICS TRANSIENTS
MULTIPHYSICS SIMULATIONS
10
MECHANISMS AND ASSEMBLIES WITH NONLINEAR CONTACT
Assembly: Head Cover, Wicket Gate, Bottom Ring, Spiral Case
Simulation with nonlinear contacts
Large displacements
2 Millions DOF
12 Iterations
SPEED
11
SPEED
ACCURACY
LARGE ASSEMBLIES
FLUID DYNAMICS TRANSIENTS
MULTIPHYSICS SIMULATIONS
COMPUTER APPLICATIONS AND CHALLENGES
12
ASSESMENT OF DISPLACEMENTS AND STIFFNESS – HEAD COVER
ACCURACY
• Criticality in large models is given by the amount of available RAM.
• On certified Clusters, the use of Virtual Memory is not allowed, so, the size of the model to run is limited.
• On Workstations, because the use of Virtual Memory is allowed, it is possible to run larger models than those runnable on the Cluster.
13
Model with 114 Million degrees of freedom
ACCURACY
14
Elapsed time for resolution: 9 hours, 20 minutes (on a Workstation)
SPEED
ACCURACY
LARGE ASSEMBLIES
FLUID DYNAMICS TRANSIENTS
MULTIPHYSICS SIMULATIONS
COMPUTER APPLICATIONS AND CHALLENGES
15
ASSEMBLY MODEL, EMBEDDED PARTS AND CONDRETE, PORCE III
LARGE ASSEMBLIES
Interactions between different components can lead to design solutions that would not be possible in an isolated analysis
16
INTERACTION ELECTROMECHANICAL EQUIPMENT - POWERHOUSE
LARGE ASSEMBLIES
Linear-elastic analysis
10.9 millions DOF
237 pair of contacts
46 parts
2 cores 2 cores 6-8 cores 6-8 cores
Default: Direct solver, SMP, 2 cores
Iterative solver, SMP, 8 cores Iterative solver,
SMP, 8 cores
Improvement in elapsed time and memory used:
Large, for different solver configurations (91%).
Low, for different hardwares (2%).
Best results running on one node.
Benefits:
• Removal of simplifications and idealizations.
• Increased accuracy.
• Employment of more realistic behaviors for interactions between parts.
• Availability of extra information for designing.
2 cores 16-32 cores
Iterative solver, SMP, 16 cores
17
SPEED
ACCURACY
LARGE ASSEMBLIES
FLUID DYNAMICS TRANSIENTS
MULTIPHYSICS SIMULATIONS
COMPUTER APPLICATIONS AND CHALLENGES
18
WIND TURBINE TRANSIENT
FLUID DYNAMICS TRANSIENTS
Objective: Analyze the flow unsteady in the nacelle region due to the effect of the rotor disturbance on the oncoming wind field.
Benefits:
• Simulate 3D Transients (Blades, nacelle and tower). • Increase the number of elements in area of interest. • Turbulence model more representative of the reality.
Control the Wind Turbine in an optimal manner, in order to optimize the energy production
Statistics
N° of elements 4 Million
N° of iterations 6500
Time in 7 cores 3 weeks
Time in 32 cores 5 days
19
CFD SIMULATION OF PRESSURE FLUCTUATION ON HYDRAULIC TURBINES
FLUID DYNAMICS TRANSIENTS
• By increasing calculation power we can perform complex simulations, complying deadlines defined in the project’s design phase.
• Evaluation of the effect of vane installation in the draft tube cone on the pressure fluctuations for a rehabilitation project.
20
CALCULATION SPEED INCREASE
FLUID DYNAMICS TRANSIENTS
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Cal
cula
tio
n S
pe
ed
Nodes Millions
Workstations
Cluster
𝑵°
𝑰𝒕𝒆
𝒓𝒂
𝒕𝒊𝒐
𝒏𝒔
𝑬𝒒
𝒖𝒂
𝒕𝒊𝒐
𝒏𝒔
∗𝑻
𝒊𝒎𝒆
∗𝑪
𝒐𝒓
𝒆𝒔
• On the range of everyday meshes (5-8 millions), Dell HPC Cluster increases the calculation speed respect to the Workstations .
• Dell HPC Cluster allows us to use meshes up to 35 millions elements.
21
NORMALIZED TIME REDUCTION
FLUID DYNAMICS TRANSIENTS
𝑻𝒊𝒎
𝒆
𝑵𝒐
𝒅𝒆
𝒔∗
𝑰𝒕𝒆
𝒓𝒂
𝒕𝒊𝒐
𝒏𝒔
1
10
100
1000
0 8 16 24 32 40
No
rmal
ize
d T
ime
N° Cores
Cluster
Workstation
22
Parallelization improves run times
BELOMONTE – ROTOR & SATOR
FLUID DYNAMICS TRANSIENTS
y = 1026x-0.519 R² = 0.9796
0 mn
50 mn
100 mn
150 mn
200 mn
250 mn
300 mn
350 mn
400 mn
0 20 40 60 80 100 120 140
Tiem
po
[m
inu
tos]
N° de Cores
HPC Cores / Nodes
7500 Cores / Nodes
Statistics
N° of elements 48 Million
N° of iterations 60
Time in 7 cores 5h50
Time in 32 cores 2h20
23
A 70% time decrease could be obtained
SPEED
ACCURACY
LARGE ASSEMBLIES
FLUID DYNAMICS TRANSIENTS
MULTIPHYSICS SIMULATIONS
COMPUTER APPLICATIONS AND CHALLENGES
24
• Multiphysics simulations of fluid/structure interactions can be approached in two different ways :
Modeling fluid through acoustic elements or
Using a bidirectional interaction CFD and FEM simultaneously.
• Multiphysics simulations associated with acoustic phenomena, ie wave, are performed within the same scope FEM.
FLUID STRUCTURE INTERACTION - ACOUSTIC
25
NATURAL FREQUENCIES, RUNNER AND SHAFT LINE
FLUID STRUCTURE INTERACTION - ACOUSTIC
26
STAY VANES NORMAL MODES
FLUID STRUCTURE INTERACTION - ACOUSTIC
Total 1,168,971
Estructura 582,401
Fluido 800,770
Total 829,350
Estructura 322,077
Fluido 507,273
2,535,484
Out of Core 10 GB
Incore 50 GBTotal used
Maximum
Memory
Statistics
Number of equations
Mesh
Number of
nodes
Number of
elements
27
EVALUATION OF DYNAMIC RESPONSE AND COUPLINGS BETWEEN HYDRAULIC DUCTS AND STRUCTURAL PARTS
FLUID STRUCTURE INTERACTION - ACOUSTIC Harmonic Response
200 m
Vibrations caused by pressure fluctuations in the draft tube can be predicted
28
CONCLUSIONS
• HPC Cluster has increased the CFD calculation capability. Now bigger and more complex models can be used in design stages (complete machines, multi-phase simulations, etc.).
• Optimal hardware configurations for CFD are not the same as FEM.
• CFD computations take advantage of cores number, in a different way that FEM.
• Time grows exponentially, when FEM resolution needs to use virtual memory.
• In FEM models, run times can be optimized greatly by using correct settings according to the hardware resources, size of the model and type of analysis.
• Optimization of the elapsed time is closely related to the solver configuration. Large improvements can be achieved even when a local workstation with 2 cores is used. In some cases best results are obtained by using only one node having a maximum amount of cores.
• Licensing strategy, "HPC Pack" or "Workgroup", must be carefully designed, depending on the amount of users and available hardware resources.
29
FUTURE CHALLENGES
• Do a benchmark with GPU platforms.
• Assess the impact on performance by swapping on "SSD" instead on a "HDD".
• Setting up a specific workstation for FEM simulations, compounds by a large amount of RAM and as many cores as possible.
• Full fluid structure interaction CFD&FEM.
30
107 years…
La Tasajera
… innovating
HP Tocoma
107 years…
Bom Jardin
… providing total solutions
Agua Doce
www.impsa.com