The COSMO Project

Overview, Status and PlansOliver Fuhrer, MeteoSwiss

Project Team: Anne, Christoph, Christoph, Daniel, David, Dominik, Florian, Isabelle, Jean-Guillaume, Jrg, Katharina, Lukas, Matthew, Men, Neil, Oliver, Thomas, Thomas, Tim, Tobias, Ulrich, Will, Xavier

The InititativeHP2C = High Performance High Productivity ComputingOverarching Goal Prepare computational sciences to make effective use of next generation supercomputersSpecific Goal Emerge with several high-impact scientific applications that scale and run efficiently on leadership computing platforms in 2012/13 timeframeWebsite http://www.hp2c.ch/Courtesy: Thomas Schulthess

The ApproachStart with a real and challenging scientific problemTake the best methods known in the fieldUnderstand their limitations on todays emerging hardwareAlgorithmic reengineering to better map method to hardwareTake aggressive view on rewriting codes dont hesitate to rewrite major portions or even starting from scratchPay close attention to hardware developmentsCourtesy: Thomas Schulthess

Project OrganizationBigDFT - Large scale Density Functional Electronic Structure Calculations in a Systematic Wavelet Basis Set (Prof. Stefan Goedecker, University of Basel)Cardiovascular - HPC for Cardiovascular System Simulations (Prof. Alfio Quarteroni, EPF Lausanne)COSMO-CCLM - Regional Climate and Weather Modeling on the Next Generations High-Performance Computers: Towards Cloud-Resolving Simulations (Dr. Isabelle Bey, ETH Zurich)Cosmology - Computational Cosmology on the Petascale (Prof. George Lake, Uni Zrich)CP2K - New Frontiers in ab initio Molecular Dynamics (Prof. Juerg Hutter, Uni Zrich)Ear Modeling - Numerical Modeling of the Ear: Towards the Building of new Hearing Devices (Prof. Bastien Chopard, University of Geneva)Gyrokinetic - Advanced Gyrokinetic Numerical Simulations of Turbulence in Fusion Plasmas (Prof. Laurent Villard, EPF Lausanne)MAQUIS - Modern Algorithms for Quantum Interacting Systems (Prof. Thierry Giamarchi, University of Geneva)Neanderthal Extinction - Individual-based Modeling of Humans under climate Stress (Prof. C. P. E. Zollikofer, University of Zurich)Petaquake - Large-Scale Parallel Nonlinear Optimization for High Resolution 3D-Seismic Imaging (Prof. Olaf Schenk, Uni Basel)Selectome - Selectome, looking for Darwinian Evolution in the Tree of Life (Prof. Marc Robinson-Rechavi, University of Lausanne)Supernova - Productive 3D Models of Stellar Explosions (Prof. Matthias Liebendrfer, University of Basel)

SetupThe COSMO ModelInputPhysicsDynamicsAssimilationBoundary ConditionsDiagnosticsOutput

Lines vs. Runtime~250k lines of Fortran 90 code% Code Lines

Solve PDE of diffusion equation (4 FLOPS, 3 load/stores)Prototype Code 1 (Dynamics)}}halo-update

Maschinebuin (XT4)AMD BarcelonaApril 2008rosa (XT5)AMD IstanbulJune 2009palu (XE6)AMD Magny-CoursApril 2010# cores4612Single core0.802 s0.836 s0.631 sAll cores0.558 s0.459 s0.183 sSpeedup1.41.83.4

Higher computational density (136 FLOPs, 3 load/stores)Prototype Code 2 (Physics)no comm

Maschinebuin (XT4)AMD BarcelonaApril 2008rosa (XT5)AMD IstanbulJune 2009palu (XE6)AMD Magny-CoursApril 2010# cores4612Single core17.6 s17.3 s18.8 sAll cores4.4 s2.9 s1.6 sSpeedup4.06.012.0

Memory ScalingExample: COSMO-2 like on Cray-XT4Problem size: 102 x 102 x 60 gridpoints on 60 coresKeep number of cores constant, vary number of cores/node used

Relative Runtime (4 cores = 100%)PPP

The COSMO ProjectTask 1 Prepare COSMO for convection resolving IPCC runs

Task 2 Incremental optimization of existing codeUnderstanding of workflow, benchmarkingHybrid parallelization MPI/OpenMPParallel I/OTalks: Anne+Jean-Guillaume, Neil, Matt

Task 3 Fundamental redesign of COSMO codeRedesign data architecture for optimal cache usageRewrite dynamical coreInvestigate shift to GPUsTalks: Tobias, Xavier, Will

Duration June 2010 December 2012

GPU ImplementationGPUs are very attractivehigher memory bandwidthhigher FLOPS per Wattlow costButneed to consider the whole workflowCOSMO is a community code and will always (?) have to run on a commodity clusterthe code should stay easy to understand and modify

New project proposal (OPCODE) for a GPU implementation of MeteoSwiss operational suite submitted

Other ApplicationsApart from plain vanilla COSMO runs, COSMO is also run in exotic configurations

ExamplesCOSMO-CLMNetCDF input/outputCOSMO-ARTgas-phase and aerosol chemistryCOSMO-M7aerosol-radiation feedbackCOSMO-CLM2land surface scheme

These configurations may potentially change everything

Example: COSMO-ART

COSMO-ARTAdditional 58 gas and 105 aerosol variables (normally 11).Of which 56 and 77 are transported and diffused.COSMO (6 min)COSMO-ART (105 min)24h simulationCray XT4 (dole)182 x 170 x 40 gridpoints10 x 11 cores

General ConclusionsPOMPA is the platform to bring the HP2C activities into the COSMO Consortium and collaborate/coordinate activities within COSMOBringing domain scientists, computational scientists and computer engineers together takes time, but it is worth the effortTake a look at https://hpcforge.org/projects/cclm-dev/

Thank you!