arsc initiatives in weather, climate and ocean modeling
Post on 09-Jan-2016
21 Views
Preview:
DESCRIPTION
TRANSCRIPT
ARSC Initiatives in Weather, ARSC Initiatives in Weather, Climate and Ocean ModelingClimate and Ocean Modeling
Dr. Gregory Newby, Chief ScientistDr. Gregory Newby, Chief Scientist
Arctic Region Supercomputing CenterArctic Region Supercomputing Center
Presentation to the HPC User FormPresentation to the HPC User Form
September 9 2008September 9 2008
Common Themes with Other Common Themes with Other User Forum PresentersUser Forum Presenters
• Lots of work with WRFLots of work with WRF• Work with CCSMWork with CCSM• Mature community-developed applications that scale Mature community-developed applications that scale
reasonably well and are hungry for more CPU powerreasonably well and are hungry for more CPU power• Desire for higher resolution to better approximate Desire for higher resolution to better approximate
realityreality• Desire to add increasingly sophisticated processes to Desire to add increasingly sophisticated processes to
the phenomena under investigationthe phenomena under investigation– Therefore, these items will not be major themes of today’s Therefore, these items will not be major themes of today’s
talktalk
ARSC Themes of potential interest to ARSC Themes of potential interest to the HPC User Forumthe HPC User Forum
• Performance analysis of current (multicore) and Performance analysis of current (multicore) and forthcoming (Cell, GPU, FPGA, T2+) processors for WRFforthcoming (Cell, GPU, FPGA, T2+) processors for WRF
• Quasi-operational WRF with NWS/NOAA usersQuasi-operational WRF with NWS/NOAA users• Wildfire smoke prediction based on WRF/ChemWildfire smoke prediction based on WRF/Chem• Model couplingModel coupling• High resolution ocean forecastsHigh resolution ocean forecasts• Arctic ice tide impactsArctic ice tide impacts• WRF sensitivity analysisWRF sensitivity analysis
– These will be the major themes of today’s talkThese will be the major themes of today’s talk
Multicore Performance Analysis on Multicore Performance Analysis on ARSC’s SupercomputersARSC’s Supercomputers
• Mostly on Midnight (dual core)Mostly on Midnight (dual core)– Sun x2200m2 & x4600 cluster nodesSun x2200m2 & x4600 cluster nodes– 2312 Opteron cores, IB, 4GB/core, Lustre2312 Opteron cores, IB, 4GB/core, Lustre– 12.88 TFLOP theoretical peak12.88 TFLOP theoretical peak
• BenchmarksBenchmarks• Real-world applicationsReal-world applications• Forthcoming: Pingo (quad core)Forthcoming: Pingo (quad core)
– Cray XT5Cray XT5– 3456 cores, SeaStar, 4GB/core, Lustre3456 cores, SeaStar, 4GB/core, Lustre– About 30 TFLOP theoretical peakAbout 30 TFLOP theoretical peak
2-socket AMD64 topology2-socket AMD64 topology
One 1 GHz 16x16 HyperTransport link per supported processor with 8GB/second bandwidth
CPU-0,1 CPU-2,3
8 socket AMD64: Sun Fire 8 socket AMD64: Sun Fire X4600 ServerX4600 Server
8 socket AMD64 topology8 socket AMD64 topology
CPU0
CPU5CPU3
CPU6CPU4
CPU1
CPU2
CPU7
* All HT links are operating at 1GHz and 8GB/s
WRF Performance on WRF Performance on MidnightMidnight
Time required to compute 1 forecast hour on 3-nest domain
Next-Generation Processor Next-Generation Processor PerformancePerformance
• GPU, Cell, quad-core Xeon & Opteron, FPGA, CMTGPU, Cell, quad-core Xeon & Opteron, FPGA, CMT– Many benchmarks completed, some applicationsMany benchmarks completed, some applications
• Forthcoming: WRF test cases for AK domains (submitted Forthcoming: WRF test cases for AK domains (submitted for AGU 2008)for AGU 2008)– GPU (nVidia 9800 GTX), possibly FirestreamGPU (nVidia 9800 GTX), possibly Firestream– FPGA: WRF doesn’t have enough “hot spots,” porting BLAS FPGA: WRF doesn’t have enough “hot spots,” porting BLAS
too much work (but RapidMind might help)too much work (but RapidMind might help)– Cell: QS22 cluster being configured; should work wellCell: QS22 cluster being configured; should work well– UltraSPARC T2+: scaled well to 8 threads per socket; WRF UltraSPARC T2+: scaled well to 8 threads per socket; WRF
is too CPU-bound to benefit much from T2+ CMTis too CPU-bound to benefit much from T2+ CMT
Quasi-Operational and Research Quasi-Operational and Research Weather ModelingWeather Modeling
• Work has been evolving since 2005Work has been evolving since 2005• Initially, efforts were directed at quasi-Initially, efforts were directed at quasi-
operational, scheduled WRF forecasts for the operational, scheduled WRF forecasts for the Fairbanks Weather Forecast Office (WFO). The Fairbanks Weather Forecast Office (WFO). The goal was to leverage lots of CPUs to provide goal was to leverage lots of CPUs to provide high-resolution forecasts, and continually high-resolution forecasts, and continually assess & improveassess & improve
• The scheduled runs continue to dominate our The scheduled runs continue to dominate our efforts, but rigorous analysis and study has efforts, but rigorous analysis and study has yielded important resultsyielded important results
2-way Nested Domains for 2-way Nested Domains for ARSCwrfARSCwrf
200x200x75 cells 421x328x75 cells 151x151x75 cells
Operational WRF – AWIPSOperational WRF – AWIPS
This is the end product of our twice-daily WRF runs, as seen on the NWS Advanced Weather Interactive Prediction System (FAI WFO)
Value to WFO FAIValue to WFO FAI• Grid resolution is Grid resolution is
approximately 5km. approximately 5km. The ability to populate The ability to populate with a reasonably with a reasonably accurate, high-accurate, high-resolution weather resolution weather model saves the model saves the forecasters a lot of forecasters a lot of time. In the best case, time. In the best case, the model output the model output isis the NDFD product.the NDFD product.
Verification ProductsVerification Products• Four days after a Four days after a
forecast, we forecast, we automatically automatically retrieve all retrieve all available available observations and observations and tabulate a tabulate a comparison of the comparison of the WRF forecast for WRF forecast for each location vs. each location vs. the observationsthe observations
Data AssimilationData Assimilation
• Motivation – better set of initial conditionsMotivation – better set of initial conditions• Current initial conditions are interpolated from coarser resolution model runs – Current initial conditions are interpolated from coarser resolution model runs –
clearly much room for errorclearly much room for error• With data assimilation, we perturb our original input grid by carefully With data assimilation, we perturb our original input grid by carefully
assimilating available observations assimilating available observations • We are currently in the process of adding this capability to our operational runs, We are currently in the process of adding this capability to our operational runs,
and once implemented will perform assimilated and non-assimilated runs side-and once implemented will perform assimilated and non-assimilated runs side-by-side for comparisonby-side for comparison
Data AssimilationData Assimilation• Test Case – 48-hour forecast starting at 00Z on 01 May 2007Test Case – 48-hour forecast starting at 00Z on 01 May 2007
Surface obs – temperature, dewpoint,pressure, winds
Vertical soundings from raobs and satellite – temperature, dewpoint, pressure, winds
Alaska Air Quality Impacted by Alaska Air Quality Impacted by WildfiresWildfires
South Fairbanks, July 6, 2004. Air quality particulate level at approximately 10 micrograms/cubic meter.
South Fairbanks, June 28, 2004.Air quality particulate level at approximately 900 micrograms / cubic meter.
Photos courtesy of Dr. James Conner, FNSBhttp://www.dec.state.ak.us/air/am/2004_wf_sum.htm
Initial architecture: WRF/Chem Smoke Dispersion SystemInitial architecture: WRF/Chem Smoke Dispersion System
FIRE EMISSIONSFOFEM
MODIS FIRE DETECTION & BURN AREA
WRF-CHEM PARTICULATE, CHEMICAL &
METEOROLOGICAL FORECAST
WRF-MET METEOROLOGY FORECAST
Gridded Hourly Emissions
PLUME DYNAMICS
FUEL MOISTURE
FIRE SPREAD
Identical WRF domain initialization (SI/WPS)
WRF/Chem
POSTPROCESSINGWRF-Chem netCDF
Static Fuel DataEmission Factors
DEM, Static Fuel
Adapted from a scheme used by our partners from the US Forest Service Fire Science Lab in Missoula.
Fire Detection and Burn Area ExampleFire Detection and Burn Area Example
Example Product for Monday, 8th September 2008, 9:00 UTCExample Product for Monday, 8th September 2008, 9:00 UTC
A weak smoke concentration due to North-Easterly winds and extended fires North/East of the Yukon has been clearly confirmed in the Fairbanks area. Source: smoke.uaf.edu
Coupling Work: Regional Arctic Climate Coupling Work: Regional Arctic Climate Model & MoreModel & More
• Practical ongoing work, DoE funded. Wieslaw Maslowski, PIPractical ongoing work, DoE funded. Wieslaw Maslowski, PI– ARSC is also looking at some more general approaches to coupled climate ARSC is also looking at some more general approaches to coupled climate
models, including adding permafrost, ice sheets, hydrology, ecosystems, human models, including adding permafrost, ice sheets, hydrology, ecosystems, human infrastructure, and coastal erosion. Major partner: International Arctic Research infrastructure, and coastal erosion. Major partner: International Arctic Research Center, UAF. Center, UAF.
• RACM is a regional Arctic climate system model including WRF, RACM is a regional Arctic climate system model including WRF, VIC (land surface model), NAPC (ocean and sea ice model). VIC (land surface model), NAPC (ocean and sea ice model). Current emphasis is coupling WRF, VIC and NAPC via cpl7. Current emphasis is coupling WRF, VIC and NAPC via cpl7.
• Cpl7 is still in development. Our work will keep pace with cpl7 Cpl7 is still in development. Our work will keep pace with cpl7 progress. CCSM4 project groups have made early-release codes progress. CCSM4 project groups have made early-release codes available to RACM partners available to RACM partners
• Since cpl7 is designed for CCSM4, where the framework is Since cpl7 is designed for CCSM4, where the framework is different with WRF and VIC, some changes of cpl7 and WRF are different with WRF and VIC, some changes of cpl7 and WRF are necessary.necessary.
ATM
LND OCN
CPL
ICE
MPI_COMM_WORLD
CPLATM
CPLATMCPLLND
CPLICE
5 COMPONENTS10 COMMUNICATION GROUPS
Partitioning of RACM Processing
ROMS and BeyondROMS and Beyond
• ROMS is an ocean model that can operate at ROMS is an ocean model that can operate at both very large and very small scales, and both very large and very small scales, and takes shorelines into accounttakes shorelines into account
• At ARSC: Ongoing research for northwest At ARSC: Ongoing research for northwest Pacific, Gulf of Alaska, Beaufort Sea & Bering Pacific, Gulf of Alaska, Beaufort Sea & Bering SeaSea
• Also, partnering to couple ROMS with other Also, partnering to couple ROMS with other best-of-breed models for ocean / ice prediction best-of-breed models for ocean / ice prediction
CCSM3 Coupling for CCSM3 Coupling for Ocean / Ice ModelingOcean / Ice Modeling
The ROMS Regional SetupsThe ROMS Regional Setups
NutrientPhytoplanktonZooplankton models
Secondary Producers(Zooplankton)
Fish
NutrientsNO3, NH4…
Primary Producers(Phytoplankton)
Physical Forcing
(Wind, temperatureSunlight, mixing)
ROMS Driving ROMS Driving Ecosystem Model Ecosystem Model
Beaufort Sea WRF Beaufort Sea WRF Sensitivity AnalysisSensitivity Analysis
• Funded by MMSFunded by MMS• Emphasis includes near-coast oil Emphasis includes near-coast oil
spills in Beaufort Sea (north of AK & spills in Beaufort Sea (north of AK & Yukon Territory)Yukon Territory)
• Many scenarios, many WRF runsMany scenarios, many WRF runs– Reanalysis studyReanalysis study– Storm case studiesStorm case studies– Impact of winds on wavesImpact of winds on waves
MMS WRF Report: Higher Resolution not needed MMS WRF Report: Higher Resolution not needed for Effective Near-Shore Wind Fieldsfor Effective Near-Shore Wind Fields
From Field to Supercomputer: Designing Next-From Field to Supercomputer: Designing Next-Generation Sea Ice Modules for Very High Generation Sea Ice Modules for Very High
Resolution Ice-Ocean ModelsResolution Ice-Ocean Models
Arctic System Modeling
Tidal and wind forcing of the ice-ocean boundary layer verses simple wind forcing (14km resolution)
These animations demonstrate the difference between simulations being purely wind driven (right), and those that are both tidally and wind driven (left). Both animations span the same period, and have the same timestep, which is roughly 50 minutes)
ConclusionsConclusions
• Climate, weather and related phenomena (sea ice, land Climate, weather and related phenomena (sea ice, land surface, physical oceanography) models tend to be surface, physical oceanography) models tend to be community developed and supportedcommunity developed and supported– Scale fairly wellScale fairly well– Ported & maintained for new hardwarePorted & maintained for new hardware– Mostly Fortran + MPI, but very complex codesMostly Fortran + MPI, but very complex codes
• Scientists are hungry for higher resolution, longer runs, Scientists are hungry for higher resolution, longer runs, and additional runs to parameterize phenomenaand additional runs to parameterize phenomena
• The Arctic requires several adjustments to models for The Arctic requires several adjustments to models for optimal verisimilitude, versus mid-latitudesoptimal verisimilitude, versus mid-latitudes
Huge THANKSHuge THANKS• Don Morton (ARSC & U. Montana): WeatherDon Morton (ARSC & U. Montana): Weather• Abdullah Kayi (GWU): HyperTransportAbdullah Kayi (GWU): HyperTransport• Martin Stuefer (UAF), Georg Grell (NOAA) & Saulo Martin Stuefer (UAF), Georg Grell (NOAA) & Saulo
Freitas (CPTEC/INPE): Wildfire smokeFreitas (CPTEC/INPE): Wildfire smoke• Kate Hedstrom (ARSC): ROMSKate Hedstrom (ARSC): ROMS• Georgina Gibson (ARSC): NPZGeorgina Gibson (ARSC): NPZ• Wieslaw Maslowski (NPS): POP, RACMWieslaw Maslowski (NPS): POP, RACM• Andrew Roberts (ARSC), Jennifer Hutchings (UAF): Tidal Andrew Roberts (ARSC), Jennifer Hutchings (UAF): Tidal
iceice• Juanxiong He (ARSC): CouplingJuanxiong He (ARSC): Coupling• Jing Zhang, Jeremy Krieger (UAF), Don Morton: MMS Jing Zhang, Jeremy Krieger (UAF), Don Morton: MMS
WRF sensitivityWRF sensitivity– Many other partners & collaboratorsMany other partners & collaborators
top related