Modernizing a Long-Lived Production Physics Code

[Extended Abstract]∗

Charles FerenbaughLos Alamos NationalLaboratory, MS B287

Los Alamos, NM 87545cferenba@lanl.gov

Sriram SwaminarayanLos Alamos NationalLaboratory, MS B287

Los Alamos, NM 87545sriram@lanl.gov

Chuck AldrichLos Alamos NationalLaboratory, MS T087

Los Alamos, NM 87545cha@lanl.gov

Matthew CalefLos Alamos NationalLaboratory, MS D413

Los Alamos, NM 87545mcalef@lanl.gov

Joann CampbellLos Alamos NationalLaboratory, MS T087

Los Alamos, NM 87545jomc@lanl.gov

Marcus DanielsLos Alamos NationalLaboratory, MS D413

Los Alamos, NM 87545mdaniels@lanl.gov

ABSTRACTLANL’s Eulerian Applications Project is modernizing itscode base to perform at large scale on Trinity and other fu-ture platforms. To make this possible, we are untangling thecomplicated dependencies that have built up over the years,and dividing the functionality into smaller, self-containedpackages. These packages will be well-documented, havewell-defined APIs, and be unit-testable. They will allow usto refactor code in individual packages with minimal impactto other parts of the code. We will then be able to work onthe optimizations for Trinity and other advanced architec-tures. This poster will describe the packagization strategy,our progress to date, initial optimization results, and futureplans.

Keywordslegacy code, code modernization, advanced architectures,optimization

1. INTRODUCTIONThe Eulerian Applications Project at LANL maintains

xRage[1], an Eulerian AMR radiation-hydronamics code.xRage is descended from Sage, a code originally writtenaround 1990. It contains about 470K lines of source code,not counting numerous third-party libraries from LANL andelsewhere. It is written mostly in Fortran 90, with some Cand C++, and uses MPI-only parallelism.

Twenty years of high-pressure mission work have left xRagewith significant technical debt. This made it difficult to:

∗The poster associated with this abstract is available fromthe SC16 website http://sc16.supercomputing.org.

This paper is authored by an employee(s) of the United States Government and is in thepublic domain. Non-exclusive copying or redistribution is allowed, provided that thearticle citation is given and the authors and agency are clearly identified as its source.

SC16 November 13–18, 2016, Salt Lake City, UT, USA

ACM ISBN 978-1-4503-2138-9.

DOI: 10.1145/1235

• understand the code flow or data flow

• maintain the code

• add new features

• train new developers

• refactor for advanced architectures

For the Trinity L1 Milestone in January 2019, EAP isrequired to run xRage across half of the Knights Landing(KNL) portion of Trinity. We needed to do major refactor-ing of the xRage code to enable the optimizations neededfor this milestone. But the high technical debt made thischallenging.

2. THE PACKAGIZATION PROCESSTo address this problem, we have created a code restruc-

turing strategy that we call “packagization.” The idea is toseparate out pieces of the code into packages, small sectionsthat implement a particular functionality, and have mini-mal, well-understood dependencies on other packages. Theend goal is to gradually transform our existing code baseinto a modular design that is cleaner, simpler, and moremaintainable.

This is done in three steps:

1. Expose physics state

• Create derived types for state of each package

• Pass state through argument lists, making dataflow explicit

2. Make packages modular

• Remove dependency cycles, levelize graph

• Create unit tests, document interfaces

3. Refactor packages

• Reorganize code for clarity, efficiency

• Optimize

By applying the first two steps to the packages of xRage,over a period of about fifteen months, we have created alevelized dependency graph with no cycles. This allows de-velopers to better understand the code structure and dataflow much more easily, and to begin the refactoring pro-cess. In particular, the xRage refactor enables optimizationefforts, to be described below.

Figure 1: xRage dependency graphs. Left: The “hairball” graph, October 2014. Right: The levelized graph,January 2016.

As part of the packagization process, we are modifyingthe xRage build system to build packages as libraries when-ever practical. This allows us to enforce the dependencyhierarchy on the libraries, and to build unit tests for eachpackage. As of this writing, over 90% of the packages inxRage are building as libraries, and the writing of unit testsis in progress.

3. THE ENAMR ABSTRACTION LAYERENAMR is a prototype abstraction layer for Fortran loop

constructs in xRage. It separates iterations over the meshfrom physics operations. It allows centralized changes toiteration and access patterns, which in particular will enableeasy implementation of threading.

ENAMR has been used to test four implementations ofcell iteration loops with different optimization strategies.Speedups of up to 40% were observed. In addition, ENAMRis being used to explore:

• OpenMP threading

• CUDA kernel execution

• Kernel aggregation

4. ADDITIONAL AUTHORSAdditional authors: Michael Hall, Scot Halverson, Thomas

Masser, Mike McKay, Zachary Medin, Ralph Menikoff, DavidNicholaeff, Bob Robey, Gabe Rockefeller, Jeremy Sauer, andJohn Wohlbier, all from Los Alamos National Laboratory.

5. REFERENCES[1] M. Gittings, R. Weaver, M. Clover, T. Betlach, N. Byrne, et

al. The RAGE radiation-hydrodynamic code. ComputationalScience & Discovery, 1(1):015005, November 2008.