ww

w.in

l.govDTSTEP:Development of an Integer Time Step Algorithm

George L MesinaINL

David L Aumiller

BMPC

September 2010

Outline

• Background and issues

• Integer time step

• Solutions and implementation

• Testing

• Conversion

Background: PVM Coupling Time Step Methods• Synchronous coupling requires all coupled codes to use the same dt.

– All codes suggest dt to the Executive every time step– The Executive limits dt based on the halving/doubling scheme

• dt = DTMAX(Exec)/2k where dt <= smallest suggested dt.

• Asynchronous coupling only requires codes to arrive at an Executive mandated time target. Can use own dt, DTMAX, DTMIN, etc.

– Parallel Asynch, each code marches independently to target

– Sequential Asynch, one code marches to target, sends data to other; other marches to target.

Background: PVM Coupling Thermal-Hydraulic Methods

• Explicit coupling exchanges data at each coupling time.– Simple method that is numerically unstable for many situations.

• Semi-implicit coupling exchanges extra data at each time step to add stability.

• Only certain combinations of time step methods and thermal-hydraulic methods are permissible.

Allowable Semi Implicit Explicit

Synch Required Allowed

Asynch Parallel Sequential

N/A Allowed Allowed

Principal Issue: Time Step Inaccuracy Issue• With 64-bit floating point variables on an Opteron chip

– Accurate to approximately 14 digits

– The final value of timehy = 1005.0000000745, not 1005.0 exactly.• This is inaccurate in the 12 digit.

• Inaccuracy builds up with longer time intervals.

• Such inaccuracy caused PVM-coupled codes to deadlock (hang).– For example, one code would hit a time target and send the proper

message while the other would step past it and eventually send a totally different message to the first code.

• Often these hangs were caused by special cases, e.g. velocity flip-flop.

T=1000.0; dt=0.005 Do 10 j = 1, 1000 10 timehy = timehy + dt

Sources of PVM Communication Issues• Asynchronous codes can have different DTMAX and DTMIN.

– Can cause time disparity for floating point sums.– Original solution was to create “epsilon-tolerance if-tests”

• Synchronous RELAP5-3D– Time targets were supplied, but change of timecard was not.

• RELAP5-3D DTSTEP fundamental algorithm required end time to operate correctly.

– Both Executive and RELAP5-3D DTSTEP routines calculated synchronous cumulative times and time-based actions.• Produced time discrepancies that deadlocked the machine.

If (abs(timehy – target) <= eps) TAKE ACTION

If DTMIN <= eps, can trigger action before actual target.

If eps too small, can step right over target.

Summary of DTSTEP Subroutines and Issues• Floating point inaccuracy caused trouble hitting time-targets exactly

– Especially problematic for PVM communications– General solution is integer time step

• There were other algorithmic and implementation errors– These will be listed later with individual solutions

• DTSTEP did too much for one subprogram

• It was very difficult to read and understand– Too many jumps – Logical variables with non-mnemonic names– Too many pre-compiler directives and conditional coding– Almost no internal documentation

Original RELAP DTSTEP Flow (Simplified)

InitializationNormal &1st time only

Abrupt ChangesRepeat with same dt, halve/double, terminate transient

Gather Statistics

New AdvancementEnd timecard or transient, SS check

Output (9 kinds)

Screen, plots, minor, major, restart, GUI

PVM Explicit ExchangeParallel or Seq, synch or asynch

New Time TargetsOptional PVM Semi-imp. rcv, New time targets

User Interfaces

Time step controldt-stretch, Courant/mass error,dt >= DTMAX/2k

Time step selectionOptional PVM Synch Exch, 105 card handler, advance time, EXIT

Integer Time Step Based upon “Clock-cycle”• RELAP5-3D and PVMEXEC use a timestep halving/doubling scheme

that guarantees hitting integer multiples of the user’s DTMAX– Time targets from RELAP5-3D timecards are such multiples– plot, minor, major, restart edits– Special targets, timecard endtimes and PVM targets, need not be.

• No step smaller than user’s DTMIN allowed. Smallest allowable dt is:– dtsmall = DTMAX/2n > DTMIN > DTMAX/2n+1.

• All normal values of dt = 2k dtsmall, k=0, 1, . . . , n.• Dtsmall is effectively the clock-cycle of RELAP5. Always has been.• To convert to integer time step, use dt = Round(dt/dtsmall) = 2k.

– Only within DTSTEP, outside DTSTEP use dt = dt * dtsmall.

dtsmall = DTMAX/2n

dt = Round(dt/dtsmall) = 2k

Integer Time Steps vs. Floating Point Time• Integer arithmetic is infinitely precise for add, subtract and multiply.

– The earlier do-loop-10 example creates no summing error.– Error tolerances are eliminated.

• Time, T = t * dtsmall, is more exact and slightly different from floating point calculated time.

– Most of your RELAP5-3D results will be slightly different.• Integer cumulative time becomes large and requires a 64-bit integer.

– 263 = 23 260 = 8(1024)6 > 8x(103)6 = 8x1018.– This is sufficient to represent 109 seconds with DTMIN = 10-9.

• Integer Time, t, starts at zero on each successive timecard.– T = t * dtsmall + T(start of timecard) = floating point time.– Required because user can change DTMAX and DTMIN.

T = t * dtsmall + T(start of timecard)

Comparison of Integer and Real Time

t (seconds)

t (cycles)0 .004

0

.008

32784 65568

1.00

819600

Exampledtmax=0.004

dtmin=1.0e-7 n = 15dtsmall=1.220703125e-7

real integer

dtsmall DTMAX/2n 1

Largest dt DTMAX 2n

Test |x-y| < eps ix == iy

Time Tbgn + Sk dtk Tbgn +

dtsmall*Sk idtk

Normal vs. Unusual Target Times• Normal time target, T, is an integer multiple of DTMAX• Unusual time target, T, is not integer multiple of DTMAX.

– e.g. DTMAX= 0.33, Timecard end = 10.0, Assume no cuts.• At T = 9.9, need dt = 0.1, but 0.1 /= DTMAX/2k.

• Unusual times are handled by using a floating point dt, which exactly reaches the unusual time, and a resynch variable.

– The resynch variable indicates need to resynch dt and dt.• e.g. Dtmax = 0.004, dtmin = 1.0e-7, dtsmall =1.22e-7,

target=1.00021

t (seconds)1.000 1.002 1.004

t2=1.0021

t3

t (cycles) t1 t2 t3=t1+dtsav

t1

Some Other DTSTEP Errors• Explosive Doubling

– When PVM and RELAP5-3D timecards differed• with R5 timecard end time exceeding PVM transient end-time,

– The timestep could double on every timestep until the R5 timecard end time was exceeded.

– The source was variable, NREPET.– The solution: PVM timecards override the RELAP5-3D timecards.

• Penultimate timecard stop– Some asynchronous processes stopped at end-time of second to

last timecard.– Cause: an error in resetting variable CURCLM in R-DTSTEP.– This error has been fixed.

Some Other DTSTEP Errors (continued)• Hang when subroutine HYDRO set variable, FAIL

– R5 incorrectly sent a message that the timestep was succesful– Solution: Set PVM success message to repeat (as R5 repeats)

• Hang in explicit asynchronous sequential– When follower received go-ahead message from leader, but had to

repeat its first advancement.• It returned to “receive go-ahead from leader message” =>

hang– Solution: Logical variable prevents return to this receive message

until time actually advances.

• Hang from repeat condition at minimum dt (expl asyn parallel)– In certain cases RELAP5 DTSTEP subroutine incorrectly set dt

twice the size of EXEC dt.– Solution: Rewrote/simplified R-DTSTEP code correctly.

Some Other DTSTEP Errors (continued)• Stretch logic error

– On a 2-step approach to an unusual time, first step was half the distance, but second step went to multiple of DTMAX, not endtime

– Source: one real target time was k*DTMAX, not endtime– Solution: (P- and R-DTSTEP) reset ALL time targets to the unusual

endtime when within one DTMAX

• Error Messages P-DTSTEP and some PVM diagnostics were improved.

Solutions and Implementation• Replace floating point time with integer based time.

– Extra integer variables were created in both R-DTSTEP and P-DTSTEP

– Replaced floating point if-tests by integer versions

– Created internal subroutines for initializing and calculating integer time from real and vice-versa.

– Create unifying mechanisms for quantities and processes shared by both DTSTEPS (next slide)• Shared module

Some Unifying Implementation Features• Isnormal – determines if time is normal or unusual

– abs(1-dt/idt*dtsmallest) < 1.0e-10– Interpretation: dt & idt*dtsmallest agree to 10 places => dt normal.

• Calctimehy– Calculate real time from certain integer time variables– Uses quad precision to guarantee correct conversion

• idtmod – f95 module– Contains internal subroutines and data common to both P- and R-

DTSTEP subroutines• Used by both

– Ensures both perform same functions in same way– Reduces coding, increases maintainability

Implementation continued• Simplified and enhanced both RELAP5 DTSTEP and PVMEXEC

DTSTEP by moving portions of coding into internal subroutines.– Separated flowchart sections were pulled together.– Create internal subroutines for reused sections of code, e.g. output– Create internal subroutines for most precompiler-protected code– Replace jumps to a section of code with call of internal subroutine

• Enhance readability– Determine definitions of all variables and document– Give mnemonic names to logical variables– Implement Outline-style documentation for sections of code– Simplify code and reduce logic paths– Eliminate dead code and unused variables

Implementation continued• Identified and fixed algorithmic and implementation errors

• Created new test cases to increase coverage–Especially for failure conditions and PVM coupling

• Verified that all test cases run correctly.

DTSTEP Test Matrix• To fully test the relevant logic paths through DTSTEP, a matrix of over

2000 cases was constructed.• BASIC – 17 tests cases that cover normal, special case, and failure

case and some combinations.

Group A (34)Basic @ normal dtBasic @ minimal dt

Group B (102)Group A @ minor editGroup A @ expl exchangeGroup A @ transient end

Group C (408)Group B w/ normal targetGroup B w/ unusual timeGroup B w/ normal then unusual targetGroup B w/ target every step

Group D (2856)Group C R5 standaloneGroup C R5-R5 expl asynch parallelGroup C R5-R5 semi-implGroup C R5-R5-R5 expl & semi-implGroup C R5-R5 expl synchGroup C R5-R5 expl asynch sequGroup C R5 controled by Exec

Conversion to F95• The integer algorithm was developed in version 2.4.

– These use the old container array (FA) and equivalences.

• All R-DTSTEP, P-DTSTEP, and related changes were ported to version 2.9.5

– Merge issues resolved (post-2.4 development and differences in integer time implementation)

• Conversion to F95– Access F95 modules, replace FA variables with module ones

• Resolution of variable declarations– Issue with precompiler directive and 32-bit PVM

• Verified again that all test cases worked correctly.