toward a standard benchmark format and suite for floating ...ztatlock/pubs/fpbench...growing pains...

TowardaStandardBenchmarkFormatandSuiteforFloating-PointAnalysis

Nasrine Damouche,Matthieu Martel,PavelPanchekha,ChenQiu,AlexanderSanchez-Stern,ZacharyTatlock.

Incredibleprogress…

AutomaticVerificationFluctuat [SAS’13]Rosa[POPL’14]FPTaylor [FM’15]

OptimizationSTOKE[PLDI’14]

ImprovementSalsa[FMICS’15]Herbie[PLDI’15]

MechanizedProofsWaveequation[ITP’10]Roundingerror[NSV’16]

Rapidimprovementinhardproblems!


AutomaticVerificationFluctuatRosaFPTaylor

OptimizationSTOKE

ImprovementSalsaHerbie

ManualVerificationWaveequationRoundingerror

Next???


Wewantourcommunitytokeepprogressing!


AutomaticVerificationFluctuatRosaFPTaylor

OptimizationSTOKE

ImprovementSalsaHerbie

ManualVerificationWaveequationRoundingerror

Next???


Wewantourcommunitytokeepprogressing!

Ascommunitygrows,growingpainsappear

Growingpains

Herbie:ulp(NaN, Inf) = UINT_MAXSTOKE:ulp(NaN, Inf) < UINT_MAX

Fluctuat:Poly, Inv, F1a, F1b, idem, …FPTaylor:sine, sqrt, verhulst, …

Rosa:def example(x: Double): Double = …Salsa:double example(double x) { … }𝑥 + 1� − 𝑥�

Similargrowingpainsincompilers,HPC,SAT,SMT,…

communities

Rosa Salsa

Compo

sition

FPTaylorFluctuatEvalua

tion

STOKEHerbie

Stan

dardiza

tion

FPBench

FPBench iscommunityinfrastructureforcooperation andcomparisonintheFPcommunity.

http://fpbench.org

BenchmarksuiteCommonformat Namedmeasures

FPBenchβ

(FPCore (x)

(- (sqrt (+ x 1)) (sqrt x)))

Arguments

S-expressionsyntax

𝑥 + 1� − 𝑥�

(FPCore (x):name “Sqrt Difference”:cite (hamming-87):pre (> x 0)(- (sqrt (+ x 1)) (sqrt x)))

𝑥 + 1� − 𝑥�

Metadata

Preconditions

(FPCore (x0):name “Sine Newton”:cite (darulova-kuncak-2014):pre (< (abs x0) 1)(while (< i 10)

([i 0 (+ i 1)][x x0 (- x (/ (+ (+ (- x (/ (pow x 3) 6.0))

(/ (pow x 5) 120.0))(/ (pow x 7) 5040.0))

(+ (+ (- 1.0 (/ (* x x) 2.0))(/ (pow x 4) 24.0))(/ (pow x 6) 720.0))))])

x))

Loops

Commonfunctions

FPCore commonformat

Expressive

Extensible

SimpletouseS-expressionsyntaxPurelyfunctionalNocontrolflowanalysis

AllC,FortranfunctionsLoops,conditionalsToolssupportparts

MetadatapropertiesTool-specificmetadataInputoroutputformat

Generatefromhigher-level,imperativeFPImp lang.

BenchmarksuiteCommonformat

Simpletoimplement

Coversallexistinguses

Simpletoextend,specialize

Namedmeasures

FPBenchβ

FPBench benchmarksuite

72totalbenchmarks

Drawnfromexistingpapers

Annotatedwithsource,ranges,description,citation

FPBench benchmarksuite

Richfeatures DiversedomainsExistingprograms

FPTaylor 29Herbie 28Rosa 6Salsa 9

Arith 72Expt 16Trig 11Loop 12Branch 3

Textbook 59Math Alg 6Emb Sys 4Sci Comp 3

Benchmarksuite

Fromexistingprojects

Covermanydomains

Growsovertime

Commonformat

Simpletoimplement



Namedmeasures

FPBenchβ

FPBench measures

Formaldefinitionsofaccuracymeasures

Describedalong5axes

Standardmeasuressotoolsagree

FPBench axesofmeasurement

Scalingvs.non-scaling

Forwardvs.backward

Maximumvs.average

Soundvs.statistical

Improvement

Absolute,relative,ULPs,bits,…

Fixedinputerrorvsfixedoutputerror

vs

Formalguaranteesvsmathematicalaccuracy

vs

Benchmarksuite

Fromexistingprojects

Covermanydomains

Growsovertime

Commonformat

Simpletoimplement



Namedmeasures

Termsformeasuringerror

Standardacrosstools

Flexiblebutrigorous

FPBenchβ

BenchmarksuiteCommonformat Namedmeasures

FPBench

http://fpbench.org

FPBench iscommunityinfrastructureforcooperation andcomparisonintheFPcommunity.

toward a standard benchmark format and suite for floating ...ztatlock/pubs/fpbench...growing pains...

Documents