how fast and fat is your probabilistic model checker? an experimental performance comparison david...
TRANSCRIPT
![Page 1: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/1.jpg)
How Fast and Fat IsYour Probabilistic Model
Checker?an experimental performance
comparison
David N. Jansen3,1, Joost-Pieter Katoen1,2, Marcel Oldenkamp2,
Mariëlle Stoelinga2, Ivan Zapreev1,2
1 MOVES Group, RWTH Aachen University2 FMT Group, University of Twente, Enschede3 ICIS, Radboud University, Nijmegen
![Page 2: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/2.jpg)
ProbabilisticModel Checker
ProbabilisticModel Checking
Probabilistic System Probabilistic Requirement
≤Probabilistic Model Probabilistic Formula
Yes No
Probability
ETMCC
MRMCPRISM(sparse)
PRISM(hybrid)
YMER
VESTA
![Page 3: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/3.jpg)
Why This Work?• Used more often
– applications: distributed systems, security, biology, quantum computing...
• Powerful tools
• Problem: Which tool to choose?
![Page 4: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/4.jpg)
ProbabilisticModel Checker
ProbabilisticModel Checkers
Probabilistic System Probabilistic Requirement
Probabilistic Model Probabilistic Formula
Yes No
Probability
Choices made
Four examples
Overall evaluation
![Page 5: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/5.jpg)
ToolsETMCC
numerical
CTMC Java
MRMC DTMC + CTMC C
PRISM hybrid
DTMC + CTMCMTBDD for transition matrix
C(++) and JavaPRISM
sparse
DTMC + CTMCsparse transition matrix
VESTAstatis
tCTMC, no S Java
YMER CTMC, only U ≤t C(++)
![Page 6: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/6.jpg)
Experiment Relevance
• Repeatable• Verifiable• Significant• Encapsulated
![Page 7: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/7.jpg)
Selected Benchmarks
Synchronous Leader Election
discrete time
Randomized Dining Philosophers
discrete time
Birth–Death Processdiscrete time
Tandem Queuing Networkcontinuous time
Cyclic Server Pollingcontinuous time
![Page 8: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/8.jpg)
SynchronousLeader Election
• nodes in a ring elect a leader– each node selects random number as id– passes it around the ring (synchronously)
– if unique id,node with maximum unique id is leader
• [Itai & Rodeh, 1990]
![Page 9: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/9.jpg)
SynchronousLeader Election
1
3
2
1
5
5
42
![Page 10: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/10.jpg)
5
4
1
2 2
3
5
14
1
2
2 3
5
1
51
2
2
3 5
1
5
4
SynchronousLeader Election
1
3
2
1
5
5
42
![Page 11: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/11.jpg)
RandomizedDining Philosophers
• Dining Philosophers– pick up chopsticks in random order
• Deadlocks resolved– if there is no second chopstick,give up eating
• [Pnueli & Zuck, 1986]
![Page 12: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/12.jpg)
Birth–Death Process
• Models a waiting queue• Standard modelin performance evaluation
• Limit queue size to get finite model
![Page 13: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/13.jpg)
Tandem Queueing Network
• Two queues after each other
• [Hermanns, Meyer-Kayser & Siegle, 1999]
checkincounter
two-phase
security check
exponential
![Page 14: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/14.jpg)
Cyclic Polling System• server cycles over n stationsand serves each one in turn– e.g. teacher walks through class,each pupil may ask a question
• [Ibe & Trivedi, 1990]
![Page 15: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/15.jpg)
Modelling
PRISMmodel
.tra formatmodel
PRISM
YMERmodel
ETMCC MRMC YMER VESTA
VESTAmodel
adaptsyntax
informaldescription
![Page 16: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/16.jpg)
Experiment 1Qualitative Properties• unbounded reachability with prob 1
• Cyclic Polling System:
busy1 P≥1(true U poll1)
If station 1 is busy,the server will poll it eventually
![Page 17: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/17.jpg)
1
10
100
1000
10000
100000
1000000
10000000
3 6 9 12 15 18number of stations n
log.scale VSZ memory (Kbytes)
mrmcprism hybridprism sparseetmccymervesta
0,00001
0,0001
0,001
0,01
0,1
1
10
100
1000
10000
3 6 9 12 15 18number of stations n
log.scale model check time (sec.)
![Page 18: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/18.jpg)
PRISM: MTBDD Size
• Multi-Terminal BDD =data structure for transition matrix
• size heavily depends on model• large MTBDD slow
Model#
states# MTBDD nodes
Synchronous leader election
500.0001.000.000
.
Cyclic polling
7.000.000
< 3.000 .
![Page 19: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/19.jpg)
CPS versus SLE runtime
0,0001
0,001
0,01
0,1
1
10
100
1000
10000
100000
4_4 4_8 4_12 4_16 8_2 8_4number of processes_random set size n_k
log.scale model check time (sec.)
0,00001
0,0001
0,001
0,01
0,1
1
10
100
1000
10000
3 6 9 12 15 18number of stations n
log.scale model check time (sec.)
mrmcprism hybridprism sparseetmccymervesta
7.077.888 states2.745 MTBDD nodes
458.847 states1.131.806 MTBDD nodes
![Page 20: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/20.jpg)
VESTA:simulation problem
• actual probability close to bound P≥p(...)
• estimate is almost always in [p–,p+]
• some irregularity stops the simulation
• 0.95 Prob(yes actual Prob≥p) Prob(actual Prob≥p yes)
![Page 21: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/21.jpg)
P≥1(... U ...)Timing Overview
1E-6
1E-3
1E+0
1E+3
1E+0 1E+3 1E+6 1E+9
number of states
time [sec]
ETMCCMRMCPRISM hybrid
PRISM sparseVESTA
![Page 22: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/22.jpg)
Analysis
ETMCC slow, out of memory
MRMC fastest tool for small models
PRISM only symbolic sparse=hybriddepends on MTBDD size
VESTAexcessive number of samples, slow
YMER not implemented
![Page 23: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/23.jpg)
depends heavilyon MTBDD sizedepends heavilyon MTBDD sizedepends heavilyon MTBDD size
Result Overview: Timing
ETMCC MRMC PRISM hybri
d
PRISM spars
e
VESTA YMER
unb’nded U – +
+/++
+/++
–/0 N/A
![Page 24: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/24.jpg)
Result Overview: Memory
ETMCC MRMC PRISM hybri
d
PRISM spars
e
VESTA YMER
unb’nded U – +
+/++
+/++
0/+ N/A
MTBDD sizevaries heavily
almost independentfrom model size
![Page 25: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/25.jpg)
Experiment 2Bounded Reachability
• Tandem Queueing Network:
P<0.01(true U ≤2 full )
Is the probabilitythat the system gets full in 2 time unitssmall?
![Page 26: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/26.jpg)
1
10
100
1000
10000
100000
1000000
10000000
10 50 100 255 511 1023queue size n
log.scale VSZ memory (Kbytes)
0,00001
0,0001
0,001
0,01
0,1
1
10
100
1000
10000
10 50 100 255 511 1023queue size n
log.scale model check time (sec.)
mrmcprism hybridprism sparseetmccymervesta
![Page 27: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/27.jpg)
Analysis
ETMCC slow, out of memory
PRISMsparse=faster, hybrid=smaller
MRMC fast for small models
VESTAokif you can afford statistical errors
YMERbest choiceif you can afford statistical errors
![Page 28: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/28.jpg)
Result Overview: Timing
ETMCC MRMC PRISM hybri
d
PRISM spars
e
VESTA YMER
unb’nded U – +
+/++
+/++
–/0 N/A
bounded U – + 0/++/++
+ ++
![Page 29: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/29.jpg)
Result Overview: Memory
ETMCC MRMC PRISM hybri
d
PRISM spars
e
VESTA YMER
unb’nded U – +
+/++
+/++
0/+ N/A
bounded U – ++/++
+ + ++
![Page 30: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/30.jpg)
Experiment 3Steady State Property
• Tandem Queuing Network
S>0.2( P>0.1(X 2nd queue full
) )
In equilibrium,the probability to satisfy
is > 0.2
P>0.1(X 2nd queue full )
P>0.1(X ... )
![Page 31: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/31.jpg)
1
10
100
1000
10000
100000
1000000
10000000
10 50 100 255 511 1023queue size n
log.scale VSZ memory (Kbytes)
0,00001
0,0001
0,001
0,01
0,1
1
10
100
1000
10000
10 50 100 255 511 1023queue size n
log.scale model check time (sec.)
mrmcprism hybridprism sparseetmccymervesta
![Page 32: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/32.jpg)
Simulating Steady State?
• simulation of bounded reachabilityhas clear stopping criterion
• simulation of unbounded reachability reachability with very large bound
• simulation of steady state? never stops
![Page 33: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/33.jpg)
Analysis
ETMCC slow, out of memory
PRISMsparse=fasterhybrid=slightly smaller
MRMC fastest
VESTA not implemented
YMER not implemented
![Page 34: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/34.jpg)
Result Overview: Timing
ETMCC MRMC PRISM hybri
d
PRISM spars
e
VESTA YMER
unb’nded U – +
+/++
+/++
–/0 N/A
bounded U – + 0/++/++
+ ++
steady state – ++ 0/+ + N/A N/A
![Page 35: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/35.jpg)
Result Overview: Memory
ETMCC MRMC PRISM hybri
d
PRISM spars
e
VESTA YMER
unb’nded U – +
+/++
+/++
0/+ N/A
bounded U – ++/++
+ + ++
steady state – +
+/++
+ N/A N/A
![Page 36: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/36.jpg)
Nested Formulas• Birth–Death Process:
P≥0.8(P≥0.9(true U ≤100 n70) U n50)
The probability to reach n50
(while the probability to reach n70
in 100 steps never drops <0.9)is ≥0.8
![Page 37: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/37.jpg)
0.00001
0.0001
0.001
0.01
0.1
1
10
100
1000
10000
100 1000 10000 100000maximum population size m
log.scale model check time (sec.)
1
10
100
1000
10000
100000
1000000
10000000
100 1000 10000 100000maximum population size m
log.scale VSZ memory (Kbytes)
mrmcprism hybridprism sparseetmccymervesta
![Page 38: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/38.jpg)
Result Overview: Timing
ETMCC MRMC PRISM hybri
d
PRISM spars
e
VESTA YMER
unb’nded U – +
+/++
+/++
–/0 N/A
bounded U – + 0/++/++
+ ++
steady state – ++ 0/+ + N/A N/A
nested – ++ 0/+ + – – N/Adid not terminate
![Page 39: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/39.jpg)
Result Overview: Memory
ETMCC MRMC PRISM hybri
d
PRISM spars
e
VESTA YMER
unb’nded U – +
+/++
+/++
0/+ N/A
bounded U – ++/++
+ + ++
steady state – +
+/++
+ N/A N/A
nested – ++/++
+ N/A N/A
![Page 40: How Fast and Fat Is Your Probabilistic Model Checker? an experimental performance comparison David N. Jansen 3,1, Joost-Pieter Katoen 1,2, Marcel Oldenkamp](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cfa5503460f949cbe9a/html5/thumbnails/40.jpg)
ConclusionsETMCC worst, only small models
MRMC fastest for small models
PRISM hybrid
fast if MTBDD is small
PRISM sparse
fast
VESTArather slow, no S , statistical errors
YMERslim & very fast, only U ≤t,few statistical errors