How Fast and Fat IsYour Probabilistic Model

Checker?an experimental performance

comparison

David N. Jansen3,1, Joost-Pieter Katoen1,2, Marcel Oldenkamp2,

Mariëlle Stoelinga2, Ivan Zapreev1,2

1 MOVES Group, RWTH Aachen University2 FMT Group, University of Twente, Enschede3 ICIS, Radboud University, Nijmegen

ProbabilisticModel Checker

ProbabilisticModel Checking

Probabilistic System Probabilistic Requirement

≤Probabilistic Model Probabilistic Formula

Yes No

Probability

ETMCC

MRMCPRISM(sparse)

PRISM(hybrid)

YMER

VESTA

Why Are Probabilities Useful?

• system performance• uncertainty in the environment

• randomized (networking) algorithms

• abstract from large populations

ProbabilisticModel Checking...

• What is inside?– temporal logics + model checking– numerical and optimisation techniquesfrom performance and operations research

• Where is it used?– powerful tools– applications: distributed systems, security, biology, quantum computing...

• Problem: Which tool to choose?

ProbabilisticModel Checker

Probabilistic Models

Probabilistic System Probabilistic Requirement

Probabilistic Model Probabilistic Formula

Yes No

Probability

Discrete timeMarkov chains

Continuous timeMarkov chains

automatatransitions are probabilistic

timing for CTMC:Prob(wait time ≤ t) = 1 – e–t

SynchronousLeader Election

• nodes in a ring elect a leader– each node selects random number as id– passes it around the ring (synchronously)

– if unique id,node with maximum unique id is leader

• [Itai & Rodeh 1990]

SynchronousLeader Election

1

3

2

1

5

5

42

5

4

1

2 2

3

5

14

1

2

2 3

5

1

51

2

2

3 5

1

5

4

SynchronousLeader Election

1

3

2

1

5

5

42

ProbabilisticModel Checker

Probabilistic Formulas

Probabilistic System Probabilistic Requirement

Probabilistic Model Probabilistic Formula

Yes No

Probability

Reachability

Bounded Reachability

Steady-State Property

extensions of CTL

ProbabilisticModel Checker

ProbabilisticModel Checkers

Probabilistic System Probabilistic Requirement

Probabilistic Model Probabilistic Formula

Yes No

Probability

Choices made

Three examples

Overall evaluation

ToolsETMCC

numerical

CTMC Java

MRMC DTMC + CTMC C

PRISM hybrid

DTMC + CTMCMTBDD for transition matrix

C(++) and JavaPRISM

sparse

DTMC + CTMCsparse transition matrix

VESTAstatis

tCTMC, reachability Java

YMER CTMC, bounded reach C(++)

Modelling

PRISMmodel

.tra formatmodel

PRISM

YMERmodel

ETMCC MRMC YMER VESTA

VESTAmodel

adaptsyntax

informaldescription

Selected Benchmarks

Synchronous Leader Election

discrete time

Randomized Dining Philosophers

discrete time

Birth–Death Processdiscrete time

Tandem Queuing Networkcontinuous time

Cyclic Server Pollingcontinuous time

Experiment Relevance

• Repeatable• Verifiable• Significant• Encapsulated

Experiment 1Reachability

• Cyclic Polling Server:server cycles over n stationsand serves each one in turn– e.g. teacher walks through class,each pupil may ask a question

• busy1 P≥1(true U poll1)If station 1 is busy,the server will poll it eventually

1

10

100

1000

10000

100000

1000000

10000000

3 6 9 12 15 18number of stations n

log.scale VSZ memory (Kbytes)

mrmcprism hybridprism sparseetmccymervesta

0,00001

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

3 6 9 12 15 18number of stations n

log.scale model check time (sec.)

Analysis

ETMCC slow, out of memory

PRISMonly symbolic sparse=hybrid

MRMCfastest tool for small models

VESTAexcessive number of samples, slow

YMER not implemented

PRISM: MTBDD Size

• Multi-Terminal BDD =data structure for transition matrix

• size heavily depends on model• large MTBDD slow

Model#

states# MTBDD nodes

Synchronous leader election

500.0001.000.000

.

Cyclic polling

7.000.000

< 3.000 .

CPS versus SLE runtime

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

100000

4_4 4_8 4_12 4_16 8_2 8_4number of processes_random set size n_k

log.scale model check time (sec.)

0,00001

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

3 6 9 12 15 18number of stations n

log.scale model check time (sec.)

mrmcprism hybridprism sparseetmccymervesta

7.077.888 states2.745 MTBDD nodes

458.847 states1.131.806 MTBDD nodes

VESTA:simulation problem

• actual probability close to bound P≥p(...)

• estimate is almost always in [p–,p+]

• some irregularity stops the simulation

• 0.95 Prob(yes actual Prob≥p) Prob(actual Prob≥p yes)

depends heavilyon MTBDD sizedepends heavilyon MTBDD sizedepends heavilyon MTBDD size

Result Overview: Timing

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

–/0 N/A

Result Overview: Memory

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

0/+ N/A

MTBDD sizevaries heavily

almost independentfrom model size

Experiment 2Bounded Reachability

• Tandem Queueing Network– two queues after each other

• P<0.01(true U ≤2 full )Is the probabilitythat the system gets full in 2 time unitssmall?

checkincounter

security check

1

10

100

1000

10000

100000

1000000

10000000

10 50 100 255 511 1023queue size n

log.scale VSZ memory (Kbytes)

0,00001

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

10 50 100 255 511 1023queue size n

log.scale model check time (sec.)

mrmcprism hybridprism sparseetmccymervesta

Analysis

ETMCC slow, out of memory

PRISMsparse=faster, hybrid=smaller

MRMC fast for small models

VESTAokif you can afford statistical errors

YMERbest choiceif you can afford statistical errors

Result Overview: Timing

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

–/0 N/A

bounded U – + 0/++/++

+ ++

Result Overview: Memory

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

0/+ N/A

bounded U – ++/++

+ + ++

Experiment 3Steady State Property

• Tandem Queuing Network

• S>0.2( P>0.1(X 2nd queue full

) )In equilibrium,the probability to satisfy

is > 0.2

P>0.1(X 2nd queue full )

P>0.1(X ... )

1

10

100

1000

10000

100000

1000000

10000000

10 50 100 255 511 1023queue size n

log.scale VSZ memory (Kbytes)

0,00001

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

10 50 100 255 511 1023queue size n

log.scale model check time (sec.)

mrmcprism hybridprism sparseetmccymervesta

Analysis

ETMCC slow, out of memory

PRISMsparse=fasterhybrid=slightly smaller

MRMC fastest

VESTA not implemented

YMER not implemented

Simulating Steady State?

• simulation of bounded reachabilityhas clear stopping criterion

• simulation of unbounded reachability reachability with very large bound

• simulation of steady state? never stops

Result Overview: Timing

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

–/0 N/A

bounded U – + 0/++/++

+ ++

steady state – ++ 0/+ + N/A N/A

Result Overview: Memory

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

0/+ N/A

bounded U – ++/++

+ + ++

steady state – +

+/++

+ N/A N/A

Nested Formulas

• we also checked nested properties

P≥0.8(P≥0.9(true U ≤100 n70) U n50)

• not detailed here

Result Overview: Timing

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

–/0 N/A

bounded U – + 0/++/++

+ ++

steady state – ++ 0/+ + N/A N/A

nested – ++ 0/+ + – – N/A

based on a singleproperty only:

did not terminate

Result Overview: Memory

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

0/+ N/A

bounded U – ++/++

+ + ++

steady state – +

+/++

+ N/A N/A

nested – ++/++

+ N/A N/A

ConclusionsETMCC worst, only small models

MRMC fastest for small models

PRISM hybrid

fast if MTBDD is small

PRISM sparse

fast

VESTArather slow, statistical errors

YMERslim & very fast, only bounded reach,few statistical errors