tmpa-2017: the quest for average response time
TRANSCRIPT
The Quest for Average Response Time
Tom Henzinger
IST Austria
Joint work with Krishnendu Chatterjee and Jan Otop
Yes/No
Model Checker
Markov
Process
Quantitative
PropertyProgram 8 (r ) Pr(}g) ¸ 0.5)
Formal Verification
Model Checker
Timed
Automaton Program (r ) } g)Property
Quantitative Answer R
Worst or average response time
Quantitative Analysis
Quantitative Analysis
From checking correctness
to measuring performance and robustness of software systems:
Quantitative temporal logics
Quantitative automata
Quantitative abstractions
Quantitative synthesis
etc.
Quantitative Analysis
From checking correctness
to measuring performance and robustness of software systems:
Quantitative temporal logics
Quantitative automata
Quantitative abstractions
Quantitative synthesis
etc.
None of this has captured average response time.
Program Behavior = Observation Sequence
x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x …
Response Times
x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x … 4,3,2
r t t t t t t t … 1
Response Monitor
rr,t,xg,t,x
S
S
g
Decomposing model checking [Pnueli et al.]
Alternating automata
Run-time verification
Bounded Response
r
gr,x
g,t,x
C := 0
C · 3
tC := C+1
gC > 3
(Discrete) clocks exponentially succinct,
but not more expressive than finite state.
Maximal Response
r
gr,x
g,t,x
C := 0
V := max(V,C)
tC := C+1
V := 0
Value of an infinite run is liminf of V.
Maximal Response Monitor
r r,xg,t,x
S
S
g
tV := V+1
V := 0V := 0
V := max(V, )
is final value of V.S
Average Response
r
gr,x
g,t,x
C := 0
N := N+1
V := avg(V,C,N)
tC := C+1
V := 0
N := 0
avg(V,C,N) = (V¢(N-1)+C) / N
(max,inc) automata:
Master automaton maintains the max of values
returned by slaves (1 max register).
Each slave automaton counts occurrences of t
(1 inc register).
(avg,inc) automata:
Master automaton maintains the avg of values
returned by slaves (1 avg register).
Slaves as above.
Nested Weighted Automata
r r,x
0g,t,x
0S
g
0
t1
S
limavg of weights
sum of weights
Function on weights instead of registers.
Unlike in the qualitative case,
nested weighted automata (“quantitative monitors”) are
more expressive than flat weighted automata:
1. value of flat limavg automaton bounded by largest weight
(cannot specify average response time)
2. flat automata have constant “width” (number of registers)
Deterministic qualitative automaton A: §! ! B
Deterministic quantitative automaton A: §! ! R
! = x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x …
Response(!) = 1
BoundedResponse(!) = 0
MaximalResponse(!) = 4
AverageResponse(!) = 3
43 2
Nondeterministic qualitative automaton A: §! ! B
A(!) = max{ value(½) | ½ run of A and obs(½) = ! }
Nondeterministic quantitative automaton A: §! ! R
A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
Nondeterministic qualitative automaton A: §! ! B
A(!) = max{ value(½) | ½ run of A and obs(½) = ! }
Emptiness: 9!. A(!) = 1Universality: 8!. A(!) = 1
Nondeterministic quantitative automaton A: §! ! R
A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
Nondeterministic qualitative automaton A: §! ! B
A(!) = max{ value(½) | ½ run of A and obs(½) = ! }
Emptiness: 9!. A(!) = 1Universality: 8!. A(!) = 1
Nondeterministic quantitative automaton A: §! ! R
A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
Emptiness: 9!. A(!) ¸ ¸
Universality: 8!. A(!) ¸ ¸
Qualitative Analysis
Given a transition system A and a qualitative property B,
Q1. does some run of A correspond to a run of B ?[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B ?
[as hard as universality of B ]
Quantitative Analysis
Given a transition system A and a quantitative property B,
Q1. does some run of A correspond to a run of B with value V ¸ ¸ ?
[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B with V ¸ ¸ ?
[as hard as universality of B ]
Qualitative Analysis
Given a transition system A and a qualitative property B,
Q1. does some run of A correspond to a run of B ?[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B ?Equivalently: does some run of A correspond to a run of :B ?
[emptiness of A £ :B ]
Qualitative Analysis
Given a transition system A and a qualitative property B,
Q1. does some run of A correspond to a run of B ?[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B ?Equivalently: does some run of A correspond to a run of :B ?
[emptiness of A £ :B ]
For deterministic B, the complement :B is easy to compute.
Nondeterministic quantitative automaton A: §! ! R
A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
Monitor: obs(½1) = obs(½2) ) value(½1) = value(½2)
Deterministic automata are monitors.
Quantitative Analysis
Given a transition system A and a quantitative property B,
Q1. does some run of A correspond to a run of B with value V ¸ ¸ ?
[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B with V ¸ ¸ ?
For monitor B, equivalently: does some run of A correspond to a run of B with V < ¸ ?
[emptiness of A £ -B ]
Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Best maximal response time: 2
Worst maximal response time: 3
Emptiness of (max,inc) automata
Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Best maximal response time: 2
Worst maximal response time: 3
Emptiness of (max,inc) automata
Best average response time: 1.5
Worst average response time: 3
Emptiness of (avg,inc) automata
Results on (max,inc) Automata
Nondet Monitor
(max,inc) (max,inc)
Emptiness PSPACE PSPACE
Universality · EXPSPACE PSPACE
¸ PSPACE
Qualitative nondeterministic universality PSPACE,
emptiness and deterministic universality PTIME.
Results on (avg,inc) Automata
Nondet Monitor
(avg,inc) (avg,inc)
Emptiness · EXPSPACE · EXPSPACE
¸ PSPACE ¸ PSPACE
Universality undecidable · EXPSPACE
¸ PSPACE
Results on (avg,inc) Automata
Nondet Monitor Monitor Monitor
(avg,inc) (avg,inc) bounded width constant width
(avg,inc) (avg,inc)
Emptiness · EXPSPACE · EXPSPACE PSPACE PTIME
¸ PSPACE ¸ PSPACE
Universality undecidable · EXPSPACE PSPACE PTIME
¸ PSPACE
How many overlapping requests?
Probabilistic System = Markov Chain
r
r
t
x
x
x
t
g
g
g
t
t
t
t
x
x
t
0.5 0.3
0.2
0.5
0.5
Defines probability for every
finite observation sequence,
and prob density function on
infinite observation sequences.
0.9
0.1
Probabilistic System = Markov Chain
r
r
t
x
x
x
t
g
g
g
t
t
t
t
x
x
t
0.5 0.3
0.2
0.5
0.5
Defines probability for every
finite observation sequence,
and prob density function on
infinite observation sequences.
0.9
0.1
Given prob density function on §!,
monitor specifies random variable V.
Probabilistic Analysis
Given a probabilistic system A and a functional quantitative property B,
Q1. compute the expected value of V on the runs of A £ B
[moment analysis]
Q2. compute the probability of V ¸ ¸ on the runs of A £ B
[distribution analysis]
Probabilistic Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Expected maximal response time: 2.5
Prob of maximal response time at most 2: 0.5
Probabilistic analysis of (max,inc) automata
0.5 0.5
Probabilistic Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Expected maximal response time: 2.5
Prob of maximal response time at most 2: 0.5
Probabilistic analysis of (max,inc) automata
Expected average response time: 2.25
Prob of average response time at most 2: 0.5
Probabilistic analysis of (avg,inc) automata
0.5 0.5
Results on (max,inc) Automata
Nondet Monitor
(max,inc) (max,inc)
Emptiness PSPACE PSPACE
Universality · EXPSPACE PSPACE
¸ PSPACE
Expectation · EXPSPACE
¸ PSPACE
Probability · EXPSPACE
¸ PSPACE
Results on (avg,inc) Automata
Nondet Monitor
(avg,inc) (avg,inc)
Emptiness · EXPSPACE · EXPSPACE
¸ PSPACE ¸ PSPACE
Universality undecidable · EXPSPACE
¸ PSPACE
Expectation PTIME
Probability PTIME
Markov Decision Process
r
r
t
x
x
x
t
g
g
g
t
t
t
t
x
x
t
0.5
0.5
0.9
0.1
Given a policy p:§!!{ $,$,$ },
monitor specifies random
variable V.
Many Open Questions …
E.g., given an MDP A and a monitor B,
compute the policy that maximizes the expected value of B on A
(generalization of mean-payoff game).
Counter Machine
r
x
g,t,xS
S
g
C = 0
tV := V+1
V := 0
C := 0V := 0
V := max(V, )
rC := C+1
g
C > 0C := C-1
Emptiness for two counters is in general undecidable.
Counter Monitor
t
x
r,g,xS
S
t
V := 0V := 0
V := max(V, )
rV := V+1
gV := V-1
No test on counter values.
Counter Monitor
t
x
r,g,xS
S
t
V := 0V := 0
V := max(V, )
rV := V+1
gV := V-1
No test on counter values. width = 1
Register Automaton
x
V := 0
C := 0
rC := C+1
gC := C-1
Emptiness of (max,inc+dec) decidable [flat, constant width: Alur et al.].
tV := max(V,C)
C := 0
Results on (max,inc+dec) Automata
Nondet Monitor
(max,inc+dec) (max,inc+dec)
Emptiness PSPACE PSPACE
Universality undecidable undecidable
Expectation undecidable
Probability undecidable
Results on (avg,inc+dec) Automata
Nondet Monitor
(avg,inc+dec) (avg,inc+dec)
Emptiness open open
Universality undecidable open
Expectation PTIME
Probability PTIME
Quantitative Monitors = Nested Weighted Automata
Unbounded width allows for natural decomposition of specifications
(incl. average response time).
More expressive and more succinct than flat weighted automata.
Emptiness decidable and sufficient for
monitor verification, model measuring, and model repair
(universality can be undecidable, even for constant width).
Probabilistic analysis polynomial for (avg,inc+dec) monitors.
Model Measuring:How much can system A be perturbed without violating qualitative property B ?
Model Repair:How much must system A be changed to satisfy qualitative property B ?
Model Measuring:How much can system A be perturbed without violating qualitative property B ?
For an observation sequence ! we can define
distance d(A,!) (e.g. edit distance) by constructing from A
monitor FA such that FA(!) = d(A,!).
Robustness of A with respect to B:r(A,B) = sup{ e | 8!. d(A,!) · e ) B(!) = 1 }.
Model Repair:How much must system A be changed to satisfy qualitative property B ?
References
Nested Weighted Automata: LICS 2015
Quantitative Automata under Probabilistic Semantics: LICS 2016
Nested Weighted Automata of Bounded Width: MFCS 2016