technische universiteit eindhoven progress 11 december 2002btheelen1 performance modelling of...
TRANSCRIPT
11 December 2002 www.ics.ele.tue.nl/~btheelen 1
technische universiteit eindhovenPROGRPROGREESSSS
Performance Modelling of Complex Hardware/Software Systems
B.D. Theelen
11 December 2002 www.ics.ele.tue.nl/~btheelen 2
technische universiteit eindhovenPROGRPROGREESSSS
Overview• Introduction• System-Level Design• Software/Hardware Engineering• Case Studies• Modelling and Validation• Performance Evaluation• Accuracy Analysis• Summary
11 December 2002 www.ics.ele.tue.nl/~btheelen 3
technische universiteit eindhovenPROGRPROGREESSSS
Introduction• Designing a complex system within limited time involves taking
important decisions in an early phase of the design process, which may have a deep impact on the performance of the system
• Early assessment of the impact of design decisions involves using system-level design methods and tools
Design
Tim
e
Concepts Requirements
Implementation
System-Level Design
Component-Level Design
(Automatic) System Implementation
11 December 2002 www.ics.ele.tue.nl/~btheelen 4
technische universiteit eindhovenPROGRPROGREESSSS
Concepts & Requirements
ConceptModel
Questions
ExecutableModel
FormalProperties
ConceptsSatisfy Requirements
Formulation
Evaluation
FomalisationValidation Fomalisation
Component-Level Design
Milestone A
Milestone C
Milestone B
Sys
tem
-Le
vel
Des
ign
Fo
rmu
lati
on
Fo
rma
lis
ati
on
Eva
lua
tio
n
Yes
No
Three Phases• FormulationFormulation
– Informal identification and specification of system concepts and design issues
• FormalisationFormalisation– Formal specification of
behaviour and architecture– Validation of adequacy of
formal system specification– Formal specification of the
properties to evaluate• EvaluationEvaluation
– Property analysis and design decisions
11 December 2002 www.ics.ele.tue.nl/~btheelen 5
technische universiteit eindhovenPROGRPROGREESSSS
Software/Hardware Engineering (SHE)• Formulation using (stereotyped) UMLUML diagrams
– Performed case studies: documents with plain texts and various diagrams• Formalisation based on system-level modelling language POOSLPOOSL
– Intended for complex real-time distributed hardware/software systems– Small set of very expressive language primitives– Formal semantics
• Defines unambiguously how to handle concurrencyconcurrency, timetime and probabilismprobabilism
• Evaluation– Performance evaluation founded on Markov chain analysisMarkov chain analysis
• Analytic computation or estimation by simulation– Formal verification of correctness properties based on model checkingmodel checking
• Tools– SHESim: Interactive creationcreation and validationvalidation of POOSL models– Rotalumis: Fast execution of POOSL models for analysisanalysis
11 December 2002 www.ics.ele.tue.nl/~btheelen 6
technische universiteit eindhovenPROGRPROGREESSSS
Case Study: Network Processor
• (Deep) packet processing in telecommunication systems
• Utilisation of shared busses• Identify possible bottlenecks• Comparison POOSL - SystemC
11 December 2002 www.ics.ele.tue.nl/~btheelen 7
technische universiteit eindhovenPROGRPROGREESSSS
Case Study: Dataflow System• Part of Alcatel’s Internet Router• Prepares/processes packets for correct routing• Arbitration of memory accesses from 3 components• Evaluate alternative arbitration mechanisms
11 December 2002 www.ics.ele.tue.nl/~btheelen 8
technische universiteit eindhovenPROGRPROGREESSSS
Case Study: Internet Router
Is the specified flow-control mechanismsuitable for various product variants?
Inputs Outputs
SourcesInput Buffers Output Buffers
SinksSwitch Core
1 1
N N
1
N
N
1
N
1
1
N
11 December 2002 www.ics.ele.tue.nl/~btheelen 9
technische universiteit eindhovenPROGRPROGREESSSS
Executable Model
• Modelling guidelinesModelling guidelines (validation)– Readability (aggregate data objects), postpone optimising for execution speed– Log considerations on the adequacy of abstractions
• Modelling patternsModelling patterns– Point abstractions, abstraction from scheduling mechanisms, stream-based
modelling, scalability of model, model refinement
11 December 2002 www.ics.ele.tue.nl/~btheelen 10
technische universiteit eindhovenPROGRPROGREESSSS
Validation
• Validation guidelinesValidation guidelines– Perform step-by-step inspections– Analyse easy-to-check properties– Investigate impact of rare events– Consider symmetries in system– Recall considerations on adequacy of abstractions
11 December 2002 www.ics.ele.tue.nl/~btheelen 11
technische universiteit eindhovenPROGRPROGREESSSS
Property Formalisation• Two options for expressing performance metrics
– Performance analysis based on external observersexternal observers: temporal rewards– ReflexiveReflexive performance analysis: POOSL
• Extend model with additional behaviour to evaluate performance metrics
• Guidelines for reflexive performance analysis– Define extensions in subclasses– Use method overriding capabilities of POOSL
Formal SemanticsFormal Semantics
Understandable POOSL Model
Markov Decision Process
+ Extensions for Performance Evaluation
+ Reward Structure
External SchedulerExternal Scheduler
Discrete-Time Markov Chain + Reward Structure
11 December 2002 www.ics.ele.tue.nl/~btheelen 12
technische universiteit eindhovenPROGRPROGREESSSS
Performance Evaluation• Analytical computationcomputation: Ergodic Theorem• Estimation by simulationsimulation: Central Limit Theorem
– Example: packet-loss probability
– Enables accuracy analysis with confidence intervalsconfidence intervals– Enables automatic terminationautomatic termination of simulation when results are accurate
N(0,1) σ
μ)(1
μ then )(1
if d.1a.s.
1
n
iin
ii
XPacketLostn
nXPacketLostn
11 December 2002 www.ics.ele.tue.nl/~btheelen 13
technische universiteit eindhovenPROGRPROGREESSSS
Accuracy Analysis• For Internet Router example: a 99% confidence interval
[0.001046, 0.001052] is obtained for the average latency– Estimation of average latency is 0.001049 seconds– We are 99% sure that the real average latency lies in [0.001046, 0.001052]– An upper bound for the relative error is 1%– Takes 0.0072 seconds of simulated real-time– Takes 11.9 hours of simulation time
• Central Limit Theorem only applicable for simple averages– Latency, packet loss probability
• Complex averages and variances: algebra of confidence intervalsalgebra of confidence intervals– Defines set operations on confidence intervals– Enables accuracy analysis for range of different averages and variances– Jitter, buffer occupancy, processor utilisation, throughput, burstiness
• POOSL library classes for analysing different long-run averages
11 December 2002 www.ics.ele.tue.nl/~btheelen 14
technische universiteit eindhovenPROGRPROGREESSSS
Summary• Three large case studies in corporation with industrial partners• SHE method distinguishes three phases
– Formulation based on informal UML diagrams– Formalisation with formal modelling language POOSL
• Guidelines for modelling and validation• Modelling patterns• Guidelines for reflexive performance analysis
– Performance evaluation based on Markov chain analysis• POOSL model implicitly defines Markov chain• Extensions for performance analysis define reward structure• Analytical computation or estimation by simulation
• Techniques for performance analysis– Confidence intervals allow automatic termination of simulation– Algebra of confidence intervals for accuracy analysis of complex metrics
• Library classes for accuracy analysis