sdrl & rtg university of pennsylvania 5/24/01 1 run-time monitoring and checking based on formal...
Post on 22-Dec-2015
214 views
TRANSCRIPT
5/24/01
1
SDRL & RTGUniversity of Pennsylvania
Run-time Monitoring and Checking Based on Formal Specifications
Insup LeeDepartment of Computer and Information Science
University of Pennsylvania
24 May 2001
Joint work with M. Kim, M. Viswanathan, S. Kannan, and O. Sokolsky
5/24/01
2
SDRL & RTGUniversity of Pennsylvania
Objectives• Specification and verification
– complete analysis, all behaviors are correct– gap between abstract model and
implementation• Testing
– tested behaviors are correct– not complete
• Run-time behavior checking– consistency between abstract model and
implementation• To provide a framework for automatic generation
of monitors and checkers
5/24/01
3
SDRL & RTGUniversity of Pennsylvania
Fundamental Issues
• How does a monitor gather information from a running system?
• How does the monitor relate to requirements?
• How do we integrate dynamic monitoring with static analysis?
• Can monitor be used to steer a system?• What mathematical guarantees do monitors
provide?
5/24/01
4
SDRL & RTGUniversity of Pennsylvania
Monitorable Properties• Run-time monitoring and checking checks whether
properties are violated or not by observing only finite traces execution at run-time
• A class of monitorable properties is a Turing computable subset of safety properties – Safety property: finite trace reveals violations.– Monitorable property: violations Turing
computable.• Equivalent to class 1 in the Arithmetic Hierarchy
Liveness
Safety
MEDL = Monitorable property
Properties on Traces
Safety closure of the halting problem
5/24/01
5
SDRL & RTGUniversity of Pennsylvania
System SpecSystem
Spec
RequirementSpec
RequirementSpec
Formal verification
Design
SystemImplementation
SystemImplementation
MonitoringScript
MonitoringScript
Implementation
Checker/CorrectorChecker/Corrector
SystemSystem FilterCommunication
Run-time Check
MaCS Framework
EventHandler
EventHandler
CorrectorCorrector
CheckerChecker
5/24/01
6
SDRL & RTGUniversity of Pennsylvania
Design Issues
• Filter– passive versus active– when to take snapshot
• Event Handler– mapping between concrete state and
abstract event• Checker
– inclusion based on trace, ready semantics, bisimulation
• Corrector– how to provide feedback
5/24/01
7
SDRL & RTGUniversity of Pennsylvania
Overview of MaCS framework• Based events and conditions• Two types of abstraction
– time abstraction by instrument filter (IF)– data abstraction by event recognizer (ER)
• Scripting languages– MEDL (Meta Event Definition Language):
property specification language– PEDL (Primitive EDF): Implementation-language
dependent• Goals: Automatic generation of IF, ER, Checker,
Steerer
5/24/01
8
SDRL & RTGUniversity of Pennsylvania
Events and Conditions
• Must be able to reason about both time instants and durations in a program execution.– Events and conditions are a natural division,
which is also found in other formalisms like SCR.
• Need temporal operators combining events and conditions in order to reason about traces.
Crecy Poitiers Agincourtstart(war) end(war)
1341 145714121344 1348
5/24/01
9
SDRL & RTGUniversity of Pennsylvania
Two-Sorted Logic
• Conditions interpreted over 3 values: true, false and undefined.
• [E,E) pairs a couple of events to define an interval.• start and end define the events corresponding to
the instant when conditions change their value.• Primitive events and conditions
212121 | | | ),[ | )(defined :: CCCCCEECC
CE
EEEECCeE
when
| | | )(end | )(start | :: 2121
5/24/01
10
SDRL & RTGUniversity of Pennsylvania
An example
• Example: correct = oldDateUsed => (f().num-retries = 1)
– If old data is used then number of retries must be 1
– f().num-retries is a local variable of function f()
invoke f() f().num-retries = 1 oldDataUsed return oldDataNotUsed
• correct is true, undefined, and true again.
5/24/01
11
SDRL & RTGUniversity of Pennsylvania
MaCS prototype architecture
Program(Java byte code)
Monitoring Script(PEDL)
Requirements(MEDL)
PEDLPEDLCompilerCompiler
MEDLMEDLCompilerCompiler
Instrumented Code
Filter Generator(JTREK)
InstrumentationInformation
Compiled PEDL
Compiled MEDL
Event Recognizer Checker
Steering Script(SADL)
SADLSADLCompilerCompiler
InstrumentationInformation
Injector class(Java byte code)
5/24/01
12
SDRL & RTGUniversity of Pennsylvania
MEDL
• Meta Event Definition Language (MEDL)• Expresses requirements using the events and conditions,
sent by event recognizer.
• Describes the safety requirements of the system, in terms of conditions that must always be true, and alarms (events) that must never be raised.– property periodic = (period == 1000)
– alarm staleData = oldDataUsed when connected
• Auxiliary variables may be used to store history.– request_info -> { num_hits’ = num_hits+1; hit_ratio’ = num_hits/num_fails; }
• Used to automatically create checker
5/24/01
13
SDRL & RTGUniversity of Pennsylvania
Structure of MEDL scriptReqSpec HexPattern
// imported events, conditions, and actionsimport event MAValert, initimport action collapse, restore// variable declarationsvar int numMAValert, prevCount1, prevCount2var float prevAverage, currAverage// definitions of eventsevent endPeriod = start(time(MAValert)-periodStart > 3000)alarm NoPattern = start(currAverage > prevAverage*1.1+10) // guarded variable updates and action invocationsupdateAvg -> {
currAverage’= (prevCount2+prevCount1+numMAValert)/3;}NoPattern -> { invoke collapse; }
End
5/24/01
14
SDRL & RTGUniversity of Pennsylvania
PEDL
• Primitive Event Definition Language (PEDL)• PEDL describes a thin interface between low-level
implementation and high-level requirement– The language maps the low-level state
information of the system to high-level events used in describing the requirements.
– Fast event recognition is key point• Provides primitives to refer to values of variables
and to certain points in the execution of the program• Depends on system implementation language• Used to create filter and event recognizer
5/24/01
15
SDRL & RTGUniversity of Pennsylvania
Primitive Event Definition Language (PEDL)
• Information about the system comes in two different forms:– Conditions, which are true or false for a finite duration of time
(e.g., is variable x >5?), and– Events, which are either present or absent at some instant of
time (e.g., is the control right now at the end of method f?)• Provides primitives to refer to values of variables and to
certain points in the execution of the program.– condition IC = (50<train_position) && (train_position<100);– Event endGD = startM(Gate.gu());
• Provides primitive “time” to refer to time when events happen – condition slowTrain = (time(endIC)-time(startIC)) > 3000;
5/24/01
16
SDRL & RTGUniversity of Pennsylvania
Structure of PEDL ScriptMonScr MAVpattern
// exported events and conditionsexport event MAValert, init;
// Overhead Reduction Options[timestamp;][valueabstract;][deltaabstract;][multithread;]
// monitored object declarationsmonmeth void Console.createMAVs(int);monobj double MAV.run().distance;
// definitions of events and conditionsevent init = startM( Console.createMAVs(int) );event MAValert = start(MAV.run().distance > 40 &&
MAV.run().distance < 120) ;End
5/24/01
17
SDRL & RTGUniversity of Pennsylvania
The Current MaCS Prototype System
• MaCS instruments Java bytecode, not a source code.
• Filter resides in the host of target program as a separate thread.
• The filter sends updated value and time stamp to the event recognizer.
• The event-recognizer (ER) evaluates condition and event description
• ER sends evaluation result to the run-time checker.• MaCS works on multi-threaded applications
5/24/01
18
SDRL & RTGUniversity of Pennsylvania
MaCSware Version 0.99
• Components– Static components
• PEDL/MEDL/SADL compiler• Bytecode Instrumentor
– Dynamic components• Filter/Injector, event recognizer and run-time
checker• GUI
• http://www.cis.upenn.edu/~rtg/macs
5/24/01
19
SDRL & RTGUniversity of Pennsylvania
Related Work
• Monitoring in process-level by input/output/message interface instrumentation – Supervisor [T.Savor98], JEM[G.Liu99] ,MOTEL[Log00]
• Monitoring in statement-level by target system instrumentation– ANNA [S.Sankar93]
• checks assertion for annotated Ada program
– Java Runtime Timing Constraint Monitor [A.Mok97]• monitors Java program using specification language RTL
– ALAMO [C.Jeffery98]• instruments C source code and monitors it
• Monitoring in instruction-level by environment instrumentation– Dynascope [R. Sosic95], JPDA [JVM99]
5/24/01
20
SDRL & RTGUniversity of Pennsylvania
Current and future work• Extend MaCS framework
– Monitoring of hybrid systems– Distributed monitoring
• Extend MaC prototype implementation– Monitoring and checking frequency optimization– Add program checking capability– Add steering capability
• Develop MaC formal foundations • Integrate with other tools
– Network simulator – Test generator
– Charon• http://www.cis.upenn.edu/~rtg/macs
5/24/01
21
SDRL & RTGUniversity of Pennsylvania
Automatic Test Generation from Formal Specifications
Insup LeeDepartment of Computer and Information Science
University of Pennsylvania
24 May 2001
Joint work with H.S. Hong and O. Sokolsky
5/24/01
22
SDRL & RTGUniversity of Pennsylvania
Specification-based Testing
Specification Spec(I)
• Determines whether an implementation conforms to its specification – Hardware and protocol conformance testing– FSM and LTS
• For each test case (I, Spec(I)),– Apply I to the implementation– Observe the set of output sequences Imp(I)– Check Imp(I) Spec(I)
Implementation
I
Imp(I)
5/24/01
23
SDRL & RTGUniversity of Pennsylvania
Specifications
• Specification-based test generation for reactive, real-time, and hybrid systems– EFSM
• FSM + data variables– Statecharts, hierarchical reactive modules
• EFSM + hierarchy + concurrency + communication – CHARON, hybrid systems
• EFSM + hierarchy + concurrency + communication + differential equations
5/24/01
24
SDRL & RTGUniversity of Pennsylvania
Test Coverage Criteria for EFSMs• Structural information in EFSMs (application-independent)
– Control flow: state or transition coverage– Data flow: all-definition, all-use, or all-def-use-path
• Properties to be validated (application-dependent)– MEDL scripts, Scenarios, MSCs, temporal logics
empty nonempty
t1: increment/money:=1
t4: decrement[money=1]/money:=0
t2: increment[money<MAX]/money:=money+1
t3: decrement[money>1]/money:=money-1
5/24/01
25
SDRL & RTGUniversity of Pennsylvania
Test Generation from EFSM
• Model checking based approach to generate tests– Finding counterexamples during the model checking of
EFSMs– State coverage: !EF nonempty– Transition coverage: !EF t4– All-definition coverage: !EF (t1 & EF (!(t1|t2|t3|t4) U t4))
• Explore other approaches
empty nonempty
t1: increment/money:=1
t4: decrement[money=1]/money:=0
t2: increment[money<MAX]/money:=money+1
t3: decrement[money>1]/money:=money-1
5/24/01
26
SDRL & RTGUniversity of Pennsylvania
Test Generation for CHARON
• Adapt EFSM-based test generation to hybrid systems
• Transforming differential equations into EFSMs– Qualitative reasoning and predicate abstraction
techniques for CHARON• Determining the executability of test cases
– Symbolic execution techniques for CHARON
5/24/01
27
SDRL & RTGUniversity of Pennsylvania
Other issues
• Test generation– How to meet coverage criteria– Test suite optimization
• Test execution– Integration of testing with MaCS
• Effectiveness of coverage criteria in practice• Apply to real examples
5/24/01
28
SDRL & RTGUniversity of Pennsylvania
Q&A