Download - Model-based Testing and and Validation - AAU
Info
rmat
ionst
eknolo
gi
CISS
AgendaWho are we??Why T&V? Testing versus VerificationModel-Driven DevelopmentTesting & MBTCourse Plan
Info
rmat
ionst
eknolo
gi
CISS
Research ProfileDistributed Systems & Semantics Unit
Info
rmati
onst
eknolo
gi
Research Evaluation, Sæby, January 12, 2006 5
Concurrency TheoryFoundation for system behavior
Verification and ValidationTools for model checking
Networks and Operating SystemsImplementation and constructionof platforms
Embedded Systems MethodologyMethods for specification, design, analysis, testing …
Industrial applications
Why CISS ?
80% of all software is embedded Demands for
increased functionalitywith minimal resources
Requires multitude of skills
Software constructionHardware platformsControl theoryComm. technology
Goal:Give a qualitative lift to current industrial practice
!!!!!
CISS Structure
Institut for Datalogi
Institut for Datalogi
Institut for Elektroniske Systemer
Institut for Elektroniske Systemer
BRICS@AalborgModelling and Validation;Programming Languages;
Software Engineering
BRICS@AalborgModelling and Validation;Programming Languages;
Software Engineering
Embedded SystemsCommunication;
HW/SWPower Management
Embedded SystemsCommunication;
HW/SWPower Management
DistributedReal Time Systems
Control Theory;Real Time Systems;
Networking.
DistributedReal Time Systems
Control Theory;Real Time Systems;
Networking.
IKT VirksomhederIKT Virksomheder
Eksterne kontakter:EE&CS BerkeleyES OldenborgES HollandARTIST
Eksterne kontakter:EE&CS BerkeleyES OldenborgES HollandARTIST
MVTU25.5 MDKK
MVTU25.5 MDKK
Nordjyllands AmtAalborg Kommune12 MDKK
Nordjyllands AmtAalborg Kommune12 MDKK
AAU12.75 MDKK
AAU12.75 MDKK
Virksomheder12.75 MDKK
Virksomheder12.75 MDKK
Partners
S-Card
RTX Telecom
Analog Devices
Aeromark
Simrad
Danfoss
Grundfos
IAR Systems
GateHouse
Ericsson Telebit
MAN B&W
Aalborg Industries
Motorola
SkovBlip Systems
Novo Nordisk
FOSS
Exhausto
ETI
TK Systemtest
SpaceCom
Panasonic
TDC Totalløsninger
Focus Areas
Applications
Technology
Tools
Modeling
MethodsProtokoller
Design- ogProg.sprog
Operativsystem
HW platform
GPSOpen source
Home automationMobile robotter
Intelligente sensorerAd hoc netværk
MobiltlfAudio/Video
Konsum elektrKontrolsystemer
AutomobileX-by wire
Algo
ritm
ik
SW-u
dvikl
ingRe
souc
e(P
ower
) Man
gem
ent
Relia
bility
Test
& Va
lider
ingHy
bride
syste
mer
Kom
mun
ikatio
nste
ori
Focus Areas
Applications
Technology
Tools
Modeling
MethodsProtokoller
Design- ogProg.sprog
Operativsystem
HW platform
GPSOpen source
Home automationMobile robotter
Intelligente sensorerAd hoc netværk
MobiltlfAudio/Video
Konsum elektrKontrolsystemer
AutomobileX-by wire
Algo
ritm
ik
SW-u
dvikl
ingRe
souc
e(P
ower
) Man
gem
ent
Relia
bility
Test
& Va
lider
ingHy
bride
syste
mer
Kom
mun
ikatio
nste
ori
Model based development
Intellingent sensor networkIT in automation
Embedded and RT OS
RT
RT Java Lab
Resource Optimal Scheduling
Testing and Verification
HW/SW Co-design / Design Space Exploration
Embedded Security
Local Regional National
HW&K
Kontrol
SW
Mekatr.
HW&K
SW
Mekatr.Kontrol
IIS
1)
2)
3)
DaNESDanish Network for Intelligent Embedded SystemsPARTNERS
CISS, IMM, MCI, PAJ SystemteknikGateHouse A/SICE Power Skov A/S Terma A/SNovo Nordisk A/S IO Technologies
Funded by Højteknologifonden
Budget63 MDKK / 4 years
Spectacular software bugsAriane 5
The first Ariane 5 rocket was launched in June, 1996. It used software developed for the successful Ariane 4. The rocket carried two computers, providing a backup in case one computer failed during launch. Forty seconds into its maiden flight, the rocket veered off course and exploded. The rocket, along with $500 million worth of satellites, was destroyed.
Ariane 5 was a much more powerful rocket and generated forces that were larger than the computer could handle. Shortly after launch, it received an input value that was too large. The main and backup computers shut down, causing the rocket to veer off course.
Spectacular software bugsU.S.S. Yorktown, U.S. Navy
When the sailor entered the mistaken number, the computer tried to divide by zero, which isn't possible. The software didn't check to see if the inputs were valid before computing and generated an invalid answer that was used by another computer. The error cascaded several computers and eventually shut down the ship's engines.
In 1998, the USS Yorktown became the first ship to test the US Navy's Smart Ship program. The Navy planned to use off-the-shelf computers and software instead of expensive U.S.S. Yorktown, courtesy of U.S. Navy custom-made machines. A sailor mistakenly entered a zero for a data value on a computer. Within minutes, Yorktown was dead in the water. It was several hours before the ship could move again.
Spectacular software bugsMoon or Missiles
The United States established the Ballistic Missile Early Warning System (BMEWS) during the Cold War to detect a Soviet missile attack. On October 5, 1960 the BMEWS radar at Thule, Greenland detected something. Its computer control system decided the signal was made by hundreds of missiles coming toward the US.
The radar had actually detected the Moon rising over the horizon. Unfortunately, the BMEWS computer had not been programmed to understand what the moon looked like as it rose in the eastern sky, so it interpreted the huge signal as Soviet missiles. Luckily for all of us, the mistake was realized in time.
Spectacular Software Bugs…. continued
INTEL Pentium II floating-point division 470 Mill US $
Baggage handling system, Denver 1.1 Mill US $/day for 9 months
Mars Pathfinder…….
Spectacular software bugsTherac 25
The Therac-25 was withdrawn from use after it was determined that it could deliver fatal overdoses under certain conditions. The software would shut down the machine before delivering an overdose, but the error messages it displayed were so unhelpful that operators couldn't tell what the error was, or how serious it was. In some cases, operators ignored the message completely.
The Therac-25 radiation therapy machine was a medical device that used beams of electrons or photons to kill cancer cells. Between 1985-1987, at least six people got very sick after Therac-25 treatments. Four of them died. The manufacturer was confident that their software made it impossible for the machine to harm patients.
“Malfunction 54”
““Malfunction 54
Malfunction 54””“H-tilt”““HH--tilttilt””
IEEE Computer, Vol. 26, No. 7, July 1993, pp. 18-41IEEE ComputerIEEE Computer, Vol. 26, No. 7, July 1993, pp. 18, Vol. 26, No. 7, July 1993, pp. 18--4141
Why T&V?
Errors in (Embedded) software are extremely expensive
Michael WilliamsResearch Director, Ericsson,
SE
Why T&V?
Errors in (Embedded) software are extremely expensive
30-40% of development time spent on (often ad-hoc) testing.
There is a enormous potential for improved methods and tools.
“Time-to-market” can be reduced through earli verification and performance analysis
Michael WilliamsResearch Director, Ericsson,
SE
System
Verification and Test
/* Wait for events */void OS_Wait(void);
/* Operating system visualSTATE process. Mimics a OS process for a* visualSTATE system. In this implementation this is the mainloop* interfacing to the visualSTATE basic API. */void OS_VS_Process(void);
/* Define completion code variable. */unsigned char cc;
void HandleError(unsigned char ccArg){printf("Error code %c detected, exiting application.\n", ccArg);exit(ccArg);
}
/* In d-241 we only use the OS_Wait call. It is used to simulate a* system. It purpose is to generate events. How this is done is up to* you.*/void OS_Wait(void){/* Ignore the parameters; just retrieve events from the keyboard and* put them into the queue. When EVENT_UNDEFINED is read from the* keyboard, return to the calling process. */SEM_EVENT_TYPE event;int num;
/* Wait for events */void OS_Wait(void);
/* Operating system visualSTATE process. Mimics a OS process for a* visualSTATE system. In this implementation this is the mainloop* interfacing to the visualSTATE basic API. */void OS_VS_Process(void);
/* Define completion code variable. */unsigned char cc;
void HandleError(unsigned char ccArg){printf("Error code %c detected, exiting application.\n", ccArg);exit(ccArg);
}
/* In d-241 we only use the OS_Wait call. It is used to simulate a* system. It purpose is to generate events. How this is done is up to* you.*/void OS_Wait(void){/* Ignore the parameters; just retrieve events from the keyboard and* put them into the queue. When EVENT_UNDEFINED is read from the* keyboard, return to the calling process. */SEM_EVENT_TYPE event;int num;
Kode
Spec
ΦΦΦΦ
• VerifikationKode/Model mht Spec
• Test System mht Model/Spec
• VerifikationKode/Model mht Spec
• Test System mht Model/Spec
Model
Test versus Verification
Airbus Control Panel
T1 T3 T5 T1 … T4 T3
E F E E G H … H A
A
A
A A
A
A A
B
B B
B BBB
2n sequences of length n
TEST VERIFIKATION
Deadlock identified by VERIFICATIONafter sequence of
2000 msgs / < 1min.
UPPAAL
Info
rmat
ionst
eknolo
gi
CISS
Traditional Software Development
The Waterfall Model
Analyse
Design
Coding
Testing♦Costly in time-to-market and money♦ Errors are detected late or never♦ Application of models as early as possible
ProblemArea
Runn
ing
Syste
m
REVIE
WS
REVIE
WS
Info
rmat
ionst
eknolo
gi
CISS
Model-Driven Development
Design Model SpecificationVerification & Refusal
AnalysisValidation
FORMAL METHODS
ImplementationTesting
UML
Monitoring
AutomaticCode generation
AutomaticTest generation
AutomaticMonitoring
Info
rmat
ionst
eknolo
gi
CISS
ModelsA model is a simplified representation of the real world.Used gain confidence in the adequacy and validity of a proposed systemModels selected aspectsRemoves irrelevant details
Implementation
Model Realization
”implements??”
Info
rmat
ionst
eknolo
gi
CISS
ModelsAbstractions of the problem-space, not solution spaceDomain Specific Modeling Languages
Simulink/StateFlowUML,
Early exploration of design-alternativesAutomatic transformation
Correctness-by-construction vs. Correctness-by-correction
Info
rmat
ionst
eknolo
gi
CISS
Model-based vs. MDDModel Driven Development:
Model is the center of focus from analysis to executionModel is gradually refined / transformed into solution
Model-based Development:(Unrelated) models used to support selected development activities where appropriate
Info
rmat
ionst
eknolo
gi
CISS
UPPAALGraphical Design Tool• state machines• datatypes• C-code• clocks• communication
Graphical Design Tool• state machines• datatypes• C-code• clocks• communication Graphical Simulator
• visualization and recording
• MSCs• Gannt Charts
Graphical Simulator• visualization
and recording• MSCs• Gannt Charts
Verifier• Exhaustive & automatic
checking of requirements
Verifier• Exhaustive & automatic
checking of requirements
UPPAAL the leading integratedtool environment for modeling, simulation and verification of
real-time systems
UPPAAL the leading integratedtool environment for modeling, simulation and verification of
real-time systems
Info
rmat
ionst
eknolo
gi
CISS
UppAal-TRONTesting Real-time systens ONline
Use model to automatically generate input stimuli Use model as oracle to automatically evaluate response
Test computer
Info
rmat
ionst
eknolo
gi
CISS
TestingTesting:
to check the quality (functionality, reliability, performance, …) of an (software) object
-by performing experiments-in a controlled way
To find errorsTo determine risk of release
• In avg. 10-20 errors per 1000 LOC•30-50 % of development time and cost in embedded software
Info
rmat
ionst
eknolo
gi
CISS
Types of Testing
unit
integration
system
efficiency
functionality
white box black box
Level
Accessibility
Aspect
usability
reliability
Info
rmat
ionst
eknolo
gi
CISS
Quality-Characteristics (ISO-9126)Functionality
Suitability, accuracy, security, compliance, interoperability
Reliabilitymaturity, fault tolerance, recoverability
Usabilityunderstandability, learnability, operability
Efficiencytime behaviour, resource utilization
MaintainabilityAnalysability, changeability, stability, testability
PortabilityAdaptability, installability, conformance, replaceability
⇒ functional testing
⇒ reliability testing
⇒ usability testing
⇒ performance testing
⇒ maintainability testing ??
⇒ portability testing ?
Info
rmat
ionst
eknolo
gi
CISS
Fundamental Testing Problems
Critical path in the development cycle Infinity of testingTest Oracle ProblemLack of failure modelsDestructive
Less prestigiousDeveloper vs. Independent Tester
Info
rmat
ionst
eknolo
gi
CISS
What is a Test?
Software under Test
Test Data Output
Test Cases
Correct result?
Oracle
Info
rmat
ionst
eknolo
gi
CISS
Testing Process
systemspecification
test cases(abstract)
executabletest cases
verdict
test generation
test implementation
test execution & analysis
Info
rmat
ionst
eknolo
gi
CISS
Manual Testing1. Figure out what to test?!2. Design good (abstract) test cases3. Implement as scripts (or manual execution and
analysis)4. Execute results5. Analyze results
Regression testing only practical when automatedRequirements and system understanding evolve or implementation change ⇒Maintenance of test-scripts ??
Info
rmat
ionst
eknolo
gi
CISS
TestGene-ratortool
TestGene-ratortool
click?x:=0
click?x<2
x>=2
DBLclick!
Automated Model Based Conformance Testing
fail
pass
Testexecution
tool
Testexecution
toolEvent
mapping
Driver
Model Test suite
TestGenerator
tool
TestGenerator
tool
Implementation Relation
Selection &optimization
Does the behavior of the (blackbox) implementation comply to that of the specification?
ImplementationUnder Test
Info
rmat
ionst
eknolo
gi
CISS
A Self-Assessment Test [Myers]
“A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.”
Write a set of test cases to test this program
Info
rmat
ionst
eknolo
gi
CISS
Test cases for:
A Self-Assessment Test [Myers]
1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?7. one side is sum of others ?8. 3 permutations of previous ?
9. one side larger than sum ofothers ?
10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?14. for each test case: is
expected output specified ?15. check behaviour after
output was produced ?
Info
rmat
ionst
eknolo
gi
CISS
Schedule: Seminar 19.00-10.00 Lecture: Course Introduction,
Introduction to Model-Driven Development 10.00-11.15 Lecture: Test case design techniques 11.15-12.30 Exercise: Test case design, Bullseye and CUnit12.30-13.30 Lunch 13.30-14.15 Lecture: Introduction to FSM-modelling14.30-15-30 Exercise: Modelling and simulation using
UppAal.
Info
rmat
ionst
eknolo
gi
CISS
Outline: Seminar 2Modeling (continued), Timed automataVerification, Model-checkingUML diagrams for testingCommercial Tool-Demo (Test-Conductor)(Offline) Test generationState-based testing of objectsMini-Project announcement
Info
rmat
ionst
eknolo
gi
CISS
Outline: Seminar 3System TestTest RealizationOnline-testing
Correctness, IOCOUppaal-TRONCase study
Mini-project Consultancy