Model-based Testing and and Validation

Arne SkouBrian Nielsen,

{ask|bnielsen}@cs.aau.dk

Info

rmat

ionst

eknolo

gi

CISS

AgendaWho are we??Why T&V? Testing versus VerificationModel-Driven DevelopmentTesting & MBTCourse Plan

Who are we?

Info

rmat

ionst

eknolo

gi

CISS

Research ProfileDistributed Systems & Semantics Unit

Info

rmati

onst

eknolo

gi

Research Evaluation, Sæby, January 12, 2006 5

Concurrency TheoryFoundation for system behavior

Verification and ValidationTools for model checking

Networks and Operating SystemsImplementation and constructionof platforms

Embedded Systems MethodologyMethods for specification, design, analysis, testing …

Industrial applications

7

Center for Indlejrede Software Systemer

Why CISS ?

80% of all software is embedded Demands for

increased functionalitywith minimal resources

Requires multitude of skills

Software constructionHardware platformsControl theoryComm. technology

Goal:Give a qualitative lift to current industrial practice

!!!!!

CISS Structure

Institut for Datalogi

Institut for Datalogi

Institut for Elektroniske Systemer

Institut for Elektroniske Systemer

BRICS@AalborgModelling and Validation;Programming Languages;

Software Engineering

BRICS@AalborgModelling and Validation;Programming Languages;

Software Engineering

Embedded SystemsCommunication;

HW/SWPower Management

Embedded SystemsCommunication;

HW/SWPower Management

DistributedReal Time Systems

Control Theory;Real Time Systems;

Networking.

DistributedReal Time Systems

Control Theory;Real Time Systems;

Networking.

IKT VirksomhederIKT Virksomheder

Eksterne kontakter:EE&CS BerkeleyES OldenborgES HollandARTIST

Eksterne kontakter:EE&CS BerkeleyES OldenborgES HollandARTIST

MVTU25.5 MDKK

MVTU25.5 MDKK

Nordjyllands AmtAalborg Kommune12 MDKK

Nordjyllands AmtAalborg Kommune12 MDKK

AAU12.75 MDKK

AAU12.75 MDKK

Virksomheder12.75 MDKK

Virksomheder12.75 MDKK

Partners

S-Card

RTX Telecom

Analog Devices

Aeromark

Simrad

Danfoss

Grundfos

IAR Systems

GateHouse

Ericsson Telebit

MAN B&W

Aalborg Industries

Motorola

SkovBlip Systems

Novo Nordisk

FOSS

Exhausto

ETI

TK Systemtest

SpaceCom

Panasonic

TDC Totalløsninger

Focus Areas

Applications

Technology

Tools

Modeling

MethodsProtokoller

Design- ogProg.sprog

Operativsystem

HW platform

GPSOpen source

Home automationMobile robotter

Intelligente sensorerAd hoc netværk

MobiltlfAudio/Video

Konsum elektrKontrolsystemer

AutomobileX-by wire

Algo

ritm

ik

SW-u

dvikl

ingRe

souc

e(P

ower

) Man

gem

ent

Relia

bility

Test

& Va

lider

ingHy

bride

syste

mer

Kom

mun

ikatio

nste

ori

Focus Areas

Applications

Technology

Tools

Modeling

MethodsProtokoller

Design- ogProg.sprog

Operativsystem

HW platform

GPSOpen source

Home automationMobile robotter

Intelligente sensorerAd hoc netværk

MobiltlfAudio/Video

Konsum elektrKontrolsystemer

AutomobileX-by wire

Algo

ritm

ik

SW-u

dvikl

ingRe

souc

e(P

ower

) Man

gem

ent

Relia

bility

Test

& Va

lider

ingHy

bride

syste

mer

Kom

mun

ikatio

nste

ori

Model based development

Intellingent sensor networkIT in automation

Embedded and RT OS

RT

RT Java Lab

Resource Optimal Scheduling

Testing and Verification

HW/SW Co-design / Design Space Exploration

Embedded Security

Local Regional National

HW&K

Kontrol

SW

Mekatr.

HW&K

SW

Mekatr.Kontrol

IIS

1)

2)

3)

DaNESDanish Network for Intelligent Embedded SystemsPARTNERS

CISS, IMM, MCI, PAJ SystemteknikGateHouse A/SICE Power Skov A/S Terma A/SNovo Nordisk A/S IO Technologies

Funded by Højteknologifonden

Budget63 MDKK / 4 years

Local Regional National

DaNES

Complex Systems

A very complex system

Klaus Havelund, NASA

Rotterdam Storm Surge Barrier

Spectacular software bugsAriane 5

The first Ariane 5 rocket was launched in June, 1996. It used software developed for the successful Ariane 4. The rocket carried two computers, providing a backup in case one computer failed during launch. Forty seconds into its maiden flight, the rocket veered off course and exploded. The rocket, along with $500 million worth of satellites, was destroyed.

Ariane 5 was a much more powerful rocket and generated forces that were larger than the computer could handle. Shortly after launch, it received an input value that was too large. The main and backup computers shut down, causing the rocket to veer off course.

Spectacular software bugsU.S.S. Yorktown, U.S. Navy

When the sailor entered the mistaken number, the computer tried to divide by zero, which isn't possible. The software didn't check to see if the inputs were valid before computing and generated an invalid answer that was used by another computer. The error cascaded several computers and eventually shut down the ship's engines.

In 1998, the USS Yorktown became the first ship to test the US Navy's Smart Ship program. The Navy planned to use off-the-shelf computers and software instead of expensive U.S.S. Yorktown, courtesy of U.S. Navy custom-made machines. A sailor mistakenly entered a zero for a data value on a computer. Within minutes, Yorktown was dead in the water. It was several hours before the ship could move again.

Spectacular software bugsMoon or Missiles

The United States established the Ballistic Missile Early Warning System (BMEWS) during the Cold War to detect a Soviet missile attack. On October 5, 1960 the BMEWS radar at Thule, Greenland detected something. Its computer control system decided the signal was made by hundreds of missiles coming toward the US.

The radar had actually detected the Moon rising over the horizon. Unfortunately, the BMEWS computer had not been programmed to understand what the moon looked like as it rose in the eastern sky, so it interpreted the huge signal as Soviet missiles. Luckily for all of us, the mistake was realized in time.

Spectacular Software Bugs…. continued

INTEL Pentium II floating-point division 470 Mill US $

Baggage handling system, Denver 1.1 Mill US $/day for 9 months

Mars Pathfinder…….

Spectacular software bugsTherac 25

The Therac-25 was withdrawn from use after it was determined that it could deliver fatal overdoses under certain conditions. The software would shut down the machine before delivering an overdose, but the error messages it displayed were so unhelpful that operators couldn't tell what the error was, or how serious it was. In some cases, operators ignored the message completely.

The Therac-25 radiation therapy machine was a medical device that used beams of electrons or photons to kill cancer cells. Between 1985-1987, at least six people got very sick after Therac-25 treatments. Four of them died. The manufacturer was confident that their software made it impossible for the machine to harm patients.

“Malfunction 54”

““Malfunction 54

Malfunction 54””“H-tilt”““HH--tilttilt””

IEEE Computer, Vol. 26, No. 7, July 1993, pp. 18-41IEEE ComputerIEEE Computer, Vol. 26, No. 7, July 1993, pp. 18, Vol. 26, No. 7, July 1993, pp. 18--4141

Why T&V?

Errors in (Embedded) software are extremely expensive

Michael WilliamsResearch Director, Ericsson,

SE

Why T&V?

Errors in (Embedded) software are extremely expensive

30-40% of development time spent on (often ad-hoc) testing.

There is a enormous potential for improved methods and tools.

“Time-to-market” can be reduced through earli verification and performance analysis

Michael WilliamsResearch Director, Ericsson,

SE

Testing vs. Verification

System

Verification and Test

/* Wait for events */void OS_Wait(void);

/* Operating system visualSTATE process. Mimics a OS process for a* visualSTATE system. In this implementation this is the mainloop* interfacing to the visualSTATE basic API. */void OS_VS_Process(void);

/* Define completion code variable. */unsigned char cc;

void HandleError(unsigned char ccArg){printf("Error code %c detected, exiting application.\n", ccArg);exit(ccArg);

}

/* In d-241 we only use the OS_Wait call. It is used to simulate a* system. It purpose is to generate events. How this is done is up to* you.*/void OS_Wait(void){/* Ignore the parameters; just retrieve events from the keyboard and* put them into the queue. When EVENT_UNDEFINED is read from the* keyboard, return to the calling process. */SEM_EVENT_TYPE event;int num;

/* Wait for events */void OS_Wait(void);

/* Operating system visualSTATE process. Mimics a OS process for a* visualSTATE system. In this implementation this is the mainloop* interfacing to the visualSTATE basic API. */void OS_VS_Process(void);

/* Define completion code variable. */unsigned char cc;

void HandleError(unsigned char ccArg){printf("Error code %c detected, exiting application.\n", ccArg);exit(ccArg);

}

/* In d-241 we only use the OS_Wait call. It is used to simulate a* system. It purpose is to generate events. How this is done is up to* you.*/void OS_Wait(void){/* Ignore the parameters; just retrieve events from the keyboard and* put them into the queue. When EVENT_UNDEFINED is read from the* keyboard, return to the calling process. */SEM_EVENT_TYPE event;int num;

Kode

Spec

ΦΦΦΦ

• VerifikationKode/Model mht Spec

• Test System mht Model/Spec

• VerifikationKode/Model mht Spec

• Test System mht Model/Spec

Model

Test versus Verification

Airbus Control Panel

T1 T3 T5 T1 … T4 T3

E F E E G H … H A

A

A

A A

A

A A

B

B B

B BBB

2n sequences of length n

TEST VERIFIKATION

Deadlock identified by VERIFICATIONafter sequence of

2000 msgs / < 1min.

UPPAAL

MDD

Info

rmat

ionst

eknolo

gi

CISS

Traditional Software Development

The Waterfall Model

Analyse

Design

Coding

Testing♦Costly in time-to-market and money♦ Errors are detected late or never♦ Application of models as early as possible

ProblemArea

Runn

ing

Syste

m

REVIE

WS

REVIE

WS

Info

rmat

ionst

eknolo

gi

CISS

Introducing, Detecting and Repairing Errors Liggesmeyer 98

Info

rmat

ionst

eknolo

gi

CISS

Introducing, Detecting and Repairing Errors Liggesmeyer 98

Info

rmat

ionst

eknolo

gi

CISS

Model-Driven Development

Design Model SpecificationVerification & Refusal

AnalysisValidation

FORMAL METHODS

ImplementationTesting

UML

Monitoring

AutomaticCode generation

AutomaticTest generation

AutomaticMonitoring

Info

rmat

ionst

eknolo

gi

CISS

ModelsA model is a simplified representation of the real world.Used gain confidence in the adequacy and validity of a proposed systemModels selected aspectsRemoves irrelevant details

Implementation

Model Realization

”implements??”

Info

rmat

ionst

eknolo

gi

CISS

ModelsAbstractions of the problem-space, not solution spaceDomain Specific Modeling Languages

Simulink/StateFlowUML,

Early exploration of design-alternativesAutomatic transformation

Correctness-by-construction vs. Correctness-by-correction

Info

rmat

ionst

eknolo

gi

CISS

Model-based vs. MDDModel Driven Development:

Model is the center of focus from analysis to executionModel is gradually refined / transformed into solution

Model-based Development:(Unrelated) models used to support selected development activities where appropriate

Info

rmat

ionst

eknolo

gi

CISS

Iterative – Multiple V

Info

rmat

ionst

eknolo

gi

CISS

UPPAALGraphical Design Tool• state machines• datatypes• C-code• clocks• communication

Graphical Design Tool• state machines• datatypes• C-code• clocks• communication Graphical Simulator

• visualization and recording

• MSCs• Gannt Charts

Graphical Simulator• visualization

and recording• MSCs• Gannt Charts

Verifier• Exhaustive & automatic

checking of requirements

Verifier• Exhaustive & automatic

checking of requirements

UPPAAL the leading integratedtool environment for modeling, simulation and verification of

real-time systems

UPPAAL the leading integratedtool environment for modeling, simulation and verification of

real-time systems

Info

rmat

ionst

eknolo

gi

CISS

UppAal-TRONTesting Real-time systens ONline

Use model to automatically generate input stimuli Use model as oracle to automatically evaluate response

Test computer

Info

rmat

ionst

eknolo

gi

CISS

TestingTesting:

to check the quality (functionality, reliability, performance, …) of an (software) object

-by performing experiments-in a controlled way

To find errorsTo determine risk of release

• In avg. 10-20 errors per 1000 LOC•30-50 % of development time and cost in embedded software

Info

rmat

ionst

eknolo

gi

CISS

Types of Testing

unit

integration

system

efficiency

functionality

white box black box

Level

Accessibility

Aspect

usability

reliability

Info

rmat

ionst

eknolo

gi

CISS

Quality-Characteristics (ISO-9126)Functionality

Suitability, accuracy, security, compliance, interoperability

Reliabilitymaturity, fault tolerance, recoverability

Usabilityunderstandability, learnability, operability

Efficiencytime behaviour, resource utilization

MaintainabilityAnalysability, changeability, stability, testability

PortabilityAdaptability, installability, conformance, replaceability

⇒ functional testing

⇒ reliability testing

⇒ usability testing

⇒ performance testing

⇒ maintainability testing ??

⇒ portability testing ?

Info

rmat

ionst

eknolo

gi

CISS

Fundamental Testing Problems

Critical path in the development cycle Infinity of testingTest Oracle ProblemLack of failure modelsDestructive

Less prestigiousDeveloper vs. Independent Tester

Info

rmat

ionst

eknolo

gi

CISS

What is a Test?

Software under Test

Test Data Output

Test Cases

Correct result?

Oracle

Info

rmat

ionst

eknolo

gi

CISS

Testing Process

systemspecification

test cases(abstract)

executabletest cases

verdict

test generation

test implementation

test execution & analysis

Info

rmat

ionst

eknolo

gi

CISS

Manual Testing1. Figure out what to test?!2. Design good (abstract) test cases3. Implement as scripts (or manual execution and

analysis)4. Execute results5. Analyze results

Regression testing only practical when automatedRequirements and system understanding evolve or implementation change ⇒Maintenance of test-scripts ??

Info

rmat

ionst

eknolo

gi

CISS

TestGene-ratortool

TestGene-ratortool

click?x:=0

click?x<2

x>=2

DBLclick!

Automated Model Based Conformance Testing

fail

pass

Testexecution

tool

Testexecution

toolEvent

mapping

Driver

Model Test suite

TestGenerator

tool

TestGenerator

tool

Implementation Relation

Selection &optimization

Does the behavior of the (blackbox) implementation comply to that of the specification?

ImplementationUnder Test

Info

rmat

ionst

eknolo

gi

CISS

A Self-Assessment Test [Myers]

“A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.”

Write a set of test cases to test this program

Info

rmat

ionst

eknolo

gi

CISS

Test cases for:

A Self-Assessment Test [Myers]

1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?7. one side is sum of others ?8. 3 permutations of previous ?

9. one side larger than sum ofothers ?

10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?14. for each test case: is

expected output specified ?15. check behaviour after

output was produced ?

Course Plan

Info

rmat

ionst

eknolo

gi

CISS

Schedule: Seminar 19.00-10.00 Lecture: Course Introduction,

Introduction to Model-Driven Development 10.00-11.15 Lecture: Test case design techniques 11.15-12.30 Exercise: Test case design, Bullseye and CUnit12.30-13.30 Lunch 13.30-14.15 Lecture: Introduction to FSM-modelling14.30-15-30 Exercise: Modelling and simulation using

UppAal.

Info

rmat

ionst

eknolo

gi

CISS

Outline: Seminar 2Modeling (continued), Timed automataVerification, Model-checkingUML diagrams for testingCommercial Tool-Demo (Test-Conductor)(Offline) Test generationState-based testing of objectsMini-Project announcement

Info

rmat

ionst

eknolo

gi

CISS

Outline: Seminar 3System TestTest RealizationOnline-testing

Correctness, IOCOUppaal-TRONCase study

Mini-project Consultancy

END