static techniques (on code) (static analysis of code) performed on the code without executing the...

Static Techniques (on code)

(Static Analysis of code)

performed on the code without executing the code

Categories of Static Techniques

• Manual

• (Semi) Mechanised

Static techniques vs. code testing• Code testing tries to characterize set of executions

throughout one test case --- (minimal) coverage (of input similar executions by using classes of input data, of paths, others) is the most important issue

• Static techniques characterize once set of executions; that’s the reason to call qualify them as static techniques

• Static techniques are usually used within a verification activity (i.e they may come before testing)

• Static techniques and testing have complementary advantages and disadvantages; additionally, some static techniques during the testing to support the test case design

Informal analysis techniques:Code walkthroughs

• Recommended prescriptions– Small number of people (three to five)

– Participants receive written documentation from the designer few days before the meeting

– Predefined duration of meeting (few hours)

– Focus on the discovery of defects, not on fixing them

– Participants: designer, moderator, and a secretary

– Foster cooperation; no evaluation of people• Experience shows that most defects are discovered by the designer

during the presentation, while trying to explain the design to other people.

Informal analysis techniques:Code inspection

• A reading code technique aiming at defect discovery

• Based on checklist (also called defect-guessing), e.g.:– use of uninitialized variables;

– jumps into loops;

– nonterminating loops;

– array indexes out of bounds;

– …

Defect Guessing

• From intuition and experience, enumerate a list of possible defects or defect prone situations

• Defect guessing can also be used to write test cases to expose those defect

Defect Guessing: Esempio

• Nel caso di array o stringhe, ogni indice è compreso nei limiti della dimensione corrispondente?

• Ci si riferisce ad una variabile non inizializzata?

• Per i riferimenti attraverso puntatore/riferimento, la corrispondente area di memoria è stata allocata (dangling reference problem)?

• Una variabile (eventualmente riferita tramite puntatore) ha tipo diverso da quello usato dal programma?

• Esistono variabili con nome simile (pratica pericolosa)?


• I calcoli coinvolgono tipi diversi e inconsistenti (ad es., stringhe e interi)?

• Esistono delle inconsistenze nei calcoli misti (ad es., interi e reali)?

• Esistono dei calcoli che coinvolgono tipi compatibili ma di precisione differente?

• In un’assegnazione (x:=exp, x=exp), il valore sinistro ha rappresentazione meno precisa del valore destro?

• È possibile una condizione di overflow o underflow (ad esempio nei calcoli intermedi)?

• Divisione per zero?


• Nelle espressioni che contengono vari operatori aritmetici, le assunzioni sull’ordine di valutazione e di precedenza degli operatori sono corrette?

• Il valore di una variabile esce dall’intervallo ragionevole? (ad es., il peso dovrebbe essere positivo, …)

• Ci sono usi sbagliati dell’aritmetica fra interi, in particolare delle divisioni?

• Stiamo prendendo in considerazione adeguatamente la precisione finita dei reali?

• Gli operatori di confronto sono usati correttamente?

• Le espressioni booleane sono corrette (uso appropriato di and, or, not)?

• Nelle espressioni che contengono vari operatori booleani, le assunzioni sull’ordine di valutazione e di precedenza degli operatori sono corrette?

• Gli operandi di un’espressione booleana sono booleani?

• La maniera in cui viene valutata l’espressione booleana ha impatto sul significato dell’espressione stessa (pratica

pericolosa)?


Correctness proofs

A program and its specification (Hoare notation)

{true}begin

read (a); read (b);x := a + b;write (x);

end{output = input1 + input2}

proof by backwards substitution

Proof rules

Claim1, Claim2

Claim3

Notation for:

If Claim 1 and Claim 2 have been proven, one can deduce Claim3

Proof rules for a language

{F1}S1{F2}, {F2}S2{F3}

{F1}S1;S2{F3}

sequence

{Pre and cond} S1 {Post},{Pre and not cond} S2 {Post}{Pre} if cond then S1 ; else S 2 ; end if; {Post}

if-then-else

while-do{I and cond} S {I}{I} while cond loop S; end loop; {I and not cond}

I is called the loop invariant

Correctness proof

• Partial correctness proof– validity of {Pre} program {Post} guarantees

that if the Pre holds before the execution of program, and if the program ever terminates, then Post will be achieved

• Total correctness proof– Pre guarantees program termination and the

truth of Post

These problems are undecidable!!!

Example

{input1 > 0 and input2 > 0}begin

read (input1); read (input2);x=input1; y=input2;div := 0;while x >= y loop

div := div + 1;x := x - y;

end loop;write (div); write (x);

end;{input1 = div * input2 + x}

Discovery of loop invariant

• Difficult and creative because invariants cannot be constructed automatically

• In the previous example

input1 = div * y + x and y=input2

Combining correctness proofs

• We can prove correctness of operations (e.g. operations on a class)

• Then use the result of each single proof to make proof for complex modules containing these operations or complex combinations of these operations

Examplemodule TABLE;exports

type Table_Type (max_size: NATURAL): ?;no more than max_size entries may be stored in a table; user modules must guarantee thisprocedure Insert (Table: in out TableType ;

ELEMENT: in ElementType);procedure Delete (Table: in out TableType;

ELEMENT: in ElementType);function Size (Table: in Table_Type) return NATURAL;provides the current size of a table…

end TABLE

{true}Delete (Table, Element);{Element Table};

{Size (Table) < max_size}Insert (Table, Element){Element Table};

Having proved these

We can then prove properties of programs using tables

For example, that after executing the sequence

Insert(T, x);Delete(T, x);

x is not present in T

An assessment ofcorrectness proofs

• Still not used in practice• However

– possibly used for very critical parts of code or in high risk software!

– assertions (any intermediate property) may be the basis for a systematic way of inserting runtime checks (instead of checking values of variables)

– proofs may become more practical as more powerful support tools are developed

– knowledge of correctness theory helps programmers being rigorous

– well written post conditions can be used to design test cases

Symbolic execution

• Can be viewed as a middle way between testing and pure verification (but it is anyway a verification technique)

• Executes the program on symbolic values (symbolic expressions)

• One symbolic execution corresponds to many usual program executions

Example (1)Consider executing the following programwith x=X, y=Y, a=A (symbolic binding)

x := y + 2;if x > a then

a := a + 2;else

y := x + 3;end if;x := x + a + y;

Build the control graph!

1

2 3

4

x := y + 2

a := a + 2y := x + 3

x := x + a + y

x > a x <= a

Example(2)• When control reaches decisions, in general,

symbolic values do not allow to select a branch

• One can choose a branch, and record the choice in a path condition

• Result:< <a = A, y = Y + 5, x = 2 * Y + A + 7>

<1, 3, 4> , Y + 2 <= A >execution pathpath condition

symbolic binding

Symbolic execution rules (1)

• read (x) – removes any existing binding for x and adds

binding x = X, where X is a newly introduced symbol

• write (expression)– output(n) = computed_symbolic_value_expression (n

counter initialized to 1 and automatically incremented after each write statement)

symbolic state: <symbolic_binding, execution_path, path_condition>


• x:= expression – construct symbolic value of expression, SV;

replace existing binding of x with x=SV

• After execution of a statement of a path that corresponds to an edge of control graph, append the edge to execution path


• if cond then S1; else S2; endif • while cond loop…endloop

– condition is symbolically evaluated • eval(cond)

– If it is possible to state eval(cond) true or eval(cond) false then execution proceeds by following the appropriate branch

– otherwise, make a choice of true or false, and conjoin eval(cond) (resp., not [eval(cond)] to the path condition

Symbolic execution and testing• The path condition describes all data are

required for the program execution follows the execution path

• Usage of symbolic execution for testing:– identify one execution path (I.e. a sequence of

arrows in a control graph)– symbolically execute the execution path (if

possible)– chose data that satisfy the path condition

• These data allow to execute that path and therefore are a test case for the path.

Example (1)found := false; counter := 1;while (not found) and counter <= number_of_items loop

if table (counter) = desired_element then found := true;

end if;counter := counter + 1;

end loop;if found then write ("the desired element exists in the table");else write ("the desired element does not exist

in the table");end if;

Example (2)1,2,3,5,6,2,4… is

not possible!1

2

3

4

6

5

7

8 9

write "the desired element exists in the table"

write "the desired element does not exist in the table"

found:= true

counter:= counter+1

table (counter) = desired_element

table (counter) < > desired_element

(not found) and counter <= number_of_items

(found) or counter > number_of_items

found := false; counter := 1;

Why so many approaches to testing and analysis?

• Testing versus static techniques

• Formal versus informal techniques

• White-box versus black-box techniques

• Fully mechanised vs. semi-mechanised techniques (for undecidable properties)

• …

view all these as complementary

OO Unit Code Defect Testing

Test Case Design Objectives

• Coverage is the always the key point (what things are covered by the test cases).

•

• Minimality of this coverage is the other key point (do not write two distinct test cases for discovering same defects).

Some techniques for making test cases• General defect testing

– The tester looks for plausible defects (i.e., aspects of the implementation of the system that may result in defects). To determine whether these defects exist, test cases are designed to exercise the code.

• Specialized defect testing: Class Testing and Class Hierarchy– Inheritance does not obviate the need for thorough testing of all derived

classes. In fact, it can actually complicate the testing process.

• Scenario-Based Test Design (defect but also acceptance testing)– Scenario-based testing concentrates on what the user does, not what the

product does. This means capturing the tasks (via use-cases) that the user has to perform, then applying them and their variants as test cases.

Two levels of test• Test of one class

– Test single operations (white and black box)– Test sequences of operations (random test)– Test sequences of states (behavior test)

• Test of sequences of operations and states is because in object-oriented code, expressing the expected-behavior or what the program does in term of input and output is not possible; therefore, the codomain R is described by sequences of operations or states

• Test of several classes with interacting objects

Esempio

Pre: pulsante P premuto

Post: la richiesta R è memorizzata

: RichiestaInterna

8: disabilitarepossibilità di fare richieste per lo stesso piano()

2: qual è il peso()

3: è possibile fare una richiesta per il piano del pulsante premuto?()

6: dimmi i pulsanti da illuminare()

LifeLine2 : Pulsante

1: pulsante premuto()

Visual Paradigm for UML Standard Edition(Universita di Torino)+pulsante premuto()+qual è il peso()+è possibile fare una richiesta per il piano del pulsante premuto?()+dimmi i pulsanti da illuminare()+disabilitarepossibilità di fare richieste per lo stesso piano()

RichiestaInternaVisual Paradigm for UML Standard Edition(Universita di Torino)

qual’è peso; è possibile…; dimmi pulsanti da illuminare…;…

Esempio

+avviaremotore()+diminuirevelocità()+fermarsi()

MotoreVisual Paradigm for UML Standard Edition(Universita di Torino)

: Partire


2: richiesta da servire()

3: qual è peso corrente()

11: qual è peso corrente

12: porte chiuse?()

ilmotore : Motore

4: avviaremotore()

Visual Paradigm for UML Standard Edition(Universita di Torino)

: Fermare

8: dimmi pulsanti da spegnere()

2: esiste una richiesta da servire?()

3: pianorichiesta=X()

ilmotore: : Motore

4: diminuirevelocità()

6: fermarsi()

5:

7:


avviare;diminuire; fermare;

Esempio

+open()+setupAccount()+deposit()+balance()+withdraw()+finalwiithdraw()+close()

AccountVisual Paradigm for UML Standard Edition(Universita di Torino)

Le operazioni sono “indipendenti” a parte alcuni vincoli che possono essere espressi con pre e post condizioni

open; setupAccount; deposit;

Behavior Testing

emptyacctopen setup Accnt

set upacct

deposit(initial)

workingacct

withdrawal(final)

deadacct close

nonworkingacct

deposit

withdrawbalance

creditaccntInfo

Figure 14.3 State diagram for Account class (adapted from [ KIR94])

The tests to be The tests to be designed designed should achieve should achieve all all state state coveragecoverage [KIR94]. That [KIR94]. That is, the is, the operation operation sequencessequences should cause should cause the the Account Account classclass to make to make transition transition through all through all allowable allowable statesstates

Black box!

+open()+setupAccount()+deposit()+balance()+withdraw()+finalwiithdraw()+close()

AccountVisual Paradigm for UML Standard Edition(Universita di Torino)

OO Software: Inter-Class Testing• Inter-class testing to exercise interactions between

classes:– For each class, use the list of class operations to generate a

series of random test sequences of operations. The operations will send messages to other classes.

– For each message that is generated, determine the destination class and the corresponding operations.

– For each operation in the destination class (that has been invoked by incoming messages), determine the messages that it transmits.

– For each of the messages, determine the next level of operations that are invoked and incorporate these into the test sequence

White box if based on class code!

Black box if based on sequence diagrams!

: RichiestaIntern...

11: disabilitarepossibilità di fare richieste per lo stesso piano()

4: qual è il peso()

5: è possibile fare una richiesta per il piano del pulsante premuto?()


: Riche...

6: creare e memorizzarsi()

8:

: Pulsan...

10: illuminarsi-per conferma richiesta()

LifeLine2 : Pulsante3 SchedulatoreTask

1: pulsante premuto

2: dimmi l'evento da trattare

3:

interface

7: rendi persistente


OO Software: Additional tests

• New issues– inheritance– genericity– polymorphism– dynamic binding

• Open problems still exist

White box!

How to test classes of a hierarchy?

• “Flattening” the whole hierarchy and considering every class as a totally independent unit– does not exploit incremental class definition

• Finding an ad-hoc way to take advantage of the hierarchy

A simple technique test case design for class hierarchy

• A test case that does not have to be repeated for any heir

• A test case that must be performed for heir class X and all of its further heirs

• A test case that must be redone by applying the same input data, but verifying that the output is not (or is) changed

• A test case that must be modified by adding other input parameters and verifying that the output changes accordingly

Black Box testing: concurrent and real-time software

• Non-determinism (of events driving the control flow) inherent in concurrency affects repeatability of failures

• For real-time software (i.e. with time constraints), a test case consists not only of usual input data, but also of the time when such data are supplied (events)

eventseventsinputinput

outputoutput

Software Testing in the large

Software Testing Strategy

unit testunit test integrationintegrationtesttest

ValidationValidation(acceptance)(acceptance)

testtest

systemsystemtesttest

Defect testing

Elaborated from Pressman

In the large

In the small: component testing

Software Architectures and Integration Testing

• Software architectures provides at least the structure of a complex software; therefore, it is natural to perform testing on integrated code by following the architecture

Call-return Layered Object-oriented

What should be tested!• We have tested units according to the expected behavior and

(some forms) of unexpected behavior• This is largely related to functional requirements and to

discovery related defects in code units; testing as such is therefore related to the correctness of code and is performed on code with possible inputs (EB) or potential inputs (P), without attention to how inputs are provided and where the software is installed

• What about non-functional requirements? They are usually related to other quality attributes than correctness.

• We need to talk about a software system (not just the software but the software installed and running) for talking about the software running in its environment! The software system is then part of the whole system and it is therefore a subsystem of it (as many others).

Software System and Software

BB/WB

BB/WB

Operating systems, Middleware,Compilers, Interpreters, N° of installation of the same module, environment parameters

BB/WBBB/WB

software system

(sub-system)

Perfect technology assumption! Assumptions during the requirement engineering!

software system

system

system

software code

Separate concerns in testingTesting for correctness is not enough, e.g.:

– Overload (stress) testing (reliability)– Robustness testing (test unexpected situations)

(safety)– Performance testing (test response time)– …

• These tests are typically related to some software quality attributes and non-functional requirements and usually performed on the software system (or subsystems) not on single units

Testing Activities in the Software Process

Software RequirementsEngineering

SRSArchitecture Design

DetailedDesign

Coding

DesignModel

SRS (analysis model)

Moduledesign

Code

Unit test

Testedmodules

Integration Test

Integratedmodules

Software System Test

Testedsoftware system

Acceptance Test

UserManual

planned

planned

planned

The word “system” refers to the “whole system” and to the “software system”. The software system implicitly encompasses hardware and allocation of software on hardware.

SystemRequirements System Test

Levels of Testing

• Low-level testing– Unit (module) testing

– Integration testing

• High-level testing– System testing

– Acceptance testing

ProgrammerDevelopment team Independent Test Group

Independent Test GroupCustomer/End Users

Type of Testing Performed By

Integration Testing

Unit Testing

• Done on individual units (or units of modules)

• Test unit with white-box and black-box

• Requires stubs and drivers

• Unit testing of several units can be done in parallel

What are Stubs, Drivers ?

• Stub – dummy module which

simulates the functionality of a module called by a given module under test

• Driver– a module which transmits

test cases in the form of input arguments to the given module under test and either prints or interprets the results produced by it

A

B

C

Driver

B

Stub for C

e.g. to test B in isolation

e.g. module call hierarchy

Esempio

P(x,y)x := y + 2;if x > a then

a := a + 2;else

y := x + 3;a := F(x, y)

end if;x := x + a + y;

Come si fa a fare il test di integrazione?

Driver

Stub for F

If x<y then y=x+5

If x>y then y=x+7

x y

Read (x,y,a); call P(x,y); if x=…then write (test ok) else write (test not ok)

Px := y + 2;if x > a then

a := a + 2;else

y := x + 3;a := STUB_F(x, y)


Esempio

x := y + 2;if x > a then

a := a + 2;else

y := x + 3;a := F(x,y)


Come si fa a fare il test della seguente unità? Esecuzione simbolica, dirà

che prima della chiamata di F(x,y):

x=Y+2, a=A e y=Y+5 (con Y+2<=a)

Definizione dei test cases di P (per esempio Black Box):

Es. A=5, Y=1, X=7

Per cui deve essere noto l’output di

F(3,6)

Esempio


a := a + 2;else

y := x + 3;a := STUB_F(x, y)



Driver

Stub for F

x y

P

Identificare i test cases per P

Input Input Output

x y

2 5 9

3 6 19

Possibili Inputs per F

Indicare manualmente i possibili outputs

Esempio


a := a + 2;else

y := x + 3;a := F(x, y)



Driver

Stub for F

Write (x)

Read (y)

x y

Read (x,y,a); call P(x,y); if x=…then write (test ok) else write (test not ok)

Px := y + 2;if x > a then

a := a + 2;else

y := x + 3;a := STUB_F(x, y)


Integration Testing

• Test integrated modules (i.e. integrated code)

• Usually focuses on interfaces (i.e. calls and parameters passing) between modules (defects are in the way modules are called)

• Largely architecture-dependent

• Non-incremental ( Big-Bang integration )– tests each module independently (black and white box)

– combines all the modules to form the integrated code in one step, and test (usually black-box)

• Incremental– instead of testing each module in isolation, the next

module to be tested is combined with the set of modules that have already been tested

– With two possible approaches: Top-down, Bottom-up

Integration Test Approaches

Comparison

Non-Incremental

• requires more stubs,drivers

• module interfacing defects are detected late

• finding defects is difficult

Incremental

• requires less stubs, drivers• module interfacing defects

detected early• finding defects is easier• not all modules should be

implemented for starting test

• results in more thorough testing of modules

Top-down Integration• Begin with the top module in the module call

hierarchy (represented as a structure chart)• Stub modules are produced

– But stubs are often complicated• The next module to be tested is any module with

at least one previously tested superordinate (calling) module (depth first or breadth first ways)

• After a module has been tested, one of its stubs is replaced by an actual module (the next one to be tested) and its required stubs

Example of a Module Hierarchy

A

B C D

F HE

Top-down Integration Testing

A

Stub B Stub C Stub D

Example:

Top-down Integration Testing

A

B Stub C Stub D

Stub FStub E

Example:

Test cases written for A are reused

Test cases white and black box for B should be combined with test cases written for A

Bottom-Up Integration• Begin with the terminal (leaves) modules

(those that do not call other modules) of the modules call hierarchy

• A driver module is produced for every module• The next module to be tested is any module

whose subordinate modules (the modules it calls) have all been tested

• After a module has been tested, its driver is replaced by an actual module (the next one to be tested) and its driver

Example of a Module Hierarchy

A

B C D

F HE

Bottom-Up Integration Testing

Driver E

FE

Driver F

Example:

Bottom-Up Integration Testing

B

FE

Driver BExample:

Test cases white and black box for B

Comparison

Top-down Integration

• Advantage– a skeletal version of the

program exists early

• Disadvantage– required stubs could be

expensive

Bottom-up Integration

• Disadvantage– the program as a whole does

not exist until the last module is added

• Effective alternative -- use hybrid of bottom-up and top-down:

- prioritize the integration of modules based on risk- highest risk modules are integration tested earlier than modules with low risk

No clear winner

Regression Testing• Re-run of previous test cases to ensure that

software already tested has not regressed to earlier defects after making changes (or integration) to the software

• Regression testing can also be performed during the entire life a software

• Reusability of test cases is the key point!

Types of System and Acceptance Testing

(Sub)System Testing• Process of attempting to demonstrate that system (or

subsystem) does not meet its original requirements and objectives as stated in the requirements (specification) document i.e. it is a defect testing

• Usually, it is not only a code testing but a test on the software system

• Test cases derived from – “system” objectives, user scenarios, possibly during

system engineering or early requirement engineering– requirement document and software requirements

specification (analysis model)– expected quality stated during the design engineering– additional aspects related to deployment of code

Usual Types of Software System Testing

• Volume testing– to determine whether the system can handle the required volumes of

data, requests, etc.

• Load/Stress testing– to identify peak load conditions at which the system will fail to handle

required processing loads within required time spans

• Usability (human factors) testing– to identify discrepancies between the user interfaces of the system

(software) and the human engineering requirements of its potential users.

• Security Testing– to show that the system’s security requirements can be subverted


• Performance testing (also as code testing)– to determine whether the system meets its performance

requirements (eg. response times, throughput rates, etc.)

• Reliability/availability testing– to determine whether the system meets its reliability and

availability requirements (here availability is related to failure; however availability may only be related to “out of service” situations not necessarily related to failures)

• Recovery testing– to determine whether the system meets its requirements for

recovery after a failure


• Installability testing – to identify ways in which the installation procedures lead to incorrect

results

• Configuration testing– to determine whether the system operates properly when the

software or hardware is configured in a required manner

• Compatibility testing– to determine whether the compatibility (interoperability) objectives

of the system have been met

• Resource usage testing– to determine whether the system uses resources (memory, disk

space, etc.) at levels which exceed requirements

• Others

Alpha and Beta TestingAcceptance testing performed on the developed software before its released to the whole user community.

• Alpha testing– conducted at the developer site by End Users (who will use

the software once delivered)– tests conducted in a controlled environment

• Beta testing– conducted at one or more customer sites by the End Users– it is a “live” use of the delivered software in an environment

over which the developers has no control

Stop conditions for Defect Test

When to Stop Defect Testing ?• Defect testing is potentially a never ending activity!

• However, an “exit condition” should be defined, e.g.:

– Stop when the scheduled time for testing expires

– Stop when all the test cases execute without detecting failures

but both criteria are not good

Better Code Defect Testing Stop conditions

• Stop on use of specific test-case design techniques.

• Example: Test cases derived from– 1) satisfying multiple condition coverage and

– 2) boundary-value analysis and

– 3) ….

– ….

– all resultant test cases are eventually unsuccessful (i.e they do not lead to failures)

Better Code Defect Testing Stop condition 1

• Sia ND il numero dei difetti• Inserire nello unit un insieme di difetti NDI• Far eseguire il test (a qualcuno che non conosce i

difetti inseriti) attraverso un certo numero di tecniche• L’efficacia di tale test è quindi:

– NumeroDifettiInseritieScoperti/NumeroDifettiInseriti

(NDIS/NDI)

• Nell’ipotesi che i difetti siano simili si può dire che – (NDS/ND)=(NDIS/NDI) e quindi

ND=NDI*NDS/NDIS ove NDS è il NumeroDifettiScoperti

Better Code Defect Testing Stop condition 2

• Miglioramento con due gruppi indipendenti di test che trovano X e Y difetti, di cui Q sono comuni

• ND, numero totale dei difetti, è quindi pari a ND=X*Y/Q (poiché si ipotizza che X/ND=Q/Y)

System Testing Stop condition

• Stop in terms of failures (rate) to be found and therefore in term of time to be spent in testing

• This stop condition is closely related to the reliability of the software system

Test Automation

Steps in Test Cases definition and Execution

Design testcases

Prepare testdata

Run pro gramwith test data

Compare resultsto test cases

Testcases

Testdata

Testresults

Testreports

Test tools

Dynamic

analyser being testedTest results

Test

predictions

File

comparator

Execution

reportSimulator

Source

code

Test

managerTest case Oracle

Test case

gener atorSpecification

Repor t

generator

Test results

report

User interface

Unit

Debugging• Task for locating and correcting defects in software code

• It can start when a failure has been detected

• It usually performed during test

• Need sometime the definition of an intermediate concept, error i.e. a situation leading to the failure and due to the defect

• Requires closing up the gap between a fault and failure– watch points

– intermediate assertions

defect (fault) error failure

discovering discovering

What is seen from an external observer

What is recognized as non correct situation

The cause

Autodebugging, System management and Fault-tolerance

• Detect errors and alert on them, may stop or may not stop the execution (leading to failure)

• Detect errors and undertake a fault management strategy (recovery, alternatives etc.) that allows to tolerate the fault!

Performance, Reliability Testing

Quality Assessment for subjective quality attributes

Performance• Types of performance analysis (can also be applied to

code)– Worst case analysis

• focus is on proving that the (sub)system response time is bounded by some function of the external requests and parameters

– Average behavior

• Analytical vs. experimental approaches, an both may concern the (software) system:– Queue models, statistics and probability theory (Markov

chains) – Simulation– Others

Correctness review• Correcteness is an absolute feature of software with a binary

result (the software is correct, the software is not correct)• Typically, correcteness is expressed in term of functional

requirements or component specifications derived from functional requirements

• Less important for real systems where hypotheses (made during requirement engineering or as perfect technology assumptions) on which these systems are built are only true in probability or sometime false

• Correcteness can be reformulated as:– Reliability (probability to work without failures over a time

period)• Robustness (management of unexpected situations (i.e. failures

elsewhere))• Safety (probability that something does not happen)

Reliability (1)• There are approaches to measure reliability on a

probabilistic basis, as in other engineering fields, i.e. the probability the (software) system will work without failure over a period of time and under some conditions (shortly, probability to do not fail within a time frame)

• Unfortunately, there are some difficulties with this approach:– independence of failures does not hold for software

If x>0 then write(y) else (write(x); write(z);)X is wrongly assigned to 7 instead of 6; Z is wrongly assigned

X is wrongly assigned to 0 instead of 1; Z is wrongly assigned

Reliability (2)• Reliability is firstly concerned with measuring

the probability of the occurrence of failures• Meaningful parameters include:

• average total number of failures observed at time t: AF(t)

• failure intensity: FI(t)=AF'(t)• mean time to fail at time t: MTTF(t)=1/FI(t)• mean time between failures MTBF(t)=MTTF(t)+MTTR(t)

(MTTR corresponds to time needed after a failure, to repair)

• Time is the execution time but also the calendar time (because in part of the software system can be shared with other software systems)

Basic reliability model• Assumes that the decrement per failure

observed (i.e., the derivative with respect to the number of observed failures) of the failure intensity function is constant– i.e., FI is a function of AF

FI(AF) = FI0 (1 - AF/AF∞)

where FI0 is the initial failure intensity and AF∞ is the total number of failures

• The model is based on optimistic hypothesis that a decrease in failures is due to the fixing of defects, sources of those failures

AF

AF

t

Basic model

AF law

FI(t)FI(0)

Af(t)=Af *(1- exp(-t* ))

Uso del modello base

Af(ti)=Af *(1- exp(-ti* ))

Stima: Af e

Calcola il tempo t per cui Af(t)=Af(T) + 1 ove T è il tempo cui si è arrivati con il test e si sono osservati Af(T) failure (quindi Af–AF(T) indica il numero di failure ancora osservabili)

Fare il test per almeno affinché il sistema sia eseguito per almeno t-T in modo da osservare un ulteriore failure, se vi sono ancora difetti

Assessment of subjective (less factual) quality attributes

• Quality assessment on code of subjective quality attributes

• Consider quality attribute like maintainability, reusability, understandability …

• There is the need of metrics

Internal and external attributes of quality

Number of procedureparameters

Cyclomatic complexity

Program siz e in linesof code

Number of errormessa ges

Length of user manual

Maintainability

Usability

Portability

A metric evaluated by

Comprehensibility

Str

uctu

re

Software quality attributes (also called external attributes)

Internal quality attributes

McCabe's source code metric• Cyclomatic complexity C on the control

graph is C = e - n + 2p – Where e is # edges, n is # nodes, p is # connected

components

• McCabe contends that well-structured modules (i.e. high quality) have C in range 3 .. 7, and C = 10 is a reasonable upper limit for the cyclomatic complexity of a single module– confirmed by empirical evidence

Halstead's software science • Tries to measure some software qualities by

measuring some quantities on code, such as– n1, number of distinct operators in the program– n2, number of distinct operands in the program– N1, number of occurrences of operators in the

program– N2, number of occurrences of operands in the

program

N= n1 log2 n1 + n2 log2 n2 (length of the program)

V = (N1+N2) log2 (n1+n2) (volume of the program) --- error in Pressman, N instead of N1+N2

Halstead's software science• Other than software qualities, quantities on code

can be used to estimate interesting features of code

• Mental Effort (effort required to understand and further develop a program)

E = [(n1) (N2) (N1+N2) log2(n1+n2)] / 2(n2)

• Estimated Number of Defects

B= E(2/3) / 3000

Esempio

if ( A > 1) and ( B = 0 ) then

X = X / A;

if ( A = 2 ) or ( X > 1) then

X = X + 1;

n1 = 6 (inclusi operatori logici)

N1 = 8

n2 = 3

N2= 7

E=[(n1) (N2) (N1+N2) log2(n1+n2)] / 2(n2)E=[6 7 (8+7) log2(6+3)]/2 3= 333 (circa)

Conclusioni• Testing in generale (anche in relazione con la verifica e

la validazione e la più generale assicurazione di qualità, distinta in previsione della qualità e valutazione della qualità)

• Testing Componenti Convenzionali (Black – White box)

• Tecniche statiche (di Verifica) e Conventional Unit Code Testing (Inspection, Walkthrougth, Symbolic Execution, Correcteness Proof)

• Testing Componenti Object-Oriented• Testing in the large (Integration and System Testing)• Testing attributi soggettivi di qualità (diversi da

correttezza, affidabilità, robustezza, safety e prestazioni)

static techniques (on code) (static analysis of code) performed on the code without executing the...

Documents

defect slide

code slide

input2 slide

esempio slide

claim3 slide

loop invariant slide

input2 x

code static analysis