ai & software testing - ubingpombo/units/docs/sq/ubi_tech_talks.pdf · title: powerpoint...

CRITICALSOFTWARE.COM

AI &

Software

Testing

© Copyright Critical Software. All rights reserved. 1

▪ Testing Myths Debunked!

▪ Where’s Testing?

▪ 7 Testing Principles

▪ AI for Testing

▪ S*** got serious

▪ SW Testing for Airborne Systems

▪ SW Testing for Railway Systems

Contents

© Copyright Critical Software. All rights reserved. Information Classification: Confidential 2


Myth #1:

Testing is easy

(aka anyone can do testing)

Testing Myths Debunked!


Myth #2:

Testers don’t write code

(aka bad coders go to Testing)



Myth #3:

Testing is random

(aka is just button-clicking)



Where’s testing?V-Model Process


Where’s testing?Agile Process


7 Testing PrinciplesSoftware Testing Principles

Artificial Intelligence Applied to

Software Testing


• Reduction of the development cycle time

• Reduction of costs

• Decrease the human error factor

• Automate processes

Only applicable to critical systems in very specific contexts

AI

ML

DL

AI for Testing

Software Testing in

Airborne Systems


S*** got serious!

Software Testing in

Airborne Systems


DAL Effect Description

Level A Catastrophic “... result in multiple fatalities, usually with the

loss of the aeroplane.”

Level B Hazardous “... serious or fatal injury to a relatively small

number of occupants other than the flight crew:”

Level C Major “... physical distress to passengers or cabin

crew, possibly including injuries.”

Level D Minor “... slight increase in crew workload, such as

routine flight plan changes, or some physical

discomfort to passengers or cabin crew.”

Level E No Safety Effect “... not affect the operational capability of the

aeroplane or increase crew workload.”

DAL: Development Assurance Level

S*** got serious!


System

Requirements

Allocated to SW

High-level

Requirements

Low-level

Requirements

HW/SW Integration Tests

Low-level

Tests

System Tests

Software Integration Tests

• Coverage in DO-178 comes in multiple forms all of which

shall be satisfied:

Coverage type Applies to Target

Requirements HLR and LLR 100% (*)

Structure Source Code,

OBJ, EOC

100% (**)

Range Normal &

Robustness

Not fully defined (***)

▪ (*) LLR coverage not required for SW Level D

▪ (**) Type of coverage depends on SW Level – i.e. statement,

decision, MC/DC

▪ (***) There are no automatic ways to measure equivalent classes

in complex function. Discontinuities, singularities, normal range

and robustness should be tested.

Software Testing in

Airborne Systems

S*** got serious!


HLR_ID_12:The function shall determine the received

message data length in bytes by decoding

the message DLC field as follows:

0001b DLC = 1

0010b DLC = 2

0011b DLC = 3

0100b DLC = 4

0101b DLC = 5

0110b DLC = 6

0111b DLC = 7

1000b DLC = 8

LLR_ID_25:set data_length_bytes to msg_header

bitwise and DLC_MASK) right shift by

DLC_START_BIT

Where,

DLC_MASK = 0xF0

DLC_START_BIT = 4

Bit 7 6 5 4 3 2 1 0

0x0000 DLC

Software Testing in

Airborne Systems

S*** got serious!


Structural coverage:

• Statement Coverage

• Decision Coverage

• Modified Condition / Decision

Coverage

Instruction 1

IF (A or B) true

THEN

Instruction 2

END_IF

Instruction 3

Software Testing in

Airborne Systems

S*** got serious!

© Copyright Critical Software. All rights reserved.

15

A and B Decision

0 1 0

1 1 1

1 0 0

A Decision

0 0

1 1

A and (B or C) Decision

0 1 0 0

1 1 0 1

1 0 0 0

1 0 1 1

B=1 doesn’t affect the decision of the first 2 TCs

Test of B=0 needs a TC where decision=1 is changed by B

MC/DC for:

A and (B or C)→ 4 TCs

C=0 doesn’t affect the decision of the first 3 TCs

Test of C=1 needs a TC where decision=0 is changed by C

Software Testing in

Airborne Systems

S*** got serious!


• DO-178C requires data and control coupling coverage (level A to C).

• Performed during Integration Testing

• Data coupling:

• Dependence of a SW component on data not exclusively under its

control, e.g.:

• Shared data: used by the component and updated by another

• Output parameter: provided by a called component

• The “coupling” exists because a component may change its

behaviour depending on the value of the data (updated or provided)

Software Testing in

Airborne Systems

S*** got serious!


• Control coupling:

• Degree by which one SW component influences the execution of

another, e.g.:

• Call of a component: execution of the component depends

on the decision of another

• Input parameter: provided by a calling component

• The “coupling” exists because the execution of a component

depends on external parameters or influences.

Software Testing in

Airborne Systems

S*** got serious!

Software Testing in

Railway Systems


S*** got serious!


Software Testing in

Railway Systems

S*** got serious!

SILTolerable Functional

Failure Rate [h-1]

0 10-5 ≤ TFFR

1 10-6 ≤ TFFR < 10-5

2 10-7 ≤ TFFR < 10-6

3 10-8 ≤ TFFR < 10-7

4 10-9 ≤ TFFR < 10-8

Failure rate according with EN 50126:


Software Testing in

Railway Systems

S*** got serious!


Software Testing in

Railway Systems

S*** got serious!

Testing Methods

Structure-Based Testing

Test of Cause Consequence Diagrams

Boundary Value Test

Equivalence Classes and Input Partition Testing

Process simulation

Error Seeding

Error Guessing


GIVEN

A

WHEN

B is received as invalid for less than 1 sec

THEN

The Train shall apply to C the last valid value of B

Software Testing in

Railway Systems

S*** got serious!


GIVEN

A

WHEN


THEN


Software Testing in

Railway Systems

S*** got serious!


S*** got serious!

Component

Testing


System and

Integration Testing

S*** got serious!

Wrap-up

▪ Testing can occur in different levels of the SW lifecycle

▪ Testing does not make the code bug-free!

▪ AI is becoming popular but its applicability to critical systems is limited

▪ For critical systems:

▪ testing coverage level depends on the criticality level

▪ testing is performed in different levels


CRITICALSOFTWARE.COM

THANK

YOU


ai & software testing - ubingpombo/units/docs/sq/ubi_tech_talks.pdf · title: powerpoint...

Documents