testing complex safety-related systems - t&vs · pdf filetesting complex safety-related...

28
Mike Bartley TVS, Founder and CEO Testing Complex Safety-Related Systems

Upload: vuongtuyen

Post on 12-Feb-2018

226 views

Category:

Documents


3 download

TRANSCRIPT

Mike Bartley

TVS, Founder and CEO

Testing Complex

Safety-Related

Systems

Copyright TVS Limited | Private & Confidential | Page 2

Agenda

Some background on your speaker and testing safety related systems

Can directed testing scale with complex systems?

Ensuring new techniques fit within safety standards

Copyright TVS Limited | Private & Confidential | Page 3

Your speaker: Mike Bartley

PhD in Mathematical Logic

MSc in Software Engineering

MBA

Worked in software testing and hardware verification for over 25 years • Praxis, IPL, ST-Micro, Infineon, Panasonic, ARM, NXP, nVidia,

ClearSpeed, Gnodal, DisplayLink, Dialog, …

• Worked in formal verification of both software and hardware

Started TVS in 2008 • Software testing and hardware verification products and services

• Offices in India, UK, France and Germany

Copyright TVS Limited | Private & Confidential | Page 4

TVS - Global Leaders in Test and Verification

India - 2011

UK - 2008

Germany - 2011

France - 2012

Singapore - 2014

China

South Korea

Continuous

geographical

expansion…

USA - 2014

0

50

100

150

200

Q3-13 Q4-13 Q1-14 Q2-14 Q3-14 Q4-14 Q1-15 Q2-15 Q3-15 Q4-15 Q1-16 Q2-16(Est.)

Q3-16(Est.)

Q4-16(Est.)

EM

PL

OY

EE

S

CALENDAR YEAR

Number of Employees by quarter

Japan 2015

Copyright TVS Limited | Private & Confidential | Page 5

Safety Standards

IEC61508: Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems

DO178C: Software considerations in airborne systems and equipment certification

EN50128: Software for railway control and protection systems

IEC60880: Software aspects for computer-based systems performing category A functions

IEC62304: Medical device software -- Software life cycle processes

ISO26262: Road vehicles – Functional safety

Copyright TVS Limited | Private & Confidential | Page 7

Safety-Critical: Verification and Testing

Depend on integrity level/class Software Verification Plan Reviews of Specifications and Code Testing (against specifications)

• Unit • Software Integration • Software System

Test Coverage Criteria Requirements and Test Traceability Independence Audit trails

Copyright TVS Limited | Private & Confidential | Page 8

Sample Differences in Safety Integrity Levels

Dynamic analysis and testing

Technique SIL 1 SIL 2 SIL 3 SIL 4

Structural test coverage (entry points) 100% HR HR HR HR

Structural test coverage (statements) 100% R HR HR HR

Structural test coverage (branches) 100% R R HR HR

Structural test coverage (conditions, MC/DC) 100% R R R HR

Test case execution from boundary value analysis R

HR HR HR

Test case execution from error guessing

R R R R

Test case execution from error seeding - R R R

Test case execution from model-based test case

generation

R R HR HR

Performance modelling R R R HR

Equivalence classes and input partition testing R R R HR

Copyright TVS Limited | Private & Confidential | Page 9

Software

Requirements

All the Right Stages but Not Necessarily

in the Right Order

High Level

Design

Unit Level

Design

Coding

Unit Testing

Integration Testing

Software System

Testing

Copyright TVS Limited | Private & Confidential | Page 10

Shift-Left “Sequential” Development Flow

Product Reqs

System Spec

Unit Build

Unit Spec Unit Verif

Spec Unit Verif

Integration

Verif Spec

System

Verif Spec

Acceptance

Verif Requirements

Verif Spec

Integration

Verif

System Verif

Static Analysis

Copyright TVS Limited | Private & Confidential | Page 13

The Unit Test Foundation

Integration of Pre-tested

Components

Copyright TVS Limited | Private & Confidential | Page 14

How do we do the system level testing?

Robotic vacuum cleaner?

Mars rover?

Drone landing on ship in rough seas?

Automotive • Automated parking?

• Lane keeping assistance?

• Driverless cars

Do we continue to perform directed testing? • Hardware verification faced the same issue 20 years ago

• Hardware adopted constrained random verification

• And ensure requirements tracing for safety standards compliance

Copyright TVS Limited | Private & Confidential | Page 15

The mechanics of an advanced test bench

Driver Stimulus generator

Test Checker Functional

Coverage

Design

Under

Test

Monitor

assert

constraint

addr data Assertions

Coverage

Coverage

Active Passive Code Coverage

Copyright TVS Limited | Private & Confidential | Page 16

Some hardware verification examples

CPU Verification

Instruction

Stream

Generator assembler

CPU RTL

CPU

C Model

Compare

Accuracy?

Copyright TVS Limited | Private & Confidential | Page 17

Some hardware verification examples

USB Verification

Packet

Generator

Score-

board

Driver DUT Response

Assertions

Coverage

Assertions

Coverage

Assertions

Coverage

Copyright TVS Limited | Private & Confidential | Page 18

Bubble Sort “Proof of Concept” for SW Testing

• Program Specification

– Input lists of integers, floats, ascii, etc.

– Reject lists of mixed types

– Convert unsorted lists to sorted lists

• Can we test the program with constrained

input generation?

– Generate valid and invalid inputs

– Direct generation towards corner cases

– Check outputs for correctness

• Without re-writing an identical checker program

– Measure what we have tested

Copyright TVS Limited | Private & Confidential | Page 19

Results of Bubble Sort “Proof of Concept”

Software

Under Test Lists

Checkers

List

Generator

Coverage

Metrics

Lists of

• Integers

• Floats

• Ascii

• etc

Constrain towards

• Empty lists

• Equal values

• Reverse ordering

• Check output list is ordered

• Output list contents == input list contents

• Empty List

• Reverse ordered

• Error cases (mix integers, floats, ascii)

• Etc.

Copyright TVS Limited | Private & Confidential | Page 20

Virtual System Level Test Environment

Software

Under Test

Sensor

Inputs Actuator

Outputs

Checkers

Event

Stream

Generator

Metrics

Logfiles

Copyright TVS Limited | Private & Confidential | Page 21

Virtual System Level Checkers

Assert “never do anything wrong” • Always fail safe

Assert “always respond correctly” • If A&B&C occur then check X happens

• Assertion coverage “check A&B&C occurs” for free

Analyse log files • Look for anomalies

• Did the actuator outputs occur in the correct order

Copyright TVS Limited | Private & Confidential | Page 23

Functional Coverage

Requirements coverage “Cross-product” coverage

Situation coverage [R Alexander et al. Situation

coverage – a coverage criterion

for testing autonomous robots.

University of York, 2015] 23

[O Lachish, E Marcus, S Ur and A Ziv. Hole Analysis for Functional Coverage Data. Design Automation Conference (DAC), June 10-14, 2002, New Orleans, Louisiana, USA.]

A cross-product coverage model is composed of the following parts:

1. A semantic description of the model (story)

2. A list of the attributes mentioned in the story

3. A set of all the possible values for each attribute (the attribute value domains)

4. A list of restrictions on the legal combinations in the cross-product of attribute values

A functional coverage space is defined as the Cartesian product

over the attribute value domains.

From Kerstin Eder of the University of Bristol

Copyright TVS Limited | Private & Confidential | Page 24

Safety compliance (asureSIGN)

Managing Requirements • Importing and editing requirements

Decomposing requirements to verification goals

Tracking test execution • Automating import of test results

• Automate accumulation and aggregation of test results

Impact analysis • Managing changes in requirements and tests

Demonstrating compliance to DO254 & DO178C

Managing multiple projects

Copyright TVS Limited | Private & Confidential | Page 25

asureSIGNTM at the heart of HW/SW V&V

Requirements - Excel - Doors - Jira - etc

Hardware Simulation • Coverage Cadence • Assertions Mentor, Aldec • Etc.

Directed test results

asureSIGNTM

Matlab

Formal Verification • OneSpin

UCIS API

Run API

Automated SW Test Tool

SW Test Tools

Manual API

Lab Results

Requirements Engineering tools

SystemC Simulation

XML API

Copyright TVS Limited | Private & Confidential | Page 26

Decomposing requirements to features and tests

Map

requirements to

verification goals

The mapped

verification goal

Sign off a requirement with a

manual test (e.g. in the lab)

Import

Requirements

(Doors, Excel,

Word, etc)

Edit

Requirements

Copyright TVS Limited | Private & Confidential | Page 27

Safety compliance (asureSIGN)

Managing Requirements • Importing and editing requirements

Decomposing requirements to verification goals

Tracking test execution • Automating import of test results

• Automate accumulation and aggregation of test results

Impact analysis • Managing changes in requirements and tests

Demonstrating compliance to DO254 & DO178C

Managing multiple projects

Copyright TVS Limited | Private & Confidential | Page 28

Tracking test execution:

Automating import of test results

asureSIGN

Hardware

Verification

Results

UCIS

Software

Test Tools

XML

API

Manual

Entry

Manual

Testing

Accellera

standard

Hardware

Simulation

Formal

Verification

Copyright TVS Limited | Private & Confidential | Page 29

Automate accumulation and aggregation of test results

Accumulate results over

multiple regressions

Record results

from each test

Aggregate results

through the hierarchy

Define and track against interim

milestones (based on % of

requirements tested)

Copyright TVS Limited | Private & Confidential | Page 30

Safety compliance (asureSIGN)

Managing Requirements • Importing and editing requirements

Decomposing requirements to verification goals

Tracking test execution • Automating import of test results

• Automate accumulation and aggregation of test results

Impact analysis • Managing changes in requirements and tests

Demonstrating safety compliance – for example • DO254/178C, ISO26262, IEC 60601, IEC 61508, EN 50128, IEC 61513

Managing multiple projects

Copyright TVS Limited | Private & Confidential | Page 31

Demonstrating compliance to DO254 & DO178C

• Export XML for import back into Doors, etc.

• Export PDF report for audit

Select level of detail

Export Metadata such as

• Tool version numbers

• Configuration data

• Data owners

Pid = unique

reference to

requirement in

external tool

Copyright TVS Limited | Private & Confidential | Page 32

Summary

Current system testing techniques will not scale with new complex systems

Lessons from hardware verification • Constrained random inputs

• Automated checking

• Functional coverage

Ensuring safety compliance • Requirements driven testing

Questions