2014 medconf unit testing · unit testing -popular misconceptions & road blocks it takes too...

Copyright 2014 – QA Systems 0Copyright 2014 – QA Systems GmbH

UNIT TESTING

The first stage of zero defect medical devices

Mr. Pierre-Henri STANEK

QA Systems GmbH - Cantata Product Manager

Copyright 2014 – QA Systems 1Copyright 2014 – QA Systems GmbH 1

INTELLIGENT NAVIGATION SYSTEM

UNIT TESTING – route map

Definition & Techniques

Popular Misconceptions

Device Testing Project Experience

IEC 62304 & FDA standards

Wrap Up and Q & A

Copyright 2014 – QA Systems Limited Test Test Slide 2

What is Unit Testing?

What is a Unit Test?

“…unit testing is a software development

method where the programmer gains

confidence that individual units of source

code are fit for use.” Source: Wikipedia

Unit verified as an executable

• Correct internal processing

• Correct use of interfaces / data

What is a Unit?

The smallest testable part of a program

(module, component, object, unit etc.)

For Procedural Code

• individual program, function or procedure

For Object Oriented Code

• A Class

(base/super/abstract/derived/child class)

Code

control

Inputs �Outputs

Without confidence in the units,

it is impossible to have confidence that the software as a whole will work…

Copyright 2014 – QA Systems 3

Imagine your (very simple!) system:

Unit AUnit A

Unit CUnit CUnit BUnit B Unit DUnit D

Unit FUnit FUnit EUnit E Unit GUnit G

Unit IUnit IUnit HUnit H Unit JUnit J

How would you test each unit and in what order?

Where do I start unit testing?


� The unit at the top is tested first

� Untested units are simulated to control the test

� Fully tested parents are used to call subunits

Unit CUnit CUnit BUnit B

Unit FUnit F Unit GUnit G


Unit DUnit D

Unit EUnit E

Unit AUnit A

Simulated Units

Simulated Units

Simulated Units

Top-Down Unit Testing


� Early integration of 'visible'

function

� Detailed design is also top

down

– Units can be coded & tested

as soon as designed (don’t

need to wait for lower

modules…)

� Produces a sequential test

plan

� Difficult to test lower units

from the top unit

� High impact of changes

Top-Down Unit Testing: Pros and Cons

Unit AUnit A




Re-testing

Required


� The lowest level units are tested first

� Tests are controlled by test drivers

� Parent units are tested using fully tested subunits




Unit DUnit D

DriverDriver

Unit AUnit A

Unit EUnit E

DriverDriver

DriverDriver

Bottom-Up Unit Testing


� Provides early integration

of units

– Test starts with simplest units

(complexity management…)

� No simulation required

– Use of real units controlled

by a test driver

� The first units to be tested are

often last units to be designed

� Hard to test higher units fully

– e.g. Error handling code (easily

met with a stub)

� Mainly sequential test plan

� High impact of changes

– All higher units need retesting

Bottom-Up Unit Testing: Pros and Cons


� Units are tested in isolation from all other units

� Test drivers are needed to call units under test; other units are

simulated.




Unit DUnit D

Unit EUnit E

DriverDriver

Simulated Units

Simulated Units

Unit AUnit A

DriverDriver

Isolation Unit Testing


� Easier to test a unit

thoroughly

– Simple drivers and

simulated units

– No dependencies between

unit tests

� Produces parallel test plan

� Minimal impact of

changes

� Does not provide an early

integration of units

– The units aren’t working

together……

� Requires both test drivers

and simulated units

– May require more effort to

develop initially

Isolation Unit Testing: Pros and Cons


� Changes to a unit do not affect any other unit

– Unless the interface has changed

Unit AUnit A




� Reduces maintenance and overall test lifecycle costs

Isolation Unit Testing: Minimal impact of changes


� Isolation unit testing is usually the best choice

– Shortest overall timescale and so lowest overall cost

– Easiest to maintain

� However, consider

– Bottom-up with low-level utility functions

– Top-down for user interfaces

– Or a mix! Can even vary within a development…

Unit Testing: Which technique do we recommend?


Unit Testing - Popular Misconceptions & Road Blocks

� It takes too much time and cost too much

� It is slower than de-bugging the code

� It only proves the code does what the code does

� Integration tests will catch all the bugs anyway

� My legacy software has no unit tests so I can’t unit test

� Maintaining unit tests is pointless

� Testing is a job for testers / QA

� The client pays to develop code, not write unit tests

� Our competition don’t unit test, it would make us uncompetitive

Developers prefer developing to testing

Sometimes popular misconceptions

are used as objections or road blocks….


Popular misconceptions about unit testing

� It takes too much time and cost too much

– Wrong: The earlier in the development cycle that bugs can be identified

and removed the cheaper it will be over the entire application

development…

� It is slower than de-bugging the code

– Wrong: de-bugging is a manually intensive activity not designed to be

consistently repeated whereas re-execution of functional unit tests is

highly if not completely automatic

� It only proves the code does what it does is

– Right and wrong: If you derive the tests only from the code then, yes;

but if tests are derived from requirements or model, tests are of much

greater benefit



� Integration tests will catch all the bugs anyway

– Wrong: Once software is running, developers are faced with the

problem of testing the software through its interfaces. It can be difficult

to create scenarios where all units are checked

� My legacy SW has no unit tests so I can’t unit test

– Wrong: would be the case if one has to reverse unit tests from the code,

but modern test tools can automatically generate unit testing safety net

based on the source code analysis…

� Maintaining unit tests is pointless

– Wrong: Having a working set of unit tests provides a powerful tool to

pin-point individual previously working units now failing tests and to

identify unwanted side-effects



� Testing is a job for testers / QA

– Right and Wrong: Separate testers or QA function are not best placed

for code level testing; developers can take structured quality control

responsibility for the code they produce

� Client pays to develop code, not write unit tests

– Wrong: the client pays you to develop working code and a good way to

achieve that is to use unit tests…

� Our competition don’t unit test, so it would make us

uncompetitive…

– Wrong: Faulty products will have to be fixed, which could work out

very expensive or damaging for the brand in a safety context, and

negate any profit made from quicker delivery


Unit testing is mandated or recommended by safety standards

[US FDA 510k]

Safety in

the loop


What is required by safety-related medical standards: IEC 62304

� IEC 62304:2006 is a harmonised standard for medical devices

that contain software, accessories to medical devices adopted

by the EU and the US

� §5.5.2 Establish Software Unit Verification Process

� The MANUFACTURER shall establish strategies, methods and procedures

for verifying each software unit

� §5.5.3 Software Unit Acceptance Criteria

� The MANUFACTURER shall establish acceptance criteria for SOFTWARE

UNITS prior to integration to larger software items […] Examples of

acceptance criteria are: does the software code implement requirements

including RISK CONTROL measures?



� §5.5.4 Additional SOFTWARE UNIT acceptance criteria

a) proper event sequence;

b) data and control flow;

c) planned resource allocation;

d) fault handling (error definition, isolation, and recovery);

e) initialisation of variables;

f) self-diagnostics;

g) memory management and memory overflows; and

h) boundary conditions.

� §5.5.5 Software Unit Verification

� The MANUFACTURER shall perform SOFTWARE UNIT VERIFICATION and

document the results



� §5.6.3 Test Integrated Software

� Software integration testing can be performed in a simulated environment,

on actual target hardware, or on the full medical device

� §5.6.6 Conduct Regression Tests

� When software items are integrated, the MANUFACTURER shall conduct

REGRESSION TESTING appropriate to demonstrate that defects have not

been introduced into previously integrated SW


What is required by safety-related medical standards: US FDA 510K

� The FDA guidance pre-date 2006 issue of IEC 62304

� §5.2.5 Testing by the software developer

� Software testing entails running software products under known

conditions with defined inputs and documented outcomes that can be

compared to their predefined expectations […] SW testing involve:

• Normal Case, Output Forcing, Robustness, Combination of Inputs

� Another aspect of software testing is the testing of software changes

• Once a SW product has been baselined, any change to that product should have its own

“mini life cycle,” including testing

� The location of testing is dependent upon the software developer's ability

to produce the target operating environment(s)


Projects experiences of enforcing medical device standards


� Most of GE Healthcare’s products are classified at the highest

level defined in the FDA guidance for SW running in medical

devices (Major Level of Concern).

⇒ The software platform underwent a risk assessment that identified

approximately 10% of the SW implementing safety-critical functions

⇒ Full functional black-box testing of all of the C/C++ code at sub-system

level was therefore required

⇒ Patient monitor system is based on a PowerPC target running the Nucleus

Plus RTOS cross-compiled using the Diab Data compiler

GE Healthcare


� At the end of the project, Mr. Jyrske comments “unit testing

has had a positive impact on the way we write the code,

reducing coupling between objects and making isolation

testing easier.”

� Another positive effect is around the stabilization of SW

interface definitions expressed in the functional requirements

specification (from which test cases for the safety-critical

classes were derived). Mr. Repo explains this became a

problem when outside companies were given batches of code

to unit test: “Design of interfaces was fluid, and

development changes were so rapid that component tests

became quickly out of date. We were a little bit surprised at

how much of a problem this was.”

GE Healthcare – Lessons Learned


� Hamilton Medical’s state-of-the-art ventilator product called

Galileo has proven its effectiveness with hard-to-ventilate

patients.

⇒ Adaptive Support Ventilation (ASV), invented by Hamilton Medical,

guarantees that the patient receives set minute ventilation, automatic fine-

tuning settings to the patient’s needs. Central to Galileo’s success is the

integrated software used to control its systems.

⇒ The software used to run Galileo needed to be tested to the highest

standards, including systematic module unit testing.

Hamilton Medical


� Unit Testing has been conducted in a company-standard way.

Mr. Keller stated “Once the first test had been run, we

developed a framework that allowed developers to use

templates. Soon all developers were testing to exactly the

same standard. Maintaining each other’s test scripts became

much easier as a result of this standardization”

� For the new generation of ventilators, Hamilton Medical

upgraded their proprietary solution to the Tornado/VxWorks

environment from Wind River Systems. Mr. Durish stated “We

ported our existing tests, and used a professional tool to

verify that our software still behaved as expected. In doing

so, we found several bugs in third party libraries.”

Hamilton Medical – Lessons Learned

Copyright 2014 – QA Systems GmbH COMMERCIAL IN CONFIDENCE Test 26

Tool support for certification

Tool Certification

• 3rd Party Certified to Class C / III devices :

• IEC 62304

• Certification kit free of charge.

Supporting Standards Compliance

• Comprehensive Standards Briefings & Safety Manual.

Device certification is critical for safety related industries.

Cantata eases the path to certification of device software, with

standards compliance and tool certification built-in.

Compliance & Automation

� APPROVED

Copyright 2014 – QA Systems 27Copyright 2014 – QA Systems GmbH 27

QA Systems Ltd

2 Palace Yard Mews

Bath BA1 2NH

United Kingdom

Phone: +44 (0)1225 321 888

Fax: +44 (0)1225 581 150

[email protected]

QA Systems GmbH

Schwieberdinger Str. 56

70435 Stuttgart

Germany

Phone: +49 (0)711 138183-0

Fax: +49 (0)711 138183-10

[email protected]

Contact details / Time for Q&A

2014 medconf unit testing · unit testing -popular misconceptions & road blocks it takes too...

Documents