IEC Certification Kit

Simulink® Test™ ISO 26262 Tool Qualification Package

R2015b

How to Contact MathWorks

Latest news: www.mathworks.com

Sales and services: www.mathworks.com/sales_and_services

User community: www.mathworks.com/matlabcentral

Technical support: www.mathworks.com/support/contact_us

Phone: 508-647-7000 (Phone)

The MathWorks, Inc.

3 Apple Hill Drive

Natick, MA 01760-2098

For contact information about worldwide offices, see the MathWorks Web site.

IEC Certification Kit: Simulink® Test™ ISO 26262 Tool Qualification Package

© COPYRIGHT 2015 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used or copied only under

the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written

consent from The MathWorks, Inc.

FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the

federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees

that this software or documentation qualifies as commercial computer software or commercial computer software documentation

as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and

conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,

reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or

other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions.

If this License fails to meet the government’s needs or is inconsistent in any respect with federal procurement law, the

government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.

Trademarks

MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a list of

additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders.

Patents

MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more information.

Revision History

September 2015 New for IEC Certification Kit Version 3.6 (Applies to Release 2015b)

v

Contents

1 Introduction ...................................................................................................................................... 1-1 1.1 Application Identification ........................................................................................................ 1-2 1.2 Tool Overview and Identification ........................................................................................... 1-3 1.3 Tool Qualification Artifacts Summary .................................................................................... 1-4

2 Software Tool Criteria Evaluation Report ........................................................................................ 2-1 2.1 Tool Environment ................................................................................................................... 2-2 2.2 Tool Configuration .................................................................................................................. 2-3 2.3 Reference Workflow ............................................................................................................... 2-4 2.4 Tool Use Cases ........................................................................................................................ 2-5

[SLTEST_UC1] Development and execution of tests for Simulink models ................................ 2-5 [SLTEST_UC2] Development and execution of tests for back-to-back testing between model and

code .............................................................................................................................................. 2-5 [SLTEST_UC3] Assessment of test results ................................................................................. 2-5 [SLTEST_UC4] Generation of test reports .................................................................................. 2-5 [SLTEST_UC5] Identification of traceability between requirements and tests cases .................. 2-6

2.5 Generic Tool Classification ..................................................................................................... 2-7 2.5.1 Potential Malfunctions and Erroneous Output ............................................................... 2-7 [SLTEST_E1] Incorrect behavior of test harness ........................................................................ 2-7 [SLTEST_E2] Incorrect run of test procedure ............................................................................. 2-7 [SLTEST_E3] Erroneous assessment of test results – false negative .......................................... 2-7 [SLTEST_E4] Erroneous assessment of test results – false positive ........................................... 2-7 [SLTEST_E5] Generation of erroneous test report ...................................................................... 2-7 [SLTEST_E6] Usage of incorrect input data ............................................................................... 2-7 [SLTEST_E7] Incorrect Tool Usage ............................................................................................ 2-8 [SLTEST_E8] Incorrect or Modified or Incompatible with Environment Tool Installation ........ 2-8 2.5.2 Error prevention and Detection Measures ..................................................................... 2-8 [SLTEST_M1] Requirements-based testing ................................................................................ 2-8 [SLTEST_M2] Tool installation integrity checks ........................................................................ 2-8 [SLTEST_M3] Configuration management ................................................................................. 2-8 [SLTEST_M4] Input data integrity checks .................................................................................. 2-8 [SLTEST_M5] Competency of project team ............................................................................... 2-8 [SLTEST_M7] Manual comparison of test results to expected results ........................................ 2-9 [SLTEST_M8] Manual review of test report content .................................................................. 2-9

2.6 Tool Classification Summary ................................................................................................ 2-10 3 Software Tool Qualification Report ................................................................................................. 3-1

3.1 Requirement for Tool Qualification ........................................................................................ 3-2 3.2 Tool Qualification Documentation .......................................................................................... 3-3

4 Confirmation Review of Tool Classification and Qualification ....................................................... 4-1 4.1 Requirement for Confirmation Review ................................................................................... 4-2 4.2 Validity of Generic Tool Classification .................................................................................. 4-3

vi

4.3 Validity of Generic Tool Qualification ................................................................................... 4-4 4.4 Conformance with Reference Workflow ................................................................................. 4-5

vii

1 Introduction

This document constitutes the ISO 26262 Tool Qualification Package for the Simulink® Test™

product. This document is intended for use in the ISO 26262 tool classification and qualification

process for software tools. It contains templates for the ISO 26262 tool qualification work

products (see ISO 26262-8, Clause 11).

The applicant shall review this template for applicability to the application under consideration,

and tailor and complete the information.

See also:

IEC Certification Kit: User’s Guide, R2015b

ISO 26262-8, Clause 11

ISO 26262-8, Clause 11 provides provisions for software tools that are used to tailor activities or

tasks required by ISO 26262. The standard outlines a two-step approach to establish the

required confidence in the tools:

Tool classification determines the required level of confidence in the software tool.

Depending on the result of the tool classification, you might need to carry out a formal

tool qualification.

The following work products need to be created when applying this approach to a software tool

(see ISO 26262-8, 11.5):

A software tool criteria evaluation report documenting the tool classification.

A software tool qualification report documenting the tool qualification, if required.

Note The applicant needs to review this template for applicability to the project under

consideration and insert missing information.

1-2

1.1 Application Identification

Applicant: <Insert information>

Application under consideration: <List application under consideration>

1-3

1.2 Tool Overview and Identification Simulink Test is a tool for authoring, managing, and executing systematic, simulation-based

tests of the Simulink models. You can create nonintrusive test harnesses to test models and

subsystems. Simulink Test includes a test sequence block that lets you construct complex test

sequences and assessments, and a test manager that lets you manage and execute tests. It enables

functional, baseline, equivalence, and back-to-back testing, including software-in-the-loop (SIL)

and processor-in-the-loop (PIL). You can generate reports, archive and review test results, rerun

failed tests, and debug the component or system under test.

Software Tool Version (Release) Tool Vendor

Simulink® Test™ Version 1.1 (R2015b) The MathWorks, Inc.

3 Apple Hill Drive

Natick, MA, 01760-2098

USA

IEC Certification Kit Version 3.6 (R2015b)

1-4

1.3 Tool Qualification Artifacts Summary The following table lists:

Prerequisites (see ISO 26262-8, 11.3.1)

Supporting information (see ISO 26262-8, 11.3.2)

Tool qualification work products (see ISO 26262-8, 11.5)

for the Simulink Test product. The table also maps these tool qualification artifacts to sections in

this document and artifacts found elsewhere.

Tool Certification Artifact Corresponding Documents / Artifacts

Safety plan <Insert document title, version, and filename / link>

Applicable prerequisites of the

lifecycle phases where software

tool is used

<Insert software lifecycle phase(s)>

<Insert prerequisite(s)>

Predetermined maximum ASIL <Insert ASIL>

Software tool documentation Simulink Test Getting Started Guide

R2015b sltest_gs.pdf

Simulink Test User’s Guide

R2015b sltest_ug.pdf

Simulink Test Reference

R2015b sltest_ref.pdf

Simulink Test Release Notes

R2015b rn.pdf

Environment and constraints of

the software tool MathWorks® bug report system at

www.mathworks.com/support/bugreports/

<Insert information>

1-5

Tool Certification Artifact Corresponding Documents / Artifacts

Software tool criteria evaluation

report Customized and completed section “Software Tool Criteria Evaluation

Report” in the Simulink Test ISO 26262 Tool Qualification Package

(this document) certkitiec_sltest_tqp.docx

Simulink Test Reference Workflow

R2015b certkitiec_sltest_workflow.pdf

Certificate Z10 15 06 67052 016

June 2015 certkitiec_sltest_certificate.pdf

Report to the Certificate Z10 15 06 67052 016

June 2015 certkitiec_sltest_certreport.pdf

Software tool qualification

report Customized and completed “Software Tool Qualification Report” in the

Simulink Test ISO 26262 Tool Qualification Package (this document) certkitiec_sltest_tqp.docx

Customized and completed Simulink Test Conformance Demonstration

Template certkitiec_sltest_cdt.docx

Certificate Z10 15 06 67052 016

June 2015 certkitiec_sltest_certificate.pdf

Report to the Certificate Z10 15 06 67052 016

June 2015 certkitiec_sltest_certreport.pdf

Confirmation review of

qualification of a software tool

Customized and completed “Confirmation Review of Tool Classification

and Qualification” in the Simulink Test ISO 26262 Tool Qualification

Package (this document)

certkitiec_sltest_tqp.docx

1-6

2 Software Tool Criteria Evaluation Report

2-2

2.1 Tool Environment It is assumed that Simulink Test will be used in the following environment (see ISO 26262-8,

11.4.4.1d):

<Insert operating system and other pertinent environment information>

2-3

2.2 Tool Configuration It is assumed that Simulink Test will be used in the following tool configuration (see ISO

26262-8, 11.4.4.1b):

Configuration Parameter Setting

Test Result Report Pane

Include MATLAB version Yes

“Include in Report” controls

<Insert relevant configuration parameter names>

<Insert application-specific setting>

File Format <Insert application-specific setting>

2-4

2.3 Reference Workflow It is assumed that Simulink Test will be used as described in the reference workflow

documented in Simulink Test Reference Workflow. To access the reference workflow document,

on the MATLAB command line, type certkitiec. The reference workflow document is in

Simulink Test.

Simulink test features integrated into the generic Model-Based Design workflow are shown in

Figure 1.

Figure 1: Simulink Test Workflow Overview

2-5

2.4 Tool Use Cases It is assumed that Simulink Test will be used as described by the following use cases (see ISO

26262-8, 11.4.4.1c). Additional information about the assumed usage of Simulink Test can be

found in the Simulink Test Reference Workflow document and Simulink Test User’s Guide

document.

[SLTEST_UC1] Development and execution of tests for Simulink models

Simulink Test is used to create and execute tests for Simulink models. Testing of Simulink

models can be leveraged to implement the following verification and testing methods:

Simulation of dynamic parts of the software architectural design including mechanisms

for error detection and handling at the architecture level (ISO 26262-6 Table 6 method

1c).

Verification of software unit design (ISO 26262-6 Table 9 method 1c).

Implementation model testing (ISO 26262-6 Table 10 methods 1a – 1c).

[SLTEST_UC2] Development and execution of tests for back-to-back testing between model and code

Simulink Test is used to create and execute tests for back-to-back testing between model and

code using equivalence test capability (ISO 26262-6 Table 10 method 1e and Table 13 method

1e).

[SLTEST_UC3] Assessment of test results

Simulink Test is used to evaluate test results comparing them with expected results. Applicable

for all testing activities identified in the use cases SLTEST_UC1 and SLTEST_UC2.

[SLTEST_UC4] Generation of test reports

Simulink Test is used to generate test reports. Applicable for all testing activities identified in

the use cases SLTEST_UC1 and SLTEST_UC2.

2-6

[SLTEST_UC5] Identification of traceability between requirements and tests cases

Simulink Test is used to establish bidirectional links between textual requirements and test cases

(ISO 26262-6 Tables 11 and 14 methods 1a and 1b).

2-7

2.5 Generic Tool Classification The tool classification for Simulink Test was performed in a generic manner, independently

from the development of a particular safety-related item or element.

For the generic tool classification, the reference use cases listed in the section Tool Use Cases

have been taken into account.

2.5.1 Potential Malfunctions and Erroneous Output

[SLTEST_E1] Incorrect behavior of test harness

Test harness developed using Simulink Test produces erroneous input test stimulus or corrupt

simulation outputs of model under test.

[SLTEST_E2] Incorrect run of test procedure

Test procedure developed using Simulink Test run erroneously, e.g. invoke simulation of wrong

model or skips test cases.

[SLTEST_E3] Erroneous assessment of test results – false negative

Comparison of model simulation results to expected results incorrectly marks test case as FAIL.

[SLTEST_E4] Erroneous assessment of test results – false positive

Comparison of model simulation results to expected results incorrectly marks test case as PASS.

[SLTEST_E5] Generation of erroneous test report

Simulink Test produces erroneous test report which does not correspond to the actual test

results.

[SLTEST_E6] Usage of incorrect input data

Incorrect input data is used, resulting in tool malfunction and erroneous output.

2-8

[SLTEST_E7] Incorrect Tool Usage

User does not follow established procedures when using the tool, resulting in tool malfunction

and erroneous output.

[SLTEST_E8] Incorrect or Modified or Incompatible with Environment Tool Installation

User does not follow established procedures when installing the tool, installs the tool in an

incorrect operational environment, or modifies a valid installation. This might result in tool

malfunction and erroneous output.

2.5.2 Error prevention and Detection Measures

To mitigate potential malfunctions and corresponding erroneous outputs of the Simulink Test

product, the following measures are provided. Additional considerations are described in

Simulink Test Reference Workflow.

[SLTEST_M1] Requirements-based testing

The test cases and expected results are derived from requirements independent of the model

under test and the test environment. The independence provides a high degree of confidence that

errors will be detected using the actual results from the model under test in the test environment.

[SLTEST_M2] Tool installation integrity checks

Integrity of tool installation can be insured by re-running the validation test suite provided with

Simulink Test in the IEC Certification Kit.

[SLTEST_M3] Configuration management

Configuration of the life cycle data shall be managed by applicant in accordance with Clause 7

of ISO 26262.

[SLTEST_M4] Input data integrity checks

Simulink Test verifies the integrity of input files using checksum.

[SLTEST_M5] Competency of project team

Training of users can be performed to ensure correct usage of tool.

2-9

[SLTEST_M6] Analysis of Available Bug Report Information

Assess and analyze bug report information for Simulink Test and comply with the

recommendations and workarounds, if applicable.

[SLTEST_M7] Manual comparison of test results to expected results

Test results are manually compared to expected results to determine whether test passed or

failed.

[SLTEST_M8] Manual review of test report content

Test report content is manually reviewed to verify that it corresponds to the actual test results.

2-10

2.6 Tool Classification Summary

Potential malfunction or erroneous output

Use cases TI Justification for TI Prevention / detection measures

TD Justification for TD TCL

[SLTEST_E1]

Incorrect behaviour

of test harness

[SLTEST _UC1]

[SLTEST _UC2]

TI2 Incorrect behavior of test

harness could prevent

errors in an object under

test from being detected.

[SLTEST _M1]

Requirements-based

testing

TD1 The test cases and expected results are

derived from requirements

independent of the model under test

and the test environment. The

independence provides a high degree

of confidence that errors will be

detected using the actual results from

the model under test in the test

environment

TCL1

[SLTEST_E2]

Incorrect run of test

procedure

[SLTEST _UC1]

[SLTEST _UC2]

TI2 Incorrect run of test

procedure could prevent

errors in an object under

test from being detected.

[SLTEST _M1]

Requirements-based

testing

TD1 Requirements-based testing will

detect incorrect run of test procedure,

see TD justification for [SLTEST_E1]

TCL1

[SLTEST_E3]

Erroneous

assessment of test

results – passed test

indicated as failed

[SLTEST _UC3] TI1 Nuisance only, failed tests

have to be manually

reviewed and explained by

user

- - - TCL1

[SLTEST_E4]

Erroneous

assessment of test

results – failed test

indicated as passed

[SLTEST _UC3] TI2 Incorrect assessment of

test results could prevent

errors in an object under

test from being detected.

None TD3 - TCL3

[SLTEST _M7]

Manual comparison of

test results to expected

results

TD1 Manual comparison of test results to

expected results can verify that results

have been correctly assessed by the

tool.

TCL1

[SLTEST_E5]

Simulink Test

produces erroneous

test report which

doesn’t correspond

to the actual test

data

[SLTEST _UC4]

[SLTEST _UC5]

TI2 Invalid test report could

prevent errors in an object

under test from being

detected.

None TD3 - TCL3

[SLTEST _M8]

Manual review of test

report content

TD1 Manual review of test report content

can verify that report has been

correctly generated by the tool.

TCL1

[SLTEST_E6]

Usage of incorrect

input data

[SLTEST _UC1]

[SLTEST _UC2]

TI2 Incorrect input data may

lead to incorrect test run

and could prevent errors in

an object under test from

being detected.

[SLTEST _M3]

Configuration

management

[SLTEST _M4]

Input data integrity

checks

TD1 Revision control and configuration

management facilitate integrity of the

input data. Using checksums allows

the unique identification the input

data.

TCL1

[SLTEST_E7]

Incorrect tool usage

All TI2 Incorrect tool usage could

prevent errors in an object

under test from being

detected.

[SLTEST _M5]

Competency of project

team

TD1 Training of tool users can prevent

these issues.

TCL1

[SLTEST _E8]

Incorrect or

Modified or

All TI2 Incorrect tool installation

may lead to incorrect test

run could prevent errors in

[SLTEST _M2]

Tool installation

integrity checks

TD1 Verification of the installed tool

version will detect invalid tool

installation.

TCL1

2-11

Potential malfunction or erroneous output

Use cases TI Justification for TI Prevention / detection measures

TD Justification for TD TCL

Incompatible with

Environment Tool

Installation

an object under test from

being detected.

Based on the preceding analysis, the maximum tool impact of the Simulink Test use cases taken

into account is TI2.

Subsequent use of error detection measures [SLTEST _M7] and [SLTEST _M8] provides high

degree of confidence that tool malfunctions SLTEST_E4 and SLTEST_E5 will be detected.

Therefore the tool confidence level for the capabilities implementing the corresponding use

cases SLTEST_UC3, SLTEST_UC4 and SLTEST _UC5 is TCL1. If no measures are applied

the tool confidence level is TCL3.

For the capabilities implementing use cases SLTEST_UC1 and SLTEST_UC2 the tool

confidence level is TCL1 provided the prevention/detection measures identified in the table

above are taken.

TÜV SÜD reviewed the generic tool classification and confirmed the results in Report to the

Certificate Z10 15 06 67052 016.

3 Software Tool Qualification Report

3-2

3.1 Requirement for Tool Qualification TCL1 can be claimed for the Simulink Test capabilities implementing use cases SLTEST_UC1

and SLTEST_UC2 given the workflow and error detection measure specified in the document

are applied. Therefore additional tool qualification methods are not necessary according to ISO

26262-8, clause 11.4.6.1.

Given the required tool confidence level TCL3 for the Simulink Test capabilities

SLTEST_UC3, SLTEST_UC4 and SLTEST_UC5 without manual comparison and review (see

Generic Tool Classification), these capabilities need to be qualified up to TCL3. Permissible

tool qualifications for TCL3 are listed in ISO 26262-8 Table 4.

3-3

3.2 Tool Qualification Documentation MathWorks carried out an application-independent prequalification of the Simulink Test

product. The Simulink Test capabilities SLTEST_UC3 (Assessment of test results) and

SLTEST_UC4 (Generation of test reports) and SLTEST_UC5 (Identification of traceability

between requirements and tests cases) were prequalified for all ASILs according to ISO 26262-

8, up to and including TCL3. These capabilities of Simulink Test were prequalified using a

combination of the following methods:

Evaluation of the tool development process (ISO 26262-8, Table 4, Method 1b).

Validation of the software tool (ISO 26262-8, Table 4, Method 1c).

According to ISO 26262-8, table 4, these two methods are permissible for all ASILs. Method 1b

is highly recommended for ASILs A, and B. Methods 1c and 1d are highly recommended for

ASIL D.

Tool qualification for the corresponding capabilities of the Simulink Test product can be

claimed for TCL1 and TCL3 by referencing the certification report and corresponding

certificate.

TÜV SÜD carried out an independent tool qualification assessment. MathWorks submitted the

results of the methods applied to pre-qualify Simulink Test to TÜV SÜD.

TÜV SÜD reviewed the generic tool qualification artifacts for Simulink Test and confirmed the

results in Report to the Certificate Z10 15 06 67052 016.

3-4

4 Confirmation Review of Tool Classification and Qualification

4-2

4.1 Requirement for Confirmation Review The tool classification (see Software Tool Criteria Evaluation Report) was carried out

independently from the development of the application under consideration. Therefore, the

resulting, predetermined tool confidence level shall be confirmed by the applicant prior to

Simulink Test being used for the development of a particular safety-related item or element for

the application under consideration (see ISO 26262-8, 11.4.2, 11.4.10).

If TCL3 is confirmed, the prequalification shall be confirmed prior to Simulink Test being used

for the development of a particular safety-related item or element for the application under

consideration (see ISO 26262-8, 11.4.2, 11.4.10).. The confirmation is required, because the

prequalification was carried out independently from the development of the application under

consideration.

If TCL1 is confirmed, tool qualification and hence confirmation of the tool qualification are not

required.

The generic tool classification is based on the assumption that Simulink Test is being used as

described in the reference workflow documented in Simulink Test Reference Workflow.

Therefore, conformance with the entire reference workflow (for TCL1) or the suitable subset

(for TCL3) in the application under consideration shall be confirmed by the applicant.

4-3

4.2 Validity of Generic Tool Classification Applicable Tool Confidence Level: < Select TCL1 or TCL3>

<Insert results of confirmation review or reference to confirmation review documentation>

4-4

4.3 Validity of Generic Tool Qualification Applicable Tool Confidence Level: < Select TCL1 or TCL3>

< Insert results of confirmation review or reference to confirmation review documentation in

case of TCL3>

4-5

4.4 Conformance with Reference Workflow Applicable Tool Confidence Level: < Select TCL1 or TCL3>

< Insert reference to customized and completed Conformance Demonstration Template >