02_system testing tools

© OMICRON

Page: 1K02 03 20060309

K02 03 20060309

System Testing ToolsSystem Testing Tools

Dr.Dr. Alexander ApostolovAlexander Apostolov

© OMICRON Page: 2K02 03 20060309

Requires:• Clear definition of what is considered a System

• An Object-Oriented approach• Good understanding of the functional hierarchy of the system

• Good understanding of the system model and the internal or external interfaces

• Good understanding of the existing testing tools

• Clear definition of the requirements for development of new testing tools

System Testing DevelopmentSystem Testing Development

© OMICRON Page: 3K02 03 20060309

System DefinitionSystem Definition

• System (from the Latin systema) is an assembly of entity/objects, real or abstract, comprising a whole with each and every component/element interacting or related to at least one other component/element.

• Any object which has no relationship with any other element of the system is not a component of that system.

• A subsystem is then a set of elements, which is a system itself, and a part of the whole system.

• Every division or aggregation of real objects/entities into systems is arbitrary, therefore it is a subjective abstract concept.

© OMICRON Page: 4K02 03 20060309

Substation Automation SystemsSubstation Automation Systems• Substation Automation Systems (SAS) consists of several subsystems, each of them performing specific substation functions

• The following are typical SAS subsystems:• Protection• Automation• Control• Measurements• Metering• Recording• Condition monitoring• Analysis

• Subsystems will be considered functions

© OMICRON Page: 5K02 03 20060309

System HierarchySystem Hierarchy

Function

System

1..*

1

1..*

1

Subfunction

1..*

1

1..*

1

Subfunction

1..*

1

1..*

1

FunctionalElement

1..*

1

1..*

1

© OMICRON Page: 6K02 03 20060309

IEC IEC 6185061850 ImpactImpact• IEC 61850 is being accepted by many

utilities as the substation automation systems (SAS) protocol of the future

• A main issue with the implementation of the standard is lack of tools that take full advantage of the capabilities of the technology

• IEC 61850 defines an object model that can be used as a starting point in the development of system testing tools

• The IEC 61850 object model is not complete and requires further clarification and development of implementation agreements

© OMICRON Page: 7K02 03 20060309

IECIEC 61850 Impact61850 Impact

• OMICRON can play an important role by taking the lead in the development of a complete object model of multifunctional IEDs

• A system for completely automatic and self-configuring testing of SAS will provide the users with significant benefits

• It can take full advantage of the experience in the creation and use of the Test Universe and the different features of IEC 61850

© OMICRON Page: 8K02 03 20060309

System Testing PrinciplesSystem Testing Principles

• System testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements.

• System testing falls within the scope of Black Box testing and as such, should require no knowledge of the inner design of the code or logic

• Black Box testing takes an external perspective of the test object to derive test cases

© OMICRON Page: 9K02 03 20060309

System Testing PrinciplesSystem Testing Principles• System testing is testing conducted on a

complete, integrated system to evaluate the system's compliance with its specified requirements.

• System testing falls within the scope of Black Box testing and as such, should require no knowledge of the inner design of the code or logic

• Black Box testing takes an external perspective of the test object to derive test cases

© OMICRON Page: 10K02 03 20060309


• System testing should be developed using a combination of a top-down and bottom-up approaches

• The design of the tools can use the top-down approach

• The testing should be performed using bottom-up approach

© OMICRON Page: 11K02 03 20060309


• The bottom-up testing approach should be based on:

• Functional element testing• Integration testing• System testing

© OMICRON Page: 12

Key Concepts in a Test ProcessKey Concepts in a Test Process• Test Case

• Scenario for test• Test Approaches

• Functional (Black-Box): test cases generated by analyzing inputs x outputs (*)

• Structural (White-Box): test cases generated by analyzing parts of the source code which should be executed

• Test Stages• Unit: a small unit is tested (e.g. a functional element)• Integration: integrated units are tested together• System: integrated system is tested as a whole in a “real”

execution environment• Acceptance: the test is done by the client/user (e.g. in a “pilot

project” during a fixed period of time)

© OMICRON Page: 13K02 03 20060309

Strategy for Unit TestsStrategy for Unit Tests

Test Function

Module to be tested

StubStubTest cases

Results

© OMICRON Page: 14K02 03 20060309

TopTop--down Strategy for down Strategy for Integration TestsIntegration Tests

A

B

E F

J K L

C

G

D

H I

© OMICRON Page: 15K02 03 20060309

TopTop--downdown StrategyStrategy for for IntegrationIntegration TestsTests

A

B

stubstub

stubstub

driver

© OMICRON Page: 16K02 03 20060309

TopTop--downdown StrategyStrategy for for IntegrationIntegration TestsTests

A

B

stubstub

stubC

stub

stub

driver

© OMICRON Page: 17K02 03 20060309

BottomBottom--upup StrategyStrategy for for IntegrationIntegration TestsTests

A

B

E F

J K L

C

G

D

H I

© OMICRON Page: 20K02 03 20060309

Some Some TestTest TypesTypes

• Functional• Fail Recovering• Fault Tolerance• Access Control• Data Integrity• Performance• Load and Stress• Configuration and Portability• Installation and uninstallion• Usability

© OMICRON Page: 21K02 03 20060309

What is a Distributed System?What is a Distributed System?

• It is a system located on multiple devices. • Some components may reside in IEDs in:

• The same panel• Same control house• Same substation• Multiple substations

• As a result, testing of these systems is more complex.

© OMICRON Page: 22K02 03 20060309

Some Challenges in the Test of Some Challenges in the Test of Distributed SystemsDistributed Systems

• Concurrency and Nondeterminism• It is very difficult to exactly replicate a test run when

the software contains multiple concurrent modules.• Temporal Correctness

• For real time systems right outputs should be produced at the correct time.

• Partial Failures• A distributed system can find that a portion of its

code cannot execute because of hardware or software failures on one of the machines hosting the system.

© OMICRON Page: 23K02 03 20060309


• Time-Outs• Distributed systems avoid deadlock by setting

timers when a request is sent to another system. • Sharing of data repository

• Access control and integrity of shared data repositories must be tested.

• Component Based Systems do not allow access to source code• Only black-box tests can be performed.

© OMICRON Page: 24K02 03 20060309


• Dynamic Nature of the Structure• A distributed system is often built with the capability of changing

its configuration, for example, some specific requests may be directed dynamically, depending on the load on the various machines.

• Synchronization• When two or more tasks must access the same memory location,

a mechanism is needed to prevent the two tasks from interfering with each other.

• Dealing with Environment Behavior/External Events• When the system reacts to external events and environment

behavior, these are not easily reproduced (stubs are necessary -e.g. simulators).

• Non-stop work• hese systems need to be on for long journeys (24/7).

• Non-visible message exchanges • Usually some message exchanges between components are

unobservable by a tester using a black-box approach.

© OMICRON Page: 25K02 03 20060309

SAS Test Configuration ProcessSAS Test Configuration Process

IEC 61850Standard IED Configuration

Tool

CIDFile

Scheme Testing Tool

IED Simulator

Test Device

IEC 61850Standard System

Configuration Tool

SCDFile

IEC 61850Standard Test Configuration

Tool

NetworkSystem

Simulator Test Universe Test Modules

CIDFileCID

File

Test Device Test Device

IED Simulator

IED Simulator

Test Device Test Device Tested Device

© OMICRON Page: 26K02 03 20060309

ConclusionConclusion

• Testing distributed systems is more complex• Non-deterministic behavior• Concurrency • Functional/Temporal correctness• Difficulty to automate (few tools)

• Non conventional techniques are needed to test these systems (e.g. the use of simulators)• High costs of fails justify the investments.

02_system testing tools

Documents