international its standards - etsi · international its standards understanding their role and...

World Class Standards

International ITS Standards

Understanding their role and benefits

Interoperability

Martin Arndt

ETSI ESP TBS© ETSI 2008. All rights reserved

7th European Congress on ITS | Geneva

2


Today’s Test Description

� What is interoperability?

� Conformance testing versus Interoperability testing

� Standards and Test Specifications development process

� ETSI Services related to Conformance and Interoperability

� ETSI Plugtests™

� Conclusions

3


What is Interoperability?

� The ultimate aim of standardization in information and communications technology

� The thread being present during the entire standards development process

� Not an isolated issue

� Not something to be somehow fixed at the end

� ETSI approach

� To produce base standards and test specifications in parallel

� Base standards should be designed with interoperability in mind

• Profiles can help to reduce potential non-interoperability risks

� Two complementary forms of testing

• Conformance testing

• Interoperability testing (i.e. a more formal form of IOT)

� Testing provides vital feedback into the standardization work

4


Characteristics of Conformance Testing

� Black-Box testing

� Stimulation and response

� Points of control and observation

� Tests are usually done layer by layer

� Implementation Under Test (IUT)

� Tests requirements specified either in a base standard

or profile standard

� Gives a high-level of confidence that key components of

a device or system are working as intended

� High IUT control and observability

� Can explicitly test error behaviour

� Can provoke and test non-normal (but legitimate)

scenarios

� Can be extended to include robustness tests

5


Limitations of Conformance Testing

� Does not prove end-to-end functionality (interoperability)

between communicating systems

� Conformance tested implementations may still not interoperate

� Likelihood of interoperability depends on the amount of test cases

• minimum set of requirements or profiles needed

� Tests individual system components

� But a system is often greater than the sum of its parts!

� Does not test the user’s ‘perception’ of the system

6


Characteristics of Interoperability Testing

� It is system testing

� Tests a complete device or a collection of devices

� Demonstrates that two or more devices interoperate

� Within a given scenario

� Testing at a high level (perception of users)

� Tests the complete system

• not parts or layers only

• protocol stacks integrated with applications

� Tests functionality

• shows system works properly but not how

� Does not need to involve test systems

� Uses existing interfaces (standard/proprietary)

� Interoperability testing looks at end-to-end functionality

� Less thorough but wide in scope

� Gives a high-level of confidence that devices (or components in a system) will interoperate with other devices (components)

7


Limitations of Interoperability Testing

� Does not prove that a device is compliant

� Interoperable devices may still interoperate even though they do

not comply with a base standard

� Cannot explicitly test error behaviour or unusual scenarios

� Invalid conditions may need to be forced

• lack of controllability

� Has limited coverage

• does not fully exercise the device

� Can be performed manually

� More difficult to automate

• due to lack of standardized control interfaces

� Does not prove 100% interoperability with other

implementations with which no testing has been done

� ‘A’ may interoperate with ‘B‘ and ‘B’ may interoperate with ‘C’. But

it doesn’t necessarily follow that ‘A’ will interoperate with ‘C’.

8


Conformance and Interoperability testing

are complementary� ETSI experience

� As you move up a system stack the emphasis should change

from conformance to interoperability testing

� Lower layer protocols and infrastructure

� Emphasis on conformance

� Middleware

� Combination of conformance and

interoperability testing

� Services, applications, systems

� Emphasis on interoperability testing

� Conformance testing can be considered as a pre-requisite to

interoperability testing

Interop TestingInterop TestingInterop TestingInterop TestingConformanceConformanceConformanceConformanceTestingTestingTestingTesting

InteroperabilityInteroperabilityInteroperabilityInteroperability

9


Typical Standards Development Process

(Unit) Conformance Testing

Interoperability Testing

Products mature from prototypes to commercial products

ETSI: Development of base standards

Certification

Industry

time

Conformance Test Specifications

Interoperability Test SpecificationsIterative

feedback

Iterative

feedback

Standards

Body

Testing provides vital feedback, e.g. shortcomings or

inconsistencies in base standards

10


Test Specification Development

Standard Test

Purposes

Test

Descriptions

TTCN-3

Test Suite

Executable

Tests

CompilationCompilationCompilationCompilation

Requirements

CatalogueTest Case Parameterization and SelectionTest Case Parameterization and SelectionTest Case Parameterization and SelectionTest Case Parameterization and Selection

11


� Specifications development services

� In-house team of experts providing direct support as a service to

ETSI Technical Bodies on the use of formal techniques in standards

� Support the development of ETSI test specifications

• conformance testing

• interoperability testing

� Assist to the development of protocol standards and profiles

• Standards Engineering

� Validation of standards

• testbed activities (e.g. IPv6, SIP/H.323)

� ETSI Plugtests™ Service

� Validation of standards through interoperability events

ETSI’s Centre for Testing & Interoperability (CTI)

12


ETSI CTI’s Detailed Portfolio of Services

� Expertise in conformance and interoperability testing

� Advice on testing strategy, planning, processes, certification etc.

� Assistance with the application of methodology, techniques, tools

� Development of test architectures and test design

� Development of Requirements Catalogues

� Development of Implementation Conformance Statement (ICS)

� Development of Test Purposes

� Development of Test Descriptions

� Detailed coding of Test Cases (mainly TTCN-3)

� Packaging of Test Cases into full Test Suites

� Synergy with interoperability events (Plugtests)

� Participation in validation of the tests (feedback)

� Advice on test implementation and links to test tool providers

13


ETSI CTI Involvement

� Cellular: GSM and 3G (UMTS) terminals

� BWA: WiMAX

� VoIP: H.323, SIP, SIGTRAN

� Service Creation: OSA/Parlay (API, IDL)

� IPv6: Core, Security, Mobility, v4-v6

� Cordless phones: DECT

� Radio communications: TETRA, DMR, dPMR

� Access terminals: FSK, SMS

� Broadband: ISDN, DSL

� Smartcards: Readers, cards, security modules

� Intelligent Transport Systems (ITS)

� NGN: IMS, GRID

� Application Enablers: OMA

14


ETSI CTI and Intelligent Transport Systems

� Electronic Fee Collection (EFC)

� CEN Dedicated Short Range Communications (DSRC) 5 GHz

• CEN Base Standards (CEN EN 12795, CEN EN 12834, CEN EN 13372)

• ETSI Test Specifications (ETSI TS 102 486 standards family)

� UNI (Italian) Dedicated Short Range Communications (DSRC) 5 GHz

• UNI/ETSI Base Standard (ETSI ES 200 674-1)

• ETSI Test Specifications (to start shortly)

� Communications Access for Land Mobiles (CALM)

� ISO Base Standards (e.g. ISO 21218 Medium Access Points)

� ETSI Test Specifications (work just started)

� Vehicular Communications

� ETSI European Profile Standard for 5 GHz frequency range

� ETSI Test Specifications (at the horizon)

15


ETSI CTI’s Plugtests™ Service

� Intended for verifying interoperability

� Feel the users‘ perception

� Fully customizable service

� Neutral and open to all organizations

� No competitive environment

� Event management

� Website, online registration, hosting site, social event facilitation,

legal aspects, shipping assistance, etc.

� Communication management

� Event promotional kit development, event presentation at key

conferences, dissemination by internal/international press, etc.

� Technical Management

� Assistance with infrastructure set-up, tests slots scheduling,

feedback, etc.

16


Why to join Plugtests™ Events?

� Provides useful feedback to the standardization process

� Accelerates time to market via quick product debugging

� Improves both quality and features of implementations

� Helps to ensure end users’ satisfaction

� Reduces deployment costs

� Allows networking with the market key players

COST & TIME SAVING!

17


How Plugtests™ look like...

Oh oh,

I found a bug

18


ITS Interoperability Events

� Vehicle-to-vehicle communications demonstration event

� Opel Test Track in Dudenhofen 22-23 October 2008

� Organized by Car 2 Car Communication Consortium

� ETSI CTI provide little guidance and act as observer

� ETSI TC ITS Plugtests™

� Vehicular communications

� Scheduled for Q3/2010

19


Testing Conclusions

� Standards can be designed for interoperability

� Standards should be engineered not hacked

� Plan for validation and testing (early)

� Do the right kind of testing and test in parallel

� Interoperability events: Plugtests™

� ETSI can help you achieve the above!

20


Questions

Just for my

understanding...

What is...

Can you please...

How does this...

Does this mean...

Is it true that...

[email protected]

international its standards - etsi · international its standards understanding their role and...

Documents