7/31/2019 ebxml-ct11-7

1/21

7/31/2019 ebxml-ct11-7

2/21

Introduction

NIST works with industry to develop

standards and tests to improve the quality of

software and achieve interoperable solutions Many years experience with

Formal standards organizations and Consortia

Developing conformance test suites, tools,

reference implementations

Developing validation and certification testing

programs

7/31/2019 ebxml-ct11-7

3/21

Todays Objectives

Present basic information about conformanceand conformance related topics

Achieve a common understanding Lead a discussion about how conformance

applies to ebXML

Present overview of TA Spec. Conformance

Clause Offer our assistance to WG in discussing and

developing conformance for their Spec.

7/31/2019 ebxml-ct11-7

4/21

Outline

Background

Terminology Conformance Testing

Testing Methodology

Testing Ingredients

ebXML Issues

TA Conformance Guidance

Conformance

clause

Conformance Testing

Test suites Test tools

Validation

Procedures, reporting

Certification

Certificates Branding

7/31/2019 ebxml-ct11-7

5/21

Background

Standards not enough to ensure interoperability

Standards only meaningful if implemented in a consistent way

Need to ensure that implementations adhere to the

standard

What is expected of implementations in order to claim

conformance i.e., what are the requirements?

How will we know if an implementation conforms?

test suites, test tools Different ideas of what conformance is

Past experience may have affected view of conformance

7/31/2019 ebxml-ct11-7

6/21

7/31/2019 ebxml-ct11-7

7/21

Terminology - Conformance

CONFORMANCE - the fulfillment of a product,process or service of specified requirements(ISO Guide 2)

These requirements are specified in a standard orspecification as part of a conformance clause or inthe body of the specification

CONFORMANCE CLAUSE - a section of a

specification that states all the requirements orcriteria that must be satisfied to claimconformance

7/31/2019 ebxml-ct11-7

8/21

7/31/2019 ebxml-ct11-7

9/21

Testing Methodology

Falsification Testing find errors by means of experimentation

show presence of errors not their absence

prove non-conformance; can never prove conformance

Tests built to check for: required functionality has been implemented (basic tests)

logical errors by misunderstanding requirements

errors from boundary conditions and divergence

common programming errors

7/31/2019 ebxml-ct11-7

10/21

Conformance Tests

Requirements

100%

non conforming

????

conforming

Test programs

Specification

Implementation

7/31/2019 ebxml-ct11-7

11/21

Terminology - Testing Process

VALIDATION - process necessary to perform

conformance testing in accordance with a prescribed

procedure and official test suite

ensures that testing can be repeatable and reproducible

ensures that conclusions are consistent with facts presented

in the evaluation

CERTIFICATION - acknowledgement that avalidation was completed and the criteria established

by CIO for issuing certificates (brands) was met.

7/31/2019 ebxml-ct11-7

12/21

Testing Ingredients

StandardConformance clause

Conformance TestingTest assertions

Test suite(test software, test scripts, test criteria)

ValidationProcess - policy and procedures for testing

Certificationqualified bodies to do the testing and certification

control board - advisory and arbiter

7/31/2019 ebxml-ct11-7

13/21

ebXML Conformance Issues

Conformance Clause should address?

1. What Needs to Conform?

2. Profiles and Levels3. Extensions

4. Implementation Defined Features

5. Alternate approaches

Scopewhat should be addressed?a. Assertions

b. Test Suite

c. Testing Service

d. Certification

7/31/2019 ebxml-ct11-7

14/21

ebXML Issues -

Conformance Clause address:1. What needs to conform

identify the class of products that will be

developed what are the conditions that need to be met in

order to claim conformance

Examples:

registry, messaging service, business service

interface, ebXML message, testing partner agreements

7/31/2019 ebxml-ct11-7

15/21

ebXML Issues -

Conformance Clause address:2. Profiles and Levels

Are profiles and/or levels needed?

Should a profile have its own levels? minimal requirements (core)

Profile is a subset of the overall specifications that includes all of the functionality

necessary to satisfy the requirements of a particular community of users.

Levels are nested subsets of the specifications. Level 1 = core that all must implement;

subsequent levels include lower level + added functionality

7/31/2019 ebxml-ct11-7

16/21

ebXML Issues -

Conformance Clause address:3. Extensions

If allowed:

shall not re-define semantics of existing functions

shall not cause standard-conforming functions to execute

incorrectly

Standardized method for extensions?

Define the mechanism for extending the specification, so

extended in a standard manner Register of extensions

document parallel to but distinct from specification

list of recognized, published extensions

may eventually migrate into future versions

7/31/2019 ebxml-ct11-7

17/21

ebXML Issues -

Conformance Clause address:4. Implementation Defined Features

may want to address implementation dependent

ranges, min/max allowed values5. Alternative Approaches

specification may describe several different ways

to accomplish an operation (e.g., choice of

bindings) specify whether or not all approaches need to be

implemented

7/31/2019 ebxml-ct11-7

18/21

ebXML Issues - Scope

Should ebXML include or address:

a. Assertions (test requirements)

- Should assertions be part of the standard?b. Test Suite

- Prescribe a test methodology?

- How test suite will be delivered/used (e.g., web based,

downloadable)?

c. Testing Service

- Guidelines for establishing a test service

d. Certification

- Guidelines for issuing certificates

7/31/2019 ebxml-ct11-7

19/21

Testing Program - Considerations

How formal should testing be? Self testing, 3rd party testing

Branding/certificates

Testing program Who will own the testing program

Policy and procedures

Testing laboratory

Issue Certificates

Control board

Test suite maintenance

7/31/2019 ebxml-ct11-7

20/21

TA Conformance Guidance

General conformance guidelines for ebXML

All ebXML specs shall contain a conformance

clause Can have levels of ebXML conformance -

hierarchical

Explains how conformance requirements can be

expressed (mandatory, conditional, optional)

Encourage use of publicly available test

suites

7/31/2019 ebxml-ct11-7

21/21

Discussion