Download - ebxml-ct11-7
-
7/31/2019 ebxml-ct11-7
1/21
-
7/31/2019 ebxml-ct11-7
2/21
Introduction
NIST works with industry to develop
standards and tests to improve the quality of
software and achieve interoperable solutions Many years experience with
Formal standards organizations and Consortia
Developing conformance test suites, tools,
reference implementations
Developing validation and certification testing
programs
-
7/31/2019 ebxml-ct11-7
3/21
Todays Objectives
Present basic information about conformanceand conformance related topics
Achieve a common understanding Lead a discussion about how conformance
applies to ebXML
Present overview of TA Spec. Conformance
Clause Offer our assistance to WG in discussing and
developing conformance for their Spec.
-
7/31/2019 ebxml-ct11-7
4/21
Outline
Background
Terminology Conformance Testing
Testing Methodology
Testing Ingredients
ebXML Issues
TA Conformance Guidance
Conformance
clause
Conformance Testing
Test suites Test tools
Validation
Procedures, reporting
Certification
Certificates Branding
-
7/31/2019 ebxml-ct11-7
5/21
Background
Standards not enough to ensure interoperability
Standards only meaningful if implemented in a consistent way
Need to ensure that implementations adhere to the
standard
What is expected of implementations in order to claim
conformance i.e., what are the requirements?
How will we know if an implementation conforms?
test suites, test tools Different ideas of what conformance is
Past experience may have affected view of conformance
-
7/31/2019 ebxml-ct11-7
6/21
-
7/31/2019 ebxml-ct11-7
7/21
Terminology - Conformance
CONFORMANCE - the fulfillment of a product,process or service of specified requirements(ISO Guide 2)
These requirements are specified in a standard orspecification as part of a conformance clause or inthe body of the specification
CONFORMANCE CLAUSE - a section of a
specification that states all the requirements orcriteria that must be satisfied to claimconformance
-
7/31/2019 ebxml-ct11-7
8/21
-
7/31/2019 ebxml-ct11-7
9/21
Testing Methodology
Falsification Testing find errors by means of experimentation
show presence of errors not their absence
prove non-conformance; can never prove conformance
Tests built to check for: required functionality has been implemented (basic tests)
logical errors by misunderstanding requirements
errors from boundary conditions and divergence
common programming errors
-
7/31/2019 ebxml-ct11-7
10/21
Conformance Tests
Requirements
100%
non conforming
????
conforming
Test programs
Specification
Implementation
-
7/31/2019 ebxml-ct11-7
11/21
Terminology - Testing Process
VALIDATION - process necessary to perform
conformance testing in accordance with a prescribed
procedure and official test suite
ensures that testing can be repeatable and reproducible
ensures that conclusions are consistent with facts presented
in the evaluation
CERTIFICATION - acknowledgement that avalidation was completed and the criteria established
by CIO for issuing certificates (brands) was met.
-
7/31/2019 ebxml-ct11-7
12/21
Testing Ingredients
StandardConformance clause
Conformance TestingTest assertions
Test suite(test software, test scripts, test criteria)
ValidationProcess - policy and procedures for testing
Certificationqualified bodies to do the testing and certification
control board - advisory and arbiter
-
7/31/2019 ebxml-ct11-7
13/21
ebXML Conformance Issues
Conformance Clause should address?
1. What Needs to Conform?
2. Profiles and Levels3. Extensions
4. Implementation Defined Features
5. Alternate approaches
Scopewhat should be addressed?a. Assertions
b. Test Suite
c. Testing Service
d. Certification
-
7/31/2019 ebxml-ct11-7
14/21
ebXML Issues -
Conformance Clause address:1. What needs to conform
identify the class of products that will be
developed what are the conditions that need to be met in
order to claim conformance
Examples:
registry, messaging service, business service
interface, ebXML message, testing partner agreements
-
7/31/2019 ebxml-ct11-7
15/21
ebXML Issues -
Conformance Clause address:2. Profiles and Levels
Are profiles and/or levels needed?
Should a profile have its own levels? minimal requirements (core)
Profile is a subset of the overall specifications that includes all of the functionality
necessary to satisfy the requirements of a particular community of users.
Levels are nested subsets of the specifications. Level 1 = core that all must implement;
subsequent levels include lower level + added functionality
-
7/31/2019 ebxml-ct11-7
16/21
ebXML Issues -
Conformance Clause address:3. Extensions
If allowed:
shall not re-define semantics of existing functions
shall not cause standard-conforming functions to execute
incorrectly
Standardized method for extensions?
Define the mechanism for extending the specification, so
extended in a standard manner Register of extensions
document parallel to but distinct from specification
list of recognized, published extensions
may eventually migrate into future versions
-
7/31/2019 ebxml-ct11-7
17/21
ebXML Issues -
Conformance Clause address:4. Implementation Defined Features
may want to address implementation dependent
ranges, min/max allowed values5. Alternative Approaches
specification may describe several different ways
to accomplish an operation (e.g., choice of
bindings) specify whether or not all approaches need to be
implemented
-
7/31/2019 ebxml-ct11-7
18/21
ebXML Issues - Scope
Should ebXML include or address:
a. Assertions (test requirements)
- Should assertions be part of the standard?b. Test Suite
- Prescribe a test methodology?
- How test suite will be delivered/used (e.g., web based,
downloadable)?
c. Testing Service
- Guidelines for establishing a test service
d. Certification
- Guidelines for issuing certificates
-
7/31/2019 ebxml-ct11-7
19/21
Testing Program - Considerations
How formal should testing be? Self testing, 3rd party testing
Branding/certificates
Testing program Who will own the testing program
Policy and procedures
Testing laboratory
Issue Certificates
Control board
Test suite maintenance
-
7/31/2019 ebxml-ct11-7
20/21
TA Conformance Guidance
General conformance guidelines for ebXML
All ebXML specs shall contain a conformance
clause Can have levels of ebXML conformance -
hierarchical
Explains how conformance requirements can be
expressed (mandatory, conditional, optional)
Encourage use of publicly available test
suites
-
7/31/2019 ebxml-ct11-7
21/21
Discussion