project proposal - opensgosgug.ucaiug.org/sgsystems/openade/shared document… · web...

INTEROPERABILITY TEST AND CERTIFICATION

MANAGEMENT ASSISTANCE ANALYSIS

Prepared for:The UCAIug OpenADE Task Force, UCAIug SG

Prepared by:The UCAIug OpenADE Task Force and QualityLogic, Inc

Managed by:UCAIug OpenADE Task Force

Version0.01

Interoperability Test and Certification Management Assistance Analysis Version – 0.01 iUCA International Users Group December 12, 2011

1

2

Revision History

Rev Date Summary Marked0.01 2011-21-12 Initial draft for team review N

Open Editorial Items and Issues LogAs open items and issues are addressed in new versions of this document, they are removed from this list.

Item No.

Date Provided By

Summary of the Issue Status / Disposition

Interoperability Test and Certification Management Assistance Analysis Version – 0.01 2UCA International Users Group December 12, 2011

3

4

56

78

Executive SummarySmart Grid Technical Standards are critically important to development of interoperable products, increasing competition, reducing costs of Smart Grid components, and speeding implementation of new technologies. These key principles enable mainstream market adoption of Smart Grid products and services. This is important for utilities and other market participants to be able to make viable infrastructure investments, and reduce project costs and product development investments required to leverage the opportunities that Smart Grid technologies make available.

The UCA International OpenADE Task Group developed the requirements that led to the North American Energy Standards Board, (NAESB) REQ18/WEQ19, PAP10 Energy Usage Information and NAESB REQ21 Energy Services Provider Interface (ESPI) standards. These are specifically referenced in the November 8, 2011, US Department of Energy Funding Opportunity to demonstrate or pilot innovative applications based on standardized energy usage availability. OpenADE/ESPI is a key standard in the NIST V2.0 Smart Grid Standards Roadmap and is the most important standard addressing technology standardization for enabling consumer access to energy usage information.

Two of the important tasks that the OpenADE Task Group plans to undertake are:

- Develop a conformance, certification, and testing process and program for OpenADE, coordinated with entities such as standard development organizations (SDOs), user groups, and Smart Grid activities.

- Develop programs to allow vendors to develop, test, and demonstrate their ability to integrate with OpenADE communications protocols.

Creating a test and certification program is a challenging activity, especially in light of the rigorous requirements embodied in V1.0 and Draft Version 2.0 of the Smart Grid Interoperability Panel (SGIP) Test and Certification Committee’s (TCC) Interoperability Process Reference Manual (IPRM)1.

This UCA International OpenADE Task Group is aiming to establish an accelerated test and certification program for OpenADE/ESPI (NAESB REQ 21) that can:

1. Accelerate the development and implementation of a successful interoperability certification program for OpenADE that meets or exceeds the applicable SGIP TCC IPRM requirements

1 See SGIP TCC TWIKI at http://collaborate.nist.gov/twiki-sggrid/bin/view/SmartGrid/SGTCCIPRM, V1.0, November 18, 2010.


9

10111213141516

1718192021222324

25

262728

2930

31323334

3536

373839

12

http://collaborate.nist.gov/twiki-sggrid/bin/view/SmartGrid/SGTCCIPRM

2. Create a standard test specification, test cases, test scripts, test harness, and other tools that can be used by vendors and others to test interoperable OpenADE products prior to certification

3. Establish a maintenance and update process and program to e ensure currency of the certification program and pre-certification tools

The balance of this requirements document will review the requirements of the SGIP TCC IPRM; provide commentary on the challenges in achieving the IPRM goals; outline the tasks required to achieve the goals of OpenADE for the interoperability test and certification; and suggest a set of next steps for the Task Group.


404142

4344

45464748

49

Table of ContentsSGIP IPRM REQUIREMENTS.................................................................................................................... 7

TRADE ALLIANCE FUNCTIONS............................................................................................................... 16

1.1. ITCA Test and Certification Tasks Based on the IPRM......................................................171.1.1. Organize the ITCA (Implied Tasks)......................................................................................181.1.2. Manage and Promote the Standard (Implied Tasks).................................................................181.1.3. Organize the Certification Program....................................................................................191.1.4. Define Certification Program (Explicit Tasks)........................................................................191.1.5. Establish Vendor Partnerships (Implied Tasks).......................................................................201.1.6. Implement Certification Program (Explicit Tasks)...................................................................211.1.7. Improvements in the Standard and the Certification Program (Explicit Tasks)................................221.1.8. Cyber-Security (Explicit Tasks)..........................................................................................231.1.9. Governance (Explicit Tasks)..............................................................................................23

NEXT STEPS.......................................................................................................................................... 24

SCHEDULE............................................................................................................................................ 26

SUMMARY........................................................................................................................................... 27


50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

6768

69

AuthorsJames Mater, QualityLogic

Steve Van Ausdall, Xtensible / SCE

Edited by: Dave Jollota, QualityLogic


70

71

72

73

74

75

SGIP IPRM RequirementsThe OpenADE Task Group is fortunate to have access to a ground-breaking guide to developing and managing a world-class test and certification program. Indeed, until the issuance of Version 1.0 of the SGIP IPRM, nothing like it existed for the Smart Grid (or any other industry) that we know of. Every trade alliance like UCA International OpenADE Task Group has had to create its own program, which has resulted in a great deal of variation in how such programs have evolved. Having a roadmap such as the IPRM can greatly accelerate achieving the goals of product interoperability based on a specific standard.

It is also critical to understand that the IPRM defines the standard against which the SGIP will be assessing the quality and maturity of certification programs for Smart Grid standards. Although NIST2 is not directly bound to accept the conclusions and recommendations of the SGIP (in terms of which standards to adopt), it is clear that the SGIP process is closely watched by NIST managers, and SGIP assessments of certification program maturity are expected to influence the decisions that NIST makes.

The IPRM Version 1 identifies some 86 formal requirements in five distinct areas that serve as a specification for what a good test and certification program for a Smart Grid technology standard should look like. In addition, a number of guidelines are discussed

2 NIST is the National Institute of Standards and Technology, US Department of Commerce. NIST established the Smart Grid Interoperability Panel specifically to engage a broad range of stakeholders in assisting with their mission to identify Smart Grid technology standards.


76

7778798081828384

858687888990

919293

345

that further clarify how such a program can achieve its goals of interoperable and conformant products based on the specification.

Version 2 of the IPRM is in draft form but contains significant changes from Version 1. These are noted in this analysis, but it should be understood that until Version 2 is approved by the SGIP these changes are subject to further modification.

The following is a high-level overview of the key requirements that the SGIP has embodied in the IPRM V1.0 and Draft Version 2. Not all may apply to the OpenADE Task Group, and those that do not will be noted as such.

The IPRM defines the structure and functions of an organization that takes industry responsibility for a test and certification program for a standard. The organization is identified as an ITCA, or Interoperability Test and Certification Authority, and is characterized by its authority and competence to design and implement a program.

1. Section 5.1: General Test Policies provides a high-level overview of key policies an ITCA should consider and adopt. These include:

a. The level and types of information provided to vendors for certifications

b. The information that should be included in the final test report

c. Any conditions or expiration limits placed on certifications

d. Conformance versus interoperability certifications. Conformance does not necessarily imply interoperability between products.

e. Trade-off between certification testing and economic/business considerations with attention to safety and Cybersecurity related issues

f. Establishing policies to ensure adequacy of test tools used in certifications

2. Section 5.2 details the requirements for a Test Suite Specification (TSS) and includes details. Section 5.3 is related and deals with attributes of a Test Profile in lieu of a complete TSS. A TSS consists of a suite of tests, categorized into logical functional areas, such as use cases or well-defined features. Each test suite consists of many related test cases corresponding to a particular feature set or use case. A test profile evaluates a subset of a TSS and is used to target specific areas of product interoperability.

a. A common Test Suite Specification (TSS)3 shall be established when multiple test labs are deployed to test the same standard and/or profile. If common unique test procedures are required to support this test suite, then

3 A Test Suite Specification (TSS) consists of a suite of tests, categorized into logical functional areas, such as use cases or well-defined features. Each test suite consists of many related test cases corresponding to a particular feature set or use case. Test cases would include both valid and invalid behavior tests. Each test case is further described step-by-step with test procedures and well defined pass / fail / indeterminate criteria, along with references.


9495

969798

99100101

102103104105

106107

108

109

110

111112

113114

115

116117118119120121122

123124125

6789

10

they shall also be defined. The TSS should be test tool agnostic. Test Suite Specifications (TSS) used for interoperability or conformance testing shall be managed in a well-defined, open and formal manner with change control.

b. The TSS shall be subject to revision control, including revision history, revision numbering, and a defect and expansion management process. The TSS should clearly identify the test purpose, references, resource requirements, test setup, procedures, observable results and possible problems / lessons learned with the test approach. Observables should clearly identify pass / fail / indeterminate requirements and informational elements.

c. A Test profile, which MUST be a subset of a TSS, specifies all mandatory and optional elements and restrictions of the standard specification and is treated as a companion to the technical standard, including submission to an SSO for formal standardization.

3. Section 5.4 enumerates the Technical Design of Test and Certification Programs. This section covers areas of general technical, inheritance, version control, general testing, conformance, testing, interoperability, performance, tools and test lead. There are 35 specific technical requirements for an ITCA. These are summarized below:

a. The ITCA MUST specify in the test program requirements those features that are mandatory, and those features that are optional. The ITCA shall require and enforce that vendors declare the optional features implemented in a product. (Tech-1 and -2)

b. The ITCA MUST require that implementations of optional features be tested and certified for conformance and interoperability. Furthermore, the ITCA shall define common test cases for that optional feature to be used by all test labs when testing for that optional feature. (Tech-3)

c. An ITCA MUST define the record handling and retention requirements to be followed by the TL and CB functions, consistent with requirements of ISO 17025 and ISO Guide 65. (Techn-4)

d. The ITCA SHALL specify conditions under which the use of components that have been certified by other programs can be used in products to be certified by the ITCA program. Basically, OpenADE could decide that specific components or classes of components are “pre-certified” but the ITCA remains responsible for ensuring conformance and interoperability of the OpenADE Task Group certified products. The ITCA SHALL implement a Compliant Portion Description (CPD)4 to be used as a guide

4 See Glossary of Terms in the IPRM for definition and further explanation of CPD


126127128129

130131132133134135136

137138139140

141142143144145

146147148149

150151152153

154155156

157158159160161162163

11

for assembling a product based on compatible sub-components. (Tech-5 to Tech-8)

e. Another section deals with differing versions of certified products and how the certification program should handle them. ITCAs need to manage re-certifications and the identification of current versions that are certified. (Tech-9 to Tech-12)

f. There shall be a defined correlation between implementations and required testing, commonly called a Proforma Implementation Conformance Statement (PICS). (Tech-14)

g. The testing and certification program shall maintain a current and upcoming list of applicable test cases to be called a Test Case Reference List. These test cases should be defined in an open, consensus-driven fashion. These test cases will be used by all test labs approved by the ITCA. There shall be a Test Plan derived from the Test Case Reference List and used by all authorized test labs. Tests shall be identified using the test plan. (Tech-13, Tech-15-16

h. The testing and certification program SHALL require that a static conformance review5 take place prior to testing a product. (Tech-17)

i. The testing and certification program shall first validate the tests, and implement them utilizing validated test tools. Golden reference test equipment may be utilized where appropriate. (Tech-18)

j. The testing and certification program shall assure that defined product test cases cover application profiles for specific feature sets and functions defined by the specific application profile, and implement interoperability evaluation within that application profile. Where practical, the testing and certification program shall assure that defined product test cases cover all feature sets and functions. (Tech-20 to Tech-22)

k. The ITCA SHALL classify common or major market products according to their application profiles and conduct certification tests based on those applications. The testing and certification program SHALL assure that defined product use cases are covered in application profiles. Interoperability testing and evaluation SHALL be implemented within those application profiles. (Tech-23)

l. A section deals with interoperability testing. The section deals with what are commonly called “plugfests” but also addresses the selection and use

5 A review of designed feature sets versus the specified PICS to determine the extent to which the features are supported by the IUT. This is the first step when a product enters a testing program. Generally the test lab requests that the implementer declare all supported feature sets in a product. This information is used to create the test plan for that product.


164165

166167168169

170171172

173174175176177178179

180181

182183184

185186187188189190

191192193194195196

197198

12131415

of “golden” reference units. Plugfests are optional for application interface standards. (Tech-24 and Tech-25)

m. ITCAs SHALL use reference test tools6 where appropriate to the technology under test (hardware and/or software) to provide a consistent and replicable approach in generating test results across ITCA test labs. Successful testing programs assure that there is a known reference or constant, to which the system is evaluated against the desired metrics to determine conformance. ITCA program tests that are performed across multiple test facilities SHALL implement processes to assure they are each measuring against a common known reference and achieving repeatable results regardless of location. (Tech-26 and Tech-27)

n. When used, a minimum of two golden units are to be selected by a defined process (Tech-28 to Tech-30)

o. If an ITCA Certification Program involves multiple Smart Grid systems, then the Program Requirements SHALL support end-to-end testing of Smart Grid systems involving multiple product implementations to the fullest extent possible. An ITCA SHALL involve all relevant parties to define various business logic models for the end-to-end system testing, and make scenarios and test harness systems available for testing. (Tech-30-31)

p. The testing and certification program shall ensure that when functional performance requirements are defined in an application profile, the performance test profile(s) shall be designed to implement test cases for evaluating these requirements. (Tech-32)

q. The ITCA SHALL ensure that test tools have a complete mandatory feature-set coverage of a standard. In cases where two or more implementations of optional features are available, the ITCA shall incorporate those feature-sets in the test tool. The ITCA shall define procedures and processes to validate the use of test tools and reference implementations. (Tech-33 to Tech-34)

r. An ITCA shall develop criteria for surveillance to insure that certified products and vendors fulfill the certification agreement(s). The surveillance is to be carried out by its certification body(ies). (Tech-35)

6 A number of terms are used in describing reference test tools such as “common test harness”, “golden reference test equipment”, and “golden reference test products”. Generally, these each represent test tools available to a test lab or end user to provide a consistent baseline test either as a standalone implementation or in concert with the many other types of test tools available.


199200

201202203204205206207208209

210211

212213214215216217218

219220221222

223224225226227228

229230231

16171819

4. Section 5.5 addresses Program and Field Experience Feedback and points to the importance of end-user and product vendor experience with the standard and the certification program.

5. Section 6 of Draft 4, Version 2 of the IPRM is new and addresses Cybersecurity Testing. ITCAs are responsible for coordinating and overseeing the Cybersecurity criteria as applicable to the testing and certification programs that they operate. Section 6.7 includes specific requirements of an ITCA for Cybersecurity testing but these are still in discussion in terms of the ITCA responsibility in this area. This section includes proposed requirements that:

a. The ITCA SHALL define the procedures and processes that will be used to validate interoperability Cybersecurity requirements. Such tests are usually initiated by each vendor, to verify that a vendor product meets an industry established level of security, and tested by independent third-party labs using the same testing procedures that are pass/fail in nature. (Sec-1)

b. The testing and certification program shall ensure that Cybersecurity functional performance requirements are defined, and test cases designed to evaluate the requirements. Further, ITCAs are responsible to qualify testing personnel for Cybersecurity training and experience. (Sec-2 and Sec-5)

c. If applicable, ITCAs are responsible for Digital Certificate programs. (Sec-3 and Sec 4)

d. ITCAs are responsible for requiring widely-accepted security stress testing, including static analysis and penetration testing; assuring security policy models drive testing; ensuring that vendors submit threat analyses as part of certification process; documenting programs and standards used for security testing; and incorporating component-based Cybersecurity concepts in the testing program.

One interpretation of this proposed section is that an ITCA needs to establish a distinct certification program just for Cybersecurity issues and include this testing in the certification program. The IPRM is not yet clear on the qualifications for a lab to do Cybersecurity testing.

6. Section 7.2, titled Governance, provides a structural prescription for an ITCA. The OpenADE Task Group intends to establish the ITCA for the OpenADE/ESPI and related standards, although one purpose of this document is to assist in determining what organization will actually operate the ITCA for OpenADE/ESPI. Key governance requirements are:


232233234

235236237238239240

241242243244245246

247248249250251

252253

254255256257258259

260261262263

264265266267268

a. The ITCA defines and documents the interoperability test program for the standard and oversees its implementation, including roles, responsibilities and resources. The ITCA SHALL provide oversight to provide confidence that implementations of Standard(s) in certified products are indeed interoperable. This will be the primary task of the OpenADE Task Group. (Gov-1 and Gov-5 in D3Rev4)

b. The ITCA determines whether first-party testing, third-party testing or both are allowed and defines the circumstances and process for submission to a certification body (CB) as well as the CB responsibilities. (Gov-2 and Gov-3 in D3Rev4)

c. The ITCA SHALL define a corrective process for resolving reported interoperability problems (e.g., in the field or as part of the test) for products for which they are responsible.7 Further, it SHALL implement preventive processes to avoid recurrence of such problems. A problem may be associated with the specification, the test processes and procedures or the test data. (Gov-4 in D3Rev4)

d. A key function is to ensure that issues that arise through the certification test process are fed back to appropriate parties for clarification or inclusion in subsequent versions of the standard. (Gov-6 in D3Rev4)

e. The ITCA SHALL maintain a publicly available certified product and systems list. (Gov 7 in D3Rev4)

f. The ITCA shall maintain a test case reference and modification history list8. This is a current master list of all tests that are to be included in a product certification test plan. This helps a product implementer in preparing fully conforming and interoperable products for an upcoming certification and launch. (Gov-8 in D3Rev4)

g. A common TSS SHALL be established when multiple test labs are deployed to test the same standard and/or profile. If common unique test procedures are required to support this test suite, then they SHALL also be defined. The TSS should be test tool agnostic. Test Suite Specifications (TSS)9 used for interoperability or conformance testing SHALL be managed in a well-defined, open and formal manner with change control. (Gov 9 and Gov 10 in D3Rev4)

h. The ITCA shall minimize divergence of interoperability requirements interpretations. If an ITCA has multiple testing laboratories and certifying

7 The ITCA should use best efforts in contacting a standards body with respect to a specification; however, it is not their responsibility to resolve issues with the specification.8 See Glossary of Terms in the IPRM for definition and explanation of the test case reference list. 9 See Glossary of Terms in the IPRM for definition and explanation of the TSS.


269270271272273274

275276277278

279280281282283284

285286287

288289

290291292293294

295296297298299300301

302303

20212223

bodies, processes shall be in place to avoid quality differences and assure repeatable testing between the laboratories. One way to minimize divergence of interpretations is to limit the number of labs to only one. Another option for minimizing divergence is to have a technical lead (also known as a lead lab) responsible for properly interpreting conformance and interoperability issues. (Gov-12 in D3Rev4)

i. Any organization that certifies the actual test labs and processes needs to conform to ISO/IEC 65 Guidelines and SHALL include the International Classification for Standards (ICS) Codes applicable to the technologies for which certification activities are performed. Further, Accreditation SHALL be by an accreditation body that is signatory, in good standing, to the International Accreditation Forum (IAF) multilateral agreement for “Product.” This set of guidelines generally specifies the formal documentation and processes required of such a body as well as criteria for eliminating or minimizing potential conflict of interest issues. A brief discussion of these Guidelines is contained in the IPRM Annex on page 66. (Gov-11 in D3Rev4)

j. The IPRM REQUIRES that product certification be issued by an ISO/IEC 65 accredited third party independent of the testing organization. (Section 1.4: Intended Audience in Draft 4 of Version 2)

k. The proposed IPRM Version 2 (Draft 4) includes an additional REQUIREMENT over and above ISO/IEC Guide 65 – the independent trusted third-party certification authority MUST only allow the statement that products are interoperable only if the products actually demonstrated interoperability during testing. They MUST not assume interoperability from conformance testing. They MUST demonstrate it before it may be part of the products certification statement. (Section 2.2: Overview of ISO/IEC Guide 65, Draft 4 of Version 2)

7. Section 7.3 is titled Lab Qualification and basically requires that lab selection SHALL be done in a uniform and transparent procedure and that any labs performing certification of products for the ITCA be ISO 17025 certified. Further, the ITCA SHALL define requirements to qualify the personnel involved in the certification and testing process for its standard. A discussion of ISO 17025 is contained in the IPRM Annex. In summary, ISO 17025 focuses on two major areas of laboratory operations: 1) management requirements; and 2) technical requirements. The management requirements address issues such as a lab’s documented practices (i.e., both administrative and technical), impartiality of the lab in its operations, responsibilities for continuous improvement and issues resolution, and the active support and involvement of lab management in assuring commitment to complying with these criteria. The technical requirements focus on areas such as ensuring that lab staff are competent in performing their testing


304305306307308309

310311312313314315316317318319320

321322323

324325326327328329330331

332333334335336337338339340341342343344

duties, assuring that the lab environment is adequate for services performed, assuring that test plans and other necessary operating instructions are documented and available, and that necessary equipment and software used for testing is calibrated, maintained and appropriate for its intended usage.

8. Section 4 deals with improvements to the overall process, the standard documentation itself, test labs, the test and certification program, etc. Reference is made to the preference that an ITCA solicit direct feedback from customers of the certified products to assess that they meet customer interoperability needs.

The IPRM includes additional recommendations for best practices for interoperability and conformance testing certification programs. Much of the material is covered in the requirements themselves but there are a number of useful recommendations that the OpenADE Task Group could utilize in designing their own programs.


345346347348

349350351352

353354355356

Trade Alliance FunctionsLike the UCA International IEC 61850 and CIM User Groups, the OpenSG Committee, OpenADE Task Group of UCA, is taking on the functions of an industry trade alliance when it assumes the responsibility for developing and operating an interoperability certification program for OpenADE/ESPI. In doing so, it is accepting responsibility for pioneering an activity within the traditional OpenSG and UCA International charter. The key characteristics of an interoperability certification program that distinguish it from prior UCA International efforts are:

1. The OpenADE Task Group is a typical OpenSG “requirements” specification activity that assumes other organizations will adopt and formalize an actual national or international technical standard to implement the requirements defined by OpenADE. Taking on the certification functions is perhaps a logical extension of the Task Group activities but is without precedent in OpenSG. Implementing a certification program implies a series of new activities that OpenADE will be responsible for including:

a. Defining the conformance certification profile

b. Making major decisions concerning implementation of the certification program as outlined in the IPRM and including additional decisions concerning resources, the certification business model, and developing and managing relations with test labs, product developers and others that are new and different from normal OpenSG relationships


357

358359360361362363364

365366367368369370371

372

373374375376377

c. Ensuring quality control and maintenance of the certification program

d. Managing the issuance and currency of certifications and the awareness of them in the user community

e. Marketing the technology standard and the certification program to ensure participation and customer awareness

2. The last standard that evolved from OpenSG Committee work was OpenADR Version 2. The functions of a trade alliance were taken on by a separate industry organization designed just for this purpose, rather than UCA International.

3. UCA International has two precedent activities in the IEC 61850 and CIM interoperability test programs. Both are conducted on a volunteer basis but neither of them meets the requirements set in the IPRM. To do so will require significant new investment by UCA International, as well as taking on additional functions.

OpenADE/ESPI could model itself after the IEC 61850 and CIM User Group activities and may choose to do so (ignoring SGTCC IPRM conformance). The concerns would be in terms of timeframe and available volunteer labor to develop and manage a successful ITCA program.

In setting up and operating an Interoperability Test and Certification Authority (ITCA) (for lack of a better term to describe the functions incumbent on UCA and OpenADE/ESPI), there are a series of activities and responsibilities that are addressed specifically or implied in the SGTCC IPRM, most of them enumerated in a separate section. These will need to be addressed whether the UCA takes on this role, or some other entity.

The following is an attempt to organize the IPRM explicit and implicit requirements and suggested best practices for an ITCA into an actual task list as summarized below:

1.1. ITCA Test and Certification Tasks Based on the IPRM

According to the IPRM proposed Version 2, Draft 4:

An Interoperability Testing and Certification Authority (ITCA) is the program management organization, providing oversight for testing and certification activities associated with one or more standards or specifications, that takes responsibility to insure that interoperable products within the scope of the specific ITCA program are brought to market. The ITCA coordinates the participation of certification bodies and test labs for its program.

The following are the tasks that OpenADE/ESPI and/or UCA International will need to complete as part of its ITCA goals under the IPRM Version 1 and proposed Version 2.


378

379380

381382

383384385

386387388389

390391392393

394395396397398399

400401

402403

404

405406407408409410

411412

1.1.1. Organize the ITCA (Implied Tasks) Develop a business plan for the ITCA

o Determine the level of conformance to the IPRM that the ITCA will aspire to implementing, including independent ISO/IEC 65 and ISO 17025 certification requirements

o Establish charter, scope and legal framework, including legal documents as needed

o Establish Governance policies and procedures

o Determine and organize ITCA budget and Treasury functions

o Determine staff support requirements and resources available

o Determine overall budget and schedule for a prudent period of time (3-5 years)

IPR Policy development and implementation

Recruit members who can contribute both financial and staff resources

Set sponsorship and dues levels or acquire other sources of funding

Select leadership

Establish meeting and working protocols

1.1.2. Manage and Promote the Standard (Implied Tasks) Recruit vendors and customers to adopt and use the OpenADE/ESPI standard

Continue to support development of the standard (working with the appropriate SSO10)

Conduct conferences, meetings, trade show exhibits and plugfests

Represent the interest of the ITCA members at appropriate events and organizations to promote the standard

Develop and maintain a web site for the standard as part of promotion, for listing certified products and for acknowledging sponsors, members and contributors

Put in place intellectual property protections – copyrights, trademarks, etc.

10 SSO = Standards Setting Organization. This may be an international standards organization such as ISO, IEC, OASIS, IEEE, etc., or in some cases it may be a trade association such as the ZigBee Alliance.


413

414

415416417

418419

420

421

422

423424

425

426

427

428

429

430

431

432433

434

435436

437438

439

2425

1.1.3. Organize the Certification Program Organize the ITCA certifications and determine what organization will act as the

Certification Body. Actual certifications must be issued by independent third parties which are not the test labs and are accredited to ISO/IEC 65 Guidelines. The ITCA could become an ISO/IEC 65 certified body and issue certifications of conformance and interoperability itself or contract with such an organization to do so.

o Certification bodies (CBs) should be accredited to ISO Guide 65, General Requirements for Bodies Operating Product Certification Systems

o Test laboratories should be accredited to ISO 17025, General Requirements for the Competence of Testing and Calibration Laboratories

o The ITCA should have an agreement with an accrediting organization(s) to assure that Certification Body and Test Lab accreditation is being performed in accordance with the ITCA program scheme.

An ITCA should have a strong relationship with the SSO associated with the standard for the purpose of feedback towards standard improvement and clarification where there may be ambiguities

1.1.4. Define Certification Program (Explicit Tasks) Define the certification program for OpenADE/ESPI products

The IPRM REQUIRES that product certification be issued by an ISO/IEC 65 accredited third party independent of the testing organization

The proposed IPRM Version 2 (Draft 3) includes an additional REQUIREMENT over and above ISO/IEC Guide 65 – the independent trusted third-party certification authority MUST only allow the statement that products are interoperable only if the products actually demonstrated interoperability during testing. They MUST not assume interoperability from conformance testing. They MUST demonstrate it before it may be part of the products certification statement.

Establish a detailed work program with schedule and resources

Determine the business model(s) to be used

o Volunteer labor for all activities

o Combination volunteer and funded staff: which activities for which

o Contractors and/or vendors and labs to implement aspects of the ITCA management and programs


440

441442443444445446

447448

449450451

452453454

455456457

458

459

460461

462463464465466467

468

469

470

471

472473

Agree on one or more Proforma Implementation Conformance Statement (PICS) or profiles that represent the most likely use cases for products based on the standard

Develop the high-level certification test specification based on the PICS

Determine how detailed test cases, scripts and test harness(es) will be developed and maintained and who will develop and maintain them

Develop a program overview and applicant preparation guide

Define defect tracking and issue tracking requirements for both the technical program and tools and the business functions

1.1.5. Establish Vendor Partnerships (Implied Tasks) Determine philosophy of ITCA management (volunteer or professional)

o Develop RFP if professional management is determined

o Identify potential ITCA managers and solicit proposals in response to the RFP

o Select and develop contract with manager

o Manage contract manager activities and set policy

Determine if one or more test labs will be required and whether or not the structure needs to be developed to add test labs in the future

o Develop RFP for test lab(s)

o Identify potential test labs and solicit proposals in response to the RFP

o Select and develop contract with test lab

o Manage contract activities with test lab

o Follow guidelines in ISO Guide 65 to ensure that Labs are ISO 17025 accredited and maintain accreditation

Determine if a separate test tool vendor will be required and whether or not the structure needs to be developed to add vendors in the future

o Develop RFP for test tool vendor

o Identify potential vendors and solicit proposals in response to the RFP

o Select and develop contract with test tool vendor

o Manage contract activities with test tool vendor


474475476

477

478479

480

481482

483

484

485

486487

488

489

490491

492

493

494

495

496497

498499

500

501

502

503

Determine if a separate marketing communications vendor will be required

o Develop RFP for marketing communications vendor

o Identify potential vendors and solicit proposals in response to the RFP

o Select and develop contract with marketing communications vendor

o Manage contract activities with marketing communications vendor

1.1.6. Implement Certification Program (Explicit Tasks) Define the General Test Policies for the certification program for OpenADE/ESPI

products (Section 5.1 of the IPRM D3V2)

Establish test report template, contents and example

Develop and maintain a test case reference and modification history list

Establish a Common Test Suite Specification (TSS11 – Section 5.2/3 of the IPRM D3V2) if multiple test labs are deployed to test the same standard and/or profile

If required, define common unique test procedures to support the TSS –these should be test tool agnostic

Manage the TSS in a well-defined, open and formal manner with change control

If there are multiple testing laboratories, put in place processes to avoid quality differences and assure repeatable testing between the laboratories

Specify in the test program requirements for those features that are mandatory, and those features that are optional (Section 5.4 of the IPRM D3V2)

o Require and enforce that vendors declare the optional features implemented in a product

o If more than one vendor implements the same optional feature in a product, require that future implementations of that optional feature be tested and certified for conformance and interoperability

o Define common test cases for optional features that need to be tested as part of the certification program

Establish certification programs, terms and conditions of award and re-certification

11 A Test Suite Specification (TSS) consists of a suite of tests, categorized into logical functional areas, such as use cases or well-defined features. Each test suite consists of many related test cases corresponding to a particular feature set or use case. Test cases would include both valid and invalid behavior tests. Each test case is further described step-by-step with test procedures and well defined pass / fail / indeterminate criteria, along with references.


504

505

506

507

508

509

510511

512

513

514515

516517

518

519520

521522

523524

525526527

528529

530531

2627282930

o Maintain a published list of certified products

o If a logo is part of the program, create the logo and licensing agreements

Determine which components, if any, certified by other industry programs can be “inherited” in product certifications

o Develop the procedures for validating that pre-certified components included in OpenADE/ESPI products do not impact interoperability and conformance to the OpenADE/ESPI specification

Develop common, well-defined standardized test cases in an open, consensus-driven fashion. These test cases will be used by all test labs approved by the ITCA

o Validate the tests and implement them utilizing validated test tools. Golden reference test equipment may be utilized where appropriate. Define procedures and processes to validate the use of test tools and reference implementations.

o Ensure that test tools have a complete mandatory feature-set coverage of a standard. In cases where optional features are included in vendor products, incorporate those feature-sets in the test tool.

Maintain a current and upcoming list of applicable test cases to be called a Test Case Reference List

o Work with authorized labs to derive a Test Plan from the Test Case Reference List. Tests shall be identified using the test plan

Establish and maintain a revision control system, including revision history, revision numbering, and a defect and expansion management process for all tests in the TSS

Assure that defined product test cases cover application profiles for specific feature sets and functions defined by the specific application profile, and implement interoperability evaluation within that application profile

Define interoperability-specific testing procedures such as “plugfests” but also the selection and use of “golden” reference units. A minimum of two golden units are to be selected.

1.1.7. Improvements in the Standard and the Certification Program (Explicit Tasks)

Develop and maintain an improvement program for the overall process, the standard documentation itself, test labs, the test and certification program, etc. (Section 5.5 in the IPRM D3V2) If possible solicit direct feedback from


532

533

534535

536537538

539540

541542543544

545546547

548549

550551

552553554

555556557

558559560

561

562

563564565

customers of the certified products to assess that they meet customer interoperability needs.

1.1.8. Cyber-Security (Explicit Tasks) The testing and certification program shall ensure that Cybersecurity functional

performance requirements are defined, and test cases designed and used to evaluate the requirements

The ITCA needs to work with NIST and the SGIP to ensure that the technical specification for OpenADE/ESPI standards is reviewed for cyber-security issues

The ITCA may need to establish a Digital Certificate Program if applicable

Determine and implement appropriate security stress testing including static analysis and penetration testing; assure security policy models drive testing; ensure that vendors submit threat analyses as part of certification process; document programs and standards used for security testing and incorporate component-based Cybersecurity concepts in the testing program

One interpretation of this proposed section is that an ITCA needs to establish a distinct certification program just for Cybersecurity issues and include this testing in the certification program. The IPRM is not yet clear on the qualifications for a lab to do Cybersecurity testing.

1.1.9. Governance (Explicit Tasks) Determine whether first party testing, third party testing or both are allowed

and define the circumstances and process for submission to a certification body (CB) as well as the CB responsibilities

Define a corrective process for resolving reported interoperability problems and implement preventative processes to avoid recurrence of such problems.


566567

568

569570571

572573

574

575576577578579

580581582583

584

585586587

588589

Next StepsThe UCA International OpenADE Task Group can move forward in different ways, but a process could be the following:

Review the above Task List and make key decisions on how to develop a formal ITCA and manage it. The initial key decisions are probably

o Whether or not to develop a formal ITCA

o Whether or not to set up formal membership and sponsorships in a legal structure for the standard

o Whether or not to raise funding to contract with a manager

o Whether or not to develop all of the artifacts with volunteer labor or raise funding to contract with appropriate vendors

o Whether or not to operate a formal third-party certification program or use a self-certification program

o Whether or not to contract with an independent test lab for certifications (depends on above)

o Whether or not to take on the marketing aspects of promoting the standard and certified vendor products


590

591592

593594

595

596597

598

599600

601602

603604

605606

o What are the organizational requirements that would be needed to implement the key decisions

o How much funding would be required and how best to pursue it

Based on the key decisions above, develop a formal or informal request for proposal to solicit proposals for operating an OpenADE/ESPI ITCA as envisioned by the OpenADE Task Group.

Based on the key decisions, conduct a specific discussion with UCA International to understand the tasks that it can take on and the funding requirements to do so. Does UCA International have (or can it develop) the needed organizational structure to accomplish the goals of the OpenADE Task Group?

Determine if the UCA can meet the requirements of the OpenADE Task Group, including the schedule, funding and marketing activities (if any), RFP and contracting activities (if any), etc.

Depending on the above determination, solicit additional responses to the OpenADE/ESPI ITCA RFP.


607608

609

610611612

613614615616

617618619

620621

Schedule

Task Completion DateTo Be Determined


622

623

SummaryIn summary, this initial requirements document has the potential to greatly accelerate the schedule and quality of both the OpenADE test and certification program and the overall OpenADE industry interoperability.


624

625626627

project proposal - opensgosgug.ucaiug.org/sgsystems/openade/shared document… · web...

Documents