Disclaimer

The present document has been produced and approved by the <long ISGname> (<short ISGname>) ETSI Industry Specification Group (ISG) and represents the views of those members who participated in this ISG.

It does not necessarily represent the views of the entire ETSI membership.

ETSI GS NFV-TST010 V0.0.7 (2018-12)

Network Function Virtualisation (NFV) Release 2;Testing;

API Conformance Testing SpecificationRelease #2

GROUP SPECIFICATION

The GS (ETSI Group Specifications) are deliverables produced by Industry Specification Groups (ISG). GSs are written with the style of a Technical Specification (TS), and represent the sole view of the ISG members.

The guidance text (green) shall be removed when no longer needed or the skeleton without guidance text also available via the editHelp! website should be used.

<<

Release #2

Reference<Workitem>

Keywords<keywords>

ETSI

650 Route des LuciolesF-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 CAssociation à but non lucratif enregistrée à laSous-préfecture de Grasse (06) N° 7803/88

Important notice

The present document can be downloaded from:http://www.etsi.org/standards-search

The present document may be made available in electronic versions and/or in print. The content of any electronic and/or print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any

existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.

Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current status of this and other ETSI documents is available at

https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx

If you find errors in the present document, please send your comment to one of the following services:https://portal.etsi.org/People/CommiteeSupportStaff.aspx

Copyright Notification

No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm except as authorized by written permission of ETSI.

The content of the PDF version shall not be modified without the written authorization of ETSI.The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI yyyy.All rights reserved.

DECTTM, PLUGTESTSTM, UMTSTM and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.3GPPTM and LTE™ are trademarks of ETSI registered for the benefit of its Members and

of the 3GPP Organizational Partners.oneM2M logo is protected for the benefit of its Members.

GSM® and the GSM logo are trademarks registered and owned by the GSM Association.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)2

Release #2

Copyrights on page 2This paragraph should be used for deliverables processed before ISG/WG approval and used in meetings.

Reproduction is only permitted for the purpose of standardization work undertaken within ETSI.The copyright and the foregoing restriction extend to reproduction in all media.

If an additonal copyright is necessary, it shall appear on page 2 after the ETSI copyright notification

The additional EBU copyright applies for EBU and DVB documents.

© European Broadcasting Union yyyy.

The additional CENELEC copyright applies for ETSI/CENELEC documents.

© Comité Européen de Normalisation Electrotechnique yyyy.

The additional CEN copyright applies for CEN documents.

© Comité Européen de Normalisation yyyy.

The additional WIMAX copyright applies for WIMAX documents.

© WIMAX Forum yyyy.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)3

Release #2

Contents (style TT)

To unlock the Table of Contents: select the Table of Contents, click simultaneously: Ctrl + Shift + F11.To update the Table of Contents: F9.To lock it: select the Table of Contents and then click simultaneously: Ctrl + F11.

Copyrights on page 2..........................................................................................................................................

Intellectual Property Rights (style H1)................................................................................................................

Foreword (style H1)............................................................................................................................................Multi-part documents............................................................................................................................................................

Modal verbs terminology (style H1)...................................................................................................................

Executive summary (style H1)............................................................................................................................

Introduction (style H1)........................................................................................................................................

1 Scope (style H1)........................................................................................................................................

2 References (style H1)................................................................................................................................2.1 Normative references (style H2)..........................................................................................................................2.2 Informative references (style H2)........................................................................................................................

3 Definitions, symbols and abbreviations (style H1)...................................................................................3.1 Definitions (style H2)..........................................................................................................................................3.2 Symbols (style H2)..............................................................................................................................................3.3 Abbreviations (style H2)....................................................................................................................................

4 Methodology (style H1).........................................................................................................................4.1 General...............................................................................................................................................................4.2 System under test...............................................................................................................................................4.3 Test configurations............................................................................................................................................4.3.1 General.........................................................................................................................................................114.3.2 Config_prod_VE..........................................................................................................................................124.3.3 Config_prod_VNFM....................................................................................................................................124.3.4 Config_prod_NFVO....................................................................................................................................134.5 Generic test description.....................................................................................................................................4.5.1 General.........................................................................................................................................................134.5.2 Scope of the tests..........................................................................................................................................154.5.3 Verification..................................................................................................................................................154.5.4 Test templates............................................................................................................................................................164.6 Test Suite Structure.......................................................................................................................................................

5 Os-Ma-Nfvo Reference Point.................................................................................................................5.1 General..........................................................................................................................................................................5.2 ICS 165.2 Test configuration.........................................................................................................................................................5.3 Interface #1 (TBD).......................................................................................................................................................5.3.1 Operation #1..............................................................................................................................................................165.3.2 Operation #2..............................................................................................................................................................165.4 Interface #2...................................................................................................................................................................

6 Ve-Vnfm Reference Point......................................................................................................................6.1 General..........................................................................................................................................................................6.2 Test configuration.........................................................................................................................................................6.3 17

7 Or-Vnfm Reference Point.......................................................................................................................7.1 General..........................................................................................................................................................................7.1.1 General characteristics of the reference point..............................................................................................177.1.2 Basic behaviours of the API producer/consumer and verification steps.....................................................187.1.2.1 Producer sends event triggered notifications based on consumer subscriptions....................................18

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)4

Release #2

7.1.2.2 Producer sends periodical notifications based on the consumer subscriptions......................................197.1.2.3 Producer executes the requested API operation.....................................................................................197.1.2.4 Consumer fetches the files/package info................................................................................................207.1.3 Verification..................................................................................................................................................207.1.3.1 Common verification aspects.................................................................................................................207.1.3.2 Verification aspects for individual API..................................................................................................217.1.3.3 Verification aspects not considered........................................................................................................217.2 Test configuration.........................................................................................................................................................7.3 22Annexes 23

Annex A (normative or informative): Title of annex (style H8)..................................................................23

Annex B (normative or informative): Title of annex (style H8)..................................................................23

B.1 First clause of the annex (style H1)........................................................................................................B.1.1 First subdivided clause of the annex (style H2).................................................................................................

Annex <X> (informative): Authors & contributors (style H8)...................................................................24

Annex <Y> (informative): Bibliography (style H8)......................................................................................24

Annex <Z> (informative): Change History (style H8).................................................................................24

History (style H1)..............................................................................................................................................

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)5

Release #2

<PAGE BREAK>

Intellectual Property Rights (style H1)

Essential patents

IPRs essential or potentially essential to the present document may have been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web server (https://ipr.etsi.org).

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document.

Trademarks

The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners. ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.

Foreword (style H1)

This unnumbered clause is mandatory and shall appear just after the IPR clause.

Replace all <parameters> with the appropriate text.

This Group Specification (GS) has been produced by ETSI Industry Specification Group <long ISGname> (<short ISGname>).

Multi-part documentsThe following block is required in the case of multi-part deliverables.

the <common element of the title> is the same for all parts;

the <part element of the title> differs from part to part; and if appropriate;

the <sub-part element of the title> differs from sub-part to sub-part.

For more details see clause 2.5 of the ETSI Drafting Rules (EDRs).

The best solution for maintaining the structure of series is to have a detailed list of all parts and subparts mentioned in one of the parts (usually it is part 1).

If you choose this solution, the following text has to be mentioned in all of the other parts and sub-parts:

The present document is part <i> of a multi-part deliverable. Full details of the entire series can be found in part [x] [Bookmark reference].

Modal verbs terminology (style H1)

This unnumbered clause is a mandatory informative element and shall appear just after the "Foreword".

In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).

"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)6

Release #2

Executive summary (style H1)

This unnumbered clause, if present, appears after the "Modal verbs terminology" and before the "Introduction". It is an optional informative element and shall not contain requirements.

The "Executive summary" is used, if required, to summarize the ETSI deliverable. It contains enough information for the readers to become acquainted with the full document without reading it. It is usually one page or shorter.

Introduction (style H1)

This unnumbered clause, if present, appears just before the "Scope". It is an optional informative element and shall not contain requirements.

CLAUSE NUMBERING STARTS HEREAFTER.

Automatic numbering may be used in ETSI deliverables but it is highly recommended to use sequence numbering.For more details see clause 2.12.1.1 and 6.9.2 of the ETSI Drafting Rules (EDRs).

Editor’s note: Describe here: How can NFV APIs conformance be beneficial for NFV interoperability. Still, conformance is necessary but not always sufficient for interoperability. Therefore, this work is complementary with NFV TST 007 [REF].

Editor’s note: Explain here the goals of the methodology such as testing both “syntactical” and “semantic” conformance of the operations – which means checking on the actual result of the operations, to leverage any possible degree of automation (generation of the test suite out of OpenAPIs as well as entirely automated test execution supported), openness towards any organizations and especially Open Source initiatives.

<PAGE BREAK>

1 Scope (style H1)

This clause numbered 1 shall start on a new page. More details can be found in clause 2.9 of the EDRs.

The Scope shall not contain requirements. Forms of expression such as the following should be used:

Scope of API conformance is the functionality test in an automated way for ETSI NFV APIs as defined by SOL WG.

Editor’s note: Insert references of all APIs in scope

Editor’s note: correct and expand.

The goal of the present document is to specify the means to test conformance of NFV implementations with specific interfaces as in the following NFV specifications: NFV SOL 002 [REF] for the Ve-Vnfm reference point, NFV SOL 003 [REF] for the Or-Vnfm reference point and NFV SOL 005 [REF] for the Os-ma-nfvo reference point.

Each deliverable specifies a set of interfaces built on the on the RESTful approach and meant to be used over the HTTP protocol. The aim of the present document is to define the methodologies and the procedures to test conformance of the exchanged HTTP payloads and the implementation of required actions for one or more of the available interfaces within a reference point.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)7

Release #2

2 References (style H1)

This clause numbered 2 appears just after the "Scope". It is a required element.

The following text block applies. More details can be found in clause 2.10 of the EDRs.

2.1 Normative references (style H2)

Clause 2.1 only shall contain normative (essential) references which are cited in the document itself. These references have to be publicly available and in English.

Legal acts can never be used as normative references

References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.

Referenced documents which are not found to be publicly available in the expected location might be found at https://docbox.etsi.org/Reference.

NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long term validity.

The following referenced documents are necessary for the application of the present document.

Use the EX style, enclose the number in square brackets and separate it from the title with a tab (you may use sequence fields for automatically numbering references, see clause 6.9.2: "Sequence numbering") (see example).

EXAMPLE:

[1][tab] ETSI GS NFV-SOL 003 v2.4.1: " Network Functions Virtualisation (NFV) Release 2; Protocols and Data Models; RESTful protocols specification for the Or-Vnfm Reference Point".

2.2 Informative references (style H2)

Clause 2.2 shall only contain informative references, which are cited in the document itself.

References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.

NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long term validity.

The following referenced documents are not necessary for the application of the present document but they assist the user with regard to a particular subject area.

[1] ETSI EG 202 237: “Methods for Testing and Specification (MTS); Internet Protocol Testing (IPT); Generic approach to interoperability testing”.

[2] ETSI EG 202 568: “Methods for Testing and Specification (MTS); Internet Protocol Testing (IPT); Testing: Methodology and Framework”.

[3] ISO/IEC 9646-1: “Information Technology - Open Systems Interconnection – Conformance Testing Methodology and Framework - Part 1: General concepts”.

[4] Robot Framework: http://robotframework.org

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)8

Release #2

3 Definitions, symbols and abbreviations (style H1)

Delete from the above heading the word(s) which is/are not applicable, (see clause 2.11 of EDRs).

Definitions and abbreviations extracted from ETSI deliverables can be useful when drafting documents and can be consulted via the Terms and Definitions Interactive Database (TEDDI) (https://webapp.etsi.org/Teddi/).

3.1 Definitions (style H2)

Clause numbering depends on applicability.

A definition shall not take the form of, or contain, a requirement.

The form of a definition shall be such that it can replace the term in context. Additional information shall be given only in the form of examples or notes (see below).

The terms and definitions shall be presented in alphabetical order.

The following text block applies. More details can be found in clause 2.11.1 of the EDRs.

For the purposes of the present document, the [following] terms and definitions [given in ... and the following] apply:

Definition format Use the Normal style.

The term shall be in bold, and shall start with a lower case letter (unless it is always rendered with a leading capital) followed by a colon, one space, and the definition starting with a lower case letter and no ending full-stop.

<defined term>: <definition>

EXAMPLE: text used to clarify abstract rules by applying them literally

NOTE: This may contain additional information.

Test Descriptions: Set of information required to define/run the API conformance test and to realize the verdict for the API conformance test.

3.2 Symbols (style H2)

Symbols should be ordered alphabetically. Clause numbering depends on applicability.

The following text block applies. More details can be found in clause 2.11.2 of the EDRs.

For the purposes of the present document, the [following] symbols [given in ... and the following] apply:

Symbol format Use the EW style and separate this from the definition with a tab. Use the EX style for the last term.

<1st symbol> [tab]<1st Explanation> (style EW)<2nd symbol> [tab]<2nd Explanation> (style EW)<3rd symbol> [tab]<3rd Explanation> (style EX)

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)9

Release #2

3.3 Abbreviations (style H2)

Abbreviations should be ordered alphabetically. Clause numbering depends on applicability.

The following text block applies. More details can be found in clause 2.11.2 of the EDRs.

For the purposes of the present document, the [following] abbreviations [given in ... and the following] apply:

Abbreviation format Use the EW style and separate this from the definition with a tab. Use the EX style for the last term.

<1st ACRONYM> [tab]<Explanation> (style EW)<2nd ACRONYM> [tab]<Explanation> (style EW)<3rd ACRONYM> [tab]<Explanation> (style EX)

4 Methodology (style H1)

4.1 General Editor’s note: explain the relationship between this WI (conformance) and Plugtest, interoperability in general (maybe in scope)

Editor’s note: Quote or reference NFV TST 002 definition of conformance

The purpose of general conformance testing is to determine to what extent a single implementation of a particular standard conforms to the individual requirements of that standard. (as defined in ETSI EDR [REF]).

The important factors which characterize conformance testing are as follows:

the System or Implementation Under Test (SUT or IUT) defines the boundaries (open interfaces) for testing; the conformance test system is a specialized tool (system) built for the purpose of testing and on which specific

test scripts can be run;  the SUT comes from a single supplier (or, at least, a single product line);  the tests are executed by a dedicated test system that has full control of the SUT and the ability to observe all

communications from the SUT;  the tests are performed at open standardized interfaces that are not (usually) accessible to a normal user (i.e.

they are specified at the protocol level);  the tests are specified at the detailed protocol level and are not usually based on functionality as experienced

by a user. the tests verify response or related request operation from SUT.

Editor’s note: Insert other relevant documents and general definitions

4.2 System under test Editor’s note: generic description of producer and consumer under test. Introduce the terminology

The system under test is identified by an implementation of the function under test producing or consuming the API under test e.g. in the case of the Or-vnfm reference point the function under test may be either a NFVO implementation or a NVFM implementation.

The function shall be tested in isolation with respect to other functional blocks in a NFV platform, to guarantee that the outcomes of the conformance tests are not result of interoperability issues with other components.

Editor’s note: there is a corner case when the SUT is going to ask the TestSys some info in order to respond (e.g. VNFM with NFVO). In this case the SUT needs to be instructed not to use the initiator of the request as a real component.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)10

Test Environment Test system

FUT

AUT

API Consumer 

Notification Endpoint 

API exchangesAPI notificationsTest interface

Release #2

4.3 Test configurationsEditor’s note: discussion on the isolation of the FUT. Test environment needed?

Editor’s note: Here are described the test configurations, i.e. function under tests, test system and its components and their reciprocal connections. All the test configurations are described and assigned a unique identifier, to be later on referenced.

4.3.1 General In accordance with Sect. 1, the scope of the present document is to define a testing methodology and test suite for both the conformant protocol exchange (i.e. valid serialization and order of messages) and the initialization or execution of the functionalities mandated for each protocol operation, including the conformant management of internal state.

In order to enable the FUT to correctly execute the operations mandated the FUT shall be tested while being executed in a test environment (TSTENV) which provides all the functional elements needed for the correct outcome of the operation.

NOTE: For example, to correctly execute an instantiation a VNFM requires to be executed in a test environment which provides a VIM and NFVI plus the NFVO to grant the operation.

The test system shall provide the implementation of an API Consumer and a Notification Endpoint for the API Under Test (AUT). Moreover, the test configuration may contain observation interfaces between the Test System and the FUT or any other functional block which is part of the test environment. The specification of the mentioned observation interfaces is out of the scope of the present document.

Stimuli to the FUT shall be injected by the Test System via the AUT only.

Conformance checks on the status and outcome of the operations triggered by the protocol shall be verified by the Test System by means of:

read operations issued via the AUT or reception of notifications on the Notification Endpoint exposed by the test system or other test interfaces to support triggers or verifications.

Fig X: Generic SUT configuration

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)11

Test Environment Test system

VNF/EM

AUT

API Consumer 

Notification Endpoint 

API exchangesAPI notificationsTest interface

Test Environment Test system

VNFM

AUT

API Consumer 

Notification Endpoint 

API exchangesAPI notificationsTest interface

Release #2

The test configurations specified in the following clauses fulfil the requirements stated below for the different FUTs and AUTs in scope of the present specification.

4.3.2 Config_prod_VEThe configuration config_prod_VE shall be implemented to test APIs which are produced by FUTs in a VNF or EM. The test environment of the VNF/EM is the NFVI where the it is executed.

Figure 0-1:Configuration for tests of APIs with a FUTs as Producer run in a VNF/EM.

4.3.3 Config_prod_VNFM

The configuration config_prod_VNFM shall be implemented to test APIs produced by FUTs which implement a VNFM. The test environment of the virtual element is the NFVI where the VE is executed.

IMG TBD

Figure 0-2: Configuration for tests of APIs with VNFM as Producer.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)12

Test Environment Test system

NFVO

AUT

API Consumer 

Notification Endpoint 

API exchangesAPI notificationsTest interface

Release #2

4.3.4 Config_prod_NFVOThe configuration config_prod_NFVO shall be implemented to test APIs produced by FUTs which implement a NFVO. The test environment of the virtual element is an NFV platform providing VNFM, VIM and NFVI.

Figure 0-3: Configuration for tests of APIs with NFVO as Producer.

4.5 Generic test description

4.5.1 General The machine readable language used for the test descriptions is the Robot Framework [4].

Test Descriptions specify the correction of the information required to run “API conformance test”. Test Descriptions are defined per test, and Table 4.5.1-1 shows the elements of the Test Description.

Editor’s NOTE: The table below may need further review.

Elements Notes

Test ID The Test ID identifies uniquely the test. In order to ensure the uniqueness of the ID, it follows a naming convention, which the Test ID is the clause number of this document referring to its Test Descriptions defined.

Test Objective The test objective clearly indicates which requirement is intended to be tested in the test. This part eases the understanding of the Test Description behaviour. This also eases the identification of the requirements, which were used as a basis for the test.

Pre-conditions The initial conditions define in which initial state the SUT has to be to apply the actual Test Description. In the process of the test, when the execution of the initial condition does not succeed, it leads to the assignment of an Inconclusive verdict. This fields includes typically other Test ID, and its test brief descriptions.

Reference In the reference row, the Test Description writer indicates, in which clauses of SOL specification, the requirement is expressed. This information is critical, because it justifies the existence and the behaviour of the Test Description.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)13

Release #2

The reference row may refer to several clauses. The reference to the base standard actually is precise enough to enable the Test Description reader to identify quickly and precisely the requirement.

Config ID The identifier of the applicable test configuration. A test configuration defines how the test system connects to the SUT.

Applicability

Editor’s NOTE: the name of elements might be reconsidered in particular change to ICS.

The identifier(s) of the applicability. See clause X.X

Editor’s NOTE: The clause number for the ICS will be updated accordingly.

Post-conditions Definition of the events that the SUT is expected to perform or shall not perform, according to the base standard and following the correct execution of the actions in the expected behaviour below.

Expected behaviour Definition of the events, which are parts of the Test Description objective, and the SUT are expected to perform in order to conform to the base specification.

Editor’s NOTE: the format of this row is FFS. E.g. test purpose and test cases, the way of describing.

Pass /fail criteria In the corresponding test, Pass or Fail verdicts can be assigned there.

Editor’s NOTE: how and in which row describe this context is FFS.

Table 4.5.1-1. Test Description entries

4.5.2 Scope of the tests Editor’s note: examples: content of the messages to be tested, logic, etc

On top of the clause 4.5.1 General, the behaviours of the SUT shall be tested in term of:

- expected handling of the messages received from Test System- expected actions inside the SUT

These above two items shall be checked by information retrieved by other operations done by Test Systems e.g. GET or appropriate notification messages. These check shall be based on the referred specifications including all error cases.

- The expected correct response messages sending back to Test system.

The above item shall be checked by analysing received messages by Test Systems. These check shall be based on the referred specifications including all error cases.

Editor’s NOTE: FFS for list of the targeted clauses/sections/statements from SOL specifications in term of test.

Editor’s NOTE: FFS for the considerations on the level of details for the test.

4.5.3 Verification Test system verifies URIs, HTTP headers and payloads of responses from SUT.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)14

Release #2

Figure X-X: verification procedure.

Editor’s note: A verification method and steps for each items is FFS.

4.5.4 Test templates

4.6 Test Suite StructureEach test case in the scope of this document will be uniquely identifies by the fields identified in Table XX:

FieldDescription

Example

Reference point

TBD TBD

Interface TBD TBD

Operation TBD TBD

Function under test

TBD TBD Editor’s note: (producer/consumer)

Expected outcome

TBD TBD

Editor’s note: Specify a table with all possible identifiers for the fields below

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)15

Release #2

5 Os-Ma-Nfvo Reference Point

5.1 General

5.2 ICS

5.2 Test configuration

5.3 Interface #1 (TBD)

5.3.1 Operation #1

5.3.2 Operation #2

5.4 Interface #2

6 Ve-Vnfm Reference Point

6.1 General

6.2 Test configuration

6.3

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)16

Release #2

7 Or-Vnfm Reference Point

7.1 General 7.1.1 General characteristics of the reference point

Or-Vnfm is the reference point between NFVO and VNFM, and the RESTful protocol for that reference point is specified in ETSI NFV SOL003 [1]. This clause describes the basic principles of the SOL 003 which shall be considered in API conformance test.

From the API conformance test point of view, Or-Vnfm reference point has below characteristics:

Asynchronous API and State handling by notifications

Due to the varieties of the reasons, e.g. consuming time of process handling, “VNF Lifecycle Management interface” defined in SOL003 [1] clause 5 is designed as asynchronous way, i.e. API designed is based on the combination of the specific methods with task resources and state handling by related attributes signalled via notification messages.

For example, the operation “Instantiate VNF” is triggered by POST message with task “Instantiate VNF” sent from NFVO to VNFM , then VNFM sends “provisional” response “202 accepted” back to NFVO, which means “accepted request” (SOL003 [1] clause5.4.4). Based on the subscriptions registered in VNFM, the appropriate notification message will be sent toward NFVO with the state of the processing in the VNFM at that time. After finishing the process of instantiating VNF in the VNFM, the notification message with the notification endpoint resource which is set to “operationState=COMPLETED” shall be sent (SOL003 [1] clause5.4.18 and clause 5.4.20), which mean that the VNF has been instantiated.

NOTE: The “notification” may be optional.

Linked resource

According to the SOL003 [1] clause 4.3.3.1, “link” pointing to the resources which may be fetched by API consumer, may be used on the reference point. It is used in order to avoid “big” response messages and to avoid repeating information request messages. Therefore the validity of such linked resources are also verified by test, as same as resources in the response message itself.

Request and response data values

According to the SOL003 [1] clause 5.5, a VNF LCM request shall use the values set in the valid VNFD, i.e. InstantiateVnfRequest, ScaleVnfRequest, ScaleVnfToLevelRequest, ChangeVnfFlavourRequest, TerminateVnfRequest HealVnfRequst, OperateVnfRequest and ChangeExtVnfConnectivityRequest.

Test system verify if each attribute of the response body use the values set in the valid VNFD.Editor’s NOTE: Whether doing this test in this version of TST010 activities is FFS. Even if this is not supported, at least it should be indicated as “this is exception for this version”.

Grant exchange

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)17

Release #2

According to SOL003 [1] clause 9.1, the VNFM executes Grant Request before certain LCM operations are executed for the direct mode. The test system verify if each attribute of Grant request body uses the values set in the valid VNFD.Editor’s NOTE: Whether doing this test in this version of TST010 activities is FFS. Even if this is not supported, at least it should be indicated as “this is exception for this version”.

Editor’s note: Whether Test system should response to continue testing VNF LCM operation is FFS, because VNFM expects Grant Response which comply with SOL003 [1] specification. For that purpose, the Test system should have the test cases, one is sending a positive grant response (code 201) and another is sending negative grant response (code 403) upon receives the Grant request from SUT..

VIM registration

According to SOL003 [1] clause 4.4.1.6, the VNFM uses VIM registration. As this registration may change according to a registration update in SOL003 [1] clause Annex C.5, the test system import the latest VIM registration before testing.

7.1.2 Basic behaviours of the API producer/consumer and verification steps

Expected behaviours of the API producer specified in the SOL specifications targeted by the tests, i.e. SOL003 [1] are categorized as follows:

7.1.2.1 Producer sends event triggered notifications based on consumer subscriptions

Typical example for this case is “Fault management”. The producer sends a notification to the consumer with mechanism “filter” and “callback”. The Test System checks if the notification aligns to the conditions of the subscription.

Editor’s NOTE: verification procedures is TBD.

7.1.2.2 Producer sends periodical notifications based on the consumer subscriptions

Typical example for this case is “Performance management”. The producer sends a periodic notification to the consumer as requested in the subscription. The Test System checks if the notification aligns to the conditions of the subscription.

Editor’s NOTE: verification procedures is TBD.

7.1.2.3 Producer executes the requested API operation

Typical example for this case is “Life Cycle Management”. Based on the concept “Asynchronous API”, the producer starts the operation by receiving a request message from the consumer, and sends some notifications back based on the operational status. The Test System checks if the procedure is correct, i.e. start with a request, send notifications with an appropriate status, complete operation and correct status of the resources and status of the producer.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)18

Release #2

Figure 7.1.2.3-1: verification procedure

Editor’s NOTE: The alignments to the texts as shown below are needed, in particular message numbers, alt indicated part, name of AUT, etc.

The basic method of verification follow the steps as illustrated in figure 7.1.2.3-1:

1) Test System sends request message#1. Before sending message#1, required features/functionalities are executed, e.g. API authentication, etc. Other tests as pre-condition are executed beforehand if required. Test System sets the data and attributes of the request body with correct/incorrect cardinalities and values.

Editor’s NOTE: What the “correct and/or incorrect” values of the attributes shall be set is FFS due to lack of consumer side specification in SOL003 [1].

2) Upon receiving the request message#1 from Test System, the SUT shall handle this request according to the requirements defined in SOL specifications, i.e. validation of the attributes in the HTTP body whether the attributes were correctly set according to the cardinalities, validation of the values of the attributes according to the requirements defined in description part of the data type tables in the SOL specifications. After executing the expected procedures in the SUT, the response message#2 are sent to the Test System with an appropriate response code and HTTP body based on the result of the SUT executed procedure.

3) Test System verifies the received response message#2, in terms of expected response code, expected HTTP header, HTTP body, cardinality of the responded data, fulfilment of the requirements described in the description field, and verification aspects shown below, etc.

a. Verification aspects: 1-C, 2-A, 2-C, 2-D, 2-E, 2-F, 2-G

Editor’s NOTE: How to verify the “correct and/or incorrect” values of the attributes is FFS due to lack of consumer side specification in SOL003 [1].

4) Optionally, the API may have more than 1 message exchange in particular for messages sent from the SUT to the Test System. In that case this step follows the step2.

5) After expected API messages exchange, The Test System may have some verification aspects which require retrieval of information from the SUT, e.g. linked resources embedded in the response message. In that case, the Test System may invoke a query message to the SUT.

6) As reaction to the step5, the SUT may send additional messages to the Test System. In this case, the verification follows step3.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)19

Release #2

7) In addition to the verification done by step3 and step6 (optional), The Test System verifies the expected post conditions for the SUT.

a. Verification aspects: 2-H

Editor’s NOTE: How to verify the “correct and/or incorrect” values of the attributes is FFS due to lack of consumer side specification in SOL003 [1].

7.1.2.4 Consumer fetches the files/package info

Typical example for this case is fetching files of a VNF package. The Test System checks if the file or package info is correctly received.

Editor’s NOTE: verification procedures is TBD.

7.1.3 VerificationSince the scope of the API conformance test includes syntax and semantics verifications, this clause defines the methodology of the verification executed by the Test System.

7.1.3.1 Common verification aspects

The common verification aspects checked by the Test System are defined in Table 7.1.3.1-1. See SOL specifications for more details of the requirements. i.e. SOL003 [1].

Verification aspects Notes

1-A: URI handling If the SUT receives an invalid URI by HTTP method with either an invalid path or invalid values for the resources, the SUT shall return the appropriate error response code to the Test System according to the SOL specification. The Test System tests the support of HTTP over TLS version 1.2.

1-B: Attribute selectors

Expected attributes or sub-resources shall be sent from the SUT according to attribute selector parameter by a query operation from the Test System. The Test System tests all of the operations with parameters if those work correctly.

1-C: HTTP header fields

Expected HTTP headers shall be sent from the SUT. The Test System tests all of the operations with parameters if those work correctly.

1-D: Error reporting In case of some types of errors occurred in the SUT, the error response with type ProblemDetails data are returned to the Test System. The Test System tests all attributes of the ProblemDetails regarding the error experienced by the SUT.

1-E: API authorization

For the case of making an API call, OAuth2.0 is required for authentication between API consumer and API producer. For the case of the notification, OAuth2.0 or HTTP Basic authentication is used. From the API producer point of view, if the passed access token is invalid, API producer should return appropriate errors, e.g. 401 unauthorized, 403 insufficient scope.

Table 7.1.3.1-1 Common verification aspects

7.1.3.2 Verification aspects for individual API

The verification aspects for individual APIs checked by the Test System are defined in Table 7.1.3.2-1.

Verification aspects Notes

2-A: Asynchronous API

As described in clause 4.5.2, one of the typical behaviours of Asynchronous API, the requested behaviours are not completed by a response message but as a combination of granting exchanges and notifications.Editor’s NOTE: How to deal with this kinds of APIs, i.e. target of the test and/or

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)20

Release #2

granularity of the test descriptions for this kinds of API is FFS. For example, if the test is separated way as VNF LCM test, Granting test and Notification test, or combined test includes all of aspects VNF LCM, Granting and Notification. For both cases, handled values shall be related in order to work correctly.

2-B: API, message, functionality those are optional for consumer but mandatory for procedure

Even if an API/message/functionality is optional to support for the consumer, it might be mandatory to support it for the procedure. To cover such cases, the Test Descriptions include optional API/message/functionality as in the scope of the test in terms of expected behaviours and verdicts for the validation of the functionality implemented by the SUT. Notification for the use of state handling is optional for the consumer, but if it is required, then the SUT shall support this. For the test description, the VNF LCM test includes testing the Notification behaviour.

2-C: Method support The Test System verifies all of the Supported HTTP method, e.g. all types of the specified method with all types of the specified response codes.

2-D: Attributes set in the request body

The Test System generates request messages with body covering the following cardinality aspects:

- Data set in the request body, e.g. if not set when cardinality is 1, treat as error.- Attributes in the data, e.g. if not set according to the specified cardinality.

2-E: Value validation of the attribute

If the values set in the attributes of the data provided in the request body are not matching to the conditions in the description part, the SUT shall treat them as error. Test System checks this aspect by sending exhaustive sets of combinations of the values. The Test System handles values coming from previous procedures as pre-conditions. For example, it is to be checked if the identifier provided by the SUT is unique according to the scope of each identifier.

Editor’s NOTE: how to deal with this check is FFS.

2-F: Response messages validation

The Test System checks the syntax of the HTTP header and body, the cardinalities and the conditions described in the description part.

Editor’s NOTE: how to deal with the check of the returned data is FFS.

2-G: Resource validation

The Test System checks if the related resources in the SUT are in the expected status after having received the response messages, by checking the notifications from the SUT or by initiating a GET request towards the SUT.

2-H: Any other dedicated conditions

The Test System verifies any other API dedicated conditions if exists. Those are described in API dedicated clauses. See SOL specifications more details of the conditions.

Table 7.1.3.2-1 Common verification aspects for each API

7.1.3.3 Verification aspects not considered

The current version of this document and test code are not considering the aspects for verification listed in table 7.1.3.3-1.

Editor’s NOTE: Need to final check if v2.5.1 is supported or note.

Verification aspects not considered by this version of document and test code

Notes

Version exchange This verification aspect is not supported in SOL specification v2.4.1 and introduced from version v2.5.1.The API producer shall support receiving and interpreting the "Version" HTTP header. The API producer shall include in the response the "Version" HTTP header signalling the used API version, including the "impl" version parameter if available. If the "impl"

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)21

Release #2

version parameter has been omitted in the request, the API producer shall use the combination of MAJOR, MINOR and PATCH as requested and the highest supported value for the "impl_version" field of the "impl" version parameter for that combination, if available.

Large query handling This verification aspect is not supported in SOL specification v2.4.1 and introduced from version v2.5.1.The API producer shall support to return an error response or to return the result in a paged manner to handle the case that a response to a query (GET request) will become so large that the response will adversely affect performance. Test System checks the SUT’s behaviours by sending a query which requires a quite large response.

Table 7.1.3.3-1 Exception of the verification aspects for API conformance test

7.2 Test configuration

7.3

From clause 4, the technical content of the deliverable shall be inserted. Each clause shall have a title which shall be placed after its number separated by a tab.

A clause can have numbered subdivisions, e.g. 5.1, 5.2, 5.1.1, 5.1.2, etc. This process of subdivisions may be continued as far as the sixth heading level (e.g. 6.5.4.3.2.1).

For numbering issues, see clause 2.12.1 of the EDRs.

Use the Heading style appropriate to its level (see clause 6.1, table 8 of the EDRs).

Separate the number of the heading and the text of the heading with a tab.

Treat clause titles as normal text (i.e. no additional capitalization), but no full stop.

IF YOUR DOCUMENT CONTAINS FIGURES, TABLES AND/OR MATHEMATICAL FORMULAE, THIS IS THE WAY THEY SHALL BE PREPARED:

- Figures shall be prepared in accordance to clauses 7.5.2 and/or 7.1 of the EDRs. The figure number and title shall be below the figure. An explicit figure title is optional. See clause 5.1.5 if you need to include notes to figures.

Use TF style for the figure number and title.

Use FL style on the paragraph which contains the figure itself.

If applicable, the figure number is followed by a colon, a space and the table title.

Maximum width for figures is 17 cm and maximum height is 22 cm.

Should you wish to number figures automatically, "Sequence numbering and bookmarking" (see clause 6.9.2 of the EDRs) is highly recommended.

- Tables shall be prepared in accordance to clause 5.2 of the EDRs. If you have tables in your document, the table number and title shall be above the table itself. An explicit table title is optional.

Use TH style for the table number and title.

If applicable, the table number is followed by a colon, a space and the table title.

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)22

Release #2

Should you wish to number tables automatically, "Sequence numbering and bookmarking" (see clause 6.9.2 of the EDRs) is highly recommended.

- Mathematical formulae shall be prepared in accordance to clause 5.3 of the EDRs.

Numbers given to the clauses, tables, figures and mathematical formulae of an annex shall be preceded by the letter designating that annex followed by a full-stop (e.g. figure B.1, table C.4). The numbering shall start afresh with each annex. A single annex shall be designated "Annex A".

NOTE: For an easy application of the ETSI styles download "the ETSI styles toolbar" from editHelp! website.

AnnexesEach annex shall start on a new page (insert a page break between annexes A and B, annexes B and C, etc.).

Numbers given to the clauses, tables, figures and mathematical formulae of an annex shall be preceded by the letter designating that annex followed by a full-stop. The numbering shall start afresh with each annex. A single annex shall be designated "Annex A".

Clauses in annex A shall be designated "A.1", "A.2", "A.3", etc. (further details in clause 2.12.1 of the EDRs).

Use the Heading 8 style. Insert a line break ("shift" + "enter") between the colon and the title.

For all annex clause headings use the appropriate Heading styles, starting from Heading 1, e.g. for clause A.1 use Heading 1, for clause A.1.1 use Heading 2. (See clause 6.1, table 8 of the EDRs).

<PAGE BREAK>

Annex A (normative or informative):Title of annex (style H8)

<Text>.

<PAGE BREAK>

Annex B (normative or informative):Title of annex (style H8)

B.1 First clause of the annex (style H1)

B.1.1 First subdivided clause of the annex (style H2)

<Text>.

<PAGE BREAK>

Annex <X> (informative):Authors & contributors (style H8)

The annex entitled "Authors & contributors" is optional. When present it describes the list of persons and companies that contributed to the elaboration of the present Group Specification.

The following people have contributed to the present document:

Rapporteur:Title, Firstname, Lastname, company

Other contributors:

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)23

Release #2

Title, Firstname, Lastname, company

<PAGE BREAK>

Annex <Y> (informative):Bibliography (style H8)

This optional informative clause shall start on a new page and be the last annex of an ETSI deliverable or the last but one if followed by the "Change history/Change request history" annex, if any. The Bibliography shall not contain requirements.

The Bibliography identifies additional reading material not mentioned within the document. Those publications might or might not be publicly available (no check is made by the ETSI Secretariat).

The Bibliography shall include list of standards, books, articles, or other sources on a particular subject which are not referenced in the document.

The Bibliography shall not include references mentioned in the deliverable.

Use Heading 8 style for the "Bibliography" annex, see clause 2.13 for examples.

For the listed material use the Normal style or bulleted lists (e.g. B1+), do not use numbered references.

<Publication>: "<Title>".

OR

<Publication>: "<Title>".

<PAGE BREAK>

Annex <Z> (informative):Change History (style H8)

The informative clause shall start on a new page and be the last annex before the "History" clause. It is an optional, informative element and shall not contain requirements.

If it is desired to keep a detailed record of the changes implemented in a new version it is recommended that this is done by inserting a "Change history/Change request" annex, see clause 2.15.

It shall be presented as a table. Apply the normal style format for tables (see clause 5.2.2 of the EDRs).

Date Version Information about changes

October 2011 1.1.1 First publication of the GS after approval(30 September - 2 October 2011; Prague)

February 2012 1.2.1

Implemented Change Requests:Error message information clarificationsRevised error message informationupdate of figure 3 clause 9.2Version 1.2.1 prepared by the Rapporteur

July 2013 1.3.1 Version 1.3.1 prepared by the Rapporteur

<PAGE BREAK>

History (style H1)

Document history

<Version> <Date> <Milestone>

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)24

Release #2

0.0.1 Skeleton

0.0.2 Apr 8 2018 NFVTST(18)000021r2: TST010 - Scope and inputs for the methodology

0.0.3 July 11, 2018 NFVTST(18)000059r1: Test configuration introduction

0.0.4 Nov 9, 2018 NFVTST(18)000106r1 - TST010 -Test Language SelectionNFVTST(18)000124r3 - TST010 Clarify Asynchronous API of Or-Vnfm reference pointNFVTST(18)000134 - TST010 Clarify linked resources of Or-Vnfm reference point

0.0.5 Nov 21 2018 NFVTST(18)000084r1 - TST010 Add verification

0.0.6 Nov 27 2018 NFVTST(18)000129r4 – TST010 Test DescriptionsNFVTST(18)000142r1 – TST010 Scope of the tests

0.0.7 Dec 17 2018 NFVTST(18)000167r4 - TST010 4.5.1 Addition of the ConfigID and Applicability in the entries of the Test DescriptionsNFVTST(18)000168r4 - TST010 7.1.1 Or-Vnfm Addition of the 3 general characteristics of the reference pointNFVTST(18)000169r3 - TST010 7.1.2 Or-Vnfm Basic behaviours of the API and Verification

Latest changes made on 2017-07-10

ETSI

ETSI GS NFV-TST010 V0.0.7 (2018-12)25