Testbed and Test ProjectsOverview

Pharos, VNF Deployment Functional Testing (Functest), Benchmarking (QTIP)

Morgan Richomme, Trevor Cooper, Palani Chinnakannan, Wenjing Chu, Fu Qiao

February 24, 2015

Test & Performance Sub-groupObjectives and Scope

• OPNFV is an original project mainly based on the integration of Open Source upstream projects

• OPNFV main goal is to define an Open Source baseline for NFV

• Testing is a key activity for achieving OPNFV objectives

Test & Performance Sub-groupProjects and Scope• TSC approved projects: Pharos, Functest, Qtip, Characterize vSwitch

Performance

• Testing Scope

– Installation/upgrade (deployment)– Representative & automated VNF functional test suites– Performance: Performance targets and baselines to be defined and

determined (e.g. compare bare-metal)– Robustness, HA, disaster recovery– NVFI/VNF interoperability with legacy network architecture– Security

• Testing with proprietary …

– VNFs, Test frameworks (SW infrastructure and hw tools)

4

Project Relationships

Test - Top Level Projects

Need a platform to test?Need guidelines to setup your own test-bed?Want to know what has been already tested?

Want to reproduce a test-suite on your test-bed?

Need functional test suite?Want to add functional tests to existing test suite?

Need tools and tests to benchmark your OPNFV platform?

Want to benchmark my VNF on OPNFV platform?

Pharos

Functest

Qtip

6

PHAROS

7

Pharos Overview

• The Pharos project deals with the creation of a distributed and federated NFV test capability that will be hosted by a number of companies in the OPNFV community

• The goals include

– Tracking availability/capacity of community platforms– Specifying capability of different community platforms– Managing timeslots and tools to perform tests– Sharing the results and the best practices– Supporting test campaigns of community projects (e.g. functional

testing project, Qtip, BGS, Oscar,…)– Automated test environment– Integration with CI (test-bed independent)

8

Pharos Main DeliverablesDeliverables Comments Release

Operational testing capabilities for the communities

Intel & E/// test-beds already available 1

Test-bed governance / code of conduct

Document if you want to set up and open your own tesbed for the community

1

Hardware recommendation Document to help you building your first OPNFV platform 1

Test-bed Installation guide 1

Test-bed Administration guide 2

Best Practices / FAQ On the wiki of FAQ section to answer to any question related to the test-beds

1

Tooling references & description Wiki page referencing the tooling and scripts for OPNFV testing (linked to Qtip project)

2

Inventory tool Tool to give an overview of the test-bed 2

Dashboard Tool to centralize the test campaign, inform any member of the federation and encourage reproduction of test campaigns to consolidate results

1 or 2

9

Pharos Demos

1. Remote test-bed access

– Access environment being used by BGS Project– Intel hosted environment … https://

wiki.opnfv.org/get_started/intel_hosting– Discuss tools / methods for remote collaboration

2. Software Fast-path Service Quality Metrics

– Project focused on measurement of Telco Traffic and Performance KPIs in the fastpath

– Demo uses RFC2889 (Frame Error Filtering) to show MAC/PHY error and discard statistics

China Mobile OPNFV Testlab

Open source Platform Open Source Testlab Provide at least 4 PODs for OPNFV testing Provide multiple hardware support Provide internet access for lab access Provide testing and measurement equipments

Partners:

11

Open discussion 1: Test Dashboard

• Goals of the Dashboard

– Inform (who is doing what, when)– Simplify reproduction of test campaigns– Identify contacts / escalations– Facilitate best practices (tools, scripts, …)– Compare results (same campaign on different configurations)– Identify covered (and non covered) domains

• Dashboard Contributors

– Community test-bed managers– Testers

12

Release 1Functest

13

Rls 1 - Assumptions and Test Steps

Assumptions

• CI deploys to Pharos compliant environment (hw infrastructure)

• Jenkins owned by CI (incl. installation scripts for master and slave modes)

• Test-bed runs Jenkins slave connected to CI environment (not for Rls 1)

• BGS constrains test configuration

Test steps

• Deploy NW

• Spin up VNF (loop-back)

• Verify functionality of VNF -> ping test

• Return result (pass/fail)

14

Rls 1 - Functest Plan• Task breakdown with milestones, leaders / owners

• Identify a simple VNF to spin up

• Identify Tempest test-suites (Open Stack)

– 5 tests identified– Deliver script to CI for invoking tests

• Setup Tempest

– Method to modify and reuse test-cases (as needed)

– Create new tests (as needed)

• Setup Jenkins in test-environment (integrated with CI after Rls 1)

• Documentation

15

Rls 1 – Test Artifacts

• Prerequisites / hardware requirements / …

• Installation and user guides

• Test suite

• Community test-bed dashboard

• Tool for maintaining testbed inventory/capability

16

Community Needs

17

Community needs / organization

• How to collect community need regarding testing?

– Mailing list, dedicated wiki page, dashboard?– Shared test-bed resources – tools to manage?

• How to manage requests for specific project requirements?

– Specific hardware• Bare metal• Vendor specific• Hardware Acceleration / Network cards

– Specific configurations (stack selection, Multi sites, VSwitch tuning..)– Specific test equipment (Spirent, IXIA, RobotFramework, SIPP, CLIF, ….)

18

Community needs / organization cont.

• How to share the results?

– Dashboard

• How to encourage test campaign reproduction?

– Any test campaign shall be automated and reproduced on any community platform

– Reference test suites, tooling and communicate on test campaign

• How to share best practices?

19

Thank You

20

Governance

• Keep it simple

• Each company may already have some internal rules, do not create new ones

• FAQ with a text sample shall be produced

21

Governance proposalThis laboratory is a controlled environment made available to OPNFV community members by XXX for improving OPNFV solution and testing interoperability with other manufacturers’ equipment for the OPNFV community.

This testing gives community members the advantages of working together to test interoperability.

Access to the laboratory is granted by the laboratory administrator. The procedure to get credentials, the documentation (architecture, user guide), the testbed inventory (hardware, bare metal capabilities, available tooling) and the local policy (physical access, specific code of conduct) shall be described in a dedicated wiki page.

Members are encouraged to share as many results as possible. The results of community tests would be collected into the Pharos dashboard. The Pharos project will disseminate the test results so any testbed of the federation could replay and consolidate test campaigns with different combination of hardware/software.

Specific tests by members using proprietary components to help improving the overall quality of the OPNFV solution are possible. Members participating in such interoperability event, dry run, practice run, certification wave, wave rehearsals must not (either before, during or after the scheduled development lab use or interoperability event, dry run, practice run, certification wave, wave rehearsal or while performing daily tasks at any of XXX test development laboratories):- Access any other members’ equipment or other facilities at XXX without expressed written permission from an authorized XXX’ project manager.- Discuss any other members information and/or interoperability problems with third parties or the press without XXX’ prior written authorization.- Reveal other members’ test results, either directly or implicitly, to his/her own company, third parties or the press.- Share or copy confidential information or intellectual property other than in accordance with the applicable OPNFV policy.

Testing resources reservation shall be managed at the testbed level by the testbed administrator. The TSC may re-prioritize test campaigns if necessary.