![Page 1: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/1.jpg)
Gym:A VNF Testing Framework -
Design and Prototype InsightsProf. Christian Rothenberg
Raphael Vicente Rosa
Claudio Bertoldo
Jan-2017
University of Campinas (UNICAMP), Brazil
in technical collaboration with Ericsson Research
![Page 2: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/2.jpg)
Imagine… an olympic athlete● Training
○ Indices
■ Blood sugar levels
■ Hemoglobine %
■ Transpiration
○ When?..
■ Start
■ Steady (long run)
■ Final Sprint
○ Why?
■ High performance
■ Sport Analytics
■ Athlete Profile
● Continuous improvement
● Competing
○ Goals:
■ Deliver the best
■ Conscious actions and body reactions
○ But…
■ Weather conditions may interfere
■ Over... [stretching | heating ]
○ In the end…
■ Break records
■ Keep competing
■ No injuries
![Page 3: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/3.jpg)
We have VNFs, execution environments… and services!
Extracted from: http://www.etsi.org/deliver/etsi gs/NFV-IFA/001 099/001/01.01.01 60/gs NFV-IFA001v010101p.pdfhttp://www.etsi.org/deliver/etsi gs/NFV-EVE/001 099/004/01.01.01 60/gs NFV-EVE004v010101p.pdf
![Page 4: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/4.jpg)
Let’s not forget: some Actors… and requirements
![Page 5: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/5.jpg)
Bring your VNF to the Gym (... and vice-versa)● Old Story: VNF Benchmark-as-a-Service (VBaaS)
● Nowadays: Gym - VNF Testing Framework
○ Modular architecture with stand-alone programmable components
○ Simple messaging system among them following generic RPC guidelines
○ Extensible set of tools and associated metrics
○ Programmability of tests in dynamic compositions of modules
○ And respective flexible methods for the interpretation/processing of
evaluations' output
![Page 6: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/6.jpg)
Why Gym?● VNF Developers
○ Develop tests for profiling VNFs performance aligned with agile Continuous
Development/Integration of DevOps methodologies, speeding their time-to-market
● Service Providers
○ Enhance offered services QoS with tested deployable scenarios (e.g., varying workloads in multiple
sites), containing transparent sets of required VNF metrics
● Cloud/Infrastructure Providers
○ Via VNF testing in their execution environments, would increase reliability with compliance
methodologies (e.g., energy consumption)
![Page 7: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/7.jpg)
Design Principles for Experimentation Methodologies➢ Comparability:
○ output of tests must be simple to understand and process, in a human-readable format, able to be
easily usable (e.g., imported by big data applications).
➢ Repeatability:○ testing setup must be defined by a handful/flexible design model, able to be interpreted and executed
by the testing platform repeatedly with customization.
➢ Interoperability:○ tests should be able to be ported in different environments with lightweight technologies providing
the means for it.
➢ Configurability:○ open interfaces and extensible messaging models among testing components must exist to provide
flexibility when composing tests descriptions and configurations.
![Page 8: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/8.jpg)
Architecture
● Core components as microservices○ e.g., containers
● Flexible messaging○ e.g., JSON and REST APIs
● Dynamic couple tools○ Extensible classes as tools’
interfaces● Freedom for design
○ Outlines and Profiles
![Page 9: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/9.jpg)
Some Concepts★ Outline:
specifies how to test a VNF that may be specific to it or applicable to several VNF types. It includes structural (e.g., agents/monitors) and functional sketches with variable parameters (e.g., probers/listeners properties), used as inputs by Gym to perform VNFs tests
★ Profile:
is composed by the outputs of an Outline execution, and represents a mapping between virtualized resources (e.g., vCPU, memory) and performance (e.g., throughput, latency between in/out or ports) at a given environment. It abstracts a VNF allocation with desired resources to deliver a given (predictable/measured) performance quality
![Page 10: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/10.jpg)
I want to benchmark a vIMS, now what?..➔ Service Provider Question:
In which cloud would be cheaper to deploy my vIMS?
➔ Why?
I want to parameterize my service, so I can:
● Extract the vIMS service performance opportunity cost
● Understand its quality, and extract some SLA
● Offer/Negotiate it with other providers
![Page 11: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/11.jpg)
1. Select some cloud providers: e.g., Microsoft, Google, Amazon
2. Get a VNF: vIMS (Project Clearwater)
3. Have a stimulus: sipp prober
4. Have a monitor for the environment: linux host listener
5. Target some metrics: CPU vs. Calls/s
6. OK, write your Outline, submit it to Player wait for the target Profilea. then, select your metrics, and…
b. apply algorithms for analysis,
c. make graphics, etc
Testing with Gym...
![Page 12: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/12.jpg)
Not so fast… I mean, we missed some steps ➔ Deploying Gym’s components
◆ How?
● Gym’s components are python modules (VM, container, baremetal)
◆ Where?
● Depends on the testing topology, e.g.,
○ Are your customers in Australia?
○ Should you group agents: one-to-many, many-to-many?
➔ How to compose the Outline?
◆ Gym provides a flexible syntax
● Under development: “It would be interesting to have some intents”
◆ Need to understand what you want
● i.e., agents/monitors requirements and structure (e.g., probers/listeners and some params)
● e.g., An specific prober metric vs. another listener metric
![Page 13: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/13.jpg)
ScenariovIMS components decomposed in OpenStack VMs (different flavors)
Monitors installed in each vIMS VM -> listening host (cpu, memory, disk i/o, network)
Agent with sipp prober in VM
Manager/Player in another VM
All components in the same network
![Page 14: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/14.jpg)
Partial Results: on-going activity in our lab
![Page 15: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/15.jpg)
What could we discuss here?❖ Consistency:
➢ Such VNF when deployed in a tested execution environment will deliver the given performance
described in its extracted Profile ?; specially when tested and put in production, making use of
multiple virtualization technologies, for instance
❖ Stability:
➢ VNF testing measurements need to present consistent results when applied over different scenarios.
So, do testing descriptions transparently handle service/resource definitions and metrics of VNFs
placed in heterogeneous execution environments?
❖ Goodness:
➢ A VNF might be tested with different allocated resources and stimulus, unlike its possible
production environment. Crucially, how good testing results, and associated stimulus, correspond
to VNF measured performance when running in execution environments under actual workloads?
![Page 16: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/16.jpg)
Gym: by the birds’ eyes➢ We leave VNF specific testing tools for users (VNF Developers) development
○ They know better a VNF state machine testing procedure (e.g., sipp)
○ e.g., it is possible to code a prober to interface NetFPGA card or DPDK pktgen
➢ Metrics and their interpretation are open field for research
○ e.g., docker containers presents dozens of features among cpu, memory, disk, etc
➢ Framework in early childhood
○ Many tests, errors, debugging going on
○ Improving design of models, messages, behaviors
➢ Important related work and foundations
○ OPNFV (yardstick, bottlenecks, etc), ToDD, many more (please, tell us about yours)
○ IETF BMWG, IRTF NFVRG, ETSI NFV (TST)
![Page 17: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/17.jpg)
In a not so far future...➔ Gym will be open source
➔ Open repository of VNF profiles
➔ New probers/listeners
➔ New scenarios coming from SDN/NFV outcomes for 5G realization
![Page 18: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/18.jpg)
...with some questions...Can we outperform continuous monitoring? When? How?
Are we modeling tests correctly?
Are tools able to express actual service workloads and heterogeneous behaviors?
“Trust, but verify” or verify, then trust?
Maybe it depends on our findings around consistency, stability
and goodness. For sure, we cannot understand and evolve
something, if not first measuring it
![Page 19: A VNF Testing Framework - Gym: Design and Prototype Insightsmaterials.dagstuhl.de/files/17/17032/17032.ChristianEsteve Rothenb… · Enhance offered services QoS with tested deployable](https://reader034.vdocument.in/reader034/viewer/2022042303/5eced3d2194eb40ca3645f0c/html5/thumbnails/19.jpg)
Thanks!