functional and scale performance tests using zopkio

Functional and scale performance tests using zopkio

{ ‘Event’: ‘PyCON HK 2015’,‘Name’: ‘Marcelo Araujo’,‘Email’: ‘[email protected]’

Agenda

● About me.● Why we test software?● Zopkio test framework.● Q&A.

PyCON HK 2015 Marcelo Araujo <[email protected]>

About me


● Name: Marcelo Araujo.● Bachelor in Computer System Networking and Telecommunications.● Post-Degree in Quality on Software Engineering. ● Python consumer since 2007.● FreeBSD Developer since 2007 (ports and *kernel).● DevOps at Gandi since 2015.

From To

Why we test software?


● Assure what we created does what it supposed to do.● The behavior with one user could be different with thousand users.● Users could do something unexpected or not planned.● Many different devices, browsers, operating systems and so forth.● We want assure a good software quality.● Validation: Are we doing the right job?● Verification: Are we doing the job right?

A typical application infra.


App Server Database

Your application starts like this!

Then, becomes like this.


App Server Master DB

Slave DB

Cache

Storage

Balancer/Proxy

API Servers

Storage Replication

VMs

Services

A question!


As a DevOps, It is not only about test software anymore!

How can I test the infrastructure?

Zopkio test framework.


● Made by Linkedin.● It is a test framework built to support at scale performance and functional

testing.● It can be installed via pip.● Latest code at: https://github.com/linkedin/Zopkio● There is documentation.● Enough pydoc for every each class.● They have some examples of code.● Under Apache 2.0 License.● They are open for pull requests.● Active development.



● Zopkio provides the ability to write tests that combine performance and functional testing across a distributed system.

● Writing tests using Zopkio should be nearly as simple as writing tests in xUnit or Nose.

● Zopkio strongly depends on:

○ Naarad: A system analysis tool that parses and plots time series data.

○ Paramiko: Python implementation of SSHv2 protocol.



● What kind of tests can we perform with zopkio?○ Functional.

■ Ensure every function produces its expected outcome.○ Load.

■ Understand the behaviour under a specific and expected load.○ Stress.

■ What is the behaviour beyond the normal expected load.○ Performance.

■ Determine the speed or effectiveness of an software or service.



● What we need from a functional and performance test framework?○ ASSERT: Must be able to test a condition!○ Get data vs time and compare with what we expected.○ Plot graphs!○ Provisioning before run any test. ○ Parallelize tests.○ Control the sequence of tests.○ Collect logs.○ Parse logs.○ Cleanup environment after tests.○ An interface with test result.○ Be able to debug! Our test is a software too.

My PYHK test lab.


bare-metal machine

FreeBSD OS

Hypervisor Type 2 (bhyve)

VM1 (BSD) VM2 (BSD) VM3 (Linux) VM4 (BSD)

Zopkio

Switch

Server

What tests I will perform in this lab?


● VM1 and VM2: Check how many files inside /etc/ and apply an ASSERT.

● VM1 and VM2: ICMP to example.com and check the lantency.

● VM3 and VM4: TCP Server/Client and make a connection between them.

The Zopkio framework.


● Zopkio has a main script that takes several optional arguments:○ Run only named tests to help debug broken tests.○ Configs overrides at execution time.○ Log level.○ Console log level.

● To run my test suite:

araujo@coxinha:/z/hk# zopkio run.py --nopassword --log-level DEBUG

Main run.


1 +-- 2 lines: import os------------------------------------------------------------------ 3 test = { 4 "deployment_code": os.path.join( 5 os.path.dirname( 6 os.path.abspath(__file__)), "deployment.py"), 7 "test_code": [ 8 os.path.join( 9 os.path.dirname( 10 os.path.abspath(__file__)), "test_suites/machine1_etc.py"), 11 os.path.join( 12 os.path.dirname( 13 os.path.abspath(__file__)), "test_suites/machine1_ping.py"), 14 os.path.join( 15 os.path.dirname( 16 os.path.abspath(__file__)), "test_suites/connection.py"), 17 os.path.join( 18 os.path.dirname( 19 os.path.abspath(__file__)), "test_suites/machine2_ping.py"), 20 os.path.join( 21 os.path.dirname( 22 os.path.abspath(__file__)), "test_suites/machine2_etc.py")],

.|-- run.py

23 "perf_code": os.path.join( 24 os.path.dirname(os.path.abspath(__file__)), "perf.py"), 25 "configs_directory": os.path.join( 26 os.path.dirname(os.path.abspath(__file__)), "configs/") 27 }

The config file.


1 { 2 "comment": "This is the first test", 3 "tip": "How many files on my /etc", 4 "pyhk_install": "/tmp/pyhk/", 5 "pyhk_exec": "scripts/scripts.tar", 6 "pyhk_cmd": "sh run.sh", 7 "ping_cmd": "bash ping-csv.sh --add-timestamp example.com >/tmp/pyhk/ping-output.csv", 8 "tcp_server_cmd": "python server.py &", 9 "tcp_client_cmd": "python client.py >connection.log", 10 "show_all_interations":true, 11 "verify_after_each_test":true, 12 "pyhk_hostname1": "10.0.1.22", 13 "pyhk_hostname2": "10.0.1.12" 14 }

|-- configs| `-- pyhk.json

The dynamic config.


1 +-- 4 lines: !/usr/bin/env python----------------------------------------------- 5 LOGS_DIRECTORY = "/tmp/pyhk/collected_logs/" 6 OUTPUT_DIRECTORY = "/tmp/pyhk/results/" 7 +-- 2 lines: -------------------------------------------------------------------------- 9 def machine_logs(): 10 return { 11 "client1": [os.path.join("/tmp/pyhk/", "connection.log")], 12 "machine1": [os.path.join("/tmp/pyhk/", "run.log")], 13 "machine2": [os.path.join("/tmp/pyhk/", "run.log")], 14 } 15 +-- 2 lines: -------------------------------------------------------------------------- 17 def naarad_logs(): 18 return { 19 "machine1": [os.path.join("/tmp/pyhk/", "run.csv")], 20 "machine2": [os.path.join("/tmp/pyhk/", "run.csv")], 21 "machine1": [os.path.join("/tmp/pyhk/", "ping-output.csv")], 22 "machine2": [os.path.join("/tmp/pyhk/", "ping-output.csv")], 23 } 24 +-- 2 lines: ----------------------------------------------------------------------------- 26 def naarad_config(): 27 return os.path.join( 28 os.path.dirname(os.path.abspath(__file__)), "naarad.cfg")

.|-- perf.py

The deployment file.


● It is in charge of provisioning machines.● A deployment file can contain four functions:

○ setup_suite() - Will run before any of tests.○ setup() - Will run before each test.○ teardown() - Will run if setup() ran successfully.○ teardown_suite() - Will run if setup_suite() ran successfully.

The deployer.


1 +-- 3 lines: !/usr/bin/env python------------------------------------------------------------------------------- 4 import zopkio.adhoc_deployer as adhoc_deployer 5 import zopkio.runtime as runtime 6 +-- 6 lines: pyhk_deployer = None-------------------------------------------------------------------------- 12 def setup_suite(): 13 +-- 2 lines: print "==> Starting tests for PYHK."--------------------------------------------------------- 15 runtime.set_user('root', '') 16 +-- 4 lines: global pyhk_deployer--------------------------------------------------------------------------- 20 tcp_server = adhoc_deployer.SSHDeployer( 21 "server", 22 {'executable': runtime.get_active_config('pyhk_exec'), 23 'extract': True, 24 'start_command': runtime.get_active_config('tcp_server_cmd'), 25 'stop_command': "ps ax | grep '[p]ython server' | awk '{print $1}' | xargs kill -9"}) 26 runtime.set_deployer("server", tcp_server) 27 28 tcp_server.install("server1", 29 {"hostname": "10.0.1.23", 30 "install_path": runtime.get_active_config('pyhk_install')}) 31

32 tcp_client = adhoc_deployer.SSHDeployer( 33 "client", 34 {'executable': runtime.get_active_config('pyhk_exec'), 35 'extract': True, 36 'start_command': runtime.get_active_config('tcp_client_cmd')}) 37 runtime.set_deployer("client", tcp_client) 38 39 tcp_client.install("client1", 40 {"hostname": "10.0.1.24", 41 "install_path": runtime.get_active_config('pyhk_install')}) 42 +-- 20 lines: pyhk_deployer = adhoc_deployer.SSHDeployer(-------- 62 def setup(): 63 for process in tcp_server.get_processes(): 64 tcp_server.start(process.unique_id) 65 66 def teardown_suite(): 67 for process in tcp_server.get_processes(): 68 tcp_server.undeploy(process.unique_id)

.|-- deployment.py

An test file.


1 +-- 6 lines: !/usr/bin/env python----------------------------------------------- 7 import zopkio.runtime as runtime 8 import zopkio.test_utils as testutilities 9 10 test_phase = 3 11 +-- 2 lines: ------------------------------------------------------------------------ 13 def test_ping_host1(): 14 print "==> ping example.com (machine1)" 15 pyhk_deployer = runtime.get_deployer("pyhk") 16 17 pyhk_deployer.start( 18 "machine1", 19 configs={ 20 "start_command": runtime.get_active_config('ping_cmd'), 21 "sync": True 22 }) 23 +-- 2 lines: ------------------------------------------------------------------------

25 def validate_ping_host1(): 26 ''' 27 Send 10 icmp to example.com . 28 ''' 29 hostname1_log_file = os.path.join( 30 perf.LOGS_DIRECTORY, "machine1-ping-output.csv") 31 hostname1_logs = testutilities.get_log_for_test( 32 "test_ping_host1", hostname1_log_file, "12:00:00") 33 34 # Number of icmp packages 35 size_p = len(hostname1_logs.split(',')) / 2 36 37 assert size_p == 10, "Less than 10 replies on host1"

|-- test_suites| |-- machine1_ping.py

Plot graphs.


1 [machine1-icmp] 2 infile=/tmp/pyhk/collected_logs/machine1-ping-output.csv 3 columns=sequence latency 4 sep=, 5 qps.sla=mean<500 6 latency.sla=mean<400 p50<300 7 8 [machine2-icmp] 9 infile=/tmp/pyhk/collected_logs/machine2-ping-output.csv 10 columns=sequence latency 11 sep=, 12 13 [GRAPH] 14 graphs=machine1-icmp.sequence,machine1-icmp.latency machine2-icmp.sequence,machine2-icmp.latency

.|-- naarad.cfg

The Zopkio GUI.


Let's take a look on my machine

The conclusion is:


We need to be able to test the infrastructure where I will run my application.

References.


● Zopkio:○ https://github.com/linkedin/Zopkio

● Naarad:○ https://github.com/linkedin/naarad/

● Code for PYHK Test Lab:○ https://github.com/araujobsd/pyhk2015

● Slides at:○ TBD


Thank you!

[email protected]@FreeBSD.org

mailto:[email protected]




functional and scale performance tests using zopkio

Internet