yueqing zhang a20306020 arthur clouet a20319763 oluseyi … · or kamailio, ip address :...

SIPP PERFORMANCE BENCHMARK

YUEQING ZHANG A20306020

ARTHUR CLOUET A20319763

OLUSEYI S AWOTAYO A20313761

FINAL REPORT

DECEMBER 2014

SIPp Performance Benchmark IIT

ITM 546 Project Report 2

Abstract The goal of our project is to use SIP technology to test the performance

of Voice Over IP automatically through a script we wrote. This SIP Performance

Benchmark project measures the maximum SIP throughput and rate of sending

invite/call with no errors that a SIP proxy can support. We have learnt to

operate the SIPp tool and also studied the RFC drafts that described the

method of testing. We have installed a SIP server and a SIP client in the lab

and measured its key performance metrics Session Capacity and Session Rate

that are defined in the IETF drafts. We will also re-test the values of these key

performance metrics that were obtained by previous students. The achieved

results will be used to write a short paper for submission to an academic

journal of IEEE.



Table of Contents

Abstract .................................................................................................................... 2

Introduction ............................................................................................................... 5

I. Project Goals ......................................................................................................... 6

II. Test Bed ............................................................................................................... 7 1. Architectures ..................................................................................................................................................... 7

a. Physical Architecture ........................................................................................................................................ 7 b. Logical Architecture .......................................................................................................................................... 8

2. Installation and configuration of SIPp tool ........................................................................................... 8 a. About SIPp Tool ................................................................................................................................................... 8 b. Installation and Configuration of SIPp Tool .......................................................................................... 9

3. Installation and configuration of Asterisk ............................................................................................. 9 a. About Asterisk ...................................................................................................................................................... 9 b. Installation and configuration of Asterisk ............................................................................................ 10

4. Installation and configuration of Kamailio ........................................................................................ 13 a) About Kamailio : ............................................................................................................................................... 13 a. Installation and configuration of Kamailio .......................................................................................... 14

III. Script ................................................................................................................ 18 1. Algorithm ......................................................................................................................................................... 18 2. Script .................................................................................................................................................................. 19 3. Improvements ................................................................................................................................................ 20

IV. Test results and analysis .................................................................................... 22 1. No DUT - No RTP ........................................................................................................................................... 24

a) 5,000 calls ............................................................................................................................................................ 24 b) 50,000 calls ......................................................................................................................................................... 26 c) Unlimited calls ................................................................................................................................................... 27

2. No DUT Yes RTP ............................................................................................................................................ 29 3. No DUT - No/Yes RTP - Comparison..................................................................................................... 31 4. Yes DUT No RTP (Asterisk) ...................................................................................................................... 32 5. Yes DUT Yes RTP (Asterisk) - Can Reinvite = NO ............................................................................ 33 6. Yes DUT - No/Yes RTP - Comparison ................................................................................................... 35 7. Yes RTP - No/Yes DUT - Comparison ................................................................................................... 36

V. Project Continuation ......................................................................................... 37

Conclusion ................................................................................................................ 38

References ................................................................................................................ 39

Appendix A - Previous Results ................................................................................... 40 NO DUT NO RTP ..................................................................................................................................................... 40 NO DUT YES RTP .................................................................................................................................................... 41 YES DUT NO RTP (Asterisk) ............................................................................................................................. 41

Appendix B - Asterisk Configuration Files .................................................................. 43 extensions.conf ....................................................................................................................................................... 43 sip.conf ....................................................................................................................................................................... 43



Introduction With the exponential growth of Internet, a new technology called Voice

over Internet Protocol (VoIP) appeared. In the beginning telephone industry

(using PSTN) ignored this evolution that meant huge modification of their

infrastructure and business model. Eventually, telephone providers adopted

this technology offering new services and applications to both public and

private sectors. One of the protocols to create VoIP services is called Session

Initiation Protocol (SIP) : this application-layer signaling tool is used to create,

modify and terminate sessions between two or more user agents (UA).

SIP has been adopted by many sectors of the telecommunications

industry: Enterprise IP PBX’es and SIP Session Border Controllers, as well as the

Emergency Services IP network (ESINet) for the delivery of Emergency calls,

Business VoIP phone services, SIP-to-Phone-PC-Telephone for cheap

International call rates are some important examples. Many commercial

systems and solutions based on SIP are available today to meet the industry’s

requirements. As a result, there is a strong need for a vendor-neutral

benchmarking methodology to allow different SIP servers to be meaningfully

compared one with the other. The goal of this program is to develop such

benchmarks and to design systems to collect the benchmarks we define. The

outcome of the program will include creation of two IETF drafts that describe

the benchmarks and the methodology to be used for their collection. The

outcome will also be a tool that collects the benchmarks and that is built in

accordance with the methodology we describe. The Methodology and

Terminology draft documents are available on the references page of this site.

After a brief explanation about our test bed configuration, we will

explain step-by-step how our script works and then provide an analysis for each

result.



I. Project Goals

This project is a continuation of work previously made by former IIT

students. After reviewing what has been done, we needed to learn how to

operate the SIPp tool and also studied the RFC drafts that describe the method

of testing. After installing and configuring our test bed, our goal was to

measure the Breaking Point and Session Attempts along CPU and Memory usage

that are defined in the IETF drafts.

In order to run those tests, we had to write a script based on IETF algorithms.

Once we were able to automate the testing, we obtained results for the

following scenarios :

No DUT No RTP : numerous calls are sent between the UAC (user-agent

client) and the UAS (user-agent server) without media.

No DUT Yes RTP : media exchange is added to the previous scenario.

Yes DUT No RTP : a proxy (Asterisk) is now filtering calls between our

UAC and UAS.

Yes DUT Yes RTP : media exchange is added to the previous scenario.

The results will be used to write a short paper for submission to an academic

journal of IEEE.



II. Test Bed

1. Architectures

a. Physical Architecture

Physical Architecture of SIPp Performance Test Bed

Five elements constitute this test bed where a switch is the link

between the four others. There are an UAC (User-Agent Client), an UAS (User-

Agent Server), a proxy (Asterisk or Kamailio) and finally a fourth computer

running Wireshark during every tests.

Computers specifications



b. Logical Architecture

Our test bed is configured on a Private LAN. The Dell 2724 Switch (IP

address : 10.200.0.3) is receiving and transmitting every packet from our UAC

(IP address : 10.200.0.248), UAS (IP address : 10.200.250) and a proxy (Asterisk

or Kamailio, IP address : 10.200.0.245).

We also configured our switch to have a mirroring port (Port 21) so we

can run Wireshark separately on an independent computer. Every packet (sent

and received) by the UAS, UAC or proxy is mirrored to this port. This

configuration is avoiding us to run Wireshark on any computer running SIPp or

Asterisk/Kamailio, thus, there is no performance interference.

2. Installation and configuration of SIPp tool

a. About SIPp Tool SIPp is a free Open Source test tool / traffic generator for the SIP

protocol. It includes a few basic SipStone user agent scenarios (UAC and UAS)

and establishes and releases multiple calls with the INVITE and BYE methods. It

can also read custom XML scenario files describing from very simple to complex

call flows. It features the dynamic display of statistics about running tests (call

rate, round trip delay, and message statistics), periodic CSV statistics dumps,

TCP and UDP over multiple sockets or multiplexed with retransmission

management and dynamically adjustable call rates.

SIPp can also send media (RTP) traffic through RTP echo and RTP / pcap

replay. Media can be audio or video.

SIPp can be used to test various real SIP equipment like SIP proxies,

B2BUAs, SIP media servers, SIP/x gateways, SIP PBX, ... It is also very useful to

emulate thousands of user agents calling your SIP system.



SIPp tool is a perfect tool for our benchmark needs. Using various

scenarios, we are able to establish numerous calls between a client and a

server as well as measure the exact performance of our test bed.

b. Installation and Configuration of SIPp Tool • Install required packages.

$ sudo apt-get install build-essential libncurses5-dev\

• Download, extract and compile SIPp $ wget

"http://downloads.sourceforge.net/project/sipp/sipp/3.2/sip

p.svn.tar.gz?r=&ts=1314783436&use_mirror=puzzle" -O

sipp.svn.tar.gz

$ tar -xzf sipp.svn.tar.gz

$ cd sipp.svn

$ make

3. Installation and configuration of Asterisk

a. About Asterisk

Asterisk is an open source framework for building communications

applications. Asterisk turns an ordinary computer into a communications

server. Asterisk powers IP PBX systems, VoIP gateways, conference servers and

other custom solutions. It is used by small businesses, large businesses, call

centers, carriers and government agencies, worldwide. Asterisk is free and

open source.

Originally designed for Linux, Asterisk also runs on a variety of different

operating systems including FreeBSD, Mac OSX , CentOS and Solaris.

The Asterisk software includes many features available in proprietary PBX

systems:

▪ Voice mail



▪ Conference calling

▪ Interactive voice response (phone menus)

▪ Automatic call distribution

Asterisk is not a SIP Proxy. A SIP proxy handles call control on behalf of other

User Agents (UA) and usually does not maintain state during a call and

therefore is never the endpoint of a call.

Asterisk, as a server, is a SIP Registrar and location server and also acts as a

user agent endpoint. Basically it is a softphone software program to operate

call through the internet which can receive REGISTER message and places the

information it receives in those requests into the location service for the

domain it handles.

If it is 'controlling' or relaying a call from a SIP phone to another SIP phone, it

simply acts as an endpoint User Agent (UA) to the originating call leg and then

creates a new call to the receiving phone. Therefore, it stays "in the middle of

the call," maintaining state and controlling, and optionally bridging, each

remote endpoints.

Asterisk can thus be described best as a "Back-to-Back User Agent" (B2BUA),

which is also consistent with the use of the term "PBX". Asterisk is a powerful

media gateway.

b. Installation and configuration of Asterisk • CentOS Updates Update your CentOS 6 Server for any possible unimplemented updates.

$ yum update -y

• Disabling SELinux (optional)



You can use any text editor (VIM ) to commit this change. Go to /etc/selinux/config and change SELINUX=enforcing to SELINUX=disabled. This can also be done by using command line :

$ sed -i s/SELINUX=enforcing/SELINUX=disabled/g

/etc/selinux/config

• Reboot Once the aforementioned change is committed and the file is updated, reboot the system using :

$ reboot

• Installation of Basic Dependencies Asterisk 11.0.0 requires some prerequisite dependencies. Here is the command line to install them:

$ yum install -y make wget openssl-devel ncurses-devel

newt-devel libxml2-devel kernel-devel gcc gcc-c++ sqlite-

devel

• Downloading Your Asterisk Source Code Move to directory /usr/src by given command :

$ cd /usr/src/

and then download the Source Code tar balls using these commands (one by one at a time) :

$ wget http://downloads.asterisk.org/pub/telephony/dahdi-

linux-complete/dahdi-linux-complete-current.tar.gz $ wget

http://downloads.asterisk.org/pub/telephony/libpri/libpri-

1.4-current.tar.gz $ wget

http://downloads.asterisk.org/pub/telephony/asterisk/asteri

sk-11-current.tar.gz

• Extraction of Downloaded Files Extract the downloaded tar balls to their corresponding directories using:



$ tar zxvf dahdi-linux-complete* $ tar zxvf libpri* $ tar zxvf asterisk*

• DAHDI Installation DAHDI (Digium Asterisk Hardware Device Interface) can be installed using the command line :

$ cd /usr/src/dahdi-linux-complete* $ make && make install && make config

• LibPRI Installation In order to enable your BRI, PRI and QSIG based hardware, you will be needing PRI Library or LibPRI. You can install these libraries using :

$ cd /usr/src/libpri* $ make && make install

• Changing Asterisk Directory Now you have to move back to the Asterisk Installation Directory :

$ cd /usr/src/asterisk*

• Running Configure Script for Asterisk At this point, you need to know your CentOS 6 Architecture (32 or 64 Bit). In many cases you are aware of it. In case you are not, try this command :

$ uname -a

For 64 Bit, system will respond with something like: 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Then, for 64 Bit :

$ ./configure --libdir=/usr/lib64 && make menuselect &&

make && make install

• Installing Sample Files Sample files are great resources. Install Sample Files using :



$ make samples

Once done, add the Asterisk Install Script in directory /etc/init.d/ using :

$ make config

• Starting DAHDI To start DAHDI Device Drivers, use:

$ service dahdi start

• Start Asterisk Finally, start Asterisk:

$ service asterisk start

Do your stuff by connecting to the Asterisk Console:

$ asterisk -c

The following step is to configure two files : sip.conf and extensions.conf. The two configurations are given in the appendix B.

4. Installation and configuration of Kamailio

a) About Kamailio Kamailio (former openser) is an open source sip server released under

gpl, able to handle thousands of call setups per second. Among features :

asynchronous tcp, udp and sctp, secure communication via tls for voip (voice,

video); ipv4 and ipv6; simple instant messaging and presence with embedded

xcap server and msrp relay; it does least cost routing; load balancing; routing

fail-over; accounting, authentication and authorization; support for many

backend systems such as mysql, postgres, oracle, radius, ldap, redis,

cassandra; xmlrpc control interface, snmp monitoring. It can be used to build

large voip servicing platforms or to scale up sip-to-pstn gateways, pbx systems

or media servers like asterisk.



Kamailio is developed in C language, runs on UNIX environments, has a modular

architectures -plenty of modules available- and has a worldwide distributed

development environment..

As a SIP server the Kamailio architecture fulfills its role:

▪ SIP registrar -record registry for specific users-

▪ SIP proxy server

▪ SIP application server -telephony services-

Contrary to the Asterisk server, the Kamailio architecture is not a back-to-back

user agent. A back-to-back user agent operates between both end-points of a

phone call or communications session and divides the communication channel

into two call legs and mediates all SIP signaling between both ends of the call,

from call establishment to termination. Typically A B2BUA may provide the

following functions:

▪ Call management (billing, automatic call disconnection, call transfer,

etc.)

▪ Network interworking (perhaps with protocol adaptation)

▪ Hiding of network internals (private addresses, network topology(Star

Topology), etc.)

The Kamailio server only handles signaling and is no more nor less than a proxy.

a. Installation and configuration of Kamailio • Installing Dependencies :

$ disable selinux : vim /etc/sysconfig/selinux $ yum update $ yum install git-core $ yum install gcc $ yum install flex $ yum install bison $ yum install make $ yum install bison pcre-devel libpcap-devel



$ yum install mysql mysql-server mysql-devel php-mysql $ yum install php php-devel php-gd php-pecl-memcache php-

pspell php-snmp php-xml php-xmlrpc php-intl php-dba php-

process

• Configuring MySQL and setting Password :

$ chkconfig --levels 235 mysqld on $ service mysqld start $ mysqladmin -u root password <password>

• Getting sources from Git :

$ mkdir -p /usr/local/src/kamailio-4.1 $ cd /usr/local/src/kamailio-4.1 $ git clone --depth 1 git://git.sip-router.org/kamailio

kamailio $ cd kamailio $ git checkout -b 4.1 origin/4.1 $ make cfg

• Edit modules.lst file

$ nano -w modules.lst edit to include_modules= db_mysql

$ make PREFIX="/usr/local/kamailio-devel"

include_modules="db_mysql dialplan" cfg $ make all $ make Q=0 all

• Installing Kamailio

$ make install $ PATH=$PATH:/usr/local/kamailio-devel/sbin $ export PATH

• Edit kamctlrc and uncomment DBENGINE and DB_PATH file to :

$ nano -w /usr/local/kamailio-devel/etc/kamailio/kamctlrc

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; DBENGINE=MYSQL

; ; DB_PATH from /usr/local/etc/kamailio/dbtext/kamailio to

/usr/local/kamailio-devel/share/kamailio/dbtext/kamailio

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;



• Creating database

$ /usr/local/kamailio-devel/sbin/kamdbctl create

Edit the config file kamailio.cfg: $ vim /usr/local/kamailio-devel/etc/kamailio/kamailio.cfg

;;;;;;;;;;;;;;;;;;;;;;;;; #!define WITH_MYSQL ; #!define WITH_AUTH ; #!define WITH_USRLOCDB ; # ----- usrloc params ----- /* enable DB persistency for location entries */

#!ifdef WITH_USRLOCDB

modparam("usrloc", "db_mode", 2) modparam("usrloc",

"db_url","mysql://root:root@localhost/openser") #!endif # ----- auth_db params ----- /* enable the DB based authentication */ #!ifdef WITH_AUTH modparam("auth_db", "calculate_ha1", yes) modparam("auth_db", "password_column", "password") modparam("auth_db", "db_url",

"mysql://root:root@localhost/openser") modparam("auth_db", "load_credentials", "") #!endif

;;;;;;;;;;;;;;;;;;;;;;;;; • Configuring the startup and default file:

Please confirm if kamailio file exists in /etc/init.d/ IF NOT,

$ cp /usr/local/src/kamailio-

4.1/kamailio/pkg/kamailio/rpm/kamailio.init

/etc/init.d/kamailio $ chmod 755 /etc/init.d/kamailio

$ vim /etc/init.d/kamailio

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;KAM=/usr/local/kamailio-devel/sbin/kamailio

;KAMCFG=/usr/local/kamailio-devel/etc/kamailio/kamailio.cfg

;RUN_KAMAILIO=yes

;DAEMON=/usr/local/kamailio-devel/sbin/kamailio

;CFGFILE=/usr/local/kamailio-

devel/etc/kamailio/kamailio.cfg

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;



Please confirm if kamailio file exists in /etc/default/ IF NOT

$ cp /usr/local/src/kamailio-

4.1/kamailio/pkg/kamailio/rpm/kamailio.default

/etc/default/kamailio

$ vim /etc/default/kamailio ;;;;;;;;;;;;;;;;;;;

;RUN_KAMAILIO=yes ;

;;;;;;;;;;;;;;;;;;;

$ mkdir -p /var/run/kamailio

by default, kamailio start with the user= kamailio and group= kamailio

$ adduser --system --shell /bin/false --home

/var/run/kamailio kamailio

If it complains that Kamailio group exists already, then

$ adduser -g kamailio kamailio

set ownership to /var/run/kamailio

$ chown kamailio:kamailio /var/run/kamailio

• Starting Kamailio:

$ service kamailio start $ service kamailio stop

ADDING ACCOUNT "kamctl add <username> <password> <email>" or without email

$ kamctl add <username> <password> <[email protected]>

SIP_DOMAIN environment

$ export SIP_DOMAIN=mysipserver.com



III. Script The purpose of this script is to automate performance testing using the

SIPp tool on various scenarios. We based our work on an algorithm conceived

for an IETF draft : "Methodology for Benchmarking SIP Networking Devices".

1. Algorithm

The main goal of this algorithm is to find the largest value of a SIP session-rate

which the SUT can process without call failure. It is based on an iterative

process until a correct value is found.

In this example, an initial call-rate of 100 calls/second is set. Without

any failure encountered, the call-rate is increased to 200 and then 400. At this

point, several fail calls occur, therefore the call-rate is decreased to 300

calls/second. The value of increase and decrease can be calculated within

different percentages.

When there is not failure, a limit left is set which is the minimum call-rate

achieve without failure. When there is a failure, a limit right is set which is the

maximum call-rate achieve with a failure. The algorithm is then going back and

fourth between those two limits (which are updated after every test) until a

breaking point is reached. In this case, 269 calls/second is the largest value

found before a call failure is observed.



The iterative process stops when a granularity point is reached : this is the

accuracy. It is calculated after every test by 2 different operations depending

if there is a failure or not. The best accuracy possible is 1 : it means that the

algorithm will test every call-rate possible between 2 limits until the most

precise value is reached (ex : Limit 2 is 270, Limit 1 is 269 : then granularity =

1; in the case of a granularity set to 5, the limit 2 could be 274, then, even if

the algorithm stops with a breaking point of 269, a call-rate of 270, 271, 272 or

273 could be reached without any failure).

2. Script

Our script exists in different versions depending on the fact that a proxy is used

or not, or that media (RTP) is exchanged or not.

#!/bin/bash #Declation of values #rate (calls per second) : r #max : maximum call rate before exiting while loop #Limit1 : left limit with no failure #Limit2 : right limit with failure(s) #fail = number of call failed (retrieved from trace_screen log) #Executing the script #on UAC : sudo ./nodut-nortp-uac.sh GRANULARITY TIMEOUT || example : ./nodut-nortp-uac.sh 3 45 #on UAS : sudo ./sipp -sn uas 10.200.0.250 r='100' max='10000' Limit1='0' Limit2='0' let "Accuracy= $1+1" echo "Call-rate = "$r echo "Timeout = "$2 while [ $Accuracy -gt $1 ] && [ $r -lt $max ] do ./sipp -i 10.200.0.248 -sn uac 10.200.0.250 -r $r -timeout $2 -trace_screen #sleeping 5sec to let the computer create and read the file log before continuing



sleep 5 #looking for log file pathlog=$(find . -maxdepth 1 -name '*.log') #looking for the failure value inlog file if [ -f $pathlog ] then Fail_temp=$(grep -w Failed $pathlog | cut -d"|" -f3 | tr -s ' ') Fail=`echo "Number of fail(s) = "$Fail_temp | awk {'print $5'}` sudo mv $pathlog Log/Archives/ fi #if no failure : the left limit with no failure = rate-call if [ $Fail -eq 0 ] then echo "No Fail Detected" Limit1=$r #if no left limit failure determined yet : rate-call is increased by 50% if [ $Limit2 -eq 0 ] then echo "Limit2 = 0" r=$(echo "$r + ($r-$Limit2)/2" | bc) #decimal calculus introduces the use of the bc option Accuracy=$(( r-Limit2 )) #if a left limit failure is already determined : rate-call is increased by 50% else echo "Limit2 ≠ 0" r=$(echo "$r + ($Limit2-$r)/2"| bc) #decimal calculus introduces the use of the bc option Accuracy=$(( Limit2-r )) fi #if Failure(s) is/are detected, rate-call is decreased else Limit2=$r echo "Fail(s) Detected" r=$(echo "$r - ($r-$Limit1)/2" | bc) Accuracy=$(( r-Limit1 )) fi done echo -e "\nTHE BREAKING POINT IS: $r call per s\n"

3. Improvements



Our script could be helpful for the SIPp community, along with the

Asterisk and Kamailio ones. Before publishing it online, a few improvements

could be made. Among them, we should review our method to read and save

the value of failed call(s). We could also save a few other interesting values

from this log : call-rate, session attempted, successful call, duration of test.

Finally, we could have only one script, even if RTP media is exchanged or if a

proxy is used, by using a function that could recognize what kind of situation

the script is running for.

This script allowed us to gain a precious time for our performance test, we are

now going to analyze the results of every scenario we ran.



IV. Test results and analysis

To set up a call, SIPp must be launch by command line on the UAS. On our UAC,

this command is directly located in the script.

./sipp -i 10.200.0.248 -sn uac 10.200.0.250 -r $r -timeout $2 -trace_screen 10.200.0.248 represents the UAC (User Agent Client). 10.200.0.250 represents the UAS (User Agent Server). -r represents the call rate (calls/second). -timeout represents the duration of the test.

In the above screen, a call-rate of 500 calls/second has been set. We can

observe that 15,000 calls have been attempted. In this particular case, there is

no failure, indeed 15,000 calls have been answered by the UAS and then

terminated.



This is the log we are looking for after each test, we are always looking for the

value of failed call. In that particular case there is 0 failed call and 15,000

successful ones.

Description of the parameters in the following tables :

Granularity : also called accuracy. This value is set when launching the script.

It is used to continue or stop the script when this specific accuracy is reached.

The lower the granularity is, the higher the accuracy of the breaking point is.

Call Length (second) : the duration time of each call. This value is set when

launching the script, only when there is no media exchange. The breaking point

is influenced by this value.

Calls or Session Attempted : the number of session attempted. This value is

set when launching the script. It limits the number of call our test bed can

attempt before stopping. This value was set to 5,000 and 50,000 but due to

illogical results, we did not use this value for the following tests.



Breakpoint (calls/s) : the highest value of a SIP session-rate we can achieve

without encountering any fail call.

Duration of Test (second) : also called timeout, this value is set when

launching the script. SIPp will stop sending calls once this time limit is reached.

CPU/MEM (%) UAC/UAS—The CPU and Memory usage when the breaking point is

reached.

1. No DUT - No RTP

This first scenario is designed to make a call between our UAC and our

UAS. The main purpose of running this test is to get the full capacity of our test

bed.

a) 5,000 calls

Granularity

Call Length

(second)

Session Attempts

Breakpoint (calls/second)

CPU/MEM (%)

UAC

CPU/MEM (%)

UAS

1 0 5,000 2768 24.2 / 0.6 37.2 / 0.5

1 2 5,000 3876 28.9 / 0.7 40.5 / 0.8

1 4 5,000 7657 24.9 / 0.6 27.2 / 0.4

1 6 5,000 11916 21.3 / 0.4 29.9 / 0.4

2 0 5,000 3534 37.5 / 1.2 41.8 / 1.6

2 2 5,000 11632 39.9 / 1.4 22.6 / 0.4

2 4 5,000 11916 23.9 / 0.5 29.2 / 0.4

2 6 5,000 11916 24.2 / 0.6 30.6 / 0.5

3 0 5,000 4754 19.6 / 0.3 39.5 / 0.8

3 2 5,000 11916 20.3 / 0.4 33.5 / 0.5

3 4 5,000 11916 23.3 / 0.6 29.9 / 0.4

3 6 5,000 11916 23.9 / 0.5 30.6 / 0.5



In this test, we set the session attempts to 5,000. We ran our script using

3 different values of granularity : 1, 2 and 3. The last configured parameter is

the call-length : 0, 2, 4 and 6 seconds.

Logically, we should have observed a decrease of the breakpoint value. Indeed,

the longer the call-length is, the more the server has to handle call at the same

time before terminating and freeing resources for next calls. In our case, we

observed the opposite with a breakpoint reaching a constant value of 11,916.

In our script, we are stopping the iterative process when a call-rate of 10,000

calls/second is reached, this is why our test stops at this particular value. If we

raise this maximum value, the breaking point could be even higher.

The problem of limiting the number of session attempts is that SIPp will stop

the test as soon as 5,000 calls has been answered and terminated. When a call-

length of 4 seconds is set with a call-rate of 3000 calls/second, the client is

sending 12000 calls to the server in 4 seconds. The server is answering and

terminating the first 5,000 calls and the test stops : the 6,000 following calls

are lost and SIPp does not take into account that information. The script is



then testing for a higher call-rate value, which will result of the same issue and

will eventually run indefinitely.

b) 50,000 calls

Granularity Call

Length (second)

Session Attempts


CPU/MEM (%) UAC

CPU/MEM (%) UAS

1 0 50,000 3345 88 / 2 88.4 / 2.7

1 2 50,000 3956 93.4 / 3.8 96.7 / 3.1

1 4 50,000 3829 92.7 / 3.4 97.1 / 3.1

1 6 50,000 4314 97.2 / 4.5 99.3 / 3.2

2 0 50,000 3566 81.4 / 0.3 89.4 / 2.7

2 2 50,000 4396 88.1 / 0.4 99.7 / 3.2

2 4 50,000 4079 86.2 / 2.5 97.2 / 3.1

2 6 50,000 4675 84.1 / 2.2 99.3 / 3.2

3 0 50,000 3464 80.4 / 0.5 88.7 / 2.7

3 2 50,000 4091 88.2 / 0.9 97.0 / 3.1

3 4 50,000 3904 86.7 / 1.4 97.1 / 3.1

3 6 50,000 4885 89.7 / 2.2 98.6 / 3.2



By raising the sessions attempted to 50,000, we basically reached the same

conclusion as before. The breaking point will get higher and higher because of

the SIPp tool limitation.

Nevertheless, we found a solution to have logical results so we can continue

our project : we are not setting a session attempt limitation anymore. Instead

of that, we are using the SIPp "timeout" argument used to set a specific

duration of test. Indeed, SIPp will send calls during the time we set, and stop

the test when the server will answer and terminate every attempted call.

c) Unlimited calls The following ladder diagram given by Wireshark shows that 3 calls have been

successfully made between the UAC and the UAS :



Granularity Duration of

Test (second) Call Length

(second) Breakpoint

(calls/second) Session Attempts

1 30 9 3941 118,200

1 60 9 3467 207,960

1 90 9 2874 258,570

2 30 9 3875 116,250

2 60 9 3211 192,640

2 90 9 2623 235,887

3 30 9 3965 118,860

3 60 9 3327 199,600

3 90 9 2740 246,430

The test with unlimited calls has been made with 3 different values of

granularity. We also added 3 different durations : 30, 60 and 90 seconds. For

each test, the call-length was set to 9 seconds so we are able to compare our

results when media RTP is flowing (indeed, a call with media approximately

lasts 9 seconds).



We can observe in the above graph that our results are now logical. Indeed, the

longer the duration of test is, the lesser the breakpoint is. We can see in the

results table that the number of sessions attempted is higher in function of the

duration of the test. The server is receiving calls while it still processes the

ones it already answered. Therefore, it can only free resources after a call is

terminated. This is why a failure is observed sooner when the duration of test

is increased, the maximum call-rate without failure will decrease over time.

2. No DUT Yes RTP

The following ladder diagram given by Wireshark shows that 3 calls have been

successfully made between the UAC and the UAS while exchanging RTP :

Granularity Duration of Test

(second) Breakpoint

(calls/second) Session Attempts

1 30 281 8,430

1 60 279 16,740

1 90 276 24,840

2 30 279 8,370

2 60 275 16,500

2 90 272 24,480

3 30 277 8,310

3 60 280 16,800

3 90 276 24,660



In this scenario, we are sending a pcap file that lasts 8 seconds, then a DTMF

Tone (1 second) closes the media exchange, thus a call lasts approximately 9

seconds. We can observe that, for any value of granularity, the breakpoint

stays constant for any duration of test (30, 60 or 90 seconds).

On Linux, there is an opened file limitation set to 1024 files by default. We

raised that limit to 4096, this is why we can observe a multiple of 4 for each

session attempts value. Only 4096 pcap files can be opened at the same time

by both UAC and UAS. We could raise this limit to infinity, however, both CPUs

were running at almost 200%. It would have not been safe for both computers

to delete this limitation. A much higher breakpoint for RTP exchange should be

measured for state-of-the-art computers.



3. No DUT - No/Yes RTP - Comparison

This chart compares both values of breakpoint for a specific duration of test

when there is RTP and when there is not. We can easily observe that

exchanging RTP reduces drastically the number of calls per second we can

achieve without failure.

Now that we measured the maximum performance that our test bed can

achieve, we can place a proxy in our configuration.



4. Yes DUT No RTP (Asterisk)

The following ladder diagram given by Wireshark shows that 2 calls have been

successfully made between the UAC, Asterisk and the UAS :

Granularity

Duration of Test

(second)

Call Length (second)


Session Attempts

1 30 9 129 3,870

1 60 9 98 5,880

1 90 9 97 8,730

2 30 9 128 3,840

2 60 9 98 5,880

2 90 9 98 5,870

3 30 9 126 3,720

3 60 9 99 5,760

3 90 9 99 8,640



When Asterisk is relaying calls from the UAC to the UAS, we can achieve

far less calls without failure than our first scenarios without a DUT. Indeed, a

lot more steps now divide a call : Asterisk receives an invite from the UAC and

must forward it to the UAS; each transaction must pass through the proxy.

Thus, we have a breakpoint of approximately 130 calls/second when the test

lasts 30 seconds, and we reach a constant value of around 100 calls/second

when the test lasts 60 seconds or more.

A call-length of 9 seconds has been set to be able to compare when media is

flowing in our next scenario.

5. Yes DUT Yes RTP (Asterisk) - Can Reinvite = NO

The can reinvite option set to no means that RTP flows can't be establish

directly between the UAS and the UAC once the call has been set up. Thus,

Asterisk is handling every media exchange.

The following ladder diagram given by Wireshark shows that 1 call have been

successfully made between the UAC, Asterisk and the UAS with media

exchange.



Granularity Duration of Test

(second) Breakpoint

(call/second) Session Attempts

1 30 88 2,640

1 60 85 5,100

1 90 76 6,780

2 30 87 2,610

2 60 86 5,160

2 90 77 6,930

3 30 85 2,460

3 60 85 4,920

3 90 79 6,840



When RTP is flowing through Asterisk, we can observe that a duration of

test of 90 seconds have an influence on the breakpoint. Indeed, it appears that

when we are reaching a session attempts value above 6,500 calls our DUT is

reaching a lower value of call-rate without encountering a failure.

We also raised the opened file limitation on our DUT to 4096 but we did not see

any modification with a limit set at 1024.

In that case, Asterisk acts as a media relay, opening pipes for every RTP

exchange, the computer resources are logically used at their maximum. By

running longer test we could eventually find a constant breakpoint value like

the one we found on our test bed without DUT.

6. Yes DUT - No/Yes RTP - Comparison

Without RTP, the utilization of Asterisk as a relay reaches a constant

breakpoint value of around 100 calls/second. With RTP, we are obviously

seeing a lesser breakpoint, a lot more resources are being used when Asterisk

acts as a media relay. Asterisk opens a pipe for each RTP sent from the UAS to



the UAC, this resource-consuming operation results on a lesser call-rate

without encountering a failure.

7. Yes RTP - No/Yes DUT - Comparison

This comparison has been made to ensure that logic is respected for our tests.

Indeed, the breakpoint for media exchange without DUT is higher than with a

DUT, it confirms the fact that lesser calls per second can be made when a

proxy acts as a media relay in between.



V. Project Continuation Once we were able to efficiently use our script, we managed to run and

get results for various scenarios. However, this project is not over as we have a

few objectives remaining : the first one is to run Asterisk with the can reinvite

option set up to yes, we might see a higher breakpoint while using RTP. Then,

we already configured another proxy : Kamailio; thus, we have to run our

scenario without RTP only (Kamailio can't relay media) and compare the results

with Asterisk. Finally, we will be able to complete a paper for submission to

IEEE.



Conclusion This project challenged our team during the entire Fall semester. While

most of the goals have been completed, a few others have to be achieved

before being able to submit our paper for publication by the IEEE. The SIPp

performance benchmark is a project that began years ago, while we confirmed

past results obtained by former IIT students, we went a step further by

conceiving a script to automate convenient and efficient testing. Our

successors, as well as the entire VoIP community, will use our work by making

our script available online to anyone.



References Previous work by former IIT students. Van Meggelen, J - Smith, J - Madsen L. (2007). 2nd Edition. Asterisk : The

Future of Telephony.

Gayraud R, Jacques O and online contributors (2004-2013). The SIPp

Technology and Documentation. Retrieved from http://sipp.sourceforge.net.

Spencer, M and online contributors (1999-2014). Asterisk Technology and

Documentation. Retrieved from http://www.asterisk.org

Davids, C - Gurbani, V - Poretsky, S. (2014). Terminology for Benchmarking

Session Initiation Protocol (SIP) Devices: Basic session setup and

registration. Retrieved from https://datatracker.ietf.org/doc/draft-ietf-

bmwg-sip-bench-term/ .

Davids, C - Gurbani, V - Poretsky, S. (2014). Methodology for Benchmarking

Session Initiation Protocol (SIP) Devices : Basic session setup and

registration. Retrieved from https://datatracker.ietf.org/doc/draft-ietf-

bmwg-sip-bench-meth/ .

(2008-2014). Kamailio SIP Server Project.

Retrieved from http://www.kamailio.org/w/



Appendix A - Previous Results

NO DUT NO RTP

Granularity Call Length

(second)

Duration of Test

(second)

Breakpoint (c/s)

CPU/MEM (%)

UAC

CPU/MEM (%) UAS

1 0 30 3749 94.0 / 2.5 93.7 / 3.1

1 2 30 3484 93.1 / 2.3 89.4 / 1.9

1 4 30 3372 92.7 / 3.4 88.4 / 1.8

1 6 30 3139 97.2 / 4.5 87.4 / 1.7

2 0 30 3831 94.5 / 2.1 95.2 / 3.5

2 2 30 3420 92.3 / 1.8 92.4 / 2.6

2 4 30 3035 90.8 / 1.7 90.9 / 1.7

2 6 30 2927 88.1 / 1.4 87.3 / 1.3

3 0 30 3598 93.3 / 2.3 93.2 / 3.1

3 2 30 3311 91.2 / 1.6 89.4 / 1.3

3 4 30 3102 89.9 / 1.3 88.5 / 1.3

3 6 30 2937 87.5 / 1.1 86.2 / 1.1



NO DUT YES RTP

Duration of Test

(second) Granularity

Breakpoint (c/s)

CPU/MEM (%) UAC

CPU/MEM (%) UAS

30 1 256 174.5 / 0.8 9.6 / 0.5

30 2 269 179.0 / 0.9 10 / 0.9

30 3 269 178.0 / 0.9 10.6 / 0.9

45 1 258 175.7 / 1.0 9.7 / 0.5

45 2 260 176.2 / 1.0 9.9 / 0.6

45 3 268 178.1 / 0.9 10.5 / 0.9

60 1 259 175.8 / 0.8 9.7 / 0.5

60 2 260 176.1 / 0.9 9.8 / 0.6

60 3 268 179.0 / 1.0 10.4 / 0.9

YES DUT NO RTP (Asterisk)

Granularity Call-Length

(second) Duration of Test

(second) Breakpoint

(c/s)

1 0 30 136

1 2 30 97

1 4 30 74

1 6 30 60

2 0 30 137

2 2 30 89

2 4 30 64

2 6 30 60

3 0 30 137

3 2 30 99

3 4 30 75

3 6 30 59



Appendix B - Asterisk Configuration Files

extensions.conf

[general] static=yes writeprotect=no autofallthrough=yes clearglobalvars=no [globals] CONSOLE=Console/dsp ; Console interface for demo IAXINFO=guest ; IAXtel username/password TRUNK=DAHDI/G2 ; Trunk interface TRUNKMSD=1 ; MSD digits to strip (usually 1 or 0) [internal] exten => sippuac,1,NoOp() exten => sippuac,n,Dial(SIP/sippuas,30) exten => sippuac,n,Playback(the-party-you-are-calling&is-curntly-unavail) exten => sippuac,n,Hangup()

sip.conf

[general] context=internal ; Default context for incoming calls. Defaults to 'default' allowoverlap=no udpbindaddr=0.0.0.0 tcpenable=no tcpbindaddr=0.0.0.0 srvlookup=yes canreinvite=no directrtpsetup=no register => [email protected]:5060/sippuac register => [email protected]:5060/sippuas [sippuas] username=sippuas type=friend canreinvite=no port=5060 context=internal



host=10.200.0.250 disallow=all allow=ulaw [sippuac] username=sippuac type=friend canreinvite=no port=5060 context=internal host=10.200.0.248 disallow=all allow=ulaw

yueqing zhang a20306020 arthur clouet a20319763 oluseyi … · or kamailio, ip address :...

Documents