jr richardson engineering for the masses [email protected] virtualizing asterisk with openvz
TRANSCRIPT
Presentation Overview
¬ Discuss various Virtualization technologies¬ Examples for Virtualization with Asterisk¬ Step by step tutorial, build a Virtual Server
for Asterisk¬ Discuss operation and maintenance¬ Considerations, what to watch out for¬ Wrap up
Overview
¬ Basic Asterisk system require very little hardware resources
¬ Many Asterisk embedded PC devices being used
¬ Modern server are overkill for many Asterisk systems
¬ Where need arises, many virtual Asterisk PBX’s can be deployed on 1 physical server
¬ Virtualizing Asterisk improves hardware efficiency
¬ Get more bang for your buck
Virtualization Introduction
Virtualization is a framework or methodology of dividing the resources of a computer into multiple execution environments.
Virtualization techniques create multiple isolated partitions — Virtual Machines (VM) or Virtual Environments (VEs) — on a single physical server
Virtualization Technologies
¬ Virtual Machines (VMs)– Emulate real or fictional hardware– Runs in guest operating system– Greater host CPU load, requires virtual
machine monitor (VMM)– VMware, QEMU, Parallels and MS Virtual Server
¬ Paravirtualization– Also uses VMM– Guest OS is modified to run under the
hypervisor (VMM)– Allows for various Guest OS– Xen and UML
Virtualization Technologies cont
¬ OS Level Virtualization– Most applications on the host can share with
other virtual environments (VEs)– Does not require each virtual machine to run
dedicated OS– Host server runs multiple instances of single
OS, can be different versions– Designed to isolate and secure each virtual
instance to run multiple applications– Very low overhead– OpenVZ, Linux-VServer, Solaris Zones,
FreeBSD Jails
Comparison
¬ VMs have a wide scope of usage, but poor performance
¬ Para-VMs have better performance, tighter integration with Host OS, but have to modify Guest OS
¬ OS level provides best performance, simpler to administer, can natively access all VEs from host, best choice for same OS server consolidation
¬ Personal Choice Open Source Project OpenVZ– Very Mature, has kernel for many Linux
distributions, great community and support– http://wiki.openvz.org
Asterisk Consideration
¬ Base OS mostly Linux¬ Same Linux Distribution used¬ Same kernel used¬ Common Asterisk versions and packages¬ Once version is selected, consecutive PBX
builds will be same¬ This is optimal for OS level Virtualization¬ Can take advantage of scalability, ease of
setup and management
Why Virtualize Asterisk?
¬ Service Providers– Offer hosted PBX solutions– Individual PBX for each customer– Simpler than Clustering Asterisk servers
¬ 2 Asterisk PBX’s on 1 Server– Soft switch application redundancy
¬ Limited hardware resources– Run production PBX on same hardware while testing
other versions or custom applications
¬ Lab– Testing various versions of Asterisk– Mock up production systems– Build Cluster arrangements for testing
Build a Virtual Server with OpenVZ
¬ Hardware Considerations– Fast processor, multi core, multi proc– Lots of RAM– Large and fast disk drive, SATA, SCSI
¬ Choose base Linux Distro– OpenVZ has several kernels for pre-packaged
linux distributions, Debian, Fedora, CentOS, SUSE, Slackware, Gentoo, Ubuntu
– Can also compile kernel package directly¬ For the following example we will use
Debian Etch with pre-build OpenVZ kernel
Install Base OS and OpenVZ Kernel
¬ Various Linux Distro install instructons on wiki– http://wiki.openvz.org/Category:Installation
¬ Example Hardware Node Specs– Asus P5MT Mobo– 3.2Ghz P-4 Proc– 4 Gig RAM– 80 Gig SATA 7200 Hard Drive
¬ BIOS setting– Disable serial and parallel prorts– Power recovery set to ‘last state’– Wait for F1 errors ‘disable’ for no keyboard
Install Base OS and OpenVZ Kernel
¬ Debian Base system, typical install– 3 partitions, / and /var and swap, majority of
space for /var– Do not install any optional packages– Virtual containers reside in /var/lib/vz/private/
o For disk quota to perform as expected, container space must not impede the root partition
– Delete CDROM as source in /etc/apt/source.list– apt-get update– apt-get dist-upgrade– Reboot so kernel updates– apt-get install ssh ftp
Install Base OS and OpenVZ Kernel
¬ Add OpenVZ repository to /ext/apt/source.list– deb http://download.openvz.org/debian etch main– deb-src http://download.openvz.org/debian etch main– deb http://debian.systs.org/ etch openvz– deb-src http://debian.systs.org/ etch openvz
¬ wget http://download.openvz.org/debian-systs/dso_archiv_signing_key.asc
¬ apt-key add dso_archiv_signing_key.asc¬ wget
http://debian.systs.org/dso_archiv_signing_key.asc¬ apt-key add dso_archiv_signing_key.asc¬ apt-get update
Install Base OS and OpenVZ Kernel
¬ apt-get install linux-image-2.6.18-openvz-13-1etch5-686 linux-headers-2.6.18-openvz-13-1etch5-686
¬ Reboot into OpenVZ kernel, verify ‘uname –r’¬ apt-get install libncurses5-dev bison libssl-dev
libnewt-dev zlib1g-dev procps gcc make binutils modconf g++ debootstrap ntp
¬ cd /usr/src/ get zaptel and install– ‘mknod /dev/zap/transcode c 196 253’– ‘chmod 660 /dev/zap/transcode’
¬ Check for rtc irq conflict, 1000Hz lost sync error– Edit grub menu.lst, add acpi=off to kernel switch
Install Base OS and OpenVZ Kernel
¬ apt-get install vzctl vzquota vzdump¬ ‘cd /’
– ‘ln –s /var/lib/vz vz
¬ Install template, ‘apt-get install [template]’– vzctl-ostmpl-debian - OpenVZ - OS Template -
Debian 4.0– vzctl-ostmpl-debian-4.0-i386-minimal - OpenVZ
- OS Template debian-4.0-i386-minimal– vzctl-ostmpl-debian-5.0-i386-minimal - OpenVZ
- OS Template debian-5.0-i386-minimal
Operation and Maintenance
¬ Config files in /etc/vz/¬ OpenVZ tools
– vzctl– vzquota– vzdump– vzlist– vzmigrate– vzcalc– vzmemcheck– vzsplit– vzcfgvalidate– beancounters
VE Config File /etc/vz/conf/101.conf
ONBOOT="yes"# UBC parameters (in form of barrier:limit)# Primary parametersNUMPROC="200:200"NUMTCPSOCK="300:300"# Secondary parametersKMEMSIZE="4874532:5058032"TCPSNDBUF="1054834:2093234"TCPRCVBUF="1054834:2093234"SHMPAGES="28842:28842"NUMFILE="3000:3000"# Disk quota parameters (in form of softlimit:hardlimit)DISKSPACE="1548576:1653434"# CPU fair sheduler parameterCPUUNITS="7000"VE_PRIVATE="/var/lib/vz/private/$VEID"OSTEMPLATE="ast4-1-2-24-ovz"NAMESERVER=“10.10.14.1"IP_ADDRESS=“10.10.14.101“DEVNODES="tty9:rw zap/channel:rw zap/ctl:rw zap/pseudo:rw zap/timer:rw
zap/transcode:rw"HOSTNAME="pulsion-ve501"
vzsplit
¬ ‘vzslpit’¬ Enter the number of VEs: 20The optimal swap space size is 7088 Mb, twice bigger than the RAM size
# Configuration file generated by vzsplit for 20 VEs# on HN with total amount of physical mem 3544 Mb# low memory 856 Mb, swap size 7632 Mb, Max treads 8000# Resourse commit level 0:# Free resource distribution. Any parameters may be increased# Primary parametersNUMPROC="400:400"NUMTCPSOCK="400:400"VMGUARPAGES="138035:2147483647"# Secondary parametersKMEMSIZE="8979701:9877671"TCPSNDBUF="1354834:2993234"TCPRCVBUF="1354834:2993234"SHMPAGES="54437:54437"NUMFILE="3488:3488"DISKSPACE="268134:294948"DISKINODES="164219:180642"CPUUNITS="7148"
Routine Operation and Maintenance
¬ Create VE¬ Create and modify templates¬ Check beancounters¬ List running VEs¬ Migrate VE to other hardware nodes¬ Monitor
– Log files– Disk space– Proc utilization– Main memory
Working with Templates
¬ ‘vzctl create 101 --ostemplate debian-4.0-i386-minimal’– OK to do this in production, Kernel guarantees VE
resources¬ Edit /etc/vz/conf/101.conf
– NAMESERVER=“10.10.14.1"– IP_ADDRESS="10.10.14.101"– DEVNODES="zap/channel:rw zap/ctl:rw zap/pseudo:rw
zap/timer:rw zap/transcode:rw“¬ ‘vzctl start 101’¬ ‘vzctl enter 101’¬ Update /etc/apt/source.lst with OpenVZ repositories¬ ‘apt-get update’¬ ‘apt-get upgrade’¬ ‘apt-get install linux-headers-`uname –r`’
Install Asterisk in Template
¬ apt-get install libncurses5-dev bison libssl-dev libnewt-dev zlib1g-dev procps gcc make binutils ftp g++ ntp ssh ftp
¬ Download and install asterisk, zaptel not needed in VE, has access to hardware node zaptel through devicenodes
¬ Configure generic Asterisk templates¬ Configure ntp, cron jobs, linux house
keeping, smarthost, syslog, etc …– Build this as the template PBX system
Update the Template
¬ Stop VE and take a snapshot– ‘vzctl stop 101’
¬ Go into the VE directory– ‘cd /vz/private/101’
¬ ‘tar -zcf /vz/template/cache/ast4-1-2-24-ovz.tar.gz .’– don’t forget the end dot
¬ Update /etc/vz/dist and /etc/vz/conf files to match the name of the new template
¬ Distribute new template files to other hardware nodes
vzctl
¬ Create and Destroy VE– ‘vzctl create 101 --ostemplate ast5-1-2-24-ovz
--config ast5-1-2-24’– ‘vzctl destroy 101’ removes private area
¬ Start, Stop, Restart VE– ‘vzctl start 101’– ‘vzctl stop 101’– ‘vzctl restart 101’
¬ Status and Enter VE– ‘vzctl status 101’
VEID 100 exist mounted running– ‘vzctl enter 101’
root@ve100:/#
vzlist
¬ ‘vzlist –a’ VEID NPROC STATUS IP_ADDR HOSTNAME 100 18 running - ve100 501 20 running 10.1.1.12 pulsionmedical-ve501 502 20 running 10.1.1.13 veritas-ve502 503 18 running 10.1.1.14 fta-ve503 504 20 running 10.1.1.15 nathealth-ve504 505 20 running 10.1.1.16 ampow-ve505 506 21 running 10.1.1.17 systemusa-ve506 507 32 running 10.1.1.18 frank-ve507 508 21 running 10.1.1.19 gclub-ve508 509 19 running 10.1.1.10 symphony-ve509
beancounters
¬ ‘cat /proc/user_beancounters’ uid resource held maxheld barrier limit failcnt 509: kmemsize 664371 1252967 4874532 5058032
0 lockedpages 0 0 338 338 0 privvmpages 4172 11690 49152 53575 0 shmpages 0 656 28842 28842 0 numproc 19 103 100 200
47 physpages 2492 3710 0 483647 0 oomguarpages 2492 3710 6144 483647
0 numtcpsock 3 4 300 300 0 tcpsndbuf 26784 49104 10834 3234 0 tcprcvbuf 49152 0 14834 93234 0 othersockbuf 24552 105272 577417 45817
0 dgramrcvbuf 0 4648 592096 92096 0 numfile 501 726 3000 3000 0
beancounters perl script
¬ astvz5:~# ./beans2 #################################################################BEANS FOR UID 509resource held maxheld barrier limit failcntkmemsize 639.07 kb 1.19 mb 4.65 mb 4.82 mb 0lockedpages 0 0 1.32 mb 1.32 mb 0privvmpages 16.30 mb 45.66 mb 192.00 mb 209.28 mb 0shmpages 0 2.56 mb 112.66 mb 112.66 mb 0numproc 19 31 200 200 0physpages 9.73 mb 14.49 mb 0 MAX_ULONG 0vmguarpages 0 0 24.00 mb MAX_ULONG 0oomguarpages 9.73 mb 14.49 mb 24.00 mb MAX_ULONG 0numtcpsock 3 4 300 300 0tcpsndbuf 26.16 kb 47.95 kb 1.01 mb 2.00 mb 0tcprcvbuf 48.00 kb 0.00 kb 1.01 mb 2.00 mb 0othersockbuf 23.98 kb 752.80 kb 563.88 kb 1.28 mb 3259dgramrcvbuf 0.00 kb 4.54 kb 578.22 kb 578.22 kb 0numothersock 14 29 300 300 0numfile 501 726 3000 3000 0
vzcalc
¬ ‘vzcalc –v 101’Resource Current(%) Promised(%) Max(%)Low Mem 0.21 3.11 3.11Total RAM 0.30 n/a n/a Mem + Swap 0.10 0.31 n/aAlloc. Mem 0.15 0.31 1.97Num. Proc 0.03 n/a 0.31--------------------------------------------Memory 0.30 3.11 3.11
vzmigrate
¬ Setup ssh root access between hardware nodes– http://wiki.openvz.org/Migration_from_one_HN_to_another
¬ ‘apt-get install rsync’¬ Correct syntaxt in vzmigrate script
– nano +382 /usr/sbin/vzmigrate– -if [ $? != 20 && $? != 21 && $? != 0 ]; then– +if [ $? != 20 ] && [ $? != 21 ] && [ $? != 0 ]; then
¬ vzmigrate -r no --keep-dst –v [ip address of new HN] [VE number to transfer]
¬ vzmigrate -r no --keep-dst -v 10.10.10.2 101¬ Migration ~120 seconds, VE only down 10
seconds
OpenVZ Considerations
¬ Hardware node VE execution scripts– Can use to run commands inside VE from HN
¬ Example of running ‘date’ and ‘ntpq –p’ commandastvz5:~# more vz-exec-date echo "################# VE 501
###################"vzctl exec 501 datesleep 1vzctl exec 501 ntpq -psleep 1echoechoecho "################# VE 502
###################"vzctl exec 502 datesleep 1vzctl exec 502 ntpq -psleep 1echoecho
OpenVZ Considerations
¬ astvz5:~# more vzcalc-all echo "################# VE 501
###################"vzcalc -v 501sleep 1echoechoecho "################# VE 502
###################"vzcalc -v 502sleep 1echoecho
OpenVZ Considerations
¬ astvz5:~# more migrate-all vzmigrate -r no --keep-dst -v 10.10.10.2 501sleep 2vzmigrate -r no --keep-dst -v 10.10.10.2 502sleep 2vzmigrate -r no --keep-dst -v 10.10.10.2 503sleep 2vzmigrate -r no --keep-dst -v 10.10.10.2 504sleep 2
Kernel Issue Workaround
¬ OpenVZ requires at least 1 active VE running to operate as expected
¬ If you have 20 production VEs, run 21 VEs on hardware node
¬ This may have been fixed in later OpenVZ kernels
¬ Beancounters require VE to be stopped at least 5 minutes to clear counters
¬ If VE kmemsize beancounters are hung, need to reboot hardware node to clear
Wrap Up
¬ Open VZ, like most open source projects have quirks but for the most part works straight out of the box
¬ The community is very knowledgeable and helpful. User list [email protected]– https://openvz.org/mailman/listinfo/users
¬ 20+ lightly loaded Asterisk VEs can be successfully deployed in production on a single hardware node on moderate to high end servers
¬ Great wiki, everything OpenVZ– http://wiki.openvz.org