europython 2014 - how we switched our 800+ projects from apache to uwsgi
Post on 14-Sep-2014
366 views
DESCRIPTION
During the last 7 years the company I am working for developed more than 800 projects in PHP and Python. All this time we were using Apache+nginx for hosting this projects. In this talk I will explain why we decided to switch all our projects from Apache+nginx to uWSGI+nginx and how we did that.TRANSCRIPT
europython 2014
1 / 26
How we switched our800+ projects fromApache to uWSGI
Max Tepkeev23 July 2014Berlin, Germany
europython 2014
2 / 26
Who We ?!
• Ailove Group• Ailove• Aitarget• iCom
• 120+ people• 800+ projects
europython 2014
3 / 26
Who I ?!
Max TepkeevRussia, Moscow
• python-redmine• architect
https://www.github.com/maxtepkeev
europython 2014
4 / 26
Previous Technology Stack
• PHP• 5.3• 5.4• 5.5
• Python• 2.7• 3.3
• nginx• Apache• mod_php• mod_wsgi
europython 2014
5 / 26
Problems
Separate Apache for each version of PHP and Python listening on different ports:
• :8080 – Apache/PHP5.3• :8081 – Apache/PHP5.4• :8082 – Apache/PHP5.5• :8083 – Apache/Python2.7• :8084 – Apache/Python3.3
europython 2014
6 / 26
Problems
Monitoring for code changes during development:
• embedded-mode – full restart• daemon-mode – 100+ lines script
mod_wsgi >= 4.1.0 (Django only):
• runmodwsgi --reload-on-changes
europython 2014
7 / 26
Problems
mod_wsgi WSGIDaemonProcess name can’t be dynamic, e.g. separate daemon process per GIT branch:
• dev.project.com – project-dev.conf• dev2.project.com – project-dev2.conf
europython 2014
8 / 26
Problems
Apache can’t load configuration files dynamically:
• reload – not so graceful restart• total crash if there are errors
europython 2014
9 / 26
Problems
• Apache configuration files are ugly (subjective)
• Apache is hard to configure properly (subjective)
• Apache is old (2.4 fixed a lot)• mod_wsgi seemed to be abandoned
(is actively developed again)
europython 2014
10 / 26
uWSGI
• Modern project• Fast development cycle• Multi-language (Python, PHP, Lua,
PERL, Ruby, Erlang, Go, Java and more)• Supports > 20 OS• nginx speaks uwsgi protocol• Ton more features
europython 2014
11 / 26
Installationhttp://projects.unbit.it/downloads/uwsgi-latest.tar.gz
To change install location add to buildconf/core.ini:
bin_name = /usr/local/uwsgi/bin/uwsgiplugin_dir = /usr/local/uwsgi/plugins
mkdir /usr/local/uwsgimkdir /usr/local/uwsgi/binmkdir /usr/local/uwsgi/plugins
europython 2014
12 / 26
InstallationCore:
python uwsgiconfig.py --build core
Plugins:
python2.7 uwsgiconfig.py --plugin plugins/python core python27python3.3 uwsgiconfig.py --plugin plugins/python core python33
UWSGICONFIG_PHPDIR=/usr/local/php5.4 python uwsgiconfig.py --plugin plugins/php core php54UWSGICONFIG_PHPDIR=/usr/local/php5.5 python uwsgiconfig.py --plugin plugins/php core php55
europython 2014
13 / 26
Solutions
Multi-version plugins:
• php53_plugin.so – PHP5.3• php54_plugin.so – PHP5.4• php55_plugin.so – PHP5.5• python27_plugin.so – Python2.7• python33_plugin.so – Python3.3
europython 2014
14 / 26
Solutions
Monitoring for code changes during development:
• --py-auto-reload N (seconds)• --touch-reload=django.wsgi
europython 2014
15 / 26
Solutions
Emperor mode - event based dynamic handling of applications (vassals):
• Monitor config files in directories• More plugins available (postgres,
mongodb, amqp, zeromq and more)
europython 2014
16 / 26
Solutions
Monitor config files in directories:uwsgi --emperor “/usr/local/uwsgi/apps/*/*.ini”
• New file “/usr/local/uwsgi/apps/app/app.ini”• Spawn vassal
• File modified• Restart vassal
• File removed• Kill vassal
europython 2014
17 / 26
Solutions
Template config files:
ln -s /usr/local/uwsgi/templates/app1_tpl/usr/local/uwsgi/apps/app1/main.ini
europython 2014
18 / 26
Solutions[uwsgi]project_dir = /srv/projects/my_project
plugins = python27virtualenv = %(project_dir)/pythonpythonpath = %(project_dir)socket = %(project_dir)/tmp/%n.sockwsgi-file = %(project_dir)/repo/%n/wsgi/django.wsgitouch-reload = %(project_dir)/repo/%n/wsgi/django.wsgilogto = %(project_dir)/logs/uwsgi-%n.log
europython 2014
19 / 26
Solutions
Create symlink for each GIT branch:
• ln -s /usr/local/uwsgi/templates/app1_template/usr/local/uwsgi/apps/app1/dev.ini
• ln -s /usr/local/uwsgi/templates/app1_template/usr/local/uwsgi/apps/app1/master.ini
europython 2014
20 / 26
Cool Features
Auto-scaling:
• broodlord mode• zerg mode• emperor• idle/die-on-idle
europython 2014
21 / 26
Cool Features
Alarm subsystem:In configuration[uwsgi]alarm = jabber xmpp:[email protected];mypass;[email protected] = jabber ^TERRIBLE ALARM
In applicationprint "TERRIBLE ALARM!"
europython 2014
22 / 26
Cool Features
Aliasing Python modules:
[uwsgi]# some configuration herepymodule-alias = foo=/opt/proj/experimental_foo.pypymodule-alias = bar=/opt/proj/experimental_bar.py
europython 2014
23 / 26
Cool Features
• built-in crontab• built-in load-balancer• clustering subsystem• offload subsystem• plugins (geoip etc.)• integration with web-servers• django admin integration
europython 2014
24 / 26
How to switch
• no fully automatic way• write scripts that generate basic
config files for you• tune them by hand afterwards
europython 2014
25 / 26
Conclusion
• apache is not bad• benchmarks are pointless• web servers perform similarly if
configured properly• it’s applications that have
problems, not web servers• choose a right tool for a right job
europython 2014
26 / 26
Questions
slides: http://slideshare.net/maxtepkeev
github: https://github.com/maxtepkeevemail: [email protected]: max.tepkeev