anatomy of a large django site
DESCRIPTION
TRANSCRIPT
![Page 1: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/1.jpg)
Anatomy of a large Django site
Andy McKay Mozilla
mozilla
![Page 2: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/2.jpg)
mozilla
Vancouver
![Page 3: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/3.jpg)
PythonZope and Plone...now at Mozilla
mozilla
![Page 4: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/4.jpg)
mozilla
Using Djangohttp://www.djangoproject.com
Credit: http://www.flickr.com/photos/abiavati/3110357974/
![Page 5: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/5.jpg)
1. About the site2. Performance3. Localisation4. Reuse
mozilla
![Page 6: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/6.jpg)
1. About the site
mozilla
![Page 7: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/7.jpg)
mozilla
![Page 8: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/8.jpg)
All code is open:https://github.com/jbalogh/zamboni
mozilla
![Page 9: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/9.jpg)
All* bugs are open:https://bugzilla.mozilla.org
mozilla
![Page 10: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/10.jpg)
Convert from CakePHP (remora)
to Django (zamboni)
mozilla
![Page 11: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/11.jpg)
mozillaCredit: http://www.flickr.com/photos/improbcat/4177702580/
![Page 12: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/12.jpg)
Changing one URL at a time from CakePHP to Django
mozilla
General trend to move away from PHP and do more Python and Django
![Page 13: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/13.jpg)
How large?250k+ addons
150 mn views month500 mn API hits day
mozilla
![Page 14: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/14.jpg)
Lines of code
PHPPython
40k18k
mozilla
![Page 15: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/15.jpg)
Lines of code
PHPPython
Unit tests
40k18k15k
mozilla
![Page 16: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/16.jpg)
No pages go out until they are faster
mozilla
running both php and python side by side. a few issues on that
![Page 17: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/17.jpg)
3 zeus load balancers24 django (and php) 1 mysql + 4 slaves3 memcached3 sphinx1 rabbitmq + 2 celeryd1 redis master + 1 slave
Credit: http://www.flickr.com/photos/tbridge/15300843/ mozilla
![Page 18: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/18.jpg)
2. Performance
mozilla
![Page 19: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/19.jpg)
mozilla
As usual, database bottleneck
![Page 20: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/20.jpg)
Cache machinehttp://bit.ly/cache-machine
mozillaCredit: http://www.flickr.com/photos/mwichary/4063534688/
![Page 21: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/21.jpg)
mozilla
from django.db import modelsimport caching.base
class Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager()
available as a mixin
need to addin the custom manager
![Page 22: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/22.jpg)
mozilla
>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13
![Page 23: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/23.jpg)
mozilla
>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13
>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13
![Page 24: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/24.jpg)
mozilla
Invalidation
![Page 25: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/25.jpg)
mozillamozilla
md5(‘select... a’) [addon 3615]
![Page 26: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/26.jpg)
mozillamozilla
md5(‘select... a’) [addon 3615]
addon 3615 md5(‘select... a’)
![Page 27: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/27.jpg)
mozillamozilla
md5(‘select... a’) [addon 3615]
md5(‘select... b’) [addon 3615, addon 1685]
addon 3615 md5(‘select... a’)md5(‘select... b’)
![Page 28: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/28.jpg)
mozilla
Memcachedrules = cache.get(3615)rules.add('select...')cache.set(3615, rules)
![Page 29: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/29.jpg)
mozilla
Redisredis.SADD(3615, ‘select...’)
![Page 30: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/30.jpg)
mozilla
Home page20+ addons
400+ sql queries
![Page 31: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/31.jpg)
mozilla
standard answer in django is select-related
add-on version
files
version
version
files
files
![Page 32: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/32.jpg)
mozilla
django: select_related()http://bit.ly/select-related
![Page 33: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/33.jpg)
mozilla
Transformerhttp://bit.ly/queryset-transform
simonw
![Page 34: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/34.jpg)
mozilla
@staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version
slightly outdated example
![Page 35: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/35.jpg)
mozilla
Home page20+ addons
~14 sql queries
big SQL statements... :(
14313 character
![Page 36: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/36.jpg)
UpdateCalled on startup
about:config ☞ extensions.update.url
mozilla
![Page 37: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/37.jpg)
IncomingUncached
8,000 req/sec1,600 req/sec
Im used to Plone in the bad old days
mozilla
![Page 38: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/38.jpg)
IncomingUncached
PHP
8,000 req/sec1,600 req/sec550 req/sec
Im used to Plone in the bad old days
mozilla
![Page 39: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/39.jpg)
PHP Python
v1Plain
Django
mozilla
![Page 40: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/40.jpg)
PHP Python
v2Min. SQL
queries
mozilla
![Page 41: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/41.jpg)
oh god
mozilla
![Page 42: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/42.jpg)
PHP Python
v3Django and
raw SQLmax-requests 200, actually we hit 210
mozilla
![Page 43: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/43.jpg)
PHP Python
v4WSGI
no Django
mozilla
![Page 44: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/44.jpg)
PHP Python
v5Pooling,
optimised queries
Thats 700 req/secwhich translates into
mozilla
![Page 45: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/45.jpg)
Reducing the SQL queries...doesn’t always help
mozilla
![Page 46: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/46.jpg)
mySQL query cache is fast
mozilla
![Page 47: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/47.jpg)
Celeryhttp://celeryproject.org
mozillaCredit: http://www.flickr.com/photos/chiotsrun/3843988392/
![Page 48: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/48.jpg)
Push things asyncemail
image processingadd-on validation
specifically fixing data changes between php and python
mozilla
![Page 49: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/49.jpg)
from celeryutils import task
@taskdef update_tag(tag, **kw): tag.update_stat()
mozilla
![Page 50: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/50.jpg)
from celeryutils import task
@task(rate_limit='60/h')def update_tag(tag, **kw): tag.update_stat()
mozilla
![Page 51: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/51.jpg)
from tasks import update_tag
update_tag.delay(tag)
mozilla
![Page 52: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/52.jpg)
mozilla
Measurement
![Page 53: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/53.jpg)
Timing Middlewarehttp://bit.ly/timing-ware
mozillaCredit: http://www.flickr.com/photos/wwarby/3297205226/
![Page 54: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/54.jpg)
mozilla
![Page 55: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/55.jpg)
3. Localization
mozilla
![Page 56: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/56.jpg)
mozilla
40+ languagesincluding rtl
show site in arabic?
![Page 57: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/57.jpg)
mozilla
Database strings
content translated
and
templates
![Page 58: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/58.jpg)
mozilla
class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation)
![Page 59: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/59.jpg)
mozilla
addon.name = 'name'addon.save()
![Page 60: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/60.jpg)
mozilla
addon.name = 'name'addon.save()
addon.name = {'fr': 'la nomme'}addon.save()
![Page 61: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/61.jpg)
mozilla
Templates
![Page 62: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/62.jpg)
mozilla
Django{% blocktrans with app=request.APP %}
Add-ons for {{ app }} {% endblocktrans %}
![Page 63: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/63.jpg)
mozilla
Jinja2http://jinja.pocoo.org/
{{ _('Add-ons for {0}')|f(request.APP) }}
![Page 64: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/64.jpg)
mozilla
Python Unicode hellUnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 16: ordinal
not in range(128)
![Page 65: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/65.jpg)
4. Reuse
mozilla
![Page 66: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/66.jpg)
Bleachhttp://pypi.python.org/pypi/
mozillaCredit: http://www.flickr.com/photos/maisonbisson/3350954463/
![Page 67: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/67.jpg)
>>> bleach.clean('an <script>evil()</script> example')
'an <script>evil()</script> example'
mozilla
![Page 68: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/68.jpg)
>>> bleach.linkify('an http://ex.com url')
'an <a href="http://ex.com" rel="nofollow">http://ex.com</a> url'
mozilla
![Page 69: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/69.jpg)
Javascript tests
mozilla
![Page 70: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/70.jpg)
django-qunithttp://bit.ly/django-qunit
mozilla
kumar
![Page 71: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/71.jpg)
mozilla
test('English', function() { z.refreshL10n('en-us'); equals($('textarea:visible', this.sandbox).text().trim(), 'Firebug integrates with Firefox to put ' + 'a wealth of development tools...');});
![Page 72: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/72.jpg)
mozilla
test('Japanese', function() { z.refreshL10n('ja'); equals($('textarea:visible', this.sandbox).text().trim(), 'Firebug は、Web ページを閲覧中にクリック一つ ' +
' で使える豊富な開発ツールを Firefox' + ' に統合します。あなたはあらゆる');});
![Page 73: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/73.jpg)
Use HudsonJenkinshttp://bit.ly/jstestnet
mozilla
So we use hudson for CI, but haven’t got the automated tests in yet
Hoping to do this via jstestnet
![Page 74: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/74.jpg)
Flake8http://bit.ly/flake8
mozillaCredit: http://www.flickr.com/photos/nebarnix/357779131/
pep 8py flakesMacCabe
![Page 75: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/75.jpg)
~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.pyapps/editors/tasks.py:1: 'datetime' imported but unusedapps/editors/tasks.py:3: 'stat' imported but unused
mozilla
![Page 76: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/76.jpg)
Playdohhttp://bit.ly/mozilla-playdoh
mozilla
fred wenzel
Credit: http://www.flickr.com/photos/ahmee/97960570/
![Page 77: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/77.jpg)
Basis for new projects
mozilla
![Page 78: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/78.jpg)
Celery supportJinja2 supportSimple migrations
By default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly flags on cookies
mozilla
fred wenzel
![Page 79: Anatomy of a large Django site](https://reader033.vdocument.in/reader033/viewer/2022052310/54bd60b64a7959a9278b4577/html5/thumbnails/79.jpg)
Take inspiration from...but not the best for you
mozilla
for example jinja2 which makes integration with lots of django addons possible, but a bit harder