the life of a web request - techniques for measuring and improving django application performance
DESCRIPTION
The connection between sub-second web application performance and revenue is becoming more and more apparent with established companies regularly reporting the benefits of reducing page load times. This talk will cover: * Designing for performance * Approaches to instrumenting and measuring application performance * Areas of focus for both front-end and back-end improvement * Techniques, tools and modules available in Django-land for improving performance * New and emerging technologies, for example SPDY protocol and Django 1.5's StreamingHttpResponseTRANSCRIPT
![Page 1: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/1.jpg)
The Life of a Web RequestMeasuring and improving
Django application performance
PyCon Australia 2013
Roger Barnes@[email protected]://slideshare.net/mindsocket/
![Page 2: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/2.jpg)
What are you here for
Aspects of site performance
Measuring everything
Fixing the right thing
AND HOW
![Page 3: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/3.jpg)
Performance focus for this talk
✔ user perspective✔ seconds per request✗ benchmarks✗ users per server✗ requests per second
![Page 4: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/4.jpg)
![Page 5: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/5.jpg)
Why should we care?
● Web application performance affects– engagement
![Page 6: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/6.jpg)
Why should we care?
● Web application performance affects– engagement– bounce rate
http://www.slideshare.net/joshfraz/sept-2012rumtalk
![Page 7: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/7.jpg)
Why should we care?
● Web application performance affects– engagement– bounce rate– conversion/abandonment
![Page 8: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/8.jpg)
Why should we care?
● Web application performance affects– engagement– bounce rate– conversion/abandonment– search engine ranking
![Page 9: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/9.jpg)
Why should we care?
● Web application performance affects– engagement– bounce rate– conversion/abandonment– search engine ranking– revenue
![Page 10: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/10.jpg)
Why should we care?
Web sites are
larger&
than ever
![Page 11: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/11.jpg)
Why should we care?
Average web page:
1.3Mb85 requests
- HTTPArchive.org
![Page 12: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/12.jpg)
Why should we care?
Users are increasingly mobile
Their expectations are up
Many "responsive" sites are no smaller than their desktop version
![Page 13: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/13.jpg)
Let's look at aweb request!
![Page 14: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/14.jpg)
![Page 15: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/15.jpg)
There's a redirect!
![Page 16: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/16.jpg)
How about some CSS?
![Page 17: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/17.jpg)
Let's get some
JavaScript and images
Start Render: 1.6s
![Page 18: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/18.jpg)
Document complete: 8.6s
Fully loaded: 9.0s
Requests: 177
Bytes in: 2,689kb
This is on a good connection
![Page 19: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/19.jpg)
The Performance Golden Rule
"80-90% of the end-user response time is spent on the frontend.
Start there."
- Steve Souders
Source: http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/
![Page 20: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/20.jpg)
![Page 21: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/21.jpg)
Front-end Measurement
● Instrument your live application for Real User Monitoring (RUM)– Google Analytics– New Relic
![Page 22: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/22.jpg)
Front-end Measurement
● Synthetic testing● Lots of tools available● 3 in particular...
![Page 23: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/23.jpg)
Chrome Developer Tools
![Page 24: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/24.jpg)
webpagetest.org
![Page 25: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/25.jpg)
Browser navigation timing
http://www.w3.org/TR/navigation-timing/
![Page 26: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/26.jpg)
django-debug-toolbar
Not in PyPi yet,see GitHub version
Back end Front end
![Page 27: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/27.jpg)
Front-end offenders
● Static resources – size and number● Lack of caching and compression● 3rd-party resources● Overdownloading
![Page 28: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/28.jpg)
Static Resources
● Minification/combination of CSS/JavaScript– django-compressor django-pipeline
● Images– Appropriate format and compression level– Correct size– Remove metadata
● jpegoptim, optiPNG
– Consider sprites
![Page 29: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/29.jpg)
Static Resources
● Web server/accelerator config– Caching/expires/compression headers
● Static files to CDN– see Django's storage hooks
● Deferred loading● Silver bullet (?)
– mod_pagespeed (apache)– ngx_pagespeed (nginx)
![Page 30: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/30.jpg)
Ok, now what?
![Page 31: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/31.jpg)
Back-end measurement
Production instrumentation with New Relic
![Page 32: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/32.jpg)
django-statsd and graphite
![Page 33: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/33.jpg)
django-debug-toolbar
![Page 34: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/34.jpg)
runprofileserver indjango-extensions
![Page 35: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/35.jpg)
Web server
WSGI application (WSGIHandler.__call__)
Initialise Request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template Rendering
Backend Services
Typical Django Request
![Page 36: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/36.jpg)
Caching Dynamic Requests
● Potentially easy wins● Beware cache invalidation● Different levels
– Caching headers– Web Server/Web Accelerator– Caching middleware– Per view caching– Template fragment caching– Low level object caching
![Page 37: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/37.jpg)
Web server / Reverse proxy
WSGI application (WSGIHandler.__call__)
Initialise request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template rendering
Backend services
Typical Django Request
Caching
Browser
![Page 38: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/38.jpg)
Compressing Dynamic Content
Web server
Web accelerator
gzip middleware
![Page 39: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/39.jpg)
Query Performance
● Use debug toolbar● Avoid unnecessary queries
– eg exists followed by separate get
● select_related to get FKs in same query– Be wary of optional foreign keys
● prefetch_related to get collections– 1 additional query instead of N+1
![Page 40: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/40.jpg)
Defer work
● Defer user-irrelevant workloads● Target
– Views– Signals (eg post_save)– Middleware
![Page 41: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/41.jpg)
Defer work with celery
def login(request): # Your login code here # ... user = form.save()
upload_to_nsa.delay(user.id) return HttpResponse(...)
![Page 42: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/42.jpg)
Ok, now what?
![Page 43: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/43.jpg)
Web server
WSGI application (WSGIHandler.__call__)
Initialise request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template rendering
Backend services
Django Works Then Responds
Bytes on the wire(Time to First Byte)
Back end
![Page 44: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/44.jpg)
Can we stream?
● StreamingHttpResponse in 1.5● Patch for streaming templates (#13910)● Caveats
– Middleware compatibility– Exception handling– YAGNI?
![Page 45: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/45.jpg)
What about eager streaming?
Hack: View middleware that streams the top of a page before even calling the view
Proof of concept:
https://github.com/mindsocket/django-perf-example
![Page 46: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/46.jpg)
Regular
Streaming
Eager streaming
![Page 47: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/47.jpg)
Visual appearance
![Page 48: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/48.jpg)
Future context
● APIs, websockets and JavaScript, oh my!● SPDY● Mobile● Full stack ownership
![Page 49: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/49.jpg)
Summary
● Measure, measure, measure● Don't ignore the front-end● Beware diminishing returns● Technology is changing
![Page 50: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/50.jpg)
Resources
● General web performance– http://www.slideshare.net/Strangeloopnet/37-lessons-
ive-learned-on-the-performance-front-lines– http://www.webpagetest.org/– https://developers.google.com/speed/pagespeed/– http://www.mnot.net/cache_docs/– http://newrelic.com/– https://dvcs.w3.org/hg/webperf/raw-
file/tip/specs/NavigationTiming/Overview.html– https://www.varnish-cache.org/
![Page 51: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/51.jpg)
Resources
● Django – https://docs.djangoproject.com/en/dev/topics/cache/– https://code.djangoproject.com/wiki/ProfilingDjango– https://code.google.com/p/django-command-extensions/wiki/RunProfileServer– https://developers.google.com/chrome-developer-tools/– https://docs.djangoproject.com/en/dev/howto/static-files/deployment/#staticfiles-
from-cdn– https://docs.djangoproject.com/en/dev/howto/custom-file-storage/– https://github.com/django-debug-toolbar/django-debug-toolbar/issues/319– https://github.com/sorl/sorl-
thumbnail/commit/e5f493a340dea8a33703c05fd66ee76d2f019881– https://github.com/aaugustin/django-c10k-demo– https://github.com/mindsocket/django-perf-example– https://www.djangopackages.com/grids/g/asset-managers/
![Page 52: The life of a web request - techniques for measuring and improving Django application performance](https://reader034.vdocument.in/reader034/viewer/2022052523/555a2e54d8b42ab3088b500f/html5/thumbnails/52.jpg)
Questions?
http://slideshare.net/mindsocket
Thank You!