tricks & challenges developing a large django application

Tricks & challengesdeveloping a large Django application

Simon WillisonEuroPython 2011, 22nd June

http://lanyrd.com/sfwcq



Lanyrd.com

Lanyrd.com

Definitive databaseof Professional Events

and Speakers

Lanyrd.com

Social event recommendationEvent networking tools

Archive of slides, notes and video

Definitive databaseof Professional Events

and Speakers

Part 1: Tricks

Trick #1: Signing

Lanyrd.com

Pass data through an untrusted source with confidence that it hasn't been tampered with

Lanyrd.com

Signing usesUnsubscribe" links in emails

lanyrd.com/un/ImN6VyI.ii0Hwm7p71DEcGfaVzziQaxeuu

Signed cookies "You are logged in as simonw" without hitting the database

http://lanyrd.com/un/ImN6VyI.ii0Hwm7p71DEcGfaVzziQaxRz9w

http://lanyrd.com/un/ImN6VyI.ii0Hwm7p71DEcGfaVzziQaxRz9w

Lanyrd.com

Signing in Django 1.4from django.core import signingsigning.dumps({"foo": "bar"})signing.loads(signed_string)

response.set_signed_cookie(key, value...)request.get_signed_cookie(key)

Trick #2: cache_version

add a conferenceadd a conference you are signed in as simonw, do you want to sign out?

calendarcalendar conferencesconferences coveragecoverage profileprofile

searchsearch

ON NOW

Django Plone Pyramid Python Twisted

EuroPython 2011Italy / Florence

19th–26th June 2011

SEPTEMBER

2011Django Open Source Python

Django Python

DjangoCon US 2011United States / Portland

6th–8th September 2011

PyCON FR 2011France / Rennes


OCTOBER PyCon DE 2011

Django events looking for participants1 Django event is looking for participants

Djangocoverage

By countryIreland 1

Django conferencesDjango conferences

19

6

17

4

52 videosMost recent added 3weeks ago

52 slide decksMost recent added 4hours ago

3 audio clipsMost recent added 1week ago

27 write-upsMost recent added 1week ago

11 handoutsMost recent added 18hours ago

3 notesMost recent added 10hours ago

Lanyrd.com

class Conference(models.Model): ... cache_version = models.IntegerField(default = 0)

def save(self, *args, **kwargs): self.cache_version += 1 super(Conference, self).save(*args, **kwargs)

def touch(self): Conference.objects.filter( pk = self.pk ).update( cache_version = F('cache_version') + 1 )

Lanyrd.com

{% cache 36000 conf-topics conference.pk conference.cache_version %} <ul class="tags inline-tags meta"> {% for topic in conference.topics.all %} <li><a href="{{ topic.get_absolute_url }}">{{ topic }}</a></li> {% endfor %} </ul>{% endcache %}

Lanyrd.com

from django.models import F

topic.conferences.all().update( cache_version = F('cache_version') + 1)

Bulk invalidation

Trick #3: NoSQL for denormalisation

Lanyrd.com

NoSQL = Not Only SQL

Lanyrd.com

Slides from a session on Lanyrd show up... Under "coverage of this conference"Under "slides by Simon Willison"Under "slides about Django"

But we like joins!

Lanyrd.com

Solution: denormalise data to Redis and Solr

Lanyrd.com

Try it

Ready for a test drive? Check this interactiveinteractive

tutorialtutorial that will walk you through the most

important features of Redis.

Redis is an open source, advanced key-value store. It is often

referred to as a data structure server since keys can contain

stringsstrings, hasheshashes, listslists, setssets and sorted sorted setssets.

Learn more Learn more →→

Download it

Redis 2.2.10 is the latest stable version.Redis 2.2.10 is the latest stable version.

Interested in legacy or unstable versions?

Check the downloads page.Check the downloads page.

What people are saying

More...More...

Comparison of CouchDB, Redis,MongoDB, Casandra, Neo4J &others http://j.mp/l32SqMhttp://j.mp/l32SqM via@DZone

@__NeverGiveup Oh YAY, oui tume redis ! *-* Hm, on s'rejoint à14h au bahut ? :o

JE L REDIS JE FOLLOW BACKSUR @Fuckement_TL

une question : "How to useServiceStack Redis in a webapplication to take advantage ofpub / sub paradigm"http://t.co/EOgyLU1http://t.co/EOgyLU1 #redis #web

Nice - Cassandra vs MongoDB vsCouchDB vs Redis vs Riak vsHBase vs Membase vs Neo4jcomparison http://bit.ly/l32SqMhttp://bit.ly/l32SqMfrom @kkovacs

This website is open source software developed by Citrusbyte. The Redis logo was designed by Carlos Prioglio.

Sponsored by

Commands Clients Documentation Community Download Issues

Lanyrd.com

simonw-follows:{144,21345,12328...}europython-attendees:{344,21345,787...}

redis.sinter( 'simonw-follows', 'europython-attendees')

Our redis sets...

Lanyrd.comadd a conferenceadd a conference you are signed in as simonw, do you want to sign out?


searchsearch

JUNE2011

Florencein Italy

EuroPython 2011EuroPython 2011The European Python Conference

You'respeakingAT THIS EVENT

(short URL)

119 speakers

9780

PEOPLE

attending

PEOPLE

tracking

TELL YOUR FRIENDS!

Tweet about thisevent

Topics

Django

Plone

Pyramid

Python

Twisted

19–26http://ep2011.europython.eu/

View the schedule on Lanyrd

Save to iCal / iPhone / Outlook /GCal

@europython

#europython

lanyrd.com/ccdpc

AndreasSchreiber@onyame

AndrewGodwin@andrewgodwin

AndriiMishkovskyi@mishok13

ArminRonacher

AlanFranzoni@franzeur

AlessandroDentella

Alex Martelli

Ali Afshar@aliafshar

AnnaRavenscroft

Anselm Kruis

Antonio Cuni@antocuni

Armin RigoEdit topics

Lanyrd.com

Main Wiki

apache > lucene > solr

Search the site with Solr Search

Powered by Lucid ImaginationLast Published: Sat, 04 Jun 2011 12:23:42 GMT

Welcome to Solr

What Is Solr?Get StartedNews

May 2011 - Solr 3.2 ReleasedMarch 2011 - Solr 3.1 Released25 June 2010 - Solr 1.4.1 Released7 May 2010 - Apache Lucene Eurocon 2010 Coming to Prague May 18-2110 November 2009 - Solr 1.4 Released20 August 2009 - Solr's first book is published!18 August 2009 - Lucene at US ApacheCon09 February 2009 - Lucene at ApacheCon Europe 2009 in Amsterdam19 December 2008 - Solr Logo Contest Results03 October 2008 - Solr Logo Contest15 September 2008 - Solr 1.3.0 Available28 August 2008 - Lucene/Solr at ApacheCon New Orleans03 September 2007 - Lucene at ApacheCon Atlanta06 June 2007: Release 1.2 available17 January 2007: Solr graduates from Incubator22 December 2006: Release 1.1.0 available15 August 2006: Solr at ApacheCon US21 April 2006: Solr at ApacheCon21 February 2006: nightly builds17 January 2006: Solr Joins Apache Incubator

What Is Solr?

PDF

About

WelcomeWho We Are

Documentation

Resources

Related Projects

Lanyrd.com



searchsearch

ON NOWFUTUREPAST

PAST

PUBLISHED

Topic: Django1 conference

5 conferences

51 conferences

Topic: GeoDjango1 conference

Practical Django ProjectsBy James Bennett

June 2009

Django Geek@djangeek

Learning Website Development with Django

FILTER BYtype

FILTER BYpast or futurePast events 55

Future events 5

FILTER BYtopicDjango 344

SearchSearchWe found 467 results for “django”

django SearchSearch

SLIDESDjango Con High Performance Django (slideshare.net)

from High Performance Django at DjangoCon 2008

Sessions 209

Coverage 190

Conferences 60

Books 5

Topics 2

People 1

Lanyrd.com



searchsearch

TODAY

We've found 182 conferences your Twitter contacts are

interested in.

From our blogWelcoming SophieBarrett to teamLanyrd

Today we have a very special

announcement (and for once,

it's not a new feature!) We

would like to welcome the

super-wonderful Sophie Barrett

to the Lanyrd team.

Session schedules inyour calendar

You can now subscribe to event

schedules in your calendar of

choice. Stay up to date at the

event with the schedule in the

pocket where you need it.

Venues (and venuemaps)

Your contacts' calendarYour contacts' calendaryours 24 contacts 182

Astronomy Science

Café Scientifique: Exploringthe dark side of starformation with the HerschelSpace Observatory

United Kingdom / Brighton

21st June 2011

4 contacts tracking

21 Attend

Track

Usability User Experience

Usability Professionals'Association – InternationalConference

United States / Atlanta

21st–24th June 2011

1 contact speaking and 3 contacts tracking

21 Attend

Track

Simon

Willison

Your profile

page

Lanyrd.com



searchsearch

Natalie DowneNatalie DowneUnfollowON TWITTER

1614

You currently follow @Natbat onTwitter

on Twitter

CONFERENCESspoken at

CONFERENCESinvolved with

Browse by year

2011 ! 2010 ! 2009 !2008 ! 2007 ! 2006 !2005

Appears with

CSS Maintainable CSS Maintainability

Design Process Internet Explorer Debugging

Show and tell Personal Projects

Web Standards Crowdsourcing

WildlifeNearYou RSI Community

Event Planning Geek Night User Experience

Usability Usability Testing JavaScript

Progressive Enhancement

Unobtrusive JavaScript Flexible design

Planning Artifacts Social Software

Motivation CSS3 CSS Layout

Project Management Dev Fort

Rapid Application Development

PAST CONFERENCES

spoken at

Five Pound App #22 Brighton, United Kingdom

30th March 2010

CSS3 WizardryWorkshop

Brighton, United Kingdom29th January 2010

Clearleft PechaKucha: Volume One

Brighton, United Kingdom22nd January 2010

BarCamp London 7 London, United Kingdom

24th–25th October 2009

Clearleft brown baglunch: Natalie Downe

Brighton, United Kingdom9th June 2009


28th–29th March 2009

Oxford Geek Night 11 Oxford, United Kingdom

25th March 2009

Flash Brighton - March24th, 2009

Brighton, United Kingdom24th March 2009

CSS systems withNatalie Downe

Brighton, United Kingdom4th March 2009

Namics InternationalTechnologyConference

Basel-City, Switzerland10th October 2008

BarCamp London 5 Richmond upon Thames,

United Kingdom27th–28th September 2008


31st May to 1st June 2008

Kiwi Foo 2008 Warkworth, New Zealand

1st–3rd February 2008


24th–25th November 2007


17th–18th February 2007

BarCamp London London, United Kingdom

2nd–3rd September 2006

PAST CONFERENCES

involved with

@media 2010 London, United Kingdom


Stage manager for the mainstage

Full Frontal 2009 Brighton, United Kingdom

20th November 2009

Volunteer

dConstruct 2009 Brighton, United Kingdom

4th September 2009

Volunteer

UX London 2009 London, United Kingdom


Volunteer

Skillswap on Speed Brighton, United Kingdom

29th October 2008

Organiser

<head> WebConference


Volunteer


5th September 2008

Volunteer

Skillswap Goes Pretty Brighton, United Kingdom

20th August 2008

Organiser

Skillswap goesCerebral

Brighton, United Kingdom9th July 2008

Organiser

Skillswap goesJavascript

Brighton, United Kingdom11th June 2008

organiser


25th July 2007

Organiser


11th April 2007

Organiser



Organiser


7th February 2007

Organiser

PAST CONFERENCES

attended

Frontiers ofInteraction 2011

Florence, Italy20th–21st June 2011

EuroPython 2011 Florence, Italy


Foo Camp 2011 Sebastopol, United States


DiscoveringMadagascar

London, United Kingdom25th May 2011

OpenTech 2011 London, United Kingdom

21st May 2011

Devnest - TwitterTuesday

London, United Kingdom3rd May 2011

Pub Standards LXV London, United Kingdom

14th April 2011

DevTank April London, United Kingdom

13th April 2011

London WebStandards: Fun andGames

Holborn, United Kingdom11th April 2011

Etsy Code as Craft Austin, United States

12th March 2011

SXSW Interactive2011

Austin, United States11th–15th March 2011

The LaunchConference

San Francisco, UnitedStates23rd–24th February 2011

Ignite Silicon Valley #3 Mountain View, United

States11th February 2011

Strata 2011 Santa Clara, United States

1st–3rd February 2011

Thirsty Bristol Bristol, United Kingdom

27th January 2011

San Francisco DjangoMeetup GroupNovember:Deployment Bonanza

San Francisco, UnitedStates17th November 2010

Think VitaminJavaScript

13th September 2010

State of the Map 2010 Girona, Spain

9th–11th July 2010

Explore 2009 London, United Kingdom


Future of Web AppsLondon 2009

London, United Kingdom30th September to 2ndOctober 2009

BarCamp Brighton 4 Brighton, United Kingdom




London Open HackDay 2009

London, United Kingdom9th–10th May 2009

Barcamp Austin 4 Austin, United States

14th April 2009



Future of Web AppsExpo 2008

London, United Kingdom8th–10th October 2008

BarCamp Brighton 3 University of Sussex, United

Kingdom6th–7th September 2008

reboot10 Copenhagen, Denmark



29th–30th May 2008

Accessibility 2.0 London, United Kingdom

25th April 2008

Future of Web Design- FOWD

London, United Kingdom17th–18th April 2008

Webstock 2008 Wellington, New Zealand


Web 2.0 Expo Berlin2007

Berlin, Germany5th–8th November 2007

Future of Web AppsExpo 2007

London, United Kingdom3rd–5th October 2007

BarCamp Brighton Brighton, United Kingdom




Hack Day London2007

London, United Kingdom16th–17th June 2007


7th–8th June 2007

Future of Web DesignLondon 2007

London, United Kingdom18th April 2007

Future of Web Apps2007

London, United Kingdom20th–22nd February 2007


8th September 2006

LUGRadio Live 2006 Wolverhampton, United

Kingdom22nd–23rd July 2006





The Future of WebApps

London, United Kingdom8th February 2006

Geek Dinner withDave Shea

London, United Kingdom23rd January 2006


11th November 2005

Google open house London, United Kingdom

10th November 2005

OpenTech 2005 Hammersmith, United

Kingdom23rd July 2005

FUTURE CONFERENCES

speaking at

London Insites TourEvening

London, United Kingdom19th July 2011

FUTURE CONFERENCES

attending

Activate - Summer2011

Kent, United Kingdom5th August 2011


2nd September 2011

Brooklyn Beta 2011 Brooklyn, United States


Full Frontal 2011 Brighton, United Kingdom


Explore 2011 London, United Kingdom


FUTURE CONFERENCES

tracking

The Guardian'sActivate Summit 2011

London, United Kingdom22nd June 2011

UX Tuesday, June2011

London, United Kingdom28th June 2011

Croydon Creatives Croydon, United Kingdom

29th June 2011

WDCNZ Wellington, New Zealand

14th July 2011

Science OnlineLondon Conference

London, United Kingdom2nd–3rd September 2011

Flash on the Beach2011

Brighton, United Kingdom11th–14th September 2011

PICNIC '11 Amsterdam, Netherlands


The Do Lectures 2011 Cardigan, United Kingdom


Fronteers 2011 Amsterdam, Netherlands


Playful 2011 London, United Kingdom

21st October 2011

Build 2011 Belfast, United Kingdom


“Co-founder of @lanyrd - Enthusiastic worldexplorer and digital creative.”

Session topics

Coverage

SEE ALL COVERAGE

11 items in total

FAVOURITE COVERAGE

See 6 items thatNatalie Downe hasfavourited

Book credits

TECHNICAL REVIEWER

The Ultimate CSSReferenceTECHNICAL EDITOR

CSS Mastery

@Natbat

9 times

Simon Willison

5 times

Nigel Crawley

4 times

Reinier Zwitserloot

4 timesNeil Crosby

4 times

Cristiano Betta

4 times

Matt Westcott

4 times

Gareth Rushgrove

bio from Twitter

Full speaking history for Natalie Downe

Online conference

Online conference

Online conference

3 videosMost recent added 7months ago

5 slide decksMost recent added 7months ago

1 write-upAdded 7 months ago

2 linksMost recent added 7months ago

faq ! blog ! privacy ! services ! colophonFollow @lanyrd on twitter. add a conferenceadd a conference

Trick #4: Hashed static asset filenames in CloudFront

Lanyrd.com

cdn.lanyrd.net/js/global.ed81d119.js

global.js

global.ed81d119.js

http://cdn.lanyrd.net/js/global.ed81d119.gz.js

http://cdn.lanyrd.net/js/global.ed81d119.gz.js

Lanyrd.com

BenefitsFar futures expiry headersCache-Control: max-age=315360000Expires: Fri, 18 Jun 2021 06:45:00 -0000 GMT

Guaranteed updated CSS in IEDeploy new assets in advance of applicationOld versions stick around for rollbacks

Lanyrd.com

./manage.py push_staticMinifies JavaScript and CSSRenames files to include sha1(contents)[:8]Pushes all assets to S3

Part 2: Challenges

Challenge #1: HTTP requests

Lanyrd.com

Talking to an API?What if it fails to load?What if it takes 30 seconds to load?

Lanyrd.com

A user gives you a URL...urllib.urlopen(url).read() ?

what if it's a huge file?what if it's a slow loading tarpit?what if it's the URL to your private, firewalled Solr instance?http://10.0.1.1/solr/select/?q=secrets

http://10.0.1.1/solr/select/?q=secrets

http://10.0.1.1/solr/select/?q=secrets

Lanyrd.com

Safe URL consumptionConnection timeoutsLogging and ProfilingHost validationHTTP caching / if-none-match / etcdjango.httpclient ?

Challenge #2: Profiling and debugging production systems

Lanyrd.com

Debugging in development rocks!

django_debug_toolbar = awesomeassert False = instant useful 500 pageimport pdb; pdb.set_trace() adds a command line debugger to your running application!

Lanyrd.com

DEBUG = FalseDEBUG = "BLIND"

Lanyrd.com

from django.views.debug import technical_500_responseimport sys

class UserBasedExceptionMiddleware(object): def process_exception(self, request, exception): if request.user.is_superuser: return technical_500_response(request, *sys.exc_info())

Lanyrd.com

mysql-proxyVery handy lua-customisable proxy for all of your MySQL trafficWorst documented software everlog.lua - logs out ALL querieshttps://gist.github.com/1039751

https://gist.github.com/1039751

https://gist.github.com/1039751

Lanyrd.com

django_instrumented(Unreleased) code I wrote for LanyrdCollects various runtime stats about the current request, stashes a profile JSON in memcachedWrites out the profile UUID as part of the HTMLA bookmarklet to view the profile

Lanyrd.com

Lanyrd.com

Django improvementsDEBUG as a global setting is an anti-patternMore low-level hooks for measuring, well, pretty much everythingLive profiling tools (as a third-party product)

MVC Mini Profiler

http://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html



Challenge #3: Zero downtime deploys

Lanyrd.com

Challenge: deploy upgrades, including database upgrades,without downtime

Lanyrd.com

Deploy DB changes separately from code changes

Make backwards compatible schema changesWe deploy code to a none-web server just so we can run "./manage.py migrate"(you are using South, right?)Use symlinks for instant rollback to previous code

Lanyrd.com

Read-only modeIf you can flip your site in to read-only mode, upgrades become a lot easier

Flip to read only modeReplicate DB to a new machineRun migrations, then testSwitch app servers to new database(On EC2, fire up an entire new web cluster)

Feature flags: finely grained version of this

Lanyrd.com

The lesson wekeep on learning

Lanyrd.com

Global settings are BADDatabase settings => multi-db (1.2)Cache settings => multi cache backends (1.3)Haystack backends => multi backends (v2.0)

Lanyrd.com

Global settings are BADDatabase settings => multi-db (1.2)Cache settings => multi cache backends (1.3)Haystack backends => multi backends (v2.0)

DEBUGTIME_ZONEMiddleware?

Lanyrd.com

Settings should be modifiable at run-time

We're hiring a developer/web operations person in London, come and talk to us!

Thank you!http://lanyrd.com/sfwcq

[ [

