buildout talk

Buildout @Formspring@martincozzi

Monday, September 19, 2011

Formspring helps people find out more about each other by sharing interesting & personal responses

Monday, September 19, 2011

26M users

Monday, September 19, 2011

3.5B responses

Monday, September 19, 2011

Our Stack

Monday, September 19, 2011

Application

Monday, September 19, 2011

Application

Memcache

Monday, September 19, 2011

ApplicationRedis

Memcache

Monday, September 19, 2011

Application

MySQL

Redis

Memcache

Monday, September 19, 2011

Application

MySQL

Redis

Cassandra

Memcache

Monday, September 19, 2011

Application

MySQL

Redis SQS

Cassandra

Memcache

Monday, September 19, 2011

Application

MySQL

Redis SQS

Kestrel

Cassandra

Memcache

Monday, September 19, 2011

Application

MySQL

Redis SQS

Kestrel

Cassandra

Memcache

PHPQWPHPQWPHPQWPHPQWPHPQW

Monday, September 19, 2011

Application

MySQL

Redis SQS

Kestrel

Cassandra

Memcache

PythonQWPythonQWPythonQWPythonQWPythonQW

PHPQWPHPQWPHPQWPHPQWPHPQW

Monday, September 19, 2011

MySQL

Redis

Kestrel

Cassandra

Memcache

PythonQWPythonQWPythonQWPythonQWPythonQW

Monday, September 19, 2011

MySQL

RedisKestrel

Cassandra

MemcachePythonQW

PythonQW

Monday, September 19, 2011

MySQL

RedisKestrel

Cassandra

MemcachePythonQW

PythonQW

Monday, September 19, 2011

MySQL

RedisKestrel

Cassandra

MemcachePythonQW

PythonQW

Monday, September 19, 2011

Monday, September 19, 2011

Monday, September 19, 2011

Monday, September 19, 2011

Monday, September 19, 2011

Monday, September 19, 2011

Monday, September 19, 2011

class StreamPush:...

class SmileOnAnswers:...

class SmileConnection:...

Workers.py

class Inbox:...

Monday, September 19, 2011

Workers.py

• ./python2.6 kestrel_worker :

• Thread Count

• Queue Name

• Class Name

Monday, September 19, 2011

Queue-Stream

Queue-Smiles

Queue-Inbox

Deploy

code.tar.gz

Monday, September 19, 2011

Problems

• Big archive

Monday, September 19, 2011

Problems

• Big archive

• Dependencies

Monday, September 19, 2011

Problems

• Big archive

• Dependencies

• Package AMI

Monday, September 19, 2011

Problems

• Big archive

• Dependencies

• Package AMI

• Difficult to maintain / collaborate

Monday, September 19, 2011

What needs to be done

• Organize Codebase

Monday, September 19, 2011

What needs to be done

• Organize Codebase

• Split in folders / packages

Monday, September 19, 2011

What needs to be done

• Organize Codebase

• Split in folders / packages

• Manage dependencies automatically

Monday, September 19, 2011

Buildout

Monday, September 19, 2011

Buildout

• No installation needed

Monday, September 19, 2011

Buildout

• No installation needed

• 3 easy steps

Monday, September 19, 2011

Buildout

• Download bootstrap.py

Monday, September 19, 2011

Buildout

• Download bootstrap.py

• Run boostrap.py

Monday, September 19, 2011

Buildout

• Download bootstrap.py

• Run boostrap.py

• Run bin/buildout

Monday, September 19, 2011

Buildout Recipe

• Plugin mechanism

Monday, September 19, 2011

Buildout Recipe

• Plugin mechanism

• Main Recipes

Monday, September 19, 2011

Buildout Recipe

• Plugin mechanism

• Main Recipes

• zc.recipe.egg

Monday, September 19, 2011

Buildout Recipe

• Plugin mechanism

• Main Recipes

• zc.recipe.egg

• zc.recipe.cmmi

Monday, September 19, 2011

Buildout Recipe

• Plugin mechanism

• Main Recipes

• zc.recipe.egg

• zc.recipe.cmmi

• djangorecipe

Monday, September 19, 2011

Organize Codebase|-- fisher| |-- bootstrap.py| |-- buildout.cfg| |-- setup.py| `-- src| `-- fisher| `-- __init__.py|-- flake| |-- bootstrap.py| |-- buildout.cfg| |-- setup.py| `-- src| `-- flake| |-- README| |-- __init__.py| |-- flake.py| `-- static| |-- favicon.ico| `-- robots.txt|-- kestrel| |-- bootstrap.py| |-- buildout.cfg| |-- setup.py| `-- src| `-- kestrel

Monday, September 19, 2011

Identify Dependencies

Monday, September 19, 2011

Identify Dependencies

Monday, September 19, 2011

Identify Dependencies

Monday, September 19, 2011

Identify Dependencies

Monday, September 19, 2011

Identify Dependencies

Monday, September 19, 2011

Identify Dependencies

Monday, September 19, 2011

List Dependencies

Monday, September 19, 2011

List Dependencies

Monday, September 19, 2011

List Dependencies

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Run Buildout

Monday, September 19, 2011

Advantages

• Flexibility with dependencies.

Monday, September 19, 2011

Advantages

• Flexibility with dependencies.

• Build your own eggs.

Monday, September 19, 2011

Advantages

• Flexibility with dependencies.

• Build your own eggs.

• Test things !

Monday, September 19, 2011

Eggs?

Monday, September 19, 2011

Queue-Stream

Queue-Smiles

Queue-Inbox

Deploy

smiles.tar.gzinbox.tar.gzstream.tar.gz

RedisKestrel

Monday, September 19, 2011

Queue-Stream

Queue-Smiles

Queue-Inbox

Deploy

stream.tar.gzinbox.tar.gzsmiles.tar.gz

KestrelMemcachePyYAML

Monday, September 19, 2011

Queue-Stream

Queue-Smiles

Queue-Inbox

Deploy

stream.tar.gzsmiles.tar.gzinbox.tar.gz

FisherKestrel

Monday, September 19, 2011

Advantages

• Targeted deploys.

Monday, September 19, 2011

Advantages

• Targeted deploys.

• Less failures.

Monday, September 19, 2011

Advantages

• Targeted deploys.

• Less failures.

• Less work for Ops.

Monday, September 19, 2011

Advantages

• Targeted deploys.

• Less failures.

• Less work for Ops.

• Manage your versions.

Monday, September 19, 2011

PyPi.python.org goes down

Monday, September 19, 2011

Downtime ?

• Recipe to proxy PyPi.

Monday, September 19, 2011

Downtime ?

• Recipe to proxy PyPi.

• Cronjob.

Monday, September 19, 2011

Downtime ?

• Recipe to proxy PyPi.

• Cronjob.

• Local PyPi.

Monday, September 19, 2011

So .. Buildout

• Manage dependencies

Monday, September 19, 2011

So .. Buildout

• Manage dependencies

• Test libraries on the fly

Monday, September 19, 2011

So .. Buildout

• Manage dependencies

• Test libraries on the fly

• Easy to scale / deploy

Monday, September 19, 2011

So .. Buildout

• Manage dependencies

• Test libraries on the fly

• Easy to scale / deploy

• Highly Recommended =)

Monday, September 19, 2011

Questions ?

• Ask me on Formspring

• formspring.me/martincozzi

• formspring.me/formspringEng

Monday, September 19, 2011

We are hiring

• Ops

• Front-end

• Back-end

martin@formspring.merob@formspring.me

Monday, September 19, 2011