buildout talk
DESCRIPTION
Tach TTRANSCRIPT
Buildout @Formspring@martincozzi
Monday, September 19, 2011
Formspring helps people find out more about each other by sharing interesting & personal responses
Monday, September 19, 2011
26M users
Monday, September 19, 2011
3.5B responses
Monday, September 19, 2011
Our Stack
Monday, September 19, 2011
Application
Monday, September 19, 2011
Application
Memcache
Monday, September 19, 2011
ApplicationRedis
Memcache
Monday, September 19, 2011
Application
MySQL
Redis
Memcache
Monday, September 19, 2011
Application
MySQL
Redis
Cassandra
Memcache
Monday, September 19, 2011
Application
MySQL
Redis SQS
Cassandra
Memcache
Monday, September 19, 2011
Application
MySQL
Redis SQS
Kestrel
Cassandra
Memcache
Monday, September 19, 2011
Application
MySQL
Redis SQS
Kestrel
Cassandra
Memcache
PHPQWPHPQWPHPQWPHPQWPHPQW
Monday, September 19, 2011
Application
MySQL
Redis SQS
Kestrel
Cassandra
Memcache
PythonQWPythonQWPythonQWPythonQWPythonQW
PHPQWPHPQWPHPQWPHPQWPHPQW
Monday, September 19, 2011
MySQL
Redis
Kestrel
Cassandra
Memcache
PythonQWPythonQWPythonQWPythonQWPythonQW
Monday, September 19, 2011
MySQL
RedisKestrel
Cassandra
MemcachePythonQW
PythonQW
Monday, September 19, 2011
MySQL
RedisKestrel
Cassandra
MemcachePythonQW
PythonQW
Monday, September 19, 2011
MySQL
RedisKestrel
Cassandra
MemcachePythonQW
PythonQW
Monday, September 19, 2011
Monday, September 19, 2011
Monday, September 19, 2011
Monday, September 19, 2011
Monday, September 19, 2011
Monday, September 19, 2011
Monday, September 19, 2011
class StreamPush:...
class SmileOnAnswers:...
class SmileConnection:...
Workers.py
class Inbox:...
Monday, September 19, 2011
Workers.py
• ./python2.6 kestrel_worker :
• Thread Count
• Queue Name
• Class Name
Monday, September 19, 2011
Queue-Stream
Queue-Smiles
Queue-Inbox
Deploy
code.tar.gz
Monday, September 19, 2011
Problems
• Big archive
Monday, September 19, 2011
Problems
• Big archive
• Dependencies
Monday, September 19, 2011
Problems
• Big archive
• Dependencies
• Package AMI
Monday, September 19, 2011
Problems
• Big archive
• Dependencies
• Package AMI
• Difficult to maintain / collaborate
Monday, September 19, 2011
What needs to be done
• Organize Codebase
Monday, September 19, 2011
What needs to be done
• Organize Codebase
• Split in folders / packages
Monday, September 19, 2011
What needs to be done
• Organize Codebase
• Split in folders / packages
• Manage dependencies automatically
Monday, September 19, 2011
Buildout
Monday, September 19, 2011
Buildout
• No installation needed
Monday, September 19, 2011
Buildout
• No installation needed
• 3 easy steps
Monday, September 19, 2011
Buildout
• Download bootstrap.py
Monday, September 19, 2011
Buildout
• Download bootstrap.py
• Run boostrap.py
Monday, September 19, 2011
Buildout
• Download bootstrap.py
• Run boostrap.py
• Run bin/buildout
Monday, September 19, 2011
Buildout Recipe
• Plugin mechanism
Monday, September 19, 2011
Buildout Recipe
• Plugin mechanism
• Main Recipes
Monday, September 19, 2011
Buildout Recipe
• Plugin mechanism
• Main Recipes
• zc.recipe.egg
Monday, September 19, 2011
Buildout Recipe
• Plugin mechanism
• Main Recipes
• zc.recipe.egg
• zc.recipe.cmmi
Monday, September 19, 2011
Buildout Recipe
• Plugin mechanism
• Main Recipes
• zc.recipe.egg
• zc.recipe.cmmi
• djangorecipe
Monday, September 19, 2011
Organize Codebase|-- fisher| |-- bootstrap.py| |-- buildout.cfg| |-- setup.py| `-- src| `-- fisher| `-- __init__.py|-- flake| |-- bootstrap.py| |-- buildout.cfg| |-- setup.py| `-- src| `-- flake| |-- README| |-- __init__.py| |-- flake.py| `-- static| |-- favicon.ico| `-- robots.txt|-- kestrel| |-- bootstrap.py| |-- buildout.cfg| |-- setup.py| `-- src| `-- kestrel
Monday, September 19, 2011
Identify Dependencies
Monday, September 19, 2011
Identify Dependencies
Monday, September 19, 2011
Identify Dependencies
Monday, September 19, 2011
Identify Dependencies
Monday, September 19, 2011
Identify Dependencies
Monday, September 19, 2011
Identify Dependencies
Monday, September 19, 2011
List Dependencies
Monday, September 19, 2011
List Dependencies
Monday, September 19, 2011
List Dependencies
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Run Buildout
Monday, September 19, 2011
Advantages
• Flexibility with dependencies.
Monday, September 19, 2011
Advantages
• Flexibility with dependencies.
• Build your own eggs.
Monday, September 19, 2011
Advantages
• Flexibility with dependencies.
• Build your own eggs.
• Test things !
Monday, September 19, 2011
Eggs?
Monday, September 19, 2011
Queue-Stream
Queue-Smiles
Queue-Inbox
Deploy
smiles.tar.gzinbox.tar.gzstream.tar.gz
RedisKestrel
Monday, September 19, 2011
Queue-Stream
Queue-Smiles
Queue-Inbox
Deploy
stream.tar.gzinbox.tar.gzsmiles.tar.gz
KestrelMemcachePyYAML
Monday, September 19, 2011
Queue-Stream
Queue-Smiles
Queue-Inbox
Deploy
stream.tar.gzsmiles.tar.gzinbox.tar.gz
FisherKestrel
Monday, September 19, 2011
Advantages
• Targeted deploys.
Monday, September 19, 2011
Advantages
• Targeted deploys.
• Less failures.
Monday, September 19, 2011
Advantages
• Targeted deploys.
• Less failures.
• Less work for Ops.
Monday, September 19, 2011
Advantages
• Targeted deploys.
• Less failures.
• Less work for Ops.
• Manage your versions.
Monday, September 19, 2011
PyPi.python.org goes down
Monday, September 19, 2011
Downtime ?
• Recipe to proxy PyPi.
Monday, September 19, 2011
Downtime ?
• Recipe to proxy PyPi.
• Cronjob.
Monday, September 19, 2011
Downtime ?
• Recipe to proxy PyPi.
• Cronjob.
• Local PyPi.
Monday, September 19, 2011
So .. Buildout
• Manage dependencies
Monday, September 19, 2011
So .. Buildout
• Manage dependencies
• Test libraries on the fly
Monday, September 19, 2011
So .. Buildout
• Manage dependencies
• Test libraries on the fly
• Easy to scale / deploy
Monday, September 19, 2011
So .. Buildout
• Manage dependencies
• Test libraries on the fly
• Easy to scale / deploy
• Highly Recommended =)
Monday, September 19, 2011
Questions ?
• Ask me on Formspring
• formspring.me/martincozzi
• formspring.me/formspringEng
Monday, September 19, 2011
We are hiring
• Ops
• Front-end
• Back-end
[email protected]@formspring.me
Monday, September 19, 2011