Plone in the CloudA CMS hosted on Amazon EC2

Nate Aune (Jazkarta.com)PyCon 2009Chicago, IL

1Tuesday, April 7, 2009

• President of Jazkarta (based in Boston)

• 3 Full-time and 10 freelance developers

• Python-based open source web applications

• Members of core Plone framework team

• Member of the Plone Foundation Board

2Tuesday, April 7, 2009

In the next 30 min...

• Amazon Web Services (AWS)

• AWS with Python

• Plone with AWS

• Tools for launching & remote deployment

• Challenges of using AWS

• Monitoring and maintaining EC2 instances

3Tuesday, April 7, 2009

What is Amazon Web Services?

4Tuesday, April 7, 2009

What is Amazon Web Services?

• Suite of elastic IT infrastructure services

• Requisition computing power and storage

• Run any application you want on any OS

5Tuesday, April 7, 2009

Benefits of AWS

• Metered pricing (per hour / per GB)

• no large up-front capital investments

• only pay for what you use

• Scaleable hosting architecture

• Redundancy for high availability

6Tuesday, April 7, 2009

AWS Architecture

• EC2 = Elastic Computing Cloud

• S3 = Simple Storage Solution

• EBS = Elastic Block Storage

• Elastic IP = Static IP addresses

7Tuesday, April 7, 2009

Elastic Computing Cloud (EC2)amazon.com/ec2

• Use an existing AMI (Amazon Machine Image) to launch a new instance

• Or create your own AMIs (even paid ones)

• Attach EBS volumes to your EC2 instance

8Tuesday, April 7, 2009

Elastic Block Storage (EBS)

9Tuesday, April 7, 2009

So how does Python play into all this?

10Tuesday, April 7, 2009

Enter Boto11Tuesday, April 7, 2009

What is Boto?

• Python package that provides interfaces to Amazon Web Services

• Thanks to author Mitch Garnaat!

• Boto supports:

• S3 via the REST API

• EC2 via the Query API

• and more...

12Tuesday, April 7, 2009

Install Boto & Bototools$ sudo easy_install boto

$ svn co http://bototools.googlecode.com/svn/trunk/ bototools

$ cd bototools

$ sudo python setup.py install

$ sudo python ./src/bototools/postinstall.py -p /usr/local/bin

• This will put these commands in /usr/local/bin:

• ec2start, ec2stop, ec2ip, ec2ls, ec2img

• s3ls, s3rm, s3cp

13Tuesday, April 7, 2009

Using Bototools

• Set these environment variables:

• AWS_ACCESS_KEY

• AWS_SECRET_ACCESS_KEY

• EC2_SSH_KEY

• Can also put in ~/.boto or /etc/boto.cfg

14Tuesday, April 7, 2009

Find an AMI

• We will use the Ubuntu 8.10 Intrepid (32-bit) from Alestic.com - ami-7cfd1a15

• More can be found at:http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=171

15Tuesday, April 7, 2009

Launch an instance$ ec2start -a ami-7cfd1a15 -k gsg-keypair

Reservation:r-0f26a666

[Instance:i-8e3154e7]

$ ec2ls

Reservation r-0f26a666

i-8e3154e7 pending None 2009-03-25T01:49:24.000Z

...wait...

$ ec2ls

Reservation r-0f26a666

i-8e3154e7 running None 2009-03-25T01:49:24.000Z ec2-174-129-238-8.compute-1.amazonaws.com

16Tuesday, April 7, 2009

Associate a static IP$ ec2ip -g

Address:174.129.238.8

$ ec2ip -l

174.129.238.8 ->

$ ec2ip -a 174.129.238.8 i-8e3154e7

True

$ ec2ip -l

174.129.238.8 -> i-8e3154e7

• Note: once you assign a static IP, you won't be able to access the machine using the original IP.

17Tuesday, April 7, 2009

Log in

$ ec2ls

Reservation r-0f26a666

i-8e3154e7 running None 2009-03-25T01:49:24.000Z ec2-174-129-238-8.compute-1.amazonaws.com

$ ssh -i ~/id_rsa-gsg-keypair root@ec2-174-129-238-8.compute-1.amazonaws.com

Linux domU-12-31-39-00-E5-14 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686

root@domU-12-31-39-00-E5-14:~#

• Time to play!

18Tuesday, April 7, 2009

Stop the instance

• Be sure to stop the instance or you will be charged!

$ ec2stop i-8e3154e7

Stopping ['i-8e3154e7']

$ ec2ls

Reservation r-0f26a666

i-8e3154e7 terminated None 2009-03-25T01:49:24.000Z

19Tuesday, April 7, 2009

Use the AWS Console

20Tuesday, April 7, 2009

Or use ElasticFox

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609&categoryID=88

• Firefox extension to start/stop EC2 instances, assign IPs, associate EBS volumes and create keypairs.

21Tuesday, April 7, 2009

Case studyPlone on EC2

22Tuesday, April 7, 2009

What is Plone?

• Open source content management system (CMS)

• Built on top of Zope

• Written in Python

• Large international community

• Used by Oxfam, NASA, Novell, Akamai, FSF, etc.

• more listed at http://plone.net/sites

23Tuesday, April 7, 2009

Why Plone on EC2?

• Too few Zope hosting providers

• Avoid renting rackspace in co-lo

• Flexible and scalable architecture

• Complete control of servers

+

24Tuesday, April 7, 2009

What is PondCMS?

• Hosted and supported version of Plone

• Eliminate the complexities of hosting

• Turnkey - automated and repeatable

• Bundled with certified add-ons

PondCMS = Plone On Demand

www.jazkarta.com/pondcms

25Tuesday, April 7, 2009

PondCMS Technology stack

!"#$%&!'()Virtual Servers

*#"+,-./+Operating System

*#"+,-./+Web App Server

*#"+,-./+PondCMS + add-ons

0%1&2

HTTP proxy

Caching proxy

Load Balancer

26Tuesday, April 7, 2009

One server deploymentHTTP

Apache

(HTTP)

Varnish

(Caching)

Pound

(Load balancing)

Zope Client #1

1

Zope Client #1

2

Zope Client #n

n

Zeo Server

Zeo Backup

27Tuesday, April 7, 2009

Add Zeo clients as demand increases

28Tuesday, April 7, 2009

HTTP

public IP: 72.44.50.237

Apache

Varnish

Pound

zope-1.web.private:8222

Zope client #1

1

zope-2.web.private:8222

Zope client #2

2

zope-n.web.private:8222

Zope client #n

n

Data.fs Backup

Apache logs

Data.fs

Frontend server

zeo-1.web.private:8221

Zeo server #1

Elastic Block Storage

Elastic Block Storage

Application Servers

Database server

Website visitor

Simple Storage Solution (S3)

Data

files

Multi server deployment

29Tuesday, April 7, 2009

Goal: automate everything

Scripts for repeatable deployment

1. Bootstrap the server

2. Install dependencies

3. Install PondCMS (includes Plone + add-ons)

4. Configure backups

5. Create the Plone site and Apache vhosts

30Tuesday, April 7, 2009

1. Bootstrap server

1. Create the EC2 instance

2. Create an EBS volume

3. Associate volume with EC2 instance

4. Acquire IP address

5. Associate IP with EC2 instance

31Tuesday, April 7, 2009

2. Install dependencies1. Install packages with apt-get

• zope, python, varnish, apache2, pound, postfix

2. Initialize EBS volume and mount as ext3 fs

3. Store Apache log files on EBS volume

4. Register server with Landscape

5. Set up Pound to start automatically

6. Set up Pound to point to Zeo clients

32Tuesday, April 7, 2009

3. Install PondCMS

1. Make a 'zope' user

2. Create a virtualenv

3. Make Zope data dirs on EBS volume

4. Download PondCMS tarball release (incl. Plone)

5. Install PondCMS into virtualenv

33Tuesday, April 7, 2009

4. Configure backups

• Configure logfile rotation

• Setup backup scripts (using repozo)

• Setup sync of backups to S3 (using Duplicity)

• Create cron jobs for backups (daily, weekly)

34Tuesday, April 7, 2009

5. Create Plone site & vhosts

1. Use Fabric (remote deployment tool) to:

• Make Plone site incl. CacheFu setup

• Make vhost for accessing site on port 80

2. Create Apache vhosts for:

• supervisor.domainname.com (start/stop Zope)

• admin.domainname.com (ZMI access)

35Tuesday, April 7, 2009

What is Fabric?

36Tuesday, April 7, 2009

What is Fabric?http://www.nongnu.org/fab/

• Simple remote deployment tool

• Similar to Capistrano (Ruby on Rails)

• Deploy applications to multiple similar hosts

37Tuesday, April 7, 2009

@require('host', 'site_id')

@fabricset(host='fab_hosts', user='fab_user', key='fab_key_filename')

def createplonesite(host=None, user=None, key=None, site_id=None,

public_host=None):

"""Create a new plone site """

if not site_id:

site_id = host

installzcml(host=host, user=user, key=key, package='pondcms.basicapp')

restart(host=host, user=user, key=key)

put('createplonesite.py', '/tmp/createplonesite.py')

s = SERVICE+'/bin/utility run /tmp/createplonesite.py -a pondcms.basicapp -s '+site_id

if public_host:

s += ' --site-url='+public_host

run(s)

Setup site Fabric script

38Tuesday, April 7, 2009

@require('host', 'vhost_name')

@fabricset(host='fab_hosts', user='fab_user', key='fab_key_filename')

def setupvhost(host=None, user=None, key=None, vhost_name=None):

"""Setup an apache2 vhost"""

site = VHOST_TEMPL % {'vhost_name': vhost_name}

tmpfile = '/tmp/'+get('fab_timestamp')+'_vhost'

f = open(tmpfile, 'w')

f.write(site)

f.close()

targetf = vhost_name+'.conf'

target = '/etc/apache2/sites-available/' + targetf

put(tmpfile, target)

run('a2ensite vhost_name')

run('/etc/init.d/apache2 reload')

Setup vhost Fabric script

39Tuesday, April 7, 2009

Running Fabric scripts$ cd fabric

$ source bin/active

(fabric)$ cd fabs/pondcms

(fabric)$ fab createplonesite:host=test.pondcms.net,user=root,key=~/id_rsa-gsg-keypair,site_id=test.pondcms.net,public_host=http://test.pondcms.net

...

(fabric)$ fab setupvhost:host=test.pondcms.net,user=root,key=~/id_rsa-gsg-keypair,vhost_name=test.pondcms.net

40Tuesday, April 7, 2009

Challenges• Persistent data must be stored on EBS

• Static IPs

• Mail sent from EC2 instances flagged as spam

• Monitoring and maintenance of VMs

• Landscape

• ZenOss

• Nagios

• Hyperic41Tuesday, April 7, 2009

42Tuesday, April 7, 2009

AWS Status history

43Tuesday, April 7, 2009

Landscapemonitoring / updates

• http://landscape.canonical.com

44Tuesday, April 7, 2009

Landscape update pkgs

45Tuesday, April 7, 2009

Thank you!Questions?

Follow me on Twitter: natea

Or email me at:natea (at) jazkarta (dot) com

46Tuesday, April 7, 2009

More info

• Amazon EC2: http://amazon.com/ec2

• Boto: http://code.google.com/p/boto

• Bototools: http://code.google.com/p/bototools/

• Fabric: http://www.nongnu.org/fabric

• Fabulatr: http://code.google.com/p/fabulatr/

• Plone: http://plone.org

• PondCMS: http://jazkarta.com/pondcms

47Tuesday, April 7, 2009

Other resources• Ubuntu on EC2:

http://groups.google.com/group/ec2ubuntu

• Ubuntu AMIs: http://www.alestic.com

• Beta program: http://www.ubuntu.com/ec2

• Landscape: http://landscape.canonical.com

• vm-builder-ec2: http://packages.ubuntu.com/intrepid-updates/utils/python-vm-builder-ec2

• Enomaly's Elastic Computing Platform:http://src.enomaly.com/

48Tuesday, April 7, 2009

Photo credits

• http://www.flickr.com/photos/docman/66137352/

• http://www.flickr.com/photos/thomashawk/126534906/

• http://www.flickr.com/photos/moeview/498544386/

• http://www.flickr.com/photos/shellysblogger/2464969383/

• http://www.flickr.com/photos/mhw/56386662/

• http://www.flickr.com/photos/mrtopf/1541527579/

49Tuesday, April 7, 2009