moving to the cloud
DESCRIPTION
"The Cloud" is a buzzword these days, but in this talk Nate Aune and Sally Kleinfeldt describe what you need to know about public and private clouds, why it's important, and why you might want to consider it for your company/organization. We'll discuss the pros/cons of public cloud providers such as Amazon Web Services and Rackspace Cloud, and how you can build your own private cloud using OpenStack or Eucalpytus, or even a hybrid that leverages both public and private clouds. Lastly, we'll look at how Plone can be deployed onto the cloud, and best practices for setting up your Plone site in a flexible and scalable way to take full advantage of all that these cloud providers have to offer. In particular, we'll examine the Connexions/Rhaptos project as a case study, and how they benefited from an Amazon-backed infrastructure. Video of the presentation is available here: https://streaming.psu.edu/media/?movieId=13392TRANSCRIPT
Moving to the Cloud
Nate Aune (Presented by Sally Kleinfeldt)
Jazkarta.comPlone Symposium East 2011
Penn State University
Agenda
• Benefits of cloud computing
• What services does Amazon provide?
• 5 minute launch of Plone on EC2
• Case study: Rhaptos hosting on EC2
• Questions?
What makes the Cloud so attractive?
Abstract resourcesFocus on your needs, not on hardware specs.
As your needs change, so should your resources.
On-demand provisioningAsk for what you need, exactly when you need it.
Get rid of it when you don’t need it.
Scalability in minutesScale out or in depending on usage needs.
Pay per consumptionNo contracts or long-term commitments.
Pay only for what you use.
Efficiency of ExpertsUtilize the skills, knowledge and resources of experts.
Most applications need:
• Compute
• Storage
• Messaging
• Payment
• Distribution
• Scale
• Analytics
Amazon delivers this
Characteristics of Truly Scalable Service
• Increasing resources results in a proportional increase in performance
• capable of handling heterogeneity, loosely coupled
• operationally efficient
• resilient
• becomes more cost effective when it grows
Flexible infrastructure
• Elastic Computing Cloud (EC2)
• Amazon Machine Images (AMI)
• Elastic IPs
• Elastic Block Storage (EBS)
• EBS bootable AMI
• Auto-scaling
• Elastic Load Balancing (ELB)
Other services
• Simple Storage Solution (S3)
• CloudFront (CDN)
• CloudWatch (monitoring)
Compared to Rackspace
• Rackspace is still catching up to AWS (just announced load balancing a few weeks ago)
• Rackspace is a bit easier to get started with
• sends you the root password
• machines already have a static IP
• the disk storage is persisted
• Prices are comparable to AWS
• Cheapest Rackspace VM is $14/mo
Private clouds
• Provision virtual machines on your own hardware
• Pros: you control your own machines.
• Cons: you have to manage your own machines
• Examples: Eucalyptus, OpenStack
http://open.eucalyptus.comhttp://openstack.org/
http://uec-images.ubuntu.com/
Text
mr.awsomeBuildout recipe to manage EC2 instances
Add an AWS part to buildout[buildout]
parts = aws
[aws]
recipe = zc.recipe.egg
eggs = mr.awsome
entry-points =
aws=mr.awsome:aws
assh=mr.awsome:aws_ssh
arguments = configpath="${buildout:directory}/etc"
Make a etc/aws.conf file[securitygroup:demo-server]description = Bristol Demo Serverconnections = tcp 22 22 0.0.0.0/0 tcp 80 80 0.0.0.0/0 tcp 8080 8080 0.0.0.0/0
[instance:demo-server]keypair = bristol-keypairsecuritygroups = demo-serverregion = us-east-1placement = us-east-1ainstance_type = t1.microimage = ami-480df921startup_script = startup-demo-serverfabfile = fabfile.py
Make a bash script: etc/startup-demo-server
#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
wget http://www.enfoldsystems.com/pubkey.gpg
apt-key add ./pubkey.gpg
echo "deb http://dist.clients.enfoldsystems.com/ubuntu lucid universe" >> /etc/apt/sources.list
apt-get update && apt-get upgrade -y
apt-get install plone-default -y
/etc/init.d/plone-default
Set your access keys
#!/bin/bash
export AWS_ACCESS_KEY_ID="<your_access_key_id>"
export AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"
Set these environment variables.
$ source setkeys.sh
$ env
File: setkeys.sh
Start the demo server$ ./bin/aws start demo-server
INFO: Instance 'demo-server' unavailable
INFO: Creating instance 'demo-server'
INFO: Instance created, waiting until it's available
....
INFO: Instance 'demo-server' available
INFO: Instance running.
INFO: Instances DNS name ec2-50-16-25-92.compute-1.amazonaws.com
INFO: Instances public DNS name ec2-50-16-25-92.compute-1.amazonaws.com
Benefits of EC2• Seamless disaster recovery with repeatable
deployments using buildout and mr.awsome
• Easy launching of testing, staging and QA servers that are isolated from your production environment (stop when not used to avoid paying for them)
• Pay-for-what-you-use metered pricing to launch servers on-demand during peak periods
• Auto-scaling to launch new Zeo client machines and shut them down when no longer needed
•
Connexions & RhaptosA case study in deploying OSS to the cloud
s
Log In Contact Us Report a Bug
Search
Home Content Lenses About Us Help MyCNX
FEATURED CONTENT
New features
1 2 3
MY ACCOUNT
SPOTLIGHT
You are here: Home
Connexions is:a place to view and share educational material made ofsmall knowledge chunks called modules that can beorganized as courses, books, reports, etc. Anyone may viewor contribute:
authors create and collaborate
instructors rapidly build and share custom collections
learners find and explore content
More about us ...
Advanced Algebra IIThe world of second-yearalgebra comes alive in KennyFelder's Advanced Algebra II.Felder, a high school mathteacher in North Carolina,designed his course using anontraditional approach to a verytraditional subject. Centeredaround a Homework andActivities book, along withaccompanying Conceptual
Explanations and Teacher's Guide collections,Advanced Algebra II tackles topics by focusing oncomprehension rather than straight lecture delivery.
Advanced Algebra II was selected by California's FreeDigital Textbook Initiative to be included in a list offree textbooks available to California schools startingin Fall 2009.
Collaborative StatisticsCollaborative Statistics waswritten by two faculty members
FIND CONTENT
16547 reusable modules woven into 1007 collections.
or browse by ...
Language
Popularity
Title, author, etc.
CREATE CONTENT
Creating content in Connexions is as easy as 1, 2, 3:
Get an accountand log in to
your workspace.
Make a modulefrom scratch or
convert it from aWord doc.
Publish yourworks, sharing
them with the world.
Jump right in
Get an account
How to create a module inminutes
How to create a collectionwith existing modules
Guides and tutorials
New author guide
Connexions Tutorial andReference
Username
Password
Log in
Get an account
Forgot your password?
User feedback"I enjoy usingtheConnexionssite and thephilosophybehind itencouragesone to learnnew fields and
to network with others. Yoursite is my new homeschool onthe internet. It is helping meto grow as a writer ofacademic materials as I readwhat others have written so asto shape my own. I am alsoamazed at the range ofinstitutions where your writerscome from. Thanks for the useof these great facilities."
Go
ArtsBusinessHumanitiesMathematics and StatisticsScience and TechnologySocial Sciences
Search Content
Subject
Search
Login Preferences Help/Guide About Trac
Wiki Timeline Roadmap Browse Source View Tickets New Ticket Search
Start Page Index History Last Change
Welcome to Connexions and Rhaptos Software DevelopmentConnexions ( cnx.org) is legos for education. It is a world-wide repository and publishing system to build textbooks, courses, lessons, andjournals. Individual authors, major content publishers, and educational organizations all contribute. Lenses provide a way for high-quality materialto be endorsed, recommended, categorized, and aligned to standards. Rhaptos is the open source software that runs the Connexions site andEnterprise Rhaptos is the software for running private content installations. Read more about Connexions and Rhaptos mission and architecturehere.
Enterprise Rhaptos Connexions and Rhaptos Development
API's for Accessing Data Quality Assurance and Testing
Enterprise Rhaptos
Enterprise Rhaptos enables the creation, sharing, modification, and vetting of educational materials accessible to anyone, anywhere, anytime viathe World Wide Web. Rhaptos' unique features of permanent versions of all materials, a simple, semantic document language, and a powerfullensing system for post-publication quality control, customized tagging, and community-based search and discovery are the engine behind a trulyreusable repository of knowledge and learning. Enterprise Rhaptos allows companies and individual organizations to host their own privateauthoring platform and publishing repository.
Installation Instructions for Enterprise Rhaptos
API's for Accessing Connexions Data
Table of ContentsWelcome to Connexions and Rhaptos Software DevelopmentEnterprise RhaptosAPI's for Accessing Connexions Data
Accessing content, feeds, and statisticsDownloading, transforming and branding contentSoftware Architecture and Application Programming Interfaces
Connexions Rhaptos DevelopmentDeveloper Resources (Getting involved)Roadmap, Designs, DesiresStarting Points for Connexions and Rhaptos Software Development
Quality Assurance and TestingTest, Release, and Configuration DocsUsing the Connexions Rhaptos Trac Bug System
Release and Configuration Details for Connexions and RhaptosMore about TRAC
Virtualization of Rhaptos
Two primary reasons to move to the cloud:
1) Hurricane territory (disaster recovery)
2) Increase adoption by other universities
Cost was also a consideration but not a primary reason for carrying out the project.
Deploy to Amazon EC2 with Python scripts
• Define server profiles with OS dependencies and turn into an AMI (Amazon Machine Image)
• Install & configure application on EBS volume
• Take snapshot of volume
• Launch new AMIs and attach EBS volume created from saved snapshot.
Plone Conference 2009 – Federico C. Guizzardi – 21 Aprile 2009
Distributed Architecture
Internet
Webserver
Cache Sys
Load Balancer
s1
s3
s4 s5
s2
10
proxy
frontend1
frontend2
frontend3
backend
[macro:base-securitygroup]connections = tcp 22 22 0.0.0.0/0 tcp 80 80 0.0.0.0/0
[securitygroup:backend]<= macro:base-securitygroupdescription = Bristol backend machine running Zeo server
[securitygroup:frontend1]<= macro:base-securitygroupdescription = Bristol frontend machine running Zeo client 1connections = tcp 8080 8080 0.0.0.0/0
[securitygroup:frontend2]<= macro:base-securitygroupdescription = Bristol frontend machine running Zeo client 2connections = tcp 8080 8080 0.0.0.0/0
[securitygroup:proxy]<= macro:base-securitygroupdescription = Bristol proxy machine running Apache, Varnish, HAProxy
aws.conf - define security groups
[macro:base-instance]image = ami-480df921keypair = bristol-keypairregion = us-east-1placement = us-east-1cinstance_type = t1.microfabfile = fabfile.py
[instance:backend]<= macro:base-instancesecuritygroups = backendstartup_script = backend-startup.ship = xxx.xxx.xxx.xxx
[instance:frontend1]<= macro:base-instancesecuritygroups = frontend1 startup_script = frontend-startup.ship = xxx.xxx.xxx.xxx
[instance:frontend2]<= macro:base-instancesecuritygroups = frontend2 startup_script = frontend-startup.ship = xxx.xxx.xxx.xxx
[instance:proxy]<= macro:base-instancesecuritygroups = proxy startup_script = proxy-startup.ship = xxx.xxx.xxx.xxx
[buildout]parts = backend-script frontend-script proxy-script
[backend-script]recipe = collective.recipe.templateinput = deployment/templates/zope-startup-script.shoutput = deployment/backend-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = backend.cfg
[frontend-script]recipe = collective.recipe.templateinput = deployment/templates/zope-startup-script.shoutput = deployment/frontend-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = frontend.cfg
[proxy-script]recipe = collective.recipe.templateinput = deployment/templates/proxy-startup-script.shoutput = deployment/proxy-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = proxy.cfg
#!/bin/bashset -e -xexport DEBIAN_FRONTEND=noninteractiveapt-get update && apt-get upgrade -y
echo "----- Install system packages for Plone"apt-get -y install python2.6-dev subversion build-essential
echo "----- Create an unprivileged user zope with password disabled"adduser zope --disabled-password --gecos ""
echo "----- Check out the buildout"cd /home/zopeecho 'p\n' | svn co http://svn.jazkarta.com/.../.../trunk buildout
echo "----- Make the zope user the owner of this directory"chown -R zope:zope buildout
echo "----- Run the buildout as user zope"cd buildoutsudo -u zope python2.6 bootstrap.py -c backend.cfgsudo -u zope ./bin/buildout -v -c backend.cfg
echo "----- Start Zeo processes"sudo -u zope ./bin/supervisordsudo -u zope ./bin/supervisorctl status
echo "----- Make sure that Supervisor starts on reboot"ln -s /home/zope/buildout/etc/supervisor /etc/init.d/supervisorchmod +x /etc/init.d/supervisorupdate-rc.d supervisor defaults
[buildout]backend-dns-name = ec2-174-129-192-63.compute-1.amazonaws.comfrontend1-dns-name = ec2-174-129-116-33.compute-1.amazonaws.comfrontend2-dns-name = ec2-184-73-240-104.compute-1.amazonaws.com
Must define the FQDN of the Zeo clients and
Zeo servers
When these names are resolved *on* the server, the DNS server returns the private IP, which means you avoid paying
bandwidth charges for outgoing/incoming traffic.
$ ./bin/aws start backend$ ./bin/aws start frontend1$ ./bin/aws start frontend2$ ./bin/aws start proxy
Start up the servers with one command
Fault tolerantEliminating single points of failure.
Plone Conference 2009 – Federico C. Guizzardi – 21 Aprile 2009
Distributed Architecture
Internet
Webserver
Cache Sys
Load Balancer
s1
s3
s4 s5
s2
10
02/21/08
The following diagram summarizes this information.
Heartbeat monitors Zeo server and moves to backup in case of failure
DRDBDistributed storage system similar in principle in RAID,but which works over the network.
Improvements to the deployment process
• time to launch a new site was days
• brittle non-repeatable installation process
• delay in procuring hardware and installing OS dependencies
Before After
Improvements to the deployment process
• time to launch a new site was days
• brittle non-repeatable installation process
• delay in procuring hardware and installing OS dependencies
• time to launch a new site was minutes
• streamlined and automated installation process
• virtual machines are launched instantly and OS dependencies are already installed on the disk image.
Before After
FabricScripts to deploy to remote hosts.
http://www.fabfile.org
Fabricset(fab_user='plone',
fab_hosts=['localhost'],
local='/home/plone/prod/',
remote='/opt/zope/prod/',
site='budapest.buildout')
def deploy():
local('cd $(local)$(site)')
local('tar cvfz $(site).tgz buildout --exclude=.svn --exclude=*.pyc')
run('cd $(remote)$(site); ./bin/supervisorctl shutdown')
run('rm -rf $(remote)$(site)')
put('$(site).tar.gz', '$(remote)$(site).tar.gz')
run('cd $(root)$(site) && tar zxf $(site).tar.gz')
start()
def start():
run('cd $(remote)$(site)')
run('./bin/supervisord')
Deploy
$ fab deploy
• Provides a local packaging of the source into a tarball
• Removes the old remotely deployed code
• Puts the source on the remote host
• Restarts the site
What does this one command do?
Benefits of Fabric
• Chain local-command, remote-command, and remote-copy commands
• Substitute variables, and the splitting of distinct operations (like in the case of the 'restart' command) so you run them separately, with a fab restart, for example.
Example Fabric file
• See Alex Clark’s Fabric file:
• http://github.com/aclark4life/Debian-Deploy-Plone/blob/master/fabfile.py
Boto
• Python library to script AWS services
• Programmatically:
• Launch new EC2 instances
• Attach EBS volumes
• Create S3 buckets
http://code.google.com/p/boto/
collective.hostoutBuildout-based recipe for deploying to remote host
http://plone.org/products/collective.hostout
[buildout]extends = buildout.cfg
parts += hostout prod
auto-checkout += collective.hostout
[sources]
collective.hostout = svn https://svn.plone.org/svn/collective/collective.hostout/trunk
[hostout]recipe = collective.hostout
user = root
#password = blahidentity-file = /Users/nateaune/id_rsa-gsg-keypair
pre-commands =
# ${buildout:directory}/bin/supervisorctl shutdown || echo 'Unable to shutdown'# post-commands =
# ${buildout:directory}/bin/supervisord
#effective-user = zope
[prod]
recipe = collective.hostout
extends = hostouthost = ec2-75-101-211-135.compute-1.amazonaws.com
buildout = buildout.cfg
path = /opt/zope/prod
Run the deploy commands
$ ./bin/buildout -NInstalling prod.Generated script '/bin/hostout'.
$ bin/hostout deployInvalid hostout hostouts are: prod
$ bin/hostout deploy prod...
Chef & Puppet• Configuration management tools to prepare
machines to behave according to their role.
• Unlike Fabric which “pushes” config to the server, with Chef/Puppet, the server “pulls” it’s configuration from a master node.
• Usually overkill for a simple deployment, but with a multi-server deployment can be useful, especially to keep machines up-to-date.
http://opscode.com/chef/http://www.puppetlabs.com/
Questions?
• http://rhaptos.org
• http://pypi.python.org/pypi/mr.awsome
• http://fabfile.org
• http://code.google.com/p/boto/
• http://pypi.python.org/pypi/collective.hostout