rails conf europe 2007 - utilizing amazon s3 and ec2 in rails
DESCRIPTION
Scaling a web application is a very hard problem, especially for small project and teams who do not have sufficient manpower, money, and time to solve this problem. Luckily Amazon already had to solve this problem in their datacenters and offers their services to other developers. This talk will introduce the two most important Amazon Web Services, the Elastic Compute Cloud (EC2) and the Simple Storage Service (S3), and will present different ways to leverage them in your own Rails application. Presented at RailsConfEurope Berlin 2007TRANSCRIPT
Utilizing Amazon S3 and EC2in Rails
Jonathan Weiss, Peritor Wissensmanagement GmbHRailsConf Europe Berlin, 2007
2
Starting Point
One machine:
• Apache
• Ruby / Rails
• MySQL
3
Worst Case Populariy
4
A Difficult Path
5
Problem: Backup
• High availability
• Redundancy
• Very big data sets
6
Problem: File System
• Important files have to be accessed by many servers
• NFS / Samba not practical
7
Problem: Spontaneous Traffic
8
Problem: Load Fluctuation
9
Don‘t reinvent the wheel!
10
Amazon Web Services
Simple Storage Service - S3
Redundant Data Store
$0,15 per GB data per month
$0,10 - $0,20 per GB traffic
Virtual server per hour
$0,10 per CPU hour
$0,10 - $0,20 per GB traffic
Elastic Compute Cloud - EC2
…….
E-Commerce S..
11
S3 - Simple Storage Service
• Redundant storage - as much as you like
• max. 5 GB per object
• Organized in „Buckets“
• Web Service API for uploads
• Downloads via
• Web Service
• HTTP / HTTPS
• BitTorrent
12
S3 - Buckets
• Unique over all S3
• Contains manykey-value-metadata tupel
• Cannot contain other buckets!
• Key can contain „/“
MyBucket_name
S3
foto_5.jpg
backups/januar.zip
fotos/2007/001.png
www.peritor.com
site/screen.css
site/script.js
FreeBSD-6.2.iso
13
S3 with AWS::S3Upload
Download
14
EC2 - Elastic Compute Cloud
• Based on XEN virtualization
• On demand virtual servers - controlled with Web Service API
• 1,7 GHz
• 1,75 GB RAM
• 160 GB local file system (not persistent!)
• Use your favorite Linux distro (Linux 2.6.16),Amazon Machine Images (AMI) are stored on S3
• ACLs for hosts/ports access control
15
EC2 ToolsList availabe images
Start a new instance
Login with SSH
Shutdown instance
16
amazon-ec2 gemSetup
Usage
17
And now?
How does this solvemy problems?
18
S3 - Backup
• s3sync.rb
• Brackup
• Jungle Disk
• S3 FUSE
• s3DAV
• Duplicity
• S3Browser
• Firefox S3 Organizer extension
• …
19
s3sync.rb
Backup
Restore
20
S3 Asset Host
21
S3 Asset Host
Setup DNS
Rails configuration
22
S3 Asset Host
welcome.rhtml template
Output
23
S3 - Authenticated User Data
24
attachment_fu Rails plugin
Setup
25
attachment_fu Rails plugin
Upload
Download
26
On-Demand Computing with EC2
Time based, e.g. with cron
27
On-Demand Computing with EC2
Load based, e.g. with Monit
28
On-Demand Computing with EC2
29
EC2 for extra capacity
30
Load Balancer / Proxy
Example mod_proxy_blancer:
• Talks to multiple backend servers(Mongrel)
• Central Proxy/Load-Balancerconfiguration that has knowledgeabout nodes
• Typically proxy restart on configchange
31
Swiftiply
Swiftiply Proxy:
• Multiple backend clients have apersistent connection to the backendport
• Proxy listens on cluster port forrequests and forwards them
No re-configuration
32
Swiftiply Proxy
Configuration
Start
Installation
33
Swiftiplied Mongrel
• Gem plugin that patches Mongrel
• Transforms Mongrel in Swiftiply client
• Experimental
Start
or
34
EC2 on Demand before Swiftiply
35
EC2 on Demand with Swiftiply
36
EC2 on Demand with Swiftiply
37
Ressources
• Amazon Web Serviceshttp://aws.amazon.com
• Swiftiplyhttp://swiftiply.swiftcore.org
• Attachment_fuhttp://svn.techno-weenie.net/projects/plugins/attachment_fu/
• Elastic Railshttp://www.elasticrails.com
• Capazonhttp://capazon.rubyforge.org
• RightScalehttp://www.rightscale.com
38
Peritor Wissensmanagement GmbH
Lenbachstraße 212157 Berlin
Internet: www.peritor.comEmail: [email protected]
© Peritor Wissensmanagement GmbH - All Rights Reserved
Telefon: +49 (0)30 69 40 11 94Telefax: +49 (0)30 69 40 11 95