Download - Effective EC2
Effective EC2PyCon Italia Qu4ttro - May 7/8/9 2010
Valentino Volonghi - [email protected]
Wednesday, May 19, 2010
Buzz
Effective EC2 - Valentino Volonghi
Why I’ll never own another server...Joe Stump - Digg Lead Architect, CTO SimpleGeo
http://stu.mp/2010/04/why-ill-never-own-another-server.html
Cloud computing economies of scaleJames Hamilton - VP & Distinguished Engineer, Amazon Web Services
http://live.visitmix.com/MIX10/Sessions/EX01
Moving to the cloudReddit Team
http://blog.reddit.com/2009/11/moving-to-cloud.html
Wednesday, May 19, 2010
Buzzwords
Agility
Cost
Location Independence
Multi-tenancy
Reliability
Scalability
Security Maintenance Metering
Source: WikipediaEffective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Competition...
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Google AppEngine
• Transparently scalable
• No maintenance headaches
• Software must be written ad hoc
• Little control over infrastructure
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Rackspace Cloud
• CloudServers/CloudFiles/CloudSites
• Some nice additional features
• Fewer datacenters and can’t pick them
• 12/11/07, 09/07/08, 03/11/09
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Others
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Amazon AWS
• 4 independent regions - 10 data centers
• 3rd party vendor support
• Well integrated services accessible via API
• “Poor” single-node performance
• Flexible upper limits
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Amazon AWS
Source: www.businessweek.com/technology/content/apr2010/tc20100428_085106.htmEffective EC2 - Valentino Volonghi
Amazon Web Services can be as bigas our retail business
Andy Jassy - Amazon Senior VP Cloud Computing Business
Revenue: $650.000.000
Market Share:77%
Wednesday, May 19, 2010
AdRoll
• Scaling High ROI Display Advertising
• 60M paid campaigns impressions
• 200M advertiser pixels impressions
• 99% of requests below 3ms think time
• Realtime Dynamic Ads
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
AdRoll 2008
• ServePath Housing
• NaviSite CDN with mod_python
• Self-Hosted DNS
• MySQL DB
• NFS
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
AdRoll 2009
• ServePath Housing
• Amazon EC2 auto-managed AdServers
• Amazon S3/CloudFront CDN
• Dynect DNS Global Server Load Blancing
• MySQL DB
• NFS
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
AdRoll 2010
• Fully Amazon EC2
• Akamai with Amazon S3 as origin
• Dynect DNS GSLB
• PostgreSQL DB
• Amazon S3 instead of NFS
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Akam
ai
S3
AdR
oll.c
om
Log
Proc
essi
ng
DB
Sche
dulin
gSy
ncin
g
3rd party platform
sA
dServers
Plan
Logs/Performance
Campaigns/Reports/Plan
Ads
Plan/Config
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
AdRoll 2010
Effective EC2 - Valentino Volonghi
Network Uptime: 100%
Avg Resp Time: 96ms
Measured by Pingdom
Wednesday, May 19, 2010
MigrationPhase I - May 2009
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
New Ad Servers
• Very Quick & faultless boot procedure
• Easy software upgrade
• Low latency network
• Real time monitoring
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
New Ad Servers
• Custom bundled AMI stored in S3
• Python deploys the AdServer on boot
• Hidden real load test
• Boto monitoring
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Boto monitoringdef regional_max_allocation(start_adserver=ec2.Image.start_adserver): zones = [zone.name for zone in ec2.bconn.get_all_zones()]
(availables_with_ip, availables_without_ip, software_failures, startings) = get_instances_by_status()
def setup_candidate(failure=None, free_ip=None): if availables_without_ip: new_instance = get_instance_in_unused_zone(availables_with_ip, availables_without_ip, lenient=True) if failure is not None: free_ip = failure.disassociate_elastic()
availables_without_ip.remove(new_instance) availables_with_ip.add(new_instance) new_instance.associate_elastic(free_ip) return
if startings: startings.pop() return
start_adserver("adserver-" + utils.uuid(), zone=get_unused_zone(availables_with_ip, zones))
for software_failure in software_failures: setup_candidate(failure=software_failure)
free_ips = ec2.get_all_free_elastic_ip(default=[]) for free_ip in free_ips: setup_candidate(free_ip=free_ip)Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
MigrationPhase 2 - March 2010
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Preparation
• SQLAlchemy is awesome
• PostgreSQL 5-10x faster than MySQL
• 1 EBS is slower than 8 in RAID!! :)
• m1.small instances are useless
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Automated Deploy
• Stock Ubuntu AMI
• Setup scripts on S3
• Fabric
➡ 5 minutes to boot a new instance
➡ No maintenance overhead
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Automated Deploy#!/bin/bash -ex
wget -O- run.alestic.com/install/runurl | bashrunurl "https://$base_s3/setup/add-keys-and-cert?SIGNATURE"runurl $base_s3/machine/dbrunurl $base_s3/deploy/db
Deploy DB
#!/bin/bash -ex
wget -O- run.alestic.com/install/runurl | bashrunurl "https://${base_s3}/setup/add-keys-and-cert?SIGNATURE"runurl $base_s3/machine/webrunurl $base_s3/deploy/webrunurl $base_s3/start/service production.ini paster
Deploy Web
#!/bin/bash -ex
wget -O- run.alestic.com/install/runurl | bashrunurl "https://${base_s3}/setup/add-keys-and-cert?SIGNATURE"runurl $base_s3/machine/apprunurl $base_s3/deploy/apprunurl $base_s3/start/service production.ini watchwoman_inotify watchwoman_pixels watchwoman_amqp spaceman \ crawlman billman generalman turtl quexecutor sphinx
Deploy App
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
DB Instance
• m2.2xlarge - 34GB RAM - 4 VCPU
• 8 EBS RAID 1+0
• PostgreSQL 8.4 log shipping to S3
• RAID snapshot every 6 hours
• Nightly SQL Backup
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Generic Instance
• m1.large - 7.5 GB RAM - 2 VCPU
• 1 EBS with 500GB space
• Store frequently changed data
• Exact copies of each other
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Web Instance
• c1.medium - 1.7 GB RAM - 2 VCPU
• No EBS
• Amazon ELB frontend
• Easily replaceable
• Logs aggregated separately in real time
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
AWS Tips/Tricks
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
runurlby Eric Hammond. Download and run any script from
any URL.
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
ec2-consistent-snapshotby Eric Hammond, atomic snapshot of EBS volumes
with xfs filesystem.
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Eric HammondIf you want to know more about Amazon AWS,
he’s the one you want to talk to.
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Use Public addressesPublic addresses are automatically resolved to internal
addresses when possible.
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Use ELB and Cloudwatch
Very useful for latency and load monitoring… It also makes scaling the web frontend extremely easy!
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
EBS RAIDEBS is cheap. Instead of one big EBS create one big
soft-RAID/LVM volume from many of them, it’s faster and safer.
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
SMTPAuth / SendGridMail sending from Amazon EC2 was/is crippled. Use a
third party service to improve deliverability.
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Use /etc/hostsSetting up a DNS is a lot of work at the beginning. Use SSH key names to setup a custom /etc/hosts file in each
instance.
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
new_lines = ["127.0.0.1 localhost"]for reservation in conn.get_all_instances(): for instance in reservation.instances: if instance.state == "running": KEYS.discard(instance.key_name) new_lines.append( "%s %s.internal %s" % ( socket.gethostbyname(instance.public_dns_name), instance.key_name, instance.key_name) )
for missing_key in KEYS: new_lines.append( "127.0.0.1 %s.internal %s" % (missing_key, missing_key) )
f = open('/etc/hosts.new', 'wb')f.write("\n".join(new_lines))f.write("\n")f.flush()f.close()shutil.move('/etc/hosts.new', '/etc/hosts')
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
AWS Wishlist
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Elastic DNS ServiceDid I say that setting up your own DNS is hard?
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Dedicated Rack Housing
Try to get a Netezza rack in AWS today...
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Improve ELBIt’s a CNAME…
No HTTPS termination
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Thank you!
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Answers?
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010