cloud to the rescue? how i learned to stop worrying and love the cloud

Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue

DCAMP

Cloud to the rescueHow I learned to stop worrying and love the cloud

1

Sunday, November 18, 12


DCAMP

2

About me

Hi, I’m Alexandru Badiu.I’m a software engineer, amateur game developer and part time Mister T impersonator.

Drupal user for 9 years, board member in Drupal Romania.

I work for Demotix.

Twitter @voidbergWeb ctrlz.ro

I pity the foolwho doesn’t use

Drupal



DCAMP

3

1The cloud



DCAMP

4

To the world“Cloud computing refers to the delivery of computing and storage capacity as a service to a heterogeneous community of end-recipients” - Wikipedia

To us“The cloud is just a buzz word meaning quick to set up VPSs when you need them, perhaps have 'unlimited' storage space and even a CDN if your lucky” - Demotix CTO

Web “2.0”Servers - Virtual Private Servers

Storage - Lots and lots of data

Delivery - CDNs and other tricks

The cloud



DCAMP

The promise

5

‣New servers anytime

‣ Scale up and down anytime

‣Unlimited storage

‣ Pay only for what you use

‣ Short / no contracts

‣ Total control



DCAMP

Demotix

6

‣ A quick 2 minute introduction

‣ http://prezi.com/_ugoyxm4qprv/demotixcom-2-minutes/

‣ Acquired by Corbis a couple of days ago


http://prezi.com/_ugoyxm4qprv/demotixcom-2-minutes/





DCAMP

Demotix numbers

7

‣ 1.5million+ page views a month

‣ Very very long tale of content

‣ Spikes where we get 100k's views in a single day

‣ 14 GB+ database

‣ 1M+ nodes, even more terms (10M+)

‣ 3-5k new nodes per day

‣ 200GB+ files directory

‣ Some folders have over 750K files

‣ 80% unauthenticated traffic

‣ 6k active users



DCAMP

Demotix before the cloud

8

‣ 1 Web server

‣ Varnish

‣ Apache, mod_php

‣ 1 Database server

‣ MySQL

‣ 1 Util server

‣ Memcached

‣ Solr

‣ FTP Server

‣ Beanstalk



DCAMP

Previous cloud experience

9

‣ Started on the cloud

‣ Prototype and alpha built a cloud

‣ Dev environment still on a cloud

‣ Moved off for better support

‣ We needed 24/7 support

‣ Site liked to crash on major Christian holidays

‣ Saw a nice speed bump when we did

‣ Better disk IO

‣ Dedicated hardware



DCAMP

Why go back?

10

‣ Spikes

‣ Single story spikes

‣ Breaking images of the Oslo bomb story went viral

‣ Our firewall (not webserver) went into melt down

‣ Event based spikes

‣ Royal wedding

‣ Elections

‣ Olympics

‣ Dev = Staging = Production

‣ Flexibility



DCAMP

Dream setup

11

‣ 1+ Load balancer

‣ Varnish

‣ 3+ Web servers

‣ Nginx, PHP-FPM

‣ 2+ Database servers

‣ MariaDB

‣ 2+ Util server

‣ Memcached

‣ Solr

‣ FTP Server

‣ Beanstalk



DCAMP

Why do it yourself?

12

Why not pay someone else to figure it out?

‣Control - SaaS is a blackbox in most cases

‣Complexity - We did not fit generic solutions

‣ If you can do it, go ahead



DCAMP

13

2Challenges



DCAMP

14

Slice ratio (How big is your share?)‣ Most (but not all) cloud providers slice servers up into equal cpu/

ram/disk space chunks and sell them off.

‣ Some let you add on resources you need, others don't.

‣ Problem for file servers.

Poor disk IO‣ Are your disks local or over the network?

#1 - Disks



DCAMP

15

Laws‣ EU has laws about where you can send and store users data.

‣ Your business may not allow for you to store data in the cloud.

Users‣ Cloud may not be where your users are.

‣ CDNs can help.

#2 - Location



DCAMP

16

More servers, more problems‣ Once your past dedicated use servers (web, db etc) you have to

do work on the app to support more servers.

Help‣ Can you get any? And how fast?

‣ How about at 3am?

‣ On Christmas day

#3 - Support



DCAMP

17

You need to be a sysadmin or become one

Servers crash at 3AM‣ Monitoring systems

‣ Emergency instructions

‣ Disaster recovery

#4 - Total control



DCAMP

18

Cloud providers differ a LOT‣ Different slice rations

‣ Different bandwidth allocation methods

‣ Different support packages

‣ Very different performance

Overselling‣ Happens a lot

#5 - 1 != 1



DCAMP

19

3Benchmarks



DCAMP

20

Where to start?

Benchmarks‣ Learn to

benchmark Drupal

‣ Cloudharmony.com



DCAMP

21

DB benchmarks



DCAMP

22

Disk/IO benchmarks



DCAMP

23

Mem/IO benchmarks



DCAMP

24

CPU benchmarks



DCAMP

25

Make sure they can do what you need‣ CDN?

‣ Storage?

‣ Good support?

‣ Is it going to break the bank?

Try before you buy‣ If there’s a free testing period, use it

‣ Put as close to a 1:1 copy of your site

Hammer it‣ AB / Siege test the cache system

‣ Replay your log files with HTTPERF or Jmeter

‣ Monitor the results with Munin/Cacti/NewRelic

Then?



DCAMP

26

4Storage



DCAMP

27

Storage = sites/*/files

More than 1 web server

The slice ratio fails for file servers

Lots of files in a single directory (10k+)

https://www.getpantheon.com/news/inside-pantheon-valhalla-filesystem

Storage







DCAMP

28

Server with big data + NFS

Tried, tested, works

Not very exciting

Single point of failure

Depending on cloud you could need a 64 GB RAM server

NFS



DCAMP

29

API storage + Fuse (Cloudfuse, s3fs)Theoretically unlimited storage

Easy CDN

A bit bleeding edge

Gluster FSImpressive

Tradeoffs that Drupal doesn’t need

Complicated but likely worth it

API / Gluster



DCAMP

30

5CDNs



DCAMP

31

The problem with CDNs



DCAMP

32

Push (CloudFiles, Amazon S3)No redundant traffic

Complex setup

Pull (Lots)Very easy to setup

Lots of redundant traffic

DNS (Cloudflare / Incapsula)Takes over your DNS

Does much more (analytics, protection, optimization)

Types of CDNs



DCAMP

33

6What did we do?



DCAMP

34

What did we do?

We picked Dediserve

http://dediserve.com





DCAMP

35

NFS StorageSimple, tried and tested

Using a NFS mount of raid disks used by all webheads

CDN PullVery easy to setup

Dediserve’s OnApp CDN

Spread assets over several domains (front end performance)

What did we do?



DCAMP

36

Our current setup‣ 1 Load balancer

‣ Varnish

‣ 4 Web servers

‣ Apache, mod_php

‣ 1 Database servers

‣ MariaDB

‣ 1 Util server

‣ Memcached

‣ Solr

‣ FTP Server

‣ Beanstalk



DCAMP

37

7Other issues and findings



DCAMP

38

Close to 1M files in a single directoryWrote custom module to fix this

Part of the file save hooks it moved the file to a folder structure based on the filename

Will be released on d.o

Cloudfuse + CDNWorking proof of concept

Uses Rackspace’s cloud files for both storage and CDN

Requires pre-generation of imagecache

Not published yet

Lots of files



DCAMP

39

Puppet is awesomeSo is any other build automation system

Spinning up new test or production servers is very very easy

Great to setup local dev, stage or merge

url_alias is dog slow on InnoDBRemoving the language conditions speeds it up

Convert term_data’s description to VARCHARMySQL won’t use in memory tables if you query TEXT fields

What we found



DCAMP

40

8Conclusions



DCAMP

41

The Cloud isn't a magic bullet

Cloud providers differ a LOT

Benchmarking can help differentiate providers

The best benchmark is your own app, with replayed traffic.

Storage can be complicated

CDNs can give you an easy win

Conclusions



DCAMP

42

Thanks!Questions?

Alexandru Badiu.Twitter @voidbergWeb http://ctrlz.ro

Email [email protected]

D.O http://drupal.org/user/8662


http://ctrlz.ro

http://ctrlz.ro

mailto:[email protected]

mailto:[email protected]

http://drupal.org/user/8662

http://drupal.org/user/8662

cloud to the rescue? how i learned to stop worrying and love the cloud

Technology