open source world '09 scaling web applications

8/14/2019 Open Source World '09 Scaling Web Applications

1/36

2009-08-12

Scaling Web ApplicationsGavin M. Roy, Chief Technology Officer - myYearbook.com

Wednesday, August 12, 2009


2/36

About myYearbook.com

Social Network, Top Teen Destination

2007

100M Page views per Month

2009 1.5B Page views per Month

Top 20 Page view site as ranked by comScore



3/36

Its a series of tubes.Former US Senator Ted Stevens



4/36

Early Stages

Emphasis on the application

Usually Monolithic and tightly coupled

Many base architectural decisions are made



5/36

Traffic InfluxPlaying whack-a-mole: the price of popularity



6/36

Pressure Point: Database

Database Strain

High IO

Slow Queries

Single Point of Failure Maintenance

High Connection Counts



7/36

Solutions for Database Scaling

Many methods for scaling databases

Replication

Sharding

Removing Joins Moving data to different servers

Caching



8/36

Solving Database IO Problems

Database in Memory

Expensive!

Linux Kernel Disk Buffer Cache

Works Very Well Faster Disks

Direct Attached > Remote Storage (SAN/NAS/etc)



9/36

Solving Slow Queries

Programmers are not DBAs

Proper indexing

Remove Complexity by reducing Joins

Data partitioning in OLTP

Good Schema > Fully de-normalized data



10/36

Solving Slow Queries

Sharding by User

CRC of Unique Key determines server where data lives

PostgreSQL via pl/proxy

MySQL via client code More complex to back-up

Support partial outages



11/36

Database Scaling via Replication

Multi-Master Replication

All servers must agree on data prior to commits (slow).

Consistent data across all servers

Master-Slave Replication

All writes on master server

Reads distributed across slaves



12/36

Service Oriented Architecture

Databases separated by application or service

Service data is stand alone from primary data

May need data from primary service

Need tools to keep data relevant

Allows for distributed application architecture



13/36

Solving Large Connection Counts

Connection Pooling

Java has this built in via JDBC connection pooling

Most other languages do not

Not Persistent Connections

Maintains Persistent Connections



14/36

Database Connection Pooling

PostgreSQL

pgBouncer - Skype

pgPool

MySQL

DBSlayer - NY Times

Cherokee Web Server



15/36

Caching

Everyones favorite daemon: memcached

Clients automatically pool and shard data

Key / Value Store

Volatile*

Economy of Scale

Failure of small memcache cluster none is impacting

Many nodes make light workWednesday, August 12, 2009


16/36

Caching Bottleneck: Network

Cache servers can transfer lots of data quickly

Keep regular insight into your network health



17/36

Removing Bottlenecks: Caching

Local Caching

Shared Memory (System V Shared Memory / Posix)

PHP - APC / Zend

File-system

memcached on web server?

Multi-tier caching, local caching then memcached cluster caching



18/36

Pressure Point: Web Application

Load Balancing

Tightly Coupled

High CPU Utilization

High IO and Shared Filesystem Data

Scaling Across Multiple Servers

Application Deployment



19/36

Spreading the Load

Hardware

Expensive

Software

By Service

DNS Round Robin

Relies on Client

Content Delivery NetworkWednesday, August 12, 2009


20/36

Decoupling Your Application

Web Applications spend a lot of time waiting

Database connection time

Database query time - Even writes

Caching service connection time

Enter Message Queues and Remote Workers



21/36

Decoupling via Message Queues

Offload data writes and actions

Offers granularity and control of impact

Scale workers to scale transaction rate

Many good brokers and standards

RabbitMQ, ActiveMQ, ZeroMQ, OpenAMQP, Java Message Service

Be language / vendor neutral



22/36

Decoupling via Remote Workers

Message Queues

Gearman

Hadoop

Map / Reduce

Offload costly CPU activity to farms of machines



23/36

Resolving File IO Slowness

Scale-Out NAS

Facebooks Haystack

PHP: Optcode Caching

Logging, syslog to remote servers



24/36

Scaling Across Many Servers

Common Configuration

memcached / tokyo tyrant

NAS stored configuration files

Health Checks

Function of Load Balancers

Need for additional insight



25/36

Application Deployment

Pull vs Push

Push works with low qty of hosts

Easy! Use scripts and ssh

Pull upon notification of code updates

Version control

Include an internal API call that returns version information



26/36

Knowing Your PipesWhy is the site down?



27/36

Clogged Pipes?

CPU Pegged

Database Bloat

Disk Issues

Network Issues

Lots of Traffic

Hardware Problems?

Crashed Daemons

Crashed Servers



28/36

Monitor Your Network

Servers

System Profile (CPU, Memory, Disk)

Network Hardware

Bandwidth

System Profile

Daemons



29/36

Monitor Your Network

Use Internal Services

Nagios

Hyperic

OpenNMS

Use External Services

Gomez

Alertra

Pingdom



30/36

Monitor your Application

Build Profiling into Application

How much time do I spend connecting to a database?

How much time do I spend in business logic?

Sample profile data in Application



31/36

Trending Application Behavior

How can I predict future utilization based upon existing growth?

Identify trends in application behaviors

Troubleshoot when something is just not right



32/36

StaplrOr whats my entire site up to?



33/36

Staplr: PostgreSQL



34/36

Staplr

Pollers

Apache HTTP, Lighttpd, Cherokee

SNMP (APC PDUs, Isilon NAS, Net-SNMPD)

Memcache

MySQL

PgBouncer

PostgreSQL

SysStat

and many more



35/36

Monitor Your Application

Use client profiling

JavaScript sends statistical data to server

Profile JavaScript action times

Page load times

Abandonment rates

Analytical data



36/36

Questions?

Follow me on twitter: http://twitter.com/crad

Contact me via email: [email protected]
http://twitter.com/cradmailto:[email protected]:[email protected]://twitter.com/cradhttp://twitter.com/crad

open source world '09 scaling web applications

Documents