high performance social networking applications php mysql

7/29/2019 High Performance Social Networking Applications PHP MySQL

1/131

!"#$%'()*#+(,-.$/#+*0#,-$1#-2

!

Writing a socialapplication in PHP/

MySQLand what happens when amillion people show up on

opening day

Duleepa Dups WijayawardhanaMySQL Community Team


2/131

Who the hell am I?


3/131

Who the hell am I?

PHP/MySQL Developer since 1999/2000


4/131

Who the hell am I?

PHP/MySQL Developer since 1999/2000 MySQL Community Relations Manager in

North America since... oh... last week :)


5/131

Who the hell am I?



Web Developer for MySQL


6/131

Who the hell am I?



Web Developer for MySQL Various positions at BioWare Corp.

2001-2007


7/131

Who the hell am I?



Web Developer for MySQL Various positions at BioWare Corp.

2001-2007

Also run the Annual St. Patricks Day DrunkDial (http://www.stpatsdrunkdial.com)
http://www.stpatsdrunkdial.com/http://www.stpatsdrunkdial.com/http://www.stpatsdrunkdial.com/http://www.stpatsdrunkdial.com/


8/131

What makes me qualified?


9/131

Spent a lot of time screaming at Murphy.



10/131

Spent a lot of time screaming at Murphy. Hired by BioWare to help create the

BioWare Community Site for Neverwinter

Nights (launched June 2002)



11/131




Had no clue what to expect, but the rest asthey say now is history.



12/131





Learned lots. Still learning...



13/131





Learned lots. Still learning... Insanity...



14/131

What are we going to cover?


15/131


A History of Disaster


16/131


A History of Disaster Launching the BioWare

Community


17/131


A History of Disaster Launching the BioWare

Community Pain Points of an Application


18/131



19/131


What skill must you have to be a webdeveloper/sysadmin in charge of a majorweb application?


20/131



The answer: You must be slightlyinsane!


21/131



The answer: You must be slightlyinsane!

... and perhaps slightly masochistic :)


22/131



23/131


Web Server Crashes (typical)


24/131


Web Server Crashes (typical) Load Balancer Crashes


25/131



Major Mail System crashes


26/131



Major Mail System crashes File System Crashes (or how I came tohate NFS)


27/131




Database Crashes (sigh)


28/131




Database Crashes (sigh) Power outage in the building


29/131



30/131


Sweden has an incredibly hot summer.Who knew.


31/131


Sweden has an incredibly hot summer.Who knew. Building nearby burns down and takes

down the city grid, we sit on thebalcony and watch the fire...


32/131




The toilet explodes and floods.


33/131




The toilet explodes and floods.

Someone connects the storm drain tothe kitchen sink. Oops.


34/131



35/131


Someone pours water on the electricmainboard and explodes your mainelectrical supply the day before a release. BigOops.


36/131



37/131


Imagine running throughone amazingly crazyblizzard, drunk as youwatch transformers

explode and the sweepingcone of darkness spreadacross the city.... You havethe presence of mind to do

a sequenced shut down butyou cant see straight tobring anything back up soyou sleep on the server

room floor to sober up.


38/131

What is a Social Application?


39/131


A site which primarily focuses on interactions

between users.


40/131



between users.

MySQL.com is not a social application,


41/131



between users.


Web 2.0 applications: Facebook, LinkedIn,

MySpace, most community sites.


42/131



between users.


Web 2.0 applications: Facebook, LinkedIn,

MySpace, most community sites.

Developing and Launching a social applicationhas special challenges


43/131

Neverwinter Nights Community...


44/131

A primer for launching a social

application...


45/131


application...

1. Plan to have about 10 times the number ofusers that you conservatively expect.


46/131


application...


2. Be prepared for getting 100 times if your

marketing has been good.


47/131


application...



marketing has been good.3. Be prepared to scale *EVERY* aspect of

your application: Web, DB, Mail etc.


48/131


application...





4. Be smart, launch softly.


49/131


application...





4. Be smart, launch softly.

5. Be even smarter, dont launch on a Fridayevening.


50/131

Before the launch

All cocky and sure of myself

What could go wrong?


51/131

After the launch

A picture is worth a thousandwords


52/131

What we did (Dont do at home!)


53/131


Launched on a Friday afternoon, idea wasto have less traffic.


54/131



Site contained a function to send an alert ifdatabase was down, when site went down,it triggered 5,000+ emails in a few mins andtook down the mail server


55/131



Site contained a function to send an alert ifdatabase was down, when site went down,it triggered 5,000+ emails in a few mins andtook down the mail server

Not enough slaves to allow the site tofunction. Ripped apart desktop computersto create functional DB slaves.


56/131

Key to Success


57/131

Key to Success

Figure out the pain points of anapplication


58/131

Key to Success


Be prepared to scale every part of yourapplication.


59/131

Key to Success


Be prepared to scale every part of yourapplication.

Be prepared to sacrifice performancefor availability, chances are good youwont be doing the other way around


60/131

Key to Success


61/131

Key to Success

Become omniscient and omnipotent.


62/131

Key to Success

Become omniscient and omnipotent. Identify Single Points of Failure (SPoF)...


63/131

Key to Success

Become omniscient and omnipotent. Identify Single Points of Failure (SPoF)...

If you have an SPoF... guaranteed it willfail

SP F d h t t


64/131

SPoFs and how to get

yourself fired :)

SP F d h t t


65/131


yourself fired :)

Do an SPoF audit on your application,SPoFs can be:

SP F d h t t


66/131


yourself fired :)


external dependencies (isp etc.)

SP F d h t t


67/131


yourself fired :)


external dependencies (isp etc.) physical infrastructure: power.

SP F d h t t


68/131


yourself fired :)



people

SP F d h t t


69/131


yourself fired :)



people

servers (db, web, load, firewall, dns...)

SPoFs and ho to get


70/131


yourself fired :)



people

servers (db, web, load, firewall, dns...)

application hooks/CRONs

P i P i t #1 Th W b d Fil


71/131

Pain Point #1: The Web and FileServers

Pain Point #1 The Web and File


72/131


A typical PHP application with lots ofvisitors will have to run on a cluster of webservers.

Pain Point #1 The Web and File


73/131



Centralized file server or pushed filesystem?

Pain Point #1: The Web and File


74/131



Centralized file server or pushed filesystem?

Centralized file server can be a bottleneck,

pushed file system limits some programmingoptions.


75/131

Pain Point #2: The Database

How will you configure the database. Master/Slave?


76/131


Sharding? More common amongst newersocial applications.


77/131


Perhaps MySQL Proxy? We ran MySQL Proxy as a test on

MySQL.com, its getting there!

http://forge.mysql.com/wiki/MySQL_Proxy
http://forge.mysql.com/wiki/MySQL_Proxyhttp://forge.mysql.com/wiki/MySQL_Proxyhttp://forge.mysql.com/wiki/MySQL_Proxy


78/131


Perhaps look at Cloud options such asAWS. Allows growth at the least cost and lets

someone else handle the problem of scalingfor traffic!


79/131

Pain Point #3: The Mail Server


80/131


Most social applications depend on vastquantities of emails to be sent out.


81/131


Most social applications depend on vastquantities of emails to be sent out. What happens when your SMTP server

gives up the ghost? Do you run SMTPservers on your web servers? Isolate theSMTP Servers?


82/131


Most social applications depend on vastquantities of emails to be sent out. What happens when your SMTP server

gives up the ghost? Do you run SMTPservers on your web servers? Isolate theSMTP Servers?

We dumped mail into a MySQL Db andsent with custom daemon.

Pain Point #4: Controlling Master/


83/131

Pain Point #4: Controlling Master/Slave Writes

Almost every application of this kindobviously splits out reads to read slaves andwrites to masters.

Use some sort of DNS based loadbalancing on your DB servers to sendqueries?


84/131

Pain Point #5: Data Caching

Replicated setups == Replication Lag. Replicated Forum software particularly

vulnerable with increased traffic, missing

posts etc.

Slave dependent queries for IDs etc. maycause issues with data integrity.


85/131



86/131


Memcached!!! When you write to thedatabase you write to your memcachedserver, read from memcached beforereading from the database again.


87/131



Are you going to the database too much?Counts of Users, Activity etc.?


88/131



Are you going to the database too much?Counts of Users, Activity etc.?

Before memcached, we used filesystem fileswritten by system processes.


89/131

Pain Point #6: The PHP Code


90/131


XDebug. If you arent using it. Download it,learn it. Use it. Improve performance ofyour application. (http://www.xdebug.org)

i i C
http://www.xdebug.org/http://www.xdebug.org/


91/131



Profile your application.

P i P i 6 Th PHP C d


92/131



Profile your application. Take a lesson from a high visibility site:Wikipedia, run a fraction of your requests

through xdebug and profile.

P i P i #6 Th PHP C d


93/131


A profile of mysql.com in April 2008

P i P i t #6 M it i


94/131

Pain Point #6: Monitoring

P i P i t #6 M it i


95/131


If a person falls in the forest do you hearthe PHP Fatal Error?

P i P i t #6 M it i


96/131


If a person falls in the forest do you hearthe PHP Fatal Error? Be omniscient in your applications. If

something goes wrong do not wait forsomeone to tell you.

P i P i t #6 M it i


97/131


If a person falls in the forest do you hearthe PHP Fatal Error? Be omniscient in your applications. If

something goes wrong do not wait forsomeone to tell you.

Build monitoring into the application, butdo you want High Performance?

P i P i t #6 M it i


98/131


P i P i t #6 M it i


99/131


Capture your errors and logging into logfiles which are then monitored.

Pain Point #6 Monitoring


100/131


Capture your errors and logging into logfiles which are then monitored. Establish a good monitoring tool which

monitors not only the Servers but yourApplication.

Pain Point #6 Monitoring


101/131


Capture your errors and logging into logfiles which are then monitored. Establish a good monitoring tool which

monitors not only the Servers but yourApplication.

Shameless plug for both MySQL EnterpriseMonitoring and my own open sourceBigDaddy (bigdaddymonitor.org) whichgrew out of all these pain points

Pain Point #7: Your SQL Queries


102/131




103/131


In the end a PHP/MySQL application livesand dies on the strength of your queries.



104/131


In the end a PHP/MySQL application livesand dies on the strength of your queries. Make sure that you have good indexes on

your tables. EXPLAIN always.



105/131


In the end a PHP/MySQL application livesand dies on the strength of your queries. Make sure that you have good indexes on

your tables. EXPLAIN always. Make sure that you have query caching

turned on go examine your slow query log.



106/131




107/131


Queries do not always scale!



108/131


Queries do not always scale! Use some sort of query analyzer,

custom or third party.



109/131


Queries do not always scale! Use some sort of query analyzer,

custom or third party.

When you develop, try to testexpensive queries against a proper

data set size.

Pain Point #8: Ajax Javascript


110/131

Pain Point #8: Ajax, Javascript



111/131


App performance is what the client sees,not what the server/server-op sees



112/131


App performance is what the client sees,not what the server/server-op sees DB Setup tuned for Web 2.0 apps? Ajax

applications tend to be less read heavy andmore write heavy.



113/131


App performance is what the client sees,not what the server/server-op sees DB Setup tuned for Web 2.0 apps? Ajax

applications tend to be less read heavy andmore write heavy.

InnoDB versus MyISAM for primary keylookups.



114/131


Client tuning is essential as much as servertuning. YSlow is one option:

Pain Point #9: All the other things


115/131




116/131


Over the years pain points have come in allshapes and sizes, including our apachelogs :)



117/131



We ended up creating a sharded db systemwith a simple perl script to dump web logsinto a MySQL database.



118/131



We ended up creating a sharded db systemwith a simple perl script to dump web logsinto a MySQL database.

Oddly worked as well if not better than afile system.

Final thought...


119/131

Final thought...


120/131

Murphy's Extended Law: If a series ofevents can go wrong, they will do so in theworst possible sequence.

Final thought...


121/131


NFS Crash

Final thought...


122/131


NFS Crash

File system corrupt

Final thought...


123/131


NFS Crash

File system corrupt

DB Crash, Table corrupted

Final thought...


124/131


NFS Crash

File system corrupt


Backup corrupted by another sequenceof events.

Final thought...


125/131


NFS Crash

File system corrupt


Backup corrupted by another sequenceof events.

I was on holiday

The moral of this sordid tale


126/131

e o a o s so d d a e



127/131

Murphy Loves Web ApplicationDevelopers.



128/131

Murphy Loves Web ApplicationDevelopers. Everything goes wrong at some point



129/131


Just be prepared



130/131


Just be prepared Eliminate every SPoF (Single Point of

Failure) in your system.


131/131

high performance social networking applications php mysql

Documents