drupal performance and scalability

Drupal Performance and ScalabilityObtaining Optimal Performance From Drupal And The LAMP Stack.

This book is written by Jeremy Andrews and licensed under a CreativeCommons Attribution-Noncommercial-No Derivative Works 3.0 UnportedLicense.

Section 1: Drupal PerformanceThe first section of this book offers details on how to get good performanceout of your Drupal powered website, and how to scale it as demand grows.The majority of the features discussed in this section are available withoutmaking any modifications to Drupal.

Chapter 1: Setting GoalsThis first chapter discusses the importance of understanding your ownperformance and scalability goals. It helps the reader to identify specificallywhat they hope to accomplish, showing them how to set concrete andattainable goals, breaking larger requirements into smaller pieces. Thischapter will teach the importance of maintaining historical performance logs,discussing several technologies and services that are available to aide in thiseffort. It will stress the importance of making regular backups, and of testingbackups before changes are made. Finally, it will explain the importance offirst testing changes on development servers, and explain best practices fordeploying tested changes onto production servers.

Setting GoalsPerformance and Scalability Checklist1.Understanding and Defining the Problem2.Goals versus Requirements3.

1.

Measuring ProgressSetting A Baseline1.Measuring Progress2.Online Services3.

2.

BackupsWhat To Backup1.

3.

Drupal Performance and Scalability http://books.tag1consulting.com/book/export/html/1

1 of 19 7/18/08 10:30 AM

Backup Schedules2.Validating Backups3.

Staging ChangesTesting Changes1.Source Control2.Database Schema Changes3.Pushing Changes To Production4.

4.

Chapter 2: Drupal InfrastructureThis chapter will provide an overview of what is coming up later in the book. Itwill talk about cheap $5/month web hosts, versus slightly more powerfulVirtual Private Servers, versus dedicated servers and server farms. It willcollect together network diagrams for the various configurations, and point tolater chapters where the various features are more fully explained.

Bargain Basement HostingAdvantages1.Squeezing Water From A Rock2.Development and Testing3.Outgrowing Your Host4.Diagram5.

1.

Virtual Private ServersAdvantages1.What Is Virtualization?2.Competing For Resources3.Outgrowing Your Host4.Diagram5.

2.

Multiple Installations versus Multi-site InstallationsAdvantages1.Security Considerations2.Diagrams3.

3.

Dedicated HostingSingle Server1.Multiple Servers2.Sharing Files And File Systems3.Load Balancers4.High Availability5.Scaling Up vs. Scaling Out6.Caching7.

4.


2 of 19 7/18/08 10:30 AM

Network Diagrams8.

Chapter 3: PerformanceConfigurationThis chapter introduces Drupal's built-in performance features. It explains howDrupal's built-in page cache works, and details how it can be configured. Thechapter also discusses Drupal's built-in CSS and JS aggregation andcompression. The importance of regularly purging Drupal's logs will bediscussed. And finally, the chapter will explore Drupal's throttle module.

Drupal's Page CacheNormal Caching1.Aggressive Caching2.Minimum Cache lifetime3.

1.

CSS and JavaScript AggregationAggregation1.Compression?2.

2.

Purging LogsWatchdog Logs1.The Access Log2.

3.

The Throttle ModuleBackground1.Configuration2.Modules3.Blocks4.Integrating Custom Themes and Modules5.Why There Won't Be A Throttle In Drupal 76.

4.

Chapter 4: Too Many ModulesThis chapter takes an in depth look at Drupal's modular design. It explores theconcept behind Drupal's “hooks”, using the nodeapi as an example. It alsolooks at Drupal's menu system. The chapter then puts all of this together bytracing what happens when you enable a single Drupal module. Finally, itdiscusses the temptation to enable hundreds of contributed modules.

Modules and HooksDrupal modules1.

1.


3 of 19 7/18/08 10:30 AM

Adding Features With Hooks2.Example: the nodeapi3.

MenusDefining Pages1.

2.

Enabling ModulesMemory Limits1..install Files2.Drupal 7 Registry Preview3.All You Can Eat?4.

3.

Chapter 5: Caching LayerThis chapter dives into Drupal's code, taking a look at the underlying cachinglayer. It will begin with an accessible, high-level description before it dives intothe actual implementation. Finally, it will teach module developers how tobetter use Drupal's built-in caching layers.

Understanding Drupal's Caching LayerOverview1.Variables2.The Many Cache Tables3.

1.

Developing With Drupal's Caching LayerDrupal's Cache API1.Caching With Custom Modules2.Sessions3.

2.

Chapter 6: The devel ModuleThis chapter will take a look at the contributed devel module, explaining itskey importance in performance tuning a Drupal website. It will discuss themany configuration options, and explain how the module can be used to profilepage loads.

More Then A Development ToolVisualizing Slow Queries1.Timing Page Creation2.Page Elements Versus The Database3.

1.

Configuration2.Profiling Database Queries3.


4 of 19 7/18/08 10:30 AM

Identifying Slow Queries1.Identifying Duplicate Queries2.Common Queries and What They Mean3.

Chapter 7: To Patch Or Not ToPatchDrupal offers considerable performance and scalability without modifying thecode in any way. However, much more performance can be obtained bypatching the core code. This chapter weighs the pros and cons of patchingDrupal, and the impact this has on keeping up to date with security patchesand upgrading to new releases.

The Case For PatchingOptimal Performance1.Community Patchsets2.Backports3.Hitting Modularity Limitations4.

1.

The Case For Not PatchingAvoiding The Unknown and Under Tested1.Keeping Up With Security Updates2.Upgrading To New Releases3.

2.

Section 2: Front End PerformanceThis second section of the book begins to look at the underlying LAMP stack,discussing how it can be optimized specifically to get the most out of Drupal.Most of the information will be presented so it is accessible to people without abackground in system administration, though advanced topics will also bediscussed.

Chapter 8: Optimizing PHPThis chapter will look at tuning PHP with php.ini. It will explain how to readphpinfo(), and discuss PHP's memory footprint. It will explain how PHP iscompiled for each page, unless you enable an opcode cache. It will then reviewsome of the most popular opcode caches, how they work with Drupal, and


5 of 19 7/18/08 10:30 AM

known issues and fixes.

Configuring PHPWhat is php.ini1.Finding php.ini2.phpinfo()3.

1.

Tuning PHPModifying php.ini1.PHP's memory footprint2.Disabling Unnecessary Features3.

2.

Writing Good CodeCommon Pitfalls1.Investment vs. Return2.

3.

Opcode CachesScripting Languages1.APC2.Xcache3.eAccelerator4.The White Screen Of Death5.

4.

Chapter 9: Optimizing ApacheThis chapter will review how Apache can be optimized to achieve better Drupalperformance. It will discuss performance oriented Apache configurationoptions. It will look at Apache modules, and will explore the importance ofminimizing Apache's memory footprint. Finally, it will look at the various webserver architectures, exploring the use of load balancers to scale out this layer.

Configuring Apachehttpd.conf1.vhosts2.Compression3.

1.

Apache ModulesPerformance Features1.Memory Considerations2.Load Testing3.

2.

Infrastructure ChoicesBasement Startups: All On One Server1.Stand Alone Web Servers2.Multiple Servers With Load Balancers3.

3.


6 of 19 7/18/08 10:30 AM

Multiple Datacenters4.

Chapter 10: Alternatives To ApacheWhile Apache is the most popular open source web server, it's not the onlyopen source web server. This chapter will review the advantages anddisadvantages of serving pages with the most popular alternative, lighttpd. Itwill detail how to get Drupal up and running with lighttpd, and exploreconfiguration options for improving performance. It will also look at usinglighttpd to compliment Apache in an infrastructure, instead of replacing it.Later, the chapter will take a brief look at running Drupal on a newer andlesser known alternative, Nginx. Finally, it will also briefly explore WAMP basedDrupal installs, tuning IIS on Windows.

LighttpdFeature Comparison1.Benchmarks2.Limitations3.Configuration4.

1.

Other AlternativesNginx1.IIS (WAMP versus LAMP)2.

2.

Chapter 11: Optimizing YourThemeDrupal themes are what give websites their own unique look. This chapterexplores the impact of creating overly complex designs with many images,CSS files, and external JavaScripts. It will take a fresh look at CSS andJavaScript aggregation, previously discussed in Chapter 2. It will also reviewbest practices for using images, and how the size of images affects page loadtimes. Finally, it will look at how to get a complex looking design withoutnegatively slowing down the time it takes for each page to load.

ImagesMultiple HTTP Requests1.File size2.

1.

CSSInline Styles1.

2.


7 of 19 7/18/08 10:30 AM

External CSS files2.Caching3.Aggregation4.Compression5.

JavaScriptInline1.External2.Caching3.Aggregation4.Compression5.

3.

OptimizationsMultiple sub-domains1.Browser Cookies2.Far-Future Expiration3.JQuery4.

4.

Chapter 12: Content DeliveryNetworksThis chapter will provide background on Content Delivery Networks, or CDNs,explaining how they speed up page load times by bringing the contents of aweb page physically closer to the visitor. It will examine contributed modulesfor quickly integrating Drupal websites with CDNs. It will also offer someinsight into the pros and cons of some of the more powerful CDN servicescurrently being offered.

BackgroundConcepts1.Building a mini-CDN2.

1.

IntegrationModules1.Themes2.

2.

CDN LineupPanther Express1.Akamai2.EdgeCast3.Limelight4.

3.


8 of 19 7/18/08 10:30 AM

Chapter 13: Front-endPerformance ToolsThere are several useful tools freely available for the open source FireFox webbrowser. This chapter will explore how to use FireBug to take apart andunderstand the elements that combine to form a web page. It will also explorethe Yslow extension, detailing how to use its extremely useful performancereports. (I will research to see if similar tools are available for other browsers,and if so will also cover them in this chapter.)

FireFoxFireBug1.YSlow2.

1.

Section 3: Improved Caching andSearchingThis section will focus in on two key areas where Drupal can benefit from thirdparty integration, caching and searching. Many of these advanced topics willrequire patching Drupal's core.

Chapter 14: Reverse ProxiesThis chapter explores the usage of reverse proxies, adding additional layers ofcaching to your web infrastructure. It explains how this improves bothperformance and scalability. It then looks at several specific reverse proxyoptions, and their configurations, including Squid, Varnish, and Apache'smod_proxy.

Reverse Proxy Architecture1.The Benefits Of Reverse Proxies

Performance1.Scalability2.Layered Caching3.

2.

Selecting and Configuring a Reverse ProxySquid1.Varnish2.

3.


9 of 19 7/18/08 10:30 AM

Apache and mod_proxy3.

Chapter 15: Integrating ThirdParty CachesThis chapter introduces the concept of integrating Drupal with a third-partycache. It will examine the use, advantages and limitation of file-based caches.It will also review the use, advantages and limitations of using PHP opcodecaches for caching other content. The chapter will detail the many projectshelping with this integration, explaining their configuration and use. It will alsoreview the many patches available for improving Drupal's caching, includingthe advcache project, block caching, and improved taxonomy caching. Finally,the chapter will provide an initial introduction to memcached.

File cachesBoost module1.Fastpath_fscache2.Cache Coherency3.

1.

Patching DrupalAdvcache1.Caching Blocks2.Caching Taxonomy3.

2.

Memory cachesOpcode Caches1.Distributed Memory Cache2.

3.

Chapter 16: Caching WithMemcachedThis chapter will offer an in depth look at what memcache is, and how itimproves both website performance and scalability. It will look at howmemcache achieves its performance, reviewing the difference between hashtables and databases, and explaining how memcache can help websites of allsizes. This chapter will explore Drupal's contributed memcache integrationmodule, and the patches that come with the project. It will look at how tomodify Drupal so anonymous pages can be served directly out of RAM, and sopages for logged-in users can be assembled from objects stored in RAM.


10 of 19 7/18/08 10:30 AM

Finally, it will look at the areas in Drupal that most benefit from memcachedintegration.

Memcache BackgroundLiveJournal.com1.Hash Table Lookups2.

1.

Infrastructure DesignSpare RAM1.Distributed Caching2.Failing Servers3.Memcache Clusters4.

2.

Memache ModuleOverview1.Base Features2.Administration3.Advanced Features4.

3.

Beyond CoreFinding Good Candidates For Caching1.Memcache Integration2.Common Mistakes3.AdvCache project4.

4.

Chapter 17: Drupal's SearchModuleThis chapter will explore how Drupal's search module works, explaininglimitations introduced by the fact that SQL was not designed as a searchinglanguage. It will discuss how to get the best performance out of Drupal'ssearch module, and how to know when it's time to look consider otheralternatives. This chapter will mostly look at search in Drupal 6, but will take abrief look at why an improved search API is likely to be one of the killerfeatures in Drupal 7.

Search Module DesignBackground1.Searching With SQL2.

1.

Performance BottlenecksInnoDB Performance1.When To Replace2.

2.


11 of 19 7/18/08 10:30 AM

Searching Drupal's FutureSearch API in Drupal 71.Introducing Third Party Search Integration2.

3.

Chapter 18: Searching WithXapian, Sphinx & SolrXapian and Sphinx are two unrelated standalone search technologies written inC++. This chapter will explain how Xapian supports real time indexing andexposes extremely flexible APIs, while Sphinx offers lightening fast searchperformance. It will also explore using the Java based Solr search engine,discussing the steeper requirements and its flexible, advanced feature set.This chapter will detail how each solution can be integrated into a Drupalwebsite, replacing or enhancing Drupal's core search functionality.

XapianBackground1.Strengths2.Weaknesses3.Benchmarks4.Integration5.

1.

SphinxBackground1.Strengths2.Weaknesses3.Benchmarks4.Integration5.

2.

SolrBackground1.Strengths2.Weaknesses3.Benchmarks4.Integration5.

3.

Section 4: Optimizing the DatabaseLayer


12 of 19 7/18/08 10:30 AM

The fourth section of the book will examine database administration for aDrupal powered website.

Chapter 19: Drupal's DatabaseAbstraction LayerThis chapter will note that Drupal is described as being “database agnostic”, asthe code strives to not depend on the underlying database that is being used.It will review the database abstraction layer, and will talk about the currentlysupported databases. It will detail how in spite of this noble aim, MySQL is stillstrongly favored. It will compare MySQL support with PostgreSQL support.Finally, it will offer a preview of the database layer rewrite that is happeningfor Drupal 7, detailing how this may finally make Drupal database agnostic.

Abstraction Layer DesignAbstraction Concepts1.MySQL Support2.PostgreSQL Support3.

1.

Database Abstraction in Drupal 72.

Chapter 20: Choosing a StorageEngineThis chapter will primarily compare MyISAM and InnoDB. It will look atDrupal's history of being designed for MyISAM, and talk about some of theDrupal-specific pitfalls with using InnoDB. It will then explain the manyadvantages to using InnoDB, presenting this as currently being the onlyserious option for large high traffic websites using MySQL. This chapter willalso briefly look at some of the up and coming MySQL storage enginescurrently being developed.

Storage EnginesConcepts1.Mix and Match2.

1.

MyISAMStrengths1.Weaknesses2.

2.

InnoDB3.


13 of 19 7/18/08 10:30 AM

Strengths1.Weaknesses2.

PreviewsFalcon1.Maria2.

4.

Chapter 21: Monitoring MySQLThis chapter will first explain the importance of monitoring your databaseserver. It will then present several useful tools for monitoring MySQL, includingmytop and innotop. It will also discuss MySQL's built in reports, includingSHOW FULL PROCESSLIST, SHOW GLOBAL STATUS, and SHOW INNODBSTATUS. This chapter will also discuss MySQL's various logs.

OverviewWhy1.How Often2.

1.

Monitoring ToolsMySQL's Built In Reports1.mysqlreport2.mytop3.innotop4.Cacti5.MySQL Enterprise Montitors6.

2.

LogsError Logs1.Slow Query Logs2.No Index Logs3.

3.

Chapter 22: Tuning MySQLThis chapter will build upon what was learned in the previous chapter, detailinghow to use that knowledge to isolate and fix performance bottlenecks. It willtake a lengthy look at the mysqlreport perl script, explaining how itsummarizes many of the reports discussed in the previous chapter, and how touse this tool to tune your server for optimal performance. It will highlight theMySQL configuration options that most affect Drupal performance.

Isolating Trouble Spots1.


14 of 19 7/18/08 10:30 AM

Tuning With mysqlreportExamples1.

2.

Deploying ChangesThe Tortoise and the Hare1.Historical Monitoring2.Controlled Experimentation3.

3.

Chapter 23: Slow Query Log,Indexes, and Query PerformanceThis chapter will take a closer look at MySQL queries. It will examine themysqlsla perl script, detailing how it is used to quickly track down thedatabase queries that are wasting the greatest amount of resources. It willthen explain how to determine why a query is performing poorly. It discusshow some queries can be optimized by adding indexes, while also looking atthe impact of adding too many indexes to your tables. It will offer an in depthlook at how MySQL indexes work, comparing indexes in MyISAM versusInnoDB. It will also review when to use multiple simple queries instead ofcomplex queries. Throughout these chapter, specific Drupal examples will beprovided.

Revisiting the Slow Query Logconfiguration1.mysqlsla2.micro-second patches3.

1.

Query PerformanceReviewing the devel Module1.Understanding Indexes2.Joining Tables3.

2.

Chapter 24: MySQL ReplicationThis chapter will define MySQL replication, explaining how it works and how itcan be used to improve a Drupal website's performance and scalability. It willexplore patches that have been deployed on Drupal.org to send somedatabase queries to a slave server, and the rest to the master server. It willexamine the idea of using Master-Master replication, arguing against this as ameans for scaling Drupal websites. It will also briefly look at the concept of


15 of 19 7/18/08 10:30 AM

sharding, and look at plans in Drupal 7 for potentially supporting theseadvanced scalability features, reviewing the limitations imposed by Drupal'sdesign in Drupal 5 and Drupal 6.

ConceptsConfiguration1.Monitoring2.Backups3.Errors4.

1.

Drupal and ReplicationMixing Storage Engines1.Redirecting Search Queries2.High Availability3.

2.

Federated DatabasesSharding1.MySQL Proxy2.Drupal 7 Preview3.

3.

Section 5: Drupal In The CloudThis final section will be considered a “bonus” in the first edition of this book.This is because cloud computing is very new and unproven. There is asignificant amount of interest in the potential for scalability with cloudcomputing, so it is important to explore this topic in these final chapters, whileacknowledging that this is a quickly changing landscape.

Chapter 25: Cloud ComputingThis chapter will offer a high level overview of what cloud computing is, andhow it potentially solves the scalability problem. It looks at the advantages tooutsourcing your underlying infrastructure, as well as the limitations thisimposes

OverviewConcepts1.Pay For What You Use2.Outsourcing3.Scalability4.Performance5.

1.


16 of 19 7/18/08 10:30 AM

Latency6.Impermanence7.

Chapter 26: Running Drupal onAmazon's EC2This chapter will provide details on how to get Drupal up and running withAmazon's EC2 cloud computing service. This chapter will include screenshots,as it will be a high level guide to getting things up and running. It will thenexamine performance concerns introduced by the high latency often found in acloud environment. It will provide specific suggestions for improving Drupal'sperformance while running in the cloud. It will also look at cloudimpermanence, and how to provide reliability through redundancy, replication,and backups.

Getting StartedRequirements1.ElasticFox2.AMIs3.32-bit versus 64-bit4.Helpful Links5.

1.

Drupal in the CloudInstallation1.Configuration2.Benchmarks3.

2.

Cloud ImpermanenceRe-installation Scripts and Images1.Rsync2.Replication Across Zones3.Load Balancing4.Preserving IP Addresses5.Automated Backups and S36.

3.

PerformanceDealing With Latency1.Striping Drives2.Layered Websites3.Revisiting Memcache4.

4.


17 of 19 7/18/08 10:30 AM

Chapter 27: Scaling In The CloudsThis chapter will take a high level look at the many benefits of scaling Drupalwebsites in the clouds. It will explore many of the advanced features Amazonis planning for EC2, and how this will continue to make cloud computing amore attractive option.

Endless Scalability?1.The Future2.

About The AuthorJeremy Andrews has been a core Drupal contributer since early 2002, when hewas originally introduced to the project by its creator. His hobby web page,KernelTrap.org, was the first online community to push Drupal to scale beyondits modest beginnings and to achieve popular recognition as a competitiveCMS solution [1]. He has worked to improve Drupal's caching layer, optimizedDrupal's bootstrap process, improved core Drupal queries, focused onimproving Drupal's overall performance, and written core modules that are stillincluded with every single copy of Drupal. He has given seminars on Drupalperformance and scalability, both in person [2] and over the Internet [3].

Jeremy formed Tag1 Consulting, Inc in 2007, a successful consulting companythat focuses on Drupal performance and scalability [4], recognized by Drupal'screator as being among the very best at what they do [5].

[1] http://luckofseven.com/vlog/episode13[2] http://www.lullabot.com/seminar/drupal_performance_and_scalability/sunny...[3] http://www.mysql.com/news-and-events/web-seminars/display-94.html[4] http://tag1consulting.com/[5] http://tag1consulting.com/blog/jeremy/Drupal_Creator_Praises_Tag1_Consul...

Back CoverDrupal is a very flexible, modular framework often used as a contentmanagement system. This book is aimed at people who have already learnedthe basics of Drupal administration and theming. Through hands on


18 of 19 7/18/08 10:30 AM

instruction, this book will take you to the next level of understanding, teachingyou how to achieve optimal Drupal performance and scalability.

Master Drupal's built in performance features, and learn to scale Drupalthrough integration with third party searching and caching tools. Gain agreater understanding of the underlying LAMP stack on which Drupal runs withuseful recipes and tips for monitoring and tuning Linux, Apache, MySQL andPHP. Learn performance secrets from other popular websites that continue topush Drupal to new levels, gaining insights from problems they've experiencedand how they solved them.

Whether your Drupal-powered website has outgrown its current infrastructure,you want to be prepared for future growth, or you want to understand whatDrupal is capable of before you commit to using it for your website, this bookwill be your guide.


19 of 19 7/18/08 10:30 AM

drupal performance and scalability

Documents

underlying lamp stack

drupal powered website

scaling drupal websites

advanced topics

database queries

cloud computing

memory footprint

load balancers