Download - Cloud, Cache, and Configs
Cloud, Cache, and ConfigsWordCamp NYC 2012
Saturday, June 9, 12
Scott TaylorLead PHP Developer, eMusic
@wonderboymusicwww.scotty-t.com
#projectmanagementsoftware
Saturday, June 9, 12
eMusic
MultisiteRegionalized ContentRegionalized Caching
Shared Content across SitesTons of Custom Post Types
Post FormatsTons of Web Services
Saturday, June 9, 12
eMusic Architecture
• 12-24 Amazon EC2 instances (CentOS)
• 4 MySQL (1 Write, 3 Read)
• 4 Memcached (~28GB RAM)
• Amazon S3 / Cloudfront CDN
• PHP 5.3 / MySQL 5.5
• PECL: APC, Memcached, HTTP
• Batcache
Saturday, June 9, 12
APC = duh
• Essential for PHP
• Opcode cache
• apc.shm_size = 64M or higher
Saturday, June 9, 12
Logs
• Never allow any PHP-related notices, errors, warnings, exceptions, etc
• Check access logs regularly for 404s, 500s etc
• make a constant for toggling debug logging
Saturday, June 9, 12
Production Data
• Always pull down, never push up
• We only push imported legacy content up
• output buffering
• code filtering
• more to come on this....
Saturday, June 9, 12
ob_start( $callback )
ob_start();
echo ‘Daryl’;
$daryl = ob_get_clean();
Output Buffering
Saturday, June 9, 12
ob_start( function ( $data ) {
return str_replace( $urls_array, EMUSIC_CURRENT_HOST, $data );
} );
Saturday, June 9, 12
Configs
• Mandatory machine configs
• HyperDB config
• Overridable sunrise.php
Saturday, June 9, 12
Machine Config• DB credentials local to that machine
• Amazon S3 bucket
• Web Service Endpoints
• Memcached Servers
if ( file_exists( ‘/wp-config/config.php’ ) ) { require_once( ‘/wp-config/config.php’ );} else {
die( ‘You must have a local config!’ );}
Saturday, June 9, 12
Memcached
• Memcached PHP Extension (not Memcache)
• Can be used locally (127.0.0.1)
• Memcached Redux supports wp_cache_get_/set_multi( )
• Johnny Cache
Saturday, June 9, 12
Batcache
• Full-page caching
• Can be configured
• You can partition cache by unique values
• Loads before plugins - any code you need has to be duped or added early (sunrise.php)
Saturday, June 9, 12
class batcache {
// This is the base configuration. You can edit these variables or move them into your wp-config.php file. var $max_age = 300;
// Expire batcache items aged this many seconds (zero to disable batcache)
var $remote = 0;
// Zero disables sending buffers to remote datacenters (req/sec is never sent)
var $times = 5;
// Only batcache a page after it is accessed this many times... (two or more)
var $seconds = 120;
// ...in this many seconds (zero to ignore this and use batcache immediately)
var $group = 'batcache';
// Name of memcached group. You can simulate a cache flush by changing this.
var $unique = array( BATCACHE_REGION, BATCACHE_COUNTRY );
// If you conditionally serve different content, put the variable values here.
var $headers = array( 'X-nananana' => 'Batcache' );
. . . . }
Saturday, June 9, 12
Sunrise• Used to alter multisite context
• sets $current_blog and $current_site
• filters all URL functions to resolve all URLs to your current domain
• registers custom locations for media
• filters Admin URLs
Saturday, June 9, 12
switch_to_blog( $blog_id )
• All dynamic functions need to account for this
• Shared content needs to resolve proper URLs
• Different sites have different media locations
Saturday, June 9, 12
$current_blog = new stdClass();$current_blog->site_id = 1;$current_blog->archived = 0;$current_blog->mature = 0;$current_blog->spam = 0;$current_blog->deleted = 0;$current_blog->lang_id = 0;$current_blog->public = 1;$current_blog->registered = '2011-02-20 03:38:22';$current_blog->last_updated = $_SERVER['REQUEST_TIME'];$current_blog->domain = EMUSIC_CURRENT_HOST;
function emusic_switch_to_blog( $blog_id, $prev_blog_id = 0 ) { if ( $blog_id === $prev_blog_id ) return;
global $current_blog, $emusic_paths; $current_blog->blog_id = $blog_id; $current_blog->path = $emusic_paths[$blog_id];}
emusic_switch_to_blog( $the_id );
add_action( 'switch_blog', 'emusic_switch_to_blog', 10, 2 );
$blog_id = $the_id;$site_id = 1;
$current_site = new stdClass();$current_site->blog_id = $the_id;$current_site->id = 1;$current_site->domain = EMUSIC_CURRENT_HOST;$current_site->site_name = 'eMusic';$current_site->path = $the_path;
Fix switch_to_blog()
Saturday, June 9, 12
add_filter( 'pre_option_upload_path', function () { $id = get_current_blog_id(); if ( 1 < $id ) return $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$id}/files";
return $_SERVER['DOCUMENT_ROOT'] . '/' . EMUSIC_UPLOADS;} );
add_filter( 'pre_option_upload_url_path', function () { $id = get_current_blog_id(); if ( 1 < $id ) return 'http://' . EMUSIC_CURRENT_HOST . "/blogs/{$id}/files";
return 'http://' . EMUSIC_CURRENT_HOST . '/' . EMUSIC_UPLOADS;} );
add_filter( 'pre_option_siteurl', function () { global $current_blog; $extra = rtrim( $current_blog->path, '/' ); return 'http://' . EMUSIC_CURRENT_HOST . $extra;} );
add_filter( 'pre_option_home', function () { global $current_blog; $extra = rtrim( $current_blog->path, '/' ); return 'http://' . EMUSIC_CURRENT_HOST . $extra;} );
Filter URLs
Saturday, June 9, 12
Plugins
• Filter active network plugins (don’t rely on database being correct)
• Filter each site’s plugins (if you have a manageable number)
• Use classes, not a bunch of functions
• Extend before you copy / paste
Saturday, June 9, 12
require_once( 'site-configs/global.php' );
if ( $the_id > 1 ) { define( 'UPLOADBLOGSDIR', 0 ); define( 'UPLOADS', 0 ); define( 'BLOGUPLOADDIR', $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$the_id}/files" );
switch ( $the_id ) { case 2: require_once( 'site-configs/bbpress.php' ); break;
case 3: require_once( 'site-configs/dots.php' ); break;
case 5: require_once( 'site-configs/support.php' ); break; }
add_filter( 'pre_option_template', function () { return 'dark'; } );} else { require_once( 'site-configs/emusic.php' );}
Site Configs
Saturday, June 9, 12
add_filter( 'pre_site_option_active_sitewide_plugins', function () { return array( 'batcache/batcache.php' => 1, 'akismet/akismet.php' => 1, 'avatar/avatar.php' => 1, 'bundle/bundle.php' => 1, 'cloud/cloud.php' => 1, 'download/download.php' => 1, 'emusic-notifications/emusic-notifications.php' => 1, 'emusic-ratings/emusic-ratings.php' => 1, 'emusic-xml-rpc/emusic-xml-rpc.php' => 1, 'johnny-cache/johnny-cache.php' => 1, 'like-buttons/like-buttons.php' => 1, 'members/members.php' => 1, //'minify/minify.php' => 1, 'apc-admin/apc-admin.php' => 1 //'debug-bar/debug-bar.php' => 1 );} );
Global Plugins
Saturday, June 9, 12
Site Plugins
add_filter( 'pre_option_active_plugins', function () { return array( 'artist-images/artist-images.php', 'catalog-comments/catalog-comments.php', 'emusic-post-types/emusic-post-types.php', 'discography.php', 'emusic-radio/emusic-radio.php', 'gravityforms/gravityforms.php', 'super-ghetto/super-ghetto.php' //,'theme-check/theme-check.php' );} );
Saturday, June 9, 12
class MyPlugin { function init() { add_action( ‘init’, array( $this, ‘register’ ) ); } function register() {}}
$my_plugin = new MyPlugin();$my_plugin->init();
class MyPlugin { function init() { add_action( ‘init’, array( ‘MyPlugin’, ‘register’ ) ); } function register() {}}
MyPlugin::init();
Saturday, June 9, 12
class FeaturePack extends eMusicPostTypes implements PostType { ..........}
Share code when possible
Saturday, June 9, 12
MySQL
• Use 5.5, better handlng of weird multibyte strings
• Use InnoDB, not MyISAM, pretty much in all cases
• HyperDB handles scaling for you
• Benchmark queries, don’t be afraid to roll your own SQL, tables, use $wpdb
Saturday, June 9, 12
HyperDB
• Can be empty locally
• Inherits wp-config DB defaults
• contains functions for replication lag detection (when used with mk-heartbeat)
Saturday, June 9, 12
Themes
• Theme setup is a class
• Extend before you repeat
• Classes are better than prefixing function names
Saturday, June 9, 12
class Theme_17Dots extends Regionalization { function __construct() { global $dots_regions_tax_map, $dots_regions_map; $this->regions_map = $dots_regions_map; $this->regions_tax_map = $dots_regions_tax_map; parent::__construct(); } function init() { add_action( 'init', array( $this, 'register' ) ); add_action( 'after_setup_theme', array( $this, 'setup' ) ); add_action( 'add_meta_boxes_post', array( $this, 'boxes' ) ); add_action( 'save_post', array( $this, 'save' ), 10, 2 ); add_filter( 'embed_oembed_html', '_feature_youtube_add_wmode' ); } . . . . . . . }
Theme Config in functions.php
Saturday, June 9, 12
class Regionalization { var $regions_map; function __construct() { add_filter( 'manage_posts_columns', array( $this, 'manage_columns' ) ); add_action( 'manage_posts_custom_column', array( $this, 'manage_custom_column' ), 10, 2 ); add_filter( 'posts_clauses', array( $this, 'clauses' ), 10, 2 ); add_filter( 'manage_edit-post_sortable_columns',array( $this, 'sortables' ) ); add_filter( 'pre_get_posts', array( $this, 'pre_posts' ) ); } . . . . . . }
Use base classes
Saturday, June 9, 12
pre_get_posts function regionalize( $query ) { global $regions_map; if ( $query->is_main_query() && !is_admin() && ( is_search() || is_archive() || is_home() ) ) {
$types = get_post_types( array( 'publicly_queryable' => true, '_builtin' => false ) ); $tax_region = array( 'taxonomy' => 'region', 'field' => 'term_id', 'terms' => array( $regions_map[ 'ALL' ], $regions_map[ THE_REGION ] ), 'operator' => 'IN' );
if ( is_home() ) { $query->set( 'posts_per_page', 10 ); } else if ( is_search() && get_option( 'editorial_search_enabled' ) ) { $ctx = get_query_var( 'search_context' ); $query->set( 's', stripslashes( urldecode( get_query_var( 's' ) ) ) );
if ( in_array( $ctx, array( 'features', 'features-books' ) ) ) { $query->set( 'posts_per_page', 48 ); } else { $query->set( 'posts_per_page', 12 ); }
if ( in_array( $ctx, array( 'books', 'features-books' ) ) ) { foreach ( $types as $type ) if ( false === strpos( $type, 'book' ) ) unset( $types[$type] ); }
if ( empty( $query->posts ) && $query->is_paged() ) $query->is_paged = false; } else if ( is_tag() ) { if ( empty( $query->posts ) && $query->is_paged() ) $query->is_paged = false; }
//error_log( 'REGIONALIZING' );
if ( !is_post_type_archive() ) $query->set( 'post_type', array_keys( $types ) );
$query->set( 'tax_query', array( $tax_region ) ); }
return $query; }
Saturday, June 9, 12
Assets
• Use remote storage
• Use a CDN
• Replace hosts using output buffer
• Cloud - pieces of W3 Total Cache
Saturday, June 9, 12
Minify
• JS / CSS concatenation speed up your front-end loading / perceived loading
• Minify is automagic
• HTML5 CSS properties are automatically inflated
• Admin tool to cache-bust URLs
• Auto-locking while files are generated
Saturday, June 9, 12
Web Services• cURL PHP extension
• curl and curl_multi()
• Memcached is essential
• hooks into parse_request to load data along with WordPress, allows us to cause WP to 404 and bail early when required data response fails
• Parallelization with curl_multi()
Saturday, June 9, 12
class eMusicRequest { var $request; var $sub_request; var $path; var $page; function load( $page = '' ) { if ( !empty( $page ) ) $this->page = $page; $file = $this->path . $this->page . '.php'; if ( file_exists( $file ) ) require_once( $file ); } function parse() { $requests = array( $this->request, $this->sub_request ); foreach ( $requests as $request ) { if ( !empty( $request ) ) { $keys = array_keys( get_class_vars( get_class( $request ) ) ); foreach ( $keys as $var ) { if ( !empty( $request->$var ) ) { $GLOBALS[$var] = $request->$var; } } } } }}
Saturday, June 9, 12
class DarkRequest extends eMusicRequest { var $genre; var $post_type; function init( $request ) { global $_GENRES; $vars =& $request->query_vars;
......... }
}
Saturday, June 9, 12
switch ( get_current_blog_id() ) {case 1: $_dark_request = new DarkRequest(); add_action( 'parse_request', array( $_dark_request, 'init' ) ); break;case 4: $_my_emusic_request = new MyEMusicRequest(); add_action( 'parse_request', array( $_my_emusic_request, 'init' ) ); break;}
Saturday, June 9, 12
<?phpclass HomeRequest extends RequestMap { var $recommendations; function __construct() { parent::__construct(); if ( !get_option( 'recs_enabled' ) ) return; if ( is_user_logged_in() && 'US' === THE_REGION ) { $user = wp_get_current_user(); $user_id = isset( $_GET['user_id'] ) ? $_GET['user_id'] : $user->ID; $params = array( 'userId' => $user_id, 'return' => true ); $this->add( get_user_recommendations( $params ), array( $this, 'parse_recommendations' ) ); $this->send(); } } function parse_recommendations( $data ) { if ( empty( $data['recommendations'] ) ) return; $data = $data['recommendations']; if ( !empty( $data ) && isset( $data[key($data)]['items'] ) && !empty( $data[key($data)]['items'] ) ) { $this->recommendations = array_slice( $data[key($data)]['items'], 0, 18 ); shuffle( $this->recommendations ); foreach ( $this->recommendations as &$rec ) { $rec = array( 'work_id' => $rec['catalogId'] ); } } }}
Saturday, June 9, 12
<?php class RequestMap extends API { private $requests; private $responses; private $ttl; private $useCache = true; protected $error = false;
public function __construct() { $this->flush(); }
public function is_error() { return $this->error; } public function flush() { $this->requests = array(); } public function getTtl() { if ( empty( $this->ttl ) ) { $this->ttl = CACHE::API_CACHE_TTL; } return $this->ttl; } public function add( $url, $callback, $vars = array() ) { $params = new stdClass(); $params->url = $url; $params->callback = $callback; $params->params = (array) $vars; $this->requests[] = $params; } private function exec( $item, $response ) { $params = array_merge( array( $response ), $item->params ); call_user_func_array( $item->callback, $params ); } public function send() { if ( !empty( $this->requests ) ) { $this->responses = self::batch( $this->getRequestUrls(), $this->getTtl(), $this->useCache ); if ( is_array( $this->responses ) ) { foreach ( $this->responses as $i => $response ) { if ( !empty( $this->requests[$i] ) ) { $this->exec( $this->requests[$i], self::parse_response( $response ) ); } } } } }}
Saturday, June 9, 12
class API {
public static function batch( $urls, $ttl = '', $usecache = true ) { $response = array(); ob_start(); if ( empty( $ttl ) ) { $ttl = CACHE::API_CACHE_TTL; } if ( is_array( $urls ) ) { if ( $usecache ) { foreach ( $urls as $index => $url ) { $in = Cache::get( Cache::API, $url );
if ( $in ) { $response[$index] = $in; unset( $urls[$index] ); } } }
$keys = array_keys( $urls ); } $calls = self::multi_request( $urls );
. . . . . . . .
Saturday, June 9, 12
if ( is_array( $calls ) && count( $calls ) > 0 ) { $calls = array_combine( $keys, array_values( $calls ) );
foreach ( $calls as $index => $c ) { if ( $c ) { $response[$index] = self::parse_response( $c ); if ( isset( $response[$index]['status']['code'] ) && $response[$index]['status']['code'] < 400 ) { if ( isset( $response[$index]['results'] ) && !empty( $response[$index]['results'] ) ) { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } else { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } } else if ( !isset( $response[$index]['status']['code'] ) ) { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } } else { $response[$index] = null; } } } else if ( !empty( $calls ) && isset( $urls[0] ) ) { $data = self::parse_response( $calls ); if ( isset( $data['status']['code'] ) && $data['status']['code'] < 400 ) { if ( isset( $data['results'] ) && !empty( $data['results'] ) ) { Cache::put( Cache::API, $urls[0], $data, $ttl ); } else { Cache::put( Cache::API, $urls[0], $data, $ttl ); } } else if ( !isset( $data['status']['code'] ) ) { Cache::put( Cache::API, $urls[0], $data, $ttl ); } $response[] = $data; } ob_end_clean(); return $response;
Saturday, June 9, 12
Because we used classes, everything is
abstracted
Saturday, June 9, 12
Custom Authentication
• WP stores slashed passwords
• wp_authentication arguments are slashed
• Inherit your base system’s rules
• DO NOT sanitize email / password (WordPress does by default)
• User tables in this case are really a transient cache
Saturday, June 9, 12
Questions / complaints?
Saturday, June 9, 12