the great consolidation - entertainment weekly migration case study - sandcamp 2015

SANDcamp, San Diego, CA - February 27, 2015

The Great ConsolidationEntertainment Weekly migration case study

Who are we?

• Four Kitchens Web Chefs

• Jon Peck - drupal.org/u/fluxsauce

• Matt Grill - drupal.org/u/drpal

• Patrick Coffey - drupal.org/u/pcoffey

Project overviewWhat did we do & who did it?

Entertainment Weekly

• Magazine that covers film, TV, music, Broadway theater, books, and pop culture

• 40th largest in the US

• ew.com

• 13.7 million consumers / week

Migration scope

• WordPress

• 10 blogs

• 51,616 terms

• 101,581 posts

• 90,435 images

• Vignette V6

• 117,326 articles

• 201,986 images

• 97,611 terms

• 4,924 galleries

Development Teams

• Four Kitchens - 3 developers to 6

• Time Inc. - 1 developer to 4

• Zoom, HipChat for regular communication

• Video really helps

• As does GitHub selfies :-)

Timeline

• First commit: April 29th, 2014

• Launch: January 29th, 2015

• 17 sprints

Delegation of responsibilities

• Four Kitchens

• Project Management

• Data structures

• Migration

• Implement design, advertising

• Time Inc.

• Product Owner

• Design

• Workflow

• Infrastructure, build process

ProcessStandards and documentation

Definition of Readiness and Completion• Stakeholders

• Context necessary for development

• Developers

• Code reviews, PO acceptance, demo, docs

• Evolving framework

Code and style standards

• Algorithmically enforced

• PHP_CodeSniffer (rules from coder)

• jshint - syntax

• jscs - code style

• EditorConfig settings

Peer code reviews

• Functional - does it…

• Fulfill the intent of the story?

• Use best practices?

• Avoid technical debt?

Development Environments

• Documented project-specific process

• Superset of existing instructions

• Greatly reduced on-boarding time

• Provided plentiful feedback to DCMS

• Beta tested new VMs

Branching standard

• Reduces clutter

• Improves communication, navigation

• Best practices

• sprint-xx (kept for one sprint)

• SEGEWCMSM-YY (deleted after merge)

• Tags added when deleting sprint branch

Front-endTheming, performance and advertising

THEMING

https://www.drupal.org/project/aurora

Aurora, mobile first theming

• Used Aurora as base theme

• HTML5, Sass & Compass

• Mobile theme is sub-theme of Aurora

• Desktop is a sub-theme of Mobile

Gulp task management

• Compiles Sass

• Runs JavaScript checkers

• jshint

• jscs

JAVASCRIPT

JavaScript global scoping

• Created global JS object for all EW functions

• Best practice

• Centralized point of entry

• Easy access

Atomic JavaScript functionality• Break related functions into individual files

• Only load what’s needed

• No monolithic structure

• Improves front-end performance

PREPROCESSING

Balancing performance with business logic• Node template data built in pre-processing

• Does not use views, panels

• Clean & reusable

• Intuitive structure

• Helper and abstractions

BEACONS

Global metadata object

• Useful for integrations requiring metadata

• Avoids repetition

• Facilitates data collection by crawlers

Beacon performance

• Delay loading of beacons (non-blocking)

• Use defer for injecting JS tags

ADVERTISING

Collaborating with DCMS

• TGX improvements

• Single-request mode

• More native GPT functionality

• Ad Manager

• Patches, recommendations

Custom ad rendering to maximise performance• No logic in templates

• Data attributes contain values

• Avoids inline JS

• Improves rendering performance

• Ad renders in footer (after page load)

Editorial UI & Workflow

Packages and Channels

• Automatic dynamic content

• Editors define rules for selection

• Optional manual curation

Custom content types

• Used existing DCMS features to begin

• Rounds of PO, editor feedback

• Usability key to adoption

Collaboration with DCMS

• Provided feedback on tools

• Search interfaces

• Transient messages

• Crop tool

• Auto save

Dynamic Entity References

• Lots of aggregate views in EW

• Editorial control of views, but with overrides

• Custom module that “fills in” empty parts of a view based on context

• View lists 15, editor defines 5, DER populates remaining

Dynamic ER challenges

• Modular, extensible, reusable

• Accurate dynamically curated nodes

• High performance

• Editoral expectations for caching

• State Machine integration

Content migrationsWordPress, Vignette to Drupal

WORDPRESS

Challenges

• 10 different blogs

• Different structures

• Multiple taxonomies

• Invalid markup

• Custom shortcodes, filters

• WordPress Migrate module not enough

Solutions

• Interviewed other teams, shared code

• Pre-processing

• Strip comments

• Combine into one monolithic file

• Separates authors, tags, images

Shortcodes and Filters

• Custom markup

• Created custom short code system

• Most will not be migrated

• Pre-processing solution

• Render as HTML

• Used WordPress, existing code

WordPress Migrate

• Can’t use directly…

• Can extend!

• Helps normalize WXR structure

• Focus on custom logic

Mapping terms on import

• Separates monolithic taxonomy

• Specific vocabularies

• New content types (people, creative works)

• CSV Spreadsheet used for defining rules

• Curated by Editors, PO

VIGNETTE

Exporting from Vignette

• Coordinated with TI team

• Defined XML structure

• Gave feedback on export UI

• Resolve edge cases

Importing

• XML content is sanitized and validated

• Transforms overloaded taxonomies to content

• People, Creative Works

• Articles, Packages, Galleries

MIGRATION PERFORMANCE

Improving migration performance• Validate all XML before ingestion

• Eliminate all PHP errors, warnings

• Avoid redundant migrations

• Separate in pre-import

• Reduce overall number of migrations

• Implement cache_counts

Reducing migration overhead

• Disabled solr, pathauto, metatag, others during migration

• Project requirement was 2.5; you should use 2.7

• Ran migrations in parallel

• Ensure HW resources are sufficient

Migration redirects

• Minimize Drupal performing redirects

• Redirect farm (rules, 1:1 redirects)

• Exported if source, target known

• DNS changes

• Redirects from migrated servers

Performance and caching

Frontend performance

• CSS

• Sass and Compass optimized

• Targeted - only load what’s needed

• JavaScript

• Linted, strict standards

• In the footer (non-blocking)

• Global objects (less is more)

Frontend caching strategy

• Editorial 1 minute publish to live

• Ensure cacheable headers are set

• Shorter TTL on Akamai than Varnish

• Ex: 5 minutes on Akamai, 10 on Varnish

• Send purges on publishing

Backend performance

• Eliminate PHP errors

• Cache, EXPLAIN custom queries

• Minimize unnecessary overhead

• Module count

• Benchmarking

Performance testing

• Load test production with migrated data

• blitz.io

• New Relic for introspection

• Exercised different content types, exceeded TTLs

• Front-end testing

• WebPageTest.org

Questions?

Thank you!