How SOA Makes Moving to the Cloud Easier

Paul RobbinsSr. Product Manager,

NYTimes.com@robbinspaul

Agenda

• Beginnings of NYTimes.com moving to the cloud• How We Started with SOA• Power of SOA• Complications of SOA and the Cloud• Example Applications• What's Next

Metrics 

• 33 million uniques• 611 million pageviews

NYTimes.com April 2011 (comScore)

Beginnings of NYTimes.com on the Cloud• Community

Infrastructure o Fairly new set of

standardized user interactions

o Unknown scale and usage

• TimesSkimmero Javascript web appo Prototypical in nature,

difficult to position for new servers 

How We Started with SOA• Services for external developers (TimesOpen)• Community services • Travel Guides application

o Previous version was reading SQLite on the flyo If a change was made, took 1 hour to replicate to

SQLiteo New version didn't live in the cloud, but built with

SOA

Power of SOA

• Service Oriented Architecture• SOA is Freedom. SOA is Flexibility.• Think about what applications should do, not how

they get data• Multiple data formats

SOA Key Concepts

• Ability to access data your wayo Sorting, filtering, and organizing is abstractedo Means services can be reused, no need to rewrite

data accessors for each new application• Ability for internal users to update data with no

direct access to the databaseo This can achieved via a REST interface to the

service

RSS ≠ SOA

But it's a good start

Complications of SOA

• Data ownership and maintenanceo Data lives in a DB, but is updated by a service

• Backwards compatibility versus new features and fieldso Versioning solves some issues, but causes new

oneso Best to keep client applications robust to

accommodate changes• Security for updates

o Adding API keys helps, but isn't fail-proof• Caching

o Clearing data cache and front-end cache on updates

Complications of the Cloud

• Many complications of mounted file system o If we replicated filesystem in the cloud, latency

and bandwidth issues• Some shared modules were hard-coded local

o Initial solution was to pull these modules over http

o Long term solution is to make these modules be accessed via service

• Ad calls were locally initiatedo Had to build a service layer

Example Applications

Skimmer and Chrome App

• Skimmer [ http://nytimes.com/skimmer ]o Newest version uses SOA and lives in the cloudo Utilizing JSON formatting and richer data than

previous RSS version• App for Chrome Web Store [

http://nytimes.com/chrome ]o Similar to Skimmer, optimized for the Chrome

browser

Best Sellers

• One of the best known brands for the Timeso [ http://nytimes.com/best-sellers ]

• Online version needed refresh, was adding E-Bookso Service was created in 2009 for TimesOpen evento Front-end was simply a static article, no ability to

navigate between weeks• Redesign meant more than aesthetic, we redesigned

architecture and interactions between platforms

Restaurants Search

• Restaurant listings around NYCo [ http://nytimes.com/restaurants/search ]

• Not only pulling data from front-end, but heavily curated set of data by producerso Built an interface into content management

system to update via services• Refine search filters handled purely by passing

query parameters to the serviceo Lightens the load on the front-end application

Demos!

What's Next

• Push-based services• Migrate more applications to SOA• Expand flexibility of services, still somewhat

restrictive• Complications with full text search via services,

need to refine that process

Items We've Open Sourced• NYT_Transformer

o Framework for converting structured data to and from different formats, from flat files to DB

o [ http://code.nytimes.com/projects/nyt_transformer/ ]• DBSlayer

o Database abstraction layero [ http://code.nytimes.com/projects/dbslayer ]

• Nimbul o Interface for managing Amazon EC2 serverso [ https://github.com/nimbul/nimbul ]

• Cloudsource o Code deployment tool for SVN cloud deployso [ https://github.com/nimbul/cloudsource ]

Questions??

• Also, be sure to check out our developer Open blogo [ http://open.blogs.nytimes.com ]

• And our developer networko [ http://developer.nytimes.com ]

• Find me on twittero [ http://www.twitter.com/robbinspaul ]