Download - Evolving IGN’s New APIs with Scala
Evolving IGN’s new APIs with Scala
Manish PanditOct 4th, 2012
About me
Manish PanditDirector of Engineering, IGN
@lobster1234linkedin.com/in/mpandit
About IGNWe are a leading online media and services company obsessed with gaming and entertainment.
56MM Monthly Unique Views737MM Monthly Page Views20MM Monthly Video Views
My Objective
Is:To get you excited about building APIs in Scala
Isn't:To incite a functional vs. imperative, or Scala vs. ____ debate
Agenda
• APIs at IGN• API Evolution• Why Scala• API Development with Scala• Monitoring, Maintaining, Evolving• Performance Management
APIs at IGN
• Offer SOA advantages• Isolate the back end systems from the FE• IGN is going where the audience is, fast– Mobile Devices, Consoles, TVs, Aggregators
• APIs are a specialized skillset• Expand the development to outside
developers
APIs at IGN : Numbers
• ~5 Billion requests a month• Average Response time of under 20ms on
cache misses• 17 APIs between Social, OAuth, and Content• Hardest hit APIs doing about 25K RPM at peak
Evolution
The API Evolution : V1 [2006]
• The CMS and the desktop site calling Java Services, which wrapped Oracle DB calls behind Hibernate.
• Caching with Memcache– Near-far Cache Architecture to address hot spots
• Not all content types were supported• Schema was still too complex
The API Evolution : V2 [2009]• Rails based Content API, Java based Social API– Standards around request/response– Automated Testing– MongoDB NoSQL storage to simplify the schema– Solr
• Video and Blogs served by V2 API• Games and Articles still used V1• Mixed-infrastructure with Rails and Java
echosystem• Performance sucked royally
The API Evolution : V3 [2012]
• A fresh look at the APIs and IGN’s direction• First attempt to – Integrate all content types– Open up the APIs to external consumers– Evolve the CMS, traditionally a monolithic system– ElasticSearch– Varnish Response Caching
• A learning opportunity for traditional Java stack engineers
API Development
• MVC Model for APIs– View, which marshals/unmarshals request and
response data– Controller, which picks the right model to serve
the request– Model, which has the necessary logic to talk to the
database, or other APIs to fetch data
Why Scala
• Performance tied to cores than processor speed, i.e. concurrency
• Ideal for API development – (relatively) simple to handle concurrency and immutability
• JVM based = Performance• Re-use Java toolkits and libraries• Concise, expressive code (1:2.5 with Java)• Actor model makes the difficult parts easy
Why Scala
• Richer Collections API• Traits enabling Mix-Ins and behavior re-use• Statically Typed w/Type inference• Functional• Growing developer community• REPL• Culture – learning, hacking and achieving
The Biggest Reason
• Just makes sense
The Biggest Reason
• Just makes sense– Pattern matching– Implicit conversions– Control abstractions for collection mapping,
filtering, iterating– Guaranteed Singletons at the classloader level– Almost non-existent boilerplate….Productive, and fun!
Picking up Scala : Week 1
• Scala for Java Developers fast-tracked the learning
• Syntax Familiarity• Java without semicolons?
def total(nums: List[Int]) :Int = { var sum:Int=0 for(num<-nums) { sum+=num } sum}
Picking up Scala : Week 4
• Reduce the lines of code, embrace immutability, use the functional paradigms built into the language
• Unmystrify the implicits magic
nums.foldLeft(0)((n,c) => n+c)
Picking up Scala : Today
• Focus on immutability• Think functional – with the knowledge gained
from using the functional aspects of the language, build control abstractions
• Keep learning and applying
API Components
• Scalatra (View, Controller)• Lift-MongoRecord for MongoDB (Model)• Casbah for MongoDB• Play! Anorm for MySQL• Actors for tasks like syndication• Maven for Builds• MongoDB for persistance
Other API Components
• Varnish Cache • Elasticsearch• Yammer Metrics (ping, healthcheck)• Swagger (self-documenting RESTful APIs)• 3Scale for Partner APIs• IntelliJ IDEA and Eclipse with ScalaIDE
Challenges with Scala
• Learning Curve depends on the culture• TPS Cover Sheets, vs. Hack Nights
• Scala is what you make of it
Awesome..?
http://scalaz.github.com/scalaz/scalaz-2.9.1-6.0.4/doc.sxr/scalaz/BKTree.scala.html
Challenges with Scala
• Scala is what you make of it– Middle ground – simple is beautiful
• Tooling not as rich as Java – Changing with Typesafe investing in ScalaIDE
Challenges with Scala
• Slow compilation based on the source– How well the code matches up to the JVM
constructs
• No (binary) compatibility of dependencies for the major releases (2.8, 2.9, 2.10(?))
• Monitoring– No native instrumentation agents as they exist for
Java
API Development : Deploying
• Continuous Deployment• Jenkins hooks to github• Pushes artifact(s) to Nexus • Fabric to push code to production (to be
automated)
API Performance Management
• As far as Scala goes, the tips are not much different than Java– GC Tuning– Avoid shared state– Parallel collections
API Performance Management
• For the overall APIs, the performance tuning has many knobs to work around with– Stateless services– Async calls with Akka Agents– Response level cache
• Keep an eye on the database and network– Control the payload size– Use reasonable defaults – Allow field filtering– Paginate your response
Monitoring and Maintenance
• Newrelic– JVM stats, SQL trace– Supports Scala via annotations, but not as rich as
Java
• Typesafe Console for Akka
Future of APIs at IGN
• Opening up the APIs• Tests, tests, tests..• Move towards simplifying the stack– sbt vs. maven– Typesafe stack
The IGN Tech Stack
Resources• Do NOT use scaladoc as your 1st reference, at
least initially• StackOverflow• Twitter’s Scala School• Online blogs and tutorials• Book: Scala for the Impatient• Scala Meetup Groups in your area• Coursera course on FP by Martin Odersky
Questions