John Nestor and Dragos Manolescu

{jnestor,dmanolescu}@whitepages.com

Whitepages Practical Experience Converting from

Ruby to Reactive

August 13, 2014

Disclaimer The following material is subjective and represents the views of the presenters.

We hope our views and experience may be of use to others making the transitive from Ruby to Scala, but your mileage may vary.

May 15, 2013

Whitepages Typesafe Case Study • WhitePages Rebuilds Core Parts of Application Stack with

Scala and Akka to Improve Scaling.

• http://j.mp/wpscala

May 15, 2013

Outline • Background

• Technology

• People

May 15, 2013

Background

About Whitepages • Top web and mobile site for finding phones, people and locations.

• 50M unique users per month.

• 35M search queries per day.

• 70 engineers, mostly in Seattle.

May 15, 2013

Why replace Ruby? Problems with Ruby legacy systems:

•  High latencies.

•  High resource consumption: memory and processors.

•  Poor tolerance of upstream degradation:

–  Limited concurrency.

–  Worker starvation when blocking on slow upstream.

–  Poor connection management and healing.

May 15, 2013

Why replace Ruby with Scala? •  Elegant blend of Functional and OO paradigms.

•  Static type system:

•  Type inference avoids the excess boilerplate of Java.

•  Compiler catches many errors.

•  Runs on the JVM:

–  Fast.

–  Near seamless interop with JVM libraries.

–  Reasonably mature tooling support.

•  Actor-based concurrency framework: Akka.

May 15, 2013

Whitepages Reactive Services •  Service-oriented-architecture where communication is Thrift

or Json over HTTP.

•  Both latency and throughput are very important.

•  We require excellent logging and monitoring.

•  Agility of development, testing, build and deployment provide leverage and competitive advantage.

May 15, 2013

Performance

Results Service p50 ms p99 ms Throughput RPS/core

DirSvc - Scala 25 300 80

DirSvc - Ruby

140 1200 7

May 15, 2013

Status of Reactive Scala Services •  We have been using Scala for a little more than 1.5 years.

•  We started with 1 development team, we now have 4.

•  We started with a single back end service. We now have 4 services in production and another 6 in development.

•  Our first project had 6 Scala developers, we now have over 20 Scala developers.

•  We changed not only language but also tools (more automation) and processes (more agile).

May 15, 2013

Future Reactive Work •  Transition additional Ruby back-end services to Scala.

•  Exploring new technologies:

•  Spark and Mesos.

•  A messaging system based on Kafka.

•  A graph store based on Titan/Cassandra.

•  A deployment and lifecycle-management system based on Docker that uses remote actors.

May 15, 2013

Technology

• GitHub Enterprise

•  IntelliJ

• Jenkins (for builds)

• Artifactory

• Yourkit

• SCoverage

• Scrooge (for thrift, from Twitter)

• Scala

• Akka

• Spray

• SBT – with custom plugins

• Typesafe Config

• Scala Test

• Coda Hale Metrics

• Persist Json

Technology Selection

May 15, 2013

Whitepages Service Framework •  Configuration (based on Typesafe Config, with multi-level overrides).

•  Monitoring (central actor that uses Coda Hale metrics and publishes to Graphite, Nagios and Riemann).

•  Logging (custom system with central actor, Json log content and real time streaming to external analytics).

•  Admin REST API.

•  REST Thrift/JSON Server (based on Spray).

•  Clients (Spray, JDBC, Redis, SOLR, Riak,… -- advanced error detection/ recovery).

May 15, 2013

Technology Successes •  Scala concise syntax. – Higher developer productivity.

•  Use of async future based code. – Great performance.

•  Immutable collections and functional programming. – Fewer bugs.

•  Strong type checking. – Fewer bugs and more maintainable code (But meta-programming becomes more difficult).

•  Concurrency. – No race conditions or deadlock on complex concurrent system.

•  Spray. – Great performance and an async API.

•  SBT. – Able to easily customize to our needs (after steep learning curve).

•  IntelliJ IDEA. – Scala support is good and keeps getting better.

•  Typesafe Developer support contract. – Fast response, great answers to complex questions. (We should have gotten it earlier!)

May 15, 2013

Technology Problems -1 •  Poor documentation:

•  SBT is the poster child.

•  We still often have to read source code of Scala and Akka libraries.

•  Stable APIs:

•  Typesafe has made great strides here.

•  Other components are still troublesome.

•  We lost many weeks as a result of breaking changes in Scala Test.

•  We currently can’t compile a 2.11 version of Kafka because the Zinc Scala compiler is not backward compatible.

May 15, 2013

Technology Problems -2

•  Performance analysis of concurrent constructs:

•  We went with the Typesafe Console.

•  It never fully worked and then was abandoned.

•  There are some new tools, but we have not had time to evaluate.

May 15, 2013

Technology Problems -3 •  Ecosystem:

•  Not as mature as the Java ecosystem and often missing Scala specific features. For example, Scala Mock does not support named parameters.

•  Some needed components were missing: until recently code coverage.

•  In some cases there were too many options: over 10 Json libraries.

•  Lots of Scala GitHub projects. Quality is variable; many become abandoned.

May 15, 2013

Technology Problems -4 •  Debugging:

•  Debuggers work on the Java byte code, that for some Scala constructs is quite convoluted.

•  Debugging chains of futures is much harder than debugging nested calls. Some kind of tracing is desperately needed.

•  Actors have lots of power but can be exceptionally hard to debug. We need both better approaches to building debug-able actors and better debugging tools.

May 15, 2013

Technology Problems -5 •  Language and Library Issues:

•  Type erasure is a major flaw in the Scala language. It causes lots of developer confusion and errors. The workarounds are awkward at best.

•  The lack of type checking on Actor constructors is an odd hole in the type checking and a real annoyance.

•  Some Scala code that looks simple and intuitive creates major difficulties when a developer needs to understand what is really going on. Implicits are one of many problems here.

May 15, 2013

Technology Overall •  The problems mentioned above create real day to day difficulties.

•  In many cases Typesafe has been actively working to incrementally correct some of these problems.

•  Since we started using Scala, the number of problems has significantly decreased and we expect that trend to continue.

•  Overall, Scala/Akka is a great technology for building reactive systems and its benefits far outweigh the problems.

May 15, 2013

People

Levels of Scala Expertise •  Basic OO features. Easy transition from most other languages including

Java and Ruby.

•  Adding functional. Ruby has functional features that are similar. But knowing when best to use functional versus OO requires experience.

•  Adding concurrency. Via futures, promises, and actors. A whole new world for Ruby programmers that requires a new way of thinking.

•  API design. Not all developers need to design reusable APIs, but those that do need a higher level of experience and feature mastery to do it well.

May 15, 2013

Building Scala Development Teams •  The are still not a large number of experienced Scala developers.

•  Those out there are in considerable demand.

•  We have followed a mixed strategy of both recruiting Scala developers and also training existing Ruby developers to become Scala developers.

•  Having an experienced Scala developer greatly accelerated the training of Ruby developers.

Note: Using Scala, Akka and other advanced technologies makes Whitepages very attractive to some of the very best developers.

May 15, 2013

Recruiting: Finding Developers •  Personal contacts. The best way to find great developers.

•  Meet-ups and Events

•  John Nestor is the organizer for the Seattle Scala meet-up.

•  Whitepages hosts the Scala meet-up (and several others)

•  Whitepages sponsors many other technology conferences and events.

•  Job posts: local Seattle and internet Scala focused.

•  College recruiting events. UW and other top CS schools.

May 15, 2013

Recruiting: Attracting Developers •  Strong support from the Whitepages CTO and CEO and other management for Scala

and other advanced technologies.

•  We have a great office (recently completely redone)

•  We provide developers with the best workstations (top-end MacBook pros).

•  We have competitive benefits, excellent work-life balance and a collaborative supportive culture.

May 15, 2013

Training: External Resources

•  Books. Odersky: Programming in Scala, and others

•  Coursera. Functional programming, reactive programming.

•  Typesafe training classes. Both Seattle public and in-house.

•  Typesafe developer support contract and consulting. Particularly valuable if you don’t have in-house Scala expertise.

•  Conferences. We sent people to Scala Days in NY and Berlin.

May 15, 2013

Training: Internal Resources •  Whitepages developers are in a single office and come in most days.

•  This permits a high degree of personal interactions.

•  Those interactions enable rapid learning and high code quality.

•  This would be much harder with multiple offices and/or widely distributed teams.

May 15, 2013

Training: Internal Approaches •  Expert mentoring.

•  Provide quick answers to questions.

•  Detailed reviews and feedback for Ruby developers initial Scala code.

•  Code reviews

•  Improve code quality.

•  Help those learning Scala.

•  Develop a shared culture and style.

•  We use a mixture of group and pair code reviews.

•  Electronic means like code review tools and pull requests were less effective.

May 15, 2013

Training: Success and Failure •  Not everyone made a successful transition from Ruby to Scala.

•  Next slide give some positive and negative indicators…

May 15, 2013

Training: What to Look For • Desire to learn new technologies.

• Desire to learn new ways to think.

• Prior functional language experience.

• Likes to work closely with others.

May 15, 2013

Scala Features to Avoid •  Scala is a very large language.

•  New developers can easily get lost on the rich feature set. Often there are several different ways of doing something, and they don’t always know how to choose.

–  We also provide the following list of features for new developers to avoid (or to fully understand before using).

–  We encourage asking more experienced developers to provide that understanding.

–  These features are also flags for discussion in code reviews.

May 15, 2013

• Reflection and macros

• asInstanceOf

• Top level objects with mutable state

• Lazy vals

• Scala enumerations

•  ? (the ask pattern)

• onComplete (unless body enclosed in try)

• Nulls

• Return, break, continue

• Mutable collections

•  var, while, until

• Generics

•  Implicits

• Actors

• Await

Scala Features to Avoid

May 15, 2013

We Are Hiring*!

May 2013

*Smart people (not just technologies) are key to our success

Thank you! Questions?