Ian McFarland, VP Technology ian@pivotallabs.com

Agile, Rails,and the Cloud

A Sea Change in Software Development

Ian McFarland, VP Technology ian@pivotallabs.com

Or...Why companies can’t afford to ignore

the efficiencies ofmodern development approaches

• We’ve developed over 100 products for client

companies

• We make Pivotal Tracker, leading agile planning tool in

its segment, will be #1 in 2010

• We’ve been doing agile for 10 years, first in Smalltalk,

then in Java, now Rails and JavaScript

• We’ve grown from 20 to 75 since starting the Rails

Practice 4 years ago

• We do enablement, but mostly implementation

Pivotal Labs

http://pivotaltracker.com/

• VP Technology at Pivotal Labs

• Been doing network distributed hypertext application

design since 1989

• Launch team at Hotwired, who brought you the

banner ad, and commercial traffic on the Internet

• Started doing web development consulting starting

1995, Java in 1996

• Chief architect at Friendster

Me

• We built it in Java first

• We rewrote in PHP + XSLT (which solved nothing)

• We didn’t have access to the cloud

• We doubled the physical plant every month the first 6

months I was there

• Scale for us was a question of access to a big enough

footprint

• 115MM Users

Friendster?

• 50MM user message bus

• The app tier isn’t what fails

Twitter?

• Horizontally scalable Rails application tier (Ruby)

• AJAX framework (JavaScript)

• Apache or Nginx (C)

• MySQL or PostgreSQL (C)

• Memcached (C)

• SOLR for Search (Java)

• XMPP Message Bus (Erlang)

• Images on S3/CloudFront (C)

• Non-relational Data Stores (Various)

What is a typical “Rails” Application?

And those other tiers aremostly just cloud services

• Performance and scalability are not

the same thing

• Actual performance and apparent

performance are not the same thing

• Apparent performance is really what

you need to manage for

By the way...

• The Rails standard approaches get you

(almost) share nothing architecture

• Build your app first

• Be asynchronous

• Use better gems and approaches

• Use the right indexes, avoid n+1 selects

• Don’t be stupid

• Tune when the app is more complete

Follow good conventions,Don’t preoptimize

• Technology selection should be driven

by economics

• Scalability and performance primarily

about architecture choices

• Developer performance is probably

your biggest cost center

How to choose your stack

• To make money

• To save money

• To manage risk

• To satisfy business and government requirements

Why do Companies build software

This revolution in Software is

about one thing: Building

business value as cheaply and

efficiently as possible

Building Business Value

• Developers

• Defects

• Deployment/Operations

• Code Maintenance

• Change

So what’s expensive about software

•Agile

•Rails

•The Cloud

Three Trends

What is it? What’s in it for Developers

What’s in it for Businesses

Agile

Cleaner, more flexible code; more fun coding;

code is more maintainable

Fewer defects, more predictable delivery, more transparency, business determines

what’s built

Rails

More powerful, more expressive, less grunt work, more gets done with less effort, more

fun.

Less effort = lower costLess effort = shorter

time to marketLess effort = lower TCO

CloudEasy Scaling, capacity planning, setup, no

pagers

Lower TCO, No initial investment, lower

operating costs, shorter deployment cycle, no

sunk cost

• Budgets are smaller

• The stakes are higher

• Departments have to do more with less money

• Failure, though always an option, is more catastrophic

Simple Economics

Agile?

• Agile is about rigor.

• Agile is about transparency.

• Agile is about testing and reliability.

• Agile is about predictability.

• Agile is about reducing the cost of

change.

How you develop software is important

• Bulleted Text Goes Here

• Bulleted Text Goes Here

• Bulleted Text Goes Here

• Bulleted Text Goes Here

• Bulleted Text Goes Here

Title

Manifesto for Agile Software DevelopmentWe are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

© 2001, the above authorsthis declaration may be freely copied in any form,

but only in its entirety through this notice.

Kent Beck James Grenning Robert C. Martin

Mike Beedle Jim Highsmith Steve Mellor

Arie van Bennekum Andrew Hunt Ken Schwaber

Alistair Cockburn Ron Jeffries Jeff Sutherland

Ward Cunningham Jon Kern Dave Thomas

Martin Fowler Brian Marick

• Business Driven: Requirements come from business stakeholders

• Iterative Development, with Short Iterations

• Test/Behavior Driven Development

• Continuous Integration, Continuous Releasability

• Pair Programming

• Productive Work Environment

That’s nice... How do we do that?

• Requirements come from business stakeholders

• One designated Customer is empowered to make decisions

• Priorities are set by that Customer

• The Customer can change priorities on anything unstarted

• The Customer accepts the work in fine-grained increments

• The Customer is intimately aware of progress, and projected

completion dates

• Closing the feedback loop is critical

Business Driven

Accept Reject

• Good tests tell us when we’ve met the customer requirements

• They tell us when we’ve broken behavior that used to work

• They tell us when we haven’t, so we can refactor with impunity

• Writing tests first keeps us from overdesigning/doing things we

don’t need to do

• Writing tests first forces cleaner API design, because we have to

call into our own code in order to write it

• It leads to looser coupling and encourages higher cohesion

• Good developer testing keeps the cost of change constant

TDD/BDD

• Because the customer is seeing the work on a daily basis, the

feedback cycle is short

• This keeps the cost of change low, by preventing unnecessary

work

• It allows for new insights to be gained from the work we’ve

already completed, and for those insights to be incorporated

into our new code

• Iterations are as short as we can make them

Iterative Development

• Knowing when things break is critical to reducing the cost of

fixing defects.

• Keep the build status visible, so you can fix it quickly

• A broken build is a ‘stop the line’ event

• Continuous releasability does not mean you release every day.

• It just means you can.

• Releases can be distracting, so weigh the cost of a release

against the value it adds to the business.

Continuous Integration,Continuous Releasability

Pair Programming

•Yes, you do.• ...but only if you want to be efficient

• This is one of the least-used practices, and one of

the most important.

• And stop whining! You do it already when you get

stuck on something.

Do we really have to pair?

• Coding

• Reading web pages about coding

• Stuck on some problem, unsure of:

• The right approach

• What the API for that object was

• How SQL indexes are selected

• How bind(this) works in JavaScript

• Checking email

• Checking news, stock price, staring blankly into space

What do developers really do all day?

• 80/20 rule: You don’t get stuck, so you spend your time on

the most interesting part of the code.

• As you eliminate the grunt work (thanks Rails) more of the

work requires real thinking, and design

• You talk through design, and refine before you code.

• You learn from your pair, everything from design and testing

techniques to SQL, CSS, and JavaScript tips.

• Focus matters: Your pair keeps you paying attention, and

can smooth over disruptions

How does pairing help?

• More developers in a smaller space

• How many truly independent fronts are there in your

codebase on which you can make progress?

• New team members: You’re really productive the first hour,

not marginally productive starting two weeks in

• They have a local sherpa to tell them how the code they’re

working on actually works.

• Knowledge Silos: Your bus number approaches ∞

How does pairing help?

• Open Workspace

• Colocated Developers and Customer

• Consistent Pairing Stations

• One big screen, 2 keyboards (we use 24” iMacs)

• No laptops on the floor

• Visible build monitors

• Everyone can see the backlog in Tracker

• Breakfast, snacks and beverages on hand

• Space for interruptions away from the workspace

Productive Workspace

• Convention over Configuration

• Fewer lines of code

• More developer comprehensibility

• Much greater developer productivity

• More extensibility (DSLs & metaprogramming

made easy)

• Agile baked into the libraries and the culture

...oh and by the way, it’s Web based!

Rails: Why should businesses care?

• When we started doing Rails, we couldn’t hire Rails

developers, because there weren’t any

• So we converted Java developers into Rails developers

• The same developers were about 2x as productive in

business terms after one month

• Once they actually got good at Rails, they were about 4x

as productive

Rails compared to Java:From an anecdotal perspective

• Some people report as much as 5-10x

• I suspect that’s because for them “Rails” includes

“Agile”

• Large client: 10 Pivots, 10 client, vs. 50 offshore

• Pivotal Labs had already been doing Agile for 10 years,

so we already got the productivity benefit.

Rails compared to Java:From an anecdotal perspective

• Predictable delivery is at a premium

• Tired developers introduce bugs

• Developer retention is still important

• Good developers are never easy to come by

• Ramp-up is still expensive

• Team changes expose companies to risk

Why Sustainability Matters

• Leading Indicator: Developer Happiness strongly

correlated to Developer Productivity

Grunt Work = Money Wasted

• Retention Matters

• Happy workers are more focused

Why Developer Happiness is Important to the Business

• The Cloud means you don’t have to buy hardware

anymore... ever. And scale is available on demand... if you

architect (or refactor) for it.

• Lets you stay focused on where you provide differentiating

value. (Unless you’re in the Ops business or have a sick

fascination with pagers, you shouldn’t do your own ops.)

[See also: Rails, Agile]

• But... enterprise business rules don’t always allow for this,

at least for some systems, at least not yet.

The Cloud

• Obvious Quantitative Wins:

• Almost zero provisioning time

• Scalable on demand

• No risk, no capex, low opex

• specialization breeds efficiency

How the Cloud changes the Game

• Less Obvious Qualitative Wins:

• Spin up test environments that match production

• Realistic load testing environments much simpler

• Clone production data when debugging

• Snapshot production to isolate production issues

How the Cloud changes the Game

• Developers

• Defects

• Deployment/Ops

• Code Maintenance

• Change

So how did we do?

Differentiating Value

• Does the work I’m doing materially relate to my company’s

core line of business?

• Does it provide competitive advantage?

• Does it make users happier or more productive?

• Does it reduce operating cost, or improve efficiency?

• Does it eliminate drudge work, so people can

concentrate on higher-value work?

• Could it be done better by an existing library or product?

• If I didn’t do this work, would it matter?

Differentiating Value:Questions to ask yourself

ARC is about getting to where

developers can work on the

interesting, differentiating problems,

at each layer

Interesting Problems

• Agile Manifesto: http://agilemanifesto.org/

• Flickr Photos used under Creative Commons:

Cloud by akakumo, Risk Factory by kyz, Laundry by T.M.O.F., Tank engine by Corvair Owner

Q&A

Photo and Text Credits

http://pivotallabs.com/talksMore Resources

Contact Info@imf or ian@pivotallabs.com