Moneyball for Software Teams?

Presented by Jonathan Alexander VP Engineering at Vocalocity Author of Codermetrics (O’Reilly 2011)

The Popularity of Moneyball

Advanced stats used to analyze baseball players and teams Bill James, the father of sabermetrics, author and consultant Michael Lewis, author of Moneyball published 2003 Moneyball starring Brad Pitt being released Sept. 23rd, 2011 O’Reilly Strata Summit: The Business of Data (NYC Sept. 20-21) has Paul

DePodesta, VP NY Mets as a featured speaker about Moneyball

Metrics Have Changed the Game(s)

Scouting Drafting Trades Coaching Player Development Salary Arbitration

Principles of Moneyball:

Identify Undervalued Assets

Analyze the skills correlated with winning Discover important skills that market undervalues Leverage knowledge to build winning teams

© Brandon Vincent | Dreamstime.com

Principles of Moneyball:

Challenge Assumptions

Gather situational statistics Objectively check assumptions against facts Adjust strategy based on analysis and results

© Richard Kane | Dreamstime.com

© Eric Broder Van Dyke | Dreamstime.com

Principles of Moneyball:

Study and Learn from Outliers

anomaly (noun): an incongruity or inconsistency, a deviation from the norm

outlier (noun): a person or thing that lies outside, a point widely separated from the main cluster

Games At Bats Hits Doubles Triples HRs RBI Avg. OPS All Stars

Piazza ‘92-’07 1,912 6,911 2,127 344 8 427 1,335 .308 .922 12

Pudge ’91-’10 2,499 9,468 2,817 565 51 309 1,313 .298 .800 14

Techniques Used For Moneyball

Leverage basic performance statistics Hits, Runs Batted In (RBI), Runs Allowed (ERA)

Add “situational” statistics gathered by “spotters” Errors, out-of-zone fielding, pressure situations

Develop “advanced” statistics through combinations and formulas OPS (on-base plus slugging) , FIP (fielding independent pitching) ,

BABIP (batting average on balls in play) , WAR (wins above replacement)

Analyze statistics to find best predictors of individual and team success

Moneyball for Software Teams?

Implement new techniques to gather metrics on a wide variety of contributions

Find ways to measure “wins” and “losses” Analyze how individual contributions and team

“chemistry” correlate to wins and losses Examine Assumptions Discover Patterns

Use metrics to create focus and help identify opportunities to change, adjust, improve

The Magic Triangle Challenge

Oft-discussed “triangle”: Features-Time-Quality Is it true? You can’t add more work unless you

lengthen time or reduce quality

Avg. Complexity

Total Complexity

Quality Problems

Release Quality %

Release 1 1.2 272 86 68%

Release 2 1.6 248 77 69%

Release 3 1.5 274 109 60%

Release 4 2.8 318 69 78%

Release 5 2.4 347 88 75%

Release 6 1.4 261 92 65%?

Release Quality % = 100 – (Quality Problems / Total Complexity)

Questions To Answer

How well do team members handle their core responsibilities? Examples: Design, Code, Test

In what ways do team members contribute beyond their core responsibilities? Examples: Innovate, Take Initiative, Handle Adversity

How much do team members help others? Examples: Assist, Mentor, Motivate

Is the software team succeeding or failing? Examples: New Users, Production Bugs, Efficiency

What Are The Roles On Your Teams?

Playmakers and Scorers Defensive Stoppers Utility and Role Players Backups Motivators Veterans and Rookies

Example: Skill Metrics

Metric Description Formula

Points Measure the overall productivity of eachcoder on assigned tasks

Points = Sum of complexity rating for all completed tasks

Utility Measure how many assigned tasks eachcoder completes

Utility = Number of tasks completed

Assists Measure the amount of coder interruptionsand how much a coder helps others

Assists = Count of times that coder helps others

Saves Measure how often a coder helps fix urgentproduction issues

Saves = Number of severe product issues coder helps fix

Tackles Measure how many potential issuesa coder handles proactively

Tackles = Number of times a coder takes initiative or innovates

Turnovers Measure the complexity of assigned tasksthat a coder fails to complete

Turnovers = Sum of complexity for all tasks that are not completed

Errors Measure the magnitude of productionissues found in areas of coder responsibility

Errors = Sum of bug severity factored by population affected

Range Measure how many areas of software a coderworks on

Range = Number of areas worked on by a coder

Example: Response MetricsMetric Description Formula

Wins Measure the number of active users added Wins = Sum (User Activations)

Losses Measure the number of active users lost Losses = Sum (User Deactivations)

Win Rate Determine the average amount of time ittakes to get a “win” (new user)

Win Rate = Time elapsed divided by the number of new users

Loss Rate Determine the average amount of time ittakes to accumulate each “loss” (lost user)

Loss Rate = Time elapsed divided by the number of lost users

Win Percentage Measure the percentage of trials that successfullyconvert to active users

Win Percentage = (Successful Trials / Trials Completed) × 100

Gain Measure the number of Wins minus themissed opportunities and Losses

Gain = Wins - ((Trials Completed – Successful Trials) + Losses)

Penalties Per Win Measure the overall urgency of customersupport issues relative to the number of new users

Penalties Per Win = Penalties / Wins

Example: “Advanced” Metrics

Metric Description Formula

Power Measure the average complexity of the tasksthat a coder completes

Power = Points / Utility

Temperature Measure how “hot” or “cold” a coder is atany given time (start with Temp. 72)

Temperature = Previous Temp. × (Current Points / Previous Points)

O-Impact “Offensive Impact” to summarize how a coderhelps move projects along

O-Impact = Points + Utility + Assists

D-Impact “Defensive Impact” to summarize how a coderhelps solve issues or avoid larger problems

D-Impact = (Saves + Tackles) × Range

Plus-Minus Measure the amount of positive contributionsversus negative issues for each coder

Plus-Minus = Points - Turnovers - Errors

Teamwork Establish a relative rating for team-orientedcontributions

Teamwork = Assists + Saves + Range - Turnovers

Fielding Establish a relative rating for the range andbreadth of work successfully handled

Fielding = (Utility + Range) - (Turnovers + Errors)

Intensity Establish a relative rating for heightenedproductivity and dealing with hot issues

Intensity = Saves + Tackles + (Avg. Temp. – Start Temp.)

Win Shares Assign a relative level of credit to each coderfor new users

Win Shares = Wins × Influence × Efficiency

Loss Shares Assign a relative level of responsibility toeach coder for lost users

Loss Shares = Losses × (1.0 - Efficiency)

Techniques to Gather and Track Metrics Get data from existing systems

Project tracking, bug tracking, customer support

Instrument your software for usage data New users, lost users, feature usage, measured benefits

Self-reporting or “spotters” for situational data Create documents or database for metric storage and tracking

Increase Team Awareness of Skills

Create Focus on Team Results

Identify Key Goals and Accomplishments

Metric Description Formula

Boost Measure the amount of additional userbenefits delivered

Boost = Sum of the percentage of users receiving each benefit

Acceleration Measure the ratio of user benefits deliveredto urgent user issues created

Acceleration = Boost / Number of Urgent User Issues) x 100

Steps for Adopting Metrics

1. Find a Sponsor

2. Create a Focus Group

3. Conduct a Trial (restart or stop if trial fails)

4. Introduce Metrics to the Team

5. Create a Metrics Storage System

6. Establish a Forum for Discourse

7. Expand Metrics Used and Analysis

Places and Times to Use Metrics Regular Team Meetings (sprint retrospectives) Project Post-Mortems Mentoring Establishing Goals and Rewards Performance Reviews (validation) Promotion Consideration

Moneyball Strategies forBuilding Better Software Teams

Recruit for “Comps” Profile your team, identify roles you need, then recruit

Improve the Farm System Use interns, contract-to-perm, promote from within

Make Trades Re-organize teams internally to fill roles and balance skills

Coach the Skills You Need Focus on those with aptitude, use target metrics

Recruiting Comps

Defensive Stopper Profile Candidate A Profile Candidate B Profile Candidate C Profile

Avg. Points Medium High Medium Medium

Avg. Utility Medium Medium Medium Medium

Avg. Assists Medium Low High Medium

Avg. Errors Low Medium Low Medium

Avg. Saves High Low High Medium

Avg. Tackles High Low Medium Low

Avg. Range Medium Low Medium Medium

Best CandidateTarget Profile

Potentially Undervalued Assets

Defensive Stoppers Utility Players Backups

Resources for Further Exploration

Codermetrics: Analytics for Improving Software Teams

262 Pages

Released August, 2011

In bookstores, Safari Online, or at http://www.oreilly.com

Codermetrics.Org – community website

Post ideas or stories

Share resources (spreadsheets, analysis)

Ask questions

Post Events

Follow on Twitter @codermetrics

Special Offer

Visit http://oreilly.com to purchase your copy of Codermetrics and enter code 4CAST to save 40% off print book & 50% off ebook with special code 4CAST

Visit http://oreilly.com/webcasts to view upcoming webcasts and online events