the roots of scrumathena.ecs.csus.edu/~buckley/csc231_files/sutherland_scrum_2010.… · chairman,...

CSM v 11.03 © Jeff Sutherland 1993-2010

THE ROOTS OF SCRUMHow the Japanese lean experience changed

global software development

With help from Citrix Online, Google, Yahoo, Microsoft, IBM, Oracle, MySpace, Adobe, GE, Siemens, Disney Animation, BellSouth, Alcatel-Lucent, GSI Commerce, Ulticom, Palm, St. Jude Medical, DigiChart, RosettaStone, Healthwise, Sony/Ericsson, Accenture, Trifork, Systematic Software Engineering, Exigen Services, SirsiDynix, Softhouse, Philips, Barclays Global Investors, Constant Contact, Wellogic, Inova Solutions, Medco, Saxo Bank, Xebia, Insight.com, SolutionsIQ, Crisp, Johns Hopkins Applied Physics Laboratory, Unitarian Universalist Association, Motley Fool, Planon, FinnTech, OpenView Venture Partners, Jyske Bank, BEC, Camp Scrum, DotWay AB, Ultimate Software, Scrum Training Institute, AtTask, Intronis, Version One, OpenView Labs, Central Desktop, Open-E, Zmags, eEye, Reality Digital, DST, Booz Allen Hamilton, Scrum Alliance, Fortis, DIPS, Program UtVikling, Sulake, TietoEnator, Gilb.com, WebGuide Partner, Emergn, NSB (Norwegian Railway), Danske Bank, Pegasystems, Wake Forest University, The Economist, iContact, Avaya, Kanban Marketing, accelare, Tam Tam

ACCU, Oxford, UK 14 Apr 2010

1Tuesday, April 13, 2010

© Jeff Sutherland 1993-2010

Jeff Sutherland, Ph.D.

Chairman, Scrum Training InstituteCEO Scrum, Inc. and Senior Advisor, OpenView Venture Partners

Agile coach for OpenView Venture Partners portfolio companies

Chief engineer for 11 software companies

Created first Scrum at Easel Corp. in 1993. Rolled out Scrum in next 5 companies

Achieved hyperproductive state in all companies. Signatory of Agile Manifesto and founder of Agile Alliance

– http://jeffsutherland.com/scrum

– [email protected]


mailto:[email protected]

mailto:[email protected]


Management Evolution

New England management culture: “The Puritan Gift”

Japanese Lean Culture

HondaToyota

AsianMiracles:China, South Korea, ...

American Business Schools

Global Financial Crash

Scrum

Takeuchi & NonakaEdwards Deming

General MacArthur

West Point

FraudGreed Corruption

Making the World a Better Place

Leadership Honesty Transparency



Roots of Scrum Making the world a better place

– Japanese manufacturing - Edward Deming– Team process – Silicon Valley entrepreneurs– Micro enterprise development – Accion and Grameen Bank

Process innovation and productivity research– Alan Kay and Xerox Parc– Takeuchi and Nonaka - knowledge generation/lean– IBM Surgical Team (Mythical Man Month)– Coplien review of Borland Quattro Project– Complex adaptive systems and iRobot subsumption

architecture



Extrem

e Program

ming

Scrum

DSDMFDD

Lean Crystal

Agile Methods



Techniques or Methodologies Used

Source: Forrester Research December 2008 Global Agile Company Online Survey



Lean Thinking Tools

Tool 1:Eliminate

Waste

Tool 2:Value Stream

Mappig

Tool 3:Feedback

Tool 4:Iterations

Tool 5:Synchronization

Tool 6:Setbased

development

Tool 7:OptionsThinking

Tool 8:Latest

ResponsibleMoment

Tool 9:DecisionMaking

Tool 10:Pull

Tool 11:QueueTheory

Tool 12:Cost of Dealy

Tool 13:Self-

determinatoion

Tool 14:Motivation

Tool 15:Leadership

Tool 16:Expertise

Tool 17:Perceivedintegritet

Tool 18:ConceptualIntegritet

Tool 19:Refactoring

Tool 20:Test

Tool 21:Measures

Tool 22:Contracts

P1Eliminate waste

P5Empower

team

P4Fast

Delivery

P3Responsibledecisions

P2 Amplify Learning

P7See theWhole

P6Buil

integrity in



Systematic’s model for Lean SW development

These are thinking tools – Projects and employees knows best how to transform them

Tools can be divided in three

dimensions

Value

Flow

Pull

Perfection

Engineering

Management

People

P6 Build Integrity in

T19 Refactoring T20 Test

P2 Amplify learning

T5 Synchronization T4 Iterations

P2 Amplify Learning

T3 Feedback T6 Setbased

development

P6 Build Integrity In

T18 Conceptual integrity

T17 Perceived integrity

P1 Eliminate Waste

T1 Eliminate Waste T2 Valuestreams

P4 Fast Delivery

T11 Queuing Theory T12 Cost of Delay

P7 See the whole

T22 Contracts T21 Measures

T10 Pull

P3 Decide in latest Responsible moment

T7 Options thinking T8 Latest responsible

MomentT9 Beslutningstagning

P5 Empower team

T16 Expertise

P5 Empower team

T14 Motivation

P5 Empower team

T15 Leadership

P5 Empower team

T13 Self-determination



Scrum applied to CMMI Level 5 company – 6 month results

10%

20%

a

30%

50%

40%

60%

CMMI 1 CMMI 5

70%

80%

90%

100%

CMMI 5SCRUM

Project effort Rework

Work

Process focusCMMI

SCRUM

50 %

50 %

50 %

10 %

9 %

6 %

25 %

4 %

100 %

69 %

35 %


© Jeff Sutherland 1993-201004/06/10

8

Company Structure

Bureaucracy•Rigid rule enforcement•Extensive written rules and procedures•Hierarchy controlsD

isciplined

Whimsical Autocracy

•Top down control•Minimum rules and procedures•Hierarchy controls

Leadership•Empowered employees•Rules and procedures as enabling tools•Hierarchy supports organizational learning

Organic•Empowered employees•Minimum rules and procedures•Little hierarchy

Coercive EmpoweringAdapted from Liker, JK (2004) The Toyota Way. McGraw Hill.



Scrum is based on complex adaptive systems

Self organization No single point of control Interdisciplinary teams Emergent behavior Outcomes emerge with high dependence on relationship and

context Team performance far greater than sum of individuals

J. Sutherland, A. Viktorov, and J. Blount, Adaptive Engineering of Large Software Projects with Distributed/Outsourced Teams, in International Conference on Complex Systems, Boston, MA, USA, 2006.



iRobot Subsumption ArchitectureSubsumption architecture is an AI concept originating from behavior based robotics. This term was invented by Rodney Brooks and colleagues in the mid to late 1980s.A subsumption architecture is a way of decomposing complicated intelligent behaviour into many "simple" behaviour modules, which are in turn organized into layers. Each layer implements a particular goal of the agent, and higher layers are increasingly more abstract. Each layer's goal subsumes that of the underlying layers, e.g. the decision to move forward by the eat-food layer takes into account the decision of the lowest obstacle-avoidance layer.For example, a robot's lowest layer could be "avoid an object", on top of it would be the layer "wander around", which in turn lies under "explore the world". The top layer in such a case could be "create a map", which is the ultimate goal. Each of these horizontal layers accesses all of the sensor data and generates actions for the actuators — the main caveat is that separate tasks can suppress (or overrule) inputs or inhibit outputs. This way, the lowest layers can work like fast-adapting mechanisms (reflexes), while the higher layers control the main direction to be taken in order to achieve the overall goal. Feedback is given mainly through the environment.


http://en.wikipedia.org/wiki/Artificial_intelligence

http://en.wikipedia.org/wiki/Artificial_intelligence

http://en.wikipedia.org/wiki/Behavior_based_robotics




http://en.wikipedia.org/wiki/Rodney_Brooks

http://en.wikipedia.org/wiki/Rodney_Brooks

http://en.wikipedia.org/wiki/Modules

http://en.wikipedia.org/wiki/Modules

http://en.wikipedia.org/wiki/Reflex

http://en.wikipedia.org/wiki/Reflex


Scrum – value driven not plan drivenEmpower lean teams to deliver more software earlier with higher quality.Demonstrate working features to the customer early and often so the customer can inspect progress and prioritize change.Deliver exactly what the client wants by directly involving the customer in the development process.Provide maximum business value to the customer by responding to changing priorities in real time.



Agile Manifestowww.agilemanifesto.org

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

Individuals and interaction

over Processes and tools

Working software over Comprehensive documentation

Customer collaboration over Contract negotiation

Responding to change over Following a plan

That is, while there is value in the items on the right, we value the items on the left more.



Scrum is a Simple Framework

Scrum

Meetings

Sprint Planning

Daily Meeting

Team

Product Owner

ScrumMaster

Burndown Charts

Sprint Backlog

Product Backlog

Sprint Review

Social Objects Roles


CSM v 11.03 © Jeff Sutherland 1993-2010M. Denne and J. Cleland-Huang, Software by Numbers : low-risk, high-return development. Upper Saddle River, N.J.: Prentice Hall PTR, 2004.



Are you doing Scrum? The Nokia Test by Bas Vodde

First, you must be doing iterative developmentIterations must be timeboxed to less than six weeks

Software must be tested and working at the end of an iteration

Iteration must start before specification is completeThen you must meet the Nokia Scrum test

1969 - Earliest published reference to Iterative Incremental development

Robert Glass. Elementary Level Discussion of Compiler/Interpreter Writing. ACM Computing Surveys, Mar 1969

See Larman, Craig and Basili, Vic. Iterative and Incremental Development: A Brief History. IEEE Computer, June 2003 (Vol. 36, No. 6) pp. 47-56


http://csdl2.computer.org/persagen/DLAbsToc.jsp?resourcePath=/dl/mags/co/&toc=comp/mags/co/2003/06/r6toc.xml

http://csdl2.computer.org/persagen/DLAbsToc.jsp?resourcePath=/dl/mags/co/&toc=comp/mags/co/2003/06/r6toc.xml


Nokia Scrum Test You know who the product owner is There is a product backlog prioritized by business value The product backlog is has estimates created by the

team The team generates burndown charts and knows their

velocity There are no project managers (or anyone else)

disrupting the work of the team

Kniberg, Henrik. Scrum and XP from the Trenches: How We Do Scrum. Version 2.1, Crisp, 5 Apr 2007.


CSM v10.21 © Jeff Sutherland 1993-2009

Venture Capital Strategy: Follow the money

Invest only in Agile projectsOne hyperproductive company out of 10 might meet investment goals for a venture group

Two or more hyperproductive could alter the marketInvest only in market leading, industry standard processes – Scrum with XP engineering practicesEnsure teams implement basic Scrum practices

Consistently improving scores on the Nokia test

Management held accountable at Board level for removing impediments

Generate hyperproductive Scrum



Toyota Motor Manufacturing North America Mission

1. As an American company, contribute to the economic growth of the community and the United States.

2. As an independent company, contribute to the stability and well-being of team members.

3. As a Toyota group company, contribute to the overall growth of Toyota by adding value to our customers.



Scrum Mission Statement

Build communities of stakeholders (customers, companies, development teams) that increase the economic well-being of all concerned.Enhance development team work environments by empowering people to work together in more creative, innovative, and productive ways.Deliver the highest possible customer value in the shortest possible time to improve the customers work experience. Make systems easier to use, more helpful to the user, and more fun to experience.

Yahoo Chief Product Owner – “Scrum is faster, better, cooler! It’s the way we first built software at Yahoo, yet is scalable to large, distributed, and outsourced teams.”



Scrum Community of Practice

!"#$%&'($)$%*'

!"#$%&'+#",&-.'

(/%&%"'

0#"1)"&'

2-3/)4'

5%3.)"4'

6)")&)'

7"&#)'

8%".)39'

2%$:%31)"&*'

;3)<#1'

=>*$3)1#)'

6:#")'

03)"?%'

Certified ScrumMasters

Computerworld estimates that over 2/3 of Internet projects in the U.S. use Agile methods, about 167,000 projects. (Sliwa, 2002)



Theory: Scrum Origins

Project Management Styles

Type A – Isolated cycles of work

Type B – Overlapping work

Type C – All at onceThe overlapping of phases does away with traditional notions about division of labor. Takeuchi and Nonaka (1986)

NASA Waterfall

Fuji-Xerox Scrum

Honda Scrum

Requirements Analysis Design Implementation Testing



Rugby Scrum


CSM v 11.03 © Jeff Sutherland 1993-2010

Good Scrum

All Black Video



Toyota synthesis of constraints

Historical assumption is that high quality, product variety, and low cost cannot be achieved simultaneously.

Toyota production system is based on totally different way of thinking.

Through knowledge creation by synthesis of contradictions, Toyota pushes the envelope.

High quality, high variety, and low cost all at once.



Prioritize and reprioritize Product Backlog to deliver business value

Time

Features

Contract

CustomerNeed



Scrum cuts through cost, time, functionality barriers

• Agile process – adapt and inspect

• Iterative, incremental – close to customer

• Used to manage complex projects since 1990;

• Delivers business functionality every 30 days;

• Extremely simple but very hard



We simultaneously want FASTER:Putnam Process Productivity Index

N = 1.27*(N-1) 33 is 2098 times 1 Myers, W., Why Software

Developers Refuse to Improve. Computer, 1998. 31(4) 110-112

Industry Average

IDXSystematic Primavera

Easel product

Easel developmentXebiaSirsiDynix

Borland Quattro



And BETTER

Productivity – product backlog requirements completed per 100,000 investment

Source: Primavera

Months since Type B Scrum implemented

3 12 24

Productivity 4.5 9.0 12.2

Quality 100+ 100 5



And MORE for LESS1. Welcome changing requirements, even late in development.2. Deliver working software frequently. 3. Business people and developers work together daily.

Scrum is ITERATIVE, customer can CHANGE requirements, and solution EMERGES through self-organization

Scaling Planning Developing Deploying



Old Organization New OrganizationCentralized DistributedUnified perspective Diversified perspectiveOriginal meaning Emergent meaningAnalytical CreativeAnalysis to action Learning by doingRational RedundantCertain UncertainStrategy concept Local actionAuthoritative ParticipativeHierarchical Flat

Cultural Change – Japanese style



Breaking down command and control

Intended strategy is developed centrally. Emergent strategy self-organizes through local actions.– Distributed cognition and actions

Scrum team must be allowed to self-organize– Autonomous– Transcendent– Cross-fertilization

Team chooses own work– Individuals manage their own work– Management gets out of the way



Distributed Organization - Google Way

When Rosing started at Google in 2001, "we had management in engineering. And the structure was tending to tell people, No, you can't do that." So Google got rid of the managers. Now most engineers work in teams of three, with project leadership rotating among team members. If something isn't right, even if it's in a product that has already gone public, teams fix it without asking anyone. Agile Principle #5, 9, 12

"For a while," Rosing says, "I had 160 direct reports. No managers. It worked because the teams knew what they had to do. That set a cultural bit in people's heads: You are the boss. Don't wait to take the hill. Don't wait to be managed." Agile Principle #1, 3

And if you fail, fine. On to the next idea. "There's faith here in the ability of smart, well-motivated people to do the right thing," Rosing says. "Anything that gets in the way of that is evil. Agile Principle #5, 12



Google and Scrum

“With the help of an experienced agile leader (Scrum Master, XP coach…) it was possible to carefully introduce agile practices into Google - an environment that does not have an affinity to processes in general. “Along with these practices came a visibility into the development status that gave the approach great management support. “All this could be done without destroying the great bottom-up culture that Google prides itself of.”

Mark Striebeck Google AdWords Project Leader and ScrumMaster



Powered by Scrum

AdWords is the “KaChing” machine at Google. It makes most of the money (2.5B Q3/06) that gives Google its market cap of $147B just behind Chevron and ahead of Intel. Arguably, the most profitable software application in the history of computing is powered by Scrum.



Diversified perspective

Cross-functional teams Scrum team has product knowledge, business

analysts, user interface design, software engineers, QA

Advanced Scrum pulls in all stakeholders – management, customers, installation, and support.



Toyota Prius – emergent strategy Revolution in product, technologies, and

process– Does not fit any product line. Designed for new

perspective. Uses many technologies

– Engine, motor, battery, braking combine into hybrid system

Developed in record time– 15 months instead of four years

Overlapping phases– Research, development, design, production

Leaders built, utilized, and energized “ba”



The concept of ba

Dynamic interaction of individuals and organization creates a synthesis in the form of a self-organizing team.

It provides a shared context in which individuals can interact with each other.

Team members create new points of view and resolve contradictions through dialogue.

Ba is shared context in motion where knowledge as a stream of meaning emerges.

Emergent knowledge codified into working software self-organizes into a product.



Prius project team managed “Ba”

Leaders can “find” and utilize spontaneously formed ba

Leaders can build ba by providing space for interactions– Physical space such as meeting rooms– Cyberspace such as computer network– Mental space such as common goals

Fostering love, care, trust, and commitment forms the foundation of knowledge creation (self-organization)

Scrum is based on TRUTH, TRANSPARENCY,



Energy of ba is given by its self-organizing nature Ba needs to be “energized” with its own

intention, direction, interest, or mission to be effective.

Leaders provide autonomy, creative chaos, redundancy, requisite variety, love, care, trust and commitment.

Prius creative chaos was generated by demanding goals. Uchiyamada demanded that his team question every norm on new car development.

Top management put Prius project team under great time pressure which caused extreme use of simultaneous engineering

Equal access to information at all levels was critical

ScrumMaster and management must “energize” ba through facilitating colocation, dynamic interaction, face to face communication, transparency, and



Local action forces self-organization

Individual self-organizes work Team self-organizes around goals Architecture self-organizes around working

code Product emerges through iterative adaptation Requires participative approach as opposed to

authoritative approach Flat organizational structure



First Scrum – Easel 1993

Abandoned GANTT charts Abandoned job titles Created ScrumMaster Created Product Owner Daily meetings to foster self-organization Shielded team from interference during Sprint Sprint planning, Sprint review, demo,

retrospective Agnostic about engineering practices Used all XP engineering practices



Scrum Influence on eXtreme Programming

From: Kent Beck To: Jeff Sutherland <jsutherland>Reply: [email protected]: Mon, 15 May 1995 18:01:15 -0400 (EDT)Subj: HBR paper______________________________________________

_______ Is there a good place to get reprints of the SCRUM paper

from HBR? I've written patterns for something very similar and I want to make sure I steal as many ideas as possible.

Kent



Scrum and XP

The first hyperproductive Scrum used all the XP engineering practices.The highest performance large project ever documented uses Scrum and XP together.You cannot get a Scrum with extreme velocity without XP.You cannot scale XP without Scrum.Example: SirsiDynix project



Distributed/Outsourcing Styles

Isolated Scrums

Distributed Scrum of Scrums

Totally Integrated Scrums



Outsourcing

Outsource $2M development Outsourcing costs - $1.6M

– Industry data show 20% cost savings on average Introduce Scrum locally

– 240% improvement Local Scrum costs – $0.83M Outsourcing is appropriate only when:

– You need expertise that is unavailable locally– You want to expand and contract development

staff without layoffs– You can linearly scale productivity with Agile

teams



SirsiDynix Distributed Scrum

Over a million lines of Java code




56 developers distributed across sites

SMDevDevDev

T LdDevDevDev

Catalogue Serials Circulation Search Reporting

StarSoftSt. Petersburg, Russia

SirsiDynixProvo, UtahDenver, COWaterloo, Canada

PO PO PO




Scrum daily meetings

7:45am Provo, Utah

St. Petersburg, Russia 17:45pm

Local Team Meeting

Scrum Team Meeting



SirsiDynix Distributed Scrum Common tools



SirsiDynix Distributed Scrum Uncommon performance

SCRUMSmall team

Waterfall SirsiDynixLarge team

PersonMonths

54 540 827

Lines of Java

51,000 58000 671,688

Function Points

959 900 12673

FP per dev/month

17.8 2.0 15.3



Linear Scalability of Large Scrum Projects

Project Size

Velocity Waterfall

Scrum Teams

•J. Sutherland, A. Viktorov, J. Blount, and N. Puntikov, "Distributed Scrum: Agile Project Management with Outsourced Development Teams," in HICSS'40, Hawaii International Conference on Software Systems, Big Island, Hawaii, 2007.•J. Sutherland, C. Jacobson, and K. Johnson, "Scrum and CMMI Level 5: A Magic Potion for Code Warriors!," in Agile 2007, Washington, D.C., 2007.



Xebia OneTeam

Since 2006, Xebia (Netherlands) started localized projects with half Dutch and half Indian team members.

After establishing localized hyperproductivity, they move the Indian members of the team to India and show increasing velocity with fully distributed teams.

After running XP engineering practices inside many distributed Scrum projects, Xebia has systematically productized a model similar to the SirsiDynix model for high performance, distributed, offshore teams with linear scalability and outstanding quality.



Dutch Velocity vs. Russian Velocity

1. M. Cohn, User Stories Applied for Agile Development. Addison-Wesley, 20042. J. Sutherland, A. Viktorov, J. Blount, and N. Puntikov, "Distributed Scrum: Agile Project Management with Outsourced Development Teams," in

HICSS'40, Hawaii International Conference on Software Systems, Big Island, Hawaii,3. J. Sutherland, G. Schoonheim, E. Rustenburg, M. Rijk. Fully Distributed Scrum: The Secret Sauce for Hyperproductive Outsourced Development

Teams. Agile 2008, Toronto, Aug 4-8 (submission, preliminary data)

SirsiDynix[2] Xebia[3]

Person Months 827 125

Lines of Java 671,688 100,000

Function Points 12673 1887

Function Points per Dev/Mon

15.3 15.1



Xebia Fully Distributed Scrum



Agile 2009TBD.com San Francisco - Xebia India

Quadrupled new user acquisition rate Quadrupled web site page views

!


Page

$Rev

isio

n:

$

The Systematic Scrum model

Value Velocity

READY

DONE

SPRINT

IMPEDIMENTS

Verify sprint delivery

Automated testContinuous IntegrationRemove impediments

DailyScrum

StoryCHK

FeatureCHK

Disciplines:

Clarify features

ReleasePlanning

Establish project environ-ment and initial PBL



Questions?


the roots of scrumathena.ecs.csus.edu/~buckley/csc231_files/sutherland_scrum_2010.… · chairman,...

Documents