why agile? why now?
DESCRIPTION
An overview of Agile and Scrum practices, and a comparison of 2 different, real-life Agile adoption experiences in a University setting. Presented at the Wharton Web Conference, July 2011TRANSCRIPT
Why Agile? Why Now?
Jorj Bauer <[email protected]> @bozoskeleton
Mike Toppa <[email protected]> @mtoppa
Wharton Web Conference – July 14, 2011Thursday, July 14, 2011
Overview
We’re going to talk through two perspectives on the adoption of Agile methodologies
What drove us
Why we approached Agile
How we approached adoption
Our perspective from where we are now
Thursday, July 14, 2011
First: who are we?
Thursday, July 14, 2011
Jorj Bauer
Manager of Engineering, Research and Development for Penn/ISC
Professional software developer since 1994
Co-owner of DejaVu Software, Inc.
2004-2008: Director of Networking, Penn/SEAS
2008-present: Manager of ERD in Penn/ISC
Thursday, July 14, 2011
Jorj: The Past
Most of my software development background has been ad-hoc, with little to no supporting infrastructure
Goals and requirements have been very fluid, with small teams
... until my current position
Thursday, July 14, 2011
Mike Toppa
Director, Web Applications for Penn/SOMIS
15 years of experience in web programming, IT project management, and functional management
Georgetown University; Stanford University; University of Pennsylvania
E*Trade; Ask Jeeves
2 failed start-ups (Finexa, Kai’s Candy Co.)
Thursday, July 14, 2011
Mike: The Past
None of the places I’ve worked followed any formal development methodology
Thursday, July 14, 2011
Mike: Becoming Director
My team was overwhelmed, but didn’t fully realize it
Like boiling frogs
Thursday, July 14, 2011
Mike: The Clients
Clients come from across the School’s administrative, academic, and research organizations
Hybrid funding model
Clients were working directly with developers
Usually no project managers
Reactive planning
Thursday, July 14, 2011
Mike: The Projects
A large number of projects
Little to no documentation
No one in a position to prioritize
Thursday, July 14, 2011
Mike: The Team
Talented
Independent
Customer service oriented
Focused on fast delivery
Thursday, July 14, 2011
Mike: The Dev Environment
No specific development procedures
An aging development framework
No automated testing
Non-critical bugs and missed requirements common
Daily firefighting
Difficulty planning
Thursday, July 14, 2011
Jorj’s challenges
Thursday, July 14, 2011
Jorj: The New Department
In 2008, I became manager of 5 direct reports, in a department of ~30, full of dotted-lines and large projects
Part of a reorganization of the department
Two large projects-in-progress with campus-wide scope were instantly “mine”
Thursday, July 14, 2011
Jorj: The Projects
Rapid shifting of priorities
Large, with no distinct requirements
Disconnect between technical requirements for a good service, level of effort to learn new services, and management expectations of level of effort required
Everything is behind schedule
A lot of fire-fighting related to the reorg as well as project management culture in the department
Thursday, July 14, 2011
Jorj: The Team
Very talented and conscientious
Not a lot of project focus; rapid shifting of priorities
A lot of unrest due to uncertainty and perceived political stupidity
Thursday, July 14, 2011
Jorj: The Dev Environment
Everyone For Themselves
Central RCS and CVS repositories, so some people chose to set up their own SVN repos
No specific development procedures
No automated testing
Projects begin and take priority based on political demand
Thursday, July 14, 2011
Jorj: The Biggest Challenge
“Project Management” happened between developers and their direct management – no higher, and no wider
The culture of “how project management works” was dysfunctional
Thursday, July 14, 2011
Prioritizing Challenges
Both of us approached our situations similarly:
1.Get a handle on existing and upcoming projects
2. Improve predictability
3. Improve quality
Thursday, July 14, 2011
Three Approaches...
In software development, you’re looking at one of these three general approaches:
Ad-hoc (informal)
Waterfall (anticipation)
Agile (adaptation)
Thursday, July 14, 2011
Waterfall
Thursday, July 14, 2011
Waterfall: why so pervasive?
“Nobody ever got fired for buying IBM”
Many people understand Waterfall
It seems sensible to wait for one thing to be complete before working on the next
The customer gets a whole solution, all at once
Thursday, July 14, 2011
Waterfall: low success rates
http://www.ambysoft.com/surveys/Thursday, July 14, 2011
Waterfall: why does it often fail?
Customers never know what they want up front; requirements always change
Customers and developers speak two different languages
Progress reports are often wrong, or set up a false expectation
If I’m 100% done the first task of 4... am I 25% done now?
Thursday, July 14, 2011
Ad-hoc
Home grown workflows, if any
Prioritization is arbitrary or politicized
Often hard to do long term planning
Success dependent on personalities and circumstances
Thursday, July 14, 2011
Agile
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value.
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
©2001 authors of The Agile Manifestoagilemanifesto.org
Thursday, July 14, 2011
Agile is incremental and iterative
Graphic by Jeff Patton
Thursday, July 14, 2011
Agile: “inspect and adapt”
Thursday, July 14, 2011
The Agile Umbrella
“Agile” applies to a large swath of development methodologies, many of which overlap
Scrum; Kanban; Crystal; Extreme Programming (XP); Lean...
And their hybrids
Scrum-Ban; ADROIT; WetAgile/Agile Waterfall...
Disclaimer: not all of these may be “good”
Thursday, July 14, 2011
Scrum: overview
Graphic by Skip Angel
Thursday, July 14, 2011
Slice vertically,not horizontally
Thursday, July 14, 2011
Thursday, July 14, 2011
Mike: Why Agile? Why Scrum?
Maintain frequent interactions with clients
Provides quick feedback
Existing good relationships
But have them take more ownership of their business process
Thursday, July 14, 2011
Mike: Why Agile? Why Scrum?
Put structure around our work
Enable planning
Make commitments, and meet them
Reduce need for firefighting
Reduce risk
Have more than one person know a project
Thursday, July 14, 2011
Mike: Why Agile? Why Scrum?
Improve quality
Reduce misunderstandings
Reduce missed requirements
Have fewer bugs
Increase technical knowledge sharing
Thursday, July 14, 2011
Authority vs. Responsibility
Cartoon by Mike Lynch
Used with permission
Thursday, July 14, 2011
What makes a job enjoyable?
Autonomy
Reward for effort
Challenging/complex work
“Work that fulfills these three criteria is meaningful.”
– Malcolm Gladwell, “Outliers: The Story of Success”
Thursday, July 14, 2011
Scrum has 3 roles
Product owner
Authority and responsibility for “what”
Programming team
Authority and responsibility for “how”
Scrum Master
Authority over the Scrum process
Responsibility for driving continuous improvement
Thursday, July 14, 2011
Scrum role: Product Owner
Responsible for what the team will work on, but now how the work is done
Thursday, July 14, 2011
Scrum role: The Team
Takes collective responsibility for doing the work
Thursday, July 14, 2011
Scrum role: Scrum Master
A “servant-leader” for the team
Thursday, July 14, 2011
Agile Adoption Patterns
Top-down vs. Bottom-up
“All in” vs. incremental
Thursday, July 14, 2011
Mike: top-down
Upper management: supportive
Team: uncertain
They recognized certain problems
But learning and conforming to a process
feels like a loss of autonomy
and yet another thing that has to be done
Thursday, July 14, 2011
Mike: all-in
Team learned core scrum concepts
I explained the new process to clients
A small team did a pilot project first
Then the whole group switched
Thursday, July 14, 2011
Mike: initial transition
It did not go very well
Team not used to working together
I misunderstood the roles
I overemphasized doing only one project at a time
Too many “scrum-buts”
Thursday, July 14, 2011
Mike: a visualization
Cartoon by Esther Derby
Used with permission
Thursday, July 14, 2011
Mike: using scrum coaches
I brought in coaches
They provided good training
Led candid discussion of problems
Most long pre-dated my introduction of scrum
Thursday, July 14, 2011
Should you use a coach?
A skilled, external coach is often key for driving change
If you haven’t done it before, it’s surprisingly easy to introduce Agile the wrong way
You need at least enough management support to pay them
You need to make sure you’re bringing in someone good
Thursday, July 14, 2011
But it’s still difficult
“In my Scrum classes I warn attendees of what I call the Scrum Promise: If you adopt Scrum, there will be a day you come into the office nearly in tears over how hard the change can be. This is because Scrum doesn’t solve problems, it uncovers them and puts them in our face. Then, through hard work we address them.”
– Mike Cohn, Agile Trainer
Thursday, July 14, 2011
Jorj: Bottom-up, Incremental
Thursday, July 14, 2011
Jorj: Bottom-up, Incremental
Ad-hoc
The “Cloak of Confusion” surrounded project management; most staff couldn’t see the problems, just the symptoms
The staff perceived control in their current methodology; nobody wants to give up control
Thursday, July 14, 2011
Why Incremental?
Convincing staff to change their culture is hard
Resources completely overcommitted
It’s difficult to “sell” a change without having data on how it will work
It’s impossible to show results for a system without using it
Thursday, July 14, 2011
NES’ Transition
Phase 1: adopt formal project workflow
Built a formal Iterative Waterfall process within the existing framework
A lot of resistance and much more project planning but improved results for existing projects
Increased awareness around product ownership, sources of delay, and available resources
Warning: this will not make you new friends
Thursday, July 14, 2011
Incrementally Improving
“Iterative Waterfall” is single-looped: allowed the team to abandon the waterfall process at many points (causing the project manager lots of additional work)
Continue working on the process itself
Thursday, July 14, 2011
Showing Results
This slowed everything down
3-4x the project management work (for me)
Allowed us to find and address deep-seated problems in workflow
Yields an inventory of projects
Gave me the ability to say “No” to new work
Thursday, July 14, 2011
Stepping back a moment...
Thursday, July 14, 2011
Saying “No” Is Important
If you take on an infinite number of projects, then your staff are constantly task-swapping (unless you have infinite staff)
Context switching between two projects eats about 20% of a full-time worker’s schedule
Cutting back on the number of active projects is key to getting back lost productivity from a bottlenecked staff
Thursday, July 14, 2011
Danger Signs of Meetings
In the meeting but doing other work while present
Being unavailable, or arriving late, for regularly scheduled
meetings
Suggesting radical departures from the current plan, at a stage that’s too
late
Never speaking in a meeting you attend regularly
Thursday, July 14, 2011
Reduce Meeting Complexity
Agile methodologies adopt more frequent meetings (generally daily). Staff specifically talk about:
What I did yesterday
What I’m doing today
What I’m waiting for
This is a more effective use of tech staff time
(But meetings may be a band-aid for other problems)
Thursday, July 14, 2011
Time/Project Management
Brooks’ Law
adding manpower to a late project will make it later
Eliyahu Goldratt’s Theory of Constraints
Identify the bottlenecks and use them to predict and control the workflow
Thursday, July 14, 2011
Thursday, July 14, 2011
Thursday, July 14, 2011
Answer: both have to be done as soon as possible.
Better answer: look at the costs, benefits, and staff resources for both and be able to shelve one
NES’ Turning Point
Two services: one brings in 2/3 of your department’s budget, and one brings in almost nothing. How do you prioritize the work?
Answer: both have to be done as soon as possible.
Thursday, July 14, 2011
No Looking Back
From here, big gains were made fairly quickly on teams that adopted these Agile practices:
Daily stand-ups; reduced queue sizes; more direct interaction with people “outside” the team
Better results on a shorter timeline within 4 months
Managers discussing and inspecting the process itself and its results, and are improving upon it
Not Done Yet!
Thursday, July 14, 2011
Mike: taking inventory
Thursday, July 14, 2011
Mike: taking inventory
9 developers, 2 product owners, and me supporting
22 clients with
124 applications
3 designers and 1 product owner supporting
about 200 static content web sites
Thursday, July 14, 2011
Mike: our maintenance curve
Thursday, July 14, 2011
Mike: project commitments for first 6 months... oh crap
Thursday, July 14, 2011
Mike: the first 6 months
Working through our over-commitment
While learning a new way to work
The continuous 2 week delivery cycle highlighted process weaknesses
Staff changes
Thursday, July 14, 2011
Planning and estimating
How did we come up with that chart?
Agile requirements gathering
Agile estimating techniques
Thursday, July 14, 2011
Requirements gathering
Breaking down a project into feature areas
Epics
Breaking down feature areas into individual features
User stories
How do you know when you’re done?
Conditions of satisfaction
Thursday, July 14, 2011
Estimating: story points and planning poker
Photo by Kelly Hirano
Used with permission
Thursday, July 14, 2011
Velocity enables scheduling
Thursday, July 14, 2011
Mike: one year later
We can plan and estimate better
Our programmers work together
Our next challenges
A project prioritization and intake process
Improving our technical practices
Thursday, July 14, 2011
Manifesto for Half-Arsed Agile Software Development
We have heard about new ways of developing software bypaying consultants and reading Gartner reports. Through
this we have been told to value:
Individuals and interactions over processes and toolsand we have mandatory processes and tools to control how those
individuals (we prefer the term ‘resources’) interact
Working software over comprehensive documentationas long as that software is comprehensively documented
Customer collaboration over contract negotiationwithin the boundaries of strict contracts, of course, and subject to rigorous change control
Responding to change over following a planprovided a detailed plan is in place to respond to the change, and it is followed precisely
That is, while the items on the left sound nicein theory, we’re an enterprise company, and there’s
no way we’re letting go of the items on the right.http://www.halfarsedagilemanifesto.org/
Thursday, July 14, 2011
Technical practices:ridiculously brief overview
“Specification by Example” by Gojko Adzic
“Clean Code” and “Agile Software Development, Principles, Patterns, and Practices” by Bob Martin
“Refactoring” by Martin Fowler
“Agile Database Techniques” by Scott Ambler
“Growing Object Oriented Software, Guided by Tests” by Steve Freeman and Nat Pryce
Thursday, July 14, 2011
Agile is relevant beyond software development
“Angry Dinosaurs: Accelerating Change and Institutional Incompetence”
Cory Ondrejka, Wharton Web Conference, 2010
http://bit.ly/bY4E15
“Let it Roll: Why more companies are abandoning budgets in favor of rolling forecasts”
CFO Magazine, May 1, 2011 http://bit.ly/k94Sfc
Thursday, July 14, 2011
Other References
Eliyahu M. Goldratt, “The Goal” (Theory of Constraints)
Fred Brooks, “The Mythical Man-Month”
Gerald Weinberg, “Quality Software Management: Systems Thinking”
“Convincing Management That Context Switching Is a Bad Idea”, Johanna Rothman http://www.jrothman.com/Papers/context-switching.html
“In Praise of Bad Programmers” http://corvusintl.com/
Thursday, July 14, 2011
Any Questions?
Jorj Bauer <[email protected]> @bozoskeleton
Mike Toppa <[email protected]> @mtoppa
Wharton Web Conference – July 14, 2011Thursday, July 14, 2011