developing metrics for agile projects compatible with cmmi graham collins, ucl...
TRANSCRIPT
Developing Metrics for agile projects
compatible with CMMI
Graham Collins, UCL
Research supported by Deutsche Bank
IntroductionIntroduction
UCL’s research and projects The problems and requests EV (Earned Value) Agile practices Combining EV and agile is it possible? Developing suitable metrics compatible with
CMMI What works, and what reduces the overheads Is this approach likely to lead to CMMI Level 5? Key changes to projects
Background - RequestsBackground - Requests
We would like to use earned value ‘We would like to predict the outcome of project end date,
cost and value’ As a project manager I cannot be there all the time What is the best way to measure progress with earned
value? ‘As developers we would like to experiment with some
agile approaches’ Your team will be working on other projects as well We are hoping to achieve metrics suitable for CMMI level
3 and higher…
Kamikaze
Suicide
Mission Impossible
Ugly
low high
high
low
Chance of success
Happiness
The Death March Project Style Quadrant
Edward Yourdon, Death March:The complete Software Developer’s guide to surviving ‘Mission Impossible’ projects, 1997 Prentice Hall
The Death March Project Style QuadrantThe Death March Project Style Quadrant
Earned Value - DefinitionEarned Value - Definition
‘The value of the useful work done at any given point in a project. The value of completed work expressed in terms of the budget assigned to that work. A measure of project progress. Note: The budget may be expressed in cost or labour hours’
APM (Association of Project Management) BoK 2006
Earned Value – Graphical RepresentationEarned Value – Graphical Representation
Earned Value
Planned
ActualCost or value
Time
Planned (BCWS = Budgeted Cost of Work Scheduled)Actual (ACWP = Actual Cost of Work Performed)Earned Value (BCWP = Budgeted Cost of Work Performed)CPI = Cost Performance Index = BCWP/ACWP (or Earned/Actual)SPI = Schedule Performance Index = BCWP/BCWS (or Earned/Planned) note this is SPIs i.e. schedule.
Use of VarianceUse of Variance
Variance (Monthly)
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
1.10
1.20
1 2 3 4 5 6
Months
Val
ue cpi
spi
spic is used in this situation
Earned Value compared to Agile Process PlanningEarned Value compared to Agile Process Planning
Based on predictive planning
Estimates effort, cost and completion date
End-to-end value tracking
Adaptive planning.
Iteration to iteration tracking
Predication of the next iterations effort
Schedule most of the activities
Adaptation to unpredictable events is problematic. Changes may require the planned to be revised or baselined
Near the beginning, it is not always possible to schedule.
Time based iterations allow initial estimate of duration which can be revised through the adaptive driven build-feedback cycle
Estimates based on past performance Estimates are based on progress being made (velocity)
Change rates often low Unpredictable change the norm
Small variations in early measurements of cost and time at the start the project give wide variation in forward predications
Unknown team development rates
Some organisations do not chart progress for an initial period
Progress is tracked immediately
Earned value well established Prioritization of the value of user stories
No earned value approaches in methods
Earned Value Agile Development
CMMI- Process AreasCMMI- Process Areas
‘Project planning is a necessity for success,Yet it still ranks on most surveys within the top three or four problem areas leading to failure.’
Tony Ciorra, Planner’s Progress, Project, APM May 2006
Category Process Area
Project Management Project Planning
Project Monitoring and Control
Quantitative Project Management
Support Process and Product Quality Assurance
Causal Analysis & Resolution
CMMI Comparative AdvantagesCMMI Comparative Advantages
Grants explicit freedom to select the order of improvement that best meets the organization’s business objectives and mitigates the organisation’s areas of risk
Enables organisations to have a predefined path
Enables increased visibility of the capability achieved in each individual process area
Focuses on a set of processes that provide an organization with a specific capability that is characterized by each maturity level
Provides a capability-level rating that is used primarily for improvement in an organisation and is rarely communicated externally
Provides a maturity-level rating that is often used in internal management communication, statements external to the organization, and during acquisitions as a means to qualify bidders
Allows improvements of different processes to be performed at different rates
Summarizes process-improvement results in a simple form – a single maturity-level number
Reflects a newer approach that does not yet have the data to demonstrate its ties to return on investment
Builds on a relatively long history of use that includes case studies and data that demonstrate proved return on investment
Continuous Representation Staged Representation
Agile ManifestoAgile Manifesto
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more
Several agile projects have achieved CMMI level 3, example David Anderson, Stretching Agile to fit CMMI Level 3, Agile Conference 2005
The Agile Principles www.agilealliance.comThe Agile Principles www.agilealliance.com
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software
Agile processes promote sustainable development
Welcome changing requirements, even late in development. Agile processes harness change for the customer competitive advantage
The sponsors, developer, and users should be able to maintain a constant pace indefinitely
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale
Continuous attention to technical excellence and good design enhances agility
Business people and developers must work together daily throughout the project
Simplicity - the art of maximizing the amount of work done – is essential
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The best architectures, requirements, and designs emerge from self-organizing teams
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation
At regular intervals the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly
Working software is the primary measure of progress
Iterative Development (Bittner-Spence)Iterative Development (Bittner-Spence)
1. Agree with the team the objectives for the iteration, including evaluation criteria, timescales, and constraints
2. Agree on a plan for how the team will achieve the objectives
3. Execute the plan
4. Assess the achievements of the team against the initial set of objectives and evaluation criteria
5. Assess the impact of the iteration’s results on the project as a whole
6. Start the next iteration.
What is iterative development? Part 3: The management perspective 15 May 2005www-128.ibm.com/developerworks/rational/library/may05/bittner-spence/index.html
Fundamental shift in measurementFundamental shift in measurement
Progress ( % complete measured in scenarios
100%
0%
Iteration 1 2 3 4
coded tested Tested & Passed
Developer PerspectiveDeveloper Perspective
Developers are less interested in the business value, benefits realization and return on investment
They work on a small number of requirements or change requests from their list of outstanding work
They anticipate a decreasing number of requirements and change requests as the product is developed
Outstanding requirements and change requests is termed the product backlog
The developer will therefore be aware of progress via work completed, product backlog and new work allocated.
User Satisfaction driving DevelopmentUser Satisfaction driving Development
User satisfaction
Iteration
ReleaseUser satisfaction
Release planning
Development IncrementIteration planning
Iteration Planning (Goal identification, story selection, tasks, estimation, team commitment)
To develop members’ capabilities; to build and exchange knowledge
Passion, commitment, and identification with the group’s expertise
To accomplish a specified task
The project’s milestones and goals
Adapted from: Communities of Practice: The organizational Frontier, Etienne C. Wenger and William M. Snyder, Harvard Business Review p139-145 Jan-Feb 2000
What is the purpose? What holds it together?
Community of practice
Project team
Project teams need to adopt some attributesProject teams need to adopt some attributes
Rate of work - velocityRate of work - velocity
Velocity
0
5
10
15
20
25
30
35
40
45
50
06/01
/06
13/01
/06
20/01
/06
27/01
/06
03/02
/06
10/02
/06
17/02
/06
24/02
/06
Sto
ry P
oin
ts
Velocity, gives an indication of the average rate of work and also a comparison of planned against delivered, each iteration
Often high variationOften high variation
20
25
30
35
40
45
50
06/0
1/20
06
13/0
1/20
06
20/0
1/20
06
27/0
1/20
06
03/0
2/20
06
10/0
2/20
06
17/0
2/20
06
24/0
2/20
06
Velocity from previous chart, showing UCL, X-bar average, and LCL
UCL upper control line
LCL lower control line
X-bar average
‘Under control’‘Under control’
35
37
39
41
43
45
47
49
51
53
55
06/0
1/06
13/0
1/06
20/0
1/06
27/0
1/06
03/0
2/06
10/0
2/06
17/0
2/06
24/0
2/06
Story points in team BNote the stable process with no values outside of UCL and LCL
LCL
Velocity measures of work rate are useful in that estimates of the next iteration can be planned in a rolling process, with plans becoming more accurate. The use of Process Control and sigma variation is supportive in this aim. This can be combined with automated colour coding
Types of meanTypes of mean
Velocity
0
5
10
15
20
25
30
1 2 3 4 5 6
iterations
sto
ry p
oin
ts
X -Bar averagecentreline (CL)
weighted average
UCL (upper control limit)
Storypoints
count Weightedpoints
0 1 0
5 2 10
18 3 54
14 4 56
24 5 120
27 6 162
14.66667 3.5 19.14286
0
5
10
15
20
25
30
35
40
45
50
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Use of MultipliersUse of Multipliers
Iterations Completed
Low Multiplier High Multiplier
1 0.6 1.60
2 0.8 1.25
3 0.85 1.15
4 or more 0.90 1.10
Multipliers for estimating velocity based on number of iterations completed from Cohn 2006
Work RemainingWork Remaining
Story Points Remaing
0
50
100
150
200
250
300
350
06/0
1/20
06
13/0
1/20
06
20/0
1/20
06
27/0
1/20
06
03/0
2/20
06
10/0
2/20
06
17/0
2/20
06
24/0
2/20
06
Sto
ry P
oin
ts
This ‘burn-down’ chart showing can be combined with an inventory line, showing the cumulative total of points
With the appropriate metrics we can improveWith the appropriate metrics we can improve
Acceptance Test Data
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12
Iteration
AT
s
Acceptance Tests
Failing ATs
Passing ATs
Additional MeasuresAdditional Measures
At different levels, project, release, iteration. Velocity and stories delivered (as a ‘burndown’ i.e. remaining), as well as on a daily basis, showing tasks agreed on and those remaining
Graphs for stories can also display inventory, to show how much additional work has been added. Iteration cumulative flow can also be displayed as ‘burn-up’ charts displaying values for inventory and complete
Issue logs should show, active, resolved, closed and blocked
WIP (Work-In-Progress) inventory charts combined with the issue log provide capability to remove special cause variation.
Earned ValueEarned Value
EV can be applied to these figures, although measuring this, can be complex, especially if more stories are added as the work progresses (e.g. agile projects).
This however would be useful if the figures have to be shown to senior managers who are used to EV figures, or used in comparison to other projects where EV figures have been tracked
When additional story points are added to the planned work it does not necessarily affect the original plan in terms of EV (or business value). In light of business priorities the team may commit to a new set of stories and drop other stories already planned. Consequently the value of stories planned within a specific time frame can remain the same. How the planned (budget) is calculated, whether based on points or business value, needs to be decided in advance.
Business ValueBusiness Value
More importantly business value (or contribution) should be considered and evaluated
Often units of measure such as story points can be valued as 0.5 or 1.0 units
The key issue in agile project management is to continually assess with the client the most important work that should be done.
Story number
‘Business Value’
Story Points
‘Points earned’
Developer hours planned
EV (earned value)
1 3 10 10 100 100
2 2 8 8 60 60
3 2 4 4 60 60
4 1 2 0 20 0
total 8 24 22 240 220
Moving RangeMoving Range
Velocity
05
10152025
3035404550
06/0
1/20
06
13/0
1/20
06
20/0
1/20
06
27/0
1/20
06
03/0
2/20
06
10/0
2/20
06
17/0
2/20
06
24/0
2/20
06
weeks
sto
ry p
oin
ts
story points mR
mR values can make variation more visible
ConclusionsConclusions
EV can be combined with agile reporting The most useful approach to achieve process improvement is via
understanding of process control, the basis for CMMI EV is useful for reporting at a programme and project level, but even with
major scope changes, re-planning can give some useful insights Acceptance tests are the most useful approach for both methods and is
the basis for metrics outlined. Issue reporting and work-in-progress are also valuable
Changes in features, where story points are maintained, on time based iterations, may not significantly change the basis for EV reporting. Projects indicate that EV can remain a viable approach, dependent on the business value weighting approach used
The agile process, the value of time based iterations, and that the team are trying to achieve maximum business value, even though the final goal and plans may evolve during the process, needs to be explained to senior managers.
Key Project ChangesKey Project Changes
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
1. Peer reporting - valuing individuals and team, moving towards self-determining teams
2. Acceptance testing - working software
3. Ensuring Business Value – continual prioritisation, estimation and understanding there is a cost to development
4. Measuring progress over shorter time periods -meeting the needs of process improvement CMMI, velocity tracking in agile methods, and better EV planning