software management by numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf ·...
TRANSCRIPT
![Page 1: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/1.jpg)
Software Management by Numbers
Towards an Engineering Discipline
© SE-CURE AG 1
SE‐CURE AG (www.se‐cure.ch) Dr. Hans SassenburgT +41 33 733 4682M +41 79 231 6600E hsassenburg@se‐cure.ch
![Page 2: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/2.jpg)
Contents
1. Software Industry Today2. Empirical Information3. Key Performance Indicators4. Products and Services5. Case Study Examples6. Conclusions
2© SE-CURE AG
![Page 3: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/3.jpg)
Questions ...
• Did you ever go after the world‘s next model/standard or hype?– If yes, how (much) did it help you?
• Are you able today to tell what targeted, actual and historical values of your main KPIs are?– If not, would you be interested to know?
© SE-CURE AG 3
![Page 4: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/4.jpg)
Software Industry Today• The software industry still isn't doing a very good job at
managing by the numbers
”Oftentimes, software projects are managed by just three metrics: schedule/effort and critical defects
found during testing”
• This is a flatland view for a multi-dimensional terrain problem: "flying a plane using only a watch and a fuel gauge“
• Other metrics must be in the equation as well, representing what will be built (product scope/quality) and how it is built (process)
© SE-CURE AG 4
![Page 5: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/5.jpg)
Biking (1/3)
© SE-CURE AG 5
My home
Summary:
32.67 km2 h 18 min
Question:
Am I fit?
![Page 6: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/6.jpg)
Mountain Biking (2/3)
© SE-CURE AG 6
My home
![Page 7: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/7.jpg)
Mountain Biking (3/3)
© SE-CURE AG 7
Elevation: 14-16%
Total ascent: 1‘107 m
Calories burnt: 1‘697
![Page 8: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/8.jpg)
Problem Statement
”You can’t fix what you refuse to measure”
• When quantitative information is missing:– Management is like an emperor without
clothes: decisions are not based on facts– Improvement initiatives result in satisfying
standards/models instead of tangibly improving measured capability
8© SE-CURE AG
![Page 9: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/9.jpg)
What is Improvement?
• It is NOT about–Satisfying a model/standard– Implementing the next hype
• It is about becoming–More predictable (performance)–Faster (efficiency)–More effective (scope)–Better (quality)
© SE-CURE AG 9
![Page 10: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/10.jpg)
Being Successful
• Knowing the (performance) capability of your organization through the collection and analysis of historical data
• Making credible commitments in terms of what will be delivered when against what cost
• Managing development once it starts; this requires earned value management, activity distribution management and effective scope and quality management
• Analyzing the impact of new initiatives by assessing how capability is affected in which areas; this prevents organizations from chasing hypes
© SE-CURE AG 10
![Page 11: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/11.jpg)
2. Empirical Information
• Empirical Models– Putnam SLIM– COCOMO II
• Empirical Laws– Minimal development time!
• Benchmarking Data– Guru: Capers Jones
• Observation– Availability hardly known, seldom used ...
© SE-CURE AG 11
![Page 12: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/12.jpg)
Empirical Models
© SE-CURE AG 12
Empirical Models(Putnam SLIM, COCOMO II)
+Benchmarking Data
features
schedule
effort
capability
characteristics (product, platform, personnel, project)
size
![Page 13: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/13.jpg)
Empirical Laws
© SE-CURE AG 13
effo
rt
scheduleTnTs Te
-25% +100%
> 100%
-50%
Ee
Es
En
Imp
oss
ible
R
eg
ion
!
Legend:
Ts , Es : fastest scheduleTn , En : nominal schedule/effortTe , Ee : lowest effort
Impractical Region
![Page 14: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/14.jpg)
Benchmark (Jones 2008)
© SE-CURE AG 14
![Page 15: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/15.jpg)
Cost Categories
© SE-CURE AG 15
Essential Non-Essential
= Quality Related
Value-Added Non-Value-Added
Core PreventionSupport Appraisal, rework
![Page 16: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/16.jpg)
Benchmark - Process Efficiency (Jones/Sassenburg)
© SE-CURE AG 16
![Page 17: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/17.jpg)
3. Key Performance Indicators
© SE-CURE AG 17
”Getting too little or too much data is easy, identifying and extracting the relevant data and converting it to
meaningful information for everyone is the challenge”
![Page 18: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/18.jpg)
KPIs versus Metrics
• KPIs (= meaningful information) are meant to gauge progress toward or benchmark against vital, strategic objectives usually defined by upper management
• Metrics (= data) at a lower detail level are measured attributes of a process or a product
• The goal of KPIs is to foster greater visibility and faster reaction to opportunities and threats, hereby enabling informed decision-making
© SE-CURE AG 18
![Page 19: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/19.jpg)
KPI Selection Criteria
• Must support project management in analysing, planning and monitoring projects
• Must inform management where a project stands and in what direction it is heading
• Must support business units in measuring their capability improvement over time
• Must support organisations in comparing/benchmarking business units
© SE-CURE AG 19
![Page 20: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/20.jpg)
Questions to Answer• Project Performance
– What is the prediction of the performance of the project?
• Process Efficiency– How efficient is the development process?
• Product Scope– How large and stable is the scope of the
planned effort in terms of features and size?
• Product Quality– What is the expected quality of the resulting
product (reliability and maintainability)?© SE-CURE AG 20
![Page 21: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/21.jpg)
Derived KPI Categories
• Project Performance (= how predictable?)– Cost, schedule, staffing rate, productivity
• Process Efficiency (= how fast?)– Effort distribution (Cost of Quality model)
• Product Scope (= how much?)– Features, deferral rate, size, re-use
• Product quality (= how well?)– Complexity, test coverage, removal efficiency,
defect density
© SE-CURE AG 21
![Page 22: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/22.jpg)
Best Practice KPI Set
© SE-CURE AG 22
Project Performance Product ScopeSchedule Feature size
Budget Deferred
Staffing rate Size
Productivity Re-use
Process Efficiency Product QualityCore Complexity
Support Test coverage
Prevention Defect density
Appraisal/rework Removal efficiency
![Page 23: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/23.jpg)
Typical Maturization Effects
© SE-CURE AG 23
Project Performance Product ScopeSchedule Feature size
Budget Deferred
Staffing rate Size
Productivity Re-use
Process Efficiency Product QualityCore Complexity
Support Test coverage
Prevention Defect density
Appraisal/rework Removal efficiency
![Page 24: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/24.jpg)
Key Points to Remember
1. Assessing real software performance capability requires a multi-dimensional view
2. Compliance to process models and/or standards is no guarantee for high performance capability, although it may help (pragmatism versus bureacracy)
3. The set of Key Performance Indicators allows measuring real performance capability improvements over time and benchmarking
© SE-CURE AG 24
![Page 25: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/25.jpg)
4. Products and Services
• See www.se-cure.ch/Products.html
• See www.se-cure.ch/Services.html
© SE-CURE AG 25
![Page 26: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/26.jpg)
Project Reporting Cockpit
© SE-CURE AG 26
The
Project
Reporting
Cockpit
will
enable
you
to
define
and
monitor
a
set
of
Key
Performance Indicators (KPIs) for your project. Historical, target
and
actual
values
are
displayed
in
overviews
and
charts,
from
which
information
can
be
consumed
at
a
glance.
![Page 27: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/27.jpg)
Project Support Toolkit
© SE-CURE AG 27
The
Project
Support
Toolkit
provides
powerful
instruments
to
project
managers
and
measurements
specialists
to
plan,
monitor
and
control
their
projects
on
a
more
detailed level, using the same set of KPIs as in the Project Reporting Cockpit.
![Page 28: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/28.jpg)
1-Day Workshops
© SE-CURE AG 28
![Page 29: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/29.jpg)
5. Case Study Examples
• Context–2 B2B manufacturers (Case A, case B)–Product functionality highly determined
by software–Case A: applications are safety-critical–Case B: security of information is crucial
• Current performance–CMMI L2/3 (internal assessments)
© SE-CURE AG 29
![Page 30: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/30.jpg)
Issues
• No benchmarking data for deferral rate and re-use level– Context specific and lack of data– Finding: deferral rate unknown
• No benchmarking data for test coverage and complexity– Market ...!– Finding: test coverage unknown, complexity high
• Feature size (FP) = Product size (KLOC)?
© SE-CURE AG 30
![Page 31: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/31.jpg)
© SE-CURE AG 31
![Page 32: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/32.jpg)
© SE-CURE AG 32
![Page 33: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/33.jpg)
© SE-CURE AG 33
![Page 34: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/34.jpg)
© SE-CURE AG 34
![Page 35: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/35.jpg)
© SE-CURE AG 35
![Page 36: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/36.jpg)
Defect Analysis (Example)
© SE-CURE AG 36= 2.5*5 + 7.5*25 + 15*20 + 35*35 + 75*15
= (1*30 + 3*10 + 2*10)/(1 + 3 + 2)
Total number of injected defects = 5‘000
= (5‘000-375)/5‘000 = 375/125
![Page 37: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/37.jpg)
© SE-CURE AG 37
![Page 38: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/38.jpg)
© SE-CURE AG 38
![Page 39: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/39.jpg)
Weakest Areas
1. Inefficient process– Sum of appraisal and internal failure costs is
high (> 45%)– Post-release distribution would even be worse!
2. Low code quality– Code complexity is high (> 15)
These are the primary causes
© SE-CURE AG 39
![Page 40: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/40.jpg)
Measure 1
• Increase prevention costs (training, reviews, inspections)
• Expected effects: – Will increase removal efficiency and lower
defect density – Will decrease appraisal costs (testing) and
rework costs (defect removal)– Will lead to faster schedule, lower effort and
higher productivity
© SE-CURE AG 40
![Page 41: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/41.jpg)
Measure 2
• Improve code quality
• Expected effects:– Will increase test coverage– Will (again) increase removal efficiency and
lower defect density – Will (again) decrease appraisal costs (testing)
and rework costs (defect removal)– Will (gain) lead to faster schedule, lower
effort and higher productivity
© SE-CURE AG 41
![Page 42: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/42.jpg)
6. Conclusions
• Process capability is not necessarily a good measure for high performance
• This does not mean that process improvement is bad
• To make it worth the investment, one should improve along a set of best practice KPIs instead of meeting the model/standard requirements only
© SE-CURE AG 42
![Page 43: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/43.jpg)
KPIs as Improvement Drivers
• Once management starts actively using such KPIs, projects are forced to bring and keep not only their measurement process in place, but many other processes as well …–Requirements Management–Project Planning–Project Monitoring and Control–…
© SE-CURE AG 43
![Page 44: Software Management by Numbers - spiderst-spider.nl/wiki/images/4/42/20091006sassenburg.pdf · Software Management by Numbers ... schedule/effort and critical defects found during](https://reader034.vdocument.in/reader034/viewer/2022051602/5ae9598a7f8b9ad73f8bbc66/html5/thumbnails/44.jpg)
Another Capability Model ...
© SE-CURE AG 44