future of software analysis & measurement_cast
TRANSCRIPT
The Future of Software Analysis & Measurement
The Future of
Software Analysis & Measurement
Daniel D. Galorath
40+ years Software & Software
Management Experience
Founder & CEO Galorath Incorporated,
SEER By Galorath
1980 MBA Management,
California State Universities
1984 Began working software
estimation, planning & control
2009 Society of Cost Estimation and
Analysis (SCEA) Lifetime Achievement
award
2001 ISPA Freiman Award,
lifetime achievement award
2006 Book: Software Sizing, Estimation,
and Risk Management
SEER parametrically estimates project
cost, effort, duration, cost & risk
Answer Fundamental Management
Questions
How much effort is required
to complete an activity?
How much calendar time is
needed to complete an activity?
What is the total cost of an activity?
How reliable will it be?
What can we do better?
What If?
www.galorath.com
SEER® Empowering Project Estimation, Planning & Control Since 1988
David Herron
Business Development Manager and VP of
Knowledge Solution Services for David
Consulting Group
Consulting and coaching services for a variety
of IT organizations throughout the US and
Canada
Acknowledged authority in the areas of
performance measurement, process
improvement and organizational change
management
Noted author and lecturer; co-authored several
books on topics relating to IT performance
measurement and Function Point Analysis
David Consulting Group
DCG is an international IT process improvement and measurement
company managing value-driven engagements with companies and
government agencies around the world
Software Process Improvement
Utilizing CMMI, Six Sigma, Lean and Agile methods
Software Sizing
Using IFPUG Function point Counting and alternative sizing techniques
Software Measurement
Providing roadmap planning, estimation models, performance benchmarks
and outsourcing SLA support
IT Performance Improvement
Improving IT operations through ITIL and IT Governance
Dr. Bill Curtis
Senior Vice President and Chief Scientist
Industry luminary responsible for influencing
CAST’s scientific and strategic direction
Best known for leading development of the
Capability Maturity Model (CMM) which has become
the global standard for evaluating the capability of
software development organizations.
Prior to joining CAST,
Co-Founder of TeraQuest, the global leader in CMM-
based services,
Directed the Software Process Program at the Software
Engineering Institute (SEI) at Carnegie Mellon University.
CAST Application Intelligence Platform
Most enterprises measure everything but the product delivered to the business
CAST Application Intelligence Platform (AIP) measures the product itself
Robustness Performance Security Changeability Transferability Size
Planning Estimation Scheduling Time Tracking Cost Tracking
Product Process
Time & Duration
Effort & Budget Function &
Scope
Quality
&Size
Requirements Earned Value User Acceptance Usability
CAST AIP
QUESTIONS
What are the single most important business drivers that lead an organization
to embark on a SAM initiative?
How does software analysis and measurement relate to business outcomes?
What organizational behaviors are most favorable for a successful SAM
program?
What is the relationship between SAM and IT governance?
How can measurement and analysis interact with estimating to provide
improved visibility and information to make the best management decisions
regarding software?
How can anyone provide a viable estimate software development, software
maintenance, or continuing innovation work and what use it is?
How can we determine which software projects with have the highest ROI?
What are the single most important
business drivers that lead an organization
to embark on a SAM initiative?
HBR Article explains this Phenomenon:
Humans seem hardwired to be optimists
We routinely exaggerate benefits and discount costs
Delusions of Success: How Optimism Undermines Executives' Decisions
(Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)
11
Solution: Temper with “outside view”
Software Analysis & Measurement can temper
Viable estimation can temper
Don’t remove optimism, but balance optimism & realism
Human Nature Optimism in Software
Cutter Consortium Software Project Survey:
62% overran original schedule by more than 50%
64% more than 50% over budget
70% had critical product quality defects after release
Standish Group CHAOS Report
46% challenged
19% failed
35% successful
~$875 billion spent on IT
~$300 billion spent on IT projects
~$57 billion wasted annually
Ever increasing “Technical Debt”
ROI of Applied Software Analysis & Measurement is HUGE
IT Failures are Pervasive: And Even Successful Projects may not be
Manual Estimates
Human reasons for error (Software Analysis & Measurement can
help)
Desire for “credibility” motivates overestimate behavior (80%
probability?)
So must spend all the time to be “reliable”
Better approach force 50% probability & have “buffer” for overruns
Technical pride causes underestimates
Buy-in causes underestimates
13
Important Business Drivers for Structural Measurement
1 Poor software
from suppliers
2 Embarrassing
disasters
3 High cost of
maintenance
How does software analysis and
measurement relate to business outcomes?
Goal Question Metric Approach
Combine goal-orientation bottoms up, decision-support & other
operational management techniques
Going to weather.com and deciding to bring an umbrella is decision support
Goal
Question
Development
Contractors
Metric
Organizational Goal
Question
Development
Organizations
Metric
How Does Measurement Relate to Business Outcomes
TRANSFERABILITY
allows new teams to quickly
begin working with an application
CHANGEABILITY
makes an application easier and quicker to modify
ROBUSTNESS
improves application stability & reduces injecting new defects
PERFORMANCE
Reduces degraded response
times and increases scalability
SECURITY
affects an application’s ability to
prevent unauthorized intrusions
Optimize work
productivity
Maximize
customer loyalty
Maximize business
agility
Minimize business
risks
Minimize IT costs
Business
Value
Application Health
Factors
Tactical Objectives
Reduce learning curves
Improve software readability
Ease team handoffs
Reduce vendor lock-in
Reduce modification effort
Reduce cost of ownership
Accelerate new function delivery
Reduce application rework
Maximize application availability
Minimize liquidated damages
Minimize degraded service
Maximize application scalability
Reduce injected defects
Reduce application mistakes
Maximize speed of response
Maximize information retrieval
Maximize information protection
Maximize customer confidence
Maximize standards compliance
Minimize unwanted breaches
Cost
Risk
What organizational behaviors are most
favorable for a successful SAM program?
Key Organizational Behaviors
Trust but verify
Viable measurement and analysis
Answering the right questions
Having actions based on measurement & analysis
Measure what can make a difference
Measure to the audience needs (Goal / Question / Metrics)
Generating the Business Value Side of the Equation (Benefits)
The business owns benefit calculations
IT should participate
Exception: projects solely improving internal IT
Beware of subjectivity translating soft benefits
Use probability and risk
Increased Revenue
Increased Profit
Reduction in cost (people, processes,
cash out) Internal benefits
Intangible Benefits
Conditions that Favor Structural Measurement
Low
maturity
Fed up
managers
Mature Disciplined
processes
High
maturity
Quantitative
management
What is the relationship
between SAM and IT governance?
(Source: Fraunhofer)
Financial
Customer Business Process
Innovation / Growth
Vision & Strategy
Goals
Data Collection
Measurement Metrics
Answers Questions
Goal Attainment
Software Development & Maintenance
6σ
GQM
PSM
CoBIT…
Business/ Organization
Level
IT Governance
IT Governance
Customer Focus
The Gap Between IT & Management Needs
How Structural Measurement is Tied to IT Governance
The Evolution of Governance
Govern the
Department
Budget headcount
Govern the
Project
Schedule defects
Govern the
Product
Structural quality
How can measurement and analysis interact
with estimating to provide improved visibility
and information to make the best management
decisions regarding software?
What a Parametric Model can Tell You
26
What is likely to happen
Feel lucky?
Firm Fixed Price?
Understand the risk before you commit!
SAM Feeds Estimation and Provides Insight & Choice
Cost, schedule, risk
Should we update or
redevelop?
What is the risk of continuing
with software with increasing
“technical debt”?
SEER-SEM Software
Estimation Model
Size
Complexity
Maintainability/
Defect Potential
People, Process, Technology
SAM Outputs
Effort
Schedule
Risk
Reliability
/Defects
Historical Results
(SAM Outputs)
Constraints
Maintenance Effort
Required
Looking at Maintenance and Technical Debt
Software Estimation Basic Model & Associated Metrics
Start Finish
Effort K
Size St ReuseDIT
Size Se (work units)
Size (Effective Se & Total St)
Defects Count (Qi Qr)
Calendar Time
On-going Iterations of Effort (ACWP or Spent)
Progress (BCWP or Earned Value) Defects (Qi Qr )
Growth (Sg)
Software Development
Process
Staffa &
Constraints
Effective Technology Cte
People Process
Technology
Stakeholder Requirements
Delivered Software
Effective complexity D
Maintenance/ Block Change Development
Process
Development Legacy,
Maintenance Specifics & Constraints
and/or
Block Changes As Redevelopment
Fundamental Metrics
Size
AKA Volume, Mass
Units: Source Lines of Code
(SLOC); Function Points (FP)
Use Cases
New versus rework
COTS & Packages
Effective Technology
AKA Productivity Potential,
Efficiency
Units: none
Time
AKA Duration, Schedule
Units: Calendar Months,
Calendar Weeks
Effort
AKA Work, Labor
Units: Staff Months, Staff Hours
Cost
AKA Budget, Money
Units: $, other currencies
Staffing
AKA Manpower Loading
Units: FTE People
Defects
AKA Reliability, Quality
Units: Defect Count
Other Key Metrics Help Track Project Performance
Track defect discovery and removal rates
against expected rates
Increased defect reporting rate
shows a worsening trend
Heath and Status Indicator shows status and trends from
the previous snapshot
Thresholds are user definable
When & Why to Collect Data
When to Collect
1. Up-front: when scoping new
project data from completed
projects
2. In-Process: During
development for management,
to identify issues and progress
3. Post Mortem: Upon
development completion to
improve corporate history
repository
4. In Service: During
maintenance to continue
learning & improving
Why People Don’t Want to
Provide Data
They could be proven wrong
It could be used against them
Data often doesn’t exist
Even if processes dictate data
requirements
If it exists it may not be clean
It may give away corporate
productivity & bid strategy
How can anyone provide a viable estimate
software development, software maintenance, or
continuing innovation work and what use it is?
10 Step System Estimation Process 2011
1. Establish Estimate Scope
2. Establish Technical Baseline, Ground
Rules, Assumptions
4. Refine Technical Baseline Into
Estimable Components
4. Collect data / estimation inputs
5. Estimate Baseline Cost, Schedule, Affordability Value
6. Validate Business Case Costs &
Benefits (go / no go)
6. Quantify Risks and Risk Analysis
8. Generate a Project Plan
9. Document Estimates and Lessons
Learned
10. Track Project Throughout
Development
Level 0 Informal or no estimating
Manual effort estimating without a process
Level 1 Direct Task Estimation
Spreadsheets Ad Hoc Process
Level 2 Formal
Sizing (e.g. function points)
Direct Task Estimation
Simple model (Size *
Productivity) or informal SEER
Use
Some measurement
& analysis
Informal Process
Level 3 Formal Sizing
Robust Parametric estimation
(SEER)
Estimate vs. actual capture
Formalized Multiple Estimate Process
Rigorous measurement
& analysis
Parametric planning &
Control
Risk Management
Repeatable process
Level 4 Formal sizing Repeatable
process
Robust parametric estimating
(SEER)
Rigorous measurement
& analysis
Parametric estimation
with tracking & control
Risk Management
Process improvement via lessons
learned
Level 5 Formal sizing Repeatable
process
Robust parametric estimating
(SEER)
Rigorous measurement
& analysis
Parametric estimation
with tracking & control
Risk Management
Continuous process
improvement
Why should we care? Maturity is related to estimate viability… With better estimation process, projects more likely to be successful in execution
Estimation Organizational Maturity V1.7
ESTIMATION & PLANNING: An Estimate Defined
An estimate is the most knowledgeable statement you can make at a
particular point in time regarding:
Effort / Cost
Schedule
Staffing
Risk
Reliability
Estimates more precise with progress
A WELL FORMED ESTIMATE IS A DISTRIBUTION
36
Avoid Surprises with Estimation Process & Tools
Challenged projects
Would you still go forward if you knew
Schedule would be significantly longer?
Cost would be dramatically higher?
Probably: but perhaps more insight could identify mitigation
Plan functionality differently
Certainly you could notify stakeholders of real costs
Ensure staffing is appropriate for the constraints
Failed Projects
Would you start a project you knew was unaffordable? Or if schedule was
completely unrealistic?
If knowing up-front could you do something about it?
often better to kill project before it begins than waste resources & let the
organization down
How can we determine which software
projects with have the highest ROI?
Software Projects Must Return Business Value
“Economics is primarily a science of choice… software
economics should provide methods for analyzing the
choices software projects must make.” Leon Levy
“Base choices on those providing the maximum business
value to the organization” Eli Goldratt
Measurement and its uses such as estimating and defect
analysis help this science of choice
Some say business value is not our problem
While others generally need to perform benefit analysis
We need to build systems that optimize the business
Make IT part of the solution
If IT & measurement don’t generate sufficient profit, money will go elsewhere
Software & IT Systems are about Business Value
Cost
Value
An ROI Analysis of an Upgrade: Software Analysis & Measurement Provides Valve Cost
Can we do better? Will stakeholders tolerate a loss for 3 years?
What is the risk?
Measurement Perspective
We measure to ultimately
produce business value to the
organization
MEASURED Measurement
should not ultimately be a cost
The analysis of measurements
produces decisions that
produce business value
Management & Stakeholder Perspective
Speak so I can understand
Give me actionable items
Don’t just give me problems…
Give me solutions
Help me make the best decisions
so we can produce business
value
Measurement itself is not the answer….
Management decision making, better performance, quality improvements and better serving stakeholders is
Measurement Manifesto for Software & IT
Learn more about CAST
www.castsoftware.comblog.castsoftware.com
www.facebook.com/castonquality www.slideshare.net/castsoftware www.twitter.com/OnQuality