11
Program Management Program Management with Requirementswith Requirements--
Based Measurement Based Measurement IFPUG Fall Conference 2007IFPUG Fall Conference 2007
Terry VogtBooz Allen Hamilton
8283 Greensboro DriveMcLean, VA 22102
22
Program Management with Program Management with RequirementsRequirements--Based MeasurementBased Measurement
Agenda
Rationale: The sizes of contemporary major software projects Rationale: The sizes of contemporary major software projects and development programs are becoming increasingly huge. and development programs are becoming increasingly huge. They are difficult to manage, and hold opportunity for enormous They are difficult to manage, and hold opportunity for enormous failure. Conversely, they provide the potential for massive failure. Conversely, they provide the potential for massive savings & success.savings & success.
• Essentials of Requirements-Based Software Measurement
• Application and Benefits• Improving Measurement Quality • Dealing with Change and Uncertainty
33
Program Management with Program Management with RequirementsRequirements--Based MeasurementBased Measurement
Essential Factors in RequirementsEssential Factors in Requirements--Based Based Software MeasurementSoftware Measurement
•• Key role of functional requirements Key role of functional requirements
•• Other significant measurement factorsOther significant measurement factors
•• Use of Function PointsUse of Function Points
•• Use of estimation toolUse of estimation tool
44
Essential Factors in RequirementsEssential Factors in Requirements--Based Based Software MeasurementSoftware Measurement
•• Key role of functional requirements Key role of functional requirements Identify what is to be providedIdentify what is to be provided
Basis for contractual obligationBasis for contractual obligation
Apply to new development, enhancements, packagesApply to new development, enhancements, packages
Key to sizing workKey to sizing work
Key to evaluating package capabilityKey to evaluating package capability
Fundamental to user satisfactionFundamental to user satisfaction
Basis for reusability considerationsBasis for reusability considerations
55
Essential Factors in RequirementsEssential Factors in Requirements--Based Based Software MeasurementSoftware Measurement
•• Other significant measurement factorsOther significant measurement factorsFor project managementFor project management
For program managementFor program management
For organizational improvementFor organizational improvement
For customer satisfactionFor customer satisfaction
For budgetingFor budgeting
For competitive successFor competitive success
1 2 3 4 5 6 7 8 9 10 11
4
8
12
16
20
Months
Est. Schedule
Defects Inserted
Defects Rem oved
Potential Defects
Delivered Defects
1 2 3 4 5 6 7 8 9 10 1 1 12 13 1 4
4 0
8 0
120
160
200
1% 10% 20% 30% 40% 50% 60% 70% 80% 90% 99%0
4
8
12
16
20
10%: Best
50%: Median
90%: Worst
66
Essential Factors in RequirementsEssential Factors in Requirements--Based Based Software MeasurementSoftware Measurement
Function points measure software size based on the Function points measure software size based on the functionality requested by and provided to the end userfunctionality requested by and provided to the end user
Function point counting resources
User/analyst interviewsRequirements documentsDesign documentsData dictionariesUse casesUser guidesScreen capturesActual softwareEntity-relationship modelsSemantic object models
Function point counting Function point counting resourcesresources
User/analyst User/analyst interviewsinterviewsRequirements Requirements documentsdocumentsDesign documentsDesign documentsData dictionariesData dictionariesUse casesUse casesUser guidesUser guidesScreen capturesScreen capturesActual softwareActual softwareEntityEntity--relationship relationship modelsmodelsSemantic object Semantic object modelsmodelsFunction points represent Function points represent logicallogical size, as size, as
opposed to opposed to physicalphysical size (like SLOC or objects)size (like SLOC or objects)
77
Essential Factors in RequirementsEssential Factors in Requirements--Based Based Software MeasurementSoftware Measurement
•• Use of Function PointsUse of Function PointsRequirements managementRequirements management
Package evaluationPackage evaluation
EstimationEstimation
Change managementChange management
EVMEVM
Process improvementProcess improvement
Performance evaluationPerformance evaluation
Program managementProgram management
88
Essential Factors in RequirementsEssential Factors in Requirements--Based Based Software MeasurementSoftware Measurement
•• Requirements Source MaterialRequirements Source MaterialMany available sources: RFPs / RFIs, Proposals, user Many available sources: RFPs / RFIs, Proposals, user documentation, Use Cases, BPM in ERP software documentation, Use Cases, BPM in ERP software development, discussions & papers, etc. development, discussions & papers, etc.
Requirements translated into function points for sizeRequirements translated into function points for size
•• Use of software development estimation toolUse of software development estimation toolUse of historic experience databaseUse of historic experience database
Qualitative performance capabilities are criticalQualitative performance capabilities are critical
Use of output reports and comparative analysisUse of output reports and comparative analysis
May be able to equate requirements and SLOCMay be able to equate requirements and SLOC
99
Essential Factors in RequirementsEssential Factors in Requirements--Based Based Software MeasurementSoftware Measurement
Example: SEERExample: SEER--SEMSEM
Parametric software Parametric software development estimation tooldevelopment estimation tool
Model general project Model general project characteristicscharacteristics•• PlatformPlatform•• ApplicationApplication•• Acquisition methodAcquisition method•• Development methodDevelopment method•• Development standardDevelopment standard
Adjust input parameters to Adjust input parameters to reflect actual development reflect actual development environmentenvironment•• Personnel capabilitiesPersonnel capabilities•• Development Development
environmentenvironment•• Labor ratesLabor rates•• Product requirementsProduct requirements
Calibrate to historical data
Tool produces a range of possible Tool produces a range of possible cost, effort, and schedule values cost, effort, and schedule values with corresponding certainties, with corresponding certainties, as well as defect estimates and as well as defect estimates and other metricsother metricsCalibrate to historical data
1010
Essential Factors in RequirementsEssential Factors in Requirements--Based Based Software MeasurementSoftware Measurement
Use of estimation tool Use of estimation tool -- ExampleExample
A framework with which proposals can be comparedA framework with which proposals can be comparedComparing certainty levels across Comparing certainty levels across the estimate continuum is key to the estimate continuum is key to understanding relative value or understanding relative value or risk of any given proposal risk of any given proposal
Cost Range
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
1% 10% 20% 30% 40% 50% 60% 70% 80% 90% 99%
Certainty Level
Cos
t ($
Mill
ions
)
Proposal B
Proposal A
Proposal C
Schedule Range
0.0
10.0
20.0
30.0
40.0
50.0
60.0
1% 10% 20% 30% 40% 50% 60% 70% 80% 90% 99%
Certainty Level
Dur
atio
n (M
onth
s)
Proposal B
Proposal A
Proposal C
Review estimates in this context Review estimates in this context to identify any idiosyncrasies or to identify any idiosyncrasies or inconsistenciesinconsistencies
1111
Program Management with Program Management with RequirementsRequirements--Based MeasurementBased Measurement
Application and Benefits of Application and Benefits of requirementsrequirements--based software based software measurementmeasurement
•• Software developmentSoftware development
•• Vendor package evaluationVendor package evaluation
•• Analysis of alternativesAnalysis of alternatives
•• Project and program managementProject and program management
1212
Application and Benefits of RequirementsApplication and Benefits of Requirements--Based Software MeasurementBased Software Measurement
•• Software developmentSoftware development
Correspondence between whatCorrespondence between what’’s required and whats required and what’’s s delivereddelivered
•• EVM, testing, user acceptanceEVM, testing, user acceptance
Clarity of the impact of change requestsClarity of the impact of change requests•• Effect on effort, cost, scheduleEffect on effort, cost, schedule
Usage as negotiation tool between customer and Usage as negotiation tool between customer and project teamproject team
•• Cost & schedule control with user buyCost & schedule control with user buy--inin
Means of identifying opportunities for reuseMeans of identifying opportunities for reuse•• Opportunistic reuse, planned reuse, product lines Opportunistic reuse, planned reuse, product lines
1313
Size estimates can be directly linked Size estimates can be directly linked to baseline requirementsto baseline requirements
Discrete requirements statements and function points are Discrete requirements statements and function points are linked in a project plan and set up an EVM framework linked in a project plan and set up an EVM framework
ID Task Name Duration Budget1 Design 24 days $3,375.002 Code 15 days $4,098.003 Test 18 days $6,027.00
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 W12Month 1 Month 2 Month 3
ID Task Name Duration Budget1 Design 24 days $3,375.002 Code 15 days $4,098.003 Test 18 days $6,027.00
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 W12Month 1 Month 2 Month 3
Planned work allocated to each requirement
Planned schedule with phase milestones
Implications for correspondence, clarity and negotiationImplications for correspondence, clarity and negotiation
1414
Application and Benefits of RequirementsApplication and Benefits of Requirements--Based Software MeasurementBased Software Measurement
•• Vendor package evaluationVendor package evaluationAssessment of package capabilityAssessment of package capability
Gap analysis: base package versus requirementsGap analysis: base package versus requirements
Estimation of effort needed to fulfill requirements Estimation of effort needed to fulfill requirements
Useful size of package evaluated for normalized valueUseful size of package evaluated for normalized value
Package fitness to purpose:Package fitness to purpose:
Vendor gap fit analysis on Vendor gap fit analysis on each user functional each user functional requirementrequirement
Gaps are identified by degree: Gaps are identified by degree: 3 = none, 2 = minor, 1 = 3 = none, 2 = minor, 1 = major, 0 = totalmajor, 0 = total
Applied as weights to effortApplied as weights to effort
1515
Application and Benefits of RequirementsApplication and Benefits of Requirements--Based Software MeasurementBased Software Measurement
•• Analysis of alternativesAnalysis of alternativesBuild versus buy comparisonBuild versus buy comparison
Package comparisonPackage comparison
Acquisition alternativesAcquisition alternatives•• Function points estimated from requirementsFunction points estimated from requirements
•• Package size adjusted for requirements fitPackage size adjusted for requirements fit
•• Requirements not fully satisfied by package become factor for Requirements not fully satisfied by package become factor for additional development costadditional development cost
1616
Application and Benefits of RequirementsApplication and Benefits of Requirements--Based Software MeasurementBased Software Measurement
•• Analysis of alternatives Analysis of alternatives Acquisition alternatives Acquisition alternatives -- continuedcontinued
•• Identification of package fixed costsIdentification of package fixed costs
•• Combine package costs with additional development Combine package costs with additional development cost required to fully satisfy user requirementscost required to fully satisfy user requirements
1717
Application and Benefits of RequirementsApplication and Benefits of Requirements--Based Software MeasurementBased Software Measurement
•• Project and program managementProject and program managementManaging to budget capsManaging to budget caps
Requirements scheduling for release managementRequirements scheduling for release management
Acquisition strategy comparisons Acquisition strategy comparisons
1818
Application and Benefits of RequirementsApplication and Benefits of Requirements--Based Software MeasurementBased Software Measurement
Acquisition strategy comparisons Acquisition strategy comparisons –– example: Reuseexample: Reuse
1919
Program Management with Program Management with RequirementsRequirements--Based MeasurementBased Measurement
Improving Measurement Quality Improving Measurement Quality
•• Reducing uncertaintyReducing uncertainty
•• Requirements traceabilityRequirements traceability
•• Performance feedbackPerformance feedback
2020
Improving Measurement QualityImproving Measurement Quality
•• Reducing uncertaintyReducing uncertaintyRequirements uncertainty affects project planningRequirements uncertainty affects project planning
•• Uncertainty increases the need to raise estimated size Uncertainty increases the need to raise estimated size and corresponding effort and costand corresponding effort and cost
•• Uncertainty expands ranges necessary to express Uncertainty expands ranges necessary to express potential outcome potential outcome ––> risk feeds on itself> risk feeds on itself
What decreases uncertaintyWhat decreases uncertainty•• User inputUser input
•• Project structureProject structure
•• Progress on a project, experience and reliable historyProgress on a project, experience and reliable history
•• Stability in environment: technical, managerial, userStability in environment: technical, managerial, user
Uncertainty may decrease with time, or may notUncertainty may decrease with time, or may not•• Time yields experienceTime yields experience
•• Speed can reduce exposure to changeSpeed can reduce exposure to change
2121
Improving Measurement QualityImproving Measurement Quality
•• Requirements traceabilityRequirements traceabilityBest practices Best practices
•• Enables requirements fulfillmentEnables requirements fulfillment
•• Part of optimal configuration managementPart of optimal configuration management
•• Essential for impact analysisEssential for impact analysis
Allows association of size with specific componentsAllows association of size with specific components•• Ensures component and total size measurements can Ensures component and total size measurements can
be maintained over the life of the project & applicationbe maintained over the life of the project & application
Enables precise adaptive measurementEnables precise adaptive measurement•• Allows measurements to change with requirementsAllows measurements to change with requirements
•• Enables reEnables re--estimation and estimation and ““what ifwhat if”” explorationexploration
•• Supports maintenance measurementSupports maintenance measurement
•• Supports portfolio management through performance Supports portfolio management through performance evaluationevaluation
2222
Improving Measurement QualityImproving Measurement Quality
•• Performance feedback of actual results to Performance feedback of actual results to measurement and estimation processesmeasurement and estimation processes
Actuals provide data to evaluate estimation processActuals provide data to evaluate estimation process•• Accurate records of staffing, effort, duration, expenseAccurate records of staffing, effort, duration, expense
•• Calibration of estimates to actual local performanceCalibration of estimates to actual local performance
Organizational info provides Organizational info provides ““opportunity dataopportunity data””
-- Keys to learning, predicting, schedulingKeys to learning, predicting, scheduling•• Utilization dataUtilization data
Delays in acquiring requested resourcesDelays in acquiring requested resources
Potential overstaffing leading to Potential overstaffing leading to ““paddingpadding””
•• Capability dataCapability data
Numbers of specific resource typesNumbers of specific resource types
Quantitative and qualitative characteristicsQuantitative and qualitative characteristics
2323
Program Management with Program Management with RequirementsRequirements--Based MeasurementBased Measurement
Dealing with Change and UncertaintyDealing with Change and Uncertainty
•• Vagueness in objectives and materialsVagueness in objectives and materials
•• Issues with performance source dataIssues with performance source data
•• Changing requirements and prioritiesChanging requirements and priorities
•• Technical and managerial changesTechnical and managerial changes
•• Appropriate modes of analysisAppropriate modes of analysis
2424
Dealing with Change and UncertaintyDealing with Change and Uncertainty
•• Vagueness in project objectives creates Vagueness in project objectives creates problems in defining scope and requirementsproblems in defining scope and requirements
Objectives of some projects and initiatives are poorly Objectives of some projects and initiatives are poorly defineddefined
•• Measurement is a matter of definitionMeasurement is a matter of definition
•• You can not measure what you can not defineYou can not measure what you can not define
•• It is critical to clarify objectives It is critical to clarify objectives
•• One approach: GOne approach: G--QQ--M methodologyM methodology
Establish database and collection requirements for Establish database and collection requirements for projectsprojects
•• Mechanisms are available, standards are notMechanisms are available, standards are not
•• Data is lifeblood of IT and must be captured and Data is lifeblood of IT and must be captured and effectively usedeffectively used
•• The problem of participation The problem of participation ––> exchange theory> exchange theory
2525
Dealing with Change and UncertaintyDealing with Change and Uncertainty
•• Issues with performance source data:Issues with performance source data:conflicts between benchmarks and realityconflicts between benchmarks and reality
Much available source material isMuch available source material is unreliableunreliable
•• No consistent methodologies or documentationNo consistent methodologies or documentation
•• Lack of a standard chart of accounts for software costsLack of a standard chart of accounts for software costs
•• Lack of engineering standards for software products, Lack of engineering standards for software products, processesprocesses
•• It is frequently necessary to fill in factual gaps with It is frequently necessary to fill in factual gaps with reasonable placeholder valuesreasonable placeholder values
•• Ultimately the most important values are localUltimately the most important values are local
Dealing with availability of performance source dataDealing with availability of performance source data•• Insufficient data Insufficient data –– attempt limited direct collectionattempt limited direct collection
•• Excessive volume Excessive volume -- sample source data for analysissample source data for analysis
•• Do not blindly accept nonDo not blindly accept non--equivalent benchmarksequivalent benchmarks
2626
Dealing with Change and UncertaintyDealing with Change and Uncertainty
•• Changing requirements and prioritiesChanging requirements and prioritiesRequirements changes impact projects and programsRequirements changes impact projects and programs
•• Changes are inevitable and consequentialChanges are inevitable and consequential
•• Rate of change and general volatility are critical issuesRate of change and general volatility are critical issues
•• Prepare to negotiate change impacts with customersPrepare to negotiate change impacts with customers
•• Manageable with effective measurementManageable with effective measurement
Changes in direction impact plans and programsChanges in direction impact plans and programs•• Unpredictable changes in priorities and budgetUnpredictable changes in priorities and budget
•• Result is wholesale changes to project & program plansResult is wholesale changes to project & program plans
•• Prepare to adapt with traceability and flexibilityPrepare to adapt with traceability and flexibility
•• Measurement should be as adaptive as the workMeasurement should be as adaptive as the work
•• Strategy should include design for reusability to Strategy should include design for reusability to preserve valuepreserve value
2727
Dealing with Change and UncertaintyDealing with Change and Uncertainty
•• Change in technical and managerial landscapeChange in technical and managerial landscape-- impacts to project measurement and outcomesimpacts to project measurement and outcomes
Technical changes Technical changes •• High risk in short termHigh risk in short term
•• Predictable and manageable in the long termPredictable and manageable in the long term
•• Tech changes sometimes affect measurement processesTech changes sometimes affect measurement processes
•• May require different measurements and experience setMay require different measurements and experience set
Managerial changesManagerial changes•• Managerial changes affect teamwork, objectives, visionManagerial changes affect teamwork, objectives, vision
•• Any staff change may be disruptive to workAny staff change may be disruptive to work
•• Usually disruptive to measurement effort Usually disruptive to measurement effort –– priorities shiftpriorities shift
•• Impacts may cause project /program cancellation or lossImpacts may cause project /program cancellation or loss
2828
Dealing with Change and UncertaintyDealing with Change and Uncertainty
•• Use of appropriate modes of analysis Use of appropriate modes of analysis ––understanding measurement requirementsunderstanding measurement requirements
Build the model to fit the purposeBuild the model to fit the purpose•• Software packages, development, maintenanceSoftware packages, development, maintenance
•• Process improvement, organizational change Process improvement, organizational change
•• Specific strategies for acquisition, other purposesSpecific strategies for acquisition, other purposes
•• Different perspectives apply to different measurement Different perspectives apply to different measurement objectivesobjectives
•• Design flexible measurement processes to adaptDesign flexible measurement processes to adapt
Model analysis to an appropriate level of detailModel analysis to an appropriate level of detail•• Proposal versus program versus projectProposal versus program versus project
•• Comparative versus budgetary needsComparative versus budgetary needs
•• Project versus processProject versus process
2929
Final ThoughtsFinal Thoughts
•• Defensibility relies on a logical approachDefensibility relies on a logical approachMust be believable, repeatable, backed by historyMust be believable, repeatable, backed by history
•• Assumptions: key to measurement approachAssumptions: key to measurement approachState & revise as assumptions change or resolveState & revise as assumptions change or resolve
•• The The ““funnel of uncertaintyfunnel of uncertainty””Wide at first, gradually narrowsWide at first, gradually narrows
•• No formulaic answer fits every problemNo formulaic answer fits every problemAdapt, extend, improvise, overcomeAdapt, extend, improvise, overcome
Ultimately, itUltimately, it’’s all about the requirement to define success s all about the requirement to define success and to know how close you come to achieving itand to know how close you come to achieving it
3030
Program Management with Program Management with RequirementsRequirements--Based MeasurementBased Measurement
Q & A
Contact Info:Terry Vogt
Booz Allen Hamilton8283 Greensboro Drive
McLean, VA 22102703-377-4567 [email protected]