cocomo iii workshop v1€“ effort, schedule, cost, quality level (defects) • cocomo®iii can be...

of 44 /44
University of Southern California Center for Systems and Software Engineering COCOMO ® III Brad Clark, PhD USC Center for Systems and Software Engineering 2017 Annual Research Review April 4, 2017

Upload: phungdieu

Post on 08-Jul-2019

217 views

Category:

Documents


0 download

TRANSCRIPT

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO® III

Brad Clark, PhDUSC Center for Systems and Software Engineering

2017 Annual Research ReviewApril 4, 2017

University of Southern CaliforniaCenter for Systems and Software Engineering

The COCOMO® III Project • COCOMO (COnstructure COst MOdel) is the most widely used,

free, open source software cost estimation model in the world. – Registered Trademark for intellectual property protection– COCOMO 81 and COCOMO II models are open and free for anyone

to use– Models have been commercialized

• It has been 17 years since the COCOMO II model has been updated and calibrated to new Software Engineering practices.

Apr 5, 2017 Copyright © USC-CSSE 2

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO® III Project Purpose• Broaden audiences of COCOMO® and address scope of modern

projects: mobile devices, web/internet, big data, cloud-targeted, and multi-tenant software

• Modernize model size inputs• Consider the impact of modern development processes (e.g.

Agile)• Improve the accuracy and realism of estimates

– Improve driver definitions– New and updated software cost drivers and adjust their ratings as

needed– Quality estimation capability– Point and range estimates based on risk

• Improve value of COCOMO® in decision-making

Apr 5, 2017 Copyright © USC-CSSE 3

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO® III Project Scope• COCOMO® III will product estimates for:

– Effort, Schedule, Cost, Quality Level (Defects)• COCOMO® III can be applied at various moments in a project’s

lifecycle:– Early Estimation, Initial Project Estimation, Project Re-estimation

• COCOMO® III’s functional vision– Single and Multiple component estimate– Analysis of alternatives– Analysis with Size-Effort-Schedule as independent variables– Support for different lifecycle processes– Lifecycle cost estimation– Legacy system transformation

Apr 5, 2017 Copyright © USC-CSSE 4

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Scope• Model – Development Process Compatibility• Application Domain• Model Breath and Depth• Model WBS• Workload Sizing• Cost estimating relationships

– Parametric models– Simple relationships

Apr 5, 2017 Copyright © USC-CSSE 5

University of Southern CaliforniaCenter for Systems and Software Engineering

Model – ICSM Common Cases • Since 2000, a plethora of development processes have arisen

– Non-Developmental Item such as COTS– Agile Development– Brownfield Development

• One size fits all estimation model is no longer feasible

• 11 common development cases are presented next with indications on where COCOMO III is suitable

Apr 5, 2017 Copyright © USC-CSSE 6

Source: Boehm, B., Lane, J., Koolmanojwong, S. and Turner, R. “The Incremental Commitment Spiral Model”, Addison-Wesley 2014, Chapter 11.

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Match to ICSM Common Cases-1Case 1: Use Non-Developmental Item (NDI)• Example: Small accounting system• Size variable, complexity low• Typical Change Rate/Month: Negligible • Criticality: n/a• NDI Support: Complete• Personnel Capability: NDI-experienced (medium)• Activities: Acquire NDI, Use NDI• Time/Build: n/a• Time/Increment: Vendor-driven

Case 2: Agile• Example: E-services• Size, Complexity: Low• Typical Change Rate/Month: 1-30%• Criticality: Low to medium• NDI Support: Good, in place• Personnel Capability: Agile-ready, medium-high experience• Activities: Skip Valuation and Architecting phases; Scrum plus agile methods of choice• Time/Build: <= 1 day• Time/Increment: 2-6 weeks

Apr 5, 2017 Copyright © USC-CSSE 7

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Match to ICSM Common Cases -2Case 3: Architected Agile• Example: Business data processing• Size, Complexity: Medium• Typical Change Rate/Month: 1-10 %• Criticality: Medium to high• NDI Support: Good, most in place• Organizational Personnel Capability: Agile-ready, medium to high experience• Activities: Combine Valuation, Architecting phases. Complete NDI preparation. Architecture-

based Scrum of Scrums• Time/Build: 2-4 weeks• Time/Increment: 2-6 months

Case 4: Formal Methods• Example: Security kernel; Safety-critical LSI chip• Size, Complexity: Low• Typical Change Rate/Month: 0.3%• Criticality: Extra high• NDI Support: None• Organizational Personnel Capability: Strong formal methods experience• Activities: Precise formal specification. Formally-based programming language; formal

verification• Time/Build: 1-5 days• Time/Increment: 1-4 weeks

Apr 5, 2017 Copyright © USC-CSSE 8

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Match to ICSM Common Cases -3Case 5: Hardware with Embedded Software Component*• Example: Multi-sensor control device• Size, Complexity: Low• Typical Change Rate/Month: 0.3 - 1 %• Criticality: Medium to very high• NDI Support: Good, in place• Organizational Personnel Capability: Experienced, medium-high• Activities: Concurrent hardware/software engineering. CDR-level review. IOC development, LRIP,

FRP. Concurrent version N+1 engineering• Time/Build: Software 1-5 days• Time/Increment: Market-driven

Case 6: Indivisible IOC*• Example: Complete vehicle platform• Size, Complexity: Medium to high• Typical Change Rate/Month: 0.3 – 1%• Criticality: High to very high• NDI Support: Some in place• Organizational Personnel Capability: Experienced, medium to high• Activities: Determine minimum-IOC likely, conservative cost. Add deferrable software features as

risk reserve. Drop deferrable features to meet conservative cost. Strong award fee for features not dropped.

• Time/Build: Software: 2-6 weeks• Time/Increment: Platform: 6-18 months

Apr 5, 2017 Copyright © USC-CSSE 9

* Means this process is suitable for COCOMO III

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Match to ICSM Common Cases -4Case 7: NDI-Intensive• Example: Supply chain management• Size, Complexity: Medium to high• Typical Change Rate/Month: 0.3 – 3%• Criticality: Medium to very high• NDI Support: NDI-driven architecture• Organizational Personnel Capability: NDI-experienced, medium to high• Activities: Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent

requirements/architecture definition. Pro-active NDI evolution influencing, NDI upgrade synchronization

• Time/Build: Software: 1-4 weeks• Time/Increment: Systems: 6-18 months

Apr 5, 2017 Copyright © USC-CSSE 10

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Match to ICSM Common Cases -5Case 8: Hybrid Agile/Plan-Driven System*• Example: C4ISR system• Size, Complexity: Medium to very high• Typical Change Rate/Month: Mixed parts; 1-10%• Criticality: Mixed parts; Medium to very high• NDI Support: Mixed parts• Organizational Personnel Capability: Mixed parts• Activities: Full ICSM, encapsulated agile in high change, low-medium criticality parts (Often

HMI, external interfaces). Full ICSM, three-team incremental development, concurrent V&V, next-increment re-baselining

• Time/Build: 1-2 months• Time/Increment: 9-18 month

Case 9: Multi-Owner System of Systems*• Example: Net-centric military operations• Size, Complexity: Very high• Typical Change Rate/Month: Mixed parts; 1-10 %• Criticality: Very high• NDI Support: Many NDIs, some in place• Organizational Personnel Capability: Related experience, medium to high• Activities: Full ICSM; extensive multi-owner team building, negotiation. Full ICSM; large

ongoing system/software engineering effort• Time/Build: 2-4 months• Time/Increment: 18-24 months

Apr 5, 2017 Copyright © USC-CSSE 11

* Means this process is suitable for COCOMO III

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Match to ICSM Common Cases -6Case 10: Family of Systems• Example: Medical device product line• Size, Complexity: Medium to very high• Typical Change Rate/Month: 1-3%• Criticality: Medium to very high• NDI Support: Some in place• Organizational Personnel Capability: Related experience, medium to high• Activities: Skip Valuation and Architecting phases. Scrum plus agile methods of choice• Time/Build: 1-2 months• Time/Increment: 9-18 months

Case 11: Brownfield• Example: Incremental legacy phaseout• Size, Complexity: High to very high• Typical Change Rate/Month: 0.3-3%• Criticality: Medium-high• NDI Support: NDI as legacy replacement• Organizational Personnel Capability: Legacy re-engineering• Activities (Incremental Definition): Re-engineer/refactor legacy into services. Incremental

legacy phaseout• Time/Build: 2-6 weeks/refactor• Time/Increment: 2-6 months

Apr 5, 2017 Copyright © USC-CSSE 12

University of Southern CaliforniaCenter for Systems and Software Engineering

Best Fits of Estimation-Types to ICSM Common Cases

• Pure Agile: Planning Poker, Agile COCOMO III• Architected Agile

– COSYSMO for architecting; Planning Poker, CAIV-SAIV for sprints, releases; IDPD for large systems

• Formal Methods: $/SLOC by Evaluated Assurance Level• NDI/Services-Intensive: Oracle, SAP, other ERP

– RICE Objects: (R)eports, (I)nterfaces, (C)onversions, (E)nhancements– COCOTS, Value-Added Function Points, Agile for portions

• Hybrid Agile/Plan-Driven– Expert Delphi, COCOMO III, Agile for portions; IDPD

• Systems of Systems– COSYSMO for Integrator; Hybrid Agile/Plan-Driven for component systems

(COCOMO III)• Family of Systems: COPLIMO• Brownfield: COSYSMO for refactoring; above for rebuilding

8/23/2016 Copyright © USC-CSSE 13

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO® III Suite of Models Concept

Copyright © USC-CSSE 14Apr 5, 2017

COPROMO

Legend:Model has been calibrated with historical project data and expert (Delphi) dataModel is derived from COCOMO III

COPLIMO

COPSEMOCOCOMO III

COINCOMO

Model Extensions to address other ICSM Common Cases

AGILECOCOMO III

University of Southern CaliforniaCenter for Systems and Software Engineering

New Feature: Application Domain Types

Real-Time• Sensor Control and Signal

Processing• Vehicle Control• Vehicle Payload• Real Time Embedded• Mission Processing

Engineering• Systems Software• Automation and Process Control• Simulation & Modeling

Automated Information Systems• Mission Planning• Training• Test• Data Processing

Apr 5, 2017 Copyright © USC-CSSE15

Selecting an Application Domain “pre-sets” model drivers

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Breadth• There are a number of different activities in software

development:– Requirements analysis– Architecting– Detailed Design– Assembling or Coding– Integration Testing– System Testing– Acceptance Testing– Deployment– Training

• COCOMO III will cover a subset of these activities

Apr 5, 2017 Copyright © USC-CSSE 16

University of Southern CaliforniaCenter for Systems and Software Engineering

Model Depth• Development activities include/exclude different types of work:

– Management– Requirements analysis– Product design– Programming– Test and evaluation– Configuration Management / Quality Assurance– Documentation

• COCOMO III covers a number of work types (next slide)– The work covered is an indicator for whether the model is suitable for

estimating a development process (re: 11 cases discussed earlier)

Apr 5, 2017 Copyright © USC-CSSE 17

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO III Depth

Copyright © USC-CSSE 18Apr 5, 2017

Subsystem Work Breakdown Structure

Management Engineering Programming Test & Evaluation Data

• Cost, Schedule, Performance Management

• Contract Management

• Subcontract Management

• Customer Interface

• Branch Office Management

• Management Reviews & Audits

• Software Requirements

• Product Design• Configuration

Management• End Item

Acceptance• Quality

Assurance

• Detailed Design• Code and Unit

Test• Integration

• Product Test• Plans• Procedures• Test• Reports

• Acceptance Test• Plans• Procedures• Test• Reports

• Test Support• Test beds• Test tools• Test data

• Manuals

University of Southern CaliforniaCenter for Systems and Software Engineering

Workload Sizing• The amount of development work to be done is expressed as

either a functional or product size

• The desire is for COCOMO III to use different size types organically as a size input– Want to move away from converting one size type to another, e.g.

Function Points to Source Lines of Code

Apr 5, 2017 Copyright © USC-CSSE 19

• Software Requirements• Function Point• SNAP Points• Fast Function Points• COSMIC Points• Automated Function Points

• Feature Points• Use Case Points• Story Points (Agile

Development)• Source Lines of Code

University of Southern CaliforniaCenter for Systems and Software Engineering

Reused Functionality• Currently COCOMO III uses the reuse model from COCOMO II

– Model is based on source lines of code

Apr 5, 2017 Copyright © USC-CSSE 20

AAF = 0.4×DM( )+ 0.3×CM( )+ 0.3× IM( )

AAM =

[AA+AAF(1+ (0.02×SU×UNFM))]100

, for AAF ≤ 50

[AA+AAF+ (SU×UNFM)]100

, for AAF > 50

#

$%%

&%%

Equivalent KSLOC = Adapted KSLOC ⋅AAM

• AAF: Adaption Adjustment Factor• DM: percent design modified• CM: percent of code and unit test modified• IM: percent of integration and test modified

• AAM: Adaption Adjustment Multiplier• SU: Software Understanding• UNFM: Programmer Unfamiliarity• AA: Assessment and Assimilation

University of Southern CaliforniaCenter for Systems and Software Engineering

Defect removal profile levels

Software development and maintenance estimates for:• Effort• Cost & Schedule

distributed by:o Phaseo Activityo Increment

• Quality

Local calibration to organization’s data

COCOMOIII

Model

Copyright © USC-CSSE 21Apr 5, 2017

COCOMO is an open and free model

Software product size estimate

Software product, platform, personnel & project attributes

Software reuse, maintenance, and increment parameters

University of Southern CaliforniaCenter for Systems and Software Engineering

Software product size estimate

COCOMO III Model Concept

Copyright © USC-CSSE 22Apr 5, 2017

DefectIntroduction

Model

DefectRemoval

Model

ScheduleModel

EffortModel

Number of est. residual defects and the residual defect density

Number of est. non-trivial defects for Requirements, Design, & Code

Defect removal profile levels

Software product, platform, personal & project attributes

Labor RatesCosts ($$)

Effort (Person Months)

Staffing Levels

Schedule (Months)

University of Southern CaliforniaCenter for Systems and Software Engineering

Where:A, B, C, D are constants determined by calibration E represents (dis)economies of scale and project-wide scale factors

COCOMO III Effort & Schedule Estimation Model

8/23/2016 Copyright © USC-CSSE23

Effort (PM) = A * SizeE * Product(19 Cost Drivers)

E = B + Sum(5 Cost Drivers)

Schedule (M) = C * PMF * SCED%/100

F = D + 0.2(E-B)

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO III Defect Introduction and Removal Model

Copyright © USC-CSSE 248/23/2016

Defect Introduction (DI) = A * SizeE * Product(DI Drivers)

E = Initially set to 1.0

Residual Defects = C * DI * Product(1 – DRF)

DRF: Defect Removal Fraction from 3 profiles:1. Automated Analysis2. People Reviews3. Execution Testing

University of Southern CaliforniaCenter for Systems and Software Engineering

Defect Estimation Scope• Estimates only “Nontrivial” defects

– Critical: causes a system to crash or unrecoverable data loss or jeopardizes personnel

– High: causes impairment of critical system function and no workaround solution exists

– Medium: causes impairment of critical system function, through a workaround solution does exist

• Defect estimates are for only 3 artifacts:– Requirements– Design– Code

Apr 5, 2017 Copyright © USC-CSSE 25

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO III Cost Drivers -1• Product Attributes

– Impact of Software Failure (FAIL) (formerly RELY)– Product Complexity (CPLX)– Developed for Reusability (RUSE)– Required Software Security (SECU) - New– Dropped:

• Documentation Match to Lifecycle Needs• Database Size

• Platform Attributes– Platform Constraints (PLAT) – New– Platform Volatility (PVOL)

Apr 5, 2017 Copyright © USC-CSSE 26

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO III Cost Drivers -2• Personnel Attributes

– Analyst Capability (ACAP)– Programmer Capability (PCAP)– Personnel Continuity (PCON)– Applications Experience (APEX)– Language and Tool Experience (LTEX)– Platform Experience (PLEX)

Apr 5, 2017 Copyright © USC-CSSE 27

University of Southern CaliforniaCenter for Systems and Software Engineering

COCOMO III Cost Drivers -3• Project Attributes

– Precedentedness (PREC)– Development Flexibility (FLEX)– Opportunity and Risk Resolution (RESL)– Stakeholder Team Cohesion (TEAM)– Process Capability & Usage (PCUS) (formerly PMAT)– Use of Software Tools (TOOL)– Multisite Development (SITE)

• Defect Removal Profile– Automated Analysis– People Reviews– Execution Testing and Tools

Apr 5, 2017 Copyright © USC-CSSE 28

University of Southern CaliforniaCenter for Systems and Software Engineering

Cost Driver Values -1

Copyright © USC-CSSE 29Apr 5, 2017

Product AttributesDriver VL L N H VH XH PRFAIL 0.82 0.92 1.00 1.10 1.26 1.54CPLX 0.73 0.87 1.00 1.17 1.34 1.74 2.38RUSE 0.95 1.00 1.07 1.15 1.24 1.31SECU

Platform AttributesDriver VL L N H VH XH PR

PLAT 1.00 1.08 1.23 1.54 1.54

PVOL 0.87 1.00 1.15 1.30 1.49

University of Southern CaliforniaCenter for Systems and Software Engineering

Cost Driver Values -2

Copyright © USC-CSSE 30Apr 5, 2017

Personnel AttributesDriver VL L N H VH XH PRACAP 1.42 1.19 1.00 0.85 0.71 2.00PCAP 1.34 1.15 1.00 0.88 0.76 1.76PCON 1.29 1.12 1.00 0.90 0.81 1.51APEX 1.22 1.10 1.00 0.88 0.81 1.51LTEX 1.20 1.09 1.00 0.91 0.84 1.43PLEX 1.19 1.09 1.00 0.91 0.85 1.40

Project AttributesDriver VL L N H VH XH PRPREC 6.20 4.96 3.72 2.48 1.24 0.00 1.33FLEX 5.07 4.05 3.04 2.03 1.01 0.00 1.26RESL 7.07 5.65 4.24 2.83 1.41 0.00 1.39TEAM 5.48 4.38 3.29 2.19 1.10 0.00 1.29PCUS 7.80 6.24 4.68 3.12 1.56 0.00 1.43TOOL 1.17 1.09 1.00 0.90 0.78 1.50SITE 1.22 1.09 1.00 0.93 0.86 0.80 1.53

University of Southern CaliforniaCenter for Systems and Software Engineering

Defect Removal Drivers

Copyright © USC-CSSE 31Apr 5, 2017

Defect RemovalProfile Artifact VL L N H VH XH

Automated AnalysisRequirements 0.00 0.00 0.10 0.27 0.34 0.40Design 0.00 0.00 0.13 0.28 0.44 0.50Code 0.00 0.10 0.20 0.30 0.48 0.55

People ReviewsRequirements 0.00 0.25 0.40 0.50 0.58 0.70Design 0.00 0.28 0.40 0.54 0.70 0.78Code 0.00 0.30 0.48 0.60 0.73 0.83

Execution TestingRequirements 0.00 0.23 0.40 0.50 0.57 0.60Design 0.00 0.23 0.43 0.54 0.65 0.70Code 0.00 0.38 0.58 0.69 0.78 0.88

University of Southern CaliforniaCenter for Systems and Software Engineering

The New “Nominal”• Cost Driver definition refinement

• The New “Nominal– Study of productivity trends over the past 40 years reveals a shift in

“nominal” ratings– Following slides show the shift in ratings for selected cost drivers,

i.e., the new “nominal”

Apr 5, 2017 Copyright © USC-CSSE 32

University of Southern CaliforniaCenter for Systems and Software Engineering

Impact of Productivity Trends

Apr 5, 2017 Copyright © USC-CSSE 33

Cost driver Kendall’s τ p-valueTOOL Use of Software Tools -0.37 2.20E-16PMAT Process Maturity (PCUS) -0.30 1.22E-13STOR Main Storage Constraint -0.29 1.31E-11TIME Execution Time Constraint -0.26 6.62E-10PLEX Platform Experience -0.17 1.98E-05PVOL Platform Volatility -0.18 2.04E-05APEX Applications Experience 0.17 4.88E-05LTEX Language and Tool Experience -0.15 2.84E-04DATA Database Size 0.13 1.81E-03RELY Required Software Reliability -0.10 1.42E-02CPLX Product Complexity -0.10 1.58E-02PREC Precedentedness of Application -0.09 2.13E-02ACAP Analyst Capability 0.08 4.87E-02

Kendall's Rank Correlation Coefficients between the Completion Year and COCOMO II Cost Drivers (sorted by degrees of correlation)

University of Southern CaliforniaCenter for Systems and Software Engineering

Use of Software Tools

Copyright © USC-CSSE 34Apr 5, 2017

University of Southern CaliforniaCenter for Systems and Software Engineering

Process Maturity (Now PCUS)

Copyright © USC-CSSE 35Apr 5, 2017

University of Southern CaliforniaCenter for Systems and Software Engineering

Execution Time Constraint-TIME

Copyright © USC-CSSE 36Apr 5, 2017

University of Southern CaliforniaCenter for Systems and Software Engineering

Main Storage Constraint-STOR

Copyright © USC-CSSE 37Apr 5, 2017

University of Southern CaliforniaCenter for Systems and Software Engineering

Platform Experience-PLEX

Copyright © USC-CSSE 38Apr 5, 2017

University of Southern CaliforniaCenter for Systems and Software Engineering

Applications Experience-APEX

Copyright © USC-CSSE 39Apr 5, 2017

University of Southern CaliforniaCenter for Systems and Software Engineering

Language and Tool Experience-LTEX

Copyright © USC-CSSE 40Apr 5, 2017

University of Southern CaliforniaCenter for Systems and Software Engineering

Next Steps• Refine the definition for Required Software Security• Shift cost driver rating definitions to accommodate changes in

“nominal”• Create a Rosetta Stone for converting COCOMO II data to the

COCOMO III format– Test the model on past data

• Setup data collection

Apr 5, 2017 Copyright © USC-CSSE 41

University of Southern CaliforniaCenter for Systems and Software Engineering

Required Software Security (SECU)*• Is this correlated to Process Maturity?• Should it be an extension to FAIL• Should it address the threat?

– Malicious hackers– Foreign governments

• Should it address exposure– On the network– Under guard, separated from outside environment

• Where are the descriptors:– Confidentiality– Integrity– Availability

Apr 5, 2017 Copyright © USC-CSSE 42

University of Southern CaliforniaCenter for Systems and Software Engineering

Invitation to Participate• CSSE invites you to collaborate on model development

– Review model formulation– Submit data for model calibration

• Actual Size• Effort• Schedule• Defects• Model Parameters

– Review of COCOMO III model– If you contribute data for model calibration, you will receive:

• An advanced copy of the new model• Comparison of your data with respect to other data points used to

calibrate the model• Please talk with me afterwards if you are interested

Apr 5, 2017 Copyright © USC-CSSE 43

Want to [email protected]

Make suggestions and be a model reviewer!

University of Southern CaliforniaCenter for Systems and Software Engineering

Copyright © USC-CSSE44

Questions?

For more information, contact:Brad [email protected]

Apr 5, 2017