COTS Metrics and Modeling Working Group

Chris Abts, Vic Basili, Jim Cannon, Brad Clark, Nancy Eickelmann, Peter Hantos, Keun Lee, Gary Thomas, Lori Vaughn

2© 2001 USC-CSE

What should I measure?

In what activities did I spend my time, money?• In what phases did I spend my money?• What problems are we having?• What advantages does the vender give us over our

internal staff?• What are the vender capabilities?• How do I characterize the environment?• How do I characterize the volatility of the system?

– How do I characterize the volatility of each COTS product?– How do I aggregate that volatility?

3© 2001 USC-CSE

SEI Activity Sets

• Engineering Activity Area Evaluation

– Construction

– Configuration Management

– Deployment and Sustainment

– System Context

– Architecture and Design

– Marketplace

• Business Activity Area Vendor Relationship

– Intergovernmental Supplier Relationships

– COTS Business Case

– COTS Cost Estimation

Ref: "An Activity Framework for COTS-Based Systems,” CMU/SEI-2000-TR-010, by Oberndorf, Brownsword, Sledge.

4© 2001 USC-CSE

Possible Effort Model Outputs

• Amount of time we should spend on evaluation– Product (effort)– Vendor (rating)

• Confidence in our assessment• Risk exposure associated with the decision

5© 2001 USC-CSE

Product Evaluation Effort Model Inputs(In what activities did I spend my time, money?)

How much effort does it take?– Number of potential products available– Technology maturity / volatility– Number of features to be evaluated– Number / complexity of interfaces– Number of other COTS products it has to interface to– Criticality to system– Organization maturity with COTS based-development– Vendor Rating

How much confidence do we have in our answers

6© 2001 USC-CSE

Vendor Relations Effort Model (What is the Vendor’s influence on my evaluation effort?)

Inputs– What size of the vendor’s market share does your need

represent?– Vendor’s market share– Vendor’s product maturity– Criticality of COTS product to system– Vendor responsiveness– Trade exposure– Vendor level of support (training, help desk, tech. support)

How much confidence do we have in our answers

7© 2001 USC-CSE

Risk Model for Exposure(How much resource should I invest?)

Effort Spent in Evaluation

Ris

k E

xpo

sure

8© 2001 USC-CSE

Evaluation Effort Model Discussion

Effort to evaluate = F (#potential products to evaluate, # interfaces, #features)

But most important factor is organization maturity with COTS

Meta Model (what have you got available)

9© 2001 USC-CSE

Priority Chart for Models

Difficulty

Imp

ort

an

ce

L M H

M

H

L

XRisk Exposure

X Product Evaluation XVendor Evaluation

XMeta Model

XSIZE

10© 2001 USC-CSE

Conclusion

• There is a lot we don’t know about how to conduct a COTS based project so that it ends successfully

• Modeling activity is very important to project success– How much time should I spend doing an activity?– What happens when I spend more or less time doing an

activity

• Building activity models is very hard– Hard to identify a primary relationship between effort and

something else (e.g. size and effort in COCOMO II)– Lots of different things affect effort in different activities– Little information is collected in addition to effort (if effort is

collected at all down to the activity level)