1 value-based software engineering i: motivation and key practices liguo huang computer science and...

1

Value-Based Software Engineering I: Motivation and Key Practices

LiGuo HuangComputer Science and Engineering

Southern Methodist University

2

Outline• Motivation and definitions• Understanding sources of value• Seven key practices

1. Benefits Realization Analysis2. Stakeholders’ Value Proposition Elicitation and

Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity Management5. Concurrent System and Software Engineering6. Value-Based Monitoring and Control7. Change as Opportunity

• Conclusions; references

3

Software Testing Business Case

• Vendor proposition– Our test data generator will cut your test costs in half– We’ll provide it to you for 30% of your test costs– After you run all your tests for 50% of your original cost,

you are 20% ahead• Any concerns with vendor proposition?

4

Software Testing Business Case• Vendor proposition

– Our test data generator will cut your test costs in half– We’ll provide it to you for 30% of your test costs– After you run all your tests for 50% of your original cost,

you are 20% ahead• Any concerns with vendor proposition?

– Test data generator is value-neutral*– Every test case, defect is equally important– Usually, 20% of test cases cover 80% of business case

* As are most current software engineering techniques

VERs for Value-Based Testing vs. Value-Neutral Testing (Bullock, 2000)

% of Valuefor

CorrectCustomer

Billing

Customer Type

100

80

60

40

20

5 10 15

Automated test generation tool - all tests have equal value

Bullock data– Pareto distribution

ROI: Value-Neutral ATG vs. Pareto Analysis (1)

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 10 20 30 40 50 60 70 80 90 100

% Tests Run

Ret

urn

On

Inve

stm

ent (

RO

I)

ATG Testing Pareto Testing

ROI: Value-Neutral ATG vs. Pareto Analysis (2)

ATG Testing Pareto Testing % of Tests Run

Cost ($K)

Value ($K)

Net Value ($K) ROI Cost

($K) Value ($K)

Net Value ($K) ROI

0 1300 0 -1300 -1.0 1000 0 -1000 -1.0 10 1350 400 -950 -.70 1100 2560 1460 +1.33 20 1400 800 -600 -.43 1200 3200 2000 1.67 40 1500 1600 100 +.07 1400 3840 2440 1.74 60 1600 2400 800 .50 1600 3968 2368 1.48 80 1700 3200 1500 .88 1800 3994 2194 1.21 100 1800 4000 2200 1.22 2000 4000 2000 1.0

Assumptions: • $1M development investments in customer billing system by beginning of testing• ATG tool costs $300K and will reduce the costs by 50% as promised• The business case for the system will produce $4M in business value in return for

$2M investment• The business case will produce a similar 80:20 distribution for the remaining 14

customer types

8

Motivation for Value-Based SE• Current SE methods are basically value-neutral

– Every requirement, use case, object, test case, and defect is equally important

– Object oriented development is a logic exercise– “Earned Value” Systems don’t track business value– Separation of concerns: SE’s job is to turn requirements into

verified code– Ethical concerns separated from daily practices

• Value – neutral SE methods are increasingly risky– Software decisions increasingly drive system value– Corporate adaptability to change achieved via software

decisions– System value-domain problems are the chief sources of

software project failures

9

The “Separation of Concerns” Legacy

• “The notion of ‘user’ cannot be precisely defined, and therefore has no place in CS or SE.”

- Edsger Dijkstra, ICSE 4, 1979

• “Analysis and allocation of the system requirements is not the responsibility of the SE group but is a prerequisite for their work”

- Mark Paulk at al., SEI Software CMM* v.1.1, 1993

*Capability Maturity Model

10

Resulting Project Social Structure

SOFTWARE

MGMT.

AERO. ELEC. G & C

MFG.

COMM PAYLOAD

I wonder whenthey'll give us ourrequirements?

11

Why Software Projects Fail

12

20% of Fires Cause 80% of Property Loss:Focus Fire Dispatching on These?

100

80

60

40

20

% of Property

Loss

% of Fires20 40 60 80 100

13

Need to Consider Fairness and Ethics:-Rawls’ Theory of Justice (1971)

• Fair rules of conduct• Principles of justice• Participants and obligations

– Provider (developer)– Buyer (acquire)– User(s)– Penumbra (general public)

• Negotiate mutually satisfactory (win-win) agreements

14

Rawls’ Theory of Justice - II• Fair rules of conduct

– Negotiation among interested parties– Veil of ignorance (about what affects whom)– Rationality

• Principles– Least Advantaged - don’t increase harm to them

• Harm = probability x magnitude (~risk exposure)– Risking Harm - don’t risk increasing harm

• Don’t use “low-threat” software in “high-threat” context

– Publicity test - defensible with honor before an informed public

• Use for difficult cost-benefit tradeoffs

15

Penumbra Negotiation Example: Fire Dispatching System

• Dispatch to minimize value of property loss– Neglect safety, least-advantaged property owners

• English-only dispatcher service– Neglect least-advantaged immigrants

• Minimal recordkeeping– Reduced accountability

• Tight budget; design for nominal case– Neglect reliability, safety, crisis performance

16

Key Definitions

• Value (from Latin “valere” – to be worth1. A fair or equivalent in goods, services, or money2. The monetary worth of something3. Relative worth, utility or importance

• Software validation (also from Latin “valere”)– Validation: Are we building the right product?– Verification: Are we building the product right?

17

Conclusions So Far• Value considerations are software success-

critical• “Success” is a function of key stakeholder

values– Risky to exclude key stakeholders

• Values vary by stakeholder role• Non-monetary values are important

– Fairness, customer satisfaction, trust• Value-based approach integrates ethics

into daily software engineering practice

18





19

What is “Value”

“relative worth, utility, or importance”− broader dictionary definition

20

The Model-Clash Spider Web: Master Net- Stakeholder value propositions (win conditions)

21

Maslow Human Need HierarchyA. Maslow, Motivation and Personality, 1954.

Self-Actualization

Esteem and Autonomy

Belongingness and love

Safety and Security

Physiological (Food and Drink)

22

Maslow Need Hierarchy• Satisfied needs aren’t motivators• Unsatisfied lower-level needs dominate higher-level needs• Management implications

– Create environment and subculture which satisfies lower-level needs

• Stability• Shared values, community• Match to special needs

– Tailor project objectives, structure to participants’ self-actualization priorities

23

People Self-Actualize in Different Ways

• Becoming a Better Manager• Becoming a Better Technologist• Helping Other Developers• Helping Users• Making People Happy• Making People Unhappy• Doing New Things• Increasing Professional Stature

24

Projecting Yourself Into Others’ Win Situations

• Do unto others

.. As you would have others do unto you

Counterexample: The Golden Rule

25

Projecting Yourself Into Others’ Win Situations

• Build computer systems to serve users and operators

.. Assuming users and operators like to write programs, and know computer science

•Computer sciences world (compilers, OS, etc.)–Users are programmers

•Applications world–Users are pilots, doctors, tellers

• Do unto others

.. As you would have others do unto you

Counterexample: The Golden Rule

26

Value-Based Software Engineering

Definition: “the explicit concern with value

concerns in the application of science and mathematics by which the properties of computer software are made useful to people”

27

VBSE Agenda• Value-based requirements engineering• Value-based architecting• Value-based design and development• Value-based verification and validation• Value-based planning and control• Value-based risk management• Value-based quality management• Value-based people management• A theory of value-based software engineering

28

7 Key Elements of VBSE1. Benefits Realization Analysis2. Stakeholders’ Value Proposition

Elicitation and Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity

Management5. Concurrent System and Software

Engineering6. Value-Based Monitoring and Control7. Change as Opportunity

29

DMR/BRA* Results Chain

INITIATIVE OUTCOMEOUTCOME

Implement a new orderentry system

ASSUMPTION

Contribution Contribution

Order to delivery time isan important buying criterion

Reduce time to process order

Reduced order processing cycle(intermediate outcome)

Increased sales

Reduce time to deliver product*DMR Consulting Group’s Benefits Realization Approach

30

Usage of Results Chain

• Identify non-software initiatives related to value-producing results

• Identify additional success-critical stakeholders (SCS)

31





32

Stakeholder Value Proposition Elicitation

• Start with Results Chain• Interview Success-Critical Stakeholders• Model Clash Spider Web as a top-level checklist• Synthetic experience techniques

– Prototypes, scenarios, stories• Shared Vision Document

– Elevator description– Results chain– Success-critical stakeholders and roles– System Block Diagram

33

Stakeholder Value Proposition Reconciliation

• Expectation Management– Be aware of the number of potential value proposition conflicts– Lessons learned retrospectives– Well-calibrated cost model– “simplifier and complicator”

• Visualization and trade-off analysis– Prototypes; scenarios, estimation models

• Prioritization– Pairwise comparison, scale-of-ten ratings of importance and difficulty

• Groupware– Easywinwin

• Business case analysis– ROI

34

EasyWinWin OnLine Negotiation Steps

35

Red cells indicate lack of Red cells indicate lack of consensus. consensus. Oral discussion of cell Oral discussion of cell graph reveals unshared graph reveals unshared information, unnoticed information, unnoticed assumptions, hidden assumptions, hidden issues, constraints, etc.issues, constraints, etc.

Tradeoffs among Cost, Schedule, and Reliability – 100K-SLOC Project

0123456789

0 10 20 30 40 50

Development Time (Months)

Cos

t ($M

)

(VL, 1)

(L, 10)

(N, 300)

(H, 10K)

(VH, 300K)

(RELY, MTBF (hours))

-- Cost/Schedule/RELY: pick any two” points

•For 100-KSLOC set of features•Can “pick all three” with 77-KSLOC set of features

37





38

Example of Business Case AnalysisROI= (Benefits-Costs)/Costs

Time

Return on Investment

-1

1

2

3 Option B

Option A

39

Example of Business Case AnalysisROI= (Benefits-Costs)/Costs

Time

Return on Investment

-1

1

2

3 Option B-Rapid

Option B

Option A

40

Unquantifiable Factors

• Benefits– Multiple criterion decision making– Stakeholder utility functions

• Uncertainties and risk– Uncertain assumptions– Buy information to reduce risk

41





42

Understanding People’s Utility Functions

• Risk-averse people• Reconcile people’s utility functions

43

How Can Risk Management Help You Deal With Risks?

• Buying information• Risk avoidance• Risk transfer• Risk reduction• Risk acceptance

44

What Else Can Risk Management Help You Do?• Determine “How much is enough?” for your products and

processes– Functionality, documentation, prototyping, COTS

evaluation, architecting, testing, formal methods, agility, discipline, …

– What’s the risk exposure of doing too much?– What’s the risk exposure of doing too little?

• Tailor and adapt your life cycle processes– Determine what to do next (specify, prototype, COTS

evaluation, business case analysis)– Determine how much of it is enough– Examples: Risk-driven spiral model and extensions

(win-win, anchor points, RUP, MBASE, CeBASE Method)

• Get help from higher management– Organize management reviews around top-10 risks

45

Example Large-System Risk Analysis:How Much Architecting is Enough?

• Large system involves subcontracting to over a dozen software/hardware specialty suppliers

• Early procurement package means early start– But later delays due to inadequate architecture– And resulting integration rework delays

• Developing thorough architecture specs reduces rework delays– But increases subcontractor startup delays

46

How Soon to Define Subcontractor Interfaces?Risk exposure RE = Prob(Loss) * Size(Loss)

-Loss due to rework delays

Time spent defining & validating architecture

RE =P(L) * S(L)

Many interface defects: high P(L)Critical IF defects: high S(L)

Few IF defects: low P(L)Minor IF defects: low S(L)

47

How Soon to Define Subcontractor Interfaces?- Loss due to rework delays

- Loss due to late subcontact startups


RE =P(L) * S(L)

Few delays: low P(L)Short Delays: low S(L)

Many delays: high P(L)Long delays: high S(L)



48


RE =P(L) * S(L)

Many delays: high P(L)Long delays: high S(L)

SweetSpot



How Soon to Define Subcontractor Interfaces?- Sum of Risk Exposures

Few delays: low P(L)Short delays: low S(L)

49

How Soon to Define Subcontractor Interfaces?- Very Many Subcontractors

RE =P(L) * S(L)

Higher P(L), S(L): many more IF’s

Mainstream Sweet

Spot

Many-SubsSweetSpot


50

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60

Percent of Time Added for Architecture and Risk Resolution

Perc

ent o

f Tim

e A

dded

to O

vera

ll Sc

hedu

le

How Much Architecting Is Enough?-A COCOMO II Analysis

Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution

Added Schedule Devoted to Rework(COCOMO II RESL factor)

Total % Added Schedule

10000KSLOC

100 KSLOC

10 KSLOC

Sweet Spot

Sweet Spot Drivers:

Rapid Change: leftward

High Assurance: rightward

51





52

Sequential Engineering Neglects Risk

$100M

$50M

Arch. A:Custommany cache processors

Arch. B:ModifiedClient-Server

1 2 3 4 5

Response Time (sec)

Original Spec After Prototyping

53

Concurrent Process Models

• Spiral process models vs. waterfall process models

• Milestone pass/fail criteria• Process maturity model

– CMM → CMMI

54





55





56

Sources of Changes

• Changes in technology• Changes in marketplace opening up new

opportunities for value creation

57

Change As Opportunity: Agile Methods

• Architecture-based– Identify product’s most likely sources of changes– Evolutionary requirements– Information hiding modularity– Schedule as Independent Variable (SAIV)

• Refactoring-based– Simple design and refactoring vs. big design up front

• Continuous customer interaction• Short value - adding increments• Tacit interpersonal knowledge

– Stories, planning game, pair programming– Explicit documented knowledge expensive to change

58

Five Critical Decision Factors• Represent five dimensions• Size, Criticality, Dynamism, Personnel, Culture

Personnel

Dynamism (% Requirements-change/month)

Culture (% thriving on chaos vs. order)

Size (# of personnel)

Criticality (Loss due to impact of defects)

5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15

Essential Funds Discretionary

Funds Comfort

Single Life

Many Lives

(% Level 1B) (% Level 2&3)

0

10

20

30

40

Agile

Plan-driven

59

Economic Value of Adaption to Change

• Change-anticipatory modular design– Real options– Buy information to reduce wrong set of changes

60





61

Conclusions• Marketplace trends favor transition to VBSE paradigm

– Software a/the major source of product value– Software the primary enabler of adaptability

• VBSE involves 7 key elements1. Benefits Realization Analysis2. Stakeholders’ Value Proposition Elicitation and


• Processes for implementing VBSE emerging– MBASE, CMMI, DMR/BRA, Balanced Scorecard, RUP

extensions, Strategic Design, Agile Methods

62

ReferencesC. Baldwin & K. Clark, Design Rules: The Power of Modularity, MIT Press, 1999.B. Boehm, “Value-Based Software Engineering,” ACM Software Engineering Notes, March 2003.B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E. Horowitz, R. Madachy, D. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, 2000.B. Boehm and L. Huang, “Value-Based Software Engineering: A Case Study, Computer, March 2003, pp. 33-41.

B. Boehm & K. Sullivan, “Software Economics: A Roadmap,” The Future of Software Economics, A. Finkelstein (ed.), ACM Press, 2000.B. Boehm and R. Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison Wesley, 2003 (to appear).

S. Faulk, D. Harmon, and D. Raffo, “Value-Based Software Engineering (VBSE): A Value-Driven Approach to Product-Line Engineering,” Proceedings, First Intl. Conf. On SW Product Line Engineering, August 2000.

R. Kaplan & D. Norton, The Balanced Scorecard: Translating Strategy into Action, Harvard Business School Press, 1996.

63

D. Reifer, Making the Software Business Case, Addison Wesley, 2002.

K. Sullivan, Y. Cai, B. Hallen, and W. Griswold, “The Structure and Value of Modularity in Software Design,” Proceedings, ESEC/FSE, 2001, ACM Press, pp. 99-108.

J. Thorp and DMR, The Information Paradox, McGraw Hill, 1998.

Economics-Driven Software Engineering Research (EDSER) web site: www.edser.org

MBASE web site : sunset.usc.edu/research/MBASE

1 value-based software engineering i: motivation and key practices liguo huang computer science and...

Documents