marshall esc lille defense r2

A Quantitative Study of the Contribution of Earned Value

Management to Project Success on External Projects Under Contract

Robert A. MarshallPh.D. Thesis Defense

August 20, 2007

Doctoral Programme in Strategy, Programme and Project Management

Candidate’s appreciation of the committee

Philippe Ruiz, Ph.D.,Committee Chairman and Professor of Statistics

Christophe Bredillet, Ph.D., Dean’s Representative and Program Director

Frank Anbari, Ph.D., Professor of Project Management George Washington University

Wayne Abba, MPA, EVM Subject Matter Expert

2


Candidate’s appreciation of the committee cont.cont.

Russell Archibald, Ph.D.

PMI/APM/IPMA Fellow

Darren Dalcher, Ph.D.

Professor of Project Management Middlesex University

Lynn Crawford, DBA

Professor of Project Management Univ. of Technology Sydney

3


Three part outline of the presentation

PART APART A Background, framework, results and contribution

PART BPART B Technical aspects of research and data

PART CPART C Summary and directions for future research

4


5

PART APART A

Background, framework, results and contribution


Research in the domain of project procurement management

6

Earned valuemanagement

ContractsProjectmanagement


The domain is in need of more research generally

“The area of contracts is one aspect of project management that is in definite need of new ideas and tools.” (Abu-Hijleh and Ibbs, 1989)

Only 2.4% of project management conference papers presented in the US and abroad from 1996-1998 were classified as procurement (Zobel & Wearne, 2000)

Only 4% of project management research articles printed in English between 1960-1999 were classified as procurement (Kloppenborg & Opfer, 2002)

7


What is Earned Value Management?

ANSI/EIA-748 defines EVM by its various attributes:

“Comprehensive planningplanning at the outset combined with the establishment and disciplined maintenance of a baselinemaintenance of a baseline for performance measurement. . . . This combination of comprehensive planning, baseline maintenance and earned value analysisearned value analysis yields earlier and better visibility into program [project] performance” (1998)

The methodology is recognized for its unique metrics which integrate the triple constraint of cost, schedule and scope

8


EVM is actively promoted by major project management organizations

Project Management Institute (PMI)

Association of Project Management (APM)

Association for the Advancement of Cost Engineering (AACE)

Project Management Association of Japan (PMAJ)

9


Accumulation of evidence indicates a low level of EVM use

Less than 1% of projects (Fleming and Koppelman, 1996)

“If such a favorable view (merits of EVM) is so strongly stated, why then is EVM methodology still not used widely as suggested by existing literature? (Kim, 2000)

High potential; low usage (Besner and Hobbs, 2006)

10


The reason for the low level of EVM use is vague and problematic – favorablefavorable research results on research results on the one handthe one hand

11

C/SCSC effected “...a significant improvement in planning [more so in budgeting than in scheduling] and a positive but less significant improvement in control…” (Marrella, 1973)

“EVM users agree that EVM use can improve cost, schedule and technical performance of a project.” (Kim, 2000)

EVM effective in evaluating and substantiating change of scope claims (Kauffmann, Keating and Considine, 2002)


The reason for the low level of EVM use is vague and problematic– unfavorableunfavorable researchresearch results on results on the other handthe other hand

12

Don’t need EVM on projects where costs are not a main concern (Kim, 2000)

Hard to apply because of difficulties in tracking actual costs (Kim, 2000)


The reason for the low level of EVM use is vague and problematic – unfavorableunfavorable research argumentsresearch arguments

13

Incompatibility of scheduling and costs at the WBS level results in an inefficient and unmanageable control system (Eldin, 1987)

“Variance analysis results as used in EVM…bring about the wrong direction in management decisions” (Y.W. Kim, 2002)

WBS ineffective at cost-schedule integration; rejects Activity Based Project Modeling (C.S. Kim, 2002)


EVM’s applicability to contracts also vague and problematic – bias towards cost-plus contractsbias towards cost-plus contracts

“Aging truth that EVM is only useful on cost-plus contracts…” (Fleming and Koppelman, 1996)

“EVM is more often used in larger companies, in cost-reimbursable contracts…” (Kim, 2000)

EVM generally used in very large-scale and cost intensive projects (Schelle, Ottmann, and Pfeiffer, 2005)

Various U.S. Government department policies favoring cost or incentive type contracts (Marshall, 2005)

14


The guiding research questions coming out of the review of literature are threefold

What is the contribution of EVM to project success?

Should contract type have a significant role in the decision to use EVM?

What if any moderating effect does contract type have on the contribution of EVM’s mechanics to project procurement management?

15


Theoretical Framework

16

D ire c t re latio n s h ip

M o d e ratin g re latio n s h ip

P r in c ip les o f E VM

W BS

P r o jec t S u c c es s

C o n tr ac t T y p e

T y p e 1 - F ixe d P ric eT y p e 2 - Co s t P lu s

S - C u r v e

M etr ic s

C o n tr ac tF o r m atio n

C o n tr ac tAd m in is tr a tio n

P roje c t P roc ure m e ntM a na ge m e nt

E V M M e c ha nic s

P o s it iv e re latio n s h ip+

+

+

+

+

+


Variable No. 1: Principles of EVM (ANSI/EIA-748)

a. Plan all work scope for the project to completion;

b. Decompose the scope into finite pieces that can be assigned to a responsible person or organization to control;

c. Integrate program work scope, schedule, and cost objective into a performance measurement baseline plan against which accomplishments may be measured. Control changes to the baseline;

d. Use actual costs incurred and recorded in accomplishing the work performed;

17


Variable No. 1: Principles of EVM (ANSI/EIA-748) cont.cont.

e. Objectively assess accomplishments at the work performance level;

f. Analyze significant variances from the plan, forecast impacts, and prepare an estimate at completion based on performance to date and work to be performed;

g. Use EVMS information in the company's management processes.

18


Variable No. 2: Work Breakdown Structure (WBS) with respect to Contract Formation Items

a. Scope development (wbs1)

b. Risk assessment (wbs2)

c. Should cost-estimates (wbs3)

d. Schedule planning (wbs4)

e. Payment planning (wbs5)

f. Evaluating bids and negotiating with bidders (wbs6)

g. Ensuring fairness and equity to contract (wbs7)

19


Variable No. 2: “S” Curves with respect to a single Contract Administration item

a. Monitoring work (scurve)

20


Variable No. 2: EVM Performance metrics with respect to multiple Contract Administration items

a. Controlling schedule (metric1)

b. Controlling scope (metric2)

c. Controlling cost (metric3)

d. Evaluating and processing change orders (metric4)

e. Evaluating and processing payment requests (metric5)

21


Variable No. 2: EVM Performance metrics with respect to multiple Contract Administration items cont.cont.

f. Evaluation and analyzing delays (metric6)

g. Evaluating and analyzing claims (metric7)

h. Acceptance of completed work (metric8)

i. Contract close-out (release of claims) (metric9)

j. Post-project audits (metric10)

22


Variable No. 3: Project Success (Shenhar et al., 1997)

23

a. Meeting Schedule and Budget Goals - how well the project satisfied the resource constraints of schedule and budget.

b. Benefit to Customer – how well the project satisfied the functional requirements and technical specifications of the organization that benefit from the project’s realization.


Variable No. 3: Project Success (Shenhar et al., 1997) cont.cont.

c. Commercial Success. How well the project satisfies the desired business results (includes return on investment for performing organization; or enhancing the profits of the benefiting organization).

d. Preparing for the Future - The contribution the project made to the performing organization in developing new business opportunities; technologies or increasing core competencies.

24


Theoretical Framework

25

D ire c t re latio n s h ip

M o d e ratin g re latio n s h ip

P r in c ip les o f E VM

W BS

P r o jec t S u c c es s

C o n tr ac t T y p e

T y p e 1 - F ixe d P ric eT y p e 2 - Co s t P lu s

S - C u r v e

M etr ic s

C o n tr ac tF o r m atio n

C o n tr ac tAd m in is tr a tio n

P roje c t P roc ure m e ntM a na ge m e nt

E V M M e c ha nic s

P o s it iv e re latio n s h ip+

+

+

+

+

+


Research hypotheses based on low EVM usage and belief that a bias may exist – H1 and H2

H1: Principles of EVM are significant positive predictors of Project Success

H2: There will be no significant difference between Fixed-price and Cost-plus contracts when relating Principles of EVM to Project Success, separately

26


Research hypotheses based on low EVM usage and belief that a bias may exist – H3, H4 and H5

H3: There will be no significant difference between Fixed-price and Cost-plus contracts on the contributions of EVM’s Work breakdown Structure during Contract Formation

H4: There will be no significant difference between Fixed-price and Cost-plus contracts on the contribution of EVM’s "S” Curve during Contract Administration

H5: There will be no significant difference between Fixed-price and Cost-plus contracts on the use of EVM’s Performance Metrics during Contract Administration

27


Findings: H1

Results offered support for H1. Linear correlation-regression showed a moderately strong relationship between the independent variable Principles of EVM and dependent variable Project Success (r = .474; β = .474, t = 6.443, n =145). The results were significant setting p at p < .05 (a = .000). The independent variable explains 22.47% of the variation in the dependent variable (R² = .2247). As predicted, Principles of EVM are significant positive predictors of Project Success

28


X-Y Scatter Plot (n=145)

29

7.006.005.004.003.002.001.00

P r i nc i pl e s o f E VM C o m po s i te

7.00

6.00

5.00

4.00

3.00

2.00

1.00

Pro

ject

Su

cces

s C

omp

osit

eX - Y Sc atte r P l o t (n= 1 4 5 )


Findings: H2

Results did not offer support for H2. Contrary to the prediction, a difference exists between fixed-price and cost-plus contracted projects with respect to the contribution of Principles of EVM to Project Success, measured separately. The results favor fixed-price contracted projects. Linear correlation-regression showed a moderately strong positive relationship between the independent variable Principles of EVM and the dependent variable Project Success for the cost-plus group (r = .405; β = .405, t = 4.432, n=102); and a moderately strong and relatively higher positive relationship for the fixed-price group (r = .585; β = .585, t = 4.622, n=43). The independent variable explains 16.40% of the variation in the dependent variable in cost-plus contracted projects; and 34.22% in fixed-price contracted projects. The results are statistically significant setting p at p < .05 (a = .000), separately

30


Findings: H3

Results offered support for H3, with exceptions. As predicted, there are no significant differences between fixed-price and cost-plus contracted projects on the contributions of EVM’s Work Breakdown Structure to Contract Formation items with the exception of wbs4 (Schedule Planning) and wbs5 (Payment Planning) (Wilks’ λ = .936, χ² = 9.389, df2, p = .009)(eigenvalue = .068; canonical correlation = .253). Five items were removed leaving a single canonical root with two discriminant coefficients (wbs4 and wbs5). No significant differences between fixed-price and cost-plus contracts were observed with respect to the contribution of EVM’s WBS to Contract Formation items for wbs1, 2, 3, 6, and 7

31


Two canonical coefficients distinguish fixed-price from cost-plus contracted projects with respect to the use of the WBS

32

Structure Matrix

.754

-.323

.151

.136

.127

.067

.041

Variables

wbs5

wbs4

wbs7a

wbs3a

wbs6a

wbs2a

wbs1a

1

Function

Variables removed.a.

The contribution of wbs5 (Payment Planning) was relatively greater in FP as compared to CP contracted projects

The contribution of wbs4 (Schedule planning) was relatively greater in CP as compared to FP contracted projects


Findings: H4

Results offered support for H4. As predicted, there is no significant difference between fixed-price and cost-plus contracted projects on the contribution of EVM’s "S” Curve to Contract Administration items (single item included in H5 analysis)

33


Findings: H5

Results offered support for H5, with exceptions. The results show no significant difference between fixed-price and cost-plus contracted projects on the contributions of EVM’s Metrics to Contract Administration (Wilks’ λ = .943, χ² = 8.434, df1, p = .004) (eigenvalue = .061; canonical correlation = .240), with the exception of metric5 (Evaluating and Processing Payment Requests). A total of ten items were removed during the analysis leaving a single canonical root with one discriminant coefficient (metric5). No significant differences between fixed-price and cost-plus contracts were observed with respect to the contribution of EVM’s Metrics to Contract Administration items for scurve, metric1, 2, 3, 4, 6, 7, 8, 9, and 10

34


A single canonical coefficient distinguishes fixed-price from cost-plus contracted projects with respect to the use of the S-curve and EVM performance metrics

35

Structure Matrix

1.000

.522

.514

.489

.469

.445

.411

.364

.325

.316

.306

Variables items

metric5

metric7a

metric4a

metric8a

metric2a

metric9a

metric10a

metric6a

scurvea

metric1a

metric3a

1

Function

Variable not used.a.

The contribution of metric5 (Evaluating and processing payment requests) was relatively greater in FP as compared to CP contracted projects


Multiple implications stem from this research

EVM directly and measurable contributes to project success

EVM is dynamic (not a checklist)

EVM alone does not explain project success

Both CP and FP contract types benefit from EVM

EVM optimized in FP contracts

36


Multiple implications stem from this research cont.cont.

EVM planning is as important as EVM control

Emphasize WBS schedule planning in CP projects

Emphasize WBS payment planning in FP projects

Emphasize metrics in processing payment requests in FP projects

37


This research increases the body of knowledge regarding project procurement management

Five publications to date have stemmed from this research which have reached a global audience

38


Related publishing's

Marshall, Robert A. (2005). “The Case for Earned Value Management with Fixed Price Contracts.” Contract Management Magazine.

Marshall, Robert A. (2006). “The contribution of earned value management to project success on contracted efforts: A quantitative statistics approach within the population of experienced practitioners.” Proceedings of the International Research Conference on Organizing by Projects (IRNOP7).

Marshall, Robert A. (Editor, 2nd Ed.). (2007). Professional Practice Guide to Earned Value. Association for the Advancement of Cost Engineering International. AACE: Morgantown, WV.

39


Related publishing's cont.cont.

Marshall, Robert A. (2007). “The Principles of Earned Value Management as Predictor Variables of Project Success.” Transactions of the 51st Annual International Meeting of the Association for the Advancement of Cost Engineering (AACE). Morgantown, WV.

Marshall, Robert A. (2007). “The contribution of earned value management to project success on contracted efforts: A quantitative statistics approach within the population of experienced practitioners.” Journal of Contract Management. (September Issue)

40


This research offers nine specific contributions

The unique (and repeatable) testing of the ANSI-748 principles of EVM as predictor variables of project success

The unique inference that fixed-price contracted projects benefit as much as, if not more, than cost-plus contracts when applying EVM

Unique quantitative approach with respect to the contribution of the principles of EVM to project success which solidify beliefs through methodological triangulation

41


This research offers nine specific contributions cont.cont.

The unique inference that the contribution of the mechanics of EVM (WBS, S-curve and metrics) do not significantly differ between fixed-price contracted projects and cost-plus contracted projects (with three exceptions noted below)

The unique inference that the contribution of the WBS to payment planning is greater in fixed-price contracted projects as compared to cost-plus contracts

The unique inference that the contribution of the WBS in schedule planning is greater in cost-plus contracted projects as compared to fixed-price contracts

42


This research offers nine specific contributions cont.cont.

The unique inference that the contribution of the EVM metrics to evaluating and processing payment requests is relatively greater in fixed-price contracted projects as compared to cost-plus contracted projects

The mechanics of EVM contribute as much to project planning (WBS) as they do to control (S-curve and metrics)

43


The results are timely and relevant

EVM shown to have high potential; low usage (Besner and Hobbs, 2006)

Schedule management techniques are more important to high-uncertainty projects (Shenhar, et, 2002)

“Schedule risk is often under appreciated for its contribution to driving contract performance and cost overruns” (U.S. DoD EVM Implementation Guide, 2005)

Budget and scope management techniques more important with projects with low uncertainty (Shenhar, et al, 2002)

44


45

PART BPART B

Technical aspects of research


Technical aspects of research and data

Pre-survey tasks

Research approach and methodology determination Research approach and methodology determination Survey instrument developmentSurvey instrument developmentData collection strategy determination and effortData collection strategy determination and effort

Post-survey tasks

Survey instrument testingSurvey instrument testingSample testing for social desirability of group responsesSample testing for social desirability of group responsesData testing for reliability and assumptionsData testing for reliability and assumptionsData treatmentsData treatmentsStatistical analysis Statistical analysis

46


Research approach selected was quantitative

Appropriate for testing hypotheses

Lends itself to relatively short duration studies

Uses known measurement and analytic techniques

Needed representative sample available

Compliments existing research and strengthens knowledge through methodological triangulation

47


Three primary statistical methods were used

Pearson’s product-moment correlation - r r

Bivariate linear regression – ββ

Canonical discriminant analysis - Wilks’ λWilks’ λ

48


Survey instrument developed for ease of use and analysis

Thirty-eight (38) questions (plus MC-10 battery)

Likert-type semantic differential scaling

Interval level scale for continuous data

Available on-line (July 6, 2005 - March 10, 2006)

Approx. 7-10 minutes to take

49


Data collection strategy included attributes of random and purposive sampling

Deliberate sampling frame included four subgroups with requisite experience

DAU EVM Community DAU EVM Community

PMI CPM PMI CPM

APM EVM SIGAPM EVM SIG

AACE EVM SIGAACE EVM SIG

Approximate sampling frame size of 2500

Participants solicited via email without bias on two separate occasions

50


Survey data attributes

148 of 256 respondents completed survey - 57.8%

145 of potentially 2500 samples used - 5.8%

Three outlier data points (residual values > 3 σ removed)

Less than one percent missing values (μ - imputation)

Residuals displayed normal homoscedasticity and normal linear relationship to expected value

51


Survey data predominantly public projects, defense and information technology

52

Industry

3 2.1 2.1

2 1.4 3.4

19 13.1 16.6

67 46.2 62.8

16 11.0 73.8

10 6.9 80.7

15 10.3 91.0

7 4.8 95.9

6 4.1 100.0

145 100.0

Description

Heavy construction

Light construction (residentialand commercial)

Government/public works

Military/defense

Information technology

Professional services (banking,consulting, management, etc.)

Aerospace

Software development

Other

Total

Frequency Percent Cum. Percent


Independent samples t-test of MC-10 social desirability data indicating no significant different in scores about CR and FP groups

53

Independent Samples Test

2.597 .109 .520 143 .604 .20565 .39546 -.57606 .98736

.487 68.9 .628 .20565 .42217 -.63658 1.048

Assumptions

Equal variancesassumed

Equal variances notassumed

Dependent variable

Contract type

F Sig.

Levene's test

for equality

of variances

t dfSig.

(2-tailed) M DifferenceSE

Difference Lower Upper

95% CI of theDifference

t-test for equality of means



54

7.006.005.004.003.002.001.00


7.00

6.00

5.00

4.00

3.00

2.00

1.00

Pro

ject

Suc

cess

Com

posi

te

1 4 8 9 3 5 2

1 4 8 9 1 8 41 7 7 7 3 8 3

X - Y Sc atte r P l o t (n= 1 4 8 )



55

7.006.005.004.003.002.001.00


7.00

6.00

5.00

4.00

3.00

2.00

1.00

Pro

ject

Su

cces

s C

omp

osit

eX - Y Sc atte r P l o t (n= 1 4 5 )


Sample data showing normal linearity

56

1 .00 .80 .60 .40 .20 .0

O b s e rv e d C u m u lativ e P ro b ab ility

1 .0

0 .8

0 .6

0 .4

0 .2

0 .0

Exp

ecte

d C

umul

ativ

e P

roba

bilit

y

C o s t-P l us C o ntr ac t Type

D e pe nde nt Var i abl e : P r o je c t s uc c e s s

N o r m al P -P P l o t o f R e g r e s s i o n Standar di ze d R e s i dual

1 .00 .80 .60 .40 .20 .0

O b s e rv e d C u m u lativ e P ro b ab ility

1 .0

0 .8

0 .6

0 .4

0 .2

0 .0

Exp

ecte

d C

umul

ativ

e P

roba

bilit

y

F i xe d-P r i c e C o ntr ac t Type

D e pe nde nt Var i abl e : P r o je c t s uc c e s s

N o r m al P -P P l o t o f R e g r e s s i o n Standar di ze d R e s i dual


Sample data showing normal homoscedasticity

57

210-1-2-3

R e g r e s s i o n Standar di ze d P r e di c te d Val ue

4

2

0

-2

-4

Reg

ress

ion

Stan

dard

ized

Res

idua

l

F i xe d-P r i c e C o ntr ac t

D e pe nde nt Var i abl e : P r o je c t Suc c e s s

210-1-2-3-4

R e g r e s s i o n Standar di ze d P r e di c te d Val ue

2

1

0

-1

-2

-3Reg

ress

ion

Stan

dard

ized

Res

idua

l

C o s t -P l us c o ntr ac t

D e pe nde nt Var i abl e : P r o je c t Suc c e s s


Tests and treatments to assure highest reliability

Composite variables created to further increase reliability

Survey instrument tested for construct validity

Principles of EVM Principles of EVM aa = .768 (7 items) = .768 (7 items)

Project Success Project Success aa = .789 (4 items) = .789 (4 items)

58


Distribution of Principles of EVM composites

59

EVM c om pos ite

7

6

5

4

3

2

1

Fixed-price contrac t

EVM c om pos ite

7

6

5

4

3

2

1

C os t-plus contrac t


Distribution of Project Success composites

60

Projec t Suc c ess c omposite

7

6

5

4

3

2

Cost-plus c ontrac t

Projec t Suc c ess c omposite

7

6

5

4

3

2

1

Fixed-pr ic e c ontrac t


61

Limitations of research approach and results

Correlation and regression analysis is inferential (deductive claim of causation never explicitly or implicitly made)

Limited stratification among project types

Limited sample size

Model has room for improvement


62

PART CPART C

Summary and direction for future research


Summary of what led to the research goal

63

W hat is thecontribution of EVM to

projec t success?

R arelyused

m etho d

E xistingresearch

inco nsistent

S ho rtageo f research

in thedo m ain

N eed fo rnew to o ls

and greaterkno wledge


Summary of contribution

64

Q u alita t iv e R es ear c h

E x p er ien tia l E v id en c e w ith F P c o n tr ac ts

Q u an tita t iv e R es ear c h

K now le dge :

E ar n ed v a lu e m an ag em en tc o n tr ib u tes to p r o jec t s u c c es so n ex te r n a l p r o jec ts . T h isp r em is e is s u p p o r ted w ithm eth o d o lo g ic a l tr ian g u la tio nan d a v er y h ig h d eg r ee o fp r o b ab ility .

Bo th F P a n d CP c o n t ra c te dp ro je c t s b e n e fit fro m e a rn e dv a lu e ma n a g e me n t , e limin a t in gv a g u e c o n t ra c t b o rd e rs . T h isp re mis e is s u p p o rte d w ith a h ig hle v e l o f p ro b a b ility .

Kim , 2 0 0 0 ; M ar r e lla , 1 9 7 3

M ar s h a ll, 2 0 0 7

An tv ik , 2 0 0 1 ; C h en , 1 9 9 1 ; Kau f f m an , Keatin g& C o n s id in e , 2 0 0 2 ; M u k h o & L is an ti, 1 9 8 2P r es u tt i, 1 9 9 3 ; Var g as , 2 0 0 3 ; Yu , 1 9 9 6


Directions for future research

Qualitative or quantitative study as they relate to EVM or are otherwise affected by the core project management processes of initiating or closing

The depth to which scope should be defined. The question yet to be answered regarding EVM is whether an optimal level of WBS exists. Second, how the utility curve for varying degrees of WBS decomposition might look

65


Directions for future research cont.cont.

Greater stratification of contract types as categorical variables with respect to the principles of EVM and project success

Exploration of variables and dimensions not covered by this study

Predictive ability of EVM principles for the specific achievement of project financial objectives

66

marshall esc lille defense r2

Documents