1 2014 report on the performance of the defense acquisition system on contract incentives dr. dan...

1

2014 Report on the Performance of the Defense Acquisition System

On Contract Incentives

Dr. Dan DavisSenior Economist

Acquisition Policy Analysis CenterOUSD(AT&L) / PARCA

[email protected]

Improving policies and performance through data analysis

7 April 2015

Not Cleared for Public ReleaseApproved for public release; distribution unlimited: 14-S-1935, 19 June 2014.

2Approved for public release; distribution unlimited: 14-S-1935, 19 June 2014.

• What contract incentives work the best?– Predetermined, formulaic?– Award fee?– Firm fixed price?– Competition– Future margin

• Are final margins affected by contractor performance?

• Are fixed-price contracts better than cost-reimbursement?

Objective: Use statistical analysis of contract data to understand incentive effectiveness


• At the median (central tendency), DoD-wide margin is systematically predicted by:Exogenous constant 5.7 %Margin change effect – 0.9 %Size (by schedule) + 1.6 % Total predicted margin 6.4 %(N=81)

Other variables (statistically insignificant or spurious):

– Cost growth– Price growth– Schedule slips– Contract type– Army, Navy, or Air Force– Quantity change– Commodity type (9 types)– Overhead share of costs– Fixed-cost share of costs– Time trend– Size of contract by spend

Final margins were not dependent on cost, price, or schedule performance in development!

Regression over 81 MDAP development contracts, 95 percent of which were started in or after 2000. Price and cost growth are adjusted for inflation.

Issue is not what the final margin is, but that it does not systematically vary by performance!


• At the median (central tendency), DoD-wide margin is systematically predicted by:Exogenous constant 14 %Margin change effect + 1 %LRIP effect – 2 %Ship effect – 1 %Space effect – 1 %Total predicted margin = 11 %(N=83)

Other variables (statistically insignificant or spurious):

– Cost growth– Price growth– Schedule slips– Contract type– Service component– Quantity change– Other 7 Commodities– Overhead share of costs– Fixed-cost share of costs– Time trend– Size of contract by schedule– Size of contract by spend

Also, final margins were not dependent on cost, price, or schedule performance in production!

Issue is not what the final margin is, but that it does not systematically vary by performance!

Regression over 81 MDAP development contracts, 95 percent of which were started in or after 2000. Price and cost growth are adjusted for inflation.


Across all phases, fixed-price (FP) contracts had significantly lower cost growth…

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Contract cost growth for all cost-type and other contracts

17%

(adjusted for inflation)

CP/H

Costgrowth

Comparison of contract cost growth by contract-type, all contracts

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Contract cost growth for all cost-type and other contracts (weighted)

Costgrowth

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Contract cost growth for all fixed-price contracts

-2%

FP

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Contract cost growth for all fixed-price contracts (weighted)

14%

FP

Not weighted

Weighted

n=71 n=95

n=71 n=95

33%

CP/H


…but this was because FP were used primarily in lower-risk cases (production)

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Contract cost growth in production (weighted)

11%

Weighted by spend

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Contract cost growth in development (weighted)

41%

Development Production

n=83 n=83

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Contract cost growth for all fixed-price contracts (weighted)

14%

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Contract cost growth for all cost-type and other contracts (weighted)

CP/H

n=95 n=71

33%

FP

90% CP/H10% FP

27% CP/H73% FP


-10%

0%

10%

20%

30%

Margin for FFPcontracts,weighted

In development, CPIF / FPIF perform better than CPAF/FF on equal or lower margins

Caution: FFP has very low sample size

-20%

0%

20%

40%

60%

80%

100%

Price growth onincentivized contracts,

weighted

-20%

0%

20%

40%

60%

80%

100%

Price growth on FFPcontracts,weighted

-20%

0%

20%

40%

60%

80%

100%

Price growth on non-incentivized contracts,

weighted

40%29%

8%

-20%

0%

20%

40%

60%

80%

100%

Price growth on FFPcontracts,

unweighted

-20%

0%

20%

40%

60%

80%

100%


unweighted

38%

9%

FFP

Price growth

CPAF, CPFF, CS, OFormulaic Incentive

unweighted weighted unweighted weighted

Statistically significant

-10%

0%

10%

20%

30%

Margin on incentivizedcontracts,weighted

-10%

0%

10%

20%

30%

Margin for non-incentivized contracts,

weighted

6%3%

-20%

0%

20%

40%

60%

80%

100%


unweighted

-10%

0%

10%

20%

30%


unweighted

6%

-10%

0%

10%

20%

30%

Margin on incentivizedcontracts,

unweighted

-10%

0%

10%

20%

30%

Margin for FFPcontracts,

unweighted

3%6%

Final margin

unweighted weighted unweighted weighted unweighted weighted

21%

8%


n = 44 n = 6 n = 33

unweighted weighted

Caution


In LRIP, FPIF / CPIF perform as well or better than FFP and have lower margins

-50%

0%

50%

100%

150%


weighted

-50%

0%

50%

100%

150%

Price growth on FFPcontracts,weighted

-50%

0%

50%

100%

150%


weighted

106%

2% 3%

-50%

0%

50%

100%

150%


unweighted

-50%

0%

50%

100%

150%

Price growth on FFPcontracts,

unweighted

- 0.5% - 1%

FFP

Price growth



Statistically significant Much less

volatility

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

Margin for incentivizedcontracts,

unweighted

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

Margin for FFPcontracts,weighted

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

30%


weighted

8%

15%

9%

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

Margin for incentivizedcontracts,

unweighted

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

Margin for FFPcontracts,

unweighted

15%

n = 4 n = 11 n = 28

10%

Final margin



-50%

0%

50%

100%

150%


unweighted

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

30%


unweighted

164%

10%

Caution


In full-rate production, FFPs had lowest unweighted price growth but LRIP FPIFs did better by spend

-20%

-10%

0%

10%

20%

30%

40%

50%

60%

70%

Price growth on FFPcontracts, weighted

11%

–1%

(none)

Price growth

FFP

Caution


unweighted weighted unweighted weighted

–1%

-20%

-10%

0%

10%

20%

30%

40%

50%

60%

70%

Price growth on FFPcontracts, unweighted

-5%

0%

5%

10%

15%

20%

25%

30%

Margin on FFPcontracts, unweighted

-5%

0%

5%

10%

15%

20%

25%

30%


-10%

-15%

–11%-5%

0%

5%

10%

15%

20%

25%

30%

Margin on FFPcontracts, weighted

15%

n = 38

–12%

n = 2

(none)

Final margin

-5%

0%

5%

10%

15%

20%

25%

30%


-5%

0%

5%

10%

15%

20%

25%

30%


-10%

-15% unweighted weighted unweighted weighted


Lowest of all unweighted

(just below formulaic LRIP at median)

Formulaic LRIPs better than this at median when weighted

-20%

-10%

0%

10%

20%

30%

40%

50%

60%

70%

Price growth on FFPcontracts, unweighted

-5%

0%

5%

10%

15%

20%

25%

30%


- 2%

15%


• Striking benefits for competition on new systems– Price growth– Schedule

growth

• Similar results for competition on system modifications/ upgrades

• Final margins were lower than all sole-sourced contracts

Benefits of competition

-25%

0%

25%

50%

75%

100%

125%

Price growth on newsole source contracts,

unweighted

-25%

0%

25%

50%

75%

100%

125%

Price growth on solesource mod contracts,

unweighted

-25%

0%

25%

50%

75%

100%

125%

Price growth oncompeted contracts,

unweighted

11% 13%

n = 52 n = 17 n = 14

48%

CompetedSole-source

or 1 bid: modSole-source or 1 bid: new

Pricegrowth

-1

0

1

2

3

4

5

6

Schedule growth on newsole source contracts,

unweighted

-1

0

1

2

3

4

5

6

Schedule growth on solesource mod contracts,

unweighted

-1

0

1

2

3

4

5

6

Schedule growth oncompeted contracts,

unweighted

0.07 yrs 0.25 yrs

n = 52 n = 17 n = 14

1.66 yrs

CompetedSole-source

or 1 bid: modSole-source or 1 bid: new

Schedulegrowth(years)

-10%

0%

10%

20%

30%

Margin for competedcontracts,

unweighted

-10%

0%

10%

20%

30%

Margin for new solesource contracts,

unweighted

-10%

0%

10%

20%

30%

Margin for sole sourcemod contracts,

unweighted

5.5%8.4%

n = 52 n = 17 n = 14

7.7%

CompetedSole-source

or 1 bid: modSole-source

or 1 bid: newFinalmargin

unweighted


• Program analysis suggests:– Higher margins in production incentivize contractors to get out

of development faster• Shorter schedule slips in development are rewarded with higher

margins in production• Statistically significant (but small sample)

Effects of schedule slip in development on margin in production (on same program)

0%

2%

4%

6%

8%

10%

12%

14%

16%

0 1 2 3 4 5 6 7

Mar

gin

in p

rodu

ction

Schedule slip during development ( in years)

margin in production Fitted trend

for every 1-year slip in development, we expect a 0.5% drop in production margin


Some Conclusions from the 2014 report

• Not all incentives work. Contractual incentives are effective if1. We use them,2. They are significant, stable, and predictable, and 3. They are tied directly to our objectives.

• “Cost-plus versus fixed-price” is a red herring• CPIF and FPIF contracts perform well and share realized

savings• FFP contracting requires knowledge of actual costs• Competition is effective—when viable• Production margins may help minimize development time


Inadequate incentives mean dissolution,

or changes of organization purpose,

or failure to cooperate.

Hence, in all sorts of organizations

the affording of adequate incentives

becomes the most definitely emphasized task

in their existence.

It is probably in this aspect of executive work

that failure is most pronounced.

Chester Barnard

The Functions of the Executive (1938)


Backup slides


Price growth by Prime Contractor(Development)

LockheedMartin

BoeingNorthrop Grumman

General Dynamics

Raytheon BAEOther

Contractors

PriceGrowth(weighted by spend)

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forOther Contractors,

weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forBAE,

weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forRaytheon,weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forGeneral Dynamics,

weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forNorthrop Grumman,

weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Contract price growthfor Boeing,weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Contract price growthfor Lockheed,

weighted

n = 14 n = 13 n = 13 n = 10 n = 7 n = 6 n = 20

37% 41%

77%

8%

32%

−3%

Overallmedian:

29%22%

Weighted by spend


Schedule growth by Prime Contractor(Development)

LockheedMartin


General Dynamics

Raytheon BAEOther

Contractors

ScheduleGrowth(years; weighted by spend)

-1

0

1

2

3

4

5

Schedule growthfor Other Contractors,

unweighted

-1

0

1

2

3

4

5

Schedule growthfor BAE ,weighted

-1

0

1

2

3

4

5

Schedule growthfor Raytheon,

weighted

-1

0

1

2

3

4

5

Schedule growthfor General Dynamics,

unweighted

-1

0

1

2

3

4

5

Schedule growthfor Northrop Grumman,

weighted

-1

0

1

2

3

4

5

Schedule growthfor Boeing,weighted

-1

0

1

2

3

4

5

Schedule growthfor Lockheed,

weighted

n = 14 n = 13 n = 13 n = 10 n = 7 n = 6 n = 20

Overallmedian:

2.5 yr2.5 yr

0.2 yr

3.4 yr

0.8 yr

2.8 yr

0.0 yr

0.9 yr

Weighted by spend


Price growth by Prime Contractor(Production)

-20%

0%

20%

40%

60%

80%

100%

120%

Price growthfor Other,weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growthfor ULA,

weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forOshkosh,weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forHuntington Ingalls,

weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forRaytheon,weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forGeneral Dynamics,

weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Price growth forNorthrop Grumman,

weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Contract price growthfor Boeing,weighted

-20%

0%

20%

40%

60%

80%

100%

120%

Contract price growthfor Lockheed,

weighted

LockheedMartin


General Dynamics

RaytheonHuntington

IngallsOshkosh ULA

Other Contractors

PriceGrowth(weighted by spend)

n = 12 n = 4 n = 8 n = 20 n = 9 n = 7 n = 4 n = 11 n = 8

Overallmedian:

12%

−1% −3% −4%

21%13%

−2%

24%

2% −1%

Weighted by spend


Schedule growth by Prime Contractor(Production)

Weighted by spend

LockheedMartin


General Dynamics

RaytheonHuntington

IngallsOshkosh ULA

Other Contractors

ScheduleGrowth(years; weighted by spend)

-1

0

1

2

3

4

5

Schedule growthfor Other,weighted

-1

0

1

2

3

4

5

Schedule growthfor ULA,

weighted

-1

0

1

2

3

4

5

Schedule growth forOshkosh,weighted

-1

0

1

2

3

4

5

Schedule growth forHuntington Ingalls,

weighted

-1

0

1

2

3

4

5

Schedule growth forRaytheon,weighted

-1

0

1

2

3

4

5

Schedule growth forGeneral Dynamics,

weighted

-1

0

1

2

3

4

5

Schedule growth forNorthrop Grumman,

weighted

-1

0

1

2

3

4

5

Schedule growth forBoeing,

weighted

-1

0

1

2

3

4

5

Schedule growthfor Lockheed,

weighted

n = 12 n = 4 n = 8 n = 20 n = 9 n = 7 n = 4 n = 11 n = 8

0.5 yr

2 yr

0.8 yr

−0.7 yr

1.1 yr 1.3 yr

2.4 yr

Overallmedian:

0.5 yr0.0 yr 0.0 yr


Operating Margins (SEC Filings)

Averages (percent) 2009 2010 2011 2012 2013 Defense 9 9 9 9 9 Commercial * 6 10 11 10 11

Capital goods 10 ‡ 13 ‡ 14 ‡ 14 ‡ 14 Engineering services * 5 * 5 * 6 * 5 * 6 Automobiles and automotive parts * −3 * 7 8 6 7

Defense

Boeing (BDS only) 10 13 10 9 10 CACI 7 6 7 8 7 General Dynamics (except Aerospace) 11 11 12 8 8 Huntington Ingalls 3 4 2 5 7 ITT Defense/Exelis 12 11 8 11 10 L-3 Communications 11 11 11 10 10 Lockheed Martin 10 9 9 9 11 Mantech 9 8 8 7 6 Northrop Grumman 8 10 12 12 13 Raytheon 12 10 11 12 12

Capital goods

* Statistically lower than defense margins‡ Statistically higher than defense margins

Commercial average is that for the 23–24 margins shown for all three sectors. SOURCE: Public SEC filings (Bloomberg, Capital Alpha Partners).


Operating Margins (SEC Filings)

Capital goods Caterpillar 2 9 12 14 9 Cummins 6 12 15 11 12 Danaher 14 16 16 17 17 Dover 10 15 15 16 16 Eaton 4 9 10 9 12 Emerson Electric 13 15 16 18 17 Flowserve 14 14 14 14 15 Honeywell 13 14 15 11 16 Illinois Tool Works 10 15 15 17 18 Ingersoll-Rand 7 9 10 9 10 Joy Global 20 20 21 21 19 Lennox 4 6 4 7 9 Parker Hannifin 8 9 12 13 11 Rockwell Automation 8 13 15 16 17

Engineering services Aecom 4 5 5 4 5 CBI 7 8 8 8 7 Fluor 5 3 4 3 4 Jacobs Engineering 5 4 5 5 6

Automobiles and automotive parts Borg Warner 1 9 11 10 12 Delphi 7 10 10 11 Ford −1 7 6 4 4 General Motors −20 4 5 −2 5 Johnson Controls 1 5 5 4 4 TRW 4 9 8 8 7

Averages (percent) 2009 2010 2011 2012 2013


2 contracts each:• General Atomics• Huntington Ingalls• Bell Textron• Sikorsky• Rockwell Collins• Alliant

1 contract each:• General Electric• Austral• VT Halter Marine• Computer Sciences

Corporation• ITT• Tybrin Electronics• SAIC• VIASAT

“Other Contractors” category

2222

In God we trust; all others must bring data.

Attributed to W. Edward Deming

Approved for public release; distribution unlimited: 14-S-1935, 19 June 2014.

1 2014 report on the performance of the defense acquisition system on contract incentives dr. dan...

Documents