austroads review of performance contract s: the potential ... edition/cases... · austroads review...

Austroads Review of Performance Contracts

Paul Hardy, Opus International Consultants

AUSTROADS REVIEW OF PERFORMANCE CONTRACTS: THE POTENTIAL BENEFITS OF PERFORMANCE

CONTRACTS

Paul Hardy Opus International Consultants

ABSTRACT The author is a highway maintenance manager with a particular interest in procurement methods. Prior to coming to New Zealand in 1995 he was involved in maintenance of trunk roads and motorways in the UK. In New Zealand he has been involved in the management of the Nelson State Highway network and has also assisted to prepare contractor lead tenders for highway maintenance management contracts in Australia and Scotland. He has recently been involved in a review of performance contracts as a project for Austroads. The paper draws on the author’s experience gained via this project and his involvement in tendering 2 PSMC contracts in Western Australia. The paper examines the potential benefits of adopting a performance contract approach. It offers the authors opinion on the feasibility of achieving these benefits and the practicality of measuring them.

Paul Hardy Opus International Consultants Ltd

Nelson Office, 4th Floor Civic House, 106 Trafalgar Street

Private Bag 36, Nelson Tel (03) 548 1099 Fax (03 548 9528

Email [email protected]



1 INTRODUCTION

1.1 THE CHANGING ROLES OF ROAD AUTHORITIES

Over the past 10 years or so Road Authorities (RAs) around the world have undergone significant changes. In many instances this has meant a redefinition of their role. Whereas previously they were providers of services, including design, construction and maintenance, with expertise in carrying out these functions, they are increasingly becoming asset managers and the purchasers of services. The levels of ‘traditional’ technical skills required in the design, construction and maintenance of road networks has been replaced in many RAs with expertise in asset management and procurement of the outcomes desired by their customers. For these RAs their role is evolving from that of expert supplier to ‘informed purchaser’.

1.2 EVOLUTION OF CONTRACTS

Evolution of contracts used to procure road construction and maintenance has occurred as a result of various changes in Government policies, in particular the outsourcing of technical and construction activities. Many RAs have adopted forms of performance contracts and specifications to the extent that some form of performance contracts or specifications have been used by RAs in most parts of the developed world. This has been particularly the case in New Zealand and Australia were 11 performance specified maintenance contracts have been let over the last 6 years. Despite this there is no common understanding of what is meant by, and what can realistically be achieved by, a performance contract.

1.3 AUSTROADS PROJECT

In response to this Austroads commissioned a project to “facilitate a common understanding between Austroads member authorities and industry of what is meant by, and what can realistically be achieved through, performance contracts”. Specifically the project looked at: • The potential benefits of performance contracts • The applicability of performance indicators and models • Experience to date with performance contracts This paper comments upon the first element, the potential benefits. It draws heavily upon the research undertaken for the study but reflects the author’s opinion on the findings.



2 CONTEXT

To define the context, within which performance contracts are used, a model of the hierarchical needs of a typical road authority is shown below, figure 1. This model will be used to explain the drivers behind the adoption of performance contracts.

2.1 ROAD AUTHORITIES HIERARCHY OF NEEDS

Figure 1, illustrates the linkage between individual tasks (inputs) and their overall objective to provide a road network that meets the needs of their customers. Increasingly, RAs are endeavouring to align their contracts with their higher-level goals, i.e. the outcomes they wish to provide to their customers. Performance contracts are in part driven by a desire to focus on the higher level needs rather than the means of achieving them and to allow suppliers increased freedom to determine how to achieve the desired outcomes. To explain this we need to consider how various current road maintenance contracts align with the model above.

Customer

Satisfaction

High Level Outcomes

Network Level Outcomes

.

Outputs

Inputs

Figure 1: Road Authorities’ Hierarchy of Needs

Customer Satisfaction: is the ultimate objective of any RA.

High Level Outcomes: look to identify the broad social, economic and environmental outcomes required and are generally reflected in the RAs mission statements in such term as “ the provision of a safe and efficient network in an environmentally acceptable manner”

Network Level Outcomes: are a means of defining how the high level outcomes are translated into actions on the road network and reflect customer expectations of the road network. These are the outcomes currently measured by most RAs e.g. Austroads National Performance Indicators measure outcomes at this level.

Outputs : are the elements of work that contribute towards achieving an outcome, for example areas of rehabilitation, reseal etc. Outputs deal with when thing get done.

Inputs: are the means by which the outputs are delivered. They are governed by technical specifications and may dictate work method, material specifications etc. Inputs deal with “what” gets done and “how”.



2.2 THE CONTRACT CONTINUUM

A range of contract types is currently used for the procurement of road maintenance. This model can be used to allow the incremental changes of moving from one contract type to another to be considered in context. Moving left to right across the continuum, the contract type focuses increasingly upon the higher levels of the hierarchy of needs i.e. there is an increasing focus on the outcome rather than the means of achieving it. Implicit in this is the notion that greater focus on the customer’s expectations is achieved. The benefits of performance contracts rely upon the premise that as you moving to the right along the contract continuum there is a progressive increase in the value achieved.

Figure 2: The Contract Continuum

Maintenance contracts align with the three drivers shown as follows:

• Input driven contracts; focus on what needs to be done and how. They typically provide detailed methodologies for each task by the use of method specifications and payment is made on the individual inputs provided i.e. labour and plant by the hour and materials by quantity. They rely the contract managers to decide the treatments and scope of works required. Examples of input driven contract include in house- service agreements with direct labour organisations

• Output driven contracts; focus on when thing need to be done. They generally comprise a schedule of items for each repair type. There is some flexibility in repair method afforded to the contractor. Payment is per item completed and may include some lump sum items that combine cyclic and routine works items. A greater level of risk and responsibility is passed to the contractor than in input driven contracts. Contractors can usually carry out output driven contracts without the need to hire in new technical skills. Conventional routine maintenance contracts fall into this category.

Inputs

Network

Level Outcomes

High Level

Outcomes

Customer

Satisfaction

Outputs

What & How When Customer Expectations

Outcome Driven Output Driven Input Driven



• End product specifications, fall somewhere between output and outcome driven. These specifications are performance-based with the time frame being end of construction with a 12-month liability period. E.g. Transit’s P17 Resurfacing contracts

• Outcome driven contracts; focus on the delivery of the outcome desired by the customers and endeavour to align the contract objectives with the higher level needs of the RA. They specify only the desired outcomes and allow flexibility in methodology. Payment is by a single lump sum, with an element of the payment at risk against measured performance. Performance contracts are outcome driven contracts. Examples include:

• Transit’s PSMCs 01,02 for Auckland Harbour Bridge and 409km of predominantly rural state highway respectively

• Transit’s hybrid contracts fall somewhere between an outcome and an output-based contract.

Outcome driven contracts and performance contracts are synonymous. They have developed generally as an evolution of existing contract arrangement with the intent of providing better value than the current contracts and a service more closely aligned with the outcomes desired.



3 TERMINOLOGY

The definition of what constitutes a performance contract or specification varies significantly between RAs and also from country to country. Defining a common terminology for use by the industry is a key output from the study.

3.1 DEFINITIONS

The following definitions were proposed as means of fostering a common understanding of what a performance contract is and were adopted for the study:

Performance Specification: A specification that describes how the end product should perform over time.

Example: “The average roughness of the road shall always be below 75 NAASRA counts”

A Performance Contract: defines obligations and outcomes in terms of performance indicators for the product or service and is of sufficient duration to enable measurement of the contractor’s performance over time.

Examples of current performance contracts are:

• Design, Construct and Maintain Contracts (DCM)

• Performance Specified Maintenance Contracts (PSMC)

• Build, Own, Operate and Transfer (BOOT)

• Design, Build, Finance and Operate (DBFO)

3.2 KEY ELEMENTS OF A PERFORMANCE CONTRACT

The key elements of a performance contract are:

• A definition of the performance required (as opposed to the definition of the method to be used or simply the end product at the completion of construction)

• A ‘requirement that a defined performance shall prevail over a suitable period of time



The definitions are required as much to exclude what isn’t a performance contract as to identify what is. Terms such as performance based and performance related have been specifically excluded.

4 THE TIME ELEMENT

4.1 PERFORMANCE OVER TIME

The definition of a performance contract requires the inclusion of a time element. Many “performance” contracts endeavour to measure performance using performance indicators that predict the expected performance, i.e. measurements taken at the end of a maintenance period are used to infer the expected life of the product. Design and construct contracts are examples of this approach as are the performance based resurfacing contracts being used by Transit New Zealand (P17) which use measures such as texture depth at the end of a maintenance period to predict the life of the new surface.

4.2 CONTRACT DURATION

The definition of a performance contract includes the time element, however it is deliberately silent on what the duration of the contract should be. Ideally, a performance contract would have a duration that covers a significant proportion of the expected life of the product. In practice this is difficult to define, e.g. what is the expected life of a road network? Elements of it may have a life cycle of 10 years or less; but the network as whole needs to endure well beyond the 10 years and needs to be managed accordingly.

4.3 CHOICE OF DURATION

The choice of duration for performance contract is a key consideration. A 10-year duration has been adopted for most current PSMCs. 10 years has been chosen to generate the economic incentives that are the trade-off for the obligation to meet performance objectives. It is long enough to allow amortisation of risks and investments by the contractor allows:

• Investment, in physical resources & an 'asset ownership' culture” • The flexibility to design appropriate treatments, • Investment up-front to reduce ongoing costs,

10 years recognises the planning horizon of most RAs. It also limits the potential changes in traffic volume, axle loading and weather events and the consequent changes in asset condition that could be expected during the contract period as a result of such changes



The definitions given above are academic. The driver behind the adoption of this form of contract is a desire to create efficiency gains and therefore obtain better value for money. A “performance” contract is merely means to an end. Debate over whether it is possible to measure performance is largely an esoteric argument.



5 POTENTIAL BENEFITS

5.1 WHAT IS A BENEFIT?

For any contracting strategy to be considered to have provided a benefit it must result in the customer receiving:

• The same level of service at a reduced cost or

• A better level of service at the same cost or

• A better level of service at a reduced cost

If one of these overall outcomes is not achieved there is no benefit.

By definition, the customer must receive the benefit, i.e. an improvement that provides a benefit for one party in the supply chain but does not provide a benefit to the customer is not a benefit. This is key point, as a number of the claimed benefits of performance contracts are not directly beneficial to the customer.

5.2 ARE THE BENEFITS UNIQUE TO PERFORMANCE CONTRACTS?

Incremental Benefi ts?

Input Driven

Output Driven

Outcome Dr iven

Benefit • Own

Forces

•Performance

Contract

Convent ional contracts

?

The scale of benefits attributed to a performance contract has to be established with reference to the contract arrangements (if any) prior to the establishment of the contract. A benefit of a performance contract is the incremental benefit that has accrued as the result of moving to a performance model over and above benefits that could have been achieved through other contracting methods. An array of benefits has been attributed to the exposure of services to competition. These have most often been achieved by the use of competitive tendering. These benefits have to be discounted before an assessment of the benefit of the performance contract can be attempted.



5.3 EXTENT OF EXPERIENCE

The majority of performance contracts have durations of 10 years. The two longest running contracts have been in existence for 5 years, the remaining contracts are all less than 3 years old. In the context of the term (duration) of the contracts, experience is limited and consequently so is the evidence to support the benefits claimed.

Contract 1995 1996 1997 1998 1999 2000

NSW RTA North Sydney Virginia USA (5 + 5.5 yrs) Tasmania Southern Road Network PSMC01 (Auckland Harbour Bridge) PSCM02 (SH3 etc)

PSMC03 (Northland) Main Roads Western Australia (6 contracts)

Table 1: Extent of Experience: Current Performance Specified Maintenance Contracts

5.4 INDUSTRY OPINION

As part of the study 3 workshops were run and were attended by 54 people representing a range of interests within the industry including contractors, consultants, road authorities and funding agencies. In addition a questionnaire was sent out to a wide industry group. 19 replies were received. Whilst this is statistically a small sample the respondents almost all had significant experience with performance contracts. Their views are consequently worthy of comment. In the questionnaire we posed the question in relation to the potential benefits, Are the claimed benefits achievable? The responses are summarised in Figure 3.



Figure 3: Industry Questionnaire Response: Are Potential Benefits Achievable? There was strong agreement amongst the respondents that the benefits claimed could be achieved. The questionnaire went on to ask are these benefits quantifiable? The responses received are summarised in figure 4.

Figure 4: Potential Benefits: Can They Be Quantified?

Potential Benefits: Can they be achieved?

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Better

Value

for M

oney

Reduce

d Adm

inistrat

ion

Reduce

d Age

ncy Risk

Budge

tary C

ertain

ty

Increa

sed In

nova

tion

Impro

ved Utilisa

tion of

Indust

ry Skills

Better

Cust

omer

Focus

Better

Definition

of Be

tter Se

rvice R

equired

Better

Undest

andin

g of P

erform

ance

by Roa

d Man

agers

Clearer

Accou

ntabilit

y betw

een Sta

kehold

ers

Increa

sed O

wnersh

ip

Per

cen

tag

e A

gre

e Industry

Road Authority

Both

Potential Benefits: Are they quantifiable?

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Better

Value

for Mone

y

Reduc

ed A

dminis

tration

Reduce

d Age

ncy Risk

Budg

etary

Certain

ty

Increa

sed Inn

ovation

Impro

ved Utilis

ation o

f Indu

stry Sk

ills

Better

Custom

er Fo

cus

Better

Definition

of Be

tter Se

rvice R

equire

d

Better

Undest

andin

g of P

erform

ance

by Roa

d Man

agers

Clearer

Accou

ntabilit

y betw

een S

takeh

olders

Increa

sed Owne

rship

Per

cen

tag

e A

gre

e Industry

Road Authority

Both



Clearly there is less confidence that the benefits can be quantified than the re is that they can be achieved. However it is worth noting that the greatest agreement on this question is with the benefits of better value for money and better definition of the level of service required. Most road authorities would be happy to achieve these two benefits alone and would consider other potential benefits to be subsidiary to them. There was discussion within the workshops are to whether these benefits could only be achieved by the use of performance contacts. This resulted in the production of the following table. A “ü” indicates general agreement. A “?” indicates a mixture of opinion between the workshop attendees.

Potential Benefit

Achieved only through a

Performance Contract?

Easier through a Performance

Contract?

Aligned with Road User Requirements ü ü

Ownership by Contractor ü (?)

Better Value ?

Better Knowledge of Asset ?

Risk transfer to those who can manage it best ü ü

Encouraging move to Pavement deterioration modelling

ü

Intervene at optimal time ü

Longer term planning ü

Encourages innovation ü

Price certainty for level of service ü

Up-skilling staff ü

Greater uniformity of measures ü

Better programming / flexibility ü Table 2: Workshop Output: Comparative Ease of Achieving Benefits

Interestingly the workshop attendees were divided on whether the two key benefits (better value and better knowledge of the asset) could be more easily achieved using a performance contract compared to a traditional contract. This I belief reflects a generally mixed opinion amongst the industry as to the credibility of the benefits claimed for performance contracts. The following section comments upon the specific benefits that have been claimed by various parties as attributable to the adoption of performance contracts and in particular relates to the use of performance specified maintenance contracts (PSMC).



6 SPECIFIC POTENTIAL BENEFITS

6.1 VALUE FOR MONEY

Performance contracts are purported to provide better value for money by providing the same, or better, level of service at reduced price. Better value for money is ostensibly easy to quantify. Every contract let to date in Australia and New Zealand has resulted in a tendered price that has been assessed as a significant “saving” to the RA letting the contract. Figures for “savings” compared with traditional contracts range from approx. 38% quoted for the latest TRANSIT PSMC to 15% in Virginia in the USA as shown in Table 2 below.

The context of the contract arrangements preceding a change to a performance contract is important. The process of contracting out and the exposure of services to competition have led in most cases to significant cost reductions. When the contracting out process is combined with a move to the use of a performance contract model it can be reasonably assumed that some of the economic benefit has been achieved through the process of competition.

The maturity of the contracting environment is another important consideration. Where contracting out has been repeated over a sustained period there has generally been successive cost reductions as contractors discover more cost effective methods of delivering the services required. Where performance contracts have been preceded by a short, 3 years or less pilot contracting initiative, it could be argued that part of the saving could have been achieved by simply re-tendering on the basis of a conventional contracting arrangement. This is not the case with the New Zealand contracts listed in Table 1, where the preceding contract arrangement was a succession of conventional contracts over a period of over 10 years.

In terms of initial reduction of cost the evidence at present would appear to be compelling.



Contract:

(Award Date)

Quoted Saving

(Source of information)

Preceding Contract Arrangement

NSW RTA

Northern Sydney

(1996)

35%

(Improving Quality and Cutting Cost through

performance contracts – M Frost 1996)

2-year pilot trial contracts split between private contractor and in house resources.

Virginia USA

(1997)

15%

(Review of VDOT’s Administration of the

Interstate Asset Management Contract –

Dec. 2000)

NB this reports questions the validity of

these savings

In house resources.

NB this contract resulted from an unsolicited offer by the contractor.

Tasmania

(July 1998)

20% ($20m)

(MRWA Position Paper Ten Year Contracting

Strategy Road Maintenance - Aug

1998)

1 no. 3 year conventional maintenance contract

Western Australia

(1999-2000)

Contracts range of savings 15 to 35 % (Outsourcing Road Design, Construction and Maintenance in WA, - G Martin, 2000)

Mixture of 3 year conventional contracts and in house resources

New Zealand: PSMC01, 02 & 03

(1999 – 2001)

15% to 38%

(TRANSIT press release – “Transit to let 4 new 10 year maintenance contracts following

success of the first “ – May 2001)

PSMC01, 02 Conventional maintenance contracts for the last 10 years re-tendered every 3 years.

PSMC03 Traditional maintenance contracts, followed by 18month hybrid contract leading into the PSMC

Table 2: Quoted Savings of PSMCs

It is too early to judge whether these contracts are providing better value for money, it is clear that they almost always lead to tendered prices, which are less than the estimated costs of using other procurement methods. The acid test will be to monitor long-term performance and determine whether:

• The level of service provided meet long term expectations

• The tendered price remains intact over the contract period



Most of the current contracts are for 10 years and only 2 of them are older than 3 years. It is therefore early days for these contracts. These apparent “savings” will only be able to be confirmed once there has been time to ratify that the specified performance measures deliver the level of service expected over the contract period. In general terms, experience is mixed although clearly positive from a Transit point of view in New Zealand.

There is concern held by some that over the period of the contract the asset will be consumed, i.e. the condition of the asset at the end of the contract will not be of equal or better value than it was at the start of the contract. The fact that it may take 10 years or more to draw definitive conclusions regarding the cost savings that might be derived from performance contracts is significant. The contracts deal with assets that may have life spans of 20 to 40 years on which changes in condition happen slowly. Currently, there is consequently little objective data from which to assess the whole of life cost benefits of these contracts. As a result, there is considerable debate within the industry over how real these cost savings are. In particular there is the question of the duration of the contract in comparison to the deterioration life cycle of the pavement, i.e. will the deterioration of the network within the contract period be masked by the short contract period. In other words will the contracts effectively mine the condition of the asset over the contract duration creating a backlog to be addressed by the following contract?

The key question is whether the same or better value is being delivered, not simply is the cost reduced. The definition of a level of service for maintenance of a road network is complex. How sure can an authority be that the key performance measures given in the contract adequately represent the service required by the road users? Even if the measures are correct can they be accurately measured?

The successful contractors have long-term revenue streams. This provides them with the potential to make long term investment decisions for both research and development (R&D) and training. This is a clear benefit to them in comparison to shorter contracts, but it prompts the question of whether the potential economic benefit is a result of the longer duration, or the form of contract.

There is to date insufficient substantive evidence to state categorically that the initial “savings” are a true reflection of increased value for money. It is, however, similarly impossible to state that this is not the case.



6.2 BETTER CUSTOMER FOCUS

Supporters of performance contracts assert that the process of defining performance measures has led to a greater understanding of customer needs. This is feasible and could be quantified by simply asking the customers. A customer satisfaction measure is, however, conspicuous by its absence from current performance contracts and it is not clear whether RAs in establishing their performance measures have involved their customers in determining the target levels.

It is true that customer satisfaction is dependent upon an array of issues; many of which are external to the contract scope e.g. progress with desirable improvements. As the customer is the ultimate arbiter of the quality of service provided, this is an important consideration. The technically oriented will look to correlate the cost with the technical interpretation of the customer’s needs but this may miss the point in terms of customer satisfaction.

RAs are increasingly focusing on the needs of their Customers. The translation of these needs into robust performance measures (KPIs), which match the performance of the asset being created or maintained, is a considerable challenge. Historically, many of these needs have been implicit with little documentary evidence of their existence and, more importantly there has been no common understanding of customer needs amongst the parties delivering them. Who determines the level of service/performance required is a crucial consideration.

The adoption of performance contracts has led to attempts to define perceived customer needs in term of measures that can be used in a contract. It does not necessarily follow, however, that these measures accurately reflect customer’s needs. This can only be determined by reference to the customers and current contracts do not explicitly address this.

Performance contracts by their nature impose a discipline on all involved, which facilitates greater communication and discussion of customer needs. A performance contract is not required for this to happen but has in many instances provided the catalyst for a greater consideration than ever before.



6.3 IMPROVED RISK RECOGNITION, ALLOCATION AND MANAGEMENT

Performance contracts transfer significant risks to the contractor that have been traditionally been held by the RA. In this regard the most significant additional risk transferred is that of the network condition. At the same time, the RA takes on the risk that their performance measures may not reflect their requirements.

Risk transfer does not necessarily mean better risk management. The transfer of risk from the RA to the contractor only provides a benefit to the customer if the contractor is better able to manage the risk

Current performance contracts tabulate the risks associated with the contract and either allocates them to the parties or provides a framework for risk sharing on a negotiated basis. There is thus a much more explicit recognition of risk than in traditional contract arrangements. The benefit of this is that risk may be managed better as a result of this greater focus. Assessing whether a risk was better managed is, however, difficult and, debatably, could only be quantified by a subjective assessment carried out if the risk eventuated.

Performance contracts offer the RA the opportunity of shedding or sharing a variety of risks previously borne solely by the RA. Most current contract offered options to the contractor in terms of how risks are allocated and managed. For example the MRWA PSMCs invited contractors to propose the costs associated with transfer of a table of identified risks from the RA to the contractor. These figures then formed part of the contract negotiation with the successful contractor. This method has the advantage of allowing the RA to evaluate the cost effectiveness of reducing their risk exposure. Other contracts have placed a cap on certain risks. For example` Transit New Zealand’s PSMC01 transferred some weather risks to the contractor but, placed a limit upon the extent of these i.e. land slips up to a specified size were allocated to the contractor and had to be allowed for in the lump sum price, but the cost of remedying larger slips remained with Transit.

There is an inherent difference in the manner in which contractors and RAs manage risks. RAs are required to be careful if not necessarily conservative in how they handle risk. In contrast, contractors are generally amenable to taking on additional risks, provided they are accompanied by a commensurate opportunity.

The notion that performance contracts in themselves force a better management of risk is in my opinion at best unproven.



6.4 REDUCED ADMINISTRATION

Customers are not interested in administration. A reduction in administration is only a benefit if it assists in providing a better level of service or a reduction in cost.

Traditional measure and value contracts can require double handling of information between the parties involved (RA, consultant and contractor). Performance contracts are generally single supplier arrangements that negate the need for a portion of this work. In addition, performance contracts advocate a “hands off” approach to contract administration. The input required from supervisory staff and contract administration staff is reduced due to the change of role from day to day surveillance to the measurement of the specified performance outcomes.

Little direct evidence is available to quantify this except for the staff reduction in some RAs following the change in contract arrangements. Some of this “administration” is, however, passed onto the contractor.

In instances where the previous contracting arrangement included 3 parties, (the RA, consultant and contractor), the consultant’s role gets absorbed into the contractor’s. There is at least one less contract to administer and it is logical to assume that whilst the contractor will take on some of the functions previously undertaken by the consultant, there will be an efficiency gain and thus some reduction in the administration.

Anecdotal evidence suggests that although performance contracts lead to a reduction in the number of supervisory staff required to administer the contract, there is a need for a higher level of skill and experience in the staff involved.

For some RA staff the change in role is challenging as it creates a feeling of loss of control. For example the decision making on treatment selection transfers to the contractor and the RA no longer controls directly what happens on the road.

Reduced administration is an “internal benefit”. It only provides a benefit to the customer if it is significant enough to provide better value for money. The scale of “savings” quoted for these contracts are significantly greater than could be achieved simply by reducing administration. Reduced administration is not an outcome and should not be a major consideration in evaluation of the benefits.



6.5 CERTAINTY OF EXPENDITURE

Most performance contracts are tendered on a single lump sum basis. The lump sum is then subject to price adjustments for performance achieved as measured by KPIs, changes to the asset etc. The potential benefit is that this approach reduces the risk of cost over-runs and budget blow- outs.

Budgetary certainty is an issue that is largely internal to the supply chain. It may not in fact be desirable to the RA as it could lead to a global inflexibility of expenditure. The customer is interested in value for money, rather than certainty of expenditure. It is therefore debatable whether this is a benefit at all.

Greater certainty of expenditure is feasible and is relatively easy to measure through a simple comparison of tendered cost against out-turn cost. In many respects a measurement of this may be useful to demonstrate whether the contract has delivered the savings indicated at the time of tender. If the price increases during the contract through variations careful consideration will need to be given as to whether these have eaten into the “savings” declared at the commencement of the contract

The downside of being committed to lump sum payment for 10 years is covered in the limitation section that follows. Many authorities, particularly local authorities that may be managing other assets as well as roads, prefer the ability to manage their budget globally. They may, for example, wish to reduce road expenditure in one year to allow funds to be allocated to utility upgrades and then catch up the deferred road maintenance in the subsequent years. “Fixed” lump sum performance contracts reduce this flexibility.

Certainty of expenditure is only a benefit if it is something that the road authority desires. Even then it only benefits the road authority rather than necessarily the customer. International experience includes examples in South America where the prime motivation for moving to a long term contract was to secure future funding by making it contractually difficult for funds to be diverted to other causes.



6.6 PROMOTE INNOVATION

There are two key considerations in determining if the customer benefits from increased innovation through the adoption of a performance contract

• Does a performance contract provide greater incentive to a contractor to innovate?

• If so, does the benefit get to the customer?

As previously described, the guaranteed long-term revenue streams provided by PSMCs might assist contractors fund the development of innovations that improve effectiveness and hence reduce the cost of their operations. It is highly likely that in tendering contractors will anticipate some efficiency gains during the contract and reduce their price accordingly. The current speed of technological advancement means that it is likely that during the life of the contract improved techniques will be available to the contractor. The longer term means that the contractor has a much greater incentive to invest in the development of new products and techniques.

The negative aspect of this is that innovations may be carried out on a commercial-in-confidence basis that may lead to duplication of effort and loss of industry wide benefits. The cost savings from innovation may not be passed on to the road user. Some contracts have endeavoured to address the potential loss of innovation, due to contractors retaining the benefits for their own commercial gain, by requiring compliance with their existing technical specifications as a starting point for the contract. The contract then allows negotiation of changes to these if they are seen as an improvement. This allows the RA the opportunity to share in the benefit of innovations developed and implemented under the contract.

Quantifying innovation is difficult in any context performance contracts are no exception. We should no confuse true innovation from efficiency gains. True innovation is generated by the forward thinkers in the industry and often need the support of RAs to come to fruition. I do not believe that performance contracts significantly increase the ability of the industry to innovate.



6.7 IMPROVED INDUSTRY SKILL BASE

If the result of the implementation of performance contracts is an overall increase in the skill level in the industry, there will ultimately be a benefit to the customer through more effective and efficient services. It is debatable, however, whether there is an overall increase is skills, or simply redistribution within the industry? I.e. have technical staff who previously worked for the RA or consultants, moved to a contracting organisation?

The shift of some RAs to more asset management focus has resulted in increased specialist technical knowledge in the contracting industry. There is clear evidence of this where some large contracting firms have recruited technical expertise and are actively targeting a greater scope of works i.e. many are looking to offer a comprehensive service that incorporates both design and construction. Whilst this trend is easy to observe and could be quantified, it would not give any true indication of whether there had been an overall increase in the skill levels in the industry.

Whilst it is clear that some skills have simply transferred from one party to another, performance contracts have created an increased focus on performance measurement, risk management and deterioration modelling/prediction. Enhanced skills in these areas have almost certainly been created as a result of being involved in the tendering and running of PSMCs.



7 THE DOWNSIDE: RISKS AND CONSTRAINTS

To obtain a balanced view of the potential benefits the potential risks and constraint associated with the adoption of a performance contract must also be evaluated. There are some significant constraints limiting the wide-scale adoption of performance contracts

7.1 INAPPROPRIAT E PERFORMANCE MEASURES

The measurement of performance is a complex and challenging area and whilst the benefits noted above are possible without robust performance measures which truly reflect the performance required these benefits may either not be realised or be overstated i.e. the desired outcome may not be achieved even if the KPIs are met.

If the base asset data is inaccurate, or not repeatable, or there is a lack of historic data this will have a big impact on the ability of the tenderers to produce maintenance programs that will be correct in the future years. Hence the risks of asset consumption or future contractual dispute over the performance measures are increased.

The development of key performance indicators is a complex and challenging task. In many respects this concept is in its infancy and it is likely that considerable knowledge will be gained over the next 5 to 10 years. This knowledge may allow this risk to be reduced significantly.

Even if appropriate measures can be determined is it possible to measure these to sufficient accuracy and repeatability? This is particularly pertinent to the expected life of a pavement and hence the inability to measure asset value and remaining life.

7.2 REDUCED COMPETITION

The majority of performance contracts let to date have been large contracts in excess of $100m. This is to ensure that there is sufficient economy of scale, to provide financial incentive to contractors to offset the high initial tendering costs and to counteract the potential downside of taking on additional risk (in particular the network condition risk). The downside to this is that the scale of the contracts and the high tendering cost is such that it effectively precludes the smaller contractors from tendering and it will become increasingly difficult for them to stay in the market.

In addition, the length of the contracts means the size of the market is reduced for the duration of the contract, thus placing greater pressure on the small and medium sized contractors.

Competition is a key driver of efficiency and needs to be retained in any long-term procurement strategy. The size of the market and the ability to retain a



competitive market place are key considerations in deciding whether to take the performance contract path. There will undoubtedly be instances whereby the adoption of a single large contract would significantly reduce future competition by locking up the majority of locally available work with one contractor. This is particularly the case in networks linking small rural communities.

7.3 THE COST OF TENDERING

Tendering is an involved and costly process requiring legal and financial advice to support the business decision in addition to the detailed technical analysis required to enable a price to be determined. This limits the parties able to tender for the contracts and raises the question of who should bear these costs. The preparation and tendering of the contract is also a complex and costly process for the RA. It generally involves not only producing a set of a typical contract documents but also compiling and verifying asset inventory and condition data and calculating programmes and estimates to be used in the tender evaluation.

7.4 INCREASED SIGNIFICANCE OF POOR CONTRACTOR’S PERFORMANCE

The possibility of poor contractor performance exists in all contracts. As with other types of contract remedies exist in performance contracts to address poor performance. The consequential effects of prematurely terminating a performance contract are, however, potentially more difficult to handle than for a traditional contract. E.g. the lead in time and cost of re-tendering a performance contact are particularly significant.

A similar situation would exist if a contractor were to go into liquidation. It could be argued that the transfer of significantly greater risk onto the contractor increases the possibility of this occurring, especially if the contractor is not able to quantify the potential costs of these risks and include adequate provision for them in their bid price.

All contracts contain the risk that a contractor may not perform to the standard required. With PSMCs the net effect of poor contractor performance is potentially greater, especially if the extent of the inadequacy is sufficient for the RA to contemplate terminating the contract. Termination of such a large contract would always be a last resort, but it is consequently particularly important that the contracts contain appropriate provisions to deal with poor performance.

7.5 REDUCED ABILITY TO DEAL WITH CHANGE

During the life of a performance contract many factors affecting the contract could change. These could be physical, political or environmental. The lump sum nature of performance contracts means that the ability to deal with such changes is potentially reduced by being “locked in” to outdated contract



provisions. It is always possible deal with variations but they invariably result in increased costs to the RA and the lack of contract rates may make negotiation of the variation more difficult.

7.6 REDUCED TECHNICAL CAPABILITY IN RAS

The transfer of responsibility for design and technical specification means that RAs will require a reduced level of technical expertise. There is a risk that if this downsizing in key technology areas is taken to an extreme level, then that RA could loose the ability to be an informed purchaser. In addition, if RAs do not retain sufficient technical skills to keep pace with developing technologies they risk being unable to take advantage of emerging technologies in future contracts. This is more of an item to be aware of than a significant risk. The developing role of RAs as procurement specialists still requires technical skills and in many respects requires a higher level of technical skills focused on asset management rather than day-to-day design and supervision.

7.7 LOSS OF CONTROL

A common concern amongst RA staff about performance contracts is the loss of control of the network. “Ownership” passes from the RA to the contractor who has the greatest say in what gets done on the network. The perceived risk is that this loss of control will lead to a reduction of the standard of service provided. The risk that is reflected by these concerns is really the risk that performance measures are inadequately defined. It is immaterial who controls or “owns” the network as long as an adequate level of service is provided.

7.8 LOSS OF INNOVATION TO THE PUBLIC DOMAIN

Whilst one of the purported potential benefit is to encourage innovation, the availability of innovation to the wider market place wi ll be restricted if they are locked in to one supplier. This is highly dependent upon how this aspect is dealt with in each contract. The risk clearly exists, but the real questions are how significant a risk is it and is it any different to what happens under normal contract arrangements?

The “risk” is predicated on the assumption that contractor developed innovation in some way replaces RA funded research and development. If RAs retain R&D then any innovations coming out of performance contracts can be considered a bonus. Comparison to RA instigated R&D is not valid if RA R&D continues.

It is likely that groundbreaking innovations will continue to come from a variety of sources. Whilst the performance contracts may provide additional incentive to investigate new methods and products the risk of these being lost to the public domain is no greater under a performance model arrangement that under a conventional contract arrangement.



7.9 LOSS OF FINANCIAL FLEXIBILITY (RA).

As described above lump sum contracts can provide greater certainty of expenditure, but this assumes that this is desirable. Not all authorities wish to be locked in to a level of expenditure for a period as long as 10 years. For example, many local authorities manage their funds more actively than this and wish to retain the ability to defer expenditure from year to year in order to balance the books and seek community input into levels of expenditure and service. Whilst performance contracts do not preclude varying the specified level of service and re-negotiation of the total expenditure it is more difficult than under a traditional contract model.

The ability to vary budgets on a year-by-year basis is restricted by PSMCs. Whether this perceived increase in price certainty is desirable or not will depend upon the wishes of the particular RA and may be a significant consideration is deciding whether or not to pursue the performance contract route.

7.10 THE ABILITY TO JUSTIFY THE BENEFITS

The potential benefits but also comments upon the difficulty of quantifying these. Currently there appears to be minimal objective data available for use in documenting the perceived benefits. Whilst this doubt exists it will always present an obstacle to the wide scale adoption of performance contracts.

7.11 LACK OF EXPERIENCE OR PREPAREDNESS (IN BOTH THE INDUSTRY AND RAS).

Performance contracts demand different skills and a different approach to that taken traditionally by contractors. If there were to be wide-scale adoption of performance contracts contractors would require sufficient warning to allow them to up-skill in some key areas particularly asset management. The RAs require a similar shift in skills with the supervision role undertaken by the RAs representative changing to one of observation/auditing requiring the acquisition of new skills. The limitation of the industry in terms of size and ability can thus be an obstacle to the wide-scale adoption of performance contracts.

The contracts themselves are large, generally over $10m for PSMCs and over $100m for DCMs. The scale and complexity of the contracts means that in order to provide a suitable team to attack the contracts contractors may wish to form joint ventures or associations with other contractors or consultants. Experience to date shows a mixture of large companies and joint ventures has been successful to date in acquiring these contracts.

7.12 LACK OF ROBUST ASSET DATA

Performance contracts rely heavily upon the use of complete, accurate and up to date asset data. Whilst most RAs will have asset inventory and condition



databases, the robustness of this data may never have been tested in earnest. If asset data is incomplete, or insufficient to allow the contractor to make reasonable pricing assumptions then significant risk to the contract is introduced.

7.13 ABILITY TO FORECAST FUTURE CONDITION

PSMC contracts use forecasts of future condition to determine the works required to maintain asset condition. For the contractor this means using a forecast as a means of determining a tender price. For the RA the forecast will be used to determine a cost estimate and to assist with the evaluation of tenders. The ability to forecast future condition is a crucial item in the success of the contract. A range of views exists within the industry over the accuracy and reliability of the tools used currently to predict future conditions. This is particularly the case with the residual life of the pavements. Without a good estimate of the effect on residual life the long-term (whole of life) benefits are difficult to assess. Confidence levels in deterioration models therefore represent a constraint to the wide scale adoption of performance contracts.

7.14 REGIONAL DEVELOPMENT AND EMPLOYMENT

In many rural areas road maintenance activities provide vital employment that assists in sustaining the viability of small communities. The efficiency gains that large region wide performance contracts create place pressures on this employment. That is contractors may not retain staff in all the communities that currently employ a small road maintenance work force. There is consequently a potential social cost to rural communities of implementing a performance contract, which may override the economic arguments.

Organisational politics within the RA may provide another constraint. The threat of job losses in the RA and the erosion of traditionally held roles and responsibilities may produce a resistance to change that makes it difficult to make the changes required to implement a performance contract.



8 SUMMARY & CONCLUSIONS

8.1 POTENTIAL BENEFITS

Significant benefits have been claimed to result from the implementation of performance contracts. The most significant of which is a substantial cost savings/increase in value for money. Other associated benefits claimed include a better identification and management of risk, greater focus on the needs of the customer and greater incentives for innovation from suppliers (contractors). Whilst significant benefits have been claimed for all the performance contracts let to date the majority of current performance contracts are less than 3 years old. It is accepted by most of the parties involved that it is too early to determine the true nature and extent of the benefits derived from these contracts.

It is also difficult to accurately quantify many of the claimed. Even the benefit that is ostensibly the easiest to measure i.e. cost savings has significant difficulties associated with it e.g. against what benchmark should “savings” be assessed and how sure can we be that the asset is being maintained. (Asset value and remaining life are difficult to measure). So, how real are these benefits? It is certainly true that costs are reduced but is sufficient effort being put into the development, testing and publishing of performance measures that can be benchmarked between networks and between procurement models?

8.2 DOWNSIDE: RISKS AND CONSTRAINTS

Is it possible to accurately define the level of performance required and if so is it possible to measure it? Current measures are based upon that which we know we can measure. Whether or not these measures truly reflect the desired outcome has never been tested in earnest. Similarly the sensitivity of these measures to relative inactivity is untried i.e. would some of the targets be met even if no work was carried out due to the slow rate of deterioration thus creating a backlog of deferred maintenance for some future contract to deal with?

Are Road Authorities able to relate the level of service they provide to the requirements of their customers? There appears to be a reluctance to involve the customers view at the level at which these contracts operate. It will be interesting to see how any local authorities that take up the model deal with this, as they by necessity keep closer contact with the road users than other road authorities.

Could the claimed benefits of performance contracts be achieved by using another model? If the objectives were as clearly stated, as performance contracts require them to be? . Massive cost reductions are not



generally achieved simply through efficiency gains. The reduction achieved by not doing a treatment is much greater than that which can be achieved by doing it more efficiently. What would be the effect on the network condition if the road authority were to gratuitously cut the funding to the level of expected saving that would be achieved by a PSMC and simply instruct their suppliers to prioritise?

Would the further expansion of the use of performance contracts ultimately lead to a loss of competitive market due to the dominance of a few large players? In New Zealand the roading a few large players already dominate industry. This applies equally to consulting and contracting. Real competition cannot in my opinion be maintained if new entrants are locked out by the entry costs of large and technically challenging contracts.

8.3 CONCLUSION

Performance contracts have prompted thought and consideration of the outcome of maintenance that had not occurred prior to their implementation. This may ultimately be the most significant benefit they provide i.e. they provide a catalyst for a much greater focus on delivering good outcomes and hence ultimately better outcomes. The method of achieving these will always be secondary. The present challenge is to provide robust methods of measuring performance that can be used to benchmark performance between networks and between procurement methods. Only when such measurements have been tested will there be a significant increase in confidence that the level of claimed benefits are real.

PSMC are not appropriate in all cases. They do however provide a viable alternative to traditional procurement methods that should be considered when determining the optimal procurement model for any network.

austroads review of performance contract s: the potential ... edition/cases... · austroads review...

Documents