netlipse infrastructure project assessment tool

NETLIPSE Infrastructure Project Assessment Tool

Stuart Baker, Deputy Director of National Rail Projects

Department for Transport, UK

Vilnius, May 18, 2010

Contents

1. Background IPAT2. IPAT – The assessment tool3. IPAT – The assessment process – pilot

cases4. Lessons learned5. The future of IPAT6. Discussion

What is the context of IPAT?

Predecessor: Netlipse I research One of the main deliverables of

Netlipse II Funding by the European

Commission / TEN-T Agency in order to: Improve the current management and

organisation of LIPs Get insight in ‘the vitality’ of projects on

certain moments, e.g. financing (gate review) Get better insight in the progress of LIPs Benchmark projects

What is the IPAT?

The IPAT is an assessment tool

The IPAT indicates the level of confidence the IPAT assessors have that the PDO and its Client have created the necessary conditions for successfully executing the next project phase

The IPAT may be used by the European Commission /remark MHE/ as one of its tools for deciding what projects will receive their support.

When to apply the IPAT?

What is the added value of the IPAT?

Project delivery organisations: to increase the certainty of successful execution of projects, resulting in particular in reduced cost overruns and time delays, and;

Clients and funders: to understand the deliverability of projects by the project delivery organisations

EU, local governments and financial institutes: to monitor and evaluate projects (ex ante and ex post) in a systematic way

NETLIPSE and research institutes: the collection of information on research forecasts and future research demands.

Focus of the IPAT

A Clear Contextand Purpose, witha competent Organisationwill bring the Required Results

Organisation

IPAT – Conceptual model and themes

Project Delivery Organisation

Client / official sponsor

T7

T3

T10T9T8

T2

T11

Political context (Usually a Ministerial

client)

T1

T5

T6

T6

T12

T12

T4

T 1 Political Context

T 2 Objectives, Purpose and Business Case

T 3 Functional Specifications

T 4 Interfaces

T 5 Stakeholder Management

T 6 Finance

T 7 Legal procedures

T 8 TechnologyT 9 KnowledgeT10

Organisation & Management

T11Contracting

T12 Risks

Stakeholders

Private companies

(contractors, advisors, operators, etc.)

Related projects

IPAT – model: Project Phases

Phases of the Project (relate to Milestones):

M 1 Initiation of the project

M 2 Funding assembly

M 3 Official approval official planning authority

M 4 Start of execution

M 5 Completion

M 6 Start operation

M 7 5 years after start of operation

M 1 M 2 M 3 M 4 M 5 M 6 M 7

T 1

T 2

T 3

T 4

T 5

T 6

T 7

T 8

T 9

T10

T11

T12

Levels of Importance byMilestone and by Theme

Prioritisation:Level 1: Minimal importanceLevel 2: Little importance

Level 3: Medium importanceLevel 4: ImportantLevel 5: Crucial

IPAT – The assessment tool

Three elements: Questionnaire Scoring model Assessment process

A pilot assessment

EX ANTE EVALUATION Femern Bælt, Denmark

A pilot assessment: Femern Bælt, Denmark

STEP 1: Search for background information

A pilot assessment: Femern Bælt one of a long term series of key Danish links

Great Belt

Øresund

Fehmern Bælt

19 km coast to coast connection (Bridge or Tunnel?)

Bridge

Tunnel

Project Organisation

Femern A/S: subsidiary of Sund & Bælt

Budget & Financing

Budget

(prices as of 2008)

• a cable-stayed bridge: 4.5 billion Euro • an immersed tunnel: 5.5 billion Euro • hinterland infrastructure on the German side: 840 million Euro (cost as of 2003).• hinterland infrastructure on the Danish side: 1 billion Euro

Financing 70-75%: by Danish loans, to be refinanced by toll income25-30%: European Commission (TEN-T) funds for the period from 2007 to 2013 of 339 million Euro. For the period from 2014 to 2020, EU support will be requested

Planning

Present stage of project. Investigations:- route corridor- geotechnical- maritime- environmental

Time of decision to build, i.e., the go/no-go decision

2012: Anticipated presentation of a construction law in the Danish Parliament

Time of start of construction 2013: start of construction phase for the coast-to-coast segment

Time of delivery 2018

Commencement of operations

End of 2018


Step 1: Search for background information

Step 2: Site visit - interviews

Site visit

Assessors team: Stuart Baker Marcel Hertogh Ellen Gehner

Interviewees: Peter Lundhus – Managing director

Femern A/S Steen Lykke – Project director Tunnel

Open interview on 12 themes Duration: 2 x 2 hours



Step 2: Site visit – interviews Step 3: Scoring

IPAT- Questionnaire (Theme 1)

1 Political context M1

1 Is there a political consensus about the need and general purpose of the project? 0

2 Is there a political consensus on the defined outcomes from the project? 0

3 Is there a political consensus on the type of infrastructure solutions proposed by the project? 0

4 Is there a political consensus on the defined route of new infrastructure? 0

5 Are there clear interfaces between the politicians and the clienting of the project? 0

6 Is there political consensus on the funding solutions? 0

7 Are there arrangements to handle any political interfaces, including differences of view between the various levels of (inter)national, regional and local, city level? 0

8 How is the relation between the Sponsor/Client and the politics/ and how do you handle differences? 0

Total score (Σ score 1-8) 0

Average score (Total score / number of questions) 0,0

Scoring process

1. Score all questions in the IPAT Questionnaire on a scale from 1 to 4 individually Questions that are not relevant to a project

phase or not applicable to a project must be indicated as such.

2. Note relevant arguments and comments that support the scoring as input for the assessment report

3. Organise a meeting with the assessors team to create consensus about the scores

Scores

The objective of scoring the criteria is to identify the main strengths and weaknesses of the project and its organisation within each theme.

Scoring table reflects weakness of the project (low score) and strength of the project (high score):

Score Qualification This reflectsScore 1 Very negative effect on the project being

successfula vital need to improve immediately

Score 2 Negative contribution to a successful project

an need to improve a weak area

Score 3 Positive contribution to a successful project

Further progress can be made to achieve a better result

Score 4 Very positive contribution to a successful project

a clear strong area for the project (probably close to best practice)

IPAT – Scoring model

PROJECT NAME PROJECT PHASE: M1

Theme Weight Score Max. score Weighted score Max. weighted score

1 Political Context 3,0 … 4,0 = Weight x Score = Weight x Max. score

2 Scope 3,6 … 4,0 “ “

3 Functional Specifications 4,8 … 4,0 “ “

4 Interfaces 4,8 … 4,0 “ “

5 Stakeholder Management 4,2 … 4,0 “ “

6 Finances 4,2 … 4,0 “ “

7 Legal Procedures 3,6 … 4,0 “ “

8 Technology 4,4 … 4,0 “ “

9 Knowledge 4,6 … 4,0 “ “

10 Organisation and Management 5,0 … 4,0 “ “

11 Contracting 4,8 … 4,0 “ “

12 Risks 4,4 … 4,0 “ “

= Total Weighted Score

= Total Maximum Weighted Score

Total Weighted Percentage Score= Total Weighted Score / Total Maximum Weighted Score

IPAT – Scoring model

PROJECT NAME PROJECT PHASE: M2

Theme Weight Score Max. score Weighted score Max. weighted score

1 Political Context 4,0 3,3 4,0 13,0 16

2 Scope 4,9 2,8 4,0 13,4 19,6

3 Functional Specifications 4,2 3,5 4,0 14,7 16,8

4 Interfaces 1,9 1,1 4,0 2,1 7,6

5 Stakeholder Management 2,4 1,0 4,0 2,4 9,6

6 Finances 4,1 3,8 4,0 15,4 16,4

7 Legal Procedures 3,6 2,0 4,0 7,2 14,4

8 Technology 3,4 3,1 4,0 11,0 13,6

9 Knowledge 2,8 3,0 4,0 8,3 11,2

10 Organisation and Management 3,3 2,8 4,0 9,4 13,3

11 Contracting 2,6 1,3 4,0 3,35 10,4

12 Risks 4,2 3,0 4,0 12,6 16,8

= 112,9 = 165,7

Total Weighted Percentage Score = 68,1%



Step 2: Site visit – interviews Step 3: Scoring Step 4: Analysis of the results

Result 1: Weight – score matrix (sub)themes

Project assessment Fehmarnbelt M2

0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0 4,5 5,0

Weight

Sc

ore

Result 2: Overall score total weighted percentage score

Overall score Fehmarnbelt M2

93,4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Milestone

To

tal w

eig

hte

d p

erce

nta

ge

sco

re

M2



Step 2: Site visit – interviews Step 3: Scoring Step 4: Analysis of the results Step 5: Writing the assessment

report

Two pilot assessments

MONITORING / EX POST EVALUATIONS

Gotthard Base Tunnel West Coast Main Line (Pre 2002)

Overall score: Total percentage scoreWest Coast Main Line – Failed Project Pre 2002

Overall score WCML M1, M2, M4

38,2%41,5%

49,8%

0,0%

10,0%

20,0%

30,0%

40,0%

50,0%

60,0%

70,0%

80,0%

90,0%

100,0%

Milestone

To

tal

wei

gh

ted

per

cen

tag

e sc

ore

M1 M2 M4

Project assessment WCML

0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0 4,5 5,0

Weight

Sco

re

M1

(Sub)theme scores: Weight – score matrixWest Coast Main Line – M1


0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0 4,5 5,0

Weight

Sco

re

M1 M2

(Sub)theme scores: Weight – score matrixWest Coast Main Line – M1, M2


0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0 4,5 5,0

Weight

Sc

ore

M1 M2 M4

(Sub)theme scores: Weight – score matrixWest Coast Main Line – M1, M2, M4

Overall score: Total percentage scoreGotthard Base Tunnel

Overall score GBT M1-M4

43,9%

69,3%

77,1%81,8%

0,0%

10,0%

20,0%

30,0%

40,0%

50,0%

60,0%

70,0%

80,0%

90,0%

100,0%

Milestone

To

tal w

eig

hte

d p

erce

nta

ge

sco

re

M1 M2 M3 M4

Project assessment GBT M1-M4

0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0 4,5 5,0Weight

Sco

re

M1

(Sub)theme scores: Weight – score matrixGotthard Base Tunnel – M1


0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0 4,5 5,0

Weight

Sco

re

M1 M2

(Sub)theme scores: Weight – score matrixGotthard Base Tunnel – M1, M2


0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0 4,5 5,0

Weight

Sco

re

M1 M2 M3

(Sub)theme scores: Weight – score matrixGotthard Base Tunnel – M1, M2, M3


0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0 4,5 5,0

Weight

Sco

re

M1 M2 M3 M4

(Sub)theme scores: Weight – score matrixGotthard Base Tunnel – M1, M2, M3, M4

Lessons learned from (pilot) pilots

Validation and fine-tuning of the questionnaire Questionnaire was comprehensive Questions were relevant Questions were recognisable for the interviewees Questions could be answered by the interviewees Some questions are reformulated

Development of the scoring methodology The assessors encountered little problems in

scoring the questions The assessors were generally unanimous about the

scoring The methodology of calculating the overall score

was considered and changed ahead of the pilots Significance of levels of importance appears to be

little Development of the final analysis of the IPAT:

Still to do: what are appropriate ‘pass scores’???

What’s next until end 2010?

Pilots still to undertake: Lock - Flanders (June 2010) French Project possibly Figueras -

Perpignan (Sep 2010) Koper – Ljubljana (June 2010)

Operational version of the IPAT (including assessors manual)

List of competent assessors Training programme

Discussion (1)

Role of the assessors during the assessment process: how to guarantee the quality of the assessment?

Discussion (2)

Pass score: what is a ‘good enough’ project? (sub)theme score – position of green

area??? Overall score > 50% ??? Bandwidth? Extra hurdle: the overall score might be

considered good enough if on each (sub)theme level the score is in or above the green area???

Discussion (3)

Added value of the application of IPAT: what makes the IPAT attractive to Clients and PDOs to do an assessment?

Time for your input now please...

Having heard all this, do you broadly understand the IPAT and are you now convinced that you we can do a reliable, fair and comparable assessment of Projects with the IPAT?

netlipse infrastructure project assessment tool

Documents