OPERATION PERFORMANCE EVALUATION REVIEW (OPER)

PREFACE

The subject of this OPER is the Technical Cooperation (TC) operation: Private Road Network Management Project – Poland in the committed amount of EUR 1,500,000 and 45,874 funded respectively through the EU Phare Program and the UK Government. The studies were carried out successively by three consulting firms. The Operation Leaders (OLs) of this TC were successively Philip Cornwell, Lucyna Stanczak, Grzegorz Gajda and Sue Barrett of the Transport Team. The Operation Team and other relevant Bank staff commented on an early draft. The Basic Data Sheet on page iii of this report is complementary to this OPER and these are designed to be read together. The evaluation was carried out by Nicolas Mathieu, Senior Economist of the Evaluation Department. Information on the operation was obtained from relevant teams and departments of the Bank and its files as well as from external sector and industry sources. Fieldwork was carried out in July 2005. EvD would like to take this opportunity to thank those who contributed to the production of this report. Post-Evaluation Selection and Process Selection of an operation for post-evaluation by EvD uses the following criteria: relevance to the Bank's likely future operations; lessons-learned potential; size of the Bank's investment commitment/exposure; balance among countries of operation; balance among sectors and types of operations; relative priority of investment operation OPERs within EvD's overall work programme priorities and resources. The Bank's post-evaluation process is described in Chapter 8 of the Operations Manual. The responsible Operation Leader first writes a TC PCR. The PCR report serves a self-evaluation function and establishes the basic facts and lessons from the operation's implementation outcome and future prospects. EvD’s independent evaluation follows, using the PCR as one of several inputs.

PRIVATE ROAD NETWORK MANAGEMENT PROJECT (POLAND)

TABLE OF CONTENTS

Page PREFACE ABBREVIATIONS AND DEFINED TERMS ii BASIC DATA SHEET iii 1. INTRODUCTION 1 2. PROJECT RATIONALE 1 3. ACHIEVEMENT OF OBJECTIVES 2 3.1 Objectives 2 3.2 Achievement of objectives 3 3.3 Rating 6 4. OVERALL ASSESSMENT 6 5. TRANSITION IMPACT AND ADDITIONALITY 7 6. BANK PERFORMANCE 8 7. KEY ISSUES AND LESSONS LEARNED 9 7.1 TC project design and implementation 9 7.2 Sector Policy dialogue 10 LIST OF APPENDICES

Appendix 1 Operations Performance Ratings Appendix 2 Transition Indicators for Exante and Exposte Application

i

PRIVATE ROAD NETWORK MANAGEMENT PROJECT (POLAND)

ABBREVIATIONS

BD Banking Department DBFO Design-Build-Finance-Operate ED Environmental Department EU European Union EvD Evaluation Department GDDKiA Directorate of National Roads and Motorways GDDP Generalna Dyrekcja Drog Publicznych (General Directorate for Public Roads) IFI International Finance Institution OCE Office of the Chief Economist (EBRD) OGC Office of the General Counsel (EBRD) OL Operation Leader OPER Operation Performance Evaluation Review OpsCom Operations Committee OT Operation Team PCR Project Completion Report PPP Public-Private Partnership project TC Technical Cooperation TOR Terms of Reference USD United States Dollar XMR Expanded Monitoring Report

DEFINED TERMS

the Bank European Bank for Reconstruction and Development. the Client Ministry of Transport of the Government of Poland the OPER Team Staff of the Project Evaluation Department and the independent sector

consultant who jointly carried out the post-evaluation the Operation Team The staff in the Banking Department and other respective departments

within the Bank responsible for the Operation appraisal, negotiation and monitoring, including the PCR.

the Project The overall objectives of the TCs were to identify under what conditions further commercialisation could assist the Road Administration to improve its operational and managerial efficiency.

ii

OPERATION PERFORMANCE REVIEW PRIVATE ROAD NETWORK MANAGEMENT PROJECT

(POLAND)

BASIC DATA SHEET Operation Code 9903, 30964 Location: Poland Operation: Private Road Network Management Project Sectors: Construction, Transport and Storage Type: Technical Cooperation Facilitators: EU Phare, UK Bank Unit: Transport A. Funding TC TC Commitment Commitment Number. Commitment title Amount (EUR) TC1 ECP 98-2000-06-68 Preparation of Private Sector

Network Management Project I 256,442

TC2 ECP 2000-2002-06-01 Preparation of Private Sector Network Management Project II

979,805

TC3 UKE 2002-11-10 Private sector Road Management Technical Support

45,874

B. Procurement Mode Sources by country Consultant services Evaluation of proposals UK C. Visits Type of Visit No. of Visits Person-days EvD/OPER 1 4

iii

OPERATION PERFORMANCE EVALUATION REVIEW PRIVATE ROAD NETWORK MANAGEMENT PROJECT (POLAND)

1. INTRODUCTION The Bank’s involvement The Ministry of Transport and Maritime Economy and the General Directorate for Public Roads (GDDP) in Poland approached the EBRD (the Bank) to seek assistance in road upgrading, maintenance and rehabilitation works through a public private partnership (PPP) financing approach. Under such an arrangement it was envisaged that firms would compete to undertake the works for a defined period on a portion of the road network, with payments being made on the basis of performance. The EBRD would consider making financing available to the successful firms in order to support the project. Between 1999 and 2004 three TC operations (TC1, TC2 and TC3) were performed in order to prepare a Road Network Management pilot project suitable for EBRD financing. The scope of work of the TC operations was wide and well diversified. It covered the engineering, environmental, legislative, financial, institutional and tendering aspects of the PPP. The main steps were as follows:

• TC1 titled “GDDP: preparation of Private Sector Road Management project’’ was approved in November 1999 by the Bank and contracted between the GDDP and the consultant in June 2002. Its content was essentially technical in nature.

• TC2 had the same title, since it was a continuation of TC1. But, because TC1 had failed, TC2 had slightly different terms of reference (TOR) and was contracted with a new consulting company in July 2002. It had both technical and institutional content.

• TC3 was approved in November 2002 and had a different objective. It was contracted by the Bank in order to further support the Bank in the preparation of the project.

TC1 and TC2 were financed by the EU PHARE Programme for a budgeted amount of €1.5 million. The amount allocated to TC2 was limited to €1 million in order to revise and complete the work from TC1, including support for the tender process, negotiation and contract award. In addition, the Bank needed to receive a high level of expertise and technical support during the project preparation, since the proposed investment project structure was new and untested in the region. TC3 was, therefore, directed at the Bank. It was financed by the UK government. A related investment project for road rehabilitation and maintenance was planned in 2002, but the project was cancelled because the Polish government changed its priorities for the sector. 2. PROJECT RATIONALE The demand for increased road capacity in Poland was growing on average by 6 per cent annually during the period 1990-98. Better road quality was, therefore, required. A survey carried out by the GDDP in 1999 indicated that 70 per cent of the national road network was in “unsatisfactory” or “poor” condition and required renewal or major maintenance in the immediate or very near future. The cost of the maintenance backlog was estimated to be approximately US$1.5 billion. In the meantime, a shortage of state budget financing led to chronic under-funding in the road sector. The foreseen solution was to seek private funds to add to the scarcity of public funds.

of 11 OPER: Private Road Network Management (Poland)

The PPP framework was considered by both parties as a way to bring in private capital and to foster more efficient private sector management practices. The project would rely upon a long-term contract between GDDP and a majority privately owned Road Network Management Company (a special purpose company or a consortium) selected through a competitive tender. Such an arrangement could require debt financing. The debt of the Road Network Management company would then be serviced from the performance related fees to be received from budgetary resources which were allocated regularly to the GDDP.1 Initially at the time the TC was launched, the PPP framework was high on the agenda of the Polish government and supported by the Bank. The objective of the investment project was to provide a commercial solution for and increase efficiency in an industry which was formerly financed from the national budget with no incentives for performance. The TC intended to bring competition and management discipline into national road operations in Poland. It was to assist the public road agency, first the GDDP and then from 2002, the Polish Directorate of National Roads and Motorways (GDDKiA) in reducing the backlog of works and improving the overall standard of the national road network.2 The TC aimed to generate transition impact by facilitating economic activity through improving transport links, increasing private ownership, increasing efficiency through a new system based on competitive tenders and transferring technical and managerial skills to the transport sector. The EU technical cooperation assignment and the direct support of the Bank through the UK grant were critical to the development of the project for national roads.3 The three TCs were, therefore, relevant to the business context in Poland and to the project.

3. ACHIEVEMENT OF OBJECTIVES

3.1 Objectives In TC1 and TC2, the more specific goals were to: (a) prepare a Strategy Report for the Implementation of the Contract; (b) advise and assist the GDDP throughout the tender process and contract award; and (c) support the negotiations and the financial closure.4 The tasks were divided accordingly in three phases. Phase 1 had to include the following elements for the preparation of the Strategy Report:

• Recommendations on engineering aspects: scope of contracts, performance criteria and costs;

• Legislative analysis; • Preliminary environmental analysis; • Financial analysis including the material implications of alternative strategies, the

1 While it was anticipated that EBRD would lend to this project, it was not a given. In practice the concessionaire had to decide how to finance the concession. 2 The GDDP was given a new name in 2002, the General Directorate of National Roads and Motorways (GDDKiA), after a restructuring resulting from the merger of the National Roads Department with the Highways Agency. 3 The Bank’s previous work in Poland on the PPP structure for the A1 motorway had highlighted issues relating to the institutional and legal impediments that needed to be addressed in order to allow the successful use of private investment. (See strategy for Poland [BDS/PO/04/01] 18 May 2004, page 7). 4 Some counterparts in the Ministry of Transport and Maritime Economy were more explicit in their request which was to investigate and learn lessons from the UK’s private Finance Initiative and comparable design-built-finance-operate (DBFO) schemes in Finland and Portugal. This more focused request was, however, not retained in the TOR for the consultants. Rather, the scope of the work for the consultants was expressed in broader terms of PPP through competitive bidding for the works and payments based on the contractors’ performance, leaving room for alternatives to DBFO, such as various forms of contracts of services, management contracts and other lease/concession formulas.

OPER: Private Road Network Management (Poland) Page 3 of 11

preparation of a financial model, recommendations on discount rates, budget implications, availability of project finance and risk analysis;

• Institutional analysis including the impact on the management structure and arrangements for monitoring changes; and

• Tendering plans, standard tender documents and procedures. Phase 2 was to advise and assist the GDDP and later the GDDKiA, through the tender process and contract award, including assistance with the management of the tender process, the determination of evaluation criteria and the evaluation of tenders received. Phase 3 was to support the negotiations and the financial close. This applied to TC1 only. In TC2 it was recognised from the outset that the remaining TC funds were insufficient to reach financial close. The TOR required assistance with negotiations for three months and to financial close only if the client provided additional resources. At the same time the Director General of the GDDKiA sent a letter to the Bank confirming that the GDDKiA would provide the additional resources. TC3 was designed to directly help the Bank prepare an investment project. The task of the consultant was to:

• Critically review the technical and institutional contents of the TC1 and TC2 reports; and • Provide technical and engineering input into the Bank’s preparation for its investment

proposal, especially on the private sectors’ performance risk under the concession agreement and the possible cash flow volatility resulting from the risk allocation between the public sector and the private sector.

3.2 Achievement of objectives

Regarding TC1, the project was approved by EBRD’s TC Committee in November 1999. The Consulting firm was selected through a tender process.5 The list of tasks under the contract covered all components of Phases 1, 2 and 3. The execution of the contract started in June 2000. Phase 1 was only partially completed by the consultants. The poor execution performance of the consulting firm was due to internal management problems and difficult relations with the client. The contract was cancelled by the client on 15 April 2002. Only €256,442 was disbursed from the €1.5 million budget originally approved (please see Table 1). Regarding TC2, the second ranked tendered consulting firm from the tendering in TC1 was engaged to carry out the remainder of Phases 1, 2 and 3 of the project under a new commitment number. The new consulting firm agreed to complete the project within the remaining budget of €979,805. While the TC objectives remained almost the same, the Terms of Reference were amended to reflect the work completed by the previous consultants.6 They also included the changes that occurred in the institutional environment of the road sector in Poland. Meanwhile, the former GDDP and the Agency for Motorway Construction merged into the General Directorate of National Roads and Motorways (GDDKiA).7 5 The consultant consortium consisted of several players, including two legal firms, one of which was based in the United Kingdom and one of which was based locally. 6 See the limitations regarding Phase 3 compared with TC1 in the above description of the TC2 objectives. 7 The consolidation aimed at simplifying the process of managing the development of road infrastructure and streamlining the decision making process. The majority of periodic works then carried out by GDDKiA were executed under a competitive contracts arrangement through joint stock companies that were then divested into in-house units. Urgent routine works were still undertaken by in-house road maintenance units, which reported to the divisional offices of GDDKiA. The balance of these works, along with periodic maintenance, was undertaken through measurement based contracts, typically of a one-year duration (see TOR TC2 – ECP 200-2002-06-01 pages 1-2).


During the execution of the contract in TC2, further changes were made to the TOR regarding the allocation of the remaining funds and the schedule for completing the assignment. The changes reflected delays in the introduction of the enabling legislation, as well as requests by the client for additional project scenarios. As expected, the remaining TC budget became insufficient to complete the tasks. In spite of its earlier commitment, however, the GDDKiA did not bring the additional financial assistance that was required to reach the next steps, up to negotiations and financial close. The legal framework. Under TC2 the consultants prepared a DBFO contractual framework that could have been applied to a road rehabilitation project. The changes in the national laws on public sector contracts that occurred at the end of 2003 could have had a vital effect on the investment project’s legal viability, that is the preparation of a special act on PPP. However, none of these changes solved the two major recurring problems attached to the implementation of a DBFO contract in Poland for national roads: (a) the uncertainty of the scope of tasks that could be entrusted by the Director to the private partner; and (b) the GDDKiA Director’s inability to undertake long-term enforceable obligations. These restrictions prevented the fulfilment of the TC2 objectives set out at the beginning of that year. Additional restrictions were applied to procurement obligations. Guidelines related to ways of eliminating the potential restrictions attached to the PPP draft law were reported under TC2. GDDKiA’s decision in early 2004 to postpone the advisory work regarding the investment project preparation until new legal conditions were known was an additional indication of the inadequacy of the legal framework, which had been modified in 2003. A PPP framework law had been in preparation since 2003. It was approved by the Polish Parliament in mid 2005. There is to be a separate law which introduces changes to existing laws resulting from the PPP law, but to EvD’s knowledge, it has not yet been approved by Parliament. Until this second law is implemented, the above mentioned restrictions still apply in principle. The assignment for TC3 was to directly support the Bank in developing the PPP and the technical components of the investment project based on the findings of TC1 and TC2. The scope of the pilot project was identified together with the technical characteristics. Constructive comments were made on the legal and technical aspects of the previous consulting reports under TC1 and TC2. The preparation of the tender documents, which were already well advanced in TC2, was pursued promptly but, again, the documents were not finalised since the new legislation, which was passed by Parliament in early 2004, did not resolve the fundamental issues of the scope and content of the obligations of GDDKiA under a DBFO contract. The investment project did not materialise because of the lack of an appropriate legal framework and the weak commitment from the government to use the PPP approach to road maintenance for the national road network. Given the current legal situation and given the status of technical preparations to execute the first pilot investment project, it has been impossible to finalise the design of the tendering process and changes into public procurement procedures. The partial achievement of the contract led to a disbursement of €25,660 compared with the €45,874 which had been committed (please see Table 1).


Table 1: TC implementation

TC Amounts (€) TC Budget

approved by Operations Committee

Commitment

Disbursement

Content

TC1 ECP 98- 2000-06-68 256,442

256,442

Review of legislative framework and identification of obstacles to be addressed; preparation of the technical design of the project; preparation of the financial model; confirmation of the contractual framework; tender design; and implementation support.

TC1 and TC2

1,500,000

TC2 ECP2000- 2002-06-01 979,805

979,805

Same as above except for a limitation on the implementation support to be provided under the reduced budget available to the replacement consultants (which was up to €1million, as compared to the €1.5 million available to the original consultants under TC1).

TC3 45,874 UKE-2002-11-10 45,874

25,660

Support Bank project appraisal

TOTAL 1,545,874 1,282,121 1,261,907

Source: Bank data files Follow on investment projects. In relation to this TC, the Bank envisaged an investment project in 2002 of €60 million in which a private sector company would finance the maintenance or the renewal of a portion of the national road network. The project did not materialise, however, because its design was still under development due to the poor performance of the original consultants. A year later the government changed its priorities for the transport sector.8 Similar efforts are currently being deployed by the Transport Team, as clearly stated in the May 2004 country strategy, but nothing has surfaced so far in terms of specific formal steps towards project preparation. 9 TC procurement process. An examination of the tender documents from the archives indicates that the Bank tendering procurement process for consultant hiring was diligently followed in TC1. An additional pre-qualification/qualification exercise was not needed for TC2. The new contract was signed with the second ranked consultant from the TC1 list, after the first consultant failed to perform under TC1.

8 Separately, the same year the Bank staff considered the financing of a National Road and Highway Fund for EUR 250 million. The project preparation was interrupted after concept review. 9 The strategy mentions that “...the Bank will focus primarily on supporting and developing opportunities for private sector involvement in the road sector...” (See strategy for Poland [BDS/PO/04/01] 18 May 2004, page 23).


3.3 Rating

Under TC1, the GDDP failed to provide suitable counterparts to work with the consultant and the consulting firm did not have the appropriately qualified staff for this task. The overall outcome of this TC is assessed as Unsatisfactory. While the problems of the local counterparts inherited from TC1 were resolved in TC2, the legal framework for designing and implementing the project were still not in place and key parts of the consultant’s assignment could not be completed. The TC2 component is, therefore, rated as Marginal. The TC3 component so far remains Marginal as well, since it is still not clear whether the Bank will support investment in road maintenance for Poland and, if it does, whether the implementation will be on a PPP basis. 4. OVERALL ASSESSMENT The objectives of TC1 were relevant to the project and aimed at covering the whole project preparation process up to the financial close. The project was discontinued prior to completion of Phase 1, however, due to the poor performance of the consultant and internal problems affecting the client’s performance. The objectives of TC2 were also relevant and well designed. The government, however, delayed clarifications of the legislative framework and requested additional analysis of project scenarios. As a result, the consultant was only able to prepare a framework for tender documents. The consultant could not finalise the tender documents or assist with the tender process, which was not launched. The objectives of TC3 were relevant to the assignment and aimed at managing the findings of TC1 and TC2 effectively and preparing an investment project. The Bank, however, has not undertaken a project that would activate all the institutional enhancements prepared by the TC studies. The major remaining issue was that the Polish government would not find a contractor or banker to invest in this project unless they were confident of the revenue stream. The lack of certainty on guarantees of payments to the contractor under the legislation, which was modified in 2003, would not give confidence to potential contractors that PPP contracts could work in practice. There were still formal limitations regarding the tasks that the Director General of the GDDKiA could delegate to a concessionaire and in the public procurement obligations which would apply to the concessionaire. Comparison with the Project Completion Report The overall assessment of the TC is summarised below: Table 2: Summary rating table for TC performance

Evaluation Department Evaluation Team PCR (or Progress Report) Rating 1

Achievement of objectives 2

Transition impact 3

Bank handling 2

Overall rating 1

TC 1 Unsuccessful TC 2 Partly Successful TC 3 Successful

Marginal

Marginal

Satisfactory

Unsuccessful

1. Ratings are highly successful, successful, partly successful or unsuccessful. PCR ratings are available for TC1 and TC2 and PR ratings are available for TC3. 2. Ratings are excellent, good, satisfactory, marginal, unsatisfactory or highly unsatisfactory. 3. Ratings are excellent, good, satisfactory, marginal, unsatisfactory or negative.


The major difference in the ratings is mainly due to the continuous assumption in the PCRs that a project would materialise. The evaluation team determined from its field findings that a project is unlikely to materialise on a PPP basis, at least in the immediate future. The Polish authorities appear to have changed their views on the subject, or may never have been fully committed in the first place. The TC assistance efforts are not likely to materialise and, therefore, the overall rating was Unsuccessful. The details of the rating of the overall assessment are provided in Appendix 1. 5. TRANSITION IMPACT AND ADDITIONALITY Given the difficulties faced by the first consultant on the legal aspect of the report and the problems coordinating with the client, the transition impact of TC1 never surfaced. The second consulting firm in TC2, who built on the work of the first consultant, generated some knowledge transfer impact to the local counterparts involved. This expertise could help the client to develop a PPP framework for road maintenance. TC2 carries some remaining potential for a higher level of transition impact. A high risk is attached to the realisation of the potential, however, since it depends upon an investment project and related sector policy decisions that have not been forthcoming for years. Potential impact at corporate level Under TC1 and TC2, the Bank investment was supposed to contribute to the increased effectiveness of the maintenance of national roads and a commercial solution in an industry which was entirely financed from the budget with no incentives for performance. The success of the EU technical cooperation was critical in achieving the transition impact of the project. No real impact can be observed so far, however, because the project has not materialised. Potential impact at industry and economy levels The assignments of TC2 led to some knowledge transfer to the client on how to develop and evaluate PPP structures. The TC provided incentives to develop the road network through PPP financing structures. The DBFO formula was to bring a life cycle solution to the perennial problem of road rehabilitation and maintenance by investing frontward and spreading the payments over time. Other aspects of economic transition were to be achieved by introducing private management of the road network, which should have increased efficiency in the sector and created demonstration effects on what could be achieved if the approach were extended to other public sector services in Poland. In addition, upgrading of the proposed road maintenance network was to create employment locally and provide better road usage, two factors that would also facilitate the transition to a market economy. Again here, none of these aspects materialised through a PPP based investment project. Overall, the transition impact is rated as Marginal. Details on the rating are provided in Appendix 2. Risks to transition Several major risks materialised, some of which were internal to the project. Under TC1 the consultants and the client performed poorly. Other risks were external to the project. At the time the project started, it was uncertain how the legal framework for PPPs on roads would develop in the country. In TC2, the framework for tender documents could not be finalised as long as the enabling legislation that passed the Parliament approval stage was still inadequate. The DBFO contract for the rehabilitation of national roads that could emerge from that legislation would not provide enough comfort to the private sector.


In addition, the Director General of the GDDP and then the GDDKiA was replaced several times and the Deputy Director, who was a strong supporter of the DBFO approach, left the Ministry in the middle of the TC operation. This created a lack of continuity and support in the dialogue on reforms in the road sector between the GDDKiA and the Ministry and the Bank. It also appeared that the motorway construction programme with its concession arrangements, was taking priority over national road projects to which the PPP framework was originally attached. Additionality The segment of the TC project which included the introduction of a PPP framework for the maintenance of the national roads was clearly additional in nature. The private sector had not intervened in that particular segment in the past. Furthermore, it would not do so at any time without a clear legal framework, a well structured project and competitive tendering, elements that only the Bank could facilitate at that stage of early institutional reforms in the sector. Environment The TC was about enhancing an institutional arrangement through a PPP which did not itself generate any significant environmental impact. Indirectly, the related Bank investment project would increase safety, reduce vehicle running costs and improve environmental conditions along the roads covered by the project. The consultant’s report in TC1 recommended, however, that an Environmental Impact Assessment be undertaken as input for DBFO contracts. The Contractor was to investigate the effects on the environment of upgrading existing roads and revising drainage and maintenance procedures.10 In fact no impact, positive or negative, materialised due to a lack of investment. 6. BANK PERFORMANCE In TC1 the Bank remained in close contact with the consultant and the client throughout the assignment. The Bank responded to the client’s concerns as to the quality of the work produced by the consultant.11 The Bank endeavoured to resolve the implementation problems in a timely manner. The major factor preventing satisfactory completion of the project was the internal conflict over PPPs among government authorities, which was outside the control of the Bank. Nevertheless, when it became noticeable at the early stages of the implementation contract that the poor relations between the client and the consultant would seriously compromise the outcome of the TC, the Bank could have intervened rather than waiting until the work was well advanced to press for a cancellation of the contract. The Banking Department did learn a lesson from TC1, which prompted it to seek a higher commitment from counterparts when preparing TC2. The original problems that were encountered with the previous consultants in TC1, namely the failure to provide suitable counterparts to work with the consultant and slow decision making, were resolved under TC2. The client actively participated in the process of revising the TOR and provided all the necessary information. The client was also involved in the selection of the consultant. Since the Bank foresaw at the outset that it would not able to mobilise donor funds of the magnitude needed for such a project in Poland, it had obtained a formal commitment from GDDKiA to provide complementary financing. The client’s attempt to raise the funds, however, proved unsuccessful. The Bank had been ready to 10 The maintenance tasks which were to be investigated included de-icing, snow clearing and road verge clean up. 11 The Client was in fact a diversified group of five key persons within the Government, each one having a different attitude regarding the relevance of a PPP system for national roads in Poland. The group included the Prime Minister, Minister and Deputy Minister of Transport and the Director and Deputy Director of the GDDP.


support the implementation of the project should conditions allow. In TC3, the consultant assisted the Bank in following up on the work accomplished by the EU-funded consultants in TC1 and TC2. Once the concessionaire was to be appointed, the advice of the consultant under the TC3 assignment would have supported the Bank in developing the investment project. But this investment did not happen under TC3 either. In retrospect, it was quite risky to embark on a TC of this magnitude while relying essentially on the support of one Deputy Director and a Minister, who was subject to frequent replacements. Usually it takes more than two persons, even when they are highly ranked in the public administration, to make an institutional client commitment in such circumstances. The EvD Team concludes that, at all stages of TC1 to TC3, the Bank could have better assessed the totality of client support before engaging in a series of large TC expenditures for an operation that never took off. It is acknowledged, however, that the Bank could not have predicted the sudden change of commitment by the client under TC2. In addition, the Bank could have reduced the risk in limiting the scope of the first TC to its institutional component, the PPP framework. This lesson was learned by the banking staff in the development of subsequent TCs for the transport sector. The project files and Donor files for TC1, TC2 and TC3 were easily accessible, quite extensive and presented in good order. 7. KEY ISSUES AND LESSONS LEARNED 7.1. TC Project design and implementation 7.1.1 The commitment of the client The client commitment appears to have had a negative impact on the TC project performance. The client was expected to provide skilled counterpart staff in TC1, but never did and it also failed to provide all the information needed for the consultant’s work. This created obvious difficulties for the consultant in performing the job effectively. Moreover, during TC1 the head administration counterparts at the GDDP, who originally fully supported the project, left the Department and were replaced by less committed counterparts Lesson: Full client commitment is necessary. The Bank should ensure that the client is fully committed to the TC project before the project starts and that experienced staff is in place and willing to support the project throughout its implementation. 7.1.2. Corrective measures by the Bank during project monitoring In TC1, three months after the first contract approval, the consultant expressed concerns regarding the client’s compliance and possible delays which endangered achieving Phases 2 and 3 of the contract. The Bank Team could have intervened at this early stage of the preparation of the report in order to convince the client to comply with the TOR. Likewise, the client could have requested an earlier termination of the contract during TC1 when it became noticeable that the consultant was unable to deal with the legal challenge and that their difficult relationship would not abate, which was long before the complete first draft was out. Instead the contract was cancelled when the draft report was well advanced.


Lesson: The importance of rapid response during monitoring. The Bank should be prompt to react when a TC project appears to go off track. Once the Bank has detected the implementation issues, it should try to solve them quickly and efficiently, before additional major TC expenditures occur.12 7.1.3 The proper sequence of tasks in the TC project design The consultant’s efforts were thinly spread across the preparation of both the technical and institution building components, since he was trying to prepare a project that had never been done in Poland before. While it is fully understandable that the Bank intended to have a project emerge from the TC and, therefore, fully supported the development of the technical component, the major bottleneck appeared to be institutional rather than technical. Changes in the legal framework for PPPs in Poland were difficult to achieve. The consultant went a long way to find out that, in the end, two key legal requirements were missing and preventing the project from going ahead. Lesson: Dealing with institutions first. In complex TCs with large institutional contents where major issues are pending, the Bank should consider tackling the institutional part first rather than committing large additional amounts of funding to technical components for a project that may never came to light. 7.1.4 The case of large consulting teams The two legal teams of the Consulting Consortium in TC1 were unable to collaborate in order to put in place legal arrangements for the PPP that were relevant and that solved outstanding problems. The failed efforts were partly due to a lack of management capabilities within the consortium and partly due to the limited level of monitoring by the Bank and the client. Most probably this could have been avoided by closer monitoring of the consulting work by the Bank. Lesson: Emphasis upon better coordination of consulting teams. It is essential that all companies that are part of a consulting consortium work in the most collaborative way possible and that the leading consultant properly manages the parties of the consortium. The Bank should pay more attention to this aspect of the performance of the consultant, ask for intermediary products and provide immediate feedback on deficiencies. 7.2 Sector Policy dialogue 7.2.1 Policy risk mitigation to enhance project execution The TC project dealt with one major uncertainty. It was unclear how the Polish legal framework would develop in the wake of a PPP for road maintenance. Lesson: A mutual understanding in addressing risks. When a TC attached to an investment project involves a major institution building risk, additional measures for risk mitigation should be taken upfront. To the extent possible, the Bank should: (a) determine the long term political willingness of the government to undertake an investment project carrying innovative and far reaching institutional changes; and (b) obtain a formal commitment to seek the required reforms with the

12 This was more difficult to achieve in this project since the GDDKiA, rather than the Bank, was the contracting party.


support of the TC. This could be done through a letter of understanding. 7.2.2. Changes in government counterparts Changes in policies regarding and by personnel towards the involvement of the private sector in road management at the leadership level of the GDDP and then the GDDKiA and the Ministry of Transport created major obstacles to the successful outcome of the project. Key public administrators who were supportive of the initiative and put in a significant effort at the initial stages of the TC project left to work on other assignments. Ultimately, it was the government’s policy that became an issue for the project. Lesson: Continuity in Bank policy dialogue to improve project success. A continuous policy dialogue is essential when the public administration which implements a TC project on institutional reforms is likely to be subject to political interference. To ensure that the prime minister, the minister and high ranked civil servant policy decisions remain in favour the project, the Bank should maintain regular contact with stakeholders at all key levels of the administration and the government.

APPENDIX 1

OPERATION PERFORMANCE RATINGS PRIVATE ROAD NETWORK MANAGEMENT PROJECT (POLAND)

Performance Indicator

Rating OVERALL TRANSITION IMPACT: Given the problems of the first consultant with the legal aspect of the report and the problems of coordination with the Client, the transition impact of TC1 never crystallized. The second consultant in TC2, who built on the work of the first consultant, transferred some knowledge to the local counterparts. This expertise could help the Client to develop PPP framework for road maintenance through a new investment project. However, the project is not likely to materialise in the near future. (Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative)

MARGINAL

ENVIRONMENTAL PERFORMANCE OF THE PROJECT AND SPONSOR: (Ratings: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Highly Unsatisfactory)

N/A

EXTENT OF ENVIRONMENTAL CHANGE: (Ratings: Outstanding, Substantial, Some, None/Negative)

N/A

ADDITIONALITY: The TC project was clearly additional in this particular segment of the introduction of a PPP framework for the maintenance of the national roads, since the private sector did not intervene in that particular segment in the past and would not do in the future without a clear legal framework and competitive tendering. (Ratings: Verified in all respects, Largely verified, Verified only in part, Not verified)

VERIFIED IN ALL ASPECTS

PROJECT FINANCIAL PERFORMANCE: (Ratings: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Highly Unsatisfactory)

N/A

COMPANY FINANCIAL PERFORMANCE: (Ratings: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Highly Unsatisfactory)

N/A

FULFILMENT OF PROJECT OBJECTIVES: The objectives of TC1 and TC2 were relevant. However, due to a poor performance from the Consultant and internal problems affecting the Clients performance, and to Government delays in clarifying the legislative framework, the consultants were not able to finalise their tasks. The objectives of TC3 were also relevant, but the Bank has undertaken no project that would activate all the institutional enhancements prepared by the TC studies. (Ratings: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Highly Unsatisfactory)

MARGINAL

BANK HANDLING: The Bank remained in close contact with the consultants and client throughout the assignment. The TC should have supported the Bank in developing the investment project. But the Bank could have assessed the overall sector policy environment more realistically before engaging the consultants to provide a PPP framework and facilitate an operation that never took off. (Ratings: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Highly Unsatisfactory)

SATISFACTORY

BANK’s INVESTMENT PERFORMANCE: (Ratings: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Highly Unsatisfactory)

N/A

OVERALL PERFORMANCE: (Ratings: Highly Successful, Successful, Partly Successful, Unsuccessful)

UNSUCCESSFUL

APPENDIX 2

TRANSITION INDICATORS FOR EXANTE AND EXPOST APPLICATION Private Road Management Project (Poland)

TI checklist

categories

STEPS OF RATING TRANSITION IMPACT EX POST

Short-term verified impact

Longer- Term

transition i

Risk to potential

TI STEP I: CHANGE BY THE PROJECT AT

CORPORATE LEVEL

Rating1

Rating2

Rating3

3

Private ownership. The success of the TC was critical to achieving participation of private companies in the maintenance of national roads through a PPP process No real impact can be observed so far, however, because the PPP framework is not in place and operational for road maintenance, and the related investment project has not yet materialised.

Unsatisfactory Satisfactory High

5

Skill transfers. The local counterparts in their collaboration with the consultants known have acquired some knowledge of PPP principles.

Satisfactory Good High

6 Demonstration effects. n.a n.a n.a

7

New standards for business conduct. Although no change in business conduct was observed in the short run, in the longer run, if the related Bank investment is undertaken, it should increase the effectiveness of the maintenance of national roads and offer a commercial solution in an industry which has been financed from budget proceeds with no performance incentives.

Unsatisfactory Satisfactory Medium

STEP II: TRANSITION IMPACT AT THE LEVEL OF THE INDUSTRY AND THE ECONOMY AS A WHOLE

Rating Rating Rating

1

Competition The design of the tendering process and the preparation of tender documents for private firms to enter into a PPP agreement was not finalised due to the absence of an appropriate legislative framework.

Unsatisfactory Unsatisfactory Low

2 Market expansion n.a n.a n.a

3 Private ownership. The technical assistance without a follow on project was unable to help increase the share of private sector participation in the road sector.

Unsatisfactory Satisfactory High

4

Frameworks for markets: The legislative framework allowing PPP in private sector maintenance was enacted in mid 2005, but, to EvD’s knowledge, the subsequent regulations releasing previous constraints on the ability of GDDKiA to handle contracts with the private sector for road maintenance have yet to be adopted.

Unsatisfactory Satisfactory Low

5

Skills transfers. The TC work that included a dialogue on transport sector reforms with high level officials led to a skills transfer that went beyond the scope of the project.

Satisfactory Good High

6 Demonstration effects n.a n.a n.a

7 New standards for business conduct. If undertaken in the long run, a PPP operation for road maintenance could have positive effects on business conduct by creating new management practices that are likely to spread to the other public sector activities where funds are scarce and inefficiencies high.

Unsatisfactory Satisfactory Medium

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Unsatisfactory Satisfactory High

OVERALL TRANSITION IMPACT RATING:4 Marginal 1 This range is: Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative. 2 This range is: Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative. 3 This range is: Low/Medium/High/Excessive. 4 This range is: Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative.

poland private road network management project (tc) [ebrd

Documents