documents.worldbank.orgdocuments.worldbank.org/curated/en/827641493410985947/... · web...

Document of The World Bank

Report No: ICR3703

IMPLEMENTATION COMPLETION AND RESULTS REPORT(IBRD-79870 IDA-42670)

ON A

CREDIT

IN THE AMOUNT OF SDR 14.8 MILLION

(US$22 MILLION EQUIVALENT)

AND ADDITIONAL FINANCING

IN THE AMOUNT OF US$19 MILLION

TO THE

REPUBLIC OF ARMENIA

FOR A

HEALTH SYSTEM MODERNIZATION PROJECT (APL2) IN SUPPORT OF THE SECOND PHASE OF THE HEALTH SECTOR REFORM PROGRAM

August 15, 2016

Health, Nutrition, and Population Global PracticeEurope and Central Asia Region

CURRENCY EQUIVALENTS(Exchange Rate Effective August 15, 2016)

Currency Unit = Armenian Dram (AMD)AMD 475.94 = US$1

US$0.70 = SDR 1

FISCAL YEARJanuary 1 – December 31

ABBREVIATIONS AND ACRONYMS

AF Additional FinancingALOS Average Length of ServiceAPL Adaptable Program LendingCPS Country Partnership StrategyECG ElectrocardiogramEMP Environmental Management PlanFM Financial ManagementGDP Gross Domestic ProductGOA Government of ArmeniaHPIU Health Project Implementation UnitHSPA Health System Performance AssessmentHSRP Health Sector Reform ProgramIFRs Interim Financial ReportsILCS Integrated Living Conditions SurveyIMR Infant Mortality RateM&E Monitoring and EvaluationMOH Ministry of HealthNCDs Non-communicable DiseasesNHA National Health AccountsOECD Organization for Economic Cooperation and DevelopmentOOP Out of PocketPAD Project Appraisal DocumentPEH Public Expenditure on HealthPHC Primary Health CareSC Supervisory CommitteesSHA State Health AgencySMU State Medical University

Regional Vice President Cyril E. MullerSenior Global Practice Director Timothy Grant Evans

Practice Manager Enis BarışProject Team Leader Susanna Hayrapetyan

ICR Team Leader Susanna Hayrapetyan

ARMENIAHealth System Modernization Project (APL2) in Support of the Second Phase of the Health

Sector Reform Program CONTENTS

Data SheetA. Basic Information.........................................................................................................B. Key Dates.....................................................................................................................C. Ratings Summary.........................................................................................................D. Sector and Theme Codes.............................................................................................E. Bank Staff....................................................................................................................F. Results Framework Analysis......................................................................................G. Ratings of Project Performance in ISRs.....................................................................H. Restructuring (if any).................................................................................................I. Disbursement Profile....................................................................................................

1. Project Context, Development Objectives, and Design...............................................

2. Key Factors Affecting Implementation and Outcomes................................................3. Assessment of Outcomes...........................................................................................

4. Assessment of Risk to Development Outcome..........................................................5. Assessment of Bank and Borrower Performance.......................................................

6. Lessons Learned.........................................................................................................7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners...........................................................................................................

Annex 1. Project Costs and Financing...........................................................................Annex 2. Outputs by Component...................................................................................

Annex 3. Economic and Financial Analysis..................................................................Annex 4. Bank Lending and Implementation Support/Supervision Processes........................................................................................................................Annex 5. Beneficiary Survey Results............................................................................

Annex 6. Stakeholder Workshop Report and Results....................................................Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR.................................................................................................................................Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders.....................................................................................................Annex 9. List of Supporting Documents........................................................................

MAP ARM33364...........................................................................................................

A. Basic Information

Country: Armenia Project Name:

Health System Modernization Project (APL2) in Support of the Second Phase of the Health Sector Reform Program

Project ID: P104467 L/C/TF Number(s): IBRD-79870,IDA-42670ICR Date: 08/15/2016 ICR Type: Core ICR

Lending Instrument:Adaptable Program Lending

Borrower:REPUBLIC OF ARMENIA

Original Total Commitment:

USD22.00 million equivalent

Disbursed Amount:USD41.86 million equivalent

Revised Amount:USD40.96 million equivalent

Environmental Category: BImplementing Agencies: Ministry of Health, Health Project Implementation UnitCo-financiers and Other External Partners:

B. Key Dates

Process Date Process Original Date Revised / Actual Date(s)

Concept Review: 01/08/2007 Effectiveness: 06/06/2007 06/06/2007

Appraisal: 01/26/2007 Restructuring(s):03/19/201012/20/201003/26/2014

Approval:03/08/2007 Mid-term Review: 05/17/2010

Closing:

12/31/2012 02/29/2016

C. Ratings Summary C.1 Performance Rating by ICROutcomes: SatisfactoryRisk to Development Outcome: Low or NegligibleBank Performance: SatisfactoryBorrower Performance: Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)Bank Ratings Borrower Ratings

Quality at Entry: Satisfactory Government: Satisfactory

Quality of Supervision: Satisfactory Implementing Agency/Agencies: Satisfactory

Overall Bank Performance: Satisfactory Overall Borrower

Performance: Satisfactory

C.3 Quality at Entry and Implementation Performance IndicatorsImplementation

Performance Indicators QAG Assessments (if any) Rating

Potential Problem Project at any time (Yes/No):

NoQuality at Entry (QEA):

None

Problem Project at any time (Yes/No):

NoQuality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Moderately Satisfactory

D. Sector and Theme Codes Original Actual

Sector Code (as % of total Bank financing)Central government administration 11 11Health 86 86Tertiary education 3 3

Theme Code (as % of total Bank financing) Administrative and civil service reform 17 17 Education for the knowledge economy 17 17 Health system performance 33 33 Injuries and non-communicable diseases 17 17 Rural services and infrastructure 16 16

E. Bank Staff Positions At ICR At Approval

Vice President: Cyril E Muller Shigeo KatsuCountry Director: Mariam J. Sherman D-M Dowsett-CoiroloPractice Manager/Manager: Enis Baris Armin H. FidlerProject Team Leader: Susanna Hayrapetyan Enis BarışICR Team Leader: Susanna HayrapetyanICR Primary Author: Renzo Efren Sotomayor Noel

F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document)The Project Development Objective is to strengthen the Ministry of Health’s capacity for more effective system governance, scaling up family medicine-based primary health care (PHC) and upgrading selected health care service delivery networks in marzes to provide more accessible, quality and sustainable health care services to the population.

Revised Project Development Objectives (as approved by original approving authority)

The PDO was not revised.

(a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1: Health System Performance Assessment report issued once every two years.Value quantitative or qualitative)

First HSPA for 2006 HSPA for 2008, 2010 and 2012 Maintained

HSPA published for 2008, 2010, 2012, 2013, 2014, and 2015

Date achieved September 1, 2007 December 31, 2012 December 31, 2014 February 29, 2016Comments (incl. % achievement)

Overachieved. Originally planned once every 2 years, HSPA is now produced annually since 2012 using government funds.

Indicator 2: National Health Accounts Report published annually. Value quantitative or Qualitative)

First NHA for 2004 A NHA for 2005 and 2006

Produce NHA until 2012

NHA produced annually from 2006 to 2015

Date achieved December 31, 2006 December 31, 2012 December 31, 2014 February 29, 2016Comments (incl. % achievement)

Achieved.

Indicator 3: Public hospitals in project sites have published financial audit reports from Independent Audits. Value quantitative or Qualitative)

0 Indicator introduced during restructuring 13 hospitals 7 hospitals

Date achieved December 31, 2006 December 31, 2012 December 31, 2014 February 29, 2016

Comments (incl. % achievement)

Partially Achieved. Armenia’s 2002 Law on Accounting states that only hospitals with annual revenue exceeding AMD 1 billion (US$2 million) must carry out independent financial audits. Only two hospitals met the requirement and did the audits. Due to HPIU proactivity and support, 5 additional hospitals decided to perform the audits.

Indicator 4: Proportion of Armenian population covered by qualified family medicine practices. Value quantitative or Qualitative)

47% 100% 95% 94.05%



Substantially Achieved. This indicator refers to the proportion of PHC-level professionals re-trained as family medicine practitioners. At the end of the project 94.05 percent of health professionals were certified as family doctors and family nurses. A small proportion of doctors and nurses who had retired or were close to retirement age were not retrained as family medicine



approval documents)




practitioners.

Indicator 5: Increased utilization of (a) outpatient services and (b) inpatient services by the poorest income quintile.

Value quantitative or Qualitative)

a) 3.9%b) 5.0% To be established Increased Utilization a) 5.3%

b) 9.7%

Date achieved December 31, 2010 December 31, 2012 December 31, 2014 December 31, 2014Comments (incl. % achievement)

Achieved

Indicator 6: Perceived quality of care in marzes increased.


64.3% Increased 75%

PHC: urban 64.1%; rural 73.2%.Hospital: urban 73.7%; rural 66.8%.

Date achieved December 31, 2007 December 31, 2012 December 31, 2014 December 31, 2012Comments (incl. % achievement)

Substantially Achieved. Based on HSPA surveys, the indicator showed some improvement. ICR team analyzed other variables related to perceived quality that showed substantial improvements.

Indicator 7: ALOS in marz hospitals decreased close to OECD average. Value quantitative or Qualitative)

7.7 Decreased 6.5 5.8

Date achieved December 31, 2006 December 31, 2012 December 31, 2014 February 29, 2016Comments (incl. % achievement)

Overachieved. OECD average (EU28) for 2014 was 7.8.

Indicator 8: Proportion of patients with (a) hypertension who had at least one ECG annually and (b) ischemic heart disease who had at least one total cholesterol test annually (Indicator introduced during the 2010 Additional Financing and restructuring)


a) 42%b) 33% Increased a) 55.2%

b) 53.7%

Date achieved December 31, 2010 December 31, 2014 December 31, 2014Comments (incl. % achievement)

Achieved.

Indicator 9: Project beneficiaries (Indicator introduced during the 2010 Additional Financing and restructuring)


PHC: 370,000 -PHC: 590 thousand-Hospitals: 2 million

-PHC: 627 thousand-Hospitals: 2.026 million

Date achieved December 31, 2006 December 31, 2014 February 29, 2016


Achieved. Based on the number of population at each marz, the facilities that provided health services at the beginning of the project aimed to reach 590,000 people with the PHC network and 2 million people with the renovated hospitals. At the end of the project, the PHC network serves 627,000 people and the hospital network 2,026,000 people.

Indicator 10: Public hospitals that are supervised by effective Supervisory Committees (SC) (Indicator



approval documents)




dropped during the 2010 Additional Financing and restructuring)Value quantitative or Qualitative)

0% 100%

Date achieved September 1, 2007 December 31, 2012Comments (incl. % achievement)

Dropped. Supervisory Committees’ functions were not fully defined and hospitals did not have the financial resources for SC remunerations.

Indicator 11: Public hospitals that issued Public Performance Reports. (Indicator dropped during the 2010 Additional Financing and restructuring)


0% 100%


Dropped. Armenia’s regulations do not enforce Public Performance Reports or audits. Hospitals did not have the financial resources for implementing these reports or audits. Though the original indicator was dropped, the project supported Independent Audits in project hospitals.

Indicator 12: Budget allocation of the prevention and control of NCDs increased. (Indicator dropped during the 2010 Additional Financing and restructuring)


To be established. To be established.


Dropped. Indicator dropped due to the difficulties to extrapolate a budget line specific to noncommunicable diseases (NCDs) from the overall health sector budget.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline ValueOriginal Target

Values (from approval documents)

Formally Revised Target

Values



Indicator 1: Component 1: Proportion of health budget allocated to PHC (percent)Value quantitative or qualitative)

36.4% 45% 43% 37.2%


Comments (including % achievement)

Not achieved. However, public expenditure on health (PEH) substantially increased from AMD 31 billion in 2005 to AMD 76.6 billion in 2014, even increased in the aftermath of the financial crisis (2009) when GDP severely constrained and growth was negative.

Indicator 2: Component 1: Abortion rate declines (per 100 births) (Indicator dropped during the 2010 Additional Financing and restructuring)

Value quantitative or qualitative)

28.9 Decreased.




Values



Date achieved December 31, 2006 December 31, 2012Comments (including % achievement)

Dropped. The project did not directly contribute to the improvement of this indicator.

Indicator 3: Component 1: Number of certified family doctors trainedValue quantitative or qualitative)

633 1650 1750 1676



Substantially achieved. The remaining number of doctors who did not receive the training was due to retirement or close to retirement age.

Indicator 4: Component 1: Number of certified family nurses trainedValue quantitative or qualitative)

568 1650 1950 1804



Substantially achieved. The remaining number of nurses who did not receive the training was due to retirement or close to retirement age.

Indicator 5: Component 1: Number of new guidelines developed and disseminated (Indicator dropped during the 2010 Additional Financing and restructuring)


18 To be established.


Dropped. This is an input indicator rather than an intermediate outcome indicator.

Indicator 6: Component 1: Number of health facilities constructed, renovated and/or equipped. Value quantitative or qualitative)

25 To be established 100 112



Overachieved.

Indicator 7: Component 1: Health personnel receiving training. Value quantitative or qualitative)

1201 To be established 3700 4118



Overachieved. Total actual value includes family doctors, family nurses, and low, middle and high level management professionals from hospitals.




Values



Indicator 8: Component 2: Reduction of square meters of hospital space (percent of original) in all marzes.Value quantitative or qualitative)

0% Reduction 85% 85%



Achieved.

Indicator 9: Component 2: All project hospitals apply updated environmental management guidelines.Value quantitative or qualitative)

No Yes Yes Yes



Achieved.

Indicator 10: Component 2: Increased number of admissions in project hospitals. Value quantitative or qualitative)

33,7334 Revised during restructuring. 40,000 49,121



Overachieved. This indicator was revised during the 2010 Additional Financing and restructuring due to lack of clarity in the labeling of the original indicator (“All public hospitals in marzes constitute part of a health care network with established gate keeping and referral rules”).

Indicator 11: Component 3: Percent of SHA contract with health services providers are concluded not later than 30 days after budget approval in the Parliament. (Indicator dropped during the 2010 Additional Financing (AF) and restructuring).


0% 100% 100%

Date achieved December 31, 2006 December 31, 2012 February 29, 2016Comments (including % achievement)

Achieved and dropped. The indicator was dropped during the 2010 AF and restructuring because it was achieved. The original target value is sustained.

Indicator 12: Component 3: Management of SHA contracts satisfactory or better to health care providers (percent of respondents). (Indicator dropped during the 2010 AF and restructuring).


To be established To be established.


Dropped. This indicator was planned to be monitored from surveys, however the baseline survey was never carried out; hence due to the inability to compare with the beginning the indicator was dropped.

Indicator 13: Component 3: Proportion of physicians licensed according to new procedures. (Indicator dropped during the 2010 AF and restructuring)




Values




0% 100%


Dropped. The Government of Armenia decided to establish a special agency for licensing physicians. Indicator became out of project scope.

Indicator 14: Component 3: Proportion of key management staff in project hospitals trained on financial management after the results of performed Independent Audits. (Indicator revised during the 2010 AF and restructuring)


0% 100%

Date achieved December 31, 2006 December 31, 2012


Not achieved. Indicator revised and target set with 39 professionals to be trained: 3 persons (1 head physician and 2 financial management staff) for each of the 13 hospitals. However, Independent Audits were not performed as expected; hence hindering the implementation of this indicator.

Indicator 15: Component 3: State Medical University has a revised curriculum in line with EU countries.Value quantitative or qualitative)

No Yes Yes Yes



Achieved. Based on technical support provided by the project, the State Medical University introduced changes in its curriculum, teaching methodologies and student evaluation system.

Indicator 16: Component 3: Monitoring and evaluation system for NCDs is in place (Indicator dropped during the 2010 AF and restructuring)


System does not exist System exists


Dropped. At the time of the 2010 AF and restructuring, the M&E system of the Ministry of Health already covered a broader range of diseases.

Indicator 17: Component 3: Reduction in OOP payments for essential health services.Value quantitative or qualitative)

PHC: 35.4%Hospital: 47.1% Reduction Reduction PHC: 20.6%

Hospital: 42.8%

Date achieved December 31, 2010 December 31, 2012 December 31, 2014 11-Dec-2014


Achieved. Data based on ILCS 2010 and ILCS 2014.

G. Ratings of Project Performance in ISRs

No. Date ISR Archived DO IP Actual Disbursements

(US$, millions) 1 08/20/2007 Satisfactory Satisfactory 0.00 2 10/15/2007 Satisfactory Satisfactory 1.50 3 06/21/2008 Satisfactory Satisfactory 1.76 4 04/02/2009 Satisfactory Satisfactory 4.18 5 08/03/2009 Satisfactory Satisfactory 6.15 6 11/03/2009 Satisfactory Satisfactory 8.41 7 06/14/2010 Satisfactory Satisfactory 13.50 8 01/04/2011 Satisfactory Satisfactory 18.70 9 10/04/2011 Satisfactory Satisfactory 23.31

10 12/03/2011 Satisfactory Satisfactory 24.14 11 06/26/2012 Moderately Satisfactory Moderately Satisfactory 29.33 12 12/03/2012 Moderately Satisfactory Satisfactory 30.66 13 07/06/2013 Satisfactory Satisfactory 33.84 14 03/30/2014 Satisfactory Satisfactory 36.56 15 10/18/2014 Satisfactory Satisfactory 38.19 16 04/08/2015 Satisfactory Satisfactory 39.58 17 09/04/2015 Moderately Satisfactory Satisfactory 41.15 18 02/12/2016 Moderately Satisfactory Satisfactory 41.83

H. Restructuring (if any)

Restructuring Date(s)

Board Approved PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes MadeDO IP

03/19/2010 No S S 11.49

Level 2 restructuring to include the rehabilitation of merged hospitals, construction of a new hospital, and provision of medical equipment.

12/20/2010 No S S 18.70

Additional Financing and Level 2 restructuring. New hospitals included in investment plan (infrastructure). Results framework revised and extension of Closing date from December 31, 2012 to December 31, 2014 for accommodating the new hospital investment plan approved.

03/26/2014 No S S 36.50

Level 2 restructuring. Extension of Closing date from December 31, 2014 to February 29, 2016 for completing civil works in hospitals.

I. Disbursement Profile

1. Project Context, Development Objectives, and Design

1. The Health System Modernization Project (APL2) in Support of the Second Phase of the Health Sector Reform Program was approved on March 8, 2007. The Financing Agreement was signed on March 9, 2007, and the credit became effective on June 6, 2007. The project was the third health project supported by IDA.1 The project was the second phase of a two-phase Adaptable Program Lending (APL) to support the Government of Armenia (GOA) Health Sector Reform Program (HSRP).2 A level 2 restructuring was approved on March 19, 2010 to include the rehabilitation of merged hospitals and the construction of a new hospital. An Additional Financing (AF) with a restructuring was approved on December 20, 2010. The Loan Agreement was signed on January 26, 2011, and the loan became effective on June 9, 2011. Finally, a level 2 restructuring was approved on March 26, 2014 to extend the Closing Date from December 31, 2014, to February 29, 2016.

1.1 Context at Appraisal

1 The first health supported operation was the Health Financing and Primary Health Care Development Project approved on July 29, 1997, for US$10 million equivalent.2 The Health Modernization Project (APL1) in support of the first phase of the Health Sector Reform Program for US$19.0 million was approved on June 10, 2004, and closed on June 30, 2009.

1

2. Economic growth and macroeconomic management were strong in Armenia. Gross domestic product (GDP) growth averaged over 10 percent per year during 2001–2006, reaching 13.9 percent in 2005 and 13.2 percent in 2006. Prudent macroeconomic policies maintained sustainable external and internal balances, kept inflation low, and reduced Armenia’s debt burden. The fiscal deficit was low and Armenia was on track with its International Monetary Fund Poverty Reduction and Growth Facility Program.

3. Armenia also was on target to achieve most, if not all, of its Millennium Development Goals by 2015. Armenia saw a significant reduction in overall poverty, with the proportion of poor declining from 51 percent in 2001 to 30 percent in 2005, and extreme poverty reduced even faster, from 16 percent in 2001 to below 5 percent in 2005. Despite changes in the methodology to calculate poverty rates during that time period, there was a clear trend towards poverty reduction. Similarly, infant mortality and maternal mortality decreased rapidly. The infant mortality rate (IMR) fell from 15.6 to 12.3 per 1,000 live births between 2000 and 2004. During the same time span, the maternal mortality ratio fell from 52.5 to 16 per 100,000 live births.

4. At the same time, Armenia was also in the midst of an epidemiological transition characterized by a decline in communicable diseases and an increase in the prevalence of chronic diseases. In 2005, the leading causes of premature death under the age of 65 were diseases of the circulatory system (hearth disease, stroke, and related conditions); cancer; external injuries and poisoning; and diseases of the respiratory and digestive systems.

5. Access to and use of health services were low, favoring polyclinics and hospitals over primary health care (PHC) facilities. There was a concern that the sick may postpone seeking care and use of services as a result of lack of resources, high out-of-pocket (OOP) payments, and low perceived quality of care, especially in rural areas. In 2003, for instance, the percentage of individuals who did not seek care when ill or injured was on average 70.5 percent, varying between 62 percent for the top quintile and 78 percent for the lowest quintile. As for the OOP informal payments, they were mostly paid in hospitals; in 2001, about 72 percent of those who sought health care in a hospital and about 60 percent of those who sought care in a polyclinic reported having made informal payments averaging AMD 20,000 (approximately US$40) and AMD 6,700 (approximately US$13), respectively, which are quite high figures with significant impoverishing effects on the household. In 2003, OOP payments accounted for 62.4 percent of the total health expenditures, 93 percent of which were informal payments.

6. Despite budgetary increases in nominal terms, the health care system remained underfunded and its resources were poorly pooled and inequitably used. Taxes and mandatory social insurance contributions constituted the main source of tax revenues for the Government through which budgetary obligations to the health sector were financed. This, however, constitutes only a small share of total health expenditures in Armenia. In 2005, according to the World Health Organization, public spending on health was US$67.9 million (1.4 percent of GDP and 7.4 percent of the government budget) while capital private spending for the same year was equal to 3.9 percent of GDP.

2

7. Armenia was in the process of developing a decentralized and pluralistic health system. The Ministry of Health (MOH) has increasingly been involved in policy making, defining broad strategies, planning, and regulation while leaving service delivery to local authorities and municipalities. All health facilities in Armenia are now joint-stock companies, with marz authorities and municipalities holding a large share of stocks of, and operating, hospitals and polyclinics. Moreover, payments to health care providers are now managed by the State Health Agency (SHA), a semiautonomous agency within the MOH. All health facilities are reimbursed based on a reimbursement rate for the services included in the state-funded basic benefit package, although they are free to charge patients for those services not covered. Once budgets are allocated to state programs and payments are made based on contracts with the SHA, health facilities have the autonomy to manage their own financial and human resources.

8. While Armenia was relatively well endowed with health professionals, the gradual decrease in the number of nurses, the relatively higher number of specialists, and geographic distribution of health care workers are of concern. Not only was the physician/nurse ratio suboptimal for adequate provision of services, but because of the oversupply of specialists and the fact that a relatively high percentage of physicians (44 percent) work in hospital settings, PHC services remain inadequately covered as well, especially in rural areas. The large-scale training of family physicians that started with APL1 aimed at addressing this issue by training family physicians and family nurses to provide PHC services, mainly in rural areas. Finally, formal medical education is provided by the Yerevan State Medical University (SMU), which graduates about 400 physicians a year, and hence there is the requirement to reform the training curricula and state medical exams to bring their training programs up to par with the European Union standards.

9. Armenia’s post-Soviet transition efforts to reform its health system have been successful and they needed to be scaled up. Compared with many other countries of the former Soviet Union, Armenia has been very successful in reducing its hospital capacity and nonmedical staffing, mainly through closure of small rural hospitals, reduction of beds, and attrition. These efforts started in 1997 with the approval of the first health operation supported by the World Bank: the Health Financing and Primary Health Care Development Project. The transformation continued with the GOA HSRP supported by the Health Modernization Project (APL1), the first phase of the APL. The HSRP was designed and financed with a view to support rationalization of health facilities and the introduction of family medicine as an organizational model for the provision of PHC services. As a result, the Government with the support from APL1, was successful in effectively reducing the large number of hospitals and consolidating networks in the city of Yerevan, with the subsequent elimination of duplicative departments and services, reduction of surface areas, and significant productivity and efficiency gains, without any compromise in access to and quality of care. Indeed, both have improved in the hospital mergers supported under APL1. Based on APL1 positive experience, the GOA decided to continue with the HSPR and approved a Marz Hospital Optimization Plan to implement similar measures in the remaining marzes outside Yerevan. APL2 intended to support the second and final phase of the GOA HSPR and its Marz Hospital Optimization Plan. This was a politically sensitive and technically challenging process, thereby requiring a timely follow-through with major investment in the facilities so as not to lose momentum.

1.2 Original Project Development Objectives (PDO) and Key Indicators

3

10. The development objective of APL2 was to strengthen the MOH’s capacity for more effective system governance, scaling up family medicine-based PHC, and upgrading selected health care service delivery networks in marzes to provide more accessible, quality, and sustainable health care services to the population. The wording of the PDO was the same in the main text of the Project Appraisal Document (PAD), annex 3 of the PAD, and the Financing Agreement.

11. The five key indicators as identified in the main text of the PAD and annexes are:

Population fully covered by qualified family medicine practices.

Key health sector quality and efficiency indicators improve in rural areas.

A culture of evidence-based impact assessment is established through the institutionalization of key health policy-monitoring documents—Health System Performance Reports and National Health Accounts (NHA) reports.

Public hospitals complete the transformation of their governance structure and make routine use of Supervisory Committees and Independent Auditing practices, for improved management, transparency, performance, and efficiency.

A gradual increase in funding for, and utilization of, preventive services for the control of noncommunicable diseases (NCDs) (for example, tobacco control, mammography, high blood pressure, diabetes, and Pap smear).

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification

12. The PDO was not revised, but the Results Framework was revised during the 2010 project restructuring as part of the Additional Financing. One IDA core indicator related to the number of beneficiaries was added. One indicator related to budget allocation for NCDs was dropped due to difficulties in extrapolating a related budget line from the overall health sector budget. However, indicators about progress on NCDs management were retained. The indicators related to implementing Supervisory Committees and issuing Public Performance Reports at the hospital level were dropped due to the undefined functions of the Supervisory Committees and the merger of public performance reporting into the PDO indicator on public hospitals publishing financial audit reports. The remaining indicators were revised for better monitoring with targets adjusted to meet the new Closing Date of December 31, 2014.

1.4 Main Beneficiaries

13. APL2 aimed to benefit the Armenian population living in the marzes that were included in the project. Based on the number of population at each marz, APL2 aimed to benefit 590,000 people with the strengthening of the PHC network and 2 million people with the renovation of hospitals.

1.5 Original Components

4

14. Component A: Family Medicine Development (US$4.70 million). This component was to support the strengthening of institutional capacity to train well-qualified family physicians and nurses as first-line PHC providers and improve their working environment. Under this component, the project planned to complete the (re)training of 1,650 physicians and an equal number of nurses to ensure 100 percent population coverage based on the ratio of about one team per 1,700 to 2,000 population. In addition, about 50 rural ambulatories were planned to be upgraded and outreach activities conducted to ensure community participation.

15. Component B: Hospital Network Optimization (US$20.77 million). This component aimed at implementing the optimization plans in the remaining eight marzes by upgrading selected hospitals and refurbishing them with modem medical, information technology, and health care waste management equipment. In addition, under this component, the project was to finance technical work for architectural design and training in hospital management, quality assurance, accountability and fiduciary management arrangements, and hospital care waste management.

16. Component C: Institutional Strengthening (US$2.58 million). This component was to strengthen the MOH’s capacity for policy making, planning, regulation, human resources development, and monitoring and evaluation (M&E) for more effective system governance and control of NCDs. It also aimed at strengthening the governance and management structures of health care facilities and the oversight function of marz administrative structures. Support was also to be made available to strengthen SHA operations, improve costing of publicly financed services, and reimbursement mechanisms. In addition, under this component, the SMU was to benefit from consultancy services to upgrade its medical curriculum, improve its teaching and training facilities, and introduce new technologies for continuous medical education.

17. Component D: Project Management (US$1.57 million). This component was to provide institutional support to the MOH through the Health Project Implementation Unit (HPIU), which was in charge of implementing day-to-day project activities and M&E. This component was to finance annual financial audits as well as training and operating costs of the HPIU, including the costs of core and short-term HPIU staff fees, office-related expenses and M&E of project implementation and performance.

1.6 Revised Components

18. Components were not revised.

1.7 Other significant changes

19. Additional Financing (AF) and restructuring of the Result Framework. An AF was requested by the GOA by letter of March 22, 2010 to scale up APL2 activities while keeping the same PDO, design, and implementation arrangements. The additional loan planned to finance the following:

Expand the Marz Hospital Optimization Plan by adding six hospitals:

o Meghri and Kapan hospitals in Syunik marz

5

o Abovian hospital in Kotayk marz

o Alaverdi and Vanadzor hospitals in Lori marz

o Berd hospital in Tavush marz

Build a new hospital in Gyumri, the second largest city in the country, instead of renovating the three severely dilapidated ones—a decision reviewed by the World Bank and found acceptable

Expand hospital buildings in Gavar and Goris

Procure modern biomedical and information technology equipment and furniture for the hospitals, which was not included in the original project’s design

Cover an increase of over 35 percent in construction costs as compared to appraisal estimates

Train medical staff on clinical case management and provide clinical reference materials to enhance the quality of hospital services

20. The Results Framework was restructured as explained in section 1.3. At the time of AF approval (December 2010), APL2 implementation progress and development objectives were rated as Satisfactory. The AF supported all the components of APL2 and hence changed the allocation of funds and schedule. The final allocation of funds is presented in table 1.

Table 1. APL2 and AF costs by Component (US$ Million Equivalent)

ComponentsOriginal Cost from APL2 Cost from AF Total Revised Cost Total %

IDA GOA IBRD GOA IDA + IBRD GOAA. Family Medicine Development

3.53 1.17 4.09 1.36 7.62 2.53 10.15 18.47

B. Hospital Network Optimization

15.58 5.19 12.75 4.25 28.33 9.44 37.77 68.74

C. Institutional Strengthening 1.70 0.88 1.69 0.55 3.39 1.43 4.82 8.77

D. Project Management 1.19 0.38 0.47 0.16 1.66 0.54 2.20 4.00

Total Project Costs 22.00 7.62 19.00 6.32 41.00 13.94 54.94 100

Source: APL2 Project Paper for AF and restructuring, November 2010.

21. Finally, a project restructuring of the AF was approved on March 26, 2014 to extend the Closing Date from December 31, 2014, to February 29, 2016. The purpose of the extension was to allow for civil works activities at the Meghri, Kapan, and Chambarak Medical Centers to be completed as part of the Hospital Network Optimization component.

2. Key Factors Affecting Implementation and Outcomes

6

2.1 Project Preparation, Design, and Quality at Entry

7

22. The ICR team rated project preparation, design and quality at entry as Satisfactory on the basis of the following features:

23. Soundness of the background analysis. APL2 preparation was based on an extensive review of health care reforms in the transitional Commonwealth of Independent States countries that showed the need to: (a) enhance allocative efficiency by reorganizing PHC; (b) invest in human resources in a strategic manner; (c) rehabilitate health facilities; and (d) strengthen the provider’s managerial capabilities. All these lessons learned were incorporated into the APL2 objective and components. In addition, another set of lessons learned were extracted from the design and implementation of World Bank-supported projects in the Eastern Europe and Central Asia Region during the previous 10 years and from the APL1 experience. Lessons were clearly documented in the PAD and incorporated into APL2. Finally, APL2 benefited from the analytical work prepared for the Europe and Central Asia Region and the country, such as the Country Assistance Strategy (2004) and the Review of Experience of Family Medicine in Europe and Central Asia (2005).

24. Assessment of project design. APL2 was the second phase of a two-phase APL to support the Government’s HSRP. APL1 had a positive story, particularly on the issue of hospital optimization, achieving fully or partially the trigger indicators that allowed the GOA to request APL2 even before the closing date of APL1. Hence, APL2 was designed based on APL1, maintaining the overall objective and its components, while improving the indicators and its scope of intervention. Therefore, the amount for APL2 was increased from the originally envisaged US$11 million equivalent to U$22 million equivalent. In addition, the APL investment vehicle was appropriate, given the scope of reforms to be pursued and the GOA’s and World Bank’s commitment to work together in further developing the sector. Finally, APL2 objectives and components were in accordance with Pillar 3 ‘reducing non-income poverty’ of the Country Assistance Strategy FY05–08 (Report No. 28991).

25. In addition, the World Bank team considered other alternatives, such as maintaining the original timetable of APL1 and requesting an AF. However, after internal consultations, the country and the Bank decided to pursue with APL2 to take advantage of the positive political climate toward a new operation generated by the strong results of APL1 and the size of the planned new investment.

26. Adequacy of the Government’s commitment. The GOA had a solid and long-term commitment to reform its health sector. The health sector reform process was launched in 1996, and during the last decade before APL2, the Government maintained the objective of increasing the utilization of essential health services, closing the gap between the rich and poor and quality and efficiency improvements. This commitment was key to partially or fully meeting the trigger indicators during APL1 that allowed APL2 to be elaborated and finally approved. In addition, despite the technical and political difficulties involved in this kind of reform, the GOA could show a proven track record to optimizing health facilities, implementing hospital mergers and networks effectively. The GOA enacted a decree regarding the consolidation of 24 public hospitals into 10 networks. Encouraged by the positive results in Yerevan, the GOA prepared and approved the master plan for the optimization of the hospitals and networks in the remaining marzes.

8

27. Assessment of risks. According to the PAD, there was only one substantial risk from outputs to objective: “Government and local authorities and staff show reluctance to revise/amend optimization plans.” There was a high degree of commitment by local marz authorities and health workers as evidenced in the finalization of the marz optimization plans that were ratified by the GOA in November 2006. However, these plans needed further refinement, and hence the GOA provided firm assurances that amendments would be made, when needed.

2.2 Implementation

28. The project was implemented by a project implementation unit, the HPIU, with extensive experience in introducing new health policies in Armenia. The Government designated the MOH as the responsible agency for the project. The HPIU, the unit within the MOH that oversaw the implementation of APL1, continued to do so for APL2. The implementation arrangements under APL2 were the same as for APL1 for continuity in implementation. The HPIU was responsible for the fiduciary aspects of the project and provided technical as well as administrative support to the MOH departments and agencies that were responsible for project activities. During the 15 years of its operation, including the first IDA-financed PHC development project, the HPIU has gained considerable experience and acquired capacity in project management. The unit was highly effective in overseeing day-to-day project activities and in being fully compliant with fiduciary requirements.

29. A transparent and participatory approach during the implementation was a key factor to overcome the politically sensitive hospital modernization process and to complete all project components. There was a Steering Committee composed of representatives from key stakeholders within and external to the MOH that provided overall oversight and supervision for the project. The Steering Committee mainly comprised the Minister of Health; First Deputy Minister of Health; First Deputy Minister of Finance and Economy; Deputy Minister of Territorial Affairs; Head of the Medical Care Provision Department; and Head of the HPIU. The committee also provided advice on the terms of reference for various assignments, participated in technical evaluations, and worked directly with consultants during the implementation of their assignments.

2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization

9

30. Design. A comprehensive set of PDO and intermediate outcome indicators was included in the PAD, which effectively captured project objectives and components. Although the final number of indicators was extensive (nine PDO and twelve intermediate outcome indicators), the HPIU monitored and systematically reported on them. When necessary, indicators were fine-tuned.

31. Implementation. The Government’s HSRP, supported by APL2, included the strengthening of the Health Information and Analytical Center of the National Institute of Health. Due to this support, the National Institute of Health has been able to regularly produce Armenia’s Health System Performance Assessment (HSPA) and Armenia’s NHA reports. HSPA and NHA reports were produced by a comprehensive set of methods: analysis of health status and health care utilization indicators constructed from routine administrative data; analysis of data from existing surveys performed on an ongoing basis by the National Statistical Service (for example, Integrated Living Conditions Survey, ILCS) or customized modules attached to such surveys; and design, implementation, and analysis of additional surveys of health care users and providers. Special attention was given to the assessment of equity issues related to health and health care and trends in OOP informal payments and evaluation of impact of health programs and policies on the poor.

32. The HPIU was successful in coordinating with several areas at the MOH to monitor and evaluate regularly the project progress indicators. Advancement on indicators was issued regularly and reported quarterly through progress reports. The regular and close monitoring of progress allowed fine-tuning of the indicators when needed, as reflected in the Aide Memoires and the approved restructuring papers.

33. During implementation, in total three PDO indicators were dropped (Supervisory Committees at hospitals, Public Performance Reports for hospitals, and budget allocation for chronic diseases). In the case of the hospital related indicators, they were dropped because there were legal and financial constraints to implement them, however they covered key aspects of the health reform. In the case of the intermediate outcome indicators, in total five were dropped for very different reasons, however, they did not cover vital aspects of the health reform.

34. Utilization. The regular M&E of the project’s Results Framework was useful not only to keep adequate track of implementation progress but also to inform important policy decisions. A good example is the case of the Marz Hospital Optimization Plan. The GOA—based on the progress indicators and positive results in areas such as the evolution of efficiency gains and increase in productivity—decided to modify the original plan and expand it to include six new hospitals. In addition, the M&E of the indicators was a key input for preparing the AF.

2.4 Safeguard and Fiduciary Compliance

10

35. Safeguard implementation is rated Satisfactory. The project triggered the safeguard policy Environmental Assessment (OP/BP 4.01); it was rated B in the environmental category (as in APL1). An environmental analysis was conducted during APL1 to identify potential adverse environmental impacts that were found to be restricted in scope and severity. As a result of the analysis, measures were designated to mitigate the risks in design, planning, and construction supervision process as well as during the operation of the health facilities. Based on the analysis and recommendations, an Environmental Management Plan (EMP) was prepared. After the review of the EMP, it was updated and continued to be valid during APL2. In addition, draft national guidelines, supported by the World Bank, on health care waste management were developed and adopted by Decree No. 03-N of the MOH of April 3, 2008. To date, compliance with the EMP has been satisfactory.

36. Financial management (FM). The financial management (FM) arrangements under the project (including accounting, reporting, budgeting, internal controls, funds flow, and staffing) has consistently been rated no less than Satisfactory during the project’s whole duration except for the last FM mission in November 2015, during which deficiencies in internal control were observed. These were related to delay in the recovery of the advance provided to the contractor for the construction of the Meghri MC, which led to a downgrade of FM rating to Moderately Satisfactory. The level and timeliness of government co-financing was satisfactory. The HPIU produced quarterly interim financial reports (IFRs) for the APL2 project, which were always received on time and found acceptable. The auditors issued unmodified (clean) opinions on the project’s annual financial statements, which were timely received. The Borrower complied with public disclosure requirement for the audited financial statements of the project3.

37. Procurement management. The HPIU was in charge of procurement for APL2 and employed three professionals (procurement specialist) for this task. The Procurement Unit at the HIPU updated the Procurement Plan as required. Due to the scope of activities under APL2, the Procurement Plan included civil works, biomedical equipment, furniture, office equipment, medical supplies, and consulting services. All these items were procured following various types of procurement methods according to the Credit and Loan Agreements. The HPIU managed procurement effectively, and procurement was rated Satisfactory during the life of the project.

2.5 Post-completion Operation/Next Phase

38. Based on the positive results and experience of APL2, a new project was prepared and is currently under implementation: the Disease Prevention and Control Project for a credit amount of US$35 million equivalent (Cr. 5222-AM). Its PDO is to improve (i) Maternal and Child Health services and the prevention, early detection, and management of selected noncommunicable disease at the primary health care level; and (ii) the efficiency and quality of selected hospitals in Armenia.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design, and Implementation

3 The disclosure was required only for IBRD Loan 7987. There was no public disclosure requirement for IDA Credit 4267.

11

39. The relevance of objectives, design, and implementation arrangements are rated Substantial. At the time of the ICR mission, as was the case during project preparation and approval, important priorities of both the Government and the World Bank were to support the Government’s health reform program to strengthen the MOH’s capacity for more effective system governance, scaling up family medicine-based primary health care, and upgrading selected health care service delivery networks in marzes.

40. According to the Country Partnership Strategy (CPS) for Armenia FY14–17 (Report No. 81647-AM), improving efficiency and targeting of social services, health care in particular, is a key objective. The CPS aims to consolidate development progress achieved to date, and health is recognized as an area with strong ownership and implementation capacity. Within this framework, there is a clear line of sight between the World Bank's support and its contribution toward boosting shared prosperity and reducing poverty. Therefore, APL2 is consistent with the country current development priorities and current World Bank country and sectoral assistance strategies and corporate goals.

3.2 Achievement of Project Development Objectives

12

41. The achievement of the PDOs is rated Substantial. The PDO covered different areas of the health system: governance, quality, efficiency, utilization, PHC and hospital networks. The analysis of the progress made in these different areas is presented in the discussion below. In addition, the ICR team also took into consideration the achievement of the outcome indicators (eight out of nine outcome indicators are fully met) and project intermediate indicators (ten out of twelve indicators were achieved or overachieved).

42. MOH’s capacity for more effective system governance has been strengthened. Decision-makers around the world face several challenges when they aim to make appropriate choices to improve the performance of their health systems. The way health systems are organized to collect resources, pooling risk, and deliver services has a profound influence in the health outcomes of the population. However, until recently, these critical decisions have been based on case studies and, sometimes, ideology. Case-studies are useful, but health systems and cultures all differ in many ways. Then, information and its permanent analysis is key for scientific evidence-base policy decisions. In this context, APL2 successfully continued its support to two well-known international analytic tools introduced in Armenia by APL1: Health System Performance Assessment (HSPA) and National Health Accounts (NHA). HSPA provides conceptual framework for the measurement and comprehensive evaluation of the different facets of health systems. NHA uses an internationally recognized methodology to track all health spending in a country regardless of the entity or the institution that finance and managed that spending. The introduction of these tools aimed to strengthen the evidence-based policies and system governance.

43. In regards to HSPA, with APL1 support, Armenia conceptualized and produced the first report in 2007. And the second in 2009. With APL2 the goal was to continue producing the report every two years. Therefore APL2 supported the 2011 report. However, due to its importance for evaluating and monitoring progress for the health sector, since 2012 the Government decided to produce it yearly adding government funds. Similarly to HSPA, the MOH successfully produced NHA reports from 2004 to 2009 (one per year was APL1 goal). APL2 continued this positive trajectory. Using APL2 and government funds, the MOH was able to produce NHA reports every year until 2015. Furthermore, in addition to the Armenian language version, an English version has also been produced for both HSPA and NHA. All the evidence gathered through the HSPA and NHA had an impact on policy decisions. For instance, the consolidation of budgets in consolidated health facilities, the increase in the unit cost (case reimbursement) for the services provided in regional hospitals, the introduction of co-payments (based on the finding of a costing study), and the introduction of global budget in hospitals (a policy measure to control costs). Furthermore, based on the HSPA findings about screening programs coverage at PHC level, the Government decided to invest and expand the screening measures for hypertension, diabetes and cervical cancer; and introduce performance based financing at PHC level. All these are important policy decisions made based on the evidence provided by these two instruments. Today, the MOH has a team of well trained professionals that are using the latest guidelines and methodologies for producing evidence for better decision making and therefore governance of the system.

13

44. Progress was made in hospitals governance. APL2 not only aimed at increasing hospital utilization in marzes as discusssed below, but it also tried to introduce modern hospital management tools for increasing hospital governance, efficiency and transparency. Hospitals are usually strong stakeholders in any health system. In the particular case of Armenia, hospitals legally are State Closed Joint-Stock companies. Usually marz governments approve hospital annual plans and provide regular supervision of the performance. Accountability to the MOH and the public is not strong. Therefore, APL2 planned to introduce Public Performance Reports, Independent Audits, and Supervisory Committees (SC) at each hospital. The Bank team assessed that it was probably premature to introduce these reforms at hospital level, however, after evaluating the risks and benefits, the team decided to pursue them. At the end of the project, despite its efforts to introduce these new measures, including an international consultancy to support the development of the Charter for Independent Boards, these new procedures were not fully implemented. First, as mentioned earlier, in the case of Armenia, hospitals have very strong stakeholders and there was not enough leverage at that time for the MOH to pursue these changes. Second, although the formation of SC was approved, their functions were not fully defined. Third, hospitals lack the financial resources to cover Board members’ remunerations.

45. Similarly, Public Performance Reports were not implemented due to their costs and that regulations did not enforce them. In the case of the Independent Audits, they intended to be a mechanism for providing feedback to improve hospital management. However, the implementation of Independent Audits faced two problems: (a) Armenia’s 2002 Law on Accounting states that only companies with annual revenue exceeding AMD 1 billion (US$2 million) or with balanced value of assets exceeding AMD 1 billion (as of the end of the year) must carry out independent financial audits; and (b) new regulations approved by the Government in 2011, motivated by the financial crisis, stated that all technical assistance and training were no longer eligible to be financed out of external funds. Two hospitals, out of thirteen included in the project, were above the AMD 1 billion threshold and therefore produced the Independent Audit report: Hrazdan Hospital in Kotayq Marz and Gyumri Hospital in Shirak Marz. Due to HPIU proactivity and support, five additional hospitals with revenues below the threshold (Ararat, Aparan, Gavar, Ijevan, and Alaverdi) decided to perform the audits using local resources. In total, despite the legal and financial difficulties, seven out of thirteen hospitals prepared the Independent Audits. The political economy of hospital management, in addition to legal and financial constraints, reduced the leverage from the MOH and the project to introduce new modern management instrument at hospitals. Initiatives such as the implementation of Supervisory Committees and Public Performance Reports could not be implemented, while there was some progress with Independent Audits.

14

46. Primary health care based on the tenets of family medicine has been scaled up and strengthened. A fundamental aspect of any health system is a strong primary health care (PHC) front line and capacity of its health care professionals. Therefore, a key aspect of the reform in Armenia was the introduction of a PHC model based on family practitioners. APL1 did a massive effort for retraining health professionals at PHC level -in this case doctors and nurses- as family practitioners. A task that was successfully continued and completed under APL2. This retraining program was developed by the MOH in collaboration with the SMU and supported by the project. It was a one-year specialization course. PHC doctors and nurses who underwent training became certified family medicine practitioners (family doctors and family nurses, respectively). At the end of APL2, 94.05 percent of health professionals were certified as family doctors and family nurses. A small proportion of doctors and nurses who had retired or were close to retirement age were not retrained as family medicine practitioners. This human resource policy clearly shows the coherence and consistency between APL1 and APL2, and how APL2 built upon the successful efforts of APL1.

Table 2. Health professionals retrained as family medicine practitioners

APL1 APL2 + AF Total Target PercentageCertified family doctors 1,082 594 1,676 1,750 95.77

Certifiedfamily nurses 988 816 1,804 1,950 92.50

Total 2,070 1,410 3,480 3,700 94.05Source: HPIU administrative reports, 2016.

47. The ICR team found supplementary information about enrollment in the health sector and ambulatories that support the objective of scaling up family medicine services. In 2006, Armenia started an enrolment pilot to provide free PHC, which became a national policy in 2011. Today, according to United Nations Population Division, Armenia has a population estimated at 3,006,154, of which 2,996,627 (99.68 percent) are enrolled in and have free access to a PHC facility. In addition, the number of health facilities constructed or renovated and equipped by the end of APL2 reached 173 (largely surpassing the original number of 100 health facilities). The training of health professionals, a very high enrollment rate, and increasing the number of health facilities all supported the scaling up of family medicine services.

48. Access to health care services has improved at both outpatient and inpatient levels, in particular for the less advantaged segment of the Armenian population. PHC is the first and most important line of health services to shield the population’s health. In 2003 (the year before APL1 was approved), per capita PHC visits were 2.0. Since then, visits have continuously increased year after year. By 2013, per capita PHC visits doubled to 4.0. Similarly, per capita hospital admissions per year in Armenia significantly increased, almost doubling, from 6.9 in 2003 to 12.3 in 2013. The access of the general population to both outpatient and inpatient health care services has remarkable increase, almost doubled.

15

Figure 1. PHC per capita visits (2003–2013)

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 20130.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

2.0 2.12.4

2.8 2.8

3.3 3.43.6 3.6

3.9 4.0

Source: National Health Information Analytical Centre, 2014.

Figure 2. Annual hospitalization rate (2003–2013)

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 20130.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

6.9 7.37.9

8.48.9

9.5 9.8 9.910.6

12.4 12.3


16

49. APL2 however, aimed not only to increase access for the general population but to improve access for the most disadvantaged population in the country. According to Armenia’s Integrated Living Conditions Survey (the ILCS), which is conducted by the National Statistical Service, in 2010 the utilization of health services by the poorest income quintile was 3.9 percent for outpatient services and for 5.0 percent for inpatient services. In 2015, the outpatient services utilization reached 5.3 percent and inpatient utilization 9.7 percent. In other words, outpatient utilization by the poor increased by 35 percent and inpatient utilization almost doubled. Increased utilization of outpatient and inpatient health services by the poor was one of the key goals of APL2.

50. Given APL2 focus on the poor, the ICR team evaluated the project impact on access to health services in relatively disadvantaged regions. According to the 2014 HSPA, the highest IMR—an indicator closely related to poverty—across marzes was in Shirak: 18.1 deaths of infants under 1 year per 1,000 live births, almost three times higher than Yerevan’s IMR of 6.7. Shirak was one of the marzes included in APL2 that benefited from significant investments in primary and hospital care. In 2006, before the APL2 intervention, the hospitalization rate per 100 persons was 6.5; in 2013 after APL2 intervention, the hospitalization rate reached 8.7 in Shirak. In the case of outpatient service utilization, there was an increase from 2.5 visits per capita (2006) to 3.2 (2013). Similar positive trends were observed in other marzes with poor health outcomes, specifically Syunik (IMR 15.4) and Gegharkunik (IMR 15.1). APL2 clearly increased utilization of outpatient and inpatient health services in relatively disadvantage marzes. Even in some cases, these marzes perform better than the capital Yerevan. Based on the above, APL2 improved access to health care services, including the disadvantage populations.

Figure 3. Hospitalization rate before and after the project (Yerevan versus disadvantaged marzes)

Yerevan Shirak Syunik Gegharkunik0

1

2

3

4

5

6

7

8

9

10

6.4 6.5

5

4

7.8

8.7

7.4

5

Before project 2006 After project 2013


17

Figure 4. PHC per capita visits before and after the project (Yerevan versus disadvantaged marzes)

Yerevan Shirak Syunik Gegharkunik0

1

2

3

4

5

6

7

3.4

2.5

4

2.5

4.6

3.2

6.3

3.1

Before project 2006 After project 2013


18

51. The quality of health care perceived by the population has improved. The project expected, as a result of significant investment for improving the infrastructure of health facilities (both PHC and hospitals) and the retraining of health personnel, an increase in the perceived quality of care by the Armenian population. To determine the perceived quality of care, the project applied surveys. The last survey was implemented in 2015, but results are still being processed at the time of this ICR. The previous survey, conducted in 2013 for 2012 data, stated that the perceived quality of care reached the following results: (a) PHC: Urban 64.1 percent and rural 73.2 percent; and (b) Hospitals: Urban 73.7 percent and rural 66.8 percent. Therefore, there have been some improvements toward perceived quality compared with the baseline (64.3 percent). However, the perceived quality of care presented above was measured for the entire Armenian population, including those who might not have benefited from the modernized and improved health services. Therefore, the ICR team decided to assess other variables (more closely related to the project geographic areas) to evaluate if there was (or was not) an improvement in the perceived quality of health care. For this purpose, the following parameters were evaluated: (a) utilization rate of health services at the marz level; and (b) specific quality perception surveys performed at the health facilities intervened.

52. According to previous HSPA reports, perceived quality in marz hospitals is lower than in Yerevan. Therefore, people tend to seek care in Yerevan hospitals. APL2 invested in the renovation of hospitals and PHC facilities in the marzes outside Yerevan. As presented in table 3, all the marzes where APL2 invested registered a substantial increase in the number of per capita ambulatory visits. For instance, in the case of Syunik marz, per capita visits increased from 4.0 (2006) to 6.3 (2013). Similarly, in the case of hospitalization rate, all the marzes, except Vayots Dzor, registered a substantial increase. In the case of Vayots Dzor, the internal optimization was proposed in the marz hospital optimization master plan supported by the project, however no major investments were done due to its proximity to Ararat Hospital. Hence, APL2 investments not only contributed to increasing the hospitalization rate in all marzes but also, in some cases, reached and even surpassed Yerevan’s hospitalization rate.

Table 3. Hospitalization rate and PHC visits per capita before and after the project in Marzes

MarzesHospitalization rate PHC visits per capita

Before project 2006

After project 2013

Before project 2006

After project 2013

Yerevan 6.4 7.8 3.4 4.6Shirak 6.5 8.7 2.5 3.2Syunik 5.0 7.4 4.0 6.3Gegharkunik 4.0 5.0 2.5 3.1Aragatston 3.6 4.4 2.1 2.5Ararat 4.2 5.2 2.3 3.3Armavir 3.6 4.7 2.3 3.4Lori 4.9 7.5 2.6 3.7Kotayk 4.4 6.1 2.4 3.4Vayots Dzor 4.1 3.5 2.6 3.5Tavoush 4.5 5.2 2.2 2.9


19

53. In addition, Yerevan reached a peak of 246,983 hospital admissions in 2012, which decreased in 2013. The private sector also registered a reduction in hospital admissions from almost 150,000 in 2012 to 139,000 in 2013. On the other hand, the total number of patient admissions in marz hospitals included in APL2 constantly increased each year, growing from 96,570 (2006) to 115,731 (2013). According to the 2014 HSPA, the reduction in the number of hospital admissions in Yerevan and in the private sector is due to the opening of renovated marz hospitals.

Figure 5. Number of hospital admissions 2006-2013

2006 2007 2008 2009 2010 2011 2012 20130

100000

200000

300000

400000

500000

600000

172976 185643 201942 196045 203767 220411 246983 237715

96570 100037 104693 107130 104629 108405111759 115731

106971 115202125891 116517 118816

131226148239 139070

Yerevan Marz Hospitals Private SectorSource: National Health Information Analytical Centre, 2014.

20

54. The ICR team considered that all these movements of people at the marz level who increasingly started to seek care at both PHC and hospital facilities, and correspondingly the reduction of care seeking in Yerevan and private health care facilities, are objective results that are underpinned by (or reflect) an improvement in the perceived quality of health care services by the population.

55. Finally, specific surveys to evaluate patient and medical personnel satisfaction were administered in eight hospitals where the project intervened: Ararat, Armavir, Aparan, Hrazdan, Gavar, Gyumri, Ijevan, and Abovyan. The results for Abovyan hospital were available in English and were reviewed by the ICR team. A total of 50 patients participated in the survey: 92 percent were satisfied with the physicians, 94 percent rated the quality of health care services provided as good or excellent, and 98 percent rated the facility conditions as good or excellent. A total of 63 physicians and nurses also participated in the survey: 78 percent were satisfied with the equipment and facilities, and 80 percent pointed out their colleagues’ high qualifications.

56. Progress has been made to increase the sustainability of the health system. The previous Soviet health model was highly inefficient and therefore unsustainable. Reform measures needed to be taken. For instance, in 2004, at the beginning of APL1, the average length of stay (ALOS) in hospitals was 11.8 days. Increasing the system’s efficiency by merging hospitals and modernizing their infrastructure was a priority. By 2010, APL1 managed to reduce ALOS to 7.6 days. This remarkable positive result continued under APL2 that supported the Government’s Hospital Network Optimization Plan. APL2 was successful at reducing the average length of stay well below OECD levels.

Table 4. ALOS in hospitals supported by the project

Baseline2006

After Intervention2015

OECD Average (EU-28)2014

ALOS 7.7 5.8 7.8 Sources: HPIU administrative reports; Health at a Glance: Europe 2014, OECD.

57. In addition, chronic diseases are now an important burden for healthcare systems globally. Therefore, the MOH decided to include and support with APL2 preventive healthcare services to contain the growing burden of noncommunicable diseases. For instance, according to the MOH, 42 percent of patients with hypertension had one ECG annually in 2010, which increased –with APL2 support- to 55.2 percent in 2014. Similarly, 33 percent of patients with ischemic heart disease had one total cholesterol test annually in 2010, which increased –with APL2 support- to 53.7 percent in 2014. These are certainly improvements, but more work needs to be done, which is one of the reasons why the Disease Prevention and Control Project was prepared.

3.3 Efficiency

21

58. Efficiency is rated Substantial. The financial crisis had a severe impact on Armenia’s economy, generating a 14 percent negative growth. Despite the financial crisis, Armenia managed to sustain its commitment to the health sector. Public expenditure on health (PEH) as percentage of GDP has not changed significantly, from 1.39 percent in 2005 to 1.58 percent in 2014. Although, in the meantime, the resources allocated to Public Expenditure on Health (PEH) has substantially increased from AMD 31 billion in 2005 to AMD 76.6 billion in 2014, almost a 2.5 times increase. PEH even increased in the aftermath of the financial crisis (2009) when GDP was severely constrained and growth was negative. See table 3.1 in Annex 3.

59. The increase in financial resources combined with consistent and continuous government efforts to rationalize the hospital sector and develop family medicine allowed the achievement of efficiency gains. The first set of economic projections done at project design were confirmed: GOA managed to increase its investment in the health sector –even during the financial crisis- while not substantially changing PEH as percentage of GDP, the number of beds was rationalized, ALOS improved to OECD levels, and both outpatient and inpatient utilization rate of health services augmented considerably by the poorest quintile. See tables 3.2, 3.3 and 3.4 in Annex 3.

60. The economic analysis performed at design calculated that there will be considerable wage increases for medical personnel. For 2015, it projected that the average monthly wage would be around US$414, while the wage of a medical doctor would double and the wage of a nurse equal to this level. Based on these projections, it was estimated that in 2015 the gross labor cost for the health sector would be equal to US$435.3 million. Based on data provided by the MOH, in 2014 the average monthly wage was around US$330. The monthly wage for doctors was approximately US$303 (91 percent of average monthly wage) and for nurses US$163 (49 percent of average monthly wage). Reproducing the original analysis performed at design, in 2014 the gross labor cost for the health sector was equal to US$144.3 million, lower than the US$435.3 million originally estimated, possibly as a result of the financial crises and the consequent budgetary constraints.

61. The economic analyses done at design projected a decrease in out-of-pocket (OOP) health payments. High OOP health payments increases the risk to either forego or postpone necessary medical interventions. High OOP health payments also have impoverishing effects on the middle classes and the poor, particularly when confronted with a catastrophic disease. Increasing Armenia’s public investments in the health sector (including salaries) were expected to reduce OOP payments for health. According to the ILCS, OOP for out-patient health services reduced from 35.4 percent in 2010 to 20.6 percent in 2014. In the meantime, in-patient OOP decreased from 47.1 percent in 2010 to 42.8 percent in 2014. It is important to note that at the time of design, all calculations on labor costs and the decline of OOP payments were made based on a pre-crisis scenario when Armenia’s economy was displaying a double-digit growth. The 2009 crisis was a ‘force majeure’ event, which resulted in significant contraction of economy (−14 percent) as shown earlier. The recovery was slow and according to the Government and World Bank officials, it did not allow to stick to the original scenario for wage increase and investments, however OOP payments reduced. See tables 3.7 in Annex 3.

22

62. At the Closing Date on February 29, 2016, APL2 activities generated significant efficiency gains, particularly as a result of the implementation of the Marz Hospital Optimization Plan and the strengthening of the PHC network. In the 1990s, the Commonwealth of Independent States countries had on average more than 1,000 acute care hospital beds per 100,000 population. An excessive number but a common situation under the Soviet health system model. Today, this indicator had a modest decrease in the region to 800. In the case of Armenia, this indicator radically decreased from more than 800 in the last decade of the twentieth century to 329.82. The only country with a better indicator than Armenia is Georgia with 211.6 acute care hospital beds per 100,000 population. Armenia achieved a remarkable reduction, which in combination with all the project indicators analyzed earlier reflects the deep transformation of Armenia’s health system. Component 1 (Family Medicine Development) and Component 2 (Hospital Network Optimization) activities enabled Armenia’s health system to transition from the previous Soviet service delivery model toward a system that is the international norm, with a strong emphasis on PHC and a streamlined hospital sector. This transition was supported not only by APL2 but also by APL1 and the earlier IDA operation that was approved in 1997. These reforms improved health services utilization and reduced OOP payments without affecting PEH as a percentage of the GDP even in the middle of a financial crisis that severely affected Armenia’s economic growth.

3.4 Justification of Overall Outcome RatingRating: Satisfactory

Table 5. Overall outcome rating

Rating Criteria Original APL2Disbursement: 45.6%

At AF and Restructuring of 2010Disbursement: 54.4%

Relevance Substantial SubstantialEfficacy Substantial SubstantialEfficiency Substantial SubstantialOutcome Satisfactory Satisfactory

63. The overall outcome of this project is Satisfactory. Even though a split rating was applied to assess outcomes against the original and restructured project, the rating for each of the two periods are the same. Therefore, no weighted calculation of the two outcomes is necessary.

3.5 Overarching Themes, Other Outcomes and Impacts(a) Poverty Impacts, Gender Aspects, and Social Development

64. As described earlier, the project improved health service access and quality in the marzes with the worst health indicators that are usually associated with poverty. In addition, the maternal mortality rate per 100 000 live births decreased from 35 in 2006 to 19 in 2013.

(b) Institutional Change/Strengthening

23

65. As manifested by the health facility managers and health authorities interviewed by the ICR team, the project supported Armenia’s health system to transition from the previous Soviet service delivery model toward a system with a strong emphasis on PHC and a streamlined hospital sector, emphasizing the notions of quality, efficiency, and the use of information (NHA, HSPA) for policy decision making.

(c) Other Unintended Outcomes and Impacts (positive or negative)

66. Although there are no official statistics, according to hospital directors interviewed, due to the renovations, hiring new and trained health personnel (doctors and specialists in particular) and the retention rate have improved.

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops

67. Not applicable.

4. Assessment of Risk to Development Outcome

68. The risk at the time of the ICR that development outcomes will not be maintained is rated Negligible to Low. The reforms were initiated with APL1 and continued and expanded during APL2. Due to the positive results achieved, the Government—MOH in particular—has a strong commitment to sustain the gains. This commitment is reflected in the trends in health investments, the analytical work, and reports. All indicate that the health reforms initiated are fully supported by the Government and are not likely to be reversed. Further, the ongoing Disease Prevention and Control Project approved in 2013 has built on the achievement of the APL series.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory

69. During project preparation, a group of professionals with vast experience in the region gathered, and evidence in modernization and rationalization was reviewed and incorporated into the project. In addition, the team relied heavily on lessons learned from APL1, particularly for hospital optimization. The design was relevant and in line with the Government and the World Bank development strategy, as described in section 1. The project design was sound and activities clearly linked to objectives that the project intended to achieve.

(b) Quality of Supervision Rating: Satisfactory

24

70. Aide Memoires provide evidence that the World Bank implementation support visits took place on a regular basis, that the World Bank team offered technically sound advice and observations, and that the project was closely monitored. The restructurings also prove that the World Bank team proactively addressed problems that emerged in a timely and comprehensive manner.

71. Evidence from internal communications, letters to the Government, and Aide Memoires prove that the World Bank team was diligent in its communication with the GOA and World Bank management, provided up-to-date information and analysis on the status and impact of project activities, and offered options to address issues that arose as a result of evolving circumstances.

72. In addition, the Bank team offered policy advice of high quality and in a timely manner. For instance, the Bank produced a public expenditure review for the health sector and conducted policy workshops that contributed significantly to global knowledge sharing and informing policy decisions.

73. During the ICR mission, the author could perceive that the World Bank team—comprising an appropriate mix of HQ- and Armenia-based staff—developed a strong and cordial relationship with the different government offices involved and all the institutions recognized that the World Bank team was proactive in identifying and offering solutions to clients’ problems.

(c) Justification of Rating for Overall Bank PerformanceRating: Satisfactory

74. The rating is based on the World Bank’s role in ensuring quality at entry as described earlier, providing diligent implementation support throughout the implementation phase, and the strong relationship with the client to overcome its challenges. As a result of these commendable efforts, the country improved its health system situation and the project achieved its PDOs.

5.2 Borrower Performance(a) Government PerformanceRating: Satisfactory

75. The GOA developed key strategic documents that provided the basis for the project, such as the Poverty Reduction and Strategy Paper and the Marz Hospital Optimization Plan, which contained the policy measures and reforms in line with APL2. In addition, the GOA developed and approved the legal resolutions needed for PHC and hospital reorganizations, supported the Steering Committee (composed of representatives from different ministries) that provided overall oversight to the project, and strongly supported the optimization process that due to its nature faced criticism from labor unions. Finally, despite the 2008–09 financial crisis with subsequent decrease in GDP, the GOA continued providing the counterpart funds needed to complete all project works and recurrent costs. It further institutionalized the production of two critical reports, HSPA and NHA, for policy decision-making.

(b) Implementing Agency or Agencies PerformanceRating: Satisfactory

25

76. The MOH, through the HPIU, was the lead executing agency for the project, and it moved decisively to implement the reorganization and optimization of the hospital networks. This reform was both technically complex by its nature and politically difficult due to labor unions. In addition, as a result of actions promoted by the HPIU and supported by the MOH, the country started the development of key documents for health policy information and decision making, such as the NHA and the HSPA reports. In general, the MOH demonstrated a strong and long-term commitment to pursue the health reform agenda.

77. The implementation of the project demanded leadership and effective collaboration among different offices within and external to the MOH. This role was occupied by the HPIU due to which the project had a good overall performance during the nine years of the project. The HPIU not only coordinated but also offered technical assistance to all related institutions. Its role was crucial to ensure effective implementation of the project. In addition to overall coordination, the HPIU had a good performance in M&E. During ICR mission, the team could perceive the good and productive relationship the HPIU established with other offices at the MOH and GOA and how critical this office was to the achievement of the PDOs.

(d) Justification of Rating for Overall Borrower PerformanceRating: Satisfactory

78. The rating is based on the GOA’s strong commitment to pursue the reform despite its complexity, provision of funds for implementing the reform, and the HPIU’s good overall performance leading and coordinating the reforms as well as the administrative and fiduciary commitments. As a result of these commendable efforts, the country improved its health system and the project achieved its PDOs.

6. Lessons Learned

26

79. In the case of Armenia, steadfast commitment to reform and to its underlying policy objectives was a key factor to achieve the conversion from a Soviet health system model to one that prioritize PHC and hospital efficiency. This is consistent with the three health projects that the Bank supported in Armenia: Health Financing and Primary Health Care Development Project (the first health supported operation approved in 1997), APL1 (2004), and APL2 (2009). And today, the effort continues with the Disease Prevention and Control Project with a focus on NCD. For instance, in the case of human resources for the PHC level, the policy work started in 1997, it continued and expanded vigorously during APL1 when the majority of health personnel was retrained, and finished during APL2 that successfully completed the retraining of PHC professionals for the whole country. Similarly, in the case of Hospital Optimization, the work started with APL1 in Yerevan. And based on its positive results, it considerably expanded to other marzes with APL2. Over nearly two decades of support to the health sector through Bank-financed operations, it is clear that short and long term objectives were maintained and that APL2 and its AF built on the progress achieved over time.

80. Instruments such as APL may still offer value and be relevant in countries with long term commitment to reforms. As explained above, the reform in Armenia expanded almost two decades with three Bank projects. The last two projects (APL1 and APL2) were APL: Adaptable Program Loans. APL offered Armenia the opportunity to continue with its reform if certain pre-defined targets were achieved. By pre-defining targets, the APL also provided an orientation to where the system was transitioning to. As reforms can expand for many years, current Bank investment tools should be able to provide long term commitments to countries’ reform programs when the reforms prove to deliver the agreed results.

81. Armenia offers a good case example for other countries of the efficiency gains that a health system can achieve. The efficiency gains obtain in Armenia are remarkable: for instance the reduction in ALOS below OECD levels, the reduction in number of beds, and OOP payments. Certainly there is more work to do and there are new challenges such as chronic diseases, but the progress made so far offers a good case to other countries facing similar inefficiencies. Therefore, if other countries decide to embark on similar reforms, they should consider the factor that allow this reform to be successful: it had consistency and a long term vision as explained above, it had a highly competent and engage core team in the Government side that was fairly constant during the reform, there was political support to pursue the optimization plan, and finally the financial and technical support from an external partner such as the Bank.

27

82. In the modernization of a health system, infrastructure investments are an important factor but a comprehensive approach is also required. Armenia achieved efficiency gains and a considerable increase in the utilization of health services at both hospital and PHC level, including the poorest quintile of the population. It is a fact that Armenia heavily invested in the renovation of its Hospital and PHC networks and that this was an important factor that increased the demand. For instance, all marz hospitals that received infrastructure investments substantially increased their utilization, the only exception is Vaots Dzor, a hospital where no investment was performed for its proximity to Ararat hospital. However, in addition to infrastructure investments, Armenia reformed other aspects of its health system: the competencies of its current PHC level work force, the training of its future health human resources (supporting the State Medical University), the financial aspects of PHC and hospital levels, etc. An optimization reform, similar to other reforms, requires to have a comprehensive approach to be successful.

83. Any reform and project that supports it should consider not only legal and financial barriers but also the political economy context. In health systems, hospitals produce the most complex health services and consume a substantial part of a country’s health budget. Therefore, there are attempts to increase their efficiency, transparency and governance. In the case of Armenia, APL2 aimed at introducing modern management tools in hospitals for enhancing their efficiency and transparency in the management of resources. However, initiatives such as the implementation of Supervisory Committees and Public Performance Reports could not be implemented, while there was some progress with Independent Audits. It is a fact that there were legal and financial constraints that limited the implementation of these initiatives, however, there were political economy aspects that also should be part of any comprehensive stakeholder evaluation.

84. It is possible to achieve the regular implementation and use of health information tools for evidence-based policy decisions. NHA and HSPA reports are key health policy documents for any country, and they were introduced in Armenia by the Ministry of Health – HPIU in collaboration with the National Institute of Health, the National Statistical Service, the National Health Analytical Centre, and supported by the project. These were activities financed by the project, and hence there was a risk that once the financial support stops, the production of these documents would also stop. However, these key policy instruments have continued to be produced by the MOH using government funds. Actually, they have been produced more frequently than originally planned. In the ICR team’s opinion, these reports are still produced and supported because decision makers and staff involved showed how relevant and useful these reports were for managing the health sector in issues such as human resources, financial aspects, efficiency gains, preparation of new projects, etc.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies

85. See Annex 7 for a summary of the Borrower’s contribution to the ICR. Comments from the Borrower on the draft ICR were received by letter No. 5-2/20334-16 of August 15, 2016 and have been addressed in the ICR. They contain minor precisions in a few economic indicators.

(b) Cofinanciers. Not applicable.

28

(c) Other partners and stakeholders. Not applicable.

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

ComponentsAppraisal Estimate (USD

millions) Actual/Latest Estimate (USD millions) Percentage of

AppraisalAPL2 AF Total APL2 AF Total

A. Family Medicine Development 3.11 4.70 7.81 3.77 6.10 9.87 126%1

B. Hospital Restructuring and Improvement

19.09 20.77 39.86 21.07 22.15 43.23 108%

C. Institutional development 2.41 2.57 4.98 0.08 0.57 0.66 13%2

D. Project Management 0.72 1.59 2.31 0.33 1.46 1.79 77%3

Total Baseline Cost 25.33 29.63 54.96 25.26 30.28 55.54 101%

Front end Fee PPF 0.00 0.00

Front end fee IBRD 0.00 0.05 0.05

Total 25.33 29.63 54.96 25.31 30.28 55.59 101%1. Family Medicine Development costs increased due to the renovation of more health facilities than originally planned. 2. Institutional Development costs were less than estimated due to GOA decision to allocate public funds to finance NHA and HSPA reports.

29

3. Project Management component cost were less than estimated due to GOA decision to use public funds to finance specialists support.

(b) Financing (in USD Million equivalent)

Source of Funds Appraisal Estimate (USD millions)

Actual/Latest Estimate (USD

millions)

Percentage of Appraisal

Borrower 13.41 13.51 101%

Local Communities 0.25 0.38 152%

International Development Association (IDA) 22.00 22.71 103%

International Bank for Reconstruction and Development (IBRD)

19.00 18.98 100%

Local Sources of Borrowing Country 0.30 0.00 0.0%

Total 54.96 55.59 101%

30

Annex 2. Outputs by Component

Component Outputs-outcomes

A. Family Medicine Development

Family medicine development component consists of 2 subcomponents implemented as follows:

1. Re-training of Family Physicians and Family NursesThe objectives of the APL-2 and AF subcomponents were the retraining family physicians and family nurses to ensure the coverage of population under the family medicine scheme. At this moment, among the total number of the re-trained personnel on family medicine, the number of family physicians trained has reached 1,676 and the number of family nurses 1,804.

2. Strengthening of the PHC InfrastructureThe objective of this sub-component was the renovation/construction of 50 rural ambulatories. All outcome indicators have been achieved and in some cases the project has over-achieved them. In 2008-2009, PHC infrastructure improved in 50 rural communities with the APL2 financial support. In particular, new ambulatories were constructed in 20 communities and reconstructed in 30 ambulatories. Within APL 2 and AF, it has been initially planned to construct 13 new rural ambulatories and carry out small scale renovations in 4ambulatories. The planned civil works were successfully done and, moreover, 1 additional ambulatory and 1 marz policlinic have been renovated as a result of savings made in the frame of construction of the AF 11 ambulatories.

B. Hospital Network Optimization

Hospital network optimization component was aimed at upgrading physical infrastructure of one network per marz; as well as providing basic modern medical, waste management and IT equipment; training and technical assistance to improve hospital management capacity, strengthen quality assurance and medical waste management systems, architectural design and supervision of the civil works.Within the project, the hospital network optimization component was successfully implemented according to functions as designed:(a) the provision of additional package of equipment, supplies and furniture for four marz medical centers (Ararat, Armavir, Aparan, and Goris); (b) the provision of a full package of equipment for two medical centers (Gavar and Gyumri); (c) the modernization of additional marz hospital networks (in Syunik marz – Construction of Meghri Regional MC and Kapan MC, Kotayk marz - Abovyan MC, and Tavush marz – Berd MC). Due to available savings, the project renovated the polyclinic sub-building of Alaverdi MC as well as Chambarak MC.

The project supported civil works at a cost of US$30.21 million in the marzes hospitals. All hospitals received medical equipment and all sites have been provided with IT equipment and staff have received training on information systems. As well as, all sites received training and supplies for medical waste management. In 2015, equipment for mortuaries of Aparan, Gyumri MC and Republican Scientific-Practical Center of Forensic Medicine in Yerevan city were provided.

C. Institutional Strengthening

The purpose of this component was to build capacities for the evaluation of health sector performance. The team focused on the development of the core instruments for health policy development and monitoring. The component supported the development of Health System Performance Assessment (HSPA) and National Health Accounts (NHA) and institutionalization of these reports. These reports have been published in two languages (in English and Armenian) and disseminated among key stakeholders. Key performance indicators have been agreed upon. TA provided on the job training in the development and implementation of reports/surveys.

31

Component Outputs-outcomesIn the frame of HSMP 2 AF project, it was also planned to implement “International expert for Training of NIH teaching personnel”, “On site Clinical Training of Clinical Staff conducted by NIH” and “TA with international experience for the development of the educational programs”.

Until 2011, continuous medical education in the Republic of Armenia was provided by the National Institute of Health after S. Avdalbekyan of the Ministry of Health of GOA, but according to the Governmental Decree N 1193 of GOA, dated on August 18, 2011, this responsibility was delegated to Yerevan State Medical University after Mkh. Heratsi. This decision aimed to improve the quality and to unify and modernize the graduate and postgraduate medical education in accordance with international standards.

Taking into consideration that procurement of consultancy services from Credit funds was considered inappropriate by the GoA, it was decided to use other resources of funding. In particular, the use of State Budget funds has been used to finance the implementation of the “Project on improvement of theoretical and practical skills of marz hospital medical specialties (physicians and nurses)”.

Project proposal has been developed by Health PIU and presented to the MoH and the World Bank. According to mutual agreements, the project has been implemented at the corresponding departments of Yerevan State Medical University after Mkh. Heratsi under the State budget of GOA.

The next objective of the component was the improvement of Public Expenditure Management, through provision of support to the State Health Agency (SHA) for improving its capacity to use efficiently public funds for purchasing health care services. Excellent progress has been made to establish capacity for measurement of true costs related to health care services and assessment of out-of-pocket payments.

Corresponding assignments (“Measurement of True Costs Related to Medical Processes in Armenian Hospitals” and “Assessment of the out-of-pocket payments in hospital facilities in Armenia”) have been successfully implemented in 2009 and 2011.

Significant progress has also been made to improve Medical Education through the provision of technical assistance to improve the Undergraduate and Continuous Medical Education.

D. Project Management

Project administration and coordination has been highly effective as the HPIU core group has been established with key professional staff, including Director, Financial Manager, Component Coordinators, Procurement Officer, Legal Specialist, Monitoring and Evaluation Specialist, Architect-Engineer, Community Mobilization Specialist, Accountant, Secretary/Translator, Office Manager, and drivers. Other consultants were hired to support specific needs under a particular component for defined assignments, as needed.

Training and incremental operating costs as well as upgrading of office infrastructure, IT equipment and supplies have been financed under the project.

32

Annex 3. Economic and Financial Analysis

1. Armenia’s national income was growing at an average rate of 12.6 percent for the 2002-2006 period that allowed a significant reduction in overall poverty, with the proportion of poor declining from 51 percent in 2001 to 30 percent in 2005. Extreme poverty reduced even faster, from 16 percent in 2001 to below 5 percent in 2005. However, the global financial crisis of 2008-09 had a significant negative impact on the country’s economy and double digit pre-crisis growth rates were replaced by an average 4.0 percent rate after the crisis. The global financial crisis also had a dramatic impact on poverty in Armenia and eroded some of the welfare gains. Poverty rate increased from 27.6 percent in 2008 to 35.9 percent in 2009, and decreased to 30 percent in 2014.

2. Despite the severe impact of the financial crisis, Armenia managed to sustain its commitment to the health sector. Public expenditure on health (PEH) as percentage of GDP has not changed significantly, from 1.39 percent in 2005 to 1.58 percent in 2014. Although, in the meantime, the resources allocated to PEH has substantially increased from AMD 31 billion in 2005 to AMD 76.6 billion in 2014, almost a 2.5 times increase. PEH even increased in the aftermath of the financial crisis (2009) when GDP severely constrained and growth was negative.

Table 3.1. Armenia GDP and PEH (AMD, billions)

Indicator 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014GDP 2,243 2,656 3,149 3,568 3,142 3,460 3,778 4,266 4,556 4,829GDP growth (%) 13.9 13.2 13.7 6.9 −14.1 2.2 4.7 7.2 3.3 3.6PEH 31.0 39.4 46.8 49.9 56.1 56.1 63.3 64.4 64.3 76.6PEH (% of GDP) 1.39 1.48 1.49 1.40 1.79 1.62 1.68 1.61 1.41 1.58

Source: Ministry of Finance of Armenia, 2016.

3. The escalation in economic resources combined with consistent and continuous government efforts to rationalize the hospital sector and develop family medicine allowed the achievement of efficiency gains. The first set of economic projections done at project design were confirmed: GOA managed to increase its investment in the health sector –even during the financial crisis- while not substantially changing PEH as percentage of GDP, the number of beds was rationalized, ALOS improved to OECD levels, and both outpatient and inpatient utilization rate of health services augmented considerably by the poorest quintile.

33

Table 3.2. Number of beds at hospitals benefited by project interventions

Marz Hospital Number of BedsBefore rationalization After rationalization

Ararat Ararat MC 80 50Aragatcotn Aparan MC 80 45Armavir Armavir MC 185 110Gegharkuniq Gavar MC 95 85

Chambarak HC 55 30Kotayq Hrazdan MC 155 80

Abovyan MC 100 55Shirak Gyumri MC 245 200Lori Alaverdi MC 50 40Syuniq Goris MC 180 90

Kapan MC 190 105Meghri MC 45 45

Tavush Berd MC 60 30Ijevan MC 120 70

Total 1,640 1,035Source: MOH of Armenia, 2016.Note: MC = Medical center.

Table 3.3. ALOS in hospitals supported by the project

Before Rationalization2006

After Rationalization2015

OECD Average (EU28)2014

ALOS 7.7 5.8 7.8Source: MOH of Armenia 2016.

Table 3.4. Utilization rate of health services by the poorest quintile

Indicator Before Rationalization After RationalizationUtilization of outpatient health services 3.9% (2010) 5.3% (2015)

Utilization of inpatient health services 5.0% (2010) 9.7% (2015)

Source: MOH of Armenia 2016.

34

4. According to the economic analysis performed at design, the government reform program supported by APL2, aimed at strengthening the provision of PHC health services and optimizing the Marz hospital network. To achieve these goals, the government would continue to increase the salary of health personnel, reduce the number of beds and buildings used in hospital networks, and invest in PHC centers and family medicine training. The project assumed that these actions will increase utilization of PHC services and hospitalizations publicly financed, as individuals would realize that more and better quality of health care services were provided. Economic analysis projected that more and better services would be provided with the same share of GDP.

5. In order to estimate the total cost of health care services in 2015, the economic analysis done at design used as a starting point an estimate of the health sector wage bill and assumed that the gross labor cost was 60 percent of the overall health care cost. In addition, the analysis employed the following assumptions:

- Population: 3.3 million;- Number of physicians: 35 per 10,000; total 11,031;- Number of nurses and midwives: 71 per 10,000; total 23,733;- Support to medical staff ratio 1:1:34,764;- Average monthly wage: US$414;- Total number of formal sector employees: 587,000;- Inflation rate: 4 percent for 2006 and 2007, 3 percent thereafter;- Gross to net wage ratio 1.4:1.0;- Physicians’ average net monthly salary (2 times the average salary): US$828;- Nurses’ average net monthly salary (50 percent of physician’s salary): US$414;- Support staff salary is estimated at US$200; and- Nominal GDP: US$ million 14,797.

6. Based on the above assumptions, the economic analysis calculated that there will be considerable wage increases for medical personnel. In 2006, the average monthly wage was around US$152. The monthly wage for doctors was approximately US$134 (88 percent of average monthly wage) and for nurses US$67 (44 percent of average monthly wage). For 2015, the economic analysis projected that the average monthly wage would be around US$414, while the wage of a medical doctor would be double this level and the wage of a nurse equal to this level. Based on these projections, it was estimated that in 2015 the gross labor cost for the health sector would be equal to US$435.3 million.

7. Based on data provided by the MOH, in 2014 the average monthly wage was around US$330. The monthly wage for doctors was approximately US$303 (91 percent of average monthly wage) and for nurses US$163 (49 percent of average monthly wage). Reproducing the original analysis performed at design, in 2014 the gross labor cost for the health sector was equal to US$144.3 million, lower than the US$435.3 million originally estimated. The considerable difference is because there were no substantial increases in doctors’ and nurses’ salaries.

Table 3.3. Average Monthly Salaries in 2014

Average Monthly Salaries 2014 (US$)

35

Average monthly wage for the country (US$) 330Average monthly wage for doctors (US$) 303Average monthly wage for nurses (US$) 163 Average monthly wage for support staff (US$) 163

Source: Ministry of Health of Armenia, 2016.

Table 3.4. Number of Health Personnel in 2014

Number of health personnel 2014Number of doctors in the country 12,896Number of nurses in the country 18,053Number of support staff in the country 10,660

Source: Ministry of Health of Armenia, 2016.

8. The economic analyses done at design projected a decrease in out-of-pocket (OOP) health payments. High OOP health payments increases the risk to either forego or postpone necessary medical interventions. High OOP health payments also have impoverishing effects on the middle classes and the poor, particularly when confronted with a catastrophic disease. Increasing Armenia’s public investments in the health sector (including salaries) were expected to reduce OOP payments for health. According to the ILCS, OOP for out-patient health services reduced from 35.4 percent in 2010 to 20.6 percent in 2014. In the meantime, in-patient OOP decreased from 47.1 percent in 2010 to 42.8 percent in 2014. It is important to note that at the time of design, all calculations on labor costs and the decline of OOP payments were made based on a pre-crisis scenario when Armenia’s economy was displaying a double-digit growth. The 2009 crisis was a ‘force majeure’ event, which resulted in significant contraction of economy (−14 percent) as shown earlier. The recovery was slow and according to the Government and World Bank officials, it did not allow to stick to the original scenario for wage increase and investments, however OOP payments reduced.

Table 3.7. OOP payments for Essential Health Services

2010percent

2014percent

Out-patient 35.4 20.6In-patient 47.1 42.8

Source: ILCS 2010 and 2014.

36

9. At the Closing date on February 29, 2016, APL2 activities generated significant efficiency gains, particularly as a result of the implementation of the Marz Hospital Optimization Plan and the strengthening of the PHC network. In the 1990s, the Commonwealth of Independent States countries had on average more than 1,000 acute care hospital beds per 100,000 population, an excessive number but a common situation under the Soviet health system model. Today, this indicator had a modest decrease to 800. However, in the case of Armenia, this indicator decreased from more than 800 in the last decade of the twentieth century to 329.82. The only country with a better indicator than Armenia is Georgia with 211.6 acute care hospital beds per 100,000 population. Armenia achieved a remarkable reduction, which in combination with all the project indicators analyzed earlier reflects the deep transformation of Armenia’s health system. Component 1 (Family Medicine Development) and Component 2 (Hospital Network Optimization) activities enabled Armenia’s health system to transition from the previous Soviet service delivery model toward a system that is the international norm, with a strong emphasis on PHC and a streamlined hospital sector. This transition was supported not only by APL2 but also by APL1 and the earlier IDA operation that was approved in 1997. These reforms improved health services utilization and reduced OOP payments without affecting PEH as a percentage of the GDP even in the middle of a financial crisis that severely affected Armenia’s economic growth.

37

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team membersNames Title Unit

Enis Barış Practice Manager GHNDR Olena Fadyeyeva Senior Operations Officer LLIOP Tamar Gotsadze Consultant GHNDR Susanna Hayrapetyan Lead Health Specialist GHNDR Satik S. Nairian Program Assistant ECCAR Svetlana Georgieva Raykova Associate Operations Officer CASPMSupervision/ICR Susanna Hayrapetyan Lead Health Specialist GHNDR Wezi Msisha Sr. Operations Officer SACKB Johanne Angers Senior Operations Officer GHNDR Alexander Astvatsatryan Consultant – Procurement Specialist GGO03 Enis Barış Practice Manager GHNDR Garik Sergeyan Sr. Financial Management Specialist GG021 Arman Vatyan Sr. Financial Management Specialist GGODR Carmen F. Laurente Senior Program Assistant GEDDR Patricio V. Marquez Lead Health Specialist GHNDR Satik S. Nairian Program Assistant ECCAR Owen K. Smith Senior Economist GHNDR Armine Aydinyan Procurement Specialist GG003 Darejan Kapanadze Safeguard Specialist GEN03 John Malmborg Consultant GHN03 Tamar Gotsadze Consultant GHNDR Gabriel Francis Program Assistant GHN03

(b) Staff Time and Cost

Stage of Project CycleStaff Time and Cost (Bank Budget Only)

No. of staff weeks USD (including travel and consultant costs)

LendingFY07 15.07 47,670

Total: 15.07 47,670Supervision/ICR

FY08 16.41 22,906FY09 16.35 28,514FY10 22.83 52,693FY11 27.25 34,710FY12 20.26 51,298FY13 11.14 46,726FY14 11.30 28,415FY15 10.68 33,292FY16 12.41 25,274

Total: 148.63 323,828

38

Annex 5. Beneficiary Survey Results

Not applicable.

39

Annex 6. Stakeholder Workshop Report and Results

Not Applicable.

40

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR

Summary of Government of Armenia completion report (May 2016).4

Since independence of 1991, the health care system in Armenia had undergone numerous changes that transformed a centrally run state soviet system into a fragmented health care system that was largely financed from out-of-pocket payments. APL-2 continued supporting the GOA to: (i) complete the family medicine based Primary Health Care reform that was launched in 1996 so to ensure that every Armenian citizen and legal resident will have access to a qualified and well-motivated family physician and nurse of his/her choice; (ii) consolidate the hospital sector to minimize waste of scarce resources and improve quality of care; and (iii) strengthen competencies for effective stewardship in policy making, regulation, oversight and public accountability to ensure effective and targeted use of public resources in accordance with the health and healthcare needs of the population, especially the vulnerable groups.

Assessment of Project Design. APL2 was adequately designed to meet the Government’s reform priorities. The objectives of the projects were crucial as they were consistent with the ongoing health reforms being carried out in Republic of Armenia. The project played a central role, while being an integral part of these reforms and with financing to support the reforms.

Assessment of World Bank performance during project implementation. The WB health team, similar to previous projects, has played an important role for successful implementation of APL-2, providing continuous support and supervision in all stages of project cycle. APL2 project has experienced a shortage of funds, and to finance this gap Additional Financing was provided to the project, which allowed successful implementation and completion of the project. The GOA, together with MOH and HPIU highly appreciate effective cooperation with the WB health team, in particular, during implementation the project benefited from continuous direct supervision conducted by the WB health team, which allowed timely identification of problems, development of prompt and feasible solutions by considering recommendations and comments provided by experienced experts. Periodic WB health team mission visits, as well as regular Financial Management Implementation Support and Supervision Missions and Procurement Post-review, provide the GOA, MOH and HPIU significant feedback on current progress and future improvements in all stages of project implementation. On the other hand, annual trainings organized by the WB for procurement specialists and financial staff are important in terms of keeping updated on WB project new requirements and approaches.

Assessment of Government’s actions during project implementation. The project benefited from strong support and active interest of the Government throughout its implementation. Healthcare is a priority area for the Government and main policy goals in healthcare are early identification of illnesses, prevention, diagnosis and treatment, as well as increased accessibility of medical services and quality of services. Overall, there were no significant delays in counterpart contributions in project lifetime. Monitoring by Ministry of Health and HPIU. The MOH implemented continuos monitoring and evaluation of the ongoing health sector reforms, in close collaboration with the Health Information Analytical Center, which was in charge of effective assessment of health sector

4 The Summary of the Government’s contribution to the ICR was prepared by the ICR team. The Government’s full report is available upon request.

41

performance, as well as financial flows, development of national reports of HSPA and NHA. At the same time the monitoring of the HSMP implemented by MOH was directed to supervision of progress of project activities, evaluation of the effectiveness and impact of main interventions, which further serves as a tool for health care facilities to use data for decision-making and project management.

Sustainability of Project Investments. The goals set upon project design and results reached upon project completion, including modernization of hospital network and PHC infrastructure, development of human resources, strengthening the institutional capacity of health care system, are consistent with the commitment of GOA to health reforms implementation. Examples of sustainability of project investments can serve: (i) increase of budget allocations to health sector in recent years, (ii) increase of utilization of health care services, both in PHC and hospital level, (iii) reforms in medical education, (iv) institutionalization of HSPA and NHA etc.

Key lessons learned.

The process of the marz hospital optimization was in a sense politically sensitive and technically challenging process. The lesson learned is that the implementation of the program was particularly successful in those regions where it was followed-up with major investment in the marz hospital networks, while not quite effective in the rest of medical centers.

Development of HSPA in the frames of HSMP, has become not only a significant tool for policy makers, but also it has shown the priority issues to be addressed in future health sector reform-oriented projects. The data presented by HSPA reports emphasized the high morbidity and mortality rates and prevalence of risk factors leading to non-communicable diseases in Armenia, and the importance of the early detection, prevention and management of those diseases. Government has paid specific attention to the last and adopted National Strategy for Early detection and prevention of cardiovascular diseases, diabetes mellitus and neoplasms in 2011. Later this lesson, was addressed in “Disease Prevention and Control Project” and implementation of country-wide screenings for early detection and prevention among the Armenian population, as well as raising awareness on NCD risk factors and promotion of healthy life-style.

The achievements in hospital modernization and increased population satisfaction with the quality of hospital care in marzes encouraged the GOA to continue the hospital reform agenda in order to address MCH and NCD (particularly cancer burden) challenges at the hospital level. This lesson is addressed in “Disease Prevention and Control Project”, including reconstruction of the Center of Hematology in Yerevan and creation of Bone Marrow Transplantation Unit within the Center, as well as creation of Radiation Therapy Center of Armenian Center of Excellence in Oncology (ACEO).

Conclusion. In overall the APL-2 was designed adequately to meet the priorities of health sector reforms set by the GOA. Most of the project targets were successfully met by the completion of the last, and have played significant role in addressing challenges in health sector, mainly providing more accessible, affordable, quality and sustainable health care services to the population, in particular to the most vulnerable groups.

42

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders

Not applicable.

43

Annex 9. List of Supporting Documents

1. Project Appraisal Document for APL 12. Project Appraisal Document for APL 23. Financing Agreement for APL 14. Financing Agreement for APL 25. Project Paper for AF6. Financing Agreement for AF7. Restructuring Papers8. ICR for the APL 19. Country Partnership Strategy10. Health Project Implementation Unit, Progress Management Reports11. Aide Memoires and Back-to-Office Reports12. Management and other important letters and memoranda13. Implementation Status and Results Reports (ISRs)14. Borrowers Contribution to ICR 15. Borrowers Comments to draft ICR, August 15, 2016. 16. Armenia Health System Performance Assessment Report 201417. Armenia National Health Accounts Report 2014

44

MAP ARM33364

45