Deepak T. Bhatia

Measuring the impact of Egovernment

Deepak T. BhatiaSectoral and E-government Applications Practice Lead

Global ICT department (GICT)The World Bank

Deepak T. Bhatia

It is no longer acceptable for technology to be utilized simply for technology’s sake. Individual e-government projects need to demonstrate their contribution to overall

government objectives – with the focus being on results and impact.

Deepak T. Bhatia 3

Why e-Government?

•Battling bureaucracy

•Improving Efficiency

•Better service delivery

Strengthening governance and accountability

Deepak T. Bhatia 4

About 150 contain ICT components (out of 250 new WB projects each year).

0% 1% 2%

8%11%

22%

40%

47%

55%

61%57%

51%

0%

10%

20%

30%

40%

50%

60%

70%

Bank projects w ith ICTcomponents (%)

0% 1% 2% 8% 11% 22% 40% 47% 55% 61% 51% 57%

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

In the World Bank the Number of Projects with ICT is growing

57% of active projects have ICT components (this is 930 projects)

ICT components’ combined volume is $6.2 billion

Deepak T. Bhatia 5

Analysis of e-Government Impact

• World Bank commissioned study to explore impact of e-government on citizens, in collaboration with the Indian Institute of Management, Ahmedabad, India and the London School of Economics

• Effort to go beyond anecdotal assessments commonly available

• Sample of 5 mature, wide scope/scale projects selected for analysis from India

– Chosen to represent different stages of overall IT readiness and evolution

Deepak T. Bhatia

Learning from Past Assessments

• Variety of approaches have been used for assessments -client satisfaction surveys, expert opinion

• Often studies have been done by agencies that may be seen as being interested in showing positive outcome

• Lack of credibility of results-different studies of the same project show very different outcomes

• Lack of rigor in sampling-results can not be easily generalized

• Lack of rigor in controlling for external influence-need for counterfactuals ignored.

• Lack of a standard methodology-making it difficult to compare projects

Deepak T. Bhatia 7

Dimensions of measurement - Stakeholders

• Impact on Clients:• All measurements on the basis of a sample of clients for each major service

availed by the client. Measurements would be done for electronic delivery of services as well as for the earlier mode of delivery of the same service. In cases where alternate (non electronic modes) are currently being used by the same set of users in similar contexts elsewhere, measurement would be recorded for such usage

• Impact on Agencies (Including Partners in Implementation):

• All measurements on the basis of data collected from agency records, a sample of employees or a group of managers. Measurements would be done for electronic delivery of services as well as for the earlier mode of delivery of the same service

• Impact on Society:

• Some of the questions to be framed as degree of improvements resulting from the implementation of a specific project

Deepak T. Bhatia

Measurement Framework - Dimensions

Stakeholders Key Dimension of Impact

Client Economic (Direct & Indirect)Governance (Corruption, Accountability, Transparency, Participation)Quality of Service (Decency, Fairness, Convenience, etc.)Over all satisfaction

Agency

(Including Partners in Implementation)

Economic (Direct & Indirect)Governance (Corruption, Accountability, Transparency, Participation) Performance on Key Non-economic Objectives Process ImprovementsWork life of employees

Society

Other Departments

Government as a Whole

Civil Society

Desirability of investments in e-Government

Impact on vulnerable groups

Image of Government (Efficiency, Corruption, Accountability, Transparency, Participation, Responsiveness)

Impact on development Goals

Deepak T. Bhatia

Proposed Framework

• Focuses on retrospective assessment of e-delivery systems(B2C and B2B)

• Balanced approach between case study and quantitative analysis

• Recognizes that some part of the value to different stakeholders can not be monetized

• Understand how inputs lead to outputs and outcomes in different project contexts

• A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of projects in countries with various delivery models and paucity of data

Deepak T. Bhatia

Methodology for Assessment

• Select mature implementations , of e-delivery of services.

• Collect data through structured survey from clients, employees, supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system)

• Customize survey instrument to each project, adapt in local language

• Data can be collected through Internet survey, face to face interviews and focus groups

• Determine sample frame and size so that results can be extrapolated to the entire population (often 300 clients may be sufficient).

• Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies

Deepak T. Bhatia 11

Projects assessed

• Bhoomi project. 2001. delivering two online services: issuance of records of rights, tenancy, and crop inspection register (RTC) and filing of requests for mutation (changes in land ownership) for affecting changes in land records.

• Karnataka Valuation and E-Registration (KAVERI) project. 2003 delivering three key services: registration of property sale and purchase deeds, issuance of nonencumbrance certificates, and issuance of copies of previously registered deeds.

• Computer-aided Administration of Registration Department (CARD). 1998. same three online services provided by KAVERI.

• eSeva. 2002, These one-stop service centers now deliver 135 services from central, state, and local governments and public utilities, and are used by 3.1 million citizens a month

• Ahmedabad Municipal Corporation (AMC) Civic Centers project. 2002, deliver three important services: annual payment of property taxes, issuance of birth and death certificates, and issuance of shop licenses.

Deepak T. Bhatia 12

Caveat

• largely relies on the survey of stakeholder perceptions of public services before and after these projects.

• Thus not an impact evaluation of investments in narrowly defined ICT, which typically includes hardware, software, and communication equipment.

• Rather, a review of the outcomes of packaged efforts involved in the transition from manual services to those using ICT as a key component.

Deepak T. Bhatia 13

Impact: Computerized service delivery is preferable to manual

Preference for Computerization (%)

0

20

40

60

80

100

Bhoomi KAVERI  CARD  e-Seva  AMC 

Preference for Computerization (%)

Deepak T. Bhatia 14

-100

-80

-60

-40

-20

0

20

40

60

80

100

120

Bhoomi

RTC

Bhoomi

Mutation

KAVERI CARD eSeva AMC

User reports of changes in travel costs, by project (number of trips * distance travelled)

Rupees per service

• Bhoomi costs went up due to change in location of service centers from villages to “taluka” headquarters

Deepak T. Bhatia 15

User reports of reductions in waiting times, by projectMinutes

• decreased in all five projects.

• With manual systems, users often had to wait in long lines to receive services. Automation has increasedthe efficiency of document processing.

0

10

20

30

40

50

60

70

80

90

100

Bhoomi

RTC

Bhoomi

Mutation

KAVERI CARD eSeva AMC

Deepak T. Bhatia 16

0.0 0.2 0.4 0.6 0.8 1.0

Bhoomi-RTC

Bhoomi-Mutation

KAVERI

CARD

eSeva

AMC

User perception of improvement in service quality

• Rating on elements such as service delivery location, accessibility, convenience, cost, transparency, and service orientation

Deepak T. Bhatia 17

Project Features

Bhoomi Error-free transactions Reduced delays in transactions

Shorter waiting times Fewer visits

KAVERI Less corruption Increased transparency Error-free transactions Shorter waiting times

CARD Increased convenience Shorter waiting times Less corruption Fair treatment

eSeva Increased convenience Shorter waiting times More convenient time schedules

Fair treatment

AMC Increased convenience Less corruption Greater transparency Good complaint handling system

Users’ top four desired features of services, by project

Goals Examples of performance indicators

Key features of the enabling environment

Outcomes Increased efficiency Increased transparency and

accountability Higher-quality public

services Better access to services

Financial and time savings in government activities

Public perceptions, such as user satisfaction and score cards

Financial and time savings for citizens

Increased public service timeliness and responsiveness

Reduced errors Financial saving per

transaction

Overall e-government strategies

Political and popular support for cross agency coordination and public sector reform

Telecommunications infrastructure and cost structures for increasing ICT access

Supportive legal and regulatory frameworks

Balances with competing priorities (such as roads and education)

Macroeconomic changes

Outputs Reengineered processes New ICT systems Increased service coverage

Comparisons of old and new business processes

Technical reviews of IT infrastructure, applications, and performance

Variety of available services IT support capacity Service training

Proposed conceptual framework

Deepak T. Bhatia 19

Additional conclusions

• Out of the three projects where significant corruption was reported in the manual system, one project was able to eliminate corruption through computerization

• Incremental operating costs for agencies per transaction are less than the reduction in the direct cost of accessing the service reported by the clients.

• Negative aspect is the observed variability of impacts across different delivery centers in a project

• Agencies: ability to cope with growth in transactions was enhanced

• Citizens strongly supported the idea that more agencies need to be computerized, but not necessarily through government investments

Deepak T. Bhatia 20

Conclusions

• Undertake ex-post evaluation of successful ongoing projects.

• Can be used for for ex-ante evaluation, with following assumptions

– extent and adoption rate of user demand need to be estimated– Project designs and investment levels must be commensurate with

anticipated benefits to users. – quality of the enabling environment should be assessed to gauge the

expected impacts of planned projects – Detailed surveys at the initial stages of project planning, would help

establish a baseline.

Deepak T. Bhatia 21

Lessons Learned

• IT cannot be a goal in itself - projects must have a direct link to process reform

• E-Government is a long term endeavor: not a silver bullet, not for immediate gains

• ‘E’ stands for ‘Everything-but-IT’: institutional change, political support, capacity development.. technology is the easy part

• Ultimately can generate significant cost and efficiency gainsBut requires…

Deepak T. Bhatia 22

Focusing on results

Deepak T. Bhatia 23

Credits

• Impact Study team

• Subhash Bhatnagar , IIM, Ahmedabad, India (Primary Author)

• Jiro Tominaga, WB

• Deepak Bhatia, WB

• Shirin Madon, London School of Economics

• Presentation research

• Asheeta Bhavnani, WB

• Mikhail Bunchuk, WB

• Reviewers

• David L. Cieslikowski• Christine Zhen-Wei Qiang

Deepak T. Bhatia 24

Annex

Deepak T. Bhatia 25

How have governments been measuring results of e-government investments?

• France – Mareva - A Method of Analysis and Value Enhancement

• Germany -WiBe - Economic Efficiency Assessment Methodology

• USA – Performance Reference Model Framework

• Australia: Demand and Value Assessment Methodology

• All these government methodologies broadly focus on the same dimensions.

• They offer two levels of impact assessment:

– first, in terms of how projects provide a business case justification for expenditure and whether they meet the targets set for them,

– and second, in terms of how projects meet the goals of the agencies concerned and, in turn, how that helps achieve wider government strategies.

Deepak T. Bhatia 26

World Bank and Impact measurement experience

• What is the WBs experience with measuring impact?

• Independent evaluation group - IEG measures impact of Bank programs and projects, and draws lessons from experience. The Bank also has a large program of impact evaluation although there may not be any that have centered around the ICT sector.

• Whether WB prescribes ONE methodology to all countries? How much tailoring is needed?

• IEG generally uses objective based methodology to measure impact. But if this question refers to tailoring strategies then the key finding from IEG work is that strategies including ICT must be tailored to country circumstances.