1st International EIBURS-TAIPS TAIPS conference on: “Innovation in the public sector

and the development of e-services”

How advanced are Italian regions in terms of public e-services?

The construction of a composite indicator to analyze patterns

of innovation in the public sector

Luigi Reggi, Davide Arduini, Marco Biagetti and Antonello Zanfei

EIBURS-TAIPS team, University of Urbino

University of Urbino

April 19-20, 2012

Aims and scope • Providing evidence on regional differences in the

diffusion of public eServices in Italy with a focus on

– different types of public eServices: beyond a monodimensional analysis based on e-gov diffusion

– not only front- but also back-end issues

– different channels for service delivery

• Providing a sound, open and transparent methodology for constructing a public eServices composite indicator based on OECD/EC-JRC Handbook

Composite indicators (CI)

A composite indicator is formed when individual indicators are compiled into a single index, on the basis of an underlying model of the multi-dimensional concept that is being measured (OECD Glossary of statistical terms)

• Composite indicators are increasingly used by statistical offices, international organizations (e.g. OECD, EU, WEF, IMF) and academic researchers to convey information on the status of countries in fields such as the environment, economy, society or technological development: Cox et al., 1992; Cribari-Neto et al., 1999; Griliches, 1990; Huggins 2003; Grupp and Mogee 2004; Munda 2005; Wilson and Jones 2002; among others

• The proliferation of these indicators is a clear symptom of their importance in policy-making, and operational relevance in macro and micro economics in general (Granger, 2001)

Searching “Composite indicator” in

Google Scholar => 5x increase in 6 years

(Saltelli, 2011)

Pros and cons of CI

Pros Cons

Can summarize complex or multi-dimensional issues in view of supporting decision-makers.

Easier to interpret than trying to find a trend in many separate indicators.

Facilitate the task of ranking countries on complex issues in a benchmarking exercise.

Can assess progress of countries over time on complex issues.

Reduce the size of a set of indicators or include more information within the existing size limit.

Place issues of country performance and progress at the centre of the policy arena.

Facilitate communication with general public (i.e. citizens, media, etc.) and promote accountability.

May send misleading policy messages if they are poorly constructed or misinterpreted.

May invite simplistic policy conclusions. May be misused, e.g., to support a desired

policy, if the construction process is not transparent and lacks sound statistical or conceptual principles.

The selection of indicators and weights could be the target of political challenge.

May disguise serious failings in some dimensions and increase the difficulty of identifying proper remedial action.

May lead to inappropriate policies if dimensions of performance that are difficult to measure are ignored.

(Saisana and Tarantola, 2002; OECD, 2008)

Field/Source Composite Indicator

Time coverag

e

Number of countries covered

Sub-indicators Aggregation

methodology

United Nations e-Government Readiness Index

2001- 2010

191 Member States

Web presence Telecommunication infrastructure Human capital

Equal weighting

Brown University

e-government index

2001 - 2007

198 Member States

availability of publications, databases and number of on line services

Equal weighting

European Commission / CapGemini

e-government index

2001 - 2010

32 European Countries

Online sophistication of the 20 basic services (4 stage maturity model: information available on-line, one-way interaction, two-way interaction and transaction) Full online availability of the 20 basic services

Equal weighting

Torres et al. (2005)

Service maturity Index

2004 33 EU municipalities

Service Maturity Breadth (number of services offered through the Internet from the 67 identified services) Service Maturity Depth (3 stage maturity model: simple information dissemination, one way communication, service and financial transactions)

Equal weighting

Kovačić (2005) e-Government Readiness Index

2003 95 Member States

Based on United Nations data and methodology Equal weighting

Baldersheim et al. (2008)

Innovation score

2004 75 Nordic municipalities

Information features of the web sites (refers to the contents of communication channels between citizens and town hall) Communication features of the web sites (refers to the extent of interactivity of web sites, or how citizens can actually communicate via municipal sites)

Equal weighting

Arduini et al. (2010)

Front Office Index

2006 1,176 Italian municipalities

Availability and level of interactiveness of 266 on line services Multiple Correspondence Analysis

eGo

vern

men

t

Selected CIs in public e-services field (1/2)

Field/Source Time

coverage

Number of countries covered

Sub-indicators Composite Indicator

Aggregation methodology

European Commission

2010 32 European Countries

- eNotification, eSubmisssion and eAwards services provided by eProcurement platforms in the public sector

- eOrdering, eInvoicing and ePayment services provided by eProcurement platforms in the public sector

eProcurement availability for the pre - award phase eProcurement availability for the post - award phase

Equal weighting

European Commission – Joint Research Centre (Seville)

2010

906 acute Hospitals in the 27 European Countries

- Infrastructure Dimension - Application and Integration Dimension - Information flows dimension - Security and privacy dimension

Composite index of eHealth deployment

Multivariate Statistical Analysis

Horan et al. (2007)

2006

2 County Metropolitan Transportation Authorities (Los Angeles and Minneapolis)

- Real-time network information - Whether traffic or transit - Traveler information such as route guidance or

destination information

Advanced Travel Information Systems Index

Equal weighting

ePro

cure

men

t eH

ealt

h

eTra

nsp

ort

atio

n

Selected CIs in public e-services field (2/2)

Existing CIs in public eServices field

• are specific to a single domain / type of eService

• employ simple equal weighting as standard aggregation method (with a few exceptions)

• do not assess results with Uncertainty or Sensitivity Analysis (UA – SA)

Critical remarks have been raised against EC eGovernment bechmarking index. Criticism is mainly focused on theoretical framework, indicators chosen, aggregation scheme adopted (Bannister, 2007; Bretschneider et al, 2005; Fariselli & Bojic 2004; Goldkuhl & Persson, 2006; Jansen, 2005)

CIs in public e-services field

What is new in our methodology for a Public eServices CI

1. Expanding the scope of the analysis of eServices diffusion – A holistic view to capture the wide spectrum of public e-services in different domains

(in our case: eGov, eEducation,eTransportation) and the different aspects of service provision (e.g. technical and organizational change within PAs and new service implementation)

2. Improving the quality of the framework – Using more sophisticated indicators both on quality of services offered and back

office changes – Robustness check of the framework / classification of indicators

3. Developing a sound, open and transparent methodology – Asking experts to assess the importance of basic indicators – Real benchmarking: measuring the distance from the efficiency frontier – Tracing back the contribution of the different aspects of eService diffusion (e.g. back-

and front-end issues) to intermediate and final indices – Checking the robustness of results by reiterating the calculation of the CI with 12

other different methods (Uncertainty Analysis)

Dimensions of ICT diffusion Service provision - front end Internal processes / interoperability / information integration - back end (Millard, 2004; Pardo and Tayi, 2007; OECD, 2007)

Decision- / policy –making (Lampathaki et al., 2010)

Channels Institutional websites, public websites Public kiosks Digital TV Mobile apps (Pieterson et al., 2008)

Domains eGovernment eEducation eTransportation eHealth Smart cities

Public e-Services diffusion: a broad definition

Aims Efficiency and effectiveness of public service (Fountain, 2001; Codagnone e Undheim, 2009)

Transparency (Wong & Welch, 2004; Meyer, 2009, Dawes 2010)

Participation (Noveck, 2008)

Providers Government: central / local agencies, public companies Third party players - PPPs, apps development (Brito, 2009; Eaves, 2010)

NGOs, citizens - self-help, collaboration (Noveck, 2008)

Data sources Government Citizens / NGOs / businesses: crowdsourcing (Osimo, 2008; Robinson et al., 2009; Chun et al., 2010)

Main focus of

existing CIs /

benchmarking

exercises

Scope of our

analysis

Public eServices CI - our framework -

• Existing theoretical frameworks are mainly focused on eGovernment and based on stage models implying linear progression (Lee, 2010)

[i.e. from stage 1 = input/eReadiness to stage n = outcome] – academic papers (Andersen & Henriksen, 2006; Hiller & Belanger, 2001; Layne & Lee,

2001; Moon, 2002; Siau & Long, 2005; Scott, 2001; West, 2004) – institutional reports (Center for Democracy & Technology, 2002; Grant & Chau, 2005;

United Nations, 2001, 2003, 2005, 2008) – private consulting firms reports (Accenture, 2003; Deloitte Research, 2000)

• Most available frameworks can hardly be applied to the construction of our CI

“Too often composite indicators include both input and output measures. […] However, only the latter set of output indicators should be included if the index is intended to measure innovation performance”

(OECD/EC-JRC Handbook on Constructing CIs, p.6)

Public eServices CI - our framework -

₋ Interactive

whiteboards

₋ Repositories of

documents

₋ Wiki platforms

₋ School website

₋ Restricted areas

for information

services

₋ Intranet

₋ Interoperability

& integration

₋ Fully interactive

service provision

₋ On line payments

₋ Multi-channel

delivery

₋ Certified e-mail

₋ eProcurement

₋ Document

workflow

₋ Technology on

board of public

transport

₋ electronic displays

on the street

₋ Travel planner

₋ Info on traffic

and parking

₋ Multi-channel

delivery

₋ Mobility

monitoring

systems

₋ Interoperability

& integration

Public eServices Composite Indicator PILLAR

SUB-PILLAR INDICATORS

didactics

Data sources

Domain Statistical units Source

eEducation 1,600 schools Between. Survey “Service e-

Platforms”, 2010

eGovernment 5,762 municipalities, 100

Provincial governments

and 22 Regional

governments

Italian Institute of Statistics. Survey

“Information and Communication

Technologies in Local Public

Administrations”, 2009

eTransportation 117 local public transport

companies

Between. Survey “Service e-

Platforms”, 2011

Valle d’Aosta and Molise (0,7% of total

Italian population) were excluded from

the analysis due to poor data quality in

the eTransportation survey

Basic indicators selection & robustness check of the framework

• An initial set of 30 indicators were assigned to each “pillar” (e-service domain) and “sub-pillar” (aspect of innovation activity being considered)

• 8 Principal Component Analyses and KMO tests were performed (1 for each sub-pillar) to check the consistency of the framework

• We applied the eigenvalue-one criterion [only one eigenvalue should exceed the unity (Kaiser, 1960)] to make sure that indicators in each sub-pillar share no more than 1 underlying dimension

• 6 indicators that did not pass this test have been discarded

Pillar Sub-pillar code BASIC INDICATORS SELECTED eG

overn

men

t ICT and changes in internal

organization

G1.1 Municipalities with certified e-mail

G1.2 Municipalities using e-procurement

G1.3 Municipalities using document workflow (full case handling)

Online services

G2.1 Municipalities providing fully interactive services on the web

G2.2 Municipalities allowing online payments

G2.3 Channels other than the web used to offer public services

eTra

ns

po

rtati

on

ICT during transportation

T1.1 No. of technological systems on board

T1.2 Cities providing information to travelers about traffic or parking by means of electronic displays

T1.3 Buses with on-board computer

ICT and changes in internal

organization

T2.1 Cities with data interchange with other entities

T2.2 Cities with a managing authority for local mobility T2.3 CIties with a mobility monitoring system

Online services

T3.1 No. of channels used to inform passengers

T3.2 Cities that provide information to travelers about traffic or parking on the web T3.3 Cities that offer timetables with route planning (travel planner) on the web

eE

du

cati

on

ICT in didactics

E1.1 Teachers using interactive whiteboard E1.2 Schools extensively using online text and file/document collections E1.3 Schools extensively using wiki platforms

Online Services

E2.1 Schools with website E2.2 Schools providing restricted access areas for web-based info services to teachers

E2.3 Schools providing tools to share training aid files on the web (assignments. audio/video of lessons. etc.)

ICT and changes in internal

organization

E3.1 School information system integrated with the National Educational Information System

E3.2 Schools information system integrated with the National Library System E3.3 Schools with Intranet

Steps for computing CI

• What is the relative importance of each Basic Indicator?

• How to aggregate the Basic Indicators in order to measure the level of development of each region in eEducation, eGovernment and eTransportation?

• How to calculate the final score?

• What is the robustness level of the results we obtained?

Gathering expert opinion through Budget Allocation (BA)

What is BA? Experts are given a “budget” of N points, to be distributed over a number of individual indicators by “paying” more for those indicators whose importance they want to stress. (Moldan and Billharz 1997)

Phases: 1. Selection of experts for

the evaluation

2. Allocation of budget to indicators

3. Calculation of weights

(a) Randomly selected from the corresponding authors of 751 top-journal articles reviewed by Arduini and Zanfei (2011). => 100 papers extracted.

(b) Also included 15 participants at the 1st International EIBURS-TAIPS Conference that present papers on eServices diffusion

An on-line questionnaire was administered. Experts were asked to allocate a 100 points budget within each sub-pillar, so that the total number of indicators to evaluate is < 4 (Bottomley et al., 2000)

Results of BA

No expert consensus on the appropriate set of weights (Mean coef of var among indicators = 0.4426)

– High variation / disagreement

– No single pair of expert suggesting similar weights

We must choose a statistical method to calculate weights, while trying not to waste the information provided by the experts

0

20

40

60

80

100

E1.1

E1.2

E1.3

E2

.1

E2.2

E2.3

E3.1

E3

.2

E3.3

G1.1

G1.2

G1

.3

G2.1

G2.2

G2.3

T1

.1

T1.2

T1.3

T2.1

T2.2

T2.3

T3.1

T3.2

T3.3

Mean Max Min Median

Combining Benefit of the Doubt (BoD) approach with expert opinion

• BoD is a method for data aggregation based on Data Evelopment Analysis (DEA) (Melyn & Moesen, 1991, Cherchye et al., 2007)

• BoD advantages – objective statistical/mathematical approach – it measures “efficiency” => compares a region’s performance

with a benchmark in a multi-dimensional space – the algorithm tends to use those indicators where the region

shows better performances • no other weighting scheme yields higher composite indicator value

(political acceptance) • reveals policy priorities / past choices • embeds concern for regional diversity

• BoD + Expert constraint (Cherchye et al., 2008)

– We impose that the use of each indicator is limited by expert opinion. The MIN (MAX) use of an indicator corresponds to the MIN (MAX) weight it has received from the experts

Indicator = ratio of the distance between the origin and the actual observed point and that of the projected point in the frontier

Source: rearranged from Mahlberg and Obersteiner (2001)

Through DEA we estimate an efficiency frontier used as a benchmark to measure the relative performance of regions

Benefit of the Doubt (BoD) approach through Data Envelopment Analysis (DEA)

In our case, CIs of the 3 pillars

are the distance from an ideal

case with 100% on all basic

indicators

Benefit of the Doubt (BoD) approach through Data Envelopment Analysis (DEA)

Linear programming problem

bounding constraint

non-negativity constraint

s.t.

(Charnes et al, 1978)

indicators weights j indicates the region

i indicates the indicator

The “pie-share” constraint

• Applying only the bounding and the non-negativity constraints may allow for extreme scenarios (Cherchye L., 2008)

– If a region’s value of one sigle indicator dominates those of other regions, that region will get the max score of 1 even if it has very low values in the other indicators

• We introduce a pie-share constraint that incorporates expert opinion (Wong and Beasley, 1990)

Li = lower bound = MIN expert weight from BA

Ui = upper bound = MAX expert weight from BA

Results

• In the following slides the resulting scores and ranks from the constrained optimisation are presented

• The score:

– represents a measure of a region’s efficiency compared to the benchmark (the “ideal case”)

– is the sum of the pie-shares of each indicators, that we have grouped toghether at a sub-pillar level (aspect of innovation activity being considered)

-

0,20

0,40

0,60

0,80

LOM EMR LAZ VEN TOS CAL BOZ SAR PMN PUG ABR MAR LIG CAM UMB BAS FVG SIC TRE

0,00

0,10

0,20

0,30

0,40

0,50

EMR BOZ TOS VEN LOM MAR FVG UMB PMN PUG SIC CAM LAZ LIG SAR CAL ABR BAS TRE

-

0,20

0,40

0,60

0,80

1,00

BOZ EMR TRE LIG FVG TOS MAR UMB CAM VEN LOM CAL PMN BAS SAR ABR LAZ PUG SIC

Online Services ICT and changes in internal organization ICT in didactics (eEdu) or during transportation (eTran)

eE

du

ca

tio

n

eG

ove

rnm

en

t e

Tra

ns

po

rtati

on

Results per pillar (1/4) scores

• The highest variation in the scores can be found in eTransportation domain, while eEducation performances seem not to vary much

• eGov results for Lombardy, Piedmont and Province of Trento are lower than expected.

– This is probably due to the high proportion of very small municipalities

Results per pillar (2/4) rankings

• The 3 rankings differ substantially

=> significantly different regional patterns

– Very high variations in the ranking for the Province of Trento and Lazio. Medium-high variation for Lombardy, Calabria, Campania

– Other regions show a more homogeneous approach to public eServices development which is characterized by different trajectories of diffusion

• High scores for EMR, TOS, BOZ | medium scores for VEN MAR PIE | low scores SIC, BAS

Results per pillar (3/4) pie shares

Tracing back pillar results through “pie shares”

• eEducation - Pie shares are more or less fixed, i.e. all regions use the same “mix” of indicators to maximize their score, under the expert constraint. This is due to quite similar relative values of each indicator and to the specific combination of bounds that experts have imposed

• eTransportation – Pie shares are flexible, so each region chooses its own set of weights revealing the areas where investments have been made

• eGovernment – intermediate case

Results per pillar (4/4) pie shares

• Indicators related to ICT diffusion in internal processes and organizational changes have a major role in computing the final score of all public e-services categories (eEdu, eGov and eTra)

• The importance of back office re-organization through ICTs has emerged in the literature on the development of organizations, which has emphasized the essential role of skills that characterize the different components of an organizational structure (Fountain and Osorio-Ursua, 2001; Fountain, 2003; West, 2005; Helfat et al., 2007)

1. Normalization: MIN-MAX, where MAX is the region with the highest score

2. Final aggregation through Geometric Mean – the marginal gain of an increase

in a low score is much higher than in a high score

– a region has more incentive to address the dimensions where it is weak

Final steps to the CI

Region CI Rank

EMR 0,94 1

BOZ 0,93 2

TOS 0,80 3

VEN 0,73 4

FVG 0,70 5

MAR 0,69 6

LIG 0,68 7

LOM 0,67 8

UMB 0,65 9

CAM 0,63 10

PMN 0,59 11

CAL 0,57 12

TRE 0,53 13

LAZ 0,52 14

PUG 0,52 15

SAR 0,51 16

ABR 0,45 17

BAS 0,45 18

SIC 0,38 19

Final scores and rank

Uncertainty Analysis (UA)

• UA is a robustness assessment of a CI (Saltelli et al, 2008)

• The uncertainties in the development of a composite

indicator will arise from some or all of the steps in the construction line (Saisiana et al, 2004)

(a) selection of subindicators (b) data selection (c) data editing (d) data normalization (e) weighting scheme (f) weights' values and (g) composite indicator formula (e) level of aggregation where the methodology applies

12 alternative scenarios (+ baseline) weighting scheme level of the method Aggregation Data normalization

S1

DEA Pie shares (min-max BA) domains Geometric No rescaling

S2 BA mean weight+EW+EW sub-pillars Additive Minmax

S3 BA median weight+EW+EW sub-pillars Additive Minmax

S4 BA mean weight+EW+EW sub-pillars Additive, geometric on

domains Minmax

S5 BA median weight+EW+EW sub-pillars Additive, geometric on

domains Minmax

S6 DEA Pie shares (min-max BA) domains Additive No rescaling

S7 EW - Additive Minmax

S8 PCA+EW+EW sub-pillars Additive on pillars and

domains Minmax

S9 PCA+EW+EW sub-pillars Additive on pillars,

geometric on domains Minmax

S10 PCA+PCA+EW sub-pillars+pillars Additive Minmax

S11 PCA+PCA+EW sub-pillars+pillars Additive, geometric on

domains Minmax

S12 PCA+PCA+PCA sub-pillars+pillars+domains Additive Minmax

S13 PCA+PCA+PCA sub-pillars+pillars+domains

Additive, geometric on domains Minmax

Results of UA

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00EM

R

BO

Z

TOS

VEN

FVG

MA

R

LIG

LOM

UM

B

CA

M

PM

N

CA

L

TRE

LAZ

PU

G

SAR

AB

R

BA

S

SIC

Highest uncertainty:

range = 0.367

Lowest

uncertainty:

range = 0.167

Differences in rankings compared to baseline scenario

Regions S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S Borda S Condorcet

PMN -3 -3 -3 -3 0 -1 -2 -2 0 0 2 0 -1 0

LOM 4 4 4 4 1 4 4 4 5 5 5 3 5 5

BOZ 0 0 0 0 0 0 0 0 0 0 0 0 0 0

TRE -2 0 -3 -3 1 -3 -3 -3 -2 -3 -3 -5 -2 -2

VEN -1 -1 -1 -1 0 -1 -1 -1 -1 -1 0 1 -1 -1

FVG -1 -1 -1 -1 0 -1 -1 -1 -1 -1 -2 -2 -1 -2

LIG -3 -3 -3 -3 1 -2 -3 -4 -5 -5 -6 -6 0 1

EMR 0 0 0 0 0 0 0 0 0 0 0 0 0 0

TOS 0 0 0 0 0 0 0 0 -1 -1 -2 -1 -1 -1

UMB 0 0 1 1 0 1 1 1 -1 1 -1 0 1 1

MAR -1 -1 -1 -1 -2 -1 -1 -1 -1 -1 0 0 -4 -6

LAZ 6 6 5 5 0 4 5 5 6 4 6 6 5 4

ABR 0 0 0 0 0 0 0 0 0 0 0 1 0 0

CAM -3 -5 -3 -3 0 -5 -1 0 1 1 -2 -2 -4 -4

PUG 4 3 4 3 0 4 1 1 2 2 4 5 4 6

BAS 0 0 0 0 0 0 0 0 -1 -1 -1 -1 0 0

CAL 0 1 0 1 -1 -2 0 0 -2 -2 -2 -2 -1 -4

SIC 0 0 0 0 0 0 0 0 1 1 1 2 0 0

SAR 0 0 1 1 0 3 1 1 0 1 1 1 0 3

Results of UA

• CI final scores based on BoD weights are among the best possible results a region can obtain

• Good robustness level, especially for top and bottom ranked regions

– 13 regions out of 19 show only a 0/1/-1 shift compared to the median rank

Conclusions 1/2

From a methodological point of view

– BoD approach combined with BA is an effective way incorporate both regional choices and expert judgment into CI

– Geometric aggregation gives higher scores to regions showing a more balanced eServices diffusion among the 3 domains

– Uncertainty analysis on rankings shows high robustness levels for top and bottom ranked regions

Conclusions 2/2

Main findings and implications from our analysis: – ranking reflects hierarchy of regions in terms of per capita

income and industrial development: current development of public eServices does not seem to correct unbalances between regions lagging behind and frontrunners

– high heterogeneity in terms of mix of e-service proficiency: need for a regional differentiation of e-service promotion policies ;

– there is more cross regional variation in terms of eEducation and eTransportation than in terms of eGov: human capital formation and mobility enhancing are bound to be the real distinctive assets of regions