ICT IMPACT INDICATORS:LINKING DATA FROM DIFFERENT SOURCES

Partnership Event, GenevaWednesday May 28th 2008

An EU initiative with 13 National Statistics Offices

Tony ClaytonUK Office for National Statistics

Developing Impact Indicators

Agenda

• Background to ICT indicator development• Limits to surveys – what we actually measure• Impact assessment; evidence from firms• Using firm level data to develop indicators –

the advantages• …. and the challenges to overcome

Background to ICT indicators

Level ofElectronicCommerceActivity

Time

Readiness

Intensity

Impact

OECD ‘diffusion’ model, based on linear view

• Readiness; ability of firms, individuals to adopt technology

• Intensity / use; measures of the proportion of firms, individuals who adopt, and the amount of use

• Impact; changes in behaviour, and in economic structure and performance as result of use

Background to EU ICT indicators

E- Europe indicators to 2005 focused on use• Citizens

- access or ownership of technology

- use of internet, and applications

• Enterprises- access to or ownership of technology

- investment in networks / availability of broadband

- use of internet and e-business links

• Government- services available on line and degree of take-up

- investment in specific areas, e.g. education

• Impact?- Only e-commerce, and some individual metrics, to represent

changed behaviour resulting from use

Background to EU ICT indicators

i2010 more focused on impacts

• EU Information space- development of broadband availability, take-up, speed and pricing

- multiplatform access in firms

• Advanced services- focused on household / individual uses of range of internet services

• Security- incidence of problems and solutions

- attitudes to risk / trust and confidence

• Impact- ICT investment, use and impact on growth / productivity

• ICT innovation- ICT R&D and effects on business transformation

Background to ICT indicators

‘Impacts’ now need to reflect innovation in ICT use

• Innovation no longer a linear process- depends on networks / feedback / co-creation

• ICT as ‘general purpose technology’- major impacts on economic performance based on use

- dependent on range of complementary investments

• Innovative uses particularly strong in services- majority of ICT investment in market services

- returns to IT investment strongest in services

- .. where growth, productivity and innovation are hardest to measure

• Pervasive presence of ICT- impacts too important not to be quantified, as part of economic

measurement

Limits to surveys

Measuring ICT investment is hard to do

• Much existing data doesn’t reflect ICT properly as capital- macroeconomic treatment of software investment is inconsistent

between countries

- firm level source data misses out large parts of ‘own account’ IT software investment – because the respondents can’t quantify it

- as ICT service purchases, and IT enabled services, substitute for ‘own investment’, can we pick up the effects?

- (Finland results for this study suggests productivity gains from IT outsourcing will ensure this grows)

• Measurement differences can be very significant …..

Limits to surveys … and macro data

0 0.5 1 1.5 2 2.5

Sweden

Britain 'New'

United States

Finland

Australia*

Japan*

Germany

France

Britain 'Old'

Software investment as % GDP 2003

* 2002

Economist Feb 2006

Limits to surveys

What else have we learned about indicators?

• Surveys about ‘self assessed’ impacts not reliable- assessment of payback from ICT can be inconsistent

- links to actual outcomes from production surveys do a better job

• Questions on technology less useful than what is done with it- technology changes, and respondents don’t always understand it

- effects on behaviour may be a better guide to economic effects

• ICT indicators are sensitive to structural differences- industry characteristics can be stronger than national variations

- member state estimates may not be comparable, in spite of similar methods.

Who we are

• 8 NSIs directly involved in developing research, metadata and method

- UK, Sweden, Finland, Denmark, Italy, France, Austria, Netherlands

Slovenia involved through Ljubljana University

• 4 more joined to use method and contribute to analysis

- Germany, Norway, Czech Republic and Ireland

• Academic support from Free University of Amsterdam

What we’ve done

• Firm level analysis within countries, linking surveys on ICT use, output, productivity, IT investment, skills, innovation …

Core variables from common EU ICT use survey‘Add-on’ analysis in countries with extra survey data

• Built metadata system to aggregate data for industries / countries consistently

Country / industry indicators built using survey dataAble to bring in other data from National AccountsAnalysis across different environmentsNew method for indicators

From firm data to macro indicators

What we’ve done …..

• Firm level analysis within countries, linking surveys on ICT use, output, etc to identify …- core variables from common EU ICT use survey

- ‘add-on’ analysis in countries with extra survey data

• Built metadata framework to aggregate this data for industries / countries on consistent basis- country / industry indicators built using survey data

- able to bring in other data from National Accounts

- analysis across different environments

- new method for constructing indicators

From firm data to macro indicators

Industry / country aggregates help

• We can aggregate to comparable intermediate aggregate data- metadata used to generate completely comparable aggregated data by

country / industry / firm size / ownership / market characteristics etc..

- then we can analyse at this level

• From ‘industry / country’ analysis we get- ability to do analysis across different national policy environments.

- ability to bring in data not available at firm level

- quantification of effects at macro level – £billion more persuasive than a regression coefficient!

Res

earc

her

Policy QuestionResearch Design Program Code

Publication

Net

wor

k Metadata

Networkmembers

Cross-countryTables

NS

Os

Provision of metadata.Approval of access.Disclosure analysis

From firm data to macro indicators

• Assemble the data- put production survey, ICT use survey and business register datasets

in single research environment

• Construct the metadata- describe each survey in standard terms; variable names, format,

range,

- test linkages between units in surveys using register identifiers

• Run analytical code on datasets, using metadata to access and interpret

- code uses SAS or STATA operating within NSI

- produces results / aggregates which can be shared

- disclosure checking before use (can build into code)

What NSIs have to do to join …..

From firm data to macro indicators

ICT metrics

Variables for most countries ‘Add-on’ variablesFirm uses computers

% workers using computers

Firm uses internet

% of workers with access to internet

Firm has website

Firm orders via internet (or EDI etc)

% of orders (or value)

Firm sells via internet (or EDI etc)

% of sales (or value)

Firm has intranet

% of workers with access to intranet

Firm has fast internet access

% of workers with access to fast internet access

Use of electronic business processes

Mobility of access to technology

Mobility of workers accessing technology

IT investment (or expenditure) at firm level

From firm data to macro indicators

Contextual variables / complements to ITVariables for most countries ‘Add-on’ VariablesFirm size

Firm age

Employment

Payroll

Capital Stock

Ownership (local or international)

Outsource ratios (measured as purchases / sales or similar)

Measures of labour ‘quality’ (measured in terms of pay per person)

Industry characteristics (growth, entry / exit, firm share turbulence)

Multinational organisation of firms (from FDI and ownership data)

Employee skills (from linked employer – employee datasets)

Changes in organisation (from some local surveys, or CIS)

Measures of innovation (from CIS)

From firm data to macro indicators

Measures of Impact

Levels Growth ratesProductivity measured as- Gross sales / employee (vs industry)- V.A. / employee (vs industry)

Productivity measured in terms of Multi-Factor Productivity (for countries which have firm level data on investment, IT investment, labour and other inputs)

Productivity change measured as- Gross sales / employee (vs industry)- V.A. / employee (vs industry)

Growth rates measured for:- Gross sales (deflated by industry PPI)- Market share of IT using firms in industry- Value added (double deflated where possible)

- Employment, or share of employment in industry

From firm data to macro indicators

country Elecom MexElec DISTR FINBU TOT

SWE7 11 9 10 11

NOR9 10 8 9 10

FIN8 9 10 8 9

GBR6 8 5 4 8

NLD4 6 4 6 7

SLO5 7 7 6

GER7 5

CZE3 4 6 5 4

AUT5 3 3 3 3

ITA1 2 2 1 2

IRE2 1 1 2 1

From firm data to macro indicators

…and what it can deliver (cont)(example; index of fast-internet enabled workers)

From firm data to macro indicators

…and what it can deliver (cont)(example; index of purchases made through the internet)

country Elecom MexElec DISTR FINBU TOTNOR

10 12 11 11 12

FRA9 10 9 7 11

AUT8 9 10 8 10

GER11 9

SWE7 7 3 10 8

FIN4 4 7 4 7

CZE5 8 6 9 6

IRE6 5 8 5 5

GBR3 6 4 6 4

NLD2 3 5 3 3

ITA1 2 2 1 2

SLO. 1 1 2 1

From firm data to macro indicators

…and what it can deliver (cont)(example; index of sales made through the internet)

country Elecom MexElec DISTR FINBU TOTNOR

10 12 11 11 12

AUT9 11 10 10 11

FRA8 10 8 9 10

GER. 9 . . 9

CZE4 8 6 5 8

FIN7 7 7 4 7

IRE6 5 5 7 6

NLD5 4 9 6 5

GBR1 6 4 2 4

SWE3 2 3 8 3

ITA2 3 2 1 2

SLO. 1 1 3 1

Impact evidence from firms

What microdata contributes to indicators

• From micro analysis we get- clear evidence on how inputs to innovation affect behaviour, competition

and performance at level where decisions are made

- pointers to policy ’levers to pull’ in terms of firm level incentives

- limited ability to compare drivers of performnce between countries

• Advantages and disadvantages- firm level analysis is flexible, fast, and uses ‘all the data’ in terms of

variation / degrees of freedom

- but international comparisons of microdata analysis are vulnerable to differences in surveys / definitions / data treatments

Impact evidence from firms

Earlier firm level analysis showed …..

• On productivity impact - IT investment and use associated with higher productivity measured

through labour productivity or MFP (for a limited number of countries)

- Productivity of IT effects stronger in US owned firms, in multinationals, in younger and more innovative firms, and in services

• On IT investment and CT deployment- links between communications infrastructure and firm investment IT are

significant (for UK at least)

- broadband using workers have a significant separate effect on productivity, over and above IT investment

Impact evidence from firms

What this study has shown from firm level analysis includes …..

• On productivity impact - consistency across 12 countries in productivity effects of core ICT use

metrics for manufacturing firms

- much more difference in services; UK NL, France and Scandinavian analyses show significant positive effects, elsewhere mixed picture

• On IT investment and CT deployment- broadband is a good indicator – or predictor - of firm level purchased

investment

- broadband using workers still have a significant separate effect on productivity in countries where use is greatest

- ‘joining up workers’ has the biggest effect in differentiated services, where much of the investment is in systems and people

Impact evidence from firms

What firm level analysis has shown

continued …..

• On Innovation - evidence that connected workers are more likely to innovate

- .. From UK that firms which get innovative ideas from third parties are more likely to invest in broadband connections

- .. From NL / SE that ICT using firms are more likely to produce innovation outputs, given other inputs, and that productivity effects associated with ICT are strongly influenced by this

- => it’s what firms change with ICT that counts, not what they spend

• On ICT and skills- higher skills levels increase returns from ICT investment, or from ICT

use, whether we use direct measures of skills, or wage measures

Impact evidence from firms

What we’ve shown from firm level analysis continued …..

• On organisation - confirmation that some business process links can be associated with

significant, positive, productivity effects, in some countries

- some of the organisation / maturity variables can be expressed in terms of a composite index … but our surveys need more work

• On flexibility / specialisation- Finnish evidence that flexibility associated with mobile access to IT has

significant productivity benefits

- Finnish evidence that outsourcing of IT services is associated with worthwhile labour productivity advantages

Impact evidence from new indicators

What we show from industry / country level analysis includes…..

• On ICT use- Evidence that broadband connected employees provide a strong

productivity indicator across the contributing countries, over the period 2003-2005

- Looks to be significant across industries, even though firm level effects are uneven

• On ICT use as driver of competition- Cross – EU evidence that higher ICT use, and productivity effects is

associated with more dynamic competition

- Consistent with US conclusion by Brynjolfson et al, that ICT ‘enterprise architecture’ enables firms to replicate successful business models

Country / industry data shows broadband gains:

100

200

300

400

500

600

0 10 20 30 40 50 60 70 80 90 100Labo

ur p

rodu

ctivi

ty in

dex

acro

ss s

ecto

rs (b

ase=

100)

% of employees linked by broadband

Labour productivity and broadband in selected EU countries (2001-04)

..and industry split shows differential impacts :

…..industry splits tend to support UK microdata results showing- IT capital matters most in manufacturing- Broadband enabled employees matter more in differentiated services

Electrical and Electronic Equipment

0

20

40

60

80

100

0 20 40 60 80 100

% of workers with access to broadband

Ho

url

y P

rod

uct

ivit

y

Paper and Publishing

0

20

40

60

80

100

0 20 40 60 80 100

% of workers with access to broadband

Ho

url

y P

rod

uct

ivit

y

Financial Services

0

20

40

60

80

100

0 20 40 60 80 100

% of workers with access to broadband

Ho

url

y P

rod

uct

ivit

y

Advantages

Micro based macro indicators can:

• Use data available at both firm and industry level - industry indicators compiled ‘bottom up’ from linked surveys

- combined with sources from National Accounts or Labour Market

- aim to include innovation indicators in next round of work

• Build comparisons across countries- firm level data can’t be shared by most NSIs, and are different

- metadata approach can iron out differences before analysis

• Allow for structural differences between countries- disaggregation possible on completely comparable basis

- only need to write analytical code once

• It works!- all 13 countries in this project have produced outputs, despite very

different systems

Challenges

Micro based macro indicators need:

• Initial investment in metadata and systems - work to set out dataset and variable descriptions from key surveys

- must be able to bring datasets together within each NSI in a common research environment, to enable them to be linked

• Systems to control confidentiality within each NSI- research / linking framework must meet national non-disclosure

requirements, but still allow firm level data to generate required outputs

• Agreement on confidentiality controls for merged dataset- agreement across NSIs on the level of detail which can be used for

research and analysis, even if it doesn’t meet national quality standards

- agreement on what detail can go into the public domain.

• Persistence!- Work in this project produced lots of data which needs simplifying!