© fedict 2005. all rights reserved user satisfaction & impact measurement belgian practice...

© fedict 2005. All rights reserved

User satisfaction & impact measurement

Belgian practice Description using the Template

Christine Mahieu, Business Analyst FEDICT

24 october 2008

© fedict 2005. All rights reserved 2

Measurement initiatives

2004 : Fed-eView / Administration • Measuring the degree of back-office development

of the federal administrations and their egov readiness 2005-2006 : Fed-eView / Citizen

• Understanding the needs of the citizens 2007 : eGov Measurement Framework Design

• Based on international best practices • Integrated instrument to continuously monitor and

benchmark the development of ICT and eGovernment in Belgium

• Partnerships are essential (Statistics department, regions, municipalities, EU, private partners)

2008 : Implementation of the framework• Consolidation of the dispersed existing indicators• Development of new indicators and collect data in

cooperation with other public and private actors


Fed-eView Citizen A strategic decision support tool

To assess the citizen’s needs, perceptions and use of electronic government, covering also ICT usage and security requirements

To monitor trends and evolutions Methodology :

• Quantitative survey : Panels from both (internet user /non-user) populations of citizens were followed in several waves over 15 months in 2005-2006

• Qualitative survey : Focus groups (for citizens and for entreprises) provided a list of potential applications to be developed end 2006-begin 2007


A. General properties Repeatability

Yes Independence of size

Maybe, not tested Estimated cost of using the methodology

About 125.000 euros for 3 waves of survey and 8 focus groups Prerequisites to carry out and use the measurement

eGov is a horizontal matter, an enabler. Satisfaction about egov services or about public service ?

Stability Yes, this is the primary objective of the methodology used

Accuracy Other surveys (like ICT Household survey - urostat) confirm the results

Conformity to WAI and W3C standards Not a requirement

Number and type of administrations using the method FEDICT and the Ministry of Economy/DG Statistics for some indicators

Multidimensionality • Quantitative survey : 2 types of citizens (internet users and non-users)• Qualitative survey : 2 types of users (citizens and entreprises)


B. Intended use of measurements results

1. Who are the intended users?• Policy makers• Researchers • Public

2. Measurements what for • To assess current status and to monitor

trends and evolution• No measurement on impact


C. Deployment properties

1. Who carries out the measurement?• Outsourced to a neutral third party (team of 3

people + research bureau) 2. How is the measurement carried out?

• Use of a ´mixed mode survey´with two random samples:

• An online panel representative for the Belgian Internet population interviewed in 3 waves via CAWI (panel members were invited by e-mail to participate)

• An offline panel representative for the Belgian internet non-users interviewed in 2 waves via CATI.

• Frequency is used as the criterion of internet use: a minimum of once a month use of the internet.



3. Measurement coverage: subset or exhaustive sampling ?

• The 2 panels constitute representative samples of the Belgian populations of internet users and internet non-users.

• Note: Belgian population is about 10 millions. Online panelInternet users

Telephone panelInternet non-users

Wave I N = 3324 N = 1421

Wave II N = 1480 N = 624

Wave III N = 1139 -


5. Testing No specific test

8. Score calculation and statistics used Percentages Scale of satisfaction : from 0 (absolutely dissatisfied) to 10

(absolutely satisfied) Use of quadrant for combining indicators :



Comparisons, e.g. time series or needs and wants/actual perception Yes


Possession of internet at home 2005

Possession of internet at home 2006

Use of internet 2006%

Yes No No 2.2

Yes Yes No 2.5

Yes No Yes .6

Yes Yes Yes 11.9

No No No 52.0

No Yes No 1.4

No No Yes 14.6

No Yes Yes 14.9

Total 100.0


9. Reporting of the evaluation results The results are summarised in a public

paper in 4 languages (on ePractice) Access to raw data will be possible when

the eGov Monitor will be implemented



> Does the measurement lead to policy actions? • Partially : definition of citizen personalised one-

stop-shop and eBox• But no optimum use of the results

> Lessons learned • Use of panels

• Low-cost way of citizen’s feedback. • Opportunity to go in-depth in the subject• Not useful to measure impact of specific communication

campaigns• Decrease representativity in 15 months time

• Focus groups: selection of people not easy• Measuring the satisfaction of eGov service or of

public service ?

D. Experience from using the methodology


Maintained in the context of the global measurement framework eGov Monitor

E. Maintenance of Measurement Method

© fedict 2005. all rights reserved user satisfaction & impact measurement belgian practice...

Documents