· d4.6 pilots evaluation results – third round page 2 of 215 deliverable factsheet project...

of 215

OpenGovIntelligence

Fostering Innovation and Creativity in Europe through Public

Administration Modernization towards Supplying and Exploiting

Linked Open Statistical Data

Deliverable 4.6

Pilots Evaluation Results – Third Round

Leading partner: Technical University of Delft (TUDelft)

Participating partners: TUDelft

Version-Status: V1.0

Dissemination level: CO

D4.6 Pilots Evaluation Results – Third Round

of 215

Deliverable factsheet

Project Number: 693849

Project Acronym: OpenGovintelligence

Project Title: Fostering Innovation and Creativity in Europe through Public Administration Modernization towards Supplying and Exploiting Linked Open Statistical Data

Deliverable title: Pilots and Evaluation Results – Second Round

Deliverable number: D4.6

Official submission date: 31/01/2019

Actual submission date: 31/01/2019

Editor(s): Ricardo Matheus (TUDelft)

Author(s): Ricardo Matheus (TUDelft), Marijn Janssen (TUDelft), Dhata Praditya (TUDelft)

Reviewer(s) All partners

Abstract: This report presents the evaluation results for the OGI project. It was updated taking into consideration the 2nd evaluation report (D4.4) and the 3rd round of Pilots plan (D4.5). The evaluation has four areas: Co-creation, ICT Toolkit, Acceptance of ICT Toolkit and Outcomes.


of 215

Effort of Participating Partners Consortium

Name Short

Name

Role Person

Months

1. Centre for Research & Technology - Hellas CERTH Coordinator 2

2. Delft University of Technology TUDelft R&D Partner 4.0

3. National University of Ireland, Galway NUIG R&D Partner 12

4. Tallinn University of Technology TUT R&D Partner 5

5. ProXML bvba ProXML R&D Partner 0.2

6. Swirrl IT Limited SWIRRL R&D Partner 0.1

7. Trafford council TRAF Pilot Partner

8. Flemish Government VLO Pilot Partner

9. Ministry of Administrative Reconstruction MAREG Pilot Partner 0.55

10. Ministry of Economic Affairs and Communication MKM Pilot Partner

11. Marine Institute MI Pilot Partner 0.5

12. Public Institution Enterprise Lithuania EL Pilot Partner


of 215

Revision History

Version Date Revised by Reason

0.1 01/11/2018 TUDelft Initial setup by Ricardo Matheus

0.4 15/01/2019 TUDelft Internal Review by Marijn Janssen

0.9 25/01/2019 Internal reviewer Internal reviewer

1.0 31/01/2019 TUDelft Final Version

Statement of originality:

This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has been made through appropriate citation, quotation or both.


of 215

Table of Contents

DELIVERABLE FACTSHEET ............................................................................................................2

EFFORT OF PARTICIPATING PARTNERS CONSORTIUM ..................................................................3

REVISION HISTORY ......................................................................................................................4

TABLE OF CONTENTS ...................................................................................................................5

LIST OF FIGURES ..........................................................................................................................7

LIST OF TABLES ...........................................................................................................................9

EXECUTIVE SUMMARY .............................................................................................................. 11

1 INTRODUCTION ................................................................................................................. 13

1.1 SCOPE .................................................................................................................................................... 13

1.2 AUDIENCE............................................................................................................................................... 13

1.3 STRUCTURE ............................................................................................................................................. 13

2 EVALUATION OVERVIEW .................................................................................................... 14

3 EVALUATION METHODOLOGY ............................................................................................ 16

3.1 PILOTS’ CO-CREATION EVALUATION METHODS .............................................................................................. 20

3.1.1 Selected Co-Creation Tools and Methods Overview ....................................................................... 21

3.2 PILOTS’ ACCEPTANCE AND OUTCOMES EVALUATION METHODOLOGY ................................................................. 25

3.2.1 Pilots’ Acceptance Selected Methodology ...................................................................................... 25

3.3 OGI ICT TOOLKIT EVALUATION METHODOLOGY ............................................................................................ 34

3.3.1 OGI Pilots’ Data Quality Evaluation Methodology .......................................................................... 38

3.3.2 The standard for Systems and Software Quality Requirements and Evaluation – ISO 25010 and ISO

25012 39

3.3.3 User Experience (UX) and User Interface (UI) testing ..................................................................... 43

4 OGI EVALUATION RESULTS ................................................................................................. 46

4.1 PILOTS’ EVALUATION RESULTS .................................................................................................................... 47

4.1.1 Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece) ........................................ 48

4.1.2 Pilot 2 – Enterprise Lithuania (Lithuania) ........................................................................................ 65

4.1.3 Pilot 3 – Tallinn Real Estate (Estonia) ............................................................................................. 81

4.1.4 Pilot 4 – Trafford Council Worklessness (England) ........................................................................ 103

4.1.5 Pilot 5 – The Flemish Environment Agency (Belgium) ................................................................... 122

4.1.6 Pilot 6 – Marine Institute (Ireland) ................................................................................................ 137

4.1.7 Overall Results From Pilots’ Evaluation ........................................................................................ 159

4.2 OGI ICT TOOLKIT EVALUATION RESULTS .................................................................................................... 163

4.2.1 Internal Evaluation ........................................................................................................................ 164

4.2.2 External Evaluation ....................................................................................................................... 178

5 CONCLUSIONS ................................................................................................................. 182

6 REFERENCES .................................................................................................................... 184


of 215

7 ANNEXES ......................................................................................................................... 187

7.1 END-USER QUESTIONNAIRES .................................................................................................................... 187

7.1.1 Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece) End-User Questionnaire

187

7.1.2 Pilot 2 – Enterprise Lithuania (Lithuania) End-User Questionnaire ............................................... 190

7.1.3 Pilot 3 – Tallinn Real Estate (Estonia) End-User Questionnaire .................................................... 194

7.1.4 Pilot 4 – Trafford Council Worklessness (England) End-User Questionnaire ................................. 197

7.1.5 Pilot 5 – The Flemish Environment Agency (Belgium) End-User Questionnaire ............................ 201

7.1.6 Pilot 6 – Marine Institute (Ireland) End-User Questionnaire ......................................................... 205

7.2 DATA QUALITY QUESTIONNAIRE ............................................................................................................... 209

7.3 OGI ARCHITECTURE QUESTIONNAIRE ......................................................................................................... 211


of 215

List of Figures FIGURE 1- INTERCONNECTIONS AND INTERDEPENDENCIES BETWEEN OGI WORKING PACKAGES AND

DELIVERABLES ............................................................................................................................................. 15 FIGURE 2 - EVALUATION DIMENSIONS ................................................................................................................... 16 FIGURE 3 - HIGH-LEVEL PROCESSES OF PILOT PLAN ............................................................................................. 18 FIGURE 4 - PILOTS' TIMELINE ................................................................................................................................ 19 FIGURE 5 – EXPLORATORY AND EXPLANATORY APPROACHES AT CO-CREATION EVALUATION SURVEYS ............ 22 FIGURE 6 - ADAPTED TRANSPARENCY MODEL FOR OGI EVALUATION ................................................................. 28 FIGURE 7- OGI FOCAL POINT ................................................................................................................................ 29 FIGURE 8- OGI TOOLS AND WORKING FLOW ........................................................................................................ 36 FIGURE 9 - UX STRUCTURE AND LAYERS OF PRODUCT AND INFORMATION ........................................................... 43 FIGURE 10 - THE RIPPLE EFFECT ............................................................................................................................ 44 FIGURE 11 - UX IMPLEMENTATION AND EVALUATION STEPS ................................................................................ 45 FIGURE 12 – COMMON FLOW OF PILOTS ACTIVITIES .............................................................................................. 47 FIGURE 13 – GREEK PILOT CO-CREATION STEPS .................................................................................................. 53 FIGURE 14 – GREEK PILOT APPLICATION CALLIMACHUS ...................................................................................... 54 FIGURE 15 – VEHICLES DATA CUBES SCREENSHOT ............................................................................................... 58 FIGURE 16 - PIE SORTED GRAPH VEHICLE TYPE ................................................................................................... 64 FIGURE 17 - BAR GRAPH FILTERED BY AREA (MUNICIPALITY) ............................................................................. 64 FIGURE 18 – LOSD USER FAMILIARITY – THIRD YEAR ........................................................................................ 76 FIGURE 19 – USER ACCEPTANCE – THIRD YEAR ................................................................................................... 77 FIGURE 20 – DATASETS ACCEPTANCE – THIRD YEAR ........................................................................................... 78 FIGURE 21 – RESULTS – THIRD YEAR .................................................................................................................... 79 FIGURE 22 - TABLE WITH BAR CHART FUNCTIONALITY ......................................................................................... 80 FIGURE 23 - TABLE WITH HEATMAP FUNCTIONALITY ............................................................................................ 80 FIGURE 24 – AUDIENCE OF TALLINN REAL ESTATE HACKATHON ......................................................................... 90 FIGURE 25 – DEVELOPERS CODING FOR THE TALLINN REAL ESTATE HACKATHON ............................................... 91 FIGURE 26 TALLINN APP RESPONDENTS’ FAMILIARITY WITH LOSD ..................................................................... 97 FIGURE 27 TALLINN APP RESPONDENTS USER ACCEPTANCE .................................................................................. 98 FIGURE 28 TALLINN APP DATASETS QUALITY RESULTS ......................................................................................... 99 FIGURE 29 TALLINN APP PERCEIVED OUTCOME ................................................................................................... 100 FIGURE 30 – TALLINN REAL ESTATE APPLICATION ............................................................................................. 101 FIGURE 31 - BAR CHART OF AVERAGE COST ....................................................................................................... 101 FIGURE 32 - PIE CHART DISTRIBUTION OF CRASHES ............................................................................................ 102 FIGURE 33 – LOSD FAMILIARITY – TRAFFORD PILOT ......................................................................................... 117 FIGURE 34 – USER ACCEPTANCE – TRAFFORD PILOT .......................................................................................... 118 FIGURE 35 – DATASETS QUALITY – TRAFFORD PILOT ......................................................................................... 119 FIGURE 36 – PERCEIVED OUTCOMES – TRAFFORD PILOT .................................................................................... 120 FIGURE 37 – TRAFFORD COUNCIL WORKLESSNESS APPLICATION ....................................................................... 121 FIGURE 38 - METADATA OF BELGIUM PILOT DATA SET ...................................................................................... 130 FIGURE 39 – LOSD FAMILIARITY – FLEMISH PILOT ............................................................................................ 132 FIGURE 40- USER ACCEPTANCE – FLEMISH PILOT .............................................................................................. 133 FIGURE 41 – DATASETS QUALITY – FLEMISH PILOT ............................................................................................ 134 FIGURE 42 – PERCEIVED OUTCOMES – FLEMISH PILOT ....................................................................................... 135 FIGURE 43 – THE FLEMISH ENVIRONMENT AGENCY APPLICATION ..................................................................... 136 FIGURE 44 - PERCENTAGE BREAKDOWN OF CO-CREATION CONTRIBUTIONS ........................................................ 145 FIGURE 45 – LOSD FAMILIARITY – MARINE INSTITUTE PILOT ........................................................................... 154 FIGURE 46 – USER ACCEPTANCE – MARINE INSTITUTE PILOT............................................................................. 155


of 215

FIGURE 47 – DATASETS QUALITY – MARINE INSTITUTE PILOT ........................................................................... 156 FIGURE 48 – PERCEIVED OUTCOMES – MARINE INSTITUTE PILOT ....................................................................... 157 FIGURE 49 - AVERAGE BAR CHART OF IRISH NATIONAL TIDE GAUGE NETWORK DATA SET ............................... 158 FIGURE 50 – OVERALL LOSD FAMILIARITY ....................................................................................................... 159 FIGURE 51 – OVERALL USER ACCEPTANCE ......................................................................................................... 160 FIGURE 52 –OVERALL DATASETS QUALITY ........................................................................................................ 161 FIGURE 53 – OVERALL PERCEIVED OUTCOMES ................................................................................................... 162 FIGURE 54 – INTERNAL AND EXTERNAL EVALUATION PROCESSES ..................................................................... 163 FIGURE 55 – DISCOVERABILITY AND USABILITY – OVERALL OF ALL PILOTS ..................................................... 165 FIGURE 48 – GRANULARITY SECTION – OVERALL OF ALL PILOTS ...................................................................... 166 FIGURE 57 – INTELLIGIBILITY SECTION – OVERALL OF ALL PILOTS.................................................................... 167 FIGURE 58 – TRUSTWORTHINESS SECTION – OVERALL OF ALL PILOTS ............................................................... 168 FIGURE 59 –LINKABLE TO OTHER DATASETS SECTION – OVERALL OF ALL PILOTS ............................................ 169 FIGURE 60 – OPEN DATA PRINCIPLES SECTION – OVERALL OF ALL PILOTS ........................................................ 170 FIGURE 61 – USE BEHAVIOUR OF LOSD OGI ARCHITECTURE ............................................................................ 172 FIGURE 62 – PERCEIVED USEFULNESS OF LOSD OGI ARCHITECTURE .............................................................. 173 FIGURE 63 – PERCEIVED EASE OF USE OF LOSD OGI ARCHITECTURE .............................................................. 174 FIGURE 64 – COMPUTER SELF-EFFICACY, PERCEPTIONS OF EXTERNAL CONTROL, COMPUTER PLAYFULNESS,

COMPUTER ANXIETY, PERCEIVED ENJOYMENT, OBJECTIVE USABILITY OF LOSD OGI ARCHITECTURE ... 175 FIGURE 65 –SUBJECTIVE NORM, VOLUNTARINESS, IMAGE, JOB RELEVANCE, OUTPUT QUALITY OF LOSD OGI

ARCHITECTURE........................................................................................................................................... 176 FIGURE 66 –RESULT DEMONSTRABILITY, BEHAVIOURAL INTENTION, USE, PRESENT SATISFACTION, FUTURE

DESIRE OF LOSD OGI ARCHITECTURE ...................................................................................................... 177 FIGURE 67 – NTTS HANDS-ON OGI ICT TOOLKIT WORKSHOP .......................................................................... 178


of 215

List of Tables TABLE 1 - CO-CREATION FRAMEWORK STAGES, METHODS FOR DATA COLLECTION AND TOOLS FOR EVALUATION

..................................................................................................................................................................... 20 TABLE 2 - USER WORKSHOP PROTOCOL FOR CO-CREATION ................................................................................. 21 TABLE 3 – OVERVIEW OF PILOTS’ SELECTED CO-CREATION TOOLS AND METHODS ............................................ 24 TABLE 4 - USER ACCEPTANCE EVALUATION FOR OGI .......................................................................................... 25 TABLE 5 - ADAPTED TRANSPARENCY MODEL FOR OGI EVALUATION .................................................................. 27 TABLE 6 - TAXONOMY FOR BENEFITS FOR OGI INNOVATION ECOSYSTEM STAKEHOLDERS .................................. 29 TABLE 7 – PILOTS’ OUTCOMES EVALUATION QUESTIONNAIRE ............................................................................. 32 TABLE 8 - OGI TOOLKIT RELEASES AND TOOLS PER YEAR ................................................................................... 35 TABLE 9 - OGI TOOLS USED IN PILOTS .................................................................................................................. 37 TABLE 10 – PILOTS’ DATA QUALITY QUESTIONNAIRE .......................................................................................... 38 TABLE 11 - CUBE DESIGN AND BUILDING BLOCKS DATA COLLECTION AND METHODOLOGY METHODS OF

EVALUATION ................................................................................................................................................ 39 TABLE 12 - OGI TOOLKIT REQUIREMENTS FOR EVALUATION ............................................................................... 39 TABLE 13 - CRITERIA FOR EVALUATION OF QUALITY IN USE ................................................................................ 42 TABLE 14 - CO-CREATION USER WORKSHOP QUESTIONS ..................................................................................... 49 TABLE 15 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS ................................................................ 50 TABLE 16 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – FIRST YEAR ......................................... 51 TABLE 16 - SUMMARY OF ACTIONS TAKEN ABOUT IDEAS AND SOLUTIONS FROM PARTICIPANTS – THIRD YEAR

STATUS ......................................................................................................................................................... 52 TABLE 17 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ............................................................................. 55 TABLE 18 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 56 TABLE 19 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 57 TABLE 21 - CO-CREATION USER WORKSHOP QUESTIONS ..................................................................................... 66 TABLE 22 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – FIRST YEAR ......................................... 67 TABLE 23 - SUMMARY OF PROBLEMS, EXPECTED SOLUTIONS AND IMPLEMENTATION - SECOND YEAR ............... 68 TABLE 24 - SUMMARY OF PROBLEMS, EXPECTED SOLUTIONS AND IMPLEMENTATION - THIRD YEAR .................. 69 TABLE 23 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ............................................................................. 71 TABLE 24 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 72 TABLE 25 – DATA QUALITY IMPROVEMENT PROGRESS – THIRD YEAR ................................................................. 73 TABLE 26 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS ................................................................ 83 TABLE 27 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS ................................................................ 88 TABLE 28 – DATA QUALITY SURVEY RESULTS ..................................................................................................... 92 TABLE 29 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 93 TABLE 30 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 94 TABLE 31 - CO-CREATION USER WORKSHOP QUESTIONS ................................................................................... 105 TABLE 32 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS .............................................................. 106 TABLE 33 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – THIRD YEAR ...................................... 108 TABLE 34 – CO-CREATION USER WORKSHOP AGENDA ............................................................................................ 109 TABLE 35 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ........................................................................... 113 TABLE 36 – DATA QUALITY IMPROVEMENT PROGRESS – THIRD YEAR ............................................................... 114 TABLE 37 – DATA QUALITY IMPROVEMENT PROGRESS – THIRD YEAR ............................................................... 115 TABLE 38 - QUESTIONS MADE TO THE AUDIENCE ................................................................................................ 123 TABLE 39 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS .............................................................. 124 TABLE 40 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – THIRD YEAR ...................................... 126 TABLE 41 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ........................................................................... 128 TABLE 42 – DATA QUALITY IMPROVEMENT PROGRESS – THIRD YEAR ............................................................... 129


of 215

TABLE 43 - SAMPLE DATA INFRASTRUCTURE BARRIERS .................................................................................... 139 TABLE 44 - SAMPLE USER CHALLENGES BARRIERS ............................................................................................ 140 TABLE 45 - SERVICE CREATION - SAMPLE OPTIONS FOR OVERCOMING BARRIERS ............................................... 141 TABLE 46 - SERVICE ENGINEERING - SAMPLE OPTIONS FOR OVERCOMING BARRIERS ......................................... 142 TABLE 47 - SERVICE MANAGEMENT - OPTIONS FOR OVERCOMING BARRIERS ..................................................... 142 TABLE 48 - SERVICE CREATION - OPTIONS FOR OVERCOMING BARRIERS ............................................................ 143 TABLE 49 - SERVICE ENGINEERING - OPTIONS FOR OVERCOMING BARRIERS ....................................................... 143 TABLE 50 - SERVICE MANAGEMENT - OPTIONS FOR OVERCOMING BARRIERS ..................................................... 144 TABLE 51 - SAMPLE CO-CREATION CONTRIBUTIONS ............................................................................................ 146 TABLE 52 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – THIRD YEAR ...................................... 148 TABLE 53 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ........................................................................... 150 TABLE 54 – DATA QUALITY IMPROVEMENT PROGRESS ...................................................................................... 151 TABLE 55 – DATA QUALITY IMPROVEMENT PROGRESS ...................................................................................... 152 TABLE 58 – PILOTS’ DATA QUALITY QUESTIONNAIRE ........................................................................................ 209


of 215

Executive Summary

OpenGovIntelligence (OGI) project aims to go beyond traditional top-down approaches and

proposes public participation in terms of the co-initiation, co-design, co-implementation, and co-

evaluation for providing innovative and data-driven public services.

During the project, building blocks for the use of Linked Open Statistical Data (LOSD) are developed.

Then, the participation processes and building blocks were implemented in six pilot projects which

have different objectives, take different approaches and look at various aspects of Linked Open

Statistical Data (LOSD). Each pilot exclusively addresses specific society’s needs by exploitation of

public sector statistical data.

The evaluation is focussed on the four main areas of this project; 1) Co-Creation Framework, 2) OGI

ICT Toolkit, 3) Acceptance of the pilots and, 4) the outcomes of the pilots (public-value creation). The

Co-creation process was evaluated in the first and second-year reports (provided in detail in D4.2

and D4.4) and in this report, we included the summary of the co-creation development.

The main focus of this 3rd-year report is on the evaluation of pilots acceptance and outcomes, which

could only be properly evaluated after the pilots have been used in practice by sufficient users. In

the final year of the project,, the final version of the pilots were delivered, and the number of the

apps users are sufficient to be evaluated. As presented in Section 3.2, the user acceptance and

perceived outcomes evaluation are divided into 4 parts: first, LOSD familiarity, user acceptance,

datasets quality and perceived outcomes.

In total, there are 219 respondents of user-questionnaire from six pilots. Most of the respondents

found the pilots’ applications are useful for them, including the creation of transparency, reduction

the administrative burden by providing a more efficient search of information and better visualizing

the results at a glance. We considered strongly agree, agree and neutral answers as positive

reactions, while negative reactions are disagree and strongly disagree. Below, we provided the key

findings of descriptive statistics of user acceptance and outcomes expectation evaluation:

The overall acceptance of the pilots’ apps was 90%;

92% pointed out an increase of Transparency after accessing the Pilots’ Apps;

The Pilots’ apps helped to 94% of surveyed people have better decision-making;

93% of people answered there is a better interpretation of Data;

For 92% of respondents, Pilots’ app reduced the time spent searching for information;

94% of end-users identified a cost reduction when using OGI pilots’ apps; and,

There is an increase in efficiency to 91% of OGI Pilots’ users.


of 215

In addition, the OGI ICT Toolkit was evaluated using a mixed qualitative approach (described in

section 3.3) by the developers. The OGI ICT toolkit consists of building blocks. All pilots used the

toolkit as their reference in designing and building their apps architecture. However, due to the

different objectives and approaches, some pilots did not need to use all OGI building blocks.

Technical partners and pilots’ leaders were interviewed and surveyed to collect information about

fundamental requirements and parameters for an ICT Toolkit development. The result shows the

potential and limits of the LOSD OGI Toolkit.

From the project we can learned that developing the ICT-toolkits took longer than expected, as this

is a new field in which there are hardly any examples. The development of the building block

encountered main challenge in deciding which level of granularity is acceptable for the both

developers and users as this requires a trade-off between the flexibility given to the developer and

ease-of-use of manipulating data. User-friendliness and having right granularity is a trade-off and

developers have different requirements in this regard.

Co-creation remains challenging in government. Co-creation is bottom-up, whereas government

traditionally takes a more top-down approach by developing pilots and providing the services.

Realizing co-creation is not only a technological problem but rather a cultural problem.

The end-users regard the pilots as successful as shown in the figure below. The pilots show that data

quality is a key aspect as garbage in is garbage out. Realizing high-data quality often took more time

than expected, whereas the OGI ICT-toolkit enables to develop the pilots within a short time frame.

They pilots help end-users to find information faster and in a more efficient way resulting in

transparency and better decision-making quality.


of 215

1 Introduction

This document reports the evaluation results of OpenGovIntelligence (OGI) project. The OGI

environment provides an ICT toolkit comprising easy-to-use and user-centric tools to facilitate

realizing a Linked Open Statistical Data (LOSD) innovation ecosystem. Pilots were deployed to

validate and prove the usability and effectiveness of OGI ICT toolkit to co-create and innovate

ecosystems.

The first part of the evaluation is related to the co-creation development which was already covered

in previous reports and will only be summarized here. The second part concerns the evaluation of

the OGI Toolkit by developers. The third part is the acceptance of the OGI ICT Toolkit by end-users.

The fourth part is the evaluation of the outcomes of pilots’, which includes Transparency,

Administrative Burden Reduction, and Costs Reduction.

1.1 Scope

The present document is the Deliverable 4.6 “D4.6 – Pilots Evaluation Results - Third Round”

(henceforth referred to as D4.6) of the OGI project. The main objective of D4.6 is to describe the

pilot’s results in this third year of OGI project and elicit the lessons learned and conclusions from the

evaluation results.

1.2 Audience

The audience for this deliverable is as follows:

European Commission (EC);

The audience interested on the exploitation of Linked Open Statistical Data (LOSD) and Data

Cubes for public service delivery

The audience interested in public service delivery based on co-creation from external and

internal stakeholders; and,

OGI Project partners.

1.3 Structure

In the next chapter, we start by presenting the pilots’ Evaluation Overview in accordance with Pilots’

Implementation Plan in the report D4.5. The structure of this document is as follows:

Section 2 provides the Evaluation Overview;

Section 3 describes the Pilots Evaluation Methodology;

Section 4 presents the OGI Evaluation Results;

Section 5 presents Conclusions;

Section 6 provides the References; and,

Section 7 shows the annexes.


of 215

2 Evaluation Overview

The OGI toolkit is implemented in 6 pilot projects. The pilots are deploying the OGI ICT Toolkit to

implement an application that can be used by others. To develop the pilots, the OGI ICT Toolkit was

used and tested and the co-creation framework is guiding the relationship between service

providers and users. Each pilot made only use of the relevant OGI building blocks. The results of the

evaluation provides insights for further improvement of the OGI toolkit and for improving the

evaluation instruments.

OGI to was evaluated by conducting six pilot projects, described in the report D1.1- Challenges and

Needs (https://www.slideshare.net/OpenGovIntelligence/deliverable-11-ogi-challenges-and-needs).

The projects use the OGI toolkit for co-creation, used LOSD to provide services and ultimately

contribute to social impact.

1. The Greek Ministry of Administrative Reconstruction (Greece)

a. http://wapps.islab.uom.gr/CubeVisualizer/vehicles/;

2. The Enterprise Lithuania (Lithuania)

a. http://vmogi03.deri.ie:8080/superset/dashboard/5/

b. http://vmogi03.deri.ie:8080/superset/dashboard/7/

3. The Tallinn Real Estate (Estonia)

a. https://rnd-tut.shinyapps.io/Estonian_Pilot/

4. Trafford Council Worklessness (England)

a. http://www.trafforddatalab.io/opengovintelligence/

5. The Flemish Environment Agency (Belgium)

a. https://www.milieuinfo.be/emissiepunten/

6. The Marine Institute (Ireland)

a. http://vis.marine.ie/DashboardsTest/#/wave_spectral

b. http://vis.marine.ie/DashboardsTest/#/wave_zero

c. http://vis.marine.ie/DashboardsTest/#/weather

The OGI environment provides an ICT toolkit comprising easy-to-use and user-centric tools to

facilitate the processing and use of Linked Open Statistical Data (LOSD). The source code of OGI

toolkit can be found at Github (https://github.com/OpenGovIntelligence)

The pilots were deployed to validate and prove the usability and effectiveness of OGI ICT toolkit to

co-create and innovate ecosystems. The resulting apps should be used by a variety of users which in

turn should result in long term effects (outcomes).

The WPs and evaluation are closely related to each other. Figure 1 summarizes the

interdependencies and interconnections between the working packages (WP) of the OGI project.

https://www.slideshare.net/OpenGovIntelligence/deliverable-11-ogi-challenges-and-needs

http://wapps.islab.uom.gr/CubeVisualizer/vehicles/


http://vmogi03.deri.ie:8080/superset/dashboard/5/


https://rnd-tut.shinyapps.io/Estonian_Pilot/

http://www.trafforddatalab.io/opengovintelligence/

https://www.milieuinfo.be/emissiepunten/

http://vis.marine.ie/DashboardsTest/#/wave_spectral

http://vis.marine.ie/DashboardsTest/#/wave_zero

http://vis.marine.ie/DashboardsTest/#/weather

https://github.com/OpenGovIntelligence


of 215

Figure 1- Interconnections and Interdependencies between OGI Working Packages and Deliverables

In an ecosystem, there are different stakeholders who view the pilots from their own perspectives.

Developers might want to evaluate the pilots based on their ability to meet the end-user

requirements using the OGI toolkit and building blocks. End-users want easy-to-use interfaces and

pilots satisfying their needs. Decision-makers might look at how participation by users and other

forms of co-creation result in an impact of the applications on the number of users and return on

investment. Policy-makers would be interested in societal impact generated by reducing

administrative burden, the creation of transparency and its contribution to solving societal

problems. Our evaluation will take into account the multiple stakeholders’ perspectives.


of 215

3 Evaluation Methodology

Based on the four stakeholder perspectives, four areas to be evaluated were derived In :

1. The first area is the co-creation framework (co-initiation, co-design, co-implement, and co-

evaluation). The evaluation is relevant for those decision-makers who want to innovate and

ensure that there is a high level of user participation.

2. The second area is the OGI Toolkit (ICT building blocks and cubes design). The evaluation is

primarily relevant for developers who build applications.

3. The third area is the acceptance of OGI Pilot Applications. End-users should accept and use

the OGI Toolkit deployed in the pilots.

4. The fourth area is the outcomes (estimative money savings, time reduction, efficiency

perception of service delivery and transparency). The outcomes are of relevance for the

policy-makers, but also the society as a whole. In the end, the purpose is to create societal

value.

These four areas are visualized in Figure 2 together with co-creation. The second area about OGI

solution platforms (in blue) contains the development of the building blocks contributing to the

platforms and the use of the platforms, and its building block for developing the apps in the pilots.

Both are closely connected to each other and therefore are integrated together.

Figure 2 - Evaluation dimensions


of 215

The co-creation framework is described in detail in Deliverable D2.1, in the WP2 Framework

Creation. The OGI toolkit is also described in detail in Deliverable D3.5, in the WP3 ICT Tools

Development.

Beyond that, this report is linked to D1.1 OGI Challenges and Needs, as part of the WP1 Challenges

and needs identification the D4.4 Pilots Evaluation results - Second round in the WP4- Pilots Planning

and Evaluation. Those interconnections are visualized in Figure 1.

Pilots are organized in three main iterations. After each iteration, the OGI toolkit will be more

advanced and enable further development in the pilot. This agile way of working enables relatively

short cycle-times and quick improvements. Furthermore, this enabled to evaluate the OGI Toolkit

and building blocks and improve them each time.

1. The first (initial) iteration resulted in an early version of the evaluation of OGI services and

tools. This feedback was used to further improve the OGI ICT Toolkit

2. The second iteration was used to develop a more advanced version. Again, this feedback

was used to further improve the OGI toolkit; and,

3. The final iteration of pilots benefitted from the lessons learned in the first and second years

of pilot iterations and resulted in a mature OGI toolkit.

Figure 3 illustrates the tasks involved in planning for and conducting a pilot and shows the high-level

process. Although the activities in this figure are presented in a waterfall manner, this is only for

communication and planning purposes, and the actual way of working was agile embracing co-

creation and focussed on short iteration resulting in working software and meeting end-user

requirements.


of 215

Figure 3 - High-Level Processes of Pilot Plan

The pilot design team was responsible for creating the pilot implementation and evaluation plans

(Deliverables 4.1, 4.3 and 4.5) as well as pilot evaluation reports (Deliverables 4.2, 4.4 and 4.6). This

team consists of the Research and Development (R&D) Partners in the OGI consortium.

The pilot implementation team was formed by the technical partners and, the people in charge of

each of the six pilots. The Pilot implementation team is responsible for executing the pilot projects

based on the plan created by the pilot design team described on this report D4.5 and the previous

version, D4.3 and D4.1.

The pilot implementation was divided into three main actions:

1) Preparation: the part that deals with collecting needed information from the pilots to fill the

implementation template;

2) Implementation: the part that executes the implementation of the OGI toolkit and co-

creation framework on the pilots by technical partners; and,

3) Evaluation: the part that measures the success of outputs and outcomes after

implementation of OGI toolkit and co-creation framework.

The findings of the evaluation of the second step were analyzed by the OGI Consortium. The result

of this analysis was used to create the pilot plan for the next iterative cycle. These steps aimed to


of 215

identify challenges and needs to improve the implementation and evaluation of OGI ICT Toolkit, and

the OGI Co-Creation Framework in the OGI pilots.

The pilots’ reports provided the processes of each pilot and evaluation on four evaluation

dimensions for each pilot (described at Figure 4), and was the source for the pilot plan of next

iteration, for example D4.4 (Evaluation results in 2nd round) was the source for D4.5 (Pilot and

Evaluation Plan 3rd release), influencing D4.6 (Evaluation results 3rd round – this report). Below,

Figure 4 describes the expected reports aforementioned.

Figure 4 - Pilots' Timeline


of 215

3.1 Pilots’ Co-Creation Evaluation Methods

Taking into consideration the types of co-creation and participant contribution an evaluation

method was developed to collect data and to analyze the data and feedback of participants. This is

explained at D2.1 OGI Framework from WP2 Framework Creation.

The objective of this section is to explain the methods to collect data and ICT tools used to evaluate

the feedback from the identified types of co-creation and participant contributions during the three

years of the OGI project. Table 2 summarises the methods to collect data and ICT tools identified as

useful to collect and analyze feedback from a participant on the four different co-creation stages.

Table 1 - Co-Creation Framework Stages, Methods for Data Collection and Tools for Evaluation

Co-Creation Step

Participant role Source to collect data Methods and Tools for evaluation

Co-Initiation

Problem & needs identification

- Social Media - R statistical analytics

- TwitteR

- Weka

- Other social media analytics

Idea generation for ways to solve problems (informed by data)

- User workshops

- Public meetings

- Social Media

Co-Design The input to service design

- User workshop

- Continued participation

- Focus groups

- User Experience and User Interface testing

- Survey (Questionnaire and interviews)

Co-Implementation

Uploading user data - Web and Phone Statistics (Number of access, download, etc.)

- Web Analytics

- Survey (Questionnaire and Interviews)

- R statistical analysis

- TwitteR

- Weka

- Other social media analytics

Suggesting changes to data sets

- Portal’s Feedback channels

Data creation for a service

- Web and Phone Statistics (Number of access, download, etc.)

Co-Evaluation

Providing feedback to service quality, usefulness, etc.

- Social Media - Portal’s Feedback channels

- Web Analytics

- Survey (Questionnaire and Interviews)

- R statistical analysis

- TwitteR

- Weka - Other social media analytics

Reporting data on service operation

- Web and Phone Statistics (Number of access, download, etc.)

The selected sources and methods are described below in the next section 3.1.1.


of 215

3.1.1 Selected Co-Creation Tools and Methods Overview

3.1.1.1 User workshop for co-creation

A workshop is an activity that aims to introduce something (idea, skill, product, etc.) to a potential

interested people. Workshops range from short workshops (45 minutes or less) to one or more days.

A critical aspect of user workshop feedback process is the inclusion of end-users in the creation of

the new data-driven public services.

The overall structure user workshop planned to be conducted on the pilots is described in Table 3:

Table 2 - User Workshop Protocol for Co-Creation

£ Stage Number of questions

1 Introduction This stage has the aim to describe the background to participants and clarify

questions. A general objective is given to participants

2 Silent

Ideation

In this stage, participants brainstorm to produce ideas. They can take notes

and be prepared to share the ideas with other people on the workshop.

3 Group

Discussion

In this stage, there is a group discussion of all the participants, presenting

the ideas that they had during the silent ideation. It is allowed to

participants to give commentaries or insert inputs from other participants

ideas presentations (discussion).

The three stages can be repeated as many times as needed. This way is possible for all individuals to

provide valuable inputs on the design and structure of the new public service. The user workshop

can be used on all the four stages of Co-Creation framework.

The user workshops can produce outcomes like :

• List of issues (problems) with the new service;

• List of potential solutions and alternatives for improvement;

• Basic thoughts on the usability and functionality of the service;

• User stories;

• List of user personas of individuals who could use the service, and

• Any other information which may come out of the workshop organically.

After participating in the user workshop, a survey was conducted to identify the participants'

feedback.


of 215

3.1.1.2 Survey Research

Whereas the workshops were focussed on deep-understanding and gaining feedback, surveys’ were

conducted to collect qualitative (interviews) and quantitative (questionnaires) data about the pilot's

performance.

Figure 5 – Exploratory and Explanatory approaches at Co-Creation Evaluation Surveys

A survey is a systematic poll of questions made to some group, or individually, to collect answers

about some problem, observation, etc. Glasow (2005) considers two types of data collection

methods:

1. Written (questionnaire); and,

2. Verbal (interviews).

Both types will be conducted with different objectives and different periods of co-creation

evaluation.

Based on the co-creation evaluation, survey research was used in all of the co-creation types.

Interviews were used to examine the co-initiation and co-evaluation, while questionnaire will be

used on the co-design and co-implementation.

The interviews used open-ended questions to seek understanding and interpretation in a different

situation. In the co-initiation stage, interviews were aimed at identifying problems and generating

ideas for problem-solving. In the co-evaluation stage, interviews were aimed at understanding the

questionnaire results as well as to seek in-depth information on the several issues found in the

questionnaire. Normally, only a specific group of stakeholders was involved, taking into

consideration there is a low degree of statistical validity due to low number and homogenous profile

of participants. In addition, there is a higher chance of bias on the answers in comparison of

questionnaires. Confidential information can be collected, the need of resource, time, etc. are spent,

and fewer errors happen on this type of survey.

Questionnaires are using closed-ended questions to gather highly standardized data. It was designed

in such so the target respondents can give more generic inputs of design and implementation on the

co-creation of public policies processes. Normally questionnaires are given to several people to


of 215

reach substantial statistical validity of a hypothesis, for example already observed on the qualitative

approach of the survey (interview).

Before conducting the survey (questionnaire or interview), Glasow (2005) suggests creating a model

that identifies the expected relationships among the variables (independent and dependent).

Variables are used to define the scope of the study, however, cannot be explicitly controlled by the

researcher. Then is possible to test the model against observations of the phenomena analyzing the

data collected on interview or questionnaire.

The survey design

In designing the survey research, Levy and Ellis (2006) suggest two steps 1) sampling strategy and

the procedure to obtain the representativeness of the population, including ensuring reliability and

2) validity. The nature of this evaluation process is in between exploratory and explanatory.

Explorative to find issues and explanatory to explain the effects of the pilots on the user-satisfaction

and outcomes. For this purpose, a the mixed method, including qualitative and quantitative method

was used. The sampling strategy should follow these methods. Population for this survey research

will be all stakeholders in each pilot, or in general, units of observation will be the Public

Administration’s employees, citizens and companies’ employees who use the OGI innovation

ecosystem. Participants of this co-creation survey should represent these units of observation.

The sample techniques were different for each co-creation type. The co-initiation and co-design

used the non-probability sampling, and the co-implementation and co-evaluation used the

probability sampling. The non-probability sampling is used because, in the two first types of co-

creation, the respondents will be selected by the ones that use the OGI tools and framework to

identify the problems and propose the improvement of the public services based on the LOSD from

PAs, citizens, and companies.

For co-implementation and co-evaluation, survey participants were selected randomly, to reach

stronger analysis to justify the use of OGI innovation ecosystem. The challenges of this technique

were to minimize sampling bias and achieve good representativeness. To deal with these issues,

each pilot partner needs to carefully acknowledge the stakeholders of the system, for example,

number of employee, the demography of users (citizens and businesses), structure of the

companies, etc. The list of questions helped partners addressing this issue.


of 215

3.1.1.3 Overview of Pilots’ Co-Creation Steps, Tools, Methods, and Results Description

Table 3 summarizes the selected co-creation tools and methods chosen by each of pilots to

implement and evaluate the co-creation steps. Further, a brief description of results from each of

tools and method.

Table 3 – Overview of Pilots’ Selected Co-Creation Tools and Methods

Pilot Co-Creation Steps

in Year 2

Co-Creation Methods and Tools

for Evaluation Brief Description of Results

Pilot 1

The Greek

Ministry of

Administrative

Reconstruction

(Greece)

Co-

Implementation

Continued participation from User

workshop

Update in application design and

data quality improvement.

Co-Evaluation End-User acceptance and

outcomes survey

Statistical analysis of Pilot

feedback from participants

Pilot 2

Enterprise

Lithuania

(Lithuania)

Co-

Implementation

Online continued participation

from User workshop

Update in application design, new

data sets included and data quality

improvement.


outcomes survey



Pilot 3

Tallinn Real

Estate

(Estonia)

Co-

Implementation

Hackathon event to exploit the

code and data sets

List of ideas from Hackathon

participants

Online discussion via Github

about code and data sets by

Hackathon participants

Github statistical analysis (views,

forks, changes) and list of

suggestions in code and data sets


outcomes survey



Pilot 4

Trafford Council

Worklessness

(England)

Co-

Implementation User workshop




outcomes survey



Pilot 5

The Flemish

Environment

Agency

(Belgium)

Co-




Co-Evaluation Didn’t evaluated because the

application is under review.

Didn’t evaluated because the


Pilot 6

Marine Institute

(Ireland)

Co-




Co-Evaluation Didn’t evaluated because the


Didn’t evaluated because the



of 215

3.2 Pilots’ Acceptance and Outcomes Evaluation Methodology

3.2.1 Pilots’ Acceptance Selected Methodology

A combination of theories was employed to create to evaluate the user perspective. The focus was on an evaluation of user acceptance and intention of use for OGI. The list of main theories is described in the report D4.3 and is listed below:

1. TAM 1 by Davis (1989);

2. TAM 2 by Venkatesh and Davis (2000);

3. TAM 3 by Venkatesh and Bala (2008);

4. UTAUT Framework by Venkatesh, Morris et al. (2003); and,

5. IS Success Framework by Delone and McLean (2003).

The presents variables and (general) measured items for the pilot's evaluation. However, as each pilot has its characteristics in term of objectives, target audiences, type of data, type of services, OGI building blocks uses, and visualization, The questionnaire created for the evaluation was slightly different to each pilot. We used the same foundation. However, we added specific questions for the pilot. The questionnaires of each pilot are described in Section 7.1.

Table 4 - User Acceptance Evaluation for OGI

Variable Measured Items Source

Job Relevance (JR)

OGI toolkit makes mine and my colleagues job tasks easier to be accomplished.

In my job, usage of the system is important.

In my job, usage of the system is relevant.

Venkatesh and Davis (2000)

Output Quality (OQ)

The quality of the output I get from the OGI toolkit is higher compared to the previous system.

I have no problem with the quality of the OGI toolkit's output.


Result Demonstrability (RD)

I have no difficulty telling others about the results of using the OGI toolkit.

I believe I could communicate to others the consequences of using the OGI toolkit.

The results of using the OGI toolkit are apparent to me.

I would have difficulty explaining why using the OGI toolkit may or may not be beneficial.


Perceived Ease of Use (PEU)

My interaction with OGI toolkit is clear and understandable.

OGI toolkit usage does not require a lot of skills.

I find it easy to get the OGI toolkit to do what I want it to do.

I find the OGI toolkit to be easy to use.

(Davis 1989)

Perceived Usefulness (PU)

Using OGI toolkit improves my performance in my job tasks.

Using OGI toolkit enhances my effectiveness in my job tasks.

Using OGI toolkit in my job increases my productivity.

I find the OGI toolkit to be useful in my job.

(Davis 1989)

Intention to Use (IU)

If I have access, I will use OGI toolkit. (Davis 1989)


of 215

3.2.1.1 Pilots’ Outcomes Selected Methodology

The outcomes evaluation is divided into a contribution to transparency, Reduction of Administrative Burden and Costs. These will be discussed next.

3.2.1.2 Transparency

Transparency has been used as a magic concept by the government and public managers (Ward

2014). The usage of transparency is sometimes used as a synonym for accountability (Bovens 2007),

openness (Coglianese 2009), and even open government data (Frank and Oztoprak 2015) as an

example. There are no clear definitions that are generally accepted by scholars in this field. This

project takes into consideration transparency as a concept of a unilateral process of disclosure of

data, information, and actions that an organization has been conducting to the public (Peixoto

2013).

Transparency is a phenomenon that can lead to accountability but does not guarantee any concrete

result of justice or mobilization, just public exposure to scrutiny (Fox 2007). In addition to the

discussion of what is and what is not transparency, only a few models are trying to explain and

evaluate transparency. Matheus and Janssen (2013) proposed an evaluation model for transparency

initiatives considering transparency as a multidimensional object based on many factors influencing

two main dimensions: interpretation and accessibility. Facilitating conditions were added

considering the background and profile of users influencing transparency.

The OGI project will test if factors and facilitating conditions influence transparency dimensions

positively or negatively. These factors designed as propositions with suggesting the direction the

factors contribute to, however, there is no proof for this in the literature. Our evaluation of the

pilots contributes to understanding these propositions. The transparency model used in OGI

evaluation is described in detail in the section 3.6.1 of D4.3. The table below summarizes the

attributes evaluated and described each proposition in detail. Figure 6 then shows the graphical

view of Transparency Model of Matheus and Janssen (2013).


of 215

Table 5 - Adapted Transparency Model for OGI Evaluation

Dimension Factor P. # Proposition

Interpretation

Ibid P1 easier interpretation of data results in higher transparency

Examples of usage P2 presence of examples of the website product, the higher has a positive influence on interpretation.

Simple Language used P3 simple language has significant positive influence on transparency

Data Quality P4 higher information quality has a significant influence on the interpretation

Updatedness of Information

P4a higher updated information has a significant influence on data quality.

Completeness P4b higher data completeness has a significant influence on data quality

Accuracy P4c higher data accuracy has a significant influence on data quality Accessibility ibid P5 higher accessibility has a significant positive influence on

transparency Simple Language P5a simple language has significant positive influence on

transparency Data Overload P5b data overload has a significant negative influence on accessibility Adhesion to Standards P5c adhesion to standards has a significant positive influence on

accessibility Unified Technology P5d unified use of technology has a positive influence on accessibility

Facilitating Conditions

Experience 6a the influence of interpretation on transparency will be moderated by experience

6b the influence of accessibility on transparency will be moderated by experience

Age 7a the influence of interpretation on transparency will be moderated by age

Level of Education 8a the influence of interpretation on transparency will be moderated by the level of education

8b the influence of accessibility on transparency will be moderated by the level of education


of 215

Figure 6 - Adapted Transparency Model for OGI Evaluation

Matheus and Janssen (2013)

3.2.1.3 Administrative Burden

Governments are facing issues to deliver more and better public services with less financial

resources, time and people. Due to these reasons, OGI evaluates how the OGI ICT toolkit can help to

reduce the administrative burden. The evaluation methods are based on the Cost-Benefit

Assessment (CBA) approach, following the Study on eGovernment and the Reduction of

Administrative Burden - SMART 2012/0061 (Gallo, Giove, et al. 2014). The taxonomy of costs and

benefits of Gallo et al. (2014) has been adapted.

The dimension considered for both Public Sector and Users Benefits was divided into two categories,

e.g. direct and indirect benefits. The reason to focus only on Benefits is that the OGI consortium is

interested oi the benefits that OGI innovation ecosystem can provide. The figure below summarises

the OGI focal point which is aimed at creating both advantages for the public sector and end-users.


of 215

Figure 7- OGI Focal Point

Table 6 summarises the adapted taxonomy for Costs and Benefits of OGI innovation ecosystem for

stakeholders.

Table 6 - Taxonomy for Benefits for OGI innovation ecosystem stakeholders

Dimension Categories Sub-Categories Description

Benefits for Public Sector

Direct Benefits

Time Savings

It includes all monetizable benefits arising from improvements on public service delivery, including time savings before/after OGI innovative ecosystem implementation.

Indirect Benefits

Efficiency perception It encompass non-magnetisable benefits related to

better service delivery and the enhancement of the decision-making process.

Quality of service

Benefits to Users (citizens, businesses)

Direct Benefits

Time-Saving It includes all monetizable benefits arising from improvements on public service delivery, including time savings before/after OGI innovative ecosystem implementation.

Indirect Benefits

Estimative Money Saving

Quality of service It encompass non-magnetisable benefits related to better service delivery and the enhancement of the decision-making process.

There were two ways to understand "benefits for Public Sector." First as an organization, and second

as people involved in the processes. Taking into consideration the OGI innovation ecosystem, we

decided to take approaches depending on pilots. Some pilots have a clear boundary between Public

Sector (organization and people – civil servants, policy makers, etc.) and users (citizens, businesses,


of 215

etc.). However, other pilots didn't have a clear boundary between Public Sector and users, because

the users are on the Public Sector (civil servants, policy makers, etc.).

The summary of data collection methods is presented in Table 6, and the planned questions of

questionnaires and interviews are at section 7.1.

1. Time savings

Time savings can be determined in two types of scenarios. If the public service already existed, it is

possible to measure how much time in days or hours a civil servant or citizen/business person

performed the task or receive the public service delivery. The measure can be collected using the log

observations of the service delivery, researcher observation of process (in person) or via a survey

(questionnaire and interview).

If the public service did not exist yet and a new service was created, measures comparing similar

processes or collect data from users (civil servants, citizens, business person, etc.) from a perception

or expectation perspective will be used. Next, a Likert scale based on the level of satisfaction with

time spent on public service (1- very dissatisfied to 5-very satisfied) will be used to measure the

perceived time savings.

2. Efficiency Perception

Efficiency perception considers the measuring of the perception of people who use public sector

data in the OGI pilots. People might have the experience to use open data and can compare this with

the use of the data in the OGI pilot. This parameter is a perception of the users, and it does not

mean quantifying the time savings like in the previous measure. The objective of this measure is to

identify if public sector users (civil servants, public managers, etc.) perceive if there is an

improvement in efficiency. To measure efficiency perception, a questionnaire will be conducted

among pilot users. The questionnaire uses a Likert scale for measuring the level of agreement (1

strongly disagree to 5- strongly agree) with an increase in efficiency.

3. Estimative Money Savings

Money savings look like a hard measurement, but is often hard to quantify and has to rely on

estimates. Taking this into consideration, the approach used a perception or expectation perspective

for expenses saving b comparing the situations before and after implementation of OGI pilots. A

questionnaire was used to collect data based on a Likert scale of agreement (1 strongly disagree to

5- strongly agree), and an open question was used to ask uses how much expenses were avoided.

4. Quality of Service

Quality of service considers two potential scenarios. If the public service already existed, it is

possible to measure the level of quality using a Likert scale (1-poor to 5- excellent) of the current

service and compare the outcomes with the ranking on the past service (1- much worse to 5- much

better). If any issue was identified on this questionnaire, further interviews were conducted to

further analyze the issue.

If the public service doesn't do exist and a new one was created, perception data about the quality if

the new service was collected from the users (civil servants, citizens, business person, etc.) using a


of 215

survey. A Likert scale was used measuring the level of satisfaction with the quality of public service

(1- very dissatisfied to 5-very satisfied).


of 215

3.2.1.4 Pilots’ Acceptance and Outcomes Evaluation Questionnaire

In this section, we surveyed Pilots’ end-users about their perceptions of using the OGI pilots

Applications. The categories collected and evaluated are described in Table 7:

Table 7 – Pilots’ Outcomes Evaluation Questionnaire

Outcomes

Categories Description

Number of

questions Source

User

Information

This section aims to introduce and explain

the pilot application, describe the survey

and identify general background

information. For example, key questions if

end-user is aware of Linked Open Data, the

OGI Project and the specific pilot end-user is

using.

Two

questions Created by authors

User

Acceptance

This section aims to identify if End-Users

accepts the functionalities that pilot

applications offer. For example, if the

functionalities are by end-users needs, or if

the app does not crash.

Thirteen

questions and

an open

question in

the final for

general

comments.

Adapted from Davis (1989);

Venkatesh and Davis (2000);

Venkatesh and Bala (2008);

Venkatesh, Morris et al.

(2003); and,

Delone and McLean (2003)

Data Sets


accepts the available data sets in the pilot

application has the desired or needed data

quality standard.

Five questions

and an open

question in

the final for

general

comments.

Adapted from Dodds (2016)

and Matheus and Janssen

(2013)

Results


accepts that pilot application can deliver

expected results and outcomes. For

example, if the pilot application can

increase Transparency or can reduce the

Administrative Burden and Costs.

Ten questions

and an open

question in

the final for

general

comments.

Adapted from Gallo, Giove,

et al. (2014)


of 215

For each pilot, an end-user questionnaire was developed and distributed among users.

1. Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece)

a. End-User Questionnaire (Section 7.1.1).

2. Pilot 2 – Enterprise Lithuania (Lithuania)

a. End-User Questionnaire (Section 7.1.2)

3. Pilot 3 – Tallinn Real Estate (Estonia)


4. Pilot 4 – Trafford Council Worklessness (England)


5. Pilot 5 – The Flemish Environment Agency (Belgium)


6. Pilot 6 – Marine Institute (Ireland)



of 215

3.3 OGI ICT Toolkit Evaluation Methodology

To evaluate the implementation of OGI ICT Toolkit in the third Year, we created the ICT Toolkit

Questionnaire to measure the OGI Data Quality, OGI Architecture, System Quality of Application,

and to externally evaluate the OGI ICT Toolkit.

The OGI ICT toolkit consists of building blocks that were developed for performing a certain

functionality. Table 8 shows the evolution of the tools (building blocks) names and the merges of

tools over the three years of projects and the accompanying deliverables (D3.1, D3.2, D3.3, D3.4,

D3.5, and D3.6). Figure 8 summarizes the OGI data cycle and the tools for each step.

Table 8 - OGI Toolkit Releases and Tools per year

1st Release

(Based on D3.1 and D3.2)

2nd Release


Full version

(Expected D3.5 and D3.6)

A- Previous

Project Results

1. Grafter 1. Grafter 1. Grafter

- 2. Data Cube Builder 2. Data Cube Builder

B- OGI Toolkit

2. Data Cube OLAP

Browser 3. Data Cube OLAP Browser

3. Cube Explorer 3. Data Cube Explorer 4. Data Cube Explorer

4. Data Cube Visualizer 5. Data Cube Visualizer

1. JSON API for Data Cube 1. JSON API for Data Cube 1. CubiQL API

2. Table2QB 2. Table2QB 2. Table2QB

3. Data Cube Aggregator 3. Data Cube Aggregator 3. Cube Aggregator

- 4. LOSD Machine Learning

Component

4. LOSD Machine Learning

Component

- 5. SPARQL connector for

Exploratory

5. SPARQL connector for

Exploratory

- - 6. DataScience SPARQL plugin

- - 7. Data Cube RDF Validator

C- Pilot

Specification

Tools

- 1. Assisted Cube Schema

Creator

1. Assisted Cube Schema

Creator

- 2. QB Multi-Dimensional

Charting

2. QB Multi-Dimensional

Charting

- 3. RDF Data Cube Geo-Data

Supported for Dashboard

3. RDF Data Cube Geo-Data


- - 4. Data Cleansing Tools

- - 5. Data Collection Tools

- - 6. Custom Mappings and Scripts


of 215

- - 7. ShinyR

- - 8. Superset + Druid

Table 8 - OGI Toolkit Releases and Tools per year

1st Release


2nd Release


Full version

(Expected D3.5 and D3.6)

A- Previous

Project Results

1. Grafter 1. Grafter 1. Grafter

- 2. Data Cube Builder 2. Data Cube Builder

B- OGI Toolkit

2. Data Cube OLAP

Browser 3. Data Cube OLAP Browser

3. Cube Explorer 3. Data Cube Explorer 4. Data Cube Explorer

4. Data Cube Visualizer 5. Data Cube Visualizer

1. JSON API for Data Cube 1. JSON API for Data Cube 1. CubiQL API

2. Table2QB 2. Table2QB 2. Table2QB

3. Data Cube Aggregator 3. Data Cube Aggregator 3. Cube Aggregator

- 4. LOSD Machine Learning

Component

4. LOSD Machine Learning

Component

- 5. SPARQL connector for

Exploratory

5. SPARQL connector for

Exploratory

- - 6. DataScience SPARQL plugin

- - 7. Data Cube RDF Validator

C- Pilot

Specification

Tools

- 1. Assisted Cube Schema

Creator

1. Assisted Cube Schema

Creator

- 2. QB Multi-Dimensional

Charting

2. QB Multi-Dimensional

Charting

- 3. RDF Data Cube Geo-Data


3. RDF Data Cube Geo-Data


- - 4. Data Cleansing Tools

- - 5. Data Collection Tools

- - 6. Custom Mappings and Scripts

- - 7. ShinyR

- - 8. Superset + Druid


of 215

Figure 8 shows how the various OGI Tools can be used in the LOSD workflow. The workflow starts

with the raw data that might need to be prepared and ends with the visualization.

Figure 8- OGI Tools and Working Flow


of 215

Table 9 shows the list of tools, the links to the OGI Github code source

(https://github.com/OpenGovIntelligence) and the pilots which are using these tools. Important to

highlight is that the colors used are the same as in Table 8 to show the connection between them.

https://github.com/OpenGovIntelligence


of 215

Table 9 - OGI Tools used in Pilots

Tools

Pilots

MAREG

(Greece)

Lithuania

Enterprise

Trafford

(England)

The Flemish

Government

(Belgium)

Marine

Institute

(Ireland)

Ministry of

Economics

(Estonia)

Grafter (Private Tool)

www.goo.gl/vaGzLc X X X

Tarql – Cube Builder

https://goo.gl/fQ9YQx X

X X

Cube Explorer (NEW)

www.goo.gl/WLC6zh X

X

Cube Browser

www.goo.gl/b57zpX X X

Cube Visualizer

www.goo.gl/hc9QGj X

X

CubiQL API (NEW)

www.goo.gl/tbZuPg X X x

X X

Table2QB (NEW)

www.goo.gl/JgMKvx X X X X X X

Cube Aggregator

www.goo.gl/urSj5V X X

X X

LOSD Machine Learning

Component (NEW)

https://goo.gl/c2gkV9

X

SPARQL connector for

Exploratory (NEW)

www.goo.gl/7AeYVF

X

Validator (NEW)

www.goo.gl/Kn6Bhh X X X X X

Assisted Cube Schema

Creator (NEW)

https://goo.gl/BCLW2s

X X

QB Multi-Dimensional

Charting X

RDF Data Cube Geo-Data


Data Cleansing Tools X X X X X X

Data Collection Tools X X X X X X

Custom Mappings and

Scripts

X X X X X X

ShinyR X

SuperSet + Druid X X

http://www.goo.gl/vaGzLc

https://goo.gl/fQ9YQx

http://www.goo.gl/WLC6zh

http://www.goo.gl/b57zpX

http://www.goo.gl/hc9QGj

http://www.goo.gl/tbZuPg

http://www.goo.gl/JgMKvx

http://www.goo.gl/urSj5V

https://goo.gl/c2gkV9

http://www.goo.gl/7AeYVF

http://www.goo.gl/Kn6Bhh

https://goo.gl/BCLW2s


of 215

3.3.1 OGI Pilots’ Data Quality Evaluation Methodology

To evaluate the implementation of OGI Data Quality in the third year, we created a Data Quality

Questionnaire. The Data Quality Questionnaire is inspired by the Open Data Institute blog post

“Exploring Open Data Quality (https://theodi.org/blog/exploring-open-data-quality) by Leigh Dodds

(Dodds, 2016). Table 10 describes the data quality attributes, their descriptions, the number of

questions in the questionnaire and the source from where they were collected or inspired. The

questionnaire with all the questions is presented in Table 58. The link to the online form is

https://docs.google.com/forms/d/e/1FAIpQLSeCl3flPZIGYEWhhASTEnwF6GkMERIxZ8OWjkKpJCmqRl

LwoA/viewform.

Table 10 – Pilots’ Data Quality Questionnaire

Data Quality

Section Description

Number of

questions Source

Discoverability

and Usability

This section aims to identify the application characteristics

that will influence some attributes of data qualities below.

For example, if the website or application has an option for

downloading data. This parameter influences the level of

granularity.

Seven

questions

Created by

authors.

Granularity

This section aims to identify the level of granularity. The

desired level of granularity varies according to the problem

being addressed. For example, if the data is in aggregated

level (average sex instead of individual data) or timely

option for download (per year average, per month, per

week, per day).

Three

questions

Frank and

Walker

(2016) Intelligibility

This section aims to identify immediate intelligibility. This

parameter gives quality to the data in the sense of

readability and increases immediate understand by users.

For example, if there is documentation giving context for

humans understand the data and see how it is possible to

use it in human or machine ways.

1 question

and open for

commentary

Trustworthiness

This section aims to identify data quality attributes that

increase the trust in data. For example, if the source is

trusted, if who is in charge of data is aware of how it was

collected and created.

Five

questions

Linkable to other

data

(5 Stars LOD)

This section aims to identify data quality attributes to

Linking the data to other opened datasets. For example, if

the data is in CSV or RDF format.

Four

questions

Bizer et al.

(2009)

15 Open Data

Principles

This section aims to identify data quality attributes related

to the 15 Open Data Principles such as timely, accessible

and primary.

Nine

questions

OPENGOV

DATA

(2018)

https://theodi.org/blog/exploring-open-data-quality

https://docs.google.com/forms/d/e/1FAIpQLSeCl3flPZIGYEWhhASTEnwF6GkMERIxZ8OWjkKpJCmqRlLwoA/viewform

https://docs.google.com/forms/d/e/1FAIpQLSeCl3flPZIGYEWhhASTEnwF6GkMERIxZ8OWjkKpJCmqRlLwoA/viewform


of 215

3.3.2 The standard for Systems and Software Quality Requirements and Evaluation – ISO 25010 and ISO 25012

Evaluation of OGI ICT toolkits is divided into two categories. The first of building blocks and the second for evaluation of cubes design. The summary of data collection and methodology of evaluation is presented at Table 11.

Table 11 - Cube Design and Building blocks data collection and methodology methods of Evaluation

Category Target groups Data Collection Approach Methodology of Evaluation

Product Quality

ICT Partners and IT Department of PAs

Questionnaire and Structured observation of application/website

ISO/IEC 25010

Quality in Use

System’s Data Quality

ISO/IEC 25012

Since the beginning, criteria for evaluation the OGI toolkit needs to be defined. Scientific literature review couldn't provide us with an extensive list of standards and requirements organized and structured. On the other hand, ISO/IEC 25010:2011, the standard for Systems and Software Quality Requirements and Evaluation (ISO/IEC, 2010), presents a structured list of requirements for building blocks and systems, which we considered for cubes design.

ISO 25010 is adopted as the evaluation method for OGI ICT toolkit. ISO 25010 is organized in 8 parameters which are divided into 30 measurement variables presented at Table 12.

Table 12 - OGI Toolkit Requirements for Evaluation

No Parameter Description Measured by Description

1 Functionality

the degree to which the OGI solution platform provides functions that meet stated and implied needs when used under specified conditions

Functional completeness

the set of functions covers all the specified tasks and user objectives

Functional correctness

the correct results with the needed degree of precision

Functional appropriateness

the accomplishment of specified tasks and objectives

2 Performance

the degree to which the OGI solution platform performs relative to the number of resources used under stated conditions

Time behavior

the response and processing times and throughput rates of a product or system, when performing its functions, meet requirements.

Resource Utilization

the amounts and types of resources used by a product or system, when performing its functions, meet requirements.

Capacity the maximum limits of a product or system parameter meet requirements.

3 Compatibility the degree to which the OGI solution Coexistence perform its required functions


of 215

platform can exchange information with other products, systems or components, and/or perform its required functions while sharing the same hardware or software environment.

efficiently while sharing a common environment and resources with other products, without detrimental impact on any other product.

Interoperability exchange information and use the information that has been exchanged.

4 Usability

the degree to which the OGI solution platform can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use

Appropriateness recognizability

users can recognize whether a product or system is appropriate for their needs.

Learnability

can be used by specified users to achieve specified goals of learning to use the product or system with effectiveness, efficiency, freedom from risk and satisfaction in a specified context of use.

Operability has attributes that make it easy to operate and control.

User error protection

protects users against making errors.

User interface Aesthetics

the user interface enables pleasing and satisfying interaction for the user.

Accessibility

can be used by people with the widest range of characteristics and capabilities to achieve a specified goal in a specified context of use.

5 Reliability

The degree to which the OGI solution platform performs specified functions under specified conditions for a specified period.

Maturity meets needs for reliability under normal operation.

Availability operational and accessible when required for use.

Fault tolerance operates as intended despite the presence of hardware or software faults.

Recoverability recover the data directly affected and re-establish the desired state of the system.

6 Security

the degree to which the OGI solution platform protects information and data so that persons or other products or systems have the degree of data access appropriate to their types and levels of authorization

Confidentiality ensures that data are accessible only to those authorized to have access

Integrity prevents unauthorized access to, or modification of, computer programs or data

Non-repudiation proven to have taken place, so that the events or actions cannot be repudiated later

Accountability actions of an entity can be traced uniquely to the entity

Authenticity the identity of a subject or resource can be proved to be the one claimed

7 Maintainability the degree to which the OGI solution Modularity composed of discrete components


of 215

platform can be modified to improve it, correct it or adapt it to changes in the environment, and in requirements

such that a change to one component has minimal impact on other components

Reusability an asset can be used in more than one system, or in building other assets

Analysability

possible to assess the impact on a product or system of an intended change to one or more of its parts, or to diagnose a product for deficiencies or causes of failures, or to identify parts to be modified

Modifiability effectively and efficiently modified without introducing defects or degrading existing product quality

8 Portability

the degree to which the OGI solution platform can be transferred from one hardware, software or other operational or usage environment to another

Adaptability

can effectively and efficiently be adapted for different or evolving hardware, software or other operational or usage environments

Installability can be successfully installed and uninstalled in a specified environment

Replaceability can replace another specified software product for the same purpose in the same environment

Source: (ISO/IEC, 2010)

The quality in use relates to the impact or outcome of the product when used in a particular context and consists of 5 parameters which are divided into 11 measurement variables as shown in Table 13.


of 215

Table 13 - Criteria for Evaluation of Quality in Use

No Parameter Description Measured by Description

1 Effectiveness accuracy and completeness with which users achieve specified goals - -

2 Efficiency resources expended about the accuracy and completeness with which users achieve goals - -

3 Satisfaction

the degree to which the user needs are satisfied when using the OGI solution platform in a specific context of use

Usefulness

the degree to which a user is satisfied with their perceived achievement of pragmatic goals, including the results of use and the consequences of use

Trust

the degree to which a user or other stakeholder has confidence that a product or system will behave as intended

Pleasure

the degree to which a user obtains pleasure from fulfilling their personal needs

Comfort the degree to which the user is satisfied with physical comfort

4 Freedom from Risk

the degree to which the OGI solution platform mitigates the potential risk of the usage

Economic Risk Mitigation

the potential risk to financial status, efficient operation, commercial property, reputation or other resources in the intended contexts of use

Health and Safety Risk Mitigation

the potential risk to people in the intended contexts of use

Environmental Risk Mitigation

the potential risk to property or the environment in the intended contexts of use

5 Context coverage

the degree to which the OGI solution platform can be used with effectiveness, efficiency, freedom from risk and satisfaction in both specified contexts of use and contexts beyond those initially explicitly identified

Context Completeness

can be used with effectiveness, efficiency, freedom from risk and satisfaction in all the specified contexts of use

Flexibility

can be used with effectiveness, efficiency, freedom from risk and satisfaction in contexts beyond those initially specified in the requirements

Source: (ISO/IEC, 2010)

During the evaluation, each measurement variable was assessed from the user perspective, such as public administration offices, citizens and businesses. Their input was used to improve the OGI solution platform by the consortium, and the revision was evaluated again during the next stage of the pilot.


of 215

3.3.3 User Experience (UX) and User Interface (UI) testing

While user experience (UX) is a term that has been used on practice and scientific literature, but is

hardly deep described or conceptualized uniformly. After conducting 275 interviews on the UX area

and deep literature review, Law, Roto et al. (2009) realized the reasons due an ill-description and ill-

conceptualization of UX:

1. First, because of the broad range of fuzzy and dynamic concepts, including emotional, affective, experiential, hedonic, and aesthetic variables.

2. Second, because of flexibility on analysis since single point to a holistic process. 3. Third, due to the fragmented and theoretical models involved on the UX domain.

Hassenzahl and Tractinsky (2006) conceptualize UX as "a term associated with a wide variety of

meanings ranging from traditional usability to beauty, hedonic, affective or experiential aspects of

technology use" (Forlizzi and Battarbee 2004). Garrett (2010) structures user experience as a project

with five dimensions and two product layers (as functionality and as information), from the more

abstract to the more concrete: strategy, scope, structure, skeleton, and surface. This structured is

presented in Figure 9.

Figure 9 - UX Structure and layers of product and information

Source: Garret (2010)

To improve the usability of software and information systems, the paradigm of user-centered

design, International Organization for Standardization (ISO) 13407, Human-centred design processes

for interactive systems, is a standard that guides user-centered design (Jokela, Iivari et al. 2003).


of 215

The ISO 9241-210 replaced the ISO 13407, which aimed to provide guidance on achieving quality in

use by incorporating user-centered design activities throughout the life cycle of interactive

computer-based systems. ISO 9241-210 standard describes six key principles that will ensure your

design is user centered (Travis 2011):

1. The design is based upon an explicit understanding of users, tasks, and environments. 2. Users are involved throughout design and development. 3. The design is driven and refined by user-centered evaluation. 4. The process is iterative. 5. The design addresses the whole user experience. 6. The design team includes multidisciplinary skills and perspectives.

ISO 9241-210 recommends the use of "ripple effect." It means to plan all the possibilities of tools

and scenarios of usage before implementing. After implementation, scenario, tools, activities, goals,

etc., can change, and influence the result. If the plan is well conceptualized, the plan is likely to

succeed. Figure 10 visualizes an example path taken due changes of plans made during the

implementation.

Figure 10 - The ripple effect

Source: Garret (2010)

Further to the explanations given on ISO 9241-210 (Travis 2011) and (13407 Travis 2011), on both

standards, there are no clear guidelines of steps to implement UX. Filling in this void, Jokela et al.

(1999) proposed a guideline to fill this blank comparing both standards. The guideline consists of 6

steps:

1. Identify the need for human-centered design;


of 215

2. Understand and specify the context of use; 3. Specify the user and organizational requirements; 4. Produce design solutions; 5. Evaluate design against requirements (loop to step 1 if not reach the desired requirement);

and, 6. A system satisfies specified user and organizational requirements.

Figure 11 - UX implementation and evaluation steps

Source: Jokela et al. (2003)

Besides the steps to conduct implementation of UX, it was also identified by Jakola et al. (1999) that

measures are not created to identify efficiency or any goal that should be reached. For this, we are

using the ISO/IEC 25010:2011 (ISO/IEC 2010).

The ISO/IEC 25010:2011 has a parameter called "usability" in which six measures define if a system is

usable. If consortium identifies the need to improve this usability, the Albert and Tullis (2013)

evaluation method can be used as an auxiliary. We consider that User Interface (UI) is a

complementary effect of UX and associated with the look, feel and interactivity of system. It is

already measured on the UX standards and ISO 25010:2011, in special on the evaluation quality of

use.


of 215

4 OGI Evaluation Results

The previous sections provided insights regarding plan and methodology used to evaluate co-

creation framework, OGI toolkit and its implementation in six pilots. This section presents the results

of the evaluation process.

This section is divided into two main sections. The first section 4.1 shows the Pilot’s Evaluation

Results. The first section includes the evaluation of:

1. Pilots’ Co-creation;

2. Pilots’ acceptance of OGI App; and,

3. Pilots’ outcomes

a. Transparency; and,

b. Administrative burden.

The second section 4.1.7 shows the OGI ICT Toolkit Evaluation Results. The second section includes

the evaluation of:

1. OGI Data Quality;

2. OGI Architecture;

3. System Quality of Application; and,

4. External evaluation of OGI ICT Toolkit.


of 215

4.1 Pilots’ Evaluation Results

This section presents the evaluation results of the six pilots. Figure 12 below summarizes the

common flow of activities performed in pilots during implementation and evaluation.

Figure 12 – Common flow of pilots activities


of 215

4.1.1 Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece)

4.1.1.1 Pilot Description and Expectation

The Greek pilot is under coordination of the Greek Ministry of Administrative Reconstruction

(MAREG) and has an initial objective to improve the monitoring and management of Government

Vehicles used by all Greek Public Agencies. The data that MAREG possesses for this monitoring and

management originate from different sources have not yet been properly defined, structured and

combined to be converted to meaningful information to facilitate internal decision-making and to

increase transparency towards the public.

Taking into consideration the results of the first iteration, the updated target groups (A) and

expected user scenarios (B) are presented below:

A- Target Groups:

1. Decision makers of MAREG

2. Public agencies operating government’s Vehicles

B- Expected User Scenarios:

1. As a user, I want to be able to have an overview of all Government Vehicles operating in

Greece based on descriptive statistical measures

2. As a user, I want to have an overview of all Government Vehicles operating in each Public

Agency

3. As a user, I want to have an overview of Government Vehicles per Municipality

4. As a user, I want to have an overview of Government Vehicles per Prefecture

5. As a user, I want to have an overview of Government Vehicles per Region

6. As a user, I want to search municipalities according to their population and have an overview

of the Government Vehicles operating in them

7. As a user, I want to search municipalities according to their altitude and have an overview of

the Government Vehicles operating in them

For a detailed description of MAREG and Greek pilot, please check report D1.1 and D4.1.

4.1.1.1.1 Identified Pilot Problems in the First Year

MAREG invited pilot’s stakeholders for this workshop with the objective to introduce them to the

pilot’s objective and extract from participants’ useful inputs for Greek pilot and OGI ICT toolkit

development. Important to highlight, all the workshops performed on this evaluation process were

conducted using the Delbecq et al. (1975) methodology. Below, the list of Greek workshop

objectives:

Optimize monitoring and management of Government Vehicles

Facilitate internal decision-making

Minimize operational costs related to Government Vehicles

Increase transparency on data related to Government Vehicles


of 215

Table 14 provides the questions for the participants.

Table 14 - Co-Creation User Workshop Questions

Question

number Question

1

Describe the main problems encountered in your interaction with the

Ministry of Administrative Reconstruction regarding Government

Vehicles?

2 What do you think are the causes of these problems?

3 What are the proposed solutions to these problems?

4 How do you think the current pilot could help towards these solutions

by using linked open data?

5 What useful information would you like to be produced by combining

the data sets available?

6 How will this information facilitate your daily work?

7 What services do you think could be available to the public?

8 What data could be openly available to the public?

The Greek pilot is mainly addressed to internal government users as it aims to improve decision-

making regarding government matter. The use cases refer to the user’s need for accurate

information on government’s Vehicles. This section presents two examples of the need for such

reporting:

The Ministry of Administrative Report needs to reply to a parliamentary question about the

number of Government Vehicles that operated by all Public Agencies. For this direct access

to accurate data about the actual number of Government Vehicles, as well as statistical

measures, such as their average cubic capacity, or their average fuel consumption per

kilometer is needed.

Public Agency X requests the permission to acquire a new four-wheel drive 2.000 cc car.

User Y of the Department in charge of Government Vehicles needs to get a detailed

overview of how many Government Vehicles operate in each Public Agency, as well as a

summary of their technical specifications (e.g., average cubic capacity).

After 30 minutes of ideation, participants were invited to explain their perspectives, bringing ideas

and solutions for the issues identified on the questions at Table 14.


of 215

Table 15 - Summary of Ideas and Solutions from participants

Problems Solutions

Data Quality Open API Solutions, Automatic dataset updates,

Manual data cleansing

Data Integrity Involve government agencies

Data is not open Open API solutions, common license template

Datasets are not

linked Use OGI Tools to link datasets

Reports from data

needed

Use visualization tools and link datasets to derive

reports

The main data set which contains descriptive data about Government Vehicles consists of data that

were collected regularly using a spreadsheet template that was sent to all Government Vehicles

beneficiaries and were not extracted from an Information System. This spreadsheet template did not

follow any standards, nor had any specific guidelines, which resulted in ad hoc data entry by users.

This is the most serious problem recognized at the workshop, and different solutions were suggested

to solve this issue by participants.

The problem of data integrity was also connected to the method of data collection, as well as with

the non-existence of an Information System available to end-users, who would also contribute in

data updates. In the current situation, Government Vehicles data needs to be updated manually by

users of the Ministry of Administrative Reconstruction. The OGI project will, therefore, serve as an

opportunity to review available datasets.

The problems of data not opened and not linked to each other, as well as the lack of standardized

reports that would facilitate internal decision-making are also connected with the absence of an

Information System for Government Vehicles and shall be addressed by the OGI project.

The list of organizations in order in English is presented below:

1. Ministry of Administrative Reconstruction (MAREG) – OGI Research team;

2. Ministry of Administrative Reconstruction (MAREG) – Government Vehicles Department;

3. Ministry of Infrastructure, Transport, and Networks;

4. Hellenic Police; and,

5. Ministry of Finance – Public Property Management Directorate.


of 215

4.1.1.1.2 Solutions for Pilot problems in Second Year

Some of the problems identified were solved in the second year and marked as “done.” If not yet

solved, it was planned to be deployed in the third year and were considered as “in progress.”

Table 16 - Summary of Ideas and Solutions from participants – First Year

Problems Solutions Actions Taken Status

Data Quality

Open API Solutions,

Automatic dataset

updates, Manual data

cleansing

Considering the main issue for data quality is

the creation of proper ontology and

metadata, the Greek pilot has 60% done.

For example, not all the agencies agree in

terms used to label the data, while part of

agencies uses abbreviations for names and

other use the full name.

In Progress.

Data Integrity Involve government

agencies

In the first year of the project, three datasets

were found with similar data to describe the

government vehicles and their status,

performance, etc.

In the second year, it was decided which

would be the selected data set. After that, the

data quality to find a common sense of what

are the useful data, metrics, and ontology to

be used. This issue is connected with the issue

of Data Quality.

Done.

Data is not

open

Open API solutions,

common license template

Technical partners linked datasets and later,

data cubes were created as a preparation to

opening the data.

In Progress.

Datasets are

not linked

Use OGI Tools to link

datasets



opening the data.

In Progress.

Reports from

data needed

Use visualization tools and

link datasets to derive

reports

Pivot table on the top of the CubiQL API.

There is no stable version for all pilots and is

still being developed by technical partners.

The visualization will be solved using the Pivot

Table on the top of the CubiQL API.

In Progress.


of 215

4.1.1.1.3 Solutions for Pilot problems in Third Year

Part of the problems identified was solved in the second year and marked as “done” or description

of the last status achieved by the pilot.

Table 17 - Summary of actions taken about Ideas and Solutions from participants – Third Year Status


Data Quality

Open API Solutions,

Automatic dataset

updates, Manual data

cleansing

Considering the main issue for data quality is

the creation of proper ontology and

metadata, the Greek pilot has 100% done.

A lot of time was spent on this task because

the data cleansing was made manually

Done in the

Third Year of

Implementation

Data Integrity Involve government

agencies

In the first year of the project, three datasets

were found with similar data to describe the

government vehicles and their status,

performance, etc.

In the second year, it was decided which

would be the selected data set. After that,

the data quality to find a common sense of

what are the useful data, metrics, and

ontology to be used. This issue is connected

with the issue of Data Quality.

Done in the

Second Year of

Implementation

Data is not

open

Open API solutions,

common license template



opening the data.

Done. Open via

SPARQL

endpoint

Datasets are

not linked

Use OGI Tools to link

datasets



opening the data.

Done. Linked by

default.

Reports from

data needed

Use visualization tools

and link datasets to

derive reports

An application was created using the Cube

Visualizer, producing data visualization.

Check Figure 15.

Done.


of 215

4.1.1.2 Pilot Co-Creation Framework Evaluation

The Co-Creation in the Greek pilot is connected with the section 4.1.1.3 about Data Quality

Evaluation. Taking into consideration the data quality situation of Greek pilot, technical partners and

the person in charge of pilot decided to focus on increasing the data quality before creating the

application. The main reason is due to the existent of datasets from different sources inside the

government with divergent data about the same vehicles. These datasets must be compared and the

only datasets that Greek pilot MAREG had was unstructured, resulting from ad hoc data entry and

full of errors. The complete list of other issues identified in the Greek pilot was already describe

above in the section 4.1.1.1. A blog post about the Greek pilot was also created in the OGI Medium

to summarise this scenario: https://medium.com/opengovintelligence/opengovintelligence-

showcase-the-greek-pilot-90639ae3fd3d.

Further, the first step was to perform data cleansing and data comparison along with CERTH, which

resulted in a total number of structured data records that amounted to 40% of the entire fleet. The

next step demanded the validation of the existing data and data collection for the remaining 60% of

the fleet. To encounter these problems, MAREG along with CERTH, elaborated a detailed plan for

data validation with input from end-users. The plan is including the development of a web app for

data validation, where users will log in, view their Government Vehicles data and validate them, by

editing, adding or deleting vehicles. This web app was created by CERTH using a free open source

tool for the development of Linked Data Applications created by the Callimachus project

(http://callimachusproject.org/), in honor of the Greek poet and scholar at the Library of Alexandria

(https://en.wikipedia.org/wiki/Callimachus).

The next step demands the contribution of end-users, mostly employees of Decentralized

Administrations in charge of Government Vehicles, who need to log in and provide feedback on the

validity of the initial data imported in the app. The success of this step demands the imposition of

the participation of all end-users by the top management in MAREG, preferably by the Minister

herself. An existing risk that needs the attention of the Greek Pilot team is the recent change in the

organization chart of MAREG, which brought by the change in the duties of the new structure, as

well as a change of people now in charge of the management and supervision of Government

Vehicles.

The following flowchart describes the main steps of data validation, as part of the co-creation

process:

Figure 13 – Greek Pilot Co-Creation Steps

The following screenshot shows the homepage of the data validation app:

https://medium.com/opengovintelligence/opengovintelligence-showcase-the-greek-pilot-90639ae3fd3d

https://medium.com/opengovintelligence/opengovintelligence-showcase-the-greek-pilot-90639ae3fd3d

http://callimachusproject.org/

https://en.wikipedia.org/wiki/Callimachus


of 215

Figure 14 – Greek Pilot Application Callimachus


of 215

4.1.1.3 Data Quality Evaluation

Table 18 – Data Quality Survey Results – Third Year

Data Quality

Attributes Results Overview

Discoverability and

Usability

1- App has links to the original data sets;

2- App has a searchable list of datasets used;

3- App allows users to download the data sets, including a partial download of selected

variables;

4- App does not have multiple format download option;

5- App allows to user preview and interacts with the data.

Granularity 1- Not only aggregate level of data (e.g., an average of cars) but individual unit data (raw

data of cars’ specifications)

Intelligibility 1- Supporting documentation exists, but has to be found separately

Trustworthiness 1- The data sets collected are known by developers, including how it was processed, with

reliable information and double checked with external sources.

Linkable to other

data

(5 Stars LOD)

1- The data sets are available in different formats to different audiences, from XLS to CSV

and RDF;

2- The RDF is linked to other data to provide context.

15 Open Data

Principles

1- Data is complete;

2- Data is primary;

3- There are still issues about timely availability;

4- Data is accessible to the widest range of users (XLS, CSV, and RDF)

5- Data is non-discriminatory because it is freely available;

6- Data has license-free (CLS, CSV, RDF);

7- Data is safe to open;

8- Data is not 100% designed with public input (decide format and parts of data sets to

visualize and download).


of 215

Table 19 – Data Quality Improvement Progress

Data Quality

Attributes

Status

1st Year 2nd Year 3rd Year

Discoverability

and Usability

The app in development,

with few functionalities.

The app developed, almost

complete and with more

functionalities.

The app is complete with all the

demanded functionalities and

datasets.

Granularity

Diverse datasets with low

quality and only raw or

aggregate level.

Task-force for data quality

unified and enriched datasets.

Ontology for all variables.

Datasets have the demanded

granularity.

Intelligibility

Low level of intelligibility,

with no context. High level

of data sensitivity.

High level of intelligibility, lower

level of data sensitivity, with

data context.

High level of intelligibility, lower

level of data sensitivity, with

data context.

Trustworthiness

Poor data quality due

incomplete, not primary

and not linked data sets.

Data was not reliable.

Increase data quality due to

linking data and enriching with

context from external sources.

Data quality can be trusted with

context from external sources.

Linkable to

other data

(5 Stars LOD)

Not following all the five

stars LOD principles.

Following all the 5 Stars LOD

principles.

Following all the 5 stars LOD

principles

15 Open Data

Principles

Not following all the 15

Open Data Principles.

Following 7 of 8 principles that

match OGI needs.

Following 7 of 8 principles that

match OGI needs.


of 215


Data Quality

Attributes Summary of Benefits Achieved Summary of Challenges Faced

Discoverability

and Usability

Greek pilot improves the discoverability

and usability. Data quality was improved,

and visualizations were created on the

top of the data cubes.

Combining data quality and end-users surveys,

Greek pilot can increase the number of data

sets in the cubes and include new asked

functionalities.

Granularity

Despite the 1st

year, the second year

Greek applications brings both raw and

aggregate data level of granularity.

This attribute is maxed out since both options

are present (raw data and aggregate data).

Intelligibility

Data were linked, new ontology to

normalize all data sets combined,

metadata giving context and data cube

enabling data visualization.

After linking the data, new ontology, metadata

and data cubes, Greek pilot almost maxed out

the attributes to bring intelligibility.

Trustworthiness

From datasets with poor quality in the

first year to a second year with high data

quality and reliable content.

The Greek pilot improved the data quality by

achieving 100%.

Linkable to

other data

(5 Stars LOD)

The first year had data sets in

spreadsheets (XLS and XLSX). The second-

year increase from 2 stars to 5 stars.

The Greek pilot maxed out the 5 stars (RDF)

with an API endpoint to access the data using

automate machine-readable access.

15 Open Data

Principles

The second-year achieved 7 of 8

principles that should be followed in OGI

pilot.

The pilot is not following the design with

public input (users can select format and part

of data to download).

4.1.1.3.1 DataSets and Data Cubes

Below the list of datasets used on the Greek Pilot:

1. Data describing Government Vehicles

2. Data on the lifecycle of Government Vehicles

3. Data on the operation and maintenance of Government Vehicles

4. Statistical data on Greek Public Agencies and their personnel

5. Statistical data on Greek Municipalities, Prefectures, and Regions (describing their

population, their topography, their climate, etc.)

A sample of data sets in both not linked (CSV file) and linked (RDF) are stored here:

https://drive.google.com/open?id=0B0_fFuauifo0M1V2REl2OG13QTQ.

https://drive.google.com/open?id=0B0_fFuauifo0M1V2REl2OG13QTQ


of 215

Currently, the Greek data cubes with linked data have the following dimensions for search and

visualization:

1. Vehicles Data cube

a. Area: containing municipalities in both Greek and English characters (e.g., Athens,

Thessaloniki);

b. Fuel Type: containing the eleven types of fuel (e.g., Diesel, Benzene);

c. Vehicle Type: containing the 15 types of vehicles (e.g., Bus, Car, Motorcycle)

Figure 15 – Vehicles data Cubes Screenshot


of 215

4.1.1.4 Outcomes Evaluation

To evaluate the application from the perspective of end-users, a questionnaire was distributed to

the end-users. In total, 45 people were surveyed in Greek Pilot using the OGI acceptance

questionnaire to evaluate the Greek Pilot Application

(http://wapps.islab.uom.gr/CubeVisualizer/vehicles/).

For this reason, the results are interpreted more qualitatively, making use of comments made by the

respondents.

1. Linked Open Statistical Data (LOSD) Familiarity;

2. User-Acceptance evaluation;

3. Datasets Quality; and,

4. Perceived Outcomes.



of 215

4.1.1.4.1 Linked Open Statistical Data (LOSD) User Familiarity

Figure 16 – Familiarity of Greek App users with LOSD

In the first part of user-questionnaire, we asked about how familiar is the respondent with open

data, linked open statistical data, open data portals and datasets provided by the Greek app. From

the responses, we can assume that the respondents are highly familiar with open data portals and

Greek app datasets, as well as with open data. However, only 52% of the respondents is familiar with

LOSD. These results indicate that LOSD is a new concept that needs to be introduced specifically to

Greek civil servants (as targeted users of Greek Pilot app), so they can use it to improve their works

and public services.


of 215

4.1.1.4.2 User Acceptance

Figure 17 – User acceptance of Greek app respondents

In the second part of user-questionnaire, we addressed an user acceptance test which aims to

validate the end to end business flow of the app. There are 12 questions (adopted from acceptance

theories and OSI standard) asked to the respondents perception in using the app, ranging from the

functionalities, technical issues, to the easiness and willingness to use as shown in Figure 17.

Regarding the functionalities, 86% of the respondents said that the app has a proper and well-

integrated functionalities. 98% of the respondents expressed that the functionalities of the app is

working properly when they are using it, however, 50% of the respondents experienced a crash

during using the app. 84% of respondents stated they have a clear and understandable interaction

with the app . Next, 77% of the respondents expressed that the Greek app have a proper design.

Regarding the easiness to use the app, we got conflicting results. 92% of the respondents claimed

that it is easy to use the app, with 98% stated that they have sufficient skills to use the app.

Furthermore, 100% of the respondents claimed that using the app does not require high level of

technical knowledge. However, in the next question, only 43% of the respondents stated that the

app is not difficult to use.

Finally, 89% of the respondents stated that the app is usefulness for their work, and 92% of the

respondents claimed that their colleagues will accept the app.

In summary, most respondents stated that the Greek application is accepted by them in terms of it

has a good level of technical functionality, is easy to use, and is useful.


of 215

4.1.1.4.3 Datasets Quality

Figure 18 – Perception of data quality provided by Greek app

The third part of the user-questionnaire is addressed the data quality provided by the Greek app.

We were using 4 data quality parameters in this questionnaire: accessibility, completeness, accuracy,

and in accordance with user requirements from the co-creation process.

95% and 93% of the respondents claimed the Greek app already provided a proper data accessibility

and data accuracy, respectively. 68% of the respondents stated that the Greek app already provided

data as requested by users, and 75% of the respondents is satisfy with the data quality level of Greek

app. The only drawback is in term of completeness which only stated by 32% of the respondents.


of 215

4.1.1.4.4 Perceived Outcomes

Figure 18 – Perception of data quality provided by Greek app

In the last part of the questionnaire, we addressed the outcome perceived by the respondents

regarding the Greek app. The respondents react positively in all of the expected outcome of the app:

89% for the better interpretation, 93% for the better decision, 91% in the efficiency (93% in reducing

time and 100% in reducing cost), 93% in increasing transparency, 93% in helping them achieving

goals. In addition, 89% of the respondents willing to use the app, while 93% asked for more

functions to improve transparency.

By this figure, we can conclude that the Greek app is useful in introducing the concept of LOSD to

the civil servants while also showing the benefits of using the LOSD in their daily routinies. In the

case of Greek app, the results align with the aims of the pilot to improve the decision making

regarding the use of government vehicles.


of 215

4.1.1.5 Screenshots of The Greek Ministry of Administrative Reconstruction (MAREG) Application

Figure 16 and Figure 17 present the screenshots of OGI ICT toolkit using data sets from the Greek

Pilot.

Figure 16 - Pie Sorted Graph Vehicle Type

Figure 17 - Bar Graph Filtered by Area (Municipality)


of 215

4.1.2 Pilot 2 – Enterprise Lithuania (Lithuania)


Enterprise Lithuania used OGI ICT Toolkit to extend service provided by Lithuanian pilot. The

objective was to identify the needs of business for exploiting LOSD, developing new user-friendly

tools for businesses to help them benchmark their business ideas in the overall context of Lithuania

business, providing tools for enabling businesses to create applications using LOSD, and helping

businesses create value from LOSD. For a detailed description of Enterprise Lithuania and Lithuanian

pilot, please check report D1.1 and D4.1. A Medium blog post was also created summarizing the

advances and challenges found by Lithuanian Pilot: https://medium.com/opengovintelligence/the-

lithuanian-pilot-a-market-research-solution-3330346a7131.


Aiming to identify what is the pilot situation, during the first year of implementation (2016) a scan

was performed into the pilots. This scan happened in a user workshop, as part of the OGI Co-

Creation Framework step “co-initiation” and “co-design.” The user workshop had the presence of

potential stakeholders presented below:

The government of the Republic of Lithuania;

Information Society Development Committee under the Ministry of Transport and

Communications (ISDC);

Vilnius city municipality administration;

Infobalt;

Kaunas University of Technology;

Mykolas Romeris University;

E TRADE, UAB; and,

Lithuanian Free Market Institute.

First, the OGI project was presented, explaining the potential solutions that LOSD and data cubes

could bring to the pilot. Then, questions were given to the participants as potential end-users,

presented in Table 21. These questions aimed to identify four situations:

1. What was the (open) data scenario?;

a. Are they in Open data format?

b. There is any data set not yet own and need to be accessed?

c. What are the barriers in the access or use of this data?

2. What was the system scenario?

a. How can data cubes (deployed as an application) help?

b. What are the functionalities needed?

https://medium.com/opengovintelligence/the-lithuanian-pilot-a-market-research-solution-3330346a7131

https://medium.com/opengovintelligence/the-lithuanian-pilot-a-market-research-solution-3330346a7131


of 215


Question

number Question

1 What type of data (open data) is available to be used? What are the forms to access them?

2 What are the Data quality issues on your datasets or accessing data sets already opened?

3 What are the legal barriers to use and access the data sets?

4 What are the needs of Lithuanian Pilot users, how could this service benefit them?

5 Which functionalities, features, and datasets could be useful for pilot users?

Second, two user scenarios were defined, based in the answers to the questions in Table 3.

1. “Julie has a children’s clothes shop in Vilnius, and she is doing so well that she decides to

expand her business to another city. She needs to know where is the best place to

open another shop and where is the highest concentration of children, the city with the

highest income and where is the lowest competition. She knows that the data is available in

the Lithuanian one-stop-shop for business. She uses the LOSD tool and finds out that the best

place to expand, according to the statistical data, is Siauliai city.”

2. “John has small IT company (IT services for medium and large companies) in London, and he

wants to open a new branch in Lithuania. He needs to know where is the best place to open

branch and where is the highest concentration of young people (21-35 years old) [from data

set – “Resident population at the beginning of the year (by age groups)“], what is average

earnings [from data set – „Average earnings (monthly (2015))”] and municipality with

highest rate of foreign direct investment [from data set –“Foreign direct investment per

capita at the end of the year“]. He knows that the data is available in the Lithuanian one-

stop-shop for business. He uses the LOSD tool and finds out that the best place to expand,

according to the statistical data, is Vilnius city municipality. But John wants more information

(e.g., about education (by sectors), unemployment rate (by sector), etc. ). He uses smart

feedback tool to inform the Lithuanian one-stop-shop for business about his needs to analyze

additional criteria (new data sets). After few days Lithuanian one-stop-shop for business adds

new criteria (new data sets) and informs John by email about these improvements”.


of 215

The summary of problems and solutions from participants is presented in Table 22.

Table 22 - Summary of Ideas and Solutions from participants – First Year

Problems Solutions

Data Sets are not Linked Encourage the government to open more data sets. Funds to

support initiatives must be addressed to municipalities.

Data Quality Quality of data does not reflect reality due to a list of issues as for

example lack of collection, storage, and centralized databases.

Civil Servants Capacity Building

(Skills and experience)

Civil servants are not prepared to deal with LOSD and opening-up

data sets.

Data Integrity Involve government agencies to work together for better

interoperability between databases and data sets.

Data Format (not open) Opening-up more data sets in programming formats such as

Application Programming Interface (API)

Lack of Metadata and low quality

of datasets description

Metadata is still missing. Effort on creating metadata to give

context on data sets.

Interactive Functionalities on

Online Platform (Data Vis, Table

for comparison, etc.)

Management level has no skill to develop data analytics properly

and data visualization projects. The government can create an

environment helping them to data-driven decision-making.

Reports from data needed Use visualization tools and link datasets to derive reports

The answers from participants show that Lithuanian pilot ca be considered on an initial maturity

stage on opening-up and linking datasets. The initial maturity stage is clearly influencing the needs of

stakeholders. Demands like a low number of data sets opened represent a big issue on creating

“good” quality of LOSD. Without open data, it is not possible to create LOSD and even harder to

create more interactive functionalities on a potential online platform, such as OGI ICT toolkit can

provide.

The lack of metadata influences the level of transparency and accountability. If new business people

cannot understand what or where is the data sets, they are unlikely to create data-driven decision-

making. The provided examples of user scenarios represent this type of case. People are looking for

data, but cannot find or identify the relationship between them.

4.1.2.2 Solutions for Pilot challenges in Second Year

The problems identified were solved in the second year and considered as “done,” summarized in

Table 23. If not yet solved, it was planned to be deployed in the third year and were considered as

“in progress.”


of 215

Table 23 - Summary of Problems, Expected Solutions and Implementation - Second Year

Problems Expected Solutions Actions are taken Status

Data Sets are not

Linked

Encourage the government to open

more data sets. Funds to support

initiatives must be addressed to

municipalities.

Data sets were linked by technical

partners, and data cubes were

created

Done.

Data Quality

Quality of data does not reflect

reality due to a list of issues as for

example lack of collection, storage,

and centralized databases.

Data quality was improved taking

into consideration the issues

identified.

Done

Data Integrity

Involve government agencies to work

together for better interoperability

between databases and data sets.

In regards to data quality, the actions

taken are presented in section 3.2.3.

Done

Lack of Metadata

and low quality of

datasets

description

Metadata is still missing. Effort on

creating metadata to give context on

data sets.

in regards to data quality, the actions


Done

Data Format (not

open)

Opening-up more data sets in

programming formats such as

Application Programming Interface

(API)

In regards to data quality, the actions


Done

Civil Servants

Capacity Building

(Skills and

experience)

Civil servants are not prepared to

deal with LOSD and opening-up data

sets.

Since the pilot app has not yet stable

version, the end-users were not

trained yet. The report “D4.7

Recommendations and tutorials for

opening-up and exploiting statistical”

data will be delivered in month 33.

In

Progress

Interactive

Functionalities on

Online Platform

(Data Vis, Table for

comparison, etc.)

Management level has no skill to

develop data analytics properly and

data visualization projects. The

government can create an

environment helping them to data-

driven decision-making.

Pivot table on the top of the CubiQL

API. There is no stable version for all

pilots and is still being developed by

technical partners.

In

Progress

Reports from data

needed

Use visualization tools and link

datasets to derive reports

Also will be solved using the Pivot

Table on the top of the CubiQL.

In

Progress


of 215

4.1.2.3 Solutions for Pilots’ challenges in the Third Year

Table 24 summarize the actions taken in both second and third year to complete the issues

identified by Pilots’ participants.

Table 24 - Summary of Problems, Expected Solutions and Implementation - Third Year

Problems Expected Solutions Actions are taken Status

Data Sets are not Linked

Encourage the government to open more data sets. Funds to support initiatives must be addressed to municipalities.

Data sets were linked by technical partners, and data cubes were created

Done in the Second Year.

Data Quality

Quality of data does not reflect reality due to a list of issues as for example lack of collection, storage, and centralized databases.

Data quality was improved taking into consideration the issues identified.


Data Integrity Involve government agencies to work together for better interoperability between databases and data sets.

In regards to data quality, the actions taken are presented in section 3.2.3.


Lack of Metadata and low quality of datasets description

Metadata is still missing. Effort on creating metadata to give context on data sets.



Data Format (not open)

Opening-up more data sets in programming formats such as Application Programming Interface (API)



Civil Servants Capacity Building (Skills and experience)

Civil servants are not prepared to deal with LOSD and opening-up data sets.

Created the report D4.7 Recommendations and tutorials for opening-up and exploiting statistical and specific training were given to pilot leaders and users.

Done in the Third Year.

Interactive Functionalities on Online Platform (Data Vis, Table for comparison, etc.)

Management level has no skill to develop data analytics properly and data visualization projects. The government can create an environment helping them to data-driven decision-making.

The stable version was released in the third year and has visual functionalities demanded by users. The evaluation of these functionalities is in section 4.1.2.6.


Reports from data needed

Use visualization tools and link datasets to derive reports

The stable version was released in the third year and has visual functionalities demanded by users. The evaluation of these functionalities is in section 4.1.2.6.



of 215


4.1.2.4.1 Co-Implementation Step

The Lithuanian pilot preferred to have a continued workshop with online participation of

stakeholders. The participants of the first workshop in the first year were invited to evaluate the

application developed in the second year via an end-user survey (section 7.1.2).

The main suggestions from stakeholders are:

1. Increase the number of data sets for a more complete search; and

2. Increase the number and improve functionalities in the application.

The suggestions will be deep described in the next sections of Data Quality Evaluation and Outcomes

Evaluation in the Second Year. For example:

“Few datasets in www.opendata.gov.lt. A too small amount of open data sets, data

formats (often only pdf (not machine-readable data format)), datasets update

frequency, lack of metadata.”

Respondent B

“Lack of metadata or low quality of metadata are one of biggest problems to reuse

open data for business or public purposes. Another huge problem is the lack of

open data, too old data, and too much-aggregated data.”

Respondent D

“The Organisation X suggests adding more functionality to allow users to inform

our institutions and suggest solutions based on open data.”

Respondent F

“The Organisation Y needs more analytical functions of our pilot: possibility to

compare the same indicator at different regions, maybe to show aggregate

indicators, to rank regions by selected indicators, etc. The Organisation Y performs

ranking of regions and wants an online tool like is planned in the pilot.”

Respondent H

http://www.opendata.gov.lt/


of 215


In the third year, a survey was conducted on technical partners to identify the stage data quality


Data Quality


Discoverability and

Usability

1- App has no links to the original data sets;

2- App has a searchable list of datasets used;

3- There is no option for users download the data sets;


5- App does not allow to user preview and interacts with the data.

Granularity 1- Not only aggregate level of data (e.g., an average of cars) but individual unit data (raw

data).

Intelligibility 1- There is no supporting documentation exists.

Trustworthiness

1- The data sets collected are not well known by developers, including how it was

processed. However, data is trusted and considered reliable, with minimum consistence

with external sources.

Linkable to other

data

(5 Stars LOD)

1- The data sets are available in different formats to different audiences, from XLS to CSV

and RDF;

2- The RDF is linked to other data to provide context.

15 Open Data

Principles


2- Data is primary;

3- There are still issues about timely availability;

4- Data is accessible to the widest range of users (XLS, CSV, and RDF)







of 215


Data Quality

Attributes

Status


Discoverability

and Usability

The app in development,

with few functionalities

and few data sets.

App developed, still being

updated and without expected

functionalities asked by users.

Updated. App has all

functionalities demanded by

users.

Granularity

Diverse Datasets with low

quality and only aggregate

level.

A task-force for data quality to

unify and enrich data sets is

needed.

Updated. App has the

demanded datasets with the

initial expected granularity.

Intelligibility

Low level of intelligibility,

with no context. Low level

of data sensitivity.

Still same scenario with Low

level of intelligibility without

proper data context.

Updated. Minimum data

context with external sources,

however, without

documentation.

Trustworthiness




Data was not 100%

reliable.

Current scenario with some

improvements. Increase data

quality with data cubes linking

data sources however without

proper context from external

sources.

Updated. Data sources are

linked with external sources.

Linkable to

other data

(5 Stars LOD)




principles.

Updated. Following the 5 stars

LOD Principles.

15 Open Data

Principles


Open Data Principles.

Following 6 of 8 OD principles

that match OGI needs.

Updated. Following 6 of 8 OD

principles that match OGI

needs.


of 215

Table 27 – Data Quality Improvement Progress – Third Year

Data Quality


Discoverability

and Usability

Lithuanian pilot improved the

discoverability and usability. Data quality

was improved, and visualizations were

created on the top of the data cubes.

New 8 datasets and functionalities

demanded by end-users survey in the 2nd

Year.

Granularity There is still the same level of granularity. Preference for aggegate level (statistics),

however, unit level of data is also presented.

Intelligibility Data were linked and data cube enabling

data visualization.

Data is linked and new functionalities

provides new types of visualization.

Trustworthiness

A still similar scenario of poor and not

reliable data sets. The focus of 2nd

year was

given in other attributes.

Data is more reliable in comparison with the

2nd

year, but there is still room to improve

data quality.

Linkable to

other data

(5 Stars LOD)

The first year had data sets in spreadsheets

(XLS and XLSX). The second-year increase

from 2 stars to 5 stars.

The Lithuanian pilot maxed out the 5 stars

(RDF) with an API endpoint to access the

data using automate machine-readable

access.

15 Open Data

Principles

The second-year achieved 6 of 8 principles

that should be followed in OGI pilot.

The pilot is not following the design with

public input (users can select format and

part of data to download).


of 215

4.1.2.5.1 Data Sets

Below the current list of data sets used on the Lithuanian Pilot:

1. Resident population at the beginning of the year (by age groups);

2. Average earnings (monthly (2015));

3. State Food and Veterinary Service;

4. State Social Insurance – Sodra;

5. Investment in tangible fixed assets at current prices (2015);

6. Economic entities in operation at the beginning of the year;

7. Resident population at the beginning of the year (by age groups);

8. Resident population at the beginning of the year (by gender);

9. Resident population at the beginning of the year (persons by municipality); and,

10. Foreign direct investment per capita at the end of the year (2014).

More datasets can be opened and linked taking into consideration the future needs of Lithuanian

pilot. The sample of data sets are here:

https://drive.google.com/open?id=0B0_fFuauifo0NmIzZFdqeTVDWjg

https://drive.google.com/open?id=0B0_fFuauifo0NmIzZFdqeTVDWjg


of 215



the end users. Several different end-users were interviewed directly, using the questionnaire. The

results from these interviews are displayed below. A questionnaire was distributed to end-users of

the Lithuanian pilot. This questionnaire was filled in by 43 users, which does not allow for rigorous

quantitative analysis taking into consideration these two Apps:

a. http://vmogi03.deri.ie:8080/superset/dashboard/5/

b. http://vmogi03.deri.ie:8080/superset/dashboard/7/

For this reason, the results are interpreted more qualitatively, making use of comments made by the

respondents.


2. User-Acceptance evaluation;






of 215

4.1.2.6.1 Linked Open Statistical Data (LOSD) User Familiarity

Surprisingly, the level of familiarity with Open Data and LOSD of Pilot App users was low.

Only 2% were Familiar with LOSD, 16% Familiar with Open Data, 23% Familiar with LOSD portals and

33% Familiar with the Datasets presented by the App (all presented in blue part of the stack). Figure

18 summarizes the results.

Figure 18 – LOSD User Familiarity – Third Year


of 215


In contrast with the familiarity with Open Data and LOSD, the Pilot App Acceptance is massively

positive. Major part pointed the functions have been working properly, few people experiencing a

crash during the use, the app interaction is acceptable, useful, and, easy to use. The people surveyed

also believe to have sufficient skills to use the app creates, your colleagues will accept the app, being

not difficult to use and having functions integrated.

However, a major part of the people understands the app still have a high level of technical

knowledge to use the app (100%), missing some functionalities (49%).

Figure 19 – User Acceptance – Third Year


of 215


The surveyed people had a positive reaction from the data set quality, with an overall of 74%

satisfaction (data quality satisfaction). The data accessibility, accuracy, and reduced time to find the

data searched.

However, by 33% of the users, not all the datasets required were present in the app. Beyond that,

35% of respondents believe the data is incomplete.

Figure 20 – Datasets Acceptance – Third Year


of 215


A major part of the dimensions surveyed about expected results had positive reactions. Al the

respondents accept the app. The main reasons to this positive reactions are associated with the

positive reactions in other dimensions such as 98% of users believing the app brought the better

interpretation of data and reduced time to look for the information they need. Further that, the app

helped 95% of surveyed people in increasing efficiency and made better decisions.

However, while 93% of them pointed an increase of transparency, users believe that more functions

can create even more transparency. This dimension can be exploited in the future with a new

qualitative search and a new round of co-creation within the users. Probably, after some rounds of

co-creation and evaluation, the expectation of users increased since the first year of the project, in

2016.

Figure 21 – Results – Third Year


of 215

4.1.2.7 Screenshots of Lithuania Enterprise Application

The next figures present the screenshots of OGI ICT toolkit using data sets from the Pilot.

Figure 22 - Table with bar chart functionality

Figure 23 - Table with heatmap functionality


of 215

4.1.3 Pilot 3 – Tallinn Real Estate (Estonia)


The Estonian Ministry of Economics will make use of the OGI ICT Toolkit to address issues in the

Estonian real estate market such as timely publication of data and information asymmetry. To best

identify the barriers facing transparency in the Estonian real estate sector interviews have been

carried out with all stakeholders from the private sector, the public sector, and the Estonian public at

large. In combination with these interviews, workshops have been conducted to gain a better

understanding of the current needs of those with interest in the real estate sector.

For a detailed description of Enterprise Lithuania and Lithuanian pilot, please check report D1.1 and

D4.1.


The purpose of the user workshop in Tallinn, Estonia, was defined by the following four points:

1. To get end-user feedback

2. To improve the initial offering of the public service

3. Involve stakeholders with the design of the service

4. Raise awareness of the Estonian Real Estate Pilot Program

Once the initial goals of the Estonian Real Estate Pilot Programs were discussed, a first of our two

sessions were performed. In the first session which was titled “Developing the Estonian Real Estate

Pilot Program” we focused on the following questions:

What problems do you see with this pilot program and its goals?

o What are the main areas in the real estate sector that you see issues?

What are the potential solutions for these problems?

o How could this pilot program better address these issues?

In what ways would this new public service be beneficial for Estonia?

All participants were given time to think carefully on these questions and write down their answers.

After the ideation process, a group discussion was held to elaborate each participants’ answers.


of 215

The second session was titled “Constructing the Functionality of the Estonian Real Estate Pilot

Program.” For this session we wanted to focus on gathering participant created personas and user

stories. As we are following an agile development approach for the pilot program creation, we made

sure to co-generate user stories, and personas with the workshop attendants as these are key parts

of agile development. They were shown the following slide information on a slide:

After that, this task was addressed:

For this session, we split the participants into pairs and had each pair generate up to three personas.

After they had time to prepare their personas and user stories, these were discussed as a group.

What we had hoped to do with this user workshop was to get individuals from many different

agencies in the same room talking about Linked Open Statistical Data in Estonia, and in this end, we

succeeded. We had many participants from 7 public sector organizations and 2 private sector

companies.

User Stories As an X I want to do Y.

As a <type of user>, I want <some goal> so that <some reason>.

Personas

Demographics - who is this person?

Needs - what are their needs, how could this service benefit them?

Example: Priit is a 35 year old Estonian who has recently moved to the United Kingdom for work. He speaks English quite well and has a bachelor's degree in computer science. Since he just arrived he is having issues when it comes to finding a career which is related to his degree.

Why do these matter?

1. Prepare at least one persona:

Demographic Information

i. Who is this person? How old are they? What is their occupation?

What do they do?

ii. Needs

a. (What sort of reasons would this person have for using our

public service)

2. Using the previously generated persona, please try to write down 10 to 15 user stories

for this person.

Reminder: As a <type of user>, I want <some goal> so that <some reason>.


of 215

Going into the workshop, we had hoped to receive information about:

Potential issues with the pilot

Potential solutions

User Stories

User Personas

Other organically generated information.

In the first session, we received the following information about potential problems and solutions

our pilot would face. They have been ordered by a number of occurrences/mentions going from the

most commonly mentioned problem/solution to the least commonly mentioned.


Problems identified

by Users Expected Solutions from Users

Data Quality Open API Solutions, Automatic dataset updates

Confidentiality

issues Anonymize the data

Competition from

existing real estate

portal

Involve users in the design process, talk with

current users of existing portals

Data Integrity Update national registries involve

governmental agencies

Data is not open Open API solutions, common license template

Datasets are not

linked Use OGI ICT Toolkit to link data sets

Needed data not

collected

Get users to provide data.

The biggest issues which participants perceived matched what previously identified by pilots’

partners on initial research. In Estonia, the data quality is quite poor for many data sets. If the data

exist, it may not even be useable due to the “confidential” nature of such data.

When looking at the solutions, mostly they are out of our hands and require government

intervention or policy changes. The problems and solutions which we can best address are


of 215

competition from other real estate portals, needed data not being collected, and, to some extent,

confidentiality issues.

When discussing the benefits that the Estonian Real Estate portal could have for Estonia, the main

five benefits were:

1. Increased transparency in the real estate sector;

2. Fairer pricing-model;

3. Happier citizens;

4. One stop shops for real estate data; and,

5. Increased availability and usage of real estate information.

In the second session, we had asked participants to generate user personas for potential users of the

Estonian Real Estate Portal. A list of the basic generated personas in order of discussion is below.

As a foreign IT specialist coming to Tallinn, I need a safe, environmentally friendly

place to live; information on recycling and public transportation, so that I could live

in a clean environment, recycle, move easily in the city.

As a family with 5 and 8-year old kids, I want to buy a house outside of Tallinn, we

need low living costs, information on schools and kindergartens, transportation

information, info on shops, sports and entertainment facilities, green areas and

parks so that I wouldn’t have to walk more than 20 minutes to school and

kindergarten

As a student moving to Tallinn, I need information on the location of universities,

public transportation, if it’s a tolerant & safe neighborhood, cheap living costs,

proximity of entertainment facilities; or using central heating.

Elderly Finnish couple, 65+, sold apartment in Helsinki, moved to Tallinn to spend

their golden years, looking for a 2-3-room apartment in very good condition with

everything necessary in walking distance or close by public transportation. Need

information on expenses related to the apartment (communication costs, mortgage,

loans taken by the house union), is there an elevator in the house, type of house,

location (safe, important PoI-s nearby)

Male, 35-years old, coming from South Estonia to find a job in Tallinn. Wants to

move his wife and 2 children to Tallinn. Looking to rent a 2-room apartment and has

a price limit, needs public transportation nearby and a municipal school in his area.

Mikk, a young graduate with a wife and baby on the way, looking for a family place

to move to, wants the portal to suggest areas that meet his requirements, and once

he finds the area, he wants to have more information about prices, quality of the

building, etc.


of 215

A guy got married, wants to sell his apartment and move to a new one. He wants to

sell his apartment and get the highest possible price, wants to promote his area and

to check out what advantages he can mention in his selling advertisement.

A real estate company wants to develop their service of selling accommodation.

They would like to make personalized offers based on user preferences, so the

portal could offer them property in the areas that interest them.

As an exchange student coming to Estonia for 3-4 months, I want to rent an

apartment or a room in a shared apartment, need the apartment to be furnished,

close to the city center and/or university, an English-speaking landlord, sports

facilities nearby, affordable meals close-by, information about pollution, traffic, etc.

She’s 45 disabled and in a wheelchair, a heavy smoker, working as an office clerk,

wants to buy an apartment, needs public transport, handicap accessible, close to

work and hospital, and if there are any public smoking places, and clubs for people

with similar situations.

The 25-year old heterosexual couple wants to buy their apartment. No children yet

but maybe soon so they need a two-room apartment. Interested in police

complaints, noise in the area, energy efficiency level of the building. Need elevator,

good view from their window (no high buildings close to their windows), have

electric cars which they need to charge and if there are any vegetarian restaurants

nearby.

When looking at these personas and what needs they had, the following datasets stood out as the

most important:

1. Transport

2. Safety

3. Price of real estate

4. Environment

5. Points of Interest

6. Information on the property

The last point of discussion will be how this information benefits our pilot program and how we can

use it for the design of the pilot. What we now know is the main problems/solutions, who may use

our service, the benefits people expect, and what datasets we should use for our initial pilot

program. Regarding future research, we need to address the issues of data confidentiality as well as

finding out how/why people use other real estate portals.

Regarding initial pilot construction, we have outlined the following for our 1st iteration.


of 215

Initial target groups:

1. Foreign students

2. Estonians new to Tallinn

Initial datasets:

1. Transport information

2. Safety statistics

3. Price statistics

4. Points of interest data (schools, doctors’ offices, etc.)

5. Information on the property (age, utilities, amenities, etc.)

Initial functionalities

1. As a user, I want to be able to search by address so that I can see the location on a map

2. As a user, I want to be able to see how safe my home is so that I can feel safe when moving

to a new location

3. As a user, I want to be able to measure the distance between my house and different points

of interest so that I can travel more efficiently

4. As a user, I want to be able to find addresses where I could live based on my criteria (like

price and safety) so that I don’t waste time searching for real estate that I don’t wish to see

5. As a user, I want to be able to find out how often public transport comes by my address so

that I can understand how much time I will have to spend commuting via public transport

In the workshop participants came up with ideas individually, and then these were discussed as a

group. The ideas put forth by every applicant were collected, and all feedback was recorded. We

believe that having individuals from many different agencies gives strength to the proposed initial

direction of the pilot program. Expected barriers are known now, giving the pilot a better

understanding of the solutions to overcome these barriers. In addition, a better understanding of

who may be the users of new Estonian Real Estate portal and how they will want to use it are

required.

It is important to highlight the importance of co-creation process, including end users in the process

of OGI ICT Toolkit development and dividing responsibility about the definition of pilots’ goals. As

noted, objectives changed from the initial common target users to international students and new

Estonians moving to Tallinn, as an example. It means a need to have further user workshops where

include more of the end users and trial our initial prototype service on them to get even further

input.


of 215

Below, the List of organizations that participated on the user workshop in Tallinn (Estonia). All names

are translated in English:

1. European Commission

2. Mooncascade

3. Ministry of Economic Affairs and Communications

4. Tallinn University of Technology

5. Finance Ministry

6. Teleport

7. Land Board

8. Estonian Statistics Board

9. Tallinn City Government

10. National Registers and Information Systems Center

11. Land board

The link to a presentation given to participants is below:

https://docs.google.com/presentation/d/1r55VkEJxT5ECyYnJleCsHqEwjm55bg5OT8-

bPHPlVH0/edit?usp=sharing

https://docs.google.com/presentation/d/1r55VkEJxT5ECyYnJleCsHqEwjm55bg5OT8-bPHPlVH0/edit?usp=sharing

https://docs.google.com/presentation/d/1r55VkEJxT5ECyYnJleCsHqEwjm55bg5OT8-bPHPlVH0/edit?usp=sharing


of 215

4.1.3.2 Solutions for Pilot problems in Second Year

The problems identified were solved in the second year and considered as “done.” If not yet solved,

it was planned to be deployed in the third year and were considered as “in progress.”



Data Quality

Open API Solutions,

Automatic dataset

updates

The OGI CubiQL API was created in the second

year and can be as a solution for this issue. In Progress

Confidentiality

issues Anonymize the data The data set is completely anonymized. Done

Competition

from existing

real estate

portal

Involve users in the

design process, talk

with current users of

existing portals

A hackathon was hosted and attracted

students and developers. There is an

opportunity for new events with the same

characteristic to improve the user-friendly and

data quality.

Done

Data Integrity

Update national

registries involve

governmental agencies

Data sets are still being updated and enriched.

The third year will finish the data quality task-

force.

In Progress

Data is not

open

Open API solutions,

common license

template

Data sets are linked and in open format. Done

Datasets are

not linked

Use OGI ICT Toolkit to

link data sets

Data sets are linked however there is no data

cube. Potentially, in the third year can have

time to use data cubes and CubiQL API.

Done

Needed data

not collected

Get users to provide

data.

Some data there is no access yet due to

organizational issues. In Progress


of 215


The user workshop done in the first year decided to use five data sets in the initial prototype of

Tallinn Real Estate Application. The selected datasets were:

Public Transportation (buses stops and lines, traffic accidents in Tallinn);

Crime (type and position of crimes in Tallinn);

Real Estate pricing (average price to buy and rent houses in Tallinn);

Schools (positions and names of schools in Tallinn); and,

Property data (property information in Tallinn).

Because of some data quality issues, described in section 4.1.3.4, the five selected data sets required

a round of cleansing to make them proper to the Tallinn Real Estate Application. The initial cleansing

was conducted by staff members at Technical Tallinn University (https://www.tlu.ee/en). Further,

part of these data sets was improved and updated in a progressing hackathon sponsored by the

Ministry of Economic Affairs and Communications (MKM) (https://www.mkm.ee/en), the

organization in charge of this pilot.

Since the data sets were cleansed, the pilot started to link the data sets, create the data cubes and

web site. For this, a team was formed with members working in the Estonian pilot and members of

the big data science team from the private sector company Portal (https://nortal.com/).

After 48 hours, the initial porotype for the Tallinn Real Estate Pilot was presented to the audience. It

was awarded an honorary mention for providing valuable location-based information. From this

hackathon, goals were achieved by participating in this hackathon:

1. Open Government Data (OGD) can be used to create new services;

2. If OGD is freely available to the society, an initial prototype created in few days with relevant

public value to society;

3. Hackathons are one of the strategies that governments and private organizations can use to

aware people and developers; and

4. If reunited, individuals from different sectors can come together and use their free time to

achieve goals in a short time.

After the hackathon, with a prototype created, between 15 and 18 March 2017, TTU and MKM

started to develop a fairly simple and easy to understand user interface for the pilot project

(https://rnd-tut.shinyapps.io/Estonian_Pilot/).

The use of GitHub and developing the service in an agile and open source using the R package

helped the success in developing together with stakeholders (co-implementation and co-evaluation)

of the pilot program in three ways:

https://www.tlu.ee/en

https://www.mkm.ee/en

https://nortal.com/



of 215

1. GitHub allows Pilot developers to easily identify issues and share the original Pilot

Application code, asking external developers to help to solve the identified issues. Then,

external developers can view, copy the code with one click (fork) and work in their

environment without any prejudice for the original code created by the Pilot developers. If

external developers solve an issue, they can ask to merge the code into the source published

by a pilot, who approves or not the new solution.

2. GitHub is flexible and free, allowing anyone to creates new functionalities and not just

solve others’ issues. Creativity from external developers can lead to new insights and new

applications; and,

3. Since GitHub has free access, the code published by Tallinn Real Estate Pilot Application is

exportable. The code and functionalities created can be used by any other city, country, or

town in the world.

The pilot was also described in the blog post at OGI Medium account:

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-estonian-pilot-

511fb4647e8 .

Figure 24 – Audience of Tallinn Real Estate Hackathon

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-estonian-pilot-511fb4647e8

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-estonian-pilot-511fb4647e8


of 215

Figure 25 – Developers coding for the Tallinn Real Estate Hackathon


of 215


Table 30 – Data Quality Survey Results

Data Quality


Discoverability and

Usability


2- App has a searchable list of data sets used;

3- App allows users to download the data sets, including a partial download of selected

variables;


5- App allows to user preview and interacts with the data.

Granularity 1- Only raw (individual) level of data is available.

Intelligibility 1- Supporting documentation can be found at the same time as the data (e.g., right next to

the link for the data). The metadata was created by a Pilot task-force.

Trustworthiness 1- The data sets collected are well-known by developers, including how it was processed;

2- The data is 100% reliable and is consistent with external sources.

Linkable to other

data

(5 Stars LOD)

1- The data sets are available in different formats to different audiences, from PDF, XLS to

CSV. RDF was not yet created by OGI developers in this pilot;

2- The RDF is not yet properly linked to other external data to provide context.

15 Open Data

Principles


2- Data is primary;

3- There are issues about timely availability;

4- Data is accessible to the widest range of users (XLS, CSV, and static visualization)


6- Data has license-free (CLS, CSV, static visualization);

7- Data is safe to open

8- Data is partially permanent;




of 215


Data Quality

Attributes

Status

1st Year 2nd Year

Discoverability

and Usability

The app in development, with few

functionalities and few data sets.

The app is still under development, still being

updated and some expected functionalities

asked by users not yet implemented.

Granularity Diverse Datasets with room to improve

data quality and only raw level.

A task-force for data quality to unify and enrich

data sets is needed (Create data cube and RDF).

Intelligibility Low level of intelligibility, with no context.

Low level of data sensitivity.

An increased level of intelligibility however still

without proper data context.

Trustworthiness

Poor data quality due incomplete, not

primary and not linked data sets. Few

datasets published.

Current scenario with some improvements.

Increase data quality with however without

proper context from external sources.

Linkable to other

data

(5 Stars LOD)

Not following all the 5 stars LOD principles.

Some data sets were not published and

accessed via a static web site.

Not following all the 5 Stars LOD principles, still

missing the RDF.

15 Open Data

Principles

Not following all the 15 Open Data

Principles.

Following 6 of 9 OD principles that match OGI

pilot needs.


of 215


Data Quality


Discoverability

and Usability

Estonian pilot improved the

discoverability and usability. Data

quality was improved, and map

visualization was created in the web.

Combining data quality and end-users’ surveys,

Estonian pilot can increase the number of data sets

and include new asked functionalities by

stakeholders that tried the app.

Granularity There is still the same level of

granularity (raw data).

Estonian pilot can consider finding new data sets

that provide aggregate level but keeping the raw

data available (primary). It helps to reach a wide

audience and create new types of visualization.

Intelligibility

Data visualization was created, but

without using the linked datasets

(RDF) or data cube.

There is still room to improve Estonian pilot taking

into consideration co-creation participation from

stakeholders.

Trustworthiness

Data sets are reliable, from official

open data portals. The focus of 2nd

year was given in other attributes.

There is still room to improve the reliability of

Estonian data sets and external connections.

Linkable to

other data

(5 Stars LOD)


spreadsheets (XLS and XLSX). The

second-year increase from 2 stars to

4 stars (RDF) and map visualization.

The Estonian pilot can improve it linking the data,

creating SPARQL endpoint (RDF) to reach the 5 LOD

stars.

15 Open Data

Principles

The second-year achieved 7 of 9

principles that should be followed in

this OGI pilot.

The pilot is not yet following the design with public

input (users can select format and part of data to

download) and only has an aggregate level (not

primary data).


of 215

4.1.3.4.1 Data Sets

Public Transportation (buses stops and lines, traffic accidents in Tallinn);

Crime (type and position of crimes in Tallinn);

Real Estate pricing (average price to buy and rent houses in Tallinn);

Schools (positions and names of schools in Tallinn); and,

Property data (property information in Tallinn).

The data sets are already in RDF (LOD format). The link to access datasets:

https://drive.google.com/drive/folders/0B89NqGUhD5HxRGQ0ZXNwMU9vNkE

https://drive.google.com/drive/folders/0B89NqGUhD5HxRGQ0ZXNwMU9vNkE


of 215



the end users. Seventeen people responded to this survey for the Tallinn Real Estate pilot. The

results are displayed below. Since only seventeen people responded, the results are more qualitative

than quantitative as no statistical tests can be run on such a sample size. The attributes collected

and evaluated were:


2. User -acceptance evaluation;




of 215

4.1.3.5.1 Linked Open Statistical Data (LOSD) Familiarity

Figure 26 Tallinn app respondents’ familiarity with LOSD

From Figure 26, we can assume that respondents of Tallinn app familiar with open data and open

data portal, however, less familiar with LOSD and Tallinn app datasets. Lack of familiarity with Tallinn

app datasets is understandable since most of the users of the Tallinn app is citizens, or international

visitor.

This results also shows that there is an opportunity to introduce LOSD to wider and varied public

group and use the LOSD to improve public services.


of 215


Figure 27 Tallinn app respondents user acceptance

In the second section of the questionnaire, we addressed respondents’ perception regarding user

acceptance of the app. Majority respondents agreed that the Tallinn app already provided a well-

integrated technical functionality and a proper design. 80% of the respondents also stated that the

functionalities of the app were working properly when they were using it and only 30% of the

respondents experienced with crash during using the app.

Majority of the respondents are also claimed that they have a clear and understandable interaction

using the app and they found the app very useful.

The percentage of respondents which perceive that the app is easy, does not require high level

technical knowledge and think that they have sufficient skills to use the app are even higher than

previous parameters. However, respondents that said the app is not difficult to use is only slightly

higher than respondents who think it is difficult. Last, 95% of respondents expressed that their

colleagues will also accept the Tallinn app.

To sum up, Tallinn app is well-accepted by majority of the respondents.


of 215


Figure 28 Tallinn app datasets quality results

In regard to datasets quality, apart from completeness, majority of the respondents stated that

Tallinn app already provide a good datasets quality. 100% for data accuracy, 95% for time to find the

data, 80% for data accessibility and 70% for meeting the user requirements from the co-creation

process. Even though 85% of the respondents required improvement in data completeness, 80% of

the respondents already satisfied with the level of datasets quality provided by Tallinn app.


of 215


Figure 29 Tallinn app perceived outcome

In the expected outcome part, the responses of questionnaire which disseminated randomly to

Tallinn app users are showing encouraging results. More than 80% of respondents claimed that the

Tallinn app bring benefits in term of better interpretation of data, improve decision making, improve

efficiency (time and cost), and increase transparency. As 85% of respondents claimed that the app

already helped them in achieving goals, majority of the respondents perceived that the app does not

require additional functions. Last, 70% of the respondents personally accept the app.


of 215

4.1.3.6 Screenshots of Tallinn Real Estate Application

The next figure presents the screenshot of Estonian pilot application.

Figure 30 – Tallinn Real Estate Application

Figure 31 - Bar chart of Average Cost


of 215

Figure 32 - Pie chart Distribution of crashes


of 215

4.1.4 Pilot 4 – Trafford Council Worklessness (England)


Trafford’s Innovation and Intelligence Lab are working on using linked open statistical data to help

support decision making relating to worklessness. They are working closely with Swirrl, who are

handling the more technical aspects of modelling and storing the linked data. The goal is to build a

tool that will bring together data from a range of sources to help understand the factors that

contribute to, or are impacted by, worklessness. Representatives from the Department for Work and

Pensions; Trafford’s Economic Growth Team and the Greater Manchester Combined Authority are

also involved. For a detailed description of Enterprise Lithuania and Lithuanian pilot, please check

report D1.1 and D4.1.


Trafford Council provides services to around 226,000 people. It is part of Greater Manchester and

works actively with other neighbouring councils to share ideas, innovations and in some cases

services. A priority for Trafford is economic growth: to support businesses, to create jobs and to

tackle unemployment and the social challenges that brings.

Trafford is a leader in the UK in bringing digital innovations into its working processes. It was

identified by the UK government as one of the ‘Open Data Champions16’: a group of local

government organisations within the UK doing the most to promote and exploit open data. Trafford

Council has established an ‘Innovation and Intelligence Laboratory’17 to bring together data and

information specialists from many organisations who work on challenging problems for the public

sector in Trafford. The Innovation Lab has four priorities: Mental Health, Aging Population,

Unhealthy Weight and Worklessness.

The purpose of the workshop was to:

1. Raise awareness of the Trafford OpenGovIntelligence project;

2. Understand the current service provision and delivery environment;

3. Identify opportunities and challenges for future service delivery;

4. Engage stakeholders and end users in developing the project, including setting the project

objectives; and,

5. Develop a single multi-agency project delivery plan with key milestones, roles and

responsibilities.


of 215

One scenario was selected to be part of OGI Trafford pilot. It will concentrate on worklessness,

within these priority areas, the objectives are to:

help to reduce demand on services;

help to redesign services;

improve people’s awareness and understanding of the area; and,

help to attract or retain investment.

Trafford is responsible for a varied area, incorporating both deprived inner city communities, rich

commuter-belt communities, Trafford Park – the largest industrial estate in Europe employing

around 35,000 people in 1400 companies, (and also ‘Old Trafford’, the home of Manchester United

football club). The pilot will focus on innovate generation and application of data from a range of

sources to tackle the problems of worklessness:

measuring and attempting to match demand for and supply of skills, gathering data from

job-seeking individuals and from businesses;

seeking to use data and digital technology to find new approaches to assisting workless

people; and,

profiling the economy, skills base and assets of the area to identify potential improvements,

and to help attract new companies to invest in the area.

The pilot will include investigation of new methods of data collection from local businesses and

citizens. The pilot will both generate requirements for and evaluate and exploit tools to be

developed in OGI for combining, analysing and visualizing statistical data. Trafford is the lead council

for the topic of worklessness in the Association of Greater Manchester Authorities, so innovations

arising from the project will have their impact enhanced by a direct route to replication in other

areas. The follow questions were made to the audience:


of 215


Question

number Question

1 Can you tell us a bit about your role in the project?

2 What do you feel are the key aspects of your pilot in the project?

3 Which audience do you want to reach with your pilot?

4 How are you going to reach your target audience for the pilot?

5 How are you going to ensure user engagement?

6 How are you going to publicise it?

7 What do you feel are the main advantages of using multidimensional statistical data?


of 215

The summary of problems and solutions from participants is presented on the table below.


Problems Solutions

Not live data in reports Interactive tool feeding in the most up-to-date information

Multiple reports required Window view with wide selection of data behind it enabling the

user to select multiple data sets

Spatial misalignment Long-term – the success of the project could influence change in

how data is collated

Accessing ‘other’ data There will be multiple data sets available across a range of

themes (work, housing, health, skills etc.)

Knowing who to go to for ‘other’

data

By acting as a first-point for all data, people will be able to build

relationships across agencies and sectors as they use each other’s

data

Demand on InfoTrafford team By making data more accessible and available, this will upskill

people to be able to self-serve

Qualitative over quantitative Available and accessible data will provide a balance to ‘local

knowledge’

Demand for information on

policy staff

It will be quicker to respond to queries (and should enable more

self-service by senior officers)

Time delay in official release of

data

Long-term – the success of the project could influence change in

when data is released

Comparing appropriate data The tool will make this easier and will explain when this is not

possible.

Long-term – it encourages more open data in standardised

formats

Taking in consideration the capacity building from Trafford Pilot, mainly the Innovation and

Intelligence Lab and the tradition to opening-up data sets, lead this pilot on a mature stage of

technical area. For example, demands like “new functionalities” and “high level of report

automation” reveals high expectation of this pilots’ stakeholders.

However, Trafford is starting a new project based on the OGI co-creation framework. It takes time to

learn and put on practice the co-creation procedures. For example, identifying all stakeholders, user

workshop for referendum of ideas, objectives, inclusion of new demands from stakeholders,

collection of new data sets, opening-up and linking them are processes that take more time than


of 215

regular top-down approach, even based on data-driven services that Trafford Innovation and

Intelligence Lab is used to perform.

After the co-creation process, Trafford pilot will concentrate on worklessness. The Initial objectives

discussed are in bold format and italics added from co-creation user workshop. Within these priority

areas, the objectives are to:

Help to reduce demand on services

o using open shared data to direct more effective service interventions and enhance

partnership collaboration, reducing duplication

Help to redesign services

o enabling efficient resource planning through evidence-based decision-making

Improve people’s awareness and understanding of the area

o increasing local insight by providing an accessible, agile and easy-to-use data and

information tool

Help to attract or retain investment

o targeting the use of public sector people, funding, assets and resources to reduce

worklessness and increase skills

The list of participants:

1. Trafford Council – Economic Growth and Prosperity;

2. Trafford Council – InfoTrafford;

3. Trafford Council – Partnerships and Communities;

4. Department for Work and Pensions; and,

5. Greater Manchester Combined Authority.


of 215

4.1.4.2 Solutions for Pilot problems in Second and Third Year

The problems identified were solved in the second year and considered as “done”. If not yet solved,

it was planned to be deployed in the third year and were considered as “in progress”.

Table 35 - Summary of Ideas and Solutions from participants – Third Year

Problems Solutions Actions Taken Progress

Not live data in

reports

Interactive tool feeding in the most

up-to-date information

The next ICT Tools to be developed

are related to visualisations (Pivot

Table).

Done in the

3rd Year.

Multiple reports

required

Window view with wide selection of

data behind it enabling the user to

select multiple data sets

The current application can have

different versions of report, but in an

interactive web site. Cannot

download results or part of data.

Done in the

3rd Year.

Spatial

misalignment

Long-term – the success of the

project could influence change in

how data is collated

Data quality task-force is improving

this type of issue (lack and low

quality ontologies).

Done in the

3rd Year.

Accessing ‘other’

data

There will be multiple data sets

available across a range of themes

(work, housing, health, skills etc.)

New data sets are being collected,

cleansed and stored properly before

be included in the Trafford App.

Done in the

3rd Year.

Knowing who to

go to for ‘other’

data

By acting as a first-point for all data,

people will be able to build

relationships across agencies and

sectors as they use each other’s data

New agencies and data owners were

identified in the second year.

Done in the

2nd Year.

Qualitative over

quantitative

Available and accessible data will

provide a balance to ‘local

knowledge’

The new application can be used in

two ways. Visualisations of maps or

downloading data sets for self-usage.

Done in the

2nd Year.

Comparing

appropriate data

The tool will make this easier and will

explain when this is not possible.

Long-term – it encourages more

open data in standardised formats

The new application can be used in

two ways. Visualisations of maps or

downloading data sets for self-usage.

Done in the

2nd Year.

Demand on

InfoTrafford team

By making data more accessible and

available, this will upskill people to

be able to self-serve

The new applications are easy to use.

Potential videos and manuals can be

created to reduce issues for users.

Done in the

2nd Year.

Demand for

information on

policy staff

It will be quicker to respond to

queries (and should enable more

self-service by senior officers)

The new applications are easy to use.

Potential videos and manuals can be

created to reduce issues for users.

Done in the

2nd Year.

Time delay in

official release of

data

Long-term – the success of the

project could influence change in

when data is released

Co-creation helped to unify data

providers and users. Data quality

task-force is improving this type of

issue (primary and timely)

Done in the

3rd Year.


of 215


The Trafford Council pilot has the objective of tackling worklessness in Greater Manchester by

providing relevant decision makers with information in the form of visualisations of Linked Open

Statistical Data (LOSD) that facilitate data-driven decision making and increases transparency

towards the public.

The workshop was divided into several group discussions that sought to explore participant views on

the use of data for decision making within their work, the challenges they encounter and what

outputs the pilot could provide to facilitate the use of data. For each discussion, the participants

were presented with themes for discussion, asking them to work in pairs or small groups to discuss

individual ideas and write them down. After these initial discussions, each participant shared their

ideas to the group and the written ideas were gathered on flip-charts for future reference.

The workshop included a presentation of tools by the technical partner Swirrl and example outputs

from the Trafford Council pilot. As part of the co-creation framework, a Data Literacy Survey was

sent via email to participants before the workshop. The survey was designed to assess familiarity

with different data formats, sources of open data and types of visualisation. Each part of the

workshop, the tools and output presentations and the Data Literacy Survey will be described in the

below. The agenda of the co-creation user workshop is presented in the Table 36.

Table 36 – Co-Creation User Workshop Agenda

Time Activity

1:30 to 1:40 Welcome and introductions

1:40 to 1:50 Introduction to the OGI Project

1:50 to 2:20 Understanding the environment

1:40 to 1:50 Current tools and systems

2:50 to 3:10 New tools and systems

3:10 to 3:20 Ideal outputs and summary

The workshop had the following participants:

Swirrl

Department of Work & Pensions

MCA

Trafford Council

o Trafford Data Lab

o Partnerships & Communities

o Performance team

o Commissioning Team

o Public Health

o Economic Growth


of 215

The participants were introduced to the OGI Project and the aim of the Trafford Council pilot and the

technical partner Swirrl. The objectives of the workshop were also explained.

1. To raise awareness of the OGI project and the Trafford pilot;

2. To understand how data is used in decision-making;

3. To understand how data is currently accessed and the challenges involved; and,

4. To understand what tools would be needed to support effective use of data.

The participants were asked to discuss in small groups and write down in Post-it notes the decisions

or activities that they currently undertake that are informed by data. After the small group

discussions, the individual decisions/activities were presented to the wider group and the Post-it

notes were organised in flip-charts, one for each small group. After this initial discussion the

participants were asked to write down in their flip-chart next to each Post-it note the data they use

and the organisations related.

Current tools and systems

The participants were asked to share with the group what type of format they use and prefer when

using data. They wrote their preferred format on the flip-chart. The group discussion continued with

the challenges they encounter when obtaining, combining and analysing data.

New tools and systems

The technical partner Swirrl presented the GM Data Store and the Department for Communities and

Local Government's dashboard that visualise LOSD stored within an instance of their Publish My

Data system. GraphQL, part of the tools developed for the OGI project, could be used to access data

from GM Data Store. Afterwards, a prototype road traffic collisions application was demonstrated by

Trafford Council to show the type of interface and functionality that could be expected as a final

output from the Trafford Council pilot.

Ideal outputs

At the end of the workshop a summary was presented and the participants wrote on Post-it notes

what would they ideally like to help them with worklessness data/tasks. The Post-it notes were fixed

to a dedicated flip-chart for future reference.

Data literacy

Prior to the workshop, a survey was sent to the workshop participants to assess data literacy. The

respondents were presented with 8 questions created to evaluate the following aspects of data

literacy: Finding data, Reading data, Creating data, Cleaning data, Managing data, Visualising data,

Analysing data and Arguing with data.

Due to the variety of positions and organisations of the participants, a wide range of uses of data

were mentioned:


of 215

Allocation of resources and training procurement for staff;

Procurement of supportive provision for Jobcentre Plus users;

Data to support and inform design of strategies and planning for year ahead and long term;

and,

Preparing reports for internal monitoring and statutory returns.

The data they use are:

Internal datasets; and,

Official sources (Office of National Statistics, DWP Data Explorer, NOMIS, Stat Xplore).

The preferred formats when analysing data and for what they find it useful are:

Spreadsheets: enable the manipulation, filter and sorting of data according to the specific

necessities;

Charts: for analysis, comparison and to show variation on time; and,

Maps: to access spatial information for design of strategies and planning.

The challenges encountered when obtaining, combining and analysing data were:

Accessibility: Excessive repetition of tasks to produce the data they need.

Poor data literacy: The workload of the data analysts is increased due the inability of data

requesters to find the information they need for themselves with the available tools.

Inconsistency of categorisation of data.

The readers of the reports produced by the participants do not understand the

information they receive.

Data protection and the necessity of anonymize data.

From this, a summary of the ideal features that the participants would like on a tool:

Visuals and raw data both available;

Information available for small areas;

Maps with indication of differences on areas using scales;

Comparison facilities by areas, GM and UK;

Reusable visuals that are simple and understandable; and,


of 215

Easy to use user interface.


of 215



Data Quality


Discoverability

and Usability


2- App has no searchable list of data sets used;

3- App allows users to download the data sets, including partial download of selected variables

and CubiQL endpoint is linked to the Trafford Data Store the app will source data from there;

4- App has multiple format download option;

5- App allows to user preview and interact with the data.

Granularity 1- There is no aggregate level of data, only raw level of data from internal data sets.

Intelligibility

1- Supporting documentation can be found at the same time as the data (e.g. right next to the

link for the data). All of the data derived from government sources so included comprehensive

metadata.

Trustworthiness

1- The data sets collected are well-known by developers, but there is neutral confidence on

how it was processed;

2- The data is 100% reliable, and is consistent with external sources.

Linkable to other

data

(5 Stars LOD)

1- The data sets are available in different formats to different audiences, from XLS to CSV. RDF

was created by OGI developers;

2- The RDF is not yet properly linked to other external data to provide context. Once the data is

converted to RDF and stored in the Trafford Data Store it will be 'linkable'.

15 Open Data

Principles


2- Data is primary;

3- There is not issues about timely availability;

4- Data is accessible to the widest range of users (XLS, CSV and RDF)

5- Data is non-discriminatory because is freely available;


7- Data is permanent;





of 215


Data Quality

Attributes

Status


Discoverability

and Usability

App in development, with

few functionalities and

few data sets.

App developed, still being

updated and without PARTIAL

expected functionalities asked

by users in the workshop.

Updated with expected

datasets and functionalities.

Granularity Diverse Data sets with low

quality and only raw level.

A task-force for data quality to

unify and enrich data sets was

made. Still room to improve.

Updated with expected level of

data granularity. Check datasets

quality section 4.1.4.5.3.

Intelligibility Low level of intelligibility,

with low level of context.

Improvements. High level of

intelligibility without proper

data context.

Updated with proper data

source context to end-users.

Check outcomes evaluation

section 4.1.4.5.

Trustworthiness




Data was not 100%

reliable.

Data is linked, but without

external context. Waiting for

stable version of OGI CubiQL API

to solve this issue.

Updated with OGI ICT toolkit

finished and data in linked

format.

Linkable to

other data

(5 Stars LOD)




principles.


principles.

15 Open Data

Principles

Not following all the Open

Data Principles.


that matches OGI needs.




of 215


Data Quality


Discoverability

and Usability

Trafford pilot improved the

discoverability and usability. Data

quality was improved and

visualizations were created.

Combining data quality and end-users surveys,

Trafford pilot can increase the number of data sets

and include new asked functionalities by end-users

after trial workshop in the second year.

Granularity The same level of granularity (raw)

and is accepted by users.

Pilot can consider to find new data sets that provides

the aggregate data instead of showing only raw level.

New functionalities can emerge from this action.

Intelligibility Data was linked and data cube


Data is linked and pilot app has all the expected

functionalities for voisualization.

Trustworthiness

Still similar scenario of poor and

not reliable data sets. Focus of 2nd

year was given in other attributes.

The data is reliable, however there is still room to

improve the external connections.

Linkable to

other data

(5 Stars LOD)


spreadsheets (PDF, XLS). The

second year increase from 2 stars

to 4 stars (RDF).

After stable version of OGI ICT Toolkit, the Trafford

Pilot App could link all the datasets and achieve the 5

stars LOD.

15 Open Data

Principles

The second year achieved 8 of 9

principles that should be followed

in Trafford OGI pilot.

Pilot is not following the design with public input

(users can select format and part of data to

download). The objective of this app is to provide an

efficient visualization of the data.

Beyond that, user can download all the data and

cleanse the data by themselves.

4.1.4.4.1 Datasets

Below the list of data sets:

https://github.com/traffordDataLab/projects/blob/master/opengovintelligence/apps/production/work%3Cnes

s/about.md

https://github.com/traffordDataLab/projects/blob/master/opengovintelligence/apps/production/work%3Cness/about.md

https://github.com/traffordDataLab/projects/blob/master/opengovintelligence/apps/production/work%3Cness/about.md


of 215


To evaluate the application from the perspective of end-users a questionnaire was distributed to the

end users of the Trafford Council Worklessness Application. The questionnaire that they distributed

is somewhat different to the questionnaire that is used for the other pilots, but allows for qualitative

views of the same themes. In the end the questionnaire had 50 valid respondents.

1. Linked Open Statistical Dataset (LOSD) Familiarity;

2. User Acceptance;




of 215


Figure 33 – LOSD Familiarity – Trafford Pilot

From Figure 33, we can assume the surveyed people in Trafford are majorly people without high

level of IT skills. 57% of people were aware about open data, and 37% of them pointed to be familiar

with LOSD technology, LOSD portals and the datasets presented in the pilot.

This result can be seen as an opportunity to improve the awareness about LOSD technology and

usage of new datasets, increasing the chances to bring value to users. The section 4.1.4.5.4 shows

the perceived outcomes such as increase of transparency, better decision-making and time spent

searching, probably due these opportunities identified in this section results about familiarity (new

technology, new datasets to users).


of 215


Figure 34 – User Acceptance – Trafford Pilot

The second section of the survey asked to users about the acceptance of the application. Checking

the results from the Figure 34 – User Acceptance – Trafford Pilot, we can assume major part of the

answers show us a positive reaction. The only negative reaction from users is the needed of high

level of technical knowledge to use the app. This makes sense, because Trafford is one of the most

complex application, in terms of methodology to present the data and combining different data.

Another questions that shows this assumption is the “not difficult to use”, which 25% believe that

app is difficult to use.

To sum up, the user acceptance of Trafford Pilot is above 90%, which means very positive reaction

and high level of acceptance of the Trafford Application.


of 215


Figure 35 – Datasets Quality – Trafford Pilot

Datasets quality was tested in this survey. The first major complaint about data quality was the

completeness, with 37% of them pointing the incompleteness of datasets in the app. Second, 25% of

them suggesting that not all the required datasets were in the Trafford application. Others pointed

that data could be more accurate.

Beyond that, 88% of the users are satisfied with the data quality presented in the Trafford

Application. In accordance with the results of user acceptance, 96% of users believe that data has a

proper accessibility and in accordance with the results seen in the perceived outcomes, a reduced

time to find the proper data.


of 215


Figure 36 – Perceived Outcomes – Trafford Pilot

Following the other sections, the perceived outcomes had majorly positive reactions. In summary,

users pointed out an acceptance of 82% of Trafford perceived The only major complain is the

needed of new functions to create more transparency (90%).

On the other hand, users are satisfied with the efficiency (84%), reducing time to search information

(86%), helping to make better decisions (88%), increase of transparency (90%) and better

interpretation of data (92%).


of 215

4.1.4.6 Screenshots of Trafford Council Application

Figure below presents the screenshot of Trafford Council Application.

Figure 37 – Trafford Council Worklessness Application


of 215

4.1.5 Pilot 5 – The Flemish Environment Agency (Belgium)


The Flemish Government will use OGI ICT Toolkit to enhance their environmental policy making in

terms of timely publication of the actual state of affairs related to environment, evaluations of the

permits policy, and develop tools to benchmark the pollution of companies to others working in the

same economical domain.

For a detailed description of Enterprise Lithuania and Lithuanian pilot, please check report D1.1 and

D4.1. A blog post at OGI Medium account was also published describing the Flemish Application:

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-belgian-pilot-

1c52872cfe65 .

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-belgian-pilot-1c52872cfe65

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-belgian-pilot-1c52872cfe65


of 215


Instead of workshops, the Flemish pilot made a series of interviews with stakeholders due the high

number of people involved. The drivers of the interviews were:

What is the link of your business with the environmental permits?

What kind of data from the permit would you like to have in a structured format?

There were two types of questions, business related and data related. It is presented on the table

below.

Table 40 - Questions made to the audience

Question

number Questions

Business Related Questions

1 Do you give permissions?

2 Do you advise about permits?

3 Do you have enforcement tasks?

4 Do you yourself manage a base registry and what is the link with permits?

5 What would be the added value of a basic registry of environmental permits for you

business?

Data-Related Questions

6 What administrative data do you want to have? (decision date, issuer, type of procedure)

7 What kind of data would like to have on an exploitation level (emissions, time series of

permitted and effectively emitted emissions, Environmental impact assessment reports)

8 What kind of spatial data would you like to have that is created during the permitting

process?


of 215


Problems Solutions

No central register of permits

Legislative action to see that all permitting authorities register

permits. Documents as well as structured data;

Building a basic register centrally to which data can be sent.

No publication of emission data in LOSD format

Publishing data via LOD technology;

Reuse of existing data sets, as example European Environment

Agency;

Adding new vocabularies on existing vocabularies.

No re-use of data gathered during permission

processes


permits;

Documents as well as structured data building a basic register

centrally to which data can be sent.

No re-use of data from base registries

Re-use of data from base registries;

Starting program to publish base registries also via LOD

technology.

No base registry build-up


permits;

Documents as well as structured data building a basic register


No possibility of linking permission data with

emission data



Agency;



enforcement data


data of new permissions for same exploitation

No possibility of benchmarking for legal persons Providing online analysis possibilities for example benchmarking

No possibility of data analysis Providing online analysis possibilities

No possibility of data-driven policy-making Providing online analysis possibilities

Data in general is locked in silo’s



Agency;


Data is not very accessible Publishing data as LOD under open data licence:

Providing analysis and reporting tools;

Starting LOD program to publish data in LOD way and as part of

process of the organisations. Not as extra work.


of 215

Considering the issues and solutions purposed by participants on the Belgium pilot, we can point out

that this pilot is on a mature stage of technical aspects. For example, while data is not accessible

because IT departments silos and lack of legislation to cooperate, we can identify capacity building

with high level of skills due the demands such as desire to “link” and “data-driven service” to

“benchmark” individual cases on an “automated way”.

However, as Trafford pilot, the Flemish government is starting this project from the scratch if co-

creation framework is considered. The pilot partners reunited all the stakeholders and identified

that face a bigger problem. More data sets were identified and probably will be opened and linked

to enable the creation for functionalities and services demanded by all stakeholders. As example,

there is no way to create dashboards with real-time data sets without connecting and linking them

before.

The list of participants is below:

Administrations on the federal level

o Public health administration

Administrations on the Flemish level

o Department of environment nature and energy,

o Agency for forest and nature,

o The heritage agency,

o The environmental planning and permit department.

o Public waste agency of Flanders,

o Flemish energy agency,

o Flemish land Agency,

o Flemish agency for regulation of the energy market

Administrations on the province level

o PVV (Organisation representing the 5 provinces together with people representing

every individual province).

Administrations on the community level

Vereniging van Vlaamse Steden en Gemeenten (VVSG)

o Organisation representing the 308 communities together with people representing

the communities of Gent, Genk, Antwerp, Leuven and Kortrijk.


of 215






No central register of permits

Legislative action to see that all

permitting authorities register

permits. Documents as well as

structured data;

Building a basic register centrally to

which data can be sent.

This issue is still being

discussed internally. In Progress

No publication of emission data in

LOSD format


Reuse of existing data sets, as

example European Environment

Agency;

Adding new vocabularies on

existing vocabularies.

The data is in Linked

Data format and

vocabularies were

created.

Done

No re-use of data gathered during

permission processes



permits;

Documents as well as structured

data building a basic register




No re-use of data from base

registries

Re-use of data from base registries;

Starting program to publish base

registries also via LOD technology.



No base registry build-up



permits;

Documents as well as structured

data building a basic register




No possibility of linking

permission data with emission

data Publishing data via LOD technology;



Agency;



The tools allow the

pilot create the link In Progress


permission data with

enforcement data




permission data with data of new

permissions for same exploitation



No possibility of benchmarking for

legal persons

Providing online analysis

possibilities for example

The tools and

vocabularies will allow In Progress


of 215

benchmarking the pilot overcome

this issue.

No possibility of data analysis Providing online analysis

possibilities

The stable version of

map visualisation and

interactive tables will

allow the analysis.

In Progress

No possibility of data-driven

policy-making

Providing online analysis

possibilities

The stable version of

map visualisation and

interactive tables will

allow the analysis and

data-driven policy-

making.

In Progress

Data in general is locked in silo’s




Agency;





Data is not very accessible

Publishing data as LOD under open

data licence:

Providing analysis and reporting

tools;

Starting LOD program to publish

data in LOD way and as part of

process of the organisations. Not as

extra work.




of 215



Data Quality


Discoverability and

Usability

1- App has no links to the original data sets;


3- App does not allow users to download the data sets, including partial download of

selected variables;



Granularity 1- Both aggregate and raw level of data from internal

Intelligibility 1- There is no supporting documentation.

Trustworthiness 1- The data sets collected are well-known by developers, including how it was processed;


Linkable to other

data

(5 Stars LOD)

1- The data sets are available in different formats to different audiences, from PDF, XLS to

CSV. RDF was created by OGI developers but not yet in stable version for end-users;

2- The RDF is not yet properly linked to other external data to provide context.

15 Open Data

Principles

1- Data has issues to be considered complete;

2- Data is primary, but aggregate level is also published;

3- There are issues about timely availability;

4- Data is not 100% accessible to the widest range of users (XLS, CSV and RDF)

5- Data is discriminatory because is not freely available;






of 215


Data Quality


Discoverability

and Usability

The Flemish application allows

users to preview and interact with

the data published in web

interactive maps and tables.

There is still no link to external data sources. Only

internal data sources have been linked. It is not ye

It is possible to download the data or part of this data

(slicing the data sets) as users’ desire.

Users can preview and interact with the data

(visualization functionality)

Granularity There is both raw and aggregate

data level in the application.

There is no aggregate level of data (e.g. average of

pollution). Data is in unit level (data from each

company, per type of pollution).

Intelligibility Data was linked and data cube


Supporting document exist, however, is found

separately.

Trustworthiness

The data is reliable, reflecting the

reality. Developers know how it is

processed and is consistent.

There is still room to improve the reliability.

Linkable to

other data

(5 Stars LOD)

The data sets are in RDF format. Achieved the max level of 5 stars LOD.

15 Open Data

Principles

The third year achieved 8 of 9

principles that can be followed in

OGI pilot.

Pilot is not following the design with public input

(users can select format and part of data to

download) and only has unit level.


of 215

4.1.5.2.1 Datasets

The data sets are already in RDF format and with metadata. The link to access data seta are at:

https://app.minbox.com/files/_fDIOp2

Figure 38 - Metadata of Belgium Pilot Data Set

https://app.minbox.com/files/_fDIOp2


of 215


To evaluate the application …


2. User Acceptance;




of 215


Figure 39 – LOSD Familiarity – Flemish Pilot

The results from Flemish pilot were not majorly positive. All of the 8 people surveyed (100%) pointed

that they were not familiar with Linked Open Statistical Data (LOSD). `Surprisingly, half of them knew

the datasets presented in the Flemish Application.


of 215


Figure 40- User Acceptance – Flemish Pilot

Major part of the dimensions surveyed resulted into positive reactions. The three mains issues found

by users were the high level of technical knowledge to use app. All users pointed that and makes

sense, considering the level of complexity of this specific datasets subject and also the functionalities

the applications has. Another problem was the experience of crashes, what could lead to the 13%

complaining about the application functionalities.

The other dimensions had positive reactions such as functions working properly, the interaction

application, usefulness, easy to use, integrated, app well designed, and, being not difficult to use.

Some of the answers are contradictory and deeper qualitative research must be done to discover

what happened specifically in this pilot.


of 215


Figure 41 – Datasets Quality – Flemish Pilot

Figure 41 – Datasets Quality – Flemish Pilot shows that the main issue of Flemish Pilot App is the

completeness of data. Surveyed people believe the data is not complete, and probably influenced

13% of them point out that all the required datasets to monitor the Flemish environment were not

presented (or incomplete). It makes sense, because Flemish pilot is still under development and new

datasets will be linked in the future to let the application even more complete.

ON the other hand, it is clear the datasets provided had positive reaction. All of them believe that

the data is accessible, accurate and they reduced time to find the data. In summary, all of the 8

surveyed people think that the overall data quality is satisfactory.


of 215


Figure 42 – Perceived Outcomes – Flemish Pilot

The positive reactions from surveyed people were better interpretation of data (100%), increase of

efficiency (100%), time reduction to search for data (100%), increase of transparency (100%). 88% of

the users pointed that can make better decision-making using the Flemish application and 75% of

them think the app helps to achieve their goals. On the other hand, they demand more functions to

enable even more transparency (88% of the surveyed people).

In general, 88% of the users believe that the Flemish Pilot app is acceptable.


of 215

4.1.5.4 Screenshots of The Flemish Environment Agency Application

The next figure presents the screenshots of OGI ICT toolkit using data sets from the Pilot. The

platform is on a different address of other pilots:

Figure 43 – The Flemish Environment Agency Application


of 215

4.1.6 Pilot 6 – Marine Institute (Ireland)


The Marine Institute pilot, in Galway (Ireland), aims to support three business cases scenarios:

1. Marine renewable energy

2. Maritime search and rescue; and,

3. Maritime tourism and leisure.

The function of the pilot will be to take non-linked tabular Comma-Separated Values (CSV) data

hosted by the Marine Institute and convert it to Linked Data formats using Data Cubes. Using Linked

Data Resource Description Framework (RDF) the new structured data management will enhance the

value of the marine data asset for the scenario purposes by structuring and enriching the data with

vocabularies and semantic value to aid the requirements of the scenarios.

The building of a Linked Data Dashboard to include data visualisations such as charts, maps and

widgets of marine Linked Data will support decision making scenarios for actors making a renewable

energy investment decision, a search and rescue expert in rescue and recovery and a maritime

sports enthusiast in their maritime sport.

For deep description of Marine Institute and Irish pilot, please check report D1.1 and D4.1. Also, a

blog post at OGI Medium account was published:

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-marine-institute-

bde38cf89ac6 .


A number of significant barriers to co-creation of data-driven services in the Marine sector were

identified by stakeholders. These barriers can be divided into two higher level categories (Data

Infrastructure, and User Challenges), each of which contain a number of sub-categories. Examples of

barriers from these categories and sub-categories are provided below.

The initial options proposed by workshop participants opened the possibility space for the creative

thinking in the next phase of the workshop, which involved participants working with specific user

scenarios and generating key needs and requirements of stakeholders involved in co-creation of

services in the Marine sector.

These sceanrios involved hypothetical users including citizens, public administrators, data scientists,

engineers, and other stakeholders. The During this section of the workshop, participants highlighted

an extensive range of 1) Informaiton needs, and 2) Decision-making needs. Informaiton needs, refers

to types of information or data which are needed for co-creation in the Marine sector. In relation to

decision-making needs, participants were asked to generate a list of key tools, services, affordances

or other types of supports which would aid in decision-making.

The scenarios purposed to participants were:

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-marine-institute-bde38cf89ac6

https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-marine-institute-bde38cf89ac6


of 215

1. Engineer Ed is considering developing a wave energy converter and would like access to

real-time data on wind, waves and currents generated by multiple public organisations

through one web-based location in order to predict the performance of his device and to

monitor its safety once it is deployed in the environment. Ed needs to know when a high

energy wave event is happening in order to provide decision support to Technician Tony.

When Ed and Tony are in agreement that the environmental conditions are in danger of

causing damage to their ocean energy converter, they need to be able to shut it down in

order to protect it. In support of the data, it would be useful to Ed to have easy access to

supporting information (such as the Offshore Renewable Energy Development plan) from

the data access site. This is useful for Ed and Manager Mike when they are choosing

potential locations in which to deploy their ocean energy converters. Deputy David also uses

this supporting information in the portal when lobbying the minister to encourage central

government policy on marine renewable energy development.

2. Captain Cillian is a racing sailor looking to gain a competitive edge in his races on Galway

Bay. He’d like to be able to access tide predictions, wind data and current speeds in a format

which he can upload into his sailing computer so he can plan his races. On shore

Commodore Colm would like to receive the locations of each of the sailing vessels in the

races he organises so he can better plan safety operations associated with those races.

Public Administrator Patricia examines race activity data and service usage in order to plan

and inform policies in relation to boat racing activities.

3. Sergeant Steve is coordinating a search and rescue operation. He needs to know both the

current conditions in the waters around the coastline, where his teams can access the

shorefront and where an object which has been dropped into the water is likely to be since it

entered the sea. Garda Grainne is a member of Steve’s team. She should be able to return

information, such as geo-located photographs, to Steve so that he can be kept up to date of

the search team’s location and conditions. In addition to the public authorities, Volunteer

Val is a member of the public involved in searching the coastline. The volunteers should have

access to the same apps and much of the same data as the authorities, but some

information may not be available to the volunteers. Inspector Ian and Coastguard Chris

review the information collected by the app after each rescue to build up dataset which

allows them to develop local search and rescue policies.

4. Drone pilot David enjoys flying his device in coastal areas. He occasionally spots strange

patterns on the water surface and would like help in identifying what these are. He would

like to submit them to a web portal where an expert can tell him what he’s seeing. Biologist

Brenda logs in to the web portal to see if there is anything of interest to her. She sees one of

David’s photos from a flight earlier today. Brenda can use any identification she makes to

alert authorities of any potential harmful algal blooms. Other experts can confirm Brenda’s

identifications and engage in further analysis of other images uploaded on the system. These

expert identifications and alerts may be used alongside model outputs and processed

satellite images to inform policy on keeping open or closing aquaculture sites.

Participants generated a total of 40 barriers in relation to Data Infrastructure. The following sub-

categories emerged from the full Data Infrastructure set: Accessibility, Awareness and Engagement,


of 215

Connectivity, Data Management, Lack of integration and collaboration, Policy Issues, Quality Issues,

and Service and Resources Issues. Table below provides a sample of barriers in each sub-category.

Table 45 - Sample Data Infrastructure Barriers

Sub-category Barrier

Accessibility

Issues

Lack of intuitive service design, such that training is not required

Lack of open access data – can everyone who wants to access the data get to it?

Awareness and

Engagement

Issues

Lack of knowledge by experts as to what data is available

Inadequate citizen data management expertise in people when collecting data on

the environment including knowledge of controlled vocabularies, metadata, GIS,

data enrichment value etc.

Connectivity

Issues

Loss of data/connectivity during e.g. storms, which is often the time when the

client wants to collect data

Failure of adequate internal/ broadband availability nationwide

Data

Management

Issues

Lack of focus on which data to prioritize and assign investment and resources to;

Lack of data sustainability, when projects end is the data still managed?

Is it accessible?

Lack of

integration/coll

aboration

Lack of integrated data infrastructure access;

Unnecessary duplication of data by different agencies

Policy Issues Legislative barriers and lack of legislation;

Conflict between the multitude of European directives on the marine

environment (e.g. MSFD, WFD, Shellfish, Habitats, EMFF) in terms of a

harmonised data standard for information describing the marine environment.

Quality Issues Lack of data quality control;

Lots of gaps in data, no guarantee of quality.

Service and

Resource Issues

Real-time data dimension;

No marine alert system for stakeholders.

Participants generated a total of 45 barriers in relation to User Challenges. The following sub-

categories emerged from the full User Challenges set: Awareness and Engagement, Collaboration,

Education and Training, Finance, Motivation, Needs Analysis, Policy Issues, Quality Issues, Resistance

and Conflict, and Service and Resource Issues. Table below provides a sample of barriers in each sub-

category.


of 215

Table 46 - Sample User Challenges Barriers

Sub-category Barrier

Awareness and

Engagement

Issues

Lack of public awareness to the data systems available

Understanding value of data to society

Collaboration

Issues

Lack of opportunities for users to engage with data developers/

technologists (stakeholder involvement in development)

Tunnel vision e.g. focus on organizing an event not on surrounding issues

Education and

Training Issues

Lack of adequate user experience in the relevant technologies

Lack of adequate ongoing training

Finance Issues Lack of funding for the “processing” of the data

Lack of financial resources

Motivation

Issues

Lack of public drive to get government to change

Lack of public drive to get government to provide the financial resources

needed to ensure a robust system is in place

Needs Analysis

Issues

Going beyond mash-up is hard, need to identify services/ needs etc.

Lack of engagement with end users to develop targeted services

Policy Issues The silo effect, focus on the individual needs of a department, not looking

at the bigger picture

Lack of vision at a high level

Quality Issues • Lack of data quality standards

• Lack of quality control of data and data delivery

Resistance and

Conflict

Resistance to using anything other than the old tried and trusted methods

Conflict between the different stakeholder stalling progress

Service and

Resource Issues

Lack of will at state level to push/implement broadband network

Difficulty of creating a data collection method that is sustainable and

iterative

While these and other barriers highlighted many challenges, the expert group identified many

options which could help to overcome these barriers. In thinking about and generating options,

participants were asked to address three core stages of the co-creation process: Service Creation

(ideas), Service Engineering (requirements, design) and Service Management (delivery, evaluation).

Participants were asked to place their option beneath one or more of these headings, depending on

the nature of the option, and it’s relevance to overcoming barriers in one of more stages of co-

creation. Again, participants worked across the higher-level categories of Data Infrastructure and

User Challenges, which were divided into a set of sub-categories.

Participants generated 72 options for overcoming barriers to Data Infrastructure, 15 of which relate

to Service Creation, 32 of which relate to Service Engineering, and 25 of which relate Service

Management. In some cases, options were suggested by participants to be relevant to more than


of 215

one stage, and thus are recorded as such in this report. Options within each of these three stages of

co-creation were further divided into sub-categories. Tables below provide a sample of options

within each stage.

Table 47 - Service Creation - Sample options for overcoming barriers

Category Option

Collaboration • Encourage government agencies to work on a common data

sharing framework

• Build up groups when starting and create multi-agencies - get-

togethers and get inputs

Education and

Training

• Training for public servants in relation to data protection

• Open data training for government

Finance • Focus finance away from policy towards data as the key value in

service development

• A data sharing unit to identify cost savings and efficiency across the

public service

Policy • Enact data sharing and governance legislation

• Build a long term sustainability plan into project applications

Services and

Resources

• A government level data storage service

• Develop a platform with plasticity to change in the future as user

needs mature


of 215

Table 48 - Service Engineering - Sample options for overcoming barriers

Category Option

Accessibility • Formats - making datasets available - update data to new format

• A system that can be accessed by most appliances, smart phones, iPads,

radar chart plotters

Collaboration • Co-ordinated field of data sharing with authorities

• Key data “collectors” meet with a view to discuss and collaborate in future

to “standardize” data collection and management

Quality • Quality is a must so the intermediate and end user know the limits of the

data used to create products

• Use standardized date “flagging”/ annotation schemes (from e.g. UNESro) to

identify data quality

Services and Resources • A government level data storage service

• Develop a platform with plasticity to change in the future as user needs

mature

Tools • Gather weather information via weather stations and ODAS buoys

• Develop linked data transformation transformers and data mapping tools

Table 49 - Service Management - Options for overcoming barriers

Category Option

Collaboration • Access to servers/ hardware requires appropriate resources - collaborations

can aid this

• Build a system used by all (the team) entities e.g. a) input race data b) input

incidents data c) input road closure data

Education and Training • Citizen data management user guides and public sector domain data

collection training

• Training for public servants in relation to data protection

Policy • Ensure that each public body has a data management plan which is

adequately resourced

• Identification of and promotion of best practice service standards to break

data silos

Services and Resources • A central document is collated on existing resources/databases/services like

a systematic review as a start point

• Investigation of new IoT methods for data collection (e.g. sig fox, LoRa) to

minimise downtime

Tools • Back up sensors that can be used in the event of a breakdown at critical

times

• Create software to identify bad data and broken sensors

Participants generated 48 options for overcoming barriers to User Challenges, 20 of which relate to

Service Creation, 16 of which relate to Service Engineering, and 12 of which relate Service

Management. Again, in some cases, options were suggested by participants to be relevant to more

than one stage, and thus are recorded as such in the tables. Options within each of these three

stages of co-creation were further divided into sub-categories. Tables below provide a sample of

options within each sub-category.


of 215

Table 50 - Service Creation - Options for overcoming barriers

Category Option

Awareness and

Engagement

• Technology road shows on the coder dojo model known as the

technology dojo for public

• Promote data availability/ service availability, create awareness

Education and Training • Provide ongoing training for all users of public data

• Educate government employees and public about what directives

are about and why they should care

Needs Analysis • Find out what exactly user needs + who are the potential users

• Avoid unnecessary cost by knowing or being able to access info on

what data is already available

Table 51 - Service Engineering - Options for overcoming barriers

Category Option

Awareness and

Engagement

• Technology road shows on the coder dojo model known as the

technology dojo for public

• Promote data availability/ service availability, create awareness

Education and Training • Induction courses when new employees start

• Publish guidelines on best practices for methodologies. Maybe

using non-technical language where possible

Needs Analysis • Run service design workshops including end-users

• Develop services/ apps from identified and documented user

requirements (not because we can!)

Policy • E.U. directive to encourage government drive

User-Friendly • Creates simple mobile apps

• Provide the user with an engaging, simple, and robust experience

to move them on to their services


of 215

Table 52 - Service Management - Options for overcoming barriers

Category Option

Awareness and

Engagement

• Require a dissemination plan for all public projects

• Identify forums for promotion of data services which will reach a

wide audience

Collaboration • Bringing businesses in as partners

Education and

Training

• Training for people (possibly older people on using apps)

Infrastructure • Real time services fail - is there backup

Policy • State support to a single data structure

Finally, participants were asked to generate a set of co-creation contributions, again based on the

four scenarios. Participants were asked to list specific contributions that each actor or co-creator in

the scenarios could make, thereby highlighting opportunities for potential co-creation by means of a

pooling of resources, expertise, and ideas.

In total, participants generated 77 co-creation contributions, across the following categories: Data

provision, Expertise, Interpretation and Analysis, Modeling and Simulation, Networking and

Promotion, Policy Input, Software Input, and Visual Data. The co-creation contributions suggested by

partiicpants here, addressed all aspects of the co-creation stages discussed during the earlier options

stage. That is, the contributions in table below, which provides a sample of contributions from each

category, as well as reference to the specific scenario from which they were derived, contains

contributions which address various aspects of service creation (ideas), service engineering

(requirements, design), and service management (delivery, evaluation).


of 215

Figure 44 - Percentage breakdown of co-creation contributions

Participants also engaged in discussion around the broader challenges of effective and fruitful co-

creation. It was highlighted that such complex scenarios, with multiple actors and contributions, may

require an incremental approach, beginning with a core and concrete starting point – a foundation

for additional actors and contributions to build upon.

This discussion then addressed the importance of validating and refining during development of co-

created services, with regular input from multiple actors and users. Given the core focus on co-

creation, this aspect was considered to be of great importance to participants.

Another point raised during the course of the discussion, consistent with some of the barriers

highlighted earlier, was the importance of accessibility and user-friendliness. It was suggested that

the platform on which the service is build must be easily accessible by citizens, for both contribution

of data and feedback.

Finally, and perhaps most crucially, the question of how to provide an environment that allows all of

the actors in a complex system, each with varying interests and levels of skills, and each providing

various kinds of information, to work together to co-create a service. This, it was agreed, is

something which will require deep thinking and consideration at each stage of the process.

D4.2 Evaluation Results – First Round

of 215

Table 53 - Sample co-creation contributions

Co-creation contribution So that we can Category Scenario

Collect samples of water and shellfish to send to the

lab, upload private data, temperature etc.

Create maps and graphs Data Provision Drone Pilot David

Helicopter data and activity, vessel data & activity,

coastguard tool

Provide search support Data Provision Sergeant Steve

A specification for data service standards Develop an infrastructure to share data from different

creators in the app

Expertise Drone Pilot David

Support on information and data enrichment services Provide a service in which data are well described for

easy integration

Expertise Drone Pilot David

Expertise in identifying what is in the photographs Inform authorities of what is taking place in the

environment

Interpretation

and analysis

Drone Pilot David

Harmful algal bloom (HAB) analysis expertise and HABs

tool expertise

Conduct HAB's status evaluation, deliver status data to

aqua-culture and public, map reporting

Interpretation

and analysis

Drone Pilot David

Model outputs, specifically physical/ biochemistry

outputs

Provide video/cartoons of projections of water

movements, properties, and characteristics,

nowcast/forecast

Modeling and

simulation

Drone Pilot David

A numerical simulation of oil spills/ harmful algal

blooms

Provide evaluations of where the photographed things

will travel to

Modeling and

simulation

Drone Pilot David

Information in easy to understand language Provide a blog post with an overview of an

environmental issue using statistics

Networking and

Promotion

Drone Pilot David


of 215

Activity, networking, promotion, lobbying Promote business and product, aid decision on potential

sites

Networking and

Promotion

Engineer Ed

Data analysis, expertise in policy Inform policy, access success, to inform data collection Policy input Captain Cillian

Influence, push for policy objectives, activity, provide

evidence

Consider other actors/interest groups, push for certain

policy objectives

Policy input Engineer Ed

A digital framework for awarding badges Develop a mechanism to provide general users with an

incentive to continue using the service

Software Input Drone Pilot David

A User Interface design Design an intuitive user interface hiding the underlying

service technology from the user

Software Input Drone Pilot David

Take photos/ describe visual photos Provide an alert system for risk levels (green, orange,

red)

Visual data Drone Pilot David

Use of my craft to assist with pictures/ videos as

required

Provide a data file of images over a defined period Visual data Drone Pilot David

This project has been funded with the support of the H2020 Programme of the European Union Copyright by the OpenGovIntelligence Consortium

.






Accessibility

Issues

Lack of intuitive service design, such that

training is not required

Lack of open access data – can everyone who

wants to access the data get to it?

Data is being linked and will

be published for end-users

access.

Done in the

3rd Year.

Connectivity

Issues

Loss of data/connectivity during e.g. storms,

which is often the time when the client wants

to collect data

Failure of adequate internal/ broadband

availability nationwide

OGI ICT Toolkit can enable the

interoperability between data

and systems reducing this

issue.

Done in the

Second Year.

Quality Issues Lack of data quality control;

Lots of gaps in data, no guarantee of quality.

Data is linked following the 5

stars LOD.

Done in the

3rd Year.

Policy Issues

Legislative barriers and lack of

legislation;

Conflict between the multitude of

European directives on the marine

environment (e.g. MSFD, WFD,

Shellfish, Habitats, EMFF) in terms of a

harmonised data standard for

information describing the marine

environment.

This still have been internally

discussed.

In Progress

and out of

control of

OGI.

Education and

Training

Issues

Lack of adequate user experience in the

relevant technologies

Lack of adequate ongoing training

User experience was

improved and outcomes

section shows positive

reaction from end-users.

A video explaining how to use

was created and linked in the

app.

Done in the

3rd Year.

Finance Issues Lack of funding for the “processing” of

the data

Lack of financial resources

There is still no sustainable

business model for the LOSD.

Exploitation report D5.8

details the OGI plan for this

issue.

In Progress

and out of

control of OGI

Pilot.


of 215


As co-creation step for co-implementation, the Marine Institute started to employ online

vocabularies to describe its online data within existing ISO 19139 metadata content and aim to

deploy these same vocabularies in Linked Open Statistical Data (LOSD) within the

OpenGovIntelligence (OGI) Irish pilots. Vocabularies provide a list of standardised and consistent

terms over a broad range of disciplines (e.g. oceanography, geography and geology) that are of

relevance to describing things clearly for key stakeholders who wish to exploit marine LOSD through

the Irish pilots. Using standardised sets of terms (otherwise known as "controlled vocabularies") in

metadata and to label LOSD solves the problem of ambiguities associated withdata markup and

enables records to be interpreted as Linked Data by machines. This opens up data sets to a whole

world of possibilities for computer aided manipulation, distribution and long term reuse.

An example of how the Irish OGI pilots may benefit from the use of controlled vocabularies is in the

using of values taken from different data sets. For instance, one data set may have a column labelled

"Temperature of the water column" and another might have "water temperature" or even

"temperature". To the human eye, the similarity is obvious but a computer would not be able to

interpret these as the same thing unless all the possible options were hard coded into its software.

If data are marked up with the same terms, this problem is resolved. Sea temperature is an

important variable in understanding the real-time conditions of the marine environment which

affect scenarios such as search and rescue because the temperature of the water column indicates

potential survival time rates for an individual requiring immediate rescue from emergency services.

The Climate and Forecast vocabulary (P07) has a mailing list which can be used to request new

terms and new terms are added, modified and deprecated on a near monthly basis. The British

Oceanographic Data Centre facilities Co-Creation from the network of SeaDataNet distributed

marine partner consortium from more than 35 countries in managing this vocabulary.


of 215



Data Quality


Discoverability and

Usability



3- App allows users to download the data sets, including partial download of selected

variables;

4- App has multiple format download option;


Granularity 1- There has aggregate level of data and unite level.

Intelligibility 1- Supporting documentation can be found at the same time as the data.

Trustworthiness

1- The data sets collected are well-known by developers and technical partners has

confidence on how it was processed;


Linkable to other

data

(5 Stars LOD)

1- The data sets are available in different formats to different audiences, from XLS to CSV.

RDF was created by OGI developers;

2- The RDF is not yet properly linked to other external data to provide context. Once the

data is converted to RDF and stored in the Trafford Data Store it will be 'linkable'.

15 Open Data

Principles


2- Data is primary;

3- There is not issues about timely availability;

4- Data is accessible to the widest range of users (XLS, CSV and RDF)

5- Data is non-discriminatory because is freely available;


7- Data is permanent;





of 215


Data Quality

Attributes

Status


Discoverability

and Usability App in development

App developed, but not

completely ready. App allows

for access to data but doesn’t

allow downloading

Updated, there is download

and slicing option for end-users

of the Irish App.

Granularity

Diverse data sets of high

quality with non-aggregate

data

Diverse data sets of high

quality with non-aggregate

data

Updated, Four datasets were

combined resulting in 2

datasets linked in aggregate

format and unit level.

Intelligibility Good intelligibility with

documentation available.

Good intelligibility with

documentation available.

Linking of datasets could be

better

Updated, existence of

documentation found at the

same time as the data.

Trustworthiness

Data Quality is good, from

trusted source. Data validity

can be controlled.

Current scenario with some

improvements. Increase data

quality with data cubes linking

data sources however without

proper context from external

sources.

Updated. Technical partners

and end-users pointed via

survey the acceptance of

datasets quality.

Linkable to

other data

(5 Stars LOD)

Not following all the 5 stars

LOD principles, no own data

cubes

Not following all the 5 Stars

LOD principles.

Updated. Following the 5 stars

LOD.

15 Open Data

Principles

Following 8 of 9 OD

Principles



Updated. Following 8 of 9 OD

principles that matches OGI

needs.


of 215


Data Quality


Discoverability

and Usability

The discoverability and usability

was improved as the app was

developed further.

Data can be downloaded and combined with new

visual functionalities improved level of acceptance by

end-users (check outcomes section).

Granularity Same high level of granularity Both unit and aggregate level are presented in the

new functionalities delivered.

Intelligibility Data contains supporting

documentation

There is documentation, however, it is close to the

data.

Trustworthiness Data is still very trustworthy and

can be validated easily

Data is reliable and trusted. Outcomes sections

shows high acceptance from users and technical

partners.

Linkable to

other data

(5 Stars LOD)

Data is still available in non-

proprietary format

Since data is linked, achieved the max level of this

dimension.

15 Open Data

Principles

Data is in accordance with 8 of 15

Open Data principles There is no public input.

4.1.6.3.1 Data Sets

The list of data sets is presented below:

Irish Weather Buoy Network

Irish Wave Buoy Network

Irish National Tide Gauge Network

East Atlantic SWAN Wave Model

The data sets are already in RDF (LOSD format). The link to access data seta are at:

https://drive.google.com/open?id=0B0_fFuauifo0R2RLR0M1dThLMWs

https://drive.google.com/open?id=0B0_fFuauifo0R2RLR0M1dThLMWs


of 215


To evaluate the application …


2. User Acceptance;




of 215


Figure 45 – LOSD Familiarity – Marine Institute Pilot

From the survey results, we can see that majority of Marine Institute App respondents stated that

they familiar with open data and open data portals, 83% and 73% respectively. Similar positive

reaction were found for familiarity with the app datasets. However, 65% of the respondents claimed

that they are not familiar with LOSD.

This result brings opportunity for the Irish pilot partner to introduce the LOSD to wider audience by

using the app so in the future more people can be aware about LOSD, thus create more benefits to

the society as seen in the perceived outcomes section.


of 215


Figure 46 – User Acceptance – Marine Institute Pilot

From the results, we should highlight that in the major part of the categories tested in this user

acceptance section, majority of the respondents gave positive reactions in terms of technical

functionality: the app provides a proper technical functionality (85%), functionality of the app

working properly (92%), clear and understandable interaction with the app (100%), functionality

well-integrated (94%), and the app provides adequate design (96%). However, 100% of the

respondents also stated that they experienced with a crash when using the app.

Regarding the easiness, 96% of the respondents stated that the app is easy to use; with less figure,

73% of the respondents stated the app is not difficult to operate. 96% of the respondents claimed

that they have sufficient skills to use the app even though 90% of the respondents said that it

requires high level technical knowledge.

From these section, improvement should be made in terms of fixing the crash, and make it less

technical to facilitate the use of the app for broader group of users.


of 215


Figure 47 – Datasets Quality – Marine Institute Pilot

In terms of data quality, majority of the respondents claimed that the app already provided data

accessibility (98%), data accuracy (100%), all required datasets from co-creation process (87%), and

less time to find the data (95%). However, only 56% of the respondents stated that the data

provided by the app are incomplete, this can be an opportunity for the improvement.

Last, 90% of the respondents satisfy with the level of data quality provided by the Marine Institute

app.


of 215


Figure 48 – Perceived Outcomes – Marine Institute Pilot

For the last section of the questionnaire, we were asking about the perceived outcomes of using the

app. For all of the potential outcomes, more than 90% of the respondents gave the positive reaction.

In addition, 90% of the respondents also asked more functions in the app to create even better

transparency.


of 215

4.1.6.5 Screenshots of Marine Institute Application

The next figure presents the screenshot of Marine Institute Application:

Figure 49 - Average Bar chart of Irish National Tide Gauge Network data set


of 215

4.1.7 Overall Results From Pilots’ Evaluation

In this section, we combine all the responses from 6 pilots, to see users’ perception about

applications provided by OGI project.

4.1.7.1 Overall Linked Open Statistical Data (LOSD) Familiarity

Figure 50 – Overall LOSD Familiarity

First, regarding familiarity of the technology, more people are familiar with open data, open data

portal and the datasets provided by all of the apps. On the other hand, less people are familiar with

LOSD.


of 215

4.1.7.2 Overall User Acceptance

Figure 51 – Overall User Acceptance

From the overall user acceptance evaluation, there are two things should be highlighted: first, even

though the apps already provided good functions and design, majority of respondents still

experienced with a crash when using the apps; second, there is a contradiction regarding the

easiness to use the apps, 94% of the respondents claimed that the apps are easy to use but only 36%

stated that the apps are not difficult to use. Each pilot partners should take a deeper qualitative

investigation to identify solution for both issues.


of 215

4.1.7.3 Overall Datasets Quality

Figure 52 –Overall Datasets Quality

From the results in data quality section, the main issues in data quality provided by the OGI apps are

regarding the data completeness and meeting the required datasets from co-creation process.


of 215

4.1.7.4 Overall Perceived Outcomes

Figure 53 – Overall Perceived Outcomes

Majority of the respondents claimed that the OGI apps are useful and bring benefits, related to

better interpretation of the data, help them to make a better decision, help them in achieving their

goals, more efficient (time and cost), and increase transparency.

However, majority of the respondents asked to add more functions to create more transparency.

In terms of acceptance, 90% of the respondents claimed that they accept the app and 95% of the

respondents stated that their colleagues are also accept the apps.


of 215

4.2 OGI ICT Toolkit Evaluation Results

This section shows the OGI ICT Toolkit Evaluation Results, including the evaluation of internal and

external evaluation. The internal evaluation contains:

1. OGI Data Quality;

2. OGI Architecture; and,

3. System Quality of Application.

The external evaluation includes the OGI ICT toolkit evaluation made external users and experts.

Figure 54 – Internal and External Evaluation Processes


of 215

4.2.1 Internal Evaluation


The LOSD OGI Data Quality evaluation was made within the technical partners in charge to design

and develop the six OGI pilots applications. The questionnaire is presented in the section 7.2,

containing 29 questions using a Likert scale of 5 level, from strongly disagree to strongly agree. This

survey evaluated the follow dimensions of LOSD OGI Data Quality:

1. Discoverability and Usability;

2. Granularity Section;

3. Intelligibility Section;

4. Trustworthiness Section;

5. Linkable to other data Section; and,

6. Open Data Principles Section.


of 215

4.2.1.1.1 Discoverability and Usability

Figure 55 – Discoverability and Usability – Overall of All Pilots

From this discoverability and usability section, all of the pilots provided a pre-release version of pilot

app. 4 pilots have already provided links to data sources and searchable list of data sources. 5 pilots

have already provided option to download the data, option to download selected parts of the data

and allowed users to preview and interact with data. Last, 3 pilots have already provided option to

download in multiple formats.


of 215

4.2.1.1.2 Granularity Section

Figure 56 – Granularity Section – Overall of All Pilots

The pilots’ apps provided different level of data granularity. All of the pilots provided at least

aggregated data. It means that the data is in statistical format but only provided as percentage or

average. On the other hand, some of the pilots also could provide the raw data (in individual unit

level), for example hourly data as unit level and daily data in average level.


of 215

4.2.1.1.3 Intelligibility Section

Figure 57 – Intelligibility Section – Overall of All Pilots

Documentation let the users to understand in detail the datasets, providing context of the data and

how the data could be used. Most the datasets apps be equipped with the documentation.

However, in some datasets the documentation is provided separately, for example, by an external

provider (the owner of the data).

On the other hand, 1 pilot included the documentation in the visualization, and another pilot is still

without documentation.


of 215

4.2.1.1.4 Trustworthiness Section

Figure 58 – Trustworthiness Section – Overall of All Pilots

Regarding the trustworthiness, there is 1 pilot which unsure how their data are collected, and 1

pilot which unsure how the data is processed.

In other categories all pilots stated that they already used trusted sources, and provided reliable and

consistency with external sources data.


of 215

4.2.1.1.5 Linkable to other data Section

Figure 59 –Linkable to other Datasets Section – Overall of All Pilots

Related to the 5-star deployment scheme for LOD, 2 pilots already reached level 5 while the rest on

level 3.


of 215

4.2.1.1.6 Open Data Principles Section

Figure 60 – Open Data Principles Section – Overall of All Pilots

Regarding open data principles, all of the pilots claimed that they already provide complete data,

primarily data, wide range of users data, non-discriminatory data, license-free data, permanent data,

safe to open data and designed with public input. While there is one pilot have not provided timely

data.


of 215

4.2.1.2 OGI Architecture Evaluation

The LOSD OGI Architecture evaluation was made within the 7 technical partners in charge to design

and develop the six OGI pilots. The questionnaire is presented in the section 7.2, containing 29

questions using a Likert scale of 5 level, from strongly disagree to strongly agree. This survey

evaluated the follow dimensions of LOSD OGI Reference Architecture:

1. User Behaviour;

2. Perceived Usefulness;

3. Perceived Ease of Use;

4. Computer Self-efficacy, Perceptions of External Control, Computer Playfulness, Computer

Anxiety, Perceived Enjoyment, Objective Usability; and,

5. Result Demonstrability, Behavioural Intention, Use, Present Satisfaction, Future Desire.


of 215

4.2.1.2.1 Use Behaviour

Figure 61 – Use Behaviour of LOSD OGI Architecture

From Figure 61, 6 out of 7 partners claimed that they referred to the LOSD OGI architecture when

designed the pilot apps.


of 215

4.2.1.2.2 Perceived Usefulness

Figure 62 – Perceived Usefulness of LOSD OGI Architecture

In the 5 perceived usefulness parameters, 100% partners perceived that the LOSD OGI architecture

is useful in designing the pilot apps: increase productivity, help them to reach their objectives, help

them to be aware of data quality issue, help them to design and build flexible architecture and

provide better requirements.

5 out of 7 partners claimed that the LOSD OGI Architecture is useful to deal with data quality.


of 215

4.2.1.2.3 Perceived Ease of Use

Figure 63 – Perceived Ease of Use of LOSD OGI Architecture

All of partners stated that the LOSD OGI architecture is clear and understandable, as well as help

them in designing linked-open data system. 6 out of 7 partners claimed that the LOSD OGI

architecture helps them to design LOD system with less cost.


of 215

4.2.1.2.4 Computer Self-efficacy, Perceptions of External Control, Computer Playfulness, Computer Anxiety, Perceived Enjoyment, Objective Usability

Figure 64 – Computer Self-efficacy, Perceptions of External Control, Computer Playfulness, Computer Anxiety, Perceived Enjoyment, Objective Usability of LOSD OGI Architecture

This section addresses partners’ computer skills. All of the partners stated that they are fluent using

computers, do not have a problem working with any software, confident to design and develop their

own architecture, enjoy using a reference architecture and feel that the reference architecture

accelerates their work in achieving the goals.

In addition, 6 out of 7 partners stated that they have fun using computer, while 5 out of 7 claimed

they are confident in working with computers.


of 215

4.2.1.2.5 Subjective Norm, Voluntariness, Image, Job Relevance, Output Quality

Figure 65 –Subjective Norm, Voluntariness, Image, Job Relevance, Output Quality of LOSD OGI Architecture

In terms of independency, all of the partners stated that their partners suggest them to use OGI

architecture, it is their choice to use OGI architecture to build the pilot app, and they have freedom

to choose to use or not to use the OGI architecture; moreover, there 5 partners stated that their

partners will think highly if they decide to use OGI architecture.

In terms of job relevance, 6 partners agreed that the use of OGI architecture is pertinent to their job-

related tasks, 6 partners stated that data quality has improved after using the OGI architecture and

all partners agreed that flexibility has improved after using OGI architecture.


of 215

4.2.1.2.6 Result Demonstrability, Behavioral Intention, Use, Present Satisfaction, Future Desire

Figure 66 –Result Demonstrability, Behavioural Intention, Use, Present Satisfaction, Future Desire of LOSD OGI Architecture

In the last part, all of the partners claimed that they can explain the benefits of using the OGI

architecture, would like to use the OGI architecture, and satisfy with current OGI architecture. In

addition, only 3 out of 7 partners welcome a different OGI architecture.


of 215

4.2.2 External Evaluation

Two external evaluations of OGI ICT Toolkit were made in the second year of project. The first in

Belgium, 16th March of 2017 and the second in 4th of September of 2017. The results are presented

below in the sections .

4.2.2.1 NTTS Workshop

The workshop was held in Brussels, in 16th March. The official web site for this conference is at:

https://ec.europa.eu/eurostat/cros/content/ntts-2017_en .

The workshop is described in the blog post at OGI Medium:

https://medium.com/opengovintelligence/hands-on-workshop-on-linked-open-statistical-data-lod-

7b9a8066f3e0 .

Figure 67 – NTTS Hands-on OGI ICT Toolkit Workshop

4.2.2.1.1 User Information

Most of the users who evaluated the OGI Toolkit in this workshop were working for some part of the

government or state-owned companies. Ages varied between 26 and 55, with only two respondents

being older and none being younger than this age group. They were highly educated and skilled

workers, who are however not very familiar with linked statistical data cubes. They seemingly were a

good representative set of end-users typical for most pilot applications, with a nice mix of managers,

statisticians, data analysts and administrative staff.

4.2.2.1.2 OGI Toolkit Evaluation

https://ec.europa.eu/eurostat/cros/content/ntts-2017_en

https://medium.com/opengovintelligence/hands-on-workshop-on-linked-open-statistical-data-lod-7b9a8066f3e0

https://medium.com/opengovintelligence/hands-on-workshop-on-linked-open-statistical-data-lod-7b9a8066f3e0


of 215

The respondents were not familiar with any competitors to the OGI Toolkit at first sight. The one

exception judged the OGI Toolkit to be better than its competitors, but did not name any specific

competitive software or toolkit. Some attractive functionalities named were its combination of

regional data, its user friendliness and the ability to save (a portion of) useful data. The interface was

judged to be slightly unattractive though. What stands out here is that most respondents however

chose not to name any attractive or unattractive feature. The basic skills necessary to use the OGI

Toolkit were said to be database browsing, domain knowledge, SparQL skills and statistics. The

respondents named different kinds of settings to teach the OGI Toolkit, with a slight majority naming

MOOCs, but both text-based education in the form of a book or a classroom course were also

named.

In terms of the results of the OGI Toolkit, most respondents were overall fairly neutral. They

however thought the OGI Toolkit design was not yet adequate and were not happy with how clear

and understandable their interaction with the OGI Toolkit was. Respondents were unconvinced that

the OGI Toolkit led to either an increase in transparency, better output or that it was useful in their

job, although they also were not convinced that it did not have value in these areas whatsoever. The

same can be said for the acceptance of the OGI Toolkit by senior management, whether they would

encourage it to their colleagues or if it would be accepted by their teams.

The respondents overall feelings on the Toolkit can best be summarized as being neutral: they didn’t

feel the OGI Toolkit was amazing and would be a significant aid in their job, but they also didn’t

come out and say it was that bad. A more refined product that has more data could very much swing

the opinion more positive.

4.2.2.2 EGOV/IFIP Hands-on Workshop

The IFIP/EGOV is an academic conference in the area of electronic government. The conference was

held in Saint Petersburg, 4th of September. The official web site of this conference is at:

http://faculty.washington.edu/jscholl/ifip.working.group.8.5/website/conference/#/ .

4.2.2.2.1 User Information

The respondents in this workshop were for the overwhelming majority working in an academic

setting. They were aged 26-55, except for one respondent who was older. They were very highly

educated, with a majority of respondents attaining a Ph.D. and everyone else completing a Master’s

Degree. As a result about half were already familiar with linked statistical data cubes. They seem to

overall be mostly academic researchers and professors who will be working with the OGI Toolkit in

an academic setting. As such, they could both be end-users and co-developers of the OGI Toolkit and

its applications. All respondents were male.

4.2.2.2.2 OGI Toolkit Evaluation

The respondents named some competitors of the OGI Toolkit: R and IBM’s Watson were named

among others. The respondents who named competitors judged the OGI Toolkit to be better than its

http://faculty.washington.edu/jscholl/ifip.working.group.8.5/website/conference/#/


of 215

competitors at its assigned purpose. Some attractive functionalities of the OGI Toolkit were judged

to be its visualization, its usability and the relative ease of using statistical data. The user interface

was found to be one of the features that need improvement, as well as a lack of SPARQL queries and

descriptive statistics. The basic skills necessary to use the OGI Toolkit as identified by the

respondents were basic and advanced statistics, basic software engineering and database

management. MOOCs were overwhelmingly named as the most effective way to teach the OGI

Toolkit.

The results of the OGI Toolkit were positively judged by the respondents. They found for example

that the OGI Toolkit design was adequate. They found the OGI interactions to be clear and

understandable and thought that the OGI Toolkit led both to an increase of transparency and better

output and found it to be useful in their job. They did think that the OGI Toolkit requires a high

amount of knowledge, and despite their educated background were overall fairly neutral in judging

if they themselves were skilled enough to use the Toolkit, although there was some variation

amongst the respondents. They were also neutral when asked if the OGI Toolkit would be accepted

by their teams, despite them answering they would encourage the use of the OGI Toolkit to their

colleagues.


of 215

4.2.2.3 Overall External Impression of OGI ICT Toolkit

The two workshops had very different respondents, which allows for a good overview and

comparison between these two sets. The existing functionalities of the OGI ICT Toolkit were overall

rated quite highly, although more functionalities need to be included. The user interface and SPARQL

queries need to be improved. Respondents need to be skilled in statistics and software engineering

to make full use of the API. The NTTS workshop participants were less content with the results from

the API than the participants of the EGOV/IFIP workshop and they disagreed very much on this

section, leading credence to the hypothesis that the OGI ICT Toolkit was still an academic project not

quite ready for non-academic end-users at the time of the workshop. Better functions and more

user friendliness could improve the transparency effects and overall output of the OGI ICT Toolkit.


of 215

5 Conclusions

The evaluation has four areas: Co-creation, ICT Toolkit, Acceptance of ICT Toolkit and Outcomes. In

this report the evaluation methodology has been updated based on the insights gained from the first

and second year.

The co-creation was evaluated using the same framework used during first and second year, to

compare the evolution and changes made on pilots influenced by the participation of pilot’s

stakeholders. The reason is that most of the co-creation aspects already took place in the first couple

of years.

The ICT toolkit evaluation has been updated to include questions in the questionnaire to evaluate

the new tools which developed after the evaluation in the second year. Finally, the part about data

quality is new, given its importance.

The 6 pilots were evaluated by employing a user – questionnaire. Most users of 219 people surveyed

found the pilots benefits include the creation of transparency, reduction the administrative burden

by more efficient search of information and visualizing the results at a glance. We considered

positive reactions strongly agree, agree and neutral answers. Negative reactions were disagree and

strongly disagree. Below the descriptive statistics of Transparency, Decision-Making, Efficiency and

Better Data Interpretation:

The overall acceptance of the pilots’ apps was 90%;

92% pointed out an increase of Transparency after accessing the Pilots’ Apps;

The Pilots’ apps helped to 94% of surveyed people have better decision-making;

93% of people answered there is a better interpretation of Data;

For 92% of respondents, Pilots’ app reduced the time spent searching information;

94% of end-users identified a cost reduction when use OGI pilots’ apps; and,

There is an increase of efficiency to 91% of OGI Pilots’ users.

Developing the ICT-toolkits more time than expected, as this is a new field in which there are hardly

any examples. The development of the building block encountered the questions of which level of

granularity would be fine as this requires a trade-off between the flexiblity given to the developer


of 215

and ease-of-use of manipulating data. User-friendliness and having right granularity is a trade-off

and developers have different requirements in this regard.

Co-creation remains challenging in government. Co-creation is bottom-up, whereas government

traditionally takes a more top-down approach by developing pilots and providing the services.

Realizing co-creation is not only a technological problem, but rather a cultural problem.

The end-users regard the pilots as successful as shown in the figure below. The pilots show that data

quality is a key aspect as garbage in is garbage out. Realizing high-data quality took often more time

than expected, whereas the OGI ICT-toolkit enables to develop the pilots within a short time frame.

They pilots help end-users to find information faster and in a more efficient way resulting in

transparency and better decision-making quality.


of 215

6 References

Dodds, Leigh (2016). Exploring Open Data Quality. Open Data Institute Website

(https://theodi.org/blog/exploring-open-data-quality). Published in: 26/10/2016. Accessed in:

26/01/2018.

Barth, Michael (2012). API Evaluation: An Overview of API Evaluation Techniques. Michael Bart

Blog A portfolio Blog about Software Development (https://www.michaelbarth.net/). Paper

available at: https://www.michaelbarth.net/files/publications/api-evaluation.pdf. Published in 2012.

Accessed in: 28/01/2018.

OPENGOVDATA (2018). Eight principles of open government data. Available at: The annotated 8

Principles of Open Government Data Website (https://opengovdata.org/) from a meeting in 7th-8th

December 2007. Accessed in: 26/01/2018.

Bizer, C., Heath, T., & Berners-Lee, T. (2009). Linked data-the story so far. Semantic services,

interoperability and web applications: emerging concepts, 205-227.

Frank, M., & Walker, J. (2016). User Centred Methods for Measuring the Value of Open Data. The

Journal of Community Informatics, 12(2).

Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-

7.

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability

scale. Intl. Journal of Human–Computer Interaction, 24(6), 574-594.

Joyce, E., & Kraut, R. E. (2006). Predicting continued participation in newsgroups. Journal of

Computer-Mediated Communication, 11(3), 723-747.

Glasow, P. A. (2005). Fundamentals of survey research methodology. Retrieved January, 18, 2013.

Levy, Y. and T. J. Ellis (2006). A systems approach to conduct an effective literature review in

support of information systems research. Informing Science: International Journal of an Emerging

Transdiscipline 9(1): 181-212.

Delbecq, Andre L., Andrew H. Van de Ven, and David H. Gustafson. Group techniques for program

planning: A guide to nominal group and Delphi processes. Scott Foresman, 1975.

20510:2010, ISO/IEC. (2010). Systems and Software Engineering—Systems and Software Product

Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models. Geneva,

International Organization for Standardization.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of

information technology.MIS quarterly: 319-340.

Davis, F. D., R. P. Bagozzi and P. R. Warshaw (1989). User acceptance of computer technology: a

comparison of two theoretical models. Management science 35(8): 982-1003.

https://theodi.org/blog/exploring-open-data-quality

https://www.michaelbarth.net/

https://www.michaelbarth.net/files/publications/api-evaluation.pdf

https://opengovdata.org/


of 215

DeLone, W. H. and E. R. McLean (1992). Information systems success: The quest for the dependent

variable. Information systems research 3(1): 60-95.

Delone, W. H. and E. R. McLean (2003). The DeLone and McLean model of information systems

success: a ten-year update. Journal of management information systems 19(4): 9-30.

Venkatesh, V. and H. Bala (2008). Technology acceptance model 3 and a research agenda on

interventions. Decision sciences 39(2): 273-315.

Venkatesh, V. and F. D. Davis (2000). A theoretical extension of the technology acceptance model:

Four longitudinal field studies. Management science 46(2): 186-204.

Venkatesh, V., M. G. Morris, G. B. Davis and F. D. Davis (2003). User acceptance of information

technology: Toward a unified view. MIS quarterly: 425-478.

Ward, R. (2013). The application of technology acceptance and diffusion of innovation models in

healthcare informatics. Health Policy and Technology 2(4): 222-228.

Ward, S. J. (2014). The magical concept of transparency. Ethics for digital journalists: Emerging best

practices: 45-58.

Bovens, M. (2007). Analysing and assessing accountability: a conceptual framework. European law

journal 13(4): 447-468.

Coglianese, C. (2009). The transparency president? The Obama administration and open

government. Governance 22(4): 529-544.

Frank, M. and A. A. Oztoprak (2015). Concepts of Transparency: Open Data in UK Local Authorities.

Proceedings of the International Conference for E-Democracy and Open Government 2015

(CeDEM15).

Peixoto, T. (2013). The Uncertain Relationship between Open Data and Accountability: A Response

to Yu and Robinson's' The New Ambiguity of Open Government'.

Matheus, R. and M. Janssen (2013). Transparency of civil society websites: towards a model for

evaluation websites transparency. Proceedings of the 7th International Conference on Theory and

Practice of Electronic Governance, ACM.

Matheus, R. and M. Janssen (2015). Transparency Dimensions of Big and Open Linked Data. Open

and Big Data Management and Innovation. M. Janssen, M. Mäntymäki, J. Hidders et al., Springer

International Publishing. 9373: 236-246.

Matheus, R., M. M. Ribeiro and J. C. Vaz (2012). New perspectives for electronic government in

Brazil: The adoption of open government data in national and subnational governments of Brazil.

Proceedings of the 6th International Conference on Theory and Practice of Electronic Governance,

ACM.

Fox, J. (2007). The uncertain relationship between transparency and accountability. Development

in Practice 17(4-5): 663-671.


of 215

Gallo, C., M. Giove, J. Millard, R. Kare and V. Thaarup (2014). Study on eGovernment and the

Reduction of Administrative Burden. Luxembourg, Luxembourg. doi 10: 42896.

Law, E. L.-C., V. Roto, M. Hassenzahl, A. P. Vermeeren and J. Kort (2009). Understanding, scoping

and defining user experience: a survey approach. Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems, ACM.

Hassenzahl, M. and N. Tractinsky (2006). User experience-a research agenda. Behaviour &

information technology 25(2): 91-97.

Forlizzi, J. and K. Battarbee (2004). Understanding experience in interactive systems. Proceedings of

the 5th conference on Designing interactive systems: processes, practices, methods, and techniques,

ACM.

Garrett, J. J. (2010). Elements of user experience, the: user-centered design for the web and

beyond, Pearson Education.

Jokela, T., N. Iivari, J. Matero and M. Karukka (2003). The standard of user-centered design and the

standard definition of usability: analyzing ISO 13407 against ISO 9241-11. Proceedings of the Latin

American conference on Human-computer interaction, ACM.

Travis, D. (2011). ISO 13407 is dead. Long live ISO 9241-210. London, UK: Userfocus Ltd., June 6.

Albert, W. and T. Tullis (2013). Measuring the user experience: collecting, analyzing, and presenting

usability metrics, Newnes.


of 215

7 Annexes

7.1 End-User Questionnaires

7.1.1 Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece) End-User Questionnaire

OGI Evaluation Questionnaire MAREG Pilot App End-users – 2018

This questionnaire is aimed to be filled by End-Users who have used the applications developed within the pilots of H2020 OpenGovIntelligence (OGI) project. When answering the questions, the application developed using the OGI Toolkit, named here as MAREG Pilot App, should be taken into account. The following topics will be asked:

General background information; Data sets; and, Resulting Application (App developed using the OGI toolkit).

More Information about MARE Pilot APP (http://wapps.islab.uom.gr/CubeVisualizer/vehicles/)

A. General Background Information 1. How familiar are you with open data applications? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar

2. What is your role or position in terms of Vehicles Cube App? A: ____________________________________________________________________________

B. Acceptance of the Vehicles Cube App 3. The Vehicles Cube App provides all the functionalities I am interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

4. All functions in the Vehicles Cube App work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

5. The Vehicles Cube App recovers well from crashes ( ) The Vehicles Cube App didn’t crash ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

6. The Vehicles Cube App helps me to achieve my goals ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

7. My interaction with the Vehicles Cube App was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

8. The Vehicles Cube App is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

9. My interaction with the Vehicles Cube App is clear and understandable




of 215

( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

10. The Vehicles Cube App design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

11. I have sufficient skills to use the Vehicles Cube App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

12. The Vehicles Cube App does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

13. The Vehicles Cube App will be accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

14. Using the Vehicles Cube App can be hard for the average user ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

15. The Vehicles Cube App is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Do you want to explain your answers or comment on anything not covered about acceptance? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

C. Vehicles Cube App Data Sets

16. The data in the Vehicles Cube App are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

17. The data in the Vehicles Cube App is inadequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

18. The data in the Vehicles Cube App are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

19. A lot of time is needed to find the information I am looking for regarding vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

20. I am able to find all the information I'm looking for when using the Vehicles Cube App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Do you want to explain your answers or comment on anything not covered about data sets? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________


of 215

D. Vehicles Cube App results 21. The Vehicles Cube App has a clear visualization ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

22. The Vehicles Cube App helps to do my job about government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

23. The Vehicles Cube App results in an increase of transparency of government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

24. The Vehicles Cube App is too complex to acquire knowledge into government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

25. The Vehicles Cube App helps me make better decisions about government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

26. The Vehicles Cube App helps my understanding of the government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

27. The Vehicles Cube App will increase efficiency of government vehicles management ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

28. The Vehicles Cube App reduces time spent looking for information about government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

29. The Vehicles Cube App reduces the costs to search and decide about government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

30. More functions in the Vehicles Cube App are needed to create transparency of government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

Thank you for your time to answer the questions!


of 215

7.1.2 Pilot 2 – Enterprise Lithuania (Lithuania) End-User Questionnaire

OGI Evaluation Questionnaire

Lithuania Enterprise App End-users – 2018

This questionnaire is aimed to be filled by End-Users who have used the applications developed within the pilots of H2020 OpenGovIntelligence (OGI) project. When answering the questions, the application developed using the OGI Toolkit, named here as Lithuania Enterprise App (LE App), should be taken into account. The following topics will be asked:

General background information; Data sets; and, Resulting Application (App developed using the OGI toolkit).Ń

A text explaining more information about Lithuania Enterprise App (http://lithuania.opengovintelligence.eu/).

A. General Background Information 1. Please select your age: ( ) Less than 21 ( ) 21 – 40 ( ) 41 – 60 ( ) Over 60 ( ) Don’t want to share

2. How familiar are you with open data applications? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar

3. What is your role using the LE App? ( ) Investors ( ) Citizens ( ) Civil servants ( ) Entrepreneurs ( ) Other ______________________________________________________________________

B. Acceptance of the Lithuania Enterprise Pilot App 4. The LE App provides all the functionalities I am interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

5. All functions in the LE App work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

6. The LE App is stable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

7. The LE App helps me to achieve my goals ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

8. My interaction with the LE App was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

http://lithuania.opengovintelligence.eu/


of 215

9. The LE App is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

10. My interaction with the LE App is clear and understandable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

11. The LE App design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

12. I have sufficient skills to use the LE App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

13. The LE App does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

14. The LE App will be accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

15. Using the LE App can be hard for the average citizen ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

16. The LE App is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree


C. Lithuania Enterprise App Data Sets 17. The data in the LE App are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

18. The data in the LE App is incomplete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

19. The data in the LE App are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree


of 215

20. A lot of time is needed to find the right data in the LE App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

21. I am able to find all the data I am looking for when using the LE App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree


D. E. Lithuania Enterprise Pilot Apps RESULTS

22. The LE App has a clear visualization ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

23. The LE App helps to create insights of Lithuanian business environment ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

24. The LE App results in an increase of transparency of Lithuanian business environment ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

25. The LE App is too complex to gain insight of Lithuanian business environment ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

26. The LE App helps me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

27. The LE App helps my understanding of the subject matter ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

28. The LE App will increase efficiency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

29. The LE App reduces time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

30. The LE App reduces the costs to search and decide ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

31. More functions in the LE App are needed to create transparency of Lithuanian business environment


of 215





of 215

7.1.3 Pilot 3 – Tallinn Real Estate (Estonia) End-User Questionnaire

OGI Evaluation Questionnaire Tallinn Real Estate App End-users – 2017

This questionnaire is aimed to be filled by End-Users who have used the applications developed within the pilots of H2020 OpenGovIntelligence (OGI) project. When answering the questions, the application developed using the OGI Toolkit, named here as Tallinn Real Estate Pilot App, should be taken into account. The following topics will be asked:


More information about Tallinn Real Estate

(https://rnd-tut.shinyapps.io/Estonian_Pilot/)

A. General Background Information 1. What is your gender? ( ) Female ( ) Male ( ) Don’t want to share

2. Please select your age: ( ) Less than 21 ( ) 21 – 40 ( ) 41 – 60 ( ) Over 60 ( ) Don’t want to share


4. What is your role using the Tallinn Real Estate App? ( ) Students ( ) Foreign employees ( ) Tourists ( ) Real Estate Agents ( ) Other ______________

B. Acceptance of the Tallinn Real Estate Pilot App 5. The Tallinn Real Estate App provides all the functionalities I’m interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

6. All functions in the Tallinn Real Estate App work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

7. The Tallinn Real Estate App recovers well from crashes ( ) The Tallinn Real Estate App didn’t crash ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

8. The Tallinn Real Estate App helps me to achieve my goals ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

9. My interaction with the Tallinn Real Estate App user was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

10. The Tallinn Real Estate App is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree



of 215

11. My interaction with the Tallinn Real Estate App is clear and understandable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

12. The Tallinn Real Estate App design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

13. I have sufficient skills to use the Tallinn Real Estate App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

14. The Tallinn Real Estate App does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

15. The Tallinn Real Estate App will be accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

16. Using the Tallinn Real Estate App can be hard for the average citizen ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

17. The Tallinn Real Estate App is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree


C. Tallinn Real Estate Pilot App Data Sets 18. The data in the Tallinn Real Estate App are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

19. The data in the Tallinn Real Estate App is incomplete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

20. The data in the Tallinn Real Estate App are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

21. A lot of time is needed to find the right data in the Tallinn Real Estate App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

22. I am able to find all the data I'm looking for when using the Tallinn Real Estate App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree



of 215

D. Tallinn Real Estate Pilot App RESULTS 23. The Tallinn Real Estate App has a clear visualization ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

24. The Tallinn Real Estate App helps to create insights ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

25. The Tallinn Real Estate App results in an increase of transparency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

26. The Tallinn Real Estate App is too complex to gain insight ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

27. The Tallinn Real Estate App helps me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

28. The Tallinn Real Estate App helps my understanding of the subject matter ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

29. The Tallinn Real Estate App will increase efficiency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

30. The Tallinn Real Estate App reduces time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

31. The Tallinn Real Estate App reduces the costs to search and decide ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

32. More functions in the Tallinn Real Estate App are needed to create transparency of Real Estate information in Tallinn ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree




of 215

7.1.4 Pilot 4 – Trafford Council Worklessness (England) End-User Questionnaire

OGI Evaluation Questionnaire Trafford Worklessness Pilot Apps End-users – 2018

This questionnaire is aimed at end-users who have used the applications developed by the pilots involved in the H2020 OpenGovIntelligence (OGI) project. When answering the questions, the applications developed using the OGI Toolkit, named here as the Trafford Worklessness Pilot Apps, should be taken into account. A separate usability survey will be used for the individual applications. The apps are designed to enable Jobcentre Plus Managers and Local Authority leads to identify areas of need and inform service delivery related to worklessness. The following topics will be asked:

Your General background information; A Technical Evaluation of the Pilot developed using the OGI toolkit The Datasets available in the Pilot; and, Pilots Outcomes (e.g. transparency, administrative burden reduction, cost

reduction).

Trafford Worklessness Pilot Apps - http://www.trafforddatalab.io/opengovintelligence/

A. General Background Information 1. Which organisation do you work for? ( ) Department for Work and Pensions ( ) Borough / City Council ( ) Other: ___________________________________ 2. Which Local Authority do you work within? ( ) Bolton ( ) Bury ( ) Manchester ( ) Oldham ( ) Rochdale ( ) Salford ( ) Stockport ( ) Tameside ( ) Trafford ( ) Wigan ( ) Other: ___________________________________ 3. How long have you been working for your employer? ( ) Less than 6 months ( ) 6 to 12 months ( ) 1 to 3 years ( ) 3 to 6 years ( ) 6 years or more 4. What is your employee status? ( ) Managerial ( ) Non-managerial 5. How comfortable do you feel using the Internet? ( ) Very uncomfortable ( ) Somewhat uncomfortable ( ) Neither comfortable nor uncomfortable ( ) Somewhat comfortable ( ) Very comfortable 6. How familiar are you with open data? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar 7. How familiar are you with Linked Open Statistical Data (LOSD)?

http://www.trafforddatalab.io/opengovintelligence/


of 215

( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar 8. How familiar are you with downloading data from open data applications like nomis and Stat-

Xplore? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar

B. Trafford Worklessness Pilot Apps Datasets 9. The data in the Trafford Worklessness Pilot Apps are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 10. The Trafford Worklessness Pilot Apps contain all of the datasets that I require ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 11. The data in the Trafford Worklessness Pilot Apps are incomplete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 12. The data in the Trafford Worklessness Pilot Apps are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 13. A lot of time is needed to find the right data in the Trafford Worklessness Pilot Apps ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 14. I am satisfied with the datasets quality provided by the Trafford Worklessness Pilot Apps ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree Do you want to explain your answers or comment on anything not covered about data sets? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

C. Acceptance of the Trafford Worklessness Pilot Apps 15. The Trafford Worklessness Pilot Apps provide all the functionalities I am interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 16. All functions in the Trafford Worklessness Pilot Apps work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 17. I experienced a crash while using the Trafford Worklessness Pilot Apps ( ) The Trafford Worklessness Pilot Apps didn’t crash ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 18. The Trafford Worklessness Pilot Apps help me in my work ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 19. My interaction with the Trafford Worklessness Pilot Apps is satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 20. The Trafford Worklessness Pilot Apps are useful to me

https://www.nomisweb.co.uk/

https://stat-xplore.dwp.gov.uk/

https://stat-xplore.dwp.gov.uk/


of 215

( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 21. The Trafford Worklessness Pilot Apps are easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 22. The Trafford Worklessness Pilot Apps have an adequate design ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 23. I have sufficient skills to use the Trafford Worklessness Pilot Apps ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 24. The Trafford Worklessness Pilot Apps do not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 25. The Trafford Worklessness Pilot Apps are accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 26. The Trafford Worklessness Pilot Apps are unnecessarily complex ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 27. I found the various functions in the Trafford Worklessness Pilot Apps are well integrated ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 28. I want to use the Trafford Worklessness Pilot Apps frequently ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree Do you want to explain your answers or comment on anything not covered about acceptance? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

D. Trafford Worklessness Pilot Apps Outcomes Evaluation 29. The visualization provided by the Trafford Worklessness Pilot Apps makes better

interpretation of data ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

30. The Trafford Worklessness Pilot Apps help me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

31. The Trafford Worklessness Pilot Apps increase efficiency of my work ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

32. The Trafford Worklessness Pilot Apps reduce time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

33. The Trafford Worklessness Pilot Apps reduce costs to find information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree


of 215

34. More functions in the Trafford Worklessness Pilot Apps are needed to create transparency to

support decision-making ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

What functionality would you like to see added to the applications to add transparency? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Finally, do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

Many thanks for taking the time to complete this questionnaire!

If you have any questions about the survey please email [email protected]

mailto:[email protected]


of 215

7.1.5 Pilot 5 – The Flemish Environment Agency (Belgium) End-User Questionnaire

OGI Evaluation Questionnaire Flemish Government App End-users – 2017

This questionnaire is aimed to be filled by End-Users who have used the applications developed within the pilots of H2020 OpenGovIntelligence (OGI) project. When answering the questions, the application developed using the OGI Toolkit, named here as Flemish Government Pilot App, should be taken into account. The following topics will be asked:


B. General Background Information 1. Please select your age: ( ) Less than 21 ( ) 21 – 40 ( ) 41 – 60 ( ) Over 60 ( ) Don’t want to share


D. Acceptance of the Flemish Government Pilot App 3. The Flemish Government App provides all the functionalities I’m interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

4. All functions in the Flemish Government App work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

5. The Flemish Government App recovers well from crashes ( ) The Flemish Government App didn’t crash ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

6. The Flemish Government App helps me to achieve my goals ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

7. My interaction with the Flemish Government App was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

8. The Flemish Government App is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

9. My interaction with the Flemish Government App is clear and understandable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

10. The Flemish Government App design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

11. I have sufficient skills to use the Flemish Government App


of 215


12. The Flemish Government App does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

13. The Flemish Government App will be accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

14. Using the Flemish Government App can be hard for the average citizen ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

15. The Flemish Government App is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Do you want to explain your answers or comment on anything not covered about acceptance? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

C. Flemish Government Pilot App Data Sets

16. The data in the Flemish Government App are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

17. The data in the Flemish Government App is incomplete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

18. The data in the Flemish Government App are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

19. A lot of time is needed to find the right data in the Flemish Government App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

20. I am able to find all the data I'm looking for when using the Flemish Government App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Do you want to explain your answers or comment on anything not covered about data sets? ______________________________________________________________________________ ______________________________________________________________________________


of 215

______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

D. Marine Institute Pilot App results 21. The Flemish Government App has a clear visualization ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

22. The Flemish Government App helps to create insights about air pollution in Flanders ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

23. The Flemish Government App results in an increase of transparency of air pollution in Flanders ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

24. The Flemish Government App is too complex to gain insight ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

25. The Flemish Government App helps me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

26. The Flemish Government App helps my understanding of the subject matter ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

27. The Flemish Government App will increase efficiency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

28. The Flemish Government App reduces time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

29. The Flemish Government App reduces the costs to search and decide ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

30. More functions in the Flemish Government App are needed to create transparency of air pollution in Flanders ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________


of 215

______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________



of 215

7.1.6 Pilot 6 – Marine Institute (Ireland) End-User Questionnaire

OGI Evaluation Questionnaire Marine Institute Pilot End-users – 2018

This questionnaire is aimed to be filled by End-Users who have used the Pilot applications developed within the H2020 OpenGovIntelligence (OGI) project. When answering the questions, the Pilot application developed using the OGI Toolkit, named here as Marine Institute Pilot, should be taken into account. The following topics will be asked:

General background information; Data sets; and, Resulting Pilot (Pilot developed using the OGI toolkit).

A. General Background Information 1. What is your organisational role using the Marine Institute Pilot? ( ) Search and rescue professional ( ) Search and rescue volunteer ( ) Marine Energy Developer ( ) Marine Energy Consultant ( ) Planning Consultant ( ) Planner ( ) Private Citizen ( ) Public Servant ( ) Civil Servant ( ) Tourism Agent ( ) Tourism business owner ( ) Sailor ( ) Volunteer ( ) IT Professional Other: ________________________________________________

2. For what purpose might you use the Marine Institute Pilot Dashboard? ( ) Recreational ( ) Work ( ) Both

3. How familiar are you with data dashboards? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar

4. How familiar are you with marine data availability suitable to your needs? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar

5. How familiar are you with Linked Open Statistical Data? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar

B. Marine Institute Pilot Evaluation 6. The Marine Institute Pilot provides all the functionalities I require ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

7. All functions in the Marine Institute Pilot work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

8. The Marine Institute Pilot system is stable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

9. The Marine Institute Pilot supports my requirements ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree


of 215

10. My interaction with the Marine Institute Pilot was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

11. The Marine Institute Pilot is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

12. My interaction with the Marine Institute Pilot is clear and understandable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

13. The Marine Institute Pilot design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

14. I have sufficient skills to use the Marine Institute Pilot ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

15. The Marine Institute Pilot does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

16. I will share the The Marine Institute Pilot with colleagues ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

17. Using the Marine Institute Pilot can be hard for the average citizen ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

18. The Marine Institute Pilot is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Have you any comments and or recommendations regarding the evaluation of this pilot? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

C. Marine Institute Pilot Data 19. The data in the Marine Institute Pilot are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

20. The data in the Marine Institute Pilot is complete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

21. The data in the Marine Institute Pilot are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

22. The data in the Marine Institute Pilot are consistent ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

23. A lot of time is needed to find the right data in the Marine Institute Pilot


of 215


24. I am able to find all the data I'm looking for when using the Marine Institute Pilot ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Are there any particular datasets you are aware of that would prove useful in the pilot evaluated? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

E. Marine Institute Pilot 25. The Marine Institute Pilot has a clear visualisation ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

26. The Marine Institute Pilot helps to support emergency response operations ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

27. The Marine Institute Pilot helps to support marine renewable energy investment decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

28. The Marine Institute Pilot helps to support planning a sailing event ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

29. The Marine Institute Pilot helps me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

30. The Marine Institute Pilot helps my understanding of ocean conditions relevant to emergency

response, marine energy and marine event planning ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

31. The Marine Institute Pilot will increase efficiency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

32. The Marine Institute Pilot reduces time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

33. The Marine Institute Pilot reduces the costs to search and decide ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

34. More functions in the Marine Institute Pilot are needed to create transparency to support

marine decision-making ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree

Do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________


of 215

______________________________________________________________________________

Thank you for your time in completing this evaluation questionnaire.


of 215

7.2 Data Quality Questionnaire

Table 58 – Pilots’ Data Quality Questionnaire

Data Quality

Section # Question Answer

A- Discoverability

and Usability

1 There is a (pre-release) version of an

application already:

( ) Yes ( ) No

If No, please skip to section B- Granularity.

2 The application contains links to the data

sources ( ) Yes ( ) No

3 The application contains a searchable list

of data sources that are used ( ) Yes ( ) No

4 The application contains an option to

download the data ( ) Yes ( ) No


download the data in multiple formats ( ) Yes ( ) No


download selected parts of the data ( ) Yes ( ) No

7 The application allows the user to

preview and interact with the data ( ) Yes ( ) No

B- Granularity

8 Your data includes only aggregated data

(e.g. national averages)? ( ) Yes ( ) No

9

Your data includes individual unit level

data, but without generic class data

(data on a small, transactional level)

( ) Yes ( ) No

10

Your data includes generic class data

(The data also includes information

about different subsets within)

( ) Yes ( ) No

C- Intelligibility

11 Is there existing supporting

documentation for the data you used?

( ) Supporting documentation exists, but has to be

found separately

( ) Supporting documentation can be found at the

same time as the data (e.g. right next to the link for

the data)

( ) Supporting documentation can be immediately

accessed from within the data set, but it is not

context sensitive

( ) Supporting documentation can be immediately

accessed from within the data set, and is context

sensitive.

If there was no supporting

documentation, how did you deal with

these datasets?

Open question.

D-

Trustworthiness

12 I know how the data I used was collected ( ) Strongly Agree

( ) Agree

( ) Neutral

13 I know how the data I used was

processed

14 The data I used was published by a


of 215

trusted source ( ) Disagree

( ) Strongly Disagree 15 The data I used is realistic

16 The data I used is consistent with

external sources

E- Linkable to

other data

(5 Stars LOD)

17 The data I used was available in a

structured format (e.g. xls) ( ) Strongly Agree

( ) Agree

( ) Neutral

( ) Disagree

( ) Strongly Disagree

18 The data I used was available in a non-

proprietary open format (e.g. csv)

19 The data I used contained URLs to

denote external things (e.g. RDF)

20 The data I used linked to other data to

provide context

F- 15 Open Data

Principles

21

The data is complete (all data that is not

subject to valid privacy, security or

privilege reasons is opened)

( ) Strongly Agree

( ) Agree

( ) Neutral

( ) Disagree

( ) Strongly Disagree

22 The data is primary (collected at the

source, not in aggregate form)

23

The data is made available as quickly as

necessary to preserve value for the end-

users

24

The data is accessible (for the widest

range of users; for the widest range of

purposes)

25 The data is non-discriminatory (available

to anyone, no registration needed)

26 The data is license-free (not subject to

copyright concerns etc.)

27 The data is permanent (made available

at a stable location indefinitely)

28 The data is safe to open (doesn’t contain

executable content)

29

The data is designed with public input

(the user decides how he can access the

data)


of 215

7.3 OGI Architecture Questionnaire

OGI Linked Open Statistical Data (LOSD) Architecture Questionnaire Evaluation – Developer Questionnaire – Third Year 2018/2019

This questionnaires is aimed to be filled by Developers at various aspects of the applications developed within the toolkit provided by H2020 OpenGovIntelligence (OGI) project. When answering the questions the application developed using the OGI Linked Open Statistical Data (LOSD) Architecture should be taken into account. The following topics will be asked:

User Behaviour; Perceived Usefulness; and, OGI Architecture characteristics aiming to improve Data Quality and Systems Flexibility.

A. User Behaviour 1. I referred to the LOSD OGI Reference Architecture to design my own architecture.

1 2 3 4 5 Almost Never [ ] [ ] [ ] [ ] [ ] Almost Always

B. Perceived Usefulness

2. Using the LOSD OGI Reference Architecture increases my architecture building work related productivity and effectiveness.

1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree

3. The LOSD OGI Reference Architecture helps me to attain my objectives with the data.


4. The LOSD OGI Reference Architecture is useful to deal with data quality issue in my work.


5. The LOSD OGI Reference Architecture helps me to become aware of data quality issues.


6. The LOSD OGI Reference Architecture helps me to design and build a flexible architecture.


7. The LOSD OGI Reference Architecture helps me to elicit better requirements.



of 215

C. Perceived Ease of Use 8. I find the LOSD OGI Reference Architecture clear and understandable.


9. I find that the LOSD OGI Reference Architecture helped me to accelerate architecting

linked open data system. 1 2 3 4 5

Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree

10. I find the LOSD OGI Reference Architecture helped me to architect linked open data system with less cost (resources).


D. Computer Self-efficacy, Perceptions of External Control, Computer Playfulness, Computer

Anxiety, Perceived Enjoyment, Objective Usability 11. I am fluent in the use of a computer.


12. I can figure out almost any software program (application) with a minimum of effort.


13. I am confident in my ability to design and develop my own architecture using the

reference architecture. 1 2 3 4 5


14. I enjoy to be creative and have fun when using computers. 1 2 3 4 5


15. I get dysfunctional nervous when working with a computer. 1 2 3 4 5


16. It feels enjoyable using the reference architecture. 1 2 3 4 5


17. The use of the reference architecture accelerates me attaining my data objectives. 1 2 3 4 5



of 215

E. Subjective Norm, Voluntariness, Image, Job Relevance, Output Quality 18. My business partners think I should use the LOSD OGI Reference Architecture.


19. It is my choice whether I use the LOSD OGI Reference Architecture to build my own

architecture. 1 2 3 4 5


20. I am given the freedom to choose whether or not I use the LOSD OGI Reference Architecture.


21. My business friends will think highly of me if I use the LOSD OGI Reference Architecture.


22. The use of the LOSD OGI Reference Architecture is pertinent to my job-related tasks.


23. I consider data quality is improved after using the LOSD OGI Reference Architecture.


24. I consider flexibility is improved from using the LOSD OGI Reference Architecture.


F. Result Demonstrability, Behavioural Intention, Use, Present Satisfaction, Future Desire

25. I believe I would have no problem explaining to someone else the benefits of using the LOSD OGI Reference Architecture.


26. I intend to make good use of the LOSD OGI Reference Architecture.


27. On average, what is the percentage do you spend on using/referring the LOSD OGI

Reference Architecture during architecting?. 1 2 3 4 5


28. I am satisfied with the present LOSD OGI Reference Architecture.


of 215


29. I would welcome a different LOSD OGI Reference Architecture.
