WP7 MULTI DOMAINS

WP7 Multi domains

WP7 Multi domains

1. Population

2. Tourism/border crossing

3. Agriculture

• Regional statistical office in Poznań

• Regional statistical office in

Bydgoszcz

Population

• Regional statistical office in Rzeszów

• Department of Social

Research

Tourism/

border crossing

• Department of Agriculture

• Regional statistical office in Olsztyn

AGRICULTURE

Country leaders of methodology

WP7 TEAM

Anna Nowicka Leader cooperation

Weronika Jaźwińska

PARTNERS

Janusz Dygaszewicz Project Manager of Polish work

John Sheridan Piet Daas Nigel Swier

may enrich statistical output in domains:

Aim of WP7 is to find out how

a combination of:

Big Data sources

administrative data

statistical data

WP7 - Future perspectives

Suggest pilots and domains with successful implementation potential for further elaboration

in the second wave of pilots in 2018

WP 7 Time frame

February 2016

February 2017 Maj 2018

FPA (Framework Partnership Agreement) 2 YEARS + 5 MONTHS (JAN. 2016 – MAY. 2018)

SGA-1 SGA-2

Milestones and deliverables(SGA-1)

Milestone 1.

Progress and technical report of

internal WP-meeting;

by M4

Milestone 2.

List of availability

big data sources in

the domain(s);

by M8

Milestone 3. Recommendation for using two or three

big data sources in the domain(s); by

M13

DELIVERABLE

THE PARTIAL REPORT FOR EACH DOMAIN CONTAINING BASIC INFORMATION ON:

The data access (with legal and privacy aspects)

The data quality issues

The methodology (focus also on combining data)

The technical aspects

by M14

Milestones and deliverables(SGA-2)

Milestone 3. Combining data

analyze;

Milestone 4. List of pilots and domains with successful

implementation potential for further elaboration in the

second wave of pilots in 2018;

Deliverable

THE GENERAL REPORT FOR DOMAINS including:

The data access (with legal and privacy aspects)

The data quality issues

The methodology (focus also on combining data)

The technical aspects

WP 7 Work plan SGA-1

Data access

Data feasibility

Data combination(SGA-2)

Summary plus future perspectives (SGA-2)

WP 7 Work plan SGA-1 TASK FROM TO

Planning and organization 2016-02-01 2016-03-31

Task 1. Data availability/Data inventory 2016-03-01 2016-09-30

1. Identify Big Data sources taking into account sustainability and availability in

several countries. 2016-03-01 2016-07-15

1.1. Establishing an inventory of these sources by: 2016-03-01 2016-06-08

1.1.1. Brainstorming - a review of potential sources, including the

2015 UNSD Survey on the Use of Big Data for Official Statistics 2016-03-01 2016-05-19

1.1.2. Preparation of a questionnaire with questions about the

sources used by the project participants 2016-04-29 2016-05-30

DONE

1.1.3. Sending the questionnaire to participants 2016-05-30 2016-05-31

1.1.4. Gathering answers and preparation for analysis 2016-05-31 2016-06-08

MILESTONE 1: Progress and technical report of internal WP-meeting 2016-05-24 2016-05-31

1.2. Assessment of the possibility of using sources for Big Data analysis in

the domains of population, tourism/border crossings, agriculture 2016-06-01 2016-06-24 STARTED

Face to face meeting Tallinn 2016-06-13 2016-06-15

WP6&WP7 internal meeting in Warsaw 2016-06-28 2016-06-30

1.3. Build the list of potential sources 2016-06-24 2016-07-15

2. Identify which results or new products from the source-oriented pilots may

contribute to these domains 2016-07-15 2016-08-26

2.1. Match the sources from the list of potential sources to each domain 2016-07-15 2016-07-22

2.2. Preliminary analysis of possibility for using sources to each domain

(legal aspects, availability, methodology, IT, quality) 2016-07-22 2016-08-12

2.3. Build the list of exploitable sources for each domain 2016-08-12 2016-08-26

3. Describe the added value of delivered linkage between these sources to

current statistics 2016-08-26 2016-09-30

3.1. Analyze the list of exploitable sources for each domain 2016-08-26 2016-09-09

3.2. Prepare the map of linkages between Big Data sources (e.g which

aspect of one data source can be used in several domains) 2016-09-09 2016-09-20

3.3. Describe the added value for each domain 2016-09-20 2016-09-30

MILESTONE 2: List of available Big Data sources in the domain(s) 2016-09-30 2016-09-30

WP 7 Work plan SGA-1 TASK FROM TO

Task 2. Data feasibility 2016-10-01 2017-02-28

1. Carry out explorative analyses in order to apply two or three Big Data sources in the

domain of population, tourism / border crossings or agriculture 2016-10-01 2016-12-30

1.1. Selection the most value big data sources for each domain (evaluation of the

legal aspects, availability, methodology, IT, quality) 2016-10-01 2016-10-31

1.2. Analying results. 2016-11-02 2016-11-30

Face to face meeting Brussels 2016-11-17 2016-11-18

1.3. Preliminary assessment of the usefulness - developing the assessment

factors. 2016-12-01 2016-12-30

2. Selection and recommendation of two or three big data sources for using in the domain

of population, tourism / border crossings, agriculture. 2017-01-02 2017-02-28

2.1. Preparing the SWOT analysis (positive and negative factors of using several

sources) 2017-01-02 2017-01-16

2.2. Recommendation of the most important and useful sources. 2017-01-16 2017-01-31

MILESTONE 3: Recommendation for using two or three Big Data sources in the

domain(s) 2017-01-31 2017-01-31

SGA-1 deliverables - report for each domain 2017-02-01 2017-02-28

Submission to the Review Board 2017-02-01 2017-02-01

Document analysis by the Review Board 2017-02-01 2017-02-15

Overview of comments and document analysis 2017-02-15 2017-02-28

Send the document to the Coordinator 2017-02-28 2017-02-28

Face to face meeting Sofia + workshop 2017-02-16 2017-02-16

Multi domains internal meeting

• Work package 7 has only one face to face internal meeting during all SGA-1 duration and therefore the meeting should be planed in the most suitable time. Under the Agreement, it is required that it will take place in the second quarter of 2016.

• For this reason, it was established that it is not necessary to organize a kick-off meeting now. At the end of June, it will be more efficient to build first list of big data sources for each domain. Sufficient means of cooperation between the participants of the package will be mails exchange and a meeting on Webex.

• Common WP6 & WP7 face to face meeting will take place on 28-30 of June in Warsaw. • FIRST DAY – common

• SECOND DAY – each WP separetly

• THIRD DAY – common

Stage: End of agreeing detailed agenda

Multi domains internal meeting

• All WP7 participants was represented by the following experts:

• The meeting was carried out via Webex. It was lead by Anna Nowicka in cooperation with Weronika Jaźwińska from Central Statistical Office in Poland.

John Sheridan IE √

Nigel Swier UK √

Piet Daas NL √

Anna Nowicka (Report)

Weronika Jaźwińska (Report)

PL

PL √

√

Multi domains internal meeting took place on 24th of May via Webex

Multi domains internal meeting

The meeting was carried out according to agenda: Nr Issues

1 Opening and agenda (Anna) 5 min

2 Information on Wiki access (WP7) (All) 5 min

3 Information on state of the art WP7 (Anna&Weronika) part 1 15 min

- Brainstorming – why we do it?

- The results of brainstorming on country level

- The results of brainstorming on international level

- Conclusion

4 Discussion and proposal on brainstorming (All) approx. 5-10 min

5 Information on state of the art WP7 (Anna&Weronika) part 2 15 min

- Questionnaire:

- the context

- the construction of the questionnaire

- the questions

- the terms of realization

6 Discussion and proposal on the questionnaire (All) approx. 5-10 min

7 Preparation to the Tallinn meeting (Anna) 5 min

8 Action plan – next steps (Anna) 5 min

9 Milestone 1: Progress & technical report of internal WP-meeting 5 min

Multi domains internal Webex

meeting – results for Wiki page

Mapping between sources and domains could be developed as a section within the ESSNet Wiki.

The idea is that we could have a list of sources, we could then click on a source,

which would then send us to page which listed relevant use cases. We could then click a use case, which would take us to a webpage containing more detailed information, including links to published research and contact information for any projects investigating these areas.

Similarly, we could have a list of domain areas, which would direct the user to

relevant use cases and the sources needed to support them. So basically, the idea would be to develop a corpus of what we known about big data sources and their application within official statistics which could be added to over time.

The Wiki would provide a platform for making this open for others to contribute to.

Multi domains internal Webex

meeting – results for Wiki page

-for a particular source, e.g. mobile phone data, a page listing the existing use cases, in which each item in this list links to a page further explaining this particular use case;

- and for a particular area, e.g. tourism statistics, a similar list of use cases linking to use cases (some of which are linking to a mobile phone use case also in the list of sources above).

All participants agreed on this way of organizing big data sources and decided to use

“The classification of types of big data” created by UNECE as a structure. As a structure for the domain participants agreed to “Classification of Statistical Activities”.

WP7 MILESTONE 1

MILESTONE 1

Progress and technical report of internal WP-meeting based on:

BRAINSTORMING RESULTS QUESTIONNAIRE PREPARATION

Multi domains internal meeting took place on 24th of May

• The aim of the task 1 in the Work Package 7 is to identify the sources of big data (including their durability and availability in different countries), assessment of the possibility of using selected sources for data analysis in the areas of population, agriculture and tourism and to identify which of the results or new products from pilot studies may be useful in these areas.

• One of the methods to build a preliminary extensive list of potential sources is conducting a brainstorming session. This method is characterized by the use of intuition to problem solving and teamwork, the advantages are: higher efficiency of a group than individuals, better detecting errors in a group, greater objectification of results in a group, greater creativity, greater degree of humanization of work in a group, learning cooperation and collaboration of the group members.

Brainstorming

Why did we do the brainstorm?

to create the widest possible range of big data sources (a cafeteria

possible sources of data that public statistics could use for new

developments or supplement existing ones, so that in the later

stages these sources can be verified from different points of view and

gradually part of them will be eliminated as the least useful.

to analyze as many as possible use cases of using big data sources

to take into account the most popular source

Big Data is a new phenomenon we should take into account that the potential of each source may still change.

Brainstorming - organization

Poland organized several brainstorming sessions:

The first on a country level, in the classic form, took

place during a meeting on 26 of April 2016

1 Electronic

brainstorming at country level 2

Electronic brainstorming

at international level (by WP7`

participants)

3

Brainstorming Created list is sorted by different levels of typology and then aggregated by the convergence of records: a. The original number of records: 122 b. Number of records on the Aggregation: 86 (including 4 without categorization). Eventually, the list was narrowed to those that initially can be considered useful in the context

of their further use (38 items) in official statistics.

Polish brainstorm

• Sources indicated in the brainstorm sessions in Poland were considered from the perspective of data. Records suggest that the overall potential of the source and the majority does not imply a clear use cases.

International brainstorm

• The material obtained from the United Kingdom, the Netherlands and Ireland was presented, but in a more compact manner, it means that the teams were limited to sources that have been identified and based on them were carried out some pilot projects or such sources are in the course of diagnosis (most have additional description and a link to the report).

• The results are presented in terms of source - use case.

Additionally, a further step in task 1 in the work package, that is, mapping sources to the areas of statistics (including, in most cases, the three set out in the WP7).

Brainstorming - results The most common source belong to the category Machine-generated data (Automated Systems): 38, of which the vast majority is a source based on various sensors / sensors: 30.

Often, the source pointed out partially processed (2. Process-mediated date): 28, including those which are administrators of tourist offices or institutions 14.

No sources which can be classified into types:

1300 - Personal documents, 1800 - User-generated maps,

But: Information on the movement of population on the basis of data from an application like Google Fit, etc. Endomondo suggests the use of log files, although such applications also provide the ability to create and spread on social networking personalized maps

Type 3131-3135 - any of the types of satellite data were specified.

from BRAINSTORMING

to the QUESTIONNAIRE

Why did WP7 carry out the questionnaire?

to find out more about the possibilities of technical, methodological quality,

access in different countries

recommending the source to the pilots after 2018 to know the plans for big data of different

countries

questionnaire was sent to countries outside the

FPA (but EU country), because we recommend beyond the period of its

duration

recognize the obstacles in the using of big data sources

QUESTIONNAIRE - preparation

1. Prepare draft questions for the questionnaire by Poland

2. Consultation and comments WP6 (in particular questions about the combining)

3. Testing of the questionnaire by the UK

4. Sharing a link to the questionnaire

QUESTIONNAIRE LINK: http://interankiety.pl/i/ma6JA9a3

Questionnaire - results Greece

BULGARIA

Belgium

Netherlands

Denmark

Hungary

Slovakia

Spain

UK

Italy

Germany

Ireland

italy

Estonia

Lithuania

Sweden

Austria

Finland

Poland

Each questionnaire is important to us. Thank you very much for completing it! WP7 has received 19 completed questionnaire.

Questionnaire - results

Now we are at the stage of analyzing the results.

Questionnaire - results

Questionnaire - results

WP7 plan for SGA-2

TASK 3. Data combination The experimental work (if practical work would be possible/if not it would be theoretical considerations including consultation with practice ex. data hub)

Data collection Data preparation Data analysis

Describe practical, technical and methodological aspects when combining big data outputs in the statistical system. For example, differences in definition, populations and volatility etc. Provide first answers on quality issues when combining big data with traditional outputs. Provide answers on the question whether micro-data have to be used when combining big data estimates with traditional outputs or data at aggregated level can be considered.

Analysis advantage and disadvantages of combining data Preparing the list of criteria for combining data Milestones 3. Combining data analyze; by M19

WP7 plan for SGA-2 TASK 4. Summary plus future perspectives

Suggest pilots and domains with successful implementation potential for further elaboration in the second wave of pilots in 2018.

Recommendation on legal aspects; Recommendation on availability; Recommendation on methodology; Recommendation on quality; Recommendation on technical requirements.

Conclusion

Milestones 4. List of potential pilots and domains with successful implementation potential for further elaboration in the second wave of pilots in 2018. by M23

M24 Deliverable - THE GENERAL REPORT FOR EACH/SEVERAL DOMAIN including:

The data access (with legal and privacy aspects) The data quality issues The methodology (focus also on combining data) The technical aspects

Sandbox

WP7 access to the Sandbox is not necessary at this stage of work

Anna Nowicka Central Statistical Office of Poland

and thank all partners for their contribution to the work!