big data in official statistics - experiences at statistics...
TRANSCRIPT
Poznań, Poland, 10 September 2015
Big Data and Official Statistics – Experiences
at Statistics Netherlands
Peter Struijs
Outline
¾ Big Data and official statistics
¾ Experiences at Statistics Netherlands with:
- use of road sensor data- use of mobile phone location data- use of public social media messages
¾ Issues and solutions
¾ Strategic, policy and organisational challenges
¾ Cooperation and collaboration
2
What is Big Data?
3
4
Data sources and approaches
Surveys / questionnaires
sampling theory
Administrative data sources
Where does Big Data fit in?
New methods may be needed, e.g. modeling fornowcasting and other methods not based on sampling theory
5
Potential Opportunities
¾ New statistics
¾ More detailed statistics
¾ More timely statistics
¾ Nowcasts and early indicators
¾ Quality improvement
¾ Response burden reduction
¾ Cost reduction and higher efficiency
6
Examples of possible Big Data sources
¾ Road sensor data
¾ Mobile phone location data
¾ Public social media messages
¾Websites
¾ Google Trends
¾ Satellite information
¾ Etc…
7
Data scientists involved in the research shown
¾ Piet Daas (photo)
¾May Offermans
¾Marco Puts
¾Martijn Tennekes
8
Statistics based on road sensor data
¾ Aim: Statistics on traffic intensities
¾ Characteristics of the data source
¾ Research on the usability of the data
¾ Process of using the data for statistics
¾ Issues when using traffic loop data
9
Road sensor data
¾ Source: National Data Warehouse for Traffic Information (NDW)
¾There are 20.000 traffic loops on Dutch motorways, and 40.000 on provincial roads
¾ Each minute (24/7) the number of passing vehicles is counted, and their average speed
¾ Three different length classes are distinguished
¾ No identification of vehicles¾ Around 230 million records a day used
Locations
10
The main roads
11
A special dike
12
Road sensors in the dike
13
Minute data of one sensor for 196 days
14
Researching the data
Cross correlation between sensor pairs- Used to validate metadata
Trajectory speed vs. point speed- Average speed is 98 Km/h
Small, medium-sized & large vehicles
22
Sensors in a road segment
17
Process of making traffic intensities statistics
¾ Select sensors on Dutch highways
¾ Preprocessing -‐ Remove non-informative variables-‐ Remove bad records-‐ Exclude bad sensors-‐ Quality indicators for daily data per sensor
¾ Processing-‐ Reduce dimensions on same road and region
-‐ Obtain number of vehicles for each road and region-‐ For each road and region, calculate monthly traffic intensity-‐ Use of R-Hadoop
¾ Validation and publication18
Data options
Historical database-‐ Request data via web interface-‐ Minute data for all highways (48 variables, Jan 2010-
April 2014: around 2.5 TB)
Data stream-‐ Every minute, all data for all active sensors-‐ Continuously collected
19
Road sensor data: Issues and non-issues
Non-issues:¾ Privacy¾ Data acquisition
Issues:¾Methodology
- Selectivity- Quality
¾ Infrastructural needs¾Other issues
- Skills needed- Transition from research to
regular statistics 20
References: statistical use of road sensor data
¾ Publication of statistical results (in Dutch):http://www.cbs.nl/nl-NL/menu/themas/verkeer-vervoer/publicaties/artikelen/archief/2015/a13-drukste-rijksweg.htm
¾ Explanation in English:http://www.cbs.nl/NR/rdonlyres/25CE3592-A756-42B7-BABF-C3E4C4E9375B/0/a13busiestnationalmotorwayinthenetherlands.pdf
¾ Research reference:Puts, M., Tennekes, M. and Daas, P. (2014) Using Road Sensor Data for Official Statistics: Towards a Big Data Methodology. Paper for Strata + Hadoop World, Barcelona, Spain.
21
Statistics based on mobile phone location data
¾ Why use mobile phone data for official statistics?
¾Characteristics of the data source
¾ Research on the usability of the data
¾ Issues when using mobile phone location data
¾ Solutions to the issues
22
Possible uses of mobile phone data
¾ Daytime population statistics
¾ Mobility statistics
¾ Tourism statistics
¾ Other uses
23
Mobile phone activity as a data source
¾ Nearly every person in the Netherlands has a mobile phone-‐ Usually on them -‐ Almost always switched on-‐ Many people are very active during the day
¾ There is a grid of antennas with good coverage
¾ Data of a single mobile company was used-‐ Hourly aggregates per area-‐ Threshold of 15 events
24
Daytime population based on mobile phone data
Issues when using mobile phone data
¾ Privacy
¾ Data acquisition
¾Methodology- Representativeness- Selectivity- Quality
¾Other issues- Infrastructure- Skills needed
26
Solutions
¾ Agreement with data provider to provide onlyaggregates and apply a threshold
¾ Data provider performed analysis of mobile phoneownership characteristics
¾ A large number of analyses were made, with the regularpopulation registration data as a reference
¾ A number of assumptions had to be made
27
References: statistical use of mobile phone data
¾ Research references:
Daas, P.J.H., Puts, M.J., Buelens, B. and van den Hurk, P.A.M. (2015) Big Data as a Source for Official Statistics. Journal of Official Statistics 31(2), pp. 249-262.
Daas, P. and Burger, J. (2014) Profiling big data sources to assess their selectivity. Paper for the 2015 New Techniques and Technologies for Statistics conference, Brussels, Belgium.
28
Mobile phone data versus road sensor data
29
Statistics based on social media data
¾ Why use social media data for official statistics?
¾Characteristics of the data source
¾ Research on the usability of the data
¾ Issues when using social media data
30
Possible uses of social media data
¾ Sentiment indicators
- e.g. consumer confidence index
¾ Social indicators
- e.g. social coherence indices
¾Other uses
31
Social media
– Dutch are very active on social media!-‐ Around 60% according to a surveyna altijd bij zich en staat vrijwel altijd aan
• Steeds meer mensen hebben een smartphone!
– Mogelijke informatiebron voor:-‐ Welke onderwerpen zijn actueel:
• Aantal berichten en sentiment hierover
-‐ Als meetinstrument te gebruiken voor:
• .
Map by Eric Fischer (via Fast Company)
32
The data
¾ All social media messages:- that are written in Dutch- and are public
¾ These messages are systematically and instantly collected by the Dutch firm Coosto
33
¾ Dataset of more than 3.5 billion messages:- covering June 2010 till the present- between 3-4 million new messages added per day
Research question
C an we replicate the consumer confidence index by only using social media data, while reducing production time?
34
Sentiment determination
¾ ‘Bag of words’ approach- list of Dutch words with their associated sentiment - added social media specific words (‘FAIL’, ‘LOL’, ‘OMG’ etc.)
¾ Use overall score to determine sentiment- is either positive, negative or neutral
¾ Average sentiment per period (day / week / month)- (#positive - #negative)/#total * 100%
35
Sentiment per platform
(~10%) (~80%)
Build a model
Idea: Fitting characteristics derived from social media messages to consumer confidence
Success: If correlation can be found that is high andremains high, that is, has predictive power
37
38
Figure 1. Development of daily, weekly and monthly aggregates of social media sentiment from June 2010 until November 2013, in green, redand black, respectively. In the insert the development of consumer confidence is shown for the identical period.
Results
¾ High correlation achieved (0.9)
¾ Changes in consumer confidence preceed changes in sentiment by one week
¾ Short processing time, so time-to-market may bereduced.
¾ Sentiment index can be produced on a weekly basis
¾ To be considered:
- Use model-based figures as early indicators- Reduce sampling of consumer confidence index
39
General sentiment indicator (draft version)
40
Issues when using social media data
Lesser issues: ¾ Privacy¾ Data acquisition
Main issues:¾Methodology
- Selectivity- Meaning of the data- Validity of methods used
¾Other issues- Skills needed
41
Questions on the validity of methods used
¾ Is it acceptable, under certain conditions, to base official statistics on correlations?
¾ If so, what are the conditions?
¾What to do if there is a shock?
42
Reference: statistical use of social media data
¾ Research reference:
Daas, P.J.H. and Puts, M.J.H. (2014) Social Media Sentiment andConsumer Confidence. European Central Bank Statistics Paper Series No. 5, Frankfurt, Germany.
43
Big Data Characteristics
44
Definition: ¾ Volume¾ Velocity¾ Variety
Data characteristics: ¾ Unstructured data¾ Selectivity¾ Population dynamics¾ Event data¾ Organic data¾ Distributed data
Data use: ¾ Other ways of processing¾ Fundamentally new applications
Overview of Issues
¾ Getting access to the data
¾ Usability of the data
- Meaning of the data, stability of the source, reproducability
¾ Methodologal issues
- Selectivity, representativeness, unknown population, quality and validity
¾ Privacy, confidentiality and reputation
¾ IT-infrastructure and security
¾ Knowledge and skills
¾ Transition from research to production
¾ Strategic challenges45
Possible responses to the issues
¾ Invest in good relations with the data provider
¾ Invest in methodological research and play with the data to get a grip on quality
¾ Use only aggregate data if possible
¾ Explore alternatives to population-based estimationmethods
¾ Keep an open mindset
¾ Take the strategic challenges seriously
46
Strategic aspects
¾ Others start producing statistics- there may be quality issues- but they are extremely rapid- and there is obviously demand
¾ Need for good, impartial information(benchmark information) will remain
- without a monopoly for NSIs
¾ There is a need for validation of information produced by others
47
Billion Prices Project MIT
48
49
The Roadmap Approach
¾ Awareness that Big Data is a strategic issue
¾ Position paper for Board of Directors
¾ Roadmap Big Data
¾ External validation of the Roadmap
¾ Roadmap updated twice a year for Board of Directors
¾ Roadmap monitor
¾ Deputy Director General responsible at strategic level
¾ Coordination group for Big Data
50
The Scope of the Roadmap
¾ Identification of outputs to be based on Big Data
¾ For each output, definition of time target and ownership
¾ Identification by owner of conditions to be fulfilled
¾ Commitment by supporting services for fulfilling the
conditions (IT, data collection, methodological support, …)
¾ Supporting programmes
51
The Roadmap Projects
Focus projectsRoad sensor data for traffic intensities statisticsMobile phone data for daytime population statistics
Other projectsInternet data for price statisticsFinancial transactions data for statisticsSocial media data for detecting trends in social cohesionInternet data for encoding enterprise purchases and sales
52
Supporting Programmes
Big Data features in:
¾ Innovation programme
¾Methodological research programme
53
Cooperation and Collaboration on Big Data
Statistics Netherlands works together with:
¾ Other NSIs¾ UN, UNECE, EU, WorldBank¾ ESSnet on Big Data (to be confirmed)¾ Government organisations¾ Universities and research organisations¾ Data providers¾ IT providers¾ Big Data firms¾ Research consortia (e.g. H2020)
54
UNECE Big Data Activities
– Classification of Big Data sources– Big Data project in 2014, with three Task Teams:
-‐ Partnerships-‐ Privacy-‐ Quality
– Sandbox in 2014, 2015 and possibly beyond– Big Data survey, together with UNSD
Results: http://www1.unece.org/stat/platform/display/bigdata/2014+Project
55
UN Big Data Activities
¾ Global Working Group on Big Data for Official Statistics with eight Task Teams:
-‐ Mobile phone data-‐ Satellite imagery-‐ Social media data-‐ Access / partnerships-‐ Advocacy / communication-‐ Big Data and SDGs-‐ Training / skills / capacity building-‐ Cross-cutting issues
¾ UNSD survey on Big Data for official statistics
56
Draft Big Data Access Principles (UN)
¾ Social responsibility
¾ Level playing field
¾ Equal treatment
¾Confidentiality and security
¾Transparency
¾ Respect for business interest
¾ Proportionality
57
Conclusion: The Way Forward
¾ Get to know Big Data
¾ Use Big Data for efficiency andresponse burden reduction
¾ Use Big Data for early indicators
¾ Use Big Data for filling gaps and new demands
¾ Use new professional methodswhere needed
¾ Create the right environment
¾ Don’t do it alone!58
General references
Glasson, M., Trepanier, J., Patruno, V., Daas, P., Skaliotis, M. and Khan, A. (2013) What does "Big Data" mean for Official Statistics? Paper for the High-Level Group for the Modernization of Statistical Production and Services, March 10.
Struijs, P., Braaksma, B. and Daas, P. (2014) Official Statistics and Big Data. Big Data & Society, April–June, pp. 1–6.
Struijs, P. and Daas, P.J.H. (2013) Big Data, Big Impact? Paper for the Seminar on Statistical Data Collection, Geneva, Switzerland.
Struijs, P. and Daas, P. (2014) Quality Approaches to Big Data in Official Statistics. Paper for the European Conference on Quality in Official Statistics 2014, Vienna, Austria. 59
The Future
60
