big data: new opportunities for official statistics content/2... · big data: new opportunities for...

22
Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

Upload: trinhthu

Post on 12-Apr-2018

221 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

Big Data:

New Opportunities for Official Statistics

MAZREHA YA’AKUB

Department of Statistics Malaysia

Page 2: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

OUTLINES

Overview Big Data Analytics

The Importance of Big data for Official Statistics

The Challenges involved in using Big data for

Official Statistics

DOSM’S Big Data Analytics

Page 3: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

Overview Big Data Analytics

“To Get Data is EASY To Get The Right Data

is Hard”

WHAT IS BIG DATA? • Huge volume of Data

• Complexity of data types and structures

• Speed of new data creation and growth

Page 4: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

Structure

Semi-Structured

“Quasi” Structured

Unstructured

Source :EMC Education Services

BIG DATA CHARACTERISTICS: DATA STRUCTURES

Page 5: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

BUSINESS INTELLIGENCE VS DATA SCIENCE

Source :EMC Education Services

Page 6: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

The Importance of Big data

for Official Statistics

Big data, the future of statistics….

National Statistical Office

(Primary Data)

New Official Statistics

Compilation of official

statistics for

evidence-based

decision–making

Data of ‘others’

=

Administrative bodies

Page 7: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

The Importance of Big data

for Official Statistics

Reduce the response burden on enterprise

and households as well as the cost for the

data collections and improve the quality

of the statistics.

Data growth rate are skyrocketing..

Source : Statistical Journal of the IAOS

Page 8: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

The Importance of Big data

for Official Statistics

Improved the timeliness of statistical products

More detailed break-downs of statistics possible

Improve accuracy

Production of new statistical indicators

Page 9: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

The Challenges involved in using

Big data for Official Statistics

Suitable statistical and IT methods

New tools and skills are needed to handle

alternative data

Quality issues on each dataset and

application

Page 10: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

The Challenges involved in using

Big data for Official Statistics

The costs of sourcing the data must in

balance with the benefits

Legislative requirement for getting access and

using the data

Legal issues (e.g. personal data

protection)

Page 11: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM’S Big Data Analytics

Used as supplements for

existing data in production of

certain statistics.

Allow high actuality

Reduce response burden on

some respondents

Page 12: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

BIG DATA ANALYTICS FOR DOSM

SOURCES Statistical Model Engine

CASE MANAGEMENT

REPORTING

PREDICTIVE MODELS

ALERT

DATAWAREHOUSE

PREDICTIVE

MODELING

SCORING

MODEL MANAGEMENT

RETRAIN / REBUILD

(Automated/Manual)

DECISION

DECISION

Integrated Big Data Analytics

Environment

SOURCES

Big Data Platform

Data Management

Unstructured Data Analytics

Advanced Analytics

Visualization

Page 13: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM’S Big Data Analytics

MAIN PROJECTS

TRADE BY ENTERPRISE CHARACTERISTICS (TEC)

PRICE INTELLIGENCE (PI)

OPINION MINING ON OFFICIAL STATISTICS

1

2

3

Page 14: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM’S Big Data Analytics

SUPPORT PROJECTS

REAL TIME BUSINESS STATUS

REAL TIME NEWS ON OFFICIAL STATISTICS

BizCode@Stats

1

2

3

Page 15: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM’S Big Data Analytics

DOSM BDA INITIATIVES 1:

TRADE BY ENTERPRISE CHARACTERISTICS (TEC)

Record what types of goods

are trading across border

between countries

The linkage of trade statistics with

business registers allows us to describe

those who are engaged in global market

and what are their characteristics.

Traditional trade statistics Trade by enterprises characteristics

(TEC)

Page 16: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM’S Big Data Analytics

DOSM BDA INITIATIVES 1:

TRADE BY ENTERPRISE CHARACTERISTICS (TEC)

TEC DATABASE

Trade

Database MSBR

TEC

Database

Integrated Trade Database with MSBR

Page 17: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM’S Big Data Analytics

DOSM BDA INITIATIVES 2:

INTERNAL PORTAL FOR PRICE INTELLIGENCE (PI)

Modernization of data collection tools

for improving quality of Consumer

Price Index (CPI). The modernization

of data collection mainly consist of the

adoption of web scraping techniques

to scrape price data from related

website for CPI compilation.

Page 18: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM’S Big Data Analytics

DOSM BDA INITIATIVES 2:

INTERNAL PORTAL FOR PRICE INTELLIGENCE (PI)

Data Warehouse

Analysis

Data Visualization

Data Mining

Report &

Dashboards

Alert

Page 19: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM’S Big Data Analytics

DOSM BDA INITIATIVES 3:

Opinion Mining on Official Statistics

The analysis and assessment of the

degree of “happiness” of Malaysia

community with regards to official

statistics published by DOSM. The

data is obtained from online social

media.

Page 21: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

DOSM utilizes Big Data as supplements

for existing data to produce

comprehensive and quality statistical

products, services and new indicators.

Source : Statistical Journal of the IAOS

Page 22: Big Data: New Opportunities for Official Statistics Content/2... · Big Data: New Opportunities for Official Statistics MAZREHA YA’AKUB Department of Statistics Malaysia

Mazreha Binti Ya'akub

[email protected]

Methodology & Research Division

Department of Statistics Malaysia