Big Data Quality, Partnerships and Privacy Teams

Agenda

IntroductionApproachResults of 3 task teams

• Quality• Partnerships• Privacy

Approach

Analysis of best practice and available documentationWork in virtual teams on two-week scheduleTesting in SandboxVirtual sprintsWorkshops

Quality

Quality framework(s) for Big Data. Testing the framework(s). Indicators and associated metadata

requirements.

Approach - the concept of hyperdimensions was taken from the administrative data quality framework.

Conclusions for Quality

There is a need for quality assessment covering the entire business process.• Input quality can be explored and assessed by using

and elaborating existing input quality frameworks.

• Throughput quality can be maintained by following quality processing principles but quality dimensions need to be further developed for Big Data processing.

• Additions have been proposed to output quality dimensions from existing frameworks, to make them suitable for Big Data applications.

Partnerships

Task: Explore current experiences and produce guidelines for partnerships

Sources:• Experiences from the Sandbox

• Experiences from Task Team participants / organisations

• Survey information: partnership questions added to a UNSD survey on Big Data for Official Statistics

Different types of partnerships - data providers design and analysis, technology partners…

Conclusions for Partnerships

A project can only exist if a working partnership can be forged with a data provider

For multinational data sources partnership agreements need to be drafted that can be used by all statistical offices

Operational guidelines for forging Big Data partnership agreements are needed

Privacy

To give an overview of existing tools for risk management in view of privacy issues

To describe how risk of identification relates to Big Data characteristics

To draft recommendations for NSOs on the management of privacy risks related to Big Data

Conclusions for Privacy

Existing tools are well-developed Privacy risk can be linked to Big Data

characteristics Recommendations have been formulated on:

• information integration and governance

• statistical disclosure limitation/control

• managing risk to reputation

But: not much experience yet with Big Data privacy issues

More InformationUNECE Wikihttp://www1.unece.org/stat/platform/display/bigdata/2014+Project

Presentations at NTTS Conference• Quality 17A

• Partnerships 4A

• Privacy 9B