enable domain experts explore, normalize and enrich their data via a self service data ... -...

refinepro.com - @RefinePro – martin@refinepro.com 1

Enable domain experts explore,

normalize and enrich their data via a

self service data preparation platform

Garbage In – Garbage Out

Data Processing Pipeline

60 to 80% of data analysis

is spent on the process of

cleaning, transformation and integration

Data Processing Pipeline

Analytics need clean data to be

reliable

Legacy data need to be

migrated to a new system

Data must be reconciled

against a master data set

Data projects needs access to

reliable data quickly

• Messy and inaccurate data.

• Individual and business units data have unique needs

• New predictive and enrichment services made available

using an API first approach.

• Current tools are challenged the speed of those new

requirements.

Data Integration Challenge

• Duplicate value & Typos

• Multi value cells

• Data in the wrong field

• Missing / Partial Values

• Encoding Errors

• Change format (text, number, date)

• Flat to relational data set

• Schema alignment

• Transpose rows and columns

• Join data-set

• Enrichment from other sources

(MDM, API calls)

Data Quality & Integration &

Is Time Consuming

• Which field should it contains?

• What format should it follow?

• What geographical scope should it support?

• Enforce data integrity rules (eg. postal code vs city)?

Individual and business units data

needs are not consistent

What is clean data?

How do you know define a clean address?

• New economy of machine learning, predictive and data

enrichment service:

• Geocoding and address cleaning

• Name recognition and extraction

• Churn prediction

• …

Those services come with an API first approach requiring

technical skills

Data Service have an

API first approach

Data Science

Spreadsheet User

Data Visualization / Interpretation

User Base

Understand the Data

(Business Skills)

Know How To Transform Data (Technical Skills)

Today's data environment challenge traditional

technologies

Excel doesn't scale or

automate well

IT can't pace with the volume

of requests

Agile Process

• Let's you adjust and adapt to a changing environment

by working in iteration

• Stop at the right level of quality

Tools: Self Service Data Preparation

• Empower the domain experts

• Allow to iterate faster through the process

New process & tools to the rescue!

Data Discovery

& Profiling

Track / Measure

Data Consumption

Data Transformation

Agile -

Incremental Data

Processing

Data Discovery & Profiling

Place data in context

Test data service

Is it useful?

What Can I do with it?

Track / Measure

Check data integrity

Find quality gaps

Learn from your experience

Agile -

Incremental Data

ProcessingData Consumption

Analytics

Migration

Reconciliation

Data Transformation

Define strategy

Perform data preparation

Self Service Data Preparation Bridges The Skill Gap

Data Science

Spreadsheet User

Data Visualization / Interpretation

OpenRefine

Excel doesn't scale or

automate well

IT can't pace with the volume

of requests

User Base

Understand the Data

(Business Skills)

OpenRefine Functionality

XLS, CSV, JSON,

XML Input &

Output Support

Point & Click

Cluster &

Deduplication

Filter &

Transpose Custom Query

Language

Enrich data via

Join, Merge

& Reconcile

Split to rows

and columns

Undo /

OpenRefine

Community developed for 5 years

Gridworks > Google Refine > OpenRefine

5,000+ monthly download

Run on a local machine

Large usage among Data Journalist, Library, Semantic

web, Open Data and Bio Science experts.

TrainingCloud & on-

premise hosting

Integration & Custom

Development

RefinePro helps teams and

organization to scale OpenRefine

1. Toronto Build Permit Data Set:

Explore what data we have available

Geocode The address

2. Salesforce dump:

Remove duplicate name

Add Facebook and Twitter profile via FullContact

Frequency- number

of use case

ProfilingPreparation

DiscoveryData Wrangling

Sense MakingData Exploration

Is the data useful?What Can I do with it?

OpenRefine in the Data Quality & Integration Pipeline

Frequency- number

of use case

Personal ETL & Analysis

Prototype

One time migration

Frequency- number

of use case

Big Data

Real -Time Processing

Enterprise ETL

Personal ETL & Analysis

Prototype

One time migration

Understand the Data

(Business Skills)

Frequency- number

of use case

Enable domain experts explore,

normalize and enrich their data via a

self service data preparation platform

OpenRefine Eco-System

Reconciliation service sit outside of Refine and

enable user to align and enrich data against

domain specific master data

Extensions add

functionality to

Refine core.

API processing plugin enable

seamless data process with

API based services

Batch Processing

library enable

lightweight ETL process

New Distributions focus on

- domain specific integration

- explore new functionality

- hosted version of Refine

enable domain experts explore, normalize and enrich their data via a self service data ... -...

Documents

enrich summer2015

effective diabetes treatment - normalize your blood sugar...

11823 cat 1400v us.ps, page 1-27 @ normalize

opportunity! register today. enhance enrich enable… ›...

175322.indd, page 1-12 @ normalize ( 175322.indd )

traccs enrich

enrich spring 2014

explore • experience • enrich

23517_gemtor.ps, page 1-30 @ normalize

capa ja228.ps, page 1 @ normalize ( untitled-1 )

octnovdec 2015_pdf version_layout 1, page 1-16 @ normalize

enrich summer 2016

220823727 cv.ps, page 1 @ normalize ( 220823727...

enrich spring 2016

enrich property group

enrich ibs

smart terminals enable intelligent services · • smart...

us16491cvr.ps, page 1-2 @ normalize 3

bdna normalize for configmgr™

1018oxford›pŁ¶.ps, page 1-32 @ normalize