02 data webinar final.pptx [read-only] · does your organization currently leverage big data? a)...

©2016 FI Consulting. All rights reserved.

DATA ESSENTIALSFOR ANALYTICS AND MODELING

September 14th, 2016


Why Does Data Matter?

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

– Jim Barksdale, Former CEO of Netscape

2



25% of critical data at the worlds top companies is flawed

37% were not satisfied with quality and availability of appropriate data for BI and Analytics

Bad data costs U.S. businesses $600 billion a year.

3



4



5

©2016 FI Consulting. All rights reserved.6

The Art of Data Curation


Data Lifecycle

Collect Prepare Store Analyze/ Report Decide

7

80% of Time


Early Stages of Data Analysis

8

C P S A D

C P S A D

C P S A D

Ope

ratio

nsFinance

Compliance

Collect Prepare Store Analyze/ DecideReport


Centralized Data Management

9

C P S A D

A D

A D

Ope

ratio

nsFinance

Compliance

Collect Prepare Store Analyze/ DecideReport


Audience Poll #1What framework best represents the current state of your organization’s data analysis?

A) We process data and perform analytics in siloes.

B) Our data is centralized, but business units perform significant additional data cleaning and preparation.

C) Data management is a centralized function.

D) All the data I use is on my desktop.

10


Challenges Data Proximity and Ownership Complexity vs. Flexibility Continued Duplication of Efforts

11


Keys to Success Data Governance Transparency Communication

12


Data Curation in the Age of Big Data

13


Audience Poll #2Does your organization currently leverage big data?

A) Yes, we use big data in some analysis

B) Yes, we use big data in all of our analysis

C) Yes, but we are just getting started

D) No, we do not currently use big data

14


Big Data“Datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze”

“The ability of society to harness information in novel ways to produce useful insights or goods and services of significant value” and “…things one can do at a large scale that cannot be done at a smaller one, to extract new insights or create new forms of value.”

15http://www.forbes.com/sites/gilpress/2014/09/03/12-big-data-definitions-whats-yours/#35b0bfc221a9


How big is a petabyte?


Big DataKey enabler of big data is Hadoop

17

“parallelizes large data sets across low‐cost commodity hardware for easy scale and dramatically reduces the cost of petabyte environments.”

http://www.forbes.com/sites/ciocentral/2012/04/16/the-big-cost-of-big-data/#4afd59456a21


Big Data + Data Curation

18

Hadoop / Unstructured Data

Email DataCall Center

Data

Mobile Clickstream

DataProduct Data

Business User Analytical Stores

Summary & Aggregation Models

BI/Data Visualization Tools

Business Users

Data Scientists & Analysts

R, Hive, Python Learnings from

Data Science

http://analytics-magazine.org/theres-no-such-thing-as-unstructured-data/


Data CurationThe process of turning independently created data sources (structured and semi‐structured data) into unified data sets ready for analytics, using domain experts to guide the process.

19


Why is data curation important?


Data is growing in size and complexity

Data in siloes spread throughout organizationTime and money spent on data preparation Need to answer business questions more quickly Self-service analytics

Image: iStock/stefanamer


Credit risk analyst needs additional metric for analysis Permission to database Different unique identifier for borrower Nuances to data aggregation

22


With data curation process…. Dataset identified as potentially useful since it has borrower

financial information Borrower unique identifier has been standardized across

database Data dictionary provides information on how data should be

aggregated

23


Challenges to an effective data curation process

24


Data Integration Separate business units / legal entities Firms acquiring other firms Data quality from legacy systems

25


Curating Golden Source Data


Data DocumentationEssential part of data governanceMitigates two types of risks:

1. Misinterpret and misuse the data2. Unaware of data availability


Data DocumentationData Dictionary Plain language definitions Data types (text, numeric, date, etc.) Valid values

Source‐to‐Target Mapping Trace to original source Show transformations Maintain transparency and accuracy


Audience Poll #3Which statement best describes your use of data dictionaries?

A) Most data sources have data dictionaries that are well‐documented and regularly used

B) We have data dictionaries, but they are not helpful or outdated

C) Data dictionaries do not exist, but would be helpful

D) My work does not require data dictionaries

29


Data Quality AssuranceFormally defined and documentedPerformed closer to sourceAutomated in ETLs toolsException handlingLeverage Data Dictionaries and STMs


Data Merging, Enrichment, DeduplicationCombine data from multiple sourcesExact matching by common elementsFuzzy matching No unique identifier (i.e. SSN) Names and addresses Spelling, punctuation, and abbreviations

I. (J. Smith vs. John Smith vs. John D. Smith)

Requires a minimum certainty threshold Remove duplicates


Feedback Loop for User‐Driven Enhancements Two‐way processEnd users perform high‐value analytics (i.e. data mining, network graphs, clustering, segmentation,

model diagnostics, etc.)

Detect outliers and anomaliesIncorporate changes at source


Tools For Your Easel

33


Databases

34


Programming Languages

35


ETLs (Extract – Transform – Load)

36


What’s next?

37


Early Stage, Siloed Data Analysis Data Assessment

Moving towards Centralized Data Management Implement Data Curation

Centralized Data ManagementIn Place Data Governance Review


Q&A

39


Speakers Christina PrevalskySenior [email protected]

Greg SteckSenior [email protected]

Vadim BondarenkoSenior [email protected]

FI Consulting, Inc.1500 Wilson BlvdArlington, VA 2009(571) 255 - 6900 www.ficonsulting.com

40

02 data webinar final.pptx [read-only] · does your organization currently leverage big data? a)...

Documents