big data visualizations · 2018. 10. 18. · experience big data & d/w integrated fabric...

Big Data VisualizationsA Dialog Surrounding the Use of Visualizations with Big Data

© Global IDs. All Rights Reserved.

Robert J. Abate, CBIP, CDMPOctober 18th, 2018

• Big Data Visualizations – A Starting Position

• Information Is The New Currency

• What Has Been Missing, The Challenge

• One Screen Is Not Enough

• Infographics And Big Data

• Best Practice Big Data Visualization Examples

• What Is Next?

Agenda

2

NOTE: Authors Opinion (not that of NY DAMA Chapter)…

Everyone Talks About It,

Few Really Know How To Do It,

It’s Been The Source Of Many Rumors,

Everyone Thinks That Everyone Else Is Doing It,

So Everyone Claims They Are Doing It !

3

Big Data Visualizations Can BeLikened To Teenage Relationships

Introduction To Big Data Visualizations

• Information is not data

• Data Science is almost the inverse of Business Intelligence (B.I.)

• Big Data Science has completely different goals - compared to Data Warehousing & B.I.

• Big Data architectures are unlike DW & BI architectures

• Examination of Big Data & Data Science requires new methods and thinking

Overview Of Big Data Vs. Business Intelligence…

4

• Real-time decision

making is critical to

business success

• Data in the enterprise

continues to grow

exponentially year

over year, making

analysis problematic

• To turn structured

and unstructured

data into actionable

intelligence, your

business needs an

effective, smart way

to harness Big Data.

• Agile / SCRUM

development is

essential today in Big

Data to learn

‘quickly’

Overview – Time Is Money & Lifecycle

5

Big Data Visualizations Defined

Authors Starting Opinion On Visualizations…

• One Screen Does Not Support “Ah-Ha” Moments1

• “Infographics” - are graphic, visual representations of information, data or knowledge2

– Intended to present complex information quickly/clearly

– They can improve cognition utilizing graphics to enhance human visual system’s ability to see patterns and trends

• Visualizations Provide A Picture Of Information

– Rapidly conveying insight & understanding when used right

1 – “The Work Of Edward Tufte & Graphics Press” www.edwardtufte.com2 – “Infographic” www./Wikipedia.com

6

A Picture Is Worth A Billion Words…

7

• Information Is Currency1

– It will be the most valuable asset in our organization

– Without information, nothing else we own will have value

– “Without knowledge, you’re powerless” 2

• Need For Trust– Legal compliance (SOX, …)

– Data integrity (Spreadmarts)

– Crypto-currency

“A testament to this new reality is the growing value of digital content. People download

songs, purchase movies online, exchange emails, and share personal information—all in

the form of digital content. Information in its many new forms has become

commoditized. In a digital world, ’information is the new currency’”3 - Aug 28, 2015

1 – http://www.pwc.com/gx/en/information-security-survey/pdf/safeguarding_the_new_currency.pdf2 – http://www.tieto.com/top-stories/information-strategic-business-asset-are-you-managing-it-such3 - https://blogs.opentext.com/why-information-is-the-new-currency/

Information is the New Currency

8

• Turning data into knowledge and insight utilizing metadata & context

• MDM / integration of assets

• Confidence and trust in data

• Linking people to information

• Making the data available

• Single Source Of Truth!

Business Architecture

Information Architecture

Application Architecture

Infrastructure Architecture

“Databases In Most OrganizationsAre Like Buildings Build Upon One-Another Without Thought Of Integration”

What Has Been Missing?

9

• Imagine Improving Productivity by

80%

What Are The Challenges Today

10

Toolset Capabilities:

• Discovery

• Profiling

• Classification

• Semantic Matching

• Global Domain Matching

• Security / PII Considerations

• Data Quality

• Data Governance

• Compliance (GDPR, Privacy)

• Data Analytics

Big Data Visualization Requirements:

• What data do we have

• How good is our data

• What types of data do I have

• Where are all of these types

• Logical groupings / MDM

• Don’t look at your neighbors data

• Let’s setup a means to monitor

• Let’s define people/tech/processes

• Let’s utilize this to manage data

• Let’ have all the semantic objects moved into a table that I can use

Global IDs - Data Ecosystem Tool Set

11

DISCOVERY

Structured Un-Structured

PROFILING

Global IDs Tool Set – Suite of Integrated Tools

12

CLASSIFICATION DOMAIN GROUPING

Global IDs Tool Set – Suite of Integrated Tools

13

DB Column Data QualitySemantic Object Governance Data SecurityData

RedundancyCompliance

Global IDs Tool Set – Purpose-Built Portals

14

DATA LINEAGE

Global IDs Tool Set – Viewing Lineage & Impact Analysis

15

Semantic Domains Identified In ColorProtected (PII/GDPR) Data

Visualizing a Data Universe

16

• Information was trapped by constraints

– Locked into silo’s of systems (dates)

– Projects created one-off solutions

• Data was everywhere – but not cataloged nor managed

– Tending to downplay quality, challenges and risks

• More than one dictionary in enterprise

– Item, Etc.

– Business Terms

• Brittle, tightly-coupled systems

Providing Insights – The Sam’s Club Challenge

17

Immersive Collaborative

Experience

Big Data & D/W

Integrated Fabric

Advanced Dynamic

Visualizations

People, Process &

Change Adoption

Objective:Create a comprehensive

repository of connected

data to enable one

version of the truth

across functional

boundaries leading to

better informed business

decisions

• Improve Productivity And Time To Finding Answers

• Visualization Of Information Provides Key Competitive Advantage

• Adding Value With “Big Data” Sources

Walmart Data CAFE

One Screen Is Not Enough

Combining Visualizations Improve Cognition

18

Infographics & Big Data - The Sam’s Club Solution

• Big Data Visualizations Are Not A Buzz Word Anymore

‒ Variety, Velocity, Volume & Veracity Of Data

Getting All The Sources Into The Data Lake Was Key

Used Extreme Agile Implementation (4 months)

‒ Visualizations “Freed The Information”

Providing Linkages Between Information Was Key

• Business Immediately Found Value

‒ Utilizing Pre-Built Visualizations With Touch-Screens

• Information Provided Insight For Data Science

19

Combined Visualizations Improve Cognition

Category

Sub Category

Fine Line

Fine

Lin

e 1

Fine

Lin

e 2

Fine

Lin

e 3

Fine

Lin

e 4

Fine

Lin

e 5

Fine

Lin

e 6

…

Best Practices in Big Data Visualizations

20

Measures and More Measures!

• Federal Reserve Chair Janet Yellen uses a dashboard of jobs data‒ In past, a single measure

(unemployment rate)‒ Make economic and labor policy

decisions• Instead, she uses dozen different

measures‒ Providing a more holistic, accurate,

and actionable view of the United States economy

• Single measure of anything complex – Poor!‒ Whether it’s the U.S. economy‒ Customer satisfaction and

predictive maintenance‒ Oversimplifying something to the

point of not being useful or actionable

More Best Practice Big Data Visualizations

21

Additional Best Practices in Big Data Visualizations

“Seeing” the Data Points Stretches the Mind…

22

Customer Views For Differential Analysis

Recency Frequency Value Tenure

Best Practice Data Visualizations Have Multiple Dimensions…

23

Showing Time Periods Geospatially

Further Best Practice Big Data Visualizations

24

Showing Movement Of Customers

Utilizing “Data Science” Groupings

Out of over 300 data elements, a total of 45 visit behavior features were found to be significant in predicting customer behavior

Further Big Data Visualization Best Practices

25

So What Is Next?

26

• Truly Integrated Data Environment – “Data Nirvana”

• Extreme Scale, Elastic Expansion• Semantic Integration – “All Visualizations & Reports Match”• Linearly Scalable

‒ Add 1x and get 2x performance

• Self – Service Provisioning‒ Users spin up a sandbox of data using movement tools with limited

duration

• Single Point Of Management‒ Resource utilization & optimization

• Secure, Unified Data Access – Single Point of Entry‒ Portal based sharing of data sandboxes (wiki-type)

• Reduce TCO By Eliminating Excessive Licensing Fees‒ Use of open source community to improve solution

The “Ideal” Big Data Visualization Ecosystem

27

• Big Data Visualizations Are The Future‒ You will not see other “reporting” as they don’t provide value

• Use More Than One “Screen or Concept” To Gain “Brain or Cranial” Insights

• Build With Best Practices‒ Build Iteratively – Data Sources & Graphics Integrations

‒ Combine Up To Four Measures Per Graphic

‒ Visualize Movement Pictorially

Customers Moving By Group

Geospatial Movement By Time

• Plan For Extreme (Web) Scale

Key Takeaways…

28

Thank You!Robert J. Abate, CBIP, CDMProbert.abate@globalids.com

CONTACT US+(1) 888 514 0192info@globalids.comwww.globalids.com