Big Data Wilfried Hoge, Software IT Architect Big Data hoge@de.ibm.com @wilfriedhoge

How is Big Data transforming the way organizations analyze information and generate actionable insights?

Paradigm shifts enabled by big data Leverage more of the data being captured

TRADITIONAL APPROACH BIG DATA APPROACH

Analyze small subsets of Information

Analyze all information

Analyzed information

All available information

All available information analyzed

Paradigm shifts enabled by big data Reduce effort required to leverage data

TRADITIONAL APPROACH BIG DATA APPROACH

Carefully cleanse information before any analysis

Analyze information as is, cleanse as needed

Small amount of carefully

organized information

Large amount of

messy information

Paradigm shifts enabled by big data Data leads the way—and sometimes correlations are good enough

TRADITIONAL APPROACH BIG DATA APPROACH

Start with hypothesis and test against selected data

Explore all data and identify correlations

Hypothesis Question

Data Answer

Data Exploration

Correlation Insight

Paradigm shifts enabled by big data Leverage data as it is captured

TRADITIONAL APPROACH BIG DATA APPROACH

Analyze data after it’s been processed and landed in a warehouse or mart

Analyze data in motion as it’s generated, in real-time

Repository Insight Analysis

Data

Data

Insight

Analysis

How have most companies made information available for decision making across the enterprise?

Traditional enterprise data and analytics environments Typical enterprise data management environment

Staging area

Actionable insight

Enterprise warehouse Data mart

Data sources

Transaction and application data

Archive

Reporting and analysis

Predictive analytics and modeling

How are leading companies transforming their data and analytics environment to provide faster, better insights at reduced costs?

Next generation architecture Starts from the current data management environment …

Enterprise warehouse

Information governance

Data mart

Analytic appliances

Information ingestion and operational information

Data sources

SYSTEMS—SECURITY—STORAGE

Transaction and application data

Enterprise warehouse

Data mart

Analytic appliances

Actionable insight

Reporting, analysis, content analytics

Predictive analytics and modeling

Next generation architecture … adds new technologies and capabilities …

Exploration, landing and

archive

Enterprise warehouse

Information governance

Real-time analytics

Data mart

Analytic appliances

Information ingestion and operational information

Data sources

SYSTEMS—SECURITY—STORAGE

Transaction and application data

Machine and sensor data

Enterprise content

Social data

Image and video

Third-party data

Enterprise warehouse

Data mart

Analytic appliances

Actionable insight

Reporting, analysis, content analytics

Predictive analytics and modeling

Decision management

Discovery and exploration

Cognitive

+ +

Next generation architecture … to enable new applications

Exploration, landing and

archive

Enterprise warehouse

Information governance

Real-time analytics

Data mart

Analytic appliances

Information ingestion and operational information

Enhanced applications

Customer experience

Operations and fraud

Risk

Financial performance

New business models

IT economics

Data sources

SYSTEMS—SECURITY—STORAGE

Transaction and application data

Machine and sensor data

Enterprise content

Social data

Image and video

Third-party data

Enterprise warehouse

Data mart

Analytic appliances

Actionable insight

Reporting, analysis, content analytics

Predictive analytics and modeling

Decision management

Discovery and exploration

Cognitive

+ +

Sample use cases that leverage the new data management environment capabilities

Dublin City Centre; Robust and efficient citywide traffic awareness system, enhance rapid action on incidents

Need

•  A budget effective solution to improve traffic awareness system.

•  To bring accuracy in event detection, inferring traffic condition (road speed) and prediction of bus arrival.

•  Challenge is to rightly analyze GPS data, which is typically high data throughput and difficult to capture

Benefits •  Monitor 600 buses across 150 routes daily •  Analyzes 50 bus location updates per

second , using InfoSphere Streams •  Collects, processes, and visualizes location

data of all public transportation vehicles

Exploration, landing and

archive

Enterprise warehouse

Information governance

Real-time analytics

Data mart

Analytic appliances

Information ingestion and operational information

Enhanced applications

Customer experience

Operations and fraud

Risk

Financial performance

New business models

IT economics

Data sources

SYSTEMS—SECURITY—STORAGE

Transaction and application data

Machine and sensor data

Enterprise content

Social data

Image and video

Third-party data

Enterprise warehouse

Data mart

Analytic appliances

Actionable insight

Reporting, analysis, content analytics

Predictive analytics and modeling

Decision management

Discovery and exploration

Cognitive

+ +

Reporting, analysis, content analytics

Real-time analytics

Architecture for traffic awareness system Real-time analytics to enhance customer experience

Customer experience

Operations and fraud

Machine and sensor data

Constant Contact Transforming Email Marketing Campaign Effectiveness with IBM Big Data

Need •  Analyze 35 billion annual emails to guide

customers on best dates & times to send emails for maximum response

Benefits •  40 times improvement in analysis performance

•  15-25% performance increase in customer email campaigns

•  Analysis time reduced from hours to seconds

Architecture for email marketing Analyze to maximize response rates

Exploration, landing and

archive

Enterprise warehouse

Information governance

Real-time analytics

Data mart

Analytic appliances

Information ingestion and operational information

Enhanced applications

Customer experience

Operations and fraud

Risk

Financial performance

New business models

IT economics

Data sources

SYSTEMS—SECURITY—STORAGE

Transaction and application data

Machine and sensor data

Enterprise content

Social data

Image and video

Third-party data

Enterprise warehouse

Data mart

Analytic appliances

Actionable insight

Reporting, analysis, content analytics

Predictive analytics and modeling

Decision management

Discovery and exploration

Cognitive

+ + Transaction and

application data

Enterprise content

Exploration, landing and

archive

+ +

Analytic appliances

Predictive analytics and modeling

Reporting, analysis, content analytics

New business models

IT economics

Battelle, helping reduce energy costs and enhancing power grid reliability and performance Need •  Assess the viability of one smart grid

technique called transactive control

Benefits •  Engages consumers and responsive assets

throughout the power system to help optimize the system and better integrate renewable resources

•  Provides the capability to analyze and gain insight from up to 10 PB of data in minutes

•  Increases grid efficiency and reliability through system self-monitoring and feedback

•  Enables a town to avoid a potential power outage

Architecture for smart grid Analytics to reduce costs and optimize grid

Exploration, landing and

archive

Enterprise warehouse

Information governance

Real-time analytics

Data mart

Analytic appliances

Information ingestion and operational information

Enhanced applications

Customer experience

Operations and fraud

Risk

Financial performance

New business models

IT economics

Data sources

SYSTEMS—SECURITY—STORAGE

Transaction and application data

Machine and sensor data

Enterprise content

Social data

Image and video

Third-party data

Enterprise warehouse

Data mart

Analytic appliances

Actionable insight

Reporting, analysis, content analytics

Predictive analytics and modeling

Decision management

Discovery and exploration

Cognitive

+ +

Machine and sensor data

Analytic appliances

Predictive analytics and modeling

Decision management

Real-time analytics

Customer experience

New business models

Operations and fraud

Trust is the most important aspect of a Big Data solution

Create foundation of trusted data

Understand usage and monitor compliance

Model exposure and understand variability

Trust the facts Ensure privacy and security

Make risk aware decisions

Trust It Be proactive about privacy, security and governance

•  Data  ac&vity  monitoring  of  NoSQL,  Hadoop,  and  Rela&onal  Systems  

•  Masking  of  sensi&ve  data  used  in  Hadoop  

   •  Ensures  sensi&ve  data    is  protected,  encrypted  and  secure  

Big Data Privacy and Security Protect a Wider Variety of Sources

InfoSphere Optim

InfoSphere Guardium

RDBMS

Hadoop

NoSQL

Data Warehouses

Application Data and Files

•  Protec&on  is  a  pre-­‐requisite  for  the  fundamental  assump&on  of  big  data  –  sharing  data  for  new  insight  

•  Automa&on  enables  protec&on  without  inhibi&ng  speed  

 

Data volumes continue to grow fast. New technologies will be needed in the future

Big Data brings Technical Challenges IBM invests in future solutions

Huge Volumes of Multimedia Data: New Storage Requirements

The Challenge of Moore’s Law: New Storage Methods Needed

IBM’s Approach: Start at the Atomic Level

how many atoms are needed to store

1-bit of data

What are your activities to leverage Big Data Analytics?

Your big data journey IBM can help wherever you are

Educate  Focused  on  knowledge  gathering  and  market  observa&ons  

25% Explore  Developing  strategy  

and  roadmap  based  on  business  needs  and  

challenges  

47% Engage  Pilo&ng  big  data  

ini&a&ves  to  validate  value  and  requirements  

Execute  Deployed  two  or  more  big  data  ini&a&ves  and  con&nuing  to  apply  advanced  analy&cs  

22% 6%

Learn  the  technology  &  gain  exper&se  

Validate  and  realize  business  value  Case  studies,                    Whitepapers  and  Value  Repors  ibmbigdatahub.com  

IBM  Briefings,  Solu&on  Centers  

Learning,  explora&on  downloads  &  test  BigDatauniversity.com,  YouTube  Big  Data  Channel  

                         IBM  Roadmap                          Workshop  • Priori&sed  business  use  cases  • Recommend  plaRorm  

           Solu&on  Design  &                Proof  of  Concept      • Validate  business  value  • Demonstrate  capabili&es  

             Enterprise-­‐wide                              big  data  ini&a&ves  • Stampede  –  exper&se  and  skills  to  get  value  straight  away  

• Enterprise  data  plaRorm  

THINK