business analytics big data integration zenterprise zos working group belux - eric... · •...

© 2014 IBM Corporation

The Evolution of Business Analytics and Big Data Integration on zEnterprise

[email protected]

Executive Architect

11 June 2014

mailto:[email protected]


Agenda of Today

2


Topics in this Session

1. The critical need for Business Analytics

2. The challenges that arise from Business Analytics Environments

3. zEnterprise as a Business Analytics Hub

4. Integrating Big Data with zEnterprise, today and in the future

5. More information


Analytics has evolved from a Business Initiative to a Business Imperative

Respondents who believe analytics creates a competitive advantage

57%

increase

More organizations are using

Analytics to create a competitive

advantage

2010

58% 2011

37%

Source: The New Intelligent Enterprise, a joint MIT Sloan

Management Review and IBM Institute of Business Value

analytics research partnership.

Copyright © Massachusetts Institute of Technology 2011

1.6x Revenue growth

2.0x EBITDA(*) growth

2.5x Stock price appreciation

Source: Outperforming in a data-rich, hyper-connected

world, IBM Center for Applied Insights study conducted

in cooperation with the Economist Intelligence Unit and

the IBM Institute of Business Value. 2012

Analytics Leaders are outperforming

their competitors in key financial

measures

4 (*) EBITDA = Earnings Before Interest, Raxes, Depreciation and Amortizaton

1.

Th

e c

ritica

l n

ee

d fo

r B

usin

ess A

na

lytics


[There is a growing desire to “weave” Analytics into the fabric of core business applications]

Sales

Marketing

Customer

service

Finance

IT

Operations

Product development

Human resources

Manage regulation, compliance and risk

Grow, retain and satisfy customers

Transform processes in finance

Increase efficiency in operations

5

Goal: achieve better business outcomes

1.

Th

e c

ritica

l n

ee

d fo

r B

usin

ess A

na

lytics


Companies want to focus on high-value initiatives

in core Business Areas

• Advanced client segmentation

• Leveraging customer sentiment

analysis

• Reducing customer churn

• Optimizing the supply chain

• Deploying predictive maintenance

capabilities

• Transform threat & fraud identification

processes

Operations

• Enabling continuous planning and

forecasting

• Automating financial and management

reporting

• Improving visibility, insight and control

Finance

• Making risk-aware decisions

• Managing financial and operational

risks

• Reducing the cost of compliance

Risk

4

2

3

Examples

1 Customers

6

1.

Th

e c

ritica

l n

ee

d fo

r B

usin

ess A

na

lytics


Shifting expectations impact severely the Business Analytics Requirements

Customers

(e.g. external, Web)

Customer Service & Support

(e.g. call centers, sales personnel)

Company

Management

Analysts

(e.g. Mktg, Research)

C Level

Mgt.

Number

of Users

User Community

Transaction

Volume

Transaction

Type

Few

Many

Small

Very Large

Complex

Simple

Availability

Less

Important

Critical

Analytics

Market is

Expanding

7

1.

Th

e c

ritica

l n

ee

d fo

r B

usin

ess A

na

lytics


Data must be brought back under control in order to meet their key Business Analytics objectives

Source: A commissioned study conducted by Forrester Consulting on behalf of IBM, October, 2012

Base: 207 Global enterprise data analytics professionals

© 2012 Forrester Research, Inc. Reproduction Prohibited

Our data is scattered in too many places (top response)

57%

What are the 3 largest

problems with your

firm’s Business

Analytics?

Above all else, our data must be secure

(tied for top response)

89%

Please rate the importance of the following

with regard to your informational needs

When we access data it should appear as a single

source of the truth (tied for top response)

89%

8

1.

Th

e c

ritica

l n

ee

d fo

r B

usin

ess A

na

lytics


[The top Business Analytics solution priorities]

Select the top 3 improvements you would make to your current

business intelligence and analytics strategy in order of importance.

(Select up to three)

Make the security of our data iron-clad to put it

beyond the reach of malicious users

Combine data in new ways to improve customer

service and create new business opportunities

Create immediate, normalized, and

transparent views of our data

Reduce the effort / speed our ability to comply

with regulatory reporting requirements

Reduce the total cost of ownership

of the environment.

60%

54%

52%

51%

28%



© 2012 Forrester Research, Inc. Reproduction Prohibited 9

1.

Th

e c

ritica

l n

ee

d fo

r B

usin

ess A

na

lytics

© 2014 IBM Corporation 10

The common approach to Business Analytics

Business

Analytics

Solution

DW & BA Administrators & Report Authors

Data Warehouse

/ Data Mart / ODS

Division ‘A’ Marketing & R&D Users

Division ‘B’ Marketing & R&D Users

Operations

Sales Finance

Executive

Management

10

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


A look at the core elements for a Business Analytics Solution

Business Analytics: •Business Intelligence •Predictive Analytics

ETL: Integrate, cleanse, transform, & load production data to Data Warehousing solution

• Data Warehouse • Operational Data

Store • Data Mart(s)

Transactional Data: •DB2 for z/OS •IMS, VSAM •Non IBM

Data In Data Movement & Cleansing

Data Warehousing

Data Analysis

Customer Interaction

Analytics Out

11

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


Today’s typical Enterprise Analytics Architecture

Sales

Marketing

R&D

Finance

Operational Systems

Transactional (DBMS, Simple Files)

Continuous feed

Departmental Business Analytics Systems Data Systems

ODS(RDBMS)

Departmental Data Marts

Enterprise Data Warehouse

Hourly, daily, weekly, monthly, batch process

Staging Area Transformation

servers

Data Mover

Sales

Finance

Marketing

Sales

Finance

R&D

Marketing

Sales

12

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


Business Analytics Life Cycle – Asynchronous and Distributed

13

Scoring Rules

Operational Systems

Staging

Area

Transformation Server

Data Mover

Departmental Data Marts

Win/Unix/Linux server

Batch

Process

Data Mining

Segmentation

Prediction

Statistical Analysis

Multi-

Dimensional

Analysis

MIS System

Budgeting

Campaign management

Financial Analysis

Selling Platforms

Customer Profit Analysis

CRM

Batch

Process

Analytical

Foresight

Online Queries & Reporting

Optimized Business Processes

Customer Support

Claims Processing

Underwriting

Fraud Management

Sales Effectiveness

Marketing

Staging

Area

Analytics

Server

Bulk

Intel/Unix/Linux Server

Win/Unix/

Linux Server

Win/Unix/

Linux Server

Enterprise Data Warehouse

(RDBMS)

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


Deployment Topology must be designed for each individual Application “Operational Model”

This architecture uses the loadbalancing and failover features that are built into the Cognos BI software architecture. Reporting load is spread across two Reporting servers and should one fail, the other will take over. This is the basis for a scaleable architecture.

50 Concurrent Users

14

• On which node do we deploy the software (function)?

• On which node do we deploy the data?

• What are the physical characteristics of each node?

What are the

Non-Functional

Requirements?

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


The Operational Model must be multiplied across each Environment

Development

Production

Quality Assurance

Disaster Recovery

15

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


The Operational Model may be multiplied across Multiple Departments

Development

Quality Assurance

Production

Disaster Recovery

Development

Quality Assurance

Production

Disaster Recovery

Development

Quality Assurance

Production

Disaster Recovery

Development

Quality Assurance

Production

Disaster Recovery

R&D

Marketing

Sales Finance

16

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


Potential issues which may limit Analytics ROI

Data transfer is limited to off peak times

Insufficient processing power to support large or complex queries

Each depart. needs to finance & acquire the HW, SW admin, facilities, training & support to meet demand

Growth = re-engineering

Duplicate environments needed for development, test, production, high availability are multiplied over applications and lines of business

Servers are run at sub-capacity

Complexity of multiple infrastructures is impacting effectiveness of DR, admin., audit ability, compliance, etc.,

Inconsistency of security controls across duplicated data

Multiple copies of the data are being created 17

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


[Executives recognize that the historic multi-platform approach for Analytics impedes moving forward]

Please evaluate each of the following statements with regard to your analytics platform:

We are looking to cloud as a way to provide greater agility in meeting analytic requirements

Having different technical environments for transactional systems and analytic systems

increases cost

Data sharing across platform silos makes it difficult to get an enterprise view

Data sharing across platform silos is excessively expensive

5 = Strongly Agree 4 = Agree

24% 44%

21% 49%

20% 42%

18% 42%

18



© 2012 Forrester Research, Inc. Reproduction Prohibited

2. T

he

ch

alle

ng

es th

at a

rise

fro

m B

usin

ess A

na

lytics E

nviro

nm

en

ts


System z: from Data Hub to Analytics Hub

19

Cognos

Differentiate with {DB2 for z/OS, System z Hardware} to integrate analytics with real time OLTP

Superior End-To-End Analytics Lifecycle Integration

Better business response,

reduced data movement, reduced complexity, reduced configuration resources,

more accurate, more secure, more available

InfoSphere

Infoserver

InfoSphere

Warehouse

SPSS

CPLEX

WebSphere

CICS/IMS

SAP

ISV Bus. App.

ILOG

Leverage System z

Operational Data Store

Operational Transactional data

Operational Analytical Data

Scoring Rules

SPSS

19

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


Data In

Business System / OLTP

Data Warehousing • Data Warehouses • Operational Data Stores • Data Marts

Business Analytics • Business Intelligence • Predictive Analytics

IBM zEnterprise Analytics Hub End-to-End Integrated Foundation for modern analytics

Insight Out Bring your analytics to your data:

• 70% of the data used for

analytics originates on

zEnterprise

Easily delivers on modern analytics

requirements for:

• Timely, accurate and secure

• Superior availability, scalability

and performance

• Rapid deployment and

expansion

• Reduced cost and complexity

Evolves with your business:

• Start where you want and grow

without re-architecting

20

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


A large percent of the data that is accessed for

analytics originates/resides on IBM zEnterprise

2/3 of business transactions for U.S. retail banks

80% of world’s corporate data

Businesses that run on zEnterprise

66 of the top 66 worldwide banks

24 of the top 25 U.S. retailers

10 of the top 10 global life/health insurance

providers

1,300+ ISVs run zEnterprise today, more than 275 of

these selling over 800 applications on Linux

The downtime of an application running on System z

equates to approximately 5 minutes per year

The System z mainframe can run over a thousand

virtual Linux images on a single frame the size of a

refrigerator

[The role of zEnterprise in Business Analytics]

21

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub

© 2014 IBM Corporation Real-Time Score/

Decision Out

ETL

Data In

R-T, min, hr, wk, mth

Business System /

OLTP

Data Warehouses, Operational Data

Stores, Data Marts

Business Analytics •Business Intelligence •Predictive Analytics

zEnterprise: an end-to-end,

integrated, rock-solid foundation

for business analytics

Analytics Out

Customer Interaction

What Sets zEnterprise Apart for Analytics?

22

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


23

[What Sets zEnterprise Apart for Analytics?]

High security (EAL5+)

High availability (99.999% )

Performs at 100% capacity

Prioritization of critical

queries & workloads

Integrated disaster recovery

Centralized, scalable

infrastructure

Virtualization

Start with your final

architecture

Processors, disk,

memory added

dynamically without

outage

Pre-install then activate

as needed

Flexible deployment

options

Co-location of data

warehousing, business

analytics, transactional

data

Reduced data

movement

Lower latency and near

real time data

Rapid acceleration of

complex queries

Superior availability, scalability and performance

Timely, accurate and secure information

Rapid deployment and expansion

Reduced costs and complexity

zEnterprise: makes the technical challenges the easy part!

23

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


IBM Cognos Business Intelligence for zEnteprise

An analytics offering that provides the breadth of business intelligence

capabilities required to supports all users from mission critical to tactical to

strategic

Provides the qualities of service required for mission critical analytics

Enables organizations to deliver

BI as a service

Provides access to the Big Data within:

Analyze the Variety of data the Business demands

24

– IBM BigInsights, Apache Hadoop

and Cloudera, Amazon Web

Services’ Elastic Map-Reduce

service (AWS EMR)

– IBM InfoSphere Streams for both

real-time processing and monitoring

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


Smart Analytics Cloud for large enterprises

This offering transforms the delivery of business intelligence into a service that is

readily available and affordable to corporate users across and beyond the enterprise

Report Authors

Cloud Administrators

BI Administrators

Smart Analytics Cloud

WAS

z/VM for Dev/Test

Cognos

Linux

DB2

WAS

z/VM for Production

Cognos

Linux

DB2

Cloud

boarding

application

Data Mart /

Warehouse

Finance

Line of Business ‘B’

Marketing R&D

Data Mart / Warehouse

Senior

Executives

Large enterprise organization

z/VM and IBM-Wave for Cloud management

Ops Mgr

for z/VM

VM Perf

Toolkit

Tivoli

Automation

Tivoli

Monitoring

z/VM

Tivoli

usage

metering

Linux Data Mart /

Warehouse

Operations

Data Mart /

Warehouse

HR

Line of Business ‘A’

Marketing R&D

Data Mart / Warehouse

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


Solution Edition for Enterprise Linux Server

26

The Enterprise Linux Server (ELS) is a System zEC12, zBC12,

z196 or z114 machine configured to run Linux-only workloads : IFL

− 2 to 13 for zBC12, 2 to 10 for z114

− 6 to 80 for z196, 6 to 101 for zEC12)

Hardware maintenance for three to five years

z/VM software with three to five years of subscription and support

Intended to allow for VERY competitive acquisition

Suse / RedHat zBC12 Linux licences promotions!

Solution Edition for Enterprise Linux Server provides attractive pricing for

consolidating Linux Applications and particularly for Database Server workloads

= Reduced

Cost

VIRTUALIZATION

+ ENERGY

EFFICIENCY

AUTOMATION +

SECURITY &

RESILIENECY +

Dynamic

Infrastructure

Extended with “ELS for Analytics”

Extended with ”Cloud Ready for Linux on z”

For customers who have database workloads such as Oracle or DB2

LUW on Linux on System z

Provides Cognos and SPSS in a Linux environment that can access a

customer’s database on Linux on z, z/OS, or databases on separate

UNIX/INTEL servers that are network attached to the zEnterprise

environment

This offering can be extended with a PureData for Analytics network

attached server, to offer exceptional query speed to these customers

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


Predictive Analytics on zEnterprise

SPSS Decision Management for Linux on System z

SPSS Collaboration and Deployment Services

for Linux on System z

Employs both predictive models and business rules to

automatically generate recommended actions

Provides role-based models and security for in scoring,

job scheduling, repository services, and integration

Predictive

Customer Analytics

Acquire

Grow

Retain

Predictive

Operational Analytics

Manage

Maintain

Maximize

Predictive

Threat & Risk Analytics

Monitor

Detect

Control

Decision Management Modeler Statistics

Collaboration and Deployment Services

IBM SPSS Statistics for Linux

on System z

IBM SPSS Modeler for Linux

on System z

Apply math to decision making and research for

commercial, government,

and academic users

Data mining tool used for generating hypotheses and scoring

Text analysis for unstructured data to model consumer behavior

IBM SPSS Modeler with Scoring Adapter for zEnterprise

Data mining tool used for generating hypotheses and scoring

Text analysis for unstructured data to model consumer behavior

In-Transaction Scoring with DB2 z/OS: Embeds the Scoring Algorithm

Directly within the Transactional Application

27

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


Scoring Options with an OLTP Application

1) Historical Scoring

DB Application

Historical Data store

Scoring and Modeling

Application

R-T, min, hr, wk, mth

ETL Copy

Delivers a static score

Batch Copy (nightly, weekly, monthly)

score value insert to DB

2) Real-time Scoring

2b) In-DB scoring

1a) Web Services scoring Call/Response

Delivers a dynamic

score

OLTP App

OLTP DB

28

• Excellent accuracy, speed and performance while reducing cost and complexity

• Improves accuracy by scoring new and relevant data directly within the OLTP application

• Scales to large data volumes to improve accuracy of data models

• Delivers the performance needed to meet and exceed SLAs of OLTP applications

• Minimizes demand on network, HW, SW and resources

Delivers better, more profitable decisions, using the latest data, at the point of customer impact • Enables more informed customer interaction • Improves fraud identification and prevention

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


Can you really scale to support the volume of an OLTP application without impacting performance?

29

Meets most demanding workload

Required: 3K to 5K trans./sec.

Measured: > 12K trans./sec.

Provides best value

Most accurate score is calculated in real time

0

2000

4000

6000

8000

10000

12000

14000

2 4 8

Inte

rnal

Th

rou

gh

pu

t R

ate

(IT

R/s

ec)

Number of CP Engines

SPSS Scoring Adaptor for DB2 for z/OS

z196 and zEC12 Throughput

z196 score

zEC12 score

Meets stringent SLA requirement

Small additional CPU cost to score

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


Degree of Complexity

Com

petitive A

dvanta

ge

Standard Reporting

Ad hoc reporting

Query/drill down

Alerts

Forecasting

Simulation

Predictive modeling

Optimization

What exactly is the problem?

How can we achieve the best outcome?

What will happen next?

What will happen if … ?

What if these trends continue?

What actions are needed?

How many, how often, where?

What happened?

Reporting

Stochastic Optimization How can we achieve the best outcome

including the effects of variability?

Descriptive

Prescriptive

Predictive

Business Analytics Styles and Solutions

Increasing prevalence of compute and data intensive parallel algorithms in commercial workloads

driven by real time decision making requirements and industry wide limitations to increasing thread speed

ILOG BRMS Business Events

Analytics

WebSphere Operational

Decision Management

30

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub

http://www.spss.com/


DB2 Analytics Accelerator for z/OS Blending zEnterprise and Netezza technologies

31

Fast

Complex queries run up to 2000x faster while retaining single record lookup speed

Cost Saving

Eliminate costly query tuning while offloading complex query processing

Appliance

No applications to change, just plug it in, load the data, and gain the value

A high performance analytics accelerator appliance

for IBM zEnterprise, delivering dramatically faster

complex business analysis transparently to all users

Times

Faster

Query

Total

Rows

Reviewed

Total

Rows

Returned Hours Sec(s) Hours Sec(s)

Query 1 2,813,571 853,320 2:39 9,540 0.0 5 1,908

Query 2 2,813,571 585,780 2:16 8,220 0.0 5 1,644

Query 3 8,260,214 274 1:16 4,560 0.0 6 760

Query 4 2,813,571 601,197 1:08 4,080 0.0 5 816

Query 5 3,422,765 508 0:57 4,080 0.0 70 58

Query 6 4,290,648 165 0:53 3,180 0.0 6 530

Query 7 361,521 58,236 0:51 3,120 0.0 4 780

Query 8 3,425.29 724 0:44 2,640 0.0 2 1,320

Query 9 4,130,107 137 0:42 2,520 0.1 193 13

DB2 Only

DB2 with

IDAA

IDAA V3 Enhancements

• High Performance

Storage Saver

• Incremental Update

• New optimization of DB2

unloads and

table/partition refreshes

• High Capacity up to 1.28

PB for a single

Accelerator

• More queries eligible for

acceleration

…and DB2 for z/OS ~ A highly tuned database for OLTP & Analytics

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub

IDAA V4 Enhancements

• Static SQL support

• Multiple encodings

• Multi-row fetch

• Workload Balancing and

High Availability

• 10 Subsystems replicate

to 1 Accelerator

• Richer monitoring

• HPSS enhancements

• DB2 11 support


IDAA can Serve as Enterprise Analytics Accelerator

32

DB2 DB2 DB2 DB2

BW non-BW BA Legacy

A single IDAA instance can be connected to multiple DB2

subsystems running various different applications.

• Applications access data exclusively via DB2, thus there is no

need to change the interface.

• Broad range of applications ensures fast ROI

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


[IBM zEnterprise Analytics System 9700/9710] Mixed Workloads for Next Generation Business Analytics

33

The next generation of System z

analytics; an integrated solution of

hardware, software and services

that enables customers to rapidly

deploy cost effective game

changing analytics across their

business.

Preselected

All the necessary

components are

identified and integrated

into an end-to-end

solution

Pretested

Over 20 different

customer typical

configurations are pre-

sized and tested

Solution Priced

Aggressively priced for a

cost-effective add-on or

new deployment for

customers with critical

data operations

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


Analytics

Accelerator

System z z196 / EC12 z/OS Operating System Stack

DB2 for z/OS

ETL/ELT

Flexibility in Critical Data Decision Systems

Operational Source Systems

Common Metadata

Organized for simplicity

and functionality

34

Data

Warehouse

Data

Mart

ODS DS8870

• Delivers a

PACKAGED

SOLUTION

• Leverages IBM

industry leading

software

• Customized with

optional

components

• Priced using

”Solution Edition”

concept

IBM zEnterprise Analytics System 9700 3

. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub

• Preselected

• Pretested

• Solution

Priced


ETL/ELT

Operational Source

Systems

Cost Effective Critical Data Decision Systems

Data

Warehouse

Data

Mart

ODS

System z z114/zBC12 z/OS Operating System Stack

DS8870

DB2 for z/OS

DB2 Analytics

Accelerator

Organized for simplicity

and functionality

35

• Delivers a

PACKAGED

SOLUTION

• Leverages IBM

industry leading

software

• Customized with

optional

components

• Priced using

”Solution

Edition” concept

IBM zEnterprise Analytics System 9710 3

. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub

• Preselected

• Pretested

• Solution

Priced


DB2 for z/OS and

IBM DB2 Analytics Accelerator

• Simplification

• Single point of

entry

• Reduced Data

Movement

• High fidelity data

• Competitive

price/performance

• Dynamic routing for

most efficient fit for

purpose runtime

architecture

• Combines the

strengths

of both System z

and Netezza

technology

• Single

environment for

security, logging,

back-up, and

recovery

* Advanced analytics is supported by Netezza but not currently exploited by DB2 Analytics Accelerator

OLTP Transactions

Operational analytics

Real time data ingestion

High concurrency

Advanced analytics*

DB2 Native

Processing

Standard reports

Complex queries

Historical queries

OLAP

OLTAP = Single Workload-Optimized System for OLTP, DW, Historical Data DB2 for z/OS with Analytics Accelerator

On-Line Transactional and Analytics Processing

36

3. zE

nte

rprise

as a

Bu

sin

ess A

na

lytics H

ub


What is Big Data?

37

Google can give you nearly 1.73 Billion options Vendors have even more definitions

Here is how Gartner defines “Big Data”

“Big data is high-volume, high-velocity and high-variety

information assets that demand cost-effective, innovative information processing for

enhanced insight and decision making”

Big Data … data from web logs, RFID, sensor

networks, social networks, social

data, Internet text and documents, Internet search

indexing, call detail records, astronomy,

atmospheric science, genomics,

biogeochemical, biological,

interdisciplinary scientific research,

military surveillance, medical records,

photography archives, video

archives, and large-scale e-commerce

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


The biggest Information Technology inflection point in 35 Years!

38

Business models

under constant

pressure

Customers are more

demanding and

connected

Great relationships

trump great products

Excitement

Confrontation

Fear

Emotions

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


Beyond the 3 traditional “V’s”

39

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


Examples of Big Data Applications

40

Each day 12 Billion Terabyte of

tweets

Imagine you could analyze them each

day to figure out what people are saying

about your products

Imagine that hospitals could take the thousands

of sensor readings collected every hour per

patient and identify subtle indications that the

patient is becoming unwell

Imagine you can

decide whether

someone qualifies for

a mortgage in minutes,

by analyzing many

sources of data, while

the client is still on the

phone or in the office

Imagine if green energy

company could use petabytes

of weather data along with

massive volumes of operational

data to optimize asset location

and utilization

Imagine if law

enforcement agencies

could analyze audio

and video feeds in real-

time without human

intervention to identify

suspicious activity

As new sources of data continue to

grow in volume, variety and velocity, so

too does the potential of these data to

revolutionize the decision making

process in every industry !

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


The “low risk” Big Data Starting Point

Where are organizations getting the most return on Big Data projects?

Source: IBM Global Big Data Online Survey

41

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


The results of a Webcast Survey …

42

Have you already implemented or are you planning to implement any Big

Data based initiatives within the next 6 months?

How would you rate the value of being able to integrate insights from

social media, telemetry, unstructured data into your analytics and

decision making processes?

Do you see the IBM System z platform as pivotal to the success of Big

Data initiatives?

Yes 31%

No 69%

Yes 90%

No 10%

High 50%

Medium 36%

Low 14%

You do not

need to build

your Big Data

infrastructure

from scratch !

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


Enterprise Integration and Governance… The key to success of incorporating Big Data

43

Information Integration

–Insights from Big

Data must be

incorporated into the

Data Warehouse and

Analytics/Decision

engines

Information Governance

–Companies need to

govern what comes

in, and the insights

that goes out

Big Data Platform Data Warehouse

Enterprise Integration

Traditional Sources New Sources

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


Big Data Exploration Find, visualize, understand

all Big Data to improve decision making

Enhanced 360o View of the Customer

Extend existing customer views (MDM, CRM, ERP) by incorporating additional

internal and external information sources

Operations Analysis Analyze a variety of machine data for improved business

results

Data Warehouse Augmentation Integrate Big Data and data warehouse

capabilities to increase operational efficiency

Security/Intelligence Extension

Lower risk, detect fraud and monitor cyber

security in real-time

Typical Current Big Data Use Cases

44

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


Hadoop, the Popular Big Data Technology

45

Open Source project of the Apache Foundation

Written in Java, originally developed by Doug Cutting

who named it after his son’s toy elephant

Optimized to handle...

– Massive amounts of data through parallelism

– A variety of data (structured, unstructured, semi-structured)

– Using inexpensive commodity hardware

Great performance

HDFS Reliability provided through replication

across different computers

MapReduce Engine Perform Calculations

• Not to process

transactions (random

OLTP access)

• Not good when work

cannot be parallelized

• Not good for low latency

data access

• Not good for processing

lots of small files

• Not good for intensive

calculations with little data

(OLAP)

Hadoop complements

existing RDBMS technology !

• Compression applied

when data are shipped

• “Cheap Commodity

Play”

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


Accelerators

Information Integration & Governance

Data

Warehouse

Stream

Computing

Hadoop

System

Discovery Application

Development

Systems

Management

BIG DATA PLATFORM

InfoSphere Data Explorer

Find, navigate, visualize all data

Accelerators

Speed time to value with analytic and application

accelerators

InfoSphere BigInsights

Bringing Hadoop to the enterprise

InfoSphere Data Warehouse

Delivers deep insight with advanced database

analytics & operational analytics

Information Integration and Governance

Governs data quality and manages the information

lifecycle

InfoSphere Streams

Analytics for data in-motion exploration

IBM BIG DATA PLATFORM Logical platform with many physical deployment options

A Big Data Platform is an essential component of a broader Big Data and Analytics Strategy

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


Highest Quality Data Services for Big Data

47

InfoSphere BigInsights for zBladeCenter Extension

for exploration & online archiving

DB2 Analytics Accelerator powered by Netezza technology

for reporting & analytics and operational analytics

DB2 for SQL & NoSQL transactions

with enhanced Hadoop integration in DB2 11 (beta)

zEnterprise

IMS for highest performance transactions with enhanced Hadoop integration

in IMS 13 (beta)

• A NoSQL or “Not Only SQL” database

provides a mechanism for storage and

retrieval of data that use looser consistency

models than traditional relational databases

in order to achieve horizontal scaling and

higher availability

• Highly optimized for retrieval and

appending operations and often offer little

functionality beyond record storage

• Aimed at scalability and performance for

certain data models

• Useful when working with a huge quantity

of data (Big Data) when the data's nature

does not require a relational model 4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


Enhancing Big Data Analytics with IMS and DB2 for z/OS

48

Unstructured data sources

are growing fast

• New user-defined

functions and generic

table UDF capability

Relational projection

of IMS model

• Two significant needs:

1. Merge this data with trusted OLTP data from zEnterprise data sources

2. Integrate this data so that insights from Big Data sources can drive business actions

DB2 is providing the connectors and the database capability to allow DB2 applications to easily and

efficiently access Hadoop data sources

IMS & DB2 provide the connectors and the database capability to allow BigInsights to efficiently

access each data source

Merge

Integrate

Much of the world’s operational

data resides on z/OS

IBM BigInsights

Phase 1

Phase 2

4. In

teg

ratin

g B

ig D

ata

with

zE

nte

rpri

se

, to

da

y a

nd

in

th

e fu

ture


More Information

• http://www-01.ibm.com/software/os/systemz/badw/

• http://www-01.ibm.com/software/data/bigdata/z/

• http://www-01.ibm.com/software/data/db2imstools/solutions/information-

governance.html

• http://www.bigdatauniversity.com/

• http://www.redbooks.ibm.com/abstracts/sg248180.html?Open

• http://www.redbooks.ibm.com/abstracts/sg247892.html

49

5. M

ore

in

form

ation

Cloud Ready

Data Ready

Security Ready

http://www-01.ibm.com/software/os/systemz/badw/



http://www-01.ibm.com/software/data/db2imstools/solutions/information-governance.html

http://www-01.ibm.com/software/data/bigdata/z/











http://www.bigdatauniversity.com/

http://www.bigdatauniversity.com/

http://www.redbooks.ibm.com/abstracts/sg248180.html?Open

http://www.redbooks.ibm.com/abstracts/sg248180.html?Open

http://www.redbooks.ibm.com/abstracts/sg247892.html

http://www.redbooks.ibm.com/abstracts/sg247892.html


Thank You for Your Attention

50

business analytics big data integration zenterprise zos working group belux - eric... · •...

Documents