analytics on z systems focus on real time - hélène lyon

© 2016 IBM Corporation

IBM Analytics on z Systems

NRB Mainframe Day 2016

Analytics on z Systems

Focus on Real-Time

Hélène Lyon – Distinguished Engineer & CTO, z

Analytics & IMS for Europe

IBM Systems, zSoftware Sales, Europe

Email: [email protected]

Tel: +33 6 84 64 19 39

mailto:[email protected]


Executive Summary

Analytics is one of the hot-button topics for agile enterprises.

– End users are looking more than ever for relevant and accurate information for

which their enterprise assets are the fundamental enabler.

– Challenges are about removing the time lag between analytics and business

impact.

One way is to augment Data Warehouse capabilities to provide Right-Time

Data Analytics including Big Data support.

– Business use case example: Improved product development and sales by

getting improved knowledge of customers past behavior.

Another way is to implement in-line analytics, aka in-transaction analytics

aka “Analytics in Business Applications” aka Real-Time Analytics

– Add analytics in z/OS based workload to improve its business value

– Business use case example: Fraud detection at the time of the fraud

2

Bring the analytics to the data for Right-Time Insight

Bring the analytics in the application for Real-Time Decision Management!


Market Trends & Analytics Needs

End customers /

consumers

Aim to improve level of

detail and frequency of

existing analytics

Have innovative ideas for

new analytics

Are driven by increasing

demand in regulatory

requirements

Lines of business/

Data scientists

both have a high demand in secure

data management

• Transactional

workload runs on the

mainframe

• Very bad perception

of the mainframe

with regards to

modernity

• DW / analytical data environment on distributed platforms

• Huge amount of analytics applications

Business IT

Overnight

batch / ETL

for analytical

purposes

Scoring Rules

A

Analytics ComponentsA

Analytics

executed

outside of

core

business

applications

zDatazApps

3

Drive increased

transactional workload

through mobile

Ask for more

personalized services

and offering


z Systems Analytics Areas – Evolution & Extension

zData

zApps

Core Business Applications and Data

ODS

EDW &

Data

Mart

ETL or ELT

ETL or ELT

IT Data

A

Today

Only Data

Analytics

z/OS based Transaction & Batch Workloads

“Transaction & Apps” Data

External Data & Master Data

“IT” Data


z Systems Analytics Areas – Evolution & Extension

Optimized

Analytics

Infrastructure

with z in mind

zData

zAppsz/OS based Transaction &

Batch Workloads

“Transaction & Apps” Data

Core Business Applications and Data

ODS

EDW &

Data

Mart

ETL or ELT

External Data & Master Data

ETL or ELT

“IT” Data IT Data

New Business Critical AnalyticsAdd, Extend, Modernize Analytics Services

Analytics in Business

Applications

or

In-Transaction Real Time

Analytics

Right-Time Data Analytics

including “Big Data”

for both structured and

unstructured data

Analytics on “IT” Data

Enhanced

data

Analytics

Platform

ODS, DW,

DM

Data lakes

…

A

A

A

A

A Analytics Components

Unstructured Data

Co

mp

ete

in

th

e c

og

nitiv

e e

ra w

ith

hig

h q

ua

lity,

rea

l tim

e in

sig

hts

ba

se

d o

n IB

M z

Syste

ms S

olu

tio

ns

Link to IBM zRTA page

5

http://www-03.ibm.com/systems/z/solutions/real-time-analytics/


z Systems Analytics Areas - How it complements existing Analytics Environments?

IBM

DB

2 A

naly

tics

Accele

rato

r

In transaction rules and

score executionIntraday capability for ad-hoc

queries & predictive analytics

Availability of historical data (in

raw format)

Accelerated reporting to

fulfill internal and regulatory

requirements

Ability to transform

data before offload to

DWH or reportingAbility to create new

SPSS models at any time

Quasi Real Time

availability of data

for analytics

Instant access to raw data

for new report generation in

hours instead of days

Load and merge of ANY non

DB2 z/OS data

Scoring Rules

A

zDatazApps

Scoring

Rules

Explore data to

uncover hidden

insights

6

A


Focus on “Analytics in Business Applications” What is “in-transaction real-time analytics?”

This is the time during which a transactional

event is still occurring

–Someone is shopping at a store

–Someone is on the phone with a customer

service representative

–An electronic payment is being processed

–…

By acting on threats or opportunities as they

arise…

–Revenues can be increased (up-sell, cross-

sell)

–Customer churn can be reduced

It’s about leveraging the power of analytics “in the transactional moment” to achieve a

more favorable outcome for a transactional event, while the event is in progress

The person visiting a

store buys more than he

or she otherwise would

have

What would have been an

over-payment is stopped

before it gets out the door

The person on the phone,

who was about to cancel

a service, instead re-ups

A commission of fraud is

stopped before it is

effected

7


Focus on “Analytics in Business Applications”What are Data Scientists doing?

DATA

Design

Build

Train

TestDeploy

Score

Act (rules)

BUILD / TRAIN / TEST phases– Data scientists choose their modeling

program (SPSS Modeler, SAS, R, Python) and the data to analyze. They train & test the models.

– They can use Spark to be more productive and benefit from more algorithms.

DEPLOY phase– Data scientists export model and deploy it

in any runtime environment, potentially z/OS.

SCORE phase– “in-transaction” scoring with an optimized

response time can ONLY be done in the z/OS transactional system.

– Frequently scoring is executed in batch, or as a service close to the data warehouse, but certainly not with a 200 milliseconds response time.

ACT phase– Business rules can be deployed to

replace or complement real-time scoring.– Colocation of rules with transaction is

needed to optimize performance.

The scoring and the rules execution can

take place in the transaction.

8


Business Challenge:

Improve fraud detection capabilities through added insight: real-time access to most current data,

integrated advanced analytics within the business flow, access to aggregated data across geographic

locations, merchants, issuer and history

Solution:

Enhance existing fraud detection with predictive scoring integrated with fraud detection transaction at

bank-transaction speeds and SLAs for *preventive* capabilities before the transaction completes

Benefit:

Institution can save significant amounts on fraud detected prior to transaction completion

Improved customer service using more advanced analytics, avoiding unnecessary account deactivations

Reduced call center costs

Able to integrate with institution’s existing fraud capabilities to provide enhancement

Transaction

Pre-

Authorization

SystemDetermine Fraud Risk

Continue, Pend/Investigate, etc.

Banking

Transaction

Initiated

Internal /

External

Fraud Rules

Engine

Transaction Data

Historical

Transaction

Data

IBM DB2

Analytics

Accelerator•Models are built outside z

Systems or on z Systems

•Models are deployed to

z/OS

Use Case: z/OS Optimized Fraud Detection for Banking Transaction

Enhanced with

Outcome from

Fraud Analytics

Alert

Data

Existing or Enhanced

Fraud Analytics

Rules mining &

predictive analytics

Other Data Other Data

Account Data

9


Use case: In-transaction, synchronous predictive analytics

• Validate inputs and channel

• Prepare for submission to System of Record

Initiate Transaction

• Invoke *existing* risk scoring mechanisms

• Get Predictive Score for transaction risk

• Determine Action

Determine Risk & Disposition • Continue

processing

• Stop Transaction

• Initiate alternate actions …

Determine Transaction Disposition

Risk Scoring Example

Customer

Account Data

Transaction

Data

Transaction

HistoryMerchant Data

In-transaction predictive analytics can address issues such as data latency, timely

execution, tight SLAs, governance of sensitive data, access to both transaction and

external data, skills gaps.

10


Use Case: IT Simplification to support growing demand with same SLAs

z Systems PR/SM

z/OS

CICS

PL1 App

DB2

Ru

ntim

eD

eve

lop

me

nt

Linux / VMware / Intel

Blaze Advisor

(WAS)

Binning Logic

PMML(Scoring model)

SOAP/HTTP

Linux / VMware / Intel

SAS

Deployment of PMML & Binning Logic

Binning Logic

PMML

(Scoring model)

Linux / Intel

JEE

Application Server

Java Split Logic

RMI/IIOP

z/OS

z Systems PR/SM

JVM

ODM Rules

Execution

Server

(zRES)

DB2

Java Split Logic

Binning Logic

CICS AOR

PL1 App

Trx

Data

Cust

Data

Acct

Data

z/O

S c

olo

ca

ted

Ru

ntim

e

PMML

(Scoring model)

Transaction Fraud Analysis done by FICO/Falcon &

Blaze Advisor on distributed servers

Scoring Model Creation with SAS

Transaction Fraud Analysis on z/OS


Model creation outside z/OS or in IDAA

Potential z/OS or non-z/OS service integration

In-Transaction Real-time Analytics technology pillars – RuntimeA 100% z/OS based solution – colocation & performance

Business

Rules

Policy

Regulation

Best Practices

Know-how

Business Critical

Queries

Tra

nsaction &

Ba

tch

Wo

rklo

ad

IBM DB2 Analytics Accelerator augment

analytics capabilities on historical data.

Predictive modeling, business rules and

orchestration together enable the most effective

decisions

API Programmable orchestration to coordinate

all activity in the same unit of work and ensure

best performance

1

2

3

Add-on

Other Analytics services

Orc

he

str

ati

on

wit

h S

imp

le

AP

I

Risk

Clustering

Segmentation

Propensity

Predictive Analytics

Model Execution

Predictive

Analytics

Model

Creation

Federated Data Analytics

12


Three technology pillars enable real-time analytics in a z/OS application runtime environment

– (1) Predictive analytics with Zementis partner solution deployed on z/OS – (2) Business Rules with Operational Decision Management – ODM on z/OS– (3) Business critical queries with IBM DB2 & DB2 Analytics Accelerator

Each of the three pillars just mentioned delivers significant value on its own.

Combined, they can work together with the data & event based exploration tools– Uncover patterns of events and behaviors– Use those patterns to develop models to predict future occurrences and identify threats

and opportunities as they arise

Optionally adding when SLAs for specific use case permit– Predictive analytics execution outside z/OS, for example in SPSS Collaboration and

Deployment Services– Integration with Spark solutions for federated data analytics, CPLEX mathematical

algorithm for Optimization and Watson-based cognitive services

In-Transaction Real-Time Analytics technology pillars - Runtime

An organization can go right to “ultimate” in-transaction real-time analytics

capability, or take a phased approach – and realize benefits at each intermediate

stage (and the order of enabler implementation is up to the client)

13


Business Benefits – Help LOB Innovate!

– Capability to richly analyze in real-time 100% of the transactions without performance

impact

– Agility to accept changes in business conditions by updating on the fly rules and models

– Auditability of business rules

– Easy to incorporate scoring in applications

IT Benefits

– When Performance Matters!

– No additional skills needed for COBOL/PLI development team

– Continous availability with non-disruptive operational analytics inherited from the core

systems (Hardware, operating system, CICS, IMS, DB2).

– Highest security level by keeping exchanged data within z Systems (no outbound data

transit)

– Simplified system management - the integrated architecture leverages existing

environment

In-Transaction Real-Time Decision solutions on z SystemsValue of deploying components on z/OS

Simple Performant AdaptiveIterative

14


Access to Mainframe data without moving it outside z Systems secured environment.

Reduce data latency Minimize Cost & Complexity Improve Data Governance & Security

A new approach for Enterprise Analytics with z Systems

Enrich transactions with real-time analytics allowing optimized decision management

Improve Business value of mainframe applications

Conviction 3: z Systems is the Most Securable for Data!

Conviction 4: Integrate Open Source technology in the z

Systems Environment

Conviction 2: Accelerate insight and simplify implementation with z13 & IBM

DB2 Analytics Accelerator

Centralized data security Tracking of activity to address audit and compliance requirements Highly available cryptography

Did you miss the LinuxONE announcement? Linux Your Way, Linux Without Limits, Linux Without Risk

Conviction 1: Bring the analytics to the data for Right-Time insight

Bring the analytics in the application for Real-Time Decision Management!

15


Video: In-transaction Analytics

https://www.youtube.com/watch?v=uPDfhYoQlHk

2.5 minutes

16

https://www.youtube.com/watch?v=uPDfhYoQlHk


Focus on Transactional Decision Management ODM, a Business Rules Engine (and more)

Operational Decision Manager can be used to extend application

functionality with improved agility, productivity, and consistency through

centralization of business rules and decision management

– What if you wanted your application to decide on how to react on a score

returned by a predictive model invoked in-transaction?

Operational

decisions

ApplicationApplication

Decision logic

Without ODM With ODM

Hard coded decisions: difficult to change

Impedes rules re-use by other systems

Externalized decisions: easy to change

Centralized decisions enable reuse and consistency

17


Focus on Transactional Decision Management ODM brings the IT and Business World together

“customer”

• the name of …

• the birthday of …

• the number of accidents

of …

• the … is a high risk driver

Business Object Model Rule Vocabulary Business Rule Language

Developer IT / Business Rule Developer / Business User

“client”

• le nom du ...

• l’anniversaire du ...

• Le nombre d’accidents du

...

• le ... est un conducteur à

risque …

01 CUST

05 NAME

05 AGE

05 NUMACCIDENTS

05 RISKLEVEL

Rule: High risk driver

if

the birthday of customer is after 12/9/1975 and

the number of accidents of customer is at least

3

then

set the customer as a high risk driver

Règle: Conducteur à risque

si

L’anniversaire du client est après le 12/9/1975

et

le nombre d’accident du client est au moins 3

alors

Classer le client comme conducteur à risque

Automatic generation

of the rule vocabulary.

Comprehensive industry

focused business terms

to define its data and

associated actions.

Localizable vocabulary

18


Focus on Predictive Analytics

Reduce Fraud /

Loss with detect &

prevent

Increase customer

retention

Drive up-sell and

cross-sell

Mitigate risk based

on predictions

A Universe of Data Example

• Pattern of consumer behavior over last 5 years of purchase

history: card present at time of sale

• Current transaction:

- Over $1000, card not present

- Merchant falls into category typical for this consumer’s

spend

• Likelihood of fraudulent transaction

• Develop insight to questions that are not binary ‘yes’ or ‘no’ likelihood of a pattern

• Predictive scores provide complementary insight to rules-based systems

19


Predictive Scoring - A 2 steps approach

AnalysesSegments

Profiles

Scoring models

...

Scoring Customer

Service

Center

Data:Demographics

Account activity

Transactions

Channel usage

Service queries

Renewals

…

Identify predictive

models/patterns found in

historical data

Use those predictive models

with variables to score

transactions & identify the

best possible future

outcomes

Practical scoring approachesOff-line: Batch Scoring

On-line: External scoring function

On-line: Within a transaction, in real time

Step 1 – Build the predictive model Step 2 – Execute the predictive model

SPSS Model

Creation enhanced

with IDAA V5SPSS

Model

Model Execution on

z/OS with Zementis

20


SPEED• Dramatically improve query response – up to

2000X faster – to support time-sensitive decisions• Right-time. Low latency. Trusted. Accurate.

SIMPLICITY• Simplify infrastructure, reduce ETL and data movement

off-platform • Non-disruptive installation

SAVINGS• Minimize data proliferation• Lower the cost of storing and managing historical data• Free up compute resources

SECURITY• Safeguard valuable data under the control and security

of DB2 for z/OS • Protected. Secured. Governed.

A workload optimized, appliance add-on to DB2 for z/OS that enables

the integration of analytic insights into operational processes to drive

business critical analytics & exceptional business value.

IBM DB2 Analytics Accelerator for z/OS


IBM DB2 Analytics Accelerator – Four Main Usage Scenarios

On average, 70% of the data that feeds data warehousing and

business analytics solutions originates on the z Systems

platform (financial information, customer lists, personal

records, manufacturing…)

Understand your workload and data:

Where transaction source data

is being analyzed todayUse Case Benefits

If the data is analyzed on the

mainframe

Rapid Acceleration of

Business Critical Queries

Performance improvements and cost

reduction while retaining System z

security and reliability

If the data is offloaded to a distributed

data warehouse or data mart

Reduce IT Sprawl for

analytics

Simplify and consolidate complex

infrastructures, low latency, reliability,

security and TCO

If the data is not being analyzed yetDerive business insight from

z/OS transaction systems

One integrated, hybrid platform,

optimized to run mixed workload.

Simplicity and time to value

If the analysis is based on a lot of

historical data

Improve access to historical

data and lower storage

costs

Performance improvements and cost

reduction

1

2

3

4

22 © GUIDE SHARE EUROPE 2016


IBM DB2 Analytics Accelerator Strategy

• Complement DB2's industry leading transactional

processing capabilities

• Provide specialized access path for data intensive

queries

• Enable real and near-real time analytics processing

• Execute transparency to the applications

• Operate as an integral part of DB2 and z Systems

• Reusing industry leading PDA's query and analytics

capabilities and take advantage of future

enhancements

• Extend query acceleration to new, innovative usage

cases, such as:

– in-database transformations

– advanced analytical capabilities

Enable DB2 transition into a truly universal DBMS that provides best

characteristics for both OLTP and analytical workloads.

DB2 for

z/OS

In-database

Transformation

Query

Accelerator

Storage

Saver

OLTP

Advanced

Analytics

23

Ultimately allow consolidation and unification of transactional and

analytical data stores.


Imperatives to implement z Systems-centric Data Lake Scenarios

• Reduce complexity of information supply chain, e.g.

Avoid data movement

Simplify data transformation

Use in-DB transformation

Use temporary tables structures

• Leverage state-of-the-art technology, e.g.

HW accelerators

Special-purpose appliances

In-memory processing

• Use federation technique whenever possible, e.g.

Federated SQL queries, leaving data in place

Federated analytical processing, leaving data in place

• Adhere to innovative and novel BI/DWH concepts, e.g.

Limit # of data marts and data cubes

Use aggregation on the fly

Allow for agile usage patterns

1

3 4

2

for Apache Hadoop

© GUIDE SHARE EUROPE 201624


Focus on Federated Data Analytics with Apache Spark

Addressing limitations of Hadoop MapReduce programming model

– No iterative programming, latency issues, ...

Using a fault-tolerant abstraction for in-memory cluster computing

– Resilient Distributed Datasets (RDDs)

Can be deployed on different cluster managers

– YARN, MESOS, standalone

Supports a number of languages

– Java, Scala, Python, SQL, R

Comes with a variety of specialized libraries

– SQL, ML, Streaming, Graph

Enables additional use cases, user roles, tasks, e.g.

– Data scientist tasks: developing analytical models

– Using languages other than Cobol or Java: R, Scala ...


Enterprise-grade, native z/OS distribution of the Apache Spark in-memory analytics engine

coupled with optimized data abstraction services that enable efficient access to a broad set

of structured and unstructured data sources in place with no data movement.

Avoid latency, costly processing and security concerns associated with data movement.

IBM z/OS Platform for Apache Spark is a no fee program product. Subscription and Support is available for purchaseLearn more Ecosystem of tools - Jupyter projecthttps://www.ibm.com/developerworks/java/jdk/spark/

Simplify data access

Reduce data access complexity with seamless Spark access to enterprise data.

Speed development

Take advantage of your expertise with popular programming languages such as Scala, Python, and SQL.

Accelerate results

Use in-memory processing for fast results. Apply a range of analytics including machine learning, iterative, streams and batch.

Developers can collaborate and build tools around IBM z/OS Platform for Apache Spark in the new GitHub organization and obtain pre-built distributions of open source components via IBM developerWorks.

Focus on Federated Data AnalyticsIBM z/OS Platform for Apache Spark

26

IBM EMEA Announcement ZP16-0103

March 22, 2016

IBM z/OS Platform for Apache Spark,

V1.1 offers an enterprise-grade platform

to enable Apache Spark on z/OS with

integrated data abstraction and

virtualization services

NEW!!

Link to Forrester paper

http://www-03.ibm.com/systems/z/os/zos/apache-spark.html

http://zos-spark.github.io/

https://www.ibm.com/developerworks/java/jdk/spark/

http://zos-spark.github.io/

https://www.ibm.com/developerworks/java/jdk/spark/

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=877&letternum=ENUSZP16-0103

http://public.dhe.ibm.com/common/ssi/ecm/zs/en/zsl03394usen/ZSL03394USEN.PDF


VSAM

z/OS

and *many* more . . .

Key Business Transacti

on Systems

Spark Applications: IBM and Partners

AdabasIMSDB2 z/OS

Distributed

Teradata

HDFS

Apache Spark Core

Spark Stream

Spark SQL

MLib GraphX

RDDDF

RDD

DF

Optimized data access

IBM z/OS Platform for Apache Spark

and *many* more . . .

Spark can run on z/OS close to z/OS-based Applications & Data

Values:Data-in place analytics, without need to ETL or move data for analytic purposes

Optimized access and z/OS governed ‘in-memory’ capabilities for core business data

Unique capability to access almost all z/OS sources with Apache Spark SQL & many non-z data sources

Almost all zIIP eligible

Integration of analytics across core systems, social data, website information, etc.

27


Spark can access z/OS based Data – Accelerated or enabled by IBM DB2 Analytics Accelerator

IMS

VSAM

z/OS

DB2

load

load

accelerated table

accelerator-only table

DB2 API

Application

SQL

val results =

sqlContext.sql("SELECT name FROM people")

JDBC to retrieve data from database

Application

Spark could run

on any platform

• z/OS

• Linux on z

• Linux on x86

• …

IBM Data Server Driver for JDBC and SQLJ

SQL access to DB2 z/OS can be

transparently accelerated by IDAA

Other data sources (IMS, VSAM,…)

loaded into IDAA accelerator-only tables

can be accessed using DB2 APIs

IDAA

sync

28

analytics on z systems focus on real time - hélène lyon

Technology