big data, big picture: can you see it?

Big Data Big Picture:Can You See It?

Mike HarerSenior Principal Product Manager, Big Data

Troy Coleman,Senior Principal Product Manager, Chorus and DB2 for z/OS

2 © 2015 CA. ALL RIGHTS RESERVED.

Today’s Discussion

THE FUTURE

EMERGING NEEDS

BIG DATA MANAGEMENT INNOVATIONS FROM CA

THE PATH FORWARD

1

2

3

4

Q & A5

The Future


THIS IS THE AGE OF THE APPLICATION

ECONOMY

AND IT’S ALL ABOUT THE DATA…

DATA IS THE FUELTHAT CAN MAKE OR

BREAK YOUR BUSINESS


DEMANDING GROWTH

SPINNING UPNEW PROJECTS

USING ALL DATA AVAILABLE

Businesses Moving Fast Into Big Data

Source: 2015 CA Sponsored Research: Vanson Bourne Global Big Data User Survey

60% want better customer experience

54% want new customers

84% will deploy an initiative in the next 1

year

94% plan to use all data available to them


How The BI Ecosystem Is EvolvingExample: Retail Store Analytics

NAS

POST-2014

HADOOP DISTRIBUTED FILE SYSTEM(UNSTRUCTURED/STRUCTURED)

NFSFTPETL

M/RHivePig

RMahout

RDBMSHBASE

StormKafka

CassandraDrill

SolrElastic Search

SAN(RDBMS/FILES)

Prod. CatalogTransaction Log

(Credit)Activity Log

Log AnalysisCustomer Insight

Recommendation Engine

Web Session AnalysisTransaction AnalysisPersonalized Offers

Transaction CaptureWeb Sessions

Shopping Carts

FLASH

INGESTION BATCH PROCESSING/MACHINE LEARNING

OLTP STREAMPROCESSING

INTERACTIVEANALYTICS

SEARCH

PRE-2014

Extract, Transform & Load all data into

DatawarehouseCapture All Transactions in

ERP/Relational DBPublish Promotions every

weekCreate Datamarts and Run Reports every day/week

ERP OPERATIONSOLTP OPERATIONS

ETL OPERATIONS BUSINESS INTELLIGENCE

BATCH OUTPUT MANAGEMENT

SAP, OraclePeoplesoft

InformaticaFTP/SFTP

Oracle DB, DB2DBSQL Server

CRON, Email


Result Is Complex Ecosystems

MULTIPLE → DOMAINS – IMPLEMENTATIONS – ENVIRONMENTS

Big Data Analytics Platforms

Traditional BI –Data Warehouse

Mainframe and Distributed

Amazon EMR Console

Unstructured Data Structured Data


Key Insight: Complexity Is The New Normal

GROWTH IN # OF BIG DATA PROJECTS OVER TIME

AS BIG DATA DISRUPTS COMPUTING PARADIGMS – GET AHEAD OF THE MANAGEMENT OF INFRASTRUCTURE NOW OR FACE THE

CHALLENGES OF DEALING WITH COMPLEXITY

Emerging Needs


What We Hear From Users


92%admit to major challenges managing their organization’s infrastructure to support current and future Big Data initiatives

31%

36%

40%

43%

48%

Lack of visibility into information andprocesses

Difficulty reintegrating analysis

Cost

Complexity of managing so many differentsolutions

Complexity of managing such a largeimplementation

COMPLEXITY IS 2 OF THE 5 LARGEST CHALLENGES IN MANAGING BIG DATA ENVIRONMENTS


Key Project Success Factors


40%

45%

47%

49%

57%

Cloud/hosted infrastructure services

Management tools for the infrastructure in place

Hire new resources with required skills

New infrastructure (storage, etc.)

Train existing resources on Big Data technologies

TOP 5 MAJOR INVESTMENTS NEEDED FOR PROJECT SUCCESS


The “Big” Big Data Management PainsThe Need to Overcome Many Challenges

Managing complex multi-vendor Big Data environments

Finding Hadoop/Big Data experts

Understanding capacity requirements for rapidly changing business needs

As complexity increases, manual processes are often required

System problems are hard to isolate, downtime increases

Unique tools and shortcomings

Driving forces… acquisitions, department consolidations demand greater operational efficiency

Big Data Management Innovations from CA

UNLOCK DATA

Getting broader: Unlocking insights from your mainframe data

VSTORM CONNECT DATA STREAMING FOR BIG DATA

MANAGE INFRASTRUCTURE

Getting to growth: Big data projects are critical for business value –protect this revenue by managing diversityCA BIG DATA

INFRASTRUCTURE MANAGEMENT

BDIM Pre-Release Registration and to Learn More:

www.ca.com/bigdata


DESIGNING FOR ROLE OF BIG DATA ADMINISTRATOR

The Role Of The Big Data Administrator

Perform day-to-day operations and support of Hadoop infrastructure

Monitor/maintain existing clusters and provision new ones

Integrate enterprise monitoring tools

Analyze current workloads and perform capacity planning

Hadoop Multi-Vendor Management

Hadoop Resource Management /Reporting

Hadoop Process Management / Automation

Hadoop Job Management & Monitoring

Hadoop System Health Monitoring & Alerts

ROLE / RESPONSIBILITIES: KEY MANAGEMENT CAPABILITIES:


360 Degree Approach

SINGLE, CONSISTENT MANAGEMENT UI EXPERIENCE

Linux / x86

SINGLE ACCESS POINT INTO

HETEROGENEOUS ENVIRONMENT

OPERATIONALIZE , MANAGE MULTI-VENDOR HADOOP MANAGEMENT DOMAINS

Big Data Infrastructure Management Server

CA BIG DATA INFRASTRUCTURE MANAGEMENT BIG DATA INFRASTRUCTURE

Current

Future Mainframe

HA

DO

OP

ECO

SYSTEM


CA Big Data Infrastructure ManagementSingle Unified View

Job monitoring

Heterogeneous system management

Intelligent alertmanagement

Resourcereporting

Cluster/Job/Node management

BDIMPre-Release Registration

and to Learn More: www.ca.com/bigdata


vStorm Connect Data Streaming for Big Data

It’s extremely difficult for Data Scientists, CMOs and other stakeholders to get access to their raw System z data in tandem with machine logs and other types of transactional information.

Need to make it easier for Mainframe customers to participate in Big Data projects. Simplifies and increases the ability to move Mainframe data into the Hadoop Big Data environment.

LAS VEGAS, November 10, 2014 — CA WORLD ’14 — CA Technologies (NASDAQ:CA) today announced a new global distribution agreement with Veristorm, a software company focused on Big Data management. The agreement strengthens CA’s ability to help customers leverage key business data on the mainframe for Big Data and analytics projects.

Logs

IMS

VSAM

Datacom

IDMS

DB2

WHY?

vStorm Connect Data Streaming for Big Data A new software solution that enables efficient data

integration into the Big Data ecosystem. Provides secure, near real-time access to mainframe

and distributed data for processing and self-service analytics.

New Extract-Hadoop-Transform technology, much faster than traditional ETL and staging processes, allowing enterprise data to be accessed by the analytics platform of choice.

WHAT?


vStorm Connect Data Streaming for Big DataHow it Works

vStorm Connect extracts data from DB2, VSAM, IMS, IDMS and Datacom and streams into Hadoop on the customer platform of choice (System z, Power Series or x86) thru a point-and-click, self-service portal.

Decreases the use of Mainframe general processors (reduction in MIPS/MSU utilization and associated expenses).

Lowers Mainframe storage costs, which can often reach up to $100k per TB, by offloading System z data into Hadoop off-platform.

Enables customer to gain new insight into key factors driving business performance.

Superior Technology - Unique because it does not require staging like traditional ETL solutions, lowering TCO & providing the data scientist real-time access to their most important transactional data.

HOW IT WORKS

KEY BENEFITS


Obstacles to Include Mainframe Data

1 2

Data Governance as the data moves off z/OS operational systems

Data Ingestion from z/OS into Hadoop (on or off platform) is a bottleneck (MIPS & ETL cost, Security around data access and transfer, Process Agility)

Existing security policies must be applied to data access and transfer.

There needs to be high speed / optimized connectors between traditional z/OS LPARs and the Hadoop clusters

Ability to serve data transparently into Hadoop clusters on mainframe AND on distributed platform

LEAD TO KEY REQUIREMENTS:


Data Ingestion Challenges

Extract from proprietary formats.

Aggregate or summarize.

Staging

Transform

Load

Hadoop, MongoDB, Cassandra, Cloud, Big Data

Ecosystem, Java, Python, C++, Interface skills

JCL, DB2, HFS, VSAM, IMS, OMVS, COBOL Copybooks, EBCDIC, Packed Decimal, Byte ordering, IPCS, z/VM,

Linux on z

z/OS

Logs

IMS

VSAM

Datacom

IDMS

DB2


vStorm Connect – Mainframe data to MainstreamBroad Platform Coverage

ELT alternative: Extract-Hadoop-Transform saves MIPS, staging storage, complexity

Dest: Cloudera, Hortonworks, etc

Metadata preserved: Use SQL on Hive

Governance, sandbox: zDoop

Datacom

IDMS

Logs

IMS

VSAM

DB2

System Z

z/OS

vStorm Connect Data Streaming for Big Data

Data +

Metad

ata

No Staging!

vStorm Enterprise

Power SystemsvStorm Enterprise

x86vStorm Enterprise

Linux on z

Linux


CA’s Big Data Management Approach

Deliver analytics solutions that remove the need for special skills and knowledge of the tools and infrastructure

Enable use of MF data (structured and unstructured) for analytics on any platform

1

Manage the complex analytics environment and simplify the complexity

2

Secure and enable compliant access control to data in analytics databases

3

4

CA DB2 Database Management for DB2 for z/OS


CA Chorus for DB2 Database Management Modernizing Management of DB2 for z/OSReleased 4.0 in April 2015

Custom Dashboard

Performance WarehouseCharting and Comparing

Policy Health Alerts and Investigation


CA Database Management Solutions for DB2 for z/OSAgile Incremental Release 19.0

CA Database Administration

Suite for DB2 for z/OS

CA SQL Performance Suite for

DB2 for z/OS

CA Sysview Performance

Management Option for DB2

for z/OS

CA Database Backup and

Recovery Suite for DB2 for z/OS

Sprint Reviewswith Users

Prototyping

User Research

User Feedback

Usability Testing

ProductBacklog

4 week sprints


BDIMPre-Release Registration and to Learn More:

www.ca.com/bigdata

http://cainc.to/BigData


For Informational Purposes Only

© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. This presentation is intended for information purposes only and does not form any type of warranty. Some of the specific slides with customer references relate to customer's specific use and experience of CA products and solutions so actual results may vary.

Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current information and resource allocations as of March 01, 2015 and is subject to change or withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.

Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product releasereferenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-available basis. The information in this presentation is not deemed to be incorporated into any contract.

CA does not provide legal advice. Neither this document nor any CA software product referenced herein shall serve as a substitute for your compliance with any laws (including but not limited to any act, statute, regulation, rule, directive, policy, standard, guideline, measure, requirement, administrative order, executive order, etc. (collectively, “Laws”)) referenced in this document. You should consult with competent legal counsel regarding any Laws referenced herein.

Terms Of This Presentation

big data, big picture: can you see it?

Technology

big data big picture

big data technologiestop

timeas big data

big data projects

data availablebusinesses

future big data initiatives316

big datasource

big datatroy coleman