rocket data virtualization - ims phoenix ug nov 10th 2016
TRANSCRIPT
1
Rocket Data Virtualization
Thursday, November 10, 2016
2
What is Data Virtualization?
Enabling data
structures that were
designed
independently to be
leveraged together,
from a single source,
in real time, and
without data
movement
Mainframe
Web/ Mobile
RDBMS
Cloud Data
Big Data
Unstructured
Logical Data Source
© 2015 Rocket Software, Inc. All Rights Reserved.
3© 2014 Rocket Software, Inc. All Rights Reserved.
Data Virtualization Drivers
4
What is Data Virtualization
© 2016 Rocket Software, Inc. All Rights Reserved.
Data Virtualization: a virtualized data
services layer that integrates data from heterogeneous data sources and content in real-time, near-real time, or batch as needed to support a wide range of applications and processes.
Forrester Research – March 2015 - Noel Yuhanna
Data Virtualization
5
Rocket Data Virtualization
© 2016 Rocket Software, Inc. All Rights Reserved..
6
How We Lower Mainframe TCO
GPP zIIP
Eligible Workloads Can Run Outside of GPP within zIIP
� Mainframes have multiple processors• General purpose processor
� all processing counts against capacity
• Specialty Engines� Eligible workloads don’t count against
capacity
� Rocket DV has patented technology that allows it to run 99% of its own processing in the zIIP engine• Enables mainframe data to be
integrated in-place without processing “penalty
7
Why Data Virtualization
Need to accommodate volume,
variety and velocity of data
Mobile driving need for more
real-time, accurate information
Increased adoption of advanced
analytics and self-service discovery
Need for agile data services
with high security
Rules of the Game Have Changed
8
Challenge of Skills and Data Compatibility
� The skills necessary to work with mainframe
data are diminishing
� More programmers today are familiar with
SQL or Java
Mainframe non-relational data structure Transformed into relational format
10
What is Wrong with Status Quo?
“There is not enough time in the
day to move all the data.” “My mobile users expect to see
current data, not yesterday’s data.”
11
Reporting
Ad-hoc
OLAP
Data WarehouseStaging Server
Staging Server
Staging Server
Moving Data Via ETL Tools
Represents ETL
S
Q
L
Data Integration Limitations
Complex, high mainframe costsData inconsistency – High latency
DB2
VSAM
IMS
Adabas
Physical Sequential
CICS
IMS
Natural
IDMS
12
Data Warehouse
Using Connectors for Data Access
Data Integration Limitations
ETL Server
Rigid, difficult to change, expensiveLots of connections – high complexity
DB2
VSAM
IMS
Adabas
Physical Sequential
CICS
IMS
Natural
IDMS
© 2016 Rocket Software, Inc. All Rights Reserved..
13
A Closer Look at Rocket Data Virtualization
14
Mapping DB2
LUW and DB2
for z/OS
Mapping mainframe
non-relational (Adabas,
VSAM) and DB2 for
z/OS data sources
Map Once Use Many
15
Virtualize & Use
© 2016 Rocket Software, Inc. All Rights Reserved..
21
21
22
z13 Exploitation
© 2015 Rocket Software, Inc. All Rights Reserved.
• Takes advantage of SIMD• Single Instruction Multiple Data (SIMD) Accelerator exploitation in
Data Virtualization core engine.
• Takes advantage of SMT2 • Simultaneous Multi-Threading 2 exploitation due to 100% zIIP offload
of DVS
• Heavily exploits zEDC for network I/O • minimum 5x improvement – depending on network topology and
speed) in elapsed time for large result sets • bidirectional exploitation in DVS, we see > 5x times reduction in
elapsed times with 0 z13 GPP cycles consumed.• 100% zIIP offload (2 zIIP per GPP ratio)
• Exploits SMC-R• Load of optimized DV engine based on hardware• for the z13, we use ARCH(11) capability in Metal C 2.1.1 compiler• we built the Rocket DV core engine with Metal C and ship optimized
builds for current z Systems platforms (z196, EC12, z13)
23© 2015 Rocket Software, Inc. All Rights Reserved.
• Exploits AT-TLS, because we use it, we inherit all of the new crypto/encryption advances
• VSAM – new SRB mode support (not ICI based)
• Log Streams – new READ parallelism
• Exploitation of 64 bit storage, Shared Memory Objects and more importantly z Flash Express to take advantage of Pageable Large Pages (reducing DAT code path)
• Flash Express exploitation for reduced Dynamic Address Translation (DAT) overhead. Exploited for all above the bar Private and Shared Memory Objects (buffer pools, Metal C heap, data areas)
• MapReduce (reading different sections of the same dataset in parallel then aggregating the Virtualization engine
• Parallel I/O, only keeping a file open for milliseconds, network and file I/O is done in parallel
• Full intra SQL and intra-partition parallelism
z13 Exploitation
24
Rocket ® Data Virtualization for IBM® z13™ and z13s™ Systems
© 2016 Rocket Software, Inc. All Rights Reserved.
25
Hybrid Cloud Data Services
© 2016 Rocket Software, Inc. All Rights Reserved.
26
Bring Mainframe Data to Spark
© 2014 Rocket Software, Inc. All Rights Reserved.
27
IBM DB2 Analytics Accelerator Loader
for z/OS Enterprise Edition
Accelerator Loader Server
IBM DB2 Analytics
Accelerator
DRDA Sources
(Oracle)
SQL Result Set
BatchDSNUTILB
SourceSQL
Statement
AcceleratorLoader
2 Back
28
RDV Use Case – Transactional Data Access
IBM z/OS Connect allows native applications
to be discovered and invoked via a
RESTful API
29
IBM z/OS Connect
Allows native applications to
be discovered and invoked via
a RESTful API
REST API consumers
Cloud / Bluemix
apps
Mobile apps
Web apps
z/OS ConnectEnterprise Edition
MobileFirstPlatform
CICS
DB2
IMS
MQ
WAS
30
IBM z/OS Connect with Data Virtualization
REST API consumers
Cloud / Bluemix
apps
Mobile apps
Web apps
z/OS ConnectEnterprise Edition
MobileFirstPlatform
CICS
DB2
IMS
MQ
WAS
VSAM
SMF
SYSLOG
Adabas / Natural
IDMS
Cloudant
Hadoop
DB2 LUW
Oracle
SQL Server
…
Allows native applications and
data to be discovered and
invoked via a RESTful API
Data
Apps
Questions