DevOps for the Mainframe

Leveraging continuous integration testing, the cloud, and beyond to deliver z/OS applications

David Myers

February, 2013

Please noteIBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

Why we need DevOps on the

Mainframe• “Business software … has become horrifically complex”• “Developers are good at new functions, but bad at ensuring nothing goes

wrong when the new software is added to the existing mix”• “No single group of people, can ever fully understand the structure of key

business transactions in an enterprise”

In 2013 alone

Major 2012 events

http://www.bbc.co.uk/news/technology-21280943

“glitch delaying wage and bill payments”

“users to face duplicate charges”

“’system issue meant over 400,000 trades had failed to fulfill its best-price guarantee.”

“bungled update to CA-7 batch processing software costs over £100M”

“flash trading error causes $440M loss – due to dead code being accidentally triggered”

Mainframe Delivery Pains…

• Bad code going into production causes crit sits and emergency fixes

• Manual dependency management and configuration tasks result in errors and delays

• Restricted dev and test capacity lead to conflict, delays, or bad test results in shared environments

• Bottlenecks due to inefficient communications between disparate platforms and teams (Dev/Test - System Programmers; mobile – distributed-mainframe)

4

Go on Offense

Play Defense

…solutions from IBM

5

Enforce base quality standards automatically prior to promote

– Continuous Testing (CIz/RTW) with automated promotion policies

Automate build, provision, configure, and deploy processes

– Cloud deploy of dev/test images

Provide cheap, isolated, production-like, fit-for-purpose development and test environments for project teams

– RD&T + RTVS Test Virtualization

Improve communication and collaboration with cross-platform release planning and monitoring

– Proactive release planning• Expand RTC planning to System

programmers.• Link RTC to incident management• Link RQM to application monitoring

TestTest

DevDev

2

DevOps for the common mainframe delivery cycle

6

Dev Test Staging Prod

Development Test System Programmer

3

1 1

4

But…It takes weeks or even months to test and fix changes due to reliance on manual processes and limited access to test resources

Development Environment

CodeCheck

In Build

Developer Tester

PromoteTo QA

Feedback

Change Request

Open Defects

Queue Request Deploy

Queue Request

Run Tests

System Programmer

Enterprises want to… deliver end-to-end application enhancements quickly to stay competitive, trust that complex enterprise systems can be broadly integrated, and bolster confidence in application quality

Fast, dependable, automatic feedback speeds time to market Lower cost of application testing using off-mainframe z/OS test environment Enables confidence by automatically tracking and promoting code health

Continuous Integration for z

DeveloperFrequent, rapid feedback earlierto improve quality

Rapid Feedback

CodeCheck In + Build +

Deploy + Run Automated Tests

Change Request

Automated, continuous testing with isolated development and test environment

Reusable integration tests simplify testing across layers

PromoteTo QA

Higher quality going into QA

Rational Development and Test Environment for System z 8.5 Rational Test Workbench (Rational Integration Tester) powered

by Green Hat Technology

Rational Team Concert 4.0 Rational Quality Manager 4.0

Fail fast and drive better quality downstreamReduced delivery time, end-to-end visibility of test activities, safer and faster upgrades (V2V)

System z

Detailed Continuous Integration for System z

Scenario

9

CI Server

(Rational Team Concert)

Test Environment (RD&T or LPAR)

Test Execution Manager

(Rational Quality

Manager) Test Automation

(Rational Test Workbench)1. Check-in code

2. Build code and zUnit tests

3. Deploy build results and test data to Test Environment

4. Execute zUnit Tests

<Deploys App+zUnit Tests><

Drives>

<Executes>

5. Kick-off Automated Test Plan 6. Run automated

interface tests against Test Environment

7. Mark execution records Pass/Fail in Test Execution Manager

<Reports>

<Starts Tests>

8. Report test results in dashboard/build results/defect records in CI server.

<Results>

Rational Development and Test Environment for System z

The ultimate in modern application development for System z

Liberate developers to rapidly prototype new applications Develop and test System z applications anywhere, anytime! Free up mainframe development MIPS for production capacity Eliminate costly delays by reducing dependencies on operations staff

Note: This Program is licensed only for development and test of applications that run on IBM z/OS. The Program may not be used to run production workloads of any kind, nor more robust development workloads including without limitation production module builds, pre-production testing, stress testing, or performance testing.

RDz & ISPF user

ISPF user

RDz user

RDz userRDz user

COBOL, PL/I, C++, Java, EGL, Batch, Assembler, Debug Tool

x86 PC running Linux

RDz user

IMS

z/OS

WAS

DB2

MQ

CICS

10

Continuous Testing with DevOps and Green Hat

Simulate subsystem dependencies– Test impact of latency– Test application response to unresponsive

services

Create a stable test environment– Simulate “public” services– Reduce capacity requirements on the cloud

infrastructure– Improve security by reducing access from

external services

Support continuous Test & Delivery

Improve testing with Green Hat service simulation

Databases Mainframe

applications

App Under Test

Third-partyServices

Packaged apps, messaging services, etc.

Rational Test Virtualization Server

Rational Test Workbench

Developers & Testers

Rational Performance Test Server

Testing with dependency virtualization

Phase 1 Phase 2 Phase 3

App

Test Case

Virtual Services

3rd Party Call

IMS Data Access

CICS Commarea Call

App

Test Case

Virtual Service

s

3rd Party Call

IMS Data Access

CICS Commarea Call

App

Test Case

Virtual Service

s

3rd Party Call

IMS Data Access

CICS Commarea Call

Controlled large system testing by isolating components under test• Easier problem determination• Lower test environment capacity requirements• Improved component quality

Each tool came with its own

• UI - Web and desktop presentations of views and tasks

• Logic – Workflow, process, search, query, scale, security and collaboration

• Storage – Availability, traceability

• Privacy, backup/archiveResulting in...

• Brittle integrations• Silos everywhere• High cost to maintain and

administer• Proprietary API's

Tool ESCM

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

Tool BRelease Planning

Tool DIncident

Management

Tool CReporting

Tool ADev work Items

Separate tools makes collaboration inefficient or

difficult...

…consolidate, standardize, and simplify to increase

collaboration

COBOL

Java

Copybooks

WSDL

DB2DDL

IMS DBD/PSB

HTML

MQ Queue

Def

DB2 Plans

Procs

JCL

Deploy

EnvironmentRunning SystemDeployable Artifacts

Build, Package,& Unit TestApplication Binaries & Platform Configuration

Library

Engage all stakeholders in early planning, share and make visible

Enterprise Applications have thousands of disparate parts• Currently maintained in separate systems • Limited linkage between systems for application dependencies• Missing assets and information is rampant

DevOps force linkages, automation, and standardized packaging…

Source Management

Rational Team Concert

Agile Development

IBM SmartCloud Provisioning

Deployment of Virtual Systems

Collaborate Define Execute Report

V2.0 supports distributed platforms

IBM SmartCloud Continuous Delivery

Continuous Delivery

16

Development

Test

IT System Admin

Test Scripts

Configuration as Code (Scripts)

App Code / Build

Validate Build With Automated Test Suite

Report Test Results

“Cloud”Provision / Configure

Virtual Environment(s)

Spend less time provisioning and comparing

environments!

17

Development

Test

IT System Admin

Benefit: On-demand access to production-like test environments

… less time comparing configurations and recreating defects due to configuration shift

Benefit: Less time spent manually configuring test environments

… improved response time to delivery demands and configuration stabilization

Benefit: On demand access to current development build and configuration

… able to test current function instead of function from 2-4 weeks ago

18

Internal Success stories

Reduced Development Cost by 25% with much

improve quality

Accelerated product development with 100s

of deployments enabling delivery of stable

deliverables for early customer feedback

Reduction in test deployment by 90%

and expected to save over $2.3M / year and accelerating product

delivery

25% 99%

Reduction in Testing time from 2 Weeks

to 3 hours accelerating product delivery with much improved quality.

90%

What we are working on in the lab…

Go on Offense

Play Defense

…solutions from IBM (future)

20

Enforce base quality standards automatically prior to promote

– Continuous Testing (CIz/RTW) with automated promotion policies

Automate build, provision, configure, and deploy processes

– Cloud-style deploy of dev/test images

Provide cheap, isolated, production-like, fit-for-purpose development and test environments for project teams

– RD&T + RTVS Test Virtualization

Improve communication and collaboration with cross-platform release planning and monitoring

– Expand RTC planning to System programmers. Link RTC to incident management via OSLC..

Characteristics of Cloud

Pay-per-useShared ResourcePools

Elastic andscalable

Rapid deploymentMinimal management

Multi-tenant

Established mainframe strengths

Self-service

Standardize z/OS and distributed images to ease

deployment

• Standard topologies exist today (production LPARs)

• Standardized images which can be repeatedly deployed are rare

• Standardized/Automated deployment of COMPLETE system is spotty

To adopt the cloud for testing…Standardize and automate provisioning of everything …

Standard Image and Middleware

CICS 4.2 IMS 11.1

DB2 10.1Z/OS 1.12

Configuration

Data

Application

RT

C/C

Iz C

apab

ilitie

s

Standardize z/OS and distributed

images to ease deployment

Standard Image and Middleware

CICS 4.2 IMS 11.1

DB2 10.1Z/OS 1.12

Configuration

Data

Application

Pattern/Platform

Application load modules

Test Data

JCL and REXX

Evolving to deploying z/OS … cloud-

style

“Cloud”RD&T OrLPAR

RD&T OrLPAR

RD&T OrLPAR

RD&T OrLPAR

Provisioning and Setup Manager

Image/TopologyPatterns

CICS+DB2

CICS+ IMS DB

Batch

Acknowledgements and

Disclaimers:

© Copyright IBM Corporation 2013. All rights reserved.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere, DB2, Maximo, Clearcase, Lotus, etc

IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml

If you have mentioned trademarks that are not from IBM, please update and add the following lines:

[Insert any special 3rd party trademark names/attributions here]

Other company, product, or service names may be trademarks or service marks of others.

Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates.

The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own

views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

26

© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

www.ibm.com/software/rational

DevOps Assessment Workshop

Objectives• Assess your current DevOps

level based on practice areas• Provide recommendations to

achieve goals• Develop adoption roadmap

Execution• Two half day sessions• Discussions to understand

current processes and pain points

• Discuss trends and directions focusing on people, process, and technology

• Present recommendations and roadmap two to three weeks after workshop

Assess your current DevOps practice levels and develop an

adoption roadmap to achieve your goals.

Evolving to Multi-stage Delivery

Pipelines

28

Delivery Pipeline

Dev Stage

SCM(Source code & configurations)

QA Stage

Library(deployables)

Performance Stage

Pre-Production

Stage

Promote to Production

Changes trigger pipeline

execution

Each stage defines tasks and

configuration settings with

entrance and exit criteria

Approval gates with

automated/manual approvals provide

compliance checking

Stages can be run in parallel

Pipeline dashboards provide

feedback for the flow of changes

through the pipeline

Build Stage

Reporting/Dashboards/Analytics

Each stage fit-for-purpose for task

execution

Extending IBM SmartCloud Continuous Delivery

Continuous Monitoring

IBM SmartCloud Application.

Performance Management

Incident Management

IBM SmartCloud Control Desk..

IBM SmartCloud Provisioning

IBM PureSystems

IBM Workload Deployer

Rational Team Concert

Continuous Delivery

Continuous Deployment

IBM Rational Automation Framework

Chef

Continuous Integration

IBM Rational Build Forge.

Hudson….

Jenkins….

Continuous Testing

IBM Rational... Quality Manager

IBM Rational Test Workbench

Extending CLM with Continuous delivery

Deployment to Cloud & Virtual Systems

29

Optional and alternative Integrations