1

9 Steps to Agile Mainframe Ops Getting Maximum Long-term Value from Your Core Systems

Keith SissonDave SeibertSeptember 2019

2

DevOps?

“DevOps is not a technology challenge. Although technology plays a part, DevOps is largely about people, culture and process.”

3

Why Now?

I. Mainframes are critically important

II. Development teams are moving faster

III. Big changes require time

4

Time to Market

5

Digital Age = Fast Beats Slow

5

#1 Concern is being caught flatfooted by competitors AND not being Able to RESPOND

6

Waterfall

TIME

Planning Design Development QAPlanning Design Development QA

Ops First Look

TIME

7

Planning Design Development QAQA

Development

Design

Planning

QA

Development

Design

Planning

QA

Development

Design

Planning

QA

Development

Design

Planning

QA

Development

Design

Planning

Waterfall

TIME

demo demo demo demo

Agile

8

What about us?

Understand dev part of DevOps … What about ops?

Where do we start?

• Concrete steps

• Reasoning

• Success Indicators

9

1. Mainframe stewardship

2. Understand mainframe in digital age

3. Seamless integration

4. Culture of cooperation

5. Success metrics

6. Process improvement

7. Problem resolution

8. Automation

9. Promote your success

Mainframe DevOps From an Ops Perspective

10

MAINFRAME STEWARDSHIP

Step 1:

11

Ignoring Situation Usually Doesn’t Make It Better

• Mainframe is here to stay

• Need people to support platform

• Need explicit executive support

12

John Deere Combine

1313

14

Five Levels of Onboarding

3. Love/Belonging• Working on big important projects• Explore new techniques• Collaboration with other teams

4. Esteem/Confidence• Training• Creating automation• Solving big problems

2. SafetyJob security and career advancement

5. Self ActualizationRealization of your full potential

1. Physical NeedsGood salary, nice place to work

15

Maslow’s Hierarchy of NeedsSelf Actualization

Encourage them to make impact in system and in organization

Encourage Initiatives

• Open-minded

• Responsibility

• Lead projects

• Change organizational systems

• Improve own working environment

Multi-disciplinary Training

• Self learning

• Training in other fields

• Certifications

16

“Business leaders are broadly:

Type 1: Passionate product innovators

Type 2: Passionate about making the company run well—process oriented.”

- Ben Horowitz

“Software is eating the world, in allsectors. In the future every company will become a software company.”

- Mark Andreessen

17

1. Exploiters: Recognize mainframe as valuable asset

2. Status Quo: Mainframe must go slow

3. Cloud Only: Get off mainframe

Camps of Thought Around Mainframe

Mainstream the mainframe

18

• Defined commitment to the mainframe

• Mission champions and allies are found

• Hire new people onto mainframe teams

Success Indicators

19

UNDERSTAND THE MAINFRAME IN THE DIGITAL AGE

Step 2:

20

1964 Chevy Corvette

21

2019 Chevy Corvette ZR1

22

The Mainframe Simply Works

• People don’t really understand mainframe’s power

• Sometimes overlooked precisely because it always works

2323

Legacy technology

No new developersto support platform

Can’t be Agile

Running code from

1960s

Expensive to operate

24

25

26

UK Bank Success Story

• Each developer delivers 36 story points per release, as opposed to 8

• Reduced time spent writing and checking code during each sprint from average of 40 hours to 7 hours

• Project delivery timeframe cut by 1/3

• Reduced time to test large applications from 2 weeks to 5 minutes

• Removed manual effort and enabled creation of repeatable tasks in Jenkins

• Overall developer productivity increased by 400%

26

“We are innovating faster on the mainframe than we are in our digital channels, which is just incredible”

27

• Stakeholder confirmation and understanding

• Champions and allies identified and ready to help

• Explicit confirmation of the understanding of the mission and its importance to the success of the business

Success Indicators

28

SEAMLESS INTEGRATION

Step 3:

29

Two-platform IT

• Enterprise platform strategy we’ve employed for 4+ years

• Helped us save $5 million annually

• Focus on continuously improving customer experience,

instead of demonizing particular platform

• Most mission-critical workloads

and strategic IP leverage mainframe

• Compulsory apps not providing competitive

advantage are server-based

• Compulsory apps became SaaS, cloud-consumed service

30

Our Data Center

31

IBM Commercial (2006)

32

• Moving on-prem x86 to cloud SaaS saves $

• Re-writing or re-platforming does not save $

• Migrating strategic IP to cloud costs millions, takes years, introduces new risks and results in failure or reduced service levels

• Even if successful, only at same place you started!

• Executive sponsorship

• Business unit buy in and co-lead

Two-platform IT

33

REST Web Service Server

REST Web Service Server

HTTP Response

HTTP Request

{"name":"Product",

"properties":

{ "id":

{ “type":“3090",

"description":“hammer",

"required":true

},

"name":

{ "description":“deluxe driver",

"type":“implement",

"required":true

},

"price":

{ "type":“11.49",

"minimum":0,

"required":true

},

"tags":

{ "type":"array",

"items":

{

"type":“red"

}

}

}}

JSON Object

REST Web Service Client

Create HTTP PUT

Read HTTP GET

Update HTTP POST

Delete HTTP DELETE

REST Web Service Server

RESTful API

34

Accepted standard for what “cloud service” is:

• On-demand self-service

• Broad network access

• Resource pooling

• Rapid elasticity

• Measured service

*NIST publication SP800-145

Source: http://www.redbooks.ibm.com/redbooks/pdfs/sg248347.pdf

• Distributed caching solution

• Unique identifier generator

• Crypto service

• No SQL data store

All accomplished on their mainframe!

35

Native CICS APIs

CICS APIs that satisfy requirement for HTTP-based communications for most cloud services:

EXEC CICS WEB EXTRACT

Gets HTTP information about incoming requests

EXEC CICS WEB RECEIVE

Gets request information from consumers

EXEC CICS WEB SEND

Sends response to consumer

Also useful APIs:

EXEC CICS WEB READ HTTPHEADER

Gets HTTP header fields and information

EXEC CICS WEB WRITE HTTPHEADER

Creates HTTP header fields and information

REST APIs

36

//O&number JOB (TECH) //*

// EXPORT SYMLIST=ORDER

// SET ORDER=‘&number//*

//PROCSTEP EXEC PRODUCTP

//*

//SYMBOLIC.SYSUT1 DD *

Description &descriptionType &type Price &price

JSON to JCL

{"name":"Product","properties": {"Order":{"Number":"0001405","required":true},

"id":{“id":"3090","description":"hammer","required":true},

"name":{"description":"deluxe",“type":"implement","required":true},

"price":{"price":"11.49","minimum":0,"required":true}

}}

//O0001405 JOB (TECH) //*

// EXPORT SYMLIST=ORDER

// SET ORDER='0001405' //*

//PROCSTEP EXEC PRODUCTP

//*

//SYMBOLIC.SYSUT1 DD *

Description hammerType implement Price 11.49

37

Services vs. Functions vs. IP

Services = Generic operations that provide valuable utility Example: Generating unique value or

temporarily storing data

Functions = Provide specific operation or result Example: Date difference calculation, string manipulation,

data/numeric conversions

IP = Code that uses logic, design and creativity to produce results that often provide strategic value

Q. What should I run on the mainframe?

A. ALL OF IT! (*anything with a lot of transactions)

38

How to Do Two-platform IT

Step 1: Conduct Audit

• Analyze all apps running

• Separate compulsory from business-critical

• Business-critical stay on mainframe

Step 3: Have Business Units Make Functional Decisions

• Elect subject matter expert (SME) to understand business unit needs

• Partner IT with SME and vet process to ensure right decisions are made

Step 2: Identify Risk

• Impact from failure

❑ Cost of failure?

❑ Who is impacted?

❑ Workarounds available?

Step 4: Decide What to Do with Data from Deprecated Technology

Have to make it accessible in some way

• Port it into new system

• Archive it in some new system

• Keep old system (eliminates some savings)

39

Step 6: Roll Out

• Train everyone on how to use new system once up and running

• Burn the boats

Step 7: Reallocate IT and Other Business Resources

• Not cost-cutting exercise

• Re-allocate people and resources

Step 5: Conduct IntensiveAudit on Data Quality

• Begin to uncover hidden constraints

• Benefit: Data is ultimately scrubbed = fresh start

How to Do Two-platform IT

40

Success Indicators

• Realization that big things can be accomplished

• No longer have groups bigoted towards certain platforms

• Stop wasting time moving workload for no real value

• Focus shifts to innovation

41

INITIATE A CULTURE OF COLLABORATION

Step 4:

42

Zero Tolerance for Silos

You can no longer afford to manage your mainframe environment as an isolated silo apart from your cross-platform enterprise.

Normalization will reduce your costs, improve your outcomes and extend the useful life of your mainframe for decades to come.

43

Agile Input

Three AmigosProduct Council

Sprint Planning and Estimation

Sprint Review and Demo

Retrospective

Sprint Cycles

44

Three Amigos

Operations

Bringing new perspectives

Product Owner

Developer

Customer

DBA Team

Marketing

45

Agile Methodology Kanban

Backlog Accepted

Next to

WIP

WIP

Next to Accepted

46

• Test automation

• Packaging

• Security

• Performance tests/benchmarking

• Build automation

• Deployment updates

• Storage demands

• Capacity

Development Release Backlog

47

Agile Methodology Kanban

Backlog Accepted WIP

48

Success Indicators

• Team problem-solving without finger-pointing

• Increased use of shared digital tools

• Tangible increases in collaborative ‘face-to-face’ meetings

• Shared and/or aligned KPIs

49

SUCCESS METRICS

Step 5:

50

KPI Success Factors

• Communicate KPIs clearly to team: Won’t affect much change if people don’t explicitly know how they are being measured

• Velocity and efficiency are as important as quality, place equal importance on measuring and continuously improving all 3

• Set small number of initial goals that are challenging within current processes but achievable by embracing new methods

51

Don’t Be Stuck in “Department of No” Mode

• Don’t solely be centered around “five 9s” availability and tight time-windows for batch completion.

• What is most important to you?

• What is your biggest constraint to moving faster?

52

SWOT Chart

Strengths Weaknesses

OpportunitiesThreats

53

Things take too long.

When will the new “X” be ready?

Measure the elapsed time from

“Go” to “Done”

Velocity

54

High priority work takes a backseat to emergencies!

It’s hard to anticipate emergencies, because they are, by definition, unplanned work

People utilized at 100 percent do not have the capacity to handle unplanned or urgent requests without dropping other high-priority work

WIP

55

Your top priorities may not alignwith other team’s top priorities

When a decision is made to do one thing,

you are delaying something else

People may just be unaware that there will

be an impact

Alignment

56

The process to get things approved slows down everything

Process Efficiency

Change control, ITIL, CMM, Quality Gate, etc

57

We have to put on maintenance, install a new release, defrag DASD, improve performance…

Technical Debt will eventually catch up to you.

Melvin Conway’s 2nd law: “There is never enough time to do something right, but there is always enough time to do it over”

Technical Debt

58

Measure and track the “Flow Rate”:

Where was the time spent from “Go” to “Done”?

Categorize and measure the time that is spent on unplanned work

Categorize and measure the time that is spent on high priorities of others

Measure the time that is spent waiting on process

https://flowframework.org Dr. Mik Kersten

Big

Thre

e

59

zAdviser

60

Success Indicators

• Basic understanding of flow roadblocks

• Improvement ideas, especially when waiting

• Roadmap created for successive rounds of process improvements

61

CONTINUOUS, QUANTIFIED PROCESS IMPROVEMENT

Step 6:

62

• Change control

• Compliance

• Security

• Audit

• Performance tests/benchmarking

• Performance monitoring

• Customer/user support

• Test automation

• Packaging

• Build automation

• Automatic deployment

• Automatic back-out

• Deployment updates

• Training

• Marketing

• Legal

DevOps

63

Insert Existing Processes into the Flow

Three AmigosProduct Council

Sprint Planning and Estimation

Sprint Review and Demo

Retrospective

Sprint CyclesChange Control

Customer Support

Automation

Performance Monitoring

Long-range Planning

Auditing/ComplianceOperations

64

DevOps

Planning

Design

Development

QA

PO

Operations

Dev

QA

TIME

Systems

Network

Customer Success

Security/Audit

Data

65

10 Items to Consider around Process1. Continuously examine your processes and eliminate

meaningless or low value ones

2. Create similar processes for all of development

3. Align approval and vetting processes with your Agile development schedule

4. Accomplish as much as you can using collaboration tools like Confluence, Slack or HipChat

5. Operational change control should be easier

6. Automate testing process

7. Automate deployment process

8. Have reliable method to verify code that was tested is same code that was deployed in production

9. Be able to quickly and automatically back out failed deployments

10. Training and constantly reinforce

66

Success Indicators

• 3-5 KPIs that align with improving quality, velocity and efficiency

• New incentives that support KPIs

• Eliminate incentives that are counterproductive

• Get explicit feedback

67

PROBLEM RESOLUTION“SHIFT LEFT”

Step 7:

68

Business Unit Wants It Fast

99.999 SLAAutomated Continuous Delivery

99.999 goal

FASTSLOW FASTSLOW

Speed and Coordination

69

Continuous Deployment

Continuous Delivery

Continuous Integration

Development

1. Analyze, edit and compile code

2. Unit test

Automation• Ideation• Product management

3. Code quality review

4. Product build

5. Integration test

6. Acceptance test

7. Deploy to production

70

71

AutoPLEX

DeployV16 | V17

Test Assets

Hiperstation Batch Jobs

V16 | V17 | V18

V16 | V17

V16 | V17

V16 | V17

V16 | V17

ZAPI

Test Results

Emailwith URL

CW01

V16 | V17

72

CW13/CW14 ISPW Promote/Deploy Process

Promote to PROD

Promote to REGR

Promote to INTG

Promote to PTFS

– Deployed to

Regression

libraries

– Automated

tests run with

product REGR

libraries

– Deployed to

Integration

libraries

– Automated

tests run with

product INTG

libraries

– Approved PTFs

auto applied

to targets

– Automated

tests run with

product SMP/e

target libraries

– Automated

SMP/e product

build-install

– Automated tests

run against all

installed product

builds

73

74

75

76

77

78

Success Indicators

• Measurable shift of QA tasks to earlier in software lifecycle

• Reduced occurrence of specific issues and alert types later in lifecycle

• Fewer customer-discovered defects

79

INCREASE AUTOMATION

Step 8:

80

• Primary focus = eliminating line separating development and operations/system programming efforts

• Enable functions formerly considered domain of operations or systems personnel and extend to dev teams with auditable, secure means

• Will not discuss integration of application database design changes into DevOps methodology

How Db2 Fits into Our DevOps Strategy

81

• Db2 tables contain our repository of PTF metadata

• Interfaces to Jira software for requirements and problem reports

• Use queries to build:

– Lists of PTFs that are customer requests

– Lists of PTFs that customers do not have installed

• Recursive Rexx and SQL

– When customer requests 1+ PTFs, our programs retrieve available prerequisite PTFs for requested PTFs

Db2 in Receive Order

82

• WITH PREREQPTF (PTF, PREREQ ) AS

• (

• SELECT PRE.PTF_ID, PRE.PREREQ_PTF_ID

• FROM PTF_PREREQ PRE

• WHERE PRE.PTF_ID = 'SBG009A'

• UNION ALL

• SELECT CHILD.PTF_ID, CHILD.PREREQ_PTF_ID

• FROM PREREQPTF PARENT, PTF_PREREQ CHILD

• WHERE PARENT.PREREQ =CHILD.PTF_ID

• )

•

• SELECT DISTINCT PTF, PREREQ

• FROM PREREQPTF

• ORDER BY PTF, PREREQ

Recursive SQL to get pre-reqs

83

• Do until PreReqs_done•

• l = l+1 • if ptot= ptfs.0 then resultlist= currPTFList• SQLSTMT = , • " SELECT DISTINCT PREREQ_PTF_ID ", • " FROM "creator".PTF_PREREQ ", • " WHERE PTF_ID IN ", • " ( " , • currPTFList, • " ) ", • " AND PREREQ_PTF_ID NOT IN ( ", • resultList ")" •

• "EXECSQL DECLARE C1 CURSOR FOR S1" • "EXECSQL PREPARE S1 FROM :SQLSTMT" • if sqlcode <> 0 then /* & sqlcode <> 347 then */ • do;DB2Function = 'Prepare Statement'; call Display_Sqlca;exit 12;end • "EXECSQL OPEN C1" • if sqlcode <> 0 then • do;DB2Function = 'Open cursor'; call Display_Sqlca;exit 12;end • do i= 1 to 9999 until SQLCODE <> 0 • "EXECSQL FETCH C1 INTO :PTFID" • select • when sqlcode = +100 & i=1 then • if ptot>0 then

Recursive Rexx & SQL

84

• Receive from network uses same Db2 metadata tables as Receive Order

• Receive from network typically retrieves entire product and all maintenance

• Our experience: Used less often for maintenance only; handled through receive order

Db2 in Receive from Network

85

• Db2 tables of nightly backup data for critical resources

– Dsname

– Volume serial

– Backup date

– Backup type

• Merge data from VTOC and Catalog information

• 24 million row table

• Replaced SAS database with Db2

• Tables queried to prep for our DR tests

• Tables also queried to build JCL for emergency DS recovery

Db2 in Disaster Recovery

86

SQL to populate DR main table

87

Before & after SQL

Having altered the logging attribute, we need an image copy to make the table recoverable.

88

WITH BACKUPS AS

(SELECT BACKUP_DATE,BACKUP_TYPE,COUNT(*) COUNTS

FROM DR.RECOV_ALL_REPORT

GROUP BY BACKUP_DATE , BACKUP_TYPE

)

SELECT COUNT(*), BACKUP_TYPE, ' '

FROM BACKUPS

GROUP BY BACKUP_TYPE

UNION ALL

SELECT * FROM

(SELECT COUNTS,BACKUP_DATE,BACKUP_TYPE

FROM BACKUPS

ORDER BY BACKUP_TYPE,DATE(BACKUP_DATE) DESC)

UNION ALL

SELECT SUM(COUNTS) ,' ',' '

FROM BACKUPS

-- S.B CW01 D=003 W=006 M=036

-- FROM DISASTER.MASTER.FILE(GDG)

Common Table Expression to query the DR table

This is a query I created to answer three questions I have of the DR data.

I want to avoid traversing the 24 million rows 3 times. I found (finally) a use for a CTE in the real world.

From the 24 million rows, I want to query:

1. The number of backups by type2. The number of datasets on each

backup date by backup type3. The total number of datasets backed up

89

WITH BACKUPS AS

(SELECT BACKUP_DATE,BACKUP_TYPE,COUNT(*) COUNTS

FROM DR.RECOV_ALL_REPORT

GROUP BY BACKUP_DATE , BACKUP_TYPE

)

SELECT COUNT(*), BACKUP_TYPE, ' '

FROM BACKUPS

GROUP BY BACKUP_TYPE

UNION ALL

SELECT * FROM

(SELECT COUNTS,BACKUP_DATE,BACKUP_TYPE

FROM BACKUPS

ORDER BY BACKUP_TYPE,DATE(BACKUP_DATE) DESC)

UNION ALL

SELECT SUM(COUNTS) ,' ',' '

FROM BACKUPS

-- S.B CW01 D=003 W=006 M=036

-- FROM DISASTER.MASTER.FILE(GDG)

Common Table Expression to query the DR table

Define the CTE

Get backup type counts

Count backups by date

Show total count

90

SELECT DATE(BACKUP_DATE) ,BACKUP_TYPE ,VOLSER ,DSNAME ,FULLGDG_DSNAME ,COMMENT ,DATA_VOLSER ,DATA_PORTION_NAME ,INDEX_VOLSER ,INDEX_PORTION_NAME ,INDEX_GDG_DSNAME FROM DR.RECOV_ALL_REPORT WHERE DSNAME [like or =] InputdsnameORDER BY Date(BACKUP_DATE) desc

Query to recover dataset from DR backup

91

• // jobcard• /*ROUTE XEQ CWCC • /*JOBPARM S=CWCC • //* • //STEP1 EXEC PGM=ADRDSSU,TIME=1440 • //SYSPRINT DD SYSOUT=* • //TAPE1 DD DSN=BACKUP.DAILY.PCC004.G0573V00,DISP=OLD, • // UNIT=V3590, • // VOL=SER=(896081), • // LABEL=(001,SL,EXPDT=98000) • //DISK1 DD UNIT=3390,VOL=SER=PCC004,DISP=SHR • //SYSIN DD * • RESTORE IDD(TAPE1) ODD(DISK1) TOL(ENQF) -• CATALOG -• DS(INCLUDE(-• BFHDJS0.DEV.EXEC -• )) SHR -• RENAMEUNCONDITIONAL( -• (BFHDJS0.DEV.EXEC, -• BFHDJS0.DEVVV.EXEC))

JCL generated for recovery

92

• Fed from problem reporting and availability facilities

• Parse syslog archives for outage and warning messages

• Store these in Db2 for analysis and recommended remedial actions

• Queries:

– What are most common messages and abends?

– What warning messages lead to abends?

– Are there clusters of messages?

– Any correlations we can find?

• Possible tie-ins to Compuware products

Db2 in Anomaly Detection

93

• Automated testing groups rely on syncing test cases with test data required for exercising those cases

• Db2 provides baseline and set of utility executions supporting groups

• Still have mix of testing group-managed Db2 procedures and ops-executed procedures

• In process of enabling all to be test-group managed

Db2 in Automated Testing

94

• REST with Db2

• SA z/OSMF POC lack of value

• Python/YAML dynamic Db2 table build

Futures for Db2 & Dev Ops

95

Success Indicators

• Elimination of repetitive ops tasks

• More frequent promotion of new code

• More cooperation between mainframe and non-mainframe ops

• Linked cross-platform deployment

96

PROMOTE YOUR SUCCESS

Step 9:

97

Success Indicators

• Publicly recognized ops successes

• Business people recognize your progress

• Improved recruitment of ops talent onto mainframe team

98

Recommended ReadingHelpful Reference Materials

• The Phoenix Project: A Novel About IT, DevOps and Helping Your Business Win

• Start and Scaling DevOps in the Enterprise

• The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

• “Digital Transformation Needs Mainframe DevOps”

• “The Need For Speed: Drive Velocity And Quality With DevOps”

• “Use Four Key Categories To Measure What Matters In Continuous Deployment”

• “Project to Product” Flow Framework and Value Stream Metrics

• Link to Craig Mullins Db2 presentation referenced

Learn more about future-enabling your mainframe ops, compuware.com.

Learn about bringing modern DevOps best practices to the mainframe by downloading our eBook

“Ten Steps to True Mainframe Agility.”

99

THANK YOUFor more information contact:

Keith.Sisson@compuware.com or

Dave.Seibert@Compuware.com

100© 2019 Compuware Corporation. All rights reserved.