best practices for optimizing mainframe application performance with ca cross-enterprise application...

Best Practices for Optimizing Mainframe Application Performance with CA Cross-Enterprise Application Performance Management

Tom Quinn

Mainframe

CA Technologies

Sr Engineering Services Architect, Mainframe Solution Center

MFX16E

@tomquinn03

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD

For Informational Purposes Only

Terms of this Presentation

© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. The presentation provided at CA

World 2015 is intended for information purposes only and does not form any type of warranty. Some of the specific slides with customer

references relate to customer's specific use and experience of CA products and solutions so actual results may vary.


Abstract

In this session you will see a technical overview of the CA Cross-Enterprise Application Performance Management (CA APM) solution and how it extends CA APM into the mainframe. You will also hear about common install or startup problems, details about tracing end to end transactions into the mainframe, and how to utilize the CA Mainframe Performance Suite integrations.

Tom Quinn

CA Technologies

Sr Engineering Services Architect, Mainframe Solution Center


Agenda

CA CROSS-ENTERPRISE APM TECHNICAL OVERVIEW

SOLUTION INSTALLATION

CA PRODUCT INTEGRATIONS

INSTALLATION VALIDATION

COMMON ERRORS AND RESOLUTIONS

UNDERSTANDING TRANSACTION TRACING

2

3

4

5

6

CA CROSS-ENTERPRISE APM ROADMAP1

7


How do I deliver a flawless experience every time an application touches the mainframe?

In the application economy it’s all about your customers.You need to think about your mainframe reframed.

Connect mobile-to-mainframe applications

Create mainframe infrastructure flexibility

for the future

Unleash the power of data on the mainframe


Agile Operations for End-to-End APMCA Cross-Enterprise APM

Transaction tracing across entire application architecture......provides reduced MTTR, greater availability and optimal customer experience

3rd Party Apps

Databases

Web Services/SAAS

MID

DLE

WA

RE

Application Server

NetworkUser Experience

CICS IMS

DatacomDB2

TRA

NSA

CTI

ON

S D

ATA

Systems of Operations Systems of RecordSystems of Engagement

Quickly pinpoint root cause of issues for

faster MTTR


End to End Application Monitoring


VALUE DRIVERS

Strategic Investment Themes

Customer Value Drives Roadmap Themes

Audit and Compliance Reporting Maximize Existing ResourcesApplication Experience Continuity

Collaborative Application Development Real Time InsightsHigh-speed Accessibility with Integrity

Data Security is ParamountCreate and Extend Mainframe Services Speed, Ease of Use and Flexibility

Connect to applications and data on the mainframe

Unleash the power of data on the mainframe

Create infrastructure flexibility for the future

Applications Data Infrastructure

ROADMAP THEMES

Secure Data and Service Enablement

Total Cost of Ownership

Business Continuity


Distributed/Mainframe Operations Analysts

CA Performance Management Portfolio VisionUnified Performance Management for z Systems

MF Systems/Network Analysts

Distributed Application Analysts

Network Analysts

CA NetMaster®

Databases

Network

CICS & MQ

z/VM, KVM & Linux on z

CA UIMCA UIM for z Systems

CA Cross Enterprise APM

Integrated Mainframe Management Console

CA APM

Planned

AvailableSystem Analysts

CA SYSVIEW®

Storage

Capacity Planner

CA MICS®

Storage Administrator

CA Vantage™


CA Cross-Enterprise Application Performance Management


Technology Goals

ProductArchitecture

Goals

Technology and Product Architecture GoalsCA CE-APM

Rapid deployment Simplified Install and deployment Open APIs

Maintain MF currency support Transaction trace across entire MF Standardize Interface with APM


Enhanced productivity a Re-Engineered Application Triage Map allowing users to naturally and easily triage application problems

Improved root cause analysis with Differential Analysis by never missing a period during which an application suffered a significant change in performance – without having to configure specifically for that application.

Improved root cause analysis -Automatically detect and monitor performance of applications at process boundaries

Roadmap: CA CE-APMTimeline as of Nov 1, 2015

Marquee Features/Business

Value

Planned Under Consideration

Delivered Planned Under Consideration

Product/Releases

CA CE-APM 10.0

Currently GA

CA CE-APM 10.2 CA CE-APM

Improved Root cause analysis through deeper visualization of the MF application components

Improved Root cause analysis through tighter and broader transaction tracing from mobile through the mainframe

Enhanced productivity with simplified install and upgrade process.

Improved Root cause analysis through additional application protocols

Improved Root cause analysis through tighter transaction tracing from mobile through the mainframe

Enhanced productivity through interface standardization for install and configuration


Secure Data and Service Enablement Business Continuity Total Cost of Ownership


Delivered

Feature Description BenefitAPM Team Center Overlay real world attributes on to complex

application relationship data to minimize the spaghetti seen in typical Application Maps

Team Center allows users to quickly look at the components that comprise their applications and roll back time to the point when they first started experiencing symptoms to find the "patient zero” causing the current issue

Differential Analysis Spikes are mapped & instability visualized across many apps and long time periods.

By using simple statistical techniques, CA APM Performance Spike Detection discovers spikes in application performance, proactively triggers evidence (trace) collection, and displays a rich performance story across applications, all with simple configuration that works out-of-the-box

Entry Point Detection Automatically detect and monitor application processes without requiring any deep knowledge of either the application or application framework , all without requiring any manual instrumentation

Broaden application monitoring coverage applications/frameworks by automatically detecting and monitoring performance of application at process boundaries


Planned CE APM 10.2

Feature Description BenefitAPM Team Center Support

Overlay real world attributes on to complex application relationship data to minimize the spaghetti seen in typical Application Maps

Team Center allows users to quickly look at the components that comprise their applications and roll back time to the point when they first started experiencing symptoms to find the "patient zero” causing the current issue

Cross system trace Provide deeper transaction visibility of the trace within the MF environment across lpars

Enables operation to see deeper into the transaction flow of the mainframe

Land Lightly Rationalization of system parameters. Reduce time to value of investment return for user base. Make the product easier to install and easier to configure


APM Team CenterRe-Engineered Application Triage Map

PAIN

Benefit

SOLUTION

[Planned Feature]

• Replaces the Application Triage Map

• Roll back time to the point when applications first started experiencing symptoms to find the "patient zero” causing the current issue

• Easily share this information with the owner of that resource to resolve the issue quickly

• Attribute-based filtering allows for custom arrangement of data

• First step in an all-new user interface designed from the ground up

• Allows users to naturally and easily triage application problems, even with little-to-no knowledge of the application (or the APM Team Center itself)

• Team Center can be used to overlay real-world attributes onto complex application relationship data to minimize the "spaghetti" seen in typical application topology maps.

EASY.

• Inability to easily triage application problems without broad knowledge of the application

• Application relationship data depicted in dense complex inter-relationships in typical application topology maps


Cross system trace enhancementsProvide deeper transaction visibility of the trace within the MF environment across LPARs

BENEFIT

[Planned Feature]• User cannot see detail beyond first transaction link within the

MF

• Get a summary of the transaction on the MF not the detail

• Need ability for deeper triage

• Provide vision of cross system trace to include multiple level of transaction

• Provide vision of the cross system trace within the same LPAR

• Improved root cause analysis by allowing the user deeper visibility into the MF

• Improved root cause analysis enabling faster triage of MF issues

PAIN

SOLUTIONInsert screenshot here


Land LightlyReduce time to value of investment return for CE APM user base

BENEFIT

[Planned Feature]

• initial install for new customer is extremely hot in CPU requirements

• install is complex and includes many steps

• Elimination of dynamic install requirement

• Elimination of the user SVC during installation

• Streamline security implementation and integration with Enterprise Security solutions

• Rationalization of system parameters for lower overhead during data collection

• Integrated Management through easier management of security across the Enterprise

• Simplify Software Usage through the rationalization of system parameters

• Accelerate Software Deployment eliminating unnecessary steps in deploying the newest updates for SYSVIEW

PAIN

SOLUTION


Under Consideration

Improved Root cause analysis through additional application protocols – IMS Connect, …

Improved Root cause analysis through tighter transaction tracing from mobile through the mainframe – can we expand to support IBM Mobile Workload pricing for the mainframe

Enhanced productivity through interface standardization for install and configuration


CA Cross-Enterprise APM Technical Overview


CA APM / CA CE APM Terminology

CA Application Performance Management (APM)– Formerly “Wily” (name is no longer used)

– Introscope – the dashboards

– Investigator – the data view

– Management Modules – java files describing the thresholds, alerts, metric groupings, dashboard definitions

– CA APM Team Center – the new browser based interface with application mapping

CA Cross-Enterprise APM– Mainframe data extension for CA APM


CA APM / CA CE APM Architecture


Licensing

The CE APM license allows you to install the following components only for the use of CA CE APM– CA Cross-Enterprise APM agent

– CA SYSVIEW PM CE APM Option

– CA SYSVIEW for DB2 PM zOS (formerly Insight for DB2)

If you are already licensed for and using these products, those installations will be modified and utilized


CA Cross-Enterprise APM

CA Cross-Enterprise APM documentation is available as a bookshelf

Includes all of the documentation for CA CE APM– Install Guide

– User Guide – contains a listing and definition of all metrics available

Includes links to the other product bookshelves– Install Guides

Documentation

Solution Installation



Installed Components

Agent Configuration

CA SYSVIEW® Configuration

CA SYSVIEW® for DB2 Configuration

CA NetMaster® for TCP/IP Configuration– CA NetMaster is not part of the CA CE APM Product, but does connect to CA

APM

Installation



The CE APM Agent must be installed

– The release of the agent must be equal to or less than the release of the CA APM

Enterprise Manager

CA SYSVIEW is used for zOS, CICS, IMS, MQ, Datacom and TCP/IP metrics

CA SYSVIEW for DB2 zOS is used for DB2 metrics

– Formerly CA Insight for DB2 zOS

– CA SYSVIEW for DB2 connects directly to the agent via XNET, it does not need to

connect to CA SYSVIEW zOS for CA CE APM

Installed Components



Agent runs as a started task on zOS

– The proc uses the included STDENV member to locate the zfs file system

Run-time files are located in zfs and are java based

– File system must be permanently mounted

– During SYSVIEW installation a zfs file system is created and must be permanently mounted if using SYSVIEW for CE APM

Edit the STDENV member of the SAMPJCL library. This member

determines the paths for java, the CE APM agent runtime and SYSVIEW

runtime files in zfs

Agent Configuration



In the zfs directory there are two files you can edit– Introscope_Cross-Enterprise_APM.profile

Allows for generic agent property information

ppz.smf.socket.port=15029

– default for tracing socket, must match what is in SYSVIEW GROUPS

for CICSWILY and IMSWILY portlist

Agent Configuration



Introscope_Cross-Enterprise_APM.profile APM EM hostname and port

– introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=hostname

– introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT=5001

# Logging Configuration

– Dictates whether log activity is sent to zOS joblog (console) or zfs log file in the

CE APM agent directory

Various debugging features available

– #log4j.logger.IntroscopeAgent=VERBOSE#com.wily.util.feedback.Log4JSeverityLevel,logfile

Agent Configuration



In the zfs directory there are two files you can edit– Cross-Enterprise_APM_Dynamic.properties

This is the primary configuration file for CE APM

The EULA must be accepted or the agent will not start!

– CA.Cross-Enterprise.APM.I.Read.And.Accept.End.User.License.Agreement=Yes

SYSVIEW.update.interval=120

– How often CA SYSVIEW data is retrieved. Default is 120 seconds. 15-300 seconds

is possible.

– This has an impact on mainframe resource utilization

Agent Configuration



In the zfs directory there are two files you can edit– Cross-Enterprise_APM_Dynamic.properties

Basic CA SYSVIEW data collection is controlled here

– SYSVIEW.Metrics.collect=yes

Detailed SYSVIEW data collection is controlled here. You can determine whether to collect all, none, or specific portions of data.

– SYSVIEW.CICS.Regions.regex=*

– This has a direct impact on resource utilization

Agent Configuration



Cross-Enterprise_APM_Dynamic.properties Basic SYSVIEW for DB2 data collection is controlled here

– Insight.metrics.collect=yes

XNET and Passticket\user information

– Insight.passticket.support=yes

Agent Configuration



Only the CEAPM option needs defined in the INSTALL job

CICS, IMS, MQ, etc. are otherwise configured if data is to be

collected for those subsystems

CA SYSVIEW Configuration – CA CE APM Only



CA SYSVIEW PM r14.0 and higher requires CA Common Services CA Master (FMID

CEI0E10) to be present

Internal product licensing for CA CE APM allows certain CA SYSVIEW data collection and

commands to be utilized

The CA CE APM Installation Guide includes a section on “Useful CA SYSVIEW

Commands”

– This is a list of commands that are available to CA CE APM only customers that can be used to validate

installation, data collection and explain how to use GROUPS in CE APM

– MENU CEAPM is available in CA SYSVIEW r14.1 showing useful commands for CA CE APM only users



CA SYSVIEW MENU CEAPM


MENU CEAPM as Primary Menu



The CEAPM Installation Guide includes what CA SYSVIEW steps must be run

and which can be skipped for a CA CE APM only installation

– LOGGER files do not need to be created, except AUDIT if your installation requires it.

– The CICS and IMS LOGR Tasks are used for transaction tracing only, the logger files are

not required.

CICS data collection is configured as normal

HELP WILY0000 provides information on supported tracing methods and CICS

data collection GLUE information




Dashboards




Cross-Process Tracing– Details of the transaction

across all of the monitored components of the transaction path, distributed and mainframe.




Supported Cross-Process Tracing Methods

– MQ to CICS or IMS

– Web Services to CICS

– CTG to CICS

Distributed APM agent passes a GUID and trace request to the

CEAPM Agent.

– The CEAPM agent tells SYSVIEW to send the SMF details to the APM EM

for that transaction record.

– The APM EM correlates the transaction information using the GUID.




CA SYSVIEW for DB2 is installed as normal

Only specific DSQ Requests are needed for CE APM– This will limit the data retrieved to only what was needed for

CE APM

THRESHDS; DCL$A$D; DCL$E$P; DCL$Q$Z

DSQPARMS; DSQAPMSS; DSQAPMGS – if datasharing

History can be turned off as it is not used for CE APM

CA SYSVIEW for DB2 zOS – CA CE APM Only



CA SYSVIEW for DB2 connects to CE APM using XNET

XNET configuration is set in Cross-Enterprise_APM_Dynamic.properties

Hostname

Port

Uses PASS Tickets for authentication– Configuration set in Cross-Enterprise_APM_Dynamic.properties

– Can also hardcode a user id and password




Does not require CA SYSVIEW to be installed

Does not require integration with CA SYSVIEW be configured

Does not provide trace data to CA APM– DB2 data is included in the end to end application trace details when

CICS calls DB2.



CA Cross-Enterprise APMCA SYSVIEW for DB2 zOS – CA CE APM Only



CA NetMaster does not use the CE APM Agent

Integration to CA APM was developed independently using an APM EPAgent

Only TCPIP data is sent to CA APM

The EPAgent must run from a Windows/Unix machine– This is the “agent” that receives the data from CA NetMaster

– Default listening port is 8000

CA NetMaster TCPIP – CA CE APM Only Installation



Connectivity parameters are set from within CA NetMaster itself, not a .profile like other agents– /PARMS

– F APMEPAGENT

Set data collection on/off

Set IP Address of EPAgent

Set port for EPAgent (default 8000)

CA NetMaster TCPIP – CA CE APM Only Installation


CA Cross-Enterprise APMCA NetMaster TCPIP – CA CE APM Only Installation

Installation Validation


CA Cross-Enterprise APMInstallation Validation

It is recommended to start CA SYSVIEW and/or CA SYSVIEW for DB2 prior to starting the CA CE APM agent

Start the CA CE APM agent task– View the task log for messages during start-up

Once the CA CE APM starts, it will attempt a connection to the CA APM EM, CA SYSVIEW and CA SYSVIEW for DB2 (via XNET) to begin requesting data


CA Cross-Enterprise APMInstallation Validation

Using the CA APM Investigator you can verify if the CA CE APM agent

connected

– The CA CE APM agent will be visible in the Investigator tree of available agents

– You will see one agent per LPAR

– If there are only agent metrics available then the agent is connected to the EM but

not gathering metrics from CA SYSVIEW or CA SYSVIEW for DB2

Using the CA APM Investigator, look for zOS, CICS, IMS, MQ, TCPIP and CA

Datacom metrics, as applicable, under the CA Cross-Enterprise APM agent

entry in the tree


CA Cross-Enterprise APMInstallation Validation – CA SYSVIEW

Determine if the CA CE APM agent is connected to CA SYSVIEW– In CA SYSVIEW, type the USERS command. Look for the CA CE APM

user name (usually WILYZOS)

– If present, type LISTL for the LISTLOG of this user to see that it is issuing commands

– If not present

Look in XSUSERS

Look at the WILYZOS job log for any error messages

– Check ASADMIN

Ensure XSXI and XSXS are running


CA Cross-Enterprise APMInstallation Validation – CA SYSVIEW for DB2 (Insight)

Using the CA APM Investigator, look for DB2 metrics under the CA Cross-Enterprise APM agent entry in the tree

Determine if the CA CE APM agent is connected to CA SYSVIEW for DB2 (Insight)– There are no CA SYSVIEW for DB2 commands that validate

connectivity

– Review the CA CE APM agent task log for error messages

– Review the XNET task log for error messages

Common Errors and Resolutions



CA CE APM Agent will not start

CA SYSVIEW zfs directory structure no longer exists

Agent connects to CA APM EM but there’s no data

Agent not connected to EM

No trace data present in Investigator

Common Errors and Resolutions



This is extremely common. Besides basic zOS proc or security settings, always check the variable paths in STDENV. Manually verify the variable paths match up to were they are used in STDENV and the files in zfs are actually there

– JAVA_HOME

– SA_INSTALL

– SYSVIEW_PATH

This is the most common mistake made. Use split screen and “f”ind to validate the paths

Often results in an “XAPI not found” type error

CA CE APM Agent Will Not Start



Scenario: The CE APM agent is present in CA APM Investigator but there is no zOS or subsystem data showing

CA SYSVIEW– Agent will start and connect to the CA APM EM even if it cannot

connect to SYSVIEW (and vice versa)

– Check CA CE APM agent log for errors

– Check ASADMIN for XSXS and XSXI

– Check USERS or XSUSERS for CA CE APM user name

Default is WILYZOS

Agent Connects to CA CE APM but There’s No Data



If CA SYSVIEW was restarted and CA CE APM agent was not, this can sometimes throw them out of synch and agent will not attempt a reconnection to CA SYSVIEW. Shutdown all tasks, restart CA SYSVIEW before the agent

Agent Connects to CA CE APM but There’s No Data (continued)



Scenario: The CA CE APM agent is present in CA APM Investigator but there is no DB2 data showing

CA SYSVIEW for DB2 (Insight)– Check CE APM agent log for errors

– Check XNET log for errors

– Check .properties for PASSTICKET information

Often try a real user’s id and password who have access to validate connectivity. Often pass tickets are not configured correctly

Agent Connects to CA CE APM but There’s No Data



In most cases the agents connect to an CA APM EM Manager of Managers (MOM) which routes them to the most available server

Validate the CA APM EM hostname and port is correct for the MOM in theIntroscope_Cross-Enterprise_APM.profile

Default APM EM listening port is 5001

PING the hostname and\or IP address of the CA APM EM MOM from the LPAR in question

Agent Won’t Connect to EM or Disconnects



Ensure firewall is open on the span of ports for the MOM and individual EM’s. This is generally a range.– You only specify the MOM in the .profile, but must have

access to all ports of all EM’s

Agent Won’t Connect to EM or Disconnects



Transaction trace data only comes from CA SYSVIEW

While DB2 information is available in the trace, it comes from the CICS

data, not from CA SYSVIEW for DB2.

This can be a more difficult issue to debug as it spans platforms

Validate application path has CA APM agents installed in all places

Gaps prevent the GUID and trace indicator from passing along

– Validate PORTLISTs are setup in the CA SYSVIEW GROUPS command

– Validate the LOGGER (CICSLOGR and IMSLOGR) have matching PORTLIST entries

No Trace Data Present



– For CICS

validate a CORID is present in the CTRANLOG record

Validate in CCONFIG WILY-INTROSCOPE is YES to indicate a trace request

Look at CICSLOGR for TCPSEND bytes count to determine if records are being sent to the EM

No Trace Data Present

Understanding Transaction Tracing


CA APM \ CA CE APM Transaction Tracing

CA Cross-Enterprise APM has the ability to provide application details to CA APM in a transaction or cross-process trace

Current supported methods are:– Web Services to CICS

– CICS Transaction Gateway (CTG) to CICS

– MQ to CICS or IMS

– We need your input! IMS Connect? Sockets?

Supported Tracing Methods



Transaction tracing is invoked through the CA APM interface and must start with a non-mainframe agent

We utilize the message header or application payload to complete the trace request– If you are using a custom middleware component we will need to

validate that it can pass this trace request

The trace will follow the CICS transaction path on the same LPAR or the first IMS transaction called

Supported Tracing Methods



The CA APM transaction trace will present the CICS transaction path on the same LPAR, showing the first program executed and relevant timing information.

Trace Details

CA Product Integrations


CA Performance Management Portfolio VisionUnified Performance Management for z Systems Planned

Available

Distributed/Mainframe Operations Analysts

MF Systems/Network Analysts

Distributed Application Analysts

Network Analysts

CA NetMaster®

Databases

Network

CICS & MQ

z/VM, KVM & Linux on z

CA UIMCA UIM for z Systems

CA Cross Enterprise APM

Integrated Mainframe Management Console

CA APM

System Analysts

CA SYSVIEW®

Storage

Capacity Planner

CA MICS®

Storage Administrator

CA Vantage™


CA CE APM Product Integrations

Alerts created from CA SYSVIEW are viewable in CA APM on a typeview page– zOS Alerts

– CICS Alerts

– IMS Alerts

– MQ Alerts

Using CA SYSVIEW with CA Cross-Enterprise APM


CA CE APM Product IntegrationsCA SYSVIEW Alerts


CA Cross-Enterprise APMPart of the bigger picture

CA SYSVIEW Event Capture

CA Mainframe Application

Tuner

CA OPS/MVS


Q & A


Recommended Sessions

SESSION # TITLE DATE/TIME

MFT56SInfrastructure Performance Management: Flexibility

Combining Breadth, Depth and Speed WED 11/18 at 3:00 pm

MFT15TNew Mainframe IT Analytics: Actionable Insight into

Root Cause Analysis of Performance IssuesWED 11/18 at 3:45 pm

MFT08SAnalyst Expert View: Mainframe in the Application Economy - Gary Barnett Chief Analyst Ovum

THUR 11/19 at 10:30 am


Follow Conversations At…

DEMO STATION TECH TALKS

Demo Station: MNFSE009

Mobile to Mainframe App Performance

Tech Talk MFT14T: WED 11/18 at 2:20 pmGo Deep & Wide: No It’s Not a Football, It’s End-to-End Performance Management

Tech Talk MFT15T: WED 11/18 at 3:45 pmNew Mainframe IT Analytics:Actionable Insight into Root Cause Analysis of Performance Issues


Because those who build the apps will own the future ...

And those apps continue to rely on the MAINFRAME


For More Information

To learn more, please visit:

http://cainc.to/Nv2VOe

CA World ’15

http://cainc.to/Nv2VOe

http://bit.ly/1wbjjqX